A lab built to be rebuilt.

Why build it this way?

I wanted a place where I could learn by actually operating things: containers, networking, secure remote access, monitoring, deployment workflows, public tools, and eventually personal data services. But I did not want the result to become a one-off machine full of hand-edited configuration that only made sense because I remembered how I set it up.

So the HomeLab is being treated as a system that should be reproducible. The physical computer is intentionally not the identity of the project. It is simply the current place where the system runs.

The design principles

The architecture has gradually settled around a few rules:

• Git is the source of truth for configuration, deployment scripts, web content, Compose definitions, and documentation.
• Runtime data is separate from application definitions, so replacing a container does not mean replacing its data.
• Administrative access stays private rather than exposing SSH, databases, or management interfaces directly to the Internet.
• Public web services enter through an outbound tunnel, avoiding inbound router port forwarding.
• Failure should be visible, with local monitoring, push alerts, and an external heartbeat for complete outages.
• Recovery steps belong in documentation, not in chat history or memory.

What the architecture looks like

At a high level, the system separates source, deployment, persistent data, private administration, and public ingress.

                   GitHub
                     │
                  git pull
                     │
                     ▼
          version-controlled source
                     │
               deploy scripts
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
   Compose definitions     persistent data
   / service config        / application state
          │                     │
          └──────────┬──────────┘
                     ▼
                 containers
                     │
        ┌────────────┴────────────┐
        ▼                         ▼
 private administration      public web apps
   secure overlay              loopback only
        │                         │
        ▼                         ▼
 trusted devices          outbound web tunnel
                                  │
                                  ▼
                              Internet

That separation matters more to me than the exact software choices. The individual tools can change over time without changing the underlying model.

From Git commit to running service

One of the most useful improvements has been making deployment boring. For a typical static service, the repository contains the website source, its container definition, and a deployment script.

A normal change now looks like:

edit / generate change
        ↓
commit to GitHub
        ↓
git pull on HomeLab
        ↓
run service deploy script
        ↓
validate local origin
        ↓
service is live

The deployment scripts also perform safety checks. For example, a service that depends on persistent storage should refuse to deploy if that storage is not mounted. That prevents an apparently successful deployment from quietly writing data into the wrong filesystem.

Public services without public administration

I treat browser-facing applications and administrative access as two different problems.

Public and selected private web applications are published through Cloudflare Tunnel. Their container ports bind only to the local host, and the tunnel makes the intentional hostname available externally without opening inbound application ports on the home router.

Administrative access uses Tailscale plus ordinary SSH. That gives trusted devices private connectivity for server administration and remote development without making SSH publicly reachable.

Knowing when something breaks

A HomeLab is much more useful when I do not have to wonder whether it is alive. Monitoring now exists at several layers rather than depending on one application to report on itself.

• Beszel provides private system and service monitoring.
• Host-level watchers check critical storage and expected containers.
• ntfy delivers immediate push notifications for failure and recovery events.
• Healthchecks.io acts as an external dead-man switch so a complete HomeLab outage can still generate an alert.

The important part is the layering: if the container platform itself stops, the storage and heartbeat checks are not dependent on that same container platform to tell me about it.

Learning from failures

The lab has already produced the kind of failure that proves why this architecture matters. Persistent storage has unexpectedly disappeared from the operating system during active writes. That exposed an important weakness: containers could remain running even though the storage they depended on was no longer available.

The response was not just to repair the filesystem. The system changed afterward: container startup is now tied to the persistent-storage mount, storage loss stops the container service, and a host-level watcher generates failure and recovery alerts.

That is exactly what I want this environment to be for: break something, understand why it failed, then improve the system so the same class of failure is handled better next time.

Portable by design

I intentionally leave the specific server hardware out of the public identity of the project. Hardware matters operationally, but it should not define the architecture.

The long-term test is whether I could install a supported Linux host, attach suitable persistent storage, clone the repository, restore required secrets and data, and rebuild the services without needing a collection of undocumented one-off steps.

I am not claiming that process is fully automatic yet. The point is that every change should move the environment closer to that standard.