Skip to content

Architecture

How thomelab is built and operated. For what it runs, see Overview.

Clusters

Cluster Role Cluster DNS domain Status
thomelab Primary cluster, 4 nodes at home. Most self-hosted apps run here, plus OpenBao and Argo Workflows. cluster.local Active
public Internet-facing edge, a single VPS node: Traefik, Netbird (management, signal, reverse proxy), Keycloak SSO, a small number of public apps. public.local Active

Both run Talos Linux. Their machine config, upgrades and bootstrap are managed with OpenTofu in thomelab-bootstrap-infra.

Repositories

Each repository holds declarative config. ArgoCD applies Kubernetes manifests; Ansible and OpenTofu apply everything below Kubernetes. No repository takes direct commits to main — every change is a GitLab issue, a branch named after it, and a merge request.

Repository Purpose Visibility
thomelab-infra Kubernetes manifests (Kustomize + ArgoCD) for both clusters. Public
thomelab-ops Ansible and OpenTofu: OS maintenance, cluster lifecycle, resource management (Keycloak realms, Netbird groups/policies, DNS records, S3 buckets). Private
thomelab-bootstrap-infra Bootstraps the layer below Kubernetes, mainly the Talos clusters today Private
thomelab-site Source for thomelab.net: homepage and this docs site (public and private builds). Private
tofu-modules OpenTofu modules shared by the ops repos (Netbird resources, Keycloak OIDC clients, a GitHub App), versioned with tags and tested with tofu test. Public
chezmoi Dotfiles and per-machine setup (laptop accounts, phone VMs, both devcontainers), with credentials rendered from OpenBao at apply time. Public

GitOps flow

Each cluster's Kubernetes state is one ArgoCD app-of-apps in thomelab-infra, in two layers:

  • cluster-tools/ — the tools a cluster needs to run anything else: ArgoCD, Traefik, cert-manager, Sealed Secrets / External Secrets + OpenBao, storage (Longhorn, NFS to the NAS, local-path on public), Velero, Netbird, NetworkPolicy enforcement, the Intel GPU device plugin.
  • applications/ — the self-hosted apps, each as base/ (the ArgoCD Application) plus overlays/<cluster>/ (cluster-specific patches and secrets).

ArgoCD watches main and auto-syncs. To test a change before merging, a temporary ArgoCD Application points at the feature branch, is validated, then deleted. Manifests are never applied from a branch directly.

Networking and exposure

Public traffic targets thomelab.net. Two systems route it:

  • Traefik, on the public cluster, for apps with an explicit IngressRoute: ntfy, Vaultwarden, Gatus, Keycloak, the homepage/docs site, ArgoCD's dashboards.
  • Netbird's reverse proxy, for everything else — apps registered as netbird_reverse_proxy_service entries in thomelab-ops. Traefik forwards unrecognized hosts to Netbird over a TLS passthrough route; Netbird terminates TLS itself (ACME, tls-alpn-01) per registered subdomain.

A subdomain registered in neither place fails the TLS handshake rather than redirecting anywhere. Fixing this requires a wildcard certificate (DNS-01, not currently configured) and would require listing every Netbird domain in Traefik as well; left unfixed as a known limitation.

Netbird's mesh is deny-by-default: a peer only reaches what an explicit policy grants its group, and routed networks (the thomelab Service range, the home LAN) carry access-control groups so a route alone doesn't open everything behind it. Admin hosts are reached with Netbird SSH, granted by policy and authenticated through Keycloak.

Inside thomelab, NetworkPolicies are enforced by kube-network-policies (Flannel, the CNI, has no enforcement of its own). It fails open: if the enforcer is down, traffic flows. Charts' own policies (ArgoCD, Longhorn) are back on under it, and app-specific policies are being added one app at a time.

SSO is Keycloak, via OIDC (for applications supporting it), oauth2-proxy (for apps on the public cluster without oidc support), and behind Netbird's authentication (for apps on thomelab that don't support OIDC).

Cluster and admin access

Administrative access (kubectl, talosctl) is kept separate from the app-traffic path above. public's host firewall permits only port 443 by default; the Kubernetes and Talos API ports are opened only temporarily, by hand, when genuinely needed. Day to day, both clusters' API servers are reached through Tailscale's Kubernetes-operator API-server proxy (the *-tailscale kubeconfig contexts, the default) rather than a direct connection. The plain public / thomelab contexts connect directly; for public that only works while its API port is open.

Netbird and Tailscale overlap in the mesh-networking sense but do different jobs here: Netbird carries the app-traffic reverse proxy, mesh access and SSH described above; Tailscale exists only for API-server access to the clusters, as a break-glass path that doesn't depend on Netbird (which itself runs on public). Apps are never exposed over Tailscale.

Secrets and encryption

  • sops + age encrypts the sensitive files committed in thomelab-bootstrap-infra and thomelab-ops (OpenTofu backend/state secrets, provider credentials), loaded through direnv. Both repos retired git-crypt for it.
  • OpenBao is where OpenTofu reads and writes runtime secrets (OIDC client secrets, tokens, kubeconfigs). Humans authenticate with bao login; there are no long-lived stored tokens for it.
  • ansible-vault additionally protects thomelab-ops/inventory/hosts.yaml; the vault password is kept outside git, in ~/.vaultpass.
  • In Kubernetes: Sealed Secrets is the mechanism for committing encrypted manifests on the public cluster, and for apps on thomelab that can't rely on OpenBao. The default mechanism for thomelab is External Secrets + OpenBao.

Automation

Scheduled and triggered jobs run as Argo Workflows on thomelab: hourly health checks of both clusters and of every ArgoCD application, a daily Velero backup check, an OpenBao seal check that also auto-unseals, a weekly Talos client-certificate renewal, and a daily digest of open merge requests. They report through ntfy, and only fail on their own errors, never on what they find.

Renovate runs on thomelab too, watching all repositories and opening merge requests for dependency and container digest bumps. In the repositories with CI, patch updates (after a three-day wait) and digest refreshes merge themselves on weekday working hours once the pipeline is green. Everything else, and anything in the OpenTofu/Ansible repositories, waits for a human merge.

Backups

Backups go to Garage (S3-compatible) on the NAS.

  • thomelab: Velero runs daily and keeps 14 days. It backs up the Kubernetes objects and, through CSI snapshots, the Longhorn volumes (stored as Longhorn backups). Postgres is skipped there and covered by CNPG barman instead (WAL archiving plus base backups). Volumes on the NAS (NFS) are skipped: their data already lives on the NAS.
  • public: Velero covers the objects only. Its local-path volumes are copied by a nightly rclone CronJob per app.
  • OpenBao takes its own snapshots to S3.

Further detail

Each repository's own README.md covers its specific commands and conventions.