Architecture¶
How thomelab is built and operated. For what it runs, see Overview.
Clusters¶
| Cluster | Role | Cluster DNS domain | Status |
|---|---|---|---|
thomelab |
Primary cluster, 4 nodes at home. Most self-hosted apps run here, plus OpenBao and Argo Workflows. | cluster.local |
Active |
public |
Internet-facing edge, a single VPS node: Traefik, Netbird (management, signal, reverse proxy), Keycloak SSO, a small number of public apps. | public.local |
Active |
Both run Talos Linux. Their machine config, upgrades and bootstrap are managed with OpenTofu in
thomelab-bootstrap-infra.
Repositories¶
Each repository holds declarative config. ArgoCD applies Kubernetes manifests; Ansible and
OpenTofu apply everything below Kubernetes. No repository takes direct commits to main — every
change is a GitLab issue, a branch named after it, and a merge request.
| Repository | Purpose | Visibility |
|---|---|---|
thomelab-infra |
Kubernetes manifests (Kustomize + ArgoCD) for both clusters. | Public |
thomelab-ops |
Ansible and OpenTofu: OS maintenance, cluster lifecycle, resource management (Keycloak realms, Netbird groups/policies, DNS records, S3 buckets). | Private |
thomelab-bootstrap-infra |
Bootstraps the layer below Kubernetes, mainly the Talos clusters today | Private |
thomelab-site |
Source for thomelab.net: homepage and this docs site (public and private builds). | Private |
tofu-modules |
OpenTofu modules shared by the ops repos (Netbird resources, Keycloak OIDC clients, a GitHub App), versioned with tags and tested with tofu test. |
Public |
chezmoi |
Dotfiles and per-machine setup (laptop accounts, phone VMs, both devcontainers), with credentials rendered from OpenBao at apply time. | Public |
GitOps flow¶
Each cluster's Kubernetes state is one ArgoCD app-of-apps in thomelab-infra, in two layers:
cluster-tools/— the tools a cluster needs to run anything else: ArgoCD, Traefik, cert-manager, Sealed Secrets / External Secrets + OpenBao, storage (Longhorn, NFS to the NAS,local-pathonpublic), Velero, Netbird, NetworkPolicy enforcement, the Intel GPU device plugin.applications/— the self-hosted apps, each asbase/(the ArgoCD Application) plusoverlays/<cluster>/(cluster-specific patches and secrets).
ArgoCD watches main and auto-syncs. To test a change before merging, a temporary ArgoCD
Application points at the feature branch, is validated, then deleted. Manifests are never applied
from a branch directly.
Networking and exposure¶
Public traffic targets thomelab.net.
Two systems route it:
- Traefik, on the
publiccluster, for apps with an explicitIngressRoute: ntfy, Vaultwarden, Gatus, Keycloak, the homepage/docs site, ArgoCD's dashboards. - Netbird's reverse proxy, for everything else — apps registered as
netbird_reverse_proxy_serviceentries inthomelab-ops. Traefik forwards unrecognized hosts to Netbird over a TLS passthrough route; Netbird terminates TLS itself (ACME,tls-alpn-01) per registered subdomain.
A subdomain registered in neither place fails the TLS handshake rather than redirecting anywhere. Fixing this requires a wildcard certificate (DNS-01, not currently configured) and would require listing every Netbird domain in Traefik as well; left unfixed as a known limitation.
Netbird's mesh is deny-by-default: a peer only reaches what an explicit policy grants its group,
and routed networks (the thomelab Service range, the home LAN) carry access-control groups so a
route alone doesn't open everything behind it. Admin hosts are reached with Netbird SSH, granted
by policy and authenticated through Keycloak.
Inside thomelab, NetworkPolicies are enforced by kube-network-policies (Flannel, the CNI,
has no enforcement of its own). It fails open: if the enforcer is down, traffic flows. Charts'
own policies (ArgoCD, Longhorn) are back on under it, and app-specific policies are being added
one app at a time.
SSO is Keycloak, via OIDC (for applications supporting it), oauth2-proxy (for apps on the public cluster without oidc support), and behind Netbird's authentication (for apps on thomelab that don't support OIDC).
Cluster and admin access¶
Administrative access (kubectl, talosctl) is kept separate from the app-traffic path above.
public's host firewall permits only port 443 by default; the Kubernetes and Talos API ports are
opened only temporarily, by hand, when genuinely needed. Day to day, both clusters' API servers
are reached through Tailscale's Kubernetes-operator API-server proxy (the *-tailscale
kubeconfig contexts, the default) rather than a direct connection. The plain public /
thomelab contexts connect directly; for public that only works while its API port is open.
Netbird and Tailscale overlap in the mesh-networking sense but do different jobs here: Netbird
carries the app-traffic reverse proxy, mesh access and SSH described above; Tailscale exists only
for API-server access to the clusters, as a break-glass path that doesn't depend on Netbird
(which itself runs on public). Apps are never exposed over Tailscale.
Secrets and encryption¶
- sops + age encrypts the sensitive files committed in
thomelab-bootstrap-infraandthomelab-ops(OpenTofu backend/state secrets, provider credentials), loaded through direnv. Both repos retired git-crypt for it. - OpenBao is where OpenTofu reads and writes runtime secrets (OIDC client secrets, tokens,
kubeconfigs). Humans authenticate with
bao login; there are no long-lived stored tokens for it. - ansible-vault additionally protects
thomelab-ops/inventory/hosts.yaml; the vault password is kept outside git, in~/.vaultpass. - In Kubernetes: Sealed Secrets is the mechanism for committing encrypted manifests on the
publiccluster, and for apps onthomelabthat can't rely on OpenBao. The default mechanism forthomelabis External Secrets + OpenBao.
Automation¶
Scheduled and triggered jobs run as Argo Workflows on thomelab: hourly health checks of
both clusters and of every ArgoCD application, a daily Velero backup check, an OpenBao seal check
that also auto-unseals, a weekly Talos client-certificate renewal, and a daily digest of open
merge requests. They report through ntfy, and only fail on their own errors, never on what they
find.
Renovate runs on thomelab too, watching all repositories and opening merge requests for
dependency and container digest bumps. In the repositories with CI, patch updates (after a
three-day wait) and digest refreshes merge themselves on weekday working hours once the
pipeline is green. Everything else, and anything in the OpenTofu/Ansible repositories, waits for
a human merge.
Backups¶
Backups go to Garage (S3-compatible) on the NAS.
thomelab: Velero runs daily and keeps 14 days. It backs up the Kubernetes objects and, through CSI snapshots, the Longhorn volumes (stored as Longhorn backups). Postgres is skipped there and covered by CNPG barman instead (WAL archiving plus base backups). Volumes on the NAS (NFS) are skipped: their data already lives on the NAS.public: Velero covers the objects only. Itslocal-pathvolumes are copied by a nightly rclone CronJob per app.- OpenBao takes its own snapshots to S3.
Further detail¶
Each repository's own README.md covers its specific commands and conventions.