<SYSTEM>This is the abridged developer documentation for homelab-iac</SYSTEM>

# homelab-iac

> Proxmox VMs and LXC containers with OpenTofu, a K3s cluster managed by GitOps, and Cloudflare for DNS, certificates and state.

The goal of this homelab is a self-sustaining, reproducible and secure environment to automate the full infrastructure lifecycle, experiment with DevOps and security tooling, host personal services, and practice architecture, automation and observability. This wiki follows the [Diátaxis](https://diataxis.fr/) framework: Tutorials Learning-oriented. Start from a fresh Proxmox node and end with a VM managed by OpenTofu. [Getting started →](/tutorials/getting-started/) How-to guides Task-oriented recipes: build templates, deploy VMs and containers, install K3s, expose apps on Hyperion. [Browse guides →](/how-to/create-rocky-template/) Reference Facts to look up: repository layout, Proxmox permissions, environment variables, components, workflows and Terraform inputs. [Open reference →](/reference/repository-layout/) Explanation Why things are built the way they are: architecture, the Hyperion and Cronos clusters, remote state, and how these docs are delivered. [Read explanations →](/explanation/architecture/)

# Architecture

> How the layers of the homelab fit together, from Proxmox up to the apps.

The homelab is described declaratively in this repository and built in layers. Each layer is owned by one tool, and each tool hands off to the next. 
```text
┌──────────────────────────────────────────────────────────────────┐
│ Apps on Hyperion            ArgoCD ← kubernetes/applications/    │
├──────────────────────────────────────────────────────────────────┤
│ K3s platform                OpenTofu ← terraform/.../hyperion    │
│ Cilium · Gateway API · cert-manager · ArgoCD                     │
├──────────────────────────────────────────────────────────────────┤
│ Guests                      OpenTofu ← terraform/proxmox/{vm,lxc}│
│ VMs (K3s node, hermes-agent) · LXC (Swarm, runner, DNS, auth…)   │
├──────────────────────────────────────────────────────────────────┤
│ Proxmox VE                  manual bootstrap (user, role, token) │
└──────────────────────────────────────────────────────────────────┘
        state: Cloudflare R2        DNS + certificates: Cloudflare
```
 ## Layers [Section titled “Layers”](#layers) **Proxmox VE.** The only manual layer: create the `terraform@pve` user and token, enable snippets, and build the VM template. Everything above is code. **Guests.** OpenTofu creates VMs from a Rocky Linux cloud-init template and LXC containers from Debian templates. Each guest is an entry in a `locals.tf` map; shared settings live in `defaults`, so adding a machine is a few lines. **K3s platform.** On the Hyperion VM, K3s runs without its bundled networking. OpenTofu installs the pieces that must exist before GitOps can work: the CNI, the certificate issuer, the shared Gateway and ArgoCD itself. **Apps.** ArgoCD takes over from there. The root app-of-apps watches `kubernetes/applications/argocd`, so adding an app is a pull request. Some services still run outside Kubernetes, as Docker Compose on LXC hosts (`lxc-auth`, `lxc-monitoring`) or as Swarm stacks on `swarm-01`/`swarm-02`. ## Why split OpenTofu and ArgoCD [Section titled “Why split OpenTofu and ArgoCD”](#why-split-opentofu-and-argocd) OpenTofu is good at creating things in order with explicit dependencies: Cilium has to run before anything can get an IP, and cert-manager has to run before the Gateway can get its certificate. ArgoCD is good at keeping many independent apps in sync with Git. The split puts the bootstrap order in OpenTofu and the day-to-day changes in ArgoCD. ## Network [Section titled “Network”](#network) | Range | Use | | -------------------- | ------------------------------------------- | | `192.168.10.0/24` | Server network: Proxmox guests, K3s nodes | | `192.168.10.1` | Gateway | | `192.168.10.150` | Pi-hole (internal DNS) | | `*.home.cervant.net` | Internal services behind the Cilium Gateway | ## CI [Section titled “CI”](#ci) Changes reach the infrastructure through GitHub Actions. Jobs that need the LAN run on a self-hosted runner in an LXC container, inside a container image with the tooling. Secrets come from Bitwarden Secrets Manager. See [CI workflows](/reference/ci-workflows/).

# Clusters: Hyperion and Cronos

> Why there are two Kubernetes clusters with different stacks, and why Hyperion replaces the K3s defaults.

The homelab plans two K3s clusters with deliberately different stacks. Hyperion is for services used every day; Cronos is for learning alternatives without risking them. ## Hyperion: the stable cluster [Section titled “Hyperion: the stable cluster”](#hyperion-the-stable-cluster) Hyperion favors stability, security and observability. Its stack: | Area | Choice | | ------------- | ----------------------------------- | | CNI and mesh | Cilium | | Gateway API | Cilium’s built-in controller | | GitOps | ArgoCD | | Certificates | cert-manager + Cloudflare DNS-01 | | LoadBalancer | K3s ServiceLB (node IP) | | Secrets | Infisical + Kubernetes operator | | Policy engine | Kyverno | | Storage | Rook + Ceph (planned) | | Observability | Prometheus, Grafana, Loki (planned) | ### Why K3s runs without Traefik, Flannel and kube-proxy [Section titled “Why K3s runs without Traefik, Flannel and kube-proxy”](#why-k3s-runs-without-traefik-flannel-and-kube-proxy) K3s ships with Flannel, kube-proxy, a network policy controller and Traefik. Cilium does all of those jobs with eBPF, so Hyperion turns them off (`--flannel-backend=none --disable-kube-proxy --disable-network-policy --disable traefik`). Running both would mean two components fighting over the same packets. One consequence: a fresh node has no networking until Cilium is installed. That is why Cilium is the first thing the Hyperion OpenTofu stack installs, followed by a 120-second wait. ### What the stack expects from K3s [Section titled “What the stack expects from K3s”](#what-the-stack-expects-from-k3s) The Hyperion stack is validated only on K3s (`v1.37.1+k3s1` on Rocky Linux 10.2). It assumes K3s in these places, and each would need changes on another distribution: * **Gateway API CRDs.** K3s installs them through its bundled `gateway-api-crd` chart. The bootstrap kustomization (`crds/`) does not include them, so on another distribution nothing would install them. * **API server address.** Cilium uses `k8sServiceHost=127.0.0.1` and `k8sServicePort=6443`, which matches a K3s server node where the API server listens locally. * **Install flags.** Flannel, kube-proxy, the network policy controller and Traefik are turned off with K3s flags (see [Install K3s on Hyperion](/how-to/install-k3s-hyperion/)). * **LoadBalancer IP.** The Gateway’s `LoadBalancer` Service gets the node IP from the K3s ServiceLB (`svclb-*` pods). Cilium’s LB IPAM is not configured. * **Service CIDR.** Pods reach the API at `10.43.0.1`, the first address of the K3s default service CIDR. ### One Gateway, one wildcard certificate [Section titled “One Gateway, one wildcard certificate”](#one-gateway-one-wildcard-certificate) All apps share a single Gateway in the `cilium-gateway` namespace, with one wildcard certificate for `*.home.cervant.net`. Apps only add an `HTTPRoute`. This keeps certificate issuance to one DNS-01 challenge, works for internal-only hostnames that aren’t reachable from the internet, and keeps per-app manifests small. ## Cronos: the experimental cluster [Section titled “Cronos: the experimental cluster”](#cronos-the-experimental-cluster) Cronos is planned. It uses the alternative for each area, so both approaches can be compared on similar workloads: | Area | Choice | | ------------- | -------------------------------------- | | CNI | Flannel (K3s default) | | Service mesh | Linkerd | | Gateway API | NGINX Gateway Fabric | | GitOps | FluxCD | | Certificates | cert-manager + Cloudflare DNS-01 | | LoadBalancer | kube-vip | | Secrets | Vault + External Secrets Operator | | Policy engine | OPA Gatekeeper | | Storage | Longhorn | | Observability | Prometheus, Grafana, Loki (via FluxCD) | ## Roadmap [Section titled “Roadmap”](#roadmap) * Deploy Rook + Ceph on Hyperion. * Build Cronos with FluxCD and Linkerd. * Add observability and dashboards per cluster.

# Docs delivery

> Why the wiki is built once and served from both Cloudflare and Hyperion, and how agents consume it.

The wiki is a static site built with Starlight. One CI build produces one `dist/` directory, which is published to two places: | URL | Where | How | | ------------------------------- | -------------------------------------- | ------------------------------- | | `https://docs.cervant.net` | Cloudflare Workers, static assets only | `wrangler deploy` | | `https://docs.home.cervant.net` | Hyperion, nginx | Image on GHCR, synced by ArgoCD | ## Why two destinations [Section titled “Why two destinations”](#why-two-destinations) The public copy is reachable from anywhere and doesn’t depend on the homelab being up. That matters most when something is broken and the docs are needed to fix it. The internal copy is reachable when the internet link is down, and it is the one internal tools and agents (such as Hermes on `hermes-agent`) can use without leaving the LAN. The two are independent. There is no failover logic between them: each URL is served from one place. ## Why one build [Section titled “Why one build”](#why-one-build) Both copies come from the same artifact, so they can’t drift. The image doesn’t build the site; it only copies `dist/` into `nginx-unprivileged`. The `site` setting in `astro.config.mjs` is the public URL, so canonical links, the sitemap and `llms.txt` point to `docs.cervant.net` in both copies. ## How the internal copy is rolled out [Section titled “How the internal copy is rolled out”](#how-the-internal-copy-is-rolled-out) CI pushes `ghcr.io/brsalcedom/homelab-docs:latest` (plus a `sha-<commit>` tag for traceability) and stops there. It has no access to the cluster and never pushes to `main`. [ArgoCD Image Updater](https://argocd-image-updater.readthedocs.io) does the rest. The `ImageUpdater` resource in `kubernetes/applications/argocd/homelab-docs/manifests/imageupdater.yaml` follows the `:latest` tag with the `digest` strategy. Every two minutes it checks GHCR, and when the digest changes it sets a Kustomize image override pinned to that digest on the `homelab-docs` Application. ArgoCD renders the new image and rolls out the Deployment. This is the `argocd` write-back method: the override lives on the Application object in the cluster, not in Git. Two consequences: * Git doesn’t record which digest is running. The Application in ArgoCD does, and the `sha-` tag in GHCR maps it back to a commit. * The root app-of-apps, which manages the `homelab-docs` Application from Git, must not undo the override. `argocd.tf` sets `ignoreDifferences` on `/spec/source/kustomize/images` for Applications, with `RespectIgnoreDifferences=true`. The app reuses the platform: an `HTTPRoute` on the shared `cilium-gateway`, covered by the existing `*.home.cervant.net` wildcard certificate from cert-manager. ## Docs for agents [Section titled “Docs for agents”](#docs-for-agents) The [`starlight-llms-txt`](https://github.com/delucis/starlight-llms-txt) plugin generates: | File | Contents | | ----------------- | --------------------------------------------------------------- | | `/llms.txt` | Index of the docs, following [llmstxt.org](https://llmstxt.org) | | `/llms-full.txt` | All pages as one Markdown file | | `/llms-small.txt` | A condensed version for smaller context windows | An agent can load `llms-full.txt` to get the whole wiki in one request. ## Reference generated from code [Section titled “Reference generated from code”](#reference-generated-from-code) The reference pages for each Terraform configuration contain a section generated by [terraform-docs](https://terraform-docs.io). CI injects it before `astro build`, so the published tables always match `main` without generated content being committed. ## Structure [Section titled “Structure”](#structure) Pages follow [Diátaxis](https://diataxis.fr/): tutorials teach, how-to guides solve one task, reference lists facts, and explanation (like this page) gives the reasons.

# Remote state on Cloudflare R2

> Why Terraform state lives in an R2 bucket and how the S3 backend is configured for it.

Every configuration under `terraform/` stores its state in the `homelab-tfstate` bucket on Cloudflare R2, each under its own key. ## Why remote state [Section titled “Why remote state”](#why-remote-state) The same configuration is applied from a workstation and from CI on the self-hosted runner. Local state would mean the two disagree about what exists. A shared bucket gives one source of truth and keeps state (which can include secrets) out of the repository. ## Why R2 [Section titled “Why R2”](#why-r2) R2 is S3-compatible, so OpenTofu’s built-in `s3` backend works with it. It has no egress fees, and it sits with the rest of the Cloudflare setup (DNS, certificates, these docs). It also lives outside the homelab, so the state survives if the Proxmox host is lost. ## How the S3 backend is configured [Section titled “How the S3 backend is configured”](#how-the-s3-backend-is-configured) R2 is not AWS, so the backend turns off the checks that only make sense against AWS: 
```hcl
terraform {
  backend "s3" {
    endpoint                    = "https://<account-id>.r2.cloudflarestorage.com"
    bucket                      = "homelab-tfstate"
    key                         = "proxmox/vm/terraform.tfstate"
    region                      = "auto"
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_s3_checksum            = true
    use_path_style              = true
  }
}
```
 Credentials come from `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, set to an R2 API token’s key pair. ## The bucket itself [Section titled “The bucket itself”](#the-bucket-itself) The bucket is created by the `cloudflare/` configuration. That configuration can’t store its state in the bucket it creates, so it keeps local state. ## Locking [Section titled “Locking”](#locking) The backend has no locking configured. Concurrent applies against the same key are avoided by running applies through CI, one workflow per configuration.

# Apply the Hyperion stack

> Install Cilium, cert-manager, the shared Gateway and ArgoCD on Hyperion with OpenTofu.

`terraform/proxmox/infraestructure/hyperion` installs the platform layer of the cluster: Cilium, cert-manager with a Cloudflare `ClusterIssuer`, the shared Gateway with a wildcard certificate, and ArgoCD with the `homelab` project and the root app-of-apps. Validated on K3s only This stack is tested only on K3s. On another distribution, see [What the stack expects from K3s](/explanation/clusters/#what-the-stack-expects-from-k3s) first. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * [K3s is installed](/how-to/install-k3s-hyperion/), including the firewalld rules for the pod network and for `80/tcp` and `443/tcp`. * The variables in [Environment variables: Hyperion](/reference/environment-variables/#hyperion) and the R2 credentials are set. ## Apply [Section titled “Apply”](#apply) Through CI: merge changes under `terraform/proxmox/infraestructure/hyperion` to `main`. The [`Tofu :: k8s hyperion`](/reference/ci-workflows/) workflow runs the bootstrap task, then plans and applies. From your workstation, run the same two steps. The bootstrap task installs the cert-manager and ArgoCD CRDs and creates the `cloudflare-api-token` and `infisical-service-token` secrets. It must run first, because the plan needs those CRDs to exist. 
```bash
cd terraform/proxmox/infraestructure/hyperion
export GITHUB_WORKSPACE="$(git rev-parse --show-toplevel)"  # the task resolves paths from it
task --taskfile "$GITHUB_WORKSPACE/.github/tasks/hyperion-bootstrap.yml"
tofu init
tofu apply
```
 The apply waits 120 seconds after Cilium is installed so the CNI is ready before the other resources. The Gateway API CRDs are already there, because K3s installs them. ## Check the result [Section titled “Check the result”](#check-the-result) 
```bash
kubectl get pods -A
kubectl get gateway -n cilium-gateway
kubectl get certificate -n cilium-gateway
```
 `wildcard-certificate` should be `Ready`, and ArgoCD should answer at `https://argocd.home.cervant.net`. ## Change chart versions [Section titled “Change chart versions”](#change-chart-versions) Chart versions are pinned in `locals.tf` (`cilium_chart_version`, `argocd_chart_version`, `cert_manager_chart_version`). Renovate opens pull requests for them. See [Hyperion components](/reference/hyperion-components/) and [Terraform: Hyperion](/reference/terraform-hyperion/).

# Create the Rocky Linux template

> Build the Rocky Linux cloud-init template that the VM configuration clones from.

The VM configuration in `terraform/proxmox/vm` clones every machine from a cloud-init template. The template ID is set in `locals.tf` (`defaults.clone_vm_id`, currently `8000`). ## 1. Download the cloud image [Section titled “1. Download the cloud image”](#1-download-the-cloud-image) Pick a Rocky Linux GenericCloud image: * Rocky 10: `https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2` * Rocky 9: `https://download.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2` Download it directly on the Proxmox node: 
```bash
ssh root@<proxmox-ip>
cd /tmp
wget https://dl.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2
```
 Or copy it from your workstation: 
```powershell
scp C:\path\to\Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 root@<proxmox-ip>:/tmp/
```
 ## 2. Create a blank VM [Section titled “2. Create a blank VM”](#2-create-a-blank-vm) In the Proxmox web UI, create a VM with: | Tab | Setting | | ------- | -------------------------------------------------------------- | | General | ID matching `clone_vm_id` (e.g. `8000`), name `rocky-template` | | OS | **Do not use any media** | | System | Defaults (SeaBIOS is fine) | | Disks | **Delete the default disk** | | CPU | 2 cores | | Memory | 2048 MB | | Network | Bridge `vmbr0`, model VirtIO | Do not start the VM. ## 3. Import the disk [Section titled “3. Import the disk”](#3-import-the-disk) On the Proxmox node: 
```bash
qm importdisk <VMID> /tmp/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 local-lvm
```
 The disk appears as **Unused Disk 0** in the VM’s hardware list. ## 4. Attach the disk and the cloud-init drive [Section titled “4. Attach the disk and the cloud-init drive”](#4-attach-the-disk-and-the-cloud-init-drive) 1. In **Hardware**, double-click **Unused Disk 0**, select the **SCSI** bus and make sure it is attached as `scsi0`. 2. Add a **CloudInit Drive** (**Hardware → Add → CloudInit Drive**) on `local-lvm`. It is added as `scsi1` or `ide2`. ## 5. Set the boot order [Section titled “5. Set the boot order”](#5-set-the-boot-order) In **Options → Boot Order**, enable `scsi0` and move it first. Network boot can be second; the cloud-init drive goes last. ## 6. Test the VM [Section titled “6. Test the VM”](#6-test-the-vm) 1. Start the VM and open the noVNC console. 2. Check that it boots, gets an IP over DHCP and that cloud-init runs. Look at `/var/log/cloud-init.log` if something fails. ## 7. Convert it to a template [Section titled “7. Convert it to a template”](#7-convert-it-to-a-template) Stop the VM, right-click it and choose **Convert to Template**. The template is ready for [Deploy VMs](/how-to/deploy-vms/).

# Deploy LXC containers

> Add, change or remove LXC containers with the terraform/proxmox/lxc configuration.

`terraform/proxmox/lxc` creates LXC containers in Proxmox using the `proxmox/lxc` module from [Brsalcedom/tf-modules](https://github.com/Brsalcedom/tf-modules). ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * Your shell has the variables listed in [Environment variables](/reference/environment-variables/#proxmox-lxc). * The OS template is downloaded on the node. ## Download the container template [Section titled “Download the container template”](#download-the-container-template) In the Proxmox UI go to **Node → local → CT Templates → Templates** and download the template set in `defaults.template` in `locals.tf` (currently `debian-13-standard_13.6-1_amd64.tar.zst`). ## Add a container [Section titled “Add a container”](#add-a-container) Containers are declared in the `containers` map in `terraform/proxmox/lxc/locals.tf`. Any key you leave out falls back to `defaults`. 
```hcl
containers = {
  # ...
  "my-container" = {
    description = "My container | IP: 192.168.10.180"
    ip          = "192.168.10.180"
    id          = 180
    cpu         = 4    # optional, default 2
    memory      = 4096 # optional, default 2048
    disk        = 30   # optional, default 15 (GB)
    tags        = ["example"]
  }
}
```
 Other optional keys: `template`, `unprivileged` (default `true`) and `mountpoint` (passed to the module as `nas_mountpoint`). The target node comes from the `node_name` variable (default `prox`). Override it with `TF_VAR_node_name`. ## Apply the change [Section titled “Apply the change”](#apply-the-change) Through CI: merge to `main`; the [`Tofu :: Proxmox LXC`](/reference/ci-workflows/) workflow plans and applies. From your workstation: 
```bash
cd terraform/proxmox/lxc
tofu init
tofu plan
tofu apply
```
 ## Check the result [Section titled “Check the result”](#check-the-result) 
```bash
ssh root@<container-ip>
```
 Then confirm CPU, memory, disk, tags, description and network settings in the Proxmox UI. See [Terraform: LXC](/reference/terraform-lxc/) for the inputs and modules of this configuration.

# Deploy VMs

> Add, change or remove QEMU virtual machines with the terraform/proxmox/vm configuration.

`terraform/proxmox/vm` creates cloud-init VMs in Proxmox using the `proxmox/vm-cloud-init` module from [Brsalcedom/tf-modules](https://github.com/Brsalcedom/tf-modules). ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * The [Rocky Linux template](/how-to/create-rocky-template/) exists. * Your shell has the variables listed in [Environment variables](/reference/environment-variables/#proxmox-vms), including SSH access to the node for the cloud-init snippet upload. ## Add a VM [Section titled “Add a VM”](#add-a-vm) VMs are declared in the `vms` map in `terraform/proxmox/vm/locals.tf`. Any key you leave out falls back to `defaults` in the same file. 
```hcl
vms = {
  # ...
  "my-new-vm" = {
    description = "My new VM | IP: 192.168.10.220"
    ip          = "192.168.10.220"
    id          = 220
    memory      = 4096 # optional, default 2048
    tags        = ["rocky", "example"]
  }
}
```
 The `terraform` tag is always added through `common_tags`. Per-VM keys that override `defaults`: `cpu`, `memory`, `clone_vm_id`, `username`, `node_name`, `bridge`, `qemu_agent`. The gateway and DNS servers always come from `defaults`. ## Apply the change [Section titled “Apply the change”](#apply-the-change) Through CI (recommended): open a pull request and merge it to `main`. The [`Tofu :: Proxmox VM`](/reference/ci-workflows/) workflow runs a plan and waits for approval on the `tofu-infra` environment before applying. From your workstation: 
```bash
cd terraform/proxmox/vm
tofu init
tofu plan
tofu apply
```
 ## Check the result [Section titled “Check the result”](#check-the-result) 
```bash
ssh rocky@<vm-ip>
```
 See [Terraform: VM](/reference/terraform-vm/) for the inputs and modules of this configuration.

# Expose an app on Hyperion

> Add an application to ArgoCD and publish it on *.home.cervant.net through the shared Cilium Gateway.

Every app on Hyperion follows the same pattern: an ArgoCD `Application` that points to a Kustomize directory, and an `HTTPRoute` attached to the shared `cilium-gateway`. TLS is already handled by the gateway’s wildcard certificate, so you don’t create a `Certificate` per app. ## 1. Create the directory [Section titled “1. Create the directory”](#1-create-the-directory) 
```text
kubernetes/applications/argocd/<app>/
├── application.yaml
└── manifests/
    ├── kustomization.yaml
    ├── namespace.yaml
    ├── deployment.yaml
    ├── service.yaml
    └── httproute.yaml
```
 The root app-of-apps picks up every file named `*application.yaml` under `kubernetes/applications/argocd`, so nothing else needs to be registered. ## 2. Write the Application [Section titled “2. Write the Application”](#2-write-the-application) 
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <app>
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  project: homelab
  source:
    repoURL: https://github.com/Brsalcedom/homelab-iac.git
    targetRevision: main
    path: kubernetes/applications/argocd/<app>/manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: <app>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
```
 Caution `path` must start with `kubernetes/`, the directory that exists in the repository. ## 3. Write the HTTPRoute [Section titled “3. Write the HTTPRoute”](#3-write-the-httproute) 
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <app>
  namespace: <app>
spec:
  parentRefs:
    - name: cilium-gateway
      namespace: cilium-gateway
      sectionName: https
  hostnames:
    - <app>.home.cervant.net
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: <app>
          port: 80
```
 The `http` listener of the gateway already redirects all `*.home.cervant.net` traffic to HTTPS. ## 4. Merge to main [Section titled “4. Merge to main”](#4-merge-to-main) ArgoCD syncs the new Application automatically. Check it with: 
```bash
kubectl -n argocd get application <app>
kubectl -n <app> get httproute
```
 The `homelab-docs` app (`kubernetes/applications/argocd/homelab-docs`) is a complete example.

# Install K3s on Hyperion

> Install K3s on the Hyperion node with the flags Cilium needs, and open the API to the LAN.

Hyperion runs K3s without its bundled networking, because Cilium replaces Flannel, kube-proxy, network policies and Traefik. [Clusters: Hyperion and Cronos](/explanation/clusters/) explains why. Validated on K3s only The Hyperion stack has been tested only on K3s `v1.37.1+k3s1` running on Rocky Linux 10.2. It relies on K3s behavior, so it will not work unchanged on other distributions such as kubeadm, RKE2, k0s or MicroK8s. See [What the stack expects from K3s](/explanation/clusters/#what-the-stack-expects-from-k3s). ## Install K3s [Section titled “Install K3s”](#install-k3s) On `hyperion-01`: 
```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik --disable-network-policy --disable-kube-proxy --flannel-backend=none --node-name=hyperion" sh -
```
 Pods stay `Pending` and the node `NotReady` until Cilium is installed by [Apply the Hyperion stack](/how-to/apply-hyperion-stack/). ### If the install fails with `SSL certificate problem: self-signed certificate` [Section titled “If the install fails with SSL certificate problem: self-signed certificate”](#if-the-install-fails-with-ssl-certificate-problem-self-signed-certificate) The script first resolves the `stable` channel against `update.k3s.io`. If a router or firewall breaks that one connection, you get this error even though GitHub and `get.k3s.io` work. Skip the lookup by pinning a version: 
```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.37.1+k3s1" INSTALL_K3S_EXEC="--disable traefik --disable-network-policy --disable-kube-proxy --flannel-backend=none --node-name=hyperion" sh -
```
 Check the [K3s releases page](https://github.com/k3s-io/k3s/releases) for the latest version. ## Open the API server to trusted networks [Section titled “Open the API server to trusted networks”](#open-the-api-server-to-trusted-networks) The API server listens on `6443/tcp` on `192.168.10.201`. Allow it only from trusted internal networks: 
```bash
sudo firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=192.168.10.0/24 port port=6443 protocol=tcp accept'
sudo firewall-cmd --permanent --zone=public --add-rich-rule='rule family=ipv4 source address=192.168.20.0/24 port port=6443 protocol=tcp accept'
sudo firewall-cmd --reload
sudo firewall-cmd --zone=public --list-rich-rules
```
 Adjust the CIDRs to the networks that need cluster access. Do not open `6443/tcp` to all sources. ## Trust the pod network [Section titled “Trust the pod network”](#trust-the-pod-network) Pods reach the API server through `10.43.0.1`, which Cilium translates to `192.168.10.201:6443` with the pod IP as source. firewalld drops that traffic unless the pod CIDR (Cilium’s `cluster-pool` default, `10.0.0.0/8`) is trusted: 
```bash
sudo firewall-cmd --permanent --zone=trusted --add-source=10.0.0.0/8
sudo firewall-cmd --reload
```
 Without this, CoreDNS never becomes ready, `metrics-server` and `local-path-provisioner` crash with `dial tcp 10.43.0.1:443: i/o timeout`, and Helm releases such as Argo CD fail to install. ## Open HTTP and HTTPS for the Gateway [Section titled “Open HTTP and HTTPS for the Gateway”](#open-http-and-https-for-the-gateway) The shared Gateway gets the node IP (`192.168.10.201`) as its LoadBalancer address, through the K3s ServiceLB. Open `80/tcp` and `443/tcp` so clients on the LAN can reach the apps: 
```bash
sudo firewall-cmd --permanent --zone=public --add-service=http --add-service=https
sudo firewall-cmd --reload
```
 Without this, DNS resolves and the certificate is `Ready`, but browsers time out on every `*.home.cervant.net` hostname. ## Configure your kubeconfig [Section titled “Configure your kubeconfig”](#configure-your-kubeconfig) Copy `/etc/rancher/k3s/k3s.yaml` from the node and change the server to `https://192.168.10.201:6443` (not `127.0.0.1`). Then: 
```bash
export KUBECONFIG=/path/to/hyperion-kubeconfig.yaml
```
 Check connectivity from Windows with: 
```powershell
Test-NetConnection 192.168.10.201 -Port 6443
```
 ## Start over [Section titled “Start over”](#start-over) To remove K3s completely: 
```bash
sudo /usr/local/bin/k3s-uninstall.sh
```

# Write and publish these docs

> Edit the wiki locally, and set up the two publication targets (Cloudflare and Hyperion).

The wiki lives in `docs/` and is built with [Starlight](https://starlight.astro.build). [Docs delivery](/explanation/docs-delivery/) explains the design. ## Edit locally [Section titled “Edit locally”](#edit-locally) 
```bash
cd docs
npm ci
npm run dev
```
 Open `http://localhost:4321`. Pages are Markdown files under `docs/src/content/docs/`, grouped by Diátaxis type: | Folder | Put here | | -------------- | --------------------------------------------------------- | | `tutorials/` | A guided lesson with a known end state | | `how-to/` | Steps to reach one goal, for someone who knows the basics | | `reference/` | Facts to look up: tables, variables, inputs | | `explanation/` | Background, design decisions, trade-offs | The sidebar is generated from the folders. Use `sidebar.order` in the frontmatter to sort pages. Check the production build before pushing: 
```bash
npm run build
npm run preview
```
 ## Publish [Section titled “Publish”](#publish) Merge to `main`. The [`Docs`](/reference/ci-workflows/#docs) workflow builds once and publishes to: * `https://docs.cervant.net` (Cloudflare Workers static assets) * `https://docs.home.cervant.net` (nginx image on Hyperion, updated by ArgoCD Image Updater) ## One-time setup [Section titled “One-time setup”](#one-time-setup) ### Cloudflare [Section titled “Cloudflare”](#cloudflare) 1. Create an API token with the **Edit Cloudflare Workers** template, scoped to the account and to the `cervant.net` zone. 2. Store the token and the account ID in Bitwarden Secrets Manager, and put their IDs in the **Get secrets from Bitwarden** step of the `cloudflare` job in `.github/workflows/docs.yaml` (they start as `00000000-...` placeholders). On the first deploy, `wrangler` creates the `homelab-docs` Worker and the `docs.cervant.net` custom domain (DNS record and certificate) from `docs/wrangler.jsonc`. To deploy by hand: 
```bash
cd docs
npm run build
npx wrangler login
npm run deploy:cloudflare
```
 ### Hyperion [Section titled “Hyperion”](#hyperion) 1. After the first image push, make the `ghcr.io/brsalcedom/homelab-docs` package public (**GitHub → Packages → homelab-docs → Package settings**). Otherwise the cluster needs an `imagePullSecret`. 2. Make sure `docs.home.cervant.net` resolves to the Cilium Gateway address in the internal DNS (Pi-hole, `192.168.10.150`), unless a wildcard `*.home.cervant.net` record already covers it. 3. Apply the Hyperion stack once after `argocd.tf` changes, so the root app ignores the image overrides (see [Docs delivery](/explanation/docs-delivery/)). Image Updater checks GHCR every two minutes, so `docs.home.cervant.net` updates a few minutes after `docs.cervant.net`. To see what it did: 
```bash
kubectl -n argocd logs deploy/argocd-image-updater-controller | grep homelab-docs
kubectl -n argocd get application homelab-docs -o jsonpath='{.spec.source.kustomize.images}'
```
 No certificate work is needed: the gateway’s wildcard certificate for `*.home.cervant.net` already covers the hostname.

# Set up the GitHub self-hosted runner

> Turn the github-runner-01 LXC container into a self-hosted GitHub Actions runner with Docker.

The `github-runner-01` container (`192.168.10.111`) runs the workflows that need access to the LAN: OpenTofu plans and applies against Proxmox and Hyperion, and the LXC bootstrap jobs. Those jobs run inside the `brsalcedom/homelab-ci` image, so the runner needs Docker. This guide is for **Debian 13**. On another OS, follow the [official Docker install docs](https://docs.docker.com/engine/install/) for step 2. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * The `github-runner-01` container is deployed (see [Deploy LXC containers](/how-to/deploy-lxc-containers/)). * The SSH private key set when the container was created. * Admin access to the repository on GitHub, to get a registration token. Connect as root: 
```bash
ssh -i ~/.ssh/id_ed25519 root@192.168.10.111
```
 Steps 1 to 3 run as `root` without `sudo`. ## 1. Install base packages [Section titled “1. Install base packages”](#1-install-base-packages) 
```bash
apt update
apt install -y curl git sudo ca-certificates
```
 ## 2. Install Docker [Section titled “2. Install Docker”](#2-install-docker) 
```bash
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc


tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/debian
Suites: $(. /etc/os-release && echo "$VERSION_CODENAME")
Components: stable
Signed-By: /etc/apt/keyrings/docker.asc
EOF


apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin


docker run hello-world
```
 ## 3. Create the runner user [Section titled “3. Create the runner user”](#3-create-the-runner-user) The runner must not run as root. 
```bash
useradd -m -s /bin/bash github
usermod -aG docker github
usermod -aG sudo github
```
 ## 4. Download and register the runner [Section titled “4. Download and register the runner”](#4-download-and-register-the-runner) Switch to the new user: 
```bash
su - github
mkdir -p ~/actions-runner && cd ~/actions-runner
```
 Download the runner. Replace `2.331.0` with the latest version from the [runner releases page](https://github.com/actions/runner/releases), and check the hash published there: 
```bash
curl -o actions-runner-linux-x64-2.331.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.331.0/actions-runner-linux-x64-2.331.0.tar.gz
echo "<hash>  actions-runner-linux-x64-2.331.0.tar.gz" | sha256sum -c
tar xzf ./actions-runner-linux-x64-2.331.0.tar.gz
```
 Get a registration token in GitHub under **Settings → Actions → Runners → New self-hosted runner**, then: 
```bash
./config.sh --url https://github.com/Brsalcedom/homelab-iac --token <TOKEN>
```
 Accept the defaults, except the runner name: use `github-runner-01`. ## 5. Install the runner as a systemd service [Section titled “5. Install the runner as a systemd service”](#5-install-the-runner-as-a-systemd-service) Go back to root (`exit`) and run: 
```bash
cd /home/github/actions-runner
sudo ./svc.sh install github
sudo ./svc.sh start
sudo ./svc.sh status
```
 The service name is `actions.runner.<owner>-<repo>.<runner>.service`, here `actions.runner.Brsalcedom-homelab-iac.github-runner-01.service`. Enable it at boot: 
```bash
sudo systemctl enable actions.runner.Brsalcedom-homelab-iac.github-runner-01.service
```
 ## 6. Verify [Section titled “6. Verify”](#6-verify) 1. In GitHub, **Settings → Actions → Runners** shows `github-runner-01` as **Idle**. 2. Run any `workflow_dispatch` workflow (for example `Tofu :: Proxmox VM`) and check that the runner picks it up. ## Operate the runner [Section titled “Operate the runner”](#operate-the-runner) 
```bash
cd /home/github/actions-runner
sudo ./svc.sh stop
sudo ./svc.sh start
sudo ./svc.sh uninstall


# Service logs
sudo journalctl -u actions.runner.Brsalcedom-homelab-iac.github-runner-01.service -f


# Runner logs
sudo -u github tail -f /home/github/actions-runner/_diag/*.log
```
 ### Update the runner [Section titled “Update the runner”](#update-the-runner) 
```bash
cd /home/github/actions-runner
sudo ./svc.sh stop
sudo -u github curl -o actions-runner-linux-x64-<VERSION>.tar.gz -L https://github.com/actions/runner/releases/download/v<VERSION>/actions-runner-linux-x64-<VERSION>.tar.gz
sudo -u github tar xzf ./actions-runner-linux-x64-<VERSION>.tar.gz
sudo ./svc.sh start
```
 ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) | Symptom | Check | | -------------------------------- | -------------------------------------------------------------------------------------------------- | | Runner doesn’t appear in GitHub | `systemctl status` and `journalctl -n 50` for the service | | Docker permission denied in jobs | `groups github` must include `docker`; if not, `usermod -aG docker github` and restart the service | | Runner shows **Offline** | Connectivity to `github.com`, then restart the service | ## Security notes [Section titled “Security notes”](#security-notes) * The `github` user can reach the Docker socket and its work directory `~/actions-runner/_work`. * Secrets come from GitHub Secrets and from Bitwarden Secrets Manager (`bitwarden/sm-action`). Never store secrets on the container. * Watch CPU, RAM and disk usage of the container; container jobs and images add up.

# CI workflows

> GitHub Actions workflows in .github/workflows, their triggers and runners.

Jobs that reach the LAN run on the [self-hosted runner](/how-to/setup-github-runner/) inside the `brsalcedom/homelab-ci` image. Everything else runs on GitHub-hosted runners. ## Infrastructure [Section titled “Infrastructure”](#infrastructure) | Workflow | File | Trigger | Runner | | --------------------------- | ------------------------------- | ------------------------------------------------------------------------ | ------------- | | Tofu :: Proxmox VM | `tofu-proxmox-vm.yaml` | Push to `main` on `terraform/proxmox/vm/**`, manual | self-hosted | | Tofu :: Proxmox LXC | `tofu-proxmox-lxc.yaml` | Push to `main` on `terraform/proxmox/lxc/**`, manual | self-hosted | | Tofu :: k8s hyperion | `tofu-k8s-hyperion.yaml` | Push to `main` on `terraform/proxmox/infraestructure/hyperion**`, manual | self-hosted | | Bootstrap :: LXC auth | `lxc-auth-bootstrap.yaml` | Manual | self-hosted | | Bootstrap :: LXC monitoring | `lxc-monitoring-bootstrap.yaml` | Manual | self-hosted | | IaC :: Security Report | `iac-security-report.yml` | Pull requests on `terraform/**`, `cloudflare/**`, manual | GitHub-hosted | The `Tofu ::` workflows have two jobs: **Plan** uploads `tfplan` as an artifact and **Apply** applies it. The VM workflow’s apply job waits for the `tofu-infra` environment. ## Docs [Section titled “Docs”](#docs) | Workflow | File | Trigger | Runner | | -------- | ----------- | --------------------------------------------------------------------------------------- | ------------- | | Docs | `docs.yaml` | Push to `main` on `docs/**` and `terraform/**/*.tf`, pull requests on `docs/**`, manual | GitHub-hosted | Jobs: 1. **Build**: runs terraform-docs into the reference pages, then `astro build`, and uploads `dist/` as an artifact. 2. **Cloudflare**: `wrangler deploy` of `dist/` to `docs.cervant.net`, with credentials from Bitwarden. Only on `main`. 3. **Image**: builds the nginx image with `dist/` and pushes `ghcr.io/brsalcedom/homelab-docs:sha-<commit>` and `:latest`. Only on `main`. ArgoCD Image Updater picks up the new `:latest` digest and rolls out `docs.home.cervant.net`; the workflow doesn’t touch the cluster. On pull requests only the build runs. ## Repository housekeeping [Section titled “Repository housekeeping”](#repository-housekeeping) | Workflow | File | Trigger | | ----------------- | --------------------- | -------------------------------------------------- | | release-please | `release-please.yaml` | Push to `main` | | Validate PR Title | `pr-title-check.yaml` | Pull requests (Conventional Commits) | | PR assignment | `pr-auto-assign.yaml` | Pull requests | | Label Sync | `label-sync.yaml` | `.github/labels.yml` changes, daily, pull requests |

# Environment variables

> Variables each Terraform configuration and the docs workflow expect.

In CI, most of these come from Bitwarden Secrets Manager through `bitwarden/sm-action`, using the `BW_TOKEN` repository secret. ## Shared: R2 state backend [Section titled “Shared: R2 state backend”](#shared-r2-state-backend) All Terraform configurations under `terraform/` store state in the `homelab-tfstate` R2 bucket through the S3 backend. | Variable | Value | | ----------------------- | ------------- | | `AWS_ACCESS_KEY_ID` | R2 access key | | `AWS_SECRET_ACCESS_KEY` | R2 secret key | ## Proxmox VMs [Section titled “Proxmox VMs”](#proxmox-vms) `terraform/proxmox/vm` | Variable | Value | | ---------------------------- | ---------------------------------------------------------------------- | | `PROXMOX_VE_ENDPOINT` | `https://<proxmox-ip>:8006/api2/json` | | `PROXMOX_VE_API_TOKEN` | `terraform@pve!iac=<TOKEN_SECRET>` | | `PROXMOX_VE_SSH_USERNAME` | `root` | | `PROXMOX_VE_SSH_PRIVATE_KEY` | Private key authorized on the node, for snippet uploads | | `TF_VAR_ssh_authorized_keys` | JSON list of public keys for the VMs, e.g. `'["ssh-ed25519 AAAA..."]'` | ## Proxmox LXC [Section titled “Proxmox LXC”](#proxmox-lxc) `terraform/proxmox/lxc` | Variable | Value | | ---------------------------- | ------------------------------------------- | | `PROXMOX_VE_ENDPOINT` | `https://<proxmox-ip>:8006/api2/json` | | `PROXMOX_VE_API_TOKEN` | `terraform@pve!iac=<TOKEN_SECRET>` | | `TF_VAR_ssh_authorized_keys` | JSON list of public keys for the containers | | `TF_VAR_node_name` | Target node (optional, default `prox`) | ## Hyperion [Section titled “Hyperion”](#hyperion) `terraform/proxmox/infraestructure/hyperion` | Variable | Value | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `KUBE_CONFIG_PATH` | Path to the Hyperion kubeconfig (API at `192.168.10.201:6443`). The `kubernetes` and `helm` providers are configured empty and read this variable, not `KUBECONFIG`. | | `KUBECONFIG` | Same path, for `kubectl` in the bootstrap task | | `CLOUDFLARE_API_TOKEN` | Cloudflare token with **Zone → DNS → Edit** and **Zone → Zone → Read** on `cervant.net`. The bootstrap task stores it as the `cloudflare-api-token` secret for cert-manager DNS-01. | | `INFISICAL_API_TOKEN` | Infisical service token, stored as the `infisical-service-token` secret for the Infisical operator | The bootstrap task (`.github/tasks/hyperion-bootstrap.yml`) reads `CLOUDFLARE_API_TOKEN`, `INFISICAL_API_TOKEN` and `KUBECONFIG` from a `.env` file. CI writes it in `terraform/proxmox/infraestructure/hyperion` before running the task. ## Docs workflow [Section titled “Docs workflow”](#docs-workflow) Bitwarden secrets used by `.github/workflows/docs.yaml`: | Output | Value | | ------------------------------ | ---------------------------------------------------------------------------------------------------- | | `CLOUDFLARE_WORKERS_API_TOKEN` | Token with the **Edit Cloudflare Workers** permissions, passed to wrangler as `CLOUDFLARE_API_TOKEN` | | `CLOUDFLARE_ACCOUNT_ID` | Cloudflare account ID | The GHCR push uses the built-in `GITHUB_TOKEN`. ## Setting variables [Section titled “Setting variables”](#setting-variables) Bash: 
```bash
export PROXMOX_VE_ENDPOINT="https://<proxmox-ip>:8006/api2/json"
```
 PowerShell: 
```powershell
$env:PROXMOX_VE_ENDPOINT="https://<proxmox-ip>:8006/api2/json"
```

# Hyperion components

> Components of the Hyperion K3s cluster, where each one is defined and its status.

## Platform (OpenTofu) [Section titled “Platform (OpenTofu)”](#platform-opentofu) Installed by `terraform/proxmox/infraestructure/hyperion`. Chart versions are in `locals.tf`. | Component | Technology | Defined in | Status | | --------------------------- | ---------------------------------------------------------------------------- | ------------------------- | ------ | | CNI, kube-proxy replacement | [Cilium](https://cilium.io) (eBPF) | `cilium.tf` | ✅ | | LoadBalancer | K3s ServiceLB (`svclb-*` pods), node IP `192.168.10.201` | Built into K3s | ✅ | | Gateway API | Cilium Gateway API controller | `cilium.tf`, `gateway.tf` | ✅ | | Certificates | [cert-manager](https://cert-manager.io) + Let’s Encrypt DNS-01 on Cloudflare | `certmanager.tf` | ✅ | | GitOps | [ArgoCD](https://argo-cd.readthedocs.io) | `argocd.tf` | ✅ | ## Applications (ArgoCD) [Section titled “Applications (ArgoCD)”](#applications-argocd) Defined under `kubernetes/applications/argocd/`. | Component | Technology | Status | | ------------- | ------------------------------------------------------------------- | ------ | | Secrets | [Infisical](https://infisical.com) Kubernetes operator | ✅ | | Policy engine | [Kyverno](https://kyverno.io) | ✅ | | PostgreSQL | [CloudNativePG](https://cloudnative-pg.io) | ✅ | | Identity | [Pocket ID](https://pocket-id.org) | ✅ | | Image updates | [ArgoCD Image Updater](https://argocd-image-updater.readthedocs.io) | ✅ | | Docs | `homelab-docs` (this wiki) | ✅ | | Storage | [Rook + Ceph](https://rook.io) | 🔜 | | Observability | Prometheus, Grafana, Loki | 🔜 | ## Shared resources [Section titled “Shared resources”](#shared-resources) | Resource | Kind | Namespace | Notes | | -------------------------- | --------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `cilium-gateway` | `Gateway` | `cilium-gateway` | Listeners `https` (443, TLS terminate) and `http` (80) for `*.home.cervant.net`; routes allowed from all namespaces | | `wildcard-certificate` | `Certificate` | `cilium-gateway` | `*.home.cervant.net`, stored in secret `tls-cert` | | `redirect-to-https` | `HTTPRoute` | `cilium-gateway` | 301 from `http` to `https` | | `cloudflare-clusterissuer` | `ClusterIssuer` | (cluster) | ACME DNS-01, token from secret `cloudflare-api-token` | | `homelab` | `AppProject` | `argocd` | Project for every app | | `root` | `Application` | `argocd` | App-of-apps over `kubernetes/applications/argocd`, `*application.yaml`; ignores `/spec/source/kustomize/images` on child Applications | ## Endpoints [Section titled “Endpoints”](#endpoints) | URL | Service | | --------------------------------- | --------- | | `https://argocd.home.cervant.net` | ArgoCD | | `https://pocket.home.cervant.net` | Pocket ID | | `https://docs.home.cervant.net` | This wiki | ## K3s install flags [Section titled “K3s install flags”](#k3s-install-flags) 
```text
--disable traefik --disable-network-policy --disable-kube-proxy --flannel-backend=none --node-name=hyperion
```
 ## Nodes [Section titled “Nodes”](#nodes) | Name | IP | Proxmox VM ID | vCPU | Memory | | ------------- | ---------------- | ------------- | ---- | ------ | | `hyperion-01` | `192.168.10.201` | 201 | 2 | 6 GiB | Hyperion is a single-node cluster. With 2 GiB the node stalls once ArgoCD syncs the applications, so keep it at 6 GiB or more.

# Proxmox permissions

> Privileges of the Terraform role used by the VM and LXC configurations.

OpenTofu authenticates as `terraform@pve` with the API token `terraform@pve!iac` (created with `--privsep 0`, so the token has the user’s privileges). The user has the `Terraform` role on `/`. ## Privileges [Section titled “Privileges”](#privileges) | Privilege | VM | LXC | Why | | ------------------------- | -- | --- | ----------------------------------------- | | `VM.Audit` | ✅ | ✅ | Read guest configuration and status | | `VM.Allocate` | ✅ | ✅ | Create and delete guests | | `VM.Clone` | ✅ | | Clone VMs from the cloud-init template | | `VM.PowerMgmt` | ✅ | ✅ | Start and stop guests | | `VM.Config.Disk` | ✅ | ✅ | Disks and root filesystem | | `VM.Config.CPU` | ✅ | ✅ | CPU cores | | `VM.Config.Memory` | ✅ | ✅ | Memory | | `VM.Config.Network` | ✅ | ✅ | Network interfaces | | `VM.Config.HWType` | ✅ | ✅ | Hardware type settings | | `VM.Config.Options` | ✅ | ✅ | Description, tags, boot options | | `VM.Config.CDROM` | ✅ | | Cloud-init drive | | `VM.Config.Cloudinit` | ✅ | | Cloud-init settings | | `Sys.Modify` | ✅ | ✅ | Node-level changes made by the provider | | `Datastore.AllocateSpace` | ✅ | ✅ | Allocate disk space | | `Datastore.Allocate` | ✅ | | Upload the cloud-init snippet | | `Datastore.Audit` | ✅ | ✅ | Read storage information | | `SDN.Use` | ✅ | ✅ | Attach guests to bridges and SDN networks | ## Create the user, role and token [Section titled “Create the user, role and token”](#create-the-user-role-and-token) Run as root on the Proxmox host. This role covers both configurations: 
```bash
pveum user add terraform@pve --password 'ChangeMeStrong!'
pveum role add Terraform --privs "VM.Audit VM.Allocate VM.Clone VM.PowerMgmt VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.HWType VM.Config.Options VM.Config.CDROM VM.Config.Cloudinit Sys.Modify Datastore.AllocateSpace Datastore.Allocate Datastore.Audit SDN.Use"
pveum acl modify / --roles Terraform --users terraform@pve
pveum user token add terraform@pve iac --privsep 0
```
 To add privileges to an existing role: 
```bash
pveum role modify Terraform --privs "<full list>"
```
 In the web UI, tokens are under **Datacenter → Permissions → API Tokens**. ## SSH access [Section titled “SSH access”](#ssh-access) The VM configuration also needs SSH access to the node as `root`, because the Proxmox API can’t upload `snippets`. See [Environment variables](/reference/environment-variables/#proxmox-vms).

# Repository layout

> What each top-level directory of homelab-iac contains.

```text
homelab-iac/
├── .github/                    # Workflows, labels, labeler and PR template
│   └── tasks/                  # Ansible-style bootstrap task files
├── cloudflare/                 # Terraform for Cloudflare (R2 state bucket)
├── docker/
│   ├── compose/                # Docker Compose stacks for LXC hosts (auth, monitoring)
│   └── swarm/                  # Docker Swarm stacks
├── docs/                       # This wiki (Starlight)
├── kubernetes/
│   └── applications/
│       └── argocd/             # Apps managed by ArgoCD on Hyperion
└── terraform/
    └── proxmox/
        ├── infraestructure/
        │   └── hyperion/       # Platform stack: Cilium, cert-manager, Gateway, ArgoCD
        ├── lxc/                # LXC containers
        └── vm/                 # QEMU virtual machines
```
 ## Terraform configurations [Section titled “Terraform configurations”](#terraform-configurations) | Path | Manages | State key in R2 | | -------------------------------------------- | --------------------------------------------------- | -------------------------------------- | | `terraform/proxmox/vm` | QEMU VMs cloned from a cloud-init template | `proxmox/vm/terraform.tfstate` | | `terraform/proxmox/lxc` | LXC containers | `proxmox/lxc/terraform.tfstate` | | `terraform/proxmox/infraestructure/hyperion` | Helm releases and manifests on the Hyperion cluster | `k8s/infra/hyperion/terraform.tfstate` | | `cloudflare` | The `homelab-tfstate` R2 bucket | local (no remote backend) | ## ArgoCD applications [Section titled “ArgoCD applications”](#argocd-applications) Each directory under `kubernetes/applications/argocd/` holds one app: an `application.yaml` plus either Helm values or a `manifests/` Kustomize directory. The root app-of-apps (created in `argocd.tf`) syncs every `*application.yaml` file recursively. | App | Purpose | | ---------------------- | ------------------------------------------------------------ | | `argocd-image-updater` | Updates app images from their registries (argocd write-back) | | `cloudnative-pg` | PostgreSQL operator | | `homelab-docs` | This wiki, served internally | | `infisical` | Infisical secrets operator | | `kyverno` | Policy engine and cluster policies | | `netbird` | NetBird agent | | `pocket-id` | OIDC identity provider | ## Requirements [Section titled “Requirements”](#requirements) * OpenTofu or Terraform 1.9 or later * A kubeconfig per cluster * A public domain for valid certificates * A Cloudflare API token * Helm, kubectl, Git * Node.js 22 or later, to work on these docs

# Terraform: Hyperion

> Providers and resources of terraform/proxmox/infraestructure/hyperion.

Source: [`terraform/proxmox/infraestructure/hyperion`](https://github.com/Brsalcedom/homelab-iac/tree/main/terraform/proxmox/infraestructure/hyperion). ## Files [Section titled “Files”](#files) | File | Contents | | ---------------- | ------------------------------------------------------------------------------------------------------------------------- | | `backend.tf` | S3 backend on Cloudflare R2, key `k8s/infra/hyperion/terraform.tfstate` | | `providers.tf` | `hashicorp/helm`, `hashicorp/kubernetes` (pinned below 3.4.0, see the comment in the file) | | `locals.tf` | Chart versions, `base_domain`, ACME email | | `cilium.tf` | Cilium Helm release and a 120 s wait for the CNI | | `certmanager.tf` | cert-manager Helm release and the `cloudflare-clusterissuer` | | `gateway.tf` | `cilium-gateway` namespace, wildcard certificate, Gateway and HTTP→HTTPS redirect | | `argocd.tf` | ArgoCD Helm release, its HTTPRoute, the `homelab` project and the root app-of-apps | | `crds/` | Kustomization for the cert-manager and ArgoCD CRDs, installed separately from the charts. Gateway API CRDs come from K3s. | ## Locals [Section titled “Locals”](#locals) | Key | Value | | ---------------------------- | --------------------------- | | `base_domain` | `home.cervant.net` | | `cilium_chart_version` | Pinned, updated by Renovate | | `argocd_chart_version` | Pinned, updated by Renovate | | `cert_manager_chart_version` | Pinned, updated by Renovate | ## Generated [Section titled “Generated”](#generated) The section below is generated by [terraform-docs](https://terraform-docs.io) in CI. Don’t edit it by hand. ### Requirements [Section titled “Requirements”](#requirements) | Name | Version | | ---------- | -------- | | terraform | >= 1.9.0 | | helm | >= 3.0.2 | | kubernetes | < 3.4.0 | ### Providers [Section titled “Providers”](#providers) | Name | Version | | ---------- | -------- | | helm | >= 3.0.2 | | kubernetes | < 3.4.0 | | time | n/a | ### Modules [Section titled “Modules”](#modules) No modules. ### Resources [Section titled “Resources”](#resources) | Name | Type | | -------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | [helm_release.argocd](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | | [helm_release.cert_manager](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | | [helm_release.cilium](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource | | [kubernetes_manifest.argocd_app_of_apps](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_manifest.argocd_project](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_manifest.argocd_route](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_manifest.certificate](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_manifest.cilium_gateway](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_manifest.cilium_redirect_httproute](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_manifest.clusterissuer](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource | | [kubernetes_namespace_v1.argocd](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | | [kubernetes_namespace_v1.cert_manager](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | | [kubernetes_namespace_v1.cilium_gateway](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace_v1) | resource | | [time_sleep.after_cilium](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource | ### Inputs [Section titled “Inputs”](#inputs) No inputs. ### Outputs [Section titled “Outputs”](#outputs) No outputs.

# Terraform: LXC

> Inputs, providers and modules of terraform/proxmox/lxc.

Source: [`terraform/proxmox/lxc`](https://github.com/Brsalcedom/homelab-iac/tree/main/terraform/proxmox/lxc). Module: [`proxmox/lxc`](https://github.com/Brsalcedom/tf-modules/tree/main/proxmox/lxc). ## Files [Section titled “Files”](#files) | File | Contents | | --------------- | ----------------------------------------------------------------------- | | `backend.tf` | S3 backend on Cloudflare R2, key `proxmox/lxc/terraform.tfstate` | | `provider.tf` | `bpg/proxmox` provider | | `locals.tf` | `defaults`, the `containers` map and the merged `containers_merged` map | | `containers.tf` | One module call per container (`for_each = local.containers_merged`) | | `variables.tf` | `ssh_authorized_keys`, `node_name` | ## Defaults [Section titled “Defaults”](#defaults) | Key | Value | | -------------- | ----------------------------------------- | | `template` | `debian-13-standard_13.6-1_amd64.tar.zst` | | `cpu` | `2` | | `memory` | `2048` | | `disk` | `15` (GB) | | `gateway` | `192.168.10.1` | | `dns` | `["192.168.10.150"]` | | `unprivileged` | `true` | ## Generated [Section titled “Generated”](#generated) The section below is generated by [terraform-docs](https://terraform-docs.io) in CI. Don’t edit it by hand. ### Requirements [Section titled “Requirements”](#requirements) | Name | Version | | ------- | ------- | | proxmox | 0.116.0 | ### Providers [Section titled “Providers”](#providers) No providers. ### Modules [Section titled “Modules”](#modules) | Name | Source | Version | | ---------- | ------------------------------------------------------------ | ------- | | containers | git::<https://github.com/Brsalcedom/tf-modules//proxmox/lxc> | v1.2.4 | ### Resources [Section titled “Resources”](#resources) No resources. ### Inputs [Section titled “Inputs”](#inputs) | Name | Description | Type | Default | Required | | ------------------- | --------------------------------------------------- | -------------- | -------- | -------- | | node_name | Proxmox node name where containers will be deployed | `string` | `"prox"` | no | | ssh_authorized_keys | SSH authorized keys | `list(string)` | n/a | yes | ### Outputs [Section titled “Outputs”](#outputs) No outputs.

# Terraform: VM

> Inputs, providers and modules of terraform/proxmox/vm.

Source: [`terraform/proxmox/vm`](https://github.com/Brsalcedom/homelab-iac/tree/main/terraform/proxmox/vm). Module: [`proxmox/vm-cloud-init`](https://github.com/Brsalcedom/tf-modules/tree/main/proxmox/vm-cloud-init). ## Files [Section titled “Files”](#files) | File | Contents | | -------------- | --------------------------------------------------------------- | | `backend.tf` | S3 backend on Cloudflare R2, key `proxmox/vm/terraform.tfstate` | | `provider.tf` | `bpg/proxmox` provider; SSH block for snippet uploads | | `locals.tf` | `defaults`, the `vms` map and the merged `vms_merged` map | | `vms.tf` | One module call per VM (`for_each = local.vms_merged`) | | `variables.tf` | `ssh_authorized_keys` | ## Defaults [Section titled “Defaults”](#defaults) | Key | Value | | ------------- | ---------------------------------------- | | `clone_vm_id` | `8000` (Rocky Linux cloud-init template) | | `cpu` | `2` | | `memory` | `2048` | | `gateway` | `192.168.10.1` | | `dns` | `["192.168.10.150"]` | | `username` | `rocky` | | `node_name` | `prox-01` | | `bridge` | `vmbr0` | | `qemu_agent` | `false` | ## Generated [Section titled “Generated”](#generated) The section below is generated by [terraform-docs](https://terraform-docs.io) in CI. Don’t edit it by hand. ### Requirements [Section titled “Requirements”](#requirements) | Name | Version | | ------- | ------- | | proxmox | 0.116.0 | ### Providers [Section titled “Providers”](#providers) No providers. ### Modules [Section titled “Modules”](#modules) | Name | Source | Version | | ---- | ---------------------------------------------------------------------- | ------- | | vms | git::<https://github.com/Brsalcedom/tf-modules//proxmox/vm-cloud-init> | v1.2.4 | ### Resources [Section titled “Resources”](#resources) No resources. ### Inputs [Section titled “Inputs”](#inputs) | Name | Description | Type | Default | Required | | ------------------- | ------------------- | -------------- | ------- | -------- | | ssh_authorized_keys | SSH authorized keys | `list(string)` | n/a | yes | ### Outputs [Section titled “Outputs”](#outputs) No outputs.

# Getting started

> From a fresh Proxmox VE node to a first VM managed by OpenTofu with remote state in Cloudflare R2.

In this tutorial you prepare a fresh Proxmox VE node so OpenTofu can manage it, then use this repository to create a virtual machine. By the end you will have: * A dedicated `terraform@pve` user with an API token. * A local shell configured to talk to Proxmox and to the Cloudflare R2 state bucket. * A VM created from the `terraform/proxmox/vm` configuration. ## Before you begin [Section titled “Before you begin”](#before-you-begin) You need: * Root access to a Proxmox VE node. * [OpenTofu](https://opentofu.org) or [Terraform](https://www.terraform.io) 1.9 or later on your workstation. * Access keys for the `homelab-tfstate` R2 bucket (see [Remote state on Cloudflare R2](/explanation/state-backend/)). * An SSH key pair. * A clone of the repository: 
  ```bash
  git clone https://github.com/Brsalcedom/homelab-iac.git
  cd homelab-iac
  ```
 ## 1. Create the Terraform user on Proxmox [Section titled “1. Create the Terraform user on Proxmox”](#1-create-the-terraform-user-on-proxmox) On the Proxmox host, as root, create a user, a role and an API token: 
```bash
# Create the Terraform user
pveum user add terraform@pve --password 'ChangeMeStrong!'


# Create a role with the permissions used by the VM and LXC configurations
pveum role add Terraform --privs "VM.Audit VM.Allocate VM.Clone VM.PowerMgmt VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.HWType VM.Config.Options VM.Config.CDROM VM.Config.Cloudinit Sys.Modify Datastore.AllocateSpace Datastore.Allocate Datastore.Audit SDN.Use"


# Grant the role at the root path
pveum acl modify / --roles Terraform --users terraform@pve


# Create an API token
pveum user token add terraform@pve iac --privsep 0
```
 The token secret is printed only once. Copy it now. Check that everything was created: 
```bash
pveum user list
pveum role list
pveum acl list
pveum user token list terraform@pve
```
 You can see why each privilege is needed in [Proxmox permissions](/reference/proxmox-permissions/). ## 2. Allow SSH uploads of cloud-init snippets [Section titled “2. Allow SSH uploads of cloud-init snippets”](#2-allow-ssh-uploads-of-cloud-init-snippets) The VM configuration uploads a cloud-init user-data snippet, which the Proxmox API cannot do. The `bpg/proxmox` provider falls back to SSH for that, so on the Proxmox node authorize your public key for `root` and enable snippets on the `local` storage: 
```bash
mkdir -p ~/.ssh && chmod 700 ~/.ssh
echo "<contents of your public key>" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys


pvesm set local --content backup,iso,vztmpl,snippets
```
 ## 3. Create the VM template [Section titled “3. Create the VM template”](#3-create-the-vm-template) The VMs are cloned from a Rocky Linux cloud-init template with ID `8000`. Follow [Create the Rocky Linux template](/how-to/create-rocky-template/) and come back when the template exists. ## 4. Configure your shell [Section titled “4. Configure your shell”](#4-configure-your-shell) Export the Proxmox and R2 credentials on your workstation. **Bash** 
```bash
export PROXMOX_VE_ENDPOINT="https://<proxmox-ip>:8006/api2/json"
export PROXMOX_VE_API_TOKEN="terraform@pve!iac=<TOKEN_SECRET>"
export PROXMOX_VE_SSH_USERNAME="root"
export PROXMOX_VE_SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519)"
export AWS_ACCESS_KEY_ID="<R2_ACCESS_KEY>"
export AWS_SECRET_ACCESS_KEY="<R2_SECRET_KEY>"
export TF_VAR_ssh_authorized_keys='["ssh-ed25519 AAAA..."]'
```
 **PowerShell** 
```powershell
$env:PROXMOX_VE_ENDPOINT="https://<proxmox-ip>:8006/api2/json"
$env:PROXMOX_VE_API_TOKEN="terraform@pve!iac=<TOKEN_SECRET>"
$env:PROXMOX_VE_SSH_USERNAME="root"
$env:PROXMOX_VE_SSH_PRIVATE_KEY=Get-Content -Raw ~\.ssh\id_ed25519
$env:AWS_ACCESS_KEY_ID="<R2_ACCESS_KEY>"
$env:AWS_SECRET_ACCESS_KEY="<R2_SECRET_KEY>"
$env:TF_VAR_ssh_authorized_keys='["ssh-ed25519 AAAA..."]'
```
 ## 5. Plan and apply [Section titled “5. Plan and apply”](#5-plan-and-apply) 
```bash
cd terraform/proxmox/vm
tofu init
tofu plan
```
 The plan lists the VMs defined in `locals.tf`. If it looks right, apply it: 
```bash
tofu apply
```
 ## 6. Log in to the VM [Section titled “6. Log in to the VM”](#6-log-in-to-the-vm) Once the apply finishes, connect with the default `rocky` user: 
```bash
ssh rocky@<vm-ip>
```
 ## Next steps [Section titled “Next steps”](#next-steps) * Add your own machine: [Deploy VMs](/how-to/deploy-vms/). * Create LXC containers: [Deploy LXC containers](/how-to/deploy-lxc-containers/). * Build the Kubernetes cluster: [Install K3s on Hyperion](/how-to/install-k3s-hyperion/).