homelab-starter/README.md

# Homelab Starter

A minimal GitOps homelab setup using **K3s** and **Flux CD**. This starter gives you a working Kubernetes cluster with core infrastructure and one example app (Gitea), all managed declaratively through Git. Secrets are encrypted at rest in Git using **SOPS** with **age**.

## Architecture

```
  ┌──────────────────────────────────────────────────────────┐
  │  Git Repository                                          │
  │                                                          │
  │  bootstrap/          Infrastructure &          apps/     │
  │  ├── ns/             app definitions           └── gitea │
  │  ├── repositories/   are organized             │         │
  │  └── kustomization/  into layers               │         │
  └──────────┬─────────────────────────────────────┘         │
             │                                               │
             ▼                                               │
  ┌──────────────────┐     Flux watches your Git repo        │
  │   Flux CD        │     and continuously reconciles       │
  │   (flux-system)  │     the cluster state to match.       │
  └──────────┬───────┘                                       │
             │                                               │
             ▼                                               │
  ┌──────────────────────────────────────────────────────────┐
  │  Kubernetes Cluster (K3s)                                │
  │                                                          │
  │  Infrastructure layer          Application layer         │
  │  ┌────────────┐                ┌────────────┐            │
  │  │ MetalLB    │─── IPs ──────▶ │ Gitea      │            │
  │  │ Traefik    │─── Routing ──▶ │ PostgreSQL │            │
  │  │ Cert-Mgr   │─── TLS ─────▶  │            │            │
  │  └────────────┘                └────────────┘            │
  └──────────────────────────────────────────────────────────┘
```

## What's Included

| Component | Purpose |
|-----------|---------|
| **MetalLB** | Assigns real LAN IPs to LoadBalancer services |
| **Traefik** | Ingress controller — routes HTTP/HTTPS traffic to apps |
| **Cert-Manager** | Automatically provisions TLS certificates via Let's Encrypt |
| **Gitea** | Self-hosted Git service (example app) with PostgreSQL |

## Prerequisites

- A Linux machine (or VM) with [K3s](https://k3s.io) installed
- `kubectl` configured to talk to your cluster
- [Flux CLI](https://fluxcd.io/flux/installation/) installed locally
- [SOPS](https://github.com/getsops/sops) and [age](https://github.com/FiloSottile/age) installed locally
- A Git repository (GitHub, Gitea, GitLab, etc.) to host this code
- A domain name pointed at your cluster (for TLS certificates)
- A Cloudflare account (or other DNS provider) for DNS-01 challenges

## Quick Start

### 1. Copy this starter into your own repository

```bash
# Create a new repo and copy the starter contents into it
# (include hidden files like .sops.yaml)
cp -r starter/{.,}* ~/my-homelab/
cd ~/my-homelab
git init && git add -A && git commit -m "Initial homelab setup"
git remote add origin <YOUR_GIT_REPO_URL>
git push -u origin main
```

### 2. Replace placeholder values

Search for all `<YOUR_...>` placeholders and replace them with your values:

| Placeholder | File(s) | Description |
|---|---|---|
| `<YOUR_IP_RANGE>` | `infrastructure/metallb-config/ipaddresspool.yaml` | LAN IP range for LoadBalancer services, e.g. `192.168.1.200-192.168.1.210` |
| `<YOUR_LAN_CIDR>` | `infrastructure/traefik-install/traefik-override.yaml` | Your local network CIDR, e.g. `192.168.1.0/24` |
| `<YOUR_EMAIL>` | `infrastructure/cert-manager-issuer/issuer.yaml` | Email for Let's Encrypt notifications |
| `<YOUR_DNS_API_TOKEN>` | `infrastructure/cert-manager-issuer/secrets.yaml` | Cloudflare API token (see below) |
| `<YOUR_DOMAIN>` | `infrastructure/routes/gitea.yaml`, `apps/gitea/install/gitea.yaml` | e.g. `git.example.com` |
| `<YOUR_DB_PASSWORD>` | `apps/gitea/install/secrets.yaml`, `apps/gitea/install/gitea.yaml` | A strong password for PostgreSQL |

```bash
# Quick way to find all placeholders
grep -r '<YOUR_' --include='*.yaml' .
```

**Why is a Cloudflare API token needed?** Cert-manager uses [DNS-01 challenges](https://cert-manager.io/docs/configuration/acme/dns01/) to prove domain ownership to Let's Encrypt. Unlike HTTP-01 challenges, DNS-01 doesn't require your server to be publicly reachable — ideal for homelabs behind NAT. It works by automatically creating a DNS TXT record on your domain, which requires API access to your DNS provider. Create a Cloudflare API token with **Zone > DNS > Edit** permission for your domain at [dash.cloudflare.com/profile/api-tokens](https://dash.cloudflare.com/profile/api-tokens).

### 3. Set up secret encryption (SOPS)

Secrets are stored in dedicated `secrets.yaml` files and must be encrypted before committing to Git. This starter uses [SOPS](https://github.com/getsops/sops) with [age](https://github.com/FiloSottile/age) encryption so that Flux can decrypt them in-cluster while they stay encrypted at rest in your repository.

#### Generate an age key pair

```bash
age-keygen -o age.agekey
# Output: Public key: age1xxxxxxxxx...
```

> **Important:** Add `age.agekey` to `.gitignore` — never commit your private key.

#### Update `.sops.yaml`

Replace `<YOUR_AGE_PUBLIC_KEY>` in `.sops.yaml` with the public key from the previous step. This file tells SOPS which files to encrypt and which YAML fields to target:

```yaml
creation_rules:
  - path_regex: infrastructure/.*/secrets\.yaml$
    age: age1xxxxxxxxx...          # your public key
    encrypted_regex: "^(data|stringData)$"
  - path_regex: apps/.*/secrets\.yaml$
    age: age1xxxxxxxxx...          # your public key
    encrypted_regex: "^(data|stringData)$"
```

Only `data` and `stringData` fields are encrypted — metadata stays readable so Git diffs remain useful.

#### Encrypt your secret files

After filling in your actual values (step 2), encrypt each secrets file:

```bash
sops --encrypt --in-place infrastructure/cert-manager-issuer/secrets.yaml
sops --encrypt --in-place apps/gitea/install/secrets.yaml
```

To edit an encrypted file later:

```bash
sops infrastructure/cert-manager-issuer/secrets.yaml
```

#### Load the private key into the cluster

Flux needs the age private key to decrypt secrets during reconciliation:

```bash
kubectl create namespace flux-system

kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=age.agekey
```

The Flux Kustomizations in this starter already reference this secret via `decryption.secretRef.name: sops-age`.

### 4. Bootstrap Flux

```bash
flux bootstrap git \
  --url=<YOUR_GIT_REPO_URL> \
  --branch=main \
  --path=bootstrap
```

This command:
- Installs Flux components into the `flux-system` namespace
- Creates `bootstrap/flux-system/` with auto-generated Flux manifests
- Commits and pushes these files to your repo
- Starts the reconciliation loop

### 5. Watch Flux reconcile

```bash
# Watch all kustomizations converge
flux get kustomizations --watch

# Check pod status across all namespaces
kubectl get pods -A
```

### 6. Access Gitea

Once everything is reconciled and the TLS certificate is issued:

```bash
# Check the certificate status
kubectl get certificate -n gitea

# Open Gitea in your browser
echo "https://<YOUR_DOMAIN>"
```

## Directory Structure

```
.sops.yaml                  # SOPS encryption rules (which files, which fields)

bootstrap/                  # Flux bootstrap layer
├── flux-system/            # Auto-generated by `flux bootstrap` (don't edit)
├── kustomization/          # Flux Kustomization CRs — tell Flux what to deploy
│   ├── infrastructure/     #   One file per infrastructure component
│   └── apps/               #   One directory per app
│       └── gitea/
├── ns/                     # Namespace definitions
└── repositories/           # Helm chart repository sources

infrastructure/             # Core cluster services
├── metallb-install/        # MetalLB Helm release
├── metallb-config/         # MetalLB IP pool and L2 advertisement
├── traefik-install/        # Traefik Helm release + config overrides
├── cert-manager-install/   # cert-manager Helm release + config overrides
├── cert-manager-issuer/    # Let's Encrypt ClusterIssuer + DNS credentials
│   ├── issuer.yaml
│   └── secrets.yaml        # encrypted with SOPS
└── routes/                 # Traefik IngressRoutes (one per app)

apps/                       # Application deployments
└── gitea/
    └── install/            # Gitea + PostgreSQL manifests
        ├── gitea.yaml
        ├── postgresql.yaml
        └── secrets.yaml    # encrypted with SOPS
```

## How Flux Works

Flux follows a **reconciliation loop**:

1. **Source Controller** watches your Git repository for changes
2. **Kustomize Controller** applies Kustomization CRs (in `bootstrap/kustomization/`)
3. **Helm Controller** installs/upgrades Helm releases (MetalLB, Traefik, Cert-Manager)
4. If you push a change to Git, Flux detects it and updates the cluster automatically

The `dependsOn` fields in the Flux Kustomizations ensure things install in the right order:

```
MetalLB Install → MetalLB Config → Traefik Install → Routes
Cert-Manager Install → Cert-Manager Issuer
Traefik Install → Gitea Install
```

Kustomizations that reference encrypted secrets include a `decryption` block pointing to the `sops-age` secret, so Flux can decrypt them transparently during apply.

## Next Steps

- **Add a backup system** — I have used Restic + Rclone to back up my data to OneDrive but there are other options.
- **Add your second app** — See [`docs/adding-an-app.md`](docs/adding-an-app.md) for a step-by-step walkthrough.

## Troubleshooting

**Flux not reconciling?**
```bash
flux logs --level=error
flux get sources git         # Check if Flux can reach your repo
flux reconcile source git flux-system   # Force a sync
```

**MetalLB not assigning IPs?**
```bash
kubectl get ipaddresspool -n infrastructure
kubectl get svc -n infrastructure   # Check if Traefik has an EXTERNAL-IP
```

**Traefik not routing?**
```bash
kubectl get ingressroute -A          # Check if routes exist
kubectl logs -n infrastructure -l app.kubernetes.io/name=traefik
```

**TLS certificate not issuing?**
```bash
kubectl get certificate -A           # Check certificate status
kubectl get certificaterequest -A    # Check pending requests
kubectl describe clusterissuer cert-issuer   # Check issuer health
kubectl logs -n infrastructure -l app=cert-manager
```

**SOPS decryption failing?**
```bash
# Check if the sops-age secret exists
kubectl get secret sops-age -n flux-system

# Check Flux kustomization status for decryption errors
flux get kustomizations
kubectl describe kustomization cert-manager-issuer--infra -n flux-system
```

**Gitea not starting?**
```bash
kubectl get pods -n gitea            # Check pod status
kubectl logs -n gitea -l app=gitea   # Check Gitea logs
kubectl logs -n gitea -l app=postgresql   # Check DB logs
```