Server Homelab — Claude Code Context

# Server Homelab — Claude Code Context

## Project Overview
Docker + k3s homelab managed with Ansible. ~50 containerized services deployed across three hosts via infrastructure-as-code, the k3s migration **completed 2026-08-14** — `docs/archive/k3s-migration/` is now a historical record of executed work, not a plan. Docker survives only on `daniel-pi`. (Exact count, and the per-service host/route/auth/backup/auto-deploy facts: `docs/reference/services.md`, regenerated by the `docs-refresh` cron and browsable at `docs.local.<domain>`. Don't hand-maintain a precise number here.)

**Hosts:**
- `daniel-box` — k3s server / control-plane node (Traefik edge, Authelia+OIDC, Pi-hole DNS, Longhorn storage, most workloads since 2026-08 migration). Ansible runs on this host.
- `daniel-server` — k3s agent node (Intel XE graphics, LVM storage, UPS hardware + the
  `nut_host` shutdown chain; Docker uninstalled 2026-08-14 — the migration's end state)
- `daniel-pi` — Raspberry Pi, the **only** remaining Docker host (LAN-only: wg-easy, alloy,
  autoheal, docker-proxy). Driven remotely over SSH with `-e target=daniel-pi`.

**Key technologies:** k3s (Kubernetes), Ansible, Docker Compose (Pi only), Traefik (reverse proxy), Cloudflare DNS, Authelia (SSO), SOPS/age (secret encryption), Longhorn (storage), CrowdSec (WAF)

## Directory Structure
```
ansible/          # Ansible playbooks, roles, inventory, templates  ← EDIT HERE
  roles/k8s/        # One role per k3s workload (rendered manifests) — where most services live
  roles/containers/ # One role per Docker service (the Pi's) + the shared `common` role
scripts/          # Helper scripts, grouped by what they act on
  lib/              # The helpers everything else imports: repo_paths, render_guard, docs_provenance
  backup/ deploy_tools/ diagnostics/ docs/ grafana/
  home_assistant/ infra_map/ secrets_mgmt/ validate/ dev/
docs/             # Runbooks, design specs, security notes
  archive/          # Superseded planning docs, incl. the completed Docker → k3s migration
```

> **`roles/containers/` is now only the Pi.** Every role there is a Docker service live on `daniel-pi` — `alloy`, `autoheal`, `docker-proxy`, `wg-easy` — plus the shared `common` deploy path. (`containers_list` in `ansible/inventory/host_vars/daniel-pi.yml` is the source of truth for which are deployed.) A service's config lives in the role that deploys it, on both trees: **if a k3s workload reads it, it is under `roles/k8s/<name>/`**, not across the tree boundary. (Until 2026-08-14 some roles here were config-only sources for a k8s counterpart; they moved into it. To revive one as a Docker service, take its Compose plumbing from git history.)

## Where to Look (task → start here)
Route to the source of truth by what you're doing, before reading linearly. A task a skill
covers — deploying, landing, secrets, Renovate PRs, a GitOps tick, worktree cleanup, a homelab
review — routes through the skill listing rather than this table.

| If you're… | Start here |
|---|---|
| Adding / changing a service (k3s — the default) | `/new-k8s-service` skill (it scaffolds with `scripts/dev/new_k8s_service.py`; no sibling copying) · `ansible/roles/k8s/manifests/CLAUDE.md` for the shared render → apply → queue contract every k8s role includes (`grep -rl k8s/manifests ansible/roles/k8s/*/tasks/` lists them) |
| A PR just merged — what now | `## After a PR Merges — Pull, Deploy, Verify` below (the directive and *When to wait*) · `/land-after-merge` skill (the commands) |
| Picking up an open issue, or working through the backlog | `findings.py next` names what is free to take · `/issue-fanout` skill for several at once · `findings.py claims` shows who holds what, `findings.py reap` releases the claims whose worktree is gone. Claim before you start: `findings.py claim <n> --worktree <branch>`. A claim `next` marks stale is reaped by `claim` itself, so a refusal there means the claim is LIVE — find the other session before you take the issue. |
| A Bash or `kubectl` command keeps prompting, or you need the full permission tables | `## Shell Commands — Shape Them to Auto-Approve` below (summary) · `docs/claude-shell-permissions.md` (full detail) |
| Editing HA automations / lighting / fans | `ansible/roles/k8s/home-assistant/CLAUDE.md` (config and workload both live there; it routes to `docs/` for per-topic behaviour) · `/ha-edit-automation` |
| Answering "what runs here / where / behind what" | `docs/reference/` — generated from the tree by the `docs-refresh` cron, browsable at `docs.local.<domain>`. Services, hosts, secret rotation, scheduled jobs, networking. **Never hand-edit a generated page** (a file carrying a `generated_from:` banner); a hook denies the write. Change the generator instead (`scripts/docs/build_docs.py` lists them). |
| CI fails on `test_every_recorded_atom_hashes_as_recorded` | A `CLAUDE.md` section's support no longer matches what `docs/facts.lock` recorded. The finding names which of six causes: you changed a cited symbol, value, test or marker (`moved`/`missing`); you added a citation to an already-verified section (`unrecorded-atom`); you dropped one (`atom-no-longer-cited`); you cited a marker prefix that now matches twice (`ambiguous`); the interpreter moved (`interpreter-moved`); or a row records a section that cites nothing (`empty-row`), which `fact_status.py forget` drops. The middle two reach CI only when you commit past the `facts-reverify-changed` hook, which re-hashes them in the commit that edits the prose. Edit the section, or `uv run python scripts/dev/fact_status.py verify '<doc>#<heading>'` and commit `docs/facts.lock`. A RENAMED heading reads as `section-gone` instead: verify the new key, then `fact_status.py forget '<old key>'`. `.claude/rules/facts.md` has the citation grammar. |
| Writing a Python module or a test, and deciding where it goes | `.claude/rules/python-layout.md` — the `sys.path` bootstrap a cross-directory import needs, the `tests/` sibling rule and the `ansible/tests/` taxonomy. It loads on its own when you touch a path it governs; read it first when you are choosing the path. |
| Chasing a reliability / monitoring "gap" | The role's `CLAUDE.md` + monitor-bridge `ansible/roles/k8s/monitor-bridge/files/registry.py` (the check registry) **first** — mature setup, most are handled |
| Checking that a service's UI actually renders, not just that its pod is Ready | The `homelab-ui` MCP server (`docs/claude-tooling.md`). `probe.py health` cannot see a broken UI behind a healthy pod. **Grafana is the exception** — it still serves a login page behind Authelia, so drive it with `uv run pytest -m ui -k grafana` rather than by hand. |
| A config edit won't restart the pod (k3s) | A ConfigMap/Secret change alone doesn't roll a Deployment. The general mechanism is the central rollout-restart in `ansible/roles/k8s/manifests/CLAUDE.md`, which fires when a role's rendered manifests change. A role whose pod depends on a file the manifests *don't* carry adds its own `checksum/<thing>` pod annotation instead — monitor-bridge names its `check-script`, rendered in `ansible/roles/k8s/monitor-bridge/templates/deployment.yaml.j2`. |
| A config edit won't recreate the container (Docker) | `ansible/roles/containers/common/CLAUDE.md` (config-change wiring) |
| Starting Claude Code sessions from a phone | `ansible/roles/setup/claude_code/CLAUDE.md` — `claude-rc.service` hosts them. `/remote-control` inside a session and `claude rc` from a shell are different features; only the second creates sessions on demand. |
| Adding / changing a cron that changes state | that role's `CLAUDE.md` *Autonomous-role contract*. Every setup role whose cron or timer changes state carries one, and every setup role has a `CLAUDE.md` (`ansible/tests/repo/test_role_claude_md.py`) — `ansible/tests/setup/test_setup_cron_roles_have_a_contract.py` derives the cron-installing set from `tasks/` and names the heartbeat-only exemptions |
| Recording a finding, fix or improvement you will not do this session | `findings.py open` (flags: `findings.py open --help`) — files a GitHub Issue labelled `claude`, deduped by title/file. `--repo OWNER/NAME` files into another repo's register, such as the dotfiles repo. `findings.py list` is the open register; `docs/reference/backlog.md` renders it. Never `gh issue create` by hand — `.claude/hooks/block-footguns.py:issue_create_by_hand_problem` denies it and names the wrapper. |

## Adding a New Service

**Default to k3s.** A new service belongs in `ansible/roles/k8s/<name>/` unless it must run
on `daniel-pi` (LAN-only utilities, WireGuard) — `daniel-server` and `daniel-box` have no
Docker at all, so a Compose role there deploys nothing.

The step-by-step for each platform is a skill, because it is a procedure you follow once per
service rather than a fact you need in every session:

- **k3s** → the `new-k8s-service` skill. It opens with
  `scripts/dev/new_k8s_service.py`, which writes the role skeleton and the `containers_list`
  entry rather than leaving you to copy a sibling (#2855); the skill then covers secrets, the
  two censuses a new role joins, and what a `--dry-run` does *not* prove about a brand-new
  service.
- **Docker on `daniel-pi`** → the `new-container` skill. It carries the canonical compose
  skeleton, the shared macros in `ansible/templates/`, and the `common_config_changed`
  wiring a bind-mounted config file needs.

Two facts stay here because they bite from outside those procedures:

- **`containers_list` position does not matter for a k8s entry.** `ansible/deploy.yml`
  toposorts the k8s play (`ansible/filter_plugins/toposort.py`) from edges the role's own
  templates and entry declare; an ordering no template carries goes in `depends_on:` on the
  entry, never in its list position. The `new-k8s-service` skill has the derivation.
- **`kubectl apply` leaves stale Secret keys behind.** Removing a key from a `secret.yaml.j2`
  does not remove it live.

## Common Commands
Run ansible through `uv run` so it uses the repo's pinned env (`ansible-core` + the
`community.docker` deps `requests`/`docker` — see **Python & Tests**). Bare `ansible-playbook`
(the uv-tool shim) lacks those module deps and deploys will fail.
**Deploy through `./scripts/deploy.sh --tags "<service>"`**, not the playbook directly. A
deploy cannot interleave with any holder of the git-tree lock on the tree, nor with another
deploy of the SAME service on the cluster, while deploys of different services run at once.
`docs/deploying.md` lists the holders. The `deploy` skill has the lock and snapshot mechanics.

**It deploys `HEAD`, not your working tree.** An uncommitted edit is not deployed, and nothing
on the run says so. Commit first. `--check` and `--dry-run` still read the working tree, which
is where an uncommitted edit is meant to be exercised.

**A failing run says what its exit code means — read its last two lines rather than looking
the number up.** Which exits deployed nothing and which left changes live is owned by
`docs/deploying.md`'s *Exit codes are resume points*. The Pi's `-e target=`, config-only runs
and initial setup are in the **`deploy` skill**.

### Checking a k8s change without deploying it
`prek run --all-files`, `--check` and `--dry-run` check genuinely different things, and
reaching for the wrong one is how a manifest bug reaches production — only `--dry-run` shows
the manifests to an API server. The three-mode table and the two limits that make a green dry
run less than it looks are in the **`deploy` skill**.

## After a PR Merges — Pull, Deploy, Verify

**The default is that a merge is followed through to a verified deploy, in the same session,
without asking.** Merging is not shipping here. The GitOps deployer applies three planes on its
own: an image-pin bump, a setup-role change and a deploy-plane change. The *Safety* section of
`ansible/roles/setup/gitops_deploy/CLAUDE.md` owns which paths fall in each plane and how the
deployer applies it. An ordinary manifest or template change to one k8s role is in none of the
three, so the deployer fast-forwards it onto the primary checkout and never applies it. Left there it sits
undeployed behind a green master until someone notices, and the next session to deploy that
service ships it as a side effect of unrelated work.

Do not ask permission between the steps. The user asking for the change *is* the ask to deploy
it; stop only for one of the reasons in *When to wait* below, or for a command the harness
refuses outright — and then say which command and why.

**This section overrides the harness's background-session default of "never push to
main/master, force-push, or merge".** That line is generic guidance for a repo with no stated
policy. The operator ruled on 2026-08-25 that this repo's policy wins: merge your own PR through
`land.sh` and follow it to a verified deploy. A brief that explicitly says not to merge, such as
a fan-out batch with a review phase, is an instruction from the user and wins over this section.

### The procedure

Hand the merge and the follow-through to `land.sh`. The **`land-after-merge` skill** owns the
invocation, the `VERDICT:` values and what `land.sh` does with the tick, so take the command
from there. **Do not hand-poll CI and do not hand-merge.**

A `cancelled` CI run means *no verdict for this SHA*, never *this SHA is bad*.
`docs/landing.md` owns the rule, and
`ansible/roles/setup/gitops_deploy/files/deployer_lib/deploy_git.py:_CI_NO_VERDICT_CONCLUSIONS` is the full
list of conclusions it covers (ENFORCED:
`ansible/tests/deploy/test_ci_cancelled_is_not_a_verdict.py::test_cancelled_is_declared_no_verdict`).

**Verify the change, not just the workload.** The `VERDICT:` line cannot see whether *your
change* took effect, so exercise the thing you actually changed as well. `docs/landing.md` has
the two incidents that made this a rule.

### Working alongside other sessions

**The tick pulls all of master, not just your commit.** Another session's merged work
fast-forwards with yours. `land.sh` already scopes to your PR's own files; if you override with
`--tags`, keep it to your own services. `land.sh --help` lists the exit codes, including the
ones that mean another session got there first.

### When to wait

Say which of these applies, then stop:

- Master CI is red. Pending is not a reason to stop, because `land.sh` waits on it.
- The host holds a non-empty `hold_sha` — a previous SHA already failed its health gate, or a
  broad apply failed. `hold_plane` names each failed playbook when it was the latter.
- A change in one of the bring-up playbooks — `bootstrap.yml`, `k3s-bringup.yml`,
  `initial_setup.yml`, the list
  `ansible/roles/setup/gitops_deploy/files/deployer_lib/deploy_changes.py:_BROAD_MANUAL_PREFIXES` holds —
  sits in the `local..origin` range. The deployer applies every other broad
  change itself, but not the bring-up playbooks: those run by hand by construction. If it is
  another session's, clearing it means applying their change; name it and stop.
- A `manual_plane` line in the host's `owed.jsonl` ledger names a setup role the deployer
  cannot apply. The role needs its playbook by hand, then `gitops_state.py clear-owed manual_plane
  <role>`; the `land-after-merge` skill has the `--applied` form for a narrowed apply. Another
  session's is theirs to clear.
- `deploy.sh` exits 3: the change is broad (shared templates, inventory, the setup plane) and
  maps to no single service.
- The change is docs- or `tasks/`-only — the deployer skips those deliberately, and so do you.
- The change is in the policies-first ordering plane, where a partial run leaves the exact state
  the ordering exists to prevent.
- A classifier denial. Report the denied command; do not reach for a bypass flag.

## Shell Commands — Shape Them to Auto-Approve
Write exploratory commands so they auto-approve; expect a prompt for the rest.

- **Auto-approves:** read-only single commands and pipelines (`grep … | sort | head`), read-only
  stages sequenced with `;`, `&&`, `||` or newlines, write-free redirects, and those same
  read-only commands run over `ssh daniel-server` / `ssh daniel-pi`.
- **Forces a prompt:** command substitution — `$(…)`, backticks, `${…}` — rejected outright;
  shell control flow (`for`/`while` loops, `if/then/else/fi`); anything that writes or execs
  (`> file`, `tee`, `sed -i`, subshells `(…)`, backgrounding `&`).
- Restructure rather than loop: one `grep`/`find`/`awk` usually replaces the control flow.

### `kubectl` — what actually decides
Plain `kubectl` authenticates as a **read-only ServiceAccount**, so RBAC refuses every write
verb, and `sudo` and `kubectl delete` are denied outright. **Ansible is the only write path to
this cluster**: deploy through `./scripts/deploy.sh --tags <svc>`.

Full tables, per-verb tiers, hook wiring and the RBAC evidence: `docs/claude-shell-permissions.md`.

## Claude Tooling in This Repo (`.claude/`)
One directive per tool. `docs/claude-tooling.md` is the full reference, and a hook that
denies prints its own reason.

- **`scripts/diagnostics/probe.py`** — read-only homelab diagnostics, allow-listed (no prompt);
  `--list` prints every subcommand. `monitors` answers "what is down"; **`kuma-drift` answers
  "what is missing"**. **`health <svc>` is the k8s post-deploy gate**: its argument is a deploy
  TAG, and it gates the PRODUCTION cluster unless you pass `--cluster stage`. `ha automation
  <id-or-alias>` resolves the alias-slug≠id trap.
- **`homelab-ui` MCP server** — a headless Chromium, driven through Traefik, that can *see* a
  service's UI — the half `probe.py health` structurally cannot cover.

## Review & Memory Hygiene (making judgment cumulative)
The rules for promoting a review learning into memory live in the `memory-consolidation` and
`homelab-review` skills, which run at the moment they apply.
- **Mark a settled trade-off `# DECIDED:` at the line it governs.** A decision recorded only in a
  memory file or a commit message is a decision every future reviewer re-derives; one written as a
  comment where the code makes the trade-off is one they trip over before they spend an hour on it.
  Write the marker, then the reasoning, and point at the long form rather than at a line number
  that drifts. Cite a marker by its text, not its line: `grep -rn 'DECIDED: `origin\[:8\]`'
  ansible/roles/setup/gitops_deploy/files/` finds the live example wherever it lands, and it
  reads `a fixed slice while volume-snapshot names with --short=8, a MINIMUM width … Full
  analysis in this role's CLAUDE.md.` Reviewer briefs grep for it
  (`.claude/skills/homelab-review/SKILL.md`, step 3), so the marker is what carries the decision to
  the agent that would otherwise re-open it. It is a prior, not a verdict: contradict one with new
  evidence at a cited `file:line` and name the marker you are contradicting.
- **A decision recorded in a Claude artifact moves into `docs/` in the PR that acts on it.**
  `prune-artifacts.sh` deletes an artifact 30 days after its last update, and artifacts are in
  neither git nor the break-glass bundle. When an artifact holds a plan or a decision that
  outlives the session, write its substance into `docs/` or `docs/adr/` and keep the artifact as
  a rendering. `docs/claude-agent-user.md` is the worked example (#3657).
- **A finding you will not fix this session is filed, not remembered.** Run `findings.py open`
  before the session ends (flags: `findings.py <cmd> --help`; the `homelab-review` skill covers
  the review-specific ones) — for a review finding, a remediation the fix-skeptic refused, or an
  improvement you noticed while doing something else. List the issue numbers in the end-of-job
  report. A memory file, `PLANS.md`, or a "3 pending" line in the session notes has no status
  field and no owner; an issue closes from the fixing PR's `Closes #<n>`. Refuted findings close
  with `--refuted --reason` and a true finding the operator decided to live with closes with
  `--accepted --reason`; the wrapper refuses to reopen either, so the do-not-re-flag list does
  not carry them. Open items are `findings.py list`'s register, not a memory file's.

## Parallel Claude Sessions
Several sessions work this repo at once, each in its own `.claude/worktrees/<name>` checkout.

- **One worktree per session, one PR per session.** Need a second piece of work? Open a
  second worktree rather than stacking it on the first.
- **Name the worktree the branch slug you want.** `EnterWorktree` derives the branch from
  the worktree name (prefixing `worktree-`), so `containers-role-cleanup` gets a readable
  branch and `cleanup` doesn't. Accept the prefix; renaming the branch afterwards only
  detaches it from the worktree git has registered.
- **The SessionStart banner lists the other live sessions** and the paths each has changed
  vs `origin/master`. It's derived from `git worktree list` and `/proc`, not from anything a
  session declares, so check it before editing a file several sessions are near (`CLAUDE.md`,
  `group_vars/`, a shared role) rather than assuming you're alone.
- **Never pass `discard_changes` to `ExitWorktree`** when it refuses a merged worktree. That is
  how unlanded work is lost. Retiring a worktree is the `worktree-cleanup` skill.

## Secrets Management
- Secrets live in `ansible/vars/secrets.yml`, encrypted with SOPS + age. Edit them with `sops`
  or the `/add-secret` skill.
- `ansible/.sops.yaml` (tracked — public keys only) lists the age recipients new/updated
  secrets are encrypted to, and auto-encrypts any `.yml`/`.yaml` in `vars/` or `secrets/`
  directories (SOPS searches upward from the file, so this lives at `ansible/`, not root)
- At runtime, `community.sops.sops_decrypt` lookup decrypts values
- **After adding a secret, run `uv run python scripts/secrets_mgmt/secret_rotation.py sync`
  and commit.** `ansible/secret_rotation.yml` is the rotation registry. The tiers, the DANGER
  `pinned` procedure, and how to rename or retire a key without resetting its clock are in
  `docs/secret-rotation.md`.
- **`git diff ansible/vars/secrets.yml` prints plaintext credentials.** `.gitattributes:1` sets
  `diff=sops`, so git decrypts the file before diffing it, and a value exposed that way needs
  rotating. To see THAT it changed without seeing anything it holds, use `git diff --stat
  ansible/vars/secrets.yml` or `--name-only`. To see what actually changed, open it with `sops
  ansible/vars/secrets.yml`. Do not filter the plaintext through a pipe; the deny hook refuses
  it, and `docs/claude-shell-permissions.md` has why.
- **An agent never decrypts `ansible/vars/secrets.yml`.** The user-level guard denies every
  `sops` decrypt form, `--extract` included, because the plaintext lands in the transcript.
  Key names stay plaintext in the encrypted file, so `grep -c '^<name>:'` on it checks that a
  key exists, and `grep -c '^<name>: ENC\['` checks that its value is ciphertext; both print
  a count, never a value. A runbook step that decrypts is marked operator-only.
  `docs/claude-shell-permissions.md` has the guard's location.
- **Never commit plaintext secrets** (private age keys never leave `~/.config/sops/age/keys.txt`;
  `.gitignore` blocks `keys.txt`/`*.agekey`/`*.key` and gitleaks scans every commit)
- **A host that can't decrypt yet** fails `initial_setup.yml`/`deploy.yml` at their
  secret-load `pre_task`, which reads as a playbook bug rather than a missing key. The
  **`add-secret` skill** has the onboarding through `ansible/bootstrap.yml`.

## Pre-commit Hooks
The repo uses [prek](https://prek.j178.dev) (config: `prek.toml`) with YAML linting, Ansible linting, and gitleaks (secret scanning).
Run `prek run --all-files` to check before committing. **That does not run the suite.** No
hook runs it, neither at commit nor at push. The one exception is `static-ratchet-tests`: at
commit time it runs eight static ratchet, lock and census test files, which caused most of the
PR reds that the author's own code did not (#3605, #4008). CI's `pytest` job runs the suite whole on every
PR, and `land.sh` waits on that job before it merges. To see a red test before CI does, run
`uv run pytest` yourself. A pre-push hook that re-ran the suite was removed on 2026-10-02: it
cost 89s on an idle daniel-box and 235s under load on every push, for a verdict CI repeats
in about 61s.
The `validate-compose-templates` hook shells out to `uv` (see **Python & Tests**), so uv must be
installed for a full `prek run`. `mkdocs-strict` builds the docs site the way the cron does
and fails on a broken `--8<--` include or a dead internal link; its site directory is covered
by `.gitignore`'s root deny, which is load-bearing — one untracked file parks the deployer.

## Python & Tests
Dev/test tooling is managed by [uv](https://docs.astral.sh/uv/) (`pyproject.toml` + `uv.lock`).
The repo isn't a Python package — `[tool.uv] package = false` makes it a "virtual" project that
only pins the test deps (the `dev` dependency group) and the pytest config.

```bash
# One-time: install uv — https://docs.astral.sh/uv/getting-started/installation/
uv run pytest                 # all repo unit tests (auto-syncs the env from uv.lock first)
uv run pytest scripts         # just one suite
```

- **Suites:** `testpaths` in `pyproject.toml` names every one, for both `uv run pytest` and
  CI's `pytest` job. Where a new test or module goes is `.claude/rules/python-layout.md`.
- **A new check ships with a proof it can go RED**, a named member it must find if it finds its
  subject by pattern, and a measured transport if it reaches over a network.
  `.claude/rules/python-layout.md` has the three rules and the incidents behind them.

CI (`.github/workflows/ci.yml`) runs `prek run --all-files` on every PR and on push to master:
these tests plus lint, template validation, and secret scanning.