Skip to content

Changelog

Infrastructure changes, notable deployments, and configuration milestones. Add an entry here whenever something meaningful changes — hardware, services, or major config.

2026-09

  • Fix: Intermittent drift-check "unreachable" alerts root-caused and closed out (2026-09-12). 8 of the 30 retained drift runs on CT104 had been failing on arrstack, in two distinct modes. Mode 1 — resource contention: the 04:00 drift timer collided nightly with Dockhand's env_update_check (cron 0 4 * * *) on both Docker hosts, arrstack and n8n. That job pulls fresh :latest images and runs grype and trivy over them (env_N_vulnerability_scanner: "both"), taking 323–610 s and pegging a core; Ansible's SSH handshake landed inside the window and returned Connection timed out during banner exchange (arrstack, n8n) or Timeout (12s) waiting for privilege escalation prompt (n8n). The hosts were never down — sshd logged Accepted publickey either side of each miss. Mode 2 — ENOSPC: arrstack's 32 G root hit 98%, so Ansible could not mkdir its per-task tmp dir and reported the host unreachable rather than failed. Backups and reboots were both ruled out as causes: vzdump pbs-daily runs 02:00→02:08–02:14 every day without exception, and both proxfold and arrstack had 84 days uptime. Timer moved 04:00 → 05:00 (verified against every OnCalendar in the 02:00–06:00 band fleet-wide — new schedule map); ansible.cfg gained timeout = 30 (was the 10 s default; the become-prompt timeout derives from it) and retries = 3 so a loaded target degrades into a slower run instead of a red Discord alert. homelab-ansible commit c573778. See Drift Detection § Timing + Troubleshooting.
  • Host: arrstack VM (101) root disk grown 32G → 64G (2026-09-12) after hitting 98% / 762 MB free — the ENOSPC half of the drift failures above. Live online grow, no downtime: qm snapshot (agent freeze/thaw, filesystem-consistent) → qm disk resize 101 scsi0 +32G → growpart /dev/sda 1 (cloud-guest-utils installed; sda1 is the only partition and extends to end of disk, thanks to the 2026-05-20 swap-partition removal) → online resize2fs. Result 63 G / 31 G free / 49%, all 20 containers up, no I/O errors. The stale pre-resize-2026-05-20 snapshot — documented as "kept for 24h" but still present 4 months on — was deleted in the same pass, returning 10.3 G to rpool (vm-101-disk-0 USED 31.2 G → 20.9 G). pre-resize-2026-09-12 retained as the rollback point. Underlying growth mechanism, not yet fixed: Dockhand runs autoUpdate: true nightly on arrstack, but env_1_image_prune is weekly (0 3 * * 0, dangling-only) and fires at 03:00 — an hour before the 04:00 job that creates the garbage — so dangling images bank up all week (last weekly pass reclaimed 2.17 GB / 10 images). See arrstack VM details.

2026-07

  • Milestone: Phase 6H — comics pipeline completed with acquisition + metadata (2026-07-18). Two services added to the komga stack: Kapowarr (mrcas/kapowarr, :5656) for acquisition — ComicVine-matched, GetComics DDL, routed through a third ProtonVPN session via a dedicated gluetun-kapowarr (no port-forwarding — DDL is outbound-only); and komf (sndxr/komf, :8085) as the metadata engine, writing ComicVine/MangaUpdates metadata + covers into Komga over its API. komf is deliberately not tunnelled (metadata APIs only). Rationale for komf: Komga has no online scraper (komga#1577) and Kapowarr doesn't write ComicInfo.xml (Kapowarr#50), so neither populates Komga's metadata alone. New /stash/rodneystash/ComicsDL temp dir (sibling of Comics — Kapowarr's non-intersect rule, same pool for atomic moves). See Kapowarr, komf, roadmap §6H.
  • Change: Komga library mount flipped :ro → :rw (2026-07-09). The read-only guard blocked deleting books/series from disk in the Komga UI; :rw enables in-UI library management (and optional ComicInfo write-back / CBR→CBZ auto-convert). Trade-off: mis-clicks in the UI now delete real files on the stash. Note: clearing entries for files deleted outside Komga still needs Scan library → Empty trash (DB cleanup, mount-mode-independent). See Komga service page.
  • Service: Komga — comic / graphic-novel library + reader added (Phase 6G, 2026-07-06). Own Dockhand stack (stacks/komga/, image gotson/komga) on arrstack VM 101 — own komga network + komga_config named volume, mirroring korrosync/mealie rather than folding into the arr acquisition pipeline. Library is a read-only bind of /stash/rodneystash/Comics (Komga writes only its DB/thumbnails to /config). Public at https://comics.rampancy.cloud via the CT 107 Caddy edge (wildcard cert + CrowdSec cover it automatically). Reads on iPad via Safari web reader or OPDS v2 (Panels/Chunky) and any OPDS-capable device. No acquisition in scope. See Komga service page + roadmap §6G.

2026-06

  • Fix: PBS datastore now survives reboots + alerts on failure (2026-06-20). The in-container NFS mount used x-systemd.automount, which is unsupported inside an LXC, so the datastore never re-mounted after the 2026-06-19 power-cycle and backups silently failed — the 2026-06-20 02:00 run errored and the last good backup was 2026-06-19 02:06 (nas-primary inactive in PVE; chunk-store errors). pbs_nfs_options switched to a plain _netdev,nofail mount; new pbs-datastore-healthcheck.timer (every 15 min) pings #homelab-ops if the mount/.chunks go missing. Host-mount+bind was rejected — the QNAP export is IP-locked to the container. See pbs role + Backup & Restore troubleshooting.
  • Fix: proxmox role now refreshes the ESP loader config after pinning the kernel (2026-06-19). The pin task only wrote /etc/kernel/proxmox-boot-pin and never ran proxmox-boot-tool refresh, and the idempotency probe parsed the wrong kernel list section so it re-pinned every play. A UPS-triggered reboot (first boot in ~8 weeks) exposed the latent loader-default drift by landing proxfold on kernel 6.17.13 instead of the pinned 6.14, dropping Nvidia 550.x / Plex HW transcode until a manual re-pin → refresh → reboot. Probe now reads the Pinned kernel: section; pin task notifies a Refresh proxmox-boot ESPs handler. See proxmox role + proxfold kernel pin.
  • Event: First real-world UPS test passed (2026-06-19). ~03:02 mains loss; CyberPower PR1500ERT2U held the load ~52 min, then NUT triggered a forced shutdown on battery-low at 03:54. All guests shut down cleanly (VM 101 force-terminated after a guest-agent timeout — no filesystem damage); host powered off ~03:58 and auto-booted 04:11; both ZFS pools clean. A prior-night 21:03–21:42 battery run recovered before low-battery.

2026-05

  • Milestone: Phase 6B.1 + 6B.3 — Home Assistant VM stood up + edge integration (2026-05-23). HAOS 17.3 on new VM 110 (hass, 192.168.1.241) on proxfold; Caddy vhost home.rampancy.cloud on edge (homelab-ansible commit b19f662); HA trusted_proxies: 192.168.1.244 + use_x_forwarded_for: true; CrowdSec coverage automatic via the wildcard handler; cellular validation green. Phase 6B.2 (core integrations) deferred — user driving hands-on. Nine scaffold-drift findings folded back into the runbook this cycle (HAOS pin bumped 17.2 → 17.3, no upstream .sha256 sidecar published, Tapo path revised to tplink-primary, Discord notify YAML deprecated split syntax, HEAD vs GET on hold-point probe, qm guest ping → qm agent ping, qm disk resize no-op on the OVA, ~90s actual first boot, and HA-Core-restart verification before declaring trusted_proxies live). See home-assistant-setup runbook Lessons appendix.
  • Change: Matrix update notifications + fleet reboot-required coverage expanded (2026-05-23). New matrix_deploy_notifier role on CT 104 fetches spantaleev's playbook monthly (first Monday 09:00) and POSTs pending upstream commits to the renamed #homelab-updates Discord channel — never pulls, never applies, honours upstream's migration-validation gate. auto_updates_notify_discord: true flipped on arrstack / matrix / n8n (the three VMs that own a kernel) → #homelab-ops. needrestart added to auto_updates fleet-wide in non-interactive 'l' mode to close the libc6/openssl/systemd gap that the home-rolled kernel postinst hook doesn't cover; LXC group_vars suppresses the stale-kernel hint (containers share the host kernel). See matrix-maintenance runbook, matrix_deploy_notifier role, and auto-updates role.
  • Milestone: Phase 6E.4 — MatrixRTC live (2026-05-22). 5 UDM port-forwards (7881/tcp + 7882/udp + 3479/udp + 5350/tcp + 30000-30020/udp → VM 111). Element Call validated end-to-end: desktop ↔ Element X mobile on cellular, audio + video + screen-share. One gotcha: apex .well-known/matrix/client had to be extended to advertise org.matrix.msc4143.rtc_foci (Element Call queries the apex, not the matrix subdomain) — fix baked into Caddy template. Phase 6E now fully complete. See matrix-setup runbook 6E.4 Lessons.
  • Fix: slskd moved off the shared gluetun container onto its own gluetun-slskd (2026-05-22) — separate ProtonVPN WireGuard session, separate NAT-PMP forwarded port. Resolves a silent port-forward collision that had crippled qBittorrent's inbound (every tracker "Unreachable", DHT empty) since the forwarded port happened to coincide with qBit's cached Session\Port. Root cause, diagnostic walk, and architectural rationale (gluetun #2381 still open) in music-acquisition-bringup follow-up and slskd VPN integration.
  • Fix: plex role apt source migrated from downloads.plex.tv/repo/deb → repo.plex.tv/deb/ and key from PlexSign.key (armored, plex.asc) → PlexSign.v2.key (dearmored, plexmediaserver.v2.gpg) (2026-05-22). Old mirror was frozen at 1.42.2.10156-f737b826c (Last-Modified: 2026-01-28); apt update && apt upgrade had stopped surfacing new builds. A previous manual host-side URL fix had been silently reverted on every drift run by the role template. Canonical URLs match Plex KB 235974187. Role's "remove legacy v2 keyring" task — previously deleting what's now the canonical keyring — inverted to remove the old plex.asc instead. See plex role.
  • Milestone: Phase 6E text + federation complete — Matrix homeserver live (2026-05-22). Tuwunel v1.7.0 on VM 111 fronted by edge Caddy at matrix.rampancy.cloud; federation green via apex well-known delegation; @rampancy:rampancy.cloud admin. 6E.4 RTC port-forwards deliberately deferred. Headline gotcha was matrix_tuwunel_config_allowed_remote_server_names filtering our OWN server's events as M_SENDER_IGNORED when the local server name was omitted from the list (variable name says "remote" but implementation applies to all senders) — cost ~8 hours of misdiagnosis. Nine documented lessons including v1.6.2 UIAA regression, MSC4222/MSC3575 noise, ensure-matrix-users-created Synapse-only, and docker role dropped from playbooks/matrix.yml to end the drift war with spantaleev. See matrix-setup runbook Lessons appendix.
  • Milestone: Phase 6E.1 — Matrix VM stood up (2026-05-21). New VM 111 matrix (192.168.1.243) on proxfold, 4 vCPU / 8 GiB / 32 GiB. New matrix.yml playbook (common/security/docker/beszel_agent — no hawser), inventory/host_vars/matrix.yml with the overlay2 pin carried over from n8n. Lessons in matrix-setup runbook.
  • Roadmap: Phase 6E (Matrix server) scoped (2026-05-21) — Tuwunel + LiveKit on VM 111 via vendored spantaleev/matrix-docker-ansible-deploy, fronted by CT 107 Caddy via well-known federation delegation. Closed federation, group voice/video in scope; mobile push, Discord bridge, OIDC deferred. Replaces the original four-line stub. See roadmap §6E + matrix-setup runbook.
  • Roadmap: Phase 6F (Music recommendations / discovery) scoped (2026-05-20) — Plex scrobble → ListenBrainz public → Explo → Plex playlists, missing tracks via the 6D slskd path. Reference-only, no work started. See roadmap §6F.
  • Host: arrstack VM (101) root disk grown 24G → 32G after hitting 86% from organic container growth (2026-05-20); installer's 1G swap partition (sda2/sda5) replaced with /swapfile so sda1 could extend to end of disk. Snapshot pre-resize-2026-05-20 kept for 24h. See arrstack VM details.
  • Follow-up: Phase 6D real-use shakeout findings folded in (2026-05-16 evening) — Tubifarry/slskd search-template + peer-throttle tuning, Lidarr import quirks on the hotio plugins-branch image, samba reload-config gotcha. See music-acquisition-bringup runbook and Lidarr service page.
  • Milestone: Phase 6D complete — music acquisition pipeline (2026-05-16). Lidarr (hotio pr-plugins image) + Tubifarry + slskd (via gluetun) + beets live on arrstack VM 101; existing 612-artist library auto-imported (877 albums, 10,691 tracks); Gotye Making Mirrors Deluxe smoke-tested end-to-end. Lessons in music-acquisition-bringup runbook.
  • Service: slskd, Lidarr (plugins-branch), and beets added to arrstack stack (2026-05-16). See arrstack services.
  • Correction: SMB share consolidated to a single read-write \\192.168.1.250\media (2026-05-15, same day as original add) — the earlier read-only/read-write split hit Windows' SMB MUP one-credential-set-per-server limit (error 1219). See Storage — SMB Export.
  • Service: SMB share added on proxfold (2026-05-15) — \\192.168.1.250\media over /stash/rodneystash for Windows clients via new samba role. Single authenticated user smbmedia, SMB2-minimum. See Storage — SMB Export.
  • Milestone: Phase 5E complete — host-level file backup to PBS (2026-05-06). Daily 02:30 proxmox-backup-client timer captures /etc, /root, /var/lib/pve-cluster into PBS namespace host/proxfold. Three CLI bugs and a PBS 4.x token-ACL footgun caught; pbs role patched same cycle. Lessons in backup-restore runbook.
  • Service: Mealie live on arrstack (2026-05-06) — recipe manager + meal planner + shopping list at http://192.168.1.252:9000. Picked over KitchenOwl on URL importer + first-party HA integration. See Mealie service page.
  • Roadmap: Phase 6B (Home Assistant) scoped (2026-05-06) — four sub-stages: HAOS VM stand-up, core integrations (HACS, Tapo via Matter, Hue V2, Bambu A1 Mini), Caddy edge at home.rampancy.cloud, dashboard + automations. Manual qm create over community-scripts installer to match existing precedent. See roadmap §6B + Home Assistant Setup runbook.
  • Milestone: Phase 6A complete — Forgejo self-hosted git, GitHub-mirrored (2026-05-05). Closes the 6A.1–6A.4 cycle across 2026-05-04/05; CF DNS token Rolled (closes the 7D leak follow-up); Discord push webhooks deferred as low-signal for solo-dev. See forgejo-setup runbook.
  • Milestone: Phase 6A.3 — 4 repos imported with full history + per-repo push-mirrors to GitHub at ~5s sync_on_commit (2026-05-05). Local origins flipped on WSL, github kept as fallback. Lessons in forgejo-setup runbook.
  • Milestone: Phase 6A.2 — Forgejo public via Caddy at git.rampancy.cloud (2026-05-05). CrowdSec coverage automatic via the existing wildcard handler. Lessons in forgejo-setup runbook.
  • Drift cleanup: Five drift items reconciled across proxfold + edge + forgejo (2026-05-05). NVENC patch repo pinned to commit 80e48e9 (was tracking master); new convention recorded — bring up new hosts with site.yml --limit <newhost> rather than per-host playbooks. See playbooks doc and nvidia role.
  • Milestone: Phase 6A.1 — Forgejo LXC stood up (2026-05-04). New forgejo role on CT 109 (192.168.1.249), forgejo-sqlite 11.0.13 from forgejo-contrib APT. Two execution-time bugs caught (apt_repository → deb822 sources on Debian 13 LXCs; INSTALL_LOCK grep instead of app.ini existence). Lessons in forgejo-setup runbook.
  • Milestone: Phase 7D complete — CrowdSec live on edge (2026-05-04). New crowdsec_engine role on CT 107 + hslatman bouncer module via xcaddy rebuild. End-to-end blocked/allowed cellular validation passed. Six execution-time bugs caught — all in the crowdsec-validation runbook. Edge security gap accepted risk closed same day.
  • Roadmap: Phase 7D scope cut (2026-05-04) — Wazuh forwarding deferred to 7A/B (gated on Phase 4B); Lynis split out and earmarked for the Wazuh piece.
  • Roadmap: Phase 7E (Pocket-ID identity + selective SSO) scoped (2026-05-03), bundled with 7D as the edge-hardening cycle. Authentik considered and deferred — Postgres/Redis footprint doesn't fit the lean ethos. See roadmap §Phase 7E.
  • Decom: VM 102 (nginx) destroyed (2026-05-03) after 24h post-cutover soak. Pre-decom vzdump on nasbackup (1.65 GB compressed). Closes Phase 5D.
  • Milestone: Phase 5D complete — edge LXC live, NPM retired (2026-05-02). New caddy role on CT 107 (edge, 192.168.1.244) replaces NPM on VM 102. Caddy serves a Let's Encrypt wildcard *.rampancy.cloud via DNS-01 against Cloudflare; four hosts migrated. CF orange-cloud attempted then rolled back. Lessons in edge-cutover runbook.
  • Milestone: Housemate access onboarded on proxfold (2026-05-01) — hazel@pve + housemate-lab pool + stash/housemate-vms (500 GiB quota) + ACLs. Two CLI bugs in the original runbook caught + corrected. Lessons in housemate-access runbook.
  • Roadmap: Phase 8 (network segmentation — VLAN scheme + PENFOLD-SW01 SwOS→RouterOS migration) added (2026-05-01). Future L2-isolation home for housemate VMs; immediate Phase 6 access ships on vmbr0 with Proxmox-side controls only. See roadmap §Phase 8.

2026-04

  • Docs: Vintage Story update procedure added (2026-04-29) — manual binary swap, no upstream update script. Captures the gotcha hit on first patch update: upstream server.sh ships vintagestory-keyed defaults that clobber the vintage-keyed customisations on extract; server.sh start then errors Username, Group or data path missing. New section at services/vintagestory#updates.
  • Service: korrosync stack added on arrstack (2026-04-29) — self-hosted KOReader progress sync server for Kobo Clara BW ↔ XTEINK X4 (Crosspoint Reader). LAN-only HTTP at 192.168.1.252:3030; reverse-proxy / TLS deferred until edge-proxy decision lands. See korrosync service page. Port-collision lesson recorded — upstream's default :3000 clashed with Dockhand's UI on the same host.
  • Roadmap: Phase 4B RAM default flipped to 8× 32 GB / 256 GB at 1 DPC (2026-04-28); 12× 16 GB and 8× 16 GB demoted to alternatives. Hynix recommended (matches existing chassis sticks). Part numbers + decision rationale in roadmap §Phase 4B.
  • Docs: Site-wide audit against live state — every page walked vs proxfold + 8 guests (2026-04-28). PVE point release, post-4C boot tooling, missing services + ports, post-Phase-5 ansible scaffolding, 8-agent Beszel fleet, broken anchor in ansible/index.md, r430 runbook RAM line + fan-count clarification.
  • Roadmap: Phase 4B re-scoped from 384 GB → 192 GB and Phase 7 (security stack — Wazuh AIO + Suricata + CrowdSec + Lynis) added, gated on 4B (2026-04-28). See roadmap §Phase 4B + §Phase 7.
  • Milestone: Phase 5C complete — n8n live as a Docker stack (2026-04-28). VM 108 (n8n, 192.168.1.248) cloud-inited; new hawser role codifies Dockhand's remote-host agent. Pivoted mid-execution from npm-on-LXC to Docker-on-VM; lessons in services/n8n.md.
  • Fix: Stopped duplicate PVE backup notifications in #homelab-ops (2026-04-27) by disabling the built-in default-matcher (mail-forward loop was re-firing into ops-all). Codified in roles/proxmox/tasks/notifications.yml.
  • Service: Vintage Story dedicated server publicly exposed at vintage.rampancy.cloud:42420 (2026-04-26). UDM port-forward, no reverse proxy (VS doesn't speak PROXY). Lessons in services/vintagestory.
  • Service: Vintage Story 1.22 dedicated server stood up — CT 201 (192.168.1.235), manually installed, side-project (2026-04-26). Host-level config Ansible-managed via playbooks/vintage.yml; VS install + mods deliberately out of scope while mods are in flux. See Vintage Story.
  • Quality: homelab-ansible handler casing sweep + baseline regen (2026-04-25). 22 handler names retitled; .ansible-lint-ignore shrank 26 → 15 entries (45 → 23 violations).
  • Quality: homelab-ansible lint profile ratcheted to production in three same-day commits (2026-04-25). Pre-commit hook now enforces it. See Linting & quality gates.
  • Infra: auto_updates role added — fleet-wide unattended-upgrades wrapping hifis.toolkit.unattended_upgrades (2026-04-25). Security-only, no auto-reboot; proxfold blocklists PVE kernels; opt-in Discord nag for pending reboots. See auto_updates role page.
  • Docs: Thermal baseline captured for proxfold (2026-04-25) — idle 74 °C, sustained 28-thread load 91–92 °C, no throttle events at 22 °C inlet. See proxfold proxmox page.
  • Extension: PBS-side Discord notifications codified via roles/pbs/tasks/notifications.yml (2026-04-25). Verify/GC/prune/sync events now fan out to #homelab-ops alongside PVE-side events.
  • Fix: Two post-5B regressions caught on the first live drift + backup cycle (2026-04-25). control switched to ansible_connection: local; PVE 9 Discord webhook body trimmed to title/severity/footer (full {{ message }} overflows the 4096-char embed limit).
  • Milestone: Phase 5B complete — server notification stack live (2026-04-24). Three paths into #homelab-ops: Beszel hub on CT 106, ZED webhook on proxfold, PVE 9 notification target. CT 104 (control) brought under Ansible management.
  • Milestone: Phase 5A complete — Proxmox Backup Server live (2026-04-23). PBS 4.x in CT 105 on proxfold, datastore nas-primary on NFSv3 from TS-269L. Daily 02:00 pbs-daily job; first full backup clean (~95 GiB after dedup). Codified in pbs role + proxmox/pbs_client.yml. Gotchas in pbs role page.
  • Milestone: Phase 4C complete — proxfold boot drive swapped to ZFS RAID1 mirror (2026-04-22). 128GB Samsung 840 PRO replaced by 2× 960GB SSDs (SM843T + Intel DC S4500); all four guests restored from NAS vzdumps; Plex HW transcode re-verified. Lessons in Boot Drive Swap runbook.
  • Infra: Phase 3D complete — scheduled drift detection live on CT104. Daily 04:00 ACST timer posts to #homelab-drift; clean runs stay silent. First live run surfaced three role bugs — all fixed in the same PR. See Drift Detection.
  • Docs: Proxfold rebuild runbook added — end-to-end auto-install procedure backed by the rebuild/ kit; WSL dual-control bootstrap documented as the DR cold-start path. proxmox and nut role pages added.
  • Infra: Phase 3C complete — Proxmox auto-install answer rendering + ISO builder merged to homelab-ansible/rebuild/. Nested VM rehearsal validated rendering, vault, disk filter fail-safe, and boot order.
  • Infra: Phase 3B complete — Plex data codification (ZFS stash/plex-data + LXC mp1 + symlink) merged into the plex role with delegate_to for the host side.
  • Infra: Phase 3A complete — proxmox and nut roles merged, adding PVE host baseline (deb822 repos, kernel pin, nouveau blacklist, sysctl, stash import, nasbackup CIFS) and NUT codification.
  • Milestone: stash pool expanded from 4-wide to 6-wide RAIDZ1 via OpenZFS 2.3 raidz_expansion (2026-04-20). Two Samsung PM1633a 3.84TB SAS SSDs attached live; 14.0T/91% → 21.0T/61%. See PVE 9 Upgrade runbook.
  • Milestone: proxfold upgraded from PVE 8.4.18 → 9.1.7 (bookworm → trixie), in-place. Kernel 6.14.11-6-pve GRUB-pinned, Nvidia 550.163.01 via DKMS, NVENC keylase patch reapplied; Plex HW transcode re-verified. Lessons in PVE 9 Upgrade runbook.
  • Infra: Nvidia cgroup2 device majors in CT 100 updated 235→234 (nvidia-uvm) and 238→237 (nvidia-caps) after the kernel swap. Silent failure mode documented (nvidia-smi keeps working via static major 195 but CUDA dies).
  • Infra: NIC names on proxfold pinned to stable nic0–nic3 via systemd .link files (MAC match), replacing kernel-assigned eno1–eno4.
  • Infra: Plex LXC (CT 100) given features: mount=nfs,nesting=1 to silence the systemd-252 AppArmor denial storm. Backup at /root/100.conf.bak-pre-nesting.
  • Infra: CyberPower PR1500ERT2U UPS deployed on proxfold — NUT 2.8.0 in standalone mode, battery transfer test passed cleanly. Phase 4A pulled forward. See UPS.
  • Docs: PVE 9 upgrade runsheet added — covers PVE 8.4→9.x upgrade, post-upgrade verification, and ZFS RAIDZ expansion. Post-execution appendix captures 8 deviations between runbook and actual run.
  • Docs: Host/runbook/reference docs reconciled with PVE 9.1.7 reality — repos (bookworm→trixie), GRUB kernel pin, proxmox-headers meta-package name, ZFS 2.4 userland / 2.3.4 kmod split, dynamic cgroup majors, LVM global_filter.
  • Infra: Plex data directory migrated from CT 100 rootfs to stash/plex-data (100G quota) via symlink — rootfs dropped from 87% to 34% usage.
  • Infra: Plex LXC rootfs resized from 16GB to 32GB (live resize, no downtime).
  • Infra: Weekly fstrim cron job added to proxfold for all running LXC containers (reclaims thin-provisioned LVM space).
  • Infra: CT 103 (stash) destroyed — legacy SMB bridge, retired March 2026, IP 192.168.1.251 freed.
  • Docs: Added implementation roadmap with fact-checked phase details.
  • Docs: Added Ansible section documenting homelab-ansible repo (all 10 roles).
  • Docs: Homelab-ansible repo scaffolded with inventory, playbooks, and all roles — control node deployment pending (Phase 1 of roadmap).
  • Milestone: Ansible control node in place and configured and tested common tag. Phase 1A complete.
  • Milestone: Portainer sunsetted and main Docker Compose stack managed by Dockhand. Nginx to be moved at a later date pending Caddy move. Phase 1B complete.

2026-03

  • Docs: Full boot drive swap runbook created — cross-referenced with storage, GPU, and Ansible sections
  • Docs: Added Proxmox repository config and IPMI fan control sections
  • Docs: ZFS ARC cap corrected to 14 GB (15032385536 bytes) across all docs
  • Service: Overseerr replaced in production with Seerr (ghcr.io/seerr-team/seerr:latest)
  • Service: MediaBot NLP branch merged to main — @mention natural language interface now live via Claude Sonnet; ANTHROPIC_API_KEY required; Discord Message Content Intent enabled
  • Docs: Added UniFi Dream Machine (The-Egg) documentation — network, Wi-Fi, firewall, security, VPN sections
  • Docs: Added PENFOLD-SW01 (MikroTik CRS326-24G-2S+RM) documentation
  • Docs: GPU passthrough section added to Plex docs (Nvidia T400, cgroup2 device allowances)

2025 and earlier

Initial stack deployment

  • Service: Proxmox VE installed on Dell PowerEdge R430 (proxfold, 192.168.1.250)
    • ZFS pools: rpool (boot, single drive — mirror upgrade planned), stash (RAIDZ1, 4× SAS SSD)
    • LXC 100: Plex Media Server with Nvidia T400 GPU passthrough
    • VM 101: arrstack (Docker host)
    • VM 102: nginx (Nginx Proxy Manager, 192.168.1.249)
  • Service: Media stack deployed on arrstack — Sonarr, Radarr, Prowlarr, qBittorrent
  • Service: Seerr deployed (media request management)
  • Service: MediaBot deployed — Discord bot for media pipeline management
  • Network: UniFi Dream Machine deployed as gateway/WAP (The-Egg, 192.168.1.1)
  • Network: MikroTik CRS326-24G-2S+RM deployed as managed switch (PENFOLD-SW01, 192.168.1.3)
  • Network: WireGuard VPN server EggLink active (UDP 51820)
  • Network: DoH via Cloudflare, IPS active mode, DPI enabled on UDM

Upcoming

See the Roadmap for planned phases:

  • Phase 1: Ansible control node + Dockhand + gluetun
  • Phase 2: PVE 9 upgrade + ZFS RAIDZ expansion
  • Phase 3: Full Ansible codification of all hosts
  • Phase 4: UPS, CPU 2 + RAM upgrade, boot drive swap to ZFS mirror
  • Phase 5: n8n automation, vulnerability management
  • Phase 6: Home Assistant, Obico, music library, Matrix (optional)