Skip to content

Changelog

Infrastructure changes, notable deployments, and configuration milestones. Add an entry here whenever something meaningful changes — hardware, services, or major config.

2026-05

  • Milestone: Phase 6B.1 + 6B.3 — Home Assistant VM stood up + edge integration (2026-05-23). HAOS 17.3 on new VM 110 (hass, 192.168.1.241) on proxfold; Caddy vhost home.rampancy.cloud on edge (homelab-ansible commit b19f662); HA trusted_proxies: 192.168.1.244 + use_x_forwarded_for: true; CrowdSec coverage automatic via the wildcard handler; cellular validation green. Phase 6B.2 (core integrations) deferred — user driving hands-on. Nine scaffold-drift findings folded back into the runbook this cycle (HAOS pin bumped 17.2 → 17.3, no upstream .sha256 sidecar published, Tapo path revised to tplink-primary, Discord notify YAML deprecated split syntax, HEAD vs GET on hold-point probe, qm guest ping → qm agent ping, qm disk resize no-op on the OVA, ~90s actual first boot, and HA-Core-restart verification before declaring trusted_proxies live). See home-assistant-setup runbook Lessons appendix.
  • Change: Matrix update notifications + fleet reboot-required coverage expanded (2026-05-23). New matrix_deploy_notifier role on CT 104 fetches spantaleev's playbook monthly (first Monday 09:00) and POSTs pending upstream commits to the renamed #homelab-updates Discord channel — never pulls, never applies, honours upstream's migration-validation gate. auto_updates_notify_discord: true flipped on arrstack / matrix / n8n (the three VMs that own a kernel) → #homelab-ops. needrestart added to auto_updates fleet-wide in non-interactive 'l' mode to close the libc6/openssl/systemd gap that the home-rolled kernel postinst hook doesn't cover; LXC group_vars suppresses the stale-kernel hint (containers share the host kernel). See matrix-maintenance runbook, matrix_deploy_notifier role, and auto-updates role.
  • Milestone: Phase 6E.4 — MatrixRTC live (2026-05-22). 5 UDM port-forwards (7881/tcp + 7882/udp + 3479/udp + 5350/tcp + 30000-30020/udp → VM 111). Element Call validated end-to-end: desktop ↔ Element X mobile on cellular, audio + video + screen-share. One gotcha: apex .well-known/matrix/client had to be extended to advertise org.matrix.msc4143.rtc_foci (Element Call queries the apex, not the matrix subdomain) — fix baked into Caddy template. Phase 6E now fully complete. See matrix-setup runbook 6E.4 Lessons.
  • Fix: slskd moved off the shared gluetun container onto its own gluetun-slskd (2026-05-22) — separate ProtonVPN WireGuard session, separate NAT-PMP forwarded port. Resolves a silent port-forward collision that had crippled qBittorrent's inbound (every tracker "Unreachable", DHT empty) since the forwarded port happened to coincide with qBit's cached Session\Port. Root cause, diagnostic walk, and architectural rationale (gluetun #2381 still open) in music-acquisition-bringup follow-up and slskd VPN integration.
  • Fix: plex role apt source migrated from downloads.plex.tv/repo/deb → repo.plex.tv/deb/ and key from PlexSign.key (armored, plex.asc) → PlexSign.v2.key (dearmored, plexmediaserver.v2.gpg) (2026-05-22). Old mirror was frozen at 1.42.2.10156-f737b826c (Last-Modified: 2026-01-28); apt update && apt upgrade had stopped surfacing new builds. A previous manual host-side URL fix had been silently reverted on every drift run by the role template. Canonical URLs match Plex KB 235974187. Role's "remove legacy v2 keyring" task — previously deleting what's now the canonical keyring — inverted to remove the old plex.asc instead. See plex role.
  • Milestone: Phase 6E text + federation complete — Matrix homeserver live (2026-05-22). Tuwunel v1.7.0 on VM 111 fronted by edge Caddy at matrix.rampancy.cloud; federation green via apex well-known delegation; @rampancy:rampancy.cloud admin. 6E.4 RTC port-forwards deliberately deferred. Headline gotcha was matrix_tuwunel_config_allowed_remote_server_names filtering our OWN server's events as M_SENDER_IGNORED when the local server name was omitted from the list (variable name says "remote" but implementation applies to all senders) — cost ~8 hours of misdiagnosis. Nine documented lessons including v1.6.2 UIAA regression, MSC4222/MSC3575 noise, ensure-matrix-users-created Synapse-only, and docker role dropped from playbooks/matrix.yml to end the drift war with spantaleev. See matrix-setup runbook Lessons appendix.
  • Milestone: Phase 6E.1 — Matrix VM stood up (2026-05-21). New VM 111 matrix (192.168.1.243) on proxfold, 4 vCPU / 8 GiB / 32 GiB. New matrix.yml playbook (common/security/docker/beszel_agent — no hawser), inventory/host_vars/matrix.yml with the overlay2 pin carried over from n8n. Lessons in matrix-setup runbook.
  • Roadmap: Phase 6E (Matrix server) scoped (2026-05-21) — Tuwunel + LiveKit on VM 111 via vendored spantaleev/matrix-docker-ansible-deploy, fronted by CT 107 Caddy via well-known federation delegation. Closed federation, group voice/video in scope; mobile push, Discord bridge, OIDC deferred. Replaces the original four-line stub. See roadmap §6E + matrix-setup runbook.
  • Roadmap: Phase 6F (Music recommendations / discovery) scoped (2026-05-20) — Plex scrobble → ListenBrainz public → Explo → Plex playlists, missing tracks via the 6D slskd path. Reference-only, no work started. See roadmap §6F.
  • Host: arrstack VM (101) root disk grown 24G → 32G after hitting 86% from organic container growth (2026-05-20); installer's 1G swap partition (sda2/sda5) replaced with /swapfile so sda1 could extend to end of disk. Snapshot pre-resize-2026-05-20 kept for 24h. See arrstack VM details.
  • Follow-up: Phase 6D real-use shakeout findings folded in (2026-05-16 evening) — Tubifarry/slskd search-template + peer-throttle tuning, Lidarr import quirks on the hotio plugins-branch image, samba reload-config gotcha. See music-acquisition-bringup runbook and Lidarr service page.
  • Milestone: Phase 6D complete — music acquisition pipeline (2026-05-16). Lidarr (hotio pr-plugins image) + Tubifarry + slskd (via gluetun) + beets live on arrstack VM 101; existing 612-artist library auto-imported (877 albums, 10,691 tracks); Gotye Making Mirrors Deluxe smoke-tested end-to-end. Lessons in music-acquisition-bringup runbook.
  • Service: slskd, Lidarr (plugins-branch), and beets added to arrstack stack (2026-05-16). See arrstack services.
  • Correction: SMB share consolidated to a single read-write \\192.168.1.250\media (2026-05-15, same day as original add) — the earlier read-only/read-write split hit Windows' SMB MUP one-credential-set-per-server limit (error 1219). See Storage — SMB Export.
  • Service: SMB share added on proxfold (2026-05-15) — \\192.168.1.250\media over /stash/rodneystash for Windows clients via new samba role. Single authenticated user smbmedia, SMB2-minimum. See Storage — SMB Export.
  • Milestone: Phase 5E complete — host-level file backup to PBS (2026-05-06). Daily 02:30 proxmox-backup-client timer captures /etc, /root, /var/lib/pve-cluster into PBS namespace host/proxfold. Three CLI bugs and a PBS 4.x token-ACL footgun caught; pbs role patched same cycle. Lessons in backup-restore runbook.
  • Service: Mealie live on arrstack (2026-05-06) — recipe manager + meal planner + shopping list at http://192.168.1.252:9000. Picked over KitchenOwl on URL importer + first-party HA integration. See Mealie service page.
  • Roadmap: Phase 6B (Home Assistant) scoped (2026-05-06) — four sub-stages: HAOS VM stand-up, core integrations (HACS, Tapo via Matter, Hue V2, Bambu A1 Mini), Caddy edge at home.rampancy.cloud, dashboard + automations. Manual qm create over community-scripts installer to match existing precedent. See roadmap §6B + Home Assistant Setup runbook.
  • Milestone: Phase 6A complete — Forgejo self-hosted git, GitHub-mirrored (2026-05-05). Closes the 6A.1–6A.4 cycle across 2026-05-04/05; CF DNS token Rolled (closes the 7D leak follow-up); Discord push webhooks deferred as low-signal for solo-dev. See forgejo-setup runbook.
  • Milestone: Phase 6A.3 — 4 repos imported with full history + per-repo push-mirrors to GitHub at ~5s sync_on_commit (2026-05-05). Local origins flipped on WSL, github kept as fallback. Lessons in forgejo-setup runbook.
  • Milestone: Phase 6A.2 — Forgejo public via Caddy at git.rampancy.cloud (2026-05-05). CrowdSec coverage automatic via the existing wildcard handler. Lessons in forgejo-setup runbook.
  • Drift cleanup: Five drift items reconciled across proxfold + edge + forgejo (2026-05-05). NVENC patch repo pinned to commit 80e48e9 (was tracking master); new convention recorded — bring up new hosts with site.yml --limit <newhost> rather than per-host playbooks. See playbooks doc and nvidia role.
  • Milestone: Phase 6A.1 — Forgejo LXC stood up (2026-05-04). New forgejo role on CT 109 (192.168.1.249), forgejo-sqlite 11.0.13 from forgejo-contrib APT. Two execution-time bugs caught (apt_repository → deb822 sources on Debian 13 LXCs; INSTALL_LOCK grep instead of app.ini existence). Lessons in forgejo-setup runbook.
  • Milestone: Phase 7D complete — CrowdSec live on edge (2026-05-04). New crowdsec_engine role on CT 107 + hslatman bouncer module via xcaddy rebuild. End-to-end blocked/allowed cellular validation passed. Six execution-time bugs caught — all in the crowdsec-validation runbook. Edge security gap accepted risk closed same day.
  • Roadmap: Phase 7D scope cut (2026-05-04) — Wazuh forwarding deferred to 7A/B (gated on Phase 4B); Lynis split out and earmarked for the Wazuh piece.
  • Roadmap: Phase 7E (Pocket-ID identity + selective SSO) scoped (2026-05-03), bundled with 7D as the edge-hardening cycle. Authentik considered and deferred — Postgres/Redis footprint doesn't fit the lean ethos. See roadmap §Phase 7E.
  • Decom: VM 102 (nginx) destroyed (2026-05-03) after 24h post-cutover soak. Pre-decom vzdump on nasbackup (1.65 GB compressed). Closes Phase 5D.
  • Milestone: Phase 5D complete — edge LXC live, NPM retired (2026-05-02). New caddy role on CT 107 (edge, 192.168.1.244) replaces NPM on VM 102. Caddy serves a Let's Encrypt wildcard *.rampancy.cloud via DNS-01 against Cloudflare; four hosts migrated. CF orange-cloud attempted then rolled back. Lessons in edge-cutover runbook.
  • Milestone: Housemate access onboarded on proxfold (2026-05-01) — hazel@pve + housemate-lab pool + stash/housemate-vms (500 GiB quota) + ACLs. Two CLI bugs in the original runbook caught + corrected. Lessons in housemate-access runbook.
  • Roadmap: Phase 8 (network segmentation — VLAN scheme + PENFOLD-SW01 SwOS→RouterOS migration) added (2026-05-01). Future L2-isolation home for housemate VMs; immediate Phase 6 access ships on vmbr0 with Proxmox-side controls only. See roadmap §Phase 8.

2026-04

  • Docs: Vintage Story update procedure added (2026-04-29) — manual binary swap, no upstream update script. Captures the gotcha hit on first patch update: upstream server.sh ships vintagestory-keyed defaults that clobber the vintage-keyed customisations on extract; server.sh start then errors Username, Group or data path missing. New section at services/vintagestory#updates.
  • Service: korrosync stack added on arrstack (2026-04-29) — self-hosted KOReader progress sync server for Kobo Clara BW ↔ XTEINK X4 (Crosspoint Reader). LAN-only HTTP at 192.168.1.252:3030; reverse-proxy / TLS deferred until edge-proxy decision lands. See korrosync service page. Port-collision lesson recorded — upstream's default :3000 clashed with Dockhand's UI on the same host.
  • Roadmap: Phase 4B RAM default flipped to 8× 32 GB / 256 GB at 1 DPC (2026-04-28); 12× 16 GB and 8× 16 GB demoted to alternatives. Hynix recommended (matches existing chassis sticks). Part numbers + decision rationale in roadmap §Phase 4B.
  • Docs: Site-wide audit against live state — every page walked vs proxfold + 8 guests (2026-04-28). PVE point release, post-4C boot tooling, missing services + ports, post-Phase-5 ansible scaffolding, 8-agent Beszel fleet, broken anchor in ansible/index.md, r430 runbook RAM line + fan-count clarification.
  • Roadmap: Phase 4B re-scoped from 384 GB → 192 GB and Phase 7 (security stack — Wazuh AIO + Suricata + CrowdSec + Lynis) added, gated on 4B (2026-04-28). See roadmap §Phase 4B + §Phase 7.
  • Milestone: Phase 5C complete — n8n live as a Docker stack (2026-04-28). VM 108 (n8n, 192.168.1.248) cloud-inited; new hawser role codifies Dockhand's remote-host agent. Pivoted mid-execution from npm-on-LXC to Docker-on-VM; lessons in services/n8n.md.
  • Fix: Stopped duplicate PVE backup notifications in #homelab-ops (2026-04-27) by disabling the built-in default-matcher (mail-forward loop was re-firing into ops-all). Codified in roles/proxmox/tasks/notifications.yml.
  • Service: Vintage Story dedicated server publicly exposed at vintage.rampancy.cloud:42420 (2026-04-26). UDM port-forward, no reverse proxy (VS doesn't speak PROXY). Lessons in services/vintagestory.
  • Service: Vintage Story 1.22 dedicated server stood up — CT 201 (192.168.1.235), manually installed, side-project (2026-04-26). Host-level config Ansible-managed via playbooks/vintage.yml; VS install + mods deliberately out of scope while mods are in flux. See Vintage Story.
  • Quality: homelab-ansible handler casing sweep + baseline regen (2026-04-25). 22 handler names retitled; .ansible-lint-ignore shrank 26 → 15 entries (45 → 23 violations).
  • Quality: homelab-ansible lint profile ratcheted to production in three same-day commits (2026-04-25). Pre-commit hook now enforces it. See Linting & quality gates.
  • Infra: auto_updates role added — fleet-wide unattended-upgrades wrapping hifis.toolkit.unattended_upgrades (2026-04-25). Security-only, no auto-reboot; proxfold blocklists PVE kernels; opt-in Discord nag for pending reboots. See auto_updates role page.
  • Docs: Thermal baseline captured for proxfold (2026-04-25) — idle 74 °C, sustained 28-thread load 91–92 °C, no throttle events at 22 °C inlet. See proxfold proxmox page.
  • Extension: PBS-side Discord notifications codified via roles/pbs/tasks/notifications.yml (2026-04-25). Verify/GC/prune/sync events now fan out to #homelab-ops alongside PVE-side events.
  • Fix: Two post-5B regressions caught on the first live drift + backup cycle (2026-04-25). control switched to ansible_connection: local; PVE 9 Discord webhook body trimmed to title/severity/footer (full {{ message }} overflows the 4096-char embed limit).
  • Milestone: Phase 5B complete — server notification stack live (2026-04-24). Three paths into #homelab-ops: Beszel hub on CT 106, ZED webhook on proxfold, PVE 9 notification target. CT 104 (control) brought under Ansible management.
  • Milestone: Phase 5A complete — Proxmox Backup Server live (2026-04-23). PBS 4.x in CT 105 on proxfold, datastore nas-primary on NFSv3 from TS-269L. Daily 02:00 pbs-daily job; first full backup clean (~95 GiB after dedup). Codified in pbs role + proxmox/pbs_client.yml. Gotchas in pbs role page.
  • Milestone: Phase 4C complete — proxfold boot drive swapped to ZFS RAID1 mirror (2026-04-22). 128GB Samsung 840 PRO replaced by 2× 960GB SSDs (SM843T + Intel DC S4500); all four guests restored from NAS vzdumps; Plex HW transcode re-verified. Lessons in Boot Drive Swap runbook.
  • Infra: Phase 3D complete — scheduled drift detection live on CT104. Daily 04:00 ACST timer posts to #homelab-drift; clean runs stay silent. First live run surfaced three role bugs — all fixed in the same PR. See Drift Detection.
  • Docs: Proxfold rebuild runbook added — end-to-end auto-install procedure backed by the rebuild/ kit; WSL dual-control bootstrap documented as the DR cold-start path. proxmox and nut role pages added.
  • Infra: Phase 3C complete — Proxmox auto-install answer rendering + ISO builder merged to homelab-ansible/rebuild/. Nested VM rehearsal validated rendering, vault, disk filter fail-safe, and boot order.
  • Infra: Phase 3B complete — Plex data codification (ZFS stash/plex-data + LXC mp1 + symlink) merged into the plex role with delegate_to for the host side.
  • Infra: Phase 3A complete — proxmox and nut roles merged, adding PVE host baseline (deb822 repos, kernel pin, nouveau blacklist, sysctl, stash import, nasbackup CIFS) and NUT codification.
  • Milestone: stash pool expanded from 4-wide to 6-wide RAIDZ1 via OpenZFS 2.3 raidz_expansion (2026-04-20). Two Samsung PM1633a 3.84TB SAS SSDs attached live; 14.0T/91% → 21.0T/61%. See PVE 9 Upgrade runbook.
  • Milestone: proxfold upgraded from PVE 8.4.18 → 9.1.7 (bookworm → trixie), in-place. Kernel 6.14.11-6-pve GRUB-pinned, Nvidia 550.163.01 via DKMS, NVENC keylase patch reapplied; Plex HW transcode re-verified. Lessons in PVE 9 Upgrade runbook.
  • Infra: Nvidia cgroup2 device majors in CT 100 updated 235→234 (nvidia-uvm) and 238→237 (nvidia-caps) after the kernel swap. Silent failure mode documented (nvidia-smi keeps working via static major 195 but CUDA dies).
  • Infra: NIC names on proxfold pinned to stable nic0–nic3 via systemd .link files (MAC match), replacing kernel-assigned eno1–eno4.
  • Infra: Plex LXC (CT 100) given features: mount=nfs,nesting=1 to silence the systemd-252 AppArmor denial storm. Backup at /root/100.conf.bak-pre-nesting.
  • Infra: CyberPower PR1500ERT2U UPS deployed on proxfold — NUT 2.8.0 in standalone mode, battery transfer test passed cleanly. Phase 4A pulled forward. See UPS.
  • Docs: PVE 9 upgrade runsheet added — covers PVE 8.4→9.x upgrade, post-upgrade verification, and ZFS RAIDZ expansion. Post-execution appendix captures 8 deviations between runbook and actual run.
  • Docs: Host/runbook/reference docs reconciled with PVE 9.1.7 reality — repos (bookworm→trixie), GRUB kernel pin, proxmox-headers meta-package name, ZFS 2.4 userland / 2.3.4 kmod split, dynamic cgroup majors, LVM global_filter.
  • Infra: Plex data directory migrated from CT 100 rootfs to stash/plex-data (100G quota) via symlink — rootfs dropped from 87% to 34% usage.
  • Infra: Plex LXC rootfs resized from 16GB to 32GB (live resize, no downtime).
  • Infra: Weekly fstrim cron job added to proxfold for all running LXC containers (reclaims thin-provisioned LVM space).
  • Infra: CT 103 (stash) destroyed — legacy SMB bridge, retired March 2026, IP 192.168.1.251 freed.
  • Docs: Added implementation roadmap with fact-checked phase details.
  • Docs: Added Ansible section documenting homelab-ansible repo (all 10 roles).
  • Docs: Homelab-ansible repo scaffolded with inventory, playbooks, and all roles — control node deployment pending (Phase 1 of roadmap).
  • Milestone: Ansible control node in place and configured and tested common tag. Phase 1A complete.
  • Milestone: Portainer sunsetted and main Docker Compose stack managed by Dockhand. Nginx to be moved at a later date pending Caddy move. Phase 1B complete.

2026-03

  • Docs: Full boot drive swap runbook created — cross-referenced with storage, GPU, and Ansible sections
  • Docs: Added Proxmox repository config and IPMI fan control sections
  • Docs: ZFS ARC cap corrected to 14 GB (15032385536 bytes) across all docs
  • Service: Overseerr replaced in production with Seerr (ghcr.io/seerr-team/seerr:latest)
  • Service: MediaBot NLP branch merged to main — @mention natural language interface now live via Claude Sonnet; ANTHROPIC_API_KEY required; Discord Message Content Intent enabled
  • Docs: Added UniFi Dream Machine (The-Egg) documentation — network, Wi-Fi, firewall, security, VPN sections
  • Docs: Added PENFOLD-SW01 (MikroTik CRS326-24G-2S+RM) documentation
  • Docs: GPU passthrough section added to Plex docs (Nvidia T400, cgroup2 device allowances)

2025 and earlier

Initial stack deployment

  • Service: Proxmox VE installed on Dell PowerEdge R430 (proxfold, 192.168.1.250)
    • ZFS pools: rpool (boot, single drive — mirror upgrade planned), stash (RAIDZ1, 4× SAS SSD)
    • LXC 100: Plex Media Server with Nvidia T400 GPU passthrough
    • VM 101: arrstack (Docker host)
    • VM 102: nginx (Nginx Proxy Manager, 192.168.1.249)
  • Service: Media stack deployed on arrstack — Sonarr, Radarr, Prowlarr, qBittorrent
  • Service: Seerr deployed (media request management)
  • Service: MediaBot deployed — Discord bot for media pipeline management
  • Network: UniFi Dream Machine deployed as gateway/WAP (The-Egg, 192.168.1.1)
  • Network: MikroTik CRS326-24G-2S+RM deployed as managed switch (PENFOLD-SW01, 192.168.1.3)
  • Network: WireGuard VPN server EggLink active (UDP 51820)
  • Network: DoH via Cloudflare, IPS active mode, DPI enabled on UDM

Upcoming

See the Roadmap for planned phases:

  • Phase 1: Ansible control node + Dockhand + gluetun
  • Phase 2: PVE 9 upgrade + ZFS RAIDZ expansion
  • Phase 3: Full Ansible codification of all hosts
  • Phase 4: UPS, CPU 2 + RAM upgrade, boot drive swap to ZFS mirror
  • Phase 5: n8n automation, vulnerability management
  • Phase 6: Home Assistant, Obico, music library, Matrix (optional)