Music acquisition bringup — Phase 6D

Stand up the music acquisition pipeline on the existing arrstack VM 101 compose: Lidarr (on the plugins branch via hotio's image) with the Tubifarry plugin, slskd routed through gluetun, and beets as an on-demand tagger. Plex / PlexAmp stay untouched — they keep indexing /stash/rodneystash/Music via the /mnt/plex/Music symlink and pick up new files as Lidarr imports them.

Stages

Stage	Scope	Hold points
6D.1	Compose deploy — gluetun port additions + lidarr / slskd / beets services up	After compose redeploy; before declaring services healthy
6D.2	slskd first-boot + Soulseek account registration	Before exposing slskd WebUI on LAN; before allowing first inbound peer
6D.3	Lidarr first-boot + flip to plugins branch + Tubifarry install + slskd wiring	Before declaring Tubifarry's slskd indexer healthy; before bulk import
6D.4	beets first-boot config	None high-risk — beets is on-demand
6D.5	Smoke test — one watchlist artist end-to-end	Before bulk import
6D.6	Bulk library import (existing 612 artists)	Lidarr metadata server health (api.lidarr.audio) before kicking off

Cross-phase decisions

• Lidarr image: ghcr.io/hotio/lidarr:pr-plugins, not linuxserver/lidarr:latest. Plugin support lives on the plugins branch only; linuxserver mainline can't load Tubifarry.
• slskd routing: shares gluetun's network namespace (network_mode: "service:gluetun") — same pattern as qBittorrent. Accepts the known discussion #2875 search-crash risk (last activity Oct 2025, no 2026 chatter — judged low-likelihood). Escape if it bites: spin up a second gluetun container dedicated to slskd, isolating blast radius from qbit.
• slskd share-back: share /downloads (slskd's own download dir) only, not the broader /stash/rodneystash/Music. Reciprocal citizenship without exposing your full listening history. Flip later via slskd.yml if you want to be a heavier seeder.
• Tubifarry, not Soularr: Lidarr plugin instead of Python sidecar — fewer moving parts, current direction of the community as of 2026.
• beets pattern: on-demand sanity-pass, not in the Lidarr import pipeline. Avoids two systems fighting over staging dirs.
• Lidarr → Plex notification: configure Lidarr's Plex Connect on completion. Faster than Plex's filesystem watcher.
• Metadata server: tolerate api.lidarr.audio flakiness. If a multi-day outage bites the bulk import, defer and revisit (LidMeta is the bolt-on insurance, deferred to a future phase).
• gluetun control API auth: required as of the deployed qmcgaw/gluetun:latest image (verified 2026-05-15 — bare API returns 401). Stage 6D.1 generates an API key, writes it into a TOML auth config under /opt/mediaserver/gluetun/auth/config.toml, and into Dockhand's env store as GLUETUN_CONTROL_SERVER_API_KEY. Same value, two destinations, never echoed to scrollback.

Pre-flight gates

• arrstack VM 101 reachable from CT 104; existing stack healthy (pct exec on proxfold + curl 8080, 7878, 8989, 9696 from CT 104)
• NFS mount on VM 101 healthy: qm guest exec 101 -- df /stash shows the 192.168.1.250:/stash line with non-zero space free
• Soulseek username + password chosen (will register in 6D.2 — pick now so they're ready in a password manager)
• Decision recorded on what slskd shares back (default: /downloads only)
• On the Lidarr metadata-server side: curl -sI https://api.lidarr.audio/api/v0.4/search?type=all&query=test returns 200 — if it doesn't, defer 6D.6 (bulk import) until it recovers

Stage 6D.1 — Compose deploy

Pre-create on-disk dirs

Dockhand creates these on first up but pre-creating avoids ownership surprises on first start:

qm guest exec 101 -- /bin/bash -c '
  mkdir -p /opt/mediaserver/{lidarr,slskd,beets} \
           /stash/rodneystash/Downloads/music
'

Set up gluetun control-server auth for slskd

Confirmed during pre-verification 2026-05-15: the deployed gluetun returns 401 Unauthorized on the control API by default (HTTP_CONTROL_SERVER_AUTH_CONFIG_FILEPATH=/gluetun/auth/config.toml, _AUTH_DEFAULT_ROLE={}). slskd 0.25 needs API-key access to fetch the NAT-PMP forwarded port, so this auth config must exist before slskd starts.

Dockhand owns the live .env inside its data volume (/opt/mediaserver/dockhand/); there is no on-disk .env at /opt/mediaserver/arrstack/. The key has to be written into Dockhand via its UI and into a host-side gluetun config file — same value, two destinations, without ever echoing the value to your terminal scrollback.

Generate the key once on VM 101 and stash it in a temp file readable only by root:

# On VM 101 (qm guest exec, no shell history retention concerns)
umask 077
openssl rand -hex 32 > /dev/shm/glkey
ls -la /dev/shm/glkey   # confirm 0600
# Do NOT cat / echo / less the file here — the next two steps consume it directly.

Write the gluetun auth config consuming /dev/shm/glkey:

mkdir -p /opt/mediaserver/gluetun/auth
KEY=$(cat /dev/shm/glkey)
cat > /opt/mediaserver/gluetun/auth/config.toml <<EOF
[[roles]]
name = "slskd"
routes = [
  "GET /v1/openvpn/portforwarded",
  "GET /v1/openvpn/status",
  "GET /v1/publicip/ip",
]
auth = "apikey"
apikey = "$KEY"
EOF
unset KEY
chmod 0600 /opt/mediaserver/gluetun/auth/config.toml

Paste the key into Dockhand:

1. Open Dockhand's UI → Stacks → arrstack → Environment.
2. Add GLUETUN_CONTROL_SERVER_API_KEY with the value from /dev/shm/glkey. You can retrieve it via cat /dev/shm/glkey on the VM — that single read is the only scrollback exposure. Close the terminal afterwards if paranoid.
3. Save → Dockhand will redeploy the stack.

Once Dockhand has redeployed, wipe the temp file:

shred -u /dev/shm/glkey

Hold point — gluetun control API now reachable with the key

After Dockhand's redeploy, verify the auth path end-to-end. The key now lives only in gluetun's config file and Dockhand's env store; the test reads it back from gluetun's config (which is the authoritative copy):

KEY=$(grep '^apikey' /opt/mediaserver/gluetun/auth/config.toml | cut -d'"' -f2)
docker exec gluetun wget -q -S \
  --header="X-API-Key: $KEY" \
  -O- http://localhost:8000/v1/openvpn/portforwarded 2>&1 | grep -E 'HTTP/|port'
unset KEY
# Expected: HTTP/1.1 200 + JSON body containing "port":<N>. If 401, the key
# in Dockhand's env doesn't match the one in config.toml — repeat the
# /dev/shm/glkey procedure with a fresh value.

Deploy via Dockhand

Commit the compose changes; Dockhand reconciles within a minute.

# from WSL
cd ~/homelab-ansible
git add stacks/arrstack/docker-compose.yml stacks/arrstack/.env.example
git commit -m "arrstack: add lidarr (plugins branch) + slskd + beets for Phase 6D"
git push
# Dockhand picks up the change; tail its logs from VM 101 to confirm
qm guest exec 101 -- docker logs dockhand --tail 50

Hold point — services healthy

qm guest exec 101 -- docker ps --format '{{.Names}}\t{{.Status}}' | grep -E 'lidarr|slskd|beets|gluetun'
# All four should report "Up <N> minutes (healthy)" or "Up <N> minutes"

Validate 6D.1 done

• curl -sI http://192.168.1.252:8686/ returns 200 (lidarr WebUI)
• curl -sI http://192.168.1.252:5030/ returns 200 (slskd WebUI through gluetun)
• curl -sI http://192.168.1.252:8337/ returns 200 (beets WebUI)
• gluetun health check still green and qBittorrent unaffected (curl -sI http://192.168.1.252:8080/)

Escape

# revert the compose commit; Dockhand reconciles back
git revert <commit>
git push
# optional state wipe
qm guest exec 101 -- rm -rf /opt/mediaserver/{lidarr,slskd,beets}

Stage 6D.2 — slskd first-boot + Soulseek account

Register on Soulseek

Manual external step — slskd needs a Soulseek-network username + password.

1. Open https://www.slsknet.org → Login → Register (no email confirmation required; you'll just pick a username + password).
2. Save credentials to your password manager — there is no password reset flow if you lose them.

Configure slskd

Open http://192.168.1.252:5030. On first boot slskd writes a default slskd.yml into /app (= /opt/mediaserver/slskd/slskd.yml on the host).

In the WebUI:

1. Soulseek credentials: paste the registered username + password.
2. Shares: add /downloads as the single shared directory (= where slskd writes its own downloads — mapped to /stash/rodneystash/Downloads/music on the host).
3. Limits: queue depth 5, upload speed cap 256 KB/s (be a decent citizen — Soulseek rewards uploaders).
4. Save & restart slskd from the WebUI's "Application" tab.

Hold point — slskd is connected and listening

In slskd's WebUI top bar: - "Server" indicator: Connected. - "Listener" indicator: Listening on port — where <N> is the dynamic NAT-PMP-forwarded port fetched from gluetun. If "Listening on port 0" or "Not Listening", check docker logs slskd for VPN-integration errors.

Validate 6D.2 done

• slskd reports Connected + Listening
• Generate an API key inside slskd: Options → Security → Generate API Key — save it; Lidarr's Tubifarry plugin needs it in 6D.3
• Test search from slskd's UI for any popular album; results appear within ~10 s

Escape

slskd is stateful only in /opt/mediaserver/slskd/ — rm -rf that dir and re-bring-up to reset.

Stage 6D.3 — Lidarr first-boot + plugins branch + Tubifarry

Auth + branch

Open http://192.168.1.252:8686. Set Forms authentication on first boot (no username/password is fine for LAN-only, but Forms is recommended).

Flip to the plugin-enabled branch — required before Tubifarry can be installed:

1. System → General → Branch: change master to plugins.
2. Save → Restart from the prompt that appears.

After the restart, System → Plugins appears in the left nav (it doesn't exist on mainline Lidarr).

Install Tubifarry

1. System → Plugins → Install From URL: paste https://github.com/TypNull/Tubifarry.
2. Wait for the install to complete (~30 s).
3. Restart Lidarr from the prompt.

Wire Tubifarry to slskd

1. Settings → Indexers → Add → slskd (provided by Tubifarry).
   • Slskd Base URL: http://localhost:5030 (slskd shares gluetun's namespace; from Lidarr's container the address depends on whether the arr network can reach gluetun's host-exposed port — use http://gluetun:5030 if container-to-container, or http://192.168.1.252:5030 from inside the LAN).
   • API key: from 6D.2.
   • Test → Save.
2. Settings → Download Clients → Add → slskd.
   • Same URL + API key.
   • Remote path: leave the auto-detected /downloads from slskd.
   • Test → Save.

Configure Lidarr root folder

1. Settings → Media Management → Add Root Folder → /media/Music (= /stash/rodneystash/Music on the host).

Wire Lidarr → Plex notification

1. Settings → Connect → Add → Plex Media Server.
   • Host: 192.168.1.230 (Plex CT 100).
   • Port: 32400.
   • Auth Token: paste a Plex token. Get one from the Plex Web UI — sign in to any media item, open the three-dot menu → Get Info → View XML → the URL in the address bar ends &X-Plex-Token=<value>. (Avoid greping the on-disk Preferences.xml; that routes the live token through your terminal scrollback. Confirmed gotcha 2026-05-15.)
   • Update Library: ON.
   • Test → Save.

Hold point — metadata server alive

curl -sI 'https://api.lidarr.audio/api/v0.4/search?type=all&query=test'
# Expected: HTTP/2 200. If 5xx, defer 6D.5 and 6D.6 until it recovers.

Validate 6D.3 done

• Lidarr is on plugins branch (System → General reflects this)
• Tubifarry plugin appears under System → Plugins as Installed + Enabled
• slskd indexer test returns success; slskd download client test returns success
• Root folder /media/Music registered and shows free space
• Plex connection test succeeds

Escape

Nothing user-visible has been done to the music library yet at this stage. Stopping the lidarr container undoes everything.

Stage 6D.4 — beets first-boot config

beets is on-demand; first boot just generates a default config.yaml at /opt/mediaserver/beets/config.yaml. Open the WebUI on :8337 to verify it loaded. No mandatory configuration for this stage — beets gets exercised manually in 6D.6's lessons section (or later, as the library accumulates) for tag cleanup.

Validate 6D.4 done

• http://192.168.1.252:8337/ loads
• docker exec -it beets beet config prints the default config

Stage 6D.5 — Smoke test (one watchlist artist)

Pick a small-discography artist

Pick one with 2–4 albums total (not 30) so the test completes quickly and uses Soulseek modestly. A favourite from your existing 612 isn't ideal — pick something Lidarr knows about but you don't already own, so you can clearly observe the import path.

1. Lidarr → Library → Add New → search for the artist.
2. Quality profile: FLAC (or whatever default fits your taste — Lossless is the broadest).
3. Metadata profile: Standard (default).
4. Monitor: All Albums → Add.

Watch the pipeline

Lidarr will: 1. Search via Tubifarry → slskd → return matches. 2. Send a download to slskd. 3. slskd downloads to /downloads/<artist>/<album>/.... 4. Lidarr detects completion → imports → moves into /media/Music/<Artist>/<Album>/.... 5. Lidarr notifies Plex → Plex re-scans Tunes.

Tail Lidarr's activity in Activity → Queue / History as it progresses.

Hold point — files actually land in the library

Before declaring the smoke test passed:

# On VM 101
qm guest exec 101 -- ls /stash/rodneystash/Music/<Artist>/
# Should show the album dir with FLACs inside, owned by root (PUID=0 in the
# compose) with sensible tags.

Plex visibility

Open Plex → Tunes → search the artist. Should appear within ~30 s of import (Lidarr's Plex Connect triggers an immediate re-scan of the relevant section).

Validate 6D.5 done

• Files exist in /stash/rodneystash/Music/<Artist>/
• Tags look reasonable (exiftool or any tag viewer on one track)
• Plex's Tunes library shows the artist + album

Stage 6D.6 — Bulk library import (existing 612 artists)

Hold point — metadata server health, again

Bulk import hits api.lidarr.audio for every artist. Re-check before kicking off:

curl -sI 'https://api.lidarr.audio/api/v0.4/search?type=all&query=test'

If 200: proceed. If 5xx or slow: defer; Lidarr will partially import and you'll be cleaning up.

Import existing files

1. Lidarr → Library → Library Import → /media/Music.
2. Lidarr scans the directory tree — expect ~30–60 minutes for 612 artists / 85 GB. Watch Activity → Queue.
3. For each unmatched album, Lidarr surfaces a manual-match dialog. Plan to spend an hour or two over a couple of evenings working through these.

Validate 6D.6 done

• Library shows ~612 artists imported
• Spot-check 5 random artists for correct metadata
• Unmatched count is 0 (or accepted-as-unmatched if Lidarr genuinely can't find them — typical for very-small-label releases)

Lessons from the 2026-05-16 run

Execution took most of a day across two sessions. The compose scaffolding committed 2026-05-15 needed several follow-up patches as we discovered execution-time bugs. Final commits to homelab-ansible/main: f60c1ca (initial scaffolding), 03da202 (VPN timeout fix), 9a39400 (downloads path redirect), 7da40d2 (citizenship config), 76f0868 (slskd env-var config), b5ca8f3 (API key env var).

Stage 6D.1 — gluetun control API auth

• Gluetun latest requires X-API-Key on the control API by default. The runbook assumed a no-auth pre-verification baseline; the deployed image returned 401 on /v1/portforward. The /dev/shm/glkey temp-file pattern + config TOML approach the runbook describes is correct; just confirm the assumption.
• Gluetun renamed /v1/openvpn/portforwarded → /v1/portforward. The legacy path 301-redirects, but wget doesn't re-send headers on redirect, so curl/wget tests against the legacy path show 401 misleadingly. Whitelist the new path.
• /v1/wireguard/status is NOT a real gluetun endpoint even though we're on WireGuard. Adding it to the auth role's whitelist crashes gluetun's control server at startup with a clear error: route path not supported by the control server. Either whitelist /v1/openvpn/status (returns {"status":"stopped"} since OpenVPN isn't running, which is fine), or omit status routes entirely if slskd doesn't need them.
• docker restart gluetun does NOT cascade-restart consumers using network_mode: service:gluetun. The compose depends_on: condition: service_healthy: restart: true only fires during docker compose up, not during runtime restarts. After restarting gluetun manually, qbit and slskd keep running but lose the namespace (which got recreated) and orphan on no network. Must manually docker restart qbittorrent slskd to re-attach. This re-confirms the Phase 3B-era observation that docker-restart breaks network_mode:service:* chains.
• qm guest exec wraps output in JSON. When extracting secrets from /dev/shm/* for paste-into-Dockhand, either pipe through jq -r '."out-data"' or use grep -oE '[a-f0-9]{64}' to extract a clean hex string. jq is the cleaner path once installed; the grep fallback works for systems without it.

Stage 6D.2 — slskd config persistence

• Soulseek doesn't have a website registration. The runbook step "Register on slsknet.org" was wrong. Account creation happens at first-client-connect — any client (slskd included) attempts login with your chosen username/password, and the server creates the account if the username is available. Usernames recycle after 30 days of no login unless you've donated for "privileges" (non-issue for 24/7 homelab).
• slskd 0.25 WebUI YAML editor saves as Run-Time Overlay (volatile). Per slskd's documented config precedence chain (Defaults < Env Vars < YAML File < CLI Args < Run-Time Overlay), the UI editor applies changes as the highest-precedence overlay — but it's lost on restart. Despite SLSKD_REMOTE_CONFIGURATION=true, the "Save" button does NOT persist edits to the on-disk slskd.yml (which remains the all-commented example file). Use env vars for all persistent config.
• slskd env var naming has non-obvious abbreviations:
  • soulseek.username → SLSKD_SLSK_USERNAME (note SLSK_, not SOULSEEK_)
  • web.authentication.password → SLSKD_PASSWORD (drops web.authentication. prefix)
  • shares.directories → SLSKD_SHARED_DIR (singular DIR, not DIRS; semicolon-separator for multiple values: /downloads;/music)
  • transfers.upload.slots → SLSKD_UPLOAD_SLOTS (drops transfers. prefix)
  • integrations.vpn.gluetun.timeout → SLSKD_VPN_GLUETUN_TIMEOUT (drops integrations. prefix)
  • directories.downloads → SLSKD_DOWNLOADS_DIR (drops directories. prefix)
• SLSKD_VPN_GLUETUN_TIMEOUT default 1000 ms causes flap-disconnect cycle. Under any gluetun load, the 1-second polling timeout elapses; slskd assumes VPN down and drops Soulseek. Repro: every ~3 min, log shows Failed to fetch status from VPN client Gluetun... Timeout of 1 seconds elapsing → VPN client disconnected → Disconnected from the Soulseek server → reconnect 1 s later. Active searches/downloads at that instant get interrupted. Bump to 5000 ms.
• SLSKD_DOWNLOADS_DIR defaults to /app/downloads which lands on the VM's 24 GB rootfs via the /opt/mediaserver/slskd:/app bind mount. The bulk-import scenario (612 artists / 85 GB+) would overflow the rootfs within minutes. Must redirect to ZFS bind mount via SLSKD_DOWNLOADS_DIR=/downloads + SLSKD_INCOMPLETE_DIR=/incomplete with separate bind mounts for each. Pre-create dirs on the host before deploy.
• SLSKD_API_KEY (single-key shorthand) is validated 16–255 characters at startup. Crash loop with Invalid configuration: Web: Authentication: API key must be between 16 and 255 characters if you paste a too-short value (e.g., a 12-char fingerprint instead of a 64-char hex key). Easy footgun when verifying secret hashes — the fingerprint is for verification, not the value to paste.
• slskd's Failed to start listening on 0.0.0.0:<port> error is cosmetic when slskd has already bound the same port on specific interfaces (tun0, docker bridge, loopback). Verify via docker exec gluetun ss -tlnp | grep <port> — multiple per-interface LISTEN entries = inbound traffic on tun0 reaches slskd, the catchall bind just lost the race.
• Soulseek citizenship: share REAL library, not downloads-only. The community treats "downloads-only share" as a leech anti-pattern; many peers auto-ban small or empty shares. The read-only bind mount /stash/rodneystash/Music:/music:ro + SLSKD_SHARED_DIR=/downloads;/music shares the curated library back (15,986 files in our case) without slskd having write access to it. Also bumped upload cap from default 256 KB/s → 1 MB/s (2026 community median) and set a polite profile blurb via SLSKD_SLSK_DESCRIPTION to signal automated-library posture.

Stage 6D.3 — Lidarr image + Tubifarry + Plex Connect

• linuxserver/lidarr:latest mainline has NO plugin support. Must use ghcr.io/hotio/lidarr:pr-plugins. The hotio image arrives pre-configured for the plugins branch (System → General shows plugins); no manual branch flip needed despite the runbook scaffolding suggesting one. System → Plugins is visible in the left nav out of the box.
• Tubifarry's slskd indexer doesn't support RSS sync. Lidarr fires the health warning No indexers available with RSS sync enabled, Lidarr will not grab new releases automatically — expected and benign for Soulseek. Soulseek has no feed-of-new-releases protocol. Discovery happens via scheduled Author Refresh (every 6h) + manual album/artist searches. Dismiss the warning or ignore.
• Tubifarry HAS per-peer throttling (the README I fetched was incomplete). The indexer config exposes a full set of citizenship controls — used to enforce Soulseek etiquette:
  • Grabs per User: default 5/day → set 3 (closer to community "≤1 album/day per user" strict norm)
  • Max Queued/User: default 0 (disabled) → set 5 (don't pile a peer's queue)
  • Min Peer Speed: default 0 → set 50 KB/s (skip very slow peers)
  • Min File Count: default 1 → set 5 (prefer full albums over single tracks)
  • Max Peer Queue: default 1000000 (disabled) → set 100 (prefer responsive peers)
• Tubifarry's slskd download client doesn't expose categories. Soulseek has no category concept (it's not torrent/usenet). Field is absent from the UI — nothing to set. The download is identified by Tubifarry's internal correlation, not a label.
• Lidarr 2.x Plex Connect uses Plex.tv OAuth, not manual token paste. The Add Connection dialog has an "Authenticate with Plex.tv" green button that opens an OAuth flow in a new tab — no &X-Plex-Token= URL scraping needed. The runbook's earlier scrape-Preferences.xml approach was both wrong AND a scrollback-hygiene violation.
• "Album Search Limit" doesn't exist in Lidarr 2.x — that was a Sonarr/Radarr concept I incorrectly transplanted. Lidarr's only global throttle for slskd-style indexers is RSS Sync Interval (Settings → Indexers → "Indexer Options" panel at bottom).
• Remote Path Mapping required. slskd writes to /downloads/... (its internal path), Lidarr sees the same files at /media/Downloads/music/... (its bind mount). Without a mapping, Lidarr health warning download client Slskd places downloads in /downloads but this directory does not appear to exist inside the container fires. Settings → Download Clients → Remote Path Mappings → Host gluetun, Remote /downloads/, Local /media/Downloads/music/.

Stages 6D.5 + 6D.6 — bulk import behavior

• Adding a root folder in Lidarr 2.x auto-scans + bulk-imports everything in it, on a delayed schedule (~minutes after add). Effectively collapses planned-deliberate 6D.6 into the 6D.3 root-folder-add step. Plan ahead: if you want chunked bulk import, don't add the root folder until you're ready for the full scan to fire. In our case the auto-import was actually fine (877 albums / 10,691 tracks registered in ~25 minutes, no errors) but it happened earlier than the runbook anticipated.
• Lidarr's import retains original filenames by default. Soulseek-sourced FLACs like 01-01 - Making Mirrors.flac stay that way; Lidarr does NOT apply its standard Artist - Album - 01 - Track rename template unless Settings → Media Management → "Rename Tracks" is enabled AND you trigger a "Rename Files" task. Either preserve the original peer naming, or schedule a rename pass later.
• UTF-8 mojibake from Soulseek peers persists through the chain. Example: Donât Worry, Weâll Be Watching You.flac — apostrophe encoded as UTF-8 (\xE2\x80\x99) but mis-rendered as Latin-1 (â\x80\x99). Neither slskd nor Lidarr corrects this on import; the file lives in the library with the broken name. beets territory for ad-hoc cleanup.
• Lidarr metadata server (api.lidarr.audio) is flaky — visible during execution as SkyHookException errors with HTTP 500 InternalServerError and Invalid response received from LidarrAPI for background tasks (Lidarr trying to map every slskd search result back to MusicBrainz releases). Doesn't block imports of pre-cached metadata (added artists) but does affect adding NEW artists from search results that aren't yet cached. Accepted risk; LidMeta is the bolt-on insurance for the future if outages bite.
• Smoke test "not in library" criterion wasn't verified. Gotye turned out to be already in the existing library (Like Drawing Blood as m4a). Doesn't break the test (Making Mirrors was a clean new-import), but next time ls /stash/rodneystash/Music/<artist> before picking the smoke-test artist.

Cross-stage / process

• Plex token retrieval should NOT use grep Preferences.xml — that routes the live token through terminal scrollback (Claude's verification grep slipped here, leaked the token; user chose accepted-not-rotated since the realistic incremental exposure was low above the already-server-resident value). Use Plex.tv OAuth in Lidarr's Connect UI (no token paste needed), or Plex Web UI's "View XML" path to get a session token from &X-Plex-Token= in the URL.
• Dockhand's per-stack .env lives inside its data volume (/opt/mediaserver/dockhand/), NOT materialized to /opt/mediaserver/arrstack/ (that dir exists on disk but is empty). Env edits go through Dockhand's UI, not host-side file paths.
• Storage terminology: stash is proxfold's own ZFS pool (6-wide raidz1 on local disks), NFS-served to its VM consumers (VMs can't bindmount host paths the way LXCs can). It's not a NAS — no separate appliance exists in this homelab.
• slskd path mangling preserves leaf folder, strips peer-specific parent path. Peer's music\Gotye\Making Mirrors\01-01.flac lands in our /downloads/Making Mirrors/01-01.flac — slskd strips music\Gotye\ and keeps just the album folder. Worked correctly for our pipeline; Lidarr imports based on tags + album-folder match, not the parent path.
• Soulseek peer concentration is naturally diluted by the *arr search model. Tubifarry issues one search per album; matching is per-album; different albums = different searches = different peer pools. Smoke test (Gotye/Making Mirrors) pulled all tracks from one peer (sandatker) because it's one album release = one Grab = one peer match. Per-peer Grabs per User=3/day cap will only kick in if you queue several different albums and the same peer keeps winning the scoring — rare in practice.

Post-execution follow-ups (2026-05-16 later session)

Additional findings from the first few hours of real-use operation after phase close:

• Tubifarry's default search query format returns ZERO results from slskd. Critical fix: set Search Templates field in the slskd indexer config to {{CleanAlbumQuery}} (album-only query). Default behaviour is Artist AlbumTitle which fails because Soulseek matches against filenames, not folder paths — and most peers organise as Artist/Album/01 - Track.flac, so the track filenames don't contain the artist name. Searching for both terms together excludes all the standard organisations. Without this fix, every album search returns "0 reports downloaded" in Lidarr while the same query in slskd's UI returns hundreds of responses. Verified diagnostic: in slskd's Search history, the Tubifarry-issued search with both terms shows 0 responses; the manual album-only search shows hundreds.
• Min Peer Speed default of 50 KB/s was too aggressive — many Soulseek peers report 0 or unknown upload speed (especially those who haven't recently uploaded). Filtering ≥50 KB/s excludes them all, even if they're functional. Set to 0.
• Max Peer Queue recommendation revised from 100 → 500. Popular albums have peers with longer queues; 100 was too aggressive. 500 still skips peers with multi-day queues but doesn't exclude all in-demand peers.
• Enable Fallback Search and Track Fallback in the slskd indexer — if the primary album search returns no matches, Tubifarry will retry with additional metadata / track-name fallback. Useful resilience for obscure releases.
• Tubifarry returns only the single "best" report to Lidarr per search, not all matches. There's no automatic peer-fallback when the chosen peer rejects the download. If the best-scored peer is unavailable, Lidarr's grab fails until the next manual search retry. Mitigation: use Tubifarry's Ignore List (Settings → Indexers → slskd → Ignore List field, takes a file path with usernames one per line) to temporarily block chronically busy peers — file lives at /opt/mediaserver/lidarr/ignored_users.txt (host) = /config/ignored_users.txt (container).
• Soulseek "Overwhelmed with requests; try again later" rejection is a normal peer-side queue-capacity state. Logged as Transfer rejected: Overwhelmed with requests with the download status Completed, Rejected (0 bytes). NOT our config issue — the peer's client is saturated. Standard mitigation: ignore-list the peer if it persists, or just retry later. When this happens, all 16 tracks of an album reject simultaneously because slskd queued them all to the same peer and the peer rejects per-request.
• Lidarr's "Rename Tracks" setting MUST be enabled to get album subfolders. Default is OFF, which means Lidarr preserves Soulseek's flat-folder structure and dumps all tracks at the artist root (no album subfolder). Plex can't reliably pick up flat-organised libraries. Toggle Settings → Media Management → Track Naming → Rename Tracks = on.
• The hotio plugins-branch image has different Track Naming fields than what the Servarr Wiki documents for mainline Lidarr. Specifically: there is NO separate "Album Folder Format" field — only Standard Track Format, Multi Disc Track Format, and Artist Folder Format. The album subfolder must come from a {Album Title}/ path prefix inside Standard/Multi Disc Track Format. Correct values for our use case:
  • Standard Track Format: {Album Title}/{Artist Name} - {Album Title} - {track:00} - {Track Title}
  • Multi Disc Track Format: {Album Title}/{medium:0}-{track:00} - {Track Title}
  • Artist Folder Format: {Artist Name} (default, fine)
  • (Helper preview shows (1) suffix in the example output — that's a Lidarr UI artifact, NOT part of the actual rendered filename.)
• Samba can need smbcontrol all reload-config when newly-created directories don't appear to clients. Hit this with the freshly-created /stash/rodneystash/Music/Linkin Park/Hybrid Theory/ not showing up via the SMB share even after Windows-client cache refresh. After reload-config on the server, the dir appeared immediately. Underlying perms (775 root, force user = root in smb.conf) were correct throughout — samba just had a stale internal state.
• Bulk import can miss existing albums whose tags/filenames don't match Lidarr's parser. Example: Linkin Park's Meteora album was on disk (/Music/Linkin Park/Meteora/ with 13 FLACs) BEFORE the bulk-import scan ran, but Lidarr's tag-matcher didn't register it as imported. Result: Meteora sat in Wanted → Missing, Lidarr later tried to download a 2023 24-bit reissue, user noticed and cancelled. Likely cause: non-canonical filename format (01. Foreword.flac with period vs dash separator) plus UTF-8 mojibake in titles (02. Donât Stay.flac). Post-bulk-import audit step: browse Wanted → Missing for albums you know are on disk; use Lidarr's Manual Import to associate existing files (Wanted → Missing → select album → "Manual Import" toolbar button → approve track-to-position matches).
• Lidarr does NOT automatically search Wanted → Missing albums via the Soulseek/Tubifarry path. RSS not supported by the slskd indexer means no unattended discovery; scheduled Author Refresh (every 6h) only refreshes artist metadata, doesn't trigger searches. Practical pattern: manual sweep of Wanted → Missing on a weekly-ish cadence. Future automation options if needed: schedule the AlbumSearch API command via external cron, or enable Lidarr's "Missing Album Search" scheduled task if it exists in this build.

Follow-up: dual-gluetun split (2026-05-22)

User-visible symptom (a few days after adding a new torrent): qBittorrent's WebUI showed every tracker as Unreachable / skipping tracker announce (unreachable) or Host not found (authoritative); downloads stalled with zero peers despite DHT/PeX/LSD all listed as "Working".

Diagnostic walk

Outbound networking from inside the gluetun netns was perfectly healthy — sanity-checked four ways:

• docker exec gluetun wget -qO- https://ipinfo.io/ip returned a valid ProtonVPN exit IP.
• docker exec qbittorrent nslookup tracker.opentrackr.org resolved fine.
• docker exec qbittorrent nc -uvz tracker.opentrackr.org 1337 succeeded (UDP egress on a tracker's announce port).
• A raw UDP BT tracker connect handshake from inside the qbit container against tracker.opentrackr.org:1337 and open.demonii.com:1337 returned valid connection_ids — i.e. the protocol works end-to-end across the VPN.

So it was internal to qBit. Pulled qBit's recent log; the CRITICAL block at startup told the whole story:

(N) Trying to listen on the following list of IP addresses: "0.0.0.0:59178,[::]:59178"
(C) Failed to listen on IP. IP: "127.0.0.1". Port: "TCP/59178". Reason: "Address in use"
(C) Failed to listen on IP. IP: "172.18.0.2". Port: "TCP/59178". Reason: "Address in use"
(C) Failed to listen on IP. IP: "10.2.0.2". Port: "TCP/59178". Reason: "Address in use"
(I) Successfully listening on IP. "::1". Port: "TCP/59178"
(I) Successfully listening on IP. "::1". Port: "UTP/59178"

qBit had silently fallen back to IPv6 loopback only because the wildcard bind for port 59178 returned EADDRINUSE. Inode → process mapping inside the netns showed slskd holding 0.0.0.0:59178. slskd's SLSKD_VPN_PORT_FORWARDING=true setting tells it to claim whatever port gluetun has currently forwarded via NAT-PMP, and ProtonVPN had assigned 59178 — coincidentally the same number qBit had cached in its Session\Port setting from a previous session. Before that coincidence, the two services were on different numbers and the conflict was invisible.

Architectural mismatch

ProtonVPN's NAT-PMP forwards one port per VPN session. With both apps in the shared gluetun namespace, only one of them could ever own the forwarded port for incoming peers. The other ends up with whatever ephemeral fallback libtorrent / soulseek picks — which doesn't match the firewall rule gluetun added on tun0 for the forwarded port, so external traffic never reaches it.

Gluetun's issue #2381 requests multi-port forwarding for ProtonVPN (NAT-PMP supports up to 5 ports per session), but it's been open since 2024-07 with status "Nearly resolved / Waiting for feedback" and no PR merged.

Fix

Split slskd onto its own dedicated gluetun container (gluetun-slskd) with a separate ProtonVPN WireGuard session, giving each P2P app its own independent forwarded port.

Compose change in homelab-ansible/stacks/arrstack/docker-compose.yml:

• Added new gluetun-slskd service (mirror of gluetun but with WIREGUARD_PRIVATE_KEY=${PROTONVPN_WIREGUARD_PRIVATE_KEY_SLSKD} and volume /opt/mediaserver/gluetun-slskd:/gluetun).
• Moved slskd's WebUI port mapping 5030:5030 from gluetun to gluetun-slskd.
• Switched slskd's network_mode from service:gluetun → service:gluetun-slskd; same change to depends_on.

Host prep on arrstack VM (one-time):

mkdir -p /opt/mediaserver/gluetun-slskd/auth
cp /opt/mediaserver/gluetun/auth/config.toml /opt/mediaserver/gluetun-slskd/auth/config.toml
chmod 600 /opt/mediaserver/gluetun-slskd/auth/config.toml

Reuses the existing GLUETUN_CONTROL_SERVER_API_KEY — slskd's SLSKD_VPN_GLUETUN_URL=http://localhost:8000 resolves to gluetun-slskd's control API automatically since slskd now lives in that netns.

ProtonVPN side: generate a second WireGuard config at account.protonvpn.com → Downloads → WireGuard configuration, with NAT-PMP enabled and a P2P server (double-arrow icon). Pick a different server from the first so the two sessions don't compete for the same exit. ProtonVPN Plus supports multiple simultaneous WireGuard configs. New PrivateKey goes into Dockhand as PROTONVPN_WIREGUARD_PRIVATE_KEY_SLSKD.

Verification

After Dockhand deploy:

Container	Public IP	Forwarded port	Status
gluetun	(one ProtonVPN exit)	e.g. 62783	Up + healthy
gluetun-slskd	(different exit)	e.g. 50211	Up + healthy

qBit log after redeploy shows Successfully listening on 0.0.0.0:<port> (TCP + UTP) plus 127.0.0.1, 172.18.0.2, and 10.2.0.2/tun0 — no more Address in use criticals. WebUI tracker list moves out of "Unreachable" within an announce cycle (or force-reannounce). slskd logs VPN client connected and ready. IP: <exit>, Forwarded port: <port> ... Connected to the Soulseek server ... Logged in as <username>.

Lessons / process

• libtorrent silently falls back to loopback on wildcard-bind EADDRINUSE. There's a CRITICAL log line but no UI surface — looks like "trackers are slow" not "I can't actually receive anything." Worth a one-time alerting hook if a similar collision class ever recurs.
• Two P2P apps cannot share a single ProtonVPN session for inbound as long as gluetun #2381 stays open. Either accept that one app runs without incoming (degraded mode — for slskd, this means search/queue work but Soulseek peers can't initiate connections in), or commit a second gluetun. Latter is essentially free on ProtonVPN Plus.
• ProtonVPN WireGuard configs expire after one year — the new PROTONVPN_WIREGUARD_PRIVATE_KEY_SLSKD joins the existing PROTONVPN_WIREGUARD_PRIVATE_KEY on the renewal calendar.
• The original 6D run's Failed to start listening on 0.0.0.0:<port> quirk note (flagged "cosmetic, just lost a race") was masking exactly this pattern. Re-read after seeing qBit's identical-shape critical: any time a libtorrent-family app inside a shared netns reports a wildcard-bind EADDRINUSE, the next debug step is "who else in this netns is holding the same port", not "ignore, peers still reach me on the per-interface bind."