Skip to content

Role: pbs

Proxmox Backup Server — runs in a dedicated LXC on proxfold, with the datastore on an NFS mount from the QNAP TS-269L.

Hosts: pbs (CT 105, 192.168.1.246)

Phase 5A executed — 2026-04-23

PBS LXC live, datastore + schedules + PBS user all codified, first full backup and verify both clean (~95 GiB on NAS across 5 guests). Daily 02:00 backup job runs from proxfold via the proxmox role's pbs_client.yml subfile.

Why these choices

  • LXC over VM — PBS is a standalone Debian service with no Docker dependency. Fits the established LXC-for-standalone / VM-for-Docker split.
  • Privileged LXC — sidesteps the uid/gid shift for the backup user (uid 34) writing to the NFS share with no_root_squash. Matches the CT 100 (plex) pattern.
  • Debian 13 template — PBS 4.x is trixie-only. The pct create call requires features: nesting=1 or systemd 257 leaves core units (journald, logind, networkd, gettys) failed on first boot.
  • Datastore on NFS, not CIFS — the TS-269L exports both, and NFSv3 is materially better-behaved than CIFS as a PBS datastore backend (no uid/gid remapping, no reported .chunks inode collisions).
  • Datastore on NAS, not on proxfold — DR requires a second failure domain. A local datastore would die with proxfold or stash.
  • PBS not on the NAS itself — TS-269L is Atom D2701 with ≤3GB RAM on EOL QTS 4.3.4. Can't run PBS 4.x.

Tasks

Task Tag Notes
Install nfs-common pbs, install
Stat Proxmox archive keyring pbs, repo gates the next task; prevents check-mode drift
Fetch Proxmox archive keyring pbs, repo from enterprise.proxmox.com/debian/proxmox-release-trixie.gpg
Add PBS no-subscription deb822 repo pbs, repo
Disable PBS enterprise repo (.list + .sources) pbs, repo both formats handled
Install proxmox-backup-server pbs, install
Mask zfs-zed, zfs-mount, zfs-share pbs, install non-functional inside LXC; prevents noise in systemctl --failed / Beszel
Ensure datastore mount point (backup:backup owner) pbs, datastore uid 34
NFS mount via /etc/fstab pbs, datastore _netdev,nofail,nfsvers=3 — NOT x-systemd.automount (unsupported in LXC; see gotcha)
Datastore healthcheck — script + service + timer + enable pbs, datastore, healthcheck every 15 min; Discord alert if mount / .chunks missing. Gated on the ops webhook var
Create PBS datastore (idempotent via .chunks/ stat) pbs, datastore one-shot
Query datastore configuration pbs, schedule for GC idempotency
Set datastore GC schedule pbs, schedule only if different from desired
List + create prune job pbs, schedule 03:00 daily, gated on job ID lookup
List + create verify job pbs, schedule sun 04:00, gated on job ID lookup
List PBS users pbs, auth for user-create idempotency
Create PBS client user pbs, auth throwaway password; real auth is the token
Ensure datastore ACLs for client user pbs, auth changed_when: false — PBS acl update is idempotent
Token creation reminder pbs, auth debug-only; token generation is deliberately manual
Include notifications.yml (Discord webhook endpoint + match-all matcher) — gated on pbs_discord_endpoint_name is defined pbs, notifications

Manual step on first install: generate the API token

The PBS API token is shown only once on creation — it cannot be retrieved later. After the role finishes, SSH to the PBS host and run:

proxmox-backup-manager user generate-token pbs-pve@pbs pve
proxmox-backup-manager cert info | grep -i Fingerprint

Capture the value field into vault_pbs_token_secret and the fingerprint into vault_pbs_fingerprint, then re-run the proxmox role's pbs tag to register storage + create the backup job.

Key variables

Variable Source Value
pbs_datastore_name defaults nas-primary
pbs_datastore_path defaults /mnt/pbs-datastore
pbs_nfs_server defaults 192.168.1.253
pbs_nfs_export defaults /backup/pbs-datastore
pbs_nfs_options defaults _netdev,nofail,nfsvers=3 (was defaults,nofail,x-systemd.automount,nfsvers=3 pre-2026-06-20 — see gotcha)
pbs_datastore_healthcheck_enabled defaults true
pbs_datastore_healthcheck_interval defaults *:0/15 (systemd OnCalendar — every 15 min)
pbs_prune_keep_daily defaults 7
pbs_prune_keep_weekly defaults 4
pbs_prune_keep_monthly defaults 6
pbs_schedule_prune defaults 03:00 — plain HH:MM (PBS rejects daily HH:MM)
pbs_schedule_verify defaults sun 04:00
pbs_schedule_gc defaults mon 04:00
pbs_prune_job_id defaults nas-primary-prune
pbs_verify_job_id defaults nas-primary-verify
pbs_archive_keyring_url defaults https://enterprise.proxmox.com/debian/proxmox-release-trixie.gpg
pbs_archive_keyring_path defaults /usr/share/keyrings/proxmox-archive-keyring.gpg
pbs_client_userid defaults pbs-pve@pbs
pbs_client_firstname defaults PVE Client (proxfold)
pbs_client_roles defaults [DatastoreBackup, DatastoreAudit]
pbs_discord_endpoint_name defaults discord-ops
pbs_discord_matcher_name defaults ops-all
vault_discord_webhook_homelab_ops vault shared with PVE + Beszel + ZED
vault_pbs_token_id vault PVE → PBS client token id (e.g. pbs-pve@pbs!pve)
vault_pbs_token_secret vault PVE → PBS client token secret
vault_pbs_fingerprint vault PBS TLS cert fingerprint for PVE trust

Gotchas captured during execution

  • x-systemd.automount is unsupported inside an LXC — never use it for an in-container mount. This was the original pbs_nfs_options value, and it silently took all backups offline after the 2026-06-19 power-cycle (the first reboot since the datastore was set up Apr 23): the 2026-06-20 02:00 job errored and the last good backup was 2026-06-19 02:06. At boot systemd logs Starting of mnt-pbs\x2ddatastore.automount unsupported. — the kernel autofs the automount unit relies on isn't permitted in the container — so the NFS datastore never mounts. PBS then opens an empty dir and every job dies with unable to open chunk store at "/mnt/pbs-datastore/.chunks" - No such file or directory; in PVE the nas-primary storage shows inactive and pvestatd logs the chunk-store error every ~10s. It "worked" for ~2 months only because the Apr 23 role run mounted it once at apply time and it stayed up until the reboot — drift runs in --check mode never re-mount. Fixed 2026-06-20: options are now a plain _netdev,nofail mount (host-side mount + bind was rejected because the QNAP NFS export is IP-locked to the container .246, and would have needed an appliance-side ACL change), paired with the datastore healthcheck timer so a missed mount is loud. Recovery for a live miss is a one-liner: pct exec 105 -- mount /mnt/pbs-datastore. See Backup & Restore — datastore not mounted.
  • PBS official docs don't list NFS/CIFS as a supported datastore filesystem (only local ext4/xfs/zfs + S3), gated on the ≥65538-subdirs-per-directory requirement, with 1–2 orders-of-magnitude performance and locking/fsync caveats. This homelab runs the unsupported-but-functional NFS path deliberately (DR second failure domain; the TS-269L can't run PBS itself). Keep an eye on GC/verify health.
  • PBS 4.x ACLs need BOTH user AND token auth-ids on the same path. Corrected 2026-05-06 during Phase 5E execution after a longstanding misreading. The original belief — that granting on the user alone caused the token to inherit — is wrong: the user grant resolves correctly for the user auth-id but the token auth-id resolves to {} permissions until it is also explicitly granted. The Phase 5A pbs-pve ACL works because /etc/proxmox-backup/acl.cfg has both authids on one comma-separated line (pbs-pve@pbs,pbs-pve@pbs!pve:DatastoreAudit,DatastoreBackup) — the original 5A operator manually added the second grant. Fixed in the role 2026-05-06: a new Ensure datastore ACLs for PBS client TOKEN auth-id task grants the same role to {{ pbs_client_userid }}!{{ pbs_client_token_name }}, gated on vault_pbs_token_id is defined (so it's a no-op on the first role run before the operator generates the token, then activates on re-run). New default pbs_client_token_name: "pve" controls the token name. Diagnostic: proxmox-backup-manager user permissions <authid> --path <p> --output-format json returns {} when broken vs the role dict when working.
  • DatastoreBackup alone is not enough for pvesm add pbs. The PVE-side registration probes /admin/datastore/<name>/status which needs Datastore.Audit. Grant both DatastoreBackup and DatastoreAudit.
  • PBS schedule parser rejects daily HH:MM. daily is a systemd OnCalendar macro (= *-*-* 00:00:00) and can't be combined with a time. Use plain HH:MM (interpreted as *-*-* HH:MM:00). Weekday + time (sun 04:00) works fine.
  • Debian 13 LXCs need features: nesting=1. Without it, systemd 257 fails ~24 core units on first boot (gettys, journald, logind, networkd, tmpfiles). pct create prints a warning but doesn't auto-apply.
  • get_url in check mode reports changed even with force: false. Stat-gate the fetch on file absence (same pattern as the docker role's armored-key task).
  • proxmox-backup-server pulls in zfsutils-linux, which ships zfs-zed, zfs-mount, and zfs-share systemd units. Inside an LXC there are no zpools visible to the container, so all three exit 1/FAILURE on start. Mask them (the role handles this) or they show as failed in systemctl --failed and trip Beszel's failed-services alerting.
  • PBS notification CLI quirks — proxmox-backup-manager notification endpoint webhook create takes <name> as a positional argument (not --name), uses per-type endpoint webhook list for idempotency probes (not a global list), and despite the CLI help text saying --body is "The HTTP body to send. Supports templating." (no mention of encoding), the body must still be base64-encoded or the endpoint stores but fails at delivery with "could not decode base64 value" — same quirk as PVE. --header value is also base64. Covered in roles/pbs/tasks/notifications.yml.

Handlers

  • restart proxmox-backup-proxy
  • restart proxmox-backup
  • Reload systemd — daemon-reload after the healthcheck unit files change

Multi-user / multi-namespace pattern (post-Phase 5E)

The role currently codifies a single PBS user (pbs_client_userid) for the PVE-side guest backup integration. Phase 5E (host-level file backup, 2026-05-06) introduced a second PBS user (host-backup@pbs) scoped to a separate namespace (host/proxfold) — created manually as a one-shot during bootstrap rather than generalising this role into a list-of-users abstraction. Rationale: one extra user didn't justify the refactor surface.

If/when a third PBS client appears (e.g. a second host backing up its own configs, or an off-site sync target), revisit: turn pbs_client_userid / pbs_client_roles into a list pbs_client_users[] and iterate the user/ACL/notification tasks. Bootstrap procedure for the manual second-user pattern lives in backup-restore runbook §host-level file backup bootstrap.

The role's existing verify-job is datastore-wide (no --ns filter), so it covers the new host/proxfold namespace automatically. The prune-job is namespace-scoped — nas-primary-prune only touches the root namespace; the host namespace gets its own nas-primary-host-prune with longer retention created during the Phase 5E bootstrap.