Skip to content

Role: nvidia

Installs and configures the Nvidia T400 GPU driver stack on the Proxmox host. Covers driver installation, NVENC session limit removal, persistence daemon, udev device rules, LXC passthrough verification, and IPMI fan speed control.

Hosts: proxfold

Tasks

Driver installation

Task Tag
Install build dependencies (build-essential, proxmox-headers, dkms) nvidia, driver
Check currently installed driver version nvidia, driver
Download driver installer (if version mismatch) nvidia, driver
Install driver silently with DKMS (--silent --dkms) nvidia, driver

Driver version is pinned in proxfold.yml:

nvidia_driver_version: "550.163.01"

A reboot is required after first install. The playbook notifies but does not auto-reboot.

PVE 9: pve-headers → proxmox-headers

On PVE 9 / Debian trixie, the meta-package name changed from pve-headers to proxmox-headers (and the versioned package is proxmox-headers-<kernel>). The role installs the versioned package for the currently booted kernel so DKMS rebuilds against the right source tree — currently proxmox-headers-6.14.11-6-pve since proxfold is GRUB-pinned to kernel 6.14.x.

DKMS rebuild on kernel updates

The 550.163.01 module is DKMS-registered, so a new 6.14.x point release will trigger an automatic rebuild. However the GRUB pin on proxfold references the exact 6.14.11-6-pve menuentry — when a newer 6.14 patch is installed, update GRUB_DEFAULT in /etc/default/grub to the new menuentry ID and update-grub before the next reboot, otherwise the box boots into the old pinned kernel.

NVENC patch

Task Tag
Clone keylase/nvidia-patch to /opt/nvidia-patch (pinned to a SHA) nvidia, nvenc
Apply patch (idempotent — skips if already patched) nvidia, nvenc

Removes Nvidia's artificial NVENC concurrent session limit, enabling unlimited simultaneous hardware transcodes.

Why the keylase repo is pinned to a specific SHA

The role pins version: on the ansible.builtin.git task to a specific upstream commit (currently 80e48e9 from 2026-04-03). The earlier version: master form caused every keylase commit to surface as drift in the daily check — the repo is updated frequently with new Windows driver patches that don't affect the Linux side. Pinning satisfies ansible-lint's git-latest rule and freezes the patch DB at a state verified working with the running driver (550.163.01).

To bump the pin (when a new Linux driver lands in the repo, or before upgrading nvidia_driver_version):

  1. On proxfold: cd /opt/nvidia-patch && git fetch && git rev-parse origin/master
  2. Smoke-test the patch DB against the running driver — bash patch.sh --list-versions 2>&1 | grep "$(nvidia-smi --query-gpu=driver_version --format=csv,noheader)" should match.
  3. Update the SHA literal in roles/nvidia/tasks/main.yml Clone NVENC patch repo, commit, push.
  4. Reconcile: ansible-playbook playbooks/site.yml --limit proxfold --tags nvenc from CT 104.

Persistence daemon

Task Tag
Deploy nvidia-persistenced.service unit nvidia, persistence
Enable and start service nvidia, persistence

Keeps the GPU initialised between uses — reduces first-transcode latency.

Udev rules

Task Tag
Deploy udev rules to /etc/udev/rules.d/70-nvidia.rules nvidia, devices

Ensures stable Nvidia device nodes persist across reboots.

LXC passthrough

Task Tag
Stat /etc/pve/lxc/<CTID>.conf (skip downstream if CT doesn't exist yet) nvidia, lxc
Remove legacy ANSIBLE MANAGED NVIDIA PASSTHROUGH block markers nvidia, lxc
Ensure each lxc.cgroup2.devices.allow line is present (per cgroup major) nvidia, lxc
Ensure each lxc.mount.entry line is present (per Nvidia device node) nvidia, lxc

The role writes the cgroup allow lines and bind-mount entries directly into /etc/pve/lxc/<container_id>.conf for the container identified by nvidia_lxc_passthrough.container_id (CT 100 on proxfold).

Why lineinfile per line, not blockinfile

PVE's LXC config parser rewrites the conf file on every container config change (pct set, pct start/stop, GUI edit) and during that rewrite it moves every raw lxc.* key to the end of the file, below the PVE-managed keys. With blockinfile the # BEGIN/# END markers stay where they were written while the managed lines migrate out of the block. The next playbook run sees an empty block, re-fills it, PVE moves the content out again — perennial drift reporting changed=1 forever while the passthrough actually works fine. Managing per-line via lineinfile state=present is transparent to the reorder: each task just checks whether one exact line exists somewhere in the file.

The first run on any host with a legacy block will strip the # BEGIN ANSIBLE MANAGED NVIDIA PASSTHROUGH / # END markers — the lines inside them are kept via the lineinfile tasks.

Fresh install / pre-restore

The stat gate makes the passthrough tasks no-op if the LXC conf doesn't exist yet — expected on a rebuild between proxmox-host.yml and vzdump restore of CT 100. After CT 100 is restored, re-run ansible-playbook playbooks/proxmox-host.yml --limit proxfold --tags lxc to apply the passthrough. After apply, pct stop 100 && pct start 100 is required once to pick up the new cgroup allowances and bind mounts.

IPMI fan fix

Task Tag
Install ipmitool (when ipmi_fan_fix: true) nvidia, ipmi, fan
Deploy /usr/local/bin/ipmi-fan-fix.sh and /etc/systemd/system/ipmi-fan-fix.service nvidia, ipmi, fan
Enable and start the service nvidia, ipmi, fan
Absent path: stop, disable, and remove script + unit (when ipmi_fan_fix: false) nvidia, ipmi, fan

The Dell R430's BMC can panic-ramp fans to 100% when it detects an unrecognised PCIe card. When that is happening, set ipmi_fan_fix: true — a script sets manual fan control and holds speed at ipmi_fan_speed_percent.

Currently off on proxfold

ipmi_fan_fix: false on proxfold — the BMC handles the T400 correctly (idle 1500–3600 RPM, CPU 55–74 °C at 20 °C inlet) and a flat duty cycle is strictly worse than BMC auto (louder at idle, no ramp headroom under load). Re-enable only if real panic-ramp events are observed under load; in that case prefer a dynamic curve script over a flat percentage.

The flag is self-healing

Setting ipmi_fan_fix: false triggers the absent path: the role stops the service, disables it, removes the script and unit file, and reloads systemd. No manual cleanup required when flipping the flag off.

Key variables

Variable Source Value
nvidia_driver_version proxfold host_vars 550.163.01
nvidia_nvenc_patch proxfold host_vars true
nvidia_persistence_daemon proxfold host_vars true
nvidia_lxc_passthrough.container_id proxfold host_vars 100
nvidia_lxc_passthrough.cgroup_majors proxfold host_vars [195, 234, 237]
nvidia_lxc_passthrough.devices proxfold host_vars /dev/nvidia0, nvidiactl, nvidia-modeset, nvidia-uvm, nvidia-uvm-tools, nvidia-caps/nvidia-cap1, nvidia-caps/nvidia-cap2
ipmi_fan_fix proxfold host_vars false (see above)
ipmi_fan_speed_percent proxfold host_vars 30 (only used when the fix is enabled)

Templates

Template Deploys to
nvidia-persistenced.service.j2 /etc/systemd/system/nvidia-persistenced.service
nvidia-udev.rules.j2 /etc/udev/rules.d/70-nvidia.rules
ipmi-fan-fix.sh.j2 /usr/local/bin/ipmi-fan-fix.sh
ipmi-fan-fix.service.j2 /etc/systemd/system/ipmi-fan-fix.service

Handlers

  • Restarts nvidia-persistenced on service unit change
  • Reloads udev on rules change
  • Notifies "reboot required" if driver is newly installed