Skip to content

n8n

Lab/glue workflow automation, running as a Docker stack on the dedicated n8n VM. Phase 5C scope: stand up the instance for hands-on exploration; workflows themselves are intentionally not yet codified.

Web UI: https://n8n.rampancy.cloud (LAN: http://192.168.1.248:5678)

Service details

Property Value
Host n8n VM (192.168.1.248) — see proxfold guests
Image n8nio/n8n:latest (Docker Hub direct — see Lessons)
Database SQLite (n8n default) under named volume n8n_data
Compose stacks/n8n/docker-compose.yml
Management Dockhand (Git-backed Docker Compose), agent = Hawser on the n8n VM
Reverse proxy n8n.rampancy.cloud → 192.168.1.248:5678 via the nginx VM
Backup Captured by pbs-daily (all-guests selector); first ad-hoc snapshot 2026-04-28

Architecture

flowchart LR
    Browser[Operator browser]
    subgraph nginx["nginx VM · 192.168.1.249"]
        Proxy[Reverse proxy<br/>n8n.rampancy.cloud → :5678]
    end
    subgraph n8nvm["n8n VM · 192.168.1.248"]
        Hawser[Hawser agent<br/>WebSocket → arrstack:3000]
        N8N[n8n container<br/>:5678]
        SQLite[(SQLite<br/>n8n_data volume)]
        N8N --> SQLite
    end
    subgraph arrstack["arrstack VM · 192.168.1.252"]
        Dockhand[Dockhand controller<br/>:3000]
    end
    Browser --> Proxy --> N8N
    Dockhand <-. WebSocket .-> Hawser
    Hawser --> N8N

Deployment flow

  1. Compose lives in homelab-ansible/stacks/n8n/ — not edited on the host.
  2. Dockhand (on arrstack) reads compose from git on its sync schedule (or manual deploy).
  3. Dockhand sends compose up over the WebSocket to the Hawser agent on the n8n VM.
  4. Hawser executes against the local Docker daemon — docker compose up -d against /data/stacks/n8n/docker-compose.yml.
  5. n8n container starts, binds 0.0.0.0:5678, persists workflows + DB to the n8n_data named volume.

The named volume sits under /var/lib/docker/volumes/n8n_data/_data on the n8n VM, which means it's part of the VM's vzdump rootfs snapshot — PBS backup is automatic, no separate volume wiring.

Database — SQLite (Phase 5C default)

n8n's bundled SQLite is sufficient for lab/glue use. The migration trigger to PostgreSQL (per n8n docs and practitioner consensus) is roughly 5–10k executions/day or 4–5 GiB DB — comfortably above lab usage.

If/when migration is needed, use n8n's export:credentials + export:workflow CLI on the SQLite side, fresh Postgres-backed instance, re-import. Execution history is dropped either way; accepted cost.

Reverse-proxy env vars

The compose currently leaves N8N_HOST / N8N_PROTOCOL / WEBHOOK_URL commented. n8n works behind the nginx proxy without them, but webhooks and OAuth callback URLs will use the bare LAN address (http://192.168.1.248:5678) instead of https://n8n.rampancy.cloud. When the first webhook-using or OAuth-using workflow is built, uncomment those vars in the compose, push, let Dockhand redeploy.

Lessons from the 2026-04-28 deploy

This stack hit four distinct problems before landing. All four worth knowing about because they map onto traps that anyone building Docker stacks on PVE 9 + a Dockhand/Hawser fleet in 2026 can step on.

1. Modern n8n is too fat for the npm-on-LXC pattern

The original Phase 5C plan was native Node.js install on a small unprivileged LXC, mirroring the Beszel hub shape. Modern n8n (v2.17+) bundles a large AI SDK ecosystem (@ai-sdk/anthropic, @ai-sdk/google, the full Vercel AI SDK, langchain, isolated-vm, etc.) which:

  • Peaks past 4 GB RAM during npm dep resolution + node-gyp native compile — the OOM killer fires on a 4 GB LXC partway through install, leaving a wedged half-installed tree.
  • Won't compile against Debian trixie's apt-managed Node.js — Debian patches V8 separately from upstream Node, and isolated-vm is written against the canonical upstream V8 ABI. Compile fails with 'SourceLocation' in namespace 'v8' does not name a type.

The pivot to Docker (n8n team's recommended path; image is prebuilt) sidesteps both. Rule of thumb captured: for Node-heavy apps that bundle native modules in 2026, use the vendor's official Docker image, not source install.

2. Docker 29's default storage driver breaks pulls

Docker 29.x defaults to containerd-snapshotter=true / storage-driver=overlayfs (io.containerd.snapshotter.v1). This new image store hits an extraction-snapshot race on certain pull patterns — dockerd logs failed to prepare extraction snapshot ... NotFound: snapshot does not exist and the layer extraction loops forever. Documented as ddev/ddev#8136, supposedly fixed in 29.2.1+ but still surfaces on 29.4.1.

Workaround: force the legacy graph driver — "storage-driver": "overlay2" in /etc/docker/daemon.json. Codified per-host in inventory/host_vars/n8n.yml as docker_daemon_config: { storage-driver: overlay2 }, which the docker role renders.

Why the other Docker hosts (arrstack, nginx) didn't hit this: Docker 29's containerd-snapshotter default only applies to fresh installs. Hosts upgraded across the 28 → 29.x boundary keep their existing overlay2 backend. arrstack and nginx were both already on overlay2 (arrstack via explicit daemon.json, nginx by upgrade-path default) before Docker 29 landed, so the new default never bit them. The n8n VM was the only fresh Docker 29 install in the fleet, which is why only it tripped the bug.

3. Hawser's default REQUEST_TIMEOUT is 30 seconds

Hawser's per-request timeout for handling a Docker compose operation defaults to 30 seconds. That's far shorter than the time required to pull any non-trivial image (n8n's first pull is ~600 MB, takes 1–3 min on home internet). Symptom: dockerd logs Not continuing with pull after error / context canceled exactly 30s after the pull starts, Dockhand reports the deploy as Success: false, and the user sees a partial extraction stream that just stops.

Fix: set REQUEST_TIMEOUT=600 (10 min) in the agent's env. Codified in hawser role defaults. Worth applying to nginx VM's existing manual install when nginx is brought under the role.

4. Don't pull on docker.n8n.io if Docker is on containerd-snapshotter

A red herring during diagnosis but worth recording: docker.n8n.io is n8n's CDN frontend that redirects to registry-1.docker.io. The redirect handling in Docker 29's containerd-snapshotter logs a "Host doesn't match" cfgHost=docker.n8n.io host=registry-1.docker.io warning. It's not the actual cause of pull failures (the snapshot bug above is) but it's noise that misleads diagnosis. The compose uses plain n8nio/n8n:latest (Docker Hub direct) instead.