Skip to content

Matrix

Self-hosted, federated Matrix homeserver. Closed-membership (token-gated registration); federation enabled to a small allowlist of trusted homeservers. Replaces day-to-day Discord chat for me + a small circle of family and friends.

Web client: https://app.element.io (set homeserver to rampancy.cloud). Server name: rampancy.cloud (users are @<localpart>:rampancy.cloud, e.g. @rampancy:rampancy.cloud). Public hostname: matrix.rampancy.cloud (where the HTTP service lives; resolved automatically via apex well-known delegation).

Phase 6E fully complete — 2026-05-22

Tuwunel v1.7.0 on VM 111 fronted by edge Caddy. Federation tester green via apex /.well-known/matrix/{server,client} delegation. First admin user @rampancy auto-promoted via grant_admin_to_first_user. MatrixRTC live via 5 UDM port-forwards; Element Call validated end-to-end (desktop ↔ Element X mobile cellular, audio + video + screen-share).

Service details

Property Value
Host matrix VM (VM 111, 192.168.1.243) — see proxfold guests
Server name rampancy.cloud (apex)
Public client URL https://matrix.rampancy.cloud
Public federation URL https://matrix.rampancy.cloud:443 (port 8448 NOT opened)
Homeserver implementation Tuwunel — Rust-based, embedded RocksDB, conduwuit lineage
Version pin matrix_tuwunel_version: v1.7.0 (see Lessons appendix)
Reverse proxy Edge Caddy on CT 107 → VM 111:81 (Traefik web entrypoint)
Federation Allowlist mode: rampancy.cloud, matrix.org, chat.dacool.zone
Registration Token-gated (allow_registration: true + invite token; no open signup)
Database RocksDB (Tuwunel embedded); Postgres installed as standby for future bridges (mautrix-discord etc.)
Backup PBS daily 02:00 VM snapshot — see pbs-daily job
Deployment spantaleev/matrix-docker-ansible-deploy vendored on CT 104 at /root/matrix-deploy/

For new users — joining instructions

Forward the block below to anyone you want on the server. Generate them a registration token via Generate a new registration token below first, and share the token through a side channel (Signal, in person, etc.) — not the same channel as these instructions.

Joining the rampancy.cloud Matrix server

  1. Install a Matrix client:
  2. Mobile (recommended): Element X — iOS / Android
  3. Browser/desktop: open https://app.element.io (no install needed)
  4. Choose "Create account" / "Sign up".
  5. When asked for the homeserver, enter: rampancy.cloud
  6. Pick a username and password.
  7. When prompted for a registration token, enter the one you were sent.
  8. Element will offer to set up a recovery key — accept and save it in a password manager. This lets you verify new devices later without losing encrypted chat history.
  9. Your Matrix ID is @<your-chosen-username>:rampancy.cloud. Send this back so I can DM you.

Recovery-key flow ≠ the cause of any send issue

Yesterday's setup turned up red-banner "messages not sent" errors after a user finished the recovery-key flow — that was a server-side configuration bug (allowlist missing the local server name), not the recovery flow. Setting up the recovery key is the right thing to do; it'll work cleanly now.

Service layout on VM 111

Spantaleev's playbook installs 9 Docker containers as systemd units. All managed via systemctl matrix-*; configs under /matrix/<service>/.

Container Role Internal port
matrix-tuwunel Homeserver (Matrix client/federation API, RocksDB storage) 6167 (HTTP) inside container
matrix-traefik Internal reverse proxy in front of all matrix-* services 81 (host bind — single entrypoint, web+federation)
matrix-livekit-server WebRTC SFU for group voice/video calls (MatrixRTC) — internal-only until 6E.4 ports open various RTC ports
matrix-livekit-jwt-service Mints short-lived JWTs to authenticate clients to LiveKit 8080
matrix-static-files Serves Matrix-spec static files from matrix.rampancy.cloud (well-known on matrix subdomain; apex statics served from edge Caddy) static
matrix-postgres Postgres 18 — currently unused by Tuwunel, ready for future bridges 5432 (internal)
matrix-client-element Self-hosted Element Web instance (not currently exposed externally; use app.element.io) nginx
matrix-container-socket-proxy Restricted Docker socket exposed read-only to services that need container introspection unix socket
matrix-exim-relay Outbound email relay for password resets / notifications (not yet configured to relay externally) 8025

Filesystem layout on VM 111:

Path Purpose
/matrix/tuwunel/config/tuwunel.toml Rendered homeserver config (do NOT edit; rendered from CT 104 by spantaleev)
/matrix/tuwunel/data/ RocksDB — the actual message store. This is the data that matters.
/matrix/traefik/config/ Internal Traefik config
/matrix/livekit-server/, /matrix/livekit-jwt-service/ RTC service configs
/etc/systemd/system/matrix-*.service Spantaleev-rendered systemd units

Admin tasks

All admin operations happen from CT 104 (/root/matrix-deploy/), not from VM 111 directly. Editing files on VM 111 will get overwritten on the next spantaleev playbook run.

Common quick-reference

Task Command (from CT 104)
Restart all matrix services on the VM ansible-playbook -i inventory/hosts setup.yml --tags=start --vault-password-file /root/.vault_pass
Pull latest spantaleev + roles cd /root/matrix-deploy && git pull && rm -rf roles/galaxy && ansible-galaxy install -r requirements.yml -p roles/galaxy/ --force
Apply a vars.yml change ansible-playbook -i inventory/hosts setup.yml --tags=install-all,start --vault-password-file /root/.vault_pass (always pipe \| tee somefile.log with set -o pipefail so errors don't get swallowed)
Stop the whole stack (e.g. for maintenance) On VM 111: sudo systemctl stop 'matrix-*'
View Tuwunel logs On VM 111: sudo journalctl -u matrix-tuwunel -f
Check container health On VM 111: sudo docker ps --filter name=matrix- --format 'table {{.Names}}\t{{.Status}}'

Generate a new registration token

Tokens are stored in the homelab-ansible vault. Rotate via the append-only /dev/shm pattern (per [[feedback_never_view_vault_to_scrollback]]):

# On WSL, in homelab-ansible/
VAULT=inventory/group_vars/all/vault.yml
WORK=/dev/shm/vault.work.$$
BACKUP="$VAULT.bak.$(date +%s)"
trap 'shred -u "$WORK" 2>/dev/null || true' EXIT
cp "$VAULT" "$BACKUP"
ansible-vault decrypt --output "$WORK" "$VAULT"
chmod 0600 "$WORK"
sed -i '/^vault_matrix_registration_token:/d' "$WORK"
VAL=$(openssl rand -hex 32)
printf 'vault_matrix_registration_token: "%s"\n' "$VAL" >> "$WORK"
unset VAL
ansible-vault encrypt --output "$VAULT" "$WORK"
shred -u "$WORK"
echo "Token rotated. Commit + push, then re-apply install-all on CT 104."

Then:

git add inventory/group_vars/all/vault.yml
git commit -m "matrix: rotate registration token"
git push origin main

# Wait ~5s for forgejo→github mirror, then on CT 104:
ssh root@192.168.1.245 'cd /root/homelab-ansible && git pull'
ssh root@192.168.1.245 'cd /root/matrix-deploy && ansible-playbook -i inventory/hosts setup.yml --tags=install-all,start --vault-password-file /root/.vault_pass'

Retrieve the new token to share with a new user — in a separate terminal (NOT via tooling that logs output):

ansible-vault view --vault-password-file ~/.vault_pass \
  ~/homelab-ansible/inventory/group_vars/all/vault.yml | grep '^vault_matrix_registration_token:'

Element Web image displays / cross-server avatars

Element Web attempts to use MSC3916 authenticated media (Tuwunel advertises support), but its implementation depends on a browser service worker that often fails to register or rewrite media URLs — particularly on Firefox, in private/incognito mode, or with stale sw.js caches. When the service worker doesn't kick in, Element Web silently falls back to the deprecated /_matrix/media/v3/* paths.

To keep the fallback working we override Tuwunel's default and re-enable the legacy media endpoints:

matrix_tuwunel_environment_variables_extension: |
  TUWUNEL_ALLOW_LEGACY_MEDIA=true

Element Desktop and Element X (mobile) don't have this issue — they hit MSC3916 endpoints natively, no service worker involved. See Lessons #9 for the full breakdown.

Add a new federation peer

Federation is in allowlist mode. Add a remote homeserver's domain to matrix_tuwunel_config_allowed_remote_server_names in /root/matrix-deploy/inventory/host_vars/matrix.rampancy.cloud/vars.yml, then re-apply install-all.

MUST keep rampancy.cloud in the list

The allowlist applies to ALL senders including the local server. Removing rampancy.cloud filters out your own messages. See Lessons appendix #1 for the full explanation.

Make an existing user admin

Tuwunel only auto-promotes the first user. To promote a later user, use the CLI mode (requires ~30s downtime — service must stop to release the RocksDB lock):

# On VM 111
sudo systemctl stop matrix-tuwunel
sudo docker run --rm \
  -v /matrix/tuwunel/data:/var/lib/tuwunel \
  -v /matrix/tuwunel/config:/etc/tuwunel:ro \
  -e TUWUNEL_CONFIG=/etc/tuwunel/tuwunel.toml \
  ghcr.io/matrix-construct/tuwunel:v1.7.0 \
  --execute "users make-user-admin @<localpart>:rampancy.cloud"
sudo systemctl start matrix-tuwunel

Don't run --execute "rooms list"

Observed to hang indefinitely under our deploy state on 2026-05-21 (caused a 6-minute outage before being killed externally). Use --execute "rooms info <room_id>" for specific rooms.

Deactivate a user

# On VM 111
sudo systemctl stop matrix-tuwunel
sudo timeout 60 docker run --rm \
  -v /matrix/tuwunel/data:/var/lib/tuwunel \
  -v /matrix/tuwunel/config:/etc/tuwunel:ro \
  -e TUWUNEL_CONFIG=/etc/tuwunel/tuwunel.toml \
  ghcr.io/matrix-construct/tuwunel:v1.7.0 \
  --execute "users deactivate @<localpart>:rampancy.cloud"
sudo systemctl start matrix-tuwunel

Deactivated users are removed from all rooms by default. Use --no-leave-rooms to keep them as ghosts.

Tuwunel admin room

When logged in as admin, you're auto-joined to a room aliased #admins:rampancy.cloud containing a server bot @conduit:rampancy.cloud (the localpart is "conduit" not "tuwunel" — legacy from Tuwunel's Conduit lineage).

Commands in that room are prefixed !admin <category> <subcommand>. Top-level categories (from !admin -h):

appservices, users, rooms, federation, server, media, debug, query, token, help

Some useful ones:

Command What it does
!admin -h Print full help in the room
!admin users list List all local users
!admin users deactivate @<user>:rampancy.cloud Deactivate via the bot (alternative to CLI mode)
!admin federation incoming-federation ~~List rooms with incoming federation activity~~ — stubbed in Tuwunel v1.7.0 (returns "This command is temporarily disabled"; the help text still lists it but the implementation is a one-line error in src/admin/federation/commands.rs). Re-check after future Tuwunel versions.
!admin server uptime Server uptime
!admin token create-and-show --uses 1 Generate a one-shot Matrix-spec registration token (alternative to rotating the vault)

Reference: Tuwunel Admin Room Commands wiki.

Backup
Schedule PBS pbs-daily job, 02:00 nightly, mode snapshot
Storage nas-primary (PBS datastore on the stash pool)
Coverage Full VM snapshot — RocksDB inside the VM is captured live (qemu-agent freeze if enabled; otherwise crash-consistent snapshot, which Tuwunel/RocksDB handle gracefully)
First snapshot 2026-05-21T16:37:49Z (~32 GiB compressed)
Restore qm restore against the PBS image — recreates VM 111 wholesale