Files

T

Sluijsens c438ec8b1d Warns against copying the reference project's nginx Sentry tunnel

The backend already serves /sentry-tunnel itself with its own rate
limit, payload cap, and DSN-derived destination - exactly what that
nginx block exists to avoid needing. Proxying the same path at nginx
intercepts it first and silently drops all three protections.

2026-07-29 13:43:33 +02:00

25 KiB

Raw Blame History

Deployment Instructions

Target: single Raspberry Pi (linux-arm64), test and production both on it, separated by directory, systemd --user unit, and local port. Domains: test.slpsoftware.nl (test), slpsoftware.nl (production).

Naming note: directories, systemd units and the website-upload path below are named slpsoftware — the customer/instance running SlpModularCms (this repo's software), not the software itself. SlpModularCms can host multiple customers; slpsoftware is this one — the first, and the owner's own site. A future second customer on the same Pi would get its own instance name throughout, following the same pattern.

1. One-Time Host Setup

Do this once, before the first automated deploy. Everything here is host configuration the workflow assumes already exists — deploy-scp.yaml (U6) never creates any of it.

1.1 Prerequisites

.NET 10 runtime installed on the Pi (ASM-03) — the publish is framework-dependent (infrastructure-design.md § 1), so the Pi needs the runtime, not the full SDK
mariadb-client (or mariadb-dump/mysqldump specifically) installed, for the backup script (§ 4) — already present on most Raspberry Pi OS images that also run mariadb-server; install mariadb-client explicitly if the dump tool isn't already there

1.2 Account Model (revised — `webadmin` is not the deploy account)

Clarified during Deployment Setup: webadmin is the FileZilla/SFTP account website-workspace authors use to upload customer sites under /mnt/storage1/www/html/ (WEBSITE_WORKSPACE.md's role) — it cannot SSH in, and should stay SFTP-only. deploy-scp.yaml needs a real SSH shell (for mkdir, ln -sfn, systemctl --user restart, the backup script), which is a different kind of access than FileZilla uses.

Preferred: a separate account for the deploy pipeline, but a generic, host-wide one — not named after this project, since it may end up deploying other projects on this Pi too (the reference SlpSoftware project's own pipeline may already have exactly this kind of account; check its PI_MAIN_USERNAME secret first and reuse it directly if it already has SSH shell access, rather than creating a second one). Example name: gitea-workflow.

sudo useradd -m -s /bin/bash gitea-workflow
sudo passwd gitea-workflow

Each project deploying through this account gets its own subdirectory under its home (§ 1.4 already namespaces by project: ~/apps/slpsoftware/<env>/), so one generic account can serve multiple projects without their release trees colliding.

Fallback, only if no SSH-capable account exists at all and creating one genuinely isn't feasible (e.g. a hosting provider that doesn't allow arbitrary new system accounts): grant webadmin SSH shell access instead (sudo usermod -s /bin/bash webadmin, plus enabling SSH password/key auth for it if currently blocked at the sshd_config level). This merges the FTP and deploy roles onto one account — acceptable as a fallback, but worth revisiting later, since it means a website-workspace author's FTP credential would also be able to run shell commands on the Pi.

The rest of this document uses gitea-workflow as the example account name — rename consistently if you pick something else.

1.3 Enable Lingering (INFRA-U6-01 — do this first, easy to forget)

sudo loginctl enable-linger gitea-workflow

Without this, the systemd --user service manager is torn down when the deploy SSH session ends, killing the just-restarted app a few seconds after every successful deploy. This also fixes the common "Failed to connect to bus" error systemctl --user can throw when invoked from a non-interactive SSH command — lingering keeps the user's systemd instance (and XDG_RUNTIME_DIR) running independent of any login session.

If systemctl --user ... still fails with Failed to connect to bus: No medium found while setting the rest of this up manually (before the deploy workflow ever runs): enabling lingering does not retroactively start the user manager — that happens on the next real login, reboot, or manually:

sudo systemctl start user@$(id -u gitea-workflow).service

And run systemctl --user commands from a real login shell for that account — not sudo -u gitea-workflow systemctl --user ..., sudo -i -u gitea-workflow, or su - gitea-workflow from your own session. All three are common ways to reach this exact error even when the user manager is already running and /run/user/<uid>/bus already exists: none of them reliably go through pam_systemd (the PAM module that actually exports XDG_RUNTIME_DIR), because /etc/pam.d/su and most sudo PAM configs don't include it, unlike /etc/pam.d/sshd or /etc/pam.d/login. Confirmed in practice: su - gitea-workflow reproduces this exactly.

Two fixes, in order of preference:

SSH in directly as gitea-workflow instead of logging in as yourself and switching user — a real SSH login does go through sshd's PAM stack and sets XDG_RUNTIME_DIR correctly
Or, after su -/sudo -i -u, just set it by hand once per shell:
```
export XDG_RUNTIME_DIR=/run/user/$(id -u)
```

This does not affect the actual deploy workflow — deploy-scp.yaml always connects over a genuine SSH session (sshpass ssh ...), which sets XDG_RUNTIME_DIR correctly on its own. This whole gotcha is specific to poking around on the host by hand via su/sudo -i.

Verify with loginctl show-user gitea-workflow | grep Linger (expect Linger=yes) and ls /run/user/<uid> (should exist once the user manager has actually started).

1.4 Directory Skeleton

Deliberately placed under gitea-workflow's own home directory, not under /mnt/storage1/www/html/ — since it's now a separate account from webadmin, there is no reason for the CMS's own release/current/shared structure to live anywhere near the other websites at all, which directly avoids interfering with them (as you asked in Q2):

mkdir -p ~/apps/slpsoftware/<env>/releases

current is created by the first deploy itself (ln -sfn) — don't pre-create it.

shared/wwwroot-web is the one exception — it must resolve to wherever webadmin actually uploads this customer's website via FileZilla, e.g. /mnt/storage1/www/html/slpsoftware/<env>/ (adjust the exact folder name to whatever convention the other sites under html/ already use, if one exists). Rather than a plain directory, make it a symlink across accounts:

mkdir -p ~/apps/slpsoftware/<env>
ln -s /mnt/storage1/www/html/slpsoftware/<env> ~/apps/slpsoftware/<env>/shared/wwwroot-web

deploy-scp.yaml's existing logic (releases/{ts}/wwwroot/web -> ../../../shared/wwwroot-web) needs no changes for this — it only ever resolves the symlink chain, it doesn't care how many hops that chain has. What does need attention: gitea-workflow needs read + traverse permission on /mnt/storage1/www/html/slpsoftware/<env>/ and its parent directories, which webadmin owns. Simplest fix: put both accounts in a shared group (e.g. webshared), chgrp -R webshared that folder, and make sure webadmin's FTP server creates new uploads group-readable (g+rx, not just owner-readable) — a one-time permission setup, not something either pipeline touches per deploy.

1.5 Runtime Configuration File

One file per environment, outside the release directory so it survives every switch:

touch ~/apps/slpsoftware/<env>/shared/env
chmod 600 ~/apps/slpsoftware/<env>/shared/env

Contents (fill in real values — this file is never read by the workflow, only by the systemd unit below):

ASPNETCORE_ENVIRONMENT=Production
# 0.0.0.0, not localhost: the reverse proxy handling TLS for this domain runs on a SEPARATE Pi
# (§ 1.7), so it must reach Kestrel over the LAN, not loopback. See § 1.7's firewall note — binding
# to all interfaces means the port must be restricted to the proxy Pi's address, not left open to
# the whole LAN.
ASPNETCORE_URLS=http://0.0.0.0:<port>
ConnectionStrings__DefaultConnection=Server=127.0.0.1;Port=3306;Database=SlpSoftware<Env>;Uid=<user>;Pwd=<password>
JwtSettings__Secret=<secure-long-random-secret>
JwtSettings__Issuer=SlpModularCms
JwtSettings__Audience=SlpModularCmsPortal

# Optional — only meaningful if this instance manages slave CMS instances via the /cms page.
# MasterModuleOptions.MasterUrl is nullable and unvalidated at startup: if left unset, the Master
# module's background reconciliation just logs a warning and skips its work ("MasterUrl not
# configured; skipping integrity check") — it never crashes or blocks startup. Safe to leave out
# entirely if you don't plan to register any slave instances under this deployment; set it to this
# environment's own public URL (e.g. https://slpsoftware.nl) if you do.
MasterModule__MasterUrl=https://<this-environment-domain>

# Optional — empty/absent is a fully supported state: Sentry is simply skipped and console logging
# continues (see Observability section in README.md).
Observability__SentryDsn=<sentry-dsn-or-empty>

# Optional — both only needed if this instance uses Umami analytics (VITE_UMAMI_SCRIPT_URL set for
# the frontend build). Leave both lines out entirely if you don't use Umami; there is no other
# origin either one needs by default.
#
# Set BOTH to the Umami script's origin (scheme + host only, no path — e.g. if
# VITE_UMAMI_SCRIPT_URL=https://analytics.slpsoftware.nl/script.js, use
# https://analytics.slpsoftware.nl for both lines below):
#   - AllowedScriptOrigins: lets the browser load Umami's tracking script (CSP script-src)
#   - AllowedConnectOrigins: lets that script send its analytics beacons back (CSP connect-src) —
#     loading a script and letting it phone home are two separate CSP directives
#
# Sentry does NOT need an entry here, on either line: U4 built a same-origin tunnel
# (Program.cs -> MapSentryTunnel()) specifically so browser error reports never leave this origin,
# precisely to avoid needing a connect-src exception (and to dodge ad blockers, which commonly
# block direct requests to Sentry's own domains).
SecurityHeaders__AllowedScriptOrigins__0=<umami-script-origin-or-omit-this-line>
SecurityHeaders__AllowedConnectOrigins__0=<same-umami-script-origin-or-omit-this-line>

Use ports 5100 (test) and 5101 (production) unless something else on the Pi already occupies them. ASPNETCORE_ENVIRONMENT=Production is used for both environments deliberately — Development disables HSTS and exposes the Scalar API explorer (Program.cs), neither of which should be true for anything reachable at a real domain, including test.

SecurityHeaders__AllowedScriptOrigins__0 / _AllowedConnectOrigins__0 must exactly match the Gitea variables SECURITY_ALLOWED_SCRIPT_ORIGINS_TEST / _PRODUCTION (see § 1.9) — REF-U5-01's CI gate only catches drift between the frontend build and that Gitea variable; it cannot see this file, so keeping the two in sync is a manual discipline, not something enforced automatically.

1.6 systemd User Units

Create ~/.config/systemd/user/slpsoftware-test.service:

[Unit]
Description=SlpModularCms API (test)
After=network.target

[Service]
WorkingDirectory=%h/apps/slpsoftware/test/current
ExecStart=/usr/bin/dotnet %h/apps/slpsoftware/test/current/SlpModularCms.Api.dll
EnvironmentFile=%h/apps/slpsoftware/test/shared/env
Restart=on-failure
RestartSec=5
KillSignal=SIGINT
TimeoutStopSec=20

[Install]
WantedBy=default.target

And ~/.config/systemd/user/slpsoftware-production.service — identical, with test replaced by production throughout (including the port inside shared/env).

Enable both (does not start them yet — nothing is deployed there until the first CI run):

systemctl --user daemon-reload
systemctl --user enable slpsoftware-test.service
systemctl --user enable slpsoftware-production.service

1.7 TLS-Terminating Reverse Proxy — On a Separate Pi, Not This One

Revised from the original design: the reverse proxy that terminates TLS for these domains runs on a different, dedicated Pi ("the proxy Pi"), not on this one ("pi-main", where gitea-workflow and the release directories from §§ 1.2–1.4 live). The proxy Pi already handles SSL and forwards plain HTTP to pi-main. Two consequences that change earlier sections:

Kestrel must be reachable over the LAN, not just loopback — hence ASPNETCORE_URLS=http://0.0.0.0:<port> in § 1.5, not http://localhost:<port>
pi-main runs no nginx and holds no certificates for these domains at all — everything in this section happens on the proxy Pi, except the firewall step at the end, which is on pi-main

As in § 1.7.1–1.7.2 below: § 1.7.1 is the repeatable, from-scratch procedure for routing any new domain through the proxy Pi to any backend (this deployment's domains included, the first time); § 1.7.2 is what that procedure produced for slpsoftware.nl — the worked example, not a separate step.

Do not add a /sentry-tunnel proxy block on the proxy Pi. The reference project (a static frontend with no backend of its own) had to tunnel through nginx directly to Sentry's ingest URL. This application does not need that: SentryTunnelExtensions.cs (built in U4) serves /sentry-tunnel itself, with its own rate limiter, a payload-size cap (Observability:TunnelMaxPayloadBytes), and a destination derived from Observability__SentryDsn rather than a hardcoded org/project ID. An nginx-level tunnel block at the same path intercepts the request before it ever reaches pi-main, silently bypassing all three of those protections — let /sentry-tunnel fall through the plain location / block like any other request instead.

1.7.1 Adding a New Domain From Scratch (on the proxy Pi)

Starting from nothing — no existing server block, no certificate — for a new domain <new-domain> that should route to <backend-host>:<backend-port> (a service on some other Pi or container on the LAN — pi-main and its two ports, in this deployment's case):

Step 1 — plain HTTP block, no TLS yet. Certbot's nginx plugin (step 2) needs a working HTTP server block for the domain to attach to and to answer the HTTP-01 validation challenge; asking for a certificate before this exists will fail.

server {
    listen 80;
    server_name <new-domain>;
    location / {
        proxy_pass http://<backend-host>:<backend-port>;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Save as /etc/nginx/sites-available/<new-domain>, symlink it into sites-enabled/, then:

sudo nginx -t && sudo systemctl reload nginx

Confirm the domain's DNS A/AAAA record already points at the proxy Pi (not the backend) before continuing — the HTTP-01 challenge in step 2 needs the domain to resolve to whichever host is answering on port 80, which is the proxy Pi.

Step 2 — request and install the certificate. The nginx plugin edits the file from step 1 in place: it adds the listen 443 ssl block, the certificate/key paths, and (by default) an HTTP→HTTPS redirect for the port 80 block:

sudo certbot --nginx -d <new-domain>

Certbot's own systemd timer handles renewal automatically — nothing further to set up for that.

Step 3 — verify. After certbot finishes:

curl -I https://<new-domain>/health   # or whichever path this new site/API actually serves

Confirm it resolves over HTTPS with a valid certificate and reaches the expected backend.

Repeat steps 1–3 once per domain. This is the same procedure regardless of whether the new domain is another environment for this feature, a completely different project's API, or a plain static site — nginx and certbot don't know or care what's actually listening on the backend they proxy to.

Step 4 — restrict the backend port on pi-main (do this once the domain works end to end). Since Kestrel now listens on 0.0.0.0:<backend-port> (§ 1.5), anything on the LAN can reach it directly, bypassing the proxy Pi's TLS entirely, unless pi-main's firewall says otherwise. On pi-main:

sudo ufw allow from <proxy-pi-lan-ip> to any port <backend-port> proto tcp
sudo ufw deny <backend-port>/tcp

(Or the equivalent nftables/iptables rules if ufw isn't what this Pi uses — the point is: only the proxy Pi's address may reach these ports, everything else is denied.)

1.7.2 Current State for This Deployment

Running the procedure above (on the proxy Pi) for test.slpsoftware.nl and slpsoftware.nl, pointing at pi-main's LAN address, produces server blocks equivalent to:

server {
    listen 443 ssl;
    server_name test.slpsoftware.nl;
    location / {
        proxy_pass http://<pi-main-lan-ip>:5100;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    # certbot-managed ssl_certificate / ssl_certificate_key / include lines omitted here
}

server {
    listen 443 ssl;
    server_name slpsoftware.nl;
    location / {
        proxy_pass http://<pi-main-lan-ip>:5101;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
    # certbot-managed ssl_certificate / ssl_certificate_key / include lines omitted here
}

1.8 Database Backup Credentials (§ 4 depends on this)

touch ~/.config/slpsoftware-db-backup.env
chmod 600 ~/.config/slpsoftware-db-backup.env

DB_HOST=127.0.0.1
DB_PORT=3306
DB_NAME=SlpSoftwareProduction
DB_USER=<a-login-with-just-SELECT-LOCK-TABLES-permission>
DB_PASSWORD=<password>

Kept separate from shared/env (§ 1.5) deliberately — the backup script needs its own credential, ideally scoped to just read access (SELECT, LOCK TABLES — everything mariadb-dump needs) rather than the application's own data-access login.

1.9 Gitea Actions Variables and Secrets

Set once, in this repository's Gitea Actions settings. This is exactly where the real path lives — deploy-scp.yaml never hardcodes it, it only reads ${{ vars.DEPLOY_PATH_TEST }} / ${{ vars.DEPLOY_PATH_PRODUCTION }}, so changing the path later is a variable edit, not a workflow change:

Name	Kind	Value
`PI_MAIN_ADDRESS`	secret	the Pi's address
`PI_MAIN_PORT`	secret	SSH port
`PI_MAIN_USERNAME`	secret	`gitea-workflow` — the generic, host-wide deploy account (§ 1.2), not `webadmin`; reuse the reference project's if it already has one
`PI_MAIN_PASSWORD`	secret	`gitea-workflow`'s password
`DEPLOY_PATH_TEST`	variable	`/home/gitea-workflow/apps/slpsoftware/test`
`DEPLOY_PATH_PRODUCTION`	variable	`/home/gitea-workflow/apps/slpsoftware/production`
`SERVICE_NAME_TEST`	variable	`slpsoftware-test.service`
`SERVICE_NAME_PRODUCTION`	variable	`slpsoftware-production.service`
`HEALTH_CHECK_URL_TEST`	variable	`https://test.slpsoftware.nl/health`
`HEALTH_CHECK_URL_PRODUCTION`	variable	`https://slpsoftware.nl/health`
`VITE_SENTRY_DSN`	variable	Sentry DSN (shared, not sensitive — safe in the client bundle)
`VITE_UMAMI_SCRIPT_URL`	variable	Umami script host (shared)
`VITE_UMAMI_WEBSITE_ID_TEST` / `_PRODUCTION`	variable	per-environment Umami website ID
`SECURITY_ALLOWED_SCRIPT_ORIGINS_TEST` / `_PRODUCTION`	variable	must match § 1.5's `SecurityHeaders__AllowedScriptOrigins__0` for that environment

2. Deploy Sequence (What Actually Happens on a Run)

Already built (U5/U6) — this is the read-only walkthrough for whoever operates it:

Push to master, or a manual workflow_dispatch → the six gates run
publish-test runs (and publish-production, only if workflow_dispatch with the flag)
deploy-test (always, if gates pass) calls deploy-scp.yaml, which uploads into a new releases/{timestamp}/, links shared/wwwroot-web in, switches current, restarts slpsoftware-test.service, verifies https://test.slpsoftware.nl/health, then prunes old releases (only test — run_db_backup: false)
deploy-production (only with the flag) does the same, plus a database backup first (run_db_backup: true) — see § 4

3. First-Ever Deploy Notes

wwwroot/web/ will be empty until a website workspace deploys into it — / serves the built-in placeholder until then (this is expected, not a failure)
The very first run has no "previous release" to keep — pruning naturally has nothing to prune
Verify manually after the first run: curl https://test.slpsoftware.nl/health and curl https://slpsoftware.nl/health (after the first production run) both return 200

4. Database Backup Script

Create ~/scripts/backup-slpmodularcms-db.sh on the Pi (this script is host-side by design — never part of this repository, so no DB credential ever reaches Gitea):

#!/usr/bin/env bash
set -euo pipefail

ENVIRONMENT="${1:?Usage: backup-slpmodularcms-db.sh <environment>}"
CREDENTIALS_FILE="$HOME/.config/slpsoftware-db-backup.env"

if [[ ! -f "$CREDENTIALS_FILE" ]]; then
  echo "Missing $CREDENTIALS_FILE — see deployment-instructions.md § 1.8" >&2
  exit 1
fi
# shellcheck source=/dev/null
source "$CREDENTIALS_FILE"
: "${DB_HOST:?}" "${DB_PORT:?}" "${DB_NAME:?}" "${DB_USER:?}" "${DB_PASSWORD:?}"

BACKUP_DIR="$HOME/backups/slpsoftware/${ENVIRONMENT}"
mkdir -p "$BACKUP_DIR"
TIMESTAMP=$(date -u +%Y%m%d%H%M%S)
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}-${TIMESTAMP}.sql.gz"

# --single-transaction: consistent snapshot without locking the tables for the whole dump duration
# (InnoDB only — every table here is, since that's EF Core's MySQL-provider default).
mariadb-dump \
  -h "$DB_HOST" -P "$DB_PORT" -u "$DB_USER" -p"$DB_PASSWORD" \
  --single-transaction --routines --triggers \
  "$DB_NAME" | gzip > "$BACKUP_FILE"

echo "Backup written to $BACKUP_FILE"

# Retention: keep the 7 most recent backups for this environment
ls -1t "$BACKUP_DIR"/*.sql.gz 2>/dev/null | tail -n +8 | xargs -r rm -f

mariadb-dump is MariaDB's own name for the tool (present since MariaDB 10.4-ish); if the host only has the older mysqldump name, substitute it — same tool, same flags.

chmod +x ~/scripts/backup-slpmodularcms-db.sh

Verify once, manually, before relying on it in a real deploy:

~/scripts/backup-slpmodularcms-db.sh production

Confirm a .sql.gz file appears under ~/backups/slpsoftware/production/ and that the dump didn't silently fail (the script uses set -euo pipefail, so a real error does propagate as a non-zero exit — which fails the calling deploy-scp.yaml step, correctly blocking the deploy). Worth a one-time restore rehearsal too — an untested backup is not a verified one:

gunzip -c ~/backups/slpsoftware/production/<file>.sql.gz | mariadb -h 127.0.0.1 -u root -p <a-scratch-database>

5. Future: Switching Production to FTPS (Shared Hosting)

D-02/NFR-09 required the workflow's transport to be swappable without restructuring — satisfied by the transport input already on deploy-scp.yaml (currently only scp is implemented). This section documents what actually changes when that day comes (OPEN-04 — not scheduled, drafted now per Q5 = B of the deployment setup plan).

5.1 What carries over unchanged

The CI workflow's gates, the two-build split, the config job pattern
The overall shape of the interface: artifact_name, environment, deploy_path

5.2 What does not carry over — read this before assuming it's a drop-in swap

Shared .NET hosting is almost always Windows/IIS-based, not Linux/systemd. That changes more than the transport:

No systemctl --user restart — IIS picks up a new deployment via an app-pool recycle, usually triggered by touching web.config or the app-pool's own recycle mechanism, not a service restart command
The atomic release-switch pattern may not be available at all — many shared hosts expose only a single web root over FTPS, with no ability to create sibling directories and swap a symlink. wwwroot/web/ persistence (FR-08, ASM-01) would need a different mechanism on such a host — e.g. never touching a specific subfolder during upload, rather than linking a persistent directory outside a swapped release tree, since "outside the release tree" may not be an available concept
Database backup — shared hosting frequently does not expose direct mariadb-dump/SSH access at all; the backup step in deploy-scp.yaml (§ 4 script) would need to become either a provider-specific API call (e.g. a hosting-panel database backup feature) or a documented manual pre-production step (the FR-20 fallback U6 already designed for)

5.3 What building `deploy-ftps.yaml` would actually require

A new reusable workflow implementing the same five required inputs (artifact_name, environment, deploy_path, service_name, health_check_url) so continuous_integration.yaml calls it identically via the transport input
An FTPS upload step (e.g. lftp mirror or curl --ftp-ssl) replacing the scp step
A new decision, not yet made: what "atomic switch" and "restart" even mean on the target host — this cannot be answered generically; it depends on which shared host is actually chosen
Re-running this feature's Infrastructure Design step for U6, scoped to the new host, once a specific shared-hosting provider is selected — not a checklist item that can be pre-answered today

Recommendation: treat this section as a starting brief for that future Infrastructure Design pass, not as a ready-to-execute procedure — the concrete host was unknown at the time this was written (OPEN-04), so several of the decisions above are necessarily provider-specific.

25 KiB Raw Blame History Unescape Escape