Adds Monitoring Setup docs and deploy-scp troubleshooting/debug fixes

Monitoring Setup: operations/plans/monitoring-setup-plan.md and operations/monitoring/monitoring-instructions.md, covering Sentry alert rules on the security_event tag, UptimeRobot's 6 liveness monitors, and the two new Umami website entries for the admin SPA. deployment-instructions.md gains a missing Observability__Environment host var (without it, both environments would tag Sentry events as "Production"), the nginx client_max_body_size fix for the 413 seen on publish-test/production artifact uploads, and two troubleshooting notes on Gitea Actions re-run behaviour: re-running deploy-test/production alone loses the run's uploaded artifact, and re-running all jobs on an existing (rather than a brand new) run can replay stale secrets. deploy-scp.yaml: step names no longer show literal unresolved ${{ inputs.* }} text (Gitea doesn't interpolate that context in step names), and a temporary debug step logs PI_MAIN_USERNAME/PASSWORD length plus a username equality check to diagnose a persistent Permission denied during the SSH steps, without ever logging the secret values themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015FffvxxJp5wG34Ru48GBig
2026-07-29 19:39:44 +02:00
parent d81168b7a9
commit 56f4f6fe0f
5 changed files with 382 additions and 7 deletions
@@ -182,6 +182,14 @@ MasterModule__MasterUrl=https://<this-environment-domain>
 # continues (see Observability section in README.md).
 Observability__SentryDsn=<sentry-dsn-or-empty>

+# REQUIRED if the DSN above is set, and must differ between this file and the other environment's.
+# D-19 uses one Sentry project for both test and production, distinguished only by this tag.
+# ASPNETCORE_ENVIRONMENT is deliberately "Production" for both environments (see note below), so
+# without this explicit override, ObservabilityOptions.Environment falls back to
+# ASPNETCORE_ENVIRONMENT and every event — test and production alike — would be tagged "Production",
+# silently defeating D-19's whole point of telling them apart in Sentry.
+Observability__Environment=<test-or-production>
+
 # Optional — both only needed if this instance uses Umami analytics (VITE_UMAMI_SCRIPT_URL set for
 # the frontend build). Leave both lines out entirely if you don't use Umami; there is no other
 # origin either one needs by default.
@@ -362,6 +370,82 @@ server {
 }
 ```

+#### 1.7.3 Artifact Upload Body Size (413 on `publish-test`/`publish-production`)
+
+The Gitea instance itself (`gitea.slpsoftware.nl`) sits behind the same proxy Pi as every other
+domain in § 1.7 — Actions' own web UI/API traffic is proxied through it exactly like
+`test.slpsoftware.nl` and `slpsoftware.nl` are. Gitea Actions' artifact upload
+(`actions/upload-artifact`) sends the build output in chunks; if a chunk exceeds nginx's
+`client_max_body_size` (default 1m), nginx itself rejects it with `413 Request Entity Too Large`
+before the request ever reaches Gitea — surfacing in the `publish-test`/`publish-production` job log
+as repeated `A 413 status code has been received, will attempt to retry the upload` followed by
+`Retry limit has been reached` on individual files once the action's retries are exhausted.
+
+**pi-main is not involved** — § 1.7 already established it runs no nginx and holds no certificates
+for any of these domains; this is purely a proxy Pi setting.
+
+**Fix — set once, globally, not per server block.** `client_max_body_size` is inherited
+(`http` → `server` → `location`); setting it in the top-level `http {}` block covers every current
+and future domain's server blocks — both the `listen 80` and `listen 443` blocks certbot manages —
+without needing to repeat it each time § 1.7.1's procedure is run for a new domain.
+
+🌐 **proxy Pi / root:**
+```nginx
+# /etc/nginx/nginx.conf, inside the http { } block:
+http {
+    client_max_body_size 512m;
+    ...
+}
+```
+```bash
+sudo nginx -t && sudo systemctl reload nginx
+```
+
+#### 1.7.4 Never Re-Run `deploy-test`/`deploy-production` Alone — Gitea Platform Limitation
+
+`deploy-test` and `deploy-production` in `continuous_integration.yaml` are not ordinary jobs — each
+one *calls* the reusable `deploy-scp.yaml` workflow (`uses: ./.gitea/workflows/deploy-scp.yaml`).
+Re-running **only** one of these two jobs after a failure (Gitea's per-job "re-run" action) is a
+known Gitea Actions limitation, not a bug in either workflow file here: Gitea does not cleanly resume
+a `workflow_call` job inside its original run — it re-executes the call in a way that loses access to
+that run's already-uploaded artifact. The `Download build artifact` step then fails immediately with:
+```
+List Artifacts - Error is not retryable
+Status Code: 404
+Error: List Artifacts failed: Artifact service responded with 404
+```
+even though `publish-test`/`publish-production` genuinely succeeded and uploaded the artifact
+moments earlier in the same run (retention had not expired — this is not the same failure mode as
+§ 1.7.3, and not a retention issue at all).
+
+**Fix: re-run the entire workflow, not just this job** — but see § 1.7.5 immediately below before
+doing that: Gitea's "re-run all jobs" on an *existing* run is not the same as triggering a genuinely
+new run, and has its own, different failure mode.
+
+#### 1.7.5 "Re-Run All Jobs" Can Silently Reuse Stale Secrets/Variables — Trigger a New Run Instead
+
+Symptom: a secret (e.g. `PI_MAIN_PASSWORD`) was wrong, causing `sshpass`/`ssh` to fail with
+`Permission denied, please try again.` (`sshpass` exit code 5 — the password itself was rejected,
+not a connectivity or config problem). The secret is corrected in Gitea's UI and confirmed working
+via a manual SSH test with the same value. **"Re-run all jobs" is used on the existing, already-failed
+run — and it fails again, identically**, as if the fix never happened.
+
+This matches a documented behaviour of Actions-style re-run implementations, seen concretely in
+GitHub's own tooling ([cli/cli#13522](https://github.com/cli/cli/issues/13522)): re-running an
+*existing* run can replay against secrets/variables as they were **when that run was first created**,
+not their current values — particularly for a job that calls a reusable workflow with
+`secrets: inherit` (exactly what `deploy-test`/`deploy-production` do here). The documented contract
+("secrets are fetched at the time of the re-run") does not hold in practice for this case. A genuinely
+**new** run (fresh push, or `workflow_dispatch`) always fetches current values correctly — only
+re-running an existing run risks the stale snapshot.
+
+**Net effect of § 1.7.4 + § 1.7.5 together**: neither of Gitea's two "re-run" options is fully safe
+for `deploy-test`/`deploy-production` — "re-run this job alone" loses the artifact, "re-run all jobs"
+can keep stale secrets. **The one reliable option is to trigger a brand new run** (an actual push, or
+`workflow_dispatch`'s "Run workflow" button) rather than using either re-run action on a failed run,
+whenever a secret or variable was just changed to fix that failure. Re-running an existing run is
+only safe when nothing about its secrets/variables changed since it was created.
+
 ### 1.8 Database Backup Credentials (§ 4 depends on this)

 👤 **pi-main / `gitea-workflow`:**