Adds Monitoring Setup docs and deploy-scp troubleshooting/debug fixes

Monitoring Setup: operations/plans/monitoring-setup-plan.md and
operations/monitoring/monitoring-instructions.md, covering Sentry alert
rules on the security_event tag, UptimeRobot's 6 liveness monitors, and
the two new Umami website entries for the admin SPA.

deployment-instructions.md gains a missing Observability__Environment
host var (without it, both environments would tag Sentry events as
"Production"), the nginx client_max_body_size fix for the 413 seen on
publish-test/production artifact uploads, and two troubleshooting notes
on Gitea Actions re-run behaviour: re-running deploy-test/production
alone loses the run's uploaded artifact, and re-running all jobs on an
existing (rather than a brand new) run can replay stale secrets.

deploy-scp.yaml: step names no longer show literal unresolved
${{ inputs.* }} text (Gitea doesn't interpolate that context in step
names), and a temporary debug step logs PI_MAIN_USERNAME/PASSWORD
length plus a username equality check to diagnose a persistent
Permission denied during the SSH steps, without ever logging the
secret values themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FffvxxJp5wG34Ru48GBig

aidlc-docs/features/gitea-deployment-workflow/operations/deployment/deployment-instructions.md

@@ -182,6 +182,14 @@ MasterModule__MasterUrl=https://<this-environment-domain>
 # continues (see Observability section in README.md).
 Observability__SentryDsn=<sentry-dsn-or-empty>
 
+# REQUIRED if the DSN above is set, and must differ between this file and the other environment's.
+# D-19 uses one Sentry project for both test and production, distinguished only by this tag.
+# ASPNETCORE_ENVIRONMENT is deliberately "Production" for both environments (see note below), so
+# without this explicit override, ObservabilityOptions.Environment falls back to
+# ASPNETCORE_ENVIRONMENT and every event — test and production alike — would be tagged "Production",
+# silently defeating D-19's whole point of telling them apart in Sentry.
+Observability__Environment=<test-or-production>
+
 # Optional — both only needed if this instance uses Umami analytics (VITE_UMAMI_SCRIPT_URL set for
 # the frontend build). Leave both lines out entirely if you don't use Umami; there is no other
 # origin either one needs by default.
@@ -362,6 +370,82 @@ server {
 }
 ```
 
+#### 1.7.3 Artifact Upload Body Size (413 on `publish-test`/`publish-production`)
+
+The Gitea instance itself (`gitea.slpsoftware.nl`) sits behind the same proxy Pi as every other
+domain in § 1.7 — Actions' own web UI/API traffic is proxied through it exactly like
+`test.slpsoftware.nl` and `slpsoftware.nl` are. Gitea Actions' artifact upload
+(`actions/upload-artifact`) sends the build output in chunks; if a chunk exceeds nginx's
+`client_max_body_size` (default 1m), nginx itself rejects it with `413 Request Entity Too Large`
+before the request ever reaches Gitea — surfacing in the `publish-test`/`publish-production` job log
+as repeated `A 413 status code has been received, will attempt to retry the upload` followed by
+`Retry limit has been reached` on individual files once the action's retries are exhausted.
+
+**pi-main is not involved** — § 1.7 already established it runs no nginx and holds no certificates
+for any of these domains; this is purely a proxy Pi setting.
+
+**Fix — set once, globally, not per server block.** `client_max_body_size` is inherited
+(`http` → `server` → `location`); setting it in the top-level `http {}` block covers every current
+and future domain's server blocks — both the `listen 80` and `listen 443` blocks certbot manages —
+without needing to repeat it each time § 1.7.1's procedure is run for a new domain.
+
+🌐 **proxy Pi / root:**
+```nginx
+# /etc/nginx/nginx.conf, inside the http { } block:
+http {
+    client_max_body_size 512m;
+    ...
+}
+```
+```bash
+sudo nginx -t && sudo systemctl reload nginx
+```
+
+#### 1.7.4 Never Re-Run `deploy-test`/`deploy-production` Alone — Gitea Platform Limitation
+
+`deploy-test` and `deploy-production` in `continuous_integration.yaml` are not ordinary jobs — each
+one *calls* the reusable `deploy-scp.yaml` workflow (`uses: ./.gitea/workflows/deploy-scp.yaml`).
+Re-running **only** one of these two jobs after a failure (Gitea's per-job "re-run" action) is a
+known Gitea Actions limitation, not a bug in either workflow file here: Gitea does not cleanly resume
+a `workflow_call` job inside its original run — it re-executes the call in a way that loses access to
+that run's already-uploaded artifact. The `Download build artifact` step then fails immediately with:
+```
+List Artifacts - Error is not retryable
+Status Code: 404
+Error: List Artifacts failed: Artifact service responded with 404
+```
+even though `publish-test`/`publish-production` genuinely succeeded and uploaded the artifact
+moments earlier in the same run (retention had not expired — this is not the same failure mode as
+§ 1.7.3, and not a retention issue at all).
+
+**Fix: re-run the entire workflow, not just this job** — but see § 1.7.5 immediately below before
+doing that: Gitea's "re-run all jobs" on an *existing* run is not the same as triggering a genuinely
+new run, and has its own, different failure mode.
+
+#### 1.7.5 "Re-Run All Jobs" Can Silently Reuse Stale Secrets/Variables — Trigger a New Run Instead
+
+Symptom: a secret (e.g. `PI_MAIN_PASSWORD`) was wrong, causing `sshpass`/`ssh` to fail with
+`Permission denied, please try again.` (`sshpass` exit code 5 — the password itself was rejected,
+not a connectivity or config problem). The secret is corrected in Gitea's UI and confirmed working
+via a manual SSH test with the same value. **"Re-run all jobs" is used on the existing, already-failed
+run — and it fails again, identically**, as if the fix never happened.
+
+This matches a documented behaviour of Actions-style re-run implementations, seen concretely in
+GitHub's own tooling ([cli/cli#13522](https://github.com/cli/cli/issues/13522)): re-running an
+*existing* run can replay against secrets/variables as they were **when that run was first created**,
+not their current values — particularly for a job that calls a reusable workflow with
+`secrets: inherit` (exactly what `deploy-test`/`deploy-production` do here). The documented contract
+("secrets are fetched at the time of the re-run") does not hold in practice for this case. A genuinely
+**new** run (fresh push, or `workflow_dispatch`) always fetches current values correctly — only
+re-running an existing run risks the stale snapshot.
+
+**Net effect of § 1.7.4 + § 1.7.5 together**: neither of Gitea's two "re-run" options is fully safe
+for `deploy-test`/`deploy-production` — "re-run this job alone" loses the artifact, "re-run all jobs"
+can keep stale secrets. **The one reliable option is to trigger a brand new run** (an actual push, or
+`workflow_dispatch`'s "Run workflow" button) rather than using either re-run action on a failed run,
+whenever a secret or variable was just changed to fix that failure. Re-running an existing run is
+only safe when nothing about its secrets/variables changed since it was created.
+
 ### 1.8 Database Backup Credentials (§ 4 depends on this)
 
 👤 **pi-main / `gitea-workflow`:**

aidlc-docs/features/gitea-deployment-workflow/operations/monitoring/monitoring-instructions.md

@@ -0,0 +1,134 @@
+# Monitoring Instructions
+
+All three services below were built into the application during Construction (U4) — this document
+only wires them to the real accounts and writes down the configuration that necessarily lives
+outside this repository (Sentry alert rules, UptimeRobot monitors, Umami website entries).
+
+**Who does what, where** — reuses the labels from `deployment-instructions.md` § "Who runs what,
+where", plus the external services themselves:
+- 👤 **pi-main / `gitea-workflow`** — the environment's `shared/env` file (§ 1.5 of
+  `deployment-instructions.md`)
+- ☁️ **Gitea web UI** — repository Variables (Settings → Actions → Variables)
+- ☁️ **Sentry web UI** — project settings and alert rules
+- ☁️ **UptimeRobot web UI** — monitors and alert contacts
+- ☁️ **Umami web UI** — website entries
+
+## 1. Sentry
+
+One project for the whole CMS (D-19); `Observability__Environment` (host-side, § 1.5) and
+`VITE_APP_ENV` (build-time) are what tell test and production events apart in that one project —
+there is no separate DSN per environment.
+
+### 1.1 Gitea variable
+
+☁️ **Gitea web UI** — Settings → Actions → Variables:
+
+| Variable | Value |
+|---|---|
+| `VITE_SENTRY_DSN` | the project's DSN (same value used by both `publish-test` and `publish-production`) |
+
+### 1.2 Host configuration
+
+Already documented in `deployment-instructions.md` § 1.5 — `Observability__SentryDsn` and
+`Observability__Environment` in each environment's `shared/env` file. `Observability__Environment`
+**must** differ between the two files (`test` / `production`); see that section's note on why
+`ASPNETCORE_ENVIRONMENT` can't be reused for this.
+
+### 1.3 Alert rules (FR-19)
+
+The six `security_event` tag values already emitted by `SecurityEvents.cs` — alert rules must filter
+on the **tag**, not the message text (`nfr-design-patterns.md` Pattern 6):
+
+| `security_event` tag value | Rule type | Threshold |
+|---|---|---|
+| `failed_login` | Count-based | > 20 events in 5 minutes |
+| `authorization_denied` | Count-based | > 20 events in 5 minutes |
+| `master_api_key_rejected` | Count-based | > 20 events in 5 minutes |
+| `admin_bypass_rejected` | Count-based | > 20 events in 5 minutes |
+| `rate_limit_triggered` | Count-based | > 20 events in 5 minutes |
+| `migration_failure` | Always-fire | any occurrence (already `Critical` level — the process is exiting) |
+
+20-in-5-minutes is a starting default (no production traffic history exists yet to tune against) —
+revisit once real traffic volume is known.
+
+☁️ **Sentry web UI** — Alerts → Create Alert Rule, once per row above:
+1. Condition: `tags.security_event equals <value>`
+2. For the five count-based rules: "when greater than 20 events, in 5 minutes" (or the platform's
+   equivalent rate-window phrasing)
+3. For `migration_failure`: no count condition — fire on every matching event
+4. Action: notify via whichever channel/integration alerts should reach (same destination used
+   elsewhere for this project is fine — nothing here requires a dedicated channel)
+
+**Note (DEV-01, accepted deviation)**: Sentry's plan in use retains events for roughly 30 days,
+short of the 90-day SECURITY-14 minimum. Already accepted as a cost decision at Requirements
+Analysis — nothing to configure here, just to be aware of if investigating an old event.
+
+## 2. UptimeRobot
+
+Six monitors — `/health`, `/`, `/admin` × test/production (D-23). What each does and does not prove
+(FR-17):
+
+| URL | Proves | Does NOT prove |
+|---|---|---|
+| `https://test.slpsoftware.nl/health` | The test process is alive (liveness only, D-21) | Database connectivity, migrations applied — no DB check by design (Q17=A) |
+| `https://test.slpsoftware.nl/` | Proxy Pi → pi-main routing and static hosting work for test | The API or admin SPA work at all |
+| `https://test.slpsoftware.nl/admin` | The admin SPA is served and its assets aren't blocked by CSP | Login/auth actually succeeds — this only checks the shell loads |
+| `https://slpsoftware.nl/health` | Same as above, production | Same as above, production |
+| `https://slpsoftware.nl/` | Same as above, production | Same as above, production |
+| `https://slpsoftware.nl/admin` | Same as above, production | Same as above, production |
+
+☁️ **UptimeRobot web UI** — Add New Monitor, once per row above:
+- Monitor Type: **HTTP(s)** (not Keyword/Port/Ping) — checks only the response status code (2xx/3xx),
+  the same liveness signal `deploy-scp.yaml`'s own `curl -f` health check already relies on
+- Interval: 5 minutes is enough for all six — `/health` recovers within a process restart
+  (`Restart=on-failure`, `RestartSec=5` in the systemd units), no need for a tighter interval
+- Alert Contact: the existing contact already configured on this account
+
+**Free-plan note**: 6 monitors at 5-minute intervals fits comfortably within the free plan's 50-monitor
+cap and 5-minute minimum interval — no paid tier needed for this feature's monitoring. One gap the
+free plan leaves open: no SSL-certificate-expiry monitoring. Certbot's systemd timer renews
+automatically (`deployment-instructions.md` § 1.7.1), so the risk is low, but a silently failed
+renewal would only be caught reactively — once an HTTP(s) check itself starts failing — rather than
+warned about in advance. Accepted as a known gap, not built around, consistent with this project's
+other documented deviations (e.g. DEV-01's log-retention gap).
+
+## 3. Umami
+
+Reuses the existing self-hosted instance at `analytics.slpsoftware.nl` (D-24/ASM-05). This section
+covers only the **admin SPA's** two website entries — the public website's Umami wiring (also
+required by D-25) belongs to whichever website workspace serves `wwwroot/web/` (FR-09/FR-16), not to
+this deploy pipeline.
+
+☁️ **Umami web UI** — Websites → Add website, twice (once per environment), domain set to the
+admin SPA's real URL (`test.slpsoftware.nl/admin`, `slpsoftware.nl/admin`) — note the resulting
+website ID for each.
+
+☁️ **Gitea web UI** — Settings → Actions → Variables:
+
+| Variable | Value |
+|---|---|
+| `VITE_UMAMI_SCRIPT_URL` | `https://analytics.slpsoftware.nl/script.js` (shared — same instance, both environments) |
+| `VITE_UMAMI_WEBSITE_ID_TEST` | the test website's ID from the step above |
+| `VITE_UMAMI_WEBSITE_ID_PRODUCTION` | the production website's ID from the step above |
+
+**These must stay in sync with two other Gitea variables that already exist from U5**
+(`SECURITY_ALLOWED_SCRIPT_ORIGINS_TEST` / `_PRODUCTION`) and with the matching
+`SecurityHeaders__AllowedScriptOrigins__0` / `_AllowedConnectOrigins__0` lines in each environment's
+host `shared/env` file (`deployment-instructions.md` § 1.5) — all four must name the same
+`https://analytics.slpsoftware.nl` origin, or either the CI gate (REF-U5-01) fails the build, or the
+script loads in CI but is silently blocked by CSP on the real host (the gate cannot see the host
+file, only the Gitea variable side of this).
+
+## 4. Gitea Variables — Full Summary
+
+For reference, every Actions variable this document and `deployment-instructions.md` § 1.9 together
+require:
+
+| Variable | Scope | Set by |
+|---|---|---|
+| `VITE_SENTRY_DSN` | shared | § 1.1 above |
+| `VITE_UMAMI_SCRIPT_URL` | shared | § 3 above |
+| `VITE_UMAMI_WEBSITE_ID_TEST` | test | § 3 above |
+| `VITE_UMAMI_WEBSITE_ID_PRODUCTION` | production | § 3 above |
+| `SECURITY_ALLOWED_SCRIPT_ORIGINS_TEST` / `_PRODUCTION` | per environment | already set at Deployment Setup — must match § 3's Umami origin |
+| `DEPLOY_PATH_*`, `SERVICE_NAME_*`, `HEALTH_CHECK_URL_*` | per environment | `deployment-instructions.md` § 1.9 (unrelated to monitoring, listed there) |

aidlc-docs/features/gitea-deployment-workflow/operations/plans/monitoring-setup-plan.md

@@ -0,0 +1,127 @@
+# Monitoring Setup Plan
+
+## Context already established (not re-asked)
+
+The observability *stack* and *what it must cover* were already decided in Requirements Analysis and
+built during Construction (U4) — this stage is about configuring the real external services and
+writing the operational documentation, not re-deciding the approach:
+
+- **Stack**: Sentry (backend `Sentry.AspNetCore` + frontend `@sentry/react`), self-hosted Umami
+  analytics, UptimeRobot uptime monitoring (D-18, D-24, requirements.md § "Monitoring and observability")
+- **Sentry projects**: **one** project for the whole CMS, `environment` tag distinguishes
+  test/production (D-19) — so one DSN, reused by both environments and both frontend/backend
+- **Sentry is optional by design**: an absent DSN disables it cleanly, console logging stays active
+  (BR-U4-08) — nothing breaks if a value is left blank temporarily
+- **Alertable events already exist in code**: six `security_event` tag values are emitted today —
+  `failed_login`, `authorization_denied`, `master_api_key_rejected`, `admin_bypass_rejected`,
+  `rate_limit_triggered`, `migration_failure` (`SecurityEvents.cs`). Alert rules must filter on the
+  `security_event` tag, not message text (`nfr-design-patterns.md` Pattern 6) — this is explicitly
+  named as **Operations — Monitoring Setup** work in `logical-components.md` (FR-19)
+- **UptimeRobot scope**: `/health`, `/` (public website) and `/admin`, per environment — 6 monitors
+  total (D-23)
+- **Umami scope**: reuse the existing self-hosted instance at `analytics.slpsoftware.nl`, add new
+  website entries for this CMS rather than standing up a new instance (D-24/ASM-05). Both the admin
+  SPA (via `VITE_UMAMI_WEBSITE_ID_<ENV>`, built by this pipeline) and the public website (via the
+  separate website-workspace, FR-09/FR-16, out of this pipeline's scope) are measured (D-25) — this
+  stage only configures the **admin SPA's** two website entries; the public website's Umami wiring
+  belongs to whichever website workspace serves it
+- **Health endpoint**: liveness only, no DB check, already built and already on the
+  `AvailabilityMiddleware` bypass list (D-21) — nothing left to build, only to point UptimeRobot at
+- **Log retention deviation**: accepted knowingly (DEV-01) — Sentry's ~30-day retention against the
+  90-day SECURITY-14 minimum. No action here beyond documenting it.
+
+## Question 1: Include Monitoring Setup?
+
+Given U4 built all the alertable-event plumbing specifically so that Operations could wire up real
+alert rules and monitors on top of it, declining this stage would leave that code inert — events
+would be emitted into a Sentry project that doesn't exist yet, with no alert rule reading the tag,
+and no UptimeRobot monitor watching `/health`. Recommended: A.
+
+A) Yes — configure the real services and document it (Recommended)
+B) No — monitoring is handled elsewhere or not needed
+C) Not sure — suggest an approach and I'll decide
+
+X) Other (please describe after [Answer]: tag below)
+
+[Answer]:A
+
+## Question 2: Monitoring Method
+
+Confirms the stack already decided in Requirements Analysis — asked per the mandatory format, not
+because it's genuinely open.
+
+A) Sentry + self-hosted Umami + UptimeRobot, exactly as scoped above (Recommended)
+B) Something else entirely — describe below
+
+X) Other (please describe after [Answer]: tag below)
+
+[Answer]:A
+
+---
+
+The remaining questions are the genuinely open items — real account/service facts that only you
+know, which nothing in Construction could decide for you.
+
+## Question 3: Sentry Project
+
+Does a Sentry project for this CMS already exist (e.g. created ad hoc while testing U4 locally), or
+does this stage need to walk through creating one from scratch?
+
+A) Already exists — I'll provide the DSN after [Answer]: below
+B) Doesn't exist yet — walk me through creating one (org, project platform picks for .NET + React,
+   where to find the DSN afterward)
+C) Not yet, and I'd rather set it up myself later — document it as a TODO with exactly which Gitea
+   secrets/variables need the DSN once it exists
+
+X) Other (please describe after [Answer]: tag below)
+
+[Answer]:A — DSN to be added directly to the Gitea secret/variable once shared (kept out of this doc)
+
+## Question 4: Sentry Alert Rule Thresholds
+
+FR-19 requires alert rules for repeated auth failures and authorization violations, filtering on the
+`security_event` tag (values: `failed_login`, `authorization_denied`, `master_api_key_rejected`,
+`admin_bypass_rejected`, `rate_limit_triggered`; `migration_failure` is `Critical`-level and worth
+its own always-fire rule regardless of count). What threshold should the repeated-failure rules use?
+
+A) A single reasonable default for all of them (e.g. 20 occurrences in 5 minutes) — document it as a
+   starting point, tune later from real traffic (Recommended)
+B) I'll specify exact thresholds per event type — describe below
+C) No count threshold — alert on every occurrence of each of the six (noisier, but nothing is ever
+   missed)
+
+X) Other (please describe after [Answer]: tag below)
+
+[Answer]:A
+
+## Question 5: UptimeRobot Account and Alert Contacts
+
+6 monitors are needed (`/health`, `/`, `/admin` × test/production). Do you have an UptimeRobot
+account already (e.g. from the reference `SlpSoftware` project), and where should alerts go?
+
+A) Existing account, reuse it — alert contact(s): describe after [Answer]: below (email, and/or
+   any other channel already configured there)
+B) Need a new account/monitor group set up from scratch — walk me through it
+C) Not sure yet — document the 6 monitors' exact URLs and recommended check interval as a checklist,
+   I'll create them myself
+
+X) Other (please describe after [Answer]: tag below)
+
+[Answer]:A — existing account, alert contact to be confirmed
+
+## Question 6: Umami Website Entries (Admin SPA)
+
+Two new website entries are needed on the existing `analytics.slpsoftware.nl` instance — one per
+environment — to obtain the `VITE_UMAMI_WEBSITE_ID_TEST`/`_PRODUCTION` values the CI workflow already
+expects as Gitea variables (`continuous_integration.yaml` reads `vars.VITE_UMAMI_WEBSITE_ID_TEST`/
+`_PRODUCTION`).
+
+A) I have access to the Umami instance — I'll create the two entries and provide the website IDs
+   after [Answer]: below
+B) Walk me through creating them (Umami's own UI/API steps)
+C) Not yet — document it as a TODO with the exact variable names the pipeline expects, I'll fill
+   them in before the first real deploy
+
+X) Other (please describe after [Answer]: tag below)
+
+[Answer]:A — website IDs to be added directly to Gitea variables once created (kept out of this doc)