Adds Monitoring Setup docs and deploy-scp troubleshooting/debug fixes #2
@@ -42,6 +42,19 @@ jobs:
|
|||||||
deploy:
|
deploy:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
|
# TEMPORARY — diagnosing a persistent "Permission denied, please try again." (sshpass exit 5)
|
||||||
|
# on the SSH steps below. Never prints the actual secret values, only lengths and a boolean
|
||||||
|
# match against the expected username, so it's safe to leave in a log — but remove this step
|
||||||
|
# once the PI_MAIN_* secrets are confirmed correct; it has no purpose beyond that diagnosis.
|
||||||
|
- name: Debug - verify PI_MAIN_USERNAME/PASSWORD secrets (remove after diagnosis)
|
||||||
|
run: |
|
||||||
|
echo "PI_MAIN_USERNAME length: ${#PI_MAIN_USERNAME}"
|
||||||
|
echo "PI_MAIN_USERNAME equals 'gitea-workflow': $([ "$PI_MAIN_USERNAME" = "gitea-workflow" ] && echo yes || echo no)"
|
||||||
|
echo "PI_MAIN_PASSWORD length: ${#PI_MAIN_PASSWORD}"
|
||||||
|
env:
|
||||||
|
PI_MAIN_USERNAME: ${{ secrets.PI_MAIN_USERNAME }}
|
||||||
|
PI_MAIN_PASSWORD: ${{ secrets.PI_MAIN_PASSWORD }}
|
||||||
|
|
||||||
- name: Compute release timestamp
|
- name: Compute release timestamp
|
||||||
id: release
|
id: release
|
||||||
run: echo "timestamp=$(date -u +%Y%m%d%H%M%S)" >> "$GITHUB_OUTPUT"
|
run: echo "timestamp=$(date -u +%Y%m%d%H%M%S)" >> "$GITHUB_OUTPUT"
|
||||||
@@ -56,9 +69,13 @@ jobs:
|
|||||||
# The backup script itself lives on the host (created once during Operations host setup) and
|
# The backup script itself lives on the host (created once during Operations host setup) and
|
||||||
# is only invoked here — no database connection string or credential is ever known to this
|
# is only invoked here — no database connection string or credential is ever known to this
|
||||||
# workflow, keeping D-16 ("runtime secrets live in host environment variables") intact.
|
# workflow, keeping D-16 ("runtime secrets live in host environment variables") intact.
|
||||||
- name: Back up database (${{ inputs.environment }})
|
# Gitea Actions does not resolve `${{ inputs.* }}` inside a step's `name:` (unlike inside
|
||||||
|
# `run:`, where it works fine — see the echo below) — kept static rather than showing the
|
||||||
|
# literal, unresolved `${{ inputs.environment }}` text in the log.
|
||||||
|
- name: Back up database
|
||||||
if: ${{ inputs.run_db_backup }}
|
if: ${{ inputs.run_db_backup }}
|
||||||
run: |
|
run: |
|
||||||
|
echo "Environment: ${{ inputs.environment }}"
|
||||||
sudo apt-get update && sudo apt-get install -y sshpass
|
sudo apt-get update && sudo apt-get install -y sshpass
|
||||||
sshpass -p "${{ secrets.PI_MAIN_PASSWORD }}" ssh \
|
sshpass -p "${{ secrets.PI_MAIN_PASSWORD }}" ssh \
|
||||||
-p ${{ secrets.PI_MAIN_PORT }} \
|
-p ${{ secrets.PI_MAIN_PORT }} \
|
||||||
@@ -74,8 +91,10 @@ jobs:
|
|||||||
# runner with "failed to attach to container: unable to upgrade to tcp, received 409", a known
|
# runner with "failed to attach to container: unable to upgrade to tcp, received 409", a known
|
||||||
# limitation of Podman's Docker-compatible API for the attach/log-streaming that container
|
# limitation of Podman's Docker-compatible API for the attach/log-streaming that container
|
||||||
# actions rely on (D-05). A plain scp command needs no nested container.
|
# actions rely on (D-05). A plain scp command needs no nested container.
|
||||||
- name: Upload release to ${{ inputs.environment }} (${{ inputs.transport }})
|
# Same Gitea Actions limitation as the step above — static name, value logged via echo instead.
|
||||||
|
- name: Upload release
|
||||||
run: |
|
run: |
|
||||||
|
echo "Environment: ${{ inputs.environment }}, transport: ${{ inputs.transport }}"
|
||||||
sudo apt-get update && sudo apt-get install -y sshpass
|
sudo apt-get update && sudo apt-get install -y sshpass
|
||||||
RELEASE_DIR="${{ inputs.deploy_path }}/releases/${{ steps.release.outputs.timestamp }}"
|
RELEASE_DIR="${{ inputs.deploy_path }}/releases/${{ steps.release.outputs.timestamp }}"
|
||||||
sshpass -p "${{ secrets.PI_MAIN_PASSWORD }}" ssh \
|
sshpass -p "${{ secrets.PI_MAIN_PASSWORD }}" ssh \
|
||||||
|
|||||||
@@ -179,8 +179,15 @@ Execution rounds (Q4 = B): **R1** = U1 + U2 · **R2** = U3 + U4 · **R3** = U5 +
|
|||||||
- [x] Build and Test — **EXECUTE** — Complete 2026-07-28. 609 unit tests (372 backend + 237 frontend), 0 vulnerable packages, 0 build errors. Plus live-host integration verification: static content/SPA-fallback/header-scoping, real master↔slave communication proving the persistent key ring survives a process restart, and W3C trace-id correlation in real log output. Artifacts in `construction/build-and-test/`
|
- [x] Build and Test — **EXECUTE** — Complete 2026-07-28. 609 unit tests (372 backend + 237 frontend), 0 vulnerable packages, 0 build errors. Plus live-host integration verification: static content/SPA-fallback/header-scoping, real master↔slave communication proving the persistent key ring survives a process restart, and W3C trace-id correlation in real log output. Artifacts in `construction/build-and-test/`
|
||||||
|
|
||||||
### OPERATIONS
|
### OPERATIONS
|
||||||
- [ ] Deployment Setup — **EXECUTE**
|
- [x] Deployment Setup — **EXECUTE** — Complete 2026-07-28, extended 2026-07-29 (MariaDB migration,
|
||||||
- [ ] Monitoring Setup — **EXECUTE**
|
reverse proxy topology correction, artifact-upload 413 fix in `deployment-instructions.md` § 1.7.3,
|
||||||
|
missing `Observability__Environment` host var added § 1.5). Artifacts: `operations/deployment/`
|
||||||
|
- [x] Monitoring Setup — **EXECUTE** — Complete 2026-07-29. Plan answered (Sentry project reused,
|
||||||
|
UptimeRobot account reused, Umami website entries created by user, 20-events/5-min default alert
|
||||||
|
threshold). User has applied the real values in Gitea variables, UptimeRobot and Sentry directly
|
||||||
|
(DSN, Umami website IDs, alert rules, 6 monitors) — not verified by execution from this session
|
||||||
|
(no access to those external accounts), taken on the user's report. Artifacts:
|
||||||
|
`operations/plans/monitoring-setup-plan.md`, `operations/monitoring/monitoring-instructions.md`
|
||||||
- [ ] Production Readiness Validation — **EXECUTE** (includes the `dotnet-appsettings` compliance gate)
|
- [ ] Production Readiness Validation — **EXECUTE** (includes the `dotnet-appsettings` compliance gate)
|
||||||
|
|
||||||
## Execution Plan Summary
|
## Execution Plan Summary
|
||||||
@@ -190,9 +197,13 @@ Execution rounds (Q4 = B): **R1** = U1 + U2 · **R2** = U3 + U4 · **R3** = U5 +
|
|||||||
|
|
||||||
## Current Status
|
## Current Status
|
||||||
- **Lifecycle Phase**: OPERATIONS
|
- **Lifecycle Phase**: OPERATIONS
|
||||||
- **Current Stage**: Build and Test **complete** 2026-07-28 (all 7 units, full-suite + live-host integration verification). CONSTRUCTION phase is now closed
|
- **Current Stage**: Deployment Setup complete (2026-07-28, extended 2026-07-29). Monitoring Setup
|
||||||
- **Next Stage**: Deployment Setup (Operations Configuration = Yes, decided at Requirements Analysis)
|
complete 2026-07-29 — user applied real values in Gitea, UptimeRobot and Sentry
|
||||||
- **Status**: Rounds 1–4 all done, nothing pushed. Entering Operations phase
|
- **Next Stage**: Production Readiness Validation (includes the `dotnet-appsettings` compliance gate)
|
||||||
|
- **Status**: CI pipeline built and merging via an open PR; `workflow_dispatch` manual testing of
|
||||||
|
`deploy-test`/`deploy-production` (and therefore a real end-to-end run touching the monitors/alerts
|
||||||
|
just configured) blocked until that PR merges to `master` (Gitea Actions only shows manual-dispatch
|
||||||
|
workflows that exist on the default branch)
|
||||||
|
|
||||||
## Round 2 Design Record (2026-07-28)
|
## Round 2 Design Record (2026-07-28)
|
||||||
- Functional Design U3 + U4 complete and committed (`357d395`)
|
- Functional Design U3 + U4 complete and committed (`357d395`)
|
||||||
|
|||||||
+84
@@ -182,6 +182,14 @@ MasterModule__MasterUrl=https://<this-environment-domain>
|
|||||||
# continues (see Observability section in README.md).
|
# continues (see Observability section in README.md).
|
||||||
Observability__SentryDsn=<sentry-dsn-or-empty>
|
Observability__SentryDsn=<sentry-dsn-or-empty>
|
||||||
|
|
||||||
|
# REQUIRED if the DSN above is set, and must differ between this file and the other environment's.
|
||||||
|
# D-19 uses one Sentry project for both test and production, distinguished only by this tag.
|
||||||
|
# ASPNETCORE_ENVIRONMENT is deliberately "Production" for both environments (see note below), so
|
||||||
|
# without this explicit override, ObservabilityOptions.Environment falls back to
|
||||||
|
# ASPNETCORE_ENVIRONMENT and every event — test and production alike — would be tagged "Production",
|
||||||
|
# silently defeating D-19's whole point of telling them apart in Sentry.
|
||||||
|
Observability__Environment=<test-or-production>
|
||||||
|
|
||||||
# Optional — both only needed if this instance uses Umami analytics (VITE_UMAMI_SCRIPT_URL set for
|
# Optional — both only needed if this instance uses Umami analytics (VITE_UMAMI_SCRIPT_URL set for
|
||||||
# the frontend build). Leave both lines out entirely if you don't use Umami; there is no other
|
# the frontend build). Leave both lines out entirely if you don't use Umami; there is no other
|
||||||
# origin either one needs by default.
|
# origin either one needs by default.
|
||||||
@@ -362,6 +370,82 @@ server {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### 1.7.3 Artifact Upload Body Size (413 on `publish-test`/`publish-production`)
|
||||||
|
|
||||||
|
The Gitea instance itself (`gitea.slpsoftware.nl`) sits behind the same proxy Pi as every other
|
||||||
|
domain in § 1.7 — Actions' own web UI/API traffic is proxied through it exactly like
|
||||||
|
`test.slpsoftware.nl` and `slpsoftware.nl` are. Gitea Actions' artifact upload
|
||||||
|
(`actions/upload-artifact`) sends the build output in chunks; if a chunk exceeds nginx's
|
||||||
|
`client_max_body_size` (default 1m), nginx itself rejects it with `413 Request Entity Too Large`
|
||||||
|
before the request ever reaches Gitea — surfacing in the `publish-test`/`publish-production` job log
|
||||||
|
as repeated `A 413 status code has been received, will attempt to retry the upload` followed by
|
||||||
|
`Retry limit has been reached` on individual files once the action's retries are exhausted.
|
||||||
|
|
||||||
|
**pi-main is not involved** — § 1.7 already established it runs no nginx and holds no certificates
|
||||||
|
for any of these domains; this is purely a proxy Pi setting.
|
||||||
|
|
||||||
|
**Fix — set once, globally, not per server block.** `client_max_body_size` is inherited
|
||||||
|
(`http` → `server` → `location`); setting it in the top-level `http {}` block covers every current
|
||||||
|
and future domain's server blocks — both the `listen 80` and `listen 443` blocks certbot manages —
|
||||||
|
without needing to repeat it each time § 1.7.1's procedure is run for a new domain.
|
||||||
|
|
||||||
|
🌐 **proxy Pi / root:**
|
||||||
|
```nginx
|
||||||
|
# /etc/nginx/nginx.conf, inside the http { } block:
|
||||||
|
http {
|
||||||
|
client_max_body_size 512m;
|
||||||
|
...
|
||||||
|
}
|
||||||
|
```
|
||||||
|
```bash
|
||||||
|
sudo nginx -t && sudo systemctl reload nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 1.7.4 Never Re-Run `deploy-test`/`deploy-production` Alone — Gitea Platform Limitation
|
||||||
|
|
||||||
|
`deploy-test` and `deploy-production` in `continuous_integration.yaml` are not ordinary jobs — each
|
||||||
|
one *calls* the reusable `deploy-scp.yaml` workflow (`uses: ./.gitea/workflows/deploy-scp.yaml`).
|
||||||
|
Re-running **only** one of these two jobs after a failure (Gitea's per-job "re-run" action) is a
|
||||||
|
known Gitea Actions limitation, not a bug in either workflow file here: Gitea does not cleanly resume
|
||||||
|
a `workflow_call` job inside its original run — it re-executes the call in a way that loses access to
|
||||||
|
that run's already-uploaded artifact. The `Download build artifact` step then fails immediately with:
|
||||||
|
```
|
||||||
|
List Artifacts - Error is not retryable
|
||||||
|
Status Code: 404
|
||||||
|
Error: List Artifacts failed: Artifact service responded with 404
|
||||||
|
```
|
||||||
|
even though `publish-test`/`publish-production` genuinely succeeded and uploaded the artifact
|
||||||
|
moments earlier in the same run (retention had not expired — this is not the same failure mode as
|
||||||
|
§ 1.7.3, and not a retention issue at all).
|
||||||
|
|
||||||
|
**Fix: re-run the entire workflow, not just this job** — but see § 1.7.5 immediately below before
|
||||||
|
doing that: Gitea's "re-run all jobs" on an *existing* run is not the same as triggering a genuinely
|
||||||
|
new run, and has its own, different failure mode.
|
||||||
|
|
||||||
|
#### 1.7.5 "Re-Run All Jobs" Can Silently Reuse Stale Secrets/Variables — Trigger a New Run Instead
|
||||||
|
|
||||||
|
Symptom: a secret (e.g. `PI_MAIN_PASSWORD`) was wrong, causing `sshpass`/`ssh` to fail with
|
||||||
|
`Permission denied, please try again.` (`sshpass` exit code 5 — the password itself was rejected,
|
||||||
|
not a connectivity or config problem). The secret is corrected in Gitea's UI and confirmed working
|
||||||
|
via a manual SSH test with the same value. **"Re-run all jobs" is used on the existing, already-failed
|
||||||
|
run — and it fails again, identically**, as if the fix never happened.
|
||||||
|
|
||||||
|
This matches a documented behaviour of Actions-style re-run implementations, seen concretely in
|
||||||
|
GitHub's own tooling ([cli/cli#13522](https://github.com/cli/cli/issues/13522)): re-running an
|
||||||
|
*existing* run can replay against secrets/variables as they were **when that run was first created**,
|
||||||
|
not their current values — particularly for a job that calls a reusable workflow with
|
||||||
|
`secrets: inherit` (exactly what `deploy-test`/`deploy-production` do here). The documented contract
|
||||||
|
("secrets are fetched at the time of the re-run") does not hold in practice for this case. A genuinely
|
||||||
|
**new** run (fresh push, or `workflow_dispatch`) always fetches current values correctly — only
|
||||||
|
re-running an existing run risks the stale snapshot.
|
||||||
|
|
||||||
|
**Net effect of § 1.7.4 + § 1.7.5 together**: neither of Gitea's two "re-run" options is fully safe
|
||||||
|
for `deploy-test`/`deploy-production` — "re-run this job alone" loses the artifact, "re-run all jobs"
|
||||||
|
can keep stale secrets. **The one reliable option is to trigger a brand new run** (an actual push, or
|
||||||
|
`workflow_dispatch`'s "Run workflow" button) rather than using either re-run action on a failed run,
|
||||||
|
whenever a secret or variable was just changed to fix that failure. Re-running an existing run is
|
||||||
|
only safe when nothing about its secrets/variables changed since it was created.
|
||||||
|
|
||||||
### 1.8 Database Backup Credentials (§ 4 depends on this)
|
### 1.8 Database Backup Credentials (§ 4 depends on this)
|
||||||
|
|
||||||
👤 **pi-main / `gitea-workflow`:**
|
👤 **pi-main / `gitea-workflow`:**
|
||||||
|
|||||||
+134
@@ -0,0 +1,134 @@
|
|||||||
|
# Monitoring Instructions
|
||||||
|
|
||||||
|
All three services below were built into the application during Construction (U4) — this document
|
||||||
|
only wires them to the real accounts and writes down the configuration that necessarily lives
|
||||||
|
outside this repository (Sentry alert rules, UptimeRobot monitors, Umami website entries).
|
||||||
|
|
||||||
|
**Who does what, where** — reuses the labels from `deployment-instructions.md` § "Who runs what,
|
||||||
|
where", plus the external services themselves:
|
||||||
|
- 👤 **pi-main / `gitea-workflow`** — the environment's `shared/env` file (§ 1.5 of
|
||||||
|
`deployment-instructions.md`)
|
||||||
|
- ☁️ **Gitea web UI** — repository Variables (Settings → Actions → Variables)
|
||||||
|
- ☁️ **Sentry web UI** — project settings and alert rules
|
||||||
|
- ☁️ **UptimeRobot web UI** — monitors and alert contacts
|
||||||
|
- ☁️ **Umami web UI** — website entries
|
||||||
|
|
||||||
|
## 1. Sentry
|
||||||
|
|
||||||
|
One project for the whole CMS (D-19); `Observability__Environment` (host-side, § 1.5) and
|
||||||
|
`VITE_APP_ENV` (build-time) are what tell test and production events apart in that one project —
|
||||||
|
there is no separate DSN per environment.
|
||||||
|
|
||||||
|
### 1.1 Gitea variable
|
||||||
|
|
||||||
|
☁️ **Gitea web UI** — Settings → Actions → Variables:
|
||||||
|
|
||||||
|
| Variable | Value |
|
||||||
|
|---|---|
|
||||||
|
| `VITE_SENTRY_DSN` | the project's DSN (same value used by both `publish-test` and `publish-production`) |
|
||||||
|
|
||||||
|
### 1.2 Host configuration
|
||||||
|
|
||||||
|
Already documented in `deployment-instructions.md` § 1.5 — `Observability__SentryDsn` and
|
||||||
|
`Observability__Environment` in each environment's `shared/env` file. `Observability__Environment`
|
||||||
|
**must** differ between the two files (`test` / `production`); see that section's note on why
|
||||||
|
`ASPNETCORE_ENVIRONMENT` can't be reused for this.
|
||||||
|
|
||||||
|
### 1.3 Alert rules (FR-19)
|
||||||
|
|
||||||
|
The six `security_event` tag values already emitted by `SecurityEvents.cs` — alert rules must filter
|
||||||
|
on the **tag**, not the message text (`nfr-design-patterns.md` Pattern 6):
|
||||||
|
|
||||||
|
| `security_event` tag value | Rule type | Threshold |
|
||||||
|
|---|---|---|
|
||||||
|
| `failed_login` | Count-based | > 20 events in 5 minutes |
|
||||||
|
| `authorization_denied` | Count-based | > 20 events in 5 minutes |
|
||||||
|
| `master_api_key_rejected` | Count-based | > 20 events in 5 minutes |
|
||||||
|
| `admin_bypass_rejected` | Count-based | > 20 events in 5 minutes |
|
||||||
|
| `rate_limit_triggered` | Count-based | > 20 events in 5 minutes |
|
||||||
|
| `migration_failure` | Always-fire | any occurrence (already `Critical` level — the process is exiting) |
|
||||||
|
|
||||||
|
20-in-5-minutes is a starting default (no production traffic history exists yet to tune against) —
|
||||||
|
revisit once real traffic volume is known.
|
||||||
|
|
||||||
|
☁️ **Sentry web UI** — Alerts → Create Alert Rule, once per row above:
|
||||||
|
1. Condition: `tags.security_event equals <value>`
|
||||||
|
2. For the five count-based rules: "when greater than 20 events, in 5 minutes" (or the platform's
|
||||||
|
equivalent rate-window phrasing)
|
||||||
|
3. For `migration_failure`: no count condition — fire on every matching event
|
||||||
|
4. Action: notify via whichever channel/integration alerts should reach (same destination used
|
||||||
|
elsewhere for this project is fine — nothing here requires a dedicated channel)
|
||||||
|
|
||||||
|
**Note (DEV-01, accepted deviation)**: Sentry's plan in use retains events for roughly 30 days,
|
||||||
|
short of the 90-day SECURITY-14 minimum. Already accepted as a cost decision at Requirements
|
||||||
|
Analysis — nothing to configure here, just to be aware of if investigating an old event.
|
||||||
|
|
||||||
|
## 2. UptimeRobot
|
||||||
|
|
||||||
|
Six monitors — `/health`, `/`, `/admin` × test/production (D-23). What each does and does not prove
|
||||||
|
(FR-17):
|
||||||
|
|
||||||
|
| URL | Proves | Does NOT prove |
|
||||||
|
|---|---|---|
|
||||||
|
| `https://test.slpsoftware.nl/health` | The test process is alive (liveness only, D-21) | Database connectivity, migrations applied — no DB check by design (Q17=A) |
|
||||||
|
| `https://test.slpsoftware.nl/` | Proxy Pi → pi-main routing and static hosting work for test | The API or admin SPA work at all |
|
||||||
|
| `https://test.slpsoftware.nl/admin` | The admin SPA is served and its assets aren't blocked by CSP | Login/auth actually succeeds — this only checks the shell loads |
|
||||||
|
| `https://slpsoftware.nl/health` | Same as above, production | Same as above, production |
|
||||||
|
| `https://slpsoftware.nl/` | Same as above, production | Same as above, production |
|
||||||
|
| `https://slpsoftware.nl/admin` | Same as above, production | Same as above, production |
|
||||||
|
|
||||||
|
☁️ **UptimeRobot web UI** — Add New Monitor, once per row above:
|
||||||
|
- Monitor Type: **HTTP(s)** (not Keyword/Port/Ping) — checks only the response status code (2xx/3xx),
|
||||||
|
the same liveness signal `deploy-scp.yaml`'s own `curl -f` health check already relies on
|
||||||
|
- Interval: 5 minutes is enough for all six — `/health` recovers within a process restart
|
||||||
|
(`Restart=on-failure`, `RestartSec=5` in the systemd units), no need for a tighter interval
|
||||||
|
- Alert Contact: the existing contact already configured on this account
|
||||||
|
|
||||||
|
**Free-plan note**: 6 monitors at 5-minute intervals fits comfortably within the free plan's 50-monitor
|
||||||
|
cap and 5-minute minimum interval — no paid tier needed for this feature's monitoring. One gap the
|
||||||
|
free plan leaves open: no SSL-certificate-expiry monitoring. Certbot's systemd timer renews
|
||||||
|
automatically (`deployment-instructions.md` § 1.7.1), so the risk is low, but a silently failed
|
||||||
|
renewal would only be caught reactively — once an HTTP(s) check itself starts failing — rather than
|
||||||
|
warned about in advance. Accepted as a known gap, not built around, consistent with this project's
|
||||||
|
other documented deviations (e.g. DEV-01's log-retention gap).
|
||||||
|
|
||||||
|
## 3. Umami
|
||||||
|
|
||||||
|
Reuses the existing self-hosted instance at `analytics.slpsoftware.nl` (D-24/ASM-05). This section
|
||||||
|
covers only the **admin SPA's** two website entries — the public website's Umami wiring (also
|
||||||
|
required by D-25) belongs to whichever website workspace serves `wwwroot/web/` (FR-09/FR-16), not to
|
||||||
|
this deploy pipeline.
|
||||||
|
|
||||||
|
☁️ **Umami web UI** — Websites → Add website, twice (once per environment), domain set to the
|
||||||
|
admin SPA's real URL (`test.slpsoftware.nl/admin`, `slpsoftware.nl/admin`) — note the resulting
|
||||||
|
website ID for each.
|
||||||
|
|
||||||
|
☁️ **Gitea web UI** — Settings → Actions → Variables:
|
||||||
|
|
||||||
|
| Variable | Value |
|
||||||
|
|---|---|
|
||||||
|
| `VITE_UMAMI_SCRIPT_URL` | `https://analytics.slpsoftware.nl/script.js` (shared — same instance, both environments) |
|
||||||
|
| `VITE_UMAMI_WEBSITE_ID_TEST` | the test website's ID from the step above |
|
||||||
|
| `VITE_UMAMI_WEBSITE_ID_PRODUCTION` | the production website's ID from the step above |
|
||||||
|
|
||||||
|
**These must stay in sync with two other Gitea variables that already exist from U5**
|
||||||
|
(`SECURITY_ALLOWED_SCRIPT_ORIGINS_TEST` / `_PRODUCTION`) and with the matching
|
||||||
|
`SecurityHeaders__AllowedScriptOrigins__0` / `_AllowedConnectOrigins__0` lines in each environment's
|
||||||
|
host `shared/env` file (`deployment-instructions.md` § 1.5) — all four must name the same
|
||||||
|
`https://analytics.slpsoftware.nl` origin, or either the CI gate (REF-U5-01) fails the build, or the
|
||||||
|
script loads in CI but is silently blocked by CSP on the real host (the gate cannot see the host
|
||||||
|
file, only the Gitea variable side of this).
|
||||||
|
|
||||||
|
## 4. Gitea Variables — Full Summary
|
||||||
|
|
||||||
|
For reference, every Actions variable this document and `deployment-instructions.md` § 1.9 together
|
||||||
|
require:
|
||||||
|
|
||||||
|
| Variable | Scope | Set by |
|
||||||
|
|---|---|---|
|
||||||
|
| `VITE_SENTRY_DSN` | shared | § 1.1 above |
|
||||||
|
| `VITE_UMAMI_SCRIPT_URL` | shared | § 3 above |
|
||||||
|
| `VITE_UMAMI_WEBSITE_ID_TEST` | test | § 3 above |
|
||||||
|
| `VITE_UMAMI_WEBSITE_ID_PRODUCTION` | production | § 3 above |
|
||||||
|
| `SECURITY_ALLOWED_SCRIPT_ORIGINS_TEST` / `_PRODUCTION` | per environment | already set at Deployment Setup — must match § 3's Umami origin |
|
||||||
|
| `DEPLOY_PATH_*`, `SERVICE_NAME_*`, `HEALTH_CHECK_URL_*` | per environment | `deployment-instructions.md` § 1.9 (unrelated to monitoring, listed there) |
|
||||||
+127
@@ -0,0 +1,127 @@
|
|||||||
|
# Monitoring Setup Plan
|
||||||
|
|
||||||
|
## Context already established (not re-asked)
|
||||||
|
|
||||||
|
The observability *stack* and *what it must cover* were already decided in Requirements Analysis and
|
||||||
|
built during Construction (U4) — this stage is about configuring the real external services and
|
||||||
|
writing the operational documentation, not re-deciding the approach:
|
||||||
|
|
||||||
|
- **Stack**: Sentry (backend `Sentry.AspNetCore` + frontend `@sentry/react`), self-hosted Umami
|
||||||
|
analytics, UptimeRobot uptime monitoring (D-18, D-24, requirements.md § "Monitoring and observability")
|
||||||
|
- **Sentry projects**: **one** project for the whole CMS, `environment` tag distinguishes
|
||||||
|
test/production (D-19) — so one DSN, reused by both environments and both frontend/backend
|
||||||
|
- **Sentry is optional by design**: an absent DSN disables it cleanly, console logging stays active
|
||||||
|
(BR-U4-08) — nothing breaks if a value is left blank temporarily
|
||||||
|
- **Alertable events already exist in code**: six `security_event` tag values are emitted today —
|
||||||
|
`failed_login`, `authorization_denied`, `master_api_key_rejected`, `admin_bypass_rejected`,
|
||||||
|
`rate_limit_triggered`, `migration_failure` (`SecurityEvents.cs`). Alert rules must filter on the
|
||||||
|
`security_event` tag, not message text (`nfr-design-patterns.md` Pattern 6) — this is explicitly
|
||||||
|
named as **Operations — Monitoring Setup** work in `logical-components.md` (FR-19)
|
||||||
|
- **UptimeRobot scope**: `/health`, `/` (public website) and `/admin`, per environment — 6 monitors
|
||||||
|
total (D-23)
|
||||||
|
- **Umami scope**: reuse the existing self-hosted instance at `analytics.slpsoftware.nl`, add new
|
||||||
|
website entries for this CMS rather than standing up a new instance (D-24/ASM-05). Both the admin
|
||||||
|
SPA (via `VITE_UMAMI_WEBSITE_ID_<ENV>`, built by this pipeline) and the public website (via the
|
||||||
|
separate website-workspace, FR-09/FR-16, out of this pipeline's scope) are measured (D-25) — this
|
||||||
|
stage only configures the **admin SPA's** two website entries; the public website's Umami wiring
|
||||||
|
belongs to whichever website workspace serves it
|
||||||
|
- **Health endpoint**: liveness only, no DB check, already built and already on the
|
||||||
|
`AvailabilityMiddleware` bypass list (D-21) — nothing left to build, only to point UptimeRobot at
|
||||||
|
- **Log retention deviation**: accepted knowingly (DEV-01) — Sentry's ~30-day retention against the
|
||||||
|
90-day SECURITY-14 minimum. No action here beyond documenting it.
|
||||||
|
|
||||||
|
## Question 1: Include Monitoring Setup?
|
||||||
|
|
||||||
|
Given U4 built all the alertable-event plumbing specifically so that Operations could wire up real
|
||||||
|
alert rules and monitors on top of it, declining this stage would leave that code inert — events
|
||||||
|
would be emitted into a Sentry project that doesn't exist yet, with no alert rule reading the tag,
|
||||||
|
and no UptimeRobot monitor watching `/health`. Recommended: A.
|
||||||
|
|
||||||
|
A) Yes — configure the real services and document it (Recommended)
|
||||||
|
B) No — monitoring is handled elsewhere or not needed
|
||||||
|
C) Not sure — suggest an approach and I'll decide
|
||||||
|
|
||||||
|
X) Other (please describe after [Answer]: tag below)
|
||||||
|
|
||||||
|
[Answer]:A
|
||||||
|
|
||||||
|
## Question 2: Monitoring Method
|
||||||
|
|
||||||
|
Confirms the stack already decided in Requirements Analysis — asked per the mandatory format, not
|
||||||
|
because it's genuinely open.
|
||||||
|
|
||||||
|
A) Sentry + self-hosted Umami + UptimeRobot, exactly as scoped above (Recommended)
|
||||||
|
B) Something else entirely — describe below
|
||||||
|
|
||||||
|
X) Other (please describe after [Answer]: tag below)
|
||||||
|
|
||||||
|
[Answer]:A
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
The remaining questions are the genuinely open items — real account/service facts that only you
|
||||||
|
know, which nothing in Construction could decide for you.
|
||||||
|
|
||||||
|
## Question 3: Sentry Project
|
||||||
|
|
||||||
|
Does a Sentry project for this CMS already exist (e.g. created ad hoc while testing U4 locally), or
|
||||||
|
does this stage need to walk through creating one from scratch?
|
||||||
|
|
||||||
|
A) Already exists — I'll provide the DSN after [Answer]: below
|
||||||
|
B) Doesn't exist yet — walk me through creating one (org, project platform picks for .NET + React,
|
||||||
|
where to find the DSN afterward)
|
||||||
|
C) Not yet, and I'd rather set it up myself later — document it as a TODO with exactly which Gitea
|
||||||
|
secrets/variables need the DSN once it exists
|
||||||
|
|
||||||
|
X) Other (please describe after [Answer]: tag below)
|
||||||
|
|
||||||
|
[Answer]:A — DSN to be added directly to the Gitea secret/variable once shared (kept out of this doc)
|
||||||
|
|
||||||
|
## Question 4: Sentry Alert Rule Thresholds
|
||||||
|
|
||||||
|
FR-19 requires alert rules for repeated auth failures and authorization violations, filtering on the
|
||||||
|
`security_event` tag (values: `failed_login`, `authorization_denied`, `master_api_key_rejected`,
|
||||||
|
`admin_bypass_rejected`, `rate_limit_triggered`; `migration_failure` is `Critical`-level and worth
|
||||||
|
its own always-fire rule regardless of count). What threshold should the repeated-failure rules use?
|
||||||
|
|
||||||
|
A) A single reasonable default for all of them (e.g. 20 occurrences in 5 minutes) — document it as a
|
||||||
|
starting point, tune later from real traffic (Recommended)
|
||||||
|
B) I'll specify exact thresholds per event type — describe below
|
||||||
|
C) No count threshold — alert on every occurrence of each of the six (noisier, but nothing is ever
|
||||||
|
missed)
|
||||||
|
|
||||||
|
X) Other (please describe after [Answer]: tag below)
|
||||||
|
|
||||||
|
[Answer]:A
|
||||||
|
|
||||||
|
## Question 5: UptimeRobot Account and Alert Contacts
|
||||||
|
|
||||||
|
6 monitors are needed (`/health`, `/`, `/admin` × test/production). Do you have an UptimeRobot
|
||||||
|
account already (e.g. from the reference `SlpSoftware` project), and where should alerts go?
|
||||||
|
|
||||||
|
A) Existing account, reuse it — alert contact(s): describe after [Answer]: below (email, and/or
|
||||||
|
any other channel already configured there)
|
||||||
|
B) Need a new account/monitor group set up from scratch — walk me through it
|
||||||
|
C) Not sure yet — document the 6 monitors' exact URLs and recommended check interval as a checklist,
|
||||||
|
I'll create them myself
|
||||||
|
|
||||||
|
X) Other (please describe after [Answer]: tag below)
|
||||||
|
|
||||||
|
[Answer]:A — existing account, alert contact to be confirmed
|
||||||
|
|
||||||
|
## Question 6: Umami Website Entries (Admin SPA)
|
||||||
|
|
||||||
|
Two new website entries are needed on the existing `analytics.slpsoftware.nl` instance — one per
|
||||||
|
environment — to obtain the `VITE_UMAMI_WEBSITE_ID_TEST`/`_PRODUCTION` values the CI workflow already
|
||||||
|
expects as Gitea variables (`continuous_integration.yaml` reads `vars.VITE_UMAMI_WEBSITE_ID_TEST`/
|
||||||
|
`_PRODUCTION`).
|
||||||
|
|
||||||
|
A) I have access to the Umami instance — I'll create the two entries and provide the website IDs
|
||||||
|
after [Answer]: below
|
||||||
|
B) Walk me through creating them (Umami's own UI/API steps)
|
||||||
|
C) Not yet — document it as a TODO with the exact variable names the pipeline expects, I'll fill
|
||||||
|
them in before the first real deploy
|
||||||
|
|
||||||
|
X) Other (please describe after [Answer]: tag below)
|
||||||
|
|
||||||
|
[Answer]:A — website IDs to be added directly to Gitea variables once created (kept out of this doc)
|
||||||
Reference in New Issue
Block a user