From e44cf493f6cc9aacafeffacb08a3b73fb10671a9 Mon Sep 17 00:00:00 2001 From: Sluijsens Date: Sun, 8 Nov 2026 00:17:09 +0100 Subject: [PATCH] Operations: deployment- en monitoringdocumentatie voor andreaslop-site-hosting Co-Authored-By: Claude Sonnet 5.5 --- aidlc-docs/active-features.md | 2 +- .../andreaslop-site-hosting/aidlc-state.md | 11 ++++- .../features/andreaslop-site-hosting/audit.md | 23 ++++++++++ .../deployment/deployment-instructions.md | 22 ++++++++++ .../operations/deployment/deployment-plan.md | 8 ++++ .../operations/deployment/rollback-plan.md | 10 +++++ .../operations/monitoring/monitoring-plan.md | 7 +++ .../operations/monitoring/monitoring-setup.md | 31 +++++++++++++ .../operations/plans/deployment-setup-plan.md | 43 +++++++++++++++++++ .../operations/plans/monitoring-setup-plan.md | 43 +++++++++++++++++++ 10 files changed, 198 insertions(+), 2 deletions(-) create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-instructions.md create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-plan.md create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/deployment/rollback-plan.md create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-plan.md create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-setup.md create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/plans/deployment-setup-plan.md create mode 100644 aidlc-docs/features/andreaslop-site-hosting/operations/plans/monitoring-setup-plan.md diff --git a/aidlc-docs/active-features.md b/aidlc-docs/active-features.md index 96b7fd8..a6c8e4d 100644 --- a/aidlc-docs/active-features.md +++ b/aidlc-docs/active-features.md @@ -9,4 +9,4 @@ | Local Dev Master/Slave Setup (local-dev-master-slave-setup) | ✅ Complete | unknown | Modules.Master, Api, Frontend | 2026-07-02 | | Gitea Deployment Workflow (gitea-deployment-workflow) | 🟡 Operations | feature/gitea-deployment-workflow | CI/CD, Api (hosting/config), Core, Modules, Frontend (build), Docs | 2026-07-27 | | SlpSoftware Production API (slpsoftware-api) | ✅ Complete | feature/slpsoftware-api | Clients (new Api.SlpSoftware), Modules (new Offerings module + Tests), Core (CmsHost), CI/CD (retargeted to Api.SlpSoftware) | 2026-08-02 | -| Andrea's site hosting (andreaslop-site-hosting) | 🟢 Construction | feature/andreaslop-site-hosting | Core (CmsHost, CSP, static content), Clients (new Api.AndreaSlopFotografie), Frontend (theme), CI/CD | 2026-11-06 | +| Andrea's site hosting (andreaslop-site-hosting) | 🟡 Operations | feature/andreaslop-site-hosting | Core (CmsHost, CSP, static content), Clients (new Api.AndreaSlopFotografie), Frontend (theme), CI/CD | 2026-11-06 | diff --git a/aidlc-docs/features/andreaslop-site-hosting/aidlc-state.md b/aidlc-docs/features/andreaslop-site-hosting/aidlc-state.md index 1b22b4c..bcfac78 100644 --- a/aidlc-docs/features/andreaslop-site-hosting/aidlc-state.md +++ b/aidlc-docs/features/andreaslop-site-hosting/aidlc-state.md @@ -5,7 +5,7 @@ - **Feature Slug**: andreaslop-site-hosting - **Project Type**: Brownfield - **Start Date**: 2026-11-06T00:06:13Z -- **Current Stage**: CONSTRUCTION - Build and Test (awaiting approval) +- **Current Stage**: OPERATIONS - Monitoring Setup (awaiting approval) - **Branch**: feature/andreaslop-site-hosting - **Request Type**: Feature (two units: U1 core-site-hosting, U2 andreaslop-client) @@ -47,3 +47,12 @@ - [x] Workflow Planning - approved - [x] Application Design - approved - [x] Units Generation - approved + +### Deployment Setup +- **Included**: Yes +- **Method**: CI/CD (Gitea Actions to on-premises Pi), test only +- **Status**: approved 2026-11-08 + +### Monitoring Setup +- **Included**: Yes +- **Approach**: Combination (logging + alerting on existing tooling; no new dashboards) diff --git a/aidlc-docs/features/andreaslop-site-hosting/audit.md b/aidlc-docs/features/andreaslop-site-hosting/audit.md index 9f06e7b..8611b1d 100644 --- a/aidlc-docs/features/andreaslop-site-hosting/audit.md +++ b/aidlc-docs/features/andreaslop-site-hosting/audit.md @@ -415,3 +415,26 @@ **Timestamp**: 2026-11-07 **User Input**: "Goedkeuren en doe maar optie A" **AI Response**: Build and Test approved. Option A done: Cryptography.Xml 10.0.14 (Core + 3 test projects), zod ^4.6.6, @tanstack/react-router 1.170.46; vulnerability scans clean, all suites green. Proceeding to Operations. + +## Operations - Deployment Setup plan +**Timestamp**: 2026-11-07 +**AI Response**: Build and Test complete. Deployment Setup question file created (operations/plans/deployment-setup-plan.md) with recommended answers that confirm the U2 infrastructure decisions. Awaiting answers. + +## Operations - Deployment Setup artifacts +**Timestamp**: 2026-11-07 +**User Input**: "alles A" +**AI Response**: Answers recorded (all A). Generated operations/deployment/{deployment-plan,deployment-instructions,rollback-plan}.md. Awaiting approval. + +## Operations - Deployment Setup approval +**Timestamp**: 2026-11-08 +**User Input**: "ga verder" +**AI Response**: Deployment Setup approved and complete. Starting Monitoring Setup. + +## Operations - Monitoring Setup plan +**Timestamp**: 2026-11-08 +**AI Response**: Monitoring question file created (operations/plans/monitoring-setup-plan.md), awaiting answers. + +## Operations - Monitoring Setup artifacts +**Timestamp**: 2026-11-08 +**User Input (via CMS & API)**: Q1 A, Q2 D, Q3 A, Q4 A, Q5 A. Also: CI proof via workflow_dispatch started by Bryan (option B), no PR, no merge; branch feature/andreaslop-site-hosting pushed (explicit go relayed 2026-11-08). +**AI Response**: Generated operations/monitoring/{monitoring-plan,monitoring-setup}.md. No Sentry alert rules were created (shared project; documented for Bryan/website session). Awaiting approval. diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-instructions.md b/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-instructions.md new file mode 100644 index 0000000..b0d9c2a --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-instructions.md @@ -0,0 +1,22 @@ +# Deployment Instructions - ASF test host + +The pipeline definition already exists in `.gitea/workflows/continuous_integration.yaml` (jobs `config-asf`, `publish-asf-test`, `deploy-asf-test`, reusing `deploy-scp.yaml`); this runbook sequences the one-time Pi work, the first deploy and every later deploy. The exact one-time steps (1-18) are in `construction/andreaslop-client/infrastructure-design/manual-steps-for-bryan.md`; the Pi-Main session drafts the commands from that list. + +## 0. Before anything is pushed +1. The vulnerability gate is clean (done: Xml 10.0.14, zod, react-router). Run `dotnet test SlpModularCms.sln` where MariaDB exists, or let CI do it. +2. Decide when to push or merge (done by Bryan; the assistant never pushes unasked). A merge to master with `ASF_TEST_DEPLOY_ENABLED` unset changes nothing for ASF, but does deploy the shared Core change (patched packages, per-client limiters, opt-in forwarded headers) to SlpSoftware test on that push. Watch that first SlpSoftware deploy. + +## 1. One-time preparation (manual-steps phase A, nothing live yet) +Verify prerequisites, create database/user, directories + symlink `shared/wwwroot-web`, `shared/env`, user unit (enabled, not started), ufw 5200 from 192.168.1.102, DB backup entry, Gitea variables (`DEPLOY_PATH_ASF_TEST`, `SERVICE_NAME_ASF_TEST`, `HEALTH_CHECK_URL_ASF_TEST`, `VITE_SENTRY_DSN_ASF`). Leave `ASF_TEST_DEPLOY_ENABLED` unset. + +## 2. Cut over (phase B, accepted short test downtime) +Save the current proxy block; verify the three forwarded headers are present; change only `proxy_pass` to `http://192.168.1.103:5200`; test config; reload. Set `ASF_TEST_DEPLOY_ENABLED=true`; push to master or run the workflow manually with "Deploy ASF test". + +## 3. What a deploy does (every time) +Build admin (`VITE_THEME=asf`, Dutch default) -> publish linux-arm64 -> upload to `releases/` -> link `wwwroot/web` to `shared/wwwroot-web` -> atomic switch of `current` -> `systemctl --user restart` -> `/health` on the public URL (10 tries) -> prune to current + previous. + +## 4. Verify after the first deploy (phase C) +`curl -I https://test.andreaslop.nl/` (CSP, HSTS, noindex), an inner page, `/Over-Mij` -> one 301, unknown path -> 404 page, `/robots.txt`, `/health`, `/api/v1/System/capabilities` (no Offerings), admin in Dutch with ASF theme, create the first administrator with the setup token, then remove `Identity__Setup__Token` from `shared/env` and restart the unit. Report forwarded-header observation and optional timing. + +## 5. Later deploys +Push to master (or manual run). The website itself is delivered by its own repository into `/mnt/storage1/www/html/andreaslop-test`; no host restart is needed. diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-plan.md b/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-plan.md new file mode 100644 index 0000000..fd05001 --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/deployment-plan.md @@ -0,0 +1,8 @@ +# Deployment Plan - andreaslop-site-hosting (test only) + +- **Method**: CI/CD pipeline (Gitea Actions) to an on-premises Pi, as decided in U2 Infrastructure Design (answers: all A). +- **Environment**: test only, `https://test.andreaslop.nl`. Production for Andrea Slop Fotografie is a separate later decision (`Website__NoIndex=false`, own database, port 5201, own route) and not part of this feature. +- **Why this shape**: it reuses the proven SlpSoftware path (`deploy-scp.yaml`: timestamped release, atomic `current` switch, restart, public `/health` check, prune to two releases) without touching it. ASF has its own jobs behind one switch, `ASF_TEST_DEPLOY_ENABLED`. +- **Triggers**: push to master, or manual run with "Deploy ASF test" ticked; both only when the switch variable is `true`. +- **Secrets**: Gitea secrets `PI_MAIN_*` (existing) and the Pi file `shared/env` (mode 600). Nothing secret is committed. No approval gate for test. +- **Sources of truth**: `construction/andreaslop-client/infrastructure-design/{infrastructure-design,deployment-architecture,manual-steps-for-bryan}.md`. diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/rollback-plan.md b/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/rollback-plan.md new file mode 100644 index 0000000..1e47b19 --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/deployment/rollback-plan.md @@ -0,0 +1,10 @@ +# Rollback Plan - ASF test host + +| What went wrong | Rollback | +|---|---| +| Bad release (health check red or wrong behaviour) | On the Pi: `ln -sfn releases/ current && systemctl --user restart andreaslopfotografie-test`. The previous release is always kept (prune keeps current + 1). | +| Proxy route broke the test site | Restore the saved nginx server block on the proxy Pi (manual step 10), test config, reload; the old nginx fallback on the app Pi serves the site again. | +| Pipeline misbehaves | Set `ASF_TEST_DEPLOY_ENABLED` to anything but `true`: all ASF jobs are skipped, SlpSoftware is unaffected. | +| Shared Core change misbehaves on another host | Redeploy the previous release of that host the same way. The new behaviour is opt-in (forwarded headers) or equivalent to before (limiter buckets share the proxy address when no proxy is configured). | +| Database | Migrations run automatically at start and are forward-only. The test database is backed up by the Pi DB backup (step 7); restore from that backup if a migration must be undone. | +| Setup token leaked | Replace `Identity__Setup__Token` in `shared/env` and restart, or remove it once the first administrator exists. | diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-plan.md b/aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-plan.md new file mode 100644 index 0000000..38d50dc --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-plan.md @@ -0,0 +1,7 @@ +# Monitoring Plan - andreaslop-site-hosting (test) + +- **Approach** (Q2 = D): logging plus alerting on tooling that already exists; no new dashboards (Q5 = A), no central log aggregator (Q3 = A). +- **Tooling reused**: systemd/journald on the Pi (host logs), the Andrea Slop Fotografie **website's** existing Sentry project (Q3 = B in U2; NOT SlpSoftware's admin project), the host's `/health` endpoint (already checked by every deploy), the admin's same-origin `/sentry-tunnel`. +- **Covered**: host errors and security events (Sentry), 301/404 website lines (journald), browser errors from the admin (via the tunnel), liveness (`/health` after each deploy and optionally an uptime monitor). +- **Not covered / known gaps**: no performance dashboard or latency alerting; the website's own browser errors are reported by the website itself; Sentry events from website, host and admin share one project, separated by the `environment` tag (`test`) and URL; test only, production monitoring comes with a future production setup. +- **Verification**: after the first deploy, trigger one wrong setup token and one 404 and confirm they show up (see `monitoring-setup.md`, section 4). diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-setup.md b/aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-setup.md new file mode 100644 index 0000000..e6c7e29 --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/monitoring/monitoring-setup.md @@ -0,0 +1,31 @@ +# Monitoring Setup - ASF test host + +## 1. Logging +- **Where**: stdout of the systemd user unit `andreaslopfotografie-test`, read with `journalctl --user -u andreaslopfotografie-test -f` (as `gitea-workflow`). +- **Levels** (from `shared/env`): `Logging__LogLevel__SlpModularCms.Website=Information` on test (one line per website 301 and 404; `Warning` on production later); everything else keeps the committed defaults; `Microsoft.EntityFrameworkCore.Database.Command` stays at `Warning` so SQL parameters (e-mail addresses) are not logged. +- **Never logged**: the setup token, passwords, tokens, full e-mail addresses. Website log lines are sanitised paths. +- **Sample lines**: `Website redirect 301 /Over-Mij -> /over-mij`; `Security event setup_token_rejected: setup token rejected on /api/v1/Setup/owner`. +- **Sentry**: `Observability__SentryDsn` (website project DSN) and `Observability__Environment=test` in `shared/env`; the admin build gets the same DSN through `VITE_SENTRY_DSN_ASF` and reports through `/sentry-tunnel`. + +## 2. Alerting (Q4 = A), in the website's Sentry project, notification by e-mail to the project owner +Create these rules once, filtered on `environment:test` (the assistant did NOT create them; they change a shared Sentry project, so Bryan or the website session adds them, or asks for the `sentry:sentry-create-alert` skill to do it): + +| Rule | Condition | Why | +|---|---|---| +| Setup token rejected | tag `security_event:setup_token_rejected`, 1+ events in 5 min | someone guesses the first-admin token | +| Failed logins | tag `security_event:failed_login`, 10+ in 5 min | brute force on the admin | +| Rate limit | tag `security_event:rate_limit_triggered`, 1+ in 5 min | limiter hit; with forwarded headers this is per visitor | +| Migration failure | tag `security_event:migration_failure`, 1 event | host cannot start; also /health goes silent | +| Unhandled errors | new issue in `environment:test`, level error or higher | host or admin bug | +| Uptime | monitor `https://test.andreaslop.nl/health` every 5 min if Sentry uptime is available (otherwise any external pinger) | the deploy only checks once | + +Each alert event carries the tag via `SecurityEventProcessor`, so the rules survive message wording changes. The first `rate_limit_triggered` events also confirm whether forwarded headers work: the limiter should count real visitors, not 192.168.1.102. + +## 3. Dashboards (Q5 = A) +None added. Use Sentry Issues filtered on `environment:test`, and on the Pi `journalctl`, `systemctl --user status andreaslopfotografie-test`, `ls -l current`. + +## 4. Verification after the first deploy +1. Open `/doesnotexist` -> journald shows a 404 line. +2. POST a wrong `X-Setup-Token` to `/api/v1/Setup/owner` (before the first administrator exists) -> 401 and a `setup_token_rejected` event in Sentry with tag `security_event`. +3. Stop the unit briefly -> uptime alert (if configured) and the deploy health check would fail. +4. Check that no event contains the token or a full e-mail address. diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/plans/deployment-setup-plan.md b/aidlc-docs/features/andreaslop-site-hosting/operations/plans/deployment-setup-plan.md new file mode 100644 index 0000000..e581918 --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/plans/deployment-setup-plan.md @@ -0,0 +1,43 @@ +# Deployment Setup Plan - andreaslop-site-hosting + +Context: most deployment decisions were already made in U2 Infrastructure Design (CI/CD via Gitea Actions to the test Pi, atomic release switch, rollback by symlink, secrets in `shared/env`, switch variable `ASF_TEST_DEPLOY_ENABLED`). The recommended answers below simply confirm that, so Operations produces one consolidated runbook instead of re-deciding it. Test environment only. + +## Question 1: Include Deployment Setup? +Do you want a consolidated deployment runbook and checks for this feature as part of this workflow? + +- A) Yes - consolidate the existing pipeline, the Pi steps and the rollback into one runbook (recommended) +- B) No - deployment is handled elsewhere (the infrastructure design and manual steps already cover it) +- C) Not sure - suggest an approach and I decide +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan: "alles A") + +## Question 2: Deployment Method +- A) CI/CD pipeline (Gitea Actions, the jobs added in U2) (recommended) +- B) Cloud platform +- C) On-premises / self-hosted server deployment (done via the pipeline above) +- D) Manual deployment +- E) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan: "alles A") + +## Question 3: Environments +- A) Test only (`test.andreaslop.nl`); production is a separate later decision (recommended) +- B) Test and production +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan: "alles A") + +## Question 4: Rollback Strategy +- A) Keep the documented approach: previous release stays on disk, `ln -sfn` + restart; route rollback by restoring the saved proxy block; CI off via the switch variable (recommended) +- B) Add an automated rollback job to the pipeline +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan: "alles A") + +## Question 5: Secrets and approval gates +- A) Keep: secrets only in Gitea secrets and the Pi `shared/env` (mode 600), no approval gate for test (recommended) +- B) Add a manual approval step before the test deploy +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan: "alles A") diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/plans/monitoring-setup-plan.md b/aidlc-docs/features/andreaslop-site-hosting/operations/plans/monitoring-setup-plan.md new file mode 100644 index 0000000..0dc8cea --- /dev/null +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/plans/monitoring-setup-plan.md @@ -0,0 +1,43 @@ +# Monitoring Setup Plan - andreaslop-site-hosting + +Context: the CMS already ships structured logging (journald on the Pi), Sentry with alertable security events (`security_event` tag, including the new `setup_token_rejected`), a same-origin `/sentry-tunnel` for the admin, and `/health` (checked by the deploy). The decision (Q3 = B) is to reuse the Andrea Slop Fotografie website's existing Sentry project. Test environment only. Recommended answers are marked. + +## Question 1: Include Monitoring Setup? +- A) Yes - document what is observed and add alert recommendations on the existing tooling (recommended) +- B) No - reuse the existing setup as is +- C) Not sure - suggest an approach +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D) + +## Question 2: Monitoring Approach +- A) Logging only +- B) Alerting only +- C) Dashboards only +- D) Combination: logging + alerting on existing tooling, no new dashboards (recommended) +- E) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D) + +## Question 3: Logging +- A) Keep journald on the Pi plus Sentry; `Information` for `SlpModularCms.Website` on test (301/404 lines), `Warning` elsewhere; token never logged (recommended) +- B) Add a central log aggregator +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D) + +## Question 4: Alerting +Which alerts, in the website's Sentry project, notified by email to the project owner? +- A) Security events (`setup_token_rejected`, `failed_login`, `rate_limit_triggered`) and any `migration_failure`/unhandled error; plus the website's uptime check on `/health` if Sentry uptime is available (recommended) +- B) Only `migration_failure` and unhandled errors +- C) No alerts yet +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D) + +## Question 5: Dashboards and existing infrastructure +- A) No new dashboard; use Sentry issues filtered by environment `test` and the existing Pi tooling (recommended) +- B) Add a dedicated dashboard +- X) Other (please describe after [Answer]: tag below) + +[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D)