Operations: deployment- en monitoringdocumentatie voor andreaslop-site-hosting
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -5,7 +5,7 @@
|
||||
- **Feature Slug**: andreaslop-site-hosting
|
||||
- **Project Type**: Brownfield
|
||||
- **Start Date**: 2026-11-06T00:06:13Z
|
||||
- **Current Stage**: CONSTRUCTION - Build and Test (awaiting approval)
|
||||
- **Current Stage**: OPERATIONS - Monitoring Setup (awaiting approval)
|
||||
- **Branch**: feature/andreaslop-site-hosting
|
||||
- **Request Type**: Feature (two units: U1 core-site-hosting, U2 andreaslop-client)
|
||||
|
||||
@@ -47,3 +47,12 @@
|
||||
- [x] Workflow Planning - approved
|
||||
- [x] Application Design - approved
|
||||
- [x] Units Generation - approved
|
||||
|
||||
### Deployment Setup
|
||||
- **Included**: Yes
|
||||
- **Method**: CI/CD (Gitea Actions to on-premises Pi), test only
|
||||
- **Status**: approved 2026-11-08
|
||||
|
||||
### Monitoring Setup
|
||||
- **Included**: Yes
|
||||
- **Approach**: Combination (logging + alerting on existing tooling; no new dashboards)
|
||||
|
||||
@@ -415,3 +415,26 @@
|
||||
**Timestamp**: 2026-11-07
|
||||
**User Input**: "Goedkeuren en doe maar optie A"
|
||||
**AI Response**: Build and Test approved. Option A done: Cryptography.Xml 10.0.14 (Core + 3 test projects), zod ^4.6.6, @tanstack/react-router 1.170.46; vulnerability scans clean, all suites green. Proceeding to Operations.
|
||||
|
||||
## Operations - Deployment Setup plan
|
||||
**Timestamp**: 2026-11-07
|
||||
**AI Response**: Build and Test complete. Deployment Setup question file created (operations/plans/deployment-setup-plan.md) with recommended answers that confirm the U2 infrastructure decisions. Awaiting answers.
|
||||
|
||||
## Operations - Deployment Setup artifacts
|
||||
**Timestamp**: 2026-11-07
|
||||
**User Input**: "alles A"
|
||||
**AI Response**: Answers recorded (all A). Generated operations/deployment/{deployment-plan,deployment-instructions,rollback-plan}.md. Awaiting approval.
|
||||
|
||||
## Operations - Deployment Setup approval
|
||||
**Timestamp**: 2026-11-08
|
||||
**User Input**: "ga verder"
|
||||
**AI Response**: Deployment Setup approved and complete. Starting Monitoring Setup.
|
||||
|
||||
## Operations - Monitoring Setup plan
|
||||
**Timestamp**: 2026-11-08
|
||||
**AI Response**: Monitoring question file created (operations/plans/monitoring-setup-plan.md), awaiting answers.
|
||||
|
||||
## Operations - Monitoring Setup artifacts
|
||||
**Timestamp**: 2026-11-08
|
||||
**User Input (via CMS & API)**: Q1 A, Q2 D, Q3 A, Q4 A, Q5 A. Also: CI proof via workflow_dispatch started by Bryan (option B), no PR, no merge; branch feature/andreaslop-site-hosting pushed (explicit go relayed 2026-11-08).
|
||||
**AI Response**: Generated operations/monitoring/{monitoring-plan,monitoring-setup}.md. No Sentry alert rules were created (shared project; documented for Bryan/website session). Awaiting approval.
|
||||
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
# Deployment Instructions - ASF test host
|
||||
|
||||
The pipeline definition already exists in `.gitea/workflows/continuous_integration.yaml` (jobs `config-asf`, `publish-asf-test`, `deploy-asf-test`, reusing `deploy-scp.yaml`); this runbook sequences the one-time Pi work, the first deploy and every later deploy. The exact one-time steps (1-18) are in `construction/andreaslop-client/infrastructure-design/manual-steps-for-bryan.md`; the Pi-Main session drafts the commands from that list.
|
||||
|
||||
## 0. Before anything is pushed
|
||||
1. The vulnerability gate is clean (done: Xml 10.0.14, zod, react-router). Run `dotnet test SlpModularCms.sln` where MariaDB exists, or let CI do it.
|
||||
2. Decide when to push or merge (done by Bryan; the assistant never pushes unasked). A merge to master with `ASF_TEST_DEPLOY_ENABLED` unset changes nothing for ASF, but does deploy the shared Core change (patched packages, per-client limiters, opt-in forwarded headers) to SlpSoftware test on that push. Watch that first SlpSoftware deploy.
|
||||
|
||||
## 1. One-time preparation (manual-steps phase A, nothing live yet)
|
||||
Verify prerequisites, create database/user, directories + symlink `shared/wwwroot-web`, `shared/env`, user unit (enabled, not started), ufw 5200 from 192.168.1.102, DB backup entry, Gitea variables (`DEPLOY_PATH_ASF_TEST`, `SERVICE_NAME_ASF_TEST`, `HEALTH_CHECK_URL_ASF_TEST`, `VITE_SENTRY_DSN_ASF`). Leave `ASF_TEST_DEPLOY_ENABLED` unset.
|
||||
|
||||
## 2. Cut over (phase B, accepted short test downtime)
|
||||
Save the current proxy block; verify the three forwarded headers are present; change only `proxy_pass` to `http://192.168.1.103:5200`; test config; reload. Set `ASF_TEST_DEPLOY_ENABLED=true`; push to master or run the workflow manually with "Deploy ASF test".
|
||||
|
||||
## 3. What a deploy does (every time)
|
||||
Build admin (`VITE_THEME=asf`, Dutch default) -> publish linux-arm64 -> upload to `releases/<timestamp>` -> link `wwwroot/web` to `shared/wwwroot-web` -> atomic switch of `current` -> `systemctl --user restart` -> `/health` on the public URL (10 tries) -> prune to current + previous.
|
||||
|
||||
## 4. Verify after the first deploy (phase C)
|
||||
`curl -I https://test.andreaslop.nl/` (CSP, HSTS, noindex), an inner page, `/Over-Mij` -> one 301, unknown path -> 404 page, `/robots.txt`, `/health`, `/api/v1/System/capabilities` (no Offerings), admin in Dutch with ASF theme, create the first administrator with the setup token, then remove `Identity__Setup__Token` from `shared/env` and restart the unit. Report forwarded-header observation and optional timing.
|
||||
|
||||
## 5. Later deploys
|
||||
Push to master (or manual run). The website itself is delivered by its own repository into `/mnt/storage1/www/html/andreaslop-test`; no host restart is needed.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Deployment Plan - andreaslop-site-hosting (test only)
|
||||
|
||||
- **Method**: CI/CD pipeline (Gitea Actions) to an on-premises Pi, as decided in U2 Infrastructure Design (answers: all A).
|
||||
- **Environment**: test only, `https://test.andreaslop.nl`. Production for Andrea Slop Fotografie is a separate later decision (`Website__NoIndex=false`, own database, port 5201, own route) and not part of this feature.
|
||||
- **Why this shape**: it reuses the proven SlpSoftware path (`deploy-scp.yaml`: timestamped release, atomic `current` switch, restart, public `/health` check, prune to two releases) without touching it. ASF has its own jobs behind one switch, `ASF_TEST_DEPLOY_ENABLED`.
|
||||
- **Triggers**: push to master, or manual run with "Deploy ASF test" ticked; both only when the switch variable is `true`.
|
||||
- **Secrets**: Gitea secrets `PI_MAIN_*` (existing) and the Pi file `shared/env` (mode 600). Nothing secret is committed. No approval gate for test.
|
||||
- **Sources of truth**: `construction/andreaslop-client/infrastructure-design/{infrastructure-design,deployment-architecture,manual-steps-for-bryan}.md`.
|
||||
@@ -0,0 +1,10 @@
|
||||
# Rollback Plan - ASF test host
|
||||
|
||||
| What went wrong | Rollback |
|
||||
|---|---|
|
||||
| Bad release (health check red or wrong behaviour) | On the Pi: `ln -sfn releases/<previous> current && systemctl --user restart andreaslopfotografie-test`. The previous release is always kept (prune keeps current + 1). |
|
||||
| Proxy route broke the test site | Restore the saved nginx server block on the proxy Pi (manual step 10), test config, reload; the old nginx fallback on the app Pi serves the site again. |
|
||||
| Pipeline misbehaves | Set `ASF_TEST_DEPLOY_ENABLED` to anything but `true`: all ASF jobs are skipped, SlpSoftware is unaffected. |
|
||||
| Shared Core change misbehaves on another host | Redeploy the previous release of that host the same way. The new behaviour is opt-in (forwarded headers) or equivalent to before (limiter buckets share the proxy address when no proxy is configured). |
|
||||
| Database | Migrations run automatically at start and are forward-only. The test database is backed up by the Pi DB backup (step 7); restore from that backup if a migration must be undone. |
|
||||
| Setup token leaked | Replace `Identity__Setup__Token` in `shared/env` and restart, or remove it once the first administrator exists. |
|
||||
@@ -0,0 +1,7 @@
|
||||
# Monitoring Plan - andreaslop-site-hosting (test)
|
||||
|
||||
- **Approach** (Q2 = D): logging plus alerting on tooling that already exists; no new dashboards (Q5 = A), no central log aggregator (Q3 = A).
|
||||
- **Tooling reused**: systemd/journald on the Pi (host logs), the Andrea Slop Fotografie **website's** existing Sentry project (Q3 = B in U2; NOT SlpSoftware's admin project), the host's `/health` endpoint (already checked by every deploy), the admin's same-origin `/sentry-tunnel`.
|
||||
- **Covered**: host errors and security events (Sentry), 301/404 website lines (journald), browser errors from the admin (via the tunnel), liveness (`/health` after each deploy and optionally an uptime monitor).
|
||||
- **Not covered / known gaps**: no performance dashboard or latency alerting; the website's own browser errors are reported by the website itself; Sentry events from website, host and admin share one project, separated by the `environment` tag (`test`) and URL; test only, production monitoring comes with a future production setup.
|
||||
- **Verification**: after the first deploy, trigger one wrong setup token and one 404 and confirm they show up (see `monitoring-setup.md`, section 4).
|
||||
@@ -0,0 +1,31 @@
|
||||
# Monitoring Setup - ASF test host
|
||||
|
||||
## 1. Logging
|
||||
- **Where**: stdout of the systemd user unit `andreaslopfotografie-test`, read with `journalctl --user -u andreaslopfotografie-test -f` (as `gitea-workflow`).
|
||||
- **Levels** (from `shared/env`): `Logging__LogLevel__SlpModularCms.Website=Information` on test (one line per website 301 and 404; `Warning` on production later); everything else keeps the committed defaults; `Microsoft.EntityFrameworkCore.Database.Command` stays at `Warning` so SQL parameters (e-mail addresses) are not logged.
|
||||
- **Never logged**: the setup token, passwords, tokens, full e-mail addresses. Website log lines are sanitised paths.
|
||||
- **Sample lines**: `Website redirect 301 /Over-Mij -> /over-mij`; `Security event setup_token_rejected: setup token rejected on /api/v1/Setup/owner`.
|
||||
- **Sentry**: `Observability__SentryDsn` (website project DSN) and `Observability__Environment=test` in `shared/env`; the admin build gets the same DSN through `VITE_SENTRY_DSN_ASF` and reports through `/sentry-tunnel`.
|
||||
|
||||
## 2. Alerting (Q4 = A), in the website's Sentry project, notification by e-mail to the project owner
|
||||
Create these rules once, filtered on `environment:test` (the assistant did NOT create them; they change a shared Sentry project, so Bryan or the website session adds them, or asks for the `sentry:sentry-create-alert` skill to do it):
|
||||
|
||||
| Rule | Condition | Why |
|
||||
|---|---|---|
|
||||
| Setup token rejected | tag `security_event:setup_token_rejected`, 1+ events in 5 min | someone guesses the first-admin token |
|
||||
| Failed logins | tag `security_event:failed_login`, 10+ in 5 min | brute force on the admin |
|
||||
| Rate limit | tag `security_event:rate_limit_triggered`, 1+ in 5 min | limiter hit; with forwarded headers this is per visitor |
|
||||
| Migration failure | tag `security_event:migration_failure`, 1 event | host cannot start; also /health goes silent |
|
||||
| Unhandled errors | new issue in `environment:test`, level error or higher | host or admin bug |
|
||||
| Uptime | monitor `https://test.andreaslop.nl/health` every 5 min if Sentry uptime is available (otherwise any external pinger) | the deploy only checks once |
|
||||
|
||||
Each alert event carries the tag via `SecurityEventProcessor`, so the rules survive message wording changes. The first `rate_limit_triggered` events also confirm whether forwarded headers work: the limiter should count real visitors, not 192.168.1.102.
|
||||
|
||||
## 3. Dashboards (Q5 = A)
|
||||
None added. Use Sentry Issues filtered on `environment:test`, and on the Pi `journalctl`, `systemctl --user status andreaslopfotografie-test`, `ls -l current`.
|
||||
|
||||
## 4. Verification after the first deploy
|
||||
1. Open `/doesnotexist` -> journald shows a 404 line.
|
||||
2. POST a wrong `X-Setup-Token` to `/api/v1/Setup/owner` (before the first administrator exists) -> 401 and a `setup_token_rejected` event in Sentry with tag `security_event`.
|
||||
3. Stop the unit briefly -> uptime alert (if configured) and the deploy health check would fail.
|
||||
4. Check that no event contains the token or a full e-mail address.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Deployment Setup Plan - andreaslop-site-hosting
|
||||
|
||||
Context: most deployment decisions were already made in U2 Infrastructure Design (CI/CD via Gitea Actions to the test Pi, atomic release switch, rollback by symlink, secrets in `shared/env`, switch variable `ASF_TEST_DEPLOY_ENABLED`). The recommended answers below simply confirm that, so Operations produces one consolidated runbook instead of re-deciding it. Test environment only.
|
||||
|
||||
## Question 1: Include Deployment Setup?
|
||||
Do you want a consolidated deployment runbook and checks for this feature as part of this workflow?
|
||||
|
||||
- A) Yes - consolidate the existing pipeline, the Pi steps and the rollback into one runbook (recommended)
|
||||
- B) No - deployment is handled elsewhere (the infrastructure design and manual steps already cover it)
|
||||
- C) Not sure - suggest an approach and I decide
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan: "alles A")
|
||||
|
||||
## Question 2: Deployment Method
|
||||
- A) CI/CD pipeline (Gitea Actions, the jobs added in U2) (recommended)
|
||||
- B) Cloud platform
|
||||
- C) On-premises / self-hosted server deployment (done via the pipeline above)
|
||||
- D) Manual deployment
|
||||
- E) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan: "alles A")
|
||||
|
||||
## Question 3: Environments
|
||||
- A) Test only (`test.andreaslop.nl`); production is a separate later decision (recommended)
|
||||
- B) Test and production
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan: "alles A")
|
||||
|
||||
## Question 4: Rollback Strategy
|
||||
- A) Keep the documented approach: previous release stays on disk, `ln -sfn` + restart; route rollback by restoring the saved proxy block; CI off via the switch variable (recommended)
|
||||
- B) Add an automated rollback job to the pipeline
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan: "alles A")
|
||||
|
||||
## Question 5: Secrets and approval gates
|
||||
- A) Keep: secrets only in Gitea secrets and the Pi `shared/env` (mode 600), no approval gate for test (recommended)
|
||||
- B) Add a manual approval step before the test deploy
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan: "alles A")
|
||||
@@ -0,0 +1,43 @@
|
||||
# Monitoring Setup Plan - andreaslop-site-hosting
|
||||
|
||||
Context: the CMS already ships structured logging (journald on the Pi), Sentry with alertable security events (`security_event` tag, including the new `setup_token_rejected`), a same-origin `/sentry-tunnel` for the admin, and `/health` (checked by the deploy). The decision (Q3 = B) is to reuse the Andrea Slop Fotografie website's existing Sentry project. Test environment only. Recommended answers are marked.
|
||||
|
||||
## Question 1: Include Monitoring Setup?
|
||||
- A) Yes - document what is observed and add alert recommendations on the existing tooling (recommended)
|
||||
- B) No - reuse the existing setup as is
|
||||
- C) Not sure - suggest an approach
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D)
|
||||
|
||||
## Question 2: Monitoring Approach
|
||||
- A) Logging only
|
||||
- B) Alerting only
|
||||
- C) Dashboards only
|
||||
- D) Combination: logging + alerting on existing tooling, no new dashboards (recommended)
|
||||
- E) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D)
|
||||
|
||||
## Question 3: Logging
|
||||
- A) Keep journald on the Pi plus Sentry; `Information` for `SlpModularCms.Website` on test (301/404 lines), `Warning` elsewhere; token never logged (recommended)
|
||||
- B) Add a central log aggregator
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D)
|
||||
|
||||
## Question 4: Alerting
|
||||
Which alerts, in the website's Sentry project, notified by email to the project owner?
|
||||
- A) Security events (`setup_token_rejected`, `failed_login`, `rate_limit_triggered`) and any `migration_failure`/unhandled error; plus the website's uptime check on `/health` if Sentry uptime is available (recommended)
|
||||
- B) Only `migration_failure` and unhandled errors
|
||||
- C) No alerts yet
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D)
|
||||
|
||||
## Question 5: Dashboards and existing infrastructure
|
||||
- A) No new dashboard; use Sentry issues filtered by environment `test` and the existing Pi tooling (recommended)
|
||||
- B) Add a dedicated dashboard
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]: A (Bryan via CMS & API, one by one: all recommended; Q2 = D)
|
||||
Reference in New Issue
Block a user