From 0e86125f127cd03d5127ad63efd2608474ed5efa Mon Sep 17 00:00:00 2001 From: Sluijsens Date: Mon, 9 Nov 2026 09:03:49 +0100 Subject: [PATCH] Handback-documenten bijgewerkt, Build-and-Test-aanvulling, Operations-vragen e-mailfeature Co-Authored-By: Claude Sonnet 5.5 --- .../andreaslop-site-hosting/handback-u2.md | 21 ++++++ .../production-readiness-checklist.md | 13 ++++ .../aidlc-state.md | 1 + .../identity-email-notifications/audit.md | 12 ++++ .../build-and-test/build-and-test-summary.md | 19 ++++++ .../identity-email-notifications/handback.md | 24 +++++++ .../operations/plans/operations-questions.md | 65 +++++++++++++++++++ 7 files changed, 155 insertions(+) create mode 100644 aidlc-docs/features/identity-email-notifications/handback.md create mode 100644 aidlc-docs/features/identity-email-notifications/operations/plans/operations-questions.md diff --git a/aidlc-docs/features/andreaslop-site-hosting/handback-u2.md b/aidlc-docs/features/andreaslop-site-hosting/handback-u2.md index 39ee07f..2e5e9ff 100644 --- a/aidlc-docs/features/andreaslop-site-hosting/handback-u2.md +++ b/aidlc-docs/features/andreaslop-site-hosting/handback-u2.md @@ -32,3 +32,24 @@ The 301 `Location` was built from the decoded path without re-encoding: `/a%3Fb/ 2. Setup token must be removed from `shared/env` after the first administrator exists (step 17). 3. The proxy block change is only the `proxy_pass` target (port 5200); the website repo generator is updated in U3. 4. Production for ASF is out of scope. + +## 7. Addendum 2026-11-09: what happened after this hand-back (all on ASF test; nothing on production) +After the first test deploy (run 608) the feature was used on `test.andreaslop.nl` and a series of defects and requests came back through the "CMS & API" session. Each has its own record under `aidlc-docs/bugs/` (cause, fix, test) and the deploys are in `audit.md`. + +| Item | Record / where | Shared-code consequence for SlpSoftware | +|---|---|---| +| Admin files gave 404 in PrerenderedPages mode (catch-all route won over the admin static files) | audit.md, run 610; `notreserved` route constraint | none | +| Invitation link pointed at the website root instead of `/admin/...` | audit.md, run 612; `buildAppUrl` | none (frontend only) | +| Nine CMS points: invited name not stored, labels, Administrator may use site settings, deactivated or deleted users refused at once (JWT active check), re-inviting a deleted address, setup title, brand logo | `bugs/cms-admin-nine-points` | deactivation takes effect immediately; two additive nullable fields later; Administrator can switch availability | +| Pending/expired invitations cannot be removed; invitation lifetime configurable | `bugs/invitation-delete-and-lifetime` | expired invitations now listed; `Identity:Invitation:ExpiryHours` | +| Role hierarchy enforced in the API (Administrator only manages Users, never hands out Owner) | same record, follow-up decisions | stricter API | +| Owner got 403 when promoting an Administrator to Owner (caller identified by a null name claim) | `bugs/owner-cannot-promote-requester-identity` | fixes the user-management API on every host | +| Availability gate stopped the website and `/admin` of a disabled CMS (503) | `bugs/availability-gate-blocks-website-and-admin` | gate now covers `/api/` only | +| Setup page shown when the setup status could not be read (429 from the strict login limiter) | `bugs/setup-page-after-failed-status-read` | status read moved to the `refresh` limiter | +| Website log level cannot be set through a systemd EnvironmentFile (dot in the variable name) | `Website__LogLevel` (README, infra docs) | none | +| Admin layout: the whole page scrolled and the sidebar moved with it | audit/commit `f242da8` | shared layout | +| Flaky CI frontend test (redirect timers left running after unmount) | commit `892c548`, hook `useDelayedAction` | shared pages | + +Also built on top of this feature, as its own feature branch: **identity-email-notifications** (mail for invitations, new links, notices and password reset; per-site sender in the site settings; three additive migrations). See `aidlc-docs/features/identity-email-notifications/handback.md`. + +ASF test currently runs the branch `feature/identity-email-notifications` (it contains this whole feature plus the e-mail feature). Still not done for this feature: merge to master (needs Bryan's explicit go and the watched SlpSoftware test deploy), production. diff --git a/aidlc-docs/features/andreaslop-site-hosting/operations/production-readiness-checklist.md b/aidlc-docs/features/andreaslop-site-hosting/operations/production-readiness-checklist.md index 248b2c7..cdfa943 100644 --- a/aidlc-docs/features/andreaslop-site-hosting/operations/production-readiness-checklist.md +++ b/aidlc-docs/features/andreaslop-site-hosting/operations/production-readiness-checklist.md @@ -34,3 +34,16 @@ Scope: the TEST environment `test.andreaslop.nl`. Production is out of scope; it 6. Merging to master deploys the shared Core change and patched packages to SlpSoftware test automatically; watch that deploy. 7. Doc-only commits are local and unpushed. 8. Production for Andrea Slop Fotografie (not prepared, listed only): own database and user, port 5201, own proxy route and certificate, `Website__NoIndex=false`, own `shared/env`, production deploy path/variables and an approval decision, production Sentry environment and alert rules, backup entry, HSTS preload/redirect review, decision on the placeholder radius and dark mode. + + +## Status update 2026-11-09 (open follow-ups above, now) +1. CI proof: done (runs 607 onward, all later runs green except one timer flake fixed in 892c548). +2. First test deploy: DONE (run 608, then 610, 612, 613-626 on the branch). ASF test is live and used by Bryan and Andrea. +3. After the first deploy: first administrators exist (Andrea Owner, Bryan Administrator); the verification list was exercised in use. Still to confirm by Bryan/Pi-Main: setup token removed from `shared/env`, forwarded-header behaviour observed (per-client rate-limit partitions), optional Pi timing. +4. Sentry alert rules: not confirmed created (owner: "CMS & API" with Bryan's go). Open. +5. Backup entry for the test database: not confirmed (Pi-Main). Open. +6. Merge to master (and the automatic SlpSoftware test deploy of the shared Core changes, which are now much larger: see handback-u2 addendum): not done, needs Bryan's explicit go; the branch has already been deployed to SlpSoftware test once (run 625/626/627 from the e-mail branch). +7. Doc commits: pushed. +8. Production: unchanged (not prepared). +Operations questions for this feature were all answered (deployment Q1-5 "alles A", monitoring Q1-5 recommended, Q2 = D). Nothing in this feature's Operations stage is waiting for an answer; what remains is the list above plus the e-mail feature's own Operations stage. +Note: the log-level variable in the documents above was replaced by `Website__LogLevel` (a variable name with a dot is ignored by systemd). diff --git a/aidlc-docs/features/identity-email-notifications/aidlc-state.md b/aidlc-docs/features/identity-email-notifications/aidlc-state.md index 1b1853a..791c6f0 100644 --- a/aidlc-docs/features/identity-email-notifications/aidlc-state.md +++ b/aidlc-docs/features/identity-email-notifications/aidlc-state.md @@ -52,3 +52,4 @@ - [x] U3 password-reset - Code Generation done, build and tests green (see construction/password-reset/code/code-summary.md); hand-back pending (D2, the security stamp in the access token, was built in U2) - [~] Build and Test - summary written (construction/build-and-test/build-and-test-summary.md); manual real test mail on ASF test pending - [ ] Operations (deployment, monitoring, production readiness incl. SPF/DKIM alignment checklist) +- Hand-back written (handback.md); Operations questions issued (operations/plans/operations-questions.md), nothing executed until answered diff --git a/aidlc-docs/features/identity-email-notifications/audit.md b/aidlc-docs/features/identity-email-notifications/audit.md index 1ee4b34..88ab474 100644 --- a/aidlc-docs/features/identity-email-notifications/audit.md +++ b/aidlc-docs/features/identity-email-notifications/audit.md @@ -93,3 +93,15 @@ **Context**: Construction U3 complete; Build and Test and Operations follow. --- + +## Documentation update and Operations questions + +**Timestamp**: 2026-11-09T08:30:00Z + +**User Input**: "(Relayed by 'CMS & API') JA, werk de handback-docs en de Build-and-Test-samenvatting bij (latere fixes: roltoekenning, poort, timers, SMTP-fases, 503-poort, setup-guard, 9 punten, bug-records), commit en push. Operations-fase e-mailfeature: stel een vragenbestand op met de keuzes voor Bryan (deployment, monitoring/alerts voor mail_configuration_invalid / identity_mail_failed / auth_failed, SPF/DKIM-checklist, productie-uitrol) en geef me de vragen; geen Operations-uitvoering zonder zijn antwoorden. Ook de ASF-feature andreaslop-site-hosting Operations: kijk of die vragen/keuzes al beantwoord zijn ..." + +**AI Response**: "Wrote handback.md, updated build-and-test-summary.md, wrote operations/plans/operations-questions.md (8 questions), added an addendum to the ASF handback-u2.md and a status update to its production-readiness checklist (all its Operations questions were answered; open items listed). No Operations execution." + +**Context**: Documentation and Operations planning; waiting for Bryan's answers. + +--- diff --git a/aidlc-docs/features/identity-email-notifications/construction/build-and-test/build-and-test-summary.md b/aidlc-docs/features/identity-email-notifications/construction/build-and-test/build-and-test-summary.md index c03d1b0..90ec563 100644 --- a/aidlc-docs/features/identity-email-notifications/construction/build-and-test/build-and-test-summary.md +++ b/aidlc-docs/features/identity-email-notifications/construction/build-and-test/build-and-test-summary.md @@ -34,3 +34,22 @@ Access tokens issued before the release are refused once (silent refresh issues ## Open items before the feature can close 1. The real test mail on ASF test (manual, above). 2. Operations stage (deployment config, monitoring, production readiness incl. the SPF/DKIM alignment checklist). 3. Bryan's approval of this stage. + + +## Update 2026-11-09: later changes on this branch (all with tests, all pushed) +| Change | Test evidence | +|---|---| +| D2 stamp in the access token (built in U2) | Core: real Identity + real tokens, password change ends earlier access tokens, refresh issues a new one | +| Soft mail configuration: a half-finished mail configuration starts the host with mail OFF and one Error line naming the settings (no values, no secrets) | `MailSoftFailureTests`, `DeployedConfigurationTests` for master, slave, SlpSoftware, ASF | +| `Website__LogLevel` (systemd drops variable names with a dot) | `WebsiteLogLevelTests` | +| Role hierarchy in the API; the caller is recognised by the user id of the token (a real token has no name claim: an Owner got 403 promoting to Owner) | `UsersApiWithRealTokensTests` (real tokens, real pipeline; 5 of 7 failed before the fix) | +| Mail diagnostics: SMTP command failure phase and status, redacted first reply line, specific codes (`rejected_sender`, `rejected_recipient`, `tls_failed`, `connection_failed`), `timeout_connect/auth/send`, a timeout of its own per step (it was one shared budget), `TimeoutSeconds` 1..120 | `SmtpFailureTests`, `SmtpSenderDiagnosticsTests` (SMTP stub that refuses or stalls), property test (control characters could re-glue an address) | +| "New invitation link" mails only on request (`sendEmail`), with a choice dialog | controller, real-HTTP and UI tests | +| Layout: only the content scrolls, the sidebar stays | layout class tests | +| Redirect timers cleaned up on unmount (a CI-only "window is not defined") | page test fails on the old code, `useDelayedAction` tests | +| Availability gate only covers `/api/` (a disabled CMS still serves the website and `/admin`) | 27 new Availability tests incl. a real in-memory pipeline; 16 failed before | +| Setup page never shown for an unreadable setup status; login notice for a disabled CMS | 6 error cases (5 failed before), limiter test | + +Current counts (local, last full run): Core 563, Identity 141, Availability 109, Master 57, Offerings 57, ASF host configuration 6, frontend 390. CI (with MariaDB): all green on the runs 626 and before; the MariaDB-backed migration up/down/up and `DateTimeOffset` round trip run there. + +Still open for this stage: the manual real test mail from the ASF test host (the SMTP sender address and TLS mode of `mail.andreaslop.nl` are being sorted out by Bryan and Pi-Main; the failure diagnostics above exist to see why a mail is refused) and Bryan's approval. diff --git a/aidlc-docs/features/identity-email-notifications/handback.md b/aidlc-docs/features/identity-email-notifications/handback.md new file mode 100644 index 0000000..fea9076 --- /dev/null +++ b/aidlc-docs/features/identity-email-notifications/handback.md @@ -0,0 +1,24 @@ +# Hand-back - identity-email-notifications (U1 + U2 + U3 and what followed) + +Branch `feature/identity-email-notifications` (contains the whole andreaslop-site-hosting feature as well). Not merged, not on production. Deployed (test only): ASF test and SlpSoftware test (mail switched OFF there until the host gets its mail settings). + +## 1. What exists +- **Mail core** (Core): `IEmailSender` (MailKit, StartTls, one timeout per step, failures are values), options with one decision point `IMailAvailability` (SMTP host set AND configuration valid, otherwise mail OFF with one Error line naming the settings), nl/en template (logo, brand colour, button plus raw link, plain-text part), language per user (user, invitation, site default, nl). +- **Site settings** (new table `SiteSettings`): sender address chosen only from `Email:AllowedFromAddresses` plus the default, sender name 1..80 characters; Owner changes, Administrator reads; Mail card on the Settings page (only when mail is on). +- **Flows**: invitation mail (language chosen by the administrator), new link (choice: copy only / copy and mail), resend, notices on deactivation, reactivation, role change, password reset (`forgot-password`, `reset-password`; request touches nothing, only a completed reset revokes sessions; link single use, 60 minutes; per-address cooldown 5 minutes; the request is handled after the response so timing does not reveal accounts). +- **Security**: the security stamp is part of the access token (a password change/reset ends earlier access tokens, refresh keeps working); deactivated/deleted users refused at once; role hierarchy in the API; no secret, address, token or link in any log. + +## 2. Configuration for a host (see construction/mail-foundation/infrastructure-design) +`Email__Smtp__Host/Port/EnableSsl/Username/Password/TimeoutSeconds`, `App__PublicBaseUrl`, `Email__FromAddress`, `Email__FromName`, `Email__AllowedFromAddresses__n`, optional `Email__LogoUrl`, `Email__BrandColor`, `App__DefaultLanguage`, `Identity__Invitation__ExpiryHours`, `Identity__PasswordReset__*`. All per host in `shared/env`. A host without a mail setting is unchanged; no new variable is required by this release. + +## 3. Behaviour changes for SlpSoftware (when it is on a release with this branch) +First deploy: three additive migrations (`SiteSettings`, `Users.PreferredLanguage`, `Invitations.Language`), existing access tokens refused once (silent refresh), a password change ends earlier access tokens, user management API stricter (role hierarchy, caller recognition by id), a deactivated or deleted user is refused at once (one extra query per authenticated request), availability gate covers `/api/` only, setup status read under the ordinary limiter, expired invitations listed, two new anonymous endpoints (404 while mail is off), default Identity token providers registered. Mail stays off without an SMTP host. + +## 4. Verification +See `construction/build-and-test/build-and-test-summary.md` (including the update of 2026-11-09). Real MariaDB: migrations up, down, up and the `DateTimeOffset` round trip (CI and locally in a throw-away container). Not yet done: the manual real test mail on ASF test. + +## 5. Open items +1. Real test mail from ASF test (needs the sender address on `andreaslop.nl` via env and the TLS mode of `mail.andreaslop.nl` confirmed; the failure log now names the phase and SMTP status). +2. Operations stage: questions in `operations/plans/operations-questions.md` (for Bryan), nothing executed. +3. Merge to master and the SlpSoftware production roll-out: not approved. +4. Optional follow-up decisions: per-user language is in, per-site template editing is not; whether "Gebruiker uitnodigen" should also ask before mailing (Bryan: no, unchanged). diff --git a/aidlc-docs/features/identity-email-notifications/operations/plans/operations-questions.md b/aidlc-docs/features/identity-email-notifications/operations/plans/operations-questions.md new file mode 100644 index 0000000..b8d977a --- /dev/null +++ b/aidlc-docs/features/identity-email-notifications/operations/plans/operations-questions.md @@ -0,0 +1,65 @@ +# Operations questions - identity-email-notifications + +Nothing here is executed until the questions are answered (through the "CMS & API" session). Context: mail is built and tested; it is live (switched on) on ASF test only once its sender address and TLS mode are settled, and off on SlpSoftware test until its host gets the settings. Production for both sites is not part of the current approval. + +## Question 1: Rolling the mail settings out +How do the mail settings reach each host? + +- A) By hand by Pi-Main in the host's `shared/env` (mode 600), using the variable list in the infrastructure design (secrets stay out of Gitea); nothing in CI changes +- B) The non-secret settings (`App__PublicBaseUrl`, `Email__FromName`, ...) come from Gitea variables written by the deploy workflow; only the SMTP credentials are set by hand +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 2: Order of the roll-out +- A) ASF test (done) then SlpSoftware test, each with its own SMTP host, then production of each site later, one at a time, each needing its own approval +- B) ASF test, then straight to both productions together after the real test mails succeed +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 3: The sender account for ASF +- A) A dedicated mailbox/account on `mail.andreaslop.nl` (for example `no-reply@andreaslop.nl`) created by Bryan for this purpose; its address goes into `Email__FromAddress` and `Email__AllowedFromAddresses__0` +- B) An existing mailbox of the site (for example `info@andreaslop.nl`), accepting that replies reach that mailbox +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 4: Which mail problems should raise an alert +Today a failed mail is logged at Warning level (journal only; Sentry sees Error events and uses Warnings only as breadcrumbs), and a bad mail configuration at start-up is an Error (Sentry sees it). What do you want? + +- A) Nothing more: failed mails are looked at in the journal when somebody reports a problem +- B) Raise the log level to Error (so Sentry sends an alert to Bryan) for mail failures that point to a configuration problem: `auth_failed`, `rejected_sender`, `tls_failed`, `not_configured`; keep timeouts and recipient problems at Warning +- C) B, plus an alert when the same host fails at least three mails in fifteen minutes whatever the reason (a Sentry rule on the events) +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 5: Start-up check after a release +- A) The deploy's existing `/health` check stays as it is; after changing mail settings the person doing it reads the journal for `mail_status enabled=true` or `mail_configuration_invalid` (written in the runbook) +- B) A) plus the CMS reports `emailEnabled` on `/health` details so a pinger can see that mail is off on a host that should have it +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 6: Deliverability (SPF, DKIM, DMARC) before production +Facts today: both `slpsoftware.nl` and `andreaslop.nl` have SPF (`a mx include:smtp.spf.ziggo.nl`), a `dkim` record and DMARC `p=none`. +- A) Before production: send test mails to a Gmail and an Outlook mailbox, check in "show original" that SPF, DKIM and DMARC pass and are aligned with the From domain; leave DMARC at `p=none` for now and revisit it later +- B) A), and tighten DMARC to `p=quarantine` for the site's domain before production +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 7: Rollback +Rolling a release back keeps the three additive database changes (older code simply ignores the new columns and table). +- A) To switch mail off quickly: remove `Email__Smtp__Host` from `shared/env` and restart the unit (the copy-link flow keeps working); for a bad release use the existing previous-release switch of the deploy +- B) A), and additionally a documented way to drop the new columns/table (only if a restore is ever needed) +- X) Other (please describe after [Answer]: tag below) + +[Answer]: + +## Question 8: Who approves production readiness for this feature +- A) Bryan, after the real test mails on ASF test and SlpSoftware test and the answers above are carried out +- X) Other (please describe after [Answer]: tag below) + +[Answer]: