E-mailfeature Operations: deployment, monitoring, productiegereedheid; configuratiefouten als Error-log
Mislukte mails door een configuratieprobleem (auth_failed, rejected_sender, tls_failed, not_configured) worden als Error gelogd zodat Sentry een alert stuurt; timeouts en ontvangerproblemen blijven Warning. Documentatie: uitrol per host met de hand in shared/env, rollback, monitoringplan, checklist; ASF-productie op IIS als open punt voor een latere feature. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -53,3 +53,8 @@
|
||||
- [~] Build and Test - summary written (construction/build-and-test/build-and-test-summary.md); manual real test mail on ASF test pending
|
||||
- [ ] Operations (deployment, monitoring, production readiness incl. SPF/DKIM alignment checklist)
|
||||
- Hand-back written (handback.md); Operations questions issued (operations/plans/operations-questions.md), nothing executed until answered
|
||||
### OPERATIONS
|
||||
- [x] Deployment Setup - approved answers Q1 A, Q2 A (ASF production separate, IIS), Q3 A, Q7 A; operations/deployment/*
|
||||
- [x] Monitoring Setup - Q4 B (config-related mail failures at Error), Q5 A; log-level change built and tested; operations/monitoring/*
|
||||
- [x] Production Readiness - checklist written; verdict: not yet (real test mails, Sentry rule, Bryan's approval); operations/production-readiness-checklist.md
|
||||
- Open for a later feature: ASF production on IIS (hosting assumptions listed in operations/deployment/deployment-plan.md)
|
||||
|
||||
@@ -105,3 +105,15 @@
|
||||
**Context**: Documentation and Operations planning; waiting for Bryan's answers.
|
||||
|
||||
---
|
||||
|
||||
## Operations stage
|
||||
|
||||
**Timestamp**: 2026-11-09T10:00:00Z
|
||||
|
||||
**User Input**: "(Relayed by 'CMS & API', Bryan's answers) 1 A (met de hand in shared/env). 2 A, MAAR: voor ASF bestaat nog geen productieomgeving en die wordt een IIS-server i.p.v. Raspberry Pi; ASF-productie pakken we dus los op (niet in deze feature). Volgorde nu: ASF test, SlpSoftware test, dan SlpSoftware-productie met eigen akkoord. 3 A (no-reply@andreaslop.nl op mail.andreaslop.nl). 4 B ... 5 A ... 6 A ... 7 A ... 8 A ... Voer nu Deployment Setup, Monitoring Setup en Production Readiness uit voor de e-mailfeature (alleen docs + de kleine logniveau-wijziging uit vraag 4 ...). Leg vast dat ASF-productie op IIS komt en wat dat betekent voor de hostingaannames in jouw docs ... markeer als open punt voor die latere ASF-productiefeature, niet nu uitwerken."
|
||||
|
||||
**AI Response**: "Recorded the answers, wrote deployment plan/instructions/rollback, monitoring plan/setup, production readiness checklist, and built the log-level change (configuration-related mail failures auth_failed, rejected_sender, tls_failed, not_configured are Error events, the rest Warning) with tests. IIS recorded as an open point for the later ASF-production feature."
|
||||
|
||||
**Context**: Operations phase complete for the e-mail feature (docs and code); production readiness verdict: not yet.
|
||||
|
||||
---
|
||||
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# Deployment instructions - turning mail on for a host
|
||||
|
||||
For whoever manages the host (Pi-Main). Do this per host and per environment; credentials never go into the repository, a chat or a ticket.
|
||||
|
||||
## Before you start
|
||||
- The release containing the mail feature is already deployed (a host without mail settings behaves exactly as before).
|
||||
- You have: the SMTP host and port, the account (username, password) that may send as the chosen sender address, the public https address of the site (no path).
|
||||
|
||||
## Steps
|
||||
1. Open the host's environment file (Pi: `~/apps/<site>/<env>/shared/env`, mode 600).
|
||||
2. Add, in one go (the order inside the file does not matter, but set `App__PublicBaseUrl` together with the host or the host starts with mail off):
|
||||
- `App__PublicBaseUrl=https://<site address>`
|
||||
- `Email__Smtp__Host=<smtp host>`, `Email__Smtp__Port=587`, `Email__Smtp__EnableSsl=true`
|
||||
- `Email__Smtp__Username=<account>`, `Email__Smtp__Password=<password>`
|
||||
- ASF: `Email__FromAddress=no-reply@andreaslop.nl`, `Email__AllowedFromAddresses__0=no-reply@andreaslop.nl`, `Email__FromName=Andrea Slop Fotografie` (SlpSoftware: nothing more is needed)
|
||||
3. Restart the unit (Pi: `systemctl --user restart <unit>`).
|
||||
4. Read the journal (Pi: `journalctl --user -u <unit> -n 50`):
|
||||
- success: a line `mail_status enabled=true smtp_host=... starttls=true default_from_domain=...`;
|
||||
- a line `mail_configuration_invalid ... <setting names>` (Error): mail stays OFF, the site runs; fix the named settings and restart.
|
||||
5. `curl https://<site>/api/v1/System/capabilities` shows `"emailEnabled":true`.
|
||||
6. Real test (Bryan): in the admin invite a mailbox of yours (Gmail and Outlook, see the readiness checklist); the dialog says "mail sent" or names the reason. Also try "Wachtwoord vergeten?" on the login page.
|
||||
7. If a mail is refused, the journal line `identity_mail_failed code=... detail=... phase=... timings=...` tells what happened (`rejected_sender`: the account may not send as that address; `auth_failed`: wrong credentials; `tls_failed`: STARTTLS problem; `timeout_*`: slow or blocked server). Settings to change: see the infrastructure design.
|
||||
|
||||
## Changing or removing mail later
|
||||
Edit the same file and restart. To switch mail off: delete `Email__Smtp__Host` (the copy-link flow keeps working) and restart.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
# Deployment plan - identity-email-notifications
|
||||
|
||||
Decisions (Operations questions, answered by Bryan via "CMS & API"): Q1 A, Q2 A (with the ASF production exception below), Q3 A, Q7 A.
|
||||
|
||||
## Method
|
||||
The code ships through the existing Gitea Actions pipeline (`deploy_asf_test`, `deploy_test`; production only with its own approval). Nothing in the workflow changes. The mail settings are NOT part of the deploy: Pi-Main sets them by hand in the host's `shared/env` (mode 600), because they contain credentials.
|
||||
|
||||
## Order
|
||||
1. ASF test (code live; mail settings in `shared/env`, real test mail pending).
|
||||
2. SlpSoftware test (code live since run 625/626; mail still off until its `shared/env` has the settings).
|
||||
3. SlpSoftware production, only after steps 1 and 2 are accepted and with Bryan's separate approval.
|
||||
4. **ASF production is not part of this feature**: it does not exist yet and will run on an IIS server instead of a Raspberry Pi. It becomes its own piece of work (see "Open point: ASF production on IIS" below).
|
||||
|
||||
## Per-host mail settings (all in that host's environment)
|
||||
| Host | `Email__Smtp__Host` | Sender | Notes |
|
||||
|---|---|---|---|
|
||||
| ASF test | `mail.andreaslop.nl`, port 587, `EnableSsl=true` | `no-reply@andreaslop.nl` in `Email__FromAddress` and `Email__AllowedFromAddresses__0` | account on that server created by Bryan; TLS mode (STARTTLS on 587) to be confirmed by the first real test mail |
|
||||
| SlpSoftware test | `mail.slpsoftware.nl`, 587, `EnableSsl=true` | default `no-reply@slpsoftware.nl` (no variable needed) | credentials of the account that may send as that address |
|
||||
| SlpSoftware production | same server as test | same | own environment, own `App__PublicBaseUrl` |
|
||||
|
||||
Common: `Email__Smtp__Username`, `Email__Smtp__Password`, `App__PublicBaseUrl` (https, no path, per environment), optional `Email__Smtp__TimeoutSeconds` (default 10 per step), `Email__FromName`. Full list: `construction/mail-foundation/infrastructure-design/infrastructure-design.md`.
|
||||
|
||||
## Open point: ASF production on IIS (decided 2026-11-09, not designed here)
|
||||
Everything in this feature's and the andreaslop-site-hosting documents that talks about the host assumes the Raspberry Pi: a systemd user unit (`andreaslopfotografie-test`), an `EnvironmentFile` named `shared/env` (mode 600, readable only by the service user), `journalctl` for logs, `systemctl --user restart` after a change, `deploy-scp.yaml` (scp + symlink `current` + restart), a reverse-proxy Pi in front, and Linux paths. For an IIS host these assumptions need an answer in the later ASF-production feature (listed, NOT worked out now):
|
||||
- **Environment variables**: the same variable names work (double underscore), but are set in `web.config` (`<aspNetCore><environmentVariables>`), as machine-level variables or per app pool, not in `shared/env`. Secrets: no file mode 600, so decide where the SMTP credentials live (ACL-protected `web.config` section or machine variables, never in the repository) and who may read them.
|
||||
- **Restart**: an app pool recycle or `app_offline.htm` instead of `systemctl restart`; a changed variable needs a recycle.
|
||||
- **Logs**: stdout logging to a file or the Windows event log instead of `journalctl`; the `mail_status`, `mail_configuration_invalid` and `identity_mail_failed` lines must be reachable there (and Sentry keeps working as is).
|
||||
- **Deployment**: the scp-plus-symlink workflow does not fit; another deploy path is needed (Web Deploy, or file copy into a release folder with a physical-path switch), with the same previous-release rollback idea.
|
||||
- **Hosting model**: ASP.NET Core module (in-process or out-of-process), the .NET runtime on the server, HTTPS binding and certificate on IIS itself (no reverse-proxy Pi), `ForwardedHeaders__KnownProxies` probably different or unnecessary, `ASPNETCORE_ENVIRONMENT`.
|
||||
- **Database and mail server**: where MariaDB lives for that host, and whether `mail.andreaslop.nl` is reachable from the IIS server (firewall, port 587).
|
||||
- **Health check**: the `/health` URL check of the workflow is fine; the way it is reached changes with the new hostname.
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
# Rollback plan - identity-email-notifications
|
||||
|
||||
Decision (Q7 A).
|
||||
|
||||
1. **Switch mail off quickly** (no code change): remove `Email__Smtp__Host` from the host's environment file and restart the unit. The admin hides every mail feature (`emailEnabled=false`), invitations work by copying the link, the two anonymous endpoints answer 404. Nothing is lost.
|
||||
2. **Bad release**: use the existing previous-release switch of the deploy (re-point `current` to the previous release and restart; see the andreaslop-site-hosting rollback plan for the Pi). The three database changes of this feature (`SiteSettings` table, `Users.PreferredLanguage`, `Invitations.Language`) are additive and nullable; the previous code simply ignores them, so no database change is needed to go back.
|
||||
3. **After a rollback of the code**: access tokens issued by the newer release carry a security-stamp claim that the old code ignores; they keep working. Tokens issued by the old code lack the claim and are refused once by a newer release (the silent refresh fixes that).
|
||||
4. A documented way to drop the new columns and table is deliberately not part of the plan (Bryan chose option A); it would only be needed for a full restore.
|
||||
5. IIS (later, ASF production): the same two switches exist in another form (remove the variable and recycle the app pool; keep the previous release folder and switch the physical path).
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
# Monitoring plan - identity-email-notifications
|
||||
|
||||
Decisions (Q4 B, Q5 A): journal plus Sentry on the existing tooling; no new dashboards; no threshold rule.
|
||||
|
||||
## Signals (structured log events, never with credentials, addresses, tokens or links)
|
||||
| Event | Level | When | Meaning |
|
||||
|---|---|---|---|
|
||||
| `mail_status enabled=true|false ...` | Information | every start | whether mail is on, with SMTP host, port, STARTTLS and the default sender DOMAIN |
|
||||
| `mail_configuration_invalid: ... {settings}` | **Error** | start, once | an SMTP host is set but the configuration is incomplete or invalid: mail stays OFF; names the settings |
|
||||
| `identity_mail_failed code=auth_failed / rejected_sender / tls_failed / not_configured` | **Error** | per failed mail | configuration problem, somebody has to fix it (wrong credentials, a sender the account may not use, TLS, no host) |
|
||||
| `identity_mail_failed code=timeout_* / rejected_recipient / rejected / connection_failed / unknown` | Warning | per failed mail | passes by itself or concerns one message; journal and breadcrumbs only |
|
||||
| `identity_mail_sent kind=... recipient_domain=...` | Information | per sent mail | volume, no address |
|
||||
| `password_reset_request_ignored`, `password_reset_completed`, `password_reset_request_failed` | Information / Warning | reset flow | cooldown hits, completed resets |
|
||||
|
||||
## Alerting (Sentry, existing project of each site, notification by e-mail to the project owner)
|
||||
Sentry turns Error events into issues. Create one rule per environment (test and production) in the site's Sentry project:
|
||||
- **Rule "mail configuration problem"**: when an issue is created or reopened and its message contains `identity_mail_failed` or `mail_configuration_invalid` (level Error), notify the project owner by e-mail. No frequency threshold (Q4 B).
|
||||
Nothing else: timeouts and recipient problems are not alerted.
|
||||
|
||||
## Where to look
|
||||
- Pi: `journalctl --user -u <unit> | grep -E "mail_status|identity_mail_|mail_configuration_invalid"`.
|
||||
- IIS (later): the stdout log or the event log of the app pool (to be settled in the ASF-production work).
|
||||
- Sentry: issues filtered on `environment:<test|production>`.
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Monitoring setup - identity-email-notifications
|
||||
|
||||
## Done in code (this feature)
|
||||
- The log level follows the kind of failure: Error for configuration problems (`auth_failed`, `rejected_sender`, `tls_failed`, `not_configured`), Warning for the rest, so Sentry alerts only when somebody has to act. Tested (`SmtpSenderDiagnosticsTests`: wrong login and refused sender are Error events, refused recipient and timeouts stay Warning; level mapping for every code; no secret in any line).
|
||||
- A start-up line states whether mail is on, and an Error names every missing or invalid setting when it is not.
|
||||
|
||||
## To be done by a person (the assistant does not change shared Sentry projects)
|
||||
1. In the Sentry project of ASF (environment `test`) and of SlpSoftware (environments `test`, later `production`): create the issue-alert rule of the monitoring plan (message contains `identity_mail_failed` or `mail_configuration_invalid`, notify the owner by e-mail). The `sentry:sentry-create-alert` skill can create it on request, with Bryan's go.
|
||||
2. Check once per host after switching mail on: the journal shows `mail_status enabled=true`.
|
||||
3. Trigger the alert once on test (for example a wrong password in `shared/env` for a minute) to see that the e-mail arrives; restore the password.
|
||||
|
||||
## Runbook lines (Q5 A)
|
||||
After any change of mail settings: restart, read the journal for `mail_status enabled=true` or `mail_configuration_invalid`, check `/System/capabilities` for `emailEnabled`.
|
||||
+8
-8
@@ -9,21 +9,21 @@ How do the mail settings reach each host?
|
||||
- B) The non-secret settings (`App__PublicBaseUrl`, `Email__FromName`, ...) come from Gitea variables written by the deploy workflow; only the SMTP credentials are set by hand
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A (Bryan: by hand in shared/env)
|
||||
|
||||
## Question 2: Order of the roll-out
|
||||
- A) ASF test (done) then SlpSoftware test, each with its own SMTP host, then production of each site later, one at a time, each needing its own approval
|
||||
- B) ASF test, then straight to both productions together after the real test mails succeed
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A, BUT: ASF has no production environment yet and it will be an IIS server instead of a Raspberry Pi, so ASF production is a separate, later piece of work (not in this feature). Order now: ASF test, SlpSoftware test, then SlpSoftware production with its own approval.
|
||||
|
||||
## Question 3: The sender account for ASF
|
||||
- A) A dedicated mailbox/account on `mail.andreaslop.nl` (for example `no-reply@andreaslop.nl`) created by Bryan for this purpose; its address goes into `Email__FromAddress` and `Email__AllowedFromAddresses__0`
|
||||
- B) An existing mailbox of the site (for example `info@andreaslop.nl`), accepting that replies reach that mailbox
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A (no-reply@andreaslop.nl on mail.andreaslop.nl)
|
||||
|
||||
## Question 4: Which mail problems should raise an alert
|
||||
Today a failed mail is logged at Warning level (journal only; Sentry sees Error events and uses Warnings only as breadcrumbs), and a bad mail configuration at start-up is an Error (Sentry sees it). What do you want?
|
||||
@@ -33,14 +33,14 @@ Today a failed mail is logged at Warning level (journal only; Sentry sees Error
|
||||
- C) B, plus an alert when the same host fails at least three mails in fifteen minutes whatever the reason (a Sentry rule on the events)
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: B (configuration-related mail failures auth_failed, rejected_sender, tls_failed, not_configured at Error level so Sentry alerts; timeouts and recipient problems stay Warning; no threshold rule)
|
||||
|
||||
## Question 5: Start-up check after a release
|
||||
- A) The deploy's existing `/health` check stays as it is; after changing mail settings the person doing it reads the journal for `mail_status enabled=true` or `mail_configuration_invalid` (written in the runbook)
|
||||
- B) A) plus the CMS reports `emailEnabled` on `/health` details so a pinger can see that mail is off on a host that should have it
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A (runbook line plus the journal)
|
||||
|
||||
## Question 6: Deliverability (SPF, DKIM, DMARC) before production
|
||||
Facts today: both `slpsoftware.nl` and `andreaslop.nl` have SPF (`a mx include:smtp.spf.ziggo.nl`), a `dkim` record and DMARC `p=none`.
|
||||
@@ -48,7 +48,7 @@ Facts today: both `slpsoftware.nl` and `andreaslop.nl` have SPF (`a mx include:s
|
||||
- B) A), and tighten DMARC to `p=quarantine` for the site's domain before production
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A (test mails to Gmail and Outlook, DMARC stays p=none for now)
|
||||
|
||||
## Question 7: Rollback
|
||||
Rolling a release back keeps the three additive database changes (older code simply ignores the new columns and table).
|
||||
@@ -56,10 +56,10 @@ Rolling a release back keeps the three additive database changes (older code sim
|
||||
- B) A), and additionally a documented way to drop the new columns/table (only if a restore is ever needed)
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A (mail off by removing Email__Smtp__Host and restarting; release rollback through the previous-release switch)
|
||||
|
||||
## Question 8: Who approves production readiness for this feature
|
||||
- A) Bryan, after the real test mails on ASF test and SlpSoftware test and the answers above are carried out
|
||||
- X) Other (please describe after [Answer]: tag below)
|
||||
|
||||
[Answer]:
|
||||
[Answer]: A (Bryan approves after the real test mails on ASF test and SlpSoftware test)
|
||||
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
# Production Readiness Checklist - identity-email-notifications
|
||||
|
||||
Scope: ASF test and SlpSoftware test now; SlpSoftware production after Bryan's approval. ASF production is outside this feature (no environment yet; it will be an IIS server).
|
||||
|
||||
## Deployment
|
||||
- **Status**: Configured (documented, no workflow change)
|
||||
- **Method**: existing CI/CD for the code; mail settings by hand in each host's environment file (Q1 A)
|
||||
- **Rollback Plan**: Yes (`deployment/rollback-plan.md`: remove the SMTP host and restart; previous-release switch; additive migrations)
|
||||
|
||||
## Monitoring
|
||||
- **Status**: Configured in code (log levels), alert rule documented and still to be created in Sentry by a person
|
||||
- **Approach**: journal plus Sentry Error events for configuration-related mail failures; no threshold, no dashboard (Q4 B)
|
||||
|
||||
## Additional Readiness Items
|
||||
- **Deliverability (Q6 A)**, before SlpSoftware production: send test mails to a Gmail and an Outlook mailbox and check in "show original" that SPF, DKIM and DMARC pass and are ALIGNED with the From domain (the From domain, the SPF-authorised server and the DKIM signing domain must match); DMARC stays `p=none` for now. Facts at 2026-11-08: both `slpsoftware.nl` and `andreaslop.nl` publish `v=spf1 a mx include:smtp.spf.ziggo.nl ~all`, a `dkim._domainkey` record and DMARC `p=none`. For ASF the sender `no-reply@andreaslop.nl` through `mail.andreaslop.nl` is expected to align (the host is the domain's own `a`/`mx`); to be proven by the real test mail.
|
||||
- **Real test mails (Q8 A)**: one per site on test (invitation, new link by mail, password reset, a notice) to an address of Bryan; the approval of this feature depends on them.
|
||||
- **Secrets Management**: Pass by design. SMTP credentials only in each host's environment file (mode 600), never in the repository, a chat or the logs (tested: no credential, address, token or link in any log line).
|
||||
- **Backups**: the new table `SiteSettings` and the two new columns are in the existing databases and so in the existing backups. Nothing new to back up.
|
||||
- **Runbook/Support Handover**: Yes: `deployment/deployment-instructions.md` (turning mail on, reading the journal), `deployment/rollback-plan.md`, `monitoring/monitoring-setup.md`, the infrastructure design (variables, SMTP modes, how to read a failure).
|
||||
- **Alert Ownership**: Bryan (Sentry e-mail as project owner); fixing with the CMS & API and Pi-Main sessions.
|
||||
|
||||
## .NET Appsettings Compliance
|
||||
- **Three-file pattern present**: Yes (unchanged; the mail section was added to the committed `appsettings.json` of the ASF and SlpSoftware hosts).
|
||||
- **No committed secrets**: Pass. Smtp host, username and password are empty in the committed files (a test asserts it for the ASF host); the committed sender default is a placeholder-style address, not a credential.
|
||||
- **`appsettings.local.json` gitignored**: Yes.
|
||||
- **Production secrets wired to env vars/secrets manager**: Yes: `Email__Smtp__Username/Password` in the host's environment file (Pi). For IIS see the open point.
|
||||
|
||||
## Open Points
|
||||
1. **ASF production on IIS** (decided 2026-11-09): not worked out here. The Pi-specific assumptions (systemd unit, `shared/env`, journalctl, scp deploy, reverse-proxy Pi) and what changes on IIS are listed in `deployment/deployment-plan.md`; a later ASF-production feature must settle them (environment variables and secrets, restart, logs, deploy path, hosting model, database and mail-server reachability).
|
||||
2. Sentry alert rule to be created (monitoring setup).
|
||||
3. Real test mails on ASF test and SlpSoftware test.
|
||||
4. Merge to master / SlpSoftware production: not approved yet (the branch is already on ASF test and SlpSoftware test).
|
||||
5. Per-site DMARC tightening later (stays `p=none` now).
|
||||
|
||||
## Overall Readiness
|
||||
- **Ready for Production**: Not yet. SlpSoftware production needs the real test mails, the alert rule and Bryan's approval; ASF production is a separate piece of work.
|
||||
Reference in New Issue
Block a user