diff --git a/aidlc-docs/features/prerendering/construction/hosting-and-targets/infrastructure-design/deployment-architecture.md b/aidlc-docs/features/prerendering/construction/hosting-and-targets/infrastructure-design/deployment-architecture.md new file mode 100644 index 0000000..2a6ed16 --- /dev/null +++ b/aidlc-docs/features/prerendering/construction/hosting-and-targets/infrastructure-design/deployment-architecture.md @@ -0,0 +1,180 @@ +# Deployment Architecture: Unit 3 "hosting-and-targets" + +Rules and rationale: [infrastructure-design.md](infrastructure-design.md) (`BR-HST-xx`). Requirements: FR-PR-14 to 18, 23; NFR-PR-P-01, P-02, S-01. Based on the frontend-shell [deployment plan](../../../../frontend-shell/operations/deployment/deployment-plan.md) and [instructions](../../../../frontend-shell/operations/deployment/deployment-instructions.md); only differences are described here. + +## 1. Topology + +```mermaid +graph LR + Visitor["Visitor or Lighthouse"] + Dev["Developer PC"] + Gitea["Gitea Actions on the Pi runner"] + Artifact["Build artifact: dist/client"] + Proxy["Proxy Pi: nginx and certbot, TLS"] + HostPi["Hosting Pi: nginx test block and WordPress"] + Webhost["Webhost: IIS with web.config"] + Preview["Preview server on dist/client"] + + Dev -->|"push or merge"| Gitea + Gitea --> Artifact + Artifact -->|"scp, test only"| HostPi + Artifact -->|"FTPS, release branch only"| Webhost + Artifact --> Preview + Visitor -->|"test.andreaslop.nl"| Proxy + Proxy -->|"HTTP, Host header kept"| HostPi + Visitor -->|"andreaslop.nl after DNS switch"| Webhost + Preview -->|"Lighthouse and browser tests"| Gitea + + classDef client fill:#f6e05e,stroke:#c05621,stroke-width:1px,color:#000; + classDef gateway fill:#63b3ed,stroke:#2b6cb0,stroke-width:1px,color:#000; + classDef service fill:#9ae6b4,stroke:#2f855a,stroke-width:1px,color:#000; + classDef database fill:#fbd38d,stroke:#92400e,stroke-width:1px,color:#000; + classDef external fill:#e9d5ff,stroke:#6b21a8,stroke-width:1px,color:#000; + class Visitor,Dev client; + class Proxy gateway; + class Gitea,Preview service; + class Artifact database; + class HostPi,Webhost external; +``` + +Text alternative: the developer pushes to Gitea, which builds one artifact; test receives it over scp on the hosting Pi behind the proxy Pi, production receives it over FTPS on the IIS webhost, and the preview server serves the same files to Lighthouse and the browser tests. + +| Component | Environment | Role in this unit | Managed by | +|---|---|---|---| +| Proxy Pi nginx (`test.andreaslop.nl.conf`, certbot) | Test | TLS and pass-through; no URL logic; optional `proxy_http_version 1.1` | Human (H4) | +| Hosting Pi nginx (`andreaslop-test.conf`) | Test | All URL, 404, header and compression rules | Generated, installed by hand (H2) | +| Webhost IIS (`web.config`) | Production | Same rules, uploaded with every deploy | Generated, automatic | +| Preview server (`previewServer.ts`) | CI and local | Same contract for Lighthouse and browser tests | Code | +| `verify-deployment.sh` | CI | Post-deploy proof against the build | Code | +| `iis-probe/` | Webhost, once | Finds locked sections without risking the site | Generated, uploaded by hand (H5) | + +## 2. Upload order, production (BR-HST-14, 15) + +```mermaid +graph TD + Art["Artifact dist with index.html at its root"] --> Guard["Guard: required files present, no server folder, no source maps"] + Guard --> Cfg["Copy web.config variant into dist"] + Cfg --> P1["Pass 1: all files except html and web.config, including /assets"] + P1 --> P2a["Pass 2a: web.config"] + P2a --> P2b["Pass 2b: all html including nested pages and 404.html"] + P2b --> P3["Pass 3: mirror with delete removes stale files"] + P3 --> Chk["verify-deployment.sh against the live host"] + classDef service fill:#9ae6b4,stroke:#2f855a,stroke-width:1px,color:#000; + classDef gateway fill:#63b3ed,stroke:#2b6cb0,stroke-width:1px,color:#000; + classDef database fill:#fbd38d,stroke:#92400e,stroke-width:1px,color:#000; + class Art,Cfg database; + class Guard,Chk gateway; + class P1,P2a,P2b,P3 service; +``` + +Text alternative: after a guard and the `web.config` copy, assets go up first, then `web.config`, then all HTML, then a delete pass, and finally the post-deploy script checks the live host. + +Why this order: new hashed assets are harmless next to the old release, so they go first. New HTML refers to them only after they exist. The `web.config` goes just before the HTML because its rules (`/x` to `x.html`) fit the new files, and the old SPA index is replaced seconds later. The delete pass runs last, so old hashed assets stay available for visitors who still hold the old HTML until the new HTML is complete. The first deploy happens before DNS moves, so the one-time mismatch (new rules, old files) has no visitors. + +Sketch of the lftp section (final syntax proven in V10): + +```text +mirror --reverse --verbose --exclude-glob '*.html' --exclude-glob web.config dist/ $FTP_DIR/; +put -O $FTP_DIR dist/web.config; +mirror --reverse --verbose --include-glob '*.html' dist/ $FTP_DIR/; +mirror --reverse --delete --verbose dist/ $FTP_DIR/; +``` + +`--exclude-glob index.html` is gone. The test deploy is unchanged except for the added guard and the new post-deploy script (BR-HST-16). + +## 3. Test deploy and nginx rollout + +```mermaid +graph TD + Cur["Today: old SPA build and old nginx config"] --> Bak["H2: back up and install the new nginx file, nginx -t, reload"] + Bak --> Gap["Test serves the old build: only / and assets work, deep links 404"] + Gap --> Dep["H3: run the workflow on the feature branch with Deploy to Test"] + Dep --> Ver["verify-deployment.sh in the job, then the matrix of section 5 from the PC"] + Ver -->|"all as expected"| Mrg["Merge the pull request: master deploys the same build"] + Ver -->|"mismatch"| Rb["Rollback: restore the backup file, reload, redeploy the last good commit"] + Mrg --> Done["Test and master agree"] + classDef client fill:#f6e05e,stroke:#c05621,stroke-width:1px,color:#000; + classDef service fill:#9ae6b4,stroke:#2f855a,stroke-width:1px,color:#000; + classDef gateway fill:#63b3ed,stroke:#2b6cb0,stroke-width:1px,color:#000; + classDef database fill:#fbd38d,stroke:#92400e,stroke-width:1px,color:#000; + class Cur,Gap database; + class Bak,Dep,Rb client; + class Ver,Mrg,Done service; +``` + +Text alternative: install the new nginx file by hand, immediately deploy the feature branch to test, verify with the matrix, and only then merge; on a mismatch restore the backed-up nginx file and redeploy the last good commit. + +Timing relative to the merge: + +- The new nginx file must be active before the first deploy of the new build. A merge deploys automatically, so the nginx update comes before the merge. With the old file and the new build, `/over-mij` would show the home page (old `try_files ... /index.html`); the new post-deploy script fails that job (BR-HST-18). +- With the new file and the old build, deep links give 404 for as long as the old build stays. That gap is kept short by running H3 right after H2 and is acceptable on a noindex test site. +- Order of actions that touch test: H2 (nginx), H3 (branch deploy), matrix, merge. Do not merge between H2 and H3. + +## 4. CI changes (code generation) + +| Job | Change | +|---|---| +| `check` | Unchanged; `pnpm test` runs the new unit and property tests (P-HST-01 to 08, 10, 11) | +| `lighthouse` | Installs nginx (`sudo apt-get install -y nginx`), runs `pnpm e2e` including `serverRules.e2e.ts` (P-HST-09), starts the preview with `--audit-route` for Lighthouse | +| `deploy-test` | Artifact guard, then the existing empty-and-copy, then `verify-deployment.sh "$HEALTH_URL" dist --test` (still skipped when `HEALTH_CHECK_URL_TEST` is empty; keep it set) | +| `deploy-production` | Guard, variant copy of `web.config`, three-pass upload, `verify-deployment.sh "$HEALTH_URL" dist` (compression only warns) | + +Repository variable added: `IIS_WEB_CONFIG_VARIANT` (default `full`; `no-httperrors` or `no-compression` per BR-HST-20). + +## 5. Verification matrix, test (`https://test.andreaslop.nl`) + +Run from the PC after H3. `B=https://test.andreaslop.nl`. + +```bash +for p in / /over-mij /soorten-fotoshoots /soorten-fotoshoots/newborn /contact '/over-mij?pakket=gezin' \ + /over-mij/ /soorten-fotoshoots/ /over-mij.html /index.html /index /Over-Mij /Over-Mij/ '/Over-Mij?x=1' \ + /404 /404.html /onbekend /onbekend/ /assets /assets/ /assets/missing.js /robots.txt /sitemap.xml; do + printf '%-30s ' "$p"; curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "$B$p" +done +curl -s -D - -o /dev/null -H 'Accept-Encoding: gzip' "$B/assets/$(curl -s $B/ | grep -o 'index-[A-Za-z0-9_-]*\.js' | head -1)" | grep -iE '^(HTTP|content-encoding|vary|cache-control)' +curl -s -D - -o /dev/null "$B/onbekend" | grep -iE '^(HTTP|content-security-policy|strict-transport|x-content-type|x-frame|referrer|x-robots)' +``` + +| Path | Expected | +|---|---| +| `/`, `/over-mij`, `/soorten-fotoshoots`, `/soorten-fotoshoots/newborn`, `/contact`, `/over-mij?pakket=gezin` | `200` | +| `/over-mij/`, `/over-mij.html`, `/Over-Mij`, `/Over-Mij/` | `301` to `/over-mij` | +| `/soorten-fotoshoots/` | `301` to `/soorten-fotoshoots` | +| `/Over-Mij?x=1` | `301` to `/over-mij?x=1` | +| `/index.html`, `/index` | `301` to `/` | +| `/404`, `/404.html`, `/onbekend`, `/assets`, `/assets/missing.js` | `404` | +| `/onbekend/`, `/assets/` | `301` to the path without slash, then `404` | +| `/robots.txt` | `200`, `Disallow: /`; `/sitemap.xml` `200` | +| JavaScript with `Accept-Encoding: gzip` | `Content-Encoding: gzip`, `Vary: Accept-Encoding`, `Cache-Control: public, max-age=31536000, immutable` | +| The 404 response | all five security headers and `X-Robots-Tag: noindex, nofollow` | + +Also open `/Over-Mij?pakket=gezin` and `/soorten-fotoshoots/` in Chrome: the address bar ends on the canonical URL, the page title is the page's own, no console error. + +## 6. Verification matrix, production rehearsal (H5, H6) + +Probe kit, per folder (`P=https://andreaslop.nl`, host via `hosts` entry; the kit README holds the exact commands): + +| Folder | Request | Expected | If not | +|---|---|---|---| +| `_probe-rewrite` | `/_probe-rewrite/a` (a.html next to directory `a/`) | `200` and the body of `a.html` | 500: rewrite locked or module missing; 301 to `a/`: rule order wrong | +| `_probe-rewrite` | `/_probe-rewrite/a/`, `/_probe-rewrite/A`, `/_probe-rewrite/dir`, `/_probe-rewrite/dir/` | `301` to `a`, `301` to `a`, `404`, `301` then `404` | A loop (too many redirects) means rule 8 is missing or not honoured | +| `_probe-errors` | `/_probe-errors/nope` | `404` with the custom page body | 500: `httpErrors` locked (F1); `404` with IIS's own page: `existingResponse` ignored | +| `_probe-errors` | `/_probe-errors/gone.html` (blocked by CustomResponse, also the ExecuteURL target) | `404` with the custom page body | Empty body or loop: F2 | +| `_probe-headers` | `/_probe-headers/x.txt`, a `301` | all probe headers on both; `Cache-Control` as set; `.woff2` MIME | 500: `customHeaders` or `staticContent` locked | +| `_probe-compress` | each probe file, 5 requests in a row, `Accept-Encoding: gzip` | `Content-Encoding: gzip` for HTML, JS, CSS, SVG, JSON, XML | 500: `urlCompression` locked (F3); 200 without encoding: ask the host (Q4) | + +Full deploy rehearsal (`P=https://andreaslop.nl` via `hosts`): the matrix of section 5 with these differences: no `X-Robots-Tag`; `/robots.txt` and `/sitemap.xml` are the build's files; `/` also tested over `http://` (301 to https) and `www.` (one 301 to `https://andreaslop.nl/`, path kept); compression after repeated requests; `/404` and `/404.html` per decision O4. + +## 7. Rollback summary + +| Situation | Action | +|---|---| +| New nginx file rejected (`nginx -t`) | Copy the `.bak-` file back, `nginx -t`, stop | +| New nginx file accepted but the matrix fails | Restore backup, reload, redeploy the last good commit to test (H2/H3 rollback) | +| Bad `web.config` live on production | Upload the previous `web.config` or a variant by FTP, or redeploy the previous release (rollback-plan.md); before DNS no visitor is affected | +| Locked section found | Set `IIS_WEB_CONFIG_VARIANT` and redeploy | +| Post-deploy script red on test after a merge | The old nginx file is probably active: re-run H2, then run the workflow on `master` with "Deploy to Test" | + +## 8. Monitoring + +No new monitoring. `verify-deployment.sh` is the deploy-time check; the UptimeRobot deep link `/portfolio` (monitoring-setup) now relies on the page file rule, so it also detects a broken rewrite after go-live. diff --git a/aidlc-docs/features/prerendering/construction/hosting-and-targets/infrastructure-design/infrastructure-design.md b/aidlc-docs/features/prerendering/construction/hosting-and-targets/infrastructure-design/infrastructure-design.md new file mode 100644 index 0000000..a0cab13 --- /dev/null +++ b/aidlc-docs/features/prerendering/construction/hosting-and-targets/infrastructure-design/infrastructure-design.md @@ -0,0 +1,457 @@ +# Infrastructure Design: Unit 3 "hosting-and-targets" + +Feature: prerendering. Requirements: FR-PR-14 to 18, 23; NFR-PR-P-01, P-02, S-01 ([requirements.md](../../../inception/requirements/requirements.md)). Companion: [deployment-architecture.md](deployment-architecture.md). Plan: [hosting-and-targets-infrastructure-design-plan.md](../../plans/hosting-and-targets-infrastructure-design-plan.md). + +Unit 3 skips Functional Design ([execution-plan.md](../../../inception/plans/execution-plan.md)), so its rules and testable properties (PBT-01) are recorded here. Rule ids are `BR-HST-xx`. Nothing in this document is implemented yet; no application or configuration file was changed. + +## 1. Summary of decisions + +| # | Decision | Rules | +|---|---|---| +| 1 | One URL contract for nginx, IIS and the preview server, generated from one page list (`sitePages`) | BR-HST-01 to 08 | +| 2 | Unknown URLs get `404.html` with a real 404; the SPA fallback is removed everywhere | BR-HST-02 | +| 3 | Trailing slash, `.html`, `/index` and letter case are canonicalised with one 301 for known pages; the client router stays unchanged | BR-HST-03, 04, 08 | +| 4 | `/404` and `/404.html` answer with status 404 on every server; Lighthouse audits the 404 page through an opt-in preview route, not through a 200 at `/404.html` | BR-HST-05, 22 | +| 5 | Directories next to page files never redirect or list (`try_files` without `$uri/`; IIS rules that pre-empt the directory redirect) | BR-HST-07 | +| 6 | nginx compresses itself (own `gzip` block, valid behind a proxy that talks HTTP/1.0). IIS compression can only be switched on per site, the rest belongs to the webhost | BR-HST-09 to 11 | +| 7 | IIS risk is limited by a probe kit (one `web.config` per section in sub folders), two committed fallback variants and a rehearsal before DNS | BR-HST-19, 20 | +| 8 | Upload order: assets, `web.config`, all HTML, cleanup. nginx on the Pi is updated by hand before the merge and proven with a branch deploy | BR-HST-14 to 18 | +| 9 | `lighthouserc.cjs` gets a data-driven exception list, empty by default; no target changes without a recorded owner decision | BR-HST-21, 23 | +| 10 | The nginx rules are proven with a real nginx process in CI (apt, same 1.24 as the Pi); IIS rules are checked by a rule simulator | BR-HST-24 | + +## 2. Findings that shape the design + +### 2.1 What is live today (read-only `curl` from this PC, 2026-11-01) + +| Check | Result | Meaning | +|---|---|---| +| `test.andreaslop.nl` and `andreaslop.nl` | Both resolve to `217.121.250.215` | One public address, one reverse proxy | +| `https://andreaslop.nl/` | `Server: nginx/1.24.0`, `Link: .../wp-json/` | Production name still serves WordPress on the Pi. The IIS host is not live; DNS has not moved (matches D12 of frontend-shell). Whether the hosting package exists is not visible from here | +| `https://test.andreaslop.nl/over-mij` | 200, 3368 bytes, the old SPA `index.html` | Test still has the old build and the old nginx config (SPA fallback) | +| `https://test.andreaslop.nl/` with `Accept-Encoding: gzip` | 200, `Content-Length: 3368`, no `Content-Encoding`, strong `ETag`, no `Vary` | Not compressed | +| `/assets/index-DE7NmAmk.js` | 536,695 bytes, `application/javascript`, no `Content-Encoding` | Not compressed | +| WordPress on `andreaslop.nl`: HTML (chunked) and `wp-content/.../*.min.css` (17,798 bytes) | no `Content-Encoding` | Same chain, same result: not specific to the test block | +| `http://192.168.1.103/` (hosting Pi) | Timeout | Private LAN address; not reachable from here | + +### 2.2 Compression cause (FR-PR-23) + +Established: nothing on the path compresses, for the test site and for WordPress alike. The repo's `andreaslop-test.conf` has no `gzip` directives, so the test block depends on the Pi's global `nginx.conf`. The repo's proxy config has no `proxy_http_version`, so nginx talks HTTP/1.0 to the hosting Pi by default, and nginx only compresses requests of version 1.1 or higher (`gzip_http_version` defaults to 1.1). + +Not established (needs the Pi, see Human task H1): which of these is the cause. + +| Candidate | Fits the evidence | Fix | +|---|---|---| +| A. HTTP/1.0 from the proxy meets `gzip_http_version 1.1` on the hosting Pi | HTML is not compressed although Ubuntu's stock `nginx.conf` ships `gzip on;` (stock `gzip_types` covers `text/html` only, so JavaScript would stay plain anyway). This is the most likely single cause of the HTML result | `gzip_http_version 1.0` in the generated test block (self-sufficient); optionally `proxy_http_version 1.1` on the proxy | +| B. `gzip` is off, or has no JS/CSS/SVG types, in the hosting Pi's global config | Explains the JavaScript result in any case | Own `gzip` block in the generated test block (BR-HST-09) | +| C. The proxy strips or empties `Accept-Encoding` | Would hide everything; the repo's proxy config does not do this | Check with H1; no change if absent | +| D. The proxy re-writes or decompresses the response | Would need `gunzip on` or Lua; not in the repo | Check with H1 | + +The generated test block fixes A and B together without a global setting, so the repo change is safe whichever candidate H1 confirms. H1 documents the cause for the record and rules out C and D, which the repo cannot fix. + +### 2.3 Spike: letter case, trailing slash and the router (BR-404-07) + +Spike (scratch, not in the repo): a case-sensitive static server over the real `dist/client`, Chrome (system, playwright-core), requests of `/over-mij` variants. Mode A is today's rule (unknown path gives `404.html` with 404). Mode B adds a 301 to the canonical form for known pages. + +| Request | Mode A (today) | Mode B (redirect) | +|---|---|---| +| `/over-mij` | 200, 1 title, 0 robots | same | +| `/Over-Mij`, `/Over-Mij?x=1` | 404 body, router renders "Over mij": 2 titles, 1 `noindex` robots meta | `301 /Over-Mij` then `200 /over-mij` (query kept): 1 title, 0 robots | +| `/over-mij/`, `/over-mij/?x=1` | 404 body, same mismatch (router tidies the address bar to `/over-mij`) | `301` then `200 /over-mij` (query kept), clean | +| `/soorten-fotoshoots/` | 404 body, 2 titles | `301` then `200 /soorten-fotoshoots`, clean | +| `/onbekend` | 404, 1 title, 1 robots (correct) | same | + +No console errors or React #418 appeared in this run (the race described in the verification log); the stale second title and `noindex` meta appear in mode A, which is the SEO consequence of BR-404-07. + +Why the router is not the fix: TanStack Router has a global `caseSensitive` option (`router.d.ts`) but the servers disagree. nginx is case-sensitive, IIS is not. With `caseSensitive: true`, nginx is consistent (404 body, router renders the catch-all) but IIS serves `over-mij.html` for `/Over-Mij` and the router renders the catch-all over it: a new mismatch. With the default (insensitive), IIS is consistent and nginx is not. No router setting suits both, so the servers must agree among themselves: redirect to the canonical form. The router then never sees a non-canonical known URL, and its settings stay as they are (default `trailingSlash: 'never'`, case-insensitive). Whether `caseSensitive: true` would additionally be good defence was not run in Chrome (it needs a rebuild); it is not adopted because of the IIS mismatch above. + +## 3. URL contract + +Applies to nginx (test), IIS (production) and the preview server (Lighthouse, browser tests). "Page" means a path in `sitePages` (11 pages: `/` and 10 others). `pageFile(path)` is `x.html`; `/` is `index.html`. + +| Rule | Statement | Source | +|---|---|---| +| BR-HST-01 | `/` serves `index.html`, status 200. `/x` (last segment without a dot) serves `x.html` with 200 when it exists, without a redirect, also when a directory `x/` exists next to `x.html`. A file request (last segment with a dot) serves that file with 200. `?query` is ignored for lookup | FR-PR-16 | +| BR-HST-02 | Anything that does not exist serves `404.html` with a real 404. There is no SPA fallback on any server (`try_files ... /index.html` and the IIS rule "SPA fallback" are removed). Responses to missing `/assets/*` also carry the 404 page | FR-PR-14; D5 | +| BR-HST-03 | A path of more than one character that ends in one or more `/` gets one 301 to the path without trailing slashes. The query string is kept byte for byte. `/` stays. Applies to every path, known or not (`/onbekend/` goes to `/onbekend`, which is a 404) | FR-PR-16 | +| BR-HST-04 | For a known page, every variant is redirected with one 301 to its canonical path (the `sitePages` key): case variants, with or without `.html`, with or without trailing slashes, in any combination (`/Over-Mij/` goes straight to `/over-mij`). `/index`, `/index.html` and their variants go to `/`. Unknown paths are never redirected for case (`/Onbekend` is a 404). Query kept | FR-PR-16 | +| BR-HST-05 | `/404` and `/404.html` answer with status 404 and the 404 page (not 200, not a redirect), on every server. The page stays `noindex` and out of the sitemap | FR-PR-14; Application Design, Consequences | +| BR-HST-06 | A redirect `Location` is built from the constant page list or from the request path minus trailing slashes, never starts with `//` or `/\`, and contains no control characters. On nginx it is never built from a decoded `$uri` in `return` (CRLF injection) | SECURITY-11 misuse case | +| BR-HST-07 | Directory trap. `soorten-fotoshoots.html` and the directory `soorten-fotoshoots/` coexist in `dist/client`. nginx: `try_files` never contains `$uri/` and the config has no `index` and no `autoindex on`. IIS: the page rule rewrites before the handler can issue its own directory redirect, and a directory-only name (`/assets`) is answered with a 404 by an explicit rule, so IIS cannot redirect it back to `/assets/` and loop against BR-HST-03. No server ever lists a directory | SECURITY-09 | +| BR-HST-08 | Server and router agree by canonicalisation (BR-HST-03, 04), not by a router setting. After a redirect the URL is lowercase, slash-free and `.html`-free, so the first browser render matches the HTML. The router configuration is unchanged | BR-404-07 | + +Empty segments inside a path (`//x`) stay undefined: nginx merges the slashes and serves the page, the preview server answers 404 (BR-SRV-07), IIS depends on HTTP.sys. This cannot produce a mismatch (the router matches the same page) and is outside the parity set; the redirect rules never produce a `//` target (BR-HST-06). + +### 3.1 Result table + +| Request | Result on all three servers | +|---|---| +| `/`, `/over-mij`, `/soorten-fotoshoots`, `/soorten-fotoshoots/newborn`, any of them with `?query` | 200, the page file | +| `/over-mij/`, `/over-mij///`, `/soorten-fotoshoots/` (+ `?x=1`) | 301 to `/over-mij`, `/over-mij`, `/soorten-fotoshoots` (+ `?x=1`) | +| `/over-mij.html`, `/soorten-fotoshoots/newborn.html` | 301 to `/over-mij`, `/soorten-fotoshoots/newborn` | +| `/index`, `/index.html`, `/index/` | 301 to `/` | +| `/Over-Mij`, `/OVER-MIJ.HTML`, `/Over-Mij/` | 301 to `/over-mij` | +| `/404`, `/404.html` | 404, the 404 page | +| `/onbekend`, `/Onbekend`, `/x.html` (unknown), `/assets/missing.js` | 404, the 404 page | +| `/onbekend/` | 301 to `/onbekend`, then 404 | +| `/assets`, `/assets/` | 404 (the second via one 301 to `/assets`); no listing | +| `/assets/index-HASH.js`, `/robots.txt`, `/sitemap.xml`, `/favicon.png` | 200, the file | +| Other methods than GET/HEAD | Preview: 405. nginx: its own 405. IIS: static handler default | + +Test host exception (unchanged): `/robots.txt` returns `Disallow: /` and every response carries `X-Robots-Tag: noindex, nofollow`. + +### 3.2 Decision flow + +```mermaid +graph TD + Req["Request path"] --> Dec["Decode once, reject NUL, backslash and dot segments"] + Dec -->|"bad"| NF["404 with 404.html"] + Dec --> Canon{"Known page in a non-canonical form?"} + Canon -->|"yes"| R1["301 to the canonical page, query kept"] + Canon -->|"no"| Slash{"Trailing slash on a path longer than one char?"} + Slash -->|"yes"| R2["301 to the path without slashes, query kept"] + Slash -->|"no"| Direct{"Is it /404 or /404.html?"} + Direct -->|"yes"| NF + Direct -->|"no"| Root{"Is it the root?"} + Root -->|"yes"| Idx["200 index.html"] + Root -->|"no"| Dot{"Last segment has a dot?"} + Dot -->|"yes"| F1{"File exists?"} + Dot -->|"no"| F2{"Page file x.html exists?"} + F1 -->|"yes"| OK["200 the file"] + F1 -->|"no"| NF + F2 -->|"yes"| OK + F2 -->|"no"| NF + classDef client fill:#f6e05e,stroke:#c05621,stroke-width:1px,color:#000; + classDef gateway fill:#63b3ed,stroke:#2b6cb0,stroke-width:1px,color:#000; + classDef service fill:#9ae6b4,stroke:#2f855a,stroke-width:1px,color:#000; + classDef database fill:#fbd38d,stroke:#92400e,stroke-width:1px,color:#000; + class Req client; + class Dec,Canon,Slash,Direct,Root,Dot,F1,F2 gateway; + class Idx,OK service; + class R1,R2,NF database; +``` + +Text alternative: a request is decoded and checked, then known pages in a non-canonical form and trailing slashes get one 301, `/404` and `/404.html` get the 404 page, and everything else is served as a file or a page file (200) or as the 404 page (404). + +## 4. Server rules + +### 4.1 Page list as the single source + +The generator and the preview server take the page list from `Object.keys(sitePages)` through one pure module (proposed `config/canonicalUrls.ts`). It exports the canonical-redirect function used by the preview server and the data (page list, `NOT_FOUND_PATH`) from which the nginx `map` and the IIS rule are generated. A new page appears in all three at once; the build already fails if a page has no HTML file (FR-PR-03). A test fails if `dist/client` holds an HTML file other than `index.html`, `404.html` and the page files, because such a file would be served by `try_files` without being canonicalised. + +### 4.2 nginx (generated `andreaslop-test.conf`) + +Normative sketch; the generator is the source of the final text. Constraints: everything lives inside the file (no change to the Pi's global `nginx.conf`), the file is included inside `http {}` (as now, through `conf.d`), variable names carry the prefix `andreaslop_` to avoid clashes with the WordPress configuration. + +```nginx +map $uri $andreaslop_canonical { + default ""; + # canonical forms: exact entries win over the regexes below + /over-mij ""; # one line per page except / + # variants: case-insensitive, optional .html, optional trailing slashes + ~*^/index(\.html)?/*$ /; + ~*^/over-mij(\.html)?/*$ /over-mij; # one line per page except / +} + +server { + listen 80; + server_name test.andreaslop.nl; + root /var/www/html/andreaslop-test; + absolute_redirect off; + autoindex off; # no index directive on purpose + + + gzip on; + gzip_vary on; + gzip_proxied any; + gzip_http_version 1.0; + gzip_comp_level 5; + gzip_min_length 256; + gzip_types text/css text/plain text/xml application/xml application/javascript + text/javascript application/json image/svg+xml; # text/html is implicit + + if ($andreaslop_canonical) { return 301 $andreaslop_canonical$is_args$args; } + rewrite ^/(.+?)/+$ /$1 permanent; # BR-HST-03, safe escaping, query kept + + error_page 404 /404.html; + + location = /robots.txt { default_type text/plain; return 200 "User-agent: *\nDisallow: /\n"; } + location = /404 { return 404; } + location = /404.html { internal; } + location /assets/ { try_files $uri =404; } + location = / { try_files /index.html =404; } + location / { try_files $uri $uri.html =404; } +} +``` + +Reasons for the non-obvious choices: + +- `try_files $uri $uri.html =404`: `$uri` without a trailing slash only matches files, so a directory `soorten-fotoshoots/` is skipped and `soorten-fotoshoots.html` is served (BR-HST-07). `$uri/` or an `index` directive would trigger nginx's own directory redirect. `index` is also dangerous here: it makes an internal redirect to `/index.html`, which BR-HST-04 redirects to `/`, a loop. Hence `location = /` with `try_files /index.html =404`. +- Redirect targets: the map values are constants. The generic slash rule uses `rewrite ... permanent`, which escapes the target; `return 301 $uri` or `$1` taken from `$uri` would allow header injection (BR-HST-06). +- `error_page 404 /404.html` keeps the status 404 (no `=`). `location = /404.html` is `internal`: a direct request is answered 404, which `error_page` turns into the 404 page with status 404 (BR-HST-05). `location = /404` returns 404 the same way; without it `$uri.html` would serve the page with 200. +- Security headers are repeated in every location (as now) and at server level; `always` makes them apply to 301, 404 and the 404 page (SECURITY-04). Whether the server-level `if` redirect carries them must be proven with a real nginx (BR-HST-24). +- `gzip_http_version 1.0` makes compression work however the proxy talks to this block. `gzip_proxied any` covers requests that carry `Via`. `gzip_types` lists MIME types, so it includes both the types nginx's `mime.types` gives (`application/javascript`, `text/xml`) and the ones the preview server uses. It is generated from the same list as the preview server's `COMPRESSIBLE` set (BR-HST-11). +- `if` is used only with `return` at server level, the one safe form. + +Caching (nginx): `/assets/*` `public, max-age=31536000, immutable`; everything else `no-cache` (HTML, `robots.txt`, `sitemap.xml`, `favicon.png`, logos). See decision O3 for why unhashed files are not long-cached. + +Proxy config (`test.andreaslop.nl.proxy.conf`, on the other Pi): not required to change for compression. Optional hygiene: add `proxy_http_version 1.1;` and `proxy_set_header Connection "";` inside `location /`. Because certbot edits this file, the change is made by hand in place (H4), never by copying the repo file over it. The generated file may carry the two lines for new installs. + +### 4.3 IIS (generated `web.config`) + +Rule order inside `` (all `stopProcessing="true"`). The two existing rules stay first; the "SPA fallback" rule is removed. + +| # | Rule | Match (url relative to the site root, ignore case) | Conditions | Action | +|---|---|---|---|---| +| 1 | `www naar andreaslop.nl` | unchanged | unchanged | unchanged | +| 2 | `HTTPS` | unchanged | unchanged | unchanged | +| 3 | `Index naar root` | `^index(?:\.html)?/*$` | none | Redirect `/`, permanent | +| 4 | `Pagina canoniek` | `^(?:pages joined by \|)(?:\.html)?/*$` | `{R:0}` does not match `^(?:pages)$` with ignoreCase false | Redirect `/{ToLower:{R:1}}`, permanent | +| 5 | `Slash weg` | `^([^/\\].*?)/+$` | none | Redirect `/{R:1}`, permanent | +| 6 | `404 direct` | `^404(?:\.html)?$` | none | CustomResponse 404 | +| 7 | `Pagina bestand` | `^(.+)$` | `{REQUEST_FILENAME}.html` IsFile | Rewrite `{R:1}.html` | +| 8 | `Map zonder pagina` | `^(.+)$` | `{REQUEST_FILENAME}` IsDirectory | CustomResponse 404 | + +Notes: + +- Rule 5 excludes a leading `/` or `\` in the captured part so that `//evil.example/` cannot become the target `//evil.example` (BR-HST-06). `ToLower` applies only to the alternation of known pages, so hashed asset names with upper-case letters are never touched. +- Rules 7 and 8 are the directory trap (BR-HST-07). URL Rewrite runs before the handler stage. Rule 7 rewrites `/soorten-fotoshoots` to the file although a directory of that name exists. Rule 8 catches directory-only names such as `/assets`; without it the handler would redirect `/assets` to `/assets/` and rule 5 would redirect back (loop). The condition of rule 7 does not use `IsDirectory negate` (the old SPA rule did). +- `/` is served by the default document `index.html` as today; a `^$` rewrite to `/index.html` is added only if the rehearsal shows the default document is not applied (V6 in section 12). +- Custom 404 page: `` with `` after a ``. `ExecuteURL` takes a site-relative URL, so `allowAbsolutePathsWhenDelegated` (which concerns physical paths in `File` mode) is not needed. Rule 6 together with `ExecuteURL` is the least certain part: if URL Rewrite also evaluates the executed child request `/404.html`, rule 6 would answer it with an empty 404. The probe kit tests exactly this (section 7); fallback F2. +- Headers (`customHeaders`) and `clientCache` stay as now: `DisableCache` site-wide, `UseMaxAge 365.00:00:00` for `/assets` (BR-HST-12). +- Compression: section 5. + +## 5. Compression (FR-PR-23) + +| Rule | Statement | +|---|---| +| BR-HST-09 | nginx: the generated test block compresses HTML, JavaScript, CSS, SVG, JSON and XML (and plain text) with the `gzip` block of section 4.2, valid behind a proxy that uses HTTP/1.0, with `Vary: Accept-Encoding`. Fonts, images and other binary files are not listed. Nothing relies on the Pi's global settings | +| BR-HST-10 | IIS: `web.config` sets `` and pins MIME types so the server's default compressible set matches (`.js` as `text/javascript`, `.css`, `.svg` as `image/svg+xml`, `.json`, `.xml` as `text/xml`, `.html`). `httpCompression` (which types, the compression level, the cache directory) cannot be set in `web.config`; it belongs to `applicationHost.config`, so the webhost owns it. The result is only proven by `curl` in the rehearsal (V6); the owner asks the webhost (section 8, Q6). A site without compression still works | +| BR-HST-11 | One list: `{extension, MIME types}` for the compressible types lives in one module. The preview server's `COMPRESSIBLE` extension set, the nginx `gzip_types`, and the IIS MIME pins are all derived from it, and a test checks that they agree | + +IIS behaviour to know: static compression starts only after a file has been requested a few times in a short time (`frequentHitThreshold`), so the first requests can be plain; and it is skipped for requests that arrive through a proxy (`noCompressionForProxies`). Rehearsal must therefore request each file several times. + +The preview server keeps its gzip (BR-SRV-08). Lighthouse therefore measures compressed transfer sizes on the preview, which is also what the test and production hosts will now deliver; before this unit, test did not. + +## 6. Caching and headers + +| Rule | Statement | +|---|---| +| BR-HST-12 | `/assets/*` (hashed names): `public, max-age=31536000, immutable`. Everything else, including HTML, `robots.txt`, `sitemap.xml`, `favicon.png` and unhashed images: `no-cache`. 301 responses carry no extra cache header (browsers may keep a 301; acceptable because the redirect set is fixed by the page list) | +| BR-HST-13 | The five security headers of `securityHeaders('production')` are present on every response: 200, 301, 404, the 404 page, 405 and error responses. Test adds `X-Robots-Tag`. nginx uses `add_header ... always` in every location and at server level; IIS uses `customHeaders`; the preview server merges them in `send()` | + +Your brief lists `robots.txt`, `sitemap.xml` and the favicon as long-cached. They are not hashed, so a long lifetime would keep a stale `robots.txt` after go-live. They revalidate instead (a 304 is cheap with `ETag`). Lighthouse's cache audit is informational and does not weigh in the score. See decision O3. + +## 7. IIS unknowns and risk limiting + +### 7.1 What may be locked or missing at the webhost + +The package is shared hosting ("ASP.NET Core 10", Windows/IIS, URL Rewrite per the hoster's page) reached over FTPS. Production has never been deployed (production-readiness-checklist: "Never run yet"), so none of the `web.config` sections below has been proven on that host, including the ones frontend-shell already uses. + +| Section / feature | Used for | Can it be locked or missing? | If it is | +|---|---|---|---| +| `system.webServer/rewrite/rules` (URL Rewrite 2.x) | all URL rules | The hoster names URL Rewrite, version unknown. `{ToLower:}` and `CustomResponse` need 2.0+ | Module missing: `web.config` is rejected (500.19) for the whole site | +| `system.webServer/httpErrors` (`existingResponse`, `responseMode`, `errorMode`) | the 404 page with status 404 | Default delegation is usually Allow, but hosts lock it. `allowAbsolutePathsWhenDelegated` matters only for `File` mode, which this design avoids | 500.19 for the whole site | +| `system.webServer/httpProtocol/customHeaders` | security headers (SECURITY-04) | Normally allowed | 500.19 for the whole site | +| `system.webServer/staticContent` (`mimeMap`, `clientCache`) | caching, MIME pins | Normally allowed | 500.19 for the whole site | +| `system.webServer/urlCompression` | enabling compression | Usually allowed; the static compression module must be installed | 500.19 if locked; no effect if the module is missing | +| `system.webServer/httpCompression` | which types and level | Server level only (`applicationHost.config`). Not used in `web.config` | Using it would give 500.19 for the whole site | +| Site root | where `web.config` takes effect | The FTP folder `wwwroot` may be the app's static web root below the real IIS site root, in which case a `web.config` placed there does not apply to `/` | Rules silently absent. Unknown until probed | + +What happens when a locked setting is used: IIS cannot read the configuration and answers every request under that folder with HTTP 500 (500.19), static files included, until a valid `web.config` is uploaded again. A remote visitor sees a generic 500 page, not the detail (which is also what we want for SECURITY-09). + +### 7.2 Risk limiting + +| Rule | Statement | +|---|---| +| BR-HST-19 | Probe kit. A generated, committed set of tiny sites under `web/deploy/iis-probe/` (not part of `dist/`, never uploaded by the pipeline), one folder per feature group, each with its own `web.config` and a few files: `_probe-rewrite` (page mapping, slash, case, directory next to file), `_probe-errors` (`httpErrors` ExecuteURL plus a CustomResponse rule on the executed page), `_probe-headers` (customHeaders, mimeMap, clientCache), `_probe-compress` (`urlCompression` and 100 KB files of every compressible type). A locked section breaks only its folder, so the owner learns which section fails without risking the site. The first real deploy (`mirror --delete`) removes the probe folders. The probe folders use the same rule text as the real `web.config` (generated from the same functions), so a pass means the real file will pass | +| BR-HST-20 | Fallback variants. The generator also writes `deploy/iis/variants/web.no-httperrors.config` (F1) and `web.no-compression.config` (F3), tested like the main file. The production job copies `deploy/iis/${IIS_WEB_CONFIG_VARIANT:-full}` as `web.config`, so switching variant is a repository variable change followed by a deploy, not a code change | + +Fallbacks, in the order they would be used: + +| Id | When | Effect | +|---|---|---| +| F1 `no-httperrors` | `httpErrors` locked | No custom 404 page. IIS answers unknown URLs with its generic 404 (status 404, no internal detail) and rules 6 and 8 answer with an empty 404 CustomResponse. FR-PR-14 is not met on production (owner decision O5); the status code stays correct | +| F2 drop rule 6 | ExecuteURL re-triggers rule 6 (empty 404 body) | `/404.html` serves the 404 page with 200 (it is `noindex` and unlinked). `/404` is excluded from rule 7 by a pattern instead. Documented parity exception (owner decision O4) | +| F3 `no-compression` | `urlCompression` locked | Site works uncompressed; ask the webhost to enable compression server-side | + +Staged rollout (also in [deployment-architecture.md](deployment-architecture.md)): A probe kit on the unlaunched site root (no DNS dependency), B full production deploy from a `release/*` branch and the curl matrix against the new host with a `hosts` entry (the existing dress rehearsal, now with the new matrix), C DNS switch, D every later change of `web.config` may use only sections proven in A and B. + +Rollback if a bad `web.config` is live: upload the previous `web.config` (or a variant) by FTP, or redeploy the previous release (rollback-plan.md). Before the DNS switch no visitor is affected. + +## 8. What the owner must ask or try at the webhost + +Ask (support ticket, one message): + +1. Q1. What is the physical path of the IIS site: the FTP root, or the FTP folder `wwwroot`? If an ASP.NET Core module (`aspNetCore` handler) is configured at the site root, may it be removed, since this site is static? +2. Q2. Is URL Rewrite Module 2.1 or newer installed and delegated (`system.webServer/rewrite/rules`) at site level? +3. Q3. Are `system.webServer/httpErrors`, `httpProtocol/customHeaders` and `staticContent` delegated (Read/Write) to `web.config` at site level? Is a host-wide custom error page configured that would override ours? +4. Q4. Is the static compression module installed, is `urlCompression` delegated, and which MIME types are in the server's `httpCompression/staticTypes` (we need HTML, JavaScript, CSS, SVG, JSON, XML)? If not, can they add them? +5. Q5. Is `IIS 10` (or later) used, and may directory browsing stay off? +6. Q6. Does the package exist and is a temporary URL or the A-record IP available for a rehearsal before DNS changes? + +Try (no webhost help needed): H5 below (probe kit). + +## 9. Deployment rules + +| Rule | Statement | +|---|---| +| BR-HST-14 | `deploy-production` uploads in three passes: (1) every file except `*.html` and `web.config`; (2) `web.config` and then all HTML files, including `404.html` and nested ones; (3) `mirror --delete`. The `--exclude-glob index.html` option disappears. A visitor can reach HTML only after every asset it refers to exists (FR-PR-18). The exact lftp exclusion syntax (glob against the relative path, or a regex) is proven locally with a dry run against a `file:` target (section 12, V10) | +| BR-HST-15 | `web.config` is copied from `deploy/iis/${IIS_WEB_CONFIG_VARIANT:-full}/web.config` to `dist/web.config`, next to `index.html` (the artifact root is `dist/`; `server/` is not in it). Before uploading, the job fails if `dist/index.html`, `dist/404.html` or `dist/over-mij.html` is missing, if `dist/server` exists, or if any `*.map` file is present (FR-PR-17, NFR-PR-SEC-03). Q1 decides whether `FTP_REMOTE_DIR` is `wwwroot` or the root | +| BR-HST-16 | `deploy-test` keeps emptying the folder and copying `dist/*`. This is acceptable for a noindex test site: the gap is a few seconds of nginx's plain 404, no old build is kept, and the WordPress guard stays. The new `dist` contains `404.html` and nested folders; `scp -r` handles them | +| BR-HST-17 | The nginx update on the Pi is a human task, done before the merge and proven by a branch deploy (H2, H3, deployment-architecture.md). A merge to `master` deploys automatically; with the old nginx config the new build would serve the home page for `/over-mij` (the old `try_files ... /index.html`). The post-deploy check (BR-HST-18) turns that case into a red job | +| BR-HST-18 | One shared post-deploy script (`web/deploy/verify-deployment.sh [--test]`) used by both jobs. It fails the job unless: `/`, `/over-mij` and `/soorten-fotoshoots/newborn` return bytes identical to the matching files in `dist` (without `Accept-Encoding`, so an old nginx that serves the home page for `/over-mij` is caught); an unknown URL returns 404 with the bytes of `404.html`; `/over-mij/` returns 301 to `/over-mij` and `/Over-Mij` to `/over-mij`; all five security headers are on the 200, the 301 and the 404; `--test` adds `X-Robots-Tag`. A gzip request for the JavaScript bundle must return `Content-Encoding: gzip` on test; on production it only warns (webhost) | + +## 10. Lighthouse targets (NFR-PR-P-01, P-02, S-01) + +| Rule | Statement | +|---|---| +| BR-HST-21 | Target state of `lighthouserc.cjs`: mobile performance 0.9 for every page; accessibility, best practices and SEO stay at 1 (S-01); desktop stays 0.9 (P-02, `lighthouserc.desktop.cjs` unchanged). The constants `homeMobilePerformance`, `photoPageMobilePerformance` and `photoPagePaths` are replaced by one exception table that is empty unless an owner decision names a page | +| BR-HST-22 | The 404 page is audited through an opt-in route. The preview server gets a flag (`--audit-route`) that only the Lighthouse start command passes; with it, `/__audit/404` serves `404.html` with status 200. Without the flag (every other use: `pnpm preview`, browser tests, parity tests) `/404.html` and `/404` are real 404s (BR-HST-05). `lighthouserc.cjs` uses `notFoundPath = '/__audit/404'` and the guard test in `app.test.tsx` follows. This replaces the unit 1 fallback (200 at `/404.html` on the preview only). Reason: Lighthouse refuses a main document with an error status (observed in unit 1), and the `ignoreStatusCode` setting (exists in the installed Lighthouse 12.6.1) would still log Chrome's console error for a 404 document, which probably fails the best-practices score of 1 on the 404 page; that was not measured here (Lighthouse was out of scope). V9 measures it; if it passes, `ignoreStatusCode: true` is the simpler alternative (decision O2) | +| BR-HST-23 | Exceptions are data. Sketch: `const mobileExceptions = { }` with entries `{ '/some-page': { minScore: 0.85, decision: 'owner, 2026-11-xx, measurement-final.md row N' } }`. A unit test enforces: every exception has a non-empty `decision`, `minScore < 0.9`, names a page that exists, and no page matches two matrix entries. The matrix builder creates one entry for all pages without exception and one per page with an exception. `numberOfRuns` for the blocking mobile run: see O6 | + +How the final measurement feeds the decision: + +1. After code generation of this unit, Build and Test runs the same method as `measurement-display.md` (3 runs per URL, median, same PC, `previewServer.ts`, configs unedited) into `construction/hosting-and-targets/code/measurement-final.md`, one row per page with master, unit 1, unit 2 and final scores. +2. Pages at 0.90 or higher on mobile need no exception. The exceptions in today's file disappear for them. +3. Each page below 0.90 goes to the owner with its score, LCP element and the options: keep an exception at a stated floor, work on it, or accept a red `pnpm check`. Nothing is lowered or left in place without that answer (NFR-PR-P-01). +4. Starting point from the unit 2 measurement (mobile, median of 3): `/` 0.85 (LCP element is the heading, 4356 ms, so it is not photo-bound; the trace V-D10 is still open), gezin 0.89, loveshoot 0.88; zwangerschap and portret exactly 0.90 (inside run-to-run noise, so a single run can flip them). All other pages are 0.91 or higher. Desktop 0.99 or better on all URLs. + +## 11. Testable properties (PBT-01) + +Generators: known pages x variants (case permutation, `.html`, 0 to 3 trailing slashes, query), unknown segments from a safe alphabet, and hostile strings (`%2e%2e`, `//host`, `\`, NUL, CR/LF, very long). + +| Id | Property | Component | Style | +|---|---|---|---| +| P-HST-01 | `resolveRequest` is total: never throws; status is 200, 301 or 404; `file` never contains `..` and is never absolute | `previewServer.ts` | Invariant | +| P-HST-02 | Every variant of a known page resolves to a 301 whose target is the canonical path, or (if canonical) to 200 with `pageFile` | `canonicalUrls.ts`, preview | Invariant | +| P-HST-03 | Termination: following redirects from any input reaches a non-redirect in at most 2 hops; a canonical target is a fixpoint | same | Idempotence | +| P-HST-04 | A redirect target starts with `/`, not with `//` or `/\`, has no control characters, and (for known pages) is one of the constants | same | Invariant (SECURITY-11) | +| P-HST-05 | The query string of the request is in the `Location` byte for byte | same | Invariant | +| P-HST-06 | Unknown, non-redirected paths give 404 with `404.html`; `/404` and `/404.html` give 404 only; `/` never redirects | same | Invariant | +| P-HST-07 | Generation: nginx map entries, IIS rule alternation and the page list are the same set; every nginx location (and the server level) contains all five headers; no `try_files` contains `$uri/`; no `index`, no `autoindex on`; no `return` or `rewrite` target built from `$uri` except `rewrite ... permanent`; `gzip_types` covers the required type list; IIS output is well-formed XML with the rule order of section 4.3 and no SPA fallback | `serverConfig.ts` | Invariant over the generated text | +| P-HST-08 | Generated files in `deploy/` equal the generator output, including the variants | `serverConfig.ts` | Existing round trip | +| P-HST-09 | Parity oracle: for generated paths, a real nginx loaded with the generated block and the preview server agree on status, `Location`, and which file is served (body hash against `dist/client`); both carry the security headers; gzip request for compressible types gives `Content-Encoding: gzip` on nginx and the preview | nginx vs preview | Oracle / differential | +| P-HST-10 | IIS simulator: the redirect rules (3, 4, 5) and the 404 rules (6, 8), parsed from the generated `web.config` and evaluated with JavaScript regexes (`{R:n}`, `{ToLower:}`, `IsFile` from a file list), agree with `resolveRequest` on the same generated paths. Limits: no real URL Rewrite, no handler stage, no ExecuteURL | `serverConfig.ts` | Oracle / differential | +| P-HST-11 | Lighthouse config: no exception without `decision`, no exception at 0.9 or above, every URL matches exactly one matrix entry, `notFoundPath` equals the preview audit route | `lighthouserc.cjs` | Invariant | + +Example-based tests keep the rows of section 3.1 as a table test, including `/soorten-fotoshoots` next to a directory. + +### 11.1 Proving the nginx rules without a production server + +| Option | Pros | Cons | +|---|---|---| +| 1. Real nginx from apt in the CI job, run unprivileged with a temporary prefix and `-c` (`sudo apt-get install -y nginx`, already the pattern for `sshpass` and `lftp`) | Same nginx 1.24 as the Pi (`Server: nginx/1.24.0`) on the Ubuntu 24.04 runner image; no nested containers (container actions do not work on Podman); seconds to run | Needs `sudo` on the runner (already used); on the PC runner image it depends on the same apt access | +| 2. `podman run nginx:1.24` in CI | Isolated | Podman inside the Podman runner job is not available; image pull each run | +| 3. Local only (owner's Podman) | No CI change | Podman machine is stopped by default; not enforced | +| 4. `nginx -t` only | Trivial | Proves syntax, not behaviour | + +**BR-HST-24**: the nginx rules are proven by a real nginx process, not by reading. Recommendation: option 1, plus a local run through option 3 when `nginx` is not on the path (skipped with a visible message only when `CI` is not set; in CI a missing nginx fails the test). The test lives in `e2e/` (`serverRules.e2e.ts`), starts nginx on a free port with a `nginxTestConf({ listen, root })` generated from the same function (parameters default to the production values, so the committed file is unchanged), serves a copy of `dist/client`, and also runs a real HTTP/1.0 request to prove `gzip_http_version`. IIS can only be read: P-HST-10 plus the rehearsal. + +## 12. To verify in code generation + +| Id | Check | +|---|---| +| V1 | `generated nginx` loads in real nginx 1.24 (`nginx -t`) as an `http`-level include without any other directive from the Pi; `map` before `server` works in `conf.d` | +| V2 | Server-level `if ... return 301` and `rewrite ... permanent` responses carry all five headers and `Cache-Control` with real nginx; `absolute_redirect off` gives a relative `Location`; `/` works without an `index` directive | +| V3 | `/soorten-fotoshoots` (file) and `/soorten-fotoshoots/` (directory) behave as in section 3.1 on nginx and preview; `/assets` and `/assets/` as well | +| V4 | `/404` and `/404.html` give 404 with the page and the five headers on nginx; `error_page` keeps the status; HEAD works | +| V5 | A HTTP/1.0 request with `Accept-Encoding: gzip` to the generated block gets `Content-Encoding: gzip` for `.html`, `.js`, `.css`, `.svg`, `.json`, `.xml` (all sizes above 256 bytes) and not for `.woff2`/images | +| V6 | IIS (rehearsal only): the 404 rule plus ExecuteURL, `/Over-Mij`, `/over-mij/`, `/assets`, `/soorten-fotoshoots`, headers on 301 and 404, compression after repeated requests, default document for `/`; recorded in `verification-log.md` | +| V7 | `previewServer.ts`: the new `resolveRequest` contract (redirect result, query handling, `--audit-route`); unit 1 tests for BR-SRV-07 are updated (the "interim" rules become final); `pageFiles.ts` stays the single path to file mapping | +| V8 | The 301 is followed by Chrome with the query string intact for `?pakket=` and `?shoot=` links, and the hydrated page has 1 title (extend the existing "URL die de server niet kent en de router wel" e2e) | +| V9 | One Lighthouse run of the 404 page on the preview audit route, and (as comparison) `/404.html` with `ignoreStatusCode`; record best-practices for both, then keep the simpler passing option (O2) | +| V10 | lftp: dry run against a `file:` target proves that pass 1 skips all nested `*.html` and `web.config`, pass 2 uploads them, pass 3 deletes only stale files | +| V11 | `verify-deployment.sh` fails against a copy of the old nginx config (home page for `/over-mij`) and passes against the new one | +| V12 | CI: the `lighthouse` job (or a new `server-rules` job) installs nginx, `pnpm e2e` includes `serverRules.e2e.ts`, and the job fails if nginx is missing | +| V13 | `lighthouserc.cjs`: the exception table is empty on delivery unless an owner decision exists; P-HST-11 passes; `pnpm lighthouse` still starts the preview with `--audit-route` and `Local` appears for `startServerReadyPattern` | +| V14 | The new nginx file works for a first-time install and for an update of an existing file (comment text in the header explains which) | + +## 13. Human tasks + +Environment: PC (repo checkout on the feature branch), hosting Pi (`192.168.1.103`, nginx for test and WordPress), proxy Pi (public `217.121.250.215`), webhost control panel. Replace `PI_USER`, `PI_HOST` with the values of `PI_MAIN_*`. + +**H1. Diagnose compression on the Pi (read-only, any time, no repo change needed).** + +On the hosting Pi: + +```bash +sudo nginx -T 2>/dev/null | grep -nE '^\s*(gzip|gzip_[a-z_]+)\s' +nginx -v +curl -s -o /dev/null -D - -H 'Host: test.andreaslop.nl' -H 'Accept-Encoding: gzip' http://127.0.0.1/ | grep -iE '^(HTTP|content-encoding|vary)' +curl -s -o /dev/null -D - --http1.0 -H 'Host: test.andreaslop.nl' -H 'Accept-Encoding: gzip' http://127.0.0.1/ | grep -iE '^(HTTP|content-encoding|vary)' +``` + +On the proxy Pi: + +```bash +sudo nginx -T 2>/dev/null | grep -nE 'proxy_http_version|Accept-Encoding|gunzip|gzip|proxy_intercept_errors|error_page|proxy_pass' +curl -s -o /dev/null -D - -H 'Host: test.andreaslop.nl' -H 'Accept-Encoding: gzip' http://192.168.1.103/ | grep -iE '^(HTTP|content-encoding|vary)' +``` + +Read: `Content-Encoding: gzip` on the 1.1 request but not on the 1.0 request means candidate A (proxy HTTP/1.0). No `gzip` lines or no gzip on either means B. `Accept-Encoding` or `gunzip` in the proxy output means C or D. Send the output back; the design does not depend on the answer, but the cause is then recorded. + +**H2. Update nginx on the hosting Pi (before the merge).** + +```bash +# on the PC, from the repo checkout on the feature branch +scp web/deploy/nginx/andreaslop-test.conf PI_USER@PI_HOST:/tmp/andreaslop-test.conf + +# on the hosting Pi +sudo cp /etc/nginx/conf.d/andreaslop-test.conf /etc/nginx/conf.d/andreaslop-test.conf.bak-$(date +%Y%m%d%H%M) +sudo install -m 644 -o root -g root /tmp/andreaslop-test.conf /etc/nginx/conf.d/andreaslop-test.conf +sudo nginx -t +``` + +If `nginx -t` fails: `sudo cp` the `.bak-...` file back over `andreaslop-test.conf`, run `sudo nginx -t`, and stop. If it passes: `sudo systemctl reload nginx`. The backup ends in `.bak-`, so the `conf.d/*.conf` include ignores it. Do not touch the WordPress block. + +**H3. Prove it with a branch deploy, then merge.** In Gitea: Actions, Continuous Integration, Run workflow, branch `feature/prerendering`, tick "Deploy to Test". When the job is green (it now runs `verify-deployment.sh`), run the matrix of deployment-architecture.md section 5 from the PC. Only then merge the pull request. The merge deploys the same build again. + +**H4. Optional, proxy Pi.** Only if H1 shows candidate A and you want it fixed at the source: edit `/etc/nginx/conf.d/test.andreaslop.nl.conf` in place (certbot owns it), add inside `location /`: `proxy_http_version 1.1;` and `proxy_set_header Connection "";`, then `sudo nginx -t && sudo systemctl reload nginx`. Not required for compression. + +**Rollback of H2/H3.** On the hosting Pi: `sudo cp /etc/nginx/conf.d/andreaslop-test.conf.bak- /etc/nginx/conf.d/andreaslop-test.conf && sudo nginx -t && sudo systemctl reload nginx`. Then redeploy the previous build: create a branch at the last good commit (`git switch -c rollback/test c4c2f83 && git push -u origin rollback/test`) and run the workflow on it with "Deploy to Test". Test is noindex and has no visitors, so a few minutes of 404 is acceptable. + +**H5. IIS probe kit (owner, once the hosting package exists; no DNS change).** + +1. Run `pnpm server-config` on the feature branch (code generation adds the probe kit); upload `web/deploy/iis-probe/*` with FileZilla ("FTP over explicit TLS") into the site root (the folder Q1 identifies). +2. Add a `hosts` entry (` andreaslop.nl`) or use the hoster's temporary URL. +3. Run the commands in `web/deploy/iis-probe/README` (one `curl` per folder; code generation writes it from the matrix in deployment-architecture.md section 6) and send the result. A 500 for a folder means that folder's section is locked: tell me which folder; I switch the variant (BR-HST-20). +4. Send the answers to Q1 to Q6 of section 8. +5. Remove the `hosts` entry afterwards. The probe folders disappear with the first real deploy. + +**H6. Production rehearsal (before DNS).** As in deployment-instructions.md "Go-live", step 1, with the matrix of deployment-architecture.md section 6 instead of the old four checks. Set `IIS_WEB_CONFIG_VARIANT` only if H5 requires a fallback. + +## 14. Decisions for the owner + +| Id | Question | Recommendation | Alternatives | +|---|---|---|---| +| O1 | How do server and router agree on letter case and trailing slash? | Server-side 301 to the canonical form on all three servers; router unchanged | Router `caseSensitive: true` (fixes nginx, breaks IIS); both (extra code, no gain); serve non-canonical URLs as 404 (impossible on case-insensitive IIS) | +| O2 | How does Lighthouse audit the 404 page while `/404.html` is a real 404? | Opt-in preview route `/__audit/404` (certain to work, default preview stays faithful) | `ignoreStatusCode: true` (no preview code; likely fails best practices because of Chrome's console error, measured in V9); keep unit 1's 200 at `/404.html` (preview unfaithful) | +| O3 | Cache lifetime of unhashed files (`robots.txt`, `sitemap.xml`, favicon, logos, OG images) | `no-cache` (revalidate); only `/assets` long-cached | Long cache for favicon, logos and OG images only (one day), at the cost of per-file IIS `location` blocks | +| O4 | `/404.html` on IIS if URL Rewrite plus ExecuteURL misbehaves | Try the 404 rule; if the probe shows an empty body, drop the rule (F2) and accept 200 with `noindex` | Block it another way (none known); accept an empty 404 | +| O5 | If the webhost locks `httpErrors` | Ask the host to unlock it first; otherwise F1 (generic IIS 404 page, correct status) | Soft 404 (rewrite to `404.html` with 200): wrong status; change host | +| O6 | `numberOfRuns` for the blocking mobile Lighthouse run | 3 (median; about 4 more minutes in `pnpm check`), because two pages sit exactly at 0.90 | Keep 1 (noise can fail a page that passes) | +| O7 | Which pages keep a mobile exception | Decided only after the final measurement (section 10); none pre-decided | Keep today's 0.8/0.85 values (not recommended: five of seven photo pages already pass) | +| O8 | Where the nginx rules are proven | Real nginx from apt in CI (option 1) | Podman container (not feasible in the Podman runner), local only | +| O9 | `pnpm check` when nginx is missing locally | Skip with a visible message locally, fail in CI | Require nginx everywhere | + +## 15. Compliance notes + +**Security Baseline** + +| Rule | Status | How | +|---|---|---| +| SECURITY-04 | Compliant by design, proven in V2 and V4 | The five headers on every response: 200, 301, 404, the 404 page. nginx: `add_header ... always` in every location and at server level; IIS: `customHeaders`; preview: `send()`. P-HST-07 and P-HST-09 test it. The CSP is unchanged | +| SECURITY-09 | Compliant by design | No directory listing: nginx has no `autoindex on` and no `index`; IIS has `directoryBrowse` off (default) and rule 8; BR-HST-07 covers the file-next-to-directory case. Error pages are the static 404 page or the server's generic error; no stack traces, paths or versions are added (`server_tokens` is a Pi-level setting and outside this file; the `Server: nginx/1.24.0` header is visible today) | +| SECURITY-11 | Compliant | Misuse cases: path traversal (BR-SRV-06, P-HST-01), open redirect through `//host` or `/\host` (BR-HST-06, P-HST-04), CRLF injection through decoded `$uri` in a redirect (BR-HST-06, P-HST-07) | +| SECURITY-10 | Compliant | New dependency: none for the tests (nginx is a CI tool installed from the distribution; the probe kit and simulator are plain code) | +| SECURITY-13 | Compliant | No external scripts; the 404 audit route is opt-in and absent from production hosts | +| SECURITY-15 | Compliant | The new post-deploy script fails closed; the artifact guard fails the upload on unexpected content | +| Others | N/A | No data stores, APIs, accounts or logging in this unit | + +**Property-Based Testing** + +| Rule | Status | Note | +|---|---|---| +| PBT-01 | Compliant | P-HST-01 to 11 above | +| PBT-09 | Compliant | fast-check is already a dependency; the oracle tests use it with a bounded number of runs because they make real HTTP requests | +| PBT-02 to 08, 10 | N/A at this stage | Enforced in Code Generation and Build and Test: seed logging, shrinking and example-based companions as in the existing suite | diff --git a/aidlc-docs/features/prerendering/construction/plans/hosting-and-targets-infrastructure-design-plan.md b/aidlc-docs/features/prerendering/construction/plans/hosting-and-targets-infrastructure-design-plan.md new file mode 100644 index 0000000..baceeea --- /dev/null +++ b/aidlc-docs/features/prerendering/construction/plans/hosting-and-targets-infrastructure-design-plan.md @@ -0,0 +1,40 @@ +# Infrastructure Design Plan - Unit 3: Hosting and targets + +Feature: prerendering. Unit: 3 "hosting-and-targets" (see [execution-plan.md](../../inception/plans/execution-plan.md) § Units of Work). Builds on unit 1 ([code summary](../start-migration/code/code-summary.md)) and unit 2 ([code summary](../display-fallbacks/code/code-summary.md), [measurement](../display-fallbacks/code/measurement-display.md)). Artifacts: [infrastructure-design/](../hosting-and-targets/infrastructure-design/). + +No questions were needed. The brief fixed the scope and the open points; the choices that belong to the owner are written as recommendations with alternatives in `infrastructure-design.md` section 14 (O1 to O9). + +Functional Design is skipped for this unit (execution plan); its rules (`BR-HST-xx`) and testable properties (PBT-01) are recorded in `infrastructure-design.md`. NFR Design is skipped; the logical components are the generated server configuration, the preview server, the CI jobs and the Lighthouse configuration. + +## Category coverage + +| Category | Applicability | Where | +|---|---|---| +| Deployment environment | Applies: test nginx on two Pis, production IIS on a shared webhost, preview server | infrastructure-design §2, §7; deployment-architecture §1 | +| Compute infrastructure | Applies: the web servers' rules; no sizing or scaling for a static site | infrastructure-design §4 | +| Storage infrastructure | N/A: static files only, no database or uploads | - | +| Messaging infrastructure | N/A: no queues or events | - | +| Networking infrastructure | Applies: reverse proxy Pi, HTTP/1.0 upstream and compression, redirects behind the proxy, FTPS | infrastructure-design §2.2, §4.2; deployment-architecture §1, §3 | +| Monitoring infrastructure | Applies narrowly: post-deploy script; no new tooling | deployment-architecture §8 | +| Shared infrastructure | Applies: the hosting Pi also runs WordPress; the global `nginx.conf` is not assumed | infrastructure-design §4.2, §13 | + +## Steps + +- [x] Step 1: Read the artifact rules (infrastructure-design, mermaid standards, content validation) and the requirements (FR-PR-14 to 18, 23; NFR-PR-P-01, P-02, S-01), execution plan and application design +- [x] Step 2: Read unit 1 (BR-404-07, BR-SRV, code summary, verification log with the `/Over-Mij` and `/over-mij/` observations) and unit 2 (code summary, measurement) +- [x] Step 3: Read `serverConfig.ts` and its test, `previewServer.ts`, `securityHeaders.ts`, `pageFiles.ts`, `web/deploy/`, `lighthouserc.cjs`, `lighthouserc.desktop.cjs`, the CI workflow and the frontend-shell operations documents +- [x] Step 4: Check the live test and production hostnames with read-only `curl` (compression, server, status); the hosting Pi address is not reachable from this PC +- [x] Step 5: Spike in Chrome: case-sensitive server over the real `dist/client`, today's rule against a canonical-redirect rule for `/Over-Mij`, `/over-mij/`, `/soorten-fotoshoots/`, query kept +- [x] Step 6: Check the installed router (`caseSensitive`, `trailingSlash`) and Lighthouse 12.6.1 (`ignoreStatusCode`, `errors-in-console`) in `node_modules`, read only +- [x] Step 7: Design the URL contract (BR-HST-01 to 08), the nginx and IIS rules, the directory trap and the redirect safety rules +- [x] Step 8: Settle the compression cause as far as the evidence allows, design nginx and IIS compression (BR-HST-09 to 11) and write the Pi diagnosis commands +- [x] Step 9: Design IIS risk limiting: lock table, probe kit, fallback variants, rehearsal stages, questions for the webhost (BR-HST-19, 20; infrastructure-design §7, §8) +- [x] Step 10: Design the deployment: upload order, `web.config` placement, guard, `deploy-test`, post-deploy script, nginx rollout order and rollback (BR-HST-14 to 18; deployment-architecture) +- [x] Step 11: Design the Lighthouse targets, the 404 audit route and the owner-decision feed (BR-HST-21 to 23) +- [x] Step 12: Record testable properties (P-HST-01 to 11) and the recommendation for proving nginx rules without a production server (BR-HST-24) +- [x] Step 13: Write `infrastructure-design.md` including Human tasks, To verify in code generation, owner decisions and compliance notes +- [x] Step 14: Write `deployment-architecture.md` with topology, upload order, nginx rollout, verification matrices and rollback +- [x] Step 15: Check diagrams against the Mermaid standards and the content validation rules (Mermaid only, ids without hyphens, quoted labels, classDef with fill, stroke and `color:#000`, one text alternative after each diagram) +- [x] Step 16: Create this plan and report to the orchestrator + +Not changed: application code, configuration, `audit.md`, `aidlc-state.md`, `requirements.md`; nothing installed, deployed or committed; no Lighthouse run; no login anywhere.