Designs where the deploy workflow puts things on the Pi

Infrastructure design for U6: single Pi split by directory not port, systemd --user services instead of sudo, a releases/current/shared layout that keeps the customer website outside every atomic switch, and a health check through the existing reverse proxy before pruning anything. Flags that user-level systemd needs enable-linger or the service dies the moment the deploy SSH session closes.
2026-07-28 15:32:51 +02:00
parent 247d8be241
commit 9a77eec0f1
3 changed files with 541 additions and 0 deletions
@@ -0,0 +1,219 @@
 # Infrastructure Design Plan — U6 Deploy Workflow
 ## Prerequisite note (deliberate deviation)
 Functional Design and NFR Design are **skipped** for U6 per the execution plan (`unit-of-work.md`,
 `aidlc-state.md`) — U6 is a Pipeline-type unit (YAML, no C# logic), and its behavioural contract is
 already fully specified by FR-02, FR-03, FR-04, FR-06, FR-08, FR-20, ASM-01, ASM-03, ASM-04, D-02,
 D-05, D-09, D-26, D-27 and the U5↔U6 shared-interface constraint in
 `unit-of-work-dependency.md` ("U5 with U6 — one interface, two files"). Infrastructure Design is the
 right entry point: it maps that contract onto an actual host layout.
 ## Steps
 - [x] Step 1: Analyze design artifacts (requirements.md §§ Decisions/FR/ASM, unit-of-work.md U6,
      unit-of-work-dependency.md, reference project `.gitea/workflows/` at
      `K:\Development\SlpSoftware\Projects\SlpSoftware`)
 - [x] Step 2: Create this plan
 - [x] Step 3: Generate context-appropriate questions (below)
 - [x] Step 4: Store plan (this file)
 - [x] Step 5: Notify user in chat
 - [x] Step 6: Collect and analyze answers (12/12 answered, no contradictions — Q1/Q10 initially
      looked like a conflict, same-port two-instance clash, resolved: port assignment is a
      host/nginx-level fact outside the workflow's concern, so "directory-only" split is consistent)
 - [x] Step 7: Generate infrastructure design artifacts (`infrastructure-design.md`,
      `deployment-architecture.md`)
 - [x] Step 8: Present completion message
 - [ ] Step 9: Wait for explicit approval
 - [ ] Step 10: Record approval and update progress
 ## What the reference project (`SlpSoftware`) already establishes — not re-asked
 - Reusable `workflow_call` deploy workflow, invoked with `secrets: inherit`
 - `sshpass` + `scp` in a plain shell step (container SCP actions fail on the Podman runner with a 409
  attach error) — D-05
 - A `config` job that turns `env:` values into job outputs, because the `env` context is unavailable
  inside a reusable workflow's `with:` block
 - Gitea Actions **variables** for paths/environment names, **secrets** for host/credentials
  (`PI_MAIN_HOST`, `PI_MAIN_PORT`, `PI_MAIN_USERNAME`, `PI_MAIN_PASSWORD` — reusable per ASM-07)
 None of that needs to be re-decided. What the reference project does **not** cover — because it only
 ever uploads a static `dist/` folder to a fixed path — is everything below: this unit publishes a
 running .NET process, must switch releases atomically without a moment of downtime for the
 customer's website, must back up a database first, and must be able to restart the process and prove
 it came back healthy.
 ---
 ## Category: Deployment Environment
 ### Question 1
 Is production the **same Raspberry Pi** as test (different port/directory), or a **second,
 separate** Pi?
 A) Same Pi, different port and directory per environment
 B) Separate Pi per environment
 C) Not decided yet — design for either (parameterize host via Gitea variables/secrets, don't assume)
 [Answer]: A, but only a different directory
 ### Question 2
 What OS architecture is the Pi running, for the `dotnet publish` runtime identifier?
 A) `linux-arm64` (Raspberry Pi 4/5, 64-bit OS)
 B) `linux-arm` (32-bit OS)
 C) `linux-x64` (not a Pi / emulated)
 D) Other (please describe after [Answer]: tag below)
 [Answer]: A, but I might need a windows package later as well for other client API's. For now it is on the Pi
 ---
 ## Category: Compute Infrastructure
 ### Question 3
 How is the published process managed/restarted on the Pi (ASM-03 assumes systemd)?
 A) systemd service, one unit per environment (e.g. `slpmodularcms-test.service`,
   `slpmodularcms-production.service`), restarted via `sudo systemctl restart <unit>` over SSH
 B) systemd service, but restart is a manual/documented step, not automated in the workflow
 C) Some other process manager (pm2, supervisor, a custom script) — describe below
 D) Other (please describe after [Answer]: tag below)
 [Answer]: A, but if there are better or more convenient ways to manage .net processes let me know
 ### Question 4
 Passwordless `sudo systemctl restart` for the deploy user — is that already configured, or does it
 need to be part of the one-time host setup documented in Operations?
 A) Already configured
 B) Needs to be added — document it as a one-time host-setup step in Operations
 C) Avoid sudo entirely — run the app as the deploy user's own systemd **user** service
   (`systemctl --user restart`), no elevation needed
 D) Other (please describe after [Answer]: tag below)
 [Answer]: C
 ### Question 5
 Should the workflow **publish** (`dotnet publish -r <RID> --self-contained ...`) framework-dependent
 or self-contained?
 A) Self-contained (Pi doesn't need a separately-installed .NET runtime; larger artifact)
 B) Framework-dependent (assumes .NET 10 runtime already installed on the Pi, per ASM-03; smaller
   artifact)
 C) Other (please describe after [Answer]: tag below)
 [Answer]: B
 ---
 ## Category: Storage Infrastructure
 ### Question 6
 Release directory layout on the host, given `deploy_path` as the single input (FR-02) — how should
 `releases/`, the atomic `current` pointer, and the persistent `wwwroot/web/` be arranged underneath
 it?
 A) `{deploy_path}/releases/{timestamp-or-sha}/` per release, `{deploy_path}/current` symlink switched
   atomically to point at one release, `{deploy_path}/shared/wwwroot-web/` persistent and
   symlinked into each new release as `wwwroot/web`
 B) Same as A, but the persistent website content lives at a path **outside** `deploy_path` entirely
   (e.g. a fixed host path unrelated to the deploy target), to make it structurally impossible for
   any future workflow change to prune it by mistake
 C) Other (please describe after [Answer]: tag below)
 [Answer]: A
 ### Question 7
 How many previous releases should be retained after pruning (FR-06: "at least the previous one")?
 A) 2 (current + 1 previous)
 B) 3 (current + 2 previous)
 C) 5
 D) Other (please describe after [Answer]: tag below)
 [Answer]: A
 ### Question 8
 Database backup before production deploy (FR-20) — is it automatable from the Gitea runner over SSH
 (ASM-04 assumes the database is reachable from the *application*, not necessarily from the runner)?
 A) Automatable: the deploy workflow runs a backup command over SSH on the Pi itself (e.g.
   `sqlcmd`/`sqlpackage` or a SQL Server backup script already on the host), before the atomic switch
 B) Not automatable from the runner: emit a clear, verifiable manual step in the deployment
   instructions instead (FR-20's documented fallback), and the workflow does not attempt it
 C) Not sure yet — design the workflow with an explicit backup step that can be a placeholder script
   the user fills in during Operations
 D) Other (please describe after [Answer]: tag below)
 [Answer]: A
 ---
 ## Category: Networking Infrastructure
 ### Question 9
 Post-switch health verification (FR-06 implies confirming the restart succeeded) — how should the
 workflow check `/health`?
 A) `curl` from the Gitea runner against the environment's public HTTPS URL
 B) `curl` executed **on the Pi itself** over the same SSH connection, against `localhost:<port>`
   (works even if the public URL is only reachable through a reverse proxy not yet configured, and
   doesn't depend on external DNS/TLS)
 C) Both — localhost check on the Pi first, then a public-URL check as a secondary confirmation
 D) Other (please describe after [Answer]: tag below)
 [Answer]: A
 ### Question 10
 Is there an existing reverse proxy (nginx, as in the reference project) in front of this
 application, or does Kestrel serve requests directly?
 A) Kestrel serves directly on a fixed port per environment (e.g. `5000` test, `5001` production) —
   no reverse proxy for this app
 B) Existing nginx reverse proxy in front, forwarding to Kestrel on a local port
 C) Not decided yet / out of scope for this workflow — document the assumption and let Operations
   confirm the real topology
 D) Other (please describe after [Answer]: tag below)
 [Answer]: B
 ---
 ## Category: Monitoring Infrastructure
 ### Question 11
 If the post-switch health check (Question 9) fails, what should the workflow do?
 A) Fail the job loudly (red pipeline) but leave the atomic switch as-is — no automatic rollback;
   manual rollback via Operations' documented procedure
 B) Automatically re-point `current` back to the previous release and restart, then fail the job
 C) Other (please describe after [Answer]: tag below)
 [Answer]: A
 ---
 ## Category: Shared Infrastructure
 ### Question 12
 Gitea Actions variable/secret naming for the deploy target — reuse the reference project's
 `PI_MAIN_*` secrets (ASM-07), or introduce project-specific names since this repo has its own
 test **and** production paths on the same or a different host?
 A) Reuse `PI_MAIN_HOST` / `PI_MAIN_PORT` / `PI_MAIN_USERNAME` / `PI_MAIN_PASSWORD` as-is (same
   values already configured for the reference repo, since it's the same Pi)
 B) New, project-scoped secrets (e.g. `SLPMODULARCMS_PI_HOST`, etc.) even if the values happen to be
   identical today, so the two repositories' deploy targets can diverge independently later
 C) Other (please describe after [Answer]: tag below)
 [Answer]: A, but rename PI_MAIN_HOST to PI_MAIN_ADDRESS
 ---
 ## Self-Validation
 - [x] Every question uses lettered multiple-choice options
 - [x] Every question ends in an explicit `Other` option
 - [x] Every question has an `[Answer]:` tag
 - [x] No process/approval question is included here (those stay in chat)
@@ -0,0 +1,137 @@
 # Deployment Architecture — U6 Deploy Workflow
 ## Overview Diagram
 ```mermaid
 graph TD
    ci["CI Workflow U5"]
    deploy["Reusable Deploy Workflow deploy-scp.yaml"]
    runner["Gitea Runner"]
    ssh["SSH SCP Transport"]
    pi["Raspberry Pi"]
    svc_test["systemd --user slpmodularcms-test"]
    svc_prod["systemd --user slpmodularcms-production"]
    rel_test["Test Releases Directory"]
    rel_prod["Production Releases Directory"]
    shared["Persistent wwwroot-web"]
    nginx["Existing nginx Reverse Proxy"]
    db["SQL Server Database"]
    ci -->|"invokes with inputs"| deploy
    deploy -->|"runs on"| runner
    runner -->|"backup then upload over"| ssh
    ssh --> pi
    pi --> rel_test
    pi --> rel_prod
    rel_test -.->|"symlink"| shared
    rel_prod -.->|"symlink"| shared
    pi --> svc_test
    pi --> svc_prod
    svc_test --> nginx
    svc_prod --> nginx
    ssh -->|"production only backup"| db
    runner -->|"health check via public URL"| nginx
    classDef pipeline fill:#63b3ed,stroke:#2b6cb0,stroke-width:1px,color:#000;
    classDef transport fill:#f6e05e,stroke:#c05621,stroke-width:1px,color:#000;
    classDef host fill:#9ae6b4,stroke:#2f855a,stroke-width:1px,color:#000;
    classDef storage fill:#fbd38d,stroke:#92400e,stroke-width:1px,color:#000;
    classDef external fill:#e9d5ff,stroke:#6b21a8,stroke-width:1px,color:#000;
    class ci,deploy,runner pipeline;
    class ssh transport;
    class pi,svc_test,svc_prod host;
    class rel_test,rel_prod,shared,db storage;
    class nginx external;
 ```
 Text alternative: U5 invokes the reusable deploy workflow on the Gitea runner, which backs up the
 database (production only) then uploads the release to the Pi over SSH/SCP; the Pi hosts separate
 release directories and systemd `--user` services for test and production, both symlinking the same
 persistent website content and sitting behind the existing nginx reverse proxy, which the runner
 also calls to verify `/health` after each switch.
 ---
 ## Release Directory Diagram
 ```mermaid
 graph TD
    deploy_path["deploy_path per environment"]
    releases["releases directory"]
    rel_new["releases timestamp new"]
    rel_prev["releases timestamp previous kept"]
    current["current symlink"]
    shared_dir["shared directory"]
    web_persist["shared wwwroot-web persistent"]
    admin["wwwroot admin part of release"]
    web_link["wwwroot web symlink"]
    deploy_path --> releases
    deploy_path --> current
    deploy_path --> shared_dir
    releases --> rel_new
    releases --> rel_prev
    shared_dir --> web_persist
    current -.->|"points to"| rel_new
    rel_new --> admin
    rel_new --> web_link
    web_link -.->|"symlink"| web_persist
    classDef path fill:#63b3ed,stroke:#2b6cb0,stroke-width:1px,color:#000;
    classDef release fill:#9ae6b4,stroke:#2f855a,stroke-width:1px,color:#000;
    classDef persistent fill:#fbd38d,stroke:#92400e,stroke-width:1px,color:#000;
    class deploy_path,releases,current path;
    class rel_new,rel_prev,admin release;
    class shared_dir,web_persist,web_link persistent;
 ```
 Text alternative: under each environment's deploy path, the current symlink points at the newest of
 two retained release directories, each containing the rebuilt admin SPA and a symlink into the one
 persistent shared website directory that survives every release.
 ---
 ## Deployment Sequence Diagram
 ```mermaid
 sequenceDiagram
    box rgba(99,179,237,0.4) Pipeline
        participant U5 as CI Workflow
        participant DW as Deploy Workflow
    end
    box rgba(154,230,180,0.4) Pi
        participant PI as Pi Host
        participant SVC as Systemd Service
    end
    box rgba(233,213,255,0.4) External
        participant NGINX as Nginx Proxy
        participant DB as Database
    end
    U5->>DW: invoke with artifact_name, environment, deploy_path, service_name, health_check_url
    alt production only
        DW->>PI: run backup script over SSH
        PI->>DB: BACKUP DATABASE
        DB-->>PI: backup file written
    end
    DW->>PI: scp artifact into new release directory
    DW->>PI: link persistent web content into release
    DW->>PI: switch current symlink atomically
    DW->>SVC: restart service
    SVC-->>DW: restart acknowledged
    DW->>NGINX: curl health check url
    NGINX->>SVC: forward to local port
    SVC-->>NGINX: health response
    NGINX-->>DW: health check result
    alt health check passed
        DW->>PI: prune releases beyond retention count
    else health check failed
        DW-->>U5: fail job, current left on new release
    end
 ```
 Text alternative: the deploy workflow optionally backs up the database, uploads and links the new
 release, atomically switches and restarts the service, then verifies health through nginx before
 pruning old releases — or fails the job without rolling back if the health check does not pass.
@@ -0,0 +1,185 @@
 # Infrastructure Design — U6 Deploy Workflow
 **Date**: 2026-07-28
 **Basis**: `u6-deploy-workflow-infrastructure-design-plan.md` (12 questions, all answered)
 ---
 ## 1. Host Topology
 **Single Raspberry Pi** (`linux-arm64`) hosts both **test** and **production** — same machine, same
 port per environment is not something this workflow manages, only the **directory differs**
 (Q1 = A, "but only a different directory").
 This is coherent with Q10 = B (an **existing nginx reverse proxy** already sits in front of the
 application): nginx routes by hostname to a **fixed, pre-configured local port per environment**.
 That port assignment is a one-time host-setup fact (documented in Operations, FR-23) — the deploy
 workflow never reads, writes or reasons about a port. Its only environment-specific inputs are the
 deploy path and which systemd unit to restart.
 **Publish target**: `linux-arm64`, **framework-dependent** (Q2 = A, Q5 = B) — assumes the .NET 10
 runtime is already installed on the Pi (ASM-03). Smaller artifact, faster upload over the same SSH
 transport every other unit already relies on.
 > **Noted, not built**: a Windows-hosted client API may be needed later for other purposes (per
 > Q2's answer). Out of scope for U6 — if it materializes, it is a second `workflow_call` with its
 > own RID and transport, following the same extensibility pattern FR-02/D-02 already established for
 > FTPS. No design decision here needs to anticipate it further.
 ---
 ## 2. Process Management
 **`systemctl --user` units**, one per environment (Q3 = A, refined by Q4 = C: avoid `sudo`
 entirely). No elevation is needed on the deploy connection — the SSH user *is* the service owner.
 **INFRA-U6-01 (new finding, not previously raised in Requirements or Application Design)**: a
 `systemd --user` service manager is torn down when the owning user's last login session ends —
 which includes the SSH session the deploy workflow opens and closes on every run. Without
 `loginctl enable-linger <deploy-user>` set **once** on the Pi, the freshly restarted service would
 be killed again a few seconds after the deploy workflow's SSH connection closes, silently taking
 test or production down right after every successful deploy. This is a one-time host-setup item,
 not a per-run workflow step — it must be carried into Operations' deployment instructions (FR-23)
 as an explicit prerequisite, verified once and never touched by the workflow itself.
 Unit naming: `slpmodularcms-test.service` / `slpmodularcms-production.service`, supplied to the
 reusable workflow via a `service_name` input (see § 5) rather than hardcoded, so the names stay
 data, not workflow logic.
 Restart command: `systemctl --user restart <service_name>` over the same SSH connection used for
 the file transfer.
 ---
 ## 3. Release and Persistence Layout
 Directory layout under each environment's `deploy_path` (Q6 = A):
 ```
 {deploy_path}/
  releases/
    {timestamp}/                 # one per deploy, e.g. 20260728143000
      ...published app files...
      wwwroot/
        admin/                   # part of the release — rebuilt by BuildAndCopyAdminFrontend on every publish
        web -> ../../../shared/wwwroot-web   # symlink, NOT part of the release
  shared/
    wwwroot-web/                 # persistent customer website content — survives every release (FR-08, ASM-01)
  current -> releases/{timestamp} # atomic switch target; systemd unit's WorkingDirectory/ExecStart points here
 ```
 **Retention (Q7 = A)**: keep **2** releases — `current` plus exactly one previous. Pruning runs
 **after** a successful health check (§ 4), deleting every `releases/*` entry except the two most
 recent by directory name (timestamps sort lexically). `shared/` is never touched by pruning.
 **wwwroot/web handling**: `dotnet publish` produces an (empty or placeholder-only) `wwwroot/web/`
 inside the release per the existing `.csproj`. Before the atomic switch, the deploy step must:
 1. `mkdir -p {deploy_path}/shared/wwwroot-web` (idempotent — safe on the very first deploy, when
   nothing has been placed there yet, per U1's tolerance for a missing `wwwroot/web` at startup)
 2. Remove whatever `wwwroot/web` the publish step produced inside the new release directory
 3. Symlink `releases/{timestamp}/wwwroot/web -> ../../../shared/wwwroot-web`
 Only then is `current` re-pointed. This ordering is load-bearing: symlinking after the switch would
 leave a window where `current` serves a release with no `web` mount at all.
 ---
 ## 4. Deployment Sequence and Health Verification
 Per-environment sequence (test runs steps 1, 3–7; production additionally runs step 2):
 1. Download the build artifact (`actions/download-artifact`, unchanged from the reference project)
 2. **Production only** (Q8 = A): trigger a database backup over the existing SSH connection —
   `ssh {user}@{host} 'bash ~/scripts/backup-slpmodularcms-db.sh'`. The backup script itself lives
   on the Pi and is created once during host setup (Operations, FR-23); the workflow never
   transmits or references database credentials, keeping D-16's "runtime secrets live in host
   environment variables, the workflow does not manage them" intact. This satisfies FR-20's
   automation branch without adding a new secret surface.
 3. Upload the artifact via `sshpass` + `scp` into a fresh `releases/{timestamp}/` directory
   (unchanged transport pattern from the reference project, D-05)
 4. Link the persistent website (§ 3, steps 1–3 above)
 5. Atomically switch `current` to the new release (`ln -sfn` — atomic on the same filesystem) and
   restart the environment's systemd `--user` unit
 6. **Verify** (Q9 = A): `curl -f` from the **Gitea runner** against the environment's public HTTPS
   `/health` URL — reachable because nginx (Q10 = B) already routes that hostname to the
   now-restarted local port. Retry with a short backoff (e.g. up to 10 attempts, 3s apart) to
   absorb ordinary process-restart time before declaring failure.
 7. Prune old releases per § 3 retention rule — **only if step 6 succeeded**
 **On health-check failure (Q11 = A)**: the job fails loudly; `current` is **left pointed at the new,
 unhealthy release** — no automatic rollback. Because retention always keeps the previous release on
 disk, a manual rollback is always a re-point-and-restart away: documented as an explicit Operations
 procedure (FR-23, D-26), not automated here. Pruning is skipped in this case, since the failed
 release must not be the only one left standing.
 ---
 ## 5. Reusable Workflow Interface (shared with U5)
 `deploy-scp.yaml` (`workflow_call`) inputs — extends FR-02's minimum set:
 | Input | Type | Required | Purpose |
 |---|---|---|---|
 | `artifact_name` | string | yes | Build artifact to download (FR-02 minimum) |
 | `environment` | string | yes | Label used in log output and health-check selection (FR-02 minimum) |
 | `deploy_path` | string | yes | Environment's release root, e.g. `${{ vars.DEPLOY_PATH_TEST }}` (FR-02 minimum) |
 | `service_name` | string | yes | systemd `--user` unit to restart, e.g. `${{ vars.SERVICE_NAME_TEST }}` |
 | `health_check_url` | string | yes | Public `/health` URL to verify, e.g. `${{ vars.HEALTH_CHECK_URL_TEST }}` |
 | `run_db_backup` | boolean | no, default `false` | Set `true` only for the production call (FR-20 gate) |
 | `transport` | string | no, default `scp` | Reserved per D-02/NFR-09; only `scp` implemented now |
 `secrets: inherit` passes through the shared Pi credentials (§ 6) unchanged from the reference
 project's pattern.
 ---
 ## 6. Gitea Variables and Secrets
 **Secrets** (shared — same Pi for both environments, Q1 + Q12):
 | Secret | Notes |
 |---|---|
 | `PI_MAIN_ADDRESS` | Renamed from the reference project's `PI_MAIN_HOST` per Q12's answer |
 | `PI_MAIN_PORT` | SSH port, unchanged convention |
 | `PI_MAIN_USERNAME` | Deploy user — the same account owning the `--user` systemd units |
 | `PI_MAIN_PASSWORD` | Unchanged convention (D-16; SSH-key migration remains a documented future step per the reference project's own note) |
 **Variables** (per environment):
 | Variable | Test | Production |
 |---|---|---|
 | `DEPLOY_PATH_TEST` / `DEPLOY_PATH_PRODUCTION` | e.g. `/home/{user}/apps/slpmodularcms-test` | e.g. `/home/{user}/apps/slpmodularcms-production` |
 | `SERVICE_NAME_TEST` / `SERVICE_NAME_PRODUCTION` | `slpmodularcms-test.service` | `slpmodularcms-production.service` |
 | `HEALTH_CHECK_URL_TEST` / `HEALTH_CHECK_URL_PRODUCTION` | e.g. `https://test.<domain>/health` | e.g. `https://<domain>/health` |
 Exact hostnames/paths are filled in during Operations host setup (FR-23) — this design fixes the
 **names and shapes** of the variables, not their runtime values.
 ---
 ## 7. Decisions Traceability
 | Decision | Source | Resolution |
 |---|---|---|
 | Same Pi, directory-only split | Q1 = A | § 1 |
 | `linux-arm64`, future Windows target noted only | Q2 = A | § 1 |
 | `systemctl --user`, no sudo | Q3 = A, Q4 = C | § 2 |
 | Framework-dependent publish | Q5 = B | § 1 |
 | Release/shared/current layout | Q6 = A | § 3 |
 | Retain 2 releases | Q7 = A | § 3 |
 | DB backup via host script over SSH | Q8 = A | § 4 |
 | Health check via public URL from the runner | Q9 = A | § 4 |
 | Existing nginx reverse proxy | Q10 = B | § 1, § 4 |
 | No automatic rollback on failed health check | Q11 = A | § 4 |
 | `PI_MAIN_ADDRESS` (renamed), shared secrets | Q12 = A + rename | § 6 |
 | **INFRA-U6-01**: `loginctl enable-linger` required for `--user` units to survive SSH disconnect | Newly identified during this design | § 2 — carried to Operations (FR-23) |
 ---
 ## 8. What Remains for Code Generation
 - `.gitea/workflows/deploy-scp.yaml` implementing §§ 3–5
 - No application code changes — U6 is `.gitea/workflows/` only (component C-11)
 - Verification in CI is necessarily limited to YAML validity and step logic review; the actual Pi,
  SSH credentials and database are real-run concerns that belong to Operations (per U6's Definition
  of Done in `unit-of-work.md`)