Continuous Integration / config-sls (pull_request) Successful in 8s
Continuous Integration / config-asf (pull_request) Successful in 6s
Continuous Integration / changes (pull_request) Successful in 13s
Continuous Integration / backend-build (pull_request) Successful in 1m53s
Continuous Integration / compile-sites (pull_request) Successful in 1m54s
Continuous Integration / vulnerability-scan (pull_request) Successful in 1m56s
Continuous Integration / frontend-prepare (pull_request) Successful in 2m24s
Continuous Integration / frontend-build (pull_request) Successful in 35s
Continuous Integration / frontend-lint (pull_request) Successful in 38s
Continuous Integration / backend-test (pull_request) Successful in 4m18s
Continuous Integration / frontend-test (pull_request) Successful in 2m35s
Continuous Integration / publish-sls-test (pull_request) Skipped
Continuous Integration / publish-sls-production (pull_request) Skipped
Continuous Integration / deploy-sls-test (pull_request) Skipped
Continuous Integration / deploy-sls-production (pull_request) Skipped
Continuous Integration / publish-asf-test (pull_request) Skipped
Continuous Integration / deploy-asf-test (pull_request) Skipped
Continuous Integration / smoke-sls-test (pull_request) Skipped
Continuous Integration / smoke-asf-test (pull_request) Skipped
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
120 lines
9.9 KiB
Markdown
120 lines
9.9 KiB
Markdown
# Migration plan: HeroImages to the library's own tables (decision 13)
|
|
|
|
Status: plan, not built, not run anywhere. Decided by Bryan 2026-11-17 (via SLP Software - SLS - Main): it may be built in the next step, **only after this plan with its back-up and rollback exists**, **rolled out to TEST only (never production)**, and **SLS-Main is told before anything is deployed to test**.
|
|
|
|
## Goal
|
|
|
|
Give every photo one library record that does not depend on the Hero: `MediaAsset` (the photo) and `MediaUse` (where it is used), with **the same ids, hashes, set hashes and public addresses** as today. Nothing in `/media/hero/<setHash>/...` changes, so ASF, SLS and the golden files stay as they are.
|
|
|
|
## Why expand then contract
|
|
|
|
| Phase | What | Reversible? |
|
|
|---|---|---|
|
|
| 0 Back-up | Back-up of the database and a listing of the media directory (below). | n/a |
|
|
| 1 Expand | New tables `MediaAssets` and `MediaUses`, filled by copying `HeroImages` and `HeroSlides`. The Hero keeps reading and writing its own tables; the library reads the new tables behind a setting. | Yes: drop the two new tables, nothing else changed. |
|
|
| 2 Switch | Hero reads photos from the library tables (setting on). New uploads write both (dual write) so the old tables stay complete. | Yes: switch the setting off; the old tables are complete. |
|
|
| 3 Contract (a later release, after acceptance) | The Hero stops writing its own image columns; they are dropped in a separate migration. | Only from the back-up of that moment. Not part of the first rollout. |
|
|
|
|
Files are never moved or renamed: the migration changes rows only, so the media directory is the same before and after.
|
|
|
|
## Mapping
|
|
|
|
| `HeroImages` column | `MediaAssets` column | Note |
|
|
|---|---|---|
|
|
| `Id` | `Id` | Same id, so references keep working. |
|
|
| `OriginalSha256`, `OriginalKey`, `SourceFormat`, `Width`, `Height`, `Bytes` | same | |
|
|
| `FocusX`, `FocusY`, `Zoom` | `DefaultFocusX`, `DefaultFocusY`, `DefaultZoom` | The photo's default focus. |
|
|
| `WideFocus*`, `TallFocus*` | `MediaUses` rows (one per slide use, ratios `wide` and `tall`) | The per-use crop; for a photo without a slide the default focus is enough. |
|
|
| `SetHash`, `Variants` | `MediaUses.SetHash` (per use) | Unchanged values; computed by the existing formula. |
|
|
| `CreatedAt`, `CreatedByUserId` | same | |
|
|
| `OriginalFileName`, `LastUsedAt`, `DeletedAt` | same | Added in the media-library step (additive columns of the Hero). |
|
|
|
|
| `HeroSlides` | `MediaUses` |
|
|
|---|---|
|
|
| `Id`, `ImageId` | `Id`, `AssetId` (module `hero`, place kind `slide`, place id = slide id) |
|
|
|
|
## Back-up (before phase 1, every environment)
|
|
|
|
1. **Database**: a full dump of the Hero database (and of the CMS database if it is one), written to a dated file outside the deploy directory, for example `mysqldump --single-transaction <database> > hero-before-media-migration-YYYYMMDD.sql`. Keep it until the contract phase is accepted.
|
|
2. **Media directory**: nothing is moved, but a listing with sizes and hashes is saved (`find <Media__Directory> -type f -exec sha256sum {} +` to a text file) so the directory can be compared afterwards. On the Raspberry Pi a copy of the directory is also a quick option (`cp -a`), if there is room.
|
|
3. **Row counts** noted: `HeroImages`, `HeroSlides`, `HeroRetiredFiles`.
|
|
|
|
## Verification (right after phase 1, every environment)
|
|
|
|
The migration job compares and **aborts** (rolls its transaction back) on any difference:
|
|
|
|
- number of `MediaAssets` = number of `HeroImages` (all rows, also those in the trash);
|
|
- every `HeroImages.Id`, `OriginalSha256` and `SetHash` has an identical `MediaAssets`/`MediaUses` counterpart;
|
|
- number of `MediaUses` of module `hero` = number of `HeroSlides`;
|
|
- a sample of ten public addresses (`/media/hero/<setHash>/<file>`) is requested before and after and gives the same bytes (checked by the smoke test).
|
|
|
|
## Rollback
|
|
|
|
| When | How |
|
|
|---|---|
|
|
| The verification fails | The migration runs in one transaction and rolls back by itself; nothing is left. |
|
|
| After phase 1, before the switch | Drop `MediaUses` and `MediaAssets` (the EF `Down` migration). The Hero never read them. |
|
|
| After the switch | Switch the setting back (`Media:Library:ReadFromLibrary=false`); the old tables are complete because of the dual write. Then drop the new tables if wanted. |
|
|
| After the contract phase (later) | Restore the database dump taken before that migration and the media listing check. |
|
|
|
|
## Roll-out rules (from Bryan)
|
|
|
|
1. **Test only.** No production variables, no production deploy, no release branch.
|
|
2. **Tell SLS-Main before deploying to test** (they run the deploy and the smoke test).
|
|
3. Back-up first (above), on the test host too.
|
|
4. Deploy with the setting **off**, check the verification result in the log, then switch it on, then the smoke test (hero markers, `/theme-init.js`, a sample of media addresses), then Bryan checks the admin.
|
|
|
|
## Status (2026-11-17): phase 1 (expand) is built, behind a setting that is OFF
|
|
|
|
Built on `feature/media-library-screen`, not yet deployed anywhere:
|
|
|
|
- New module `SlpModularCms.Modules.Media` (own `MediaDbContext`, tables `MediaAssets` and `MediaUses`, additive migration `InitialCreate`; no existing table is touched). It is referenced by the three hosts, so a deploy of this code creates the two empty tables at start-up and nothing else.
|
|
- `IMediaMigrationSource` (Core) and `HeroMediaMigrationSource` (Hero, read-only): the hero's photos (also those in the trash) and slides with their own ids, hashes and set hashes.
|
|
- `MediaLibraryCopier`: copies and **verifies** (counts, ids, hash, key, size, set hash) in one transaction; any difference or error rolls the whole copy back and nothing is left. It MIRRORS the source, so running it again gives the same result; rows of other modules are not touched.
|
|
- The copy runs only when `Media__Library__CopyOnStartup=true` (default **false**) and a failure is logged, never fatal.
|
|
- Tested in memory and against a real MariaDB (migration, copy twice, rollback on a failing copy).
|
|
|
|
Nothing reads from the new tables yet (phase 2, the switch, is not built). The library screen still reads the hero through the usage provider.
|
|
|
|
### Roll-out on TEST (needs SLS-Main to be told first)
|
|
|
|
1. Back-up (database dump and media listing) as above.
|
|
2. Deploy the branch with `Media__Library__CopyOnStartup` unset: the two empty tables appear; check the log for the Media migration.
|
|
3. Set `Media__Library__CopyOnStartup=true` and restart: the log says `Media library copy verified: N photos and M uses from hero` (or the error and the rollback).
|
|
4. Rollback at any point: set it back to false (the tables stay, unused) or drop `MediaUses` and `MediaAssets`.
|
|
5. Never on production.
|
|
|
|
### Still to build (phases 2 and 3)
|
|
|
|
The switch (`Media:Library:ReadFromLibrary`, dual write of new uploads, library reading from the new tables) and the contract phase. They need Bryan's acceptance of the copy on test first.
|
|
|
|
## Test host: what to check around the copy (prepared 2026-11-17; the copy is NOT switched on yet)
|
|
|
|
**Deploy without the copy (decided on assumption by SLS-Main, Bryan away):** the test deploy of the branch carries two additive database changes, the Hero migration `AddMediaLibraryColumns` (three nullable columns) and the empty tables of the Media module. No separate database dump is taken on the Raspberry Pi (SLS-Main cannot make one); the assumption is that the test data can be rebuilt with the seed scripts and that both changes are additive. Bryan checks this assumption later.
|
|
|
|
After that deploy, check in the log:
|
|
|
|
- `Applying migration '..._AddMediaLibraryColumns'` and `Applying migration '..._InitialCreate'` (Media) appear once, no error;
|
|
- the Hero admin still opens, a photo can be uploaded and a slide saved (smoke test), and `/media/hero/<setHash>/...` addresses still answer;
|
|
- `Media__Library__CopyOnStartup` is NOT set: no line `Media library copy verified` appears.
|
|
|
|
**To switch the copy on (only after Bryan's agreement):**
|
|
|
|
1. Make a database dump first if it can be done, and note the number of rows: `SELECT COUNT(*) FROM HeroImages; SELECT COUNT(*) FROM HeroSlides;`.
|
|
2. Set the environment variable `Media__Library__CopyOnStartup=true` on the test host and restart the application.
|
|
3. The log must say `Media library copy verified: N photos and M uses from hero.` with N equal to the `HeroImages` count and M equal to the `HeroSlides` count. An error line `The media library copy failed and was rolled back` means nothing was copied.
|
|
4. Check: `SELECT COUNT(*) FROM MediaAssets; SELECT COUNT(*) FROM MediaUses;` equals the same counts; the library screen still shows the same photos (it still reads the hero).
|
|
5. Switch it off again (remove the variable, restart) when done: the copy can be repeated at any time; it is a mirror.
|
|
|
|
**Rollback:** remove the variable (the tables stay, unused), or drop the tables with the EF `Down` migration of the Media module; the Hero's own tables are never changed by the copy.
|
|
|
|
## Switching the copy on at the test hosts (no shell on the Pi needed)
|
|
|
|
The application reads `Media__Library__CopyOnStartup` from the host's `shared/env` (the file the service starts with). The deploy workflow can set exactly this one key, so nobody needs a shell on the Raspberry Pi:
|
|
|
|
1. Dispatch the CI workflow (`Continuous Integration`) on the branch with **deploy_sls_test** and/or **deploy_asf_test** checked and **media_library_copy = on**. Each test deploy then sets `Media__Library__CopyOnStartup=true` in its own `shared/env` before restarting the service, and the run log shows the line `Media library copy verified: N photos and M uses from hero` (or the error line that says the copy was rolled back).
|
|
2. Check N and M against the hero (photos and slides) and the library screen.
|
|
3. Dispatch once more with **media_library_copy = off** (and the deploy flags again): the key is removed and the next start does not copy. (`unchanged`, the default, never touches the file.)
|
|
|
|
The input is refused for any environment other than `test`, and only the three values are accepted; the workflow never prints the file (it holds secrets). Production is not reachable.
|