Switches the database from SQL Server to MariaDB
The target Pi only has MariaDB, and SQL Server has no ARM64 build at all - not a config problem, a real gap discovered during deployment setup. Swapped the EF Core provider, regenerated every migration, updated connection strings and the backup script everywhere they appear. Took two tries to land on a provider that actually works: Pomelo builds fine against this project's EF Core 10 packages but fails at runtime (it's compiled against 9's internal API surface, which moved in 10 wherever Identity/DataProtection force the newer packages). Oracle's official provider builds and migrates fine but has a real MariaDB bug in its own migration-lock code, reproduced against a live database. Kept Oracle's provider and worked around just that one broken method - everything else it does is correct - rather than give up more of the stack to chase a workaround. Verified against a real local MariaDB end to end: all three migrations applied, both hosts start clean, full suite still green.
This commit is contained in:
@@ -57,6 +57,72 @@
|
||||
- **Artifacts**: `operations/deployment/deployment-plan.md`, `deployment-instructions.md`,
|
||||
`rollback-plan.md`
|
||||
|
||||
### Database Provider Migration — SQL Server → MariaDB (2026-07-29)
|
||||
|
||||
Discovered while finalizing Deployment Setup: the target Pi only runs MariaDB, and Microsoft SQL
|
||||
Server has **no ARM64 build at all** (the `mcr.microsoft.com/mssql/server` image is `linux/amd64`
|
||||
only; Azure SQL Edge, the former ARM answer, is retired). This invalidates ASM-04. Confirmed no
|
||||
other machine is available, and the eventual production host (`mijnhostingpartner.nl`) will also
|
||||
run MariaDB, plus the workload is small enough that MariaDB's performance is not a concern — user
|
||||
decided: switch the database provider, not the deployment target.
|
||||
|
||||
**What changed** (application code, not just Operations docs):
|
||||
- `SlpModularCms.Core.csproj`: `Microsoft.EntityFrameworkCore.SqlServer` → `MySql.EntityFrameworkCore`
|
||||
10.0.7 (Oracle's official provider)
|
||||
- `UseSqlServer(...)` → `UseMySQL(...)` in `ServiceCollectionExtensions.cs`, `AvailabilityModule.cs`,
|
||||
`MasterModule.cs`
|
||||
- `DatabaseMigrationExtensions.cs` + its test: `Microsoft.Data.SqlClient.SqlException` →
|
||||
`MySql.Data.MySqlClient.MySqlException` for the transient-failure classifier
|
||||
- All three DbContexts' migrations deleted and regenerated (`ApplicationDbContext`,
|
||||
`AvailabilityDbContext`, `MasterDbContext`) — no MySQL/MariaDB model-compatibility issues surfaced
|
||||
(no index-length problems, no raw SQL anywhere in the codebase to translate)
|
||||
- Connection strings updated across `appsettings.json` (Api + Api.Slave, all three tiers) from
|
||||
SQL Server format to MySQL format
|
||||
- `README.md` dev setup: MariaDB container command replacing the SQL Server one, with a migration
|
||||
note for anyone returning to old instructions
|
||||
- `deployment-instructions.md` / `rollback-plan.md`: connection string format, backup script
|
||||
rewritten around `mariadb-dump` (was `sqlcmd`/`BACKUP DATABASE`), restore procedure rewritten
|
||||
|
||||
**Provider selection — two failed attempts before the working one, both reproduced against a real
|
||||
MariaDB instance, not just reasoned about:**
|
||||
1. **Pomelo.EntityFrameworkCore.MySql** (the usual first choice, explicit first-class MariaDB
|
||||
support) — caps at EF Core 9.x, no EF Core 10 release exists. A Pomelo maintainer states
|
||||
([PR #2017](https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/pull/2017))
|
||||
that Pomelo 9 / EF Core 9 packages work fine on a net10.0 TFM — confirmed true only when nothing
|
||||
else forces EF Core 10 packages. This project's `Microsoft.AspNetCore.Identity.EntityFrameworkCore`
|
||||
and `Microsoft.AspNetCore.DataProtection.EntityFrameworkCore` are versioned in lockstep with the
|
||||
.NET 10 runtime and hard-require EF Core >= 10.0.9, so the resolved `Microsoft.EntityFrameworkCore.Abstractions`
|
||||
ends up at 10.0.9 regardless. Restore only warns (NU1608), and it builds — but fails at runtime
|
||||
with `MissingMethodException: AbstractionsStrings.ArgumentIsEmpty` the moment EF tooling touches
|
||||
a DbContext: Pomelo's compiled assembly calls an internal EF Core 9 helper that no longer exists
|
||||
in the 10.0.9 assembly actually loaded.
|
||||
2. **MySql.EntityFrameworkCore 10.0.7** (Oracle's official provider) — its net10.0 dependency group
|
||||
targets EF Core 10.0.7, compatible with 10.0.9. Builds and migrates cleanly, but
|
||||
`dotnet ef database update` against the real MariaDB throws
|
||||
`InvalidCastException: Unable to cast object of type 'System.DBNull' to type 'System.Int64'` in
|
||||
`MySQLHistoryRepository.AcquireDatabaseLock()` — a confirmed MariaDB-incompatibility bug (MariaDB's
|
||||
`GET_LOCK()` apparently returns `NULL` in a case Oracle's code doesn't handle, and Oracle's
|
||||
provider is tested against real MySQL Server, not MariaDB). This isn't limited to CLI tooling —
|
||||
the same code path runs on every application startup via `MigrateCoreDatabase()`.
|
||||
|
||||
**Working solution**: kept Oracle's `MySql.EntityFrameworkCore` 10.0.7 (otherwise fully compatible)
|
||||
and added `NonLockingMySQLHistoryRepository` (`SlpModularCms.Core/Hosting/`), wired in via
|
||||
`options.ReplaceService<IHistoryRepository, NonLockingMySQLHistoryRepository>()` on all three
|
||||
`AddDbContext` registrations. Oracle's internal `MySQLHistoryRepository` class can't be subclassed
|
||||
(it's `internal`, only its constructor is public), so the workaround constructs a real instance of
|
||||
it via reflection and forwards every `IHistoryRepository` member to it **except**
|
||||
`AcquireDatabaseLock`/`AcquireDatabaseLockAsync`, which return a no-op lock instead of ever reaching
|
||||
the broken `GET_LOCK` call. Accepted as safe because this deployment model never runs migrations
|
||||
from more than one place at a time (`MigrateCoreDatabase()` at startup, one deploy at a time via the
|
||||
atomic-release sequence) — a genuinely concurrent multi-instance migration race is not a scenario
|
||||
this architecture produces.
|
||||
|
||||
**Verified end to end**: all three migrations applied successfully to the user's real local
|
||||
MariaDB (`dotnet ef database update`, all expected tables present including `DataProtectionKeys`
|
||||
and the renamed Identity tables); both `SlpModularCms.Api` and `SlpModularCms.Api.Slave` start
|
||||
cleanly against it (`/health` → 200, `MigrateCoreDatabase()` logs "already up to date" on the
|
||||
second run); full backend suite re-confirmed at 372/372 passed, 0 build errors.
|
||||
|
||||
## Scope Decisions (from feature-selection.md)
|
||||
- **Public website**: documentation/instructions only — where the website build lands in `wwwroot/`, how it coexists with `wwwroot/admin/`, and what a per-website workspace must deliver. The website's own build/deploy workflow stays out of scope (Q4 = A).
|
||||
- **Environments**: local, test, production only.
|
||||
|
||||
@@ -1345,3 +1345,14 @@ by Infrastructure Design. U5 carries REF-U3-01's Umami origin gate plus FR-21 an
|
||||
**Context**: OPERATIONS — Deployment Setup, configuration guidance clarified before final approval
|
||||
|
||||
---
|
||||
## Database Provider Migration — SQL Server to MariaDB
|
||||
|
||||
**Timestamp**: 2026-07-29T00:00:00Z
|
||||
|
||||
**User Input**: "Op de Pi draait op dit moment alleen MariaDB en geen mssql. Maakt dat uit?" ... "Nee, geen andere machine, maar kan het misschien met een podman container geregeld worden? Die draait wel op die machine" ... "De beoogde productieomgeving zal ook MariaDB hebben (mijnhostingpartner.nl) en de websites zullen niet mega groot worden dus ik denk dat voor performance MAriaDB ook prima is. Laten we optie 1 doen met MariaDB. Ik heb de lokale connection string aangepast. Die zou moeten werken. Dus als het lukt mag je meteen de migraties mogen runnen als dat nodig is" ... "In deze pull request: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/pull/2017 [...] Als je dat kan controleren dan zouden we door kunnen" ... "Ja, probeer die override"
|
||||
|
||||
**AI Response**: "Identified that SQL Server has no ARM64 build and Azure SQL Edge (the former ARM path) is retired, explained the tradeoffs (remote SQL Server, QEMU emulation, or switching to MariaDB), user chose MariaDB. Swapped Microsoft.EntityFrameworkCore.SqlServer for MySql.EntityFrameworkCore across Core.csproj and the 3 UseSqlServer call sites, regenerated all migrations, updated connection strings and README/deployment docs. Tried Pomelo per the user's PR link — confirmed it restores/builds against EF Core 10.0.9 (matching the maintainer's claim) but fails at runtime with a MissingMethodException, since this project's Identity/DataProtection EF Core integration packages force EF Core 10 packages regardless of Pomelo's own 9.x dependency range. Reverted to Oracle's provider, which then surfaced a second, independent bug: AcquireDatabaseLock() throws InvalidCastException against real MariaDB (GET_LOCK() returns NULL where Oracle's code expects an integer). Implemented NonLockingMySQLHistoryRepository as a targeted workaround (constructs Oracle's internal history repository via reflection, forwards everything except the two lock methods). Verified end to end: all 3 migrations applied to the user's real local MariaDB, both hosts (Api and Api.Slave) start cleanly against it, full 372-test backend suite still green. Corrected ASM-04 in requirements.md and recorded the full saga in aidlc-state.md."
|
||||
|
||||
**Context**: OPERATIONS — Deployment Setup revealed a genuine Construction-phase technology gap; resolved with application code changes (not just documentation), verified against a real database
|
||||
|
||||
---
|
||||
|
||||
+1
-1
@@ -331,7 +331,7 @@ DEV-01…04 are **pre-existing or cost-driven** and none is introduced by this f
|
||||
| ASM-01 | With the atomic release switch (FR-06), `wwwroot/web/` must live **outside** the swapped release directory and be linked into it (e.g. a symlink to a persistent path on the host). | Otherwise switching releases silently discards the customer's website — precisely the failure mode D-06 was chosen to prevent. This follows necessarily from combining Q4 = C with CQ3 = C, so it is stated rather than asked. | Raise it and the deploy design changes materially; flag before Construction if this is not acceptable. |
|
||||
| ASM-02 | **No `wwwroot` folder is needed for the API.** The API is not static content — its assemblies live in the application root and it serves `/api/v1` through routing. The option is kept open but nothing is built for it. | Avoids building an unused folder. | If something static under an API path is intended, say so and FR-07 gains a third mount. |
|
||||
| ASM-03 | The Pi already runs, or can run, a .NET 10 runtime, and the app is managed by a process manager (systemd) that the deploy can restart over SSH. | The atomic switch requires restarting the process. | Restart mechanism changes; deploy step is rewritten. |
|
||||
| ASM-04 | The Pi's SQL Server database is reachable from the application, and a backup can be taken before a production deploy. | FR-20 depends on it. | FR-20 becomes a documented manual precondition only. |
|
||||
| ASM-04 | ~~The Pi's SQL Server database is reachable from the application~~, and a backup can be taken before a production deploy. | FR-20 depends on it. | **Wrong — corrected at Deployment Setup, 2026-07-29**: SQL Server has no ARM64 build at all, and the Pi only runs MariaDB. The EF Core provider was migrated from `Microsoft.EntityFrameworkCore.SqlServer` to `MySql.EntityFrameworkCore` (Oracle's official MySQL/MariaDB provider), all migrations regenerated, connection strings and the backup script (`mariadb-dump`, not `sqlcmd`) updated accordingly. See `aidlc-state.md` § Operations for the full record, including a confirmed MariaDB-compatibility bug in Oracle's provider (`AcquireDatabaseLock`) and its workaround (`NonLockingMySQLHistoryRepository`). |
|
||||
| ASM-05 | The existing Umami instance at `analytics.slpsoftware.nl` remains available and its script origin can be added to the CSP. | FR-16, FR-18. | Umami setup gains host work, as in the reference project. |
|
||||
| ASM-06 | One Sentry project with environment tags is acceptable for both backend and frontend events of this CMS. | D-19. | Split into more projects; only configuration changes. |
|
||||
| ASM-07 | Existing Gitea secrets for the Pi (`PI_MAIN_*` in the reference project) can be reused or replicated for this repository. | FR-02. | New secrets are created; documented in FR-23. |
|
||||
|
||||
+32
-18
@@ -20,8 +20,9 @@ assumes already exists — `deploy-scp.yaml` (U6) never creates any of it.
|
||||
### 1.1 Prerequisites
|
||||
- .NET 10 runtime installed on the Pi (ASM-03) — the publish is framework-dependent
|
||||
(`infrastructure-design.md` § 1), so the Pi needs the runtime, not the full SDK
|
||||
- `sqlcmd` installed, for the backup script (§ 4) — e.g. `mssql-tools18` / `unixodbc` on Debian-based
|
||||
Raspberry Pi OS
|
||||
- `mariadb-client` (or `mariadb-dump`/`mysqldump` specifically) installed, for the backup script (§ 4)
|
||||
— already present on most Raspberry Pi OS images that also run `mariadb-server`; install
|
||||
`mariadb-client` explicitly if the dump tool isn't already there
|
||||
|
||||
### 1.2 Account Model (revised — `webadmin` is not the deploy account)
|
||||
|
||||
@@ -106,7 +107,7 @@ below):
|
||||
```ini
|
||||
ASPNETCORE_ENVIRONMENT=Production
|
||||
ASPNETCORE_URLS=http://localhost:<port>
|
||||
ConnectionStrings__DefaultConnection=Server=127.0.0.1,1433;User ID=<user>;Password=<password>;Database=SlpSoftware<Env>;TrustServerCertificate=True
|
||||
ConnectionStrings__DefaultConnection=Server=127.0.0.1;Port=3306;Database=SlpSoftware<Env>;Uid=<user>;Pwd=<password>
|
||||
JwtSettings__Secret=<secure-long-random-secret>
|
||||
JwtSettings__Issuer=SlpModularCms
|
||||
JwtSettings__Audience=SlpModularCmsPortal
|
||||
@@ -213,14 +214,15 @@ touch ~/.config/slpsoftware-db-backup.env
|
||||
chmod 600 ~/.config/slpsoftware-db-backup.env
|
||||
```
|
||||
```ini
|
||||
DB_SERVER=127.0.0.1,1433
|
||||
DB_HOST=127.0.0.1
|
||||
DB_PORT=3306
|
||||
DB_NAME=SlpSoftwareProduction
|
||||
DB_USER=<a-login-with-backup-database-permission>
|
||||
DB_USER=<a-login-with-just-SELECT-LOCK-TABLES-permission>
|
||||
DB_PASSWORD=<password>
|
||||
```
|
||||
Kept **separate** from `shared/env` (§ 1.5) deliberately — the backup script needs its own
|
||||
credential, ideally scoped to just `BACKUP DATABASE` permission rather than the application's own
|
||||
data-access login.
|
||||
credential, ideally scoped to just read access (`SELECT`, `LOCK TABLES` — everything `mariadb-dump`
|
||||
needs) rather than the application's own data-access login.
|
||||
|
||||
### 1.9 Gitea Actions Variables and Secrets
|
||||
Set once, in this repository's Gitea Actions settings. This is exactly where the real path lives —
|
||||
@@ -286,22 +288,29 @@ if [[ ! -f "$CREDENTIALS_FILE" ]]; then
|
||||
fi
|
||||
# shellcheck source=/dev/null
|
||||
source "$CREDENTIALS_FILE"
|
||||
: "${DB_SERVER:?}" "${DB_NAME:?}" "${DB_USER:?}" "${DB_PASSWORD:?}"
|
||||
: "${DB_HOST:?}" "${DB_PORT:?}" "${DB_NAME:?}" "${DB_USER:?}" "${DB_PASSWORD:?}"
|
||||
|
||||
BACKUP_DIR="$HOME/backups/slpsoftware/${ENVIRONMENT}"
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
TIMESTAMP=$(date -u +%Y%m%d%H%M%S)
|
||||
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}-${TIMESTAMP}.bak"
|
||||
BACKUP_FILE="$BACKUP_DIR/${DB_NAME}-${TIMESTAMP}.sql.gz"
|
||||
|
||||
sqlcmd -S "$DB_SERVER" -U "$DB_USER" -P "$DB_PASSWORD" -C -Q \
|
||||
"BACKUP DATABASE [$DB_NAME] TO DISK = N'$BACKUP_FILE' WITH INIT, COMPRESSION"
|
||||
# --single-transaction: consistent snapshot without locking the tables for the whole dump duration
|
||||
# (InnoDB only — every table here is, since that's EF Core's MySQL-provider default).
|
||||
mariadb-dump \
|
||||
-h "$DB_HOST" -P "$DB_PORT" -u "$DB_USER" -p"$DB_PASSWORD" \
|
||||
--single-transaction --routines --triggers \
|
||||
"$DB_NAME" | gzip > "$BACKUP_FILE"
|
||||
|
||||
echo "Backup written to $BACKUP_FILE"
|
||||
|
||||
# Retention: keep the 7 most recent backups for this environment
|
||||
ls -1t "$BACKUP_DIR"/*.bak 2>/dev/null | tail -n +8 | xargs -r rm -f
|
||||
ls -1t "$BACKUP_DIR"/*.sql.gz 2>/dev/null | tail -n +8 | xargs -r rm -f
|
||||
```
|
||||
|
||||
`mariadb-dump` is MariaDB's own name for the tool (present since MariaDB 10.4-ish); if the host only
|
||||
has the older `mysqldump` name, substitute it — same tool, same flags.
|
||||
|
||||
```bash
|
||||
chmod +x ~/scripts/backup-slpmodularcms-db.sh
|
||||
```
|
||||
@@ -310,9 +319,13 @@ chmod +x ~/scripts/backup-slpmodularcms-db.sh
|
||||
```bash
|
||||
~/scripts/backup-slpmodularcms-db.sh production
|
||||
```
|
||||
Confirm a `.bak` file appears under `~/backups/slpsoftware/production/` and that `sqlcmd` didn't
|
||||
silently fail (the script uses `set -euo pipefail`, so a real SQL error does propagate as a non-zero
|
||||
exit — which fails the calling `deploy-scp.yaml` step, correctly blocking the deploy).
|
||||
Confirm a `.sql.gz` file appears under `~/backups/slpsoftware/production/` and that the dump didn't
|
||||
silently fail (the script uses `set -euo pipefail`, so a real error does propagate as a non-zero
|
||||
exit — which fails the calling `deploy-scp.yaml` step, correctly blocking the deploy). Worth a
|
||||
one-time restore rehearsal too — an untested backup is not a verified one:
|
||||
```bash
|
||||
gunzip -c ~/backups/slpsoftware/production/<file>.sql.gz | mariadb -h 127.0.0.1 -u root -p <a-scratch-database>
|
||||
```
|
||||
|
||||
## 5. Future: Switching Production to FTPS (Shared Hosting)
|
||||
|
||||
@@ -336,9 +349,10 @@ than the transport:
|
||||
`wwwroot/web/` persistence (FR-08, ASM-01) would need a **different** mechanism on such a host —
|
||||
e.g. never touching a specific subfolder during upload, rather than linking a persistent directory
|
||||
outside a swapped release tree, since "outside the release tree" may not be an available concept
|
||||
- **Database backup** — shared hosting frequently does not expose direct `sqlcmd`/SSH access at all;
|
||||
the backup step in `deploy-scp.yaml` (§ 4 script) would need to become either a provider-specific
|
||||
API call or a documented manual pre-production step (the FR-20 fallback U6 already designed for)
|
||||
- **Database backup** — shared hosting frequently does not expose direct `mariadb-dump`/SSH access
|
||||
at all; the backup step in `deploy-scp.yaml` (§ 4 script) would need to become either a
|
||||
provider-specific API call (e.g. a hosting-panel database backup feature) or a documented manual
|
||||
pre-production step (the FR-20 fallback U6 already designed for)
|
||||
|
||||
### 5.3 What building `deploy-ftps.yaml` would actually require
|
||||
1. A new reusable workflow implementing the **same five required inputs**
|
||||
|
||||
+1
-2
@@ -46,8 +46,7 @@ A backup is taken before every **production** deploy (`operations/deployment/dep
|
||||
§ 4, invoked by `deploy-scp.yaml` when `run_db_backup: true`). To restore:
|
||||
|
||||
```bash
|
||||
sqlcmd -S <server> -U <user> -P <password> -C -Q \
|
||||
"RESTORE DATABASE [SlpModularCmsProduction] FROM DISK = N'<path-to-backup>.bak' WITH REPLACE"
|
||||
gunzip -c <path-to-backup>.sql.gz | mariadb -h <server> -u <user> -p SlpSoftwareProduction
|
||||
```
|
||||
|
||||
Restoring a database backup and rolling back the application release are **independent actions** —
|
||||
|
||||
Reference in New Issue
Block a user