Website: redirect-map voor oude URLs (prerendered sites) #36

Merged
Sluijsens merged 1 commits from feature/website-redirect-map into master 2026-11-18 21:47:00 +01:00
7 changed files with 871 additions and 3 deletions
+1
View File
@@ -377,6 +377,7 @@ In productie moeten gevoelige instellingen worden doorgegeven via Environment Va
- `SecurityHeaders__AllowedScriptOrigins__0`, `SecurityHeaders__AllowedConnectOrigins__0`, ... — externe origins die de CSP van `/admin` en `/api/v1` mag toestaan (bijv. de Umami-scripthost en Sentry's ingest-endpoint). Zie `appsettings.json → SecurityHeaders` voor de volledige optiesvorm; een onbekende `PathPolicies`-policy-naam laat de applicatie bij **opstarten** falen, niet pas bij de eerste request - `SecurityHeaders__AllowedScriptOrigins__0`, `SecurityHeaders__AllowedConnectOrigins__0`, ... — externe origins die de CSP van `/admin` en `/api/v1` mag toestaan (bijv. de Umami-scripthost en Sentry's ingest-endpoint). Zie `appsettings.json → SecurityHeaders` voor de volledige optiesvorm; een onbekende `PathPolicies`-policy-naam laat de applicatie bij **opstarten** falen, niet pas bij de eerste request
- **Voorgerenderde website (`Website:Mode=PrerenderedPages`)** — een client kiest dit in zijn `Program.cs` via `CmsHostOptions` (standaard `SpaFallback`: alles blijft zoals het was). Instellingen voor deze modus: `Website__NoIndex=true` (testinstantie: `X-Robots-Tag` plus `robots.txt` met `Disallow: /`), `SecurityHeaders__DefaultPolicy=StrictSite` met `SecurityHeaders__AllowedSiteScriptOrigins__0`, `SecurityHeaders__AllowedSiteConnectOrigins__0`, ... (alleen gebruikt door `StrictSite`). Logging van de website (301's en 404's op `Information`) gaat via de logger-categorie `SlpModularCms.Website`: zet per omgeving `Website__LogLevel` (test `Information`, standaard en productie `Warning`; leeg = `Logging:LogLevel` beslist). Gebruik niet `Logging__LogLevel__SlpModularCms.Website` in een systemd-`EnvironmentFile`: een variabelenaam met een punt wordt daar genegeerd. Gedrag per pad staat in [`WEBSITE_WORKSPACE.md`](WEBSITE_WORKSPACE.md). - **Voorgerenderde website (`Website:Mode=PrerenderedPages`)** — een client kiest dit in zijn `Program.cs` via `CmsHostOptions` (standaard `SpaFallback`: alles blijft zoals het was). Instellingen voor deze modus: `Website__NoIndex=true` (testinstantie: `X-Robots-Tag` plus `robots.txt` met `Disallow: /`), `SecurityHeaders__DefaultPolicy=StrictSite` met `SecurityHeaders__AllowedSiteScriptOrigins__0`, `SecurityHeaders__AllowedSiteConnectOrigins__0`, ... (alleen gebruikt door `StrictSite`). Logging van de website (301's en 404's op `Information`) gaat via de logger-categorie `SlpModularCms.Website`: zet per omgeving `Website__LogLevel` (test `Information`, standaard en productie `Warning`; leeg = `Logging:LogLevel` beslist). Gebruik niet `Logging__LogLevel__SlpModularCms.Website` in een systemd-`EnvironmentFile`: een variabelenaam met een punt wordt daar genegeerd. Gedrag per pad staat in [`WEBSITE_WORKSPACE.md`](WEBSITE_WORKSPACE.md).
- **Redirect-map voor oude URL's (alleen `PrerenderedPages`)** - een website levert `redirects.json` in de root van de websitemap (`wwwroot/web/redirects.json`): `{"version":1,"redirects":[{"from":"/oud","to":"/nieuw","status":301},{"from":"/weg","status":410}]}`. Alleen exacte paden (hoofdletter- en slot-slash-ongevoelig, geen query-string- of patroonregels), status 301/404/410 (404 en 410 geven de eigen `404.html` als body), maximaal 5000 regels en 1 MB, geen dubbelen of kringen; ketens worden afgevlakt. De regel gaat vooraf aan de gewone padherleiding en is dus op test en productie gelijk. Het bestand wordt hooguit elke 5 seconden opnieuw gecontroleerd (wijzigingstijd en lengte); een ongeldig nieuw bestand laat de vorige regels staan en geeft een waarschuwing in het log (categorie `SlpModularCms.Website`). `GET /redirects.json` is een 404. Ontwerp: [aidlc-docs/features/andreaslop-site-hosting/redirect-map-design.md](aidlc-docs/features/andreaslop-site-hosting/redirect-map-design.md).
### 2a. Data Protection key ring (Master CMS Module) ### 2a. Data Protection key ring (Master CMS Module)
De API keys van geregistreerde slaves worden versleuteld opgeslagen met ASP.NET Core Data Protection. De key ring wordt **automatisch** persistent opgeslagen in de database (`PersistKeysToDbContext`, met een expliciete, stabiele applicatie-discriminator) — dit hoeft niet meer apart geconfigureerd te worden. Dit is bewust zo gebouwd omdat de atomische release-switch (zie hierboven) bij elke deploy een nieuwe content-root-map gebruikt: zonder een database-backed key ring zou dat elke keer de key ring weggooien en alle opgeslagen slave-API-keys onleesbaar maken. De API keys van geregistreerde slaves worden versleuteld opgeslagen met ASP.NET Core Data Protection. De key ring wordt **automatisch** persistent opgeslagen in de database (`PersistKeysToDbContext`, met een expliciete, stabiele applicatie-discriminator) — dit hoeft niet meer apart geconfigureerd te worden. Dit is bewust zo gebouwd omdat de atomische release-switch (zie hierboven) bij elke deploy een nieuwe content-root-map gebruikt: zonder een database-backed key ring zou dat elke keer de key ring weggooien en alle opgeslagen slave-API-keys onleesbaar maken.
@@ -0,0 +1,424 @@
using System.Net;
using System.Text;
using FluentAssertions;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using SlpModularCms.Core.Hosting;
using SlpModularCms.Core.Hosting.Security;
using SlpModularCms.Core.Hosting.Website;
using Xunit;
namespace SlpModularCms.Core.Tests.Hosting.Website;
public class RedirectMapParserTests
{
private static RedirectMapParseResult Parse(string json) => RedirectMapParser.Parse(Encoding.UTF8.GetBytes(json));
private static string Doc(params string[] rules) => "{\"version\":1,\"redirects\":[" + string.Join(",", rules) + "]}";
private static string R(string from, int status, string? to = null)
=> "{\"from\":\"" + from + "\"" + (to is null ? "" : ",\"to\":\"" + to + "\"") + ",\"status\":" + status + "}";
[Fact]
public void AValidFile_GivesRules_KeyedOnTheNormalisedPath()
{
var result = Parse(Doc(R("/Over-Mij-2/", 301, "/over-mij"), R("/feed", 410), R("/x.jpg", 404)));
result.Errors.Should().BeEmpty();
result.Rules!.Keys.Should().BeEquivalentTo("/over-mij-2", "/feed", "/x.jpg");
result.Rules["/over-mij-2"].Should().Be(new RedirectRule(301, "/over-mij"));
result.Rules["/feed"].Should().Be(new RedirectRule(410, null));
}
[Fact]
public void AnEmptyList_IsValid() => Parse(Doc()).Rules.Should().BeEmpty();
[Theory]
[InlineData("{\"version\":2,\"redirects\":[]}")]
[InlineData("{\"redirects\":[]}")]
[InlineData("{\"version\":1}")]
[InlineData("{\"version\":1,\"redirects\":{}}")]
[InlineData("not json")]
[InlineData("[]")]
public void ABrokenDocument_IsInvalid(string json) => Parse(json).Rules.Should().BeNull();
[Theory]
[InlineData("/a", 302, "/b")]
[InlineData("/a", 200, "/b")]
[InlineData("/a", 301, null)]
[InlineData("/a", 410, "/b")]
[InlineData("/a", 404, "/b")]
public void StatusAndTarget_MustMatch(string from, int status, string? to)
=> Parse(Doc(R(from, status, to))).Rules.Should().BeNull();
[Theory]
[InlineData("a")]
[InlineData("//a")]
[InlineData("/a//b")]
[InlineData("/a/../b")]
[InlineData("/a/./b")]
[InlineData("/a?x=1")]
[InlineData("/a#top")]
[InlineData("/")]
[InlineData("/admin/x")]
[InlineData("/ADMIN")]
[InlineData("/api/v1/x")]
[InlineData("/health")]
[InlineData("/%2e%2e/x")]
public void AnInvalidFrom_IsRejected(string from) => Parse(Doc(R(from, 410))).Rules.Should().BeNull();
[Theory]
[InlineData("b")]
[InlineData("//evil.example/x")]
[InlineData("/b?x=1")]
[InlineData("http://example.com/b")]
[InlineData("/b c")]
[InlineData("/b\\\\c")]
[InlineData("/café")]
public void AnInvalidTarget_IsRejected(string to) => Parse(Doc(R("/a", 301, to))).Rules.Should().BeNull();
[Fact]
public void AnAbsoluteHttpsTarget_IsAllowed()
=> Parse(Doc(R("/a", 301, "https://example.com/b"))).Rules!["/a"].To.Should().Be("https://example.com/b");
[Fact]
public void Duplicates_AreRejected_AlsoWhenOnlyCaseOrTrailingSlashDiffers()
{
Parse(Doc(R("/a", 410), R("/a", 404))).Rules.Should().BeNull();
Parse(Doc(R("/a", 410), R("/A/", 410))).Rules.Should().BeNull();
}
[Fact]
public void FromEqualToTo_IsRejected() => Parse(Doc(R("/a", 301, "/A/"))).Rules.Should().BeNull();
[Fact]
public void AChain_IsFlattened()
{
var result = Parse(Doc(R("/a", 301, "/b"), R("/b", 301, "/c"), R("/c", 301, "https://example.com/z")));
result.Rules!["/a"].To.Should().Be("https://example.com/z");
result.Rules["/b"].To.Should().Be("https://example.com/z");
}
[Fact]
public void ACircle_IsRejected() => Parse(Doc(R("/a", 301, "/b"), R("/b", 301, "/a"))).Rules.Should().BeNull();
[Fact]
public void ATargetThatIsGone_IsRejected() => Parse(Doc(R("/a", 301, "/b"), R("/b", 410))).Rules.Should().BeNull();
[Fact]
public void AMissingAnchor_InChainOverTheLimit_IsRejected()
{
var rules = Enumerable.Range(0, RedirectMapParser.MaxChain + 3).Select(i => R("/p" + i, 301, "/p" + (i + 1))).ToArray();
Parse(Doc(rules)).Rules.Should().BeNull();
}
[Fact]
public void TooManyRules_AreRejected()
{
var rules = Enumerable.Range(0, RedirectMapParser.MaxRules + 1).Select(i => R("/p" + i, 410)).ToArray();
Parse(Doc(rules)).Rules.Should().BeNull();
Parse(Doc(rules[..RedirectMapParser.MaxRules])).Rules.Should().HaveCount(RedirectMapParser.MaxRules);
}
[Fact]
public void ATooBigFile_IsRejected() => RedirectMapParser.Parse(new byte[RedirectMapParser.MaxBytes + 1]).Rules.Should().BeNull();
[Fact]
public void PercentEncodedAndUnicodePaths_MeetInTheSameKey()
{
var result = Parse(Doc(R("/caf%C3%A9", 410)));
result.Rules!.Keys.Should().Contain(RedirectMapParser.Normalize("/café"));
}
}
public sealed class WebsiteRedirectMapReloadTests : IDisposable
{
private readonly string _root = Path.Combine(Path.GetTempPath(), "slp-redirects-" + Guid.NewGuid().ToString("N"));
private readonly TestClock _clock = new();
public WebsiteRedirectMapReloadTests() => Directory.CreateDirectory(_root);
public void Dispose() => Directory.Delete(_root, recursive: true);
private sealed class TestClock : TimeProvider
{
private long _ticks;
public override long GetTimestamp() => _ticks;
public override long TimestampFrequency => TimeSpan.TicksPerSecond;
public void Advance(TimeSpan by) => _ticks += by.Ticks;
}
private void Put(string json, DateTime? modified = null)
{
var path = Path.Combine(_root, WebsiteRedirectMap.FileName);
File.WriteAllText(path, json);
File.SetLastWriteTimeUtc(path, modified ?? DateTime.UtcNow.AddSeconds(Random.Shared.Next(1, 100000)));
}
private WebsiteRedirectMap Map() => new(_root, NullLogger.Instance, _clock);
private const string One = "{\"version\":1,\"redirects\":[{\"from\":\"/a\",\"to\":\"/b\",\"status\":301}]}";
private const string Two = "{\"version\":1,\"redirects\":[{\"from\":\"/c\",\"status\":410}]}";
[Fact]
public void WithoutAFile_NothingIsRedirected() => Map().Find("/a").Should().BeNull();
[Fact]
public void ARule_IsFound_CaseAndSlashInsensitive()
{
Put(One);
var map = Map();
map.Find("/a").Should().Be(new RedirectRule(301, "/b"));
map.Find("/A/").Should().Be(new RedirectRule(301, "/b"));
map.Find("/").Should().BeNull();
map.Find("/other").Should().BeNull();
}
[Fact]
public void AChangedFile_IsPickedUp_AfterTheInterval_NotBefore()
{
Put(One);
var map = Map();
map.Find("/a").Should().NotBeNull();
Put(Two);
_clock.Advance(TimeSpan.FromSeconds(1));
map.Find("/a").Should().NotBeNull("the file is only looked at again after the interval");
_clock.Advance(WebsiteRedirectMap.RecheckInterval);
map.Find("/a").Should().BeNull();
map.Find("/c").Should().Be(new RedirectRule(410, null));
}
[Fact]
public void AnInvalidNewFile_KeepsThePreviousRules()
{
Put(One);
var map = Map();
map.Find("/a").Should().NotBeNull();
Put("{ broken");
_clock.Advance(WebsiteRedirectMap.RecheckInterval);
map.Find("/a").Should().Be(new RedirectRule(301, "/b"));
}
[Fact]
public void ARemovedFile_EmptiesTheRules()
{
Put(One);
var map = Map();
map.Find("/a").Should().NotBeNull();
File.Delete(Path.Combine(_root, WebsiteRedirectMap.FileName));
_clock.Advance(WebsiteRedirectMap.RecheckInterval);
map.Find("/a").Should().BeNull();
}
[Fact]
public void AFileThatAppearsLater_IsPickedUp()
{
var map = Map();
map.Find("/a").Should().BeNull();
Put(One);
_clock.Advance(WebsiteRedirectMap.RecheckInterval);
map.Find("/a").Should().NotBeNull();
}
[Fact]
public void AnInvalidFirstFile_MeansNoRules() { Put("{ broken"); Map().Find("/a").Should().BeNull(); }
}
public sealed class WebsiteRedirectMapIntegrationTests : IDisposable
{
private readonly string _root = Path.Combine(Path.GetTempPath(), "slp-redirect-site-" + Guid.NewGuid().ToString("N"));
private readonly List<WebApplication> _apps = [];
public WebsiteRedirectMapIntegrationTests()
{
Write("index.html", "<!doctype html><title>Home</title>");
Write("over-mij.html", "<!doctype html><title>Over mij</title>");
Write("portfolio.html", "<!doctype html><title>Portfolio</title>");
Write("404.html", "<!doctype html><title>Niet gevonden</title>");
Write(WebsiteRedirectMap.FileName,
"{\"version\":1,\"redirects\":["
+ "{\"from\":\"/portfolio-item/decoration\",\"to\":\"/portfolio\",\"status\":301},"
+ "{\"from\":\"/feed\",\"status\":410},"
+ "{\"from\":\"/gone\",\"status\":404},"
+ "{\"from\":\"/over-mij\",\"to\":\"/portfolio\",\"status\":301}]}");
}
public void Dispose()
{
foreach (var app in _apps)
{
app.DisposeAsync().AsTask().GetAwaiter().GetResult();
}
Directory.Delete(_root, recursive: true);
}
private void Write(string relative, string content)
{
var path = Path.Combine(_root, "wwwroot", "web", relative);
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
File.WriteAllText(path, content);
}
private async Task<HttpClient> StartAsync(bool noIndex, bool prerendered = true)
{
Directory.CreateDirectory(Path.Combine(_root, "wwwroot"));
var builder = WebApplication.CreateBuilder(new WebApplicationOptions
{
EnvironmentName = Environments.Production,
ContentRootPath = _root,
WebRootPath = Path.Combine(_root, "wwwroot")
});
builder.WebHost.UseTestServer();
builder.Logging.ClearProviders();
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
{
["SecurityHeaders:DefaultPolicy"] = "StrictSite",
["Website:NoIndex"] = noIndex ? "true" : "false"
});
builder.Services.AddCmsSecurityHeaders(builder.Configuration);
if (prerendered)
{
builder.Services.AddCmsWebsite(builder.Configuration);
}
var app = builder.Build();
var mode = prerendered ? WebsiteMode.PrerenderedPages : WebsiteMode.SpaFallback;
app.UseCmsSecurityHeaders();
app.UseCmsStaticContent(mode);
if (prerendered)
{
app.UseCmsPrerenderedSite();
}
app.MapGet("/api/v1/ping", () => Results.Text("pong"));
app.MapPost("/portfolio-item/decoration", () => Results.Text("posted"));
app.MapCmsSpaFallbacks(mode);
if (prerendered)
{
app.MapCmsWebsiteNotFound();
}
await app.StartAsync();
_apps.Add(app);
return app.GetTestClient();
}
[Theory]
[InlineData(true)]
[InlineData(false)]
public async Task A301Rule_Redirects_WithLocationAndCache_OnTestAndProduction(bool noIndex)
{
var client = await StartAsync(noIndex);
foreach (var path in new[] { "/portfolio-item/decoration", "/portfolio-item/decoration/", "/Portfolio-Item/Decoration/?utm=x" })
{
var response = await client.GetAsync(path);
response.StatusCode.Should().Be(HttpStatusCode.MovedPermanently, path);
response.Headers.Location!.OriginalString.Should().Be("/portfolio", path);
response.Headers.CacheControl!.MaxAge.Should().Be(TimeSpan.FromHours(1));
}
}
[Fact]
public async Task A410Rule_AnswersWithThe404Page_AndTheStatus()
{
var client = await StartAsync(noIndex: false);
var response = await client.GetAsync("/feed");
response.StatusCode.Should().Be(HttpStatusCode.Gone);
(await response.Content.ReadAsStringAsync()).Should().Contain("Niet gevonden");
response.Headers.CacheControl!.NoCache.Should().BeTrue();
}
[Fact]
public async Task A404Rule_AnswersWithThe404Page()
{
var client = await StartAsync(noIndex: false);
var response = await client.GetAsync("/gone");
response.StatusCode.Should().Be(HttpStatusCode.NotFound);
(await response.Content.ReadAsStringAsync()).Should().Contain("Niet gevonden");
}
[Fact]
public async Task Head_Works_AndPost_IsNotRedirected()
{
var client = await StartAsync(noIndex: false);
var head = await client.SendAsync(new HttpRequestMessage(HttpMethod.Head, "/feed"));
head.StatusCode.Should().Be(HttpStatusCode.Gone);
(await head.Content.ReadAsStringAsync()).Should().BeEmpty();
var post = await client.PostAsync("/portfolio-item/decoration", new StringContent(""));
post.StatusCode.Should().Be(HttpStatusCode.OK);
(await post.Content.ReadAsStringAsync()).Should().Be("posted");
}
[Fact]
public async Task ARule_BeatsAnExistingPage()
{
var client = await StartAsync(noIndex: false);
var response = await client.GetAsync("/over-mij");
response.StatusCode.Should().Be(HttpStatusCode.MovedPermanently);
}
[Fact]
public async Task APathWithoutARule_IsHandledAsBefore()
{
var client = await StartAsync(noIndex: false);
(await client.GetAsync("/portfolio")).StatusCode.Should().Be(HttpStatusCode.OK);
(await client.GetAsync("/unknown")).StatusCode.Should().Be(HttpStatusCode.NotFound);
(await client.GetAsync("/api/v1/ping")).StatusCode.Should().Be(HttpStatusCode.OK);
}
[Theory]
[InlineData("/redirects.json")]
[InlineData("/Redirects.json")]
public async Task TheMapItself_IsNotServed(string path)
{
var client = await StartAsync(noIndex: false);
var response = await client.GetAsync(path);
response.StatusCode.Should().Be(HttpStatusCode.NotFound);
(await response.Content.ReadAsStringAsync()).Should().NotContain("redirects");
}
[Fact]
public async Task InSpaMode_TheFileDoesNothing()
{
var client = await StartAsync(noIndex: false, prerendered: false);
var response = await client.GetAsync("/feed");
response.StatusCode.Should().NotBe(HttpStatusCode.Gone);
response.StatusCode.Should().NotBe(HttpStatusCode.MovedPermanently);
}
}
@@ -57,6 +57,12 @@ public static class PrerenderedSiteExtensions
provider.GetRequiredService<ILoggerFactory>().CreateLogger(PrerenderedSiteMiddleware.LoggerCategory)); provider.GetRequiredService<ILoggerFactory>().CreateLogger(PrerenderedSiteMiddleware.LoggerCategory));
}); });
// Old URLs of a previous site (redirects.json in the website directory): 301, 404 or 410 per exact path.
services.AddSingleton(provider => new WebsiteRedirectMap(
provider.GetRequiredService<WebsiteFileAccess>().RootPath,
provider.GetRequiredService<ILoggerFactory>().CreateLogger(PrerenderedSiteMiddleware.LoggerCategory),
provider.GetService<TimeProvider>()));
// Lets the catch-all 404 route refuse the reserved prefixes (see NotReservedPathConstraint). // Lets the catch-all 404 route refuse the reserved prefixes (see NotReservedPathConstraint).
services.Configure<RouteOptions>(options => services.Configure<RouteOptions>(options =>
options.ConstraintMap[NotReservedPathConstraint.Name] = typeof(NotReservedPathConstraint)); options.ConstraintMap[NotReservedPathConstraint.Name] = typeof(NotReservedPathConstraint));
@@ -31,6 +31,7 @@ public sealed class PrerenderedSiteMiddleware
private readonly RequestDelegate _staticFiles; private readonly RequestDelegate _staticFiles;
private readonly WebsiteFileAccess _files; private readonly WebsiteFileAccess _files;
private readonly HtmlMarkerInjector? _injector; private readonly HtmlMarkerInjector? _injector;
private readonly WebsiteRedirectMap? _redirects;
private readonly bool _noIndex; private readonly bool _noIndex;
private readonly ILogger _logger; private readonly ILogger _logger;
@@ -40,11 +41,13 @@ public sealed class PrerenderedSiteMiddleware
IOptions<WebsiteOptions> options, IOptions<WebsiteOptions> options,
ILoggerFactory loggerFactory, ILoggerFactory loggerFactory,
WebsiteFileAccess files, WebsiteFileAccess files,
HtmlMarkerInjector? injector = null) HtmlMarkerInjector? injector = null,
WebsiteRedirectMap? redirects = null)
{ {
_next = next; _next = next;
_files = files; _files = files;
_injector = injector; _injector = injector;
_redirects = redirects;
_noIndex = options.Value.NoIndex; _noIndex = options.Value.NoIndex;
_logger = loggerFactory.CreateLogger(LoggerCategory); _logger = loggerFactory.CreateLogger(LoggerCategory);
@@ -107,6 +110,26 @@ public sealed class PrerenderedSiteMiddleware
return; return;
} }
// The redirect map (old URLs of a previous site) goes first: an explicit rule beats a file of the same name.
if (_redirects?.Find(request.Path.Value) is { } rule)
{
var from = Sanitize(request.Path.Value);
if (rule.Status == StatusCodes.Status301MovedPermanently)
{
_logger.LogInformation("Website redirect-map 301 {From} -> {To}", from, Sanitize(rule.To));
context.Response.StatusCode = StatusCodes.Status301MovedPermanently;
context.Response.Headers[HeaderNames.Location] = rule.To;
context.Response.Headers[HeaderNames.CacheControl] = "public, max-age=3600";
}
else
{
_logger.LogInformation("Website redirect-map {Status} {From}", rule.Status, from);
await WebsiteNotFoundHandler.WritePageAsync(context, _files, _logger, rule.Status);
}
return;
}
var resolution = WebsitePathResolver.Resolve(request.Path.Value, request.QueryString.Value, _files.Exists); var resolution = WebsitePathResolver.Resolve(request.Path.Value, request.QueryString.Value, _files.Exists);
switch (resolution) switch (resolution)
@@ -48,8 +48,17 @@ public sealed class WebsiteFileAccess : IFileProvider, IDisposable
} }
} }
/// <summary>The directory the website is served from (also where <see cref="WebsiteRedirectMap.FileName"/> lives).</summary>
public string RootPath => _root;
public IFileInfo GetFileInfo(string subpath) public IFileInfo GetFileInfo(string subpath)
{ {
// The redirect map is data for the host, not a page of the site: it is never served to visitors.
if (string.Equals(subpath.Trim('/'), WebsiteRedirectMap.FileName, StringComparison.OrdinalIgnoreCase))
{
return new NotFoundFileInfo(subpath);
}
var provider = GetProvider(); var provider = GetProvider();
return provider is null ? new NotFoundFileInfo(subpath) : provider.GetFileInfo(subpath); return provider is null ? new NotFoundFileInfo(subpath) : provider.GetFileInfo(subpath);
} }
@@ -36,9 +36,15 @@ public static class WebsiteNotFoundHandler
logger.LogInformation("Website 404 {Path}", PrerenderedSiteMiddleware.Sanitize(request.Path.Value)); logger.LogInformation("Website 404 {Path}", PrerenderedSiteMiddleware.Sanitize(request.Path.Value));
context.Response.StatusCode = StatusCodes.Status404NotFound; await WritePageAsync(context, context.RequestServices.GetRequiredService<WebsiteFileAccess>(), logger, StatusCodes.Status404NotFound);
}
/// <summary>Answers with <paramref name="statusCode"/> and, when the site has one, the body of its <c>404.html</c> (also used for 410 by the redirect map).</summary>
internal static async Task WritePageAsync(HttpContext context, WebsiteFileAccess files, ILogger logger, int statusCode)
{
var request = context.Request;
context.Response.StatusCode = statusCode;
var files = context.RequestServices.GetRequiredService<WebsiteFileAccess>();
var page = files.GetFileInfo("404.html"); var page = files.GetFileInfo("404.html");
if (!page.Exists || page.IsDirectory) if (!page.Exists || page.IsDirectory)
@@ -0,0 +1,399 @@
using System.Text;
using System.Text.Json;
using Microsoft.Extensions.Logging;
namespace SlpModularCms.Core.Hosting.Website;
/// <summary>One rule of the redirect map: a status (301, 404 or 410) and, for 301, where to.</summary>
public sealed record RedirectRule(int Status, string? To);
/// <summary>What <see cref="RedirectMapParser.Parse"/> found: the rules (null when the file is invalid) and the problems.</summary>
public sealed record RedirectMapParseResult(IReadOnlyDictionary<string, RedirectRule>? Rules, IReadOnlyList<string> Errors);
/// <summary>
/// Reads and validates <c>redirects.json</c> (design: aidlc-docs/features/andreaslop-site-hosting/redirect-map-design.md):
/// <c>{ "version": 1, "redirects": [ { "from": "/a", "to": "/b", "status": 301 }, { "from": "/c", "status": 410 } ] }</c>.
/// Pure: no I/O, no clock, never throws on bad input (problems are returned).
/// </summary>
public static class RedirectMapParser
{
public const int MaxRules = 5000;
public const int MaxBytes = 1024 * 1024;
public const int MaxPathLength = 2048;
public const int MaxChain = 10;
private const int MaxReportedErrors = 50;
/// <summary>The key a path is looked up under: Unicode-normalised, lower case, without a trailing slash (except "/").</summary>
public static string Normalize(string path)
{
var normalized = path.Normalize(NormalizationForm.FormC).ToLowerInvariant();
return normalized.Length > 1 && normalized[^1] == '/' ? normalized[..^1] : normalized;
}
public static RedirectMapParseResult Parse(byte[] json)
{
var errors = new List<string>();
if (json.Length > MaxBytes)
{
return Invalid($"the file is larger than {MaxBytes} bytes");
}
JsonDocument document;
try
{
document = JsonDocument.Parse(json);
}
catch (JsonException ex)
{
return Invalid("not valid JSON: " + ex.Message);
}
using (document)
{
var root = document.RootElement;
if (root.ValueKind != JsonValueKind.Object
|| !root.TryGetProperty("version", out var version)
|| version.ValueKind != JsonValueKind.Number
|| !version.TryGetInt32(out var versionNumber)
|| versionNumber != 1)
{
return Invalid("\"version\" must be 1");
}
if (!root.TryGetProperty("redirects", out var list) || list.ValueKind != JsonValueKind.Array)
{
return Invalid("\"redirects\" must be an array");
}
if (list.GetArrayLength() > MaxRules)
{
return Invalid($"more than {MaxRules} rules");
}
var rules = new Dictionary<string, RedirectRule>(StringComparer.Ordinal);
var index = -1;
foreach (var item in list.EnumerateArray())
{
index++;
var where = $"redirects[{index}]";
if (item.ValueKind != JsonValueKind.Object)
{
Add($"{where}: not an object");
continue;
}
var from = ReadString(item, "from");
var to = ReadString(item, "to");
var status = item.TryGetProperty("status", out var s) && s.ValueKind == JsonValueKind.Number && s.TryGetInt32(out var code) ? code : 0;
if (status is not (301 or 404 or 410))
{
Add($"{where}: status must be 301, 404 or 410");
continue;
}
var key = string.Empty;
var fromProblem = string.Empty;
var toProblem = string.Empty;
if (from is null || !TryNormalizeFrom(from, out key, out fromProblem))
{
Add($"{where}: \"from\" {(from is null ? "is missing" : fromProblem)}");
continue;
}
if (status == 301)
{
if (to is null || !IsValidTarget(to, out toProblem))
{
Add($"{where}: \"to\" {(to is null ? "is required for 301" : toProblem)}");
continue;
}
if (to.StartsWith('/') && Normalize(DecodeOrSelf(to)) == key)
{
Add($"{where}: \"to\" is the same path as \"from\"");
continue;
}
}
else if (to is not null)
{
Add($"{where}: \"to\" is not allowed for {status}");
continue;
}
if (!rules.TryAdd(key, new RedirectRule(status, to)))
{
Add($"{where}: \"from\" {from} is a duplicate (case and a trailing slash do not count)");
}
}
if (errors.Count > 0)
{
return new RedirectMapParseResult(null, errors);
}
return Flatten(rules);
void Add(string message)
{
if (errors.Count < MaxReportedErrors)
{
errors.Add(message);
}
}
}
static RedirectMapParseResult Invalid(string message) => new(null, [message]);
}
/// <summary>Follows chains so a visitor sees one redirect; a cycle, a chain over <see cref="MaxChain"/> or a target that is gone is an error.</summary>
private static RedirectMapParseResult Flatten(Dictionary<string, RedirectRule> rules)
{
var errors = new List<string>();
var result = new Dictionary<string, RedirectRule>(rules.Count, StringComparer.Ordinal);
foreach (var (key, rule) in rules)
{
if (rule.Status != 301 || rule.To is null || !rule.To.StartsWith('/'))
{
result[key] = rule;
continue;
}
var target = rule.To;
var seen = new HashSet<string> { key };
for (var step = 0; ; step++)
{
var next = Normalize(DecodeOrSelf(target));
if (!rules.TryGetValue(next, out var onward))
{
break;
}
if (!seen.Add(next))
{
errors.Add($"{key}: redirects in a circle");
break;
}
if (onward.Status != 301)
{
errors.Add($"{key}: points at {target}, which is itself answered with {onward.Status}");
break;
}
if (step >= MaxChain)
{
errors.Add($"{key}: more than {MaxChain} redirects in a row");
break;
}
if (!onward.To!.StartsWith('/'))
{
target = onward.To;
break;
}
target = onward.To;
}
result[key] = rule with { To = target };
}
return errors.Count > 0 ? new RedirectMapParseResult(null, errors) : new RedirectMapParseResult(result, []);
}
private static string? ReadString(JsonElement item, string name)
=> item.TryGetProperty(name, out var value) && value.ValueKind == JsonValueKind.String ? value.GetString() : null;
private static string DecodeOrSelf(string value)
{
try
{
return Uri.UnescapeDataString(value);
}
catch (UriFormatException)
{
return value;
}
}
private static bool TryNormalizeFrom(string from, out string key, out string problem)
{
key = string.Empty;
problem = string.Empty;
if (from.Length == 0 || from.Length > MaxPathLength || from[0] != '/')
{
problem = $"must start with a single / and be at most {MaxPathLength} characters";
return false;
}
if (from.Contains('?') || from.Contains('#') || from.Contains('\\'))
{
problem = "must be a path without query string or fragment";
return false;
}
var decoded = DecodeOrSelf(from);
if (decoded.Any(char.IsControl) || decoded.StartsWith("//", StringComparison.Ordinal) || decoded.Contains("//", StringComparison.Ordinal))
{
problem = "has control characters or an empty segment (//)";
return false;
}
if (decoded.Split('/').Any(segment => segment is "." or ".."))
{
problem = "has a . or .. segment";
return false;
}
if (WebsitePathResolver.IsReservedPath(decoded))
{
problem = "starts with a path the host keeps for itself (/admin, /api/v1, /health)";
return false;
}
key = Normalize(decoded);
if (key == "/")
{
problem = "cannot be the home page /";
return false;
}
return true;
}
private static bool IsValidTarget(string to, out string problem)
{
problem = string.Empty;
if (to.Length == 0 || to.Length > MaxPathLength || to.Any(c => c <= ' ' || c > '~'))
{
problem = "must be plain ASCII without spaces or control characters (percent-encode anything else)";
return false;
}
if (to.StartsWith('/'))
{
if (to.StartsWith("//", StringComparison.Ordinal) || to.Contains('\\') || to.Contains('?') || to.Contains('#'))
{
problem = "a path must start with a single / and have no query string or fragment";
return false;
}
return true;
}
if (Uri.TryCreate(to, UriKind.Absolute, out var uri) && uri.Scheme == Uri.UriSchemeHttps && !string.IsNullOrEmpty(uri.Host))
{
return true;
}
problem = "must be a path starting with / or an absolute https:// URL";
return false;
}
}
/// <summary>
/// The redirect map of the website (<c>redirects.json</c> in the root of the website directory), loaded on first use and
/// re-checked at most every <see cref="RecheckInterval"/> by modified time and length (no file watcher: unreliable on
/// symlinks and network drives). A new file replaces the rules only when it is valid; an invalid one keeps the previous
/// rules and is logged once; a removed file means no rules. Thread-safe; the request path costs a dictionary lookup.
/// </summary>
public sealed class WebsiteRedirectMap
{
public const string FileName = "redirects.json";
public static readonly TimeSpan RecheckInterval = TimeSpan.FromSeconds(5);
private readonly string _root;
private readonly ILogger _logger;
private readonly TimeProvider _time;
private readonly Lock _gate = new();
private State _state = new(new Dictionary<string, RedirectRule>(), default, -1, long.MinValue);
public WebsiteRedirectMap(string root, ILogger logger, TimeProvider? time = null)
{
_root = root;
_logger = logger;
_time = time ?? TimeProvider.System;
}
private sealed record State(IReadOnlyDictionary<string, RedirectRule> Rules, DateTime Modified, long Length, long CheckedAt);
/// <summary>The rule for a request path (as ASP.NET exposes it, decoded), or null.</summary>
public RedirectRule? Find(string? path)
{
if (string.IsNullOrEmpty(path) || path[0] != '/' || path == "/")
{
return null;
}
var state = Current();
return state.Rules.Count == 0 ? null : state.Rules.GetValueOrDefault(RedirectMapParser.Normalize(path));
}
private State Current()
{
var state = _state;
var now = _time.GetTimestamp();
if (state.CheckedAt != long.MinValue && _time.GetElapsedTime(state.CheckedAt, now) < RecheckInterval)
{
return state;
}
lock (_gate)
{
state = _state;
now = _time.GetTimestamp();
if (state.CheckedAt != long.MinValue && _time.GetElapsedTime(state.CheckedAt, now) < RecheckInterval)
{
return state;
}
_state = Reload(state, now);
return _state;
}
}
private State Reload(State previous, long now)
{
var path = Path.Combine(_root, FileName);
try
{
var info = new FileInfo(path);
if (!info.Exists)
{
if (previous.Rules.Count > 0)
{
_logger.LogInformation("The website redirect map {File} is gone; no paths are redirected any more.", FileName);
}
return new State(new Dictionary<string, RedirectRule>(), default, -1, now);
}
if (info.LastWriteTimeUtc == previous.Modified && info.Length == previous.Length)
{
return previous with { CheckedAt = now };
}
var bytes = info.Length > RedirectMapParser.MaxBytes ? new byte[RedirectMapParser.MaxBytes + 1] : File.ReadAllBytes(path);
var result = RedirectMapParser.Parse(bytes);
if (result.Rules is null)
{
// Remember this version of the file so the same problem is not logged every few seconds.
_logger.LogWarning(
"The website redirect map {File} is invalid and is ignored (the previous rules stay): {Problems}",
FileName,
string.Join("; ", result.Errors.Take(5)));
return new State(previous.Rules, info.LastWriteTimeUtc, info.Length, now);
}
_logger.LogInformation("The website redirect map {File} was loaded: {Count} rules.", FileName, result.Rules.Count);
return new State(result.Rules, info.LastWriteTimeUtc, info.Length, now);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
// Being written or not readable right now: keep what we have and look again at the next interval.
_logger.LogWarning(ex, "Reading the website redirect map {File} failed; the previous rules stay.", FileName);
return previous with { CheckedAt = now };
}
}
}