feat(seo): server-render sponsor page content for crawlers - #111
Merged
Conversation
Bots previously received the React shell with only <title>/meta swapped — an empty <div id="root"> for all 124k /sponsor/:id pages, which Google treats as thin/duplicate content. Injects real sponsor content (status, licence details, change history, FAQ) inside #root, cached 6h per sponsor and flushed nightly by the monitor job; React replaces it on hydration for real browsers. Adds FAQPage + BreadcrumbList JSON-LD. Reintegrated from a June 2026 branch that never merged. Kept current main's simpler Cache-Control (public, max-age=3600, no stale-while-revalidate) rather than silently reintroducing the value a later main commit (13c5b82) deliberately dropped from this same line. Unrelated to this change: the SubscriptionModal.tsx deletion originally bundled with this work was already done independently on main (3b46e6b). Full suite green, tsc clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ion claim, silent failures) Critical fixes: - JSON-LD script-tag breakout: JSON.stringify() doesn't escape "<", so a register field containing "</script><script>..." (e.g. company name) could close the ld+json block early and inject live markup. Added toSafeJsonLd() (escapes "<" to \u003c) and used it for every JSON-LD block, replacing the raw JSON.stringify() calls. - $-pattern corruption in the #root injection: seoBody was spliced into the replacement *string* passed to html.replace(), where $&/$$/$'/$1 etc. are special substitution tokens — a literal "$" in DB-sourced sponsor data could silently corrupt the injected HTML. Switched to a replacer function. - Silent injection failure: if the #root regex ever stops matching the built template (bundler/whitespace change), replace() was a no-op with no error, and the broken page got cached for 6h — reintroducing the exact thin-content problem this feature exists to fix. Now compares html before/after the replace, logs a warning and skips the cache write when nothing was injected. - isRevoked collapsed GRACE_PERIOD into "revoked": sponsors 1 day into a register gap (status still unconfirmed) were shown "no longer a licensed sponsor... cannot support a UK visa application" — a false, legally-sensitive claim directly contradicting the correctly-hedged statusSentence() text on the same page. isRevoked now only matches REMOVED_REVOKED; GRACE_PERIOD gets its own softer "under review" notice. Also: - Removed duplicated status-label and HTML-escaping logic between seo.ts and sponsorSeoHtml.ts (statusLabel, escapeHtml both now exported and shared; escapeAttr in seo.ts is an alias of the same function). - Extracted the #root injection regex into a shared, exported constant (ROOT_INJECTION_REGEX) instead of duplicating it with a "must stay in sync" comment between seo.ts and its test. - Escaped the changeType fallback in the history list (defense-in-depth; was unescaped raw DB value). - Corrected comments claiming #root is "empty" pre-injection (it holds a generic SSR fallback) and that React "hydrates" it (main.tsx calls createRoot().render(), a plain client-side remount, not hydration). - Added route-level tests for server/routes/seo.ts (previously zero coverage): cache hit/miss, DB-not-found fallthrough, non-bot/non-html bypass, and the JSON-LD escaping fix. - Added sponsorSeoHtml.ts coverage for GRACE_PERIOD/NEWLY_GRANTED statuses, null townCity/route/typeRating/removedAt branches, escaping beyond currentName (townCity, history previousValue/newValue), the changeType fallback, the $-corruption fix, and the JSON-LD script-breakout fix. Verified: tsc clean, lint clean, build clean, 475 tests pass (43 files). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Extract the nested "(removed around ...)" template literal into a removedClause variable (S4624: nested template literals). - Turn the isRevoked/isGracePeriod nested ternary for warningBlock into an if/else (S3358: nested ternary). - Build the JSON-LD "<" unicode-escape via String.fromCharCode(92) + "u003c" instead of a backslash-escaped string literal (S7780), keeping the critical XSS-prevention behavior from the prior commit unchanged (verified by the existing script-breakout test). Verified: tsc clean, lint clean, build clean, 475 tests pass (43 files). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Behaviorally identical for this ASCII backslash constant; fromCodePoint is the modern preferred API. Verified: tsc clean, lint clean, sponsor SEO tests pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#rootfor bot requests, so crawlers index real content instead of the generic SPA fallback shell — Google was treating ~124k sponsor pages as thin/duplicate and not indexing them.9fac47f): fixed a JSON-LD script-tag breakout (unescaped<inJSON.stringifyoutput), a$-pattern corruption bug in the#rootreplace, a silent-failure path that could cache a broken/empty page for 6h with no error, and a bug where sponsors inGRACE_PERIODwere shown a false "licence revoked, cannot support a UK visa application" claim.Test plan
tsc --noEmitcleannpm run lintcleannpm run buildcleanserver/routes/seo.ts(cache hit/miss, DB-not-found fallthrough, non-bot bypass, JSON-LD escaping) and expandedsponsorSeoHtml.test.tscoverage (grace-period status, null fields, extra escaping paths, the$-corruption fix, the script-breakout fix)/sponsor/:id/:slugwith a bot User-Agent against a real sponsor id in a running instance and confirm rendered HTML + JSON-LD🤖 Generated with Claude Code