test(e2e): DNS/TLS/email/CLI + observability/data-storage e2e coverage - #582
Open
dviejokfs wants to merge 30 commits into
Open
test(e2e): DNS/TLS/email/CLI + observability/data-storage e2e coverage#582dviejokfs wants to merge 30 commits into
dviejokfs wants to merge 30 commits into
Conversation
…tests Extends the e2e suite from PR #579 with real external-service coverage and closes a major gap in apps/temps-cli's own test coverage: - Pebble (ACME v2 test server) + pebble-challtestsrv + Mailpit test infra (apps/temps-e2e/docker-compose.e2e.yml) - CLI Pebble DNS provider support (temps-cli dns-providers) - apps/temps-e2e tls-scenario/email-scenario: full lifecycle against a real deployed app - real ACME HTTP-01 issuance, cert issuer verified, real SMTP send verified via Mailpit's own API, open/click tracking verified - web/e2e Playwright specs driving the actual console UI through the same flows (domain-tls-pebble, email-provider-mailpit) - apps/temps-cli: unit tests for 54/64 command groups (up from 4/64), 788 tests, following the existing bun:test pure-function pattern - extracting and testing validators/formatters/parsers that AI agents and scripts depend on via bunx @temps-sdk/cli Known gap documented in web/e2e/README.md: no console UI to link a provisioned certificate to a project's custom domain (API-only today).
📓 Changelog previewThis is what your commits will add to the generated ## [Unreleased]
### Added
- **e2e:** Monitoring/status-page lifecycle scenario
- **e2e:** Error-tracking (Sentry-compatible) lifecycle scenario
- **e2e:** Add backup-restore-scenario via MinIO; fix unrestorable WAL-G backups
- **e2e:** Add git-deploy-scenario against a real public GitHub repo
### Documentation
- **e2e:** Document the 9 scenario commands added since PR #582 opened
- **e2e:** Document Slack notifications as a deliberate, permanent e2e gap
### Fixed
- **cli:** Environments vars get never revealed real values
- **cli:** Correct instance-wide user roles to admin/user
- **audit:** Correct ISO 8601 doc example for from/to query params
- **status-page:** Recognize major_outage/partial_outage as down/degraded
- **status-page:** Make current-status/uptime/bucketed time params optional
- **status-page:** Exclude the bootstrap 'unknown' check row from uptime%
- **status-page:** Share ONE MonitorService so API-created monitors get an immediate check
- **status-page:** Cast AVG(response_time_ms) to double precision
- **analytics-session-replay:** Route method mismatch + soft-delete never enforced
- **ci:** Wire up Pebble + Mailpit test infra for e2e-tests.yml
### Miscellaneous
- **sdk:** Regenerate @temps-sdk/api from the live OpenAPI spec
### Testing
- **e2e:** DNS/TLS/email coverage via Pebble+Mailpit, temps-cli unit tests
- **e2e:** Genuine CLI subprocess e2e coverage (cli-scenario)
- **e2e:** Kv-storage scenario
- **e2e:** Audit-logs scenario
- **e2e:** Managed-services scenario; fix scenario --with-db
- **e2e:** RBAC/teams permission-boundary scenario
- **e2e:** Wire up kv/audit/managed-services/rbac scenario commands
- **e2e:** Add logs-scenario for log-aggregation lifecycle
- **e2e:** Add analytics-scenario for web-analytics lifecycle
- **e2e:** Add session-replay-scenario for replay ingest+playback lifecycle
- **e2e:** Dns01-wildcard-scenario -- real wildcard cert issuance via Pebble DNS-01
- **e2e:** Blob-scenario -- real RustFS data-plane round trip, 3 real bugs found+fixed
- **e2e:** Flags-scenario -- feature flags driven through the real @temps-sdk/node-sdk FlagsClient |
Adds real CLI e2e testing: cli-scenario spawns the actual @temps-sdk/cli binary as a subprocess for every step against a live instance, rather than driving the SDK client directly like scenario/tls-scenario/email-scenario do. This proves argv parsing, Commander's command wiring, and stdout/--json formatting actually work end to end -- exactly what breaks an AI agent running `bunx @temps-sdk/cli ...` even when the underlying API is fine, which the apps/temps-cli unit tests from the previous commit can't catch. Covers the flows an agent chains in practice: project CRUD, env var set/list/export/delete, service create/inspect/remove, a real image deploy polled to a terminal state via the CLI's own `deployments status`, domain CRUD, and apikeys read paths. Every resource is torn down via the CLI's own delete/remove commands, falling back to the SDK-based teardown for anything the CLI itself fails to remove. Two real, pre-existing bugs found live while building this (documented in cli-scenario.ts, not fixed there -- both need backend changes out of scope for a CLI e2e suite): - `environments vars get` / `vars list --show-values` both read from the list endpoint, which masks EVERY variable's value to "***" regardless of is_secret -- so --show-values never reveals anything. Only `vars export` genuinely resolves real values. - `domains status` 404s on a domain that hasn't finished ACME provisioning yet -- exactly the domain state you'd want to check status on. Root cause: check_domain_status resolves via list_certificates(CertificateFilter:: default()), which only returns domains with an already-issued certificate. Also confirmed (not a bug): apikeys create is correctly rejected under API-key auth as an anti-privilege-escalation guard, so it's structurally untestable from a pure CLI e2e run -- the scenario exercises the apikeys read paths that are reachable instead. Verified passing 3x back-to-back; instance left with zero leftover resources after each run.
\`vars get\` read from the environment-variables list endpoint, which the
server masks to the literal string "***" for every variable regardless of
is_secret -- so a command whose entire point is showing one variable's
value never actually showed it. Found live by cli-scenario (real CLI
subprocess e2e), reported by the user with the actual endpoint that does
work: GET /projects/{id}/env-vars/{key}/value.
vars export already resolves real values through that same endpoint via
resolveEnvVarValues() -- vars get just never called it. Reuse it: resolve
plaintext for the non-secret matched rows before printing, still routing
secrets through formatEnvVarValue()'s existing withheld-value message.
cli-scenario's env-var step now asserts the real value comes back from
`vars get` (previously it could only assert via `vars export`, with a
comment explaining why `get` didn't work). Verified live 3x back-to-back,
plus a manual secret-vs-non-secret check confirming secrets are still
correctly withheld.
AVAILABLE_ROLES was ['admin','developer','viewer'] and defaulted new
users to 'viewer' -- none of which except 'admin' are valid
instance-wide roles. RoleType (crates/temps-entities/src/types.rs)
only ever accepts "admin"/"user"; 'developer'/'viewer' are TEAM
roles, a separate concept. Every 'users create' without an explicit
--roles flag was silently sending a role the backend 400s on
("Unsupported role").
ListAuditLogsQuery.from/.to docs claimed epoch milliseconds (example = 1), but temps_core::DateTime's Deserialize only accepts RFC3339/naive ISO 8601 strings -- passing millis 400s with "Invalid datetime format".
14-step round trip: enable KV, create 2 projects, set/get, incr
(sequential + fresh-key-defaults-to-1), keys pattern exact-match,
nx/xx conditional-write semantics, ttl sentinels (-1 no-expiry / -2
missing-key), expire, del (exact count + idempotent second call),
cross-project isolation (kv:p{project_id}: namespace), kv_status
read-back, and 400-on-missing-project_id.
No console UI exists for this feature beyond an enabled/healthy
status badge, so this is the only coverage that exists.
Create + delete a project, then read back the exact
PROJECT_CREATED/PROJECT_DELETED rows via GET /audit/logs and
/audit/logs/{id} (id/data/actor/timestamp match, not just "the
list is non-empty"), and confirm the read endpoint itself is
RBAC-gated (401/403 with no bearer token).
Full lifecycle: build+push a throwaway Go probe app, provision + link a postgres service to a project BEFORE deploying, deploy the probe (writes through the injected POSTGRES_URL), verify an exact row-count round trip through /probe, verify the resolved-env-vars reveal endpoint returns a real (non-masked) connection string, then unlink and confirm disappearance. Also fixes a real, pre-existing bug found while building this: 'scenario --with-db' created its postgres service with no parameters, and PostgresParameterStrategy::validate_for_creation requires 'database'/'username' with no defaults -- every --with-db run was 400ing.
Proves the actual permission BOUNDARY (not just that team/access CRUD returns 2xx): an independently-authenticated second, low-privilege user is granted team access to a project, then escalated viewer -> deployer -> admin, asserting the exact 200/403 transitions and required_permission strings the guard enforces at each tier, plus the exact audit trail (PROJECT_ACCESS_GRANTED / TEAM_MEMBER_ROLE_UPDATED / PROJECT_ACCESS_REVOKED / TEAM_DELETED). Uses a second 'guard' team (granted and never revoked) so that revoking the primary team's grant at the end doesn't drop the project's grant count to zero -- which would (correctly) reopen the project and defeat the revoke assertion for the wrong reason. Zero grants means unrestricted is documented, tested platform behavior (checker.rs's own unit tests, the CLI's revokeAccessAction warning), not a bug. Also fixes a real, pre-existing CLI bug found while building this: apps/temps-cli's AVAILABLE_ROLES offered 'developer'/'viewer' as instance-wide user roles (fixed in a prior commit on this branch) -- neither is valid; those are TEAM roles, a separate concept. Needs DB-direct access (crates/temps-cli's own 'api-key' subcommand) to mint the second user's bearer key: minting a NEW key while already authenticated via API key is deliberately blocked (anti-privilege-escalation), and login only sets a session cookie.
Registers the four new 'temps-e2e' subcommands added on this branch: kv-scenario, audit-scenario, managed-services-scenario, rbac-scenario.
health_check_service.rs writes the finer-grained 'major_outage' (5xx after retries, connection failures) and 'partial_outage' (unexpected status codes) to status_checks.status -- not the plain 'down'/ 'degraded' two of its consumers assumed: - calculate_overall_status compared monitor.current_status == "down" literally, so a monitor that was completely unreachable was invisible to the project-level overall-status computation unless an incident also happened to exist to catch it via the severity checks. - get_bucketed_status's SQL only counted literal 'down'/'degraded' in its FILTER clauses; a bucket made entirely of real 'major_outage' checks fell through to status="unknown" (gray/neutral) instead of "down" (red) on the status-page bucketed uptime chart -- silently hiding real outages. temps-monitoring's own MonitorStatus::from_str already normalizes both vocabularies for outage-detection purposes; these two call sites never got the same treatment. Found while building live e2e coverage for the monitoring feature -- toggling a deployed app to 5xx and checking the bucketed chart reproduced exactly this.
…ional
CurrentStatusQuery, UptimeQuery, and BucketedQuery declared
start_time/end_time as non-optional temps_core::DateTime fields, but
their utoipa docs (and, for UptimeQuery, the 'days' field and the
service methods they call) all promised sensible defaults when
omitted: last 24h for current-status, 'days' (default 60) for uptime,
24h-ago/7-days-ago for the two bucketed endpoints. Since the fields
were required, every call had to pass an explicit range or 400 --
making the documented default-timeframe behavior (and get_current_status's
own no-arg 24h helper) unreachable dead code, and 'days' a parameter
that could never actually take effect.
Found while building live e2e coverage for the monitoring feature: a
scenario polling GET /monitors/{id}/current-status the way any real
client naturally would -- with no query params -- 400'd immediately.
create_monitor inserts a synthetic status_checks row (status='unknown', 'Monitor created - awaiting first health check') to bootstrap uptime queries. Every uptime-percentage calculation (calculate_uptime, get_current_status_for_timeframe, get_current_status_with_timeframes, get_bucketed_status) counted it toward total_checks without it ever being able to count toward successful_checks -- so a monitor with a 100% real pass rate still reported less than 100% uptime for as long as that placeholder row stayed within the query window (its first 24h for current-status, its first "days" window for uptime history/ calculate_uptime, its first bucket interval for the bucketed chart). Found live while building e2e coverage for the monitoring feature: a brand-new monitor whose only real check ever succeeded reported uptime_percentage=50 instead of 100.
…t an immediate check
plugin.rs built TWO separate MonitorService instances: one via
StatusPageService::new (no job queue) registered for the HTTP route
handlers to call through, and a second via MonitorService::with_job_queue
used only by the plugin's own job-listener loop
(EnvironmentCreated/EnvironmentDeleted/DeploymentSucceeded).
create_monitor's emit_monitor_created only emits the MonitorCreated
event when a job queue is present. Since POST /projects/{id}/monitors
-- the real, user-facing create path -- went through the job-queue-less
instance, that emission silently no-opped for every API-created
monitor. Only the internal auto-provisioning path
(ensure_monitor_for_environment, invoked from the job listener's own
instance) ever got the "checked immediately" behavior
start_scheduler's own doc comment promises ("ensuring new monitors are
checked immediately without waiting for the next scheduled cycle").
An API-created monitor instead waited for the next periodic cycle
(interval ticks every 60s from server start, not from monitor
creation), which can be nearly a full cycle late.
Fixed by adding StatusPageService::with_job_queue (mirrors
MonitorService's own constructor pair) and a monitor_service_arc()
getter so plugin.rs builds exactly one MonitorService and shares it
between the HTTP-facing StatusPageService and its job-listener loop,
closing off this whole class of divergent-instance bug rather than
just patching today's symptom.
Found live while building e2e coverage for the monitoring feature: an
explicitly-created monitor's current-status stayed "unknown" for over
30s while an auto-created monitor on the same environment flipped to
"operational" within ~14s.
Postgres' AVG() over an integer column returns NUMERIC, not double precision. calculate_avg_response_time's FromQueryResult struct declared the field as Option<f64>, so sqlx's try_get<f64> errored on the type mismatch. get_status_overview silently swallowed that Err into a fake "unknown" status for every monitor on the page, even ones that were fully healthy and had just reported "operational" moments earlier via the current-status endpoint (which computes response time via a sibling query that already used Decimal correctly). Also log the swallowed error instead of discarding it silently, so a future regression here is diagnosable from server logs instead of requiring DB archaeology to root-cause.
Real uptime-monitoring e2e coverage: confirms a fresh project auto- gets a default monitor, creates an explicit second one via the CRUD API, deploys a toggleable Go app (new lib/toggle-app.ts -- health flips over HTTP so it works identically against a remote instance), waits for the fixed 60s health-check cycle to catch a real 5xx outage and later its recovery, and asserts the incident/bucketed- chart/status-overview side effects at each step. Building this surfaced and fixed four real platform bugs (see the three preceding commits + this branch's status-page fixes): status-literal mismatches in overall/bucketed status, required-vs- optional query params blocking documented defaults, an uptime- denominator bug from the synthetic bootstrap check row, two divergent MonitorService instances that skipped the immediate-check path for API-created monitors, and the AVG() NUMERIC/f64 cast bug fixed in the prior commit. The incident auto-resolve step polls rather than asserting once: current-status reads status_checks directly and flips the instant a check commits, but incident resolution is a side effect of the same check processed asynchronously through the job queue, which can trail by anywhere from a few ms to most of another 60s cycle.
Real error-tracking e2e coverage: creates a DSN (not auto-provisioned,
unlike the monitoring auto-monitor), sends real Sentry-shaped events
authenticated with the DSN's public key via X-Sentry-Auth (the one
route in the platform using a different auth scheme than the normal
bearer token), and proves fingerprint-based grouping actually groups
an identical repeat into the same issue while keeping a genuinely
different exception in its own issue -- not just that the error-group
CRUD routes return 2xx.
Also covers: computed group title ("{type}: {value}"), event detail
round-tripping the stored exception data, error-stats splitting
resolved/unresolved after a status update, and the DSN auth boundary
(a garbage key gets a real 401, not silently accepted).
New lib/sentry-events.ts: a payload builder + raw sender, since the
ingest endpoint's generated SDK type (SentryEventRequest) only
declares event_id/message/platform/timestamp -- the real handler
takes arbitrary JSON and expects a full exception/stacktrace shape,
so the OpenAPI schema for this one route is decorative only and the
typed client can't express a realistic payload.
No platform bugs found this time -- ran clean end-to-end on the first
real attempt (one assertion fix on my own end: event data is stored
wrapped under a `sentry` key alongside a `source` discriminator, not
at the top level, since this crate ingests from multiple SDK sources).
Real container-log round trip through the Docker log collector: deploy a throwaway Go app that emits structured JSON stdout/stderr on demand, wait out the actual chunk-flush window, then verify full-text search, level filtering, JSONB `fields` passthrough, env filtering, grep-style context via chunk_id/line_offset, and purge -- through the real ingestion pipeline, not synthetic rows inserted into storage. Ran clean 3x back-to-back (incl. one --keep + one --json run) with zero platform bugs found; this scenario is purely additive coverage. Also promotes the pollUntil polling helper (previously duplicated inside monitoring-scenario.ts) into the shared flows.ts module now that a third caller needs it.
packages/api/openapi.json had drifted to 590 of the server's 674 paths (84 endpoints missing), including the whole temps-analytics query surface (getEventEntries, checkAnalyticsHasEvents, getVisitorJourney, and friends) needed to write the analytics e2e scenario. Regenerated from a live dev server's /api/api-docs/openapi.json, canonicalized the same way apps/temps-cli/scripts/openapi-canonical.ts does (sorted keys, 2-space indent) so the on-disk diff reflects only real API changes. Also adds the missing `prettier` devDependency: openapi-ts.config.ts sets `format: 'prettier'` but the package never declared it, so `bun run generate` failed with ENOENT in any environment without a prettier binary already on PATH (nothing else in this package pulls one in — apps/temps-cli's equivalent config has no formatter step at all). apps/temps-e2e typechecks unchanged against the new client; no other package in this repo depends on @temps-sdk/api.
Deploys a throwaway HTML app (temps-proxy only issues visitor/session cookies for HTML responses, per should_track_page) so the scenario can capture real _temps_visitor_id/_temps_sid Set-Cookie values, then replays them on two POST /api/_temps/event calls the way a real tracking snippet would. Verifies the custom event's event_data round-trips into event-entries' props JSON, and that both events land in the same visitor-journey session -- proving cookie-based session stitching end-to-end, not just that the query routes return 2xx. Ran clean 3x back-to-back (incl. one --keep + one --json run) with zero Rust-side bugs found; this scenario is purely additive coverage. The SDK staleness that blocked writing it was fixed in the preceding commit.
…ver enforced
Two real bugs found while building session-replay e2e coverage:
1. PUT /visitors/{visitor_id}/session-replays/{session_id}/duration was
registered as POST in configure_routes(), but the utoipa docs (and every
generated SDK) declare PUT. Any client following the documented contract
got a 405. Fixed the route registration to match.
2. delete_session_replay correctly soft-deletes a session (and its events,
is_active=false), but none of the four read paths ever checked the flag:
get_sessions_for_project, get_sessions_for_visitor, get_session_replay,
and get_session_replay_without_events all read is_active=false rows as
if they were live. A "deleted" recording -- DOM mutations, potentially
keystrokes -- stayed fully visible in both list views and fetchable in
full by ID. Added is_active=true filters to all four (including the
events query inside get_session_replay, which had the same gap on the
session_replay_events table).
Verified: `cargo test --package temps-analytics-session-replay --lib` (15
passed), clippy clean, and live end-to-end -- confirmed a deleted session's
direct-by-ID playback now correctly 404s where it previously returned the
full event stream.
…ecycle Deploys the same HTML app used by analytics-scenario (session-replay's public init endpoint requires a real _temps_visitor_id cookie and 400s without one), inits a replay session, POSTs two base64+zlib event batches, then verifies: the project/visitor-scoped lists both surface it once duration is computed from real timestamps, full playback returns all events with a custom field round-tripped intact, a manual duration override sticks, and delete actually removes it from every read path. Found and root-cause-fixed two real bugs while building this (see preceding commit): a PUT/POST route method mismatch on the duration endpoint, and soft-delete never being enforced by any read path. Ran clean 3x back-to-back (incl. one --keep + one --json run) after the fixes landed.
README only covered scenario/tls-scenario/email-scenario/cli-scenario/ examples. Adds usage + a "steps" section for kv-scenario, audit-scenario, managed-services-scenario, rbac-scenario, monitoring-scenario, error-tracking-scenario, logs-scenario, analytics-scenario, and session-replay-scenario -- including the real platform bugs each one found and fixed, mirroring the existing tls-scenario/cli-scenario style.
dviejokfs
changed the title
…e gap SSRF guard (validate_external_url) + SlackProvider::initialize()'s hardcoded hooks.slack.com prefix check block any local target reaching the real create-to-test API flow. send() itself is already covered by a wiremock Rust unit test, which is the right layer for this feature.
… Pebble DNS-01 Registers a Pebble DNS provider, issues a wildcard certificate through a real DNS-01 challenge (TXT record pushed to pebble-challtestsrv, then Pebble's real validator queries it back), and asserts the resulting certificate's issuer and SAN. Closes a real coverage gap rather than a code bug: temps-domains' own test_dns01_wildcard_with_pebble is #[ignore]d and sets PEBBLE_VA_ALWAYS_VALID=1, bypassing Pebble's real validator entirely -- nothing before this proved the DNS-01 pipeline works end-to-end. Verified live 3x back-to-back against a dev instance configured with ACME_DIRECTORY_URL/ACME_INSECURE/TEMPS_ALLOW_PEBBLE_PROVIDER=1.
… bugs found+fixed put/download/head/list/copy/delete against the platform-wide blob-storage singleton, deleted-blob 404, cross-project isolation, and a 400 on a missing project_id -- mirrors kv-scenario's shape. Three real bugs found and fixed: 1. blob_put's #[utoipa::path] never declared its query params at all, and blob_list's declared params were missing project_id -- both existed on the Rust query-extractor structs but were invisible to the OpenAPI spec, so no client could call these with a project scope. Fixed both annotations and regenerated packages/api. 2. temps-proxy unconditionally stripped Content-Length from every HEAD response regardless of downstream HTTP version, when the reason given in the existing comment only applies to HTTP/2. An HTTP/1.1 client got a HEAD response with no Content-Length and no chunked encoding on a keep-alive connection -- no way to tell the response was complete. Confirmed live: curl -I through the proxy port hung indefinitely; the same request against the console port (same handler, no proxy) returned instantly with the header present. Fixed by only stripping when session.is_http2(). 3. BlobService::del counted every successful S3 DeleteObject call as a deletion, but DeleteObject is idempotent by design and succeeds whether or not the key existed -- so a second delete of already-gone keys kept reporting them deleted. Fixed by checking existence via HeadObject first. Verified live 3x back-to-back against a dev instance.
…emps-sdk/node-sdk FlagsClient Defaults, per-environment overrides, the kill switch outranking an override (not just the default), ETag/If-None-Match 304 caching, exposure reporting round-tripped through last_evaluated_at, and the delivery endpoint's deployment-token-only auth boundary. Every read goes through the real FlagsClient (@temps-sdk/node-sdk), not just the raw HTTP API -- that's the actual client apps deployed on Temps use. Required adding @temps-sdk/node-sdk as a workspace dependency of apps/temps-e2e and building its dist/ (it had none yet). No platform bugs found -- every step passed clean on the first attempt. Verified live 3x back-to-back.
…L-G backups apps/temps-e2e: new backup-restore-scenario command exercising the real backup/restore lifecycle for a postgres external service backed by a local MinIO -- real wal-g backup-push, real S3 upload, real in-place restore, proven by reverting writes made after the backup (not just 2xx checks). Bug found and fixed: PostgresWalgEngine::run (crates/temps-backup) ran `wal-g backup-push` without ever enabling continuous WAL archiving on the target container, so the base backup's checkpoint LSN had no archived WAL segment behind it. Every restore failed at Postgres startup with "could not locate required checkpoint record", the container crash-looped, and wait_for_container_health timed out at 90s -- an unrestorable backup that still reported `completed`. Fixed by adding ExternalService::enable_continuous_archiving (no-op default, implemented on PostgresService via the existing enable_wal_archiving) and calling it *before* backup-push, not after: pg_stop_backup() force-completes the current WAL segment, and Postgres only marks a completed segment `.ready` for archiving if archive_mode is already on at that moment -- enabling it afterward doesn't retroactively archive a segment that closed while archiving was off. Skips the container-recreating setup entirely once archiving is already active on a service, so it's a no-op after the first backup. Confirmed live 3x against a real MinIO + postgres service.
apps/temps-e2e: new git-deploy-scenario command proving the real git
pipeline end to end -- clone github.com/gotempsh/temps-examples, build a
language-preset subdirectory (examples/starters/go/net-http), trigger via
POST /projects/{id}/trigger-pipeline, and verify the exact response bodies
baked into the real repo source (not just 2xx). Deliberately hits real
github.com, unlike every other scenario in this suite, since there's no
local substitute for "does the git-clone-and-build pipeline work against a
real repo host".
Test-side bug found and fixed (not a platform bug): creating a Git-type
project always auto-queues an initial deployment as a side effect of
POST /projects itself, regardless of automatic_deploy, and asynchronously.
The first version raced that auto-deployment's row against the explicit
trigger-pipeline call and ended up polling the wrong (auto-triggered)
deployment, which the platform correctly cancels/supersedes the moment the
real one is created -- surfacing as a false "cancelled" failure while the
real deployment kept building. Fixed by having
triggerPipelineAndGetDeploymentId wait for the auto-queued deployment to
land first, so the baseline id used to detect "a genuinely new deployment"
is deterministic.
Confirmed live 3x against a real GitHub clone + build.
domain-tls-pebble.spec.ts and email-provider-mailpit.spec.ts (added earlier on this branch) each document that they need `docker compose -f apps/temps-e2e/docker-compose.e2e.yml up -d`, but no CI workflow ever ran it -- so both failed in every run: - domain-tls-pebble: `docker run --network temps-e2e-pebble-net ...` failed with "network temps-e2e-pebble-net not found" (the compose file that creates it was never brought up). - email-provider-mailpit: the "Send Test Email" UI action never showed success, because nothing was listening on Mailpit's SMTP port for temps to actually deliver to. Also, `temps serve` was starting with `--address 0.0.0.0:3000`, but Pebble's baked-in config sends real HTTP-01 validation requests to its own fixed httpPort (5002) -- so even with Pebble reachable, the challenge response would never land on temps' proxy. Fixed by: - adding a step that brings up pebble/pebble-challtestsrv/mailpit from docker-compose.e2e.yml and waits for all three ports before `temps serve` starts (mirrors README.md's "External-service test infra" section) - setting ACME_DIRECTORY_URL/ACME_INSECURE/TEMPS_ALLOW_PEBBLE_PROVIDER on the `temps serve` step - changing --address to 0.0.0.0:5002 to match Pebble's fixed httpPort Verified locally: ran both specs against a one-off instance configured identically to this workflow (Pebble/Mailpit from the same compose file, --address=5002, the same three env vars) -- both pass, confirmed 2x.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Continues the e2e coverage work from #579 with real DNS/TLS/email verification, and adds two complementary layers of
apps/temps-clitest coverage: fast unit tests, and genuine CLI subprocess e2e tests.External-service e2e coverage (Pebble + Mailpit):
apps/temps-e2e/docker-compose.e2e.yml): Pebble (Let's Encrypt's own ACME v2 test server), pebble-challtestsrv, Mailpitapps/temps-cli: Pebble DNS provider added todns-providers(was API-only before)apps/temps-e2eCLI:tls-scenarioandemail-scenariocommands — full lifecycle against a real deployed app: real ACME HTTP-01 issuance (cert issuer parsed and asserted as Pebble's test root, not just a 200), real SMTP send verified via Mailpit's own REST API (not just that the send call returned success), open/click tracking verifiedweb/e2ePlaywright specs (domain-tls-pebble,email-provider-mailpit) driving the actual console UI through the same flowsapps/temps-cliunit test coverage:bun:testunit tests (up from 4/64), 788 tests totaldata/index.test.ts): extract pure validators/formatters/parsers into named exports, test those — fast, offline, no live network callsai,blob,dev,email-domains,init,instances,kv,open,rollback,skills) were individually verified to have no extractable pure logicapps/temps-cligenuine CLI subprocess e2e coverage (cli-scenario):--jsonoutput — exactly what breaks an AI agent runningbunx @temps-sdk/cli ...even when the API is fine.cli-scenariocloses that gap: it spawns the real CLI binary as a subprocess for every step against a live instance (project CRUD, env vars, a service, a real image deploy polled to a terminal state via the CLI's owndeployments status, domain CRUD, apikeys read paths), unlikescenario/tls-scenario/email-scenario, which drive the SDK client directly.environments vars getread from the list endpoint, which masks every variable's value to"***"regardless ofis_secret— so a command whose entire point is showing one variable's value never actually showed it. Fixed to resolve real values through the same audited per-key endpointvars exportalready used (getEnvironmentVariableValue,GET /projects/{id}/env-vars/{key}/value); secrets are still correctly withheld.cli-scenario's env-var step is now a regression test for this fix, verified live 3x.domains status404s on a domain that hasn't finished ACME provisioning yet — exactly the domain state you'd actually want to check status on. Root cause:check_domain_statusresolves vialist_certificates(CertificateFilter::default()), which only returns domains with an already-issued certificate row.crates/temps-auth/src/sensitive_action.rs, an anti-privilege-escalation guard) — since API-key auth is this CLI's only non-interactive auth mode,apikeys createis structurally untestable from a pure CLI e2e run; the scenario exercises the read paths that are reachable instead.Known gap found and documented (
web/e2e/README.md): there's no console UI to link a provisioned TLS certificate to a project's custom domain — that's API-only today (POST /projects/{id}/custom-domains/{domain_id}/link-certificate/{cert_id}).domain-tls-pebblecovers what the UI does offer (standalone cert provisioning); the full app-serving path is covered via the API intls-scenario.Bug found and fixed along the way:
fetchOverTls(apps/temps-e2e) calledgetPeerCertificate()after the socket's'end'event, which returns{}under Bun — fixed by capturing the cert at handshake-complete time instead. Documented inapps/temps-e2e/README.mdfor anyone touching that code later.Extended: 9 more scenario commands + real bugs found (since this PR opened)
Kept growing on the same branch rather than splitting into a new PR, since each addition follows the exact same shape and verification discipline as the original scope. All commands, steps, and bugs are documented in detail in
apps/temps-e2e/README.md; summarized here.Data & Storage / Security & Auth:
kv-scenario— real Redis-backed kv-storage round trip (set/get/incr/expire/ttl/keys/del, nx/xx conditionals, cross-project isolation). No console UI exists for this feature beyond an enabled/healthy badge, so this is the only coverage that exists.audit-scenario— realPROJECT_CREATED/PROJECT_DELETEDaudit rows read back exactly (id/data/actor/timestamp), plus an RBAC-gate check on the read endpoint.managed-services-scenario— provision + link a postgres service to a project before deploying, an app writing through the injectedPOSTGRES_URL, exact row-count round trip, unlink. Bug found+fixed:scenario --with-dbcreated its postgres service with no parameters, andPostgresParameterStrategy::validate_for_creationrequiresdatabase/usernamewith no defaults — every--with-dbrun was 400ing.rbac-scenario— a second, independently-authenticated low-privilege user escalated viewer → deployer → admin, asserting exact 200/403 transitions and the audit trail at each tier. Bug found+fixed:apps/temps-cli'sAVAILABLE_ROLESoffereddeveloper/vieweras instance-wide user roles — neither is valid; those are team roles, a separate concept.Observability:
monitoring-scenario— auto-provisioned + explicit monitors, a real 5xx outage caught by the fixed 60s check cycle, incident lifecycle (create/resolve), bucketed chart, status overview. Four real bugs found+fixed: status-literal mismatches (major_outage/partial_outagevs the literal"down"/"degraded"the SQL checked for) in both overall and bucketed status; required-vs-optional query params blocking documented default time ranges; an uptime-denominator bug from the synthetic bootstrap check row; two divergentMonitorServiceinstances that skipped the immediate-check path for API-created monitors; plus theAVG(response_time_ms)PostgresNUMERIC→f64cast bug that madeget_status_overviewsilently report every healthy monitor as "unknown".error-tracking-scenario— Sentry-compatible ingestion authenticated via DSN public key (X-Sentry-Auth, the one route using a different auth scheme than the platform bearer token), fingerprint-based grouping proven live (identical repeat groups, distinct exception doesn't). No platform bugs found — clean on the first attempt.logs-scenario— real container stdout/stderr through the actual Docker log collector (not synthetic rows): full-text search, level filtering, JSONBfieldspassthrough, env filtering, grep-style context, purge. No platform bugs found.analytics-scenario— real visitor/session cookies issued by the proxy (should_track_pageonly fires for HTML responses), replayed on the public/api/_temps/eventingest path, customevent_dataround-trip, and session stitching proven viavisitor journey(two independent POSTs sharing one cookie pair land in one session). Bug found+fixed:packages/api/openapi.json— the source for the SDK this whole suite depends on — had drifted to 590 of the live server's 674 paths; the entiretemps-analyticsquery surface was unreachable from any TypeScript consumer until regenerated from a live spec. Not a Rust bug — every route was already fully wired, just unreachable from generated clients.session-replay-scenario— real rrweb-shaped event batches (base64+zlib), init/ingest/playback, list visibility once duration is computed, manual duration override, soft delete. Two real bugs found+fixed: the duration-override endpoint was registered asPOSTbut documented (and SDK'd) asPUT— any client following the docs got a 405. Anddelete_session_replaycorrectly soft-deletes a session, but none of the four read paths ever checkedis_active— a "deleted" recording (potentially DOM mutations/keystrokes) stayed fully visible in both list views and fetchable in full by ID; fixed all four, confirmed live that deleted-session playback now 404s.Test plan
bun testinapps/temps-cli: 788 pass / 0 failbun run typecheckinapps/temps-cliandapps/temps-e2e: clean (one pre-existing, unrelated error inopenapi-ts.config.ts, confirmed viagit diffto predate this PR)tls-scenariorun live 3x back-to-back against a dedicated Pebble-configured instance: real project → real deploy → real ACME HTTP-01 exchange → real HTTPS fetch of the deployed app → issuer confirmed as Pebble's test rootemail-scenariorun live 3x back-to-back: real SMTP send → real Mailpit receipt → real open/click tracking eventscli-scenariorun live 3x back-to-back (real CLI subprocess against a live instance), including a regression test for thevars getfix: every step green, instance left with zero leftover resources each timevars getfix manually verified against both a non-secret var (reveals the real value) and a secret var (still correctly withheld)domain-tls-pebble.spec.tsandemail-provider-mailpit.spec.tsrun live 3x back-to-back against the console UI, both greenkv-scenario,audit-scenario,managed-services-scenario,rbac-scenariorun live, each green, zero leftover resourcesmonitoring-scenario,error-tracking-scenario,logs-scenario,analytics-scenario,session-replay-scenarioeach run live 3x back-to-back (incl.--keepand--jsonvariants), all green after their respective fixes landedcargo test/clippy clean for every crate touched by a backend fix in this PR (temps-status-page,temps-analytics-session-replay)