Container reported not-running / crashed while genuinely alive during slow startup, even with an immediate placeholder port

[oliver-sturrock]

Summary

Running a container whose entrypoint takes anywhere from several seconds to tens of minutes before binding its configured port, we consistently see the container torn down / reported as not-running while it is still genuinely alive and making progress — even after adding a placeholder listener that answers the port-ready health check (waitForPort()'s tcpPort.fetch('http://ping')) immediately. Across many real attempts we've seen at least three different error signatures, and log evidence of overlapping/duplicate container start attempts within the same short window.

Environment

• @cloudflare/containers: 0.3.7
• wrangler: 4.107.0
• Instance type: standard-2 (1 vCPU / 6GiB / 12GB disk)
• max_instances: 1
• Container image: Debian-based (nextcloud:apache), PHP/Apache workload with a slow first-boot install step (occ maintenance:install against a remote Postgres DB, ~20-50 min end-to-end) and a separate fast (~seconds) config-restore path used on subsequent boots.

What we're seeing

Error 1: Container crashed while checking for ports, did you start the container and setup the entrypoint correctly?

This fires from waitForPort()'s retry loop (src/lib/container.ts) whenever this.container.running reads false on any poll iteration — regardless of our own configured portReadyTimeoutMS (we set this to 3600000ms specifically to tolerate a long install). We confirmed via direct database inspection that the underlying process was still alive and making real progress (creating schema, inserting rows) when this fired.

Error 2 (after adding a placeholder listener): The container is not running, consider calling start()

This string doesn't appear anywhere in the @cloudflare/containers package source — it must come from the native getTcpPort().fetch() binding itself. We saw this thrown from containerFetch()'s proxy call immediately after our own explicit startAndWaitForPorts() call had just returned successfully (same-second timestamps in wrangler tail), and — in the same test — the container's own outbound callback to our Worker (proving it was alive) succeeded ~18 seconds after this "not running" error was reported.

Error 3: There is no container instance that can be provided to this Durable Object, try again later

Seen in a later attempt, in the same test run as a duplicate/unexpected second incoming request to the same endpoint we only called once, and an onStop callback firing with a non-zero exit code — suggestive of more than one container start attempt overlapping for the same Durable Object within a ~30 second window.

The placeholder-listener workaround (partial)

Since waitForPort() accepts any valid HTTP response (not just 2xx) as "ready," we tried opening a trivial placeholder HTTP server (php -S) on the target port immediately, before any of our slow startup work, specifically to make the readiness check pass on its first poll instead of spending the whole slow-startup window looking unready. This did change the failure signature (we stopped seeing Error 1 as consistently) but did not eliminate the underlying issue — we still hit Errors 2 and 3 above on subsequent attempts, including cases where the container appeared to serve a request against a filesystem that hadn't finished syncing yet, alongside signs of overlapping start attempts.

What we'd like to understand

1. Is there a platform-level (not portReadyTimeoutMS/instanceGetTimeoutMS) timeout or liveness-check interval that can independently mark a container as not-running while its main process is still alive?
2. Is this.container.running guaranteed to be a real-time, non-stale reflection of platform state at the moment it's read, or can it lag/flap? (We found 🐛 Bug Report — container status is incorrect after destroying container without monitor workerd#4362 describing a related staleness issue with this exact flag, in the opposite direction — reporting true when it should be false.)
3. Are concurrent/overlapping calls into startAndWaitForPorts() / containerFetch() for the same Durable Object instance expected to be safe, or is there a known race when multiple requests each try to ensure the container is started around the same time?
4. Is the placeholder-listener pattern (open a cheap port immediately, do slow work, then hand off to the real listener) a supported/sane approach for long-starting workloads, or is there a better-supported mechanism we're missing for this class of workload?

Happy to share full wrangler tail logs, our Dockerfile, and Worker code if useful — trimmed the above for length and to remove app-specific identifiers, but the mechanism and error strings are copied verbatim from real runs.

Related, possibly relevant issues we found while investigating: #162, #147.

[oliver-sturrock]

Additional evidence that strengthens this: I ran a fully isolated test — exactly one curl request (--http1.1, no client-side retries, verified only one invocation was made), with wrangler tail watching passively.

The client received exactly one HTTP response. But the server-side tail log shows two separate GET .../status.php - Ok events, 5 seconds apart, for that single client request:

NextcloudNeonContainer.startWithRestore - Ok @ 11:49:50 AM
GET .../status.php - Ok @ 11:49:50 AM
  (log) fetch: before startWithRestore, path=/status.php
  (log) fetch: startWithRestore returned, about to proxy
  (log) fetch: container.fetch returned status=404
GET .../status.php - Ok @ 11:49:55 AM        <-- second, unexplained event
GET .../sfc-restore-fetch?secret=REDACTED - Ok @ 11:50:15 AM
Alarm @ 11:49:51 AM - Ok
NextcloudNeonContainer.getRestoreTar - Ok @ 11:50:00 AM
Alarm @ 11:50:15 AM - Ok
  (log) onStop: exitCode=1 reason=exit
Alarm @ 11:50:15 AM - Canceled

Nothing in our own Worker code issues a second request to /status.php — container.fetch(request) proxies to the container over the TCP port binding, it doesn't re-invoke the Worker's own fetch() handler. The second event also has no correlated (log) lines from our own console.log calls, which fire as the very first thing in every real invocation of our handler — suggesting it may not be going through our own code path the same way the first one did.

This looks like the platform itself (not client retries, not our code) may be internally re-driving request handling for this Durable-Object/container-backed Worker under some condition — which would plausibly explain the overlapping start attempts and inconsistent error signatures we're seeing across different runs.

[oliver-sturrock]

Follow-up: built a minimal, non-Nextcloud test container (plain Python stdlib http.server, no PHP/Apache/database) to isolate whether this is a platform-level issue or specific to our actual workload. Tested every individual factor from our real setup, each in isolation, against a genuinely fresh container instance per test (deleted and recreated the container app between rounds to rule out state leaking from prior attempts):

1. Slow startup alone (90s, then 300s sleep before binding the port) — clean.
2. Placeholder-port pattern (open port immediately, real listener after the delay) — clean.
3. Container calling back to its own Worker mid-startup (mirroring our restore-fetch pattern) — clean.
4. sleepAfter set to 24h (ruling out activity-tracking interference) — clean.
5. Real CPU load during the entire slow-start window (busy-loop processes, not just sleep) — clean.
6. Real outbound WAN network traffic to our actual external Postgres host (repeated TCP connections over the full 300s window, mirroring the many small round trips our real install does) — clean.

Every single variant completed successfully on the first or second attempt, no crash, no duplicate/overlapping request events, no unexplained errors. This strongly suggests the instability we're seeing is not a generic "any container with X characteristic gets killed" platform bug — it's something specific to our actual (larger, PHP/Apache/multi-process) workload that we haven't isolated yet, possibly the combination of several of the above simultaneously rather than any single factor, or something about resource usage/process patterns a minimal Python container doesn't replicate. Flagging so this doesn't read as a confirmed platform-wide bug based on our original report - happy to keep narrowing it down and will update if we isolate the actual trigger.

[oliver-sturrock]

Found the actual root cause on our real workload - it turned out to have nothing to do with slow startup, resource contention, instance size, or any platform-level container-lifecycle behavior. Posting in case it saves someone else the same multi-day rabbit hole.

Root cause: the official nextcloud:apache image (Debian-based Apache) ships with access.log, error.log, and other_vhosts_access.log symlinked to /dev/stdout//dev/stderr - a completely standard convention for container-friendly logging, and it works fine in plain docker run (which is exactly why every one of our local reproductions succeeded while every real Cloudflare deployment eventually crashed - we never once reproduced this locally). On Cloudflare's container runtime specifically, opening those symlinks fails with ENXIO ("No such device or address"), which kills Apache immediately with AH00015: Unable to open logs - before it ever binds its port. That's what our onStop callback was reporting as exitCode=1, and what surfaced upstream (through several layers of the container lifecycle) as the various "crashed while checking for ports" / "not running" errors in our earlier reports on this thread.

We only found this by bypassing log forwarding entirely and having the container POST its own failure detail directly to our Worker over HTTP - wrangler tail's forwarding of container stdout/stderr had become completely unreliable by the time this started failing consistently (lines that appeared fine on earlier successful runs stopped showing up at all, even though the container was still emitting them - which itself might be a related/interesting data point: this ENXIO condition seems to correlate with degraded stdout/stderr forwarding more generally, not just those specific log files failing to open).

Fix: replace the symlinked log files with real files (rm + touch + chown) and disable the optional other-vhosts-access-log conf, all at image build time. Once done, the container starts and stays up reliably.

This obviously isn't a bug in @cloudflare/containers or a startAndWaitForPorts/portReadyTimeoutMS issue at all - it's a mismatch between a common base-image convention (/dev/stdout log symlinks) and Cloudflare's container runtime's handling of those device files. Might be worth a mention in the docs (something like "if your image logs to /dev/stdout or /dev/stderr via symlinks and crashes immediately on start, check for ENXIO") since it's such a standard pattern in official Docker Hub images (not just this one) that others will likely hit it too. Considering this issue resolved on our end - thanks for the reference materials that helped narrow the search space along the way.