Skip to content

fix(wrangler): keep wrangler dev alive when a proxied request to the UserWorker fails transiently - #15252

Merged
dario-piotrowicz merged 5 commits into
cloudflare:mainfrom
GregoryCollett:gc-dev-proxy-transient-fetch-failures
Sep 7, 2026
Merged

dario-piotrowicz merged 5 commits into
cloudflare:mainfrom
GregoryCollett:gc-dev-proxy-transient-fetch-failures

Conversation

@GregoryCollett

@GregoryCollett GregoryCollett commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #14926. Also mitigates the empty-message symptom of #14906 for this path by including the request method, URL, attempt count and underlying exception in the logged error.

The bug

wrangler dev runs the ProxyWorker (binding the public port) and the UserWorker runtime as separate workerd processes. When a proxied request to the UserWorker fails while the UserWorker's origin is unchanged, the ProxyWorker reports {type: "error"}, ProxyController.onProxyWorkerMessage escalates it via emitErrorEvent, DevEnv.handleErrorEvent falls through to the fatal re-emit, and the entire dev server exits with an empty ✘ [ERROR] (the error crosses a JSON boundary, so castErrorCause produces a message-less Error), leaving the port unbound. In CI test suites, one transient failure kills every remaining test with ERR_CONNECTION_REFUSED — several independent reports on #14926 describe exactly this.

Root cause of the transient failure itself

kj HTTP defaults collide: the client pool closes idle connections after 5s (HttpClientSettings.idleTimeout) and the server closes idle keep-alive connections after 5s (HttpServerSettings.pipelineTimeout). For a pooled ProxyWorker→UserWorker connection idling ≈5s, both timers fire ~simultaneously; a request written into the connection as the server's close is in flight gets an RST, surfaced as Error: Network connection lost..

Reproduced organically (no fault injection): bursts of ~48 concurrent GETs separated by idle gaps swept 3.0→7.9s in 0.1s steps against wrangler dev hit the fatal within one sweep, at a 4.2s outer gap (per-connection idle 4.2–5.2s given the ~1s burst spread) — failed requests interleaved with same-millisecond 200s (stale pool checkouts die, fresh connections succeed), while the UserWorker runtime never exited and its isolate was never re-created. Deterministic variant: SIGKILL the UserWorker runtime under request load (miniflare restarts it reusing the port, so the in-flight failures are same-origin).

The fix (two small pieces)

  1. ProxyWorker.ts: same-origin fetch failures now retry bodyless (GET/HEAD) requests — immediately, then once more after 250ms — before reporting. The retry draws a fresh connection, absorbing the race. This mirrors the existing requeue-on-reload behaviour and its rationale comment ("it would be incorrect to retry non-idempotent requests"); non-GET/HEAD behaviour is unchanged. Recovered requests log a console.warn so absorbed events remain visible. The reported error now includes method, URL and attempt count.
  2. DevEnv.ts: handleErrorEvent classifies "Error inside ProxyWorker" reports as recoverable — logged via logger.error with the underlying exception — instead of falling through to the fatal top-level re-emit. This sits alongside the existing recoverable carve-out for other ProxyController reasons. One failed proxied request should never take down the dev session.

With both pieces, the kill-the-runtime repro goes from "dev server exits, port unbound" to: in-flight GETs retry across the restart (5 of 11 recovered in my run), the rest fail individually with the error logged, and the server keeps serving.


  • Tests
    • Tests included/updated — DevEnv.test.ts gains a test asserting a ProxyWorker error report is logged and does not re-emit a fatal top-level error event.
    • Automated tests not possible - manual testing has been completed as follows: the retry path was validated against a real wrangler dev session two ways — (a) the organic idle-gap sweep described above (7,200 requests: zero server-side failures with the fix; the fatal within one sweep without it), and (b) SIGKILL-ing the UserWorker runtime under load (server survives; GETs recover across the restart). Note: I could not run the wrangler vitest suite locally because @cloudflare/remote-bindings#build fails in my environment on a pristine main checkout (its tsdown embed-workers plugin resolves template paths incorrectly there), so I'm relying on CI for the full suite; turbo check:type --filter=wrangler and oxfmt pass locally.
  • Public documentation
    • Documentation not necessary because: behavioural bug fix in wrangler dev internals; a changeset is included.

Note

This is a contribution from an AI agent: Claude Code (Claude Fable 5), working under the direction and review of Gregory Collett.


Devin Review

…UserWorker fails transiently

When a request proxied to the UserWorker failed while the UserWorker's
origin was unchanged — most commonly a reused keep-alive connection the
UserWorker's HTTP server closed at the same moment the request was
written to it (kj's client-pool idleTimeout and server pipelineTimeout
both default to 5s, so a connection idling ~5s races the close) — the
ProxyWorker reported a fatal error and the whole dev server exited with
an empty error message, leaving the port unbound.

The ProxyWorker now retries bodyless (GET/HEAD) requests before
reporting, absorbing the transient failure on a fresh connection, and
DevEnv classifies an exhausted or non-retriable report as recoverable:
it is logged with the request method, URL, attempt count and underlying
exception, and the dev server keeps serving. Only the affected request
fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: fc9c3f1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
wrangler Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-pool-workers Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-project-automation github-project-automation Bot moved this to Untriaged in workers-sdk Aug 18, 2026
@workers-devprod
workers-devprod requested review from a team and dario-piotrowicz and removed request for a team August 18, 2026 15:29
@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/tidy-moons-provide.md: [@cloudflare/wrangler]
  • packages/wrangler/src/tests/api/startDevWorker/DevEnv.test.ts: [@cloudflare/wrangler]
  • packages/wrangler/src/api/startDevWorker/DevEnv.ts: [@cloudflare/wrangler]
  • packages/wrangler/templates/startDevWorker/ProxyWorker.ts: [@cloudflare/wrangler]

@pkg-pr-new

pkg-pr-new Bot commented Aug 18, 2026 •

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@15252

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@15252

@cloudflare/codemods

npm i https://pkg.pr.new/@cloudflare/codemods@15252

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@15252

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@15252

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@15252

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@15252

miniflare

npm i https://pkg.pr.new/miniflare@15252

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@15252

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@15252

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@15252

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@15252

@cloudflare/vitest-plugin

npm i https://pkg.pr.new/@cloudflare/vitest-plugin@15252

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@15252

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@15252

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@15252

wrangler

npm i https://pkg.pr.new/wrangler@15252

commit: fc9c3f1

devin-ai-integration[bot]

This comment was marked as resolved.

@dario-piotrowicz dario-piotrowicz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much @GregoryCollett 😄

The PR looks overall good to me but there are a few bits that I think should be looked at before merging, could you please have a look? 🙏

Comment thread packages/wrangler/src/__tests__/api/startDevWorker/DevEnv.test.ts Outdated
Comment thread .changeset/tidy-moons-provide.md Outdated
Comment thread packages/wrangler/src/api/startDevWorker/DevEnv.ts Outdated
Comment thread packages/wrangler/templates/startDevWorker/ProxyWorker.ts Outdated
Comment thread packages/wrangler/templates/startDevWorker/ProxyWorker.ts Outdated
Comment thread packages/wrangler/templates/startDevWorker/ProxyWorker.ts Outdated
Comment thread packages/wrangler/templates/startDevWorker/ProxyWorker.ts Outdated
- Move the GET/HEAD retry into fetch's rejection handler so errors
  thrown while post-processing a received response are reported, never
  retried (and can never re-run the UserWorker's handler).
- Make the attempt parameter optional and 0-indexed.
- Report exhausted retries explicitly ("failed after 3 attempts").
- data: {} in the DevEnv test; full issue URL in the DevEnv comment.
- Rewrite the changeset to be user-facing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@GregoryCollett

Copy link
Copy Markdown
Contributor Author

All review comments addressed in 8adab46:

  • Retry is attached to fetch's rejection handler, so only connection-level failures with no received response are retried; post-processing errors report on the first attempt and can never re-run the UserWorker's handler.
  • Exhausted retries are explicit in the logged error ((failed after 3 attempts)).
  • attemptUserWorkerFetch(attempt = 0) with a bare call, plus a JSDoc (kept as an arrow — the handlers need the ProxyWorker's lexical this to re-read this.proxyData across async boundaries).
  • Changeset rewritten user-facing; data: {} in the test; full issue URL in the DevEnv comment.

Re-verified end-to-end on the updated branch: SIGKILLing the UserWorker runtime under a 4-thread GET storm — 1034 requests, 0 client-visible failures, 4 recovered on the delayed third attempt, dev server kept serving. DevEnv.test.ts also passes locally now — the local-vitest blocker in the PR description turned out to be Node 22.14: below 22.18, tsdown bundles its config file, which rewrites import.meta.dirname inside scripts/embed-workers.ts to the package root, so ../templates resolves to packages/templates and @cloudflare/remote-bindings#build fails. On Node ≥ 22.18 it builds fine. Happy to file that separately.

While in here I noticed packages/remote-bindings/templates/startDevWorker/ProxyWorker.ts — the trimmed fork used for remote proxy sessions — still has the pre-fix error path. Happy to open a follow-up issue/PR if useful.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

The ProxyWorker's report crosses a JSON boundary, so castErrorCause wraps
a plain object in a message-less Error and the real detail (method, URL,
attempt count, underlying error) sits on .cause. Reading event.cause.message
logged nothing after the colon — the same empty-error symptom this PR fixes.

The test now builds its cause through JSON and castErrorCause, so it can no
longer pass with a message the production path never has.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@workers-devprod

workers-devprod commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • ✅ @cloudflare/wrangler
Show detailed file reviewers

devin-ai-integration[bot]

This comment was marked as resolved.

A retry scheduled before a reload begins would fire with the captured
proxyData, so it could reach the pre-reload Worker or send a stale
preview token. Re-check proxyData identity when the timer fires and
requeue instead, letting processQueue rebuild the request from current
routing and headers. Compares identity rather than origin because the
origin can be unchanged while the headers are stale.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@workers-devprod workers-devprod left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codeowners reviews satisfied

@github-project-automation github-project-automation Bot moved this from Untriaged to Approved in workers-sdk Sep 7, 2026
@dario-piotrowicz

Copy link
Copy Markdown
Member

Thanks for the changes @GregoryCollett 🙏

@dario-piotrowicz
dario-piotrowicz merged commit 682cd44 into cloudflare:main Sep 7, 2026
88 of 92 checks passed
@github-project-automation github-project-automation Bot moved this from Approved to Done in workers-sdk Sep 7, 2026
LiyanChen-X added a commit to LiyanChen-X/workers-sdk that referenced this pull request Sep 9, 2026
The rejected UserWorker fetch is now answered with a 502 inside the
ProxyWorker itself, so the "Error inside ProxyWorker" report can only
mean the ProxyWorker failed while post-processing a received response —
a genuine proxy defect. Drop the recoverable branch DevEnv gained in
cloudflare#15252 so such reports are fatal again, and flip its regression test to
assert the re-emit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RBs8hK2eBFkHBgik22QB97
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

3 participants