v0.6.46: mothership streaming fixes, brightdata integration#4190
v0.6.46: mothership streaming fixes, brightdata integration#4190icecrasher321 merged 2 commits intomainfrom
Conversation
…ry ordering (#4188) * fix(brightdata): use params for echo-back fields in transformResponse transformResponse receives params as its second argument. Use it to return the original url, query, snapshotId, and searchEngine values instead of hardcoding null or extracting from response data that may not contain them. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(brightdata): handle async Discover API with polling The Bright Data Discover API is asynchronous — POST /discover returns a task_id, and results must be polled via GET /discover?task_id=... The previous implementation incorrectly treated it as synchronous, always returning empty results. Uses postProcess (matching Firecrawl crawl pattern) to poll every 3s with a 120s timeout until status is "done". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(brightdata): alphabetize block registry entry Move box before brandfetch/brightdata to maintain alphabetical ordering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * lint * fix(brightdata): return error objects instead of throwing in postProcess The executor wraps postProcess in try-catch and falls back to the intermediate transformResponse result on error, which has success: true with empty results. Throwing errors would silently return empty results. Match Firecrawl's pattern: return { ...result, success: false, error } instead of throwing. Also add taskId to BrightDataDiscoverResponse type to eliminate unsafe casts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(brightdata): use platform execution timeout for Discover polling Replace hardcoded 120s timeout with DEFAULT_EXECUTION_TIMEOUT_MS to match Firecrawl and other async polling tools. Respects platform- configured limits (300s free, 3000s paid). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview Refactors the home Fixes Bright Data tool behavior by polling the async Discover API until completion (platform timeout-aligned), echoing key output fields from request params across several tools, and adding Reviewed by Cursor Bugbot for commit 377712c. Configure here. |
Greptile SummaryThis PR delivers two independent fixes: (1) BrightData Discover now correctly polls for async task results via Confidence Score: 5/5Safe to merge; all findings are P2 quality-of-life improvements that do not affect the primary streaming or tool-execution paths. Three P2 findings: polling elapsed-time precision, no retry on transient HTTP errors in BrightData Discover, and reduced cache invalidation for passive observers. None break core functionality — the streaming experience is correctly handled, and the BrightData issues only affect edge-case reliability in the polling loop. apps/sim/tools/brightdata/discover.ts — timing and error-handling in the polling loop Important Files Changed
Sequence DiagramsequenceDiagram
participant Client as useChat browser
participant RQ as React Query Cache
participant Server as Next.js API
participant Redis as Redis stream events
Client->>Server: POST copilot/chat send message
Server-->>Client: SSE stream text/tool/complete events
Client->>RQ: upsertTaskChatHistory on each flush
alt Stream completes normally
Client->>Client: finalize
Client->>RQ: invalidateChatQueries includeDetail true
RQ->>Server: GET copilot/chat
Server->>Redis: fetch streamSnapshot
Server->>Server: buildEffectiveChatTranscript
Server-->>RQ: persisted messages plus live snapshot
end
alt SSE task_status event any tab
Server-->>Client: task_status event
Client->>RQ: invalidateQueries taskKeys.list workspaceId only
end
|
| let elapsedTime = 0 | ||
|
|
||
| while (elapsedTime < MAX_POLL_TIME_MS) { | ||
| try { | ||
| const pollResponse = await fetch( | ||
| `https://api.brightdata.com/discover?task_id=${encodeURIComponent(taskId)}`, | ||
| { | ||
| method: 'GET', | ||
| headers: { | ||
| Authorization: `Bearer ${params.apiKey}`, | ||
| }, |
There was a problem hiding this comment.
Elapsed-time counter under-counts wall-clock duration
elapsedTime only accumulates the sleep interval and ignores the time spent in each fetch call. On a slow network, where each poll round-trip takes seconds, the loop can run well past MAX_POLL_TIME_MS before the guard fires.
Replace with a wall-clock deadline:
| let elapsedTime = 0 | |
| while (elapsedTime < MAX_POLL_TIME_MS) { | |
| try { | |
| const pollResponse = await fetch( | |
| `https://api.brightdata.com/discover?task_id=${encodeURIComponent(taskId)}`, | |
| { | |
| method: 'GET', | |
| headers: { | |
| Authorization: `Bearer ${params.apiKey}`, | |
| }, | |
| const deadline = Date.now() + MAX_POLL_TIME_MS | |
| while (Date.now() < deadline) { |
And remove elapsedTime += POLL_INTERVAL_MS at the end of the loop body.
| if (!pollResponse.ok) { | ||
| return { | ||
| ...result, | ||
| success: false, | ||
| error: `Failed to poll discover results: ${pollResponse.statusText}`, | ||
| } | ||
| } |
There was a problem hiding this comment.
No retry on transient HTTP errors during polling
Any non-2xx response (including 429 Too Many Requests or 503 Service Unavailable) immediately terminates the poll and returns a failure. A single transient error aborts the entire discovery task, even though the Bright Data job itself may still be running.
Consider retrying on retriable status codes (e.g. 429, 5xx) up to a small limit before giving up, or at minimum logging the status code in the error message so callers can distinguish transient from permanent failures.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 377712c. Configure here.

fix(brightdata): fix async Discover API (polling), echo-back fields via params, registry ordering, improved error handling, and platform-aligned timeout (#4188)
fix(mothership): chat stream structuring + logs resource post fix (#4189)