Skip to content

v0.6.46: mothership streaming fixes, brightdata integration#4190

Merged
icecrasher321 merged 2 commits intomainfrom
staging
Apr 16, 2026
Merged

v0.6.46: mothership streaming fixes, brightdata integration#4190
icecrasher321 merged 2 commits intomainfrom
staging

Conversation

@icecrasher321
Copy link
Copy Markdown
Collaborator

fix(brightdata): fix async Discover API (polling), echo-back fields via params, registry ordering, improved error handling, and platform-aligned timeout (#4188)
fix(mothership): chat stream structuring + logs resource post fix (#4189)

waleedlatif1 and others added 2 commits April 15, 2026 16:20
…ry ordering (#4188)

* fix(brightdata): use params for echo-back fields in transformResponse

transformResponse receives params as its second argument. Use it to
return the original url, query, snapshotId, and searchEngine values
instead of hardcoding null or extracting from response data that may
not contain them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(brightdata): handle async Discover API with polling

The Bright Data Discover API is asynchronous — POST /discover returns
a task_id, and results must be polled via GET /discover?task_id=...
The previous implementation incorrectly treated it as synchronous,
always returning empty results.

Uses postProcess (matching Firecrawl crawl pattern) to poll every 3s
with a 120s timeout until status is "done".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(brightdata): alphabetize block registry entry

Move box before brandfetch/brightdata to maintain alphabetical ordering.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* lint

* fix(brightdata): return error objects instead of throwing in postProcess

The executor wraps postProcess in try-catch and falls back to the
intermediate transformResponse result on error, which has success: true
with empty results. Throwing errors would silently return empty results.

Match Firecrawl's pattern: return { ...result, success: false, error }
instead of throwing. Also add taskId to BrightDataDiscoverResponse type
to eliminate unsafe casts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(brightdata): use platform execution timeout for Discover polling

Replace hardcoded 120s timeout with DEFAULT_EXECUTION_TIMEOUT_MS to
match Firecrawl and other async polling tools. Respects platform-
configured limits (300s free, 3000s paid).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
)

* fix(mothership): chat streaming structure

* fix logs resource thinking bug"

* address comments

* address comments
@vercel
Copy link
Copy Markdown

vercel bot commented Apr 15, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Apr 15, 2026 11:45pm

Request Review

@cursor
Copy link
Copy Markdown

cursor bot commented Apr 15, 2026

PR Summary

Medium Risk
Touches core chat streaming and cancellation paths plus client-side caching, which can affect message ordering/duplication and queued follow-up delivery. Bright Data changes add polling and output-shaping that may impact runtime behavior and timeouts.

Overview
Improves mothership/copilot chat streaming by introducing buildEffectiveChatTranscript to merge Redis stream snapshots into the returned chat messages, and updating both copilot and mothership chat GET APIs to normalize persisted messages and append a consistent “live assistant” tail while a stream is active.

Refactors the home useChat hook to drive streaming updates through the React Query TaskChatHistory cache (stable live-assistant:<streamId> IDs), improve stop/cancel handling (mark executing tools as cancelled), and add a sessionStorage-backed queued-send handoff to safely resume queued follow-ups across navigation/stop boundaries. The /api/copilot/chat/stop route now always persists a cancelled completion block (and uses generateId), with a new route test.

Fixes Bright Data tool behavior by polling the async Discover API until completion (platform timeout-aligned), echoing key output fields from request params across several tools, and adding taskId to Discover outputs; also expands resource attachment types, tweaks handleTaskStatusEvent to invalidate only the workspace task list, and includes minor doc/registry/lockfile updates.

Reviewed by Cursor Bugbot for commit 377712c. Configure here.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Apr 15, 2026

Greptile Summary

This PR delivers two independent fixes: (1) BrightData Discover now correctly polls for async task results via postProcess and echoes input params back through output fields across all BrightData tools; (2) Mothership streaming is refactored so the live assistant transcript is materialized from Redis-backed stream events (effective-transcript.ts) and pushed directly into the React Query cache, replacing the previous local useState approach and eliminating the need for server refetches during an active stream.

Confidence Score: 5/5

Safe to merge; all findings are P2 quality-of-life improvements that do not affect the primary streaming or tool-execution paths.

Three P2 findings: polling elapsed-time precision, no retry on transient HTTP errors in BrightData Discover, and reduced cache invalidation for passive observers. None break core functionality — the streaming experience is correctly handled, and the BrightData issues only affect edge-case reliability in the polling loop.

apps/sim/tools/brightdata/discover.ts — timing and error-handling in the polling loop

Important Files Changed

Filename Overview
apps/sim/tools/brightdata/discover.ts Rewrites Discover tool to use async polling via postProcess; elapsed-time counter only counts sleep intervals not fetch latency, and any transient HTTP error immediately terminates the poll.
apps/sim/lib/copilot/chat/effective-transcript.ts New module that materializes a live assistant message from Redis-backed stream events, replacing the previous per-component state approach; logic is thorough and well-tested.
apps/sim/app/workspace/[workspaceId]/home/hooks/use-chat.ts Large refactor moving live stream state from local useState to React Query cache; adds queued-send handoff via sessionStorage; looks architecturally sound.
apps/sim/hooks/use-task-events.ts Scopes list invalidation to the current workspace (improvement), but removes detail invalidation for completed events — passive observers in other tabs may see stale chat transcripts.
apps/sim/app/api/copilot/chat/stop/route.ts Now always appends a stopped assistant message (even with empty content), fixes UUID generation to use generateId(), and synthesizes correct contentBlocks with a stopped marker.

Sequence Diagram

sequenceDiagram
    participant Client as useChat browser
    participant RQ as React Query Cache
    participant Server as Next.js API
    participant Redis as Redis stream events

    Client->>Server: POST copilot/chat send message
    Server-->>Client: SSE stream text/tool/complete events
    Client->>RQ: upsertTaskChatHistory on each flush

    alt Stream completes normally
        Client->>Client: finalize
        Client->>RQ: invalidateChatQueries includeDetail true
        RQ->>Server: GET copilot/chat
        Server->>Redis: fetch streamSnapshot
        Server->>Server: buildEffectiveChatTranscript
        Server-->>RQ: persisted messages plus live snapshot
    end

    alt SSE task_status event any tab
        Server-->>Client: task_status event
        Client->>RQ: invalidateQueries taskKeys.list workspaceId only
    end
Loading

Comments Outside Diff (1)

  1. apps/sim/hooks/use-task-events.ts, line 41-54 (link)

    P2 Stale detail cache for passive observers after stream completes

    The previous implementation invalidated taskKeys.detail(chatId) when a completed event arrived. That guaranteed any tab watching a chat (but not actively streaming) would re-fetch the final persisted messages.

    With that invalidation removed, a passive tab (e.g. the same chat open in a second browser tab, or a tab that navigated away mid-stream) will not see the final transcript until its own staleTime expires or it navigates. The active streaming tab is fine because finalize() still calls invalidateChatQueries. If passive-observer freshness matters, consider re-adding a targeted detail invalidation here.

Reviews (1): Last reviewed commit: "fix(mothership): chat stream structuring..." | Re-trigger Greptile

Comment on lines +127 to +137
let elapsedTime = 0

while (elapsedTime < MAX_POLL_TIME_MS) {
try {
const pollResponse = await fetch(
`https://api.brightdata.com/discover?task_id=${encodeURIComponent(taskId)}`,
{
method: 'GET',
headers: {
Authorization: `Bearer ${params.apiKey}`,
},
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Elapsed-time counter under-counts wall-clock duration

elapsedTime only accumulates the sleep interval and ignores the time spent in each fetch call. On a slow network, where each poll round-trip takes seconds, the loop can run well past MAX_POLL_TIME_MS before the guard fires.

Replace with a wall-clock deadline:

Suggested change
let elapsedTime = 0
while (elapsedTime < MAX_POLL_TIME_MS) {
try {
const pollResponse = await fetch(
`https://api.brightdata.com/discover?task_id=${encodeURIComponent(taskId)}`,
{
method: 'GET',
headers: {
Authorization: `Bearer ${params.apiKey}`,
},
const deadline = Date.now() + MAX_POLL_TIME_MS
while (Date.now() < deadline) {

And remove elapsedTime += POLL_INTERVAL_MS at the end of the loop body.

Comment on lines +141 to +147
if (!pollResponse.ok) {
return {
...result,
success: false,
error: `Failed to poll discover results: ${pollResponse.statusText}`,
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No retry on transient HTTP errors during polling

Any non-2xx response (including 429 Too Many Requests or 503 Service Unavailable) immediately terminates the poll and returns a failure. A single transient error aborts the entire discovery task, even though the Bright Data job itself may still be running.

Consider retrying on retriable status codes (e.g. 429, 5xx) up to a small limit before giving up, or at minimum logging the status code in the error message so callers can distinguish transient from permanent failures.

Copy link
Copy Markdown

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 377712c. Configure here.

Comment thread apps/sim/app/api/copilot/chat/stop/route.ts
Comment thread apps/sim/lib/copilot/chat/effective-transcript.ts
@icecrasher321 icecrasher321 merged commit c0bc62c into main Apr 16, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants