Skip to content

Add stream error handlers to prevent process crashes#3385

Open
rajin-kichannagari wants to merge 4 commits into
redhat-developer:mainfrom
rajin-kichannagari:fix/stream-error-handling-rhidp-13064
Open

Add stream error handlers to prevent process crashes#3385
rajin-kichannagari wants to merge 4 commits into
redhat-developer:mainfrom
rajin-kichannagari:fix/stream-error-handling-rhidp-13064

Conversation

@rajin-kichannagari

@rajin-kichannagari rajin-kichannagari commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Attach error handlers to streaming responses to prevent unhandled error events from crashing the Node.js process when the LCS connection drops mid-stream.

Addresses security vulnerability identified in Red Hat Product Security threat model (dated Mar 15, 2026).

Changes

  • Add error handler to nodeStream in POST /v1/query endpoint (router.ts)
  • Add error handler to response object to detect client disconnections
  • Add error handlers to both source and transform streams in notebooks query endpoint (/v1/sessions/:sessionId/query)
  • Ensure streams are properly destroyed on error to prevent hanging connections

Affected Endpoints

  • /v1/query endpoint (Lightspeed router)
  • /v1/sessions/:sessionId/query endpoint (Notebooks router)

Checklist

  • A changeset describing the change and affected packages
  • Added or Updated documentation
  • Tests for new functionality and regression tests for bug fixes
  • Screenshots attached (for UI changes)

Testing note: Verified with manual tests simulating LCS stream failures and client disconnections. Error handlers work correctly and prevent process crashes. Existing unit tests verify the happy path.

Attach error handlers to streaming responses to prevent unhandled error events
from crashing the Node.js process when the LCS connection drops mid-stream.

Changes:
- Add error handler to nodeStream in POST /v1/query endpoint (router.ts)
- Add error handler to response object to detect client disconnections
- Add error handlers to both source and transform streams in notebooks query endpoint
- Ensure streams are properly destroyed on error to prevent hanging connections

Addresses security vulnerability identified in Red Hat Product Security
threat model (dated Mar 15, 2026).

Co-Authored-By: Rajin Kichannagari <rkichann@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

This pull request adds a new top-level directory under workspaces/. Please follow Submitting a Pull Request for a New Workspace in CONTRIBUTING.md.

@rhdh-gh-app

rhdh-gh-app Bot commented Jun 11, 2026

Copy link
Copy Markdown

Missing Changesets

The following package(s) are changed by this PR but do not have a changeset:

  • @red-hat-developer-hub/backstage-plugin-lightspeed-backend

See CONTRIBUTING.md for more information about how to add changesets.

Changed Packages

Package Name Package Path Changeset Bump Current Version
@red-hat-developer-hub/backstage-plugin-lightspeed-backend workspaces/lightspeed/plugins/lightspeed-backend none v2.9.1

JslYoon and others added 2 commits June 12, 2026 11:12
Fixes TS7006 'Parameter implicitly has an any type' errors in CI.
Also removes leftover debug console.log.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 10.34483% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.85%. Comparing base (6a1b1f0) to head (1063b5a).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3385      +/-   ##
==========================================
- Coverage   52.87%   52.85%   -0.02%     
==========================================
  Files        2408     2408              
  Lines       87639    87668      +29     
  Branches    24294    24295       +1     
==========================================
+ Hits        46336    46341       +5     
- Misses      39800    39824      +24     
  Partials     1503     1503              
Flag Coverage Δ *Carryforward flag
adoption-insights 83.58% <ø> (ø) Carriedforward from 8a1740e
ai-integrations 70.03% <ø> (ø) Carriedforward from 8a1740e
app-defaults 69.60% <ø> (ø) Carriedforward from 8a1740e
augment 46.39% <ø> (ø) Carriedforward from 8a1740e
bulk-import 72.69% <ø> (ø) Carriedforward from 8a1740e
cost-management 17.48% <ø> (ø) Carriedforward from 8a1740e
dcm 60.27% <ø> (ø) Carriedforward from 8a1740e
extensions 62.17% <ø> (ø) Carriedforward from 8a1740e
global-floating-action-button 74.30% <ø> (ø) Carriedforward from 8a1740e
global-header 61.63% <ø> (ø) Carriedforward from 8a1740e
homepage 52.60% <ø> (ø) Carriedforward from 8a1740e
install-dynamic-plugins 56.23% <ø> (ø) Carriedforward from 8a1740e
konflux 91.01% <ø> (ø) Carriedforward from 8a1740e
lightspeed 68.19% <10.34%> (-0.27%) ⬇️
mcp-integrations 85.46% <ø> (ø) Carriedforward from 8a1740e
orchestrator 37.33% <ø> (ø) Carriedforward from 8a1740e
quickstart 62.09% <ø> (ø) Carriedforward from 8a1740e
sandbox 79.56% <ø> (ø) Carriedforward from 8a1740e
scorecard 83.93% <ø> (ø) Carriedforward from 8a1740e
theme 64.44% <ø> (ø) Carriedforward from 8a1740e
translations 8.49% <ø> (ø) Carriedforward from 8a1740e
x2a 57.83% <ø> (ø) Carriedforward from 8a1740e

*This pull request uses carry forward flags. Click here to find out more.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a1b1f0...1063b5a. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JslYoon JslYoon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 3 issues:

  1. Transform stream error handler doesn't destroy upstream body stream
    (resource leak)

    .status(500)
    .json({ status: 'error', error: 'Stream error occurred' });
    }
    transformStream.destroy();
    });

    When transformStream errors, body.destroy() is never called. The
    upstream stream keeps reading data, causing memory leaks.

  2. Double-destroy race condition in response error handler

logger.error(
`Transform stream error while processing notebook query: ${error}`,
);
if (!res.headersSent) {
res.status(500).json({
status: 'error',
error: 'Processing error occurred',

body.destroy() triggers body.on('error') which calls
transformStream.destroy(), then line 536 calls it again.
transformStream.destroy() executes twice.

  1. Upstream fetch not cancelled on client disconnect (connection pool
    exhaustion)

    `Upstream stream error while processing query: ${error}`,
    );
    if (!response.headersSent) {
    response.status(500).json({ error: 'Stream error occurred' });
    }
    });
    // Handle client disconnection (RHIDP-13064)
    response.on('error', error => {
    logger.warn(`Client disconnected while processing query: ${error}`);
    nodeStream.destroy();
    });
    nodeStream.pipe(response);

    nodeStream.destroy() doesn't cancel the underlying fetch. The upstream
    HTTP connection stays open, draining data from Lightspeed backend even after
    client disconnects.

}
});

// Handle client disconnection (RHIDP-13064)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to add in task #

if (fetchResponse.body) {
const nodeStream = Readable.fromWeb(fetchResponse.body as any);

// Handle stream errors (RHIDP-13064)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for (RHIDP-13064)

- Add AbortController to cancel underlying fetch on client disconnect,
  preventing connection pool exhaustion
- Add body.destroy() + abortController.abort() in transform stream
  error handler to prevent resource leaks
- Remove redundant transformStream.destroy() from client disconnect
  handler to avoid double-destroy race condition
- Remove internal ticket references from inline comments

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@rajin-kichannagari rajin-kichannagari force-pushed the fix/stream-error-handling-rhidp-13064 branch from 13b99a6 to 1063b5a Compare June 12, 2026 20:58
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants