Fix 500 error on operation logs download due to JSON serialization#3366
Open
Bhanunamikaze wants to merge 2 commits into
Open
Fix 500 error on operation logs download due to JSON serialization#3366Bhanunamikaze wants to merge 2 commits into
Bhanunamikaze wants to merge 2 commits into
Conversation
a967a4e to
7466e68
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description of Issue
When operations execute agent abilities that output untrusted or malformed data—such as random binary dumps, unescaped null characters (
\x00), or invalid UTF-16 surrogates (U+D800 - U+DFFF)—the Caldera server can silently store these outputs in the raw result logs.When a user attempts to download the operation's Event Logs or Report via the web UI or API (
/api/v2/operations/<id>/event-logs), these stored characters break Python'sjson.dumpsmechanism locally insideweb.json_response(). This leads directly to a500 Internal Server Error (UnicodeEncodeError), entirely blocking the user from downloading the logs for operations that encountered this scenario.Proposed Fix
This PR implements a defense-in-depth sanitization approach to harden the serialization pipeline, prioritizing availability and readability of logs over pure verbatim rendering of corrupted agent output blocks:
BaseWorld.decode_bytesto strictly strip surrogate character anomalies dynamically to lock the initial layer._sanitize_for_jsondynamically insidec_operation.py. If a specificLinkor ability output string is un-serializable, it falls back to safely encoding it as pure ASCII, escaping dangerous non-printable bytes.reportparsing iterators and_convert_link_to_event_loggenerators in activetry/exceptbounds. If a single agent link returns a fundamentally malformed payload that physically breaksjson.loadsor customdictmapping, it isolates that specific block, falls back to raw data, and lets the rest of the report generate gracefully rather than killing the entire request.operation_api.pyto enforceensure_ascii=True, locking any dynamically missed payloads down before they are dispatched through the asynchronous web router.