[Bug]: ToolCallMatchingProperty.manipulation_indices crashes on orphaned observation, permanently bricking conversations after a restart

[rajshah4]

Is there an existing issue for the same bug?

• I have searched existing issues and this is not a duplicate.

Bug Description

ToolCallMatchingProperty (in openhands/sdk/context/view/properties/tool_call_matching.py) has two methods that make inconsistent assumptions about the same invariant:

• enforce() (line 25) gracefully drops an observation-like event whose tool_call_id has no matching ActionEvent (an "orphaned observation").
• manipulation_indices() (line 93) raises KeyError on the same state, via a bare set.remove(event.tool_call_id), with an explicit comment saying it intentionally raises to enforce the 1-action→1-result invariant.

View.manipulation_indices (openhands/sdk/context/view/view.py:44-49) calls manipulation_indices() on every registered property without first calling enforce_properties(). So if the view ever reaches an orphaned-observation state, the conversation becomes permanently unrecoverable: every subsequent LLM completion rebuilds the view and throws the same KeyError, classified internal / retryable: false with user_action: "none".

The orphaned-observation state is reached through the crash-recovery path in openhands/agent_server/event_service.py:1116: on agent-server restart, it emits an AgentErrorEvent (which is an ObservationBaseEvent) for an interrupted in-flight tool call. That AgentErrorEvent carries a tool_call_id, and the matching ActionEvent can later be absent from a view — dropped by enforcement on a prior turn, lost to a parse/forward-incompat issue (observed here: 220/408 on-disk events failed to parse because agent-server tool types like BrowserNavigateTool/TerminalAction are unknown to the SDK install), or removed by future condensation. Once the action is gone but the AgentErrorEvent remains, the next View.manipulation_indices call crashes.

Expected Behavior

A view containing an orphaned observation (an ObservationBaseEvent whose tool_call_id has no matching ActionEvent) should be handled consistently by both methods of ToolCallMatchingProperty:

• enforce() already drops the orphan gracefully.
• manipulation_indices() should tolerate the same state (e.g. via discard() instead of remove(), or by guarding the removal) rather than raising KeyError.

Equivalently: View.manipulation_indices should never raise on a view that View.enforce_properties is able to repair. A restart-interrupted tool call should not brick the conversation; the user should be able to continue after the recovery AgentErrorEvent is emitted.

Actual Behavior

The conversation enters execution_status=error with a non-retryable KeyError: '<tool_call_id>' and cannot be continued. Every subsequent turn hits the same KeyError because the view is rebuilt from the same persisted events. The failure is latent — in the reported instance it surfaced ~7 minutes and ~230 events after the restart that produced the orphaned AgentErrorEvent.

Reproduced against a local Agent Canvas backend running the SDK-backed agent server (openhands-agent-server==1.42.1, openhands-sdk==1.42.1). The defect is reproducible without an agent-server restart, model credentials, or a live conversation — running the minimal script below with python raises the exact KeyError:

# against openhands-sdk==1.42.1
python repro_synthetic.py

which prints:

A) View.enforce_properties -> OK (1 event dropped, view repaired)
B) ToolCallMatchingProperty.manipulation_indices -> KeyError: 'chatcmpl-tool-8018b0bacc905a1b'
C) View.manipulation_indices -> KeyError: 'chatcmpl-tool-8018b0bacc905a1b'

The full script is under Minimal Code Sample below. The same KeyError is also reproducible by loading all on-disk events from the affected conversation and running each property's manipulation_indices() individually (python repro_full_fidelity.py) — only ToolCallMatchingProperty raises; ObservationUniquenessProperty, BatchAtomicityProperty, and ToolLoopAtomicityProperty all succeed.

Steps to Reproduce

A) Synthetic repro (self-contained, no restart, no network, <1s):

1. With openhands-sdk==1.42.1 installed, save the script under Minimal Code Sample as repro_synthetic.py.
2. Run python repro_synthetic.py.
3. Observe:
   • enforce() succeeds (drops the orphan, view repaired).
   • manipulation_indices() raises KeyError: 'chatcmpl-tool-8018b0bacc905a1b'.
   • View.manipulation_indices raises the same KeyError.

B) Full-fidelity repro from the affected conversation:

1. Load all on-disk events from the affected conversation's persistence directory (see Screenshots and Additional Context for the conversation id and path).
2. Run python repro_full_fidelity.py.
3. Observe ToolCallMatchingProperty raises KeyError: 'chatcmpl-tool-8018b0bacc905a1b' while ObservationUniquenessProperty, BatchAtomicityProperty, and ToolLoopAtomicityProperty all succeed.

Acceptance Criteria

• ToolCallMatchingProperty.manipulation_indices no longer raises KeyError on a view containing an orphaned observation (observation with no matching ActionEvent).
• View.manipulation_indices and View.enforce_properties agree on every view state — if enforce can repair it, manipulation_indices must not crash on it.
• The duplicate-observation detection intent preserved (the existing comment says remove() is intentional to catch a second observation for the same tool_call_id); a regression test covers both the duplicate-observation case and the zero-action orphan case.
• A conversation that previously errored with this KeyError can be continued (or at least produces a recoverable, agent-correctable error rather than a permanent internal / retryable: false KeyError).
• A regression test constructs an orphaned observation (e.g. an AgentErrorEvent with no matching ActionEvent) and asserts View.manipulation_indices returns without raising.

Installation Method

npm install -g @openhands/agent-canvas (bundled uvx invocation of openhands-agent-server==1.42.1 + openhands-sdk==1.42.1).

SDK Version

openhands-sdk==1.42.1, openhands-agent-server==1.42.1 (bundled by Agent Canvas).

Version Confirmation

• I have confirmed this bug exists on the LATEST version of OpenHands SDK

Confirmed on 1.42.1 (the version bundled by the latest Agent Canvas release at reproduction time).

Python Version

3.12.12 (Agent Canvas uv env); the defect is not Python-version-specific.

Model Name (if applicable)

openhands/glm-5.2 with LLMSummarizingCondenser (max_size=240, keep_first=2). The bug is independent of the model — it fires in the view/property layer before the LLM is called.

Operating System

macOS (arm64); not platform-specific.

Logs and Error Messages

Persisted ConversationErrorEvent (repeated 5 times, all internal / retryable: false):

{
  "kind": "ConversationErrorEvent",
  "code": "KeyError",
  "detail": "'chatcmpl-tool-8018b0bacc905a1b'",
  "classification": {
    "kind": "internal",
    "retryable": false,
    "user_action": "none"
  }
}

Traceback from the synthetic repro:

Traceback (most recent call last):
  File ".../repro_synthetic.py", line 92, in <module>
    mi = View(events=list(view_events)).manipulation_indices
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../openhands/sdk/context/view/view.py", line 49, in manipulation_indices
    results &= property.manipulation_indices(self.events)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../openhands/sdk/context/view/properties/tool_call_matching.py", line 93, in manipulation_indices
    pending_tool_call_ids.remove(event.tool_call_id)
KeyError: 'chatcmpl-tool-8018b0bacc905a1b'

Minimal Code Sample

repro_synthetic.py — self-contained, runs in under a second against openhands-sdk==1.42.1:

import os, sys, traceback
os.environ["OPENHANDS_SUPPRESS_BANNER"] = "1"
sys.path.insert(0, "<site-packages>")  # openhands-sdk 1.42.1

from openhands.sdk.event import AgentErrorEvent, EventID, MessageEvent
from openhands.sdk.event.base import LLMConvertibleEvent
from openhands.sdk.context.view.view import View
from openhands.sdk.context.view.properties.tool_call_matching import (
    ToolCallMatchingProperty,
)
from openhands.sdk.llm.message import Message, TextContent

TOOL_CALL_ID = "chatcmpl-tool-8018b0bacc905a1b"

# An observation-like event (AgentErrorEvent IS an ObservationBaseEvent) with a
# tool_call_id for which there is NO ActionEvent in the view. This is exactly
# what event_service.py:1116 writes after a restart-interrupted tool call, once
# the matching ActionEvent has been dropped from the active view.
orphaned_error = AgentErrorEvent(
    id=EventID("evt-154"), source="agent", tool_name="task",
    tool_call_id=TOOL_CALL_ID,
    error="A restart occurred while this tool was in progress. "
          "This may indicate a fatal memory error or system crash. "
          "The tool execution was interrupted and did not complete.",
)
user_msg = MessageEvent(
    id=EventID("evt-387"), source="user",
    llm_message=Message(role="user",
        content=[TextContent(text="ok, cool, double check there isn't an existing ticket")]),
)
view_events = [orphaned_error, user_msg]

# A) enforce() handles the orphan gracefully:
v = View(events=list(view_events))
v.enforce_properties(all_events=view_events)   # OK -> 1 event remains

# B) manipulation_indices() crashes on the same state:
try:
    ToolCallMatchingProperty().manipulation_indices(list(view_events))
except KeyError as e:
    print(f"KeyError: {e}")   # -> KeyError: 'chatcmpl-tool-8018b0bacc905a1b'

# C) View.manipulation_indices (called every turn / during condensation) crashes:
try:
    View(events=list(view_events)).manipulation_indices
except KeyError as e:
    print(f"KeyError: {e}")   # -> KeyError: 'chatcmpl-tool-8018b0bacc905a1b'

Output (observed):

A) View.enforce_properties -> OK (1 event dropped, view repaired)
B) ToolCallMatchingProperty.manipulation_indices -> KeyError: 'chatcmpl-tool-8018b0bacc905a1b'
C) View.manipulation_indices -> KeyError: 'chatcmpl-tool-8018b0bacc905a1b'

Screenshots and Additional Context

Affected real conversation. Conversation 2caeb75f-1a60-4014-a66a-24c945644917 on a local Agent Canvas backend errored with this exact KeyError. On-disk events at …/.openhands/agent-canvas/dev_conversations/2caeb75f1a604014a66a24c945644917/events/ (events 151, 154, 386, 391, 396, 401, 406 reference the orphaned id chatcmpl-tool-8018b0bacc905a1b).

Timeline of the real failure:

• Event 151: agent called the task tool (subagent delegation), tool_call_id = chatcmpl-tool-8018b0bacc905a1b.
• Event 154: AgentErrorEvent on the same id — "A restart occurred while this tool was in progress…" — emitted by the restart-recovery path at event_service.py:1116. No ObservationEvent was ever written for that id.
• The conversation continued normally for ~7 minutes and ~230 events (the orphaned pair was not at the tail of the active view yet).
• Event 387: user sent a new message. The next View.manipulation_indices call (called during completion / condensation at llm_summarizing_condenser.py:282) hit the orphaned observation and raised KeyError: 'chatcmpl-tool-8018b0bacc905a1b'. Conversation stuck in error.

Why the orphan survived enforce. AgentErrorEvent is an ObservationBaseEvent, so when both the ActionEvent and the AgentErrorEvent are present, enforce keeps the pair. The orphan arises when the ActionEvent is later dropped from the active view while the AgentErrorEvent survives. In the full-fidelity repro, 220 of the 408 on-disk events failed to parse with the installed SDK because the agent-server persisted tool types (BrowserNavigateTool, TerminalAction) that the local SDK install does not know — a version skew between the agent-server and the SDK that left the ActionEvent unparsed while the AgentErrorEvent (which carries no tool-specific action payload) parsed fine. Any future condensation or property that drops the action without also dropping the paired error would produce the same state.

Suggested fix (two complementary options):

1. Minimal: make manipulation_indices consistent with enforce — use discard() instead of remove(), or guard the removal so an orphaned observation (no matching action) is tolerated. Preserve the duplicate-detection intent from the comment with an explicit if event.tool_call_id in pending_tool_call_ids: check and a warning log otherwise. This directly stops the crash.

2. Stronger invariant: make View.manipulation_indices call enforce_properties first (or guarantee the view is enforced before any property reads it), so the two methods never disagree.

Option 1 is the lower-risk fix and directly stops the crash regardless of how the orphan arises. Option 2 is the stronger invariant but may change condensation behavior.

Separately, the restart-recovery path (event_service.py:1116) could be hardened so the matching ActionEvent is never later droppable without also dropping the AgentErrorEvent — but the view-property fix (Option 1) is what stops the crash regardless of how the orphan arises.

This issue was created by an AI agent (OpenHands) on behalf of the requester.

[github-actions]

✅ ready-for-dev label applied

This issue meets the readiness criteria and is ready for contribution.
A contributor can now open a PR that links this issue (Fixes #<number>).

This is an automated check — no AI was used to generate this comment.