Skip to content

fix(multi_format_api): avoid zero-column DataFrame on empty JSON arrays#93

Merged
gabrielbressan-tfy merged 1 commit into
mainfrom
fix/multi-format-empty-array-zero-columns
Jul 10, 2026
Merged

fix(multi_format_api): avoid zero-column DataFrame on empty JSON arrays#93
gabrielbressan-tfy merged 1 commit into
mainfrom
fix/multi-format-empty-array-zero-columns

Conversation

@gabrielbressan-tfy

Copy link
Copy Markdown
Collaborator

Problem

The multi_format_api handler (display name URL Reader) crashes when an API returns a structurally valid but empty record array — e.g. {"results": [], "next_url": "..."}.

parse_json returned a bare pd.DataFrame(), which is shape (0, 0) — zero rows and zero columns (pandas can't infer column names from an empty list of dicts). When the API-handler planner then runs SELECT * FROM df, DuckDB rejects the zero-column frame:

Invalid Input Error: Need a DataFrame with at least one column

This is a DuckDB schema-inference limitation, not a data problem — the query is valid and the data is legitimately empty. But the error that bubbles up to the agent is a generic DuckDB traceback with no hint that "your filters matched zero rows," so the LLM misreads a data condition as a syntax problem and starts mutating the query (guessing record_path, column names, etc.) — a wasteful retry-storm.

Fix

The empty-record branch in parse_json now always returns a 0-row frame with at least one column:

  • Preserve the envelope's top-level scalar siblings (next_url, count, status, …) as columns — mirroring what the non-empty path already does when it reattaches envelope metadata.
  • Fall back to a single _empty sentinel column when there's no envelope metadata to borrow (e.g. a bare top-level []).

A valid-but-empty response now returns a clean empty result set, giving the agent an unambiguous "valid query, zero rows" signal instead of a DuckDB traceback.

XML and CSV paths always emit ≥1 column, so JSON was the only affected format.

Tests

  • Tightened the existing empty-array test to assert df.shape[1] >= 1.
  • Added a regression for the {"results": [], "next_url": "..."} envelope case (columns preserved, results key excluded).
  • Added a regression for the bare [] sentinel fallback.

All 15 tests in the handler module pass (/venv/bin/python -m pytest in the dev-mindsdb container).

🤖 Generated with Claude Code

When a JSON response contained a structurally valid but empty record array
(e.g. {"results": [], "next_url": "..."}), parse_json returned a bare
pd.DataFrame() — shape (0, 0). DuckDB rejects a zero-column frame when the
planner runs `SELECT * FROM df`, raising "Need a DataFrame with at least one
column". That opaque error surfaced to the agent as a query problem rather
than an empty result, triggering wasteful query-rewrite retries.

The empty branch now returns a 0-row frame with at least one column:
the envelope's top-level scalar siblings (next_url, count, status, ...)
when present, else an `_empty` sentinel. A valid-but-empty response now
returns a clean empty result set instead of a DuckDB traceback.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gabrielbressan-tfy gabrielbressan-tfy merged commit cda26cd into main Jul 10, 2026
3 of 5 checks passed
@gabrielbressan-tfy gabrielbressan-tfy deleted the fix/multi-format-empty-array-zero-columns branch July 10, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant