feat: charts and exports - #69
Open
vrtornisiello wants to merge 19 commits into
Open
Conversation
vl-convert compiles and renders a Vega-Lite spec in-process (embedded V8, no Node or browser), used to validate generated chart specs before they ship. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Offer CSV/JSONL/PARQUET/AVRO. materialize_export extracts a result table to a deterministic, message-scoped GCS object (reused on repeat) and reports its size and a sanitized filename. Adds is_result_expired, collect_query_handles, and the expired / too-large typed signals the API maps to 410 / 400. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make query_ref the sole primary key of query_handles and add thread/message scoped lookups; add a STREAMING message status. run_agent creates the assistant row up front (STREAMING), persists query handles as tool outputs stream, and writes the terminal state deterministically: a fail-fast placeholder and an update-only finalize. Includes the query_ref-PK and STREAMING-status migrations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read a result's capped rows, generate a spec via a data-visualization specialist (with_structured_output over function-calling), sanitize it (strip any data / datasets / URL), and validate it (real columns + renders) in a bounded repair loop. The exact result rows are bound server-side, so the model cannot invent the numbers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
app/agent/tools/dataviz.py adds list_query_results, export_query_result, and chart_query_result, registered in the toolkit. execute_bigquery_sql now returns a query_ref by which its result can be exported or charted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
POST /messages/{id}/exports signs a download for a chosen format; POST
/messages/{id}/charts generates a chart spec for a result. Expired, too-large,
and unproducible-spec cases map to 410, 400, and 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the Exporting results and Charting results sections and the capability line: the model calls the tools with a result's query_ref and describes charts in words. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- type the export endpoint's `format` param as str so OFFERED_EXPORT_FORMATS (not the Literal) is the source of truth and the unsupported-format guard is actually reachable, returning a helpful 400 instead of FastAPI's generic 422 - add tests for the export tool's ResultTableExpired branch, _persist_query_handles with no query_result artifacts, and _finalize_message's update-returns-None and update-raises paths Feature modules (charts, exports, dataviz, chatbot, agent_runner) now at 100%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hand-written 9a3c7b2e1f04 with 91426eac7604 (alembic-generated, then completed by hand — autogenerate detects neither PK-constraint nor enum-value changes). 91426eac7604 makes query_ref the sole PK and indexes message_id (reversible downgrade); c4e1a9d2f6b8 adds the STREAMING enum value and repoints onto it (irreversible — Postgres can't drop enum values). Applied and verified on a fresh local DB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
read_chart_data guards on settings.CHART_MAX_BYTES (the serialized JSON size of the rows bound inline into the spec) rather than a 1000-row cap, so dense but legitimate charts — e.g. a ~5.5k-row municipal choropleth — are allowed while pathological payloads are still refused. Add CHART_MAX_BYTES under a new "Exports & Charts" settings section (default 5 MiB, capped at 10 MiB), moving the export/chart limits out of the Google Cloud section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Choropleth maps for Brazilian states and municipalities: the model writes a
geoshape+lookup spec referencing named data sources, and the server injects the
geometry (static TopoJSON under app/assets/) and the result rows — no external fetch.
- Remove the on-demand chart-button endpoint (POST /messages/{id}/charts); charts now
come only from the conversational chart_query_result tool.
- Rework the chart module into a clear pipeline — fetch_chart_data -> generate_chart_spec
(sanitize + validate) -> inject_chart_data — with JsonValue-typed spec walkers.
- Restructure the chart-spec system prompt into markdown sections; make color
discretionary (a scheme only for a semantic convention, else Vega defaults) and drop
the injected house palette. Note the choropleth capability in the agent prompt.
- Reorganize the chart tests into one class per function.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Credit the source repo (dados-referenciais-abertos), note the ODbL/DbCL license and that the derived TopoJSON stays under the ODbL/DbCL, in Portuguese and English. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses code-review findings on the chart pipeline: - BigQuery NUMERIC/BIGNUMERIC values were serialized with default=str, so numeric columns reached the spec as strings and broke quantitative axes and choropleth color scales. Coerce Decimal -> float (dates stay ISO strings, which Vega-Lite parses). - Choropleth geometry is no longer inlined into the spec. It bloated the persisted messages.events (a chart artifact is stored verbatim) by ~2 MB per municipality map, re-streamed on every thread load. Specs now reference the geometry by URL (GEO_ASSET_URL_BASE), which the browser fetches and caches; the TopoJSON files move to the website's static assets. Rows stay inline, bounded by CHART_MAX_BYTES. - Spec validation rendered the full geometry to SVG on every retry. It now resolves geo named-sources to a minimal one-feature stub for the offline compile check, so a choropleth no longer re-projects thousands of features up to MAX_CHART_SPEC_ATTEMPTS times per request. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A model-generated spec could carry an `href` mark property or encoding channel, whose `javascript:` URI would execute on click in the viewer's browser. Analytical charts never need clickable-link marks, so add `href` to _UNTRUSTED_KEYS alongside url/datasets; the sanitizer now drops it anywhere in the spec. A data column literally named "href" is unaffected (it appears as a field value, not a key). Defense-in-depth on top of the already-patched Vega library and the same-user/self-XSS blast radius. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Coerce integer-valued Decimals to int in _json_default so NUMERIC id/code columns (e.g. id_municipio) keep their exact digits with no trailing .0, which also lets a string-keyed geo lookup join match. rstrip the geo asset URL base so a trailing slash no longer yields a double slash, and correct two stale docstrings/comments left over from the inline-TopoJSON switch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add two guidelines to the chart spec generator: relabel SI-abbreviated axis prefixes (k/M/G/T) to the naming convention the measured quantity uses in the response's language, and filter out non-positive rows before a log scale so a stray zero/null can't collapse the axis. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1MB of already-aggregated JSON rows is several thousand points — well above what a readable chart needs — while sparing the SSE payload and the in-browser Vega-Lite render the sluggishness a 5MB spec could cause. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Charts & Exports
Turn a query's results into downloadable files and charts, all derived server-side from the exact rows, with no code sandbox.
export_query_result): download a result as CSV / JSON Lines / Parquet /AVRO, materialized on demand from BigQuery and served via a signed URL. Query results
are persisted as handles, so they stay exportable/chartable across turns.
chart_query_result): the agent describes a chart in words; a generatorproduces a validated Vega-Lite v6 spec (validate-and-repair loop), and the server injects
the exact rows into it (the client renders with Vega-Embed).
geoshapemaps fromstatic TopoJSON injected server-side (
app/assets/).named sources, and all rows/geometry are injected by the server.
Adds
vl-convertfor spec validation, aquery_handlestable + migration, and related settings.