Skip to content

feat: expose extracted images in scrape response#322

Merged
us merged 2 commits into
mainfrom
feat/scrape-images-format
Jul 18, 2026
Merged

feat: expose extracted images in scrape response#322
us merged 2 commits into
mainfrom
feat/scrape-images-format

Conversation

@us

@us us commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Requested in #317 by @Antardas: return discovered images as structured data so consumers don't have to re-parse the HTML for RAG, indexing, media downloading, or recursive crawling.

What

New images output format:

  • /v1 (native): images: [{ url, alt }] — resolved absolute URLs plus alt text where the source carries it (handy as caption/embedding text for RAG).
  • /v2 + /firecrawl/v2: images: string[] — flat URL list, matching Firecrawl's images shape so the compat surface stays a drop-in.

Sources covered: <img> src/data-src/srcset, <picture><source>, og/twitter/itemprop meta, icon links, <video poster>, and inline background-image. srcset is parsed per the WHATWG algorithm, so a comma inside a data: URI is kept whole instead of being split into phantom URLs. Deduplicated by URL in traversal order; a later duplicate upgrades an empty alt.

Also threaded through the CLI (--format images) and the MCP tool schemas.

Notes

  • Additive: images is only produced when requested, no change to other formats or the scrape-success path.
  • The v1-objects / v2-strings split is deliberate: /v2 stays a byte-compatible Firecrawl drop-in while native /v1 adds the alt metadata.

ref #317

us added 2 commits July 18, 2026 21:27
Add an `images` output format so consumers get discovered images as
structured data instead of re-parsing the HTML. The native /v1 surface
returns `[{url, alt}]` objects; /v2 and /firecrawl/v2 flatten them to a
plain string[] of URLs to stay a Firecrawl drop-in.

Extraction mirrors Firecrawl's source set (img src/data-src/srcset,
picture source, og/twitter/itemprop meta, icon links, video poster,
inline background-image) with WHATWG-correct srcset parsing so a comma
inside a data: URI is not shredded into phantom URLs. Deduplicated by
URL in traversal order; a later duplicate upgrades an empty alt.

Threaded through the CLI (--format images), MCP tool schemas, and the
v2 format decomposition.
Keep the OutputFormat drift-check tables and both SKILL.md copies in
sync with the new `images` variant.

@us us left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All checks green (build/test, clippy, conformance, fuzzing, CodeQL, OpenAPI + docs drift guards). Verified end-to-end on 36 live sites with a Firecrawl-parity oracle (0 divergence) plus adversarial fixtures. Merging.

@us
us merged commit 8a66b7f into main Jul 18, 2026
13 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 18, 2026
@us
us deleted the feat/scrape-images-format branch July 18, 2026 19:15
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant