[pull] main from firecrawl:main - #745
Merged
Merged
Conversation
…n document.markdown (#4365) * feat(v2): pageMarkers PDF parser option — per-page attribution in document.markdown Adds `parsers: [{ type: "pdf", pageMarkers: true }]` (default false) to v2 /parse and /scrape. When set, PDF pages in document.markdown are joined with `\n\n---\n\n<!-- page N -->\n\n` where N is the 1-based physical page of the content that follows. No new response field. Semantics: markers appear between pages only (no leading marker for page 1); numbering may skip pages merged by cross-page stitching — callers that need every physical page should use `pages: true`. Composes with pages, blocks, maxPages, and mode. Plumbing follows the pages/blocks precedent (#4347): - schema field flows through the pageMarkdown-alias transform + getPDFPageMarkers() accessor - firePDF sync engine sends page_markers; async /jobs submit sends pageMarkers; both force the fire-pdf engine (no silent MinerU fallback without markers) - excluded from URL-index reads/writes and agentIndexOnly (index markdown never carries markers) - PDF cache: pageMarkers mutates markdown itself (unlike the pages/blocks sidecars), so marker requests read/write a fully disjoint …markers… variant family — a base-variant entry is never served for a marker request and vice versa (mode:ocr precedent) - JS/Python SDK parser types updated Requires fire-pdf's page_markers support on the sync /ocr path to be deployed first (older fire-pdf builds ignore unknown fields). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: fail loud on missing page_markers echo, drop SDK changes, fix fallback log - Revert JS/Python SDK changes — this PR is API-only for now; SDK support lands separately. - Validate the new fire-pdf page_markers response echo (firecrawl/fire-pdf#629): markers are baked into markdown and their absence is not detectable by content, so a build that ignores the unknown request field must fail loud instead of silently caching unmarked markdown under marker cache variants. - Document why the async /jobs option is camelCase pageMarkers: that is the field fire-pdf's jobs schema defines, and its handler 400s unknown option keys, so snake_case would be rejected. - Mention markers in the async→sync fallback warn message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * review: validate the page_markers echo on the async /jobs result too The sync path validates fire-pdf's page_markers acknowledgment, but the async path cached results with none — an api/worker version skew (submit accepts the pageMarkers option, an older worker silently ignores the stored key) would persist unmarked markdown under marker cache variants. fire-pdf now echoes page_markers on the /jobs result JSON (firecrawl/fire-pdf#629); reject marker results lacking the echo and fall back to the sync path, where the same contract applies. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Expose the v2 PDF parser pageMarkers option across the JS, Python, Go, Java, Ruby, PHP, Rust, and .NET SDKs so scrape/parse requests can join PDF pages in document.markdown with <!-- page N --> separators. Bump each package version so merge auto-publishes. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Abimael Martell <abimaelmartell@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )