fix: stream compression and DEFLATE encoding - #3
Merged
Conversation
- Implement tests for js_pdf stream compression in `compression.test.mjs`, verifying default compression behavior, stream readability, and compression efficiency. - Introduce tests for the DEFLATE encoder in `deflate.test.mjs`, ensuring round-trip integrity through both Node's zlib and the port's own inflate implementation. - Create a support module `pdf-text.mjs` to facilitate reading generated PDFs as text, handling compressed streams appropriately.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds document-wide stream compression (DEFLATE/zlib) and a diagnostics channel, updates PDF parsing helpers in tests to account for compressed streams, and improves memory usage in PNG inflate/UTF-8 encoding.
Changes:
- Implement RFC1951/1950 deflate encoder and apply
/FlateDecodeto eligible PDF streams by default (opt-out viacompress: false). - Add
PdfSettingsplumbing + diagnostic handler API; warn on oversized decoded images when nodpiis provided. - Update/extend tests (compression/deflate) and refactor PDF-bytes-to-text helpers to transparently inflate streams for assertions.
Reviewed changes
Copilot reviewed 46 out of 61 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| test/widgets_phase_5_7.test.mjs | Switches PDF text extraction to shared helper that expands streams. |
| test/ttf_font.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/table.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/svg_widget.test.mjs | Uses shared latin1 helper for PDF source assertions. |
| test/svg_gradient.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/support/pdf-text.mjs | Adds helper to read generated PDFs as text while inflating /FlateDecode streams. |
| test/rich_text_phase_3_7.test.mjs | Switches PDF text extraction to shared helper that expands streams. |
| test/progress_phase_5_5.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/placeholders_phase_3_9.test.mjs | Switches PDF text extraction to shared helper that expands streams. |
| test/page_orientation.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/object_model.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/layout_phase_3_6.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/js_pdf.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/image_phase_4_3.test.mjs | Uses shared helper for matching content in PDFs with compressed streams. |
| test/icon_phase_5_4.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/forms_metadata_phase_5_6.test.mjs | Switches PDF text extraction to shared helper that expands streams. |
| test/flex_phase_3_4.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/example_parity_regressions.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/deflate.test.mjs | Adds unit tests validating new deflate encoder against two decoders. |
| test/decoration_phase_3_5.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/content_phase_3_8.test.mjs | Switches PDF text extraction to shared helper that expands streams. |
| test/compression.test.mjs | Adds integration tests for document stream compression behavior and diagnostics. |
| test/clip_phase_3_10.test.mjs | Switches PDF text extraction to shared helper that expands streams. |
| test/chart_phase_5_1.test.mjs | Replaces direct latin1 conversion with shared helper. |
| test/basic_phase_3_3.test.mjs | Reuses shared latin1 helper instead of local byte-to-string logic. |
| test/barcode_phase_5_2.test.mjs | Replaces direct latin1 conversion with shared helper. |
| src/widgets/image_provider.ts | Adds oversized-image warning via diagnostic handler when dpi is omitted. |
| src/widgets/document.ts | Introduces compress option and passes PdfSettings through serialization. |
| src/pdf/obj/xobject.ts | Threads per-stream compress flag into XObject streams. |
| src/pdf/obj/object_stream.ts | Plumbs document settings.compress into stream objects. |
| src/pdf/obj/object.ts | Extends object registry contract to expose document PdfSettings. |
| src/pdf/obj/metadata.ts | Ensures XMP metadata stream is never compressed. |
| src/pdf/image/png.ts | Replaces number[] inflate buffer with growable Uint8Array for memory/perf. |
| src/pdf/format/object_base.ts | Adds PdfSettings type and default settings constant. |
| src/pdf/format/dict_stream.ts | Implements conditional /FlateDecode compression with “keep only if smaller” rule. |
| src/pdf/format/deflate.ts | Adds DEFLATE/zlib encoder implementation. |
| src/pdf/document.ts | Stores PdfSettings on PdfDocument and threads settings through serializePdf. |
| src/pdf/diagnostics.ts | Adds diagnostic handler API with safe optional console fallback. |
| src/index.ts | Exports deflate APIs, diagnostics APIs, and PdfSettings type. |
| src/base/utf8.ts | Switches to typed-array implementations to reduce allocations and heap usage. |
| src/barcode/code128.ts | Updates shortestCode signature to match new codeUnits() return type. |
| package.json | Bumps package version to 0.1.5. |
| examples/generation-results.json | Updates generated example sizes/timestamp (reflecting compression). |
| docs/ROADMAP.md | Documents compression/diagnostics work and explains memory improvements. |
| docs/PORTING-STATUS.md | Updates porting status to reflect new compression/settings/diagnostics modules. |
| AI_USAGE.md | Updates guidance to recommend dpi and documents compression & diagnostics usage. |
| dist/types/widgets/document.d.ts | Updates public types for compress option and exposes settings. |
| dist/types/pdf/obj/xobject.d.ts | Updates type signature to include optional compress parameter. |
| dist/types/pdf/obj/object_stream.d.ts | Updates constructor signature to include optional compress parameter. |
| dist/types/pdf/obj/object.d.ts | Updates registry type to expose settings. |
| dist/types/pdf/obj/metadata.d.ts | Updates docstring to reflect “never compressed” behavior. |
| dist/types/pdf/format/object_base.d.ts | Exposes PdfSettings and default settings in type declarations. |
| dist/types/pdf/format/dict_stream.d.ts | Updates stream dictionary types to include compress and new constructor param. |
| dist/types/pdf/format/deflate.d.ts | Adds type declarations for new deflate exports. |
| dist/types/pdf/document.d.ts | Threads PdfSettings through PdfDocument and serializePdf types. |
| dist/types/pdf/diagnostics.d.ts | Adds type declarations for diagnostics handler APIs. |
| dist/types/index.d.ts | Re-exports deflate/diagnostics/settings in top-level typings. |
| dist/types/base/utf8.d.ts | Updates typings for codeUnits() return type and new utf8 docs. |
| dist/types/barcode/code128.d.ts | Updates shortestCode signature to accept Uint16Array. |
| dist/js_pdf.mjs | Updates bundled build with compression, diagnostics, and utf8/png changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+64
to
+76
| // Data that names its own filter is already in the form the reader wants. | ||
| let data = this.data; | ||
| if (this.compress && !this.has('/Filter')) { | ||
| const deflated = deflateZlib(data); | ||
| if (deflated.length < data.length) { | ||
| this.set('/Filter', new PdfName('/FlateDecode')); | ||
| data = deflated; | ||
| } | ||
| } | ||
|
|
||
| // `/Length` is derived, so it is set at write time rather than by the | ||
| // caller, and last — the key order is part of the output contract. | ||
| this.set('/Length', new PdfNum(this.data.length)); | ||
| this.set('/Length', new PdfNum(data.length)); |
Comment on lines
+80
to
+81
| s.putBytes(data); | ||
| if (data.length === 0 || data[data.length - 1] !== 0x0a) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
compression.test.mjs, verifying default compression behavior, stream readability, and compression efficiency.deflate.test.mjs, ensuring round-trip integrity through both Node's zlib and the port's own inflate implementation.pdf-text.mjsto facilitate reading generated PDFs as text, handling compressed streams appropriately.