Conversation
@workglow/cli
@workglow/ai
@workglow/ai-provider
@workglow/job-queue
@workglow/knowledge-base
@workglow/storage
@workglow/task-graph
@workglow/tasks
@workglow/util
workglow
commit: |
@workglow/cli
@workglow/ai
@workglow/ai-provider
@workglow/job-queue
@workglow/knowledge-base
@workglow/storage
@workglow/task-graph
@workglow/tasks
@workglow/util
workglow
commit: |
7-phase plan covering @workglow/util/media GpuImage abstraction (WebGPU + sharp + CPU backends), filter task rewrites, schema-driven cache hooks, AI vision interop, and builder app preview rewrite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… (Phase 1)
Introduces GpuImage as the canonical inter-task image type for the
GPU image pipeline refactor. Three concrete backends selected at
construction time:
- CpuImage — universal fallback; wraps an ImageBinary
- WebGpuImage — owns a GPUTexture (rgba8unorm); apply() runs
a render pipeline via the shader registry,
materialize() round-trips through copyTextureToBuffer
- SharpImage — wraps sharp.Sharp with a deferred pipeline;
operations stay deferred until materialize()/encode()
Supporting infrastructure:
- GpuImageSchema — JSON Schema literal with format:"image" so the
runner's input resolver can hydrate values
- GpuDevice — browser singleton with device.lost recovery
- TexturePool — size-class buckets, length-based eviction
- shaderRegistry — per-source shader-module cache + 17 named
shader source constants (passthrough plus 16
per-filter slots, all currently passthrough
until Phase 3 fragments land)
- GpuImageFactory — Proxy-based registry with then-guard so the
value plays nicely with Promise resolution.
Async fromDataUri/fromBlob/fromImageBitmap
are wired per platform; sync fromImageBinary
returns CpuImage.
Tests cover each backend's API surface, with WebGPU paths gated
behind describe.skipIf(!navigator.gpu) and Sharp paths gated to
node/bun. sharp added as optionalDependencies on @workglow/util.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ImageOp<P> = { webgpu, sharp, cpu } declares a triple of
backend-specific filter implementations. applyOp dispatches on
image.backend and forwards typed params to the matching arm.
ImageFilterTask<P> abstract base wires execute() and
executePreview() through applyOp(input.image, this.op,
this.opParams(input)) — modes diverge only at the runner edges
(cache materialization on execute, none on executePreview), not
in the task body.
(Note: this triple-arm design is replaced in Phase 8 by a
codec-style registry that eliminates the cross-platform type
dependency in imageOp.ts.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…; delete produceImageOutput (Phase 3)
Eliminates the per-stage data-URI encode/decode that the spec
identified as the source of the 70-185x perf gap.
All 18 image filter tasks (Blur, Border, Brightness, Contrast,
Crop, Flip, Grayscale, Invert, Pixelate, Posterize, Resize,
Rotate, Sepia, Threshold, Tint, Transparency, Text, Watermark)
move from the per-task `produceImageOutput` wrapper into the new
ImageFilterTask + ImageOp pattern. Each filter gets its own
{filter}/ subdirectory with {filter}Op.ts (cpu/sharp/webgpu arms)
and Image{Filter}Task.ts (subclass).
CPU arms reuse the existing per-pixel math relocated into pure
functions. Sharp arms map to the matching sharp builder method
(.flip, .blur, .recomb, .extend, .extract, .resize, .rotate,
.tint, .threshold, etc.); a few (Posterize, Transparency) lack
clean sharp equivalents and fall back to no-op TODOs that
Phase 7 cross-backend tests document. WebGPU arms call
image.apply() with the spec'd shader name; all per-filter shader
slots currently point at PASSTHROUGH_SRC until per-filter
fragments land.
Text and Watermark are special — they extend Task directly
(not ImageFilterTask) because the compositing math is the same
across backends, so the cpu/sharp/webgpu triple doesn't apply.
Final cleanup deletes:
- produceImageOutput from packages/tasks/src/task/image/imageTaskIo.ts
- ImageBinarySchema, ImageBinaryOrDataUriSchema,
ImageBinarySchemaOptions, ImageFromSchema from ImageSchemas.ts
(color schemas retained — still used by Tint/Border)
- ImageJson discriminated union and toJSON/fromJSON methods on
the Image class (Image class itself stays for now;
Phase 5 audits and deletes)
Also fixes a discovered issue in
@workglow/task-graph's InputResolver: Phase 2 recursion was
spreading format-annotated GpuImage instances into plain records,
losing prototype methods. Discriminator now skips class instances
(non-Object.prototype prototype) while still recursing into plain
objects so nested format annotations resolve.
Tests for migrated filters use CpuImage wrappers; deprecated
"preserves data-uri" cases removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t resolver (Phase 4)
Two foundational pieces wire the new GpuImage type into the
runner's input/output paths.
format:"image" input resolver (string-only):
- imageHydrationResolver registers under the "image" prefix.
When TaskRunner's resolveSchemaInputs encounters a property
schema with format:"image" and a string value, the resolver
hydrates a data: URI to a GpuImage via GpuImageFactory.fromDataUri.
Non-string shapes (Blob, ImageBitmap, ImageBinary) flow through
unchanged — consumers convert at the call site (useGpuImage in
Phase 6).
PortCodecRegistry + TaskRunner cache hooks:
- PortCodec<Live, Wire> declares { serialize, deserialize }.
Registry lives in @workglow/util/di (avoids cycles); a thin
re-export from @workglow/task-graph keeps the public-facing
import path stable.
- TaskRunner consults the registry at the cache write site
(serialize output ports whose schema declares a registered
format) and at the cache read site (deserialize cached
values back to live form).
- Inputs are also normalized through the codec for cache-key
determinism — two CpuImage instances with identical pixels
now collapse to the same wire form and hit the same cache
entry instead of forcing re-execution.
format:"image" port codec:
- imageCacheCodec registers a PortCodec under "image" that
materializes GpuImage to a CachedImage (raw pixels + dims +
channels) and deserializes back via CpuImage.fromImageBinary.
Cache holds raw bytes — no PNG/JPEG re-encode.
End-to-end: a filter task with format:"image" input/output ports
gets data-URI hydration, deterministic cache keys, and raw-byte
cache materialization for free.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…uImage; delete Image class (Phase 5) AiVisionTask.getJobInput now materializes the input GpuImage at the worker boundary instead of running it through Image.from(...) .toFirstSupported(...). Default materialization yields ImageBinary; TENSORFLOW_MEDIAPIPE providers get an ImageBitmap via createImageBitmap. Workers see only bytes — WebGpuImage handles never cross the worker boundary. TypeImageInput collapses from a 60-line oneOf discriminated union (image:data-uri | image:ImageBitmap | image:OffscreenCanvas | image:VideoFrame | image:ImageBinary) to a single GpuImageSchema() literal. The runner's input resolver hydrates strings; non-string shapes flow through to Phase 5's getJobInput materialization. ImageEmbeddingTask, ImageClassificationTask, ImageToTextTask, and ImageSegmentationTask all keep their existing imports — TypeImageInput is the same name, just a different shape underneath. Provider integrations updated for HuggingFace Transformers (provider-hf-transformers): six files now call imageBinaryToBlob() to produce the Blob shape the @huggingface/transformers Node pipelines accept (HFT rejects data: URIs in node). Adds four helpers in @workglow/util/media: encodeImageBinaryToPng, imageBinaryToBase64Png, imageBinaryToDataUri, imageBinaryToBlob. With AiVisionTask migrated, the legacy Image class (packages/util/src/media/Image.ts + Image.browser.ts) had no remaining production callers. Deleted along with its tests and the corresponding re-exports from media-browser.ts and media-node.ts. Post-cleanup: tightens TS for VisionTasks/ZeroShotTasks/CacheSerialization test files (signature mismatches uncovered after the schema/SpyRepo shape changes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ation + perf benchmark (Phase 7) Cross-backend filter equality: each migrated filter (16 with ImageOp triples) runs the same 32×32 input through cpu and sharp backends; asserts maxAbsDiff per channel ≤ 2. WebGPU cases gated behind navigator.gpu (skip in node, run in browser). Some sharp arms have known algorithmic gaps vs cpu (Posterize TODO, Transparency alpha semantics, Threshold binary vs grayscale, etc.) — those cases are documented as test.skip with the cpu-vs-sharp delta noted. Cache round-trip: two equivalent CpuImage instances (distinct refs, identical pixels) run ImageBlurTask through an in-memory cache. The materialized outputs match and the cache holds exactly one entry — proving the normalize-inputs-for-cache-key path (Phase 4) collapses opaque instance identity to wire form. 7-stage chain integration: runs the canonical Text → Flip → Sepia → Blur → Posterize → Border → Pixelate chain on a 64×64 CpuImage, asserts the chain completes without throwing and produces non-trivial output. Per-filter correctness covered by the cross-backend test; this is a smoke test for the full pipeline. imageChainPerf benchmark: standalone bun script. 480p / 720p / 1080p timings post-warm. Reproduces the spec's perf-validation methodology. Targets per spec: - 720p WebGPU: ≤ 33 ms/run (browser only) - 720p sharp: ≤ 100 ms/run (node) - 720p cpu fallback: ≤ 1500 ms/run Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…odec sub-export
End-to-end review (independent code-reviewer agent) flagged two
non-blocking improvements and the final-sweep uncovered two
test-failure surfaces. Bundled here:
1. imageHydrationResolver fails loud on unknown string schemes.
Previously HTTP URLs / file paths / etc. passed through as
raw strings and crashed the consuming task with a confusing
TypeError. Now throws with a helpful message pointing at
registerInputResolver("image:http", fn) for sub-prefix
resolvers.
2. media-node WebGpuImage doc comment — explains why the symbol
is re-exported in node despite always throwing if invoked
(cross-package type imports in ImageOp<P> need it). Calling
it in node always throws via getGpuDevice() === null.
3. media-browser SharpImage type re-export. (Symmetric to (2):
imageOp.ts is shared across packages and references all four
GpuImage backend types; the type re-export makes both
bundles type-check uniformly. This re-export is removed in
Phase 8 once imageOp.ts becomes platform-agnostic.)
4. ZeroShotTasks.integration.test.ts adds the side-effect import
of @workglow/tasks/codec so the raster codec is registered
before vision tasks try to hydrate the data: URI test fixture
through the new format:"image" resolver. (VisionTasks already
has this import; ZeroShotTasks just missed it.)
5. New @workglow/tasks/codec sub-export — a lean side-effect entry
that does ONLY codec registration (no SafeFetch, no undici, no
other tasks). Three util-side tests that needed codec registration
for vitest (CpuImage, GpuImage, imageHydrationResolver) switch to
it. Bun was unaffected (native fetch); vitest was failing at
suite-load time on `webidl.util.markAsUncloneable` because the
full @workglow/tasks barrel pulled undici@8.1.0 which is
incompatible with vitest's environment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ss-platform type leak (Phase 8)
The Phase 2 design used `ImageOp<P> = { webgpu, sharp, cpu }` and a
central `applyOp` dispatcher that imported all three concrete backend
types. That forced cross-platform type re-exports (SharpImage from
media-browser, WebGpuImage from media-node) — code smells that
surfaced when bun's tsc tried to type-check the web example.
Replace with a string-keyed registry mirroring the existing image
raster codec / text renderer pattern:
registerFilterOp("cpu", "blur", fn); // <f>.cpu.ts (cross-platform)
registerFilterOp("sharp", "blur", fn); // <f>.sharp.ts (server-only)
registerFilterOp("webgpu", "blur", fn); // <f>.webgpu.ts (browser-only)
// imageOp.ts: zero platform-specific imports
applyFilter(image, "blur", { radius: 5 });
Each of 16 filters splits into 2-3 backend files (<f>.cpu.ts is
mandatory; <f>.sharp.ts and <f>.webgpu.ts are conditional on the
backend having a real impl). ImageFilterTask now declares
`filterName = "blur"` instead of `op = blurOp`.
Side-effect registration wired through codec.node.ts (cpu + sharp)
and codec.browser.ts (cpu + webgpu). Tasks's full node.ts/browser.ts
entries import the codec entry instead of just the raster codec
registration so all filter dispatch is set up automatically.
Posterize and Transparency drop their no-op sharp arms — applyFilter
throws a clear "no sharp impl registered" error instead of silently
returning the input unchanged.
Cleanup: media-browser.ts no longer re-exports SharpImage; media-node.ts
keeps WebGpuImage as type-only (needed for *.webgpu.ts source files
to type-check under node tsc) but no longer exports the runtime value.
Adding a new backend (WASM, dawn-on-node, GPU.js) is now new files
plus entry-point updates — no central edits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sroussey
force-pushed
the
feat/gpu-image-pipeline
branch
from
180bdc6 to
c025325
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a new “GpuImage” abstraction and GPU-oriented image pipeline across @workglow/util, @workglow/tasks, and @workglow/task-graph, including schema hydration and cache serialization via port codecs.
Changes:
- Add
GpuImageinterface + platform factories (CPU/Sharp/WebGPU) and runner-side hydration forformat:"image". - Introduce
PortCodecRegistryand use it inTaskRunnerto normalize cache keys and serialize/deserialize cached outputs for formatted ports (notably images). - Migrate image filter tasks to a backend-dispatch model (
cpu/sharp/webgpu) with side-effect “codec” entrypoints and add broad test coverage.
Reviewed changes
Copilot reviewed 167 out of 169 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/util/src/media/texturePool.browser.ts | Adds a WebGPU texture pool singleton for reuse across filters. |
| packages/util/src/media/sharpImage.bun.ts | Bun entry re-export for Sharp-based image backend. |
| packages/util/src/media/imageHydrationResolver.ts | Registers format:"image" input resolver (string → GpuImage via data URI). |
| packages/util/src/media/imageCacheCodec.ts | Registers format:"image" port codec for cache key normalization + output caching. |
| packages/util/src/media/gpuImageSchema.ts | Adds a schema helper that annotates ports with format:"image". |
| packages/util/src/media/gpuImage.ts | Introduces GpuImage interface and proxy-based factory registration. |
| packages/util/src/media/gpuDevice.browser.ts | Adds a cached WebGPU device getter with reset helper for tests. |
| packages/util/src/media/encode.ts | Adds helpers to encode ImageBinary to PNG/base64/data-uri/blob. |
| packages/util/src/media-node.ts | Updates node media barrel exports; registers node GpuImage factories and codecs. |
| packages/util/src/media-browser.ts | Updates browser media barrel exports; registers browser GpuImage factories and codecs. |
| packages/util/src/di/index.ts | Re-exports PortCodecRegistry from DI index. |
| packages/util/src/di/PortCodecRegistry.ts | Adds codec registry API (register/get/reset). |
| packages/util/src/di/Container.ts | Makes globalContainer truly global via globalThis + Symbol.for. |
| packages/util/package.json | Adds optional dependency on sharp. |
| packages/test/src/test/util/imageHydrationResolver.test.ts | Tests format:"image" hydration behavior and passthrough cases. |
| packages/test/src/test/util/imageCacheCodec.test.ts | Tests image port codec serialize/deserialize behavior. |
| packages/test/src/test/util/WebGpuImage.test.ts | Smoke/API tests for WebGpuImage (skips appropriately by environment). |
| packages/test/src/test/util/SharpImage.test.ts | Sharp backend behavior tests (skipped in browser). |
| packages/test/src/test/util/ShaderRegistry.test.ts | Tests shader cache behavior and shader source availability. |
| packages/test/src/test/util/GpuDevice.test.ts | Tests WebGPU device caching and node fallback behavior. |
| packages/test/src/test/util/CpuImage.test.ts | Tests CPU backend encode/materialize/release semantics. |
| packages/test/src/test/task/watermark.test.ts | Adds CPU smoke tests for watermark task using new image type. |
| packages/test/src/test/task/transparency.test.ts | Adds CPU correctness tests for transparency task. |
| packages/test/src/test/task/tint.test.ts | Adds CPU correctness tests for tint task. |
| packages/test/src/test/task/threshold.test.ts | Adds CPU correctness tests for threshold task. |
| packages/test/src/test/task/text.test.ts | Adds CPU tests for text rendering output shape/content. |
| packages/test/src/test/task/sepia.test.ts | Adds CPU tests for sepia coefficients behavior. |
| packages/test/src/test/task/rotate.test.ts | Adds CPU tests for rotate behavior and dimension swaps. |
| packages/test/src/test/task/resize.test.ts | Adds CPU tests for resize behavior/dimensions. |
| packages/test/src/test/task/posterize.test.ts | Adds CPU tests for posterize quantization behavior. |
| packages/test/src/test/task/pixelate.test.ts | Adds CPU tests for pixelate block averaging behavior. |
| packages/test/src/test/task/invert.test.ts | Adds CPU tests for invert behavior and alpha preservation. |
| packages/test/src/test/task/grayscale.test.ts | Adds CPU tests for grayscale conversion behavior. |
| packages/test/src/test/task/flip.test.ts | Adds CPU tests for flip behavior across channel counts. |
| packages/test/src/test/task/crop.test.ts | Adds CPU tests for crop bounds/clamping and channels. |
| packages/test/src/test/task/contrast.test.ts | Adds CPU tests for contrast LUT behavior. |
| packages/test/src/test/task/brightness.test.ts | Adds CPU tests for brightness adjustment/clamping. |
| packages/test/src/test/task/border.test.ts | Adds CPU tests for border expansion/color handling. |
| packages/test/src/test/task/blur.test.ts | Adds CPU tests for blur behavior and invariants. |
| packages/test/src/test/task/ImageOp.test.ts | Tests backend/filter dispatch via register/apply. |
| packages/test/src/test/task/ImageFilterTask.test.ts | Tests ImageFilterTask execute/preview and opParams plumbing. |
| packages/test/src/test/task/ImageColorInput.test.ts | Updates color tests to work with GpuImage outputs/materialize. |
| packages/test/src/test/task/ImageChainIntegration.test.ts | Adds a multi-stage CPU integration chain smoke test. |
| packages/test/src/test/task/ImageCacheRoundTrip.test.ts | Tests cache key normalization producing a single cache entry for equivalent images. |
| packages/test/src/test/task-graph/PortCodecRegistry.test.ts | Tests format and prefix lookup in PortCodecRegistry. |
| packages/test/src/test/ai-provider/ZeroShotTasks.integration.test.ts | Ensures tasks side-effects register image handling for vision task tests. |
| packages/test/src/test/ai-provider/VisionTasks.integration.test.ts | Updates vision tests to rely on hydration/materialization pipeline. |
| packages/test/src/perf/imageChainPerf.ts | Adds a perf harness for a 7-stage image chain. |
| packages/tasks/src/task/image/transparency/transparency.webgpu.ts | Registers WebGPU filter op for transparency. |
| packages/tasks/src/task/image/transparency/transparency.cpu.ts | Adds CPU transparency implementation + registration. |
| packages/tasks/src/task/image/transparency/ImageTransparencyTask.ts | Adds new transparency task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/tint/tint.webgpu.ts | Registers WebGPU filter op for tint. |
| packages/tasks/src/task/image/tint/tint.sharp.ts | Registers Sharp filter op for tint. |
| packages/tasks/src/task/image/tint/tint.cpu.ts | Adds CPU tint implementation + registration. |
| packages/tasks/src/task/image/tint/ImageTintTask.ts | Adds new tint task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/threshold/threshold.webgpu.ts | Registers WebGPU filter op for threshold. |
| packages/tasks/src/task/image/threshold/threshold.sharp.ts | Registers Sharp filter op for threshold. |
| packages/tasks/src/task/image/threshold/threshold.cpu.ts | Adds CPU threshold implementation + registration. |
| packages/tasks/src/task/image/threshold/ImageThresholdTask.ts | Adds new threshold task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/sepia/sepia.webgpu.ts | Registers WebGPU filter op for sepia. |
| packages/tasks/src/task/image/sepia/sepia.sharp.ts | Registers Sharp filter op for sepia. |
| packages/tasks/src/task/image/sepia/sepia.cpu.ts | Adds CPU sepia implementation + registration. |
| packages/tasks/src/task/image/sepia/ImageSepiaTask.ts | Adds new sepia task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/rotate/rotate.webgpu.ts | Registers WebGPU filter op for rotate. |
| packages/tasks/src/task/image/rotate/rotate.sharp.ts | Registers Sharp filter op for rotate. |
| packages/tasks/src/task/image/rotate/rotate.cpu.ts | Adds CPU rotate implementation + registration. |
| packages/tasks/src/task/image/rotate/ImageRotateTask.ts | Adds new rotate task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/resize/resize.webgpu.ts | Registers WebGPU filter op for resize. |
| packages/tasks/src/task/image/resize/resize.sharp.ts | Registers Sharp filter op for resize. |
| packages/tasks/src/task/image/resize/resize.cpu.ts | Adds CPU resize implementation + registration. |
| packages/tasks/src/task/image/resize/ImageResizeTask.ts | Adds new resize task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/posterize/posterize.webgpu.ts | Registers WebGPU filter op for posterize. |
| packages/tasks/src/task/image/posterize/posterize.cpu.ts | Adds CPU posterize implementation + registration. |
| packages/tasks/src/task/image/posterize/ImagePosterizeTask.ts | Adds new posterize task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/pixelate/pixelate.webgpu.ts | Registers WebGPU filter op for pixelate. |
| packages/tasks/src/task/image/pixelate/pixelate.sharp.ts | Registers Sharp filter op for pixelate. |
| packages/tasks/src/task/image/pixelate/pixelate.cpu.ts | Adds CPU pixelate implementation + registration. |
| packages/tasks/src/task/image/pixelate/ImagePixelateTask.ts | Adds new pixelate task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/invert/invert.webgpu.ts | Registers WebGPU filter op for invert. |
| packages/tasks/src/task/image/invert/invert.sharp.ts | Registers Sharp filter op for invert. |
| packages/tasks/src/task/image/invert/invert.cpu.ts | Adds CPU invert implementation + registration. |
| packages/tasks/src/task/image/invert/ImageInvertTask.ts | Adds new invert task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/imageTaskIo.ts | Removes legacy image IO helper. |
| packages/tasks/src/task/image/imageOp.ts | Adds backend-specific filter-op registry + dispatch + reset helper. |
| packages/tasks/src/task/image/grayscale/grayscale.webgpu.ts | Registers WebGPU filter op for grayscale. |
| packages/tasks/src/task/image/grayscale/grayscale.sharp.ts | Registers Sharp filter op for grayscale. |
| packages/tasks/src/task/image/grayscale/grayscale.cpu.ts | Adds CPU grayscale implementation + registration. |
| packages/tasks/src/task/image/grayscale/ImageGrayscaleTask.ts | Adds new grayscale task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/flip/flip.webgpu.ts | Registers WebGPU filter op for flip. |
| packages/tasks/src/task/image/flip/flip.sharp.ts | Registers Sharp filter op for flip. |
| packages/tasks/src/task/image/flip/flip.cpu.ts | Adds CPU flip implementation + registration. |
| packages/tasks/src/task/image/flip/ImageFlipTask.ts | Adds new flip task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/crop/crop.webgpu.ts | Registers WebGPU filter op for crop. |
| packages/tasks/src/task/image/crop/crop.sharp.ts | Registers Sharp filter op for crop. |
| packages/tasks/src/task/image/crop/crop.cpu.ts | Adds CPU crop implementation + registration. |
| packages/tasks/src/task/image/crop/ImageCropTask.ts | Adds new crop task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/contrast/contrast.webgpu.ts | Registers WebGPU filter op for contrast. |
| packages/tasks/src/task/image/contrast/contrast.sharp.ts | Registers Sharp filter op for contrast. |
| packages/tasks/src/task/image/contrast/contrast.cpu.ts | Adds CPU contrast implementation + registration. |
| packages/tasks/src/task/image/contrast/ImageContrastTask.ts | Adds new contrast task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/brightness/brightness.webgpu.ts | Registers WebGPU filter op for brightness. |
| packages/tasks/src/task/image/brightness/brightness.sharp.ts | Registers Sharp filter op for brightness. |
| packages/tasks/src/task/image/brightness/brightness.cpu.ts | Adds CPU brightness implementation + registration. |
| packages/tasks/src/task/image/brightness/ImageBrightnessTask.ts | Adds new brightness task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/border/border.webgpu.ts | Registers WebGPU filter op for border. |
| packages/tasks/src/task/image/border/border.sharp.ts | Registers Sharp filter op for border. |
| packages/tasks/src/task/image/border/border.cpu.ts | Adds CPU border implementation + registration. |
| packages/tasks/src/task/image/border/ImageBorderTask.ts | Adds new border task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/blur/blur.webgpu.ts | Registers WebGPU filter op for blur. |
| packages/tasks/src/task/image/blur/blur.sharp.ts | Registers Sharp filter op for blur. |
| packages/tasks/src/task/image/blur/blur.cpu.ts | Adds CPU blur implementation + registration. |
| packages/tasks/src/task/image/blur/ImageBlurTask.ts | Adds new blur task using ImageFilterTask dispatch. |
| packages/tasks/src/task/image/ImageTransparencyTask.ts | Removes legacy transparency task implementation. |
| packages/tasks/src/task/image/ImageTintTask.ts | Removes legacy tint task implementation. |
| packages/tasks/src/task/image/ImageThresholdTask.ts | Removes legacy threshold task implementation. |
| packages/tasks/src/task/image/ImageSepiaTask.ts | Removes legacy sepia task implementation. |
| packages/tasks/src/task/image/ImageSchemas.ts | Removes legacy ImageBinary schema helpers (now using GpuImageSchema). |
| packages/tasks/src/task/image/ImageRotateTask.ts | Removes legacy rotate task implementation. |
| packages/tasks/src/task/image/ImageResizeTask.ts | Removes legacy resize task implementation. |
| packages/tasks/src/task/image/ImagePosterizeTask.ts | Removes legacy posterize task implementation. |
| packages/tasks/src/task/image/ImagePixelateTask.ts | Removes legacy pixelate task implementation. |
| packages/tasks/src/task/image/ImageInvertTask.ts | Removes legacy invert task implementation. |
| packages/tasks/src/task/image/ImageGrayscaleTask.ts | Removes legacy grayscale task implementation. |
| packages/tasks/src/task/image/ImageFlipTask.ts | Removes legacy flip task implementation. |
| packages/tasks/src/task/image/ImageFilterTask.ts | Adds base class dispatching filters via applyFilter. |
| packages/tasks/src/task/image/ImageCropTask.ts | Removes legacy crop task implementation. |
| packages/tasks/src/task/image/ImageContrastTask.ts | Removes legacy contrast task implementation. |
| packages/tasks/src/task/image/ImageBrightnessTask.ts | Removes legacy brightness task implementation. |
| packages/tasks/src/node.ts | Switches task node entry to import new codec side-effect entry. |
| packages/tasks/src/electron.ts | Switches electron entry to import new codec side-effect entry. |
| packages/tasks/src/codec.node.ts | Adds lean side-effect entry registering raster codec + cpu/sharp filter ops. |
| packages/tasks/src/codec.browser.ts | Adds lean side-effect entry registering raster codec + cpu/webgpu filter ops. |
| packages/tasks/src/bun.ts | Switches bun entry to import new codec side-effect entry. |
| packages/tasks/src/browser.ts | Switches browser entry to import new codec side-effect entry. |
| packages/tasks/package.json | Adds ./codec export and modifies conditional exports. |
| packages/task-graph/src/task/TaskRunner.ts | Adds port codec-based cache key normalization and cached output serialization/deserialization. |
| packages/task-graph/src/task/InputResolver.ts | Prevents Phase 2 recursion from spreading class instances / resolver-owned objects; adds plain-object detection. |
| packages/task-graph/src/storage/PortCodecRegistry.ts | Re-exports codec registry API from util for task-graph consumers. |
| packages/task-graph/src/common.ts | Exposes codec registry API via task-graph public surface. |
| packages/ai/src/task/base/AiVisionTask.ts | Changes vision tasks to accept hydrated GpuImage and materialize at worker boundary. |
| packages/ai/src/task/base/AiTaskSchemas.ts | Simplifies image schema to format:"image" for runner hydration. |
| packages/ai-provider/src/provider-tf-mediapipe/common/TFMP_JobRunFns.ts | Tightens typing for task run fn registry. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_ObjectDetection.ts | Converts materialized ImageBinary to Blob for pipeline inputs. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_JobRunFns.ts | Tightens typing for task run fn registry. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_ImageToText.ts | Converts ImageBinary to Blob before calling pipeline. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_ImageSegmentation.ts | Converts ImageBinary to Blob before calling pipeline. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_ImageEmbedding.ts | Converts ImageBinary (single/array) to Blob before calling pipeline. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_ImageClassification.ts | Converts ImageBinary to Blob before calling pipeline. |
| packages/ai-provider/src/provider-hf-transformers/common/HFT_BackgroundRemoval.ts | Converts ImageBinary to Blob before calling pipeline. |
| bun.lock | Updates lockfile for new optional dependencies (sharp). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
30
to
46
| "exports": { | ||
| ".": { | ||
| "react-native": { | ||
| "types": "./dist/browser.d.ts", | ||
| "import": "./dist/browser.js" | ||
| "types": "./src/browser.ts", | ||
| "import": "./src/browser.ts" | ||
| }, | ||
| "browser": { | ||
| "types": "./dist/browser.d.ts", | ||
| "import": "./dist/browser.js" | ||
| "types": "./src/browser.ts", | ||
| "import": "./src/browser.ts" | ||
| }, | ||
| "bun": { | ||
| "types": "./dist/bun.d.ts", | ||
| "import": "./dist/bun.js" | ||
| "types": "./src/bun.ts", | ||
| "import": "./src/bun.ts" | ||
| }, | ||
| "types": "./dist/node.d.ts", | ||
| "import": "./dist/node.js" | ||
| "types": "./src/node.ts", | ||
| "import": "./src/node.ts" | ||
| }, |
There was a problem hiding this comment.
Package exports now point at TypeScript source files under ./src, but the published "files" list only includes "dist" (and docs). This will break consumers after publish because the exported paths won't exist in the package. Update exports back to ./dist/.js and ./dist/.d.ts (and add ./dist/codec.* outputs for the new entry), or include the necessary src files in "files"/publish output.
Comment on lines
+28
to
+34
| const hint = image.backend === "webgpu" | ||
| ? "tasks/src/codec.browser.ts" | ||
| : "tasks/src/codec.node.ts"; | ||
| throw new Error( | ||
| `No "${image.backend}" implementation registered for filter "${filter}". ` + | ||
| `Ensure the corresponding side-effect file is imported via ${hint}.`, | ||
| ); |
There was a problem hiding this comment.
The thrown error suggests importing internal paths like "tasks/src/codec.browser.ts" / "tasks/src/codec.node.ts", which won't exist for package consumers. Update the hint to reference the public entrypoint (e.g. import "@workglow/tasks/codec" or the appropriate platform barrel) so the remediation is actionable.
Comment on lines
+52
to
+66
| release(texture) { | ||
| const entry = owners.get(texture); | ||
| if (!entry) return; | ||
| const k = sizeClassKey(entry.width, entry.height, entry.format); | ||
| let bucket = buckets.get(k); | ||
| if (!bucket) { | ||
| bucket = []; | ||
| buckets.set(k, bucket); | ||
| } | ||
| if (bucket.length >= capacity) { | ||
| texture.destroy(); | ||
| return; | ||
| } | ||
| bucket.push(entry); | ||
| }, |
There was a problem hiding this comment.
TexturePool.release() is not idempotent: releasing the same texture twice will push the same PooledTexture entry into the bucket multiple times (and later destroy it multiple times). Track in-pool state (e.g., a Set or a boolean on the entry) and ignore/detect duplicate releases; also consider removing from the owners map when a texture is destroyed to avoid stale entries.
Comment on lines
+15
to
+19
| return gpu.apply({ | ||
| shader: "passthrough", | ||
| uniforms: new Float32Array([angle, 0, 0, 0]).buffer, | ||
| outSize: { width: outW, height: outH }, | ||
| }); |
There was a problem hiding this comment.
This WebGPU op registers the rotate filter but uses shader:"passthrough". Since shader sources are keyed by name (and rotate has its own key), this will silently remain a no-op even after ROTATE_SRC is replaced with a real implementation. Use shader:"rotate" so the filter automatically picks up the correct WGSL when implemented.
Comment on lines
+84
to
92
| * Image input schema — hydrated to GpuImage by the runner via the image input resolver. | ||
| */ | ||
| export const TypeImageInput = { | ||
| oneOf: [ | ||
| { | ||
| type: "string", | ||
| title: "Image Data", | ||
| description: "Image as data-uri", | ||
| format: "image:data-uri", | ||
| }, | ||
| { | ||
| type: "object", | ||
| additionalProperties: false, | ||
| properties: { | ||
| data: { | ||
| oneOf: [ | ||
| { | ||
| type: "object", | ||
| format: "image:ImageBitmap", | ||
| title: "ImageBitmap", | ||
| }, | ||
| { | ||
| type: "object", | ||
| format: "image:OffscreenCanvas", | ||
| title: "OffscreenCanvas", | ||
| }, | ||
| { | ||
| type: "object", | ||
| format: "image:VideoFrame", | ||
| title: "VideoFrame", | ||
| }, | ||
| { | ||
| type: "object", | ||
| properties: { | ||
| data: { | ||
| type: "array", | ||
| items: { type: "number", format: "Uint8Clamped" }, | ||
| format: "Uint8ClampedArray", | ||
| title: "Data", | ||
| description: "Data of the image", | ||
| }, | ||
| width: { type: "number", title: "Width", description: "Width of the image" }, | ||
| height: { type: "number", title: "Height", description: "Height of the image" }, | ||
| channels: { | ||
| type: "number", | ||
| title: "Channels", | ||
| description: "Channels of the image", | ||
| }, | ||
| rawChannels: { | ||
| type: "number", | ||
| title: "Raw Channels", | ||
| description: "Raw channels of the image", | ||
| }, | ||
| }, | ||
| additionalProperties: false, | ||
| required: ["data", "width", "height", "channels"], | ||
| format: "image:ImageBinary", | ||
| title: "ImageBinary", | ||
| }, | ||
| ], | ||
| }, | ||
| width: { type: "number", title: "Width", description: "Width of the image" }, | ||
| height: { type: "number", title: "Height", description: "Height of the image" }, | ||
| channels: { | ||
| type: "number", | ||
| title: "Channels", | ||
| description: "Channels of the image", | ||
| minimum: 1, | ||
| maximum: 4, | ||
| }, | ||
| }, | ||
| required: ["data", "width", "height", "channels"], | ||
| }, | ||
| ], | ||
| type: "object", | ||
| properties: {}, | ||
| title: "Image", | ||
| description: "Image as URL, data URI, ImageBitmap, ImageBinary, or GpuImage — hydrated by the runner", | ||
| format: "image", | ||
| description: "Image as URL or base64-encoded data", | ||
| } as const satisfies JsonSchema; |
There was a problem hiding this comment.
TypeImageInput's description claims the runner hydrates URLs and other string schemes, but the registered format:"image" resolver currently only supports data: URIs (and throws for other schemes). Either narrow the description to reflect current behavior or implement/register resolvers for the additional supported schemes (e.g. image:http).
Comment on lines
+27
to
+30
| export async function imageBinaryToBlob(bin: ImageBinary, mimeType = "image/png"): Promise<Blob> { | ||
| const bytes = await encodeImageBinaryToPng(bin); | ||
| return new Blob([bytes.buffer as ArrayBuffer], { type: mimeType }); | ||
| } |
There was a problem hiding this comment.
imageBinaryToBlob() always encodes PNG bytes via encodeImageBinaryToPng(), but sets the Blob MIME type from the mimeType parameter. For non-PNG mimeType values this produces a Blob whose declared type doesn't match its content. Either encode using the requested mimeType (via encodeDataUri(bin, mimeType)) or rename/scope the helper to PNG only.
Comment on lines
+24
to
+26
| const DEFAULT_CAPACITY_PER_SIZE = 8; | ||
| const TEXTURE_USAGE = 0x04 | 0x10 | 0x01 | 0x02; // TEXTURE_BINDING | RENDER_ATTACHMENT | COPY_SRC | COPY_DST | ||
|
|
There was a problem hiding this comment.
TEXTURE_USAGE is defined via hard-coded bitmasks. Prefer GPUTextureUsage.* flags (or a small local enum alias) so the intent stays clear and future WebGPU changes/types are respected.
Comment on lines
+12
to
+19
| const codecs = new Map<string, PortCodec>(); | ||
|
|
||
| export function registerPortCodec<Live = unknown, Wire = unknown>( | ||
| formatPrefix: string, | ||
| codec: PortCodec<Live, Wire>, | ||
| ): void { | ||
| codecs.set(formatPrefix, codec as PortCodec); | ||
| } |
There was a problem hiding this comment.
PortCodecRegistry stores codecs in a module-scoped Map. In environments where multiple bundle copies of @workglow/util can be loaded (similar to the globalContainer issue addressed in Container.ts), registrations may not be shared. Consider backing the registry with globalThis + Symbol.for (same pattern as globalContainer) so codecs are truly global across entrypoints/bundles.
Comment on lines
+10
to
+15
| registerFilterOp<ResizeParams>("webgpu", "resize", (image, { width, height }) => { | ||
| return (image as WebGpuImage).apply({ | ||
| shader: "passthrough", | ||
| uniforms: undefined, | ||
| outSize: { width, height }, | ||
| }); |
There was a problem hiding this comment.
This WebGPU op uses shader:"passthrough" for resize. Since RESIZE_SRC exists as a distinct shader key, keeping passthrough here will prevent the resize shader from ever being used once implemented. Use shader:"resize" so the op matches the intended shader name.
Comment on lines
+10
to
+15
| registerFilterOp<CropParams>("webgpu", "crop", (image, { left, top, width, height }) => { | ||
| return (image as WebGpuImage).apply({ | ||
| shader: "passthrough", | ||
| uniforms: new Float32Array([left, top, width, height]).buffer, | ||
| outSize: { width, height }, | ||
| }); |
There was a problem hiding this comment.
This WebGPU op uses shader:"passthrough" for crop even though CROP_SRC exists as a distinct shader key. Use shader:"crop" so the filter will actually crop once the crop WGSL is implemented (and to avoid future mismatches).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.