diff --git a/README.md b/README.md index a45382c..86e7cb0 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,10 @@ Computer Vision ML inference in C++ Based on [ggml](https://github.com/ggml-org/ggml) similar to the [llama.cpp](https://github.com/ggml-org/llama.cpp) project. +New here? [**Getting started**](docs/getting-started.md) walks through a first run in five minutes. + +**Docs:** [Getting started](docs/getting-started.md) · [Overview](docs/overview.md) · [Command line](docs/using-the-cli.md) · [Library API](docs/using-the-library.md) · [Implementing a model](docs/model-implementation-guide.md) + ### Features | Model | Task | Backends | diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..ddcc474 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,101 @@ +# Getting Started + +In this tutorial you will cut an object out of a photo using vision.cpp. It takes about five +minutes and needs nothing but the release package, one model file and one image — no build, no +Python, no conversion. + +At the end you will have this: + +| | | +| :--- | :--- | +| `mask.png` | a black-and-white mask of the object | +| `object.png` | the original photo with the background dimmed away | + +## Step 1 — Get the executable + +Download a [release package](https://github.com/Acly/vision.cpp/releases) and extract it. You +will find `vision-cli` in the `bin` folder. + +Check that it runs: + +```sh +vision-cli --help +``` + +You should see a list of commands: `sam`, `birefnet`, `depthany`, `migan`, `esrgan`. + +> If you would rather build from source, follow [Building](../README.md#building) first, then +> come back here. `vision-cli` ends up in `build/bin`. + +## Step 2 — Get a model and an image + +The executable contains the network structure, but not the weights. Download them: + +```sh +curl -L -O https://huggingface.co/Acly/BiRefNet-GGUF/resolve/main/BiRefNet-lite-F16.gguf +``` + +This is BiRefNet, a model that separates a subject from its background. The file is a +[GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) — the weights and nothing else. + +For the input, use any photo with a clear subject. If you cloned the repository, there is one +at `docs/media/input.jpg`. Put it next to the model file and call it `input.jpg`. + +## Step 3 — Run it + +```sh +vision-cli birefnet -m BiRefNet-lite-F16.gguf -i input.jpg -o mask.png --composite object.png +``` + +The output tells you what it is doing: + +``` +Initializing backend... done (1.1 ms) +- device: CPU - Intel(R) Core(TM) i3-14100 +Loading model weights from 'BiRefNet-lite-F16.gguf'... done (151.3 ms) +- float type: f16 +- tensor layout: cwhn +- model image size: 1024 +- inference image size: 1024x1024 +- flash attention: off +Running inference... complete (5372.6 ms) +-> mask saved to mask.png +-> image composited and saved to object.png +``` + +Inference takes a few seconds on a desktop CPU. Loading the weights takes a fraction of a +second — that number is the point of the project, and it is the same on any machine. + +## Step 4 — Look at the result + +Open `object.png`. The subject is untouched and the background has faded away. + +`mask.png` is what the model actually produced: white where the subject is, black elsewhere. +Everything in `object.png` was computed from it. + +That is the whole loop. An executable that already knows the network, a `.gguf` that carries +the weights, an image in, a result out. + +## Try one more + +The same executable runs the other built-in models. Only the command and the weights change: + +```sh +curl -L -O https://huggingface.co/Acly/Real-ESRGAN-GGUF/resolve/main/RealESRGAN-x4plus_anime-6B-F16.gguf + +vision-cli esrgan -m RealESRGAN-x4plus_anime-6B-F16.gguf -i input.jpg -o upscaled.png +``` + +This one upscales the image four times. It works on tiles and takes noticeably longer — you will +see it count them off. + +## Where to go next + +- [Overview](overview.md) — what the library is and why weights and structure are separate. +- [Using the command line](using-the-cli.md) — every option, every built-in model. +- [Using the library](using-the-library.md) — the same models from your own code. +- [README](../README.md#features) — the other built-in models, and what each one does. +- [Model implementation guide](model-implementation-guide.md) — when the model you want is not + in the list, and you want to add it. +- [MMDetection detectors](mmdet-detectors.md) — running detectors whose structure is generated + rather than hand-written. diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..5b50afd --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,111 @@ +# Overview + +vision.cpp is a C++ library for running computer-vision neural networks. It loads weights +from a GGUF file, builds a compute graph with [ggml](https://github.com/ggml-org/ggml), and +executes it on CPU or GPU. The result is a single native binary with no Python, no framework +runtime, and no interchange-format interpreter. + +It is the same idea as [llama.cpp](https://github.com/ggml-org/llama.cpp), applied to vision +models instead of language models. + +## The idea: a model is code, not a file + +Most inference stacks treat a model as data. You export a graph to ONNX or TorchScript, and +a general-purpose runtime loads that graph, matches its operators against a kernel library, and +interprets it. The runtime has to support every operator anyone might export, so it is large, +and it has to plan the graph at startup, so loading takes time. + +vision.cpp splits the model in two: + +| Part | Form | Where it lives | +| :--- | :--- | :--- | +| Structure | C++ that builds a ggml graph | compiled into your binary | +| Weights | GGUF tensors | a `.gguf` file loaded at run time | + +Nothing interprets a graph description at run time, because there is no graph description — the +graph is the code you compiled. That is what makes the deployment small and start-up fast, and +it is the trade-off at the centre of the project: adding a model that isn't supported yet +means writing or generating code, not exporting a file. + +Weights stay external, so swapping checkpoints, changing precision, or quantising does not +require rebuilding. + +## What you get + +- Self-contained. The only dependencies are ggml, `stb` for image I/O, and optionally + `fmt`. There is no Python in the runtime path. +- CPU and GPU. CPU works everywhere; Vulkan covers NVIDIA, AMD and Intel from one build. +- Small and quick to start. Deployment size and model-load time are explicit goals of the + project — see the [Performance](../README.md#performance) section for the current numbers. +- Modular. The same primitives the built-in models are made of are public, so you can + assemble your own. + +## How it fits together + +The library is layered. Each layer is usable on its own; you can stop at whichever one matches +how much control you need. + +| Layer | Header | What it gives you | +| :--- | :--- | :--- | +| Model APIs | `visp/vision.h` | Ready-made models — load, run, get a result. | +| Image I/O | `visp/image.h` | Load, save, resize, tile, convert. | +| Neural network layers | `visp/nn.h` | `conv_2d`, `group_norm`, attention, and other building blocks. | +| Graph and backends | `visp/ml.h` | GGUF loading, weight transfer, graph construction, execution. | +| Detection post-processing | `visp/postproc.h` | Anchors, decoding, NMS, RoIAlign, masks. | +| Tracking | `visp/tracker.h` | ByteTrack association across frames. | + +Two front-ends are built on top: + +- `vision-cli` — a command-line tool for the built-in models + (`vision-cli sam -m MobileSAM-F16.gguf -i image.jpg -p 100 200 -o mask.png`). +- Python bindings — `bindings/python`, for scripting and comparison against reference + implementations. + +## Running a model + +Most of the time there is nothing to add. The models in the +[README](../README.md#features) are already implemented, so running one means downloading its +weights and pointing at them: + +```sh +vision-cli birefnet -m BiRefNet-lite-F16.gguf -i photo.jpg -o mask.png +``` + +Or from your own program, in three calls — pick a device, load the weights, compute. See +[using the command line](using-the-cli.md) and [using the library](using-the-library.md). + +If you have your own checkpoint for one of those architectures, convert it with +`scripts/convert.py`. The structure is already in the library; only the weights change. + +## Adding a model + +When the architecture is not implemented yet, it has to be written. The +[model implementation guide](model-implementation-guide.md) walks through it: describe the +network with the `nn.h` primitives, and provide a conversion function that turns the original +checkpoint into GGUF. Every built-in model was added this way, and it gives the most control +over layout and precision. + +The result is what the library loads: a `_forward` function that builds a graph, plus a +GGUF file of weights. + +For model families with hundreds of variants, hand-writing each one is not realistic and the +C++ can be generated from a traced PyTorch module instead. The +[MMDetection guide](mmdet-detectors.md) covers that case — the interface generated code must +satisfy, and how to handle what tracing cannot capture. + +## Scope + +vision.cpp is an inference library. There is no training, no autograd, and no optimizer. + +It is also not a general model runtime: it does not aim to execute arbitrary exported graphs. +Supported models are the ones that have been implemented or generated, which is why the model +list in the [README](../README.md#features) is finite and why growing it is a code change. + +## Next + +- [Getting started](getting-started.md) — run a model end to end in five minutes. +- [Using the command line](using-the-cli.md) — every built-in model, no code. +- [Using the library](using-the-library.md) — the same models from C++ or Python. +- [README](../README.md) — install, build, supported models, performance. +- [Model implementation guide](model-implementation-guide.md) — write a model by hand. +- [MMDetection detectors](mmdet-detectors.md) — run detectors from a compiled backbone. diff --git a/docs/using-the-cli.md b/docs/using-the-cli.md new file mode 100644 index 0000000..1b7fa8d --- /dev/null +++ b/docs/using-the-cli.md @@ -0,0 +1,160 @@ +# Using the command line + +`vision-cli` runs every built-in model without writing any code. All you need is the executable +and a `.gguf` weights file. + +If you have not run anything yet, start with [Getting started](getting-started.md). + +## The shape of a command + +```sh +vision-cli -m -i -o +``` + +The command selects the model, `-m` says which weights to load, `-i` and `-o` are files. + +| Command | Task | Input | Output | +| :--- | :--- | :--- | :--- | +| `birefnet` | Background removal | image | mask | +| `sam` | Segment one object you point at | image + prompt | mask | +| `depthany` | Depth estimation | image | depth map | +| `migan` | Inpainting — fill a region | image + mask | image | +| `esrgan` | Upscaling | image | larger image | + +## Options + +`-m, --model ` +: The `.gguf` weights. Required. + +`-i, --input [ ...]` +: Input image. `migan` takes two — the image and the mask. + +`-o, --output ` +: Output file. Defaults to `output.png`. + +`-p, --prompt [ ...]` +: Prompt for models that take one. `sam` accepts a point (`x y`) or a box + (`x1 y1 x2 y2`) in pixels, origin top-left. + +`-b, --backend ` +: Which device to run on. Defaults to automatic — GPU if the build has Vulkan and a device is + available, CPU otherwise. + +`--composite ` +: Also write the input image combined with the resulting mask, instead of the mask alone. + +`--tile ` +: Split large inputs into tiles of this size. Used by `esrgan` to keep memory bounded. + +`-h, --help` +: Print the command list and exit. + +## Getting weights + +Each model has its own GGUF repository. Download the file and pass it with `-m`. + +| Model | Weights | +| :--- | :--- | +| MobileSAM | [Acly/MobileSAM-GGUF](https://huggingface.co/Acly/MobileSAM-GGUF) | +| BiRefNet | [Acly/BiRefNet-GGUF](https://huggingface.co/Acly/BiRefNet-GGUF) | +| Depth-Anything V2 | [Acly/Depth-Anything-V2-GGUF](https://huggingface.co/Acly/Depth-Anything-V2-GGUF) | +| MI-GAN | [Acly/MIGAN-GGUF](https://huggingface.co/Acly/MIGAN-GGUF) | +| Real-ESRGAN | [Acly/Real-ESRGAN-GGUF](https://huggingface.co/Acly/Real-ESRGAN-GGUF) | + +Several variants are usually available per model — different sizes or resolutions. The +executable reads which one it got from the file's metadata, so no extra flag is needed. + +## Remove a background + +```sh +vision-cli birefnet -m BiRefNet-lite-F16.gguf -i photo.jpg -o mask.png --composite cutout.png +``` + +`mask.png` is white where the subject is. `cutout.png` is the photo with the background removed. + +## Segment one object + +Unlike background removal, this needs to be told which object. Give a point inside it: + +```sh +vision-cli sam -m MobileSAM-F16.gguf -i photo.jpg -p 300 200 -o mask.png +``` + +or a box around it: + +```sh +vision-cli sam -m MobileSAM-F16.gguf -i photo.jpg -p 420 120 650 430 -o mask.png +``` + +A box is usually more reliable when the object touches others. + +## Estimate depth + +```sh +vision-cli depthany -m Depth-Anything-V2-Small-F16.gguf -i photo.jpg -o depth.png +``` + +The output is a single-channel image — bright is near, dark is far. Values are relative to the +image, not metric distances. + +## Fill a region + +Inpainting takes two inputs: the image, and a mask marking what to replace. + +```sh +vision-cli migan -m MIGAN-512-places2-F16.gguf -i photo.jpg mask.png -o filled.png +``` + +White in the mask is the region to fill. You can produce that mask with `birefnet` or `sam`, +which makes removing an object a two-step operation. + +## Upscale + +```sh +vision-cli esrgan -m RealESRGAN-x4plus_anime-6B-F16.gguf -i photo.jpg -o large.png +``` + +The scale factor comes from the weights — the model above is 4×. Large inputs are processed in +tiles; you will see them counted off, and the whole run takes considerably longer than the other +models. + +## Choosing a device + +```sh +vision-cli birefnet -m BiRefNet-lite-F16.gguf -i photo.jpg -o mask.png -b gpu +``` + +GPU requires a build with Vulkan enabled — see [Building](../README.md#building). Without it, +`-b gpu` has nothing to select and the run stays on CPU. The first two lines of output always +name the device that was actually used. + +## Using your own weights + +If you have a checkpoint for an architecture the library already implements, convert it to GGUF +rather than looking for a pre-made file. + +```sh +uv run scripts/convert.py MyModel.pth +``` + +`` is one of `sam`, `sam3`, `birefnet`, `depth-anything`, `migan`, `esrgan`. +The result lands in `models/`. + +| Option | Description | +| :--- | :--- | +| `-o, --output` | Output directory or file. Default `models`. | +| `-q, --quantize f16` | Store float weights as f16 — roughly half the file size. | +| `-l, --layout whcn\|cwhn` | Tensor layout for 2D operations. Leave unset unless you know you need the other one. | +| `--model-name` | Name recorded in the file's metadata. | +| `-v, --verbose` | Print every tensor as it is converted. | + +Conversion also rearranges and precomputes tensors, so it is not a pure format change — this is +why a checkpoint cannot be loaded directly. + +This route only covers architectures that exist in the library. For anything else, see the +[model implementation guide](model-implementation-guide.md). + +## Next + +- [Using the library](using-the-library.md) — the same models from your own C++ or Python code. +- [Overview](overview.md) — why weights and structure are separate files. diff --git a/docs/using-the-library.md b/docs/using-the-library.md new file mode 100644 index 0000000..422c413 --- /dev/null +++ b/docs/using-the-library.md @@ -0,0 +1,125 @@ +# Using the library + +Everything `vision-cli` does is a few calls into `libvisioncpp`. This page shows the shape of +those calls so you can put the models inside your own program. + +## The pattern + +Every built-in model follows the same three steps: pick a device, load the weights, compute. + +```c++ +#include +using namespace visp; + +int main() { + backend_device dev = backend_init(); // 1. device + birefnet_model model = birefnet_load_model("BiRefNet-lite-F16.gguf", dev); // 2. weights + + image_data input = image_load("photo.jpg"); + image_data mask = birefnet_compute(model, input); // 3. compute + + image_save(mask, "mask.png"); +} +``` + +The structure of the network is already in the library — that is why loading takes only the +weights file. Nothing is parsed or planned at start-up beyond reading the tensors. + +## Devices + +```c++ +backend_device dev = backend_init(); // best available +backend_device cpu = backend_init(backend_type::cpu); +backend_device gpu = backend_init(backend_type::gpu); +``` + +`backend_init()` with no argument picks the GPU when the build has Vulkan and a device is +present, and falls back to CPU. One device is used for the whole model; pass it to the load +function and it decides where weights and computation live. + +## The models + +Each model has a `_load_model` and a `_compute`. The differences are only in what goes in and +what comes out. + +| Model | Load | Compute | +| :--- | :--- | :--- | +| BiRefNet | `birefnet_load_model(path, dev)` | `birefnet_compute(m, image)` → alpha mask | +| Depth-Anything | `depthany_load_model(path, dev)` | `depthany_compute(m, image)` → depth, f32 in [0, 1] | +| MI-GAN | `migan_load_model(path, dev)` | `migan_compute(m, image, mask)` → filled image | +| ESRGAN | `esrgan_load_model(path, dev)` | `esrgan_compute(m, image)` → upscaled image | +| MobileSAM | `sam_load_model(path, dev)` | see below — two calls | + +SAM is split because the expensive part does not depend on the prompt. Encode the image once, +then ask for as many objects as you like: + +```c++ +sam_model sam = sam_load_model("MobileSAM-F16.gguf", dev); + +sam_encode(sam, image); // once per image + +image_data a = sam_compute(sam, i32x2{300, 200}); // by point +image_data b = sam_compute(sam, box_2d{{420, 120}, {650, 430}}); // by box +``` + +Prompt coordinates are pixels with the origin in the top-left corner. + +## Images + +`image_data` owns its pixels; `image_view` refers to pixels someone else owns. Functions take +views, so you can pass data you already have without copying it. + +```c++ +image_data img = image_load("photo.jpg"); // from disk +image_save(img, "out.png"); // to disk + +image_view v{extent, image_format::rgba_u8, my_buffer}; // wrap your own memory +``` + +That last form is the one to reach for when frames come from a camera, a decoder, or another +part of your application — nothing needs to go through a file. + +## Going lower + +The one-call functions above are compositions. Each model also exposes the steps separately: +parameter detection, pre-processing, graph construction, post-processing. + +```c++ +birefnet_params p = birefnet_detect_params(file); // read shape/variant from the GGUF +image_data in = birefnet_process_input(image, p); // resize, normalise +tensor out = birefnet_predict(m, input_tensor, p); // build the graph +image_data mask = birefnet_process_output(data, target_extent, p); +``` + +Use these when you need to batch work, keep tensors on the device between stages, run +pre-processing somewhere else, or share a compute graph across calls. `visp/ml.h` has the +pieces underneath — `model_load`, `model_transfer`, `compute_graph_init`, `compute`. + +## Detection post-processing + +If you are building a detector rather than using a built-in model, `visp/postproc.h` has the +parts that are not neural networks: anchor generation, box decoding, NMS, RoIAlign, mask +pasting. `visp/tracker.h` has ByteTrack for keeping identities across frames. Both are plain +CPU code and take structs, not framework config. + +The [MMDetection guide](mmdet-detectors.md) shows them assembled into a working detector. + +## Python + +The bindings cover the same models for scripting and comparison work. + +```python +from visioncpp import Device, Model, Backend + +device = Device.init(Backend.auto) +model = Model.load("BiRefNet-lite-F16.gguf", device) +mask = model.compute(image) +``` + +They live in `bindings/python`. The C++ API is the reference; the bindings follow it. + +## Next + +- [Using the command line](using-the-cli.md) — the same models without writing code. +- [Model implementation guide](model-implementation-guide.md) — adding a model the library does + not have.