diff --git a/README.md b/README.md index a794acd..eacbe6f 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ New here? [**Getting started**](docs/getting-started.md) walks through a first r | [**ESRGAN**](#real-esrgan) | Super-resolution | CPU, Vulkan | | [**YOLOv9t**](#yolov9t) | Object detection | CPU | | [**MMDetection** models](docs/mmdet-detectors.md) | Object detection, segmentation, tracking | CPU | -| [**Compiled PyTorch models**](docs/mmdet-detectors.md#models-whose-head-survives-tracing) | Any traceable `nn.Module` — ultralytics YOLO, torchvision · full guide in the compiler checkout, `docs/vision-cpp-mmdet-guide-en.md` | CPU | +| [**Compiled PyTorch models**](docs/mmdet-detectors.md#models-whose-head-survives-tracing) | Any traceable `nn.Module` — ultralytics YOLO, torchvision · needs the compiler, [GTX_Compiler](https://github.com/Sudo42b/GTX_Compiler), which carries this repository as a submodule | CPU | | [_Implement a model [**Guide**]_](docs/model-implementation-guide.md) | | | **Backbones:** SWIN (v1), DINO (v2), TinyViT @@ -36,6 +36,10 @@ Get the library and executables: * Download a [release package](https://github.com/Acly/vision.cpp/releases) and extract it, * or [build from source](#building). +> Those releases come from upstream and carry the built-in models only. A compiled PyTorch +> model is added by rebuilding this fork — see [MMDetection models](docs/mmdet-detectors.md), +> so [build from source](#building) if that is what you are here for. + ### Example: Select an object in an image Let's use MobileSAM to generate a segmentation mask of the plushy on the right @@ -51,7 +55,7 @@ You can download the model and input image here: [MobileSAM-F16.gguf](https://hu Find the `vision-cli` executable in the `bin` folder and run it to generate the mask: ```sh -vision-cli -m MobileSAM-F16.gguf -i input.jpg -p 420 120 650 430 -o mask.png +vision-cli sam -m MobileSAM-F16.gguf -i input.jpg -p 420 120 650 430 -o mask.png ``` Pass `--composite output.png` to composite input and mask. Use `--help` for more options. @@ -61,14 +65,14 @@ Pass `--composite output.png` to composite input and mask. Use `--help` for more #include using namespace visp; -void main() { +int main() { backend_device cpu = backend_init(backend_type::cpu); sam_model sam = sam_load_model("MobileSAM-F16.gguf", cpu); image_data input_image = image_load("input.jpg"); sam_encode(sam, input_image); - image_data object_mask = sam_compute(sam, box_2d{{420, 120}, {650, 320}}); + image_data object_mask = sam_compute(sam, box_2d{{420, 120}, {650, 430}}); image_save(object_mask, "mask.png"); } ``` @@ -109,7 +113,7 @@ vision-cli birefnet -m BiRefNet-lite-F16.gguf -i input.png -o mask.png --composi [Model download](https://huggingface.co/Acly/Depth-Anything-V2-GGUF/tree/main) | [Paper (arXiv)](https://arxiv.org/abs/2406.09414) | [Repository (GitHub)](https://github.com/DepthAnything/Depth-Anything-V2) | License: Apache-2 / CC-BY-NC-4 ```sh -vision-cli depth-anything -m Depth-Anything-V2-Small-F16.gguf -i input.png -o depth.png +vision-cli depthany -m Depth-Anything-V2-Small-F16.gguf -i input.png -o depth.png ``` #### MI-GAN @@ -150,7 +154,7 @@ To convert a model, install [uv](https://docs.astral.sh/uv/) and run: ```sh uv run scripts/convert.py MyModel.pth ``` -where `` is one of `sam, birefnet, esrgan, ...`. +where `` is one of `sam`, `sam3`, `birefnet`, `depth-anything`, `migan`, `esrgan`. This will create `models/MyModel.gguf`. See `convert.py --help` for more options. @@ -160,7 +164,7 @@ Building requires CMake and a compiler with C++20 support. **Get the sources** ```sh -git clone https://github.com/Acly/vision.cpp.git --recursive +git clone https://github.com/Sudo42b/vision.cpp.git --recursive cd vision.cpp ``` @@ -170,6 +174,10 @@ cmake . -B build cmake --build build --config Release ``` +The configure step downloads the five built-in models — about 180 MB into `models/` — because +tests are on by default in a standalone clone and the tests need them. `-D VISP_TESTS=OFF` +skips both the tests and the download. + ### Vulkan _(Optional)_ Building with Vulkan GPU support requires the [Vulkan SDK](https://www.lunarg.com/vulkan-sdk/) to be installed. @@ -178,9 +186,12 @@ Building with Vulkan GPU support requires the [Vulkan SDK](https://www.lunarg.co cmake . -B build -D VISP_VULKAN=ON ``` -### Tests _(Optional)_ +### Tests -Build with `-DVISP_TESTS=ON`. Run all C++ tests with the following command: +Tests are **on by default** whenever vision.cpp is the project CMake was pointed at — the clone +above, and equally `cmake -S vision.cpp -B vision.cpp/build` from a parent checkout. They are +off only when a parent `CMakeLists.txt` pulls this one in with `add_subdirectory`. +`-D VISP_TESTS=OFF` turns them off in every case. Run all C++ tests with the following command: ```sh cd build ctest -C Release diff --git a/docs/getting-started.md b/docs/getting-started.md index ddcc474..4363fd4 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -27,6 +27,10 @@ You should see a list of commands: `sam`, `birefnet`, `depthany`, `migan`, `esrg > If you would rather build from source, follow [Building](../README.md#building) first, then > come back here. `vision-cli` ends up in `build/bin`. +> The release packages come from upstream and carry the five built-in models. A model compiled +> from PyTorch is added by rebuilding this fork, so that route starts from source — see +> [MMDetection detectors](mmdet-detectors.md). This tutorial only needs the release. + ## Step 2 — Get a model and an image The executable contains the network structure, but not the weights. Download them: @@ -38,6 +42,9 @@ curl -L -O https://huggingface.co/Acly/BiRefNet-GGUF/resolve/main/BiRefNet-lite- This is BiRefNet, a model that separates a subject from its background. The file is a [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) — the weights and nothing else. +> Built from source instead? You already have it. Configuring the build downloads all five +> built-in models into `models/`, so use `models/BiRefNet-lite-F16.gguf` and skip the `curl`. + For the input, use any photo with a clear subject. If you cloned the repository, there is one at `docs/media/input.jpg`. Put it next to the model file and call it `input.jpg`. diff --git a/docs/mmdet-detectors.md b/docs/mmdet-detectors.md index f7ff0a5..5710eb0 100644 --- a/docs/mmdet-detectors.md +++ b/docs/mmdet-detectors.md @@ -4,13 +4,15 @@ This guide describes how to run detectors from [MMDetection](https://github.com/open-mmlab/mmdetection) with vision.cpp. MMDetection defines hundreds of detectors as compositions of a backbone, a neck and a head. +The neck is usually an FPN — a feature pyramid network, which turns one backbone output into +several feature maps at different resolutions, the *levels* this document keeps referring to. The backbone and neck are plain feed-forward networks and translate directly into a ggml graph. vision.cpp splits the model there: the feature extractor runs as a compiled ggml graph, and the head, decoding and post-processing are C++ built from library primitives. One head assembled that way serves every family that shares its structure. ``` - image ──▶ backbone + neck (compiled ggml graph) ──▶ FPN features + image ──▶ backbone + neck (compiled ggml graph) ──▶ FPN levels │ head │ tools/detect/head.cpp ▼ @@ -90,15 +92,23 @@ Outputs: they are compiled into the runner rather than read at run time. See [Configuration reference](#configuration-reference). -The `.pt` pickles by module name `mmdet_wrap`. The export writes `mmdet_wrap.py` and -`mmdet_compat.py` beside it, and the compiler's loader puts the `.pt`'s own directory on the -import path, so the file opens anywhere with nothing set in the environment. +Those two are what you read. Two more land beside them, and they are not optional: saving a +module pickles its classes by module name, so whatever opens the `.pt` has to be able to +import `mmdet_wrap`. The export copies `mmdet_wrap.py` and `mmdet_compat.py` next to the `.pt` +for exactly that reason. The compiler puts the `.pt`'s own directory on `sys.path`, so the +file opens with **nothing set in the environment** — no `PYTHONPATH`. + +Keep the four together when you move them. A `.pt` separated from its wrapper fails at load in +a way that reads like a model problem. ### Step 2 — Compile the backbone -`backbone.pt` is a plain PyTorch module and is compiled to a vision.cpp arch module by a -PyTorch-to-ggml model compiler. That compiler is outside the scope of this document; what -matters is the interface the generated code must satisfy. +`backbone.pt` is a plain PyTorch module and is compiled to a vision.cpp arch module by +**g2c**, the PyTorch-to-ggml compiler at +[GTX_Compiler](https://github.com/Sudo42b/GTX_Compiler) — the project that carries this +repository as a submodule. Its own workings are outside the scope of this document; what +matters here is the interface the generated code must satisfy. The same compiler handles the +whole-model route at the end of this document, so it is one tool, not two. Compile at the resolution `--size` used in step 1. Tracing records the operations for one input shape, so the graph runs at that shape and no other. A graph built for a different size @@ -230,7 +240,7 @@ which keeps a font out of the runner. | Variable | Effect | | :--- | :--- | | `VISP_DRAW_THRESHOLD` | Minimum score to draw. Default `0.3` | -| `VISP_PRINT_DETS` | How many rows to print. `0` turns the table off | +| `VISP_PRINT_DETS` | How many rows to print. Default `10`; `0` turns the table off | In the raw form each detection is six `float32` values written back to back: @@ -257,8 +267,9 @@ then none of the steps above apply. A compiler emits the entire graph, `install_ it into `src/visp/arch/` with a registration unit beside it, and `vision-cli` dispatches on the architecture name recorded in the GGUF: -Run these from the **compiler checkout root** — the directory that holds `vision.cpp/` and -`pyproject.toml`. `g2c` is that project's console script, not one of vision.cpp's, and `uv run` +Run these from the **compiler checkout root** — your clone of +[GTX_Compiler](https://github.com/Sudo42b/GTX_Compiler), the directory that holds `vision.cpp/` +and `pyproject.toml`. `g2c` is that project's console script, not one of vision.cpp's, and `uv run` started inside `vision.cpp/` finds no project there: it builds a second virtual environment and then fails to spawn. @@ -267,9 +278,18 @@ uv run g2c --model "ultralytics.YOLO('yolo26m.pt')" --name Yolo26m \ --output out --input-shape 1,3,640,640 python vision.cpp/tools/install_arch.py out --name Yolo26m --detect-yolo cmake --build vision.cpp/build -j4 -./vision.cpp/build/bin/vision-cli yolo26m -m out/Yolo26m.gguf -i photo.jpg -o detected.jpg +./vision.cpp/build/bin/vision-cli yolo26m -m out/Yolo26m.gguf \ + -i vision.cpp/tests/input/cat-and-hat.jpg -o detected.png ``` +> ⚠️ **Read the `→ gguf:` line, not the exit code.** A `g2c` run that prints `done` without a +> `→ gguf:` line has failed, and it still exits `0`. A script that checks the exit status will +> carry on with no weights file. +> +> ⚠️ **Registration needs a shared library.** The registration object is referenced by nothing, +> so a static link discards the whole translation unit. The build succeeds and the command +> silently disappears from `vision-cli --help`. + `--detect-yolo` supplies what the GGUF does not carry — class count, strides, whether the head is NMS-free — so the result comes back as boxes: `vision-cli` draws them and prints their coordinates and scores. Registered without the flag, the same command writes the graph @@ -288,395 +308,18 @@ within 0.1 px, and the pre-decode tensors are at relative L1 1.7e-03 on the boxe ## What decodes to boxes -**Eighty-six families are verified against MMDetection** — forty-one single-stage and -thirty-eight two-stage agree on boxes; seven more are checked at the compiled-graph level -because they emit no boxes to compare (five mask-only families and three text-conditioned ones, -one of which is also single-stage). - -**The tables below will not add up to those figures, and that is deliberate.** A table lists -only what clears the strict bar — box under 2 px, score under 0.05, no label or count -mismatch — so the single-stage table holds forty rows and the two-stage table thirty-five. -The remainder are verified but sit outside the bar for a stated reason, and each is accounted -for in the sections that follow: `free_anchor` and `yolact` land on a score-cut boundary, -`dynamic_rcnn`, `pafpn` and `res2net` are several pixels out from fp16 weights alone (0.00 px -when recompiled in fp32), and `double_heads` agrees at 0.16 px. Counting rows and expecting the -summary is the mistake; the rows are the strict set, not the verified set. - -Assembling a head and decoding its output are -separate steps, and a family can pass the first and fail the second. The runner picks a decoder from what the box prediction *is* — a delta -against an anchor, a distance from a grid point, a normalised `cxcywh` query — not from the -shape of the tower that produced it. YOLOX and RPN build the same tower as RetinaNet and decode -nothing like it. - -Each row below was measured against MMDetection's own `predict_by_feat` on the same pixels at -512 (`cat-and-hat.jpg`), with the trained checkpoint the harness pairs with that config. -The harness is `tools/verify/dense_head/verify_postproc.py`; it passes at 2 px, 0.05 score, -no label mismatch and no difference in how many boxes survive. - -| Family | Decoder | Worst box | Worst score | -| :--- | :--- | ---: | ---: | -| `cornernet` | `detect_corner` (embedding pairing) | 0.03 px | 0.003 | -| `bytetrack` | `detect_yolox` (tracker wrapper) | 0.06 px | 0.007 | -| `ddq` | `detect_detr` (distinct queries) | 0.06 px | 0.018 | -| `yolof` | `detect_anchor` (ctr_clamp) | 0.12 px | 0.002 | -| `ocsort` | `detect_yolox` (tracker wrapper) | 0.15 px | 0.041 | -| `centernet` | `detect_centernet` (heatmap peaks) | 0.13 px | 0.001 | -| `atss` | `detect_anchor` | 0.14 px | 0.000 | -| `dyhead` | `detect_anchor` (center_offset 0.5) | 0.21 px | 0.000 | -| `efficientnet` | `detect_anchor` | 0.15 px | 0.005 | -| `pisa` | `detect_anchor` | 0.18 px | 0.005 | -| `nas_fcos` | `detect_fcos` | 0.19 px | 0.001 | -| `condinst` | `detect_fcos` (mask branch ignored) | 0.19 px | 0.003 | -| `gfl` | `detect_fcos` | 0.23 px | 0.004 | -| `yolo` | `detect_yolov3` | 0.23 px | 0.005 | -| `boxinst` | `detect_fcos` (mask branch ignored) | 0.25 px | 0.001 | -| `strongsort` | `detect_yolox` (tracker wrapper) | 0.25 px | 0.001 | -| `reppoints` | `detect_fcos` (xyxy offset) | 0.25 px | 0.006 | -| `retinanet` | `detect_anchor` | 0.27 px | 0.000 | -| `conditional_detr` | `detect_detr` | 0.27 px | 0.002 | -| `dab_detr` | `detect_detr` | 0.28 px | 0.001 | -| `dino` | `detect_detr` | 0.28 px | 0.004 | -| `deformable_detr` | `detect_detr` | 0.26 px | 0.002 | -| `autoassign` | `detect_fcos` | 0.30 px | 0.006 | -| `ld` | `detect_fcos` | 0.30 px | 0.006 | -| `fcos` | `detect_fcos` | 0.32 px | 0.004 | -| `foveabox` | `detect_fcos` (base_edge) | 0.40 px | 0.002 | -| `ddod` | `detect_anchor` | 0.41 px | 0.001 | -| `yolox` | `detect_yolox` | 0.41 px | 0.002 | -| `ssd` | `detect_anchor` (per-level anchors) | 0.42 px | 0.003 | -| `ghm` | `detect_anchor` | 0.46 px | 0.005 | -| `vfnet` | `detect_fcos` | 0.46 px | 0.003 | -| `paa` | `detect_paa` (score voting) | 0.54 px | 0.011 | -| `pvt` | `detect_anchor` (PVT-Tiny) | 0.55 px | 0.005 | -| `sabl` | `detect_sabl` (buckets) | 0.55 px | 0.003 | -| `fsaf` | `detect_anchor` (TBLR coder) | 0.56 px | 0.003 | -| `tood` | `detect_tood` (decoded in-graph) | 0.63 px | 0.003 | -| `rtmdet` | `detect_fcos` | 0.68 px | 0.002 | -| `lad` | `detect_paa` (score voting) | 0.74 px | 0.001 | -| `nas_fpn` | `detect_anchor` | 1.04 px | 0.004 | -| `detr` | `detect_detr` | 1.85 px | 0.044 | - -Every row above was re-measured together in one run, so the numbers are comparable with each -other. That matters more than it sounds: the harness picks a representative config per family -from a hand-written override list, and comparing against the config `metafile.yml` would have -chosen instead silently pairs a compiled graph with someone else's checkpoint. Thirteen -families differ between those two choices, and the mismatch reads as a decode failure — -`retinanet` looked 20 px out and `dino` looked like a regression until the pairing was fixed. - -`pvt` is measured on PVT-Tiny, the variant the table has always used. Selecting the family's -representative config from `metafile.yml` instead picks PVTv2-B5, a different architecture -(overlapping patch embedding, linear spatial reduction), which lands at 15.74 px and is not -covered. Two variants of one family can disagree completely; naming the variant is not -optional. - -Six of those rows were added after the first pass, and every one of them had been recorded as -out of scope. That is worth saying plainly: **"needs its own decoder" is not the same as -"cannot be done"** — the six closed in a day. What they needed was reading the family's own -config rather than assuming defaults. `ssd` lays down a different number of anchors per level; -`yolact` gives base sizes and centres separately from the strides, so the usual -stride-times-scale reconstruction lands 0.859× small and half a cell off; `paa` and `lad` rank -by anchor rather than by (anchor, class) and re-average boxes by score voting, which is -selected by the `with_score_voting` attribute and not by the class name — `lad` subclasses -`PAAHead`, so matching on the name misses it. - -`ld` needs its command run from the MMDetection root. Distillation configs name the teacher as -`teacher_config='configs/gfl/...'`, relative to the working directory rather than to the config -file, so running from anywhere else fails to find it and the family looks broken. - -`condinst` was added without writing a line of code, and finding it was an accounting exercise -rather than an engineering one. Every table in this chapter classifies the families the two -harnesses *run* — thirty-four single-stage plus forty two-stage — but `configs/` holds a -hundred. Subtracting gives twenty-six that neither harness has ever touched, and a family that -is never run cannot appear as a failure. Most of the twenty-six are legitimately outside the -question: seven trackers that wrap a detector rather than being one, five mask-only families -that emit no boxes at all, the three text-conditioned families, `reid`, and the five already -discussed above. Two were not: `condinst` and `boxinst` both have an ordinary box branch, and -both were already listed in `verify_heads.py` with their checkpoints downloaded. Nobody had -run them. - -`condinst` passes because `CondInstBboxHead` extends `FCOSHead` and leaves the box path alone. -Its `forward_single` adds a fourth branch — a `controller` convolution predicting 169 mask -parameters — and its `_predict_by_feat_single` carries `param_pred`, `points` and `strides` -alongside the boxes, but every one of those feeds the mask head. The decode is the FCOS decode: -`DistancePointBBoxCoder` against grid points, sigmoid centerness as the score factor, -`filter_scores_and_topk`, the standard `_bbox_post_process`. The harness classified it as -`kind fcos` on its own and `detect_fcos` was already right. - -`boxinst` followed for free, at 0.25 px — `BoxInstBboxHead` subclasses `CondInstBboxHead` and -overrides neither `forward_single` nor either `predict_by_feat`, so it runs the decoder above -verbatim; only `num_params` changes, from 169 to 593. What had blocked it was not the family -but `BoxInstDataPreprocessor.__init__`, which raises unconditionally when `scikit-image` is -absent. The only use of `skimage` in that class is inside an `if training:` branch, computing -LAB colour similarity for the pseudo-masks that box-supervised *training* needs; inference -never reaches it. Installing the package was the whole fix — no code changed. A constructor -guard on a training-only dependency reads exactly like an unsupported architecture in a results -table, and the two deserve different columns. - -Five families produce no boxes at all and are measured differently. `solo` and `solov2` predict -masks by location, `maskformer` and `mask2former` classify mask embeddings, and -`mask2former_vis` does the same over video; none of them has a `bbox_head`, so a box comparison -has nothing to compare. What they do have is a compiled graph, and that is what is checked: the -harness compiles the backbone and neck as usual, runs `tools/verify/backbone/run_dump.cpp` — -which emits the graph's `out_*` tensors with no head and no decoding — and compares them against -torch at the same relative-L1 threshold the box families use. - -| Family | Head attribute | Worst rel L1 | -| :--- | :--- | ---: | -| `solov2` | `mask_head` | 6.30e-04 | -| `solo` | `mask_head` | 6.99e-04 | -| `mask2former_vis` | `track_head` | 1.87e-03 | -| `maskformer` | `panoptic_head` | 1.94e-03 | -| `mask2former` | `panoptic_head` | 2.19e-03 | - -Read that table for what it is: **the compiled portion agrees with torch**, not "the family runs -end to end". The mask heads were never ported to C++, and `maskformer`/`mask2former` declare no -neck at all, so for those two the compiled portion is the backbone alone. Recording this as -"masks verified" would be the same mistake as recording a harness limitation as a model -limitation. `mask2former_vis` is measured on a single frame; multi-frame tracking is untested, -not unsupported. - -These families were failing as `HEAD_NONE` before, which read like a defect and was not: their -heads are simply named `mask_head`, `panoptic_head` or `track_head` rather than `bbox_head`, and -the export produced a perfectly good `bb.pt` the whole time. A missing classification is not a -missing capability. - -Two-stage families are measured separately, at 800 and against the detector's own `predict` -rather than a head's `predict_by_feat`, because the boxes do not exist until RPN proposals, -RoIAlign and the RoI head have run. The harness is `tools/verify/roi/verify_postproc_roi.py` -and the thresholds are the same. Thirty-five of the forty-five families with a `roi_head` agree — -forty-five rather than forty because the harness now looks one layer inside wrapper detectors -(see below): - -| Family | Decoder | Worst box | Worst score | -| :--- | :--- | ---: | ---: | -| `detectors` | `detect_roi` (SAC) | 0.03 px | 0.0006 | -| `panoptic_fpn` | `detect_roi` | 0.04 px | 0.0001 | -| `qdtrack` | `detect_roi` (tracker wrapper) | 0.03 px | 0.0003 | -| `deepsort` | `detect_roi` (tracker wrapper) | 0.04 px | 0.0000 | -| `sort` | `detect_roi` (tracker wrapper) | 0.04 px | 0.0000 | -| `dcnv2` | `detect_roi` | 0.05 px | 0.0006 | -| `carafe` | `detect_roi` | 0.06 px | 0.0007 | -| `hrnet` | `detect_roi` | 0.06 px | 0.0002 | -| `mask_rcnn` | `detect_roi` | 0.06 px | 0.0009 | -| `crowddet` | `detect_roi` (set-NMS, 2 instances) | 0.07 px | 0.0001 | -| `ms_rcnn` | `detect_roi` (+ mask-IoU rescoring) | 0.07 px | 0.0015 | -| `gn+ws` | `detect_roi` | 0.08 px | 0.0008 | -| `masktrack_rcnn` | `detect_roi` (tracker wrapper) | 0.09 px | 0.0012 | -| `tridentnet` | `detect_roi` (C4, no neck) | 0.09 px | 0.0013 | -| `cascade_rcnn` | `detect_roi` (3 stages) | 0.09 px | 0.0025 | -| `gn` | `detect_roi` | 0.09 px | 0.0003 | -| `empirical_attention` | `detect_roi` | 0.09 px | 0.0003 | -| `seesaw_loss` | `detect_roi` (NormedLinear, custom activation) | 0.09 px | 0.0002 | -| `faster_rcnn` | `detect_roi` | 0.10 px | 0.0008 | -| `libra_rcnn` | `detect_roi` | 0.11 px | 0.0007 | -| `htc` | `detect_roi` (3 stages + semantic) | 0.12 px | 0.0011 | -| `resnest` | `detect_roi` | 0.12 px | 0.0002 | -| `gcnet` | `detect_roi` | 0.14 px | 0.0010 | -| `albu_example` | `detect_roi` | 0.14 px | 0.0008 | -| `grid_rcnn` | `detect_roi` (grid heatmap, no reg branch) | 0.15 px | 0.0007 | -| `point_rend` | `detect_roi` | 0.15 px | 0.0012 | -| `regnet` | `detect_roi` | 0.16 px | 0.0012 | -| `scnet` | `detect_roi` (+ global context) | 0.18 px | 0.0020 | -| `simple_copy_paste` | `detect_roi` | 0.19 px | 0.0007 | -| `swin` | `detect_roi` | 0.22 px | 0.0003 | -| `resnet_strikes_back` | `detect_roi` | 0.24 px | 0.0023 | -| `fpg` | `detect_roi` (at 1024) | 0.28 px | 0.0036 | -| `dcn` | `detect_roi` | 0.37 px | 0.0002 | -| `instaboost` | `detect_roi` | 0.39 px | 0.0079 | -| `soft_teacher` | `detect_roi` (semi-supervised wrapper) | 0.42 px | 0.0017 | - -`panoptic_fpn` is back in the table, and the round trip it took is the useful part. It was -recorded at 0.04 px, then removed when the harness started deleting each stage's outputs before -that stage ran: the old run's export had failed and a stale `frcnn.json` from an earlier run had -carried it through, so the number could no longer be trusted. It was marked unverified rather -than wrong. Installing `panopticapi` and re-measuring returns **0.04 px** — the original number -was right all along. "Unverified" and "wrong" are different claims, and only one of them -survived contact with the measurement. - -What blocked the re-measurement was the interpreter, not the environment, and the distinction -matters because two virtualenvs on this machine disagreed: the one the harness uses had a -working `import mmdet.models` but no `panopticapi`, while the other had `panopticapi` and could -not import `mmdet.models` at all — a stale `mmpretrain` install makes its -`reid_data_preprocessor` raise `TypeError` at class-definition time. The harness resolves child -processes through `sys.executable`, so which Python starts it decided the answer. Two sessions -measuring the same package reached opposite conclusions, each correct about its own interpreter. - -Checking this needs the failing call, not an import. `import panopticapi` and even -`import mmdet.datasets.coco_panoptic` both succeed without the package, because the check is -deferred to `LoadPanopticAnnotations.__init__` (`mmdet/datasets/transforms/loading.py:572`). -The discriminating command is -`python -c "from mmdet.datasets.transforms.loading import LoadPanopticAnnotations as L; L()"`. - -All eight **wrapper** families are now measured: seven trackers and one semi-supervised -detector. Trackers and semi-supervised -detectors are not detectors themselves: they put one inside `model.detector` and keep only their -own machinery at the top level, so reading `model.backbone` raises -`'ConfigDict' object has no attribute 'backbone'` and the export stops. Unwrapping the config -fixes the export, and the harness now also looks one layer inside when it decides which families -are two-stage — otherwise a family that passes is never swept again. - -Unwrapping the config is only half of it, and the other half fails silently. The checkpoint is -keyed by the **attribute name the wrapper class created**, which is not the config key: both -write `model.detector`, but `SoftTeacher` builds `self.student` and `self.teacher`, so its -weights are stored under `teacher.` / `student.` and the config decides which one inference uses -(`semi_test_cfg.predict_on`). Stripping `detector.` from such a checkpoint matches nothing, -`load_state_dict` reports it and carries on, and the graph is built on random weights — which -reads as a decode bug, not a loading bug: boxes 562 px out, 91 labels wrong, 100 detections -against MMDetection's 5. Check the count of unloaded tensors, not whether loading raised. -Trackers mostly use `detector.`, but not all of them: MMDetection ships some tracker weights as -the **detector alone**, already flat (`deepsort`, `sort` and `strongsort` are keyed -`backbone.` / `neck.` / `bbox_head.`). So a prefix that matches nothing means one of two things, -and they need opposite handling — already-flat, which should be used as-is, or a wrong guess, -which must stop. Every detector has a `backbone`, and that is the signature that separates them. - -The other half of a wrapper is its `data_preprocessor`, and it is easy to drop. Six of the seven -trackers declare it on the **wrapper** and give the inner detector none (`soft_teacher` is the -exception, which is why it worked first). Unwrap without carrying it down and normalisation -disappears — mean 0, std 1 — and the detector returns nothing at all. The reference side reads -the same unwrapped config, so both sides return nothing; the harness reports `EMPTY` rather than -a pass, so the failure is visible, but the family cannot be measured until the preprocessor comes -down with it. The wrapper's preprocessor is a `TrackDataPreprocessor`, which expects video-shaped -batches, so it is rewritten to `DetDataPreprocessor` on the way down — same numbers, no trap for -whoever opens the dumped config with `inference_detector`. - -Trackers are trained on pedestrians with `num_classes=1`, so the harness picks a test image per -family (`mmdet_families.test_image`); the cat photo the other families use yields no detections -on either side, which reports as `EMPTY`. The chosen image is printed beside the numbers -whenever it differs from the default, so a zero can be told apart from a wrong photo later. - -`fpg` is measured at 1024 rather than 800, and the reason is worth stating: it builds levels -below P5, where 800 stops dividing evenly — a 25-wide map meets a 26-wide one and the export -aborts. That is a property of the resolution, not of the family. - -The ten that do not agree split four ways, and the split matters more than the count: - -- **The RPN is not a standard anchor RPN**, so host `rpn_proposals` cannot lay down the priors: - `cascade_rpn` refines across stages, `guided_anchoring` predicts anchor shapes, and - `queryinst`/`sparse_rcnn` learn proposals outright. `groie` is refused for the neighbouring - reason — `GenericRoIExtractor` aggregates every level through per-level convolutions, which - host RoIAlign cannot express. All five stop at export with a stated reason rather than a - wrong number. -- **FP16 weights, not a defect.** `dynamic_rcnn` (10.24 px), `pafpn` (8.77 px) and `res2net` - (6.81 px) return the right count and the right labels with the coordinates several pixels - out. Recompiling with fp32 weights makes all three exact at **0.00 px**, so the gap is the - half-precision the compiler deliberately stores — the NPU this targets is FP16-native. - Isolating it took swapping one tensor at a time: substituting the C++ RPN *class* scores - into torch changed nothing, while substituting the *box deltas* reproduced the full 10.25 px - from a maximum delta error of 0.0023. Do not replace these numbers with their fp32 twins; - they are what the deployment precision produces. -- **Neither the graph nor the decoder is at fault.** `double_heads` agrees at 0.16 px and - differs by exactly one box: mmdet scores it 0.2954 and the compiled graph 0.3010, on either - side of the 0.30 cut the harness itself applies. That is the fp16 score error landing on a - threshold, not a decode difference, so the harness now prints how many boxes sit in the - 0.30–0.35 band beside every count mismatch. `fast_rcnn` is not measured at all — its - metafile lists no weights, and random initialisation cannot judge a decoder, so it is - recorded as untried rather than failing. -- **The family post-processed its own way.** This group is now empty, and each of the three - is worth keeping because none of them was visible in the tensors. `ms_rcnn` multiplies every - score by a predicted mask IoU, which needs the whole mask branch — RoIAlign at 14, the mask - head, then a second head over the features concatenated with the chosen mask channel. - `seesaw_loss` is two things at once: a `NormedLinear` classifier (the weight normalisation is - constant at inference and folds away; the input normalisation stays) and a custom activation - over `num_classes + 2` channels, so reading the class count as "output size minus one" shifts - every label by one. `crowddet` needed set-NMS — boxes from the same proposal do not suppress - each other — but three of its four differences were RPN settings that are not the defaults: - fixed anchor `centers`, `clip_border=False`, and an objectness head with two channels rather - than one because its `loss_cls` omits `use_sigmoid`. Its RPN tensors matched at 3e-04 from - the start, which is what said the problem was in the host code and not in the graph. -- **An operator was missing, approximated, or silently reduced along the wrong axis.** This - group is now empty, and how each was found is worth keeping. `carafe` rendered - `pixel_shuffle` as a pass-through identity, skipping CARAFE's upsampling outright - (29 px → 0.06 px). `libra_rcnn` approximated with a fixed kernel the non-integer - `adaptive_max_pool2d` that BFP uses to scatter back to P6 (12 px → 0.11 px). Both announced - themselves as `TODO` comments in the generated `.cpp`, so grep for those before reading - anything else. - - `gcnet` (37 px → 0.14 px) had no such marker. `ContextBlock` uses - `nn.LayerNorm([planes, 1, 1])` — three normalised axes — but the renderer always emitted - `ggml_norm`, which reduces `ne0` alone. At that point the tensor is `ne [1, 1, C, N]`, so - `ne0` is 1: normalising a single element gives `x - mean(x) = 0`, and the whole channel - branch collapses to a constant bias. A renderer that cannot express an operation still emits - shape-correct code, which passes compilation and every shape assertion while returning wrong - values. - - The three cascade families joined them later, and each was a different missing piece rather - than a shared cascade bug. `htc` (94 px → 0.12 px) feeds a semantic segmentation branch back - into every RoI: RoIAlign at stride 8 onto a 14×14 grid, average-pooled to 7×7 and added to - the box features **at every stage**, not only the first. `scnet` (29 px → 0.18 px) adds a - global-context vector to all RoI positions the same way. `detectors` (30 px → 0.03 px) was - not a fusion at all — its switchable atrous convolution ran at one dilation. `swin` - (0.22 px) failed earlier still, at load: `GGML_MAX_NAME` is 64 and its tensor names are - longer, and separately the shifted-window attention mask is built by slice assignment that - tracing drops, which silently zeroes the mask instead of crashing. The length is resolved - by **shortening the names, not by patching ggml** — a patched submodule commit lives on no - remote we can push to, so it would break every fresh clone. What made this one hard to see - is that the shortener already existed and still let the name through: it folds the module - prefix against a budget that assumes a short suffix (`running_mean`, twelve characters), - and `relative_position_bias_table` is twenty-eight. The prefix here is short enough to pass - untouched, and nothing checked the finished length, so the bake wrote a GGUF that could not - be loaded and said so only much later, as one line from the runner. Counting the longest - weight name is not the same as checking what the shortener does with it. - -Nothing is left unsorted. `tridentnet` used to return no boxes at all, and it took two -different C4-only faults to explain that. Its RPN puts five scales on **one** level -(`scales=[2,4,8,16,32]`, stride 16) where an FPN RPN puts one scale on each of five, so -reading only `scales[0]` built three anchors instead of fifteen and then read a -fifteen-channel objectness map as if it had three — every proposal landed somewhere else. -Fixing that moved the crash rather than removing it: with one level, NMS left 107 proposals -where the RoI graph had been compiled for 1000, and the `flatten` inside it bakes the row -count into a reshape. Proposals are now padded up to the cap and only the real rows are -decoded. A first fix that does not make the symptom go away usually means a second cause, -not a wrong first fix. - -`rpn` decodes proposals rather than detections, so a worst case over the whole set says little: -of 185 proposals the median is 0.09 px and 182 are within 1 px, with one proposal of 186 -falling on the other side of the score cut. `free_anchor` agrees to 0.35 px and 0.025 with one -box likewise on the boundary. - -Three groups do not decode, and they fail for different reasons: - -- **Something before the decoder already disagrees with torch.** This group is now empty, and - emptying it took no new code — the three families recorded here (`tood`, `deformable_detr`, - `dyhead`) were re-measured after the compiler fixes landed, and two of them simply passed: - `tood` at 0.63 px and `deformable_detr` at 0.26 px. The note that `tood`'s "box branch blows - up" and that `dyhead`'s neck sat at 0.7 relative L1 described a tree that no longer exists. - Re-running a recorded failure after unrelated fixes is cheaper than reading it. - - `dyhead` closed too, and where it hid is worth keeping. Its neck agreed at 2e-03 relative L1 - and its head at 1e-04, yet the boxes were 112 px out with the score identical to four digits — - the right cell won and was placed at the wrong pixel. The cause was in anchor generation: - `AnchorGenerator` keeps `base_sizes` equal to the strides and puts `octave_base_scale` into - `scales`, but the host folded the two together into `base_size = stride * octave_base_scale`. - Anchor *sizes* come out the same either way, which is why no shape check ever complained; the - *centre* does not, because it is `center_offset * base_size`. At stride 32 with - `octave_base_scale` 8 that is 128 against MMDetection's 16 — exactly the 112 px observed. - Families with `center_offset` 0 (retinanet, atss, gfl, …) are immune, since both readings give - zero, so a single family carried the defect for the whole anchor decoder. Only `dyhead` and - `glip` set it non-zero. Now 0.21 px, with the anchor families re-measured unchanged. - - `MMDET_DUMP_HEAD` writes the neck output beside the head output for exactly this split — a - family whose `feat` dumps match and whose `cls`/`box` dumps do not is a head problem, and the - reverse is a compiler problem. When both match, as here, what is left is the decoder. - -- **Beware the pairing when you re-measure by hand.** `tood` first re-measured at 13.89 px with - a count mismatch, which looked like a live defect and was not: `verify_heads.py` exports from - its hand-written override list (`tood_r50_fpn_1x_coco.py`, the anchor-free variant) while the - config that `mmdet_families.resolve()` returns is the anchor-based one. Compiling from one and - judging against the other compares a graph with someone else's weights. Take the pair from the - harness, never from the config directory. -- **The family post-processes its own way.** This group is down to `yolact`, and it is a - boundary case rather than a decode failure: fast NMS is implemented and the boxes agree to - 0.74 px, but one box sits either side of the harness's own 0.30 cut (mmdet 0.3013, compiled - 0.2951). It is counted with `free_anchor` and `double_heads`, not with the failures. -- **The priors or the coder are outside `det_params`.** This group is now empty. `ssd`, - `fsaf`, `paa`, `lad`, `cornernet` and `centernet` all decode, and `yolov3` did earlier. - `centripetalnet` decodes too but lands at 3.80 px on one of two boxes, where a single - top-k rank flips between two nearly-equal heatmap peaks; running mmdet's own - `_decode_heatmap` on our tensors returns the same answer (0.4387 against 0.4393), so the - formula is equivalent and what remains is half-precision, not the decoder. - -A family in the first group is not silently wrong: without anchor parameters `detect_anchor` -generates no candidates and the runner reports zero boxes. +Every family that has been verified, the decoder each one uses, and the measured error are in +the verification report in the compiler repository, +[`docs/verification-report-en.md`](https://github.com/Sudo42b/GTX_Compiler/blob/main/docs/verification-report-en.md). +It also records what did not verify and why. + +Two measures appear there and they answer different questions. **Box** — coordinates within +2 px of MMDetection's own output, scores within 0.05, no label or count mismatch. **Tensor** — +whether the compiled graph reproduces the head's output *before* decoding, at relative L1/L2 +under 5e-02. Neither is a correction of the other, and they must not be put in the same table. + +If you only need to know whether your family is covered, the list in the report is the answer. +If you need to know how far to trust the number, read the method section above it. ## Detection heads @@ -686,7 +329,8 @@ expect. They are declared in `tools/detect/head.h`. ### `anchor_head_forward` Shared convolution tower followed by classification and regression convolutions — the layout -used by RetinaNet, ATSS, GFL and other anchor-based dense heads. All levels share one set of +used by RetinaNet, ATSS, GFL and other anchor-based **dense** heads — dense because they +predict at every cell of every level, with no proposal step to narrow the field first. All levels share one set of weights. ```c++ @@ -777,12 +421,14 @@ struct detection { thresholding and NMS. Used by RetinaNet, ATSS, PAA and other delta-coded heads. `score_factors` is the optional centerness/IoU branch. MMDetection thresholds and takes top-k on the class score **alone** and multiplies the factor in afterwards, so passing it - here rather than folding it into `cls_scores` is what keeps the surviving set the same. + here rather than folding it into `cls_scores` keeps the surviving set the same. `std::vector detect_fcos(cls_scores, bbox_preds, centerness, feat_hw, fcos_params const& p)` : Anchor-free distance decoding. `centerness` may be empty — GFL and VFNet fold quality into the class score and have no such branch. `bbox_preds` are already pixel distances: the head - component applies the DFL integral, the stride multiply and the exponent, so this function + component applies the DFL integral — DFL is Distribution Focal Loss, which predicts each + box edge as a distribution over bins and recovers the distance by integrating it — plus the + stride multiply and the exponent, so this function must not apply a stride again. `point_offset` is 0.5 for FCOS and 0 for the heads built on an `AnchorGenerator`. diff --git a/docs/model-implementation-guide.md b/docs/model-implementation-guide.md index 7752ff3..070fc99 100644 --- a/docs/model-implementation-guide.md +++ b/docs/model-implementation-guide.md @@ -13,15 +13,15 @@ vision.cpp and ggml. 1. Inspect the model architecture and weights 2. Write a script that converts the weights to GGUF format -4. Implement the compute graph +3. Implement the compute graph * Copy a module/layer from the reference into a Python test file * Implement the `forward` function in C++ and expose it * Run the reference and C++ implementation on dummy data from Python and compare * Repeat until everything is implemented (and tested) -5. Implement pre-/post-processing steps in C++ -6. Add the model to the CLI -7. Add the model to the API -8. Add the model to `test-models` +4. Implement pre-/post-processing steps in C++ +5. Add the model to the CLI +6. Add the model to the API +7. Add the model to `test-models` This might sound like a lot, but most of the steps are pretty straight-forward. The process has a pretty good chance to result in something that works at the @@ -57,7 +57,7 @@ functionality is missing, you can quickly hack it in. Make sure to use that. _vision.cpp_ adds some infrastructure on top of ggml to reduce boilerplate for common tasks. It's designed to amend functionality, not wrap or replace it. The -[include/visp/ml.h](/include/visp/ml.h) public header contains all the +[include/visp/ml.h](../include/visp/ml.h) public header contains all the interesting bits. If you take a look at the existing model implementations in `src/visp/arch`, you @@ -120,7 +120,27 @@ for name, tensor in model.state_dict().items(): writer.add_tensor(name, tensor) ``` -You might have to shorten weight names to fit the 64-characters limit. +Weight names have to fit in 64 characters, NUL included, so 63 is the real budget. Deeply +nested modules go over it — `backbone.stages.0.blocks.0.attn.w_msa.relative_position_bias_table` +is 66 — and the file is then **refused at load**: + +``` +gguf_init_from_file_ptr: tensor name 53 is too long: 66 >= 64 +``` + +Shorten the module prefix, not the suffix; `.weight` and `.bias` are what tells the loader +which kind of tensor it is. Two things make this easy to get wrong: + +- **Counting the longest weight name is not the same as checking the finished one.** A short + prefix with a long suffix still overflows, and a shortener that budgets only for the prefix + will pass it straight through. +- **Check the length you actually wrote.** Nothing between the shortener and the loader looks + at it, so a file that cannot be opened is written in silence and the failure surfaces much + later, as one line from the runner. + +Generated models already handle this: `shared/compile/tensor_names.py` in the compiler folds +the prefix, and the code generator calls the same function, so both sides arrive at the same +name without passing a table between them. ### 3. The Compute Graph @@ -183,7 +203,7 @@ uv run pytest tests/test_piong.py ### C++ Implementation I'd usually put something as basic as layer-norm in -[src/visp/nn.cpp](/src/visp/nn.cpp). And in fact, it's already there. But for +[src/visp/nn.cpp](../src/visp/nn.cpp). And in fact, it's already there. But for this example, let's pretend it's a more model-specific operation, and put it into a new file [src/visp/arch/piong.cpp](). @@ -206,7 +226,7 @@ by printing the state-dict in the test. To make the test work, we're missing some glue. It's tempting to export and use the function directly, but having a separate "invoker" function has proven to be -more flexible. So I go to [tests/workbench.cpp](/tests/workbench.cpp) and add a +more flexible. So I go to [tests/workbench.cpp](../tests/workbench.cpp) and add a little bit of boilerplate: ```c++ @@ -291,14 +311,14 @@ Some examples where this helped: scaling/changing the distribution to include negative numbers. 3. **Complexity** - For the most top-level operation it's sometimes not practical to come up with dummy input, especially if it includes downsample/upsample steps - which require large input tensors. So just skip the test and hope for the - best :) + which require large input tensors. Skip the test for that one and rely on the + end-to-end comparison instead. ## 4. Pre- and Postprocessing It's common for vision models to process images and masks with a wild mix of PIL/numpy/OpenCV/torchvision/whatever. The -[include/visp/image.h](/include/visp/image.h) header has a collection of +[include/visp/image.h](../include/visp/image.h) header has a collection of common transformations. If that doesn't cover it, it also has some tools to implement custom per-pixel operations. @@ -340,17 +360,17 @@ invoked like this: ```sh vision-cli -m -i [...] -o ``` -Adding a new model arch in [src/cli/cli.cpp](/src/cli/cli.cpp) is pretty +Adding a new model arch in [src/cli/cli.cpp](../src/cli/cli.cpp) is pretty straight-forward by following one of the existing implementations. It usually includes some practical post-processing too. ## 6. API -Models are exported in [include/visp/vision.h](/include/visp/vision.h). This +Models are exported in [include/visp/vision.h](../include/visp/vision.h). This includes a high-level API which represents the most common use cases. It should be simple, and does not need to support configuration options. Typically that means a function to load the model, and one to run inference. These are -implemented in [src/visp/vision.cpp](/src/visp/vision.cpp). +implemented in [src/visp/vision.cpp](../src/visp/vision.cpp). Below there is space for a more modular API, which directly exports the functions specific to the model: parameter detection, pre-/post processing, and @@ -360,22 +380,20 @@ graph building. Finally, it is good to have a test that actually runs the entire model on some sensible input (an image!) and spits out something nice to look at and go "yep, -it works". This is what [tests/test-models.cpp](/tests/test-models.cpp) is for. +it works". This is what [tests/test-models.cpp](../tests/test-models.cpp) is for. With all the previous work, those tests are really simple to implement: load an image, call the high level API, compare the result to a reference and store it. -_Note on reference images:_ Those aren't checked into the repository to avoid -bloat. GitHub's LFS support is kinda bullshit, so it currently involves me -invoking a script to upload new images. If you're making a PR, just leave them -out. +_Note on reference images:_ Those aren't checked into the repository, to keep it +small. Adding new ones is a manual step on the maintainer's side, so leave them out +of a pull request. -## Afterword +## A note on the Python tests -This is my process, it works for me. I don't expect anyone to follow it by -heart. All contributions are welcome as long as the results are good! +They are a means, not a deliverable. Their purpose is to make the implementation +faster and the bugs easier to find; if they ever cost more to maintain than they +save, they may be dropped. Treat them as scaffolding while you work, not as +something a contribution has to preserve. -I don't actually value all the Python tests much as an end result, and may -decide to scrap them if they increase maintenance burden. For now they're -included in the repository, but their main purpose is to make the implementation -faster and finding bugs less painful. And increasing the chance that things just -work! +The steps above are one way through, not the only one. A contribution is judged on +the result. diff --git a/docs/overview.md b/docs/overview.md index 7ac63f8..b548c7f 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -23,7 +23,7 @@ vision.cpp splits the model in two: | Weights | GGUF tensors | a `.gguf` file loaded at run time | Nothing interprets a graph description at run time, because there is no graph description — the -graph is the code you compiled. That is what makes the deployment small and start-up fast, and +graph is the code you compiled. That keeps the deployment small and start-up fast, and it is the trade-off at the centre of the project: adding a model that isn't supported yet means writing or generating code, not exporting a file. diff --git a/docs/using-the-cli.md b/docs/using-the-cli.md index 1b7fa8d..a5b023f 100644 --- a/docs/using-the-cli.md +++ b/docs/using-the-cli.md @@ -24,13 +24,18 @@ The command selects the model, `-m` says which weights to load, `-i` and `-o` ar ## Options `-m, --model ` -: The `.gguf` weights. Required. +: The `.gguf` weights. Omit it and each command looks for its own default name — + `MobileSAM-F16.gguf`, `BiRefNet-lite-F16.gguf`, and so on — under `models/`, + `$VISION_MODEL_DIR`, `$XDG_DATA_HOME/visioncpp`, `~/.local/share/visioncpp` and the + install directory, in that order. `-i, --input [ ...]` : Input image. `migan` takes two — the image and the mask. `-o, --output ` -: Output file. Defaults to `output.png`. +: Output file. Defaults to `output.png`. Images are always written as **PNG**, whatever + the name says — `-o out.jpg` produces a PNG file called `out.jpg`, which some viewers + refuse to open. Give it a `.png` name. `-p, --prompt [ ...]` : Prompt for models that take one. `sam` accepts a point (`x y`) or a box @@ -140,6 +145,10 @@ uv run scripts/convert.py MyModel.pth `` is one of `sam`, `sam3`, `birefnet`, `depth-anything`, `migan`, `esrgan`. The result lands in `models/`. +Two of those names do not carry over to the command line unchanged. `depth-anything` here is +`depthany` there — same model, two spellings. And `sam3` converts but has no `vision-cli` +subcommand yet, so the GGUF it writes can only be reached from the library API. + | Option | Description | | :--- | :--- | | `-o, --output` | Output directory or file. Default `models`. | diff --git a/docs/using-the-library.md b/docs/using-the-library.md index 422c413..3446dbe 100644 --- a/docs/using-the-library.md +++ b/docs/using-the-library.md @@ -85,20 +85,45 @@ The one-call functions above are compositions. Each model also exposes the steps parameter detection, pre-processing, graph construction, post-processing. ```c++ -birefnet_params p = birefnet_detect_params(file); // read shape/variant from the GGUF -image_data in = birefnet_process_input(image, p); // resize, normalise -tensor out = birefnet_predict(m, input_tensor, p); // build the graph -image_data mask = birefnet_process_output(data, target_extent, p); +// once — load the weights onto a device +model_file file = model_load("BiRefNet-lite-F16.gguf"); +birefnet_params p = birefnet_detect_params(file, {1024, 1024}); +model_weights w = model_init(file.n_tensors()); +model_transfer(file, w, dev, dev.preferred_float_type(), dev.preferred_layout()); + +// once per graph — build it and allocate +compute_graph graph = compute_graph_init(6 * 1024); +model_ref m(w, graph); +birefnet_buffers bufs = birefnet_precompute(m, p); +tensor input = compute_graph_input(m, GGML_TYPE_F32, {3, p.image_extent[0], p.image_extent[1], 1}); +tensor output = birefnet_predict(m, input, p); +compute_graph_allocate(graph, dev); +for (tensor_data const& buf : bufs) transfer_to_backend(buf); + +// per image — the only part that repeats +image_data prepared = birefnet_process_input(image, p); +transfer_to_backend(input, prepared); +compute(graph, dev); +tensor_data result = transfer_from_backend(output); +image_data mask = birefnet_process_output(result.as_f32(), image.extent, p); ``` -Use these when you need to batch work, keep tensors on the device between stages, run +The three blocks are why the split exists: the graph is built once and reused, so only the +last block runs per frame. Note where the types change — `birefnet_predict` takes a `tensor` +that lives on the device, not the `image_data` that came out of `process_input`, and +`birefnet_process_output` reads back a plain `span`. `transfer_to_backend` and +`transfer_from_backend` are what cross that line. + +Reach for this when you need to batch work, keep tensors on the device between stages, run pre-processing somewhere else, or share a compute graph across calls. `visp/ml.h` has the pieces underneath — `model_load`, `model_transfer`, `compute_graph_init`, `compute`. ## Detection post-processing If you are building a detector rather than using a built-in model, `visp/postproc.h` has the -parts that are not neural networks: anchor generation, box decoding, NMS, RoIAlign, mask +parts that are not neural networks: anchor generation, box decoding, NMS (non-maximum +suppression — drop the lower-scoring of two overlapping boxes), RoIAlign (crop a fixed-size +feature patch for one region of interest), mask pasting. `visp/tracker.h` has ByteTrack for keeping identities across frames. Both are plain CPU code and take structs, not framework config. diff --git a/src/visp/arch/Yolo26m.cpp b/src/visp/arch/Yolo26m.cpp new file mode 100644 index 0000000..15f7229 --- /dev/null +++ b/src/visp/arch/Yolo26m.cpp @@ -0,0 +1,436 @@ +// GENERATED BY SuperGate GTX Compiler (ggml/vision.cpp backend), DO NOT EDIT! +#include "visp/arch/Yolo26m.h" +#include "visp/ml.h" +#include "visp/nn.h" +#include "visp/vision.h" +#include "util/string.h" +#include + +#include + +namespace visp { + +tensor Yolo26m_forward(model_ref m, tensor x, Yolo26m_params const& p) { + (void)p; + // 입력 레이아웃 변환 (TODO: cwhn/whcn 자동 판별) + x = cwhn_to_contiguous_2d(m, x); + + tensor const1 = ggml_arange(m, 0.5f, 1.0f, 1.0f) /* const 0.5 */; + tensor const3 = ggml_arange(m, 0.5f, 1.0f, 1.0f) /* const 0.5 */; + tensor const5 = ggml_arange(m, 0.5f, 1.0f, 1.0f) /* const 0.5 */; + tensor const7 = ggml_arange(m, 0.1767766922712326f, 0.6767766922712326f, 1.0f) /* const 0.1767766922712326 */; + tensor const8 = ggml_arange(m, 0.1767766922712326f, 0.6767766922712326f, 1.0f) /* const 0.1767766922712326 */; + tensor conv9 = conv_2d(m["model.0.conv"], x, 2, 1); + tensor act10 = ggml_silu_inplace(m, conv9); + tensor conv11 = conv_2d(m["model.1.conv"], act10, 2, 1); + tensor act12 = ggml_silu_inplace(m, conv11); + tensor conv13 = conv_2d(m["model.2.cv1.conv"], act12, 1, 0); + tensor act14 = ggml_silu_inplace(m, conv13); + tensor sl15 = ggml_view_4d(m, act14, 160, 160, 64, 1, act14->nb[1], act14->nb[2], act14->nb[3], 0); + tensor sl16 = ggml_view_4d(m, act14, 160, 160, 64, 1, act14->nb[1], act14->nb[2], act14->nb[3], 64*act14->nb[2]); + tensor conv17 = conv_2d(m["model.2.m.0.cv1.conv"], sl16, 1, 0); + tensor act18 = ggml_silu_inplace(m, conv17); + tensor conv19 = conv_2d(m["model.2.m.0.m.0.cv1.conv"], act18, 1, 1); + tensor act20 = ggml_silu_inplace(m, conv19); + tensor conv21 = conv_2d(m["model.2.m.0.m.0.cv2.conv"], act20, 1, 1); + tensor act22 = ggml_silu_inplace(m, conv21); + tensor add23 = ggml_add(m, act18, act22); + tensor conv24 = conv_2d(m["model.2.m.0.m.1.cv1.conv"], add23, 1, 1); + tensor act25 = ggml_silu_inplace(m, conv24); + tensor conv26 = conv_2d(m["model.2.m.0.m.1.cv2.conv"], act25, 1, 1); + tensor act27 = ggml_silu_inplace(m, conv26); + tensor add28 = ggml_add(m, add23, act27); + tensor conv29 = conv_2d(m["model.2.m.0.cv2.conv"], sl16, 1, 0); + tensor act30 = ggml_silu_inplace(m, conv29); + tensor cat31 = ggml_concat(m, add28, act30, 2); + tensor conv32 = conv_2d(m["model.2.m.0.cv3.conv"], cat31, 1, 0); + tensor act33 = ggml_silu_inplace(m, conv32); + tensor cat34 = ggml_concat(m, ggml_concat(m, sl15, sl16, 2), act33, 2); + tensor conv35 = conv_2d(m["model.2.cv2.conv"], cat34, 1, 0); + tensor act36 = ggml_silu_inplace(m, conv35); + tensor conv37 = conv_2d(m["model.3.conv"], act36, 2, 1); + tensor act38 = ggml_silu_inplace(m, conv37); + tensor conv39 = conv_2d(m["model.4.cv1.conv"], act38, 1, 0); + tensor act40 = ggml_silu_inplace(m, conv39); + tensor sl41 = ggml_view_4d(m, act40, 80, 80, 128, 1, act40->nb[1], act40->nb[2], act40->nb[3], 0); + tensor sl42 = ggml_view_4d(m, act40, 80, 80, 128, 1, act40->nb[1], act40->nb[2], act40->nb[3], 128*act40->nb[2]); + tensor conv43 = conv_2d(m["model.4.m.0.cv1.conv"], sl42, 1, 0); + tensor act44 = ggml_silu_inplace(m, conv43); + tensor conv45 = conv_2d(m["model.4.m.0.m.0.cv1.conv"], act44, 1, 1); + tensor act46 = ggml_silu_inplace(m, conv45); + tensor conv47 = conv_2d(m["model.4.m.0.m.0.cv2.conv"], act46, 1, 1); + tensor act48 = ggml_silu_inplace(m, conv47); + tensor add49 = ggml_add(m, act44, act48); + tensor conv50 = conv_2d(m["model.4.m.0.m.1.cv1.conv"], add49, 1, 1); + tensor act51 = ggml_silu_inplace(m, conv50); + tensor conv52 = conv_2d(m["model.4.m.0.m.1.cv2.conv"], act51, 1, 1); + tensor act53 = ggml_silu_inplace(m, conv52); + tensor add54 = ggml_add(m, add49, act53); + tensor conv55 = conv_2d(m["model.4.m.0.cv2.conv"], sl42, 1, 0); + tensor act56 = ggml_silu_inplace(m, conv55); + tensor cat57 = ggml_concat(m, add54, act56, 2); + tensor conv58 = conv_2d(m["model.4.m.0.cv3.conv"], cat57, 1, 0); + tensor act59 = ggml_silu_inplace(m, conv58); + tensor cat60 = ggml_concat(m, ggml_concat(m, sl41, sl42, 2), act59, 2); + tensor conv61 = conv_2d(m["model.4.cv2.conv"], cat60, 1, 0); + tensor act62 = ggml_silu_inplace(m, conv61); + tensor conv63 = conv_2d(m["model.5.conv"], act62, 2, 1); + tensor act64 = ggml_silu_inplace(m, conv63); + tensor conv65 = conv_2d(m["model.6.cv1.conv"], act64, 1, 0); + tensor act66 = ggml_silu_inplace(m, conv65); + tensor sl67 = ggml_view_4d(m, act66, 40, 40, 256, 1, act66->nb[1], act66->nb[2], act66->nb[3], 0); + tensor sl68 = ggml_view_4d(m, act66, 40, 40, 256, 1, act66->nb[1], act66->nb[2], act66->nb[3], 256*act66->nb[2]); + tensor conv69 = conv_2d(m["model.6.m.0.cv1.conv"], sl68, 1, 0); + tensor act70 = ggml_silu_inplace(m, conv69); + tensor conv71 = conv_2d(m["model.6.m.0.m.0.cv1.conv"], act70, 1, 1); + tensor act72 = ggml_silu_inplace(m, conv71); + tensor conv73 = conv_2d(m["model.6.m.0.m.0.cv2.conv"], act72, 1, 1); + tensor act74 = ggml_silu_inplace(m, conv73); + tensor add75 = ggml_add(m, act70, act74); + tensor conv76 = conv_2d(m["model.6.m.0.m.1.cv1.conv"], add75, 1, 1); + tensor act77 = ggml_silu_inplace(m, conv76); + tensor conv78 = conv_2d(m["model.6.m.0.m.1.cv2.conv"], act77, 1, 1); + tensor act79 = ggml_silu_inplace(m, conv78); + tensor add80 = ggml_add(m, add75, act79); + tensor conv81 = conv_2d(m["model.6.m.0.cv2.conv"], sl68, 1, 0); + tensor act82 = ggml_silu_inplace(m, conv81); + tensor cat83 = ggml_concat(m, add80, act82, 2); + tensor conv84 = conv_2d(m["model.6.m.0.cv3.conv"], cat83, 1, 0); + tensor act85 = ggml_silu_inplace(m, conv84); + tensor cat86 = ggml_concat(m, ggml_concat(m, sl67, sl68, 2), act85, 2); + tensor conv87 = conv_2d(m["model.6.cv2.conv"], cat86, 1, 0); + tensor act88 = ggml_silu_inplace(m, conv87); + tensor conv89 = conv_2d(m["model.7.conv"], act88, 2, 1); + tensor act90 = ggml_silu_inplace(m, conv89); + tensor conv91 = conv_2d(m["model.8.cv1.conv"], act90, 1, 0); + tensor act92 = ggml_silu_inplace(m, conv91); + tensor sl93 = ggml_view_4d(m, act92, 20, 20, 256, 1, act92->nb[1], act92->nb[2], act92->nb[3], 0); + tensor sl94 = ggml_view_4d(m, act92, 20, 20, 256, 1, act92->nb[1], act92->nb[2], act92->nb[3], 256*act92->nb[2]); + tensor conv95 = conv_2d(m["model.8.m.0.cv1.conv"], sl94, 1, 0); + tensor act96 = ggml_silu_inplace(m, conv95); + tensor conv97 = conv_2d(m["model.8.m.0.m.0.cv1.conv"], act96, 1, 1); + tensor act98 = ggml_silu_inplace(m, conv97); + tensor conv99 = conv_2d(m["model.8.m.0.m.0.cv2.conv"], act98, 1, 1); + tensor act100 = ggml_silu_inplace(m, conv99); + tensor add101 = ggml_add(m, act96, act100); + tensor conv102 = conv_2d(m["model.8.m.0.m.1.cv1.conv"], add101, 1, 1); + tensor act103 = ggml_silu_inplace(m, conv102); + tensor conv104 = conv_2d(m["model.8.m.0.m.1.cv2.conv"], act103, 1, 1); + tensor act105 = ggml_silu_inplace(m, conv104); + tensor add106 = ggml_add(m, add101, act105); + tensor conv107 = conv_2d(m["model.8.m.0.cv2.conv"], sl94, 1, 0); + tensor act108 = ggml_silu_inplace(m, conv107); + tensor cat109 = ggml_concat(m, add106, act108, 2); + tensor conv110 = conv_2d(m["model.8.m.0.cv3.conv"], cat109, 1, 0); + tensor act111 = ggml_silu_inplace(m, conv110); + tensor cat112 = ggml_concat(m, ggml_concat(m, sl93, sl94, 2), act111, 2); + tensor conv113 = conv_2d(m["model.8.cv2.conv"], cat112, 1, 0); + tensor act114 = ggml_silu_inplace(m, conv113); + tensor conv115 = conv_2d(m["model.9.cv1.conv"], act114, 1, 0); + tensor pool116 = ggml_pool_2d(m, conv115, GGML_OP_POOL_MAX, 5, 5, 1, 1, 2, 2); + tensor pool117 = ggml_pool_2d(m, pool116, GGML_OP_POOL_MAX, 5, 5, 1, 1, 2, 2); + tensor pool118 = ggml_pool_2d(m, pool117, GGML_OP_POOL_MAX, 5, 5, 1, 1, 2, 2); + tensor cat119 = ggml_concat(m, ggml_concat(m, ggml_concat(m, conv115, pool116, 2), pool117, 2), pool118, 2); + tensor conv120 = conv_2d(m["model.9.cv2.conv"], cat119, 1, 0); + tensor act121 = ggml_silu_inplace(m, conv120); + tensor add122 = ggml_add(m, act121, act114); + tensor conv123 = conv_2d(m["model.10.cv1.conv"], add122, 1, 0); + tensor act124 = ggml_silu_inplace(m, conv123); + tensor sl125 = ggml_view_4d(m, act124, 20, 20, 256, 1, act124->nb[1], act124->nb[2], act124->nb[3], 0); + tensor sl126 = ggml_view_4d(m, act124, 20, 20, 256, 1, act124->nb[1], act124->nb[2], act124->nb[3], 256*act124->nb[2]); + tensor conv128 = conv_2d(m["model.10.m.0.attn.qkv.conv"], sl126, 1, 0); + tensor rs129 = ggml_reshape_4d(m, ggml_cont(m, conv128), 400, 128, 4, 1); + tensor sl130 = ggml_view_4d(m, rs129, 400, 32, 4, 1, rs129->nb[1], rs129->nb[2], rs129->nb[3], 0); + tensor sl131 = ggml_view_4d(m, rs129, 400, 32, 4, 1, rs129->nb[1], rs129->nb[2], rs129->nb[3], 32*rs129->nb[1]); + tensor sl132 = ggml_view_4d(m, rs129, 400, 64, 4, 1, rs129->nb[1], rs129->nb[2], rs129->nb[3], 64*rs129->nb[1]); + tensor t133 = ggml_cont(m, ggml_permute(m, sl130, 1, 0, 2, 3)); + tensor mm134 = ggml_mul_mat(m, ggml_cont(m, ggml_permute(m, sl131, 1, 0, 2, 3)), t133); + tensor t135 = ggml_mul(m, mm134, const8); + tensor sm136 = ggml_soft_max(m, t135); + tensor t137 = ggml_cont(m, ggml_permute(m, sm136, 1, 0, 2, 3)); + tensor mm138 = ggml_mul_mat(m, ggml_cont(m, ggml_permute(m, t137, 1, 0, 2, 3)), sl132); + tensor rs139 = ggml_reshape_4d(m, ggml_cont(m, mm138), 20, 20, 256, 1); + tensor rs140 = ggml_reshape_4d(m, ggml_cont(m, sl132), 20, 20, 256, 1); + tensor dwconv141 = conv_2d_depthwise(m["model.10.m.0.attn.pe.conv"], rs140, 1, 1); + tensor add142 = ggml_add(m, rs139, dwconv141); + tensor conv143 = conv_2d(m["model.10.m.0.attn.proj.conv"], add142, 1, 0); + tensor add144 = ggml_add(m, sl126, conv143); + tensor conv145 = conv_2d(m["model.10.m.0.ffn.0.conv"], add144, 1, 0); + tensor act146 = ggml_silu_inplace(m, conv145); + tensor conv147 = conv_2d(m["model.10.m.0.ffn.1.conv"], act146, 1, 0); + tensor add148 = ggml_add(m, add144, conv147); + tensor cat149 = ggml_concat(m, sl125, add148, 2); + tensor conv150 = conv_2d(m["model.10.cv2.conv"], cat149, 1, 0); + tensor act151 = ggml_silu_inplace(m, conv150); + tensor up152 = ggml_interpolate(m, act151, 40, 40, 512, 1, GGML_SCALE_MODE_NEAREST) /* interpolate -> target ne */; + tensor cat153 = ggml_concat(m, up152, act88, 2); + tensor conv154 = conv_2d(m["model.13.cv1.conv"], cat153, 1, 0); + tensor act155 = ggml_silu_inplace(m, conv154); + tensor sl156 = ggml_view_4d(m, act155, 40, 40, 256, 1, act155->nb[1], act155->nb[2], act155->nb[3], 0); + tensor sl157 = ggml_view_4d(m, act155, 40, 40, 256, 1, act155->nb[1], act155->nb[2], act155->nb[3], 256*act155->nb[2]); + tensor conv158 = conv_2d(m["model.13.m.0.cv1.conv"], sl157, 1, 0); + tensor act159 = ggml_silu_inplace(m, conv158); + tensor conv160 = conv_2d(m["model.13.m.0.m.0.cv1.conv"], act159, 1, 1); + tensor act161 = ggml_silu_inplace(m, conv160); + tensor conv162 = conv_2d(m["model.13.m.0.m.0.cv2.conv"], act161, 1, 1); + tensor act163 = ggml_silu_inplace(m, conv162); + tensor add164 = ggml_add(m, act159, act163); + tensor conv165 = conv_2d(m["model.13.m.0.m.1.cv1.conv"], add164, 1, 1); + tensor act166 = ggml_silu_inplace(m, conv165); + tensor conv167 = conv_2d(m["model.13.m.0.m.1.cv2.conv"], act166, 1, 1); + tensor act168 = ggml_silu_inplace(m, conv167); + tensor add169 = ggml_add(m, add164, act168); + tensor conv170 = conv_2d(m["model.13.m.0.cv2.conv"], sl157, 1, 0); + tensor act171 = ggml_silu_inplace(m, conv170); + tensor cat172 = ggml_concat(m, add169, act171, 2); + tensor conv173 = conv_2d(m["model.13.m.0.cv3.conv"], cat172, 1, 0); + tensor act174 = ggml_silu_inplace(m, conv173); + tensor cat175 = ggml_concat(m, ggml_concat(m, sl156, sl157, 2), act174, 2); + tensor conv176 = conv_2d(m["model.13.cv2.conv"], cat175, 1, 0); + tensor act177 = ggml_silu_inplace(m, conv176); + tensor up178 = ggml_interpolate(m, act177, 80, 80, 512, 1, GGML_SCALE_MODE_NEAREST) /* interpolate -> target ne */; + tensor cat179 = ggml_concat(m, up178, act62, 2); + tensor conv180 = conv_2d(m["model.16.cv1.conv"], cat179, 1, 0); + tensor act181 = ggml_silu_inplace(m, conv180); + tensor sl182 = ggml_view_4d(m, act181, 80, 80, 128, 1, act181->nb[1], act181->nb[2], act181->nb[3], 0); + tensor sl183 = ggml_view_4d(m, act181, 80, 80, 128, 1, act181->nb[1], act181->nb[2], act181->nb[3], 128*act181->nb[2]); + tensor conv184 = conv_2d(m["model.16.m.0.cv1.conv"], sl183, 1, 0); + tensor act185 = ggml_silu_inplace(m, conv184); + tensor conv186 = conv_2d(m["model.16.m.0.m.0.cv1.conv"], act185, 1, 1); + tensor act187 = ggml_silu_inplace(m, conv186); + tensor conv188 = conv_2d(m["model.16.m.0.m.0.cv2.conv"], act187, 1, 1); + tensor act189 = ggml_silu_inplace(m, conv188); + tensor add190 = ggml_add(m, act185, act189); + tensor conv191 = conv_2d(m["model.16.m.0.m.1.cv1.conv"], add190, 1, 1); + tensor act192 = ggml_silu_inplace(m, conv191); + tensor conv193 = conv_2d(m["model.16.m.0.m.1.cv2.conv"], act192, 1, 1); + tensor act194 = ggml_silu_inplace(m, conv193); + tensor add195 = ggml_add(m, add190, act194); + tensor conv196 = conv_2d(m["model.16.m.0.cv2.conv"], sl183, 1, 0); + tensor act197 = ggml_silu_inplace(m, conv196); + tensor cat198 = ggml_concat(m, add195, act197, 2); + tensor conv199 = conv_2d(m["model.16.m.0.cv3.conv"], cat198, 1, 0); + tensor act200 = ggml_silu_inplace(m, conv199); + tensor cat201 = ggml_concat(m, ggml_concat(m, sl182, sl183, 2), act200, 2); + tensor conv202 = conv_2d(m["model.16.cv2.conv"], cat201, 1, 0); + tensor act203 = ggml_silu_inplace(m, conv202); + tensor conv204 = conv_2d(m["model.17.conv"], act203, 2, 1); + tensor act205 = ggml_silu_inplace(m, conv204); + tensor cat206 = ggml_concat(m, act205, act177, 2); + tensor conv207 = conv_2d(m["model.19.cv1.conv"], cat206, 1, 0); + tensor act208 = ggml_silu_inplace(m, conv207); + tensor sl209 = ggml_view_4d(m, act208, 40, 40, 256, 1, act208->nb[1], act208->nb[2], act208->nb[3], 0); + tensor sl210 = ggml_view_4d(m, act208, 40, 40, 256, 1, act208->nb[1], act208->nb[2], act208->nb[3], 256*act208->nb[2]); + tensor conv211 = conv_2d(m["model.19.m.0.cv1.conv"], sl210, 1, 0); + tensor act212 = ggml_silu_inplace(m, conv211); + tensor conv213 = conv_2d(m["model.19.m.0.m.0.cv1.conv"], act212, 1, 1); + tensor act214 = ggml_silu_inplace(m, conv213); + tensor conv215 = conv_2d(m["model.19.m.0.m.0.cv2.conv"], act214, 1, 1); + tensor act216 = ggml_silu_inplace(m, conv215); + tensor add217 = ggml_add(m, act212, act216); + tensor conv218 = conv_2d(m["model.19.m.0.m.1.cv1.conv"], add217, 1, 1); + tensor act219 = ggml_silu_inplace(m, conv218); + tensor conv220 = conv_2d(m["model.19.m.0.m.1.cv2.conv"], act219, 1, 1); + tensor act221 = ggml_silu_inplace(m, conv220); + tensor add222 = ggml_add(m, add217, act221); + tensor conv223 = conv_2d(m["model.19.m.0.cv2.conv"], sl210, 1, 0); + tensor act224 = ggml_silu_inplace(m, conv223); + tensor cat225 = ggml_concat(m, add222, act224, 2); + tensor conv226 = conv_2d(m["model.19.m.0.cv3.conv"], cat225, 1, 0); + tensor act227 = ggml_silu_inplace(m, conv226); + tensor cat228 = ggml_concat(m, ggml_concat(m, sl209, sl210, 2), act227, 2); + tensor conv229 = conv_2d(m["model.19.cv2.conv"], cat228, 1, 0); + tensor act230 = ggml_silu_inplace(m, conv229); + tensor conv231 = conv_2d(m["model.20.conv"], act230, 2, 1); + tensor act232 = ggml_silu_inplace(m, conv231); + tensor cat233 = ggml_concat(m, act232, act151, 2); + tensor conv234 = conv_2d(m["model.22.cv1.conv"], cat233, 1, 0); + tensor act235 = ggml_silu_inplace(m, conv234); + tensor sl236 = ggml_view_4d(m, act235, 20, 20, 256, 1, act235->nb[1], act235->nb[2], act235->nb[3], 0); + tensor sl237 = ggml_view_4d(m, act235, 20, 20, 256, 1, act235->nb[1], act235->nb[2], act235->nb[3], 256*act235->nb[2]); + tensor conv238 = conv_2d(m["model.22.m.0.0.cv1.conv"], sl237, 1, 1); + tensor act239 = ggml_silu_inplace(m, conv238); + tensor conv240 = conv_2d(m["model.22.m.0.0.cv2.conv"], act239, 1, 1); + tensor act241 = ggml_silu_inplace(m, conv240); + tensor add242 = ggml_add(m, sl237, act241); + tensor conv244 = conv_2d(m["model.22.m.0.1.attn.qkv.conv"], add242, 1, 0); + tensor rs245 = ggml_reshape_4d(m, ggml_cont(m, conv244), 400, 128, 4, 1); + tensor sl246 = ggml_view_4d(m, rs245, 400, 32, 4, 1, rs245->nb[1], rs245->nb[2], rs245->nb[3], 0); + tensor sl247 = ggml_view_4d(m, rs245, 400, 32, 4, 1, rs245->nb[1], rs245->nb[2], rs245->nb[3], 32*rs245->nb[1]); + tensor sl248 = ggml_view_4d(m, rs245, 400, 64, 4, 1, rs245->nb[1], rs245->nb[2], rs245->nb[3], 64*rs245->nb[1]); + tensor t249 = ggml_cont(m, ggml_permute(m, sl246, 1, 0, 2, 3)); + tensor mm250 = ggml_mul_mat(m, ggml_cont(m, ggml_permute(m, sl247, 1, 0, 2, 3)), t249); + tensor t251 = ggml_mul(m, mm250, const7); + tensor sm252 = ggml_soft_max(m, t251); + tensor t253 = ggml_cont(m, ggml_permute(m, sm252, 1, 0, 2, 3)); + tensor mm254 = ggml_mul_mat(m, ggml_cont(m, ggml_permute(m, t253, 1, 0, 2, 3)), sl248); + tensor rs255 = ggml_reshape_4d(m, ggml_cont(m, mm254), 20, 20, 256, 1); + tensor rs256 = ggml_reshape_4d(m, ggml_cont(m, sl248), 20, 20, 256, 1); + tensor dwconv257 = conv_2d_depthwise(m["model.22.m.0.1.attn.pe.conv"], rs256, 1, 1); + tensor add258 = ggml_add(m, rs255, dwconv257); + tensor conv259 = conv_2d(m["model.22.m.0.1.attn.proj.conv"], add258, 1, 0); + tensor add260 = ggml_add(m, add242, conv259); + tensor conv261 = conv_2d(m["model.22.m.0.1.ffn.0.conv"], add260, 1, 0); + tensor act262 = ggml_silu_inplace(m, conv261); + tensor conv263 = conv_2d(m["model.22.m.0.1.ffn.1.conv"], act262, 1, 0); + tensor add264 = ggml_add(m, add260, conv263); + tensor cat265 = ggml_concat(m, ggml_concat(m, sl236, sl237, 2), add264, 2); + tensor conv266 = conv_2d(m["model.22.cv2.conv"], cat265, 1, 0); + tensor act267 = ggml_silu_inplace(m, conv266); + tensor conv268 = conv_2d(m["model.23.cv2.0.0.conv"], act203, 1, 1); + tensor act269 = ggml_silu_inplace(m, conv268); + tensor conv270 = conv_2d(m["model.23.cv2.0.1.conv"], act269, 1, 1); + tensor act271 = ggml_silu_inplace(m, conv270); + tensor conv272 = conv_2d(m["model.23.cv2.0.2"], act271, 1, 0); + tensor rs273 = ggml_reshape_3d(m, ggml_cont(m, conv272), 6400, 4, 1); + tensor conv274 = conv_2d(m["model.23.cv2.1.0.conv"], act230, 1, 1); + tensor act275 = ggml_silu_inplace(m, conv274); + tensor conv276 = conv_2d(m["model.23.cv2.1.1.conv"], act275, 1, 1); + tensor act277 = ggml_silu_inplace(m, conv276); + tensor conv278 = conv_2d(m["model.23.cv2.1.2"], act277, 1, 0); + tensor rs279 = ggml_reshape_3d(m, ggml_cont(m, conv278), 1600, 4, 1); + tensor conv280 = conv_2d(m["model.23.cv2.2.0.conv"], act267, 1, 1); + tensor act281 = ggml_silu_inplace(m, conv280); + tensor conv282 = conv_2d(m["model.23.cv2.2.1.conv"], act281, 1, 1); + tensor act283 = ggml_silu_inplace(m, conv282); + tensor conv284 = conv_2d(m["model.23.cv2.2.2"], act283, 1, 0); + tensor rs285 = ggml_reshape_3d(m, ggml_cont(m, conv284), 400, 4, 1); + tensor cat286 = ggml_concat(m, ggml_concat(m, rs273, rs279, 0), rs285, 0); + tensor dwconv287 = conv_2d_depthwise(m["model.23.cv3.0.0.0.conv"], act203, 1, 1); + tensor act288 = ggml_silu_inplace(m, dwconv287); + tensor conv289 = conv_2d(m["model.23.cv3.0.0.1.conv"], act288, 1, 0); + tensor act290 = ggml_silu_inplace(m, conv289); + tensor dwconv291 = conv_2d_depthwise(m["model.23.cv3.0.1.0.conv"], act290, 1, 1); + tensor act292 = ggml_silu_inplace(m, dwconv291); + tensor conv293 = conv_2d(m["model.23.cv3.0.1.1.conv"], act292, 1, 0); + tensor act294 = ggml_silu_inplace(m, conv293); + tensor conv295 = conv_2d(m["model.23.cv3.0.2"], act294, 1, 0); + tensor rs296 = ggml_reshape_3d(m, ggml_cont(m, conv295), 6400, 80, 1); + tensor dwconv297 = conv_2d_depthwise(m["model.23.cv3.1.0.0.conv"], act230, 1, 1); + tensor act298 = ggml_silu_inplace(m, dwconv297); + tensor conv299 = conv_2d(m["model.23.cv3.1.0.1.conv"], act298, 1, 0); + tensor act300 = ggml_silu_inplace(m, conv299); + tensor dwconv301 = conv_2d_depthwise(m["model.23.cv3.1.1.0.conv"], act300, 1, 1); + tensor act302 = ggml_silu_inplace(m, dwconv301); + tensor conv303 = conv_2d(m["model.23.cv3.1.1.1.conv"], act302, 1, 0); + tensor act304 = ggml_silu_inplace(m, conv303); + tensor conv305 = conv_2d(m["model.23.cv3.1.2"], act304, 1, 0); + tensor rs306 = ggml_reshape_3d(m, ggml_cont(m, conv305), 1600, 80, 1); + tensor dwconv307 = conv_2d_depthwise(m["model.23.cv3.2.0.0.conv"], act267, 1, 1); + tensor act308 = ggml_silu_inplace(m, dwconv307); + tensor conv309 = conv_2d(m["model.23.cv3.2.0.1.conv"], act308, 1, 0); + tensor act310 = ggml_silu_inplace(m, conv309); + tensor dwconv311 = conv_2d_depthwise(m["model.23.cv3.2.1.0.conv"], act310, 1, 1); + tensor act312 = ggml_silu_inplace(m, dwconv311); + tensor conv313 = conv_2d(m["model.23.cv3.2.1.1.conv"], act312, 1, 0); + tensor act314 = ggml_silu_inplace(m, conv313); + tensor conv315 = conv_2d(m["model.23.cv3.2.2"], act314, 1, 0); + tensor rs316 = ggml_reshape_3d(m, ggml_cont(m, conv315), 400, 80, 1); + tensor cat317 = ggml_concat(m, ggml_concat(m, rs296, rs306, 0), rs316, 0); + tensor conv318 = conv_2d(m["model.23.one2one_cv2.0.0.conv"], act203, 1, 1); + tensor act319 = ggml_silu_inplace(m, conv318); + tensor conv320 = conv_2d(m["model.23.one2one_cv2.0.1.conv"], act319, 1, 1); + tensor act321 = ggml_silu_inplace(m, conv320); + tensor conv322 = conv_2d(m["model.23.one2one_cv2.0.2"], act321, 1, 0); + tensor rs323 = ggml_reshape_3d(m, ggml_cont(m, conv322), 6400, 4, 1); + tensor conv324 = conv_2d(m["model.23.one2one_cv2.1.0.conv"], act230, 1, 1); + tensor act325 = ggml_silu_inplace(m, conv324); + tensor conv326 = conv_2d(m["model.23.one2one_cv2.1.1.conv"], act325, 1, 1); + tensor act327 = ggml_silu_inplace(m, conv326); + tensor conv328 = conv_2d(m["model.23.one2one_cv2.1.2"], act327, 1, 0); + tensor rs329 = ggml_reshape_3d(m, ggml_cont(m, conv328), 1600, 4, 1); + tensor conv330 = conv_2d(m["model.23.one2one_cv2.2.0.conv"], act267, 1, 1); + tensor act331 = ggml_silu_inplace(m, conv330); + tensor conv332 = conv_2d(m["model.23.one2one_cv2.2.1.conv"], act331, 1, 1); + tensor act333 = ggml_silu_inplace(m, conv332); + tensor conv334 = conv_2d(m["model.23.one2one_cv2.2.2"], act333, 1, 0); + tensor rs335 = ggml_reshape_3d(m, ggml_cont(m, conv334), 400, 4, 1); + tensor cat336 = ggml_concat(m, ggml_concat(m, rs323, rs329, 0), rs335, 0); + tensor dwconv337 = conv_2d_depthwise(m["model.23.one2one_cv3.0.0.0.conv"], act203, 1, 1); + tensor act338 = ggml_silu_inplace(m, dwconv337); + tensor conv339 = conv_2d(m["model.23.one2one_cv3.0.0.1.conv"], act338, 1, 0); + tensor act340 = ggml_silu_inplace(m, conv339); + tensor dwconv341 = conv_2d_depthwise(m["model.23.one2one_cv3.0.1.0.conv"], act340, 1, 1); + tensor act342 = ggml_silu_inplace(m, dwconv341); + tensor conv343 = conv_2d(m["model.23.one2one_cv3.0.1.1.conv"], act342, 1, 0); + tensor act344 = ggml_silu_inplace(m, conv343); + tensor conv345 = conv_2d(m["model.23.one2one_cv3.0.2"], act344, 1, 0); + tensor rs346 = ggml_reshape_3d(m, ggml_cont(m, conv345), 6400, 80, 1); + tensor dwconv347 = conv_2d_depthwise(m["model.23.one2one_cv3.1.0.0.conv"], act230, 1, 1); + tensor act348 = ggml_silu_inplace(m, dwconv347); + tensor conv349 = conv_2d(m["model.23.one2one_cv3.1.0.1.conv"], act348, 1, 0); + tensor act350 = ggml_silu_inplace(m, conv349); + tensor dwconv351 = conv_2d_depthwise(m["model.23.one2one_cv3.1.1.0.conv"], act350, 1, 1); + tensor act352 = ggml_silu_inplace(m, dwconv351); + tensor conv353 = conv_2d(m["model.23.one2one_cv3.1.1.1.conv"], act352, 1, 0); + tensor act354 = ggml_silu_inplace(m, conv353); + tensor conv355 = conv_2d(m["model.23.one2one_cv3.1.2"], act354, 1, 0); + tensor rs356 = ggml_reshape_3d(m, ggml_cont(m, conv355), 1600, 80, 1); + tensor dwconv357 = conv_2d_depthwise(m["model.23.one2one_cv3.2.0.0.conv"], act267, 1, 1); + tensor act358 = ggml_silu_inplace(m, dwconv357); + tensor conv359 = conv_2d(m["model.23.one2one_cv3.2.0.1.conv"], act358, 1, 0); + tensor act360 = ggml_silu_inplace(m, conv359); + tensor dwconv361 = conv_2d_depthwise(m["model.23.one2one_cv3.2.1.0.conv"], act360, 1, 1); + tensor act362 = ggml_silu_inplace(m, dwconv361); + tensor conv363 = conv_2d(m["model.23.one2one_cv3.2.1.1.conv"], act362, 1, 0); + tensor act364 = ggml_silu_inplace(m, conv363); + tensor conv365 = conv_2d(m["model.23.one2one_cv3.2.2"], act364, 1, 0); + tensor rs366 = ggml_reshape_3d(m, ggml_cont(m, conv365), 400, 80, 1); + tensor cat367 = ggml_concat(m, ggml_concat(m, rs346, rs356, 0), rs366, 0); + tensor ar370 = ggml_arange(m, 0.0f, 80.0f, 1.0f); + tensor add371 = ggml_add(m, ar370, const5); + tensor mg372 = ggml_repeat(m, add371, ggml_new_tensor_2d(m, GGML_TYPE_F32, 80, 80)); + tensor stk373 = ggml_concat(m, ggml_reshape_3d(m, mg372, 1, 80, 80), ggml_reshape_3d(m, mg372, 1, 80, 80), 0); + tensor rs374 = ggml_reshape_2d(m, ggml_cont(m, stk373), 2, 6400); + tensor full376 = ggml_fill(m, ggml_new_tensor_2d(m, GGML_TYPE_F32, 1, 6400), 8.0f); + tensor ar379 = ggml_arange(m, 0.0f, 40.0f, 1.0f); + tensor add380 = ggml_add(m, ar379, const3); + tensor mg381 = ggml_repeat(m, add380, ggml_new_tensor_2d(m, GGML_TYPE_F32, 40, 40)); + tensor stk382 = ggml_concat(m, ggml_reshape_3d(m, mg381, 1, 40, 40), ggml_reshape_3d(m, mg381, 1, 40, 40), 0); + tensor rs383 = ggml_reshape_2d(m, ggml_cont(m, stk382), 2, 1600); + tensor full385 = ggml_fill(m, ggml_new_tensor_2d(m, GGML_TYPE_F32, 1, 1600), 16.0f); + tensor ar388 = ggml_arange(m, 0.0f, 20.0f, 1.0f); + tensor add389 = ggml_add(m, ar388, const1); + tensor mg390 = ggml_repeat(m, add389, ggml_new_tensor_2d(m, GGML_TYPE_F32, 20, 20)); + tensor stk391 = ggml_concat(m, ggml_reshape_3d(m, mg390, 1, 20, 20), ggml_reshape_3d(m, mg390, 1, 20, 20), 0); + tensor rs392 = ggml_reshape_2d(m, ggml_cont(m, stk391), 2, 400); + tensor full394 = ggml_fill(m, ggml_new_tensor_2d(m, GGML_TYPE_F32, 1, 400), 32.0f); + tensor cat395 = ggml_concat(m, ggml_concat(m, rs374, rs383, 1), rs392, 1); + tensor cat396 = ggml_concat(m, ggml_concat(m, full376, full385, 1), full394, 1); + tensor t397 = ggml_cont(m, ggml_permute(m, cat395, 1, 0, 2, 3)); + tensor t398 = ggml_cont(m, ggml_permute(m, cat396, 1, 0, 2, 3)); + tensor uns399 = ggml_reshape_3d(m, ggml_cont(m, t397), 8400, 2, 1); + tensor sl400 = ggml_view_3d(m, cat336, 8400, 2, 1, cat336->nb[1], cat336->nb[2], 0); + tensor sl401 = ggml_view_3d(m, cat336, 8400, 2, 1, cat336->nb[1], cat336->nb[2], 2*cat336->nb[1]); + tensor t402 = ggml_sub(m, uns399, sl400); + tensor add403 = ggml_add(m, uns399, sl401); + tensor cat404 = ggml_concat(m, t402, add403, 1); + tensor t405 = ggml_mul(m, cat404, t398); + tensor t406 = ggml_sigmoid(m, ggml_concat(m, ggml_concat(m, rs346, rs356, 0), rs366, 0)); + tensor cat407 = ggml_concat(m, t405, t406, 1); + + tensor out_0 = compute_graph_output(m, contiguous_2d_to_cwhn(m, cat286), "out_0"); + tensor out_1 = compute_graph_output(m, contiguous_2d_to_cwhn(m, act203), "out_1"); + tensor out_2 = compute_graph_output(m, contiguous_2d_to_cwhn(m, act230), "out_2"); + tensor out_3 = compute_graph_output(m, contiguous_2d_to_cwhn(m, act267), "out_3"); + tensor out_4 = compute_graph_output(m, contiguous_2d_to_cwhn(m, cat317), "out_4"); + tensor out_5 = compute_graph_output(m, contiguous_2d_to_cwhn(m, cat336), "out_5"); + tensor out_6 = compute_graph_output(m, contiguous_2d_to_cwhn(m, act203), "out_6"); + tensor out_7 = compute_graph_output(m, contiguous_2d_to_cwhn(m, act230), "out_7"); + tensor out_8 = compute_graph_output(m, contiguous_2d_to_cwhn(m, act267), "out_8"); + tensor out_9 = compute_graph_output(m, contiguous_2d_to_cwhn(m, cat367), "out_9"); + return out_9; +} + +Yolo26m_params Yolo26m_detect_params(model_file const& f) { + Yolo26m_params p{}; + // GGUF general.architecture 검증 (모델은 정적 unroll 이라 추가 하이퍼파라미터 없음). + if (std::string_view arch = f.arch(); arch != "yolo26m") { + throw except( + "Architecture expected to be 'yolo26m', but was '{}' ({})", + arch, f.path); + } + return p; +} + +} // namespace visp diff --git a/src/visp/arch/Yolo26m.h b/src/visp/arch/Yolo26m.h new file mode 100644 index 0000000..edcf069 --- /dev/null +++ b/src/visp/arch/Yolo26m.h @@ -0,0 +1,14 @@ +// GENERATED BY SuperGate GTX Compiler (ggml/vision.cpp backend), DO NOT EDIT! +#pragma once +#include "visp/ml.h" + +namespace visp { + +struct Yolo26m_params { + // TODO(ggml): 모델 하이퍼파라미터 (block_count 등) +}; + +tensor Yolo26m_forward(model_ref m, tensor x, Yolo26m_params const& p); +Yolo26m_params Yolo26m_detect_params(model_file const& f); + +} // namespace visp diff --git a/src/visp/arch/yolo26m_register.cpp b/src/visp/arch/yolo26m_register.cpp new file mode 100644 index 0000000..f5e8b07 --- /dev/null +++ b/src/visp/arch/yolo26m_register.cpp @@ -0,0 +1,47 @@ +// GENERATED BY tools/install_arch.py — 손으로 고치지 마라(재설치하면 덮어쓴다). +// +// 이 파일은 **아무도 부르지 않는다.** 전역 객체 생성자가 프로그램 시작 시 스스로 +// `arch_registry` 에 등록한다. 그래서 CLI 에 case 를 추가할 필요가 없다. +#include "visp/arch/Yolo26m.h" +#include "visp/arch_registry.h" + +namespace { + +visp::tensor forward(visp::model_ref m, visp::tensor x, visp::model_file const& f) { + // `Yolo26m_params` 는 arch 마다 타입이 달라 레지스트리 시그니처에 못 싣는다 — + // 여기서 만들어 소화한다. + return visp::Yolo26m_forward(m, x, visp::Yolo26m_detect_params(f)); +} + +visp::arch_task make_task() { + visp::arch_task t; + t.kind = visp::arch_kind::detect_yolo; + t.num_classes = 80; + t.strides = {8.0f, 16.0f, 32.0f}; + t.nms_free = true; + t.score_thr = 0.25f; + t.input_size = 640; + t.mean = {0.0f, 0.0f, 0.0f}; + t.stdv = {255.0f, 255.0f, 255.0f}; + t.class_names = { + "person", "bicycle", "car", "motorcycle", "airplane", "bus", + "train", "truck", "boat", "traffic light", "fire hydrant", "stop sign", + "parking meter", "bench", "bird", "cat", "dog", "horse", + "sheep", "cow", "elephant", "bear", "zebra", "giraffe", + "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", + "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", + "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", + "fork", "knife", "spoon", "bowl", "banana", "apple", + "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", + "donut", "cake", "chair", "couch", "potted plant", "bed", + "dining table", "toilet", "tv", "laptop", "mouse", "remote", + "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", + "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", + "hair drier", "toothbrush", + }; + return t; +} + +const visp::arch_registrar reg{"yolo26m", &forward, make_task()}; + +} // namespace diff --git a/tools/example_custom.sh b/tools/example_custom.sh new file mode 100755 index 0000000..6e79021 --- /dev/null +++ b/tools/example_custom.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +# example_custom.sh — **직접 정의한 nn.Module** 을 끝까지 돌린다. +# +# ./tools/example_custom.sh +# +# 앞의 세 예제는 라이브러리가 모델을 준다(ultralytics·torchvision·mmdet). 이건 아니다 — +# 클래스를 내가 쓰고, `.pt` 로 저장하고, g2c 에 그 `.pt` 를 준다. 다른 부서가 자기 모델을 +# 얹을 때 밟는 경로가 이것이다. +# +# ⚠️ **모델 클래스는 별도 `.py` 에 둔다.** `torch.save(model)` 는 클래스를 **모듈 이름으로** +# 피클하므로, `__main__` 에서 정의하면 로드하는 쪽이 그 이름을 못 찾아 +# `ModuleNotFoundError` 로 죽는다. 그래서 아래도 `mymodel.py` 를 따로 쓴다. +# +# ⚠️ **가중치는 실제 학습된 것을 쓴다.** 사전학습 ResNet 의 앞단을 그대로 가져와 조립한다. +# 랜덤 초기화로 재면 항등 초기값이 빠진 연산을 덮어 검증을 통과시킨다. +set -euo pipefail + +SIZE="${SIZE:-224}" +VCPP="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +G2C="${G2C_ROOT:-$(cd "$VCPP/.." && pwd)}" +WORK="${WORK:-/tmp/visp-example-custom}" + +if [ ! -f "$G2C/shared/compile/pipeline.py" ]; then + echo "g2c 를 못 찾았다: $G2C" >&2 + echo "G2C_ROOT= 로 지정할 것." >&2 + exit 1 +fi + +mkdir -p "$WORK" + +echo "== 1/4 모델 클래스 (별도 .py — 피클이 이름으로 찾는다) ==" +cat > "$WORK/mymodel.py" <<'PY' +"""사전학습 ResNet18 의 앞단 + 직접 쓴 head. 가중치는 전부 실제 학습된 것이다.""" +import torch.nn as nn +import torchvision + + +class MyNet(nn.Module): + def __init__(self): + super().__init__() + r = torchvision.models.resnet18(weights="DEFAULT") + # 사전학습 레이어를 그대로 가져온다 — 여기까지가 "실제 가중치" 다. + self.stem = nn.Sequential(r.conv1, r.bn1, r.relu, r.maxpool, r.layer1, r.layer2) + # 직접 쓴 부분. 사전학습 conv 를 1x1 로 줄여 쓰므로 여기도 학습된 값에서 나온다. + self.head = nn.Sequential( + nn.Conv2d(128, 64, 1, bias=False), + nn.BatchNorm2d(64), + nn.ReLU(inplace=True), + nn.AdaptiveAvgPool2d(1), + ) + # head 도 학습된 값에서 채운다 — 난수를 한 값도 안 쓴다. + # ⚠️ 채널 수가 다르다: layer2 는 128, 우리 head 는 64다. **앞 64개만 자른다.** + # `load_state_dict` 로 통째로 넣으면 size mismatch 로 죽는다(strict=False 도 못 막는다). + src_conv = r.layer2[-1].conv2.weight.detach() # [128,128,3,3] + self.head[0].weight.data.copy_( + src_conv.mean(dim=(2, 3))[:64].unsqueeze(-1).unsqueeze(-1)) # → [64,128,1,1] + bn = r.layer2[-1].bn2 + self.head[1].weight.data.copy_(bn.weight.detach()[:64]) + self.head[1].bias.data.copy_(bn.bias.detach()[:64]) + self.head[1].running_mean.data.copy_(bn.running_mean.detach()[:64]) + self.head[1].running_var.data.copy_(bn.running_var.detach()[:64]) + + def forward(self, x): + return self.head(self.stem(x)) +PY + +echo "== 2/4 실제 가중치로 저장 ==" +OMP_NUM_THREADS=1 uv run --project "$G2C" python - "$WORK" <<'PY' +import sys, torch +sys.path.insert(0, sys.argv[1]) +from mymodel import MyNet + +torch.set_num_threads(1) +m = MyNet().eval() +torch.save(m, f"{sys.argv[1]}/mynet.pt") +print(f" saved: {sys.argv[1]}/mynet.pt ({sum(p.numel() for p in m.parameters()):,} params)") +PY + +echo "== 3/4 g2c 컴파일 ==" +# PYTHONPATH 에 .pt 가 있는 폴더를 넣어야 피클이 `mymodel` 을 찾는다. +OMP_NUM_THREADS=1 PYTHONPATH="$WORK:$G2C" \ + uv run --project "$G2C" python -m shared.compile.pipeline \ + --model "$WORK/mynet.pt" --name MyNet \ + --output "$WORK" --input-shape "1,3,$SIZE,$SIZE" + +# 성공 판정은 종료코드가 아니라 **파일 유무**다 — g2c 는 실패해도 exit 0 + "완료!" 를 낸다. +[ -f "$WORK/MyNet.gguf" ] || { echo "gguf 가 안 나왔다 — 위 로그를 볼 것" >&2; exit 2; } + +echo "== 4/4 torch 와 대조 ==" +OMP_NUM_THREADS=1 PYTHONPATH="$WORK" \ + uv run --project "$G2C" --extra ggml python - "$WORK" "$SIZE" <<'PY' +import importlib.util, sys, numpy as np, torch +# ⚠️ 이름이 겹친다 — 내가 쓴 클래스도 `MyNet`, g2c 가 생성한 클래스도 `MyNet` 이다. +# (후자는 `nn.QuantModel` 을 상속한 별개 타입이다.) 별칭으로 갈라 둔다. +from mymodel import MyNet as TorchNet + +work, size = sys.argv[1], int(sys.argv[2]) +torch.manual_seed(0); torch.set_num_threads(1) +x = torch.randn(1, 3, size, size) + +with torch.no_grad(): + want = TorchNet().eval()(x).numpy().ravel() + +# 생성 .py 는 클래스만 정의한다. 실행 진입점은 `nn.run_gguf(Model(), gguf, 파일경로, x)` 다. +import nn +nn.set_backend("ggml") +gen_path = f"{work}/MyNet.py" +spec = importlib.util.spec_from_file_location("gen", gen_path) +mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod) +out = nn.run_gguf(mod.MyNet(), f"{work}/MyNet.gguf", gen_path, x.numpy()) +got = np.asarray(out[0] if isinstance(out, (list, tuple)) else out).ravel() + +# ⚠️ cosine 은 스케일 불변이라 크기가 통째로 틀려도 1.0 이 나온다. 거리로 잰다. +rel = np.abs(got - want).sum() / np.abs(want).sum() +print(f"상대 L1 {rel:.2e} ({want.size} 값)") +print("PASS" if rel < 5e-2 else "FAIL") +PY + +echo +echo "생성물: $WORK" diff --git a/tools/example_mmdet.sh b/tools/example_mmdet.sh new file mode 100755 index 0000000..894cf45 --- /dev/null +++ b/tools/example_mmdet.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# example_mmdet.sh — MMDetection 검출기를 **학습된 체크포인트**로 박스까지 돌린다. +# +# ./tools/example_mmdet.sh # retinanet +# ./tools/example_mmdet.sh fcos # 다른 계열 +# MMDET=~/mmbuild/mmdetection ./tools/example_mmdet.sh +# +# yolo·torchvision 예제와 갈리는 지점: **mmdet head 는 트레이스를 통과하지 못한다.** +# 그래서 `.pt` 로 나가는 것은 backbone+neck 뿐이고, head 는 C++ 부품이 조립한다. +# `bbox_head` 는 속성으로 남아 state_dict(→GGUF)에 실린다 — 연산만 빼고 가중치는 남긴다. +# 자세한 배경은 `docs/mmdet-detectors.md`. +# +# ⚠️ **학습된 체크포인트로 잰다.** config 만으로 지은 랜덤 초기화는 항등 초기값 +# (γ=1·β=0, scale=1)이 빠진 연산을 덮어 검증을 통과시킨다. +set -euo pipefail + +FAM="${1:-retinanet}" +SIZE="${SIZE:-512}" + +VCPP="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +MMDET="${MMDET:-$HOME/mmbuild/mmdetection}" +WORK="${WORK:-/tmp/visp-example-$FAM}" +DH="$VCPP/tools/verify/dense_head" + +# ⚠️ 이 예제만 **mmdet 이 설치된 인터프리터**가 필요하다. 앞의 세 예제와 갈리는 지점이다 — +# 거긴 `uv run --project` 로 g2c 환경을 쓰면 됐지만, 여기선 하네스가 mmdet 을 import 한다. +# `python3` 를 그냥 쓰면 시스템 파이썬으로 가서 `No module named 'yaml'` 같은 +# 엉뚱한 곳에서 죽는다 — 진짜 원인(mmdet 환경이 아님)이 안 보인다. +PYTHON="${PYTHON:-python3}" + +if [ ! -d "$MMDET/configs" ]; then + echo "MMDetection 체크아웃을 못 찾았다: $MMDET" >&2 + echo "MMDET= 로 지정할 것. configs/ 와 checkpoints/ 가 있어야 한다." >&2 + exit 1 +fi + +if ! "$PYTHON" -c "import mmdet, yaml" 2>/dev/null; then + echo "'$PYTHON' 에 mmdet(또는 pyyaml)이 없다." >&2 + echo "PYTHON= ./tools/example_mmdet.sh $FAM" >&2 + echo "확인: \$PYTHON -c 'import mmdet; print(mmdet.__version__)'" >&2 + exit 1 +fi + +echo "== 1/2 config·체크포인트 짝 고르기 ==" +# ⚠️ 짝을 손으로 고르지 마라 — 계열마다 변종이 여럿이라 **남의 가중치를 재게** 된다. +# metafile.yml 이 계열마다 Config → Weights 를 갖고 있으므로 그걸 읽는다. +read -r CFG CKPT <&2; exit 2; } + +echo "== 2/2 박스 대조 (mmdet 자신의 predict_by_feat 와) ==" +# 하네스가 export → g2c → head 조립 → 실행 → 대조를 한 번에 한다. +# ⚠️ vision.cpp 를 먼저 빌드해 둬야 한다 — 러너가 libvisioncpp 에 링크한다. +cd "$DH" +OMP_NUM_THREADS=1 "$PYTHON" verify_heads.py "$FAM" \ + --set "paths.workdir=$WORK" --set run.workers=1 --set "run.size=$SIZE" + +GEN="$WORK/$FAM/out" +[ -x "$GEN/run_mmdet" ] || { echo "러너가 안 나왔다 — 위 로그를 볼 것" >&2; exit 3; } + +OMP_NUM_THREADS=1 "$PYTHON" verify_postproc.py \ + "$GEN" "$CFG" "$CKPT" "$VCPP/tests/input/cat-and-hat.jpg" "$SIZE" + +echo +echo "생성물: $WORK/$FAM" diff --git a/tools/example_torchvision.sh b/tools/example_torchvision.sh new file mode 100755 index 0000000..3c799e8 --- /dev/null +++ b/tools/example_torchvision.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# example_torchvision.sh — torchvision 분류 모델을 **실제 ImageNet 가중치**로 끝까지 돌린다. +# +# ./tools/example_torchvision.sh # resnet18 +# ./tools/example_torchvision.sh resnet50 +# +# yolo 예제와 다른 점: 분류 모델은 박스가 없다. `vision-cli` 대신 생성된 `.py` 로 +# ggml 커널에서 돌리고 **torch 와 값을 대조**한다 — 그게 이 갈래의 "끝까지" 다. +# +# ⚠️ `weights='DEFAULT'` 를 쓴다. 랜덤 초기화로 재지 마라 — 항등 초기값(γ=1·β=0)이 +# 빠진 연산을 덮어 검증을 통과시킨다. 실제로 VFNet 의 scale 하드코딩이 그렇게 숨었다. +set -euo pipefail + +MODEL="${1:-resnet18}" +SIZE="${SIZE:-224}" + +VCPP="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +G2C="${G2C_ROOT:-$(cd "$VCPP/.." && pwd)}" +WORK="${WORK:-/tmp/visp-example-$MODEL}" + +# resnet18 → ResNet18 (g2c --name 규약: 클래스명이 파일명이 된다) +CLS="$(printf '%s' "${MODEL:0:1}" | tr '[:lower:]' '[:upper:]')${MODEL:1}" + +if [ ! -f "$G2C/shared/compile/pipeline.py" ]; then + echo "g2c 를 못 찾았다: $G2C" >&2 + echo "G2C_ROOT= 로 지정할 것." >&2 + exit 1 +fi + +echo "== 1/3 g2c 컴파일 ($MODEL, 실제 ImageNet 가중치 → $WORK) ==" +# torchvision 이름을 그대로 준다. g2c 가 weights='DEFAULT' 로 받아 온다. +OMP_NUM_THREADS=1 PYTHONPATH="$G2C" \ + uv run --project "$G2C" python -m shared.compile.pipeline \ + --model "torchvision.models.$MODEL(weights='DEFAULT')" --name "$CLS" \ + --output "$WORK" --input-shape "1,3,$SIZE,$SIZE" + +# 성공 판정은 종료코드가 아니라 **파일 유무**다 — g2c 는 실패해도 exit 0 + "완료!" 를 낸다. +[ -f "$WORK/$CLS.gguf" ] || { echo "gguf 가 안 나왔다 — 위 로그를 볼 것" >&2; exit 2; } + +echo "== 2/3 ggml 커널로 실행 ==" +# 생성된 .py 는 GGUF 를 libggml.so 로 eager 실행하는 진입점이다. 파이썬 바인딩이 필요하다. +OMP_NUM_THREADS=1 uv run --project "$G2C" --extra ggml python "$WORK/$CLS.py" + +echo "== 3/3 torch 와 대조 ==" +OMP_NUM_THREADS=1 uv run --project "$G2C" --extra ggml python - "$MODEL" "$WORK" "$CLS" <<'PY' +import importlib.util, os, sys, numpy as np, torch, torchvision + +name, work, cls = sys.argv[1], sys.argv[2], sys.argv[3] +torch.manual_seed(0); torch.set_num_threads(1) +x = torch.randn(1, 3, 224, 224) + +ref = getattr(torchvision.models, name)(weights="DEFAULT").eval() +with torch.no_grad(): + want = ref(x).numpy().ravel() + +# 생성 .py 는 클래스만 정의한다. 실행은 `nn.run_gguf(Model(), gguf, 파일경로, x)` 다 +# (그 파일의 `__main__` 블록이 쓰는 것과 같은 진입점). +import nn +nn.set_backend("ggml") +gen_path = os.path.join(work, cls + ".py") +spec = importlib.util.spec_from_file_location("gen", gen_path) +mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod) +out = nn.run_gguf(getattr(mod, cls)(), os.path.join(work, cls + ".gguf"), gen_path, x.numpy()) +got = np.asarray(out[0] if isinstance(out, (list, tuple)) else out).ravel() + +# ⚠️ cosine 으로 재지 마라 — 스케일 불변이라 크기가 통째로 틀려도 1.0 이 나온다. +# rtmdet 이 cos 0.999450 으로 통과했는데 값은 97% 틀렸다. 거리로 잰다. +rel = np.abs(got - want).sum() / np.abs(want).sum() +print(f"상대 L1 {rel:.2e} · argmax torch={want.argmax()} ggml={got.argmax()}") +print("PASS" if rel < 5e-2 and want.argmax() == got.argmax() else "FAIL") +PY + +echo +echo "생성물: $WORK" diff --git a/tools/example_yolo.sh b/tools/example_yolo.sh index 3250571..a148d05 100755 --- a/tools/example_yolo.sh +++ b/tools/example_yolo.sh @@ -46,7 +46,7 @@ echo "== 3/4 빌드 ==" cmake --build "$BUILD" -j"$(nproc)" > /dev/null echo "== 4/4 실행 ==" -OUT="$WORK/detected.jpg" +OUT="$WORK/detected.png" # vision-cli 는 확장자와 무관하게 PNG 로 쓴다 "$BUILD/bin/vision-cli" "$MODEL" -m "$WORK/$CLS.gguf" -i "$IMAGE" -o "$OUT" echo diff --git a/tools/install_arch.py b/tools/install_arch.py index 4b1035d..e45a528 100644 --- a/tools/install_arch.py +++ b/tools/install_arch.py @@ -206,7 +206,9 @@ def _triple(spec, what): print() print("next:") print(f" cmake --build {os.path.join(VCPP, 'build')} -j4") - print(f" {os.path.join(VCPP, 'build/bin/vision-cli')} {arch} -m -i -o out.jpg") + # vision-cli 는 확장자와 무관하게 PNG 를 쓴다(image_save → stbi_write_png). + # .jpg 를 권하면 PNG 인데 이름만 .jpg 인 파일이 나와 뷰어가 거부한다. + print(f" {os.path.join(VCPP, 'build/bin/vision-cli')} {arch} -m -i -o out.png") if __name__ == "__main__": diff --git a/tools/verify/dense_head/verify_heads.py b/tools/verify/dense_head/verify_heads.py index 9f60931..96836da 100644 --- a/tools/verify/dense_head/verify_heads.py +++ b/tools/verify/dense_head/verify_heads.py @@ -212,6 +212,48 @@ def save(tag, lst): f"먼저 빌드해라 —\n" f" cmake -S {V} -B {V}/build\n" f" cmake --build {V}/build -j4") + + +def _abi_name_len(): + """헤더의 `GGML_MAX_NAME` 과 **라이브러리에 실제로 박힌 값**을 돌려준다. + + ⚠️ **존재 검사만으로는 부족하다.** `tensor_name = fixed_string` 이 공개 API + 시그니처에 들어 있어 이 매크로는 사실상 **ABI 버전**이다. 헤더가 64인데 라이브러리가 + 128 로 구워져 있으면 맹글링이 갈려 계열마다 + `undefined reference … fixed_string<64ul>` 로 죽는다 — 라이브러리 파일은 멀쩡히 **있다.** + 실측: 서브모듈 포인터를 되돌린 뒤 재빌드를 안 해서 **100계열 × 47초를 태웠다.** + 위키 `헤더와-라이브러리는-같은-트리여야-한다`. + """ + hdr = None + try: + with open(os.path.join(V, "depend", "llama", "ggml", "include", "ggml.h"), + encoding="utf-8") as f: + m = re.search(r"define\s+GGML_MAX_NAME\s+(\d+)", f.read()) + hdr = int(m.group(1)) if m else None + except OSError: + pass + lib = None + try: + out = subprocess.run(["nm", "-DC", os.path.join(V, "build", "lib", "libvisioncpp.so")], + capture_output=True, text=True, timeout=60).stdout + # ⚠️ `fixed_string` 을 통째로 세면 **오탐한다** — 예외 메시지 타입이 + # `fixed_string<128>` 로 따로 있고(`include/visp/util.h`) GGML_MAX_NAME 과 무관하다. + # ABI 를 실제로 가르는 함수 **하나의 시그니처**로 좁혀 읽는다. + m = re.search(r"compute_graph_output\([^)]*fixed_string<(\d+)ul>", out) + if m: + lib = int(m.group(1)) + except (OSError, subprocess.SubprocessError): + pass # nm 이 없으면 검사를 건너뛴다(막지 않는다) + return hdr, lib + + +_hdr_n, _lib_n = _abi_name_len() +if _hdr_n and _lib_n and _hdr_n != _lib_n: + sys.exit(f"헤더와 라이브러리의 GGML_MAX_NAME 이 다르다 — 헤더 {_hdr_n} · 라이브러리 {_lib_n}.\n" + f"이 매크로는 `fixed_string` 으로 공개 API 시그니처에 들어가 **ABI 버전**이다.\n" + f"그대로 두면 계열마다 `undefined reference … fixed_string<{_hdr_n}ul>` 로 죽는다.\n" + f"라이브러리를 지금 헤더로 다시 구워라 —\n" + f" cmake --build {V}/build -j4") # ⚠️ **메모리 가드.** 위키 `wsl-계속-터짐` — 병렬 torch 스윕이 WSL 을 통째로 죽인 적이 있다. # 가용 메모리가 이 밑으로 내려가면 새 계열을 안 띄우고 기다린다. 느려질지언정 안 죽는다. MIN_FREE_MB = CFG.min_free_mb @@ -302,6 +344,16 @@ def run(cmd, cwd, env_extra=None, timeout=2400, phase=None): # `ValueError: too many values to unpack` 로 죽고, 앞 둘만 쓰면 절반이 검증 안 된다. _o = list(_o) if isinstance(_o, (list, tuple)) else [_o] assert len(_o) >= 2 and len(_o) %% 2 == 0, "SubB 출력이 (cls, box) 쌍이 아니다: %%d" %% len(_o) +# ⚠️ **개수만 세면 안 된다.** `with_reg=False` 인 head 는 자리는 채우되 값이 `None` 이다 +# (`grid_rcnn` — 좌표를 별도 `grid_head` 가 낸다). 그대로 두면 다음 줄에서 +# `'NoneType' object has no attribute 'numpy'` 로 죽어 **원인이 안 보인다.** +# 여기서 이유를 대고 멈춘다 — 이 하네스가 못 재는 구조지 계열의 결함이 아니다. +for _k in range(len(_o) // 2): + if _o[2 * _k + 1] is None: + raise SystemExit( + "SubB 의 bbox_pred 가 None 이다 — with_reg=False 인 head 다(좌표를 다른 head 가 낸다). " + "이 하네스는 (cls, box) 쌍을 전제하므로 이 계열은 two-stage 하네스로 재라: " + "tools/verify/roi/verify_postproc_roi.py") for _k in range(len(_o) // 2): _c, _b = _o[2 * _k], _o[2 * _k + 1] np.ascontiguousarray(_c.numpy()).tofile("ref.cls.%%d.bin" %% _k)