Moving build system to pixi - #2037
Conversation
A devcontainer is a small wrapper around Docker to make it easy to set up a clean development environment. Most IDE's and code editors support these now as well as many cloud environments. I've included a very simple environment to start with which we can extend as necessary later to streamline the build process.
This diff adds the pixi file and updates some SuperBuild scripts to compile. With this we can run `pixi install` and `pixi run build` and the compilation works.
Output report from before and after pixi are almost identical.
roughly a 6 minute build now
It certainly does! That's much clearer 😄 Thank you - I should probably close the linked issue then, as I think it's handled well. |
jmeshlib's vertex ring traversals advance with Triangle::oppositeEdge(), which returns NULL when the vertex is not part of the triangle, but only test against the start edge. The NULL is carried into the next iteration and dereferenced, segfaulting the -edgeSwapThreshold path used to build the 2.5D mesh.
This fixes a build failure in CI for MacOS which failed as it couldn't resolve rasterio which was only available on MacOS 14. This locks that as the minimum version and reworks the platform format support to avoid errors on the latest pixi.
- opendm/remote.py and stages/splitmerge.py, on windows this would fetch the host python env, patching to force the same interpreter. - opendm/osfm.py - we do not need win.bat anymore.
The runtime stage copies a curated set of paths out of the builder, which did not include contrib, so the tools are absent from the image. .dockerignore also excludes contrib. That has no effect on the published images, which are built from a Git context where it is not applied, but it does apply to a local `docker build`, where the copy would then fail. Drop the entry so both paths behave the same. The documented invocations used --entrypoint, which bypasses the pixi activation the tools need. Pass the interpreter as the command instead: the entrypoint execs an argument that resolves on PATH, so the tool runs inside the activated environment.
This is a temporary fix to avoid clobbering working images.
It seems most big projects keep the "primary" version first and the variant second. This flips the tag to follow. This also means it should be for example :3.6.2-gpu which I did differently but we can retag them.
The gpu and gpu-prod environments target the linux-64-cuda-12-0 platform, so pixi requires the __cuda virtual package to install them. No NVIDIA driver is visible inside docker build, so both pixi install steps aborted with unsupported-platform. Set CONDA_OVERRIDE_CUDA in the shared dev stage, which covers builder and prod-env alike.
The gpu and gpu-prod environments target win-64-cuda-12-0, so pixi requires the __cuda virtual package to install them. The windows-2022 runner has no NVIDIA driver, so setup-pixi and both pixi run steps would abort the same way the GPU docker build did.
OpenMVS resolves CUDA_CUDA_LIBRARY through FindCUDA, whose toolkit-local search covers lib and lib64 but not stubs, and whose default-path search only reaches a real driver library. Neither exists inside docker build, so the configure step failed with CUDA_CUDA_LIBRARY-NOTFOUND linked by target MVS. The existing CMAKE_LIBRARY_PATH hint only covered the toolkit layout used by the NVIDIA base images; the conda toolkit keeps its stub under the environment prefix instead. Look there too, and only when no real driver library is found, so builds on a CUDA machine keep linking the real one.
The smoke test launches DensifyPointCloud and ReconstructMesh, which carry a NEEDED libcuda.so.1 once OpenMVS is built with CUDA. Nothing injects a driver during a build, and the gpu-prod environment ships only the CUDA runtime, so the launch check had no libcuda to resolve. The base image already carries a compat driver, just not on the loader path. Point the smoke step at it and no further: the pixi hook appends this to LD_LIBRARY_PATH, so it resolves only what the environment cannot, and a container started with the NVIDIA runtime still binds the host driver.
The PR matrix builds the default CPU environment and the docker job builds the CPU image, so nothing exercised gpu.Dockerfile before a merge. Build it the same way the CPU image is built, through the runtime target so the smoke test runs with it. Cancel superseded runs for the ref, since the GPU image takes about an hour.
A hosted runner could not finish it: the job died 58 minutes in with the runner losing contact, its log stopping at [129/152] Completed 'lastools', immediately before OpenMVS compiles across 19 GPU architectures. Fitting the build into a hosted runner meant deleting parts of its filesystem and repointing its docker daemon, none of which belongs in a workflow. Build through the daemon's own store so layers survive between runs, which spares the two pixi install steps on a rerun, and age out old layers afterwards so a shared runner does not fill up. The repository is public, so gate the job to branches pushed to the repository itself: a self-hosted runner must not execute code from a fork.
A hosted runner cannot build this image: the equivalent pull request job died 58 minutes in with the runner losing contact, just as OpenMVS began compiling across 19 GPU architectures. The same build finishes in 15 minutes on the self-hosted runner, which is where the CPU image is already published from.
GitHub Actions
Failure
Two GPU builds filled the runner's disk, to the point that the next job could not write its own log. Both came from building through the daemon's own store to keep layers as cache, which on a build this size means tens of GB kept per run against a reclaim that dropped nothing under a fortnight old. The cache was not worth it: it spared the two pixi install steps, about ten minutes of a fifteen minute build. Build through the default container driver instead, which is torn down with the job, and reclaim unconditionally afterwards. The publish workflows never reclaimed anything at all, and the GPU one builds with no cache on every push to master, so give them the same step.
The build ran the runner out of disk again with 34GB free, dying under four minutes in, while still unpacking environments. The builder and prod-env stages are independent, so buildkit materialises both at once, and each carries a multi-GB conda environment before anything is compiled. Hold buildkit to one worker, as the CPU publish already does on this runner. Reclaim before building as well as after, since the trailing reclaim is skipped when a job is killed outright, which is how the runner filled in the first place. Report free space either side so the cost of a build is a number in the log rather than an inference from what broke.
|
Sorry the CI jobs are causing problems due to disk space but I reverted to the last full successful commit. That gives us a good baseline. I want to do a full CI review anyway but I don't want to hold this branch up. Testing of it has been good with no noticable drop in performance. It passes the full oats suite with no obvious differences to the existing master. |
4 participants
This moves the entire ODM buildsystem to pixi. This allows us to use prebuild binaries for our libraries massively cutting down the build time, while also introducing a single pixi.toml file to store dependencies.
With this change, anyone should be able to build and run ODM like this:
This should work on Linux (x86, aarch64), Windows, Mac. GPU support is also available by adding a
-e gpuargument to the commands.Changes
Testing
To ensure this doesn't cause any major regressions, I've been keeping an eye on the following canaries:
Block-Michenzaniandodm_data_aukerman. I have been running these with no additional flags and to my untrained eye the results look pretty consistent.Missing testing
While I have tested the basics, there are some areas I'm aware I haven't tested enough yet:
Future work
pixi.lockfile to lock the exact versions.