Gate 0 smoke: CARLA RenderThread timeout / SIGSEGV — need working Dockerfile (Vulkan + shm 2g)

[karmugilen]

Summary

Follow-up to #1. Repo commit fa8d091 is fine. Gate 0 smoke still fails on the A4000 Docker box because CARLA/Unreal cannot render.

Please send a Dockerfile + exact docker run that you have already used to pass:

[job] Smoke test passed; no training requested.

Do not keep patching Python. This is a container GPU/Vulkan/shm problem.

What already works (fa8d091)

Host: rajkumar_12497@10.30.161.69 → docker exec -it mywork bash
Container: originally 70dac519e9f3, RTX A4000 16 GiB
Project: /home/rajkumar_12497/student1/Auto_Thinker_Engine

git pull --ff-only origin main          # a98431e -> fa8d091  OK
git submodule status
  160132436aeda1de54956f9910e56f3970a565aa third_party/CarDreamer
  6ef8646d807cd10ce0c88e10a7e943211e7fc44c third_party/dreamerv3_torch
uv sync --frozen --extra dev --extra carla   OK
bash scripts/setup_cardreamer.sh "$CARLA_ROOT"
  SETUP_RC=0
  Unit tests: 80 passed, 8 warnings in 4.82s
python run.py doctor --require-cuda --require-carla
  DOCTOR_RC=0
  CUDA: available (1 device(s))  NVIDIA RTX A4000  torch 2.4.1+cu121
  CARLA Python API: importable

Root refusal is gone with CARLA_RUN_USER=carla.

Command that still fails

CARLA_ROOT=/home/rajkumar_12497/student1/software \
CARLA_RUN_USER=carla \
RUN_MODE=smoke \
bash jobs/slurm_run.sh
# SMOKE_RC=1

Log: outputs/carla_2000.log

ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5182:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5182:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5182:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5705:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2664:(snd_pcm_open_noupdate) Unknown PCM default
LowLevelFatalError [File:Unknown] [Line: 1214]
GameThread timed out waiting for RenderThread after 60.00 secs
Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
4.26.2-0+++UE4+Release-4.26 522 0
Disabling core dumps.
CommonUnixCrashHandler: Signal=11
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

Console:

SMOKE TEST FAILED: time-out of 30000ms while waiting for the simulator, make sure the simulator is ready and connected to localhost:2000
⚠️  Smoke test FAILED. Do NOT proceed to Phase 1.
ERROR: CARLA exited before becoming ready. Log: .../outputs/carla_2000.log
SMOKE_RC=1

Docker facts (docker inspect mywork)

Image=mywork:with-tools
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility    # CUDA only — no graphics/Vulkan
ShmSize=67108864                              # 64 MiB  (CARLA wants ~2g)
Binds=["/home/rajkumar_12497:/home/rajkumar_12497"]

Inside the original container:

$ ls -l /etc/vulkan/icd.d/nvidia_icd.json
-rw-r--r-- 1 root root 0 Jul 29 08:51 /etc/vulkan/icd.d/nvidia_icd.json

$ ls -l /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.580.173.02
-rw-r--r-- 1 root root 0 Jul 29 08:51 .../libGLX_nvidia.so.580.173.02

$ ls /usr/share/vulkan/icd.d
intel_hasvk_icd.x86_64.json  intel_icd.x86_64.json
lvp_icd.x86_64.json          radeon_icd.x86_64.json
virtio_icd.x86_64.json

Vulkan:

Cannot create Vulkan instance.
ERROR at ./vulkaninfo/vulkaninfo.h:649:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER
loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0

Host has the real driver files (1.2M libGLX_nvidia, 36M libnvidia-glcore, valid nvidia_icd.json). The container only got CUDA (compute,utility), so Unreal has nothing to render with.

What we need

A Dockerfile (or compose) plus the exact docker run you used when smoke passed. Minimum:

docker run -d --name auto-thinker \
  --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=all \
  -e NVIDIA_VISIBLE_DEVICES=all \
  --shm-size=2g \
  --user <non-root-uid> \
  -v "$PWD":/work -w /work \
  <your-image>

Must succeed inside that image:

python run.py doctor --require-cuda --require-carla
CARLA_ROOT=... RUN_MODE=smoke bash jobs/slurm_run.sh

Also document CARLA 0.9.15 path, non-root user, VK_ICD_FILENAMES to a non-empty NVIDIA ICD, and SDL_AUDIODRIVER=dummy / -nosound.

CNN 10k is blocked until smoke passes.

[JayacharanR]

## Resolution: reproducible CARLA GPU/Vulkan Docker runtime

The original Gate 0 failure was caused by the container runtime, not the Python repository:

- `NVIDIA_DRIVER_CAPABILITIES=compute,utility` exposed CUDA but not graphics.
- `/dev/shm` was only 64 MiB.
- NVIDIA Vulkan/OpenGL libraries were missing or zero-byte.
- CARLA/Unreal therefore failed in `RenderThread` and crashed.

The fix is included in commit `57ed72b`:

- Added a top-level `Dockerfile`.
- Added `.dockerignore`.
- Added `DOCKER_CARLA_RUNBOOK.md`.
- Added NVIDIA Vulkan ICD validation.
- Added dummy ALSA audio configuration.
- CARLA now starts with `-RenderOffScreen -nosound`.
- The container runs as a non-root UID.
- Shared memory is configured at 2 GiB.
- CARLA 0.9.15 is mounted from persistent host storage.
- The locked Python environment is installed with Python 3.10.
- `setup_cardreamer.sh` and `jobs/slurm_run.sh` use the image virtual environment.
- No Python model code was changed.

The full instructions are also available in:

`DOCKER_CARLA_RUNBOOK.md`

## Host prerequisites

Run these commands on the A4000 host:

```bash
nvidia-smi
docker info

The host must have:

• NVIDIA Container Toolkit configured for Docker;
• CARLA 0.9.15 extracted to persistent storage;
• a CARLA directory writable by the container UID;
• Docker support for --gpus all.

Set the repository and CARLA paths:

export REPO_ROOT=/path/to/Auto_Thinker_Engine
export CARLA_HOST_ROOT=/persistent/software/CARLA_0.9.15

cd "$REPO_ROOT"

test -x "$CARLA_HOST_ROOT/CarlaUE4.sh"
test -f "$CARLA_HOST_ROOT/CarlaUE4/Binaries/Linux/CarlaUE4-Linux-Shipping"
test -w "$CARLA_HOST_ROOT"

CARLA_HOST_ROOT must point to the directory containing CarlaUE4.sh.

Build the image

Build using the same UID/GID that will own the repository and CARLA files:

cd "$REPO_ROOT"

docker build \
  --build-arg USER_UID="$(id -u)" \
  --build-arg USER_GID="$(id -g)" \
  -t auto-thinker:carla-gpu .

Exact container command

Run this command from the repository directory:

docker rm -f auto-thinker 2>/dev/null || true

docker run -d --name auto-thinker \
  --gpus all \
  -e NVIDIA_DRIVER_CAPABILITIES=all \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e CARLA_ROOT=/opt/carla \
  -e VK_ICD_FILENAMES=/etc/vulkan/icd.d/nvidia_icd.json \
  -e SDL_AUDIODRIVER=dummy \
  --shm-size=2g \
  --user "$(id -u):$(id -g)" \
  -v "$PWD":/work \
  -v "$CARLA_HOST_ROOT":/opt/carla:rw \
  -w /work \
  auto-thinker:carla-gpu

This uses a non-root container user, so CARLA_RUN_USER is not required.

The important runtime settings are:

NVIDIA_DRIVER_CAPABILITIES=all
NVIDIA_VISIBLE_DEVICES=all
/dev/shm=2g
CARLA_ROOT=/opt/carla
VK_ICD_FILENAMES=/etc/vulkan/icd.d/nvidia_icd.json
SDL_AUDIODRIVER=dummy

Verify GPU, Vulkan, shared memory, and user

Run:

docker exec auto-thinker bash -lc '
set -Eeuo pipefail

test "$(id -u)" -ne 0
echo "UID: $(id -u)"

df -h /dev/shm

test -s "$VK_ICD_FILENAMES"
grep -Eqi "nvidia|libGLX_nvidia" "$VK_ICD_FILENAMES"

vulkaninfo --summary
nvidia-smi
python --version
'

The NVIDIA ICD JSON must be non-empty and must reference NVIDIA. Vulkan must successfully create an instance and report the NVIDIA GPU.

The expected shared-memory size is approximately 2 GiB.

If /etc/vulkan/icd.d/nvidia_icd.json is not available but the NVIDIA toolkit mounted the file at /usr/share/vulkan/icd.d/nvidia_icd.json, the repository launcher detects and uses that alternate path.

If Vulkan reports:

ERROR_INCOMPATIBLE_DRIVER

or the ICD file is empty, recreate the container with the exact --gpus all and NVIDIA_DRIVER_CAPABILITIES=all settings. Do not attempt to fix this with Python packages.

Set up the repository inside the container

Run the existing setup script:

docker exec -it auto-thinker bash -lc \
  'bash scripts/setup_cardreamer.sh "$CARLA_ROOT"'

This performs the following:

• verifies both submodules;
• applies the tracked Dreamer compatibility patch;
• runs uv sync --frozen --extra dev --extra carla;
• installs or verifies CARLA Python API 0.9.15;
• verifies CarDreamer imports;
• runs the unit tests.

Expected results include:

CARLA API: OK
PyTorch: OK (CUDA: True, devices: 1)
Unit tests: 80 passed

Run the required Gate 0 doctor check

docker exec -it auto-thinker bash -lc \
  'python run.py doctor --require-cuda --require-carla'

Expected result:

CUDA: available
CARLA Python API: importable
[doctor] Passed.

Run the required CARLA smoke test

docker exec -it auto-thinker bash -lc \
  'CARLA_ROOT=/opt/carla RUN_MODE=smoke bash jobs/slurm_run.sh'

The launcher will:

1. validate the NVIDIA Vulkan ICD;
2. set SDL_AUDIODRIVER=dummy;
3. run CARLA with -RenderOffScreen -nosound;
4. start CARLA on port 2000;
5. connect the CARLA Python client;
6. spawn a vehicle and camera;
7. advance 100 synchronous simulation steps;
8. stop CARLA after the smoke test.

The required final line is:

[job] Smoke test passed; no training requested.

The CARLA log is written to:

outputs/carla_2000.log

Run the CNN 10k training validation

Only start training after Gate 0 passes:

docker exec -it auto-thinker bash -lc \
  'CARLA_ROOT=/opt/carla \
   RUN_MODE=cnn \
   STEPS=10000 \
   SEED=42 \
   bash jobs/slurm_run.sh'

Monitor the GPU from another host terminal:

watch -n 2 nvidia-smi

Inspect outputs:

find outputs -maxdepth 4 -type f | sort | tail -40
tail -f outputs/carla_2000.log

Run the full CNN baseline

After the 10,000-step validation succeeds:

docker exec -it auto-thinker bash -lc \
  'CARLA_ROOT=/opt/carla \
   RUN_MODE=cnn \
   STEPS=500000 \
   SEED=42 \
   bash jobs/slurm_run.sh'

Optional JEPA and comparison runs

Phase 2 requires the comma2k19 dataset mounted at:

/work/data/comma2k19

After mounting the dataset, run:

docker exec -it auto-thinker bash -lc \
  'RUN_MODE=phase2 bash jobs/slurm_run.sh'

The transfer arms can be run after the required checkpoints are available:

docker exec -it auto-thinker bash -lc \
  'CARLA_ROOT=/opt/carla \
   RUN_MODE=custom_jepa \
   STEPS=10000 \
   SEED=42 \
   bash jobs/slurm_run.sh'

docker exec -it auto-thinker bash -lc \
  'CARLA_ROOT=/opt/carla \
   RUN_MODE=vjepa2 \
   STEPS=10000 \
   SEED=42 \
   bash jobs/slurm_run.sh'

Gate 0 closure criteria

This issue can be closed after the reviewer confirms:

• the container runs with a non-root UID;
• NVIDIA_DRIVER_CAPABILITIES=all is present;
• /dev/shm is 2 GiB;
• the NVIDIA ICD JSON is non-empty;
• vulkaninfo --summary succeeds;
• nvidia-smi works inside the container;
• python run.py doctor --require-cuda --require-carla passes;
• the smoke command prints:

[job] Smoke test passed; no training requested.

- the CNN 10k validation starts successfully.