Drivemind

A reproducible six-camera Graph-VQA and Agentic RL pipeline for autonomous-driving VLMs

Six synchronized cameras | scene-isolated Graph-VQA | CE/DPO | Mixture of LoRA | DAPO/SAPO | Graph Agentic RL

Demos

Daytime urban driving	Wet night with a crossing pedestrian

Both animations cover the complete 32-second sequence. Click either preview for the original 24 FPS MP4. See the technical report and resume-ready summary.

Overview

Drivemind preserves the official DriveLM-nuScenes question IDs, six-camera ordering and submission schema, then adds a complete open-weight VLM pipeline:

• deterministic 26,095 train / 3,355 scene-isolated dev QA construction with leakage checks;
• assistant-token-only CE-LoRA SFT and balanced, CE-anchored preference optimization;
• answer-independent routing over Perception, Prediction, Planning and Behavior LoRA experts;
• causal Graph trajectories with predicted, masked, shuffled and oracle context audits;
• GRPO, GSPO, DAPO and SAPO training under frozen promotion gates;
• calibrated reward modeling and multi-policy Graph Agentic RL;
• resumable inference, official-style local evaluation, same-ID checks and challenge JSON export.

This is a runnable modern equivalent, not a parameter-identical reproduction of the paper's unpublished BLIP-2 stack. The active benchmark is single-frame, six-camera and camera-only: radar, LiDAR, map labels, history frames and dev references are not model inputs.

Pipeline

The solid green path is the promoted route. Blue branches are completed research stages; dashed orange branches are controlled experiments that did not replace the formal baseline.

Results

Official-style local leaderboard

The complete local leaderboard retains the public DriveLM columns: Accuracy, semantic judge, BLEU-1..4, ROUGE-L, CIDEr, Match and Final. The semantic column uses cached, temperature-zero DeepSeek-V4-Flash because the official GPT judge is unavailable, so these are local DriveLM-DS proxy scores, not hidden-server results.

Complete official-style leaderboard | machine-readable metrics | all experiment metrics

rank	method	accuracy	judge*	match	final_score	decision
1	Frozen Graph-A + four AuxDPO experts	0.813814	73.3716	30.9280	0.613725	Current baseline
2	CE-anchored hybrid DPO	0.813814	73.1498	30.9848	0.612952	Promoted
3	Graph-A anchor + MoL downstream	0.813154	73.0197	30.7513	0.612293	Promoted
4	Global null-space AuxDPO	0.814925	72.7156	30.9280	0.611632	Rejected by full-system gate
5	Pure Graph-CE	0.802691	71.2974	34.0947	0.609998	Rejected by same-ID gate
6	Mixture of LoRA experts	0.808735	72.4769	30.7513	0.608245	Promoted component

judge* is the DeepSeek proxy for the official chatgpt column. Candidate rank and promotion are deliberately different: a model advances only after 100% coverage, complete semantic judging, regression limits and an exact same-ID audit pass.

End-to-end promotion chain

stage	promoted method	local DriveLM-DS Final	outcome
Raw VLM	Qwen2.5-VL-7B zero-shot	0.257961	Control
Domain SFT	Six-camera CE-LoRA	0.594636	Promoted
Preference tuning	Task-balanced CE-anchored DPO	0.596356	Promoted
Expert specialization	Four-task Mixture of LoRA	0.608245	Promoted
Graph routing	Graph-A anchor + MoL downstream	0.612293	Promoted
Hybrid preference tuning	CE-anchored hybrid DPO	0.612952	Promoted
Expert-wise AuxDPO	Frozen Graph-A + four downstream experts	0.613725	Current formal baseline

GRPO/GSPO, Graph-CE, global-null-space AuxDPO and Graph Agentic RL remain controlled results rather than silent substitutions into the promoted route. Their comparable and diagnostic records are in the unified metrics table.

Method evolution and commands

The labels below describe methods rather than internal version numbers. Commands assume the repository root; replace machine-specific model, data and output paths as documented in each method directory.

1. Dataset preparation and six-camera SFT

DriveLM graph nodes are flattened into supervised QA records while scene boundaries remain intact. Each sample receives six synchronized cameras in a fixed order, and CE is applied only to assistant answer tokens.

python reproduction/qwen_vl/build_dataset.py \
  --train-json data/QA_dataset_nus/v1_1_train_nus.json \
  --val-json data/QA_dataset_nus/v1_1_val_nus_q_only.json \
  --output-dir data/reproduction/qwen_vl --seed 42 --dev-ratio 0.1

bash reproduction/qwen_vl_v036/posttrain_b10.sh

2. Task-balanced CE-anchored DPO

Preference pairs are balanced across four task families. A chosen-answer CE anchor limits language drift while frozen reference log-probabilities, full-dev sweeps and same-ID gates prevent preference over-optimization.

python reproduction/qwen_vl_v037b/build_balanced_preferences.py \
  --input-jsonl data/v037a/preferences_with_reference.jsonl \
  --output-jsonl data/v037b/balanced_preferences.jsonl --per-task 1026

accelerate launch --multi_gpu \
  reproduction/qwen_vl_v037b/train_anchored_dpo_ddp.py \
  --experiment-name anchored-dpo-seed42 \
  --model-path models/Qwen2.5-VL-7B-Instruct \
  --adapter-path models/six-camera-ce-lora \
  --reference-jsonl data/v037b/balanced_preferences.jsonl \
  --output-dir models/anchored-dpo \
  --learning-rate 1e-6 --beta 0.05 --chosen-ce-weight 0.1 \
  --max-steps 100 --save-steps 25 --seed 42

See the complete DPO arguments and gate sequence.

3. Mixture of LoRA experts

One frozen VLM backbone hosts four task adapters. The official task key routes each question without inspecting answers or dev labels; an adaptive sweep stops when validation no longer improves.

python reproduction/qwen_vl_v039_mol/build_mol_manifests.py --help
bash reproduction/qwen_vl_v039_mol/run_adaptive_controller.sh

4. Planning trajectory RL with GRPO and GSPO

Frozen Perception and Prediction outputs form leakage-free Planning context. Both algorithms share the same deterministic reward and undergo Planning-only selection plus a 3,355-QA full-system audit.

bash reproduction/qwen_vl_v040_trajectory_rl/run_v040_pipeline.sh
bash reproduction/qwen_vl_v040_trajectory_rl/run_final_eval.sh

5. Qwen3.8-Max API full zero-shot evaluation

The API model uses the same six-camera prompts at temperature zero, with no domain fine-tuning and no access to reference answers. It is a full zero-shot control and never participates in checkpoint promotion.

python reproduction/qwen_api_v041/infer_api.py --help

6. Graph trajectory CE and fixed anchor routing

Frame-level QAs are reconnected as Perception -> Prediction -> Planning -> Behavior conversations. Graph-CE improves the first important-object anchor but can hurt Planning under predicted context, so the retained route uses the Graph-trained adapter for Graph-A and delegates downstream nodes to MoL experts.

bash reproduction/qwen_vl_v042_graph/run_graph_sft.sh
python reproduction/qwen_vl_v043_internvl_transfer/build_fixed_ensemble.py --help

7. Hybrid preference optimization and AuxDPO

Mixed candidate sources reduce chosen/rejected distribution mismatch, semantic validation handles non-unique open answers, and CE anchors preserve direct image-to-answer ability. The accepted composition freezes Graph-A and updates only the four downstream experts.

bash reproduction/qwen_vl_v045_gvqa_auxdpo/run_candidate_generation.sh
bash reproduction/qwen_vl_v045_gvqa_auxdpo/run_controller.sh

8. Context-causal Graph RL: DAPO versus SAPO

The Prediction edge is trained under predicted, masked, shuffled and oracle contexts. Both pass the edge-level causal gate; DAPO gives the stronger routed full-dev gain under the frozen protocol.

bash reproduction/qwen_vl_v050_graph_dapo/run_training.sh
bash reproduction/qwen_vl_v051_graph_sapo/run_training.sh

9. Reward modeling and Graph Agentic RL

A single calibrated reward model scores semantic Prediction/Planning quality while verifiable nodes keep deterministic rewards. Five policy adapters execute one sampled path, consume actual upstream outputs and receive reward-to-go. The current candidate improves predicted-context quality but is rejected because independent Planning regresses.

bash reproduction/qwen_vl_v054b_reward_calibration/run_pipeline.sh
bash reproduction/qwen_vl_v055_graph_agentic_rl/run_full_training.sh
bash reproduction/qwen_vl_v055_graph_agentic_rl/run_selected_full_evaluation.sh

Quick start

Environment

git clone git@github.com:1zhiCheng/Drivemind.git
cd Drivemind
conda create -n drivemind python=3.11 -y
conda activate drivemind
python -m pip install -r requirements.txt

Install a CUDA-compatible PyTorch build before the remaining requirements.

Data and model

Never commit Hugging Face or evaluator credentials.

hf auth login
hf download OpenDriveLab/DriveLM \
  drivelm_nus_imgs_train.zip drivelm_nus_imgs_val.zip \
  v1_1_train_nus.json v1_1_val_nus_q_only.json \
  --repo-type dataset --local-dir downloads

hf download Qwen/Qwen2.5-VL-7B-Instruct \
  --local-dir models/Qwen2.5-VL-7B-Instruct
python scripts/check_setup.py

The audit must report six camera directories and ready: true. See data preparation.

Infer and evaluate

python reproduction/qwen_vl/infer.py \
  --model-path models/Qwen2.5-VL-7B-Instruct \
  --adapter-path models/drivemind \
  --input-jsonl data/reproduction/qwen_vl/qwen_dev.jsonl \
  --output-json outputs/dev_predictions.json \
  --device cuda:0 --max-new-tokens 256 --resume

python reproduction/qwen_vl/evaluate_offline.py \
  --references-jsonl data/reproduction/qwen_vl/qwen_dev.jsonl \
  --predictions-json outputs/dev_predictions.json \
  --output-json outputs/dev_metrics.json

For the local semantic proxy, keep the DeepSeek key outside the repository and run reproduction/drivelm_ds_eval/evaluate.py. Report it only as a local DriveLM-DS result.

Project status

• Dataset preparation, leakage audit and six-camera SFT baseline.
• Balanced DPO, Mixture of LoRA, Graph-SFT and fixed Graph routing.
• GRPO/GSPO and context-causal DAPO/SAPO comparisons with frozen promotion gates.
• Calibrated reward model and end-to-end Graph Agentic RL pilot/full evaluation.
• Improve independent Planning retention, rerun the semantic gate and submit the official challenge JSON.

Repository layout

challenge/                     official extraction and submission tools
reproduction/qwen_vl/         dataset builder, SFT, inference and lexical evaluation
reproduction/qwen_vl_v03*/    CE, DPO, grounding and Mixture-of-LoRA stages
reproduction/qwen_vl_v04*/    trajectory RL, Graph-SFT and hybrid/AuxDPO stages
reproduction/qwen_vl_v05*/    DAPO, SAPO, reward modeling and Graph Agentic RL
reproduction/drivelm_ds_eval/ local official-style structural/semantic evaluator
results/                       official leaderboard and unified experiment metrics
docs/                          reproduction, method and demo documentation
reports/                       illustrated technical reports
assets/                        architecture diagram, GIF previews and MP4 demos

Limitations

• The scene-isolated local split supports reproducible model selection; official validation answers are hidden.
• Published models use visual key frames rather than raw temporal camera streams.
• Coordinate grounding, causal context use and independent Planning retention remain bottlenecks.
• Datasets and model weights are intentionally excluded from Git.

Acknowledgement and citation

Built on DriveLM, nuScenes and Qwen2.5-VL. Please cite the original projects. The DriveLM citation is in CITATION.cff.

License

Code is released under Apache-2.0. DriveLM data, nuScenes images and model weights retain their own licenses.