A reproducible, non-clinical case study of adapting Qwen/Qwen3.5-2B to
Chinese medical exam data with text-only LoRA.
The repository covers the full prototype path: pinned input download, dataset admission, train/test isolation, single-GPU LoRA training, paired base-versus-adapter evaluation, fixed dialogue checks, and a terminal demo. It intentionally does not include model weights, adapters, raw datasets, or large predictions.
Not a clinical system. Do not use this project for diagnosis, treatment, emergency care, prescribing, or clinical decision support.
| Item | Verified result |
|---|---|
| Base model | Qwen/Qwen3.5-2B at revision 15852e8c...a8fc |
| Training data | 27,516 public CMB rows + 384 disclosed replay rows |
| Train/test exact normalized overlap | 0 across all 11,200 CMB test questions |
| Training | 1 epoch, 872 optimizer steps, 27.71 minutes |
| Hardware | 1x NVIDIA RTX 4090 24GB |
| Peak allocated VRAM | 13.996 GiB |
| CMB-560 base | 274/560, 48.93% |
| CMB-560 adapter | 332/560, 59.29% |
| Paired change | +10.36 percentage points |
| Fixed critical dialogue checks | Adapter passed 7/7 after rule screening and manual review |
The CMB result is from a frozen 560-question subset and an exact answer parser. It is evidence of task-specific adaptation, not a full-benchmark, clinical, or safety result. The seven dialogue checks are fixed-prompt observations, not a safety certification. The figure combines the non-medical and exact-output checks in one display row; the protocol reviewed them as two criteria.
Numeric source: presentation/results.json.
Detailed interpretation: docs/FINAL_BRIEF.md.
- Data rejection as evidence. Huatuo26M-Lite remained factually unreliable after multiple filters and manual review, so it was excluded from training.
- A frozen comparison. Base and adapter use the same prompts, decoding, parser, CMB subset, and evaluation code.
- Measured engineering cost. Training time, peak VRAM, throughput, steps, losses, revisions, and artifact hashes are recorded.
- Separated claims. Exam accuracy, dialogue behavior, and application boundaries are reported independently.
- Research provenance. Failed routes, decisions, experiment configs, and reproducibility pitfalls remain inspectable.
The lightweight checks do not install PyTorch or download any model:
python -m venv .venv
source .venv/bin/activate
python -m pip install pytest ruff
export PYTHONPATH=src
python -m pytest -q
python -m ruff check src tests scripts
python scripts/validate_repository.pyThe final command checks relative Markdown links, JSON/JSONL syntax, expected headline values, accidental model artifacts, oversized files, and known private path/hostname patterns.
Full reproduction requires Linux, Bash, an NVIDIA CUDA environment, access to the pinned Hugging Face inputs, and enough local storage for dependencies, model files, datasets, and outputs. The recorded run used one RTX 4090 24GB; smaller memory configurations were not validated.
cp .env.example .env
# Edit .env with machine-local paths.
set -a
source .env
set +a
bash scripts/bootstrap_gpu.sh
bash scripts/run_data.sh
bash scripts/run_replay_data.shThe final CMB-only route is the default. The rejected Huatuo route is retained only as historical evidence and is not downloaded unless explicitly requested.
After reviewing the generated manifests and manual-review samples:
EXPERIMENT_ID=TRAIN-003-qwen35-2b-cmb-replay-1epoch \
bash scripts/run_train.sh
EXPERIMENT_ID=EVAL-005-train003-formal-paired \
ADAPTER_ROOT="${MEDSLM_OUTPUTS}/TRAIN-003-qwen35-2b-cmb-replay-1epoch/adapter" \
bash scripts/run_paired_eval.sh
bash scripts/run_demo.shDo not start a promoted run blindly. The exact stages, expected hashes,
inspection gates, commands, and interpretation rules are in
docs/REPRODUCE.md.
Pinned public inputs
-> deterministic filtering and sampling
-> zero-overlap audit against all CMB test questions
-> 10-step save/reload smoke
-> one-epoch text-only LoRA
-> same-run base vs adapter CMB-560 evaluation
-> fixed dialogue screening and full manual review
-> terminal demo with an explicit educational-use boundary
The final adapter used assistant-only loss, BF16, SDPA, LoRA rank 16, effective
batch size 32, learning rate 5e-5, and maximum sequence length 512.
| Path | Purpose |
|---|---|
src/medical_slm/ |
Download, data preparation, training, evaluation, and demo code |
scripts/ |
Environment-driven launch and repository validation |
assets/ |
Fixed qualitative prompts and 32 reviewed replay records |
registry/ |
Immutable experiment configs, lifecycle tags, and metric summaries |
memory/ |
Decisions, dataset ledger, failures, pitfalls, and experiment conclusions |
docs/ |
Reproduction guide, experiment report, and historical route survey |
presentation/results.json |
Machine-readable source for the public headline numbers |
For a short reading path:
docs/FINAL_BRIEF.mddocs/REPRODUCE.mdmemory/DATASETS.mdmemory/FAILED_ATTEMPTS.mdregistry/config/TRAIN-003-qwen35-2b-cmb-replay-1epoch.yaml
This project is a compact engineering and evaluation prototype. It does not establish:
- clinical correctness, general medical safety, or physician-level behavior;
- effectiveness outside the pinned model, data revisions, and prompts;
- statistical significance or robustness across seeds, model sizes, or GPUs;
- redistribution rights for upstream model weights or datasets;
- that replay-trained boundary behavior generalizes beyond the fixed checks.
The 384 replay rows are deterministic copies of 32 project-authored, human-reviewed examples. They are disclosed separately from public medical knowledge data and must not be described as safety evidence.
The memory/ and registry/ layers are a real, desensitized example of the
gated workflow implemented by
ResearchMemoryKit. Medical SLM is
useful on its own; ResearchMemoryKit explains how the project retained current
state, negative results, decisions, and completion evidence during agent-assisted
execution.
Citation metadata is provided in CITATION.cff. Cite the
upstream Qwen and CMB sources separately when reusing their model or data.
Code in this repository is released under the MIT License. Upstream models and datasets remain subject to their own licenses.