[Feature]: Interleave training and inference jobs when they share a single GPU

[calebrob6]

Problem Statement

Training and inference are served by two independent queue triggers in api/hastefuncqueues/function_app.py — GetCreateModelRunQueueTrigger (train_queue_name) and GetRunInferenceQueueTrigger (inference_queue_name) — but both submit work to the same Azure Batch training-pool. When that pool has a single GPU, there is nothing coordinating the drain order across the two queues.

The result is that each queue drains independently and training work starves the inference work that depends on it. With multiple training runs queued, the observed order is:

Training job 1
Training job 2
Inference job 1
Inference job 2

whereas the useful order is:

Training job 1
Inference job 1
Training job 2
Inference job 2
...

The practical impact is that a user who queues several runs gets no usable inference output until every training job in the queue has finished. Completing one run end-to-end is almost always more valuable than getting all the training done first, since the first set of predictions is what unblocks review.

Proposed Solution

Coordinate scheduling across the training and inference queues so a run reaches inference before the next training job claims the GPU. Options worth evaluating:

• Single ordered work queue for GPU jobs, with a job-type discriminator, so ordering is explicit rather than emergent from two independent triggers.
• Priority scheduling where inference messages outrank queued training messages, so a freed GPU picks up pending inference first.
• Chained submission — hold the next training job until the inference triggered by the previous training run has been submitted, making the pipeline explicitly run-at-a-time.
• Azure Batch job priority on the training-pool tasks, if pool-level ordering is sufficient without changing the queue topology.

Alternatives Considered

• Scale the pool so training and inference no longer contend for one GPU. Sidesteps the ordering problem rather than fixing it, and doesn't help single-GPU deployments.
• Manual operator sequencing (pause the training queue while inference drains). Not viable as a standing workaround.

Area

API (Azure Functions)

Additional Context

Relevant code:

• api/hastefuncqueues/function_app.py — GetCreateModelRunQueueTrigger (~L257) and GetRunInferenceQueueTrigger (~L604)
• api/hastefuncqueues/host.json — currently batchSize: 1, newBatchThreshold: 0, which serializes each queue individually but does nothing across queues
• api/hastefuncqueues/README.md — documents both functions targeting training-pool

Worth confirming as part of the fix whether the same contention affects the embedding queue (GetRunEmbeddingQueueTrigger), which may also share GPU capacity.

Checklist

• I have searched existing issues and this has not already been requested.