Questions after finetune: how to generate sport instructions from two motions, and how to train with video input?

[Leocoding0971]

Hi, thanks for releasing the code!

I have finished finetuning with the skating_gt setting. I have two questions:

After finetuning (skating_gt), how can I input two motion sequences (learner vs. reference) to generate Sport Instruction outputs?
Is there an inference script or a recommended way to run generation with two motion inputs?

I saw in previous issues that raw videos cannot be released for ethical reasons.
If I want to train a video-input version of the model, what is the expected pipeline?
For example, should I extract poses first and then use this repo as-is, or is there any code/config for video-based training that is not in this release?

Any pointers or example scripts would be very helpful. Thanks

[weihsinyeh]

We extracted poses using HybrIK.

Since the learner and reference videos may have different frame lengths, we adopted CARL as the concept encoder to obtain frame embeddings. This concept encoder was independently trained on skating datasets with ground-truth segmentation annotations.

Next, we perform video alignment to determine the start and end frames by first finding the pair of segments with the minimum embedding distance, and then running inference on those aligned segments.

Finally, run evaluation.py with your test_pkl_file.