Order packed microbatches largest-first per DP rank

[wdykas]

What does this PR do ?

This PR makes it so the largest micro batches go first in packed sequence training. This is an optimization for memory. If go with the default path now where we blow up reserved memory because we keep needing new reservations for larger shapes. Then when stuff like collective buffers try to establish we can hit ooms even though we have low active memory

On nano run before:
162,286 MiB (158.5 GiB) PyTorch reserved. The job failed with only 19,191 MiB (18.7 GiB) live
After:
72,916 MiB (71.2 GiB) maximum reserved, with a 63,997 MiB (62.5 GiB) live peak

Issues

List issues that this PR closes (syntax):

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

• ...

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.