rl_algos

Modular on-policy RL: PPO, AMP, WASABI. Pure PyTorch — no simulator dependency.

Ported and refactored from instinct_rl, designed to be driven by manager-based RL training (IsaacLab) via a thin wrapper living in rl_training.

Install

Isaac Sim 4.5 → Python 3.10.

pip install -e .
pytest tests/          # 55 tests, CPU only, ~22s, no simulator needed

Quick start

from rl_algos.env import DummyVecEnv
from rl_algos.runners import OnPolicyRunner

env = DummyVecEnv(num_envs=64, num_actions=4)
cfg = {
    "num_steps_per_env": 16,
    "policy": {
        "class_name": "ActorCritic",
        "actor_hidden_dims": [64, 64],
        "critic_hidden_dims": [64, 64],
        "activation": "elu",
        "init_noise_std": 1.0,
    },
    "algorithm": {
        "class_name": "PPO",
        "num_learning_epochs": 5,
        "num_mini_batches": 4,
        "learning_rate": 1e-3,
        "schedule": "adaptive",
        "desired_kl": 0.01,
    },
}
runner = OnPolicyRunner(env, cfg, log_dir="logs/run0", device="cpu")
runner.learn(150)

Swap "PPO" for "AMPPPO" or "WasabiPPO" and give the env amp_policy / amp_reference observation groups. Swap "ActorCritic" for "EncoderActorCritic" and add encoder_configs for camera input.

What's here

Algorithms	PPO, AMPPPO, WasabiPPO, BCEAMPPPO, QuadWasabiPPO
Models	ActorCritic, EncoderActorCritic (camera)
Blocks	MlpModel, Conv2dHeadModel
Storage	RolloutStorage (GAE), AdversarialStorage

Everything is reached by name from config (class_name), including classes outside this package ("my_pkg.module:MyClass"). Adding a policy, block, or algorithm requires no core edits — see PLAN.md and .claude/rules/.

Design in one paragraph

Observations travel as flat tensors described by an obs_format ({group: {component: shape}}), which maps 1:1 onto manager-based observation terms. Algorithms return a dict of named losses that the base update() sums with per-name coefficients — so a mixin adds a loss without touching the update loop. AMP and WASABI share one AdversarialMixin and differ in exactly three methods: the discriminator loss, the style-reward transform, and what the gradient penalty constrains.

Not supported yet

Recurrent policies, MoE, VAE, state estimators, transformers, distillation. These raise NotImplementedError rather than silently misbehaving. See PLAN.md §8.

License

BSD-3-Clause. Contains code derived from rsl_rl / instinct_rl (© 2021 ETH Zurich, Nikita Rudin; © 2021 NVIDIA CORPORATION) and normalization code © 2020 Preferred Networks, Inc. See licenses/.