Conversation
Integrate G1/Franka/PiperGo2 simulation configs, watchdog routing and resilience improvements, embodied action updates, and bundled robot/scene assets so the end-to-end onboard workflow runs out-of-the-box. Made-with: Cursor
Feat/multi robot sim integration : Add the function of simulation part
- Introduced a new runtime watchdog supervisor to manage session execution. - Added session registry for tracking session states and transitions. - Implemented error handling with a dedicated error hierarchy for runtime failures. - Created result writer to handle writing runtime results to artifacts and environment summaries. - Added templates for session, skill, and target definitions. - Developed initialization script for setting up runtime workspace files. - Implemented tests for session management, supervisor functionality, and environment handling.
# Conflicts: # .gitignore
…implement runtime watchdog components - Added abstract methods to BaseRolloutTarget for better contract enforcement. - Introduced FailureEscalator and HealthMonitor for session error handling and preflight checks. - Implemented session scheduling with priority handling in SessionScheduler. - Created WorkspaceWatcher for monitoring changes in runtime protocol files. - Updated README and README_zh for new features and usage instructions.
- Added `plugin_pipeline.py` to execute perception plugins in a pipeline. - Introduced `preflight.py` for strict checks on perception configurations. - Created `sensor_frame.py` to define the structure of sensor frames. - Developed `sensor_frame_builder.py` to build sensor frames from observations. - Defined new schemas in `environment.py` and `perception.py` for environment and perception data. - Established sensor configuration schemas in `sensor_config.py`. - Added example perception and sensor configuration YAML files for dummy simulation. - Implemented comprehensive tests for perception runtime, including preflight checks and environment writing.
- Introduced a new runtime contract for the dummy simulation in `dummy_sim.runtime.yaml`, defining action contracts, safety requirements, and capabilities. - Implemented local and remote target design - Updated protocol file format - Optimized the watchdog to perform a unified preflight function - Modified the behavior of the target and adapter, including local/remote targets, TargetAdapter / PolicyAdapter / ActionBridge. - Implemented unit tests for runtime protocol alignment, ensuring compatibility and validation of runtime contracts and adapter plans.
- Updated target specifications to use `target_class` and `target_kind` instead of `type` and `backend`. - Modified skill specifications to include `runtime_kind` and `loop_mode`, aligning with the new structure. - Introduced a new session management system with `SessionRunner` and `TargetSessionHandle` for better encapsulation of session logic. - Added new data models for session management, including `RuntimeObservation`, `EnvironmentRequest`, and `SkillContext`. - Implemented `PolicySkillRuntime` and `BuiltinSkillRuntime` as base classes for skill runtimes, enhancing modularity. - Updated tests to reflect changes in target and skill specifications, ensuring compatibility with the new runtime structure. - Created a README for runtime skills to clarify organization and usage.
…eatures and timeout handling
- Removed the test file for XLerobot2WheelsRemoteDriver. - Added Minecraft target adapter in PhyAgentOS, including: - `MinecraftTargetAdapter` for converting protocol observations to runtime format. - `MinecraftTarget` for connecting to a mineflayer bridge and handling actions. - `MinecraftSkillRuntime` for executing episodes in Minecraft without VLA policy. - Introduced necessary modules and methods for observation, action handling, and session management. - Added framework and legacy asset images to documentation.
- Implemented a new websocket server for LeRobot PI0/PI0.5 checkpoints in `lerobot_pi0_server.py`. - Added functions for observation and action processing, including conversion to numpy arrays. - Integrated msgpack serialization for efficient data transfer in `msgpack_numpy.py`. - Created documentation for the execution flow of BuiltinSkillRuntime and PolicySkillRuntime.
…ion and batch processing
…tion, development guidelines, code style, and contribution rules
…pervision - Added `RuntimeWorkspaceManager` to provision runtime workspace files from agent config and user-authored templates. - Introduced `RuntimeWorkspaceReport` for reporting workspace provisioning results. - Created templates for runtime contracts and workspace files including `TARGETS.md`, `SKILLS.md`, `SESSIONS.md`, and `ENVIRONMENT.md`. - Implemented a `BackgroundWatchdog` class to run the `WatchdogSupervisor` in a daemon thread. - Added scripts for end-to-end testing with real LIBERO target and PI0.5 policy server, including `run_pi05_libero_real_e2e.py` and `run_pi05_libero_sweep.py`. - Included a YAML configuration for the LIBERO target in `libero_real.runtime.yaml`.
…ntime compatibility checks
feat: update runtime workspace and adapter contracts
…nd action handling
- Created usage guide in English and Chinese for PhyAgentOS × Minecraft, detailing terminal control, in-game chat, bot teleporting, script references, troubleshooting, and code changes. - Introduced a viewer HTML file for real-time monitoring of the Minecraft bot's state, including position, health, and nearby entities. - Added deployment documentation in Chinese, covering architecture overview, environment requirements, Windows deployment steps, and action space for bot interactions.
…nd maintainability,
- Renamed `SKILLS.md` to `SKILLRUNTIME.md` to better reflect its purpose. - Updated references in the codebase to align with the new naming convention. - Introduced a new schema for skill runtimes in `PhyAgentOS/runtime/schemas/skillruntime.py`. - Created a new directory structure for skill runtimes under `PhyAgentOS/runtime/skillruntime/`. - Updated deployment scripts to accommodate the new skill runtime structure. - Revised the `SESSIONS.md` format to use YAML instead of JSON for better readability. - Enhanced the `init_runtime_workspace.py` script to initialize the new `SESSIONS.md` format. - Added a new README for the skill runtime directory to clarify usage and organization. - Updated various scripts to reference the new skill runtime structure and naming conventions.
# Conflicts: # PhyAgentOS/agent/context.py # PhyAgentOS/agent/tools/embodied.py # PhyAgentOS/cli/minecraft_commands.py # PhyAgentOS/runtime/skillruntime/game/minecraft_skill_runtime.py # PhyAgentOS/runtime/watchdog/runtime_registry.py # PhyAgentOS/runtime/workspace/manager.py # PhyAgentOS/templates/SKILLS.md # PhyAgentOS/templates/TARGETS.md # PhyAgentOS/templates/TOOLS.md # PhyAgentOS/templates/configs/runtime/contracts/minecraft.runtime.yaml # PhyAgentOS/templates/configs/runtime/embodied/minecraft.md # README.md # README_zh.md # docs/scenarios/game/minecraft/0_start.md # docs/scenarios/game/minecraft/2_agent_loop.md
…y requirements and add a benchmark skill.
Pre commit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.