physicalai-train-benchmarking-a-policy
Benchmarks a trained Physical AI Studio policy in a simulation gym and reports success metrics. Use when running physicalai benchmark, editing configs under library/configs/benchmark, adding or changing a Benchmark class in physicalai.benchmark, tuning rollout/episode/env settings, recording rollout videos, or interpreting results.json / results.csv.
How do I install this agent skill?
npx skills add https://github.com/open-edge-platform/physical-ai-studio --skill physicalai-train-benchmarking-a-policyIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill is safe and provides standard documentation and workflow instructions for benchmarking machine learning policies within the physicalai framework.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Benchmarking a Studio Policy
Benchmarking evaluates a trained policy by rolling it out in a gym and scoring success. Benchmark classes live in library/src/physicalai/benchmark/gyms/benchmark.py (Benchmark, PushTBenchmark, LiberoBenchmark); results types in benchmark/gyms/results.py (BenchmarkResults, TaskResult); rollout logic in library/src/physicalai/eval/rollout.py (evaluate_policy). The library supports both direct Python API use and the physicalai benchmark CLI wrapper (library/src/physicalai/cli/benchmark.py).
Python API invocation
Use this path for notebooks, tests, custom scripts, or direct library integrations.
from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.policies import ACT
policy = ACT.load_from_checkpoint("experiments/act/version_0/checkpoints/last.ckpt")
benchmark = PushTBenchmark(num_episodes=1)
results = benchmark.evaluate(policy)
print(results.summary())
results.to_json("results/benchmark/results.json")
results.to_csv("results/benchmark/results.csv")
For exported artifacts, load the Runtime-facing model first:
from physicalai.benchmark.gyms import PushTBenchmark
from physicalai.inference import InferenceModel
model = InferenceModel("./exports/act_policy")
results = PushTBenchmark(num_episodes=1).evaluate(model)
CLI invocation
physicalai benchmark \
--config configs/benchmark/pusht.yaml \
--policy physicalai.policies.ACT \
--ckpt_path experiments/act/version_0/checkpoints/last.ckpt \
--output_dir ./results/benchmark
--policy— policy class path.--ckpt_path— a.ckptor an export directory.--config— a benchmark config (configs/benchmark/pusht.yaml,configs/benchmark/libero.yaml) selecting theBenchmarkclass and its settings.--output_dir— defaults to./results/benchmark.
Override benchmark settings on the CLI, e.g. --benchmark.num_episodes 10 --benchmark.num_envs 8.
Output
- Prints
results.summary()to stdout. - Writes
results.jsonandresults.csvinto--output_dir. - Optional video via config
video_dir+record_mode(all|failures|successes|none).
Workflow
- Choose API or CLI deliberately. Use the Python API for code-level tasks; use CLI for config/docs/entry-point tasks.
- Done when: the selected path matches the user's requested surface area.
- Confirm the policy loads from the checkpoint/export before a full sweep:
physicalai benchmark --config configs/benchmark/<suite>.yaml --policy <ClassPath> --ckpt_path <path> --benchmark.num_episodes 1- Done when: one episode runs end-to-end and a summary prints.
- Run the full benchmark with the intended episode/env counts.
- Done when:
results.jsonandresults.csvare written and the success metric is populated.
- Done when:
- Interpret results via
BenchmarkResults/TaskResultfields; compare against a baseline checkpoint on the same config. - Record videos for qualitative review when a task regresses (
record_mode: failures).
Adding or changing a Benchmark
- Subclass
Benchmarkinbenchmark/gyms/(studyPushTBenchmark/LiberoBenchmark); the gym itself comes fromphysicalai.gyms(pusht.py,libero.py, …). - Add a matching config in
library/configs/benchmark/. - Add tests under
library/tests/unit/benchmark/.- Done when:
uv run pytest tests/unit/benchmarkpasses and a 1-episode run succeeds.
- Done when:
Required checks
- The policy runs from both a
.ckptand an export dir if both are supported paths. - The Python API path (
Benchmark(...).evaluate(...)) and CLI wrapper agree on supported inputs for user-facing benchmark changes. - Success/episode metrics are populated (not zero/NaN by accident) and reproducible across runs.
- Env/episode counts match hardware; large
num_envsfits memory. - Heavy gym deps (e.g.
libero,robocasa) are gated behind their optional extras and imported lazily.
Related skills
physicalai-train-training-a-policy— to produce the checkpoint being benchmarked.physicalai-train-exporting-and-validating— when benchmarking an exported artifact for deployment parity.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/open-edge-platform/physical-ai-studio/physicalai-train-benchmarking-a-policy">View physicalai-train-benchmarking-a-policy on skillZs</a>