From Language to Physics: Validating LLM-Generated Robot Plans Under Physical Constraints

Large language models can now produce syntactically coherent, semantically plausible action plans for embodied agents — sequences such as pick up the wrench, navigate to the engine bay, apply torque to the bolt. These plans parse cleanly and often align with commonsense priors. They frequently fail under physical execution: payload limits exceeded, targets occluded or out of reach, gripper aperture insufficient, torque requirements unmet, or action preconditions violated by incorrect sequencing.

The failure mode is structural, not incidental. LLMs optimize for linguistic coherence; robot execution requires satisfaction of spatial, kinematic, dynamic, temporal, and resource constraints over a grounded world state. Self-critique from the generating model tends to reproduce the same class of errors. Full physics simulation can detect execution failure but typically lacks structured violation typing, minimal repair proposals, and benchmark-grade evaluation against annotated ground truth.

We frame the core research question as follows: given a candidate plan P = (a₁, …, aₙ), a natural-language task specification τ, and a symbolic world-state description W, identify the subset of steps that are physically infeasible, classify each violation under a shared ontology, and propose minimal repairs R such that the revised plan P′ satisfies the constraints checked by the validation pipeline.

Praxis is our open-source framework for this problem (MIT license). Source, benchmark, and paper: https://github.com/Phukon-Industries/praxis. It is not a planning algorithm and does not claim theoretical novelty. It integrates modular constraint validation, minimal unsatisfiable subset (MUS/IIS) analysis, and model-based diagnosis into a pipeline targeted at LLM-generated high-level plans — a setting where language-level plausibility and physical feasibility systematically diverge.

The validation stage composes domain-specific checkers over each plan step against W. Spatial validators verify object existence, containment, and reachability. Kinematic validators check gripper aperture, grasp feasibility, and workspace limits. Dynamic validators enforce force, velocity, acceleration, and payload bounds. Temporal validators track object custody and action preconditions — e.g., a place action requires prior pick of the same object instance. Resource validators model energy consumption and battery sufficiency over the plan horizon. Each validator emits a structured Violation record: ontology category, leaf constraint type, severity, affected step index, and a human-readable explanation suitable for downstream repair or logging.

Violations are typed under the Physical Feasibility Ontology (PFO): nine root categories — Spatial, Kinematic, Dynamic, Payload, Temporal, Sensor, Environmental, Safety, Resource — decomposed into 19 leaf constraint types (object existence, reachability, gripper aperture, payload mass, surface stability, collision risk, battery sufficiency, and others). The ontology serves dual roles: as a programmatic scaffold mapping validators to constraint nodes, and as a shared vocabulary for comparing infeasibility across systems, benchmarks, and papers.

The repair stage consumes the violation set and proposes localized edits rather than full replanning. A rule-based repairer handles recurrent patterns: inserting missing pick steps before place, removing duplicate actions, substituting objects within payload limits, re-ordering steps to restore temporal consistency. For violations outside the rule coverage envelope, an LLM-assisted repairer conditions on the structured violation report, τ, and W to generate targeted step-level fixes — constrained by the original task intent rather than open-ended plan generation.

The evaluation stage scores detection (precision, recall, F1 against ground-truth violation labels), explanation accuracy (correct ontology category and causal attribution), repair coverage (whether the proposed edit resolves the annotated violation), and a composite overall score S = 0.5·F₁ + 0.25·E + 0.25·R. In safety-critical deployment contexts, false negatives in detection dominate the error cost; we report recall alongside precision for this reason.

PraxisBench accompanies the framework: 280 annotated YAML scenarios spanning 12 task domains (kitchens, warehouses, workshops, offices, hospitals, factories, construction sites, laboratories, farms, gardens, restaurants, and homes) and seven robot platforms (Franka Panda, UR5, UR10, KUKA iiwa, TIAGo, Fetch, and Spot). The release includes 15 feasible plans to measure false-positive rates, 18 multi-category cases with interacting constraints, and 323 ground-truth violations with 310 reference repair annotations. Each scenario contains τ, W, a candidate plan (typically seeded with deliberate violations), labeled violations with ontology identifiers, and ground-truth repairs. Mean plan length is 2.93 steps (821 steps total across the benchmark).

PraxisBench isolates feasibility reasoning as a measurable capability distinct from end-to-end task success — evaluating whether a system can detect violations, explain their type, and propose fixes. A Python evaluation harness (praxisbench.harness.EvaluationHarness) automates loading scenarios, running a user-specified pipeline, computing per-scenario and aggregate metrics with optional bootstrap confidence intervals, and saving structured results for reproducible comparison.

On the 210 scenarios whose entire ground-truth violation set lies within categories with dedicated rule-based validators (spatial, kinematic, dynamic, temporal, resource), Praxis achieves detection F1 of 0.75 (95% CI [0.73, 0.77]), explanation accuracy of 0.99, repair coverage of 0.97, and overall score 0.87. Per-category detection F1 on the full benchmark ranges from 0.67 (resource) to 0.81 (spatial) where validators exist. Evaluated on all 280 scenarios — including violations in sensor, environmental, safety, and payload categories without dedicated rule-based checkers — overall detection F1 drops to 0.58, which delineates the current coverage boundary rather than hiding it.

We cross-validated all 280 scenarios against PyBullet simulation. For physics-grounded constraint types (spatial, kinematic, dynamic), per-step agreement between analytical checks and simulation is 83.3% (684/821 steps), with 194/280 scenarios in full agreement. Overall per-step agreement is lower (56.2%) because Praxis also flags semantic constraints — sensor visibility, environmental conditions, human safety — that our simulation backend does not model. When simulation failure is treated as proxy ground truth for physics-grounded checks, precision is 0.78, recall 0.92, and F1 0.84. The rule-based pipeline averages 0.047 ms per scenario; the PyBullet pass averages 22.7 ms per scenario — roughly 480× faster analytical screening on the same hardware.

Implementation details: validators subclass a single abstract base with a validate(plan, world_state) interface; the pipeline composes checkers dynamically and supports sequential rule-based and LLM-assisted passes. All interchange structures use Pydantic models. The package installs via pip, exposes a CLI for single-scenario validation, and carries no hard dependency on a specific simulator or robot middleware.

Praxis is MIT-licensed with full paper source under paper/ in the repository: https://github.com/Phukon-Industries/praxis. We welcome contributions: new validators, benchmark scenarios, platform-specific integrations, and empirical evaluation on real hardware. Contact: phukonindustries@protonmail.com.