PARLAM: A Large Action Model for General-Purpose Robotics

Palo Alto Robotics (PAR) — Technical Report v1.0

Abstract

We present PARLAM (Palo Alto Robotics Large Action Model), a foundation model for embodied intelligence that unifies perception, world modeling, planning, and control in a single architecture trained across heterogeneous robotic embodiments. PARLAM's central design commitment is a two-level universal action interface: (i) a discrete semantic action vocabulary — typed, parameterized action tokens with option semantics — produced autoregressively by a vision-language backbone, and (ii) a continuous latent motion code — a residual-vector-quantized, embodiment-agnostic representation of short-horizon motion learned from both action-labeled robot data and action-free internet video via inverse/forward-dynamics (IDM/FDM) objectives. Lightweight morphology-conditioned adapters decode latent motion into embodiment-specific control at 50–200 Hz, while the backbone plans at 3–10 Hz, yielding a dual-system architecture in the lineage of π0/GR00T-class vision-language-action (VLA) models but with an explicit, predictive world model in the loop: PARLAM is trained jointly as a world-action model, predicting future latent world states (JEPA-style) alongside actions, which supplies model-predictive fallback control, uncertainty estimation, and failure prediction. We formalize cross-embodiment control as amortized inference in a shared latent action space, give an identifiability argument for latent actions learned from passive video, and specify the full training stack: a four-tier data pyramid (web video → simulation → teleoperation → fleet telemetry), a composite objective combining flow-matching action generation, FAST-style frequency-domain action tokenization for autoregressive pretraining, latent-dynamics prediction, and contrastive cross-modal alignment; and a post-training phase of simulation RL and advantage-weighted fleet learning. We detail the safety architecture (conformal uncertainty gating, control-barrier-function shielding, simulation rehearsal) and the continuous fleet-learning loop over RobotNet, PAR's experience-sharing substrate, and propose scaling-law and evaluation protocols. Throughout, we position PARLAM against RT-2, OpenVLA, Octo, π0/π0-FAST, GR00T N1, LAPA/UniVLA, and V-JEPA 2-AC, arguing that the missing unification in current systems is a single latent language of motion shared by prediction, planning, and control.

1. Introduction

1.1 From token prediction to behavior prediction

Large language models demonstrated that a single sequence model, trained at sufficient scale on a universal token interface, absorbs and transfers knowledge across superficially unrelated tasks. Robotics has no such interface. Actions are emitted in per-robot coordinate frames, at per-robot rates, through per-robot controllers; consequently knowledge is trapped in the embodiment that produced it. The Open X-Embodiment effort showed that even naive pooling of 1M+ trajectories from 22 embodiments produces positive transfer when model capacity is sufficient (RT-2-X roughly tripling emergent-skill performance over its single-embodiment counterpart) [1, 2]; embodiment-scaling analyses further indicate that diversity of bodies can matter more than trajectories per body [3]. The lesson is that cross-embodiment structure exists and is learnable — but current VLAs inherit it implicitly, through co-training, rather than through an explicit representational commitment.

PARLAM makes that commitment explicit:

Hypothesis (Universal Motion Language). There exists a low-dimensional, discrete-continuous hybrid action representation 𝒵 = 𝒱_sem × 𝒵_motion such that (a) behavior across a wide class of embodiments factors through 𝒵 with small decoder heads, (b) 𝒵 is recoverable, up to reparameterization, from passive observation of behavior (video) without action labels, and (c) prediction of futures and generation of behavior can share 𝒵, making the world model and the policy two heads on one latent interface.

Language models predict tokens; world models predict futures; action models predict behavior. PARLAM is built on the observation that these are the same prediction problem posed over the right latent space.

1.2 Why current VLAs are insufficient

Modern VLAs fall into three architectural families, each with a characteristic deficiency:

1. Discretize-and-predict (RT-2 [2], OpenVLA [4]): continuous actions binned per-dimension into 256 tokens overwriting rare vocabulary entries of an LLM, predicted autoregressively. Simple and scalable, but per-step binning fails on high-frequency dexterous control — adjacent tokens carry near-zero marginal information, collapsing the next-token objective [7] — and the action space remains embodiment-specific (a 7-DoF delta-EEF convention).
2. Continuous heads (Octo's diffusion head [5], π0's flow-matching action expert [6], OpenVLA-OFT's parallel L1 regression [8], RDT-1B [9]): expressive multi-modal action chunks at high control rates, but the action interface is again a fixed per-embodiment vector convention; cross-embodiment sharing happens only in upstream features.
3. Latent-action pretraining (Genie's latent action model [10], LAPO, LAPA [11], UniVLA [12], UniAct [13]): VQ-learned latent actions from unlabeled video give a genuinely embodiment-agnostic pretraining signal — LAPA outperforms label-dependent pretraining and even OXE-pretrained baselines in transfer settings [11, 12] — but published systems treat the latent space as a pretraining scaffold to be discarded or narrowly decoded, not as the runtime interface for planning, memory, and world modeling.

Separately, action-free world models have crossed a capability threshold: V-JEPA 2-AC post-trains a 300M action-conditioned latent predictor on ~62 hours of unlabeled DROID interaction atop a video encoder pretrained on ~1M hours of internet video, then executes zero-shot pick-and-place on unseen Franka arms via CEM planning over a goal-conditioned latent energy — outperforming both a behavior-cloned VLA baseline and a generative video world model while planning ~16× faster [14, 15]. Yet V-JEPA 2-AC has no policy: every action is an optimization. PARLAM's thesis is that the policy (fast, amortized) and the world model (slow, deliberative) must share one latent action space so that each can supervise, verify, and rescue the other.

1.3 Contributions

1. Formulation of general-purpose robot control as amortized inference over a shared discrete-continuous latent action space, with an identifiability analysis for latent actions learned from passive video (§3).
2. The PARLAM architecture: a mixture-of-transformers backbone coupling a VLM planner (System 2, 3–10 Hz) to a flow-matching action expert (System 1, 50–200 Hz) through the universal action interface, with morphology-conditioned embodiment adapters and a JEPA-style predictive world head (§4–5).
3. A four-level hierarchical action generator — mission → task graph → motion graph → motor commands — with formal option semantics linking semantic tokens to latent motion codes (§6).
4. Training methodology: data pyramid, six-term composite objective with staged curriculum, FAST-tokenized autoregressive pretraining before continuous-head attachment, simulation-native RL, and advantage-weighted fleet post-training (§7).
5. Memory subsystem (episodic/semantic/procedural) implemented as retrieval-conditioned context rather than parametric growth (§8).
6. Safety stack: conformal action gating, CBF shielding, world-model rehearsal, and rollback (§9).
7. Fleet learning over RobotNet with validation-gated model propagation, and scaling/evaluation protocols (§10–11).

2. Related Work

Vision-language-action models. RT-1/RT-2 established transformer policies over web-pretrained backbones with actions as text-like tokens [2]; OpenVLA showed a 7B open model (fused DINOv2+SigLIP features, Llama-2 backbone, 256-bin action tokens) outperforming larger closed models and supporting LoRA/quantized deployment [4]; Octo introduced a lightweight generalist with a diffusion action head and block-wise attention permitting new observation streams at fine-tuning time [5]; π0 attached a flow-matching action expert to a PaliGemma-class VLM for 50 Hz dexterous chunked control [6]; OpenVLA-OFT demonstrated that parallel decoding + chunking + continuous L1 heads dominate autoregressive decoding for fine-tuned success and latency [8]; GR00T N1 introduced the dual-system architecture (VLM reasoner + DiT action expert) for humanoids [16]; RDT-1B scaled diffusion policies to bimanual manipulation [9]. PARLAM adopts the dual-system, continuous-head consensus but inserts an explicit universal action interface between the systems.

Action tokenization. Per-dimension binning [2, 4] fails on high-frequency data because marginal token information collapses; FAST resolves this with DCT-domain compression of normalized action chunks followed by quantization and BPE, achieving ~10× compression and enabling autoregressive VLAs that match diffusion-head performance at ~5× lower training cost, with a universal tokenizer (FAST+) trained on 1M trajectories [7]. VQ-VLA scales convolutional residual VQ-VAE action tokenizers with simulation data [17]. PARLAM uses FAST-style tokens for Stage-A autoregressive pretraining (aligning the VLM prefix with action semantics before the continuous expert is attached — a recipe validated in π0.5-class systems [18]) and residual-VQ latent motion codes as the permanent runtime interface.

Latent action models. Genie learned latent actions from unlabeled video sufficient to make generated worlds playable [10]; LAPO introduced IDM/FDM latent-action recovery; LAPA scaled VQ latent-action pretraining to VLA backbones, beating supervised pretraining in cross-embodiment transfer [11]; UniVLA's task-centric latent actions decouple task-relevant dynamics from nuisance visual change and decode through a 10.8M-parameter head, surpassing OXE-pretrained OpenVLA while pretraining on less data [12]; UniAct and ViLLA-x pursue universal action spaces across embodiments [13]. PARLAM extends this line by making the latent action space bidirectional — shared by the policy (generation) and the world model (prediction) — and by regularizing it with morphology-conditioned decodability constraints (§3.3).

World models and world-action models. RSSM-family latent dynamics (PlaNet, DreamerV3) established imagination-based control with fixed hyperparameters across domains [19, 20]; V-JEPA 2(-AC) demonstrated internet-video pretraining → small-data action conditioning → zero-shot MPC on real arms with a locally convex goal-energy landscape [14, 15]; generative video world models (Genie 2/3, Cosmos, GAIA-1) supply high-fidelity but computationally heavy rollouts [10, 21, 22]. GR-2 and WHAM-style models couple video generation with action prediction [23]. PARLAM's world head follows the JEPA design point — predict in representation space, not pixel space — for a ~16×-class planning-speed advantage at deliberation time [14].

Hierarchy, options, and language planners. The options framework formalizes temporally extended actions with initiation sets and termination conditions [24]; SayCan grounds LLM task decomposition in learned affordances [25]; ACT introduced action chunking with temporal ensembling for fine manipulation [26]; Diffusion Policy established score-based multimodal visuomotor control [27]. PARLAM's four-level generator composes these: LLM task graphs bind to semantic action tokens with learned affordance/precondition models, executed as chunked latent motion.

Cross-embodiment and morphology conditioning. Beyond OXE co-training [1], HPT aligns heterogeneous proprioception/vision into a shared token space [28]; CrossFormer trains one transformer across manipulation, navigation, locomotion, and aviation [29]; MetaMorph and GET-Zero condition on kinematic-graph encodings for zero-shot morphology generalization [30]. PARLAM's embodiment adapters take the graph-conditioning route: decoders are hypernetwork-generated from a morphology embedding, not enumerated per robot.

Fleet learning and safety. QT-Opt pooled real grasping experience across robot farms [31]; Fleet-DAgger formalizes supervisor allocation [32]; federated averaging [33] and validation-gated deployment structure PARLAM's RobotNet loop, which is designed to run atop persistent simulation substrates such as SimNet. Safety draws on control barrier functions [34], conformal prediction for LLM/robot uncertainty (KnowNo) [35], and shielded RL.

3. Problem Formulation

3.1 Cross-embodiment control as amortized inference

Let ℰ be a space of embodiments; each e ∈ ℰ defines a control space 𝒰_e (dimension d_e, rate f_e), sensor suite Ω_e, and dynamics 𝒯_e. A task is (g, R_g) with g a language/image/latent goal. The naive object of robot learning is a family {π_e : Ω_e × 𝒢 → 𝒰_e}, one policy per embodiment — the regime PARLAM abolishes.

PARLAM instead posits the factorization

π_e(u | o, g) = ∫ Dec_e(u | z, s_e) · π(z | Enc(o), g, m) dz (1)

where z ∈ 𝒵 is the universal action, π is the shared PARLAM core (conditioned on memory m, §8), s_e is proprioceptive state, and Dec_e is a small embodiment adapter. The learning problem is: choose 𝒵, π, {Dec_e} to maximize task performance across (e, g) while minimizing per-embodiment parameters |Dec_e| — an information-bottleneck view in which 𝒵 must be a sufficient statistic of intended world change, invariant to how a body realizes that change.

3.2 The two-level action space

𝒵 = 𝒱_sem × 𝒵_motion:

• Semantic level 𝒱_sem: a typed vocabulary of parameterized action tokens — MoveTo(x), Pick(obj), ApplyForce(F, frame), Inspect(target), HandOver(agent), Assemble(a,b,constraint) — each carrying option semantics [24]: an initiation classifier I_σ(s), a termination β_σ(s), and an effect model P(s′|s,σ). Semantic tokens are the interface to language planning, verification, and memory indexing. Arguments are continuous slots (poses, object embeddings, wrenches) predicted by pointer/regression heads, not discretized — avoiding the resolution pathologies of binning [7].
• Motion level 𝒵_motion: residual-VQ codes for H-step motion chunks (H ≈ 0.5–1 s). A chunk of intended motion is encoded as K residual codebook indices (K = 4–8, codebooks of 1024), quantizing not raw actuator commands but a scene-relative motion abstraction: end-effector/body-frame twists, contact-mode switches, and interaction-force profiles expressed in object-centric frames. This is the representation that transfers: "approach along −ẑ of the handle frame, close to 20 N grip" is embodiment-free; joint-7 velocity is not.

3.3 Identifiability of latent actions from passive video

PARLAM pretrains 𝒵_motion partly on action-free video via the IDM/FDM construction [10, 11]: an inverse model q(z_t | o_t, o_{t+H}) and forward model p(ô_{t+H} | o_t, z_t) trained with a VQ bottleneck. Two known failure modes must be controlled:

1. Nuisance capture. Unconstrained latents absorb task-irrelevant appearance change (lighting, camera motion) rather than agent-caused change. Following the task-centric argument of UniVLA [12], PARLAM factors the latent transition as z = (z_act, z_nuis) with only z_act made available to the policy, trained adversarially so z_act is predictive of manipulanda state change but not of background change; DINO-feature-space prediction (rather than pixels) further suppresses nuisance gradients [12, 14].
2. Reparameterization ambiguity. From passive data, z is identifiable only up to an invertible relabeling; grounding requires a small action-labeled anchor set. PARLAM interleaves labeled robot data so the same codebook is trained under both (video-IDM) and (proprioception-supervised) encodings; empirically (LAPA, UniVLA) a few hundred labeled trajectories suffice to pin the mapping [11, 12]. Formally, if two embodiments' labeled data induce decoders Dec{e1}, Dec{e2} over a shared code z, the anchor loss ‖Dec_{e}(z) − u^{label}‖ selects, among observationally equivalent codebooks, one in which codes are co-decodable — the property (1) requires.

3.4 The world-action objective

Let w_t = Enc(o_t) be the latent world state. PARLAM jointly learns

Policy π(z_t | w_t, g, m) and World model ŵ_{t+1} = f(w_t, z_t) (2)

with a shared z. This coupling has three consequences exploited throughout the paper: (i) planning by search — CEM/MPPI over z through f gives an MPC controller with the same action interface as the amortized policy, so the world model can override the policy (V-JEPA-2-AC-style deliberation [14]) without any representational conversion; (ii) self-supervision — f's prediction error on policy-generated z is a dense correction signal (Dyna-style) and an anomaly detector; (iii) verification — before execution, sampled z are rolled through f and scored against constraints (§9).

4. Architecture

                         Human instruction / mission
                                    │
       ┌────────────────────────────▼─────────────────────────────┐
       │  SYSTEM 2 — VLM BACKBONE (7B, 3–10 Hz)                   │
       │  · language understanding & task-graph synthesis          │
       │  · semantic action tokens σ ∈ 𝒱_sem (+ argument heads)   │
       │  · FAST-token action literacy (from Stage-A pretraining)  │
       └───────┬──────────────────────────────┬────────────────────┘
               │ semantic tokens + KV cache   │ latent world state w_t
       ┌───────▼──────────────┐      ┌────────▼──────────────────┐
       │ SYSTEM 1 — ACTION    │◄─────┤ WORLD HEAD f(w,z) (JEPA)  │
       │ EXPERT (300M–1B DiT, │  MPC │ · latent future prediction │
       │ flow matching,       │ over │ · collision/failure/       │
       │ 50–200 Hz)           │  z   │   uncertainty estimation   │
       │ emits z_motion chunks│      └────────────────────────────┘
       └───────┬──────────────┘
               │ z ∈ 𝒵_motion (RVQ codes / continuous pre-quant)
       ┌───────▼───────────────────────────────────────────────────┐
       │ EMBODIMENT ADAPTERS Dec_e (1–20M each, hypernet-generated │
       │ from morphology embedding ψ(e))  → u_t at native rate     │
       └───────┬───────────────────────────────────────────────────┘
               │
       Low-level control (whole-body QP / impedance / joint PD)

4.1 Multimodal perception encoder

Inputs are tokenized per-modality and fused by a perceiver-style resampler into a fixed budget (≈512 tokens/step):

• Vision: fused SigLIP (semantics) + DINOv2 (geometry) patch features — the pairing OpenVLA found decisive for manipulation [4] — over base + wrist streams; an optional Ego3D positional encoding injects metric 3D structure [SpatialVLA-style].
• Depth/LiDAR: voxelized point-cloud tokens via a sparse 3D encoder.
• Force/torque & tactile: 1D-conv tokenization of 100 Hz wrench streams; contact events as discrete tokens.
• Proprioception: per-joint tokens carrying (q, q̇, τ) with kinematic-tree positional encodings — the morphology-aware scheme of HPT/GET-Zero [28, 30], which is what allows one encoder to ingest a 7-DoF arm and a 43-DoF humanoid.
• Audio: log-mel patches (verbal instructions, contact acoustics).
• Environment graph / simulation state: when PARLAM runs inside a simulation substrate (SimNet), ground-truth scene-graph tokens can replace estimated ones — a privileged-information channel used for teacher policies and distilled out for deployment.

4.2 Backbone: mixture-of-transformers with dual systems

Following the π0/GR00T design point [6, 16], PARLAM is a mixture of transformers: a 7B VLM (System 2) and a 300M–1B action expert (System 1) share layer-wise self-attention over a joint sequence but keep disjoint weights, so web-scale semantic knowledge is not overwritten by control gradients. System 2 attends over (instruction, perception, memory, semantic-action history) and emits the task graph and next semantic tokens; System 1 attends over (System-2 KV cache, proprioception at full rate, current chunk noise) and emits latent motion via conditional flow matching: with x_0 ~ 𝒩(0,I), x_1 = z*_motion, the expert v_θ regresses the probability-flow velocity

ℒ_FM = 𝔼_{t∼U[0,1]} ‖ v_θ(x_t, t, c) − (x_1 − x_0) ‖², x_t = t·x_1 + (1−t)·x_0 (3)

integrated with 5–10 Euler steps at inference — the fast, multimodal chunk generator validated by π0 [6]. Asynchronous real-time chunking overlaps generation of chunk k+1 with execution of chunk k, with inpainting-style consistency on the overlap region to eliminate discontinuities.

4.3 Embodiment adapters

Each embodiment contributes a descriptor: kinematic graph, actuator limits, sensor extrinsics. A graph transformer produces morphology embedding ψ(e) ∈ ℝ^{256}; a hypernetwork emits adapter weights Dec_e = h(ψ(e)) — so unseen morphologies get a functional (if unrefined) decoder zero-shot, refined with minutes of self-calibration data (the GET-Zero/MetaMorph result generalized to the latent-action setting [30]). Dec_e maps (z_motion, s_e) to native commands through a differentiable kinematics layer: latent scene-relative twists → task-space targets → whole-body QP or impedance references. The adapter is deliberately too small (1–20M) to contain behavior; behavior lives in the core, which is what makes (1) an actual bottleneck rather than a rhetorical one.

4.4 World head

The world head f is a block-causal transformer (V-JEPA-2-AC's architecture class [14]) over (w_{t−k:t}, z_{t−k:t}) predicting ŵ_{t+1:t+H} in representation space with teacher-forced + rollout losses. Auxiliary decoders over ŵ predict: object trajectories, human-motion forecasts, collision probability fields, task-failure hazard λ(t), and epistemic uncertainty (via a small ensemble of world heads; disagreement = novelty). Because prediction is latent, deliberative planning is 1–2 orders of magnitude cheaper than generative-video simulation at comparable control quality [14, 15], keeping MPC within the real-time budget for recovery situations.

5. The Universal Action Interface in Detail

5.1 Residual VQ motion codebooks

Motion chunks a_{t:t+H} (in scene-relative canonical form, §3.2) are encoded by a causal conv-transformer encoder and quantized by K-level residual VQ:

z = (c_1, …, c_K), c_k = argmin_j ‖ r_{k−1} − e_j^{(k)} ‖, r_k = r_{k−1} − e_{c_k}^{(k)} (4)

with EMA codebook updates, codebook-usage entropy regularization, and dead-code resampling. Coarse levels (k ≤ 2) carry motion type and direction — these are what transfer across embodiments and what the video-IDM pathway can supervise; fine levels carry dexterity-critical detail — these are refined only on labeled/teleop data. Cross-embodiment transfer of a skill is, mechanically, reuse of coarse codes with re-fit fine codes, which the training curriculum exploits (§7.3).

5.2 FAST tokens as the autoregressive on-ramp

Attaching a continuous expert to a VLM that has never represented actions is unstable; PARLAM therefore runs Stage A pretraining in which action chunks are compressed by DCT → quantization → BPE (the FAST pipeline, ~10× denser than binning, matching diffusion-head quality at ~5× lower training cost [7]) and predicted by the VLM with plain next-token loss. This teaches the backbone action semantics in its native objective; the flow-matching expert is attached afterward (Stage B) — the ordering reported to stabilize π0.5-class training [18]. FAST tokens are retired at runtime; RVQ codes are not.

5.3 Semantic ↔ motion binding

Every semantic token σ owns a conditional prior p(z_motion | σ, w, args) trained by maximum likelihood over demonstrations segmented by a learned boundary detector (β_σ). This is the options framework instantiated at scale [24]: σ's initiation set I_σ is a classifier over w used by the planner for affordance-feasibility checks (SayCan-style grounding, but with learned rather than value-derived affordances [25]); β_σ triggers replanning; the effect model P(w′|w,σ) is a coarse-grained view of the world head used for task-graph search. The binding makes plans executable by construction: a task graph is a lattice of σ's whose motion realizations are guaranteed to exist in the policy's support.

6. Hierarchical Action Generation

Level 1 — Mission. Free-form instruction ("assemble the chair") parsed by System 2 with retrieval from semantic memory (§8).

Level 2 — Task graph. System 2 synthesizes a DAG of semantic tokens with pre/post-conditions; graph search over effect models prunes infeasible orderings; the world head simulates the maximum-hazard edges before commitment. Task graphs are emitted as structured tokens and are themselves training targets (learned from hindsight-segmented demonstrations), not just prompt-time chain-of-thought.

Level 3 — Motion graph. Each σ expands to a sequence of latent motion chunks sampled from p(z|σ, w, args) by the action expert, with chunk-level closed-loop feedback: re-encoding of w_t every chunk, termination via β_σ.

Level 4 — Motor commands. Dec_e + whole-body control at native rate; joint impedance/torque limits enforced in the QP, so learned components can never command constraint-violating torques directly.

Cross-level consistency is trained, not assumed: hindsight relabeling converts executed motion into (task-graph, chunk) supervision, and level-2 predictions are scored against level-4 outcomes to calibrate the planner's effect models.

7. Training

7.1 Data pyramid

The pyramid ordering encodes the identifiability argument of §3.3: T1 shapes the coarse latent space cheaply; T3 anchors it; T2 gives controllable breadth (lighting, friction, materials, gravity, geometry, sensor noise, morphology randomization — the DR axes with LLM-proposed ranges where hand priors are weak [37, 40]); T4 closes the loop.

7.2 Composite objective

ℒ = λ₁ℒ_FM (flow matching, Eq. 3) + λ₂ℒ_AR (FAST/semantic next-token CE) + λ₃ℒ_WM (latent future prediction, teacher-forced + k-step rollout) + λ₄ℒ_IDM/FDM (video latent actions, Eq. 4 commitment losses) + λ₅ℒ_CL (InfoNCE across vision/language/motion triplets — the alignment that lets language index motion) + λ₆ℒ_aux (affordance I_σ, termination β_σ, hazard λ, uncertainty calibration).

Gradient conflicts between generation (λ₁, λ₂) and prediction (λ₃) terms are managed by head-specific LR scaling and PCGrad-style projection on shared trunk layers.

7.3 Stages

A. Action literacy — VLM + FAST tokens on T3+T2, world head on T1+T2. B. Expert attachment — freeze most of VLM, train System 1 + RVQ + adapters; interleave T1 IDM batches to keep coarse codes video-consistent. C. Simulation RL — massively parallel PPO/DreamerV3-style imagination fine-tuning in GPU-batched environments [19, 36] against LLM-synthesized rewards audited by humans (Eureka-class reward generation [40]); policies constrained by KL-to-BC-prior to prevent mode collapse of 𝒵. D. Fleet post-training — offline, advantage-weighted (AWAC/IQL-family) updates from T4 with the world head providing counterfactual value baselines; preference optimization (DPO-style) over paired execution videos rated by humans and by VLM judges for style/safety constraints that rewards under-specify.

7.4 What is not trained

Dec_e kinematics layers are analytic; safety shields (§9) are verified code, not learned; the QP is a solver. PARLAM's philosophy is learned intent, verified execution.

8. Memory

PARLAM's weights are frozen between fleet updates; adaptation between updates is contextual, via three retrieval-indexed stores:

• Episodic: trajectory segments embedded by (w, z, outcome); retrieval key = current (w, g). Supplies in-context examples ("last time this drawer stuck, force profile X worked") appended as tokens to System 2 — in-context imitation rather than weight change.
• Semantic: persistent scene graphs (objects, rooms, tools, topological maps) with object permanence; maintained by an assimilation filter over w; queried by task-graph synthesis and by argument heads (object grounding).
• Procedural: the skill library — (σ, p(z|σ,·), certificates, provenance) — shared fleet-wide through RobotNet; per-robot LoRA deltas for site-specific refinements, merged into the core only at validated fleet updates.

Human preferences ("this household stacks cups upside-down") live in episodic/semantic memory with per-site scoping — a privacy boundary as much as an engineering one.

9. Safety Architecture

Defense in depth, ordered from prediction to actuation:

1. Uncertainty gating. Ensemble disagreement of world heads + policy entropy over 𝒵 feed a conformal calibration layer (split-conformal over fleet outcomes, KnowNo-style [35]) yielding distribution-free bounds: if the 1−α prediction set over outcomes contains a hazard, PARLAM asks for help or slows down rather than acts.
2. Rehearsal. Candidate chunks are rolled through f; predicted collision fields/hazard λ above threshold trigger resampling or MPC override (the shared-𝒵 property making override seamless, §3.4).
3. CBF shield. A control barrier function layer filters Dec_e output: u_safe = argmin ‖u − u_π‖² s.t. ḣ(x,u) ≥ −γh(x) for verified safety sets h (workspace limits, human proximity, force caps) [34] — actuation-level guarantees independent of learned components.
4. Physics validation & rollback. Executed outcomes scored against world-head predictions; systematic surprise quarantines the responsible skill (procedural-memory certificate revocation, fleet-wide) and triggers rollback to the previous validated policy version.
5. Human override at all levels, with shared-autonomy blending rather than binary handoff, and supervisor allocation across the fleet per interactive-fleet-learning policy [32].

Neuro-symbolic plan validation — type-checking task graphs against precondition logic before execution — is what bounds "action hallucination": a plan that references a nonexistent object or violates an invariant is rejected at Level 2, before any motion exists.

10. Fleet Learning over RobotNet

The loop: robot → experience (w, z, outcome, surprise) → upload (prioritized by surprise/novelty; on-device filtering + DP noise for human-adjacent data) → RobotNet aggregation → Stage-D retraining → validation gauntlet (simulation regression suite over SimNet branches; shadow-mode prediction on live twins; conformal recalibration) → staged deployment (canary fleet → global). Model deltas propagate as LoRA/adapter patches (MB-scale) rather than full weights; federated aggregation [33] is used where raw trajectories cannot leave premises. A skill learned in Tokyo improves Palo Alto at the latency of validation, not of data collection — the QT-Opt/OpenBot-Fleet result [31, 41] promoted from task-specific to foundation-model scope. The quantitative regime shift (argued in our SimNet analysis) is that ecosystem learning rate becomes validation-bound, which is why the gauntlet is engineered for throughput: amortized adversarial probes, surrogate safety models, branch-parallel evaluation.

11. Scaling and Evaluation

Scaling protocol. Axes: parameters (0.5B→70B core), T3 hours, T2 episode diversity, embodiment count |ℰ|, codebook capacity. Predictions to test: (i) transfer to a held-out embodiment scales with |ℰ| faster than with per-embodiment hours (the embodiment-diversity law [3]); (ii) coarse-code reuse rate across embodiments rises with codebook entropy regularization and predicts zero-shot adapter quality; (iii) FAST-pretrained backbones dominate cold-started continuous heads at matched compute [7, 18]; (iv) world-head accuracy (latent AUC on failure prediction) is the best single predictor of safe-deployment rate — the metric that should gate scale-up.

Benchmarks. Cross-embodiment: OXE-derived evaluation + held-out morphologies; manipulation: LIBERO, dexterous PAR-internal suites; long-horizon: multi-stage assembly with mid-episode perturbation; language: instruction paraphrase/preposition sensitivity [2]; world modeling: action-conditioned latent prediction vs. V-JEPA-2-AC on DROID splits [14]; safety: hazard-recall at fixed intervention rate, conformal coverage validity; fleet: skill-propagation half-life, forgetting on certified-skill regression suite.

12. Limitations

(i) The universal-motion-language hypothesis is strongest for quasi-static and moderately dynamic manipulation; highly dynamic whole-body behavior (parkour-class locomotion) may resist scene-relative canonicalization, requiring embodiment-specific coarse codes. (ii) Identifiability from video (§3.3) degrades when the observed agent's affordances differ radically from any fleet embodiment (human hands vs. parallel-jaw grippers) — the anchor set must be diverse. (iii) Latent-space MPC inherits world-head blind spots; conformal gating bounds risk only under exchangeability, which distribution shift violates — hence rollback, not just gating. (iv) Frequency-domain and RVQ compression bound achievable dexterity bandwidth; surgical-class precision may need dedicated fine-code stacks. (v) Fleet learning concentrates capability and failure: a bad validated update is a correlated fleet-wide event, which is why canary staging and instant rollback are load-bearing, not optional.

13. Conclusion

PARLAM operationalizes a specific bet: that perception, prediction, planning, and control should meet in one discrete-continuous latent action space — learned from video, anchored by robots, generated by a flow-matching expert, verified by a JEPA-style world head, and decoded through morphology-conditioned adapters. Every component has independent empirical support in the current literature — dual-system VLAs [6, 16], FAST-tokenized action literacy [7], latent-action pretraining [10–13], representation-space world models with zero-shot planning [14], cross-morphology conditioning [28–30], fleet-scale experience pooling [31, 41]. PARLAM's contribution is the closed algebra among them: the policy and the world model speak the same language of motion, so each can train, check, and rescue the other — the property we take to be necessary for robots that improve continuously, collectively, and safely at fleet scale.

References

[1] Open X-Embodiment Collaboration. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. ICRA 2024. arXiv:2310.08864. [2] A. Brohan et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL 2023. [3] B. Ai et al. Towards Embodiment Scaling Laws in Robot Locomotion. arXiv:2505.05753. [4] M. J. Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024. arXiv:2406.09246. [5] Octo Model Team. Octo: An Open-Source Generalist Robot Policy. RSS 2024. [6] K. Black et al. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164. [7] K. Pertsch et al. FAST: Efficient Action Tokenization for Vision-Language-Action Models. arXiv:2501.09747. [8] M. J. Kim, C. Finn, P. Liang. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT). arXiv:2502.19645. [9] S. Liu et al. RDT-1B: A Diffusion Foundation Model for Bimanual Manipulation. ICLR 2025. [10] J. Bruce et al. Genie: Generative Interactive Environments. ICML 2024. [11] S. Ye et al. Latent Action Pretraining from Videos (LAPA). ICLR 2025. arXiv:2410.11758. [12] Q. Bu et al. UniVLA: Learning to Act Anywhere with Task-centric Latent Actions. RSS 2025. arXiv:2505.06111. [13] Z. Zheng et al. UniAct: Universal Actions for Enhanced Embodied Foundation Models. CVPR 2025. [14] M. Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv:2506.09985. [15] NVIDIA. Cosmos World Foundation Model Platform for Physical AI. arXiv:2501.03575. [16] NVIDIA. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv:2503.14734. [17] Y. Wang et al. VQ-VLA: Scaling Vector-Quantized Action Tokenizers. ICCV 2025. [18] Physical Intelligence. π0.5: A VLA with Open-World Generalization. 2025. [19] D. Hafner et al. Mastering Diverse Control Tasks through World Models (DreamerV3). Nature 2025. [20] D. Hafner et al. Learning Latent Dynamics for Planning from Pixels (PlaNet). ICML 2019. [21] J. Parker-Holder et al. Genie 2: A Large-Scale Foundation World Model. Google DeepMind, 2024. [22] A. Hu et al. GAIA-1: A Generative World Model for Autonomous Driving. arXiv:2309.17080. [23] ByteDance. GR-2: A Generative Video-Language-Action Model. arXiv:2410.06158. [24] R. Sutton, D. Precup, S. Singh. Between MDPs and Semi-MDPs: The Options Framework. AIJ 1999. [25] M. Ahn et al. Do As I Can, Not As I Say (SayCan). CoRL 2022. [26] T. Z. Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT/ALOHA). RSS 2023. [27] C. Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023. [28] L. Wang et al. Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (HPT). NeurIPS 2024. [29] R. Doshi et al. Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation (CrossFormer). CoRL 2024. [30] A. Gupta et al. MetaMorph: Learning Universal Controllers with Transformers. ICLR 2022; A. Patel, S. Song. GET-Zero. ICRA 2025. [31] D. Kalashnikov et al. QT-Opt: Scalable Deep RL for Vision-Based Robotic Manipulation. CoRL 2018. [32] R. Hoque et al. Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision. CoRL 2022. [33] B. McMahan et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS 2017. [34] A. Ames et al. Control Barrier Functions: Theory and Applications. ECC 2019. [35] A. Z. Ren et al. Robots That Ask For Help: Uncertainty Alignment for LLM Planners (KnowNo). CoRL 2023. [36] V. Makoviychuk et al. Isaac Gym: High Performance GPU-Based Physics Simulation for Robot Learning. NeurIPS D&B 2021. [37] J. Tobin et al. Domain Randomization for Sim-to-Real Transfer. IROS 2017; X. B. Peng et al. Dynamics Randomization. ICRA 2018. [38] A. Handa et al. DeXtreme: Transfer of Agile In-Hand Manipulation from Simulation to Reality. ICRA 2023. [39] A. Khazatsky et al. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS 2024. [40] Y. J. Ma et al. Eureka: Human-Level Reward Design via Coding LLMs. ICLR 2024; DrEureka. RSS 2024. [41] M. Müller et al. OpenBot-Fleet: A System for Collective Learning with Real Robots. ICRA 2024.