Evaluating \( \pi_0 \) in the Wild:
Strengths, Problems, and the Future of Generalist Robot Policies

Robotics, particularly manipulation, has never had trained models that work out of the box on new objects, locations, and tasks. Roboticists have had the unsatisfying experience of going through tedious engineering and data collection to acquire robot policies, and then finding that even small environmental changes break those policies. One promising direction is to train generalist models on large datasets, in the hope that they will produce sensible behavior in new situations, reducing the burden on the end user. This last year has been exciting because of the first wave of models that are beginning to promise that this dream of generalist robots is possible. So, when Physical Intelligence made their models public, we were keen to try it out ourselves, and we were largely impressed and excited about the possibilities as these models continue to improve.

Our evaluations were conducted using the π₀-FAST-DROID model, specifically fine-tuned on the DROID robot setup, which consists of the Franka Panda robot with side and wrist cameras. We found it refreshingly easy to setup the platform for policy inference - no camera / controller calibration, or workspace-specific tuning was required. To output actions, the model requires a prompt from the user, describing the task, and images from the wrist and side cameras (see video above).

Our experiments took place in a kitchen environment, see images above. The kitchen has a diverse selection of objects, backgrounds and lighting conditions, making it ideal for coming up with a large variety of tasks.

Evaluating robot policies is difficult because it is hard to come up with a selection of tasks that cover the wide range of behaviors an arbitrary user would find useful.

We conducted over 300 trials of \( \pi_0 \) on various manipulation tasks. It is important to stress that our evaluations were conducted to satisfy our own curiosity about the capabilities of the model (e.g. how does \( \pi_0 \) handle articulated objects, or occluded viewpoints), and do not provide a comprehensive evaluation of the model's capabilities. We summarize our findings below:

1. Strong Prior for Sensible Behaviors: \( \pi_0 \) produces sensible behaviors across a wide variety of our tasks, although it is important to note that sensible behaviors are often insufficient for task completion.
2. Prompt Engineering Matters: Although \( \pi_0 \) can produce reasonable actions for many prompts and camera viewpoints, we observed that its success rate for the same task can fluctuate dramatically when the phrasing or viewpoint changes. To achieve consistent performance, use canonical prompts(verb + object) and select camera angles that clearly shows the target object.
3. Unexpected Quirks: \( \pi_0 \) can recover from failures, and handle moving humans in the scene, but it struggles with mid-task freezing, collision avoidance, and fine-grained manipulation.

Below, we elaborate on our findings in detail, going through success and failure behavior of \( \pi_0 \), as well as some additional interesting phenomena we discovered.

Powered by PaliGemma, Google DeepMind's 3B VLM, as its vision encoder (Beyer etc, 2024), \( \pi_0 \) demonstrates robust scene comprehension and adaptability. Despite relying solely on uncalibrated monocular RGB inputs (224x224 pixels after compression), it can handle very challenging objects and environments, including transparent or camouflaged items, and items it has not seen during training.

1. It can grasp transparent objects

\( \pi_0 \) is capable of identifying and manipulating transparent objects, as shown below. It picks up the bottle with a stable grasp, aligns it to the small cup, and precisely drops it in. Many traditional grasp detection techniques require an accurate 2D or 3D reconstruction of the scene, and transparent objects can cause issues in reconstruction accuracy. What makes it even more impressive is that the model can detect transparent objects solely from uncalibrated, mono-RGB images.

2. It can grasp an object even when it is camouflaged into a colorful background

\( \pi_0 \) can identify the 'yellow fish' here even when it is placed on top of a colorful board game. This object has an unusual and difficult shape, and it blends in well with the background, but \( \pi_0 \) detects it well enough to grasp it up.

3. It is robust to human activity in the input

During evaluation, there were many times where the side-view camera captured humans moving around in the background. However, \( \pi_0 \) can always focus on its task, keeping the robotic arm's movements focused on object manipulation.

We believe there are two reasons for \( \pi_0 \)'s robustness to human movement. First, the pre-trained VLM backbone of \( \pi_0 \) is trained on images involving humans (Sharma et al., 2018, Changpinyo et al., 2022a, Kuznetsova et al. 2020), so humans are in-distribution. Next, as our occlusion experiments in Section 2.3.1 show, the policy seems to prioritize the wrist camera's images during pick-and-place tasks, so distractors in the side-view camera seem to minimally affect the policy.

Here are two side-view videos involving humans in the scene. Please refer to Appendix B.6 for more experiments on human-robot interaction.

(All videos featuring humans were uploaded with permission from the individuals involved.)

Many existing works in Computer Vision and Robotics focus on transparent object detection and manipulation. But the nice part here is that we have an end-to-end, data-driven system that does it, without any special logic or care for transparent objects.

"\( \pi_0 \)'s ability to handle transparency, clutter and distractors hints at a future where robots see the world as humans do—through semantics, not just pixels."

If you are a human, you can easily imitate the behavior of the robot in the videos above. This is because the robot's behavior is sequential, and each step is independent of the previous step. However, it was not true for the history of robotics. Traditional behavior-cloning models may be able to memorize one exact path; But changing a scene and or start from different height could cause model fails. This is because the variance of the data could cause robot learning very bad behaviors, like collision and failure. Through our experiments, we observed that \( \pi_0 \) exhibits similar behavior patterns across a wide range of manipulation tasks. Although it is an autoregressive model without any memory or history, \( \pi_0 \) often executes the tasks step by step like:

Reach → Grasp → Transfer → Release → Reset → Idle

What's remarkable is that this pattern is not hard-coded in model structure, but emerges naturally from millions of demonstration data — suggesting that \( \pi_0 \) learns consistent task execution priors across environments. For instance, even when \( \pi_0 \) is unfamiliar with an object or task, it often proactively explores near affordance-rich areas, using its wrist camera to decide whether to grasp or not.

In certain trials, we also observed reset-like behaviors: if \( \pi_0 \) perceives the task as complete (e.g., after placing an item into a bowl), it may return to its home configuration and stop. While this often indicates a well-formed task boundary, it can also lead to early stopping/freezing, especially in multi-object scenes — see Section 4.1 for analysis of early stopping failure cases.

While this sequencing might suggest that \( \pi_0 \) has learned an internal understanding of the task, we caution against such framing. These patterns may reflect properties of the data distribution (e.g., Markovian, short-horizon tasks), rather than indicating the policy has acquired explicit task inference or memory.

However, it doesn't mean we have solved imitation learning. \( \pi_0 \) follows subtask sequences in a sensible way is maybe more of an observation about the task family rather than the algorithm at hand — many of the tested tasks may be sufficiently markovian that a history-less policy can follow a sensible chain of subtasks. We will discuss more in the Section 4.

One common failure case is that policy may freeze unexpectedly during execution, leaving tasks incomplete until a human intervenes. This behavior comes from two related factors: semantic ambiguity and autoregressive action decoding limitations.

Possible Causes

1. The VLM part doesn't understand the instruction

Unlike those commercial chatbot with large parameters, \( \pi_0 \) is built upon PaliGemma, a very small VLM model. Therefore, it lacks the commonsense reasoning that LLMs can use to recognize unfamiliar object categories. When it does not understand a command, it gets stuck. In some experiments, we found that some objects / instructions are out of distribution (OOD), causing early stopping. To show how bad the PaliGemma model performs, we attached a Vision-Question-Answering (VQA) example in Appendix C.

2. \( \pi_0 \) remembers only the now, but many tasks need a sense of before-and-after

\( \pi_0 \) is a memory-less policy, meaning its next move depends only on the current camera images, it never "remembers" what it did a moment ago. That works for single, snap-shot actions(e.g. pick up a cup), but can fail when a task needs several coordinated steps. For example, here is an articulation task that requires multiple steps and freeze in the middle:

• Case: "Open the drawer" → stops after grasping the handle.
• Behavior: \( \pi_0 \) reaches out, grabs the handle... and then freezes.

Why? In the training data, most frames showing a robot holding a handle are idle frames—nothing moves. \( \pi_0 \) always chooses the most common action it has seen for a given image. So when it sees "hand-on-handle," the safest bet in its experience is "do nothing."

How could we fix this? We asked folks from Physical Intelligence, and the answer is: Shake the dice a little. Instead of always taking the single most likely (arg-max) action, we can allow a bit of randomness—called "sampling with temperature." By letting \( \pi_0 \) occasionally pick the second-most likely action, it may start pulling instead of freezing, and the drawer finally slides open.

\( \pi_0 \) often struggles with spatial reasoning about height. For example, when asked to pick up an object and place it into a container, the policy cannot lift the object high enough to clear the height of the container. This suggests one drawback of image-based policies: the policy does not have a metrically accurate method to determine the distance between the gripper and the surrounding environment.

What's more, when \( \pi_0 \) is told to manipulate a household appliance that it did not see during training, it will tend to collide with the device or stop during trials. As shown below, \( \pi_0 \) can not use the coffee machine in our lab.

One possible solution is to use techniques such as voxel maps and planning constraints. Including depth information with a depth camera could also be helpful to implement collision avoidance.

Additionally, purely image-based policies lack tactile feedback. In our trials, \( \pi_0 \) sometimes applied too much force to delicate objects like human fingers, or too little to firmly grasp heavier ones like the plastic bottle. Complementing vision with tactile sensors or with low-level force controllers could help overcome these issues.

We investigated how variations in the prompts provided affect the policy's behavior, and found that \( \pi_0 \)'s performance heavily depends on the instructions given by the user, leaving space for prompt engineering.

1.1 You need to tune your prompt carefully to operate the robot

\( \pi_0 \) freezes or fails when instructions contain typos, grammatical errors, or ambiguous phrasing. For example, when we try to let \( \pi_0 \) manipulate the articulated objects, we may need to try multiple different prompts to find the 'in-distribution' instructions.

Instruction	Success Rate	Behavior
"Close the toilet"	0%	Wanders aimlessly, unable to localize the target.
"Close the white lid of the toilet"	100%	Always closes the toy toilet.

1.2 \( \pi_0 \)'s behavior without language goals

When given no specific language instruction, \( \pi_0 \) defaults to interacting with the most familiar objects from its training data:

• Given nonsense text like "dgbfzjkfhjilawhdfkAWHDKLWHADFiQAWFHqawipfjcasklfmdc", it picks up marker pens
• Given "xxx", it reaches for blocks repeatedly

In the DROID dataset, marker pens comprise 16.67% of objects, which could influence \( \pi_0 \) to pick the pen up when it is only given vision guidance. Default behaviors are heavily influenced by training data distribution. Overcoming this ambiguity and rejecting invalid instructions is still an ongoing problem.

Quirk 2: How robust is \( \pi_0 \) under partial observability?

Our Observations

• Dependence on Wrist Camera:
  • In our pick and place tasks, \( \pi_0 \) heavily relies on the wrist camera. It still works even if the side camera is blocked,
  • \( \pi_0 \) works worse in the alternative case, where the wrist camera is occluded but not the side camera.
• Viewpoint Robustness:
  • \( \pi_0 \) can tolerate changes to the side-view camera position and orientation during the task
  • If the camera is blocked and then unblocked, \( \pi_0 \) can recover.
• Common Failure Modes under Partial Observability:
  • Total occlusion of wrist camera results in the robot freezing.
  • \( \pi_0 \) is memoryless, predicting the action per frame auto-regressively, so it will be able to continue executing the task if observation is available.
  • However, the exploration behavior of \( \pi_0 \) is not efficient and is limited to certain areas of the scene. This makes it difficult to actively search the environment.

Our evaluations show that \( \pi_0 \) is a promising generalist policy: it demonstrates intelligent behaviour in unseen manipulation scenes. However, many challenges remain —-- saying we are "very impressed" is true, but only with right context. Remember, we've had roughly 50 years of robotics research. Until now, you couldn't just download someone else's controller, load it onto your own robot, and expect it to do even simple things. If \( \pi_0 \) can do that --- even at 20 - 50 % success on simple tasks straight out of the box, that alone marks a major leap forward.

As discussed in section Problems and Quirks, our experiments reveals that performance is sensitive to prompt phrasing, and the policy still struggles with instruction following, fine-grained manipulation and partial observability. We don't expect \( \pi_0 \) to be installed in everyone's home tomorrow, but we hope to see more advancement. Let robot go and run , let it do reasonable things step by step, which may not finish task every time, but it moves in the right direction. We are optimistic that continued research will address these issues and bring truly generalist robot policies closer to the practical reality.

We are grateful to Will Liang, Hungju Wang, and Sam Wang for their assistance in setting up the π0 environment. We further thank Kaustubh Sridhar, Tianyou Wang, Ian Pedroza, Ethan Yu, Tim Song, and Yiqian Li for their help doing the experiments.

We also thank Junyao Shi, Aurora Qian, Leon Kim, and Jason Ma for their insightful suggestions on evaluating a generalist manipulation policy.

We extend our appreciation to Karl Pertsch from Physical Intelligence for his constructive feedback on the early blog draft.

This project was supported in part by the National Science Foundation Graduate Research Fellowship Program under Grant No DGE-2236662. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

If you find this evaluation useful for your research, please consider citing our repository:

@misc{pi0-experiment-wild,
  author = {J. Wang, M. Leonard, K. Daniilidis, D. Jayaraman, & E. S. Hu},
  title = {Evaluating pi0 in the Wild: Strengths, Problems, and the Future of Generalist Robot Policies},
  year = {2025},
  publisher = {GRASP Lab, University of Pennsylvania},
  url = {https://penn-pal-lab.github.io/pi0-Experiment-in-the-Wild}
}

The following are details of our experiment set up.

• Franka Research 3 Arm: 7-DOF force-sensitive robot with a 3 kg payload.
• Robotiq 2F-85 gripper: two-finger gripper with 5mm stroke and adjustable force control.
• Cameras:
  • Side-view: ZED 2 stereo camera for global scene understanding
  • Wrist-mounted: ZED Mini for close-range object manipulation
  • Perception Mode: Pure RGB (no depth calibration)

GPU Server:

• GPUs: 1x NVIDIA RTX A6000 (48GB VRAM)
• CUDA Version: 12.3
• Usage: \( \pi_0 \) model inference.

Workstation

• GPU: NVIDIA GeForce RTX 3080 (16GB VRAM)
• CUDA Version: 12.6
• Usage: DROID low level control.

• Vision-Language Model: Paligemma 3B for spatial and semantic understanding.
• FAST+: Frequency-space Action Sequence Tokenization (FAST), a universal robot action tokenizer, trained on 1M real robot action trajectories. It can be used as a black-box tokenizer for a wide range of robot action sequences, with diverse action spaces and control frequencies.
• Training Data: Pretrained on π cross-embodiment robot dataset & Open X-Embodiment, fine tuned on DROID dataset.

• We divide the tasks into 7 categories:
  • Pick-and-Place
  • Pour
  • Articulated Objects
  • Fabric Manipulation
  • YCB Benchmark
  • Human Robot Interaction
  • Coffee Machine Challenge
• For each task:
  • We list the progress score and success rate for a group of similar trials.
  • Example rollouts and instructions are provided for each category.
• Note: This is a "vibe-check" style evaluation, not a rigorous benchmark. It can indicate some strengths and weaknesses of \( \pi_0 \), but does not reflect the full capabilities of the model.
• If you want to see more policy rollouts, please check our recent CoRL paper: RoboArena!