Introducing Foresight

Today we are publicly naming something we have been building for years. Foresight is Dexterity’s world model, the intelligence layer that lets our robots reason about the physical world, predict what will happen next, and act with confidence in environments where mistakes are expensive and safety is non-negotiable.

Foresight has been trained on experience from over 100 million autonomous actions in production across enterprise logistics operations. Not in simulation. Not in a lab. In real warehouses, on real shifts, handling real packages continuously.

We are sharing it now because we think the Physical AI community deserves to understand what production-grade world models actually look like. And because we built a game that lets you experience one small piece of the problem yourself.

Why World Models Must Be Different for Manipulation

Most world models in AI today are built for observation. They compress visual data, generate plausible future frames, or reconstruct scenes from novel viewpoints. They are impressive. They are also insufficient for robots that need to touch things.

Manipulation demands a fundamentally different kind of understanding. When a robot picks up a box, the world changes. Objects shift. Previously hidden surfaces become visible. Stability conditions evolve. The world model must not only perceive these changes: it must predict them before they happen, reason about them as they unfold, and update its beliefs based on what actually occurred.

Interpretable by Design

Foresight is built as an operator, not a database. It transforms the current physical state of the world into an updated state at each timestep, integrating multi-modal sensor data, robot body state, and the outcomes of physical interactions. Every update maintains explicit uncertainty bounds, physics constraints, and rollback capability. Every decision the system makes can be traced, inspected, and understood.

In industrial settings, a black box is not deployable. Operators must understand why a robot did what it did. Safety engineers must verify that behavior is bounded. When something goes wrong, which it will because the physical world is adversarial, the failure must be diagnosed, not simply trained around.

Foresight is interpretable by design. Not as an afterthought. Not as a dashboard bolted onto an opaque model. Interpretability is architectural, woven into every layer of the system.

Four Core Capabilities

A world model is only as valuable as what it enables. Foresight makes decisions possible by providing the rich, simulatable, physically grounded state that planners, orchestrators, and skill agents require.

1. Predictive branching: Foresight can roll the world state forward under hypothetical actions, simulating the consequences of each candidate placement before anything moves. This gives planners the ability to explore multiple futures, score them against physics constraints, and reject commitments the system cannot recover from. The branching algorithm is replaceable. The world model that makes branching meaningful is not.

2. Pragmatic decision-making: In production, globally optimal placement is rarely achievable. Time pressure, imperfect sensing, and an unpredictable object sequence define the operating reality. Foresight provides rich enough state prediction that algorithms can evaluate the tradeoff between optimal and good-enough, because it can simulate the downstream consequences of both. Knowing when good enough is the right answer separates systems that work in demonstrations from systems that work in production.

3. Capability-aware orchestration: Dexterity deploys teams of specialized skill agents: perception, motion planning, grasp selection, force control. What makes orchestration capability-aware is the world model underneath it. Foresight does not just model the state of the environment. It models the feasibility of every action in that state: whether a grasp will hold, whether a path is clear, whether a placement will remain stable. The orchestrator does not need its own understanding of physics. Foresight gives it one.

4. Predictive pipelining: While a robot arm is still executing an action, Foresight is already simulating the likely outcome. When the action completes and sensor data arrives, Foresight reconciles its prediction with reality and updates the world state. The next action is ready before the current one finishes.

Careful and Fast

On average, Foresight’s packing agent makes a placement decision in under 400 milliseconds, while simultaneously optimizing for density, physical stability, robot reachability, and dual-arm parallelism. Most academic packing solvers optimize a single objective and take orders of magnitude longer. Foresight optimizes all of them jointly, in real time, on sequences it has never seen before.

Other frontier robotics companies publish videos that are sped up, or show robots operating at a fraction of human pace. Foresight operates at production speed: thousands of actions per shift, every shift.

Speed matters because logistics does not wait. A truck loading dock has a schedule. A fulfillment center has throughput targets. A robot that thinks carefully but acts slowly is a robot that gets replaced by a human. Foresight is built to be both safe and performant, because in production these properties are not optional and cannot be traded off against each other.

The Packing Problem

Packing a truck is one of the hardest spatial reasoning challenges in Physical AI. The problem presents a constrained three-dimensional volume. Boxes arrive in random sizes, weights, and order. The objective is to maximize density while maintaining stability: walls of boxes that will not collapse, weight distributed safely, fragile items protected. All of this must be accomplished at production speed, with no advance knowledge of what is coming next.

Humans are surprisingly good at this. We have deep spatial intuition built over a lifetime of interacting with physical objects. Most people can pack a truck to 40-60% density without thinking too hard about it.

Getting beyond that is where it gets interesting. The combinatorial space explodes. Every placement constrains every future placement. A locally optimal choice can lead to a globally suboptimal outcome. All of this must be navigated under time pressure, with imperfect information, in a sequence that cannot be controlled.

This is just one of the problems Foresight solves. Spatial packing is one dimension of a much broader system. In production, Foresight simultaneously reasons about physical stability to prevent wall collapses, incorporates robot motion and reachability constraints, trades off density goals against dual-arm parallelism for higher throughput, imagines how packages will shift under gripper forces, and handles uncertainty in box dimensions and rigidity to produce robust placements. Each of these is a hard problem on its own. Composing them into a coherent, real-time system is where most approaches fail.

Try It Yourself

We built a game that lets you experience the packing problem firsthand. It is a browser-based truck loading challenge: boxes arrive one at a time, and you place them to maximize density. It is simple to understand and genuinely difficult to master.

We are sharing this not as a competition against Foresight, but as an invitation. Try the problem. Feel why it is hard. Gain an appreciation for the spatial reasoning that Physical AI systems need to handle, and then consider that packing is just one piece of what a production robot needs to do.

Play the game at dexterity.ai/play.

The Foresight API Challenge

Later this month, we are launching the Foresight API Challenge: a competition where student teams can build their own packing agents, compete on a public leaderboard, and win up to $50,000 in prizes.

The format is straightforward: a REST API, a stream of boxes, and a truck to fill. Build an agent that decides where each box goes. Agents are ranked by average packing density across random sequences.

No simulator is provided. You will need to build your own understanding of the physics. One box at a time, full state returned. This is a real engineering challenge, the kind of problem where strong fundamentals in spatial reasoning, search, and physics modeling will matter more than compute budget.

Details and signups at dexterity.ai/challenge. The API challenge opens in the coming weeks.

Foresight is the result of years of work by a team that believes Physical AI should be interpretable, safe, and fast. Deployed at production scale, it has proven that these properties are not in tension: all three are achievable together.

The game is a window into one small piece of what Foresight does. The challenge is an invitation to go deeper. And this is just the beginning; future rounds will expand the problem space and bring the full robot into the challenge.

For students, researchers, and engineers who believe Physical AI is the most important unsolved problem in technology: the challenge is open.