Inside Foresight: From Packing Evaluation to Real-Time Coordination

In Introducing Foresight, we described what Foresight does. This companion post is about how we know it works — and how we measure its impact.

Part I covers the Foresight-powered packing agent — the planning intelligence that decides where every box goes. Part II covers the Foresight-powered execution layer — the real-time coordination that turns those plans into 400+ picks per hour across two robot arms.

These aren’t cherry-picked results. They’re the systematic evaluations and production measurements we use internally to make deployment decisions.

01

Baseline Performance Across Distributions

We evaluated Foresight on four distinct box distributions drawn from real warehouse data. Each brings a different mix of box sizes, weights, and shapes. These results represent the agent simultaneously optimizing for all objectives — density, stability, throughput, and placement quality — in a realistic physics environment with robot kinematic constraints and gripper force limits.

We evaluate under two constraint regimes. Full robot constraints enforce kinematic reachability, gripper force limits, and dual-arm collision avoidance — the conditions the agent actually faces in production. Light robot constraints relax reachability checks to isolate algorithmic packing quality from robot-specific penalties, giving us a useful calibration baseline. The gap between regimes quantifies the cost of production constraints — or conversely, the benefit we stand to gain if we can lift them.

02

Building Statistical Confidence

A single number means nothing without a confidence interval. We use distribution-free order statistics to bound the true median of each metric — no distributional assumptions required.

The idea is clean: sort your n observations and pick the k-th and (n−k+1)-th smallest values. The probability that the true population median lies between them is determined entirely by the binomial distribution — not by any assumption about the shape of the underlying distribution.

The charts below show how the 95% CI narrows as we add more sequences. For low-variance metrics like density, 100 sequences is often sufficient. For higher-variance metrics like fissure area, we need substantially more.

03

Not All Metrics Are Equal

The coefficient of variation (CV = std / mean) reveals which metrics have inherently higher variance and which are stable enough to detect small policy improvements.

We track a number of metrics beyond the primary KPIs for detailed monitoring of how any update to the packing agent affects the behavior and performance of the entire system. The chart below ranks them by CV.

Fissure metrics have CVs of 0.5+ — the standard deviation is half the mean. A single sequence can't tell you much about fissure quality — you need many runs to separate signal from variance. By contrast, density has a CV of just 0.037 — the wall volume fraction is tightly constrained by the geometry of the task, so it's extremely stable across sequences.

This asymmetry matters: an agent change that looks like it improved fissures on 5 sequences might just be sampling variance.

The practical implication: when we make algorithmic changes targeting higher-variance metrics (like fissure area), we need 3–5x more evaluation sequences than when targeting stable metrics (like density) to achieve the same statistical power for detecting improvements.

04

Why Random Shuffling Is Misleading

A tempting shortcut for generating more evaluation data: randomly shuffle the box sequence. But real warehouse sequences have local structure that shuffling destroys — and this can silently bias your evaluator.

We looked at real-world package sequence data with over 500K total boxes, using a sliding 25-box window. For each window, we computed simple aggregates: how many small boxes? How many heavy ones? Total weight? We then compared these distributions between the original ordered sequences and uniformly shuffled copies.

Real-world sequences are rarely random walks. WMS batching logic releases orders in waves (e.g. all small electronics first); supplier packaging means consecutive boxes from the same vendor share similar dimensions; and physical constraints often group heavy items together. These effects create runs of similar boxes that shuffling destroys.

The result is striking. Shuffling preserves global rates (the means are identical) but collapses local variance by ~1.4x. An agent evaluated on shuffled data faces a fundamentally different — and easier — sub-sequence distribution than one evaluated on real data.

05

No Free Lunch, But a Good Pareto Curve

Foresight jointly optimizes for multiple objectives. There's no escaping the tradeoff — but we can design models to protect the metrics that matter most while accepting cost elsewhere.

The parallelism objective encourages placing boxes in positions where a second robot arm can work simultaneously. Higher parallelism means higher throughput — but it constrains placement choices, potentially affecting other metrics.

By sweeping the weight λ on the parallelism objective, we trace out the Pareto frontier. The key insight: the Foresight packing agent preserves density even as parallelism increases. In the regime we care about (λ ≤ 1), going from 38% to 72% parallelizable placements costs only 0.00–0.01 in density — the curve is essentially flat. The agent has learned to find placements that are simultaneously dense and parallelizable. Only at extreme parallelism targets (λ » 1) does density begin to drop.

Not all metrics are so easily maintained. Tilt stays remarkably flat across the entire frontier, but fissures increase meaningfully. Density is the metric we've specifically designed to protect; fissures are where the tradeoff cost surfaces.

06

Robustness Under Perception Noise

In production, Foresight doesn't see the world perfectly. Boxes have uncertain dimensions, positions drift, and sensor noise is ever-present. A packing agent that only works under perfect perception is useless.

We inject controlled noise into the physics simulation — perturbing observed box positions, dimensions, orientations, and weights — to measure how gracefully each metric degrades. Four noise axes, three magnitudes each, three key metrics tracked.

Density is remarkably resilient. Even at ±3cm position noise — a substantial perturbation — median density drops only from 0.68 to 0.66 (3%). Size noise at ±3cm costs just 1.5%. Weight noise up to ±15% has effectively zero impact on density.

Tilt is the most sensitive metric. Position noise at ±3cm increases mean tilt from 2.31° to 3.66° (+58%). This makes physical sense: when boxes are placed slightly off from where the agent intended, they settle at larger angles.

Fissures stay controlled. The worst case (position ±3cm) increases fissure sum by ~14%. Weight noise has essentially no effect on any metric, confirming that the agent is robust to moderate relative error in weight.

These results give us confidence that Foresight's approach degrades gracefully rather than catastrophically under real-world sensing conditions.

07

Predictive Pipelining

Predictive pipelining lets both robot arms operate concurrently — one arm picks while the other places, with no idle waiting. In production, 65% of placement decisions plan against predicted future state rather than waiting for fresh observations. This is how dual-arm throughput stays high.

In a serial system, each arm waits for the other to complete before planning its next move. Predictive pipelining eliminates this wait. When Arm A picks a box, the system predicts where that box will land — and critically, how that placement will affect the surrounding neighborhood. Placements use force-controlled tight-packing motions that push boxes into contact with their neighbors, so a single placement can shift, compress, or tilt adjacent boxes.

The system predicts this neighborhood effect, and Arm B plans its next action against this predicted world state — both arms plan concurrently rather than taking turns. Before execution, the planned trajectory is validated against the system's current estimate of the world.

The following metrics come from the last 5,000 boxes from production, analyzed from execution audit logs.

Two-thirds of all placement decisions plan against predicted in-flight state. Pipelining also meaningfully constrains the search space: 29% of candidate placements are filtered out because the predicted uncertainty in their placement neighborhood is too high — a nearby box is still in flight, and its effect on surrounding boxes hasn't settled yet. This isn't wasted computation — it's the system proactively avoiding regions of the trailer where the outcome is uncertain.

To isolate the throughput impact of parallelism, we ran an apples-to-apples comparison in simulation: the same ~1,500-box workload with parallel arm motion enabled vs. disabled.

08

When Predictions Are Wrong

When the prediction is wrong — the world changes significantly between planning and execution — the trajectory fails validation. This happens in only 0.6% of cases. When it does, the system automatically replans and recovers 80% of the time.

Before executing any pipelined trajectory, the system validates it against its current estimate of the world — not the predicted one it was planned against. Small changes are absorbed naturally; the validation catches cases where the world has changed enough to warrant re-evaluation. When that happens, the system replans against the updated state. This validation step adds negligible latency — it is a read-only sensor query. If replanning also fails, the box is returned to its pick location for a retry on the next cycle. No crash, no damage — just a small throughput hiccup.

Net effect: predictive pipelining delivers a 33% throughput improvement with a <0.2% effective failure rate — 9 unrecovered mismatches out of 5,000 placement searches. The CI bands in Section 02 comfortably contain this noise.