Dexterity Supercharges Foresight by 17x with NVIDIA

02 | Solution and Impact Summary

17x Faster: From 1508ms to 90ms

TensorRT-optimized inference. Massively parallel geometric reasoning. Purpose-built physics CUDA kernels. Three stages of NVIDIA acceleration compound to deliver a 17x speedup, bringing visual perception from 1508ms down to 90ms per cycle.

After optimization, the full sense-think-act loop fits comfortably within the cadence needed for real-time dual-arm production operation. The result: higher throughput and fewer placement errors, enabling denser, more reliable trailer loads in production. What follows is the deep dive into each pipeline stage and how these gains were achieved.

03 | Deep Dive

Three Pipeline Stages, Three NVIDIA Solutions

Foresight's visual perception pipeline has three stages. Each one was a bottleneck. Each one was solved with a different layer of NVIDIA technology. Benchmarks were run on production data on an NVIDIA L4 Tensor Core GPU.

03.1 | Foresight Perception

Object-Level Scene Decomposition: 2x Faster with TensorRT

Decomposing raw sensor data into individual objects, each with its own spatial embedding, depth features, and downstream attributes. TensorRT optimization halves median inference latency and dramatically tightens the tail.

Before Foresight can reason about the world, it must decompose raw multi-camera imagery into object-level representations. This isn't simple segmentation. For each object in the scene, Foresight Perception localizes not by image patches but by object instance, isolating each box's region across RGB and depth channels. From that localized attention area, the system extracts a rich set of per-object embeddings and attributes: spatial priors, surface features, and operational properties like pickability scores that predict whether removing a box would destabilize its neighbors. In a full trailer, this means reasoning about 100+ objects per frame.

The Foresight 0.1 pipeline was already fast, but not fast enough. TensorRT optimization, combined with architectural improvements to eliminate unnecessary overhead, delivers a 2x median speedup and, critically, compresses the P95 tail from 207ms down to 114ms. That tail matters: in a real-time pipeline, one slow frame can stall everything downstream.

03.2 | Foresight Reconstruction

Geometric Scene Understanding: 4.6x Faster, 32x More Data

Forming geometric hypotheses about every object in the world. Massive GPU parallelism doesn't just make it faster; it makes it possible to use the full, uncompromised sensor resolution.

Once objects are identified, Foresight must build a geometric understanding of each one, inferring shape, pose, and spatial extent from raw point cloud data. This is a dense optimization problem: the system generates and refines geometric hypotheses for every object in the scene, converging on a representation that captures how each object exists in 3D space.

The Foresight 0.1 CPU-based implementation was fast enough for subsampled point clouds, but subsampling means throwing away data, which means less precise reconstructions. On the GPU, Foresight explores a massive space of geometric hypotheses in parallel, evaluating candidates simultaneously across thousands of threads. This processes the full, unsubsampled point cloud from every camera: 32x more data points per object, and still runs 4.6x faster than the CPU version did on the reduced data.

03.3 | Foresight Synthesis

Physical Consistency at GPU Speed

Reconciling multi-sensor observations into a single, physically consistent world state, at the speed that real-time robotics demands. Purpose-built CUDA kernels enforce the laws of physics across every object simultaneously.

Multiple cameras observe the same scene from different viewpoints. Their observations disagree. Objects occlude each other, deform under contact, compress under load, and sag under gravity. Foresight Synthesis must reconcile all of this into one coherent world state: a representation where every physical constraint is satisfied and every observation is explained.

This is a continuous optimization over a suite of physical consistency constraints. Each pass evaluates how well the current world state explains reality, then refines it. The constraints span geometric agreement, volumetric accountability, contact mechanics, and physical plausibility, each demanding its own compute-intensive evaluation across every object in the scene.

This is the most compute-intensive stage of the pipeline, and the one where purpose-built CUDA kernels had the most dramatic impact.

Each constraint evaluates a different dimension of physical consistency, computed across every object in the scene:

Every constraint benefits from GPU acceleration, but the gains are not uniform. Volumetric Reasoning and Geometric Consistency see the most dramatic speedups (277x and 73x) because they scale as O(N×M) where N is the number of objects and M is the number of sensor observations, exactly the kind of embarrassingly parallel workload where purpose-built CUDA kernels dominate. Contact Mechanics and Plausibility Enforcement, while lighter, still achieve 8x speedups on GPU.

03.4 | Cumulative Impact

How Every Optimization Compounds

Perception drops from 57ms to 28ms, Reconstruction from 225ms to 49ms, and Synthesis from 1226ms to 13ms. The three stages sum to 90ms, down from 1508ms: a 17x end-to-end speedup.

04 | Dexterity's Broader Use of NVIDIA Ecosystem

Dexterity × NVIDIA

TensorRT for inference. Massive GPU parallelism for geometric understanding. Purpose-built physics CUDA kernels for physical consistency. This is what it takes to give a robot real-time understanding of the physical world. This acceleration work is just the foundation. Below is a snapshot of the broader NVIDIA ecosystem we are actively building on.