Foresight API Challenge

Solve a hard problem.
Win $50K.

Truck loading is one of the hardest spatial reasoning challenges in Physical AI. Build an agent, compete with the best student/researcher teams across the world, and win $50,000 — plus an interview at Dexterity.

Now Live!

Read the Docs ↓

Duration ExtendedThe challenge has been extended through April 30.

Prize Structure

Grand Prize

$50K

#1 on private evaluation with ≥70% IQM density

Eligibility

.edu+

Researchers and students with a valid .edu or equivalent university email address

Duration

→ Apr 30

Public leaderboard: ~April 6 – 28

Private Eval

48 hrs

April 29 12pm – April 30 12pm PT

How It Works

Play the Game & Get Your API Key

Play the truck loading game and score ≥50% packing density. Your API key will be displayed on the end screen — copy it and save it. No separate sign-up needed.

Play Now →

Build Your Agent

Use the REST API to send placements and receive game state. One box at a time, full state returned. No simulator provided - build your own.

Top the Leaderboard

Your agent plays random sequences. Minimum 25 games to qualify. Top 5 on the public leaderboard advance to the private evaluation — 50 held-out sequences, each played exactly once. #1 wins.

Scoring

Your score is packing density — the fraction of usable truck volume you fill with boxes.

// Density formula
density = sum(box_volumes) / (max_x_reached × truck_width × truck_height)

Numerator

Total volume of all placed boxes

Denominator

Deepest box face × truck cross-section

Implication

Gaps behind your pack front are penalized

How You Win

The competition has two phases. The public leaderboard determines who advances. The private evaluation determines who wins.

Phase 1~April 6 – 28

Public Leaderboard

Play as many games as you want (up to 50/day in compete mode). Each game draws from the same box pool, but the sequence order is randomly shuffled — every game is different. Your leaderboard rank is the interquartile mean (IQM) of your most recent 50 compete games — the average of the middle 50%, dropping your best and worst outliers.

Min Games

To appear on the leaderboard

Rate Limit

50 / day

Compete mode, per API key

Advance

Top 5

Advance to private evaluation

Phase 2April 29 12:00 PM – April 30 12:00 PM PT

Private Evaluation

The top 5 from the public leaderboard play 50 held-out sequences that no one has seen before. These sequences are pre-generated and frozen — every finalist plays the exact same 50 sequences in the same order. Each sequence can be played exactly once per API key. If your agent crashes, disconnects, or causes a collapse mid-sequence, that score stands. No retries.

Your final score is the interquartile mean (IQM) across all 50 private sequences — the middle 50% of your scores, averaged. You must achieve ≥70% IQM density to qualify for the prize. Note: there are no robot reachability constraints, visibility concerns, or observation noise in this round — it is pure packing. #1 with ≥70% wins $50,000.

Sequences

Held-out, never seen before

Retries

None

Each sequence played exactly once

Window

48 hours

Complete all 50 within the eval window

Validation

Before any prize is awarded, top submissions are validated:

→

Visual Review: Top submissions produce a full replay that is reviewed by a human. Your agent's placements are watched step-by-step.

→

Autonomy: Submissions must be generated by automated agents, not human teleoperation. Anomalous patterns will be flagged for review.

→

Final Check: Top submission may be asked to run against additional sequences while observed by Dexterity. This will not affect the ranking.

Rules

Prize$50K to #1 on private eval (≥70% IQM required)

Eligibility.edu or equivalent university email

TeamsAllowed — prize splits among members

Rate Limit50 compete games/day per API key

RankingIQM of packing density (most recent 50 games)

SimulatorNot provided — build your own

Agents OnlyAutomated agents required, no human teleoperation

InterviewWinner invited to interview at Dexterity

Questions? Reach us at foresight-api-challenge@dexterity.ai

API Reference & Guide

Everything you need to build and test your truck-packing agent.

Get Started ↓

Overview

The Foresight Packing Challenge API is a REST service where your agent packs boxes into a truck, one at a time. You receive a box, choose where to place it (position + orientation), and get back the full post-placement state including physics-settled positions.

Base URL

https://dexterity.ai/challenge/api

Auth

API Key

Passed in the request body, not headers

Format

JSON

All requests and responses are JSON

Modes

Dev Mode

dev

No physics simulation — boxes land exactly where you place them. No rate limiting, not scored. Use this to build and test your agent.

Compete Mode

compete

Full physics simulation. Boxes settle realistically. Rate limited, scored, and ranked on the leaderboard. This is where it counts.

Truck Segment Dimensions

Depth (X-axis)

2.0 m

Width (Y-axis)

2.6 m

Height (Z-axis)

2.75 m

Quickstart

Get from zero to your first placement in three steps.

Start a game

POST /challenge/api/start

# curl
curl -X POST https://dexterity.ai/challenge/api/start \
  -H "Content-Type: application/json" \
  -d '{
    "api_key": "YOUR_API_KEY",
    "mode": "dev"
  }'

// Response
{
  "game_id": "g_a1b2c3d4e5f6",
  "truck": { "depth": 2.0, "width": 2.6, "height": 2.75 },
  "current_box": {
    "id": "108148",
    "dimensions": [0.406, 0.310, 0.211],
    "weight": 3.13
  },
  "boxes_remaining": 120,
  "mode": "dev"
}

Place the box

Send the box ID, a 3D position [x, y, z], and an orientation quaternion [w, x, y, z].

POST /challenge/api/place

curl -X POST https://dexterity.ai/challenge/api/place \
  -H "Content-Type: application/json" \
  -d '{
    "game_id": "g_a1b2c3d4e5f6",
    "box_id": "108148",
    "position": [0.203, 0.155, 0.106],
    "orientation_wxyz": [1.0, 0.0, 0.0, 0.0]
  }'

// Response
{
  "status": "ok",
  "placed_boxes": [
    {
      "id": "108148",
      "position": [0.203, 0.155, 0.106],
      "orientation_wxyz": [1.0, 0.0, 0.0, 0.0],
      "dimensions": [0.406, 0.310, 0.211]
    }
  ],
  "current_box": {
    "id": "209311",
    "dimensions": [0.350, 0.280, 0.190],
    "weight": 2.45
  },
  "boxes_remaining": 119,
  "density": 0.0003,
  "game_status": "in_progress"
}

Repeat until done

Keep placing boxes from current_box in each response. The game ends when game_status becomes "completed" — either too many boxes collapsed or you stopped early. Your score is the final density.

Tip: In compete mode, physics simulation means boxes settle and can shift. The placed_boxes array returns post-settle positions — use these to plan your next placement.

Game Loop

Here's a minimal Python agent that plays a complete game with naive left-to-right packing:

import requests

BASE = "https://dexterity.ai/challenge/api"

# 1. Start a game
game = requests.post(f"{BASE}/start", json={
    "api_key": "YOUR_API_KEY",
    "mode": "dev"
}).json()

game_id = game["game_id"]
box = game["current_box"]
y_cursor = 0.0

# 2. Place boxes one at a time
while box:
    dims = box["dimensions"]  # [length, width, height]

    resp = requests.post(f"{BASE}/place", json={
        "game_id": game_id,
        "box_id": box["id"],
        "position": [dims[0]/2, y_cursor + dims[1]/2, dims[2]/2],
        "orientation_wxyz": [1, 0, 0, 0]
    }).json()

    y_cursor += dims[1]
    box = resp.get("current_box")

    if resp["game_status"] == "completed":
        print(f"Game over: {resp['termination_reason']}, density={resp['density']:.4f}")
        break

Termination Conditions

Your game ends when one of these happens:

Reason	Description
unstable	3+ boxes displaced >10cm after a placement
player_stop	You called `/stop` to end the game early

Endpoints

POST/challenge/api/start

Start a new game session. Returns the truck config and first box.

Request Body

Field	Type	Description
api_key*	string	Your API key
mode	string	Defaults to `"compete"`. Use `"dev"` for testing

Response

Field	Type	Description
game_id	string	Unique game session ID
truck	object	Dimensions in meters: `{ depth, width, height }`
current_box	object	First box: `{ id, dimensions, weight }`
boxes_remaining	int	Number of boxes left in the queue
mode	string	Confirmed game mode

POST/challenge/api/place

Place the current box in the truck. Returns the updated state with physics-settled positions.

Request Body

Field	Type	Description
game_id*	string	From the `/start` response
box_id*	string	Must match `current_box.id`
position*	[x, y, z]	Center position in meters
orientation_wxyz*	[w, x, y, z]	Rotation quaternion (auto-normalized)

Response

Field	Type	Description
status	string	`"ok"` or `"terminated"`
placed_boxes	array	All placed boxes with post-settle positions and orientations
current_box	object \| null	Next box to place, or `null` if game over
boxes_remaining	int	Boxes left in the queue
density	float	Current packing density (your score)
game_status	string	`"in_progress"` or `"completed"`
termination_reason	string \| null	Why the game ended (if it did)

Important: You must place the box whose id matches current_box.id. You cannot skip or reorder boxes.

GET/challenge/api/status/{game_id}

Get the current state of a game. Useful for debugging or resuming after a crash.

Response

Field	Type	Description
game_id	string	The game session ID
game_status	string	`"in_progress"` or `"completed"`
mode	string	`"dev"` or `"compete"`
boxes_placed	int	Number of boxes placed so far
boxes_remaining	int	Boxes left in the queue
density	float	Current packing density
placed_boxes	array	All placed boxes with current positions
current_box	object \| null	Next box to place

POST/challenge/api/stop

End a game early. Your score is the density at time of stopping.

Request Body

Field	Type	Description
game_id*	string	The game session to stop
api_key*	string	Your API key

GET/challenge/api/my-games

Retrieve all your games and a summary of your stats. Requires your API key.

Query Parameters

Field	Type	Description
api_key*	string	Your API key
mode	string	Filter by `"dev"` or `"compete"`
status	string	Filter by `"in_progress"` or `"completed"`

Response

Field	Type	Description
api_key	string	Your API key
display_name	string \| null	Your display name
summary	object	Aggregate stats: total/completed/in-progress game counts, avg & best compete density, games today, daily limit
games	array	All matching games (newest first), each with game_id, mode, status, density, boxes_placed, total_boxes, termination_reason, timestamps

GET/challenge/api/health

Health check. Returns {"status": "ok"}.

Coordinate System

The truck uses a right-handed coordinate system with the origin at the front-bottom-left corner.

        Z (up, height: 2.75m)
        ^
        |
        |
        +---------> Y (width: 2.6m)
       /
      /
     v  X (depth: 2.0m, loading direction)

Axis	Direction	Range
X	Depth (loading direction, front → back)	0 → 2.0 m
Y	Width (left → right)	0 → 2.6 m
Z	Height (floor → ceiling)	0 → 2.75 m

Position

The position field is the center of the box. To place a box flush on the floor in the front-left corner:

// Box dimensions: [length, width, height] = [0.4, 0.3, 0.2]
"position": [0.2, 0.15, 0.1]  // half-extents from origin

Orientation

Quaternion in [w, x, y, z] format. Common rotations:

Rotation	Quaternion	Effect
Identity	`[1, 0, 0, 0]`	No rotation
90° around Z	`[0.707, 0, 0, 0.707]`	Swap length ↔ width
90° around Y	`[0.707, 0, 0.707, 0]`	Swap length ↔ height
90° around X	`[0.707, 0.707, 0, 0]`	Swap width ↔ height

Tip: Quaternions are auto-normalized by the API. You don't need to worry about exact unit length, but near-zero magnitude will be rejected.

Error Handling

All errors return a JSON body with error (code) and message (human-readable) fields.

Error Code	HTTP	Description
invalid_api_key	401	API key is missing, invalid, or revoked
invalid_game_id	404	Game not found or already completed
invalid_box_id	400	box_id doesn't match the current box
rate_limited	429	Daily compete-mode game limit reached
(validation)	422	Invalid position, degenerate quaternion, missing fields

// Example error response
{
  "error": "invalid_box_id",
  "message": "Expected box_id '209311', got '108148'"
}

Tips for Building Your Agent

Start with dev mode

Use "mode": "dev" while building. Boxes land exactly where you place them — no physics surprises. Switch to "compete" when you're ready to test against real physics and get on the leaderboard.

Use placed_boxes for planning

Every /place response includes the full placed_boxes array with post-physics positions. In compete mode, boxes may shift from where you placed them. Use the settled positions to compute valid placement locations for the next box.

Think about stability

Placing boxes without support underneath will cause them to fall and potentially topple the stack. If 3 or more boxes displace by more than 10cm, the game terminates with unstable.

Optimize rotations

Each box can be placed in 6 orientations (3 axes × 2 flips that matter). Trying all valid rotations for each box is a key optimization. A box that doesn't fit one way might fit perfectly rotated.

Rate limits

Dev Mode

Unlimited

No daily game limit

Compete Mode

50 / day

Per API key

Leaderboard

25 games min

IQM of most recent 50 games

Interactive docs

Swagger UI is available at /docs for interactive API exploration.

Happy packing. We'll see you on the leaderboard.

Solve a hard problem.Win $50K.

Prize Structure

How It Works

Play the Game & Get Your API Key

Build Your Agent

Top the Leaderboard

Scoring

How You Win

Public Leaderboard

Private Evaluation

Validation

Rules

API Reference & Guide

Overview

Modes

Truck Segment Dimensions

Quickstart

Start a game

Place the box

Repeat until done

Game Loop

Termination Conditions

Endpoints

Request Body

Response

Request Body

Response

Response

Request Body

Query Parameters

Response

Coordinate System

Position

Orientation

Error Handling

Tips for Building Your Agent

Start with dev mode

Use placed_boxes for planning

Think about stability

Optimize rotations

Rate limits

Interactive docs

Solve a hard problem.
Win $50K.