Early in building Botpylon's dispatch engine, we relied on a static congestion map: a zone-by-zone weight overlay derived from initial floor mapping and a few days of observed traffic patterns. It worked reasonably well for the first few hours of a shift. Then the warehouse would change, a wave of high-priority orders would shift traffic density to a different area of the floor, and the static weights would become wrong. We were routing robots away from last hour's congestion into this hour's new bottleneck.
The core problem with static congestion modeling is that warehouse floor traffic is not stationary. Congestion hotspots shift based on time-of-day wave patterns, incoming dock activity, robot battery charge states clustering at charging stations, and sudden demand spikes on specific SKU zones. A map built at 8 AM is meaningfully wrong by 10 AM.
We spent a few months exploring rule-based dynamic weighting, adjusting path costs based on observed robot density in real time. That helped. But the rule system kept growing, and the interaction effects between rules created cases we had not anticipated. The floor graph had become a maintenance problem as much as an engineering problem. That pushed us toward a learned approach.
Why Reinforcement Learning, Not Supervised Learning
The first instinct when you want to predict congestion is to treat it as a supervised regression problem: given current floor state as features, predict congestion level at each zone N seconds from now. You can train this on historical shift data, validate it on held-out shifts, and it will achieve reasonable accuracy on typical patterns.
The problem is that a supervised prediction model tells you what congestion will look like if dispatch continues unchanged. But we are the dispatch engine. We change what robots do, and those changes affect congestion outcomes. A supervised model trained on historical data learns the correlations under historical dispatch decisions, not the correlations under optimal dispatch decisions. The model's training distribution diverges from its deployment distribution as soon as we start acting on its outputs.
Reinforcement learning frames the problem differently. The agent (our dispatch policy) takes actions (task assignments to specific robots), observes the resulting floor state transitions, and receives a reward signal based on how well the fleet performed. Over many episodes, the policy learns to select task assignments that lead to low congestion and high throughput, not just to predict what congestion would be under a fixed prior policy.
We are not claiming RL is always the right tool for warehouse operations problems. It is harder to train, harder to debug, and its behavior in edge cases can be surprising in ways that rule-based systems are not. For congestion-aware dispatch specifically, the closed-loop nature of the problem made supervised prediction insufficient.
State Representation: What the Policy Sees
Defining the state space for a warehouse floor RL policy requires some care. The full state of a warehouse with 80+ robots is high-dimensional, and most of that information is not useful for the dispatch decision at any given moment.
We represent floor state as a graph where nodes are floor zones (defined by the physical zone grid, typically 3x3 meter cells) and edges represent navigable paths between adjacent zones. Each node carries a state vector encoding: current robot density (robots per unit area), inbound traffic rate (robots heading toward this zone in the last 10 seconds), task queue depth in this zone (pending pick tasks assigned to positions within this zone), and historical congestion frequency for this zone over the last 20 minutes.
Robot-level state feeds into zone-level aggregates rather than being tracked at the individual robot level in the policy input. This keeps the state space bounded regardless of fleet size and makes the policy more generalizable across different floor configurations and robot counts.
The action space is a task assignment: given a robot that just became available and a queue of pending tasks, select which task to assign to which robot. The policy outputs a ranking over candidate task-robot pairs, and we select the top-ranked assignment that passes safety and precedence constraints.
The Reward Function Design
Getting the reward function right took more iterations than the policy architecture. The obvious choice, maximize pick rate, sounds correct but creates problems. A policy trained purely on pick rate learns to assign all robots to the highest-density pick zones, which creates severe congestion in those zones and actually reduces throughput. The reward needs to encode the tradeoff between task completion and floor state health.
Our current reward function combines three components. The first is task completion reward: a positive signal for each task completed within its expected travel time. The second is congestion penalty: a negative signal proportional to the number of robot-seconds spent in stop-and-wait states across the fleet in each time step. The third is zone balance reward: a small positive signal for distributing active tasks across floor zones rather than concentrating them.
The congestion penalty component required careful tuning. Too high, and the policy becomes overly conservative, avoiding any zone with more than two robots even when throughput requires it. Too low, and congestion still builds at peak load. We found that weighting congestion penalty at roughly 30-40% of the task completion reward, with the exact ratio tuned per floor configuration, produced the right tradeoff for the deployments we have seen.
Training Infrastructure and Episode Construction
Training this policy in production is not feasible: you cannot run thousands of training episodes in a live warehouse. We built a simulation layer that replays historical shift data, allowing the policy to explore different dispatch decisions and observe counterfactual outcomes.
The simulation uses recorded robot trajectories and task streams from real shifts, then perturbs the dispatch decisions to explore alternatives. For each episode step, the policy selects a task assignment, and the simulation advances the floor state using a physics-approximated movement model for each robot. This is not a perfect physics simulation, it uses simplified kinematic models that match typical AMR and AGV behavior reasonably well, but the simulation gap is a real limitation we account for with conservative deployment margins.
We used Proximal Policy Optimization (PPO) as the base algorithm. PPO's clipped objective prevents large policy updates that could destabilize training on the relatively noisy warehouse simulation environment. Episodes are constructed from full 8-hour shifts extracted from historical data, with the episode start time randomized to expose the policy to different shift phases (shift start, mid-shift steady state, end-of-shift drawdown, wave transitions).
Training converges within roughly 2,000 episodes per floor configuration, which at 8-hour simulated episodes means we are learning from the equivalent of about 16,000 hours of shift data. This is feasible on a single GPU in overnight runs. The policy checkpoints are validated on held-out shifts before deployment.
What the Policy Actually Learned
After training, the policy exhibits several behaviors we did not explicitly encode as rules. During shift wave transitions, when WMS releases a large batch of new pick tasks, the policy spreads initial assignments across multiple floor zones rather than dispatching all newly available robots toward the heaviest pick zone. This effectively pre-distributes robot density before congestion builds, which is a form of predictive avoidance that rule-based systems achieve only with explicit look-ahead logic.
The policy also learned to distinguish between transient high-density zones (two robots briefly converging at an aisle intersection) and persistent congestion zones (a pick area that stays at high density because task queue depth keeps new robots flowing in). It routes aggressively through the former and avoids the latter, which is the right behavior but required no explicit rule.
One behavior we did not anticipate: the policy sometimes defers a high-priority task assignment for 2-3 seconds to allow a nearly-complete zone clearance before routing a robot into the area. This delays individual task completion slightly but reduces the cascade effect of sending a robot into a zone at peak density. The congestion penalty in the reward function was enough for the policy to learn this deferral strategy without it being explicitly specified.
Deployment Constraints and Honest Limitations
We run the RL policy as the path-weight update component of the floor graph engine, not as the sole dispatch decision-maker. The policy outputs updated edge weights for the floor graph every 15 seconds. The dispatch engine then uses those updated weights for route planning and task assignment. This separation means the policy's outputs go through the same safety and constraint-checking layer as any other input to the dispatcher.
The policy does not generalize perfectly across all floor configurations. A policy trained on a two-floor warehouse layout with 85 AMRs will perform well on similar layouts but will need retraining for a substantially different floor geometry or a significantly different robot count. We treat floor-specific policy training as part of the onboarding process for new deployments.
We also do not use the RL policy for intraday cold starts when the floor is in an unusual state with no historical data to inform the initial zone weights. In those cases, we fall back to rule-based initialization and allow the learned policy to take over after the first 30-45 minutes of shift data accumulates. The boundary between these modes is explicit and logged, which matters for operators who want to understand why the dispatch engine behaved in a particular way at shift start.
The performance improvement over static congestion weighting is consistent in our simulation benchmarks: roughly 18-28% reduction in fleet-wide stop-and-wait time during peak load periods, with most of the gain coming from wave transition handling where static maps fail the most. Whether those simulation gains translate fully to production depends on how closely the simulation matches the specific floor, which is why we treat the simulation gap honestly rather than claiming exact production numbers.