Why RL beats heuristics on the factory floor
A retrospective on replacing a rule-based scheduler at Dekali with an RL policy — and the silent rules nobody had written down.
When I joined the production-planning team at Dekali, they were running a heuristic scheduler that had grown for four years. Every week somebody added a rule because of a specific customer escalation. Nobody had deleted a rule in eighteen months. The result was a 1,400-line if/else chain that worked on average and failed catastrophically twice a quarter — usually at the worst possible moment, when a high-priority order arrived and the scheduler "found" an empty slot two weeks away because three operators happened to be on holiday on the only days that machine had the right tool fitted.
The move to Reinforcement Learning was framed inside the team as a research bet. In hindsight, it was an organisational change disguised as a technical one. Replacing 1,400 lines of conditionals with a learned policy forced us to articulate, for the first time, what we were actually optimising. The honest answer was: we did not know. The heuristic was a fossil of every operator's implicit knowledge.
We built a discrete-event simulator of the plant first — orders arriving, machines breaking, operators ending shifts, materials missing. The simulator was the real artefact. Once we trusted it, we trained a PPO policy against it with three reward components: makespan, operator-skill match, and machine-changeover cost. Our first reward was wrong: it weighted makespan too heavily, and the policy learned to game the metric by stalling low-priority jobs indefinitely. The second reward shape was closer, but the policy preferred operators it had seen often during training and avoided the ones whose shifts only overlapped with rare orders. We added an entropy bonus over the operator distribution and the problem went away.
The deployment was the hard part. We did not flip a switch — we shadow-ran the policy for six weeks, logging every disagreement between the policy's plan and the supervisor's actual plan. Eighty percent of disagreements were the policy being right. The remaining twenty percent were the supervisor knowing something the policy did not — that operator X was scheduled for an internal training, that the toolbox for machine M2 was being repaired. Those twenty percent became features the next week.
Two non-obvious lessons. First: capture the silent rules. Every shift supervisor in a manufacturing plant has a half-page of priorities only they know. RL does not learn those rules by watching the data — it learns to be right on average and wrong on exactly the cases the supervisor handles in their head. You have to interview the supervisor, surface those rules, and bake them into the reward or the action mask. Second: explainability beats accuracy. The day the model is right but unreadable is the day trust collapses, and trust is the only currency you have in an SME. We added a "why" panel that tells the operator which constraints drove a particular schedule, in plain Italian. Adoption tripled the week we shipped it.