Bandits are the reinforcement learning you will actually use

Forty promotional banners for the homepage, nobody knows which ones work, the set turns over every few weeks, and the product manager wants the system to keep learning rather than freezing on one winner. Someone says reinforcement learning. Six weeks later there is a half-built simulator of user behaviour that nobody trusts and not one banner has been ranked. The problem was a bandit, and a bandit would have shipped in three days.

Full RL asks for things a business does not have

Reinforcement learning is a genuinely different way to learn: no labelled answers, just consequences, and an agent that must work out which of its actions earned them. It needs three things games supply free and products do not.

It needs cheap exploration. AlphaGo could play millions of games against itself at no cost. Your checkout page gives a few thousand sessions a day, and every bad action is a real customer having a worse time. Without a simulator you cannot afford the sample counts, and a simulator of human behaviour good enough to train against is harder than the original problem.

It needs credit assignment over a long horizon. The reward arrives at the end and something in the middle of the sequence earned it — that is the part of RL that took decades of research. In a product the sequence is also polluted by everything else happening to the user, so the signal is weaker and the horizon longer.

And it needs a reward function that survives an optimiser attacking it. An agent told to maximise session length learns to make the interface slow to navigate. That is the default outcome of handing a proxy to an unconstrained learner.

A bandit is RL with the state removed

Strip the sequence out. There is a set of actions — the arms. Pull one, get a reward straight away, and nothing about the world changes as a result: the next decision starts where the last one did.

That single simplification deletes credit assignment, the simulator, the discount factor, the value function and most of the reward engineering. What survives is the explore-exploit trade: spend a pull on an arm you are unsure about, or take the one that currently looks best. That is the interesting half of reinforcement learning, and the half that shows up constantly in ordinary work — which banner, which price band, which of six subject lines, which ranking model gets the next request.

Three ways to choose an arm

Epsilon-greedy. With probability epsilon, pull a random arm; otherwise pull the best one so far. It is four lines of code and it works. The weakness is undirected exploration: with forty arms and epsilon at 0.1, it keeps spending pulls on arms it has already established are terrible. It also needs a decay schedule, which becomes a constant nobody revisits.

UCB. Pull the arm with the highest mean plus a bonus that grows with how little you have tried it — optimism in the face of uncertainty. Exploration goes to the genuinely uncertain arms, and the regret guarantees are strong: Auer, Cesa-Bianchi and Fischer’s 2002 analysis is where the standard bound comes from. The awkwardness is practical: the bonus is tied to the reward scale, so it needs care with anything that is not a bounded rate, and being deterministic it hands the same arm to every request until the next update lands — thousands of identical pulls under batched traffic.

Thompson sampling. Keep a posterior over each arm’s reward rate, draw one sample from each posterior, pull the argmax. Arms explore in proportion to the probability that they are best, which is the right amount, and because the choice is random it spreads across simultaneous requests, so batching and delayed updates hurt much less. The method is Thompson’s, from 1933, and it sat unused for decades. For a conversion rate the posterior is a beta distribution and the whole thing is a counter per arm, drawn with Generator.beta:

import numpy as np

rng = np.random.default_rng(0)
wins   = np.ones(n_arms)   # beta prior: one imagined success
losses = np.ones(n_arms)   # and one imagined failure

def choose():
    return int(np.argmax(rng.beta(wins, losses)))

def update(arm, converted):
    if converted:
        wins[arm] += 1
    else:
        losses[arm] += 1

Thompson sampling is the sensible default; reach for UCB when the guarantee has to be on paper.

Context, and when a plain test is still better

A contextual bandit picks the arm using features of the request — country, device, referrer, what the user looked at last. In practice that means a model predicting reward from context and arm, with exploration layered on top: sampled model parameters, or a per-arm bonus, rather than sampled counters. It is where bandits stop competing with A/B tests and do something a test cannot: learn a different answer for different users.

Bandits win when the arms are many, the traffic is continuous, and one wrong impression costs little. Forty banners in a fixed test means splitting traffic forty ways and showing thirty-nine losers to real users for the whole window; a bandit starves the bad ones within days. They also win when the answer expires — seasonality, a catalogue that turns over, a promotion that gets stale — because a bandit has no end date and a test does.

They lose whenever the point is a clean causal readout. The allocation shifts over time, so a naive comparison of arm means is confounded with everything else that moved during the run, and reading a bandit’s output as if it were an experiment reintroduces exactly the failure modes a proper test is designed to avoid. If the number goes into a finance forecast or a launch decision, run the test; two arms and a ship-or-not call is a test, not a bandit. And if the reward only lands after thirty days — retention, refunds, lifetime value — there is nothing to learn from inside the window where the choice has to be made.

Three traps

Delayed rewards. A bandit optimises what it can observe, so when one arm’s reward arrives quickly and another’s slowly, it converges on the fast one, not the better one. Define the reward on a window the algorithm sees, and watch the long-horizon metric separately.

Non-stationarity. The standard formulation assumes fixed arm rates. After a month of traffic the posteriors are so concentrated that a genuinely better challenger cannot get enough pulls to prove itself. Discount old observations — multiply the counters by 0.99 each day — or floor the exploration so no arm ever drops below a minimum share.

It optimises the metric you gave it. Reward clicks and you get the banner that looks most like a mistake. This is reward hacking — the problem that makes full RL dangerous — arriving through a much smaller door.

One more thing, free at the time: log the arm chosen and the probability of choosing it. Those propensities let you evaluate a new policy against traffic you already served, without running anything. Teams that skipped the column always want it later, and it cannot be reconstructed.

A reasonable way to start this week: take a surface where the team already runs A/B tests and the reward lands within a day, put its variants behind Thompson sampling, and log the arm and its propensity from the very first request. Four arms is plenty. Discounting, non-stationarity and off-policy replay can all wait until that one is serving traffic and someone has read a week of its logs.