A fixed-N A/B test sends half your traffic to the worse variant for the whole run, then stops at a preset sample size. BayesDecide uses Thompson sampling to route more traffic to the variant that looks best while staying calibrated, and a stopping rule promotes a winner with no fixed-N A/B test.
BayesDecide runs your prompt variants as a live experiment. For each request it picks a variant, you serve
it, and you tell it the reward you saw. It steadily sends more traffic to the variant that looks best while
staying calibrated, so you stop paying the cost of a worse variant, and when the evidence is strong enough
it tells you which one won. Your code does not change shape: you call experiment.select(),
serve the variant, then experiment.observe(variant, reward).
Each variant carries a Beta-Bernoulli posterior over its reward rate, or a Gaussian arm for unbounded metrics like latency and cost. Every reward in [0, 1] is folded in with a closed-form conjugate update in O(1) with zero inference latency. Thompson sampling draws from each posterior and routes to the best draw, best-arm identification is estimated by Monte Carlo, and a Bayesian stopping heuristic or an anytime-valid confidence sequence decides when to stop. Per-cohort categorical context, time-decay, an advisory promotion engine, and a tamper-evident hash-chained audit trail.
Take a real production scene: you have four prompt variants in front of live traffic, one is clearly better, and you need to find it without burning weeks of conversions on the losers.
decide returns a winner.The "With BayesDecide" timeline is the behavior of the calibrated decision as rewards accumulate; the stopping rule decides when to promote, and the promotion engine is advisory, it recommends a winner and does not move traffic itself.
install to your first calibrated decision.pnpm add @takk/bayesdecidenpm install @takk/bayesdecideyarn add @takk/bayesdecidebun add @takk/bayesdecideThe core has no runtime dependencies and a node-free core. Every @takk sibling is an optional
peer; install only what you compose with.
# nothing else is required to start experimenting
# optional: install a sibling you compose with, for example
pnpm add @takk/keymesh @takk/modelchainimport { createExperiment } from '@takk/bayesdecide';
const experiment = createExperiment({
variants: [{ id: 'concise' }, { id: 'detailed' }],
confidence: 0.95,
});
// Pick a variant with Thompson sampling, serve it, then record the reward.
const { variant } = experiment.select();
// ...serve the variant, score the response...
experiment.observe(variant, reward); // reward in [0,1]
const decision = experiment.decide(); // { done, best, probabilityBest, reason }import { createExperiment } from '@takk/bayesdecide';
import { createPromptRouter } from '@takk/bayesdecide/router';
const experiment = createExperiment({
variants: [{ id: 'concise' }, { id: 'detailed' }],
confidence: 0.95,
});
// A framework-agnostic adapter: bind to Vercel AI SDK, Mastra, Genkit, raw
// fetch, or a non-human-entity loop. It selects, you serve, you report reward.
const router = createPromptRouter(experiment);
const { id, report } = router.route();
// ...call the model with prompt `id`, then:
report(reward); // reward in [0,1]A Beta-Bernoulli posterior per variant, updated in closed form on each reward. Each request draws from every posterior and routes to the best draw, so traffic follows the evidence.
More traffic reaches the variant that looks best while it stays calibrated, instead of an even split paying for the losers all run long.
The Beta-Bernoulli arm takes any reward in [0, 1], binary or a fractional LLM-as-judge score. A Gaussian arm models unbounded metrics like latency and cost.
Optimize a judge score and a cost metric with the same experiment, instead of forcing every signal through a single binary outcome.
The probability each variant is best is estimated by Monte Carlo over the posteriors, so you read how confident the experiment is, not just a point estimate of the mean.
You get a calibrated probability the winner is really the winner, and a reproducible, auditable decision for a given seed.
A Bayesian stopping heuristic fires when the probability the best variant is best clears a threshold. An anytime-valid confidence-sequence mode controls error even under unlimited peeking.
Promote early when the gap is obvious, and check the dashboard as often as you like in the anytime-valid mode without inflating the false-positive rate.
A categorical context splits one experiment into independent posteriors per cohort, so a variant that wins for new users and loses for power users is never pooled into one averaged number.
You learn the right winner for each cohort separately, instead of shipping a prompt that is only best on average.
When a stopping rule fires, the promotion engine returns a recommended winner with the probability it is best and the reason. It is advisory: it recommends, it does not move traffic itself.
A human or a non-human entity stays in control of the rollout, with a clear, logged recommendation rather than a silent auto-route.
An append-only log of every selection and decision, with a SHA-256 hash-chain you append to
and verify via the Web Crypto API.
Prove the experiment's decisions were made on the recorded evidence, in Node, edge runtimes, or the browser, when a promotion is questioned.
Snapshot the posteriors to portable JSON and restore them on the next run. Opt-in time-decay relaxes stale evidence toward the prior, so an old run does not anchor a new one.
Learning survives restarts and cold starts, and a variant that won a month ago does not weigh on today's decision forever.
Every published version signed with npm publish --provenance through GitHub Actions OIDC.
Lockfile committed, zero required runtime dependencies, node-free core.
Verify in one command that the tarball you installed was built from the source commit you trust.
Pass a prior globally, or override it per variant. The confidence target stays yours; the
inference is the library's.
| Prior | What it encodes | When to use it |
|---|---|---|
UNIFORM_PRIOR |
Beta(1, 1): every reward rate equally likely before any evidence. | Default. No prior knowledge about a variant yet. |
JEFFREYS_PRIOR |
Beta(0.5, 0.5): the reference objective prior for a Bernoulli rate. | When you want the standard objective prior with minimal influence. |
OPTIMISTIC_PRIOR |
Beta(8, 2): a high reward rate assumed until evidence says otherwise. | A new variant you expect to do well should be explored before it is written off. |
perVariantPrior |
A per-variant override map, so each prompt starts from its own profile. | A known-good baseline and a fresh experimental prompt should not share one belief. |
Any Beta prior is accepted; pass your own pseudo-counts as Beta(alpha, beta) globally or per
variant. The Gaussian arm takes a Normal-inverse-gamma prior for unbounded metrics.
| Entry point | Subpath export | Runtime | Use it when |
|---|---|---|---|
| Facade | @takk/bayesdecide |
node-free | You want the createExperiment facade and the full toolkit. |
| Prompt router | @takk/bayesdecide/router |
node-free | You want the adapter that binds to Vercel AI SDK, Mastra, Genkit, or raw fetch. |
| Stopping rules | @takk/bayesdecide/stopping |
node-free | You want the Bayesian heuristic or the anytime-valid stopping mode on its own. |
| Node store | @takk/bayesdecide/node |
Node only | You need durable, file-backed persistence that survives restarts. |
The bayesdecide binary runs the compiled engine, so every number it prints comes from execution.
simulate compares the adaptive bandit to a uniform A/B test on a synthetic set of variants;
replay folds a recorded sequence of rewards and prints the decision for the next selection.
# compare adaptive bandit vs uniform A/B on the same synthetic variants
npx @takk/bayesdecide simulate \
--n 2000 \
--variants 4# variant,reward pairs, reward in [0,1], from a file or stdin
echo "concise,1 detailed,0 concise,1 detailed,1" | npx @takk/bayesdecide replay
npx @takk/bayesdecide --version
npx @takk/bayesdecide help
# exit codes: 0 ok, 64 usage, 65 data error, 66 missing inputRecord every selection and decision in a tamper-evident, hash-chained log. And because the decision is
explicit, decide returns the best variant, the probability it is best, and the reason, so a
promotion, or a wait, is never a black box. Both are node-free and need no external service.
import { AuditLog } from '@takk/bayesdecide/audit';
const log = new AuditLog();
await log.append({
variant: 'concise', reward: 1,
probabilityBest: 0.82, decided: false,
reason: 'still-exploring', at: Date.now(),
});
await log.verify(); // true, until any entry is alteredconst decision = experiment.decide();
// {
// done: true, // the stopping rule has fired
// best: 'concise', // the recommended winner (advisory)
// probabilityBest: 0.97, // P(best variant is best), by Monte Carlo
// reason: 'threshold-reached'
// }
The reason is one of threshold-reached, anytime-valid-stop, max-samples, still-exploring, or min-samples, so
you can log and alert on exactly why a winner was or was not promoted. Wire the observer hook to
emit an OpenTelemetry span per decision and a metric per reward, with no runtime dependency, the governance
seam for a non-human entity's prompt selection.
The other approaches solve part of the problem. The contrast clarifies where BayesDecide sits.
| Capability | BayesDecide | Fixed-N A/B test | epsilon-greedy | Feature-flag SaaS | Hand-rolled |
|---|---|---|---|---|---|
| Distribution | npm library | a spreadsheet | npm library | hosted service | your repo |
| Calibrated traffic routing | Thompson sampling | even split | fixed epsilon | varies | rarely |
| Best-arm identification | Monte Carlo | manual | no | partial | no |
| Anytime-valid stopping | yes | fixed-N only | no | some | no |
| Unbounded-metric arm | Gaussian | manual | no | varies | rarely |
| Tamper-evident audit | yes | no | no | partial | no |
| Zero deps, node-free | yes | n/a | deps | no | varies |
| License | Apache-2.0 | n/a | MIT | proprietary | your call |
The honest summary: a fixed-N A/B test or a simple epsilon-greedy loop is fine when traffic is cheap and you can wait for a preset sample size. Pick BayesDecide when serving the worse variant costs real money, you want calibrated routing, best-arm identification, an anytime-valid stop, and a tamper-evident trail composed in one node-free TypeScript library you embed in production.
BayesDecide is exact Bayesian inference, and exactness has assumptions. They are documented, and where real traffic breaks them, there is an explicit mitigation rather than a silent fudge.
| Assumption or property | What it means | How BayesDecide handles it |
|---|---|---|
| Independent rewards | Each observation is its own draw from the variant. | A minimum-samples floor and the anytime-valid mode keep early peeks honest while evidence accrues. |
| Stable reward rate | The underlying rate does not drift over time. | Opt-in time-decay relaxes stale evidence toward the prior. |
| Cohort matters | A variant can win for one audience and lose for another. | A categorical context keys separate posteriors per cohort, so regimes are never pooled. |
| Reward in [0, 1] | The Beta-Bernoulli arm expects a bounded reward. | Pass a binary or fractional LLM-as-judge score; use the Gaussian arm for unbounded metrics. |
| Calibrated probability | P(best) is a Monte Carlo estimate, not an exact integral. | A seeded sampler with a configurable draw count trades estimate variance for compute. |
| Heuristic vs valid stop | The default stop is a P(best) threshold, not an error-controlled test. | The anytime-valid confidence-sequence mode controls error even under unlimited peeking. |
| Advisory promotion | A winner is recommended, not auto-routed. | The promotion engine returns the best variant and the reason; you own the rollout. |
| Integrity, not identity | The audit seal proves a log was not altered after sealing. | A SHA-256 hash chain via Web Crypto, an integrity seal, not a signature. |
106 tests passing under Vitest 4 with 96% coverage. The benchmark cuts a uniform A/B
test's regret by up to 99.5%, regret 4.4 against 800 in the 2-variant large-gap scenario. Run
pnpm test on a fresh clone to reproduce.
TypeScript 6 in maximum strict mode (exactOptionalPropertyTypes,
useUnknownInCatchVariables, noUncheckedIndexedAccess,
noImplicitOverride, noImplicitReturns). Zero errors under tsc
--noEmit.
Biome 2.5 clean across src and tests. publint clean and
@arethetypeswrong/cli green across all sixteen subpaths. Dual ESM + CJS with separate
.d.ts and .d.cts per subpath.
The special functions reproduce known identities: exp(lgamma(5)) equals 24, the Beta(1, 1)
CDF is the identity, and the quantile inverts the CDF. The credible interval is the true Beta quantile, not
an approximation.
A smoke test exercises the compiled ESM and CJS artifacts and spawns the compiled CLI as a single Node process, run on the Node 20, 22, and 24 matrix.
Committed pnpm lockfile, zero required runtime dependencies, node-free core, and SLSA provenance
attestation on every published version. Verify with npm view @takk/bayesdecide@1.0.0 --json | jq
.dist.attestations.
Yes. 106 tests pass under Vitest 4 with 96% coverage; TypeScript 6 maximum strict mode is clean; Biome 2.5
lint is clean; publint and @arethetypeswrong/cli are clean across all sixteen
subpaths. The suite and the distribution smoke test are run on the Node 20, 22, and 24 matrix. Every
published release carries SLSA provenance produced by GitHub Actions.
A fixed-N A/B test holds an even split for the whole run, so it keeps paying the cost of every losing variant until it hits a preset sample size. BayesDecide uses Thompson sampling to send more traffic to the variant that looks best while staying calibrated, and a stopping rule promotes a winner when the evidence is strong. In the 2-variant large-gap benchmark that cuts the uniform A/B test's regret by up to 99.5%, regret 4.4 against 800.
No. The promotion engine is advisory: decide returns a recommended winner with the probability
it is best and the reason, but it does not move traffic itself. You stay in control of the rollout, a
deliberate seam so a human or a non-human entity owns the promotion rather than a silent auto-route.
The default stopping rule is a Bayesian decision heuristic: it fires when the probability the best variant is best clears a threshold. The anytime-valid mode is an error-controlled confidence-sequence test that stays valid even under unlimited peeking, so you can check it as often as you like without inflating the false-positive rate. The docs are explicit about which guarantee each one gives.
Yes. The core is node-free; the audit seal uses the Web Crypto API, not node:crypto. Import
@takk/bayesdecide on any runtime with Web Crypto, and bind the prompt router to Vercel AI SDK,
Mastra, Genkit, raw fetch, or a non-human-entity loop. Only @takk/bayesdecide/node requires the
Node standard library.
BayesDecide records the variant identifiers and cohort labels you choose, the posterior parameters, the observed reward counts, and timestamps. It never sees your prompts or model outputs and makes no outbound network calls of its own. With the file store, only that experiment state reaches disk; no secrets are involved at any point.
A new variant starts at its prior, the uniform Beta(1, 1) by default, so it is explored early before the
data can write it off. To start from a better belief, pass OPTIMISTIC_PRIOR or a
perVariantPrior override so a promising new prompt gets sampled until it has data of its own.
In-process memory by default, with portable JSON snapshots via snapshot() and
load(). For durability across restarts, use createFileStore from
@takk/bayesdecide/node. For an edge runtime, snapshot to your own KV store between invocations.
No, and the docs are explicit about it. The seal is a SHA-256 hash-chain root that proves a log was not altered after sealing. It is an integrity seal, not a signature: it does not prove who produced the log. Signed and timestamped seals for stronger third-party evidence are on the roadmap.
For the Beta-Bernoulli arm, a reward in [0, 1]: a binary success or a fractional LLM-as-judge
score. For unbounded metrics like latency or cost in dollars, use the Gaussian arm, which models a real
value with a Normal posterior. Pick the arm that matches the signal you are optimizing.
Every release is published with npm publish --provenance. Check the attestations with
npm view @takk/bayesdecide@<version> --json | jq .dist.attestations. The attestation
links the tarball you installed to the GitHub Actions workflow that built it from a specific source commit.
Yes. Snapshot the posteriors to portable JSON with snapshot() and restore them with
load(), persisting that JSON wherever you like, SQLite, Postgres, Redis, or a KV store. The
createFileStore file store is the zero-dependency reference for Node.
Strict SemVer 2.0.0, starting from 1.0.0. The binding stability surface is documented in SPEC.md section 5. Major bumps require a deprecation cycle; security fixes follow the disclosure flow in SECURITY.md.