A fixed cutoff, "accept above 0.8", is a guess that ignores how good and bad outputs actually separate. BayesOutputGate calibrates a high-quality and a low-quality hypothesis from labeled history, weighs each new output's scores with a Bayes Factor on the Jeffreys scale, and stamps it pass, fail, or escalate.
BayesOutputGate is the passport control for your outputs. It reads the labeled history of known-good and
known-bad outputs, calibrates a model of each, and for every new output reports the Bayes Factor between a
high-quality and a low-quality hypothesis, then stamps it pass, fail, or escalate, with the evidence and the
rationale attached. Your code stays small: you call evaluate(scores, models, policy) and read the
decision.
For each dimension the engine fits a Beta density to the scores of known-high outputs and another to known-low outputs, calibrated by method of moments and updatable online. The Bayes Factor is the ratio of the likelihoods, summed in log space across weighted dimensions, interpreted on the Jeffreys scale. A pure policy thresholds the evidence; a decision-theoretic policy turns it into a posterior and minimizes expected loss. Goodness-of-fit and dependence diagnostics, measured calibration, a tamper-evident hash-chained audit trail, and an MCP and LLM tool round it out.
Take a real production scene: an LLM-as-judge returns quality scores for every output, and you must decide whether to ship each one, accepting a bad output misleads a user, rejecting a good one wastes work.
The "With BayesOutputGate" path is the behavior of the calibrated Bayes Factor; the gate produces the recommendation, pass, fail, or escalate, and your orchestrator acts on it. The gate decides on evidence calibrated from your own labeled data, not on a constant you picked.
install to your first calibrated verdict.pnpm add @takk/bayesoutputgatenpm install @takk/bayesoutputgateyarn add @takk/bayesoutputgatebun add @takk/bayesoutputgateThe core has no runtime dependencies and a node-free core. Every @takk sibling is an optional
peer; install only what you compose with.
# nothing else is required to start gating outputs
# optional: install a sibling you compose with, for example
pnpm add @takk/keymesh @takk/modelchainimport { evaluate, HypothesisManager } from '@takk/bayesoutputgate';
// Calibrate the two hypotheses from labeled outputs scored by any scorer you run.
const manager = new HypothesisManager();
manager.fit([
{ scores: [{ dimension: 'factuality', value: 0.96 }], label: 'high' },
{ scores: [{ dimension: 'factuality', value: 0.12 }], label: 'low' },
]);
// Weigh a new output's scores and decide.
const decision = evaluate(
[{ dimension: 'factuality', value: 0.88 }],
manager.models(),
{ kind: 'bayes-factor', passAbove: 10, failBelow: 0.1 },
);
console.log(decision.action); // 'pass' | 'fail' | 'escalate'
console.log(decision.bayesFactor); // evidence ratio for high quality over low
console.log(decision.strength); // Jeffreys-scale labelimport { evaluate } from '@takk/bayesoutputgate';
// Two errors are not equally costly: shipping a bad output beats rejecting a good one.
const decision = evaluate(scores, models, {
kind: 'decision-theoretic',
priorHighQuality: 0.8, // base rate of good outputs
lossFalsePass: 10, // cost of shipping a low-quality output
lossFalseFail: 1, // cost of rejecting a high-quality output
escalationCost: 0.5, // cost of routing to a human reviewer
});
console.log(decision.action); // expected-loss-minimizing action
console.log(decision.posteriorHighQuality); // P(high quality | scores)
console.log(decision.expectedLoss); // { pass, fail, escalate }For each dimension the engine fits a Beta density to known-high and known-low scores, then the per-dimension density ratio, summed in log space, is the combined Bayes Factor, read on the Jeffreys scale.
You decide on calibrated evidence with a meaning, substantial, strong, decisive, instead of a constant you guessed.
Likelihoods are calibrated by method of moments from labeled outputs, regularized by a prior treated as pseudo-observations, and updated online from sufficient statistics as new outcomes arrive.
The gate gets sharper as labels accrue, with a one-step update and no full refit.
Factuality, fluency, safety, relevance, each carries its own weight in the combined Bayes Factor, so the dimensions that matter more count more, instead of being averaged into one number.
A great factuality score can no longer hide a failing safety score behind an average.
The decision-theoretic policy turns the Bayes Factor into a posterior through an explicit prior, then picks the action that minimizes expected loss under the asymmetric cost of each error type.
Shipping a wrong answer and rejecting a good one are priced separately, the way they cost in production.
A Kolmogorov-Smirnov goodness-of-fit test checks whether the Beta assumption holds, and a dependence diagnostic measures pairwise correlation, because correlated dimensions double-count evidence.
You learn when the independence assumption is unsafe before the Bayes Factor misleads you.
Once the gate emits posterior probabilities, the library scores them against realized outcomes with the Brier score, the expected calibration error, and a reliability diagram.
Calibrated means a number you can verify on your own data, not a word the library asserts.
An append-only log of every decision, with a SHA-256 hash-chain you append to and
verify via the Web Crypto API, the validation evidence the EU AI Act expects.
Prove a decision was made on the recorded evidence, in Node, edge runtimes, or the browser, when it is questioned.
The whole gate is exposed as a framework-agnostic tool, name, description, JSON Schema, handler, that drops into an MCP server or an LLM tool-calling API, with defensive input validation.
A non-human entity validates its own output before acting on it, no SDK to wire in.
Every published version signed with npm publish --provenance through GitHub Actions OIDC.
Lockfile committed, zero required runtime dependencies, node-free core.
Verify in one command that the tarball you installed was built from the source commit you trust.
Choose how the gate decides: threshold the evidence on the Jeffreys scale, or minimize expected loss under your own costs, and weight each dimension and set its priors. The scores are yours; the calibration is the library's.
| Setting | What it encodes | When to use it |
|---|---|---|
kind: "bayes-factor" |
Pass at or above the Bayes Factor, fail at or below, escalate in between, on the Jeffreys scale. | Default. The two error types cost about the same and you want principled evidence thresholds. |
kind: "decision-theoretic" |
Turn the Bayes Factor into a posterior through a prior, then minimize expected loss under asymmetric costs. | Shipping a bad output and rejecting a good one cost differently, the usual production case. |
passAbove / failBelow |
The Bayes Factor pass and fail thresholds; the defaults, 10 and 0.1, are the Jeffreys strong-evidence boundaries. | You want to tune how much evidence a pass or a fail requires before escalating. |
weight / priorHigh / priorLow |
Per-dimension weight in the combined Bayes Factor and the two regularizing Beta priors that anchor a cold model. | A dimension matters more than another, or a known base rate should anchor it until labels accrue. |
Any Beta prior is accepted per dimension, and the goodness-of-fit and dependence diagnostics report when the model assumptions strain. Each new labeled outcome folds into the calibration online from sufficient statistics.
| Entry point | Subpath export | Runtime | Use it when |
|---|---|---|---|
| Gate facade | @takk/bayesoutputgate |
node-free | You want the evaluate gate, OutputGateMonitor, and the full toolkit. |
| Bayes Factor | @takk/bayesoutputgate/bayesfactor |
node-free | You want the combined Bayes Factor and the Jeffreys-scale interpretation on their own. |
| Tool adapter | @takk/bayesoutputgate/adapter |
node-free | You want the framework-agnostic MCP and LLM tool a non-human entity can call. |
| Node loaders | @takk/bayesoutputgate/node |
Node only | You need to load labeled history or score vectors from a JSON or CSV file on disk. |
The bayesoutputgate binary runs the compiled engine, so every number it prints comes from execution.
gate decides each output; bayes-factor computes the evidence ratio;
calibrate reports goodness-of-fit and dependence; audit-verify checks an audit-chain file.
# history is a JSON array of { scores, label }; scores is an array of score vectors, or a CSV
npx @takk/bayesoutputgate gate history.json scores.jsonnpx @takk/bayesoutputgate bayes-factor models.json scores.json
npx @takk/bayesoutputgate calibrate history.json
npx @takk/bayesoutputgate audit-verify chain.json
npx @takk/bayesoutputgate --version
npx @takk/bayesoutputgate --help
# exit codes: 0 ok, 2 usage or input error, 30 fail, 40 escalate, 20 broken chainRecord every decision in a tamper-evident, hash-chained log. And because the verdict is explicit,
evaluate returns the action, the Bayes Factor, the Jeffreys-scale strength, and the per-dimension
contributions, so a pass, a fail, or an escalation is never a black box. Both are node-free and need no external
service.
import { AuditChain, verifyChain } from '@takk/bayesoutputgate/audit';
const chain = new AuditChain();
await chain.append({
action: 'pass', bayesFactor: 42.1,
strength: 'strong-high',
});
await verifyChain(chain.toArray()); // { valid: true }, until any entry is alteredconst decision = evaluate(scores, models, policy);
// {
// action: 'pass',
// bayesFactor: 42.1,
// logBayesFactor: 3.74,
// strength: 'strong-high', // on the Jeffreys scale
// rationale: 'bayes-factor',
// contributions: [
// { dimension: 'factuality', score: 0.88, logBayesFactor: 2.6, weight: 1 },
// { dimension: 'safety', score: 0.91, logBayesFactor: 1.14, weight: 1 },
// ],
// }
The per-dimension contributions show exactly which axis drove the verdict, so you can log and alert on the
evidence, not just the action. The OutputGateMonitor records every decision into the hash-chained
audit log with no runtime dependency, the governance seam for a non-human entity's own outputs.
The other approaches solve part of the problem. The contrast clarifies where BayesOutputGate sits.
| Capability | BayesOutputGate | Fixed threshold | Guardrail rules | LLM judge alone | Hand-rolled |
|---|---|---|---|---|---|
| Distribution | npm library | your constant | rule engine | API call | your repo |
| Calibrated evidence, not a cutoff | Bayes Factor | a guess | no | raw score | rarely |
| Multi-dimension weighting | yes | no | partial | no | rarely |
| Utility-aware (expected loss) | yes | no | no | no | no |
| Goodness-of-fit and dependence checks | yes | no | no | no | no |
| Tamper-evident audit | yes | no | partial | no | no |
| Zero deps, node-free | yes | n/a | varies | network | varies |
| License | Apache-2.0 | your call | varies | proprietary | your call |
The honest summary: a fixed threshold is fine when one dimension and a known cutoff are all you have, and an LLM-as-judge is a fine scorer. BayesOutputGate is the layer on top: it turns those scores into a calibrated pass, fail, or escalate decision with a Bayes Factor, weighs dimensions, prices the two error types, and seals a tamper-evident trail, in one node-free TypeScript library you embed in a JavaScript or TypeScript service.
BayesOutputGate is a Bayesian model, and that has assumptions. They are documented, and where a real workload strains them, there is an explicit diagnostic rather than a silent fudge.
| Assumption or property | What it means | How BayesOutputGate handles it |
|---|---|---|
| The score model is a Beta | Each dimension's scores are modeled by a Beta density; real scores may not be Beta-distributed. | A Kolmogorov-Smirnov goodness-of-fit test reports when the Beta assumption fails, so you do not trust a bad fit. |
| Dimensions are conditionally independent | Summing log-Bayes-Factors across dimensions assumes they do not move together given the hypothesis. | A dependence diagnostic measures pairwise correlation and flags pairs that double-count evidence. |
| Calibration is measured | A posterior probability is only worth its match to reality on real outcomes. | The Brier score, expected calibration error, and a reliability diagram measure it on your data. |
| Calibrated from labeled history | The two hypotheses are only as good as the labeled known-good and known-bad outputs behind them. | A regularizing prior anchors a cold model, and labeled outcomes fold in online to sharpen it. |
| Evidence versus utility | A Bayes Factor measures evidence; a decision needs the cost of each error too. | The decision-theoretic policy adds an explicit prior and asymmetric losses to minimize expected loss. |
| Scores are bounded in [0, 1] | The Beta density lives on the open unit interval; an exact 0 or 1 has no finite density. | Boundary scores are clamped inward by a small epsilon so the log-density never becomes non-finite. |
| Recommends, does not act | The library returns a pass, fail, or escalate recommendation, it does not ship or block your outputs. | Your orchestrator consumes the decision; turnkey runtime integrations are on the roadmap. |
| Integrity, not identity | The audit seal proves a log was not altered after sealing. | A SHA-256 hash chain via Web Crypto, an integrity seal, not a signature. |
90 tests passing under Vitest 4 with 94% coverage (statements 94.64%, branches 87.58%,
functions 96.24%, lines 95.91%). Run pnpm test on a fresh clone to reproduce.
TypeScript 6 in maximum strict mode (exactOptionalPropertyTypes,
useUnknownInCatchVariables, noUncheckedIndexedAccess,
noPropertyAccessFromIndexSignature, noImplicitOverride). Zero errors under tsc
--noEmit.
Biome 2.5 clean across src and tests. publint clean and
@arethetypeswrong/cli green across all thirteen subpaths. Dual ESM + CJS with separate
.d.ts and .d.cts per subpath.
The special functions are verified against closed forms, the Beta-Binomial against its uniform-prior identity, the decision-theoretic policy against expected loss, and the audit chain against a known SHA-256 digest and against tampering. The size-limit core facade is 5.29 kB brotli.
A smoke test exercises the compiled ESM and CJS artifacts and spawns the compiled CLI as a single Node process, run on the Node 20, 22, and 24 matrix.
Committed pnpm lockfile, zero required runtime dependencies, node-free core, and SLSA provenance
attestation on every published version. Verify with npm view @takk/bayesoutputgate@1.0.0 --json | jq
.dist.attestations.
Yes. 90 tests pass under Vitest 4 with 94% coverage; TypeScript 6 maximum strict mode is clean; Biome 2.5
lint is clean; publint and @arethetypeswrong/cli are clean across all thirteen
subpaths. The suite and the distribution smoke test are run on the Node 20, 22, and 24 matrix. Every
published release carries SLSA provenance produced by GitHub Actions.
A threshold compares one number to a constant you guessed. BayesOutputGate compares how likely the observed scores are under a high-quality hypothesis against a low-quality one, calibrated from your labeled history, and reports the strength of that evidence on the Jeffreys scale. It accounts for how good and bad outputs actually separate, weighs dimensions, and can escalate when uncertain rather than forcing a binary call.
The dependence diagnostic measures pairwise correlation across your history and flags any pair that crosses a threshold. Correlated dimensions double-count evidence and inflate the Bayes Factor, so when a pair is flagged you drop, merge, or down-weight one of them. The check surfaces a real limitation instead of hiding it.
Use the Bayes Factor policy when the two error types cost about the same and you want principled evidence thresholds on the Jeffreys scale. Use the decision-theoretic policy when passing a bad output and rejecting a good one have different costs, which is the usual production case; it turns the Bayes Factor into a posterior and picks the action that minimizes expected loss.
Yes. The core is node-free; the audit seal uses the Web Crypto API, not node:crypto. Import
@takk/bayesoutputgate or @takk/bayesoutputgate/edge on any runtime with Web Crypto,
and call the tool adapter from a non-human-entity loop. Only @takk/bayesoutputgate/node requires
the Node standard library, for reading history files from disk.
BayesOutputGate records the per-dimension scores, the labels, the calibrated model parameters, and the decisions you supply. It never sees your raw outputs or payloads and makes no outbound network calls of its own. With the file loaders, it only reads the history file you point it at; no secrets are involved at any point.
The weakly-informative default priors, Beta(2, 1) for the high hypothesis and Beta(1, 2) for the low, keep the models well-defined before any labels arrive, with only a few pseudo-observations of weight so real labeled data dominates quickly. To anchor a dimension to a known base rate, pass your own priors; each new labeled outcome then folds into the calibration online.
The calibration is a small set of numbers, the Beta parameters and weight per dimension, that you can
serialize wherever you like via the model snapshot. Labeled history and score vectors are read from a JSON or
CSV file with the loaders in @takk/bayesoutputgate/node, or fed in directly. For an edge runtime,
keep the calibration in your own KV store between invocations.
No, and the docs are explicit about it. The seal is a SHA-256 hash-chain that proves a log was not altered after sealing. It is an integrity seal, not a signature: it does not prove who produced the log. Signed and timestamped seals for stronger third-party evidence are on the roadmap.
Per-output score vectors, each a set of { dimension, value } pairs in [0, 1] from any scorer
you run, an LLM-as-judge, a classifier, or a regex. To calibrate, feed labeled observations of known-high and
known-low outputs; to decide, feed the scores of a new output. The gate weighs only the dimensions that have
both a score and a calibrated model.
Every release is published with npm publish --provenance. Check the attestations with
npm view @takk/bayesoutputgate@<version> --json | jq .dist.attestations. The attestation
links the tarball you installed to the GitHub Actions workflow that built it from a specific source commit.
No. BayesOutputGate produces the recommendation, pass, fail, or escalate, with the Bayes Factor, the strength, and the per-dimension contributions. Your orchestrator, an MCP server, an LLM tool loop, a Hermes runtime hook, or your own controller, consumes that decision and acts. Turnkey runtime integrations are on the roadmap.
Strict SemVer 2.0.0, starting from 1.0.0. The binding stability surface is documented in SPEC.md section 5. Major bumps require a deprecation cycle; security fixes follow the disclosure flow in SECURITY.md.