hiphopsllm.reliability

Reliability: operational profiles, measured evidence, and HIP-LLM.

Three things live here.

profile — the operational profile, the mix of work the system will actually meet. Every estimate downstream is conditional on it.

hipllm — the whole of HIP-LLM, re-exported. OperationalFailureProb turns observed correctness outcomes plus a profile into a hierarchical imprecise posterior for the probability that the next item fails.

calibration — the bridge. It runs that inference per component and writes the resulting intervals onto the fault tree’s basic events, replacing engineering-judgement placeholders with measurement, and recording exactly what it did and did not touch.

Operational profiles

Operational profiles: what the system will actually be asked to do.

A benchmark accuracy is a statement about the benchmark’s mix of tasks. A reliability claim has to be a statement about the mix the system will meet in service, and those two are rarely the same — a model measured on a corpus that is 18% long multi-hop questions, then deployed on a workload that is 70% of them, does not keep its measured failure rate.

The operational profile is that mix: a partition of the input space into strata with the probability of each. Everything downstream — the imprecise posterior, the calibrated basic events, the top-event probability — is conditional on it, so it is a first-class object here rather than a dictionary passed around.

The class is deliberately thin over hip_llm.schemas.OperationalProfile, so a profile built here can be handed straight to HIP-LLM’s inference and back.

class hiphopsllm.reliability.profile.OperationalProfile[source]

Bases: object

A partition of the input space into strata, with a weight on each.

Build one from a mapping:

profile = OperationalProfile({"short": 0.30, "long": 0.70})

or from observed usage:

profile = empirical_profile(["short", "long", "long", "long"])
weights: Dict[str, float]
name: str = 'operational profile'
provenance: str = 'declared by the analyst'

how the weights were arrived at — printed in every report that uses them

normalise: bool = True

renormalise weights that do not sum to one, rather than raising

property labels: Tuple[str, ...]
property vector: ndarray
items()[source]
expected(per_stratum)[source]

Profile-weighted average of a per-stratum quantity.

Raises if a stratum has no value: a silently dropped stratum is a silently reweighted profile, which is the kind of error that survives all the way into a published number.

Return type:

float

Parameters:

per_stratum (Mapping[str, float])

restricted_to(labels)[source]

The profile conditioned on a subset of strata, renormalised.

Return type:

OperationalProfile

Parameters:

labels (Sequence[str])

to_hipllm(level='benchmark_stratum')[source]

The equivalent hip_llm.schemas.OperationalProfile.

Parameters:

level (str)

classmethod coerce(profile)[source]

Accept this class, a plain mapping, or a HIP-LLM profile.

Return type:

OperationalProfile

Parameters:

profile (OperationalProfile | Mapping[str, float] | Any)

summary()[source]
Return type:

str

__init__(weights, name='operational profile', provenance='declared by the analyst', normalise=True)
Parameters:
Return type:

None

hiphopsllm.reliability.profile.dataset_proportional_profile(strata, name='dataset-proportional profile')[source]

Weights proportional to the benchmark’s own composition.

HIP-LLM names this choice explicitly (paper Section 4.2, Remark 7) rather than letting it happen by default, and so does this. The paper’s whole argument is that a benchmark accuracy is a descriptive statistic about the dataset, and becomes a reliability claim only once it is reweighted to the mix of work the system will actually meet. Taking the dataset’s own mix as that workload asserts they are the same — sometimes true, never automatic.

Use empirical_profile() when the labels come from production traffic; the two are computed identically and differ only in what they claim, which is exactly why they are separate functions.

Return type:

OperationalProfile

Parameters:
hiphopsllm.reliability.profile.empirical_profile(strata, name='empirical operational profile')[source]

The profile implied by an observed sequence of stratum labels.

Use this when the workload has been logged. It records that provenance, so a reader can tell a measured profile from an assumed one.

Return type:

OperationalProfile

Parameters:
hiphopsllm.reliability.profile.uniform_profile(labels, name='uniform operational profile')[source]

Equal weight on every stratum — the honest default when nothing is known.

It is a choice, not an absence of one, and it is recorded as such.

Return type:

OperationalProfile

Parameters:
hiphopsllm.reliability.profile.stratify(items, key, profile=None)[source]

Label each item with its stratum, checking the labels against a profile.

key is a callable, or the name of a mapping key / attribute. Passing a profile turns an unexpected label into an error rather than a stratum that silently carries zero weight.

Return type:

List[str]

Parameters:

Evidence calibration

Turning measured outcomes into fault-tree basic-event probabilities.

This is the join between the two halves of the package.

HiP-HOPS gives the structure: which combinations of component faults produce a system-level hazard. Its basic-event probabilities, though, arrive as engineering judgement — llm.halluc = 0.15 and so on — which is fine for ranking cut sets and useless for a reliability claim.

HIP-LLM gives the numbers: a hierarchical imprecise-Bayesian posterior for the probability that a future item fails, given observed outcomes and an explicit operational profile. Crucially it returns an interval, because a few hundred benchmark items do not identify a point.

EvidenceCalibrator runs the second and writes its answer into the first. The result is a fault tree whose leaves carry measured intervals with recorded provenance, and therefore a top event that can be quoted as P(hazard) ∈ [lower, upper] rather than as a number nobody can defend.

Two design decisions worth stating.

Where a component’s measured failure rate goes. An LLM agent’s subtle-value deviation is VS-out = BE-HALLUC OR BE-NONDET OR VS-in. A measured end-to-end wrong-answer rate for that agent, on correct input, is the probability of the union of its internal events — not of any one of them. The default policy therefore rescales the whole set so their OR reproduces the measurement while their prior ratios are preserved: with weights w_i summing to one, p_i = 1 - (1 - P)**w_i, which satisfies 1 - prod(1 - p_i) = P exactly.

Nothing is calibrated silently. Every event this touches gets an evidence string naming the sample size and the profile, and every component the calibrator could not match is reported. A network quietly running on default priors is the specific failure this module exists to prevent.

class hiphopsllm.reliability.calibration.ComponentEvidence[source]

Bases: object

What was measured about one component, and what it implies.

interval is the imprecise posterior for the probability that this component fails on an item drawn from the operational profile. It is the number that goes into the fault tree.

component: str
n_trials: int
n_failures: int
empirical: float
point: float
interval: Tuple[float, float]
profile: OperationalProfile
by_stratum: Dict[str, float]

per-stratum observed failure rate, for the diagnostic table

posterior: Any = None

the raw HIPLLM result, when the full inference was run

method: str = 'hip-llm imprecise posterior'
property width: float
property success_interval: Tuple[float, float]

the per-task probability of not failing.

Type:

[1 - upper, 1 - lower]

reliability(n_tasks=1)[source]

R(n) — the probability of failure-free operation over n tasks.

This is HIP-LLM’s definition of reliability (paper Section 3, Theorems 4-6): the probability of failure-free operation over a specified number of future tasks under a given operational profile, not a benchmark accuracy. Returned as an envelope, because the underlying failure probability is itself an interval.

Computed as \(\mathbb{E}[p^{n}]\) per hyperparameter configuration and then enveloped, never as \(\mathbb{E}[p]^{n}\) — by Jensen’s inequality those differ, and the second understates reliability.

Return type:

Tuple[float, float]

Parameters:

n_tasks (int)

statement(n_tasks=1)[source]

The claim this evidence supports, in the form HIP-LLM defines it.

Return type:

str

Parameters:

n_tasks (int)

evidence_string()[source]
Return type:

str

summary()[source]
Return type:

str

__init__(component, n_trials, n_failures, empirical, point, interval, profile, by_stratum=<factory>, posterior=None, method='hip-llm imprecise posterior')
Parameters:
Return type:

None

class hiphopsllm.reliability.calibration.CalibrationReport[source]

Bases: object

Which events were calibrated, which were not, and why.

evidence: Dict[str, ComponentEvidence]
updated: Dict[str, Tuple[float, float, Tuple[float, float]]]

basic event id -> (old probability, new probability, new interval)

uncalibrated_components: List[str]

components in the model that no measurement matched

unmatched_evidence: List[str]

measurement keys that matched no component

skipped_events: Dict[str, str]

basic events left on placeholders, with the reason

notes: List[str]
property n_updated: int
to_frame()[source]

The before/after table, for a notebook.

evidence_frame()[source]
summary()[source]
Return type:

str

__init__(evidence=<factory>, updated=<factory>, uncalibrated_components=<factory>, unmatched_evidence=<factory>, skipped_events=<factory>, notes=<factory>)
Parameters:
Return type:

None

class hiphopsllm.reliability.calibration.EvidenceCalibrator[source]

Bases: object

Fit operational failure probabilities and write them into a fault tree.

Parameters:

  • profile (OperationalProfile) – The operational profile every estimate is conditional on.

  • settings (Any) – HIP-LLM inference settings. Defaults to HIPLLM.quick_inference_settings(), which is interactive-speed; pass HIPLLM.paper_inference_settings() for published sizes.

  • bound (str) – Which HIP-LLM envelope becomes the basic-event interval. "credible" (the default) uses the outer equal-tail credible envelope across hyperparameter configurations — the widest, most cautious reading. "expected" uses the envelope of posterior means, which is tighter and is the right choice when the interval feeds a design decision rather than a safety argument.

  • point (str) – Where the point estimate inside the interval comes from: "midpoint", "lower", "upper", or "empirical".

  • policy (str) – How a component’s measured probability is spread over its basic events; see distribute_union().

  • classes (Tuple[FClass, ...]) – Failure classes a wrong-answer measurement is allowed to speak about. Latency and unsafe-execution events keep their placeholders — a correctness benchmark says nothing about them.

  • exact (bool) – Run the full HIP-LLM hierarchical inference. False substitutes a Jeffreys-interval approximation, which is far faster and adequate for exploration; it is labelled as such in the evidence string, so it can never be mistaken for the real thing in a report.

profile: OperationalProfile
settings: Any = None
bound: str = 'credible'
point: str = 'midpoint'
policy: str = 'share'
classes: Tuple[FClass, ...] = (FClass.VALUE_SUBTLE, FClass.VALUE_COARSE)
credible_level: float = 0.95
exact: bool = True
fit_component(component, outcomes, strata)[source]

Estimate one component’s operational failure probability.

outcomes[k] is 1/True when the component answered item k correctly, matching HIP-LLM’s convention; strata[k] names the operational stratum of the same item.

Return type:

ComponentEvidence

Parameters:
fit_many(observations)[source]

Fit several components at once.

observations maps a component name to (outcomes, strata), or to a mapping with outcomes and strata keys:

calibrator.fit_many({
    "react_agent": (react_outcomes, strata),
    "cot_agent":   (cot_outcomes, strata),
})
Return type:

Dict[str, ComponentEvidence]

Parameters:

observations (Mapping[str, Any])

apply(failure_model, evidence, *, component_map=None, in_place=True)[source]

Write measured intervals onto the model’s basic events.

component_map forces a measurement key onto a component id when the automatic name matching cannot be trusted. Everything the matcher does or fails to do is recorded in the returned CalibrationReport.

Return type:

CalibrationReport

Parameters:
__init__(profile, settings=None, bound='credible', point='midpoint', policy='share', classes=(FClass.VALUE_SUBTLE, FClass.VALUE_COARSE), credible_level=0.95, exact=True)
Parameters:
Return type:

None

hiphopsllm.reliability.calibration.calibrate_failure_model(failure_model, observations, profile, **kwargs)[source]

Fit and apply in one call.

report = calibrate_failure_model(
    model.failure_model,
    {"react_agent": (outcomes_a, strata), "cot_agent": (outcomes_b, strata)},
    profile={"short": 0.3, "long": 0.7},
)
print(report.summary())
Return type:

CalibrationReport

Parameters:
hiphopsllm.reliability.calibration.distribute_union(target, weights, mode='share')[source]

Split a union probability over independent events.

Given P(E1 or ... or En) = target and prior weights w_i, return probabilities p_i with 1 - prod(1 - p_i) = target.

mode="share" uses p_i = 1 - (1 - target) ** (w_i / sum(w)), which is exact and keeps the events in roughly their prior proportions. mode="dominant" puts the whole probability on the heaviest event and leaves the rest at zero — blunter, but easier to defend when only one of the events is really what was measured.

Return type:

ndarray

Parameters:

HIP-LLM, re-exported

Every HIP-LLM function, re-exported under this package’s namespace.

HIP-LLM ships in two layers and both are vendored here in full (see docs/source/vendoring.md for the provenance and the commit they were taken from):

HIPLLM

The high-level API — FailureProb, OperationalFailureProb, the StrategyQA loader — which is what most users want.

hip_llm

The replication engine underneath it: hyperposteriors, imprecise envelopes, reliability transforms, baselines, benchmark evaluation, plotting, and the schemas the two layers share.

Both are importable directly (from HIPLLM import OperationalFailureProb keeps working exactly as before). This module makes them reachable through one import as well, so an analysis that spans the fault tree and the reliability model does not need three import lines:

from hiphopsllm import OperationalFailureProb, quick_inference_settings

One name is deliberately not passed through unchanged. HIP-LLM’s OperationalProfile is a schema with parallel labels/weights arrays; this package’s hiphopsllm.reliability.profile.OperationalProfile is a mapping-shaped class that converts to it. The HIP-LLM one is available here as HIPLLMOperationalProfile.

The HIP-LLM classes themselves are documented upstream at https://hipllm.readthedocs.io. The names reachable from here are:

FailureProb, FailureProbResult, LogprobsUnavailableError, OperationalFailureProb, OperationalFailureResult, StrategyQALoadError, decomposition_stratum, load_strategyqa, paper_inference_settings, parse_strategyqa_answer, quick_inference_settings, HIPLLM_VERSION, BenchmarkResult, CDFEnvelope, DomainData, GlobalSettings, HIPLLMOperationalProfile, HyperparameterConfiguration, HyperparameterInterval, HyperposteriorGrid, ModelResult, PosteriorSamples, ReliabilityEnvelope, ReproductionRecord, ReproductionStatus, RunMode, SourceRecord, SubdomainData, OFFICIAL_REPOSITORY, PAPER_DOI,

plus the engine modules api_clients, baselines, benchmark_eval, envelopes, grids, hyperposterior, numerics, operational_profile, plotting, posterior, reliability, scalability, schemas, validation.

A test asserts this list stays complete against HIPLLM.__all__ and hip_llm.__all__.