Calculations reference

Sample size and reliability

Sample size classes

sample_size_class is a banded label on observation_count (the number of PO lines in the window):

if observation_count < 30:  "sparse"
elif observation_count < 100: "adequate"
else: "robust"

Why the bands matter:

• sparse: stats are unstable, CIs are wide, outlier detection is unreliable. Triggers quality_flags.low_sample_size. Use IQR for outlier flagging instead of z-score (see Outlier detection below).
• adequate: stats are usable but not rock-solid. Z-score is acceptable but compute the CI against a t-distribution rather than a normal.
• robust: Central Limit Theorem applies; z-distribution CI and z-score outlier detection are both fine.

Confidence interval for the mean

Always emit confidence_in_mean at the 95% level. Formula depends on sample size class:

sample stddev s = sqrt( sum( (x_i - mean)**2 ) / (n - 1) )
standard error  = s / sqrt(n)
 
if class == "robust":
    margin_of_error = 1.96 * standard_error
else:
    margin_of_error = t_critical(0.025, df = n - 1) * standard_error
 
interval = [mean - margin_of_error, mean + margin_of_error]

For sparse samples, the t-critical at df = 5 is ~2.57; at df = 29 it is ~2.05. The CI gets visibly wider as n shrinks: that is a feature, not a bug. Use sample stddev (divide by n - 1), not population stddev.

Population and spend share

population_size and population_rank

For a manifold of kind K over time_window W:

population       = distinct entities of kind K observed in the
                   source table during window W
population_size  = count(population)
population_rank  = rank of this entity by total_spend_usd within
                   population, descending (1 = largest spender)

Do not extrapolate beyond W. If the window is 24 months, population_size is "active entities in the last 24 months", not "all entities the company has ever transacted with".

pct_of_total_spend

pct_of_total_spend = this_entity.total_spend_usd
                   / sum(e.total_spend_usd for e in population)

The denominator is total spend across the entire population of kind K in the window. It is NOT total company procurement spend across all kinds; that is a different metric and TMS does not cross kinds.

For a supplier_category manifold, denominator = sum across all supplier_categories in the window. For a supplier manifold, denominator = sum across all suppliers in the window. And so on.

population_total_spend_usd

population_total_spend_usd = sum(e.total_spend_usd for e in population)

The pct_of_total_spend denominator, published as a field. Emit it so consumers never have to recover the base figure by inverting the ratio on a single entity, which is unreliable at stored precision and unsafe when unattributed spend is present (see below).

Scoped to the kind and the window like every other population field. It is not comparable across kinds unless every kind partitions the same source rows, and unattributed_spend_pct is how you tell.

Attributed and unattributed spend

A kind's population partitions spend only if every source row resolves to exactly one entity of that kind. Where resolution can fail (a PO line whose supplier never matched a parent_supplier_id, an item carrying no commodity assignment), the unresolved remainder is unattributed spend.

attributed_spend_usd   = spend resolving to this entity through the
                         kind's join path
unattributed_spend_usd = total_spend_usd - attributed_spend_usd
unattributed_spend_pct = unattributed_spend_usd / total_spend_usd

total_spend_usd is always the gross figure, inclusive of the residue.

All three fields are required on every entity_profile manifold, including kinds that resolve every row by construction. Where a kind INNER JOINs its mapping upstream and cannot carry a residue (supplier, supplier_category), emit attributed_spend_usd == total_spend_usd, unattributed_spend_usd = 0, unattributed_spend_pct = 0. Say it rather than implying it: an absent field is indistinguishable from a zero one, and a consumer cannot tell those apart from the manifold alone.

pct_of_total_spend uses total_spend_usd in both numerator and denominator. Computing it on attributed_spend_usd breaks the first invariant below and leaves the denominator unrecoverable: inverting the ratio then overstates it by 1 / (1 - unattributed_spend_pct), which at 34% unattributed is a 52% error in the headline figure, silently.

Any rollup, concentration or HHI computation that walks a join path (a supplier_rollup on an item manifold, for instance) covers the attributed share only. When unattributed_spend_pct > 0.05, set has_unattributed_supplier_spend and treat supplier-level conclusions as covering the attributed share alone.

Invariants

Every entity_profile manifold satisfies these. They are the cheapest conformance checks available and should be asserted by the generator before emit.

pct_of_total_spend * population_total_spend_usd == total_spend_usd
attributed_spend_usd + unattributed_spend_usd   == total_spend_usd
1 <= population_rank <= population_size

Use a 0.5% relative tolerance on the first two: pct_of_total_spend is stored at six decimal places, so at pct = 0.000001 the rounding error alone is roughly 50%. Restrict the first check to entities with pct_of_total_spend >= 0.0005, below which stored precision cannot support it. Raising stored precision on pct_of_total_spend would let the check run corpus-wide and is worth doing.

Data quality

data_quality_score

data_quality_score = 1 - (imputed_or_null_count / total_row_count)

Range 0 to 1. Anything below 0.90 means a meaningful fraction of rows needed imputation or had missing critical fields. When below 0.90, emit data_quality_notes explaining what was filled and why.

What counts as imputed

An "imputed" value is anything derived rather than directly observed. Common cases:

• unit_price computed from spend / qty when the source row has spend but no unit_price.
• qty computed from spend / unit_price when the source row has spend and unit_price but no qty.
• site_code defaulted (e.g., to "UNKNOWN") when the source row lacks site attribution.
• Currency converted via FX (see Currency normalization).
• Date defaulted to month-end or quarter-end when the source row carries only a coarse period.

Imputed values count against data_quality_score AND trigger quality_flags.imputation_applied = true. Each imputation rule applied must appear in lineage.transformations so an auditor can reproduce.

Staleness threshold

days_since_last = today - staleness.last_observation
is_stale = days_since_last > stale_threshold_days

Default stale_threshold_days:

• 90 for routine PO-line procurement data (default).
• 365 for slow-cadence categories like capital equipment (long replenishment cycles; sparse observations are expected).
• 30 for real-time-ish flows (e.g., daily transactional categories in CPG or hospitality).

If you override the default, record it in reliability.staleness.stale_threshold_days and call out the choice in data_quality_notes.

Currency normalization

TMS assumes a single reporting currency in the manifold (typically USD). If the source table carries multiple currencies, normalize to one BEFORE building the manifold, not after.

Process:

1. Pick a reporting currency. Record it on the relevant financial block if you need to be explicit (e.g., currency: "USD" on the subject or financial_summary).
2. Pick an FX policy:
   • Monthly average FX: simple, suitable for spend analytics where intra-month volatility is small.
   • End-of-period FX: matches accounting reconciliation cadence.
   • Transaction-date FX: most accurate but most expensive to compute. Use when individual lines matter (e.g., commodity_group analyses for hedged commodities).
3. Document the policy in lineage.transformations (one prose line) AND set quality_flags.imputation_applied = true. FX conversion is imputation.
4. Never mix currencies in distribution stats, HHI math, or rollups. The numbers come out meaningless: the CV blows up, the HHI bands stop mapping to the FTC scale, and rank ordering is wrong.

Discipline rating bands

discipline_rating is a banded label on weighted_avg_item_cv:

if weighted_avg_item_cv < 0.10: "Excellent"
elif weighted_avg_item_cv < 0.25: "Good"
elif weighted_avg_item_cv < 0.50: "Fair"
else: "Poor"

Always emit a rating_basis object alongside discipline_rating:

"rating_basis": {
  "metric": "weighted_avg_item_cv",
  "threshold_excellent": 0.10,
  "threshold_good": 0.25,
  "threshold_fair": 0.50,
  "note": "Spend-weighted per-item CV. Cross-category comparisons are not meaningful."
}

This lets the consumer re-band against their own tolerance without re-reading the spec.

HHI on a rollup output

Herfindahl-Hirschman Index measures concentration. Compute over all rows including the Pareto-truncated tail:

hhi = sum( (row.spend / total_spend) ** 2 for row in all_rows )
top_supplier_pct = max(row.spend for row in all_rows) / total_spend

Bands (FTC convention, scaled to 0-1):

• < 0.15 unconcentrated
• 0.15 <= hhi <= 0.25 moderately concentrated
• > 0.25 highly concentrated (sets high_supplier_concentration flag)
• 1.0 monopoly (single source)

The tail belongs in the HHI math even when it is summarized away in the rollup. Using only the truncated head will under-state concentration.

Weighted average per-item CV

Isolates pricing volatility from product-mix effects:

items_eligible = [i for i in items if i.po_line_count >= 2]
total_eligible_spend = sum(i.spend for i in items_eligible)
 
weighted_avg_item_cv = sum(
  i.cv * (i.spend / total_eligible_spend)
  for i in items_eligible
)

Single-PO items have no stddev and are excluded. Track the eligibility share separately:

pricing_data_coverage_pct = total_eligible_spend / total_entity_spend

If pricing_data_coverage_pct < 0.80, add a note in price_stability.rating_basis.note explaining the gap. The discipline_rating is only as trustworthy as its coverage.

Pareto truncation algorithm

function pareto_truncate(rows, target_coverage, min_rows, max_rows,
                         rank_metric="spend"):
    sorted_rows = sort(rows, key=rank_metric, descending=True)
    total = sum(r[rank_metric] for r in sorted_rows)
 
    emitted = []
    cumulative = 0.0
    for row in sorted_rows:
        if len(emitted) >= max_rows:
            break
        if (len(emitted) >= min_rows and
            cumulative / total >= target_coverage):
            # tie_extended_by_spend_v2: include this row only if it
            # adds non-trivial incremental spend at the boundary
            if row[rank_metric] / total >= 0.005:
                emitted.append(row)
                cumulative += row[rank_metric]
            break
        emitted.append(row)
        cumulative += row[rank_metric]
 
    tail = sorted_rows[len(emitted):]
    tail_summary = summarize(tail)
 
    return {
        "rows": emitted,
        "rows_truncated": len(tail),
        "tie_break": "tie_extended_by_spend_v2",
        "tail_summary": tail_summary,
    }

Default target_coverage:

• supplier_rollup: 0.80
• item_rollup: 0.80
• industry_rollup: 0.95
• sub_category_rollup: 0.95
• commodity_rollup: 0.80

Default min_rows = 5, max_rows = 50 (supplier/item) or max_rows = 20 (industry/sub_category).

tail_summary structure

Minimum fields for any tail_summary:

{
  "rows_truncated": 0,
  "spend": 0,
  "pct_of_spend": 0,
  "po_count": 0,
  "po_line_count": 0
}

Type-specific extras:

• supplier_rollup tail → add item_count and tail_industry_mix (top 5 industries by tail spend, each {industry, spend, pct_of_tail})
• item_rollup tail → add distinct_commodity_groups and tail_commodity_mix (top 5 commodity groups)
• commodity_rollup tail → add distinct_sub_categories
• industry_rollup tail → add distinct_industries (typically equals rows_truncated since each row is already an industry)

entity_profile quality flags

The seven universal flags apply to every manifold kind. entity_profile adds eight more. All are required and all are booleans: emit false rather than omitting a flag that does not apply, for the same reason S1 requires zero-valued spend residues.

Two things these names do not say on their own.

is_single_supplier and is_single_item exist because HHI is tautologically 1.0 when there is one counterparty. That is not concentration risk in the usual sense, and a consumer reading hhi = 1.0 without the flag will report a finding that is really a statement about population size. Set the flag and let the reader distinguish.

level_1_*_truncated is a materiality signal, not a "was anything truncated" signal. A rollup that emitted 50 rows and truncated a 2% tail does not set it. Raw counts live in truncation.tail_summary either way, so the flag is asking whether the truncation changes a conclusion.

Earlier manifolds emitted is_single_* with the entity kind interpolated into the key (is_single_supplier_commodity). That is the only place in TMS where a key name varies by subject.entity_type, it cannot be expressed in JSON Schema without enumerating every kind, and it breaks typed consumers. Use the fixed keys above. Generators may emit the kind-suffixed forms alongside them for one version; consumers should read the fixed keys.

Outlier detection

Used to populate quality_flags.suspected_outliers (boolean) and to flag individual rows in level_2_telemetry.inline_rows.

Z-score (default for adequate and robust samples)

z_score = (row.unit_price - entity.mean) / entity.stddev

Emit the line with flag set when |z_score| >= 3. Common flag values used in the TechnoFlex examples:

• "price_outlier_high" for z_score >= 3
• "price_outlier_low" for z_score <= -3
• "price_peak_period" for 1.5 <= z_score < 3
• "uom_likely_misencoded" for z_score >= 8 (typically a per-railcar price mistakenly entered per-unit; verify the spend math to confirm)
• "high_volume_lane" for normal-priced rows with qty in the top decile (not an outlier per se, but useful evidence)

IQR fallback (for sparse samples)

When sample_size_class == "sparse" (n < 30), z-score breaks down because stddev is unreliable. Use IQR instead:

Q1 = 25th percentile of unit_price
Q3 = 75th percentile of unit_price
IQR = Q3 - Q1
 
flag row as outlier if:
    row.unit_price < Q1 - 1.5 * IQR     (low)
    or row.unit_price > Q3 + 1.5 * IQR  (high)

Flag vocabulary for IQR-detected outliers:

• "iqr_outlier_high" for the upper tail
• "iqr_outlier_low" for the lower tail

Treat IQR and z-score as parallel detectors with the same downstream behavior. quality_flags.suspected_outliers is true if EITHER detector fires on any row. Record which detector was used in level_2_telemetry.retrieval notes or in data_quality_notes.