Methodology — how Skill Trust scores are made

Canonical source: docs/methodology.md (analyzer v1).

Version: analyzer v1. Last updated 2026-09-09.

A Skill Trust score is a 0–100 risk number for one MCP skill at one pinned

revision (0 is safest, 100 means do-not-install). Every score ships with the

per-finding evidence behind it. This document is the full recipe: what we

check, how findings convert to a number, what the number does and does not

mean, and the operating limits we will not cross.

1. Scope: Layer 1 only

Scoring is Layer-1-only: manifest review, AST-pattern matching over code,

override-language matching over tool descriptions, provenance checks, plus

manual review of the batch output. There is no sandbox execution, no canary

token detonation, no fuzzing, and no live probing of the skill or its

endpoints.

Sandbox execution, canary analysis, and fuzzing stay out of V1 on purpose:

they need isolated VMs with egress control, which Workers cannot provide.

They are re-costed as V2 work on dedicated compute, not as an in-Worker

addition. Until that lands, any claim that a score reflects runtime behavior

is false, and we do not make it.

2. The pipeline

1. Pin. The batch resolves the exact bits under test: a commit sha for

git skills, a tarball sha256 for npm skills, or a manifest hash for remote

skills. The score always belongs to the pin, never to the skill in general.

1. Static pass. The analyzer (versioned, see §6) scans the manifest, tool

descriptions, and file contents at the pin. Findings carry rule id,

severity, line number where one applies, and a short detail string.

1. One judge call. A single bounded LLM call reviews the static findings

in context (full SKILL.md plus helper-script context). The judge can

confirm, downgrade, or add context — it cannot invent findings outside the

static evidence, and its cost is capped (see §8).

1. Manual review. A person reads the batch output before scores publish.

Nothing goes public straight from the pipeline.

1. Publish. Score, grade, findings, pin, analyzer version, and timestamp

are written as one immutable snapshot keyed by (skill, sha). The badge and

evidence page render from the snapshot.

3. Rule catalog (analyzer v1)

Each rule lists its definition, severity, and weight. Weights sum per finding;

the total caps at 100.

No static rule in analyzer v1 emits critical. The critical weight exists for

judge-confirmed findings: when the bounded judge call confirms a static

finding as actively exploitable rather than merely present, that finding

scores at the critical weight.

4. Severity weights and grade bands

Severity weights: critical 40, high 10, medium 2, low 1, info 0. The score is

the sum of finding weights, capped at 100.

Grade bands: low 0–24, medium 25–49, high 50–74, critical 75–100.

5. What a score means for install checks

Two different questions, two different answers:

• Pinned-hash match (integrity): does the score's sha equal the bits you

are about to install? If yes, the findings describe exactly what you are

installing. Install verifiers must compare the full pin, not the skill name.

• Latest-published-sha (freshness): is the score's sha still the skill's

current revision? If the upstream moved, the score is marked stale until

the next batch re-scores the new pin. A stale score describes the old pin,

not what you would install today.

Never install off the skill name alone. Match the pin for integrity, check the

freshness flag for currency.

6. Versioning and history

The analyzer carries a version (analyzer_v, currently v1). Every snapshot

records the analyzer version that produced it, and history is immutable per

(skill, sha): a re-score of new bits writes a new snapshot row, never an

update. Evidence pages show the pin and analyzer version for the snapshot

they render, so a score from v1 stays readable and comparable after v2

ships.

7. Remote skills: what we do and do not promise

Remote skills (HTTP/SSE endpoints) cannot be pinned the way git or npm skills

can — the server can change under the URL at any time. Our limit for remote

skills is endpoint + manifest-hash pinning with a short-TTL re-check: we

record the endpoint and the manifest hash we scored, and we re-check on a

short TTL. This catches drift; it does not prevent it. We make no rug-pull

claim — a remote skill can change between our checks, and its score describes

the last hash we saw, flagged stale the moment the hash moves.

8. Cost guardrails

The judge is the only variable cost in scoring. Each score carries a $0.20

judge ceiling: one bounded call, truncated context, small judge. The weekly

batch aborts before writing anything if projected judge spend exceeds $5/run

(25 skills × the $0.20 ceiling) — the abort fires first, the alert pages the

backlog, and no partial batch publishes. Typical spend lands at $0.01–0.03

per score; the ceiling, not the typical, is what the budget enforces.

9. Cache, badge, and opt-out timing

• Unpinned badge and score reads serve from a KV gate with a TTL bound of at

most 5 minutes, or bust-on-write when the batch rewrites a key — whichever

hits first.

• The 5-minute-or-bust bound is what preserves the R5 1-hour badge-404 SLA:

when a dispute opens or a maintainer opts out, the key is busted on write

and any surviving edge copy expires within 5 minutes, so the badge is gone

everywhere long before the hour is up.

• One propagation caveat is outside our control: GitHub-camo and other image

CDNs cache badge bytes on their own schedule. The 1-hour opt-out SLA covers

our badge endpoint returning 404; a camo-proxied copy embedded in a README

can linger until that CDN re-fetches. We document the lag instead of

promising what a third-party cache will do.

10. No-warranty, paid-badge, and comms rules

• No warranty. A Verified badge ($199/yr) is not a safety guarantee.

Scores are point-in-time assessments at a pinned revision, produced by a

static pass plus one bounded judge call. They do not certify that a skill

is safe, secure, or fit for any purpose, and they do not replace the

installer's own review.

• Never a paid "untrusted" badge. A Verified or attested skill that fails

re-scoring loses its badge: the badge endpoint returns 404 and the evidence

page shows "suspended pending re-review". We never sell, render, or display

a badge that reads "untrusted" — suspension replaces shaming, and the

dispute path runs before any public change.

• No public debate. We never argue a score in a public thread. On dispute

we acknowledge once, move to the ticket, and publish the rationale with the

resolution. Every public claim about methodology links the evidence page.