Methodology

public · how assets are evaluated
← Methodology

Signals

A signal is one observation about an asset, produced by a stated method and stored as a single record.

Every signal belongs to one category, carries the method that produced it, and records where it came from. Some signals describe what an asset declares about itself; others describe what Vettd measured or inferred from the artifact.

A shape is what kind of claim a signal is. Five shapes can be produced by a rule; absence is a display state, not a produced signal. The most useful question is not how big the number is but whether a change in it is real.

MeasurementReproducible under the stated method

An exact value produced by a stated method

EstimateNot reproducible; it can move from sampling noise alone

A statistic over samples

FactDirectly read from the artifact

A neutral descriptor read directly off the artifact

ClassificationNot reproducible; a judgment

A bucket assigned by inference, with a confidence

FindingNot applicable; a judgment

Something wrong; the only shape that obligates action

AbsenceNot applicable

Not assessed; assessed with nothing found; or a notice about the assessment itself

Popularity figures are public repository statistics. They are context and can change on their own, without the asset changing.

A measurement is exact given the method that produced it. If the method changes, the measurement changes with it: widening what a method counts is a new method, never a silent redefinition of the old one.

Not every number is a measurement. Popularity figures are public repository statistics: they can move on their own, without the asset changing, and they are context rather than a grade.

Each source observes a different input and produces a different kind of evidence. The blocks below say who produces a signal, what the source observes, and what it does not see.

ScanLive
Producer

Vettd static analysis

Input observed

The submitted skill bundle: SKILL.md plus its scripts, references, and evals

Method

Deterministic static analysis of those files

Evidence produced

Findings, measurements, facts, and classifications (see the rule inventory below)

Freshness

Observed at scan or import time, not refreshed on a schedule

Limits

It only sees the submitted artifact; it does not observe runtime behavior

ResearchLive
Producer

Public repository data, read from the repository's public metadata

Input observed

The repository's public fields

Method

Reading those public repository fields

Evidence produced

The Popularity measurements plus the archived characteristic

Freshness

Observed at scan or import time, not refreshed on a schedule

Limits

Repository statistics can move on their own

EvaluationPlanned
Producer

Not yet defined

Input observed

Not yet defined

Method

Not yet defined; Evaluation is not active

Evidence produced

None yet

Freshness

Not applicable while planned

Limits

It does not observe runtime behavior today

Registered rules

The rule IDs below are the identifiers Vettd uses internally for each check. Vettd registers 38 signal rules today, grouped by data category, and each rule is tagged with the source that produces it.

Characteristics

characteristics/declared-licensefact · Scan
characteristics/primary-languageclassification · Scan
characteristics/archivedfact · Research
characteristics/repository-linkfact · Scan
characteristics/eval-file-formatfact · Scan

Performance

performance/static-context-tokensmeasurement · Scan
performance/progressive-disclosurefact · Scan

Reliability & Quality

reliability/unresolvable-internal-referencesfinding · Scan
reliability/gotchas-sectionfact · Scan
reliability/examplesfact · Scan
reliability/checklist-patternfact · Scan
reliability/validation-loopfact · Scan
reliability/step-by-step-workflowfact · Scan
reliability/generic-instructionfinding · Scan
reliability/description-presencefinding · Scan
reliability/description-usage-contextfact · Scan
reliability/description-briefnessfinding · Scan
reliability/description-overclaimfinding · Scan
reliability/eval-test-case-countmeasurement · Scan
reliability/eval-assertionsfact · Scan
reliability/eval-test-cases-sufficientfinding · Scan

Cost

cost/declared-external-servicesfact · Scan
cost/declared-required-env-varsfact · Scan
cost/description-lengthfinding · Scan

Compatibility

compatibility/declared-required-toolsfact · Scan
compatibility/declared-mcp-serversfact · Scan
compatibility/declared-harness-targetsfact · Scan
compatibility/declared-environment-assumptionsfact · Scan
compatibility/declared-namefact · Scan
compatibility/cli-helpfact · Scan
compatibility/interactive-promptsfinding · Scan
compatibility/structured-outputfact · Scan
compatibility/unpinned-dependenciesfinding · Scan

Popularity

sentiment/starsmeasurement · Research
sentiment/forksmeasurement · Research
sentiment/watchersmeasurement · Research
sentiment/open-issuesmeasurement · Research
sentiment/last-commit-agemeasurement · Research

Safety has no registered rules: safety findings are stored as findings rather than as signals.

When a category has no signals, the record says so rather than staying blank. Both represent an absence, but they mean different things: not assessed, where no producer has reported anything for the asset, and nothing found, where a check ran and found nothing to report. Neither is a clean bill of health.