Methodology

How Vettd evaluates AI assets.

A plain-language overview of the signals Vettd looks for today across skills, prompts, MCP servers, agents, and agentic apps.

Evaluation process

Structure

Vettd examines the submitted package or repository for required files, naming conventions, documentation completeness, and the presence of expected directories.

Security

Static analysis looks for credential exposure, unsafe shell patterns, metadata endpoint probing, name-squatting signals, and multi-step attack chains.

Safety

Signals are weighted and combined into a verdict. External scanners (where configured) contribute additional findings before the final grade is written.

High level by design

This page describes the categories of evidence Vettd looks at and the scoring thresholds used to assign letter grades and classify individual findings.

Evidence-based, not absolute

Vettd analyzes submitted packages, repository content, and scan data available at the time of submission. A scan with no findings is inconclusive, not a pass.

Letter grades

Grades are derived from structure and security findings only. Quality and usability findings appear in the finding list but do not affect the grade. Thresholds are evaluated in order from F to A — the first tier whose condition is met wins.

Low-severity findings only (fewer than 4 lows)

No mediums, highs, or criticals detected in structure or security checks.

4 or more lows, or any medium present (fewer than 3 mediums)

Pattern-level findings with no confirmed data flow to a vulnerability, or at least one known weakness class.

3 or more mediums, or any high present (fewer than 3 highs)

One or more findings with conditional exploitability, or serious harm conditional on an additional factor.

3 or more highs, or any critical present

Adversarial intent detected, or findings that fire unconditionally without exploitability preconditions.

Finding severities

Each finding is assigned one of five severity levels. Only findings in the structure and security categories count toward the grade. Info findings are always excluded from grade computation.

critical

Adversarial intent, or no exploitability preconditions required. Fires unconditionally in the context where the pattern appears. intent:malicious auto-elevates any finding to critical regardless of base assignment.

high

Serious harm but conditional on one additional factor: caller-controlled input reaching the vulnerable operation, a specific execution environment, or a permissive configuration. Negligent intent.

medium

Known weakness class with conditional exploitability, limited impact in isolation, or meaningful false-positive risk. Warrants review but insufficient to classify on the exploitability or intent axes alone.

low

Heuristic match only. High false-positive rate, no confirmed data flow to a vulnerability. Pattern associates with a risk class but is not a confirmed instance.

info

Observation. No direct harm path. Human review only, or passing state. Info findings do not contribute to the grade.

What Vettd checks, by asset type

Skills

8 checks

For public skills in the directory, Vettd evaluates package structure, documentation quality, and security signals. It looks at signals both in isolation and as sequences, because the difference between a careless mistake and an intentional attack is often visible in the chain of behaviors, not any single one.

What Vettd looks at

Whether required structure is present, such as SKILL.md and supporting directories like scripts, references, assets, or evals.
Whether the description is clear enough to explain what the skill does and when it should be used.
Whether the instructions appear to include concrete workflow guidance, examples, validation steps, or checklists.
Whether the package contains security red flags like embedded secrets, unsafe shell patterns, destructive commands, or environment files that should not ship.
Whether scripts access known credential file paths such as cloud provider credentials, SSH private keys, or container registry configs.
Whether scripts probe cloud instance metadata endpoints, which have no legitimate use in a skill and are a known credential-theft vector.
Whether the skill name is suspiciously close to a known popular skill, which is a common pattern in supply-chain attacks.
Whether any dangerous signals form a chain (credential access followed by encoding and outbound transmission, or remote code fetched and piped to a shell) that suggests intentional attack design rather than negligent coding.

Why it matters

These checks help users judge whether a skill looks maintainable, testable, and safe to install.
They separate well-documented submissions from packages that need more scrutiny before use.
A well-documented skill can still be harmful if it ships credentials or unsafe commands, so security signals matter as much as documentation quality.
Treating every dangerous API as equally suspicious produces too much noise. A skill that calls eval() is not the same as one that fetches remote code, decodes it from base64, and executes it on install. Vettd tries to surface that difference.

A skill score reflects the evidence visible in the submitted package. It is not a guarantee of safety in every environment. See the decision boundary guide for how Vettd distinguishes negligent code from intentional malice.

Reference frameworks

Framework labels in the directory are reference context. They indicate which standards a submitter or reviewer is using, not automated end-to-end compliance certifications.

OWASP

A broad application security reference used to reason about common software and web risk patterns.

It is often used as shorthand for issues like injection, broken access control, unsafe defaults, and weak validation.
In Vettd today, this label is reference context for reviewers and submitters, not an automated certification claim.

NIST 800-53

A large control catalog for security and privacy programs, frequently used in regulated or enterprise environments.

It covers control families such as access control, audit logging, configuration management, incident response, and system integrity.
When shown in the directory, it should be read as alignment context or reviewer intent, not proof that Vettd has mapped every control.

CMMC

A maturity model used in the defense industrial base to assess cybersecurity practices and process rigor.

It builds on structured security domains and is typically relevant when software or workflows touch controlled defense data or supplier requirements.
Vettd does not currently perform a formal CMMC assessment on assets surfaced in the public directory.

ISO 42001

An AI management system standard focused on governance, accountability, risk management, and organizational controls for AI.

It is more about management processes and oversight than a narrow code-only checklist.
A directory tag here should be interpreted as reference framing for AI governance conversations, not a completed ISO 42001 audit.

EU AI Act

A regulatory framework that classifies AI use cases by risk and imposes obligations based on how the system is used.

It matters most when an AI system is deployed into regulated contexts, user-facing decision flows, or high-risk operational domains.
Vettd can surface signals relevant to review, but a public label is not a legal determination that an asset satisfies the Act.

CISA

A practical security reference point associated with US cyber defense guidance, advisories, and operational best practices.

Teams often use it as a shorthand for security hygiene expectations around hardening, exposure reduction, and response readiness.
In the directory, this tag is descriptive context rather than evidence of a formal CISA review.