How Vettd evaluates AI assets.
Evaluation process
Structure
Vettd examines the submitted package or repository for required files, naming conventions, documentation completeness, and the presence of expected directories.
Security
Static analysis looks for credential exposure, unsafe shell patterns, metadata endpoint probing, name-squatting signals, and multi-step attack chains.
Safety
Each finding carries a severity. The severity levels, and the thresholds that compute a grade from them, are listed below.
High level by design
This page describes the categories of evidence Vettd looks at, the thresholds used to assign letter grades, and the severity levels applied to findings.
Evidence-based, not absolute
Vettd analyzes submitted packages, repository content, and scan data available at the time of submission. A scan with no findings is inconclusive, not a pass.
Asset types
What these five agentic asset types mean. Repository is a separate directory kind.
The evidence model
Every signal shown on the site belongs to one of these categories. The category decides how the signal is judged, not how much data we happen to have.
A shape is what kind of claim a signal is. Five shapes can be produced by a rule; absence is a display state, not a produced signal. The most useful question is not how big the number is but whether a change in it is real.
Popularity figures are public repository statistics. They are context and can change on their own, without the asset changing.
Vettd reports where each signal came from. Not every source is active yet.
Letter grades
Grades are computed from findings that carry a severity. Both Safety and Reliability & Quality are graded categories: each produces its own letter grade, computed independently from its own findings by the thresholds below. Info findings never count. Categories judged as measured or unjudged do not produce a grade. Thresholds are evaluated in order from F to A; the first tier whose condition is met wins.
Finding severities
Each finding is assigned one of five severity levels. Severity decides whether a finding counts toward a grade: info findings never do, and every other level feeds the grade for its category.
What Vettd checks, by asset type
Skills
8 checksFor public skills in the directory, Vettd evaluates package structure, documentation quality, and security signals. It looks at signals both in isolation and as sequences, because the difference between a careless mistake and an intentional attack is often visible in the chain of behaviors, not any single one.
What Vettd looks at
- Whether required structure is present, such as SKILL.md and supporting directories like scripts, references, assets, or evals.
- Whether the description is clear enough to explain what the skill does and when it should be used.
- Whether the instructions appear to include concrete workflow guidance, examples, validation steps, or checklists.
- Whether the package contains security red flags like embedded secrets, unsafe shell patterns, destructive commands, or environment files that should not ship.
- Whether scripts access known credential file paths such as cloud provider credentials, SSH private keys, or container registry configs.
- Whether scripts probe cloud instance metadata endpoints, which have no legitimate use in a skill and are a known credential-theft vector.
- Whether the skill name is suspiciously close to a known popular skill, which is a common pattern in supply-chain attacks.
- Whether any dangerous signals form a chain (credential access followed by encoding and outbound transmission, or remote code fetched and piped to a shell) that suggests intentional attack design rather than negligent coding.
Why it matters
- These checks help users judge whether a skill looks maintainable, testable, and safe to install.
- They separate well-documented submissions from packages that need more scrutiny before use.
- A well-documented skill can still be harmful if it ships credentials or unsafe commands, so security signals matter as much as documentation quality.
- Treating every dangerous API as equally suspicious produces too much noise. A skill that calls eval() is not the same as one that fetches remote code, decodes it from base64, and executes it on install. Vettd tries to surface that difference.
Reference frameworks
Framework labels in the directory are reference context. They indicate which standards a submitter or reviewer is using, not automated end-to-end compliance certifications.
OWASP
A broad application security reference used to reason about common software and web risk patterns.
NIST 800-53
A large control catalog for security and privacy programs, frequently used in regulated or enterprise environments.
CMMC
A maturity model used in the defense industrial base to assess cybersecurity practices and process rigor.
ISO 42001
An AI management system standard focused on governance, accountability, risk management, and organizational controls for AI.
EU AI Act
A regulatory framework that classifies AI use cases by risk and imposes obligations based on how the system is used.
CISA
A practical security reference point associated with US cyber defense guidance, advisories, and operational best practices.
Limits
Vettd analyzes the submitted artifact and public repository data. It does not run the asset or observe runtime behavior, and signals are only as fresh as the last scan or import.
What Vettd does not checkMore detail
This page is the summary. These pages go deeper on the parts a skeptical reader asks about.