AI Tool Curator: A Skill That Builds and Maintains Your Own Best-Of List

There are now more AI tools than any person could test in a lifetime. New agent skills, CLIs, MCP servers, frameworks, and "wrappers of wrappers" ship every single day. That abundance is a gift and a tax. The gift is that whatever you want to build, something probably already exists. The tax is that finding the right thing — the one that actually works, stays maintained, and won't lock you in — costs hours you don't have.

This is why curation is leverage. The person who can reliably tell you "use these three, avoid those five, and here's exactly why" is doing more valuable work than the person who built a tenth competing tool. Curation compresses thousands of hours of collective trial-and-error into a decision you can make in two minutes. That compression is worth money, attention, and trust.

The AI Tool Curator skill turns your agent into a disciplined research analyst. It tracks tools, evaluates them against a consistent rubric, stores results in a living database, and turns that database into roundups, comparison tables, and newsletters you'd actually stake your name on. No hype. No "10 mind-blowing tools" clickbait. Just defensible judgments you can update as the world changes.

Why trustworthy curation is genuinely valuable

Let me be blunt about the economics, because this is the part people skip. Trustworthy curation is one of the few content formats that gets more valuable as the underlying space gets noisier. When there were ten AI coding tools, a roundup was a convenience. Now that there are hundreds, a roundup that's actually correct is a public service — and a defensible business.

Here's why it works:

• Decision fatigue is real and expensive. Every developer evaluating a tool is spending salaried hours doing it. If your curation saves them a day of evaluation, you've delivered hundreds of dollars of value for free. That's why people subscribe, share, and come back.
• Trust compounds. The first time someone follows your recommendation and it works, they remember. The fifth time, they stop evaluating tools themselves and just check your list first. That's the moment curation becomes leverage — you've become the filter.
• It's monetizable without being sleazy. Sponsorships, affiliate links, a paid tier with deeper comparisons, a job board, a course — all of these sit naturally on top of a trusted list. The catch is the order of operations: trust first, money second. Reverse it and the whole thing collapses.

Curation is a promise: "I did the work so you don't have to, and I'll tell you when I'm uncertain." Break that promise once with a paid placement dressed up as a recommendation, and you've spent the only asset you had.

The opposite of curation is what most of the internet does: aggregate everything, rank by popularity, and call it a day. That's not curation, that's a leaderboard. A leaderboard tells you what's popular. Curation tells you what's good for your specific situation. The gap between those two is where all the value lives.

The rubric is the whole game

If there's one idea to take from this piece, it's this: a roundup is only as trustworthy as the rubric behind it. Without a rubric, "best" means "whatever I happened to like this week," and your readers can feel the inconsistency even if they can't name it. With a rubric, every judgment is traceable. Someone can disagree with your weights but they can't accuse you of being arbitrary.

A good rubric for AI tools and skills covers eight dimensions. Here's the one this skill ships with, along with what each actually measures.

Criterion	What it measures	Weight	Scoring notes
Problem fit	Does it solve a real, specific problem well?	25%	The single most important axis. A tool that nails one job beats a Swiss Army knife that's mediocre at ten.
Ease of setup	Time from "install" to "first useful result"	15%	Measure in minutes. Count every dependency, API key, and config file. Friction kills adoption.
Reliability	Does it work consistently across real inputs?	20%	Run it 5+ times on varied inputs. Flaky output is worse than no tool.
Documentation	Can a stranger succeed from the README alone?	10%	Check for a runnable quickstart, examples, and honest "limitations" sections.
Maintenance signals	Is it alive and likely to stay alive?	10%	Recent commits, responsive issues, named maintainer, changelog. Stars are NOT this.
Pricing & limits	Total cost at realistic usage, including hidden ones	8%	Free tier traps, per-seat creep, token costs, rate limits.
Lock-in	How hard to leave if it goes bad?	7%	Open formats, exportable data, standard protocols vs. proprietary cages.
Security & trust	What does it touch, and can you read the source?	5%	Permissions requested, network calls, secrets handling, supply-chain hygiene.

The weights aren't sacred — they're a starting point. If you're curating for an enterprise audience, security and lock-in jump. If you're curating for hobbyists, ease of setup and pricing dominate. The point isn't the exact numbers. The point is that you decide them once, publish them, and apply them the same way every time.

Scoring on a 1-5 scale, with anchors

Numbers without anchors are noise. "I'd give it a 4" means nothing unless 4 has a definition. The skill uses concrete anchors so two different evaluations stay comparable:

• 5 — Exceptional. Best in class on this axis; you'd recommend it on this dimension alone.
• 4 — Strong. Clearly above average; minor gaps that don't affect the recommendation.
• 3 — Adequate. Does the job; you can live with it; nothing special.
• 2 — Weak. Notable problems; only acceptable if alternatives are worse.
• 1 — Poor. A real liability; counts against using the tool.

A final weighted score becomes a single number out of 5, but — and this matters — you never publish the number alone. The number is a sorting key. The reasons are the product. A 4.2 with a one-line justification per axis is trustworthy. A 4.2 floating in space is exactly the hype you're trying to replace.

Stars are a popularity signal, not a review

This deserves its own subsection because it's the most common mistake in tool curation. GitHub stars, npm download counts, and "trending" badges measure attention, not quality. They're heavily lagged, easily gamed, and biased toward whoever had the best launch tweet.

Treat stars as exactly one weak input under "maintenance signals," and even there, read them skeptically:

• A repo with 30k stars and no commit in 14 months is a museum piece, not a recommendation.
• A repo with 200 stars, weekly commits, a responsive maintainer, and a clean changelog might be the better pick.
• Stars tell you a tool was interesting at some point. They tell you nothing about whether it works today on your inputs.

Before you recommend or install anything, read the source. For an agent skill, that means reading the SKILL.md and any scripts it ships. For an MCP server or CLI, it means scanning the entry point, the permissions it requests, and the network calls it makes. This is non-negotiable: you're about to give a tool access to your environment and put your name on a recommendation. Five minutes of reading the source is the cheapest insurance you'll ever buy. If a tool's source is unreadable, obfuscated, or the install script does things it doesn't explain — that's your review, and it's a low score.

A repeatable research workflow

Trust comes from process, and process comes from doing the same steps in the same order every time. Here's the four-phase workflow the skill follows. The discipline is what separates a curator from someone who just has opinions.

Phase 1 — Source and capture

You can't evaluate what you haven't found, and you can't find good tools by only reading what's trending. Cast a wide, diverse net:

• Primary sources: the official repo, the README, the changelog, the docs site. Always start here.
• Ecosystem hubs: skill registries, awesome-lists, package registries, and curated directories.
• Practitioner channels: newsletters from people who actually build, conference talks, and the GitHub feeds of engineers you respect.
• The cold corners: issue trackers (where the real bugs live), "show HN" threads, and the second page of search results that nobody reads.

When you find a candidate, capture it immediately in a structured record — don't trust memory. At minimum: name, URL, one-line description, category, the date you found it, and where you found it. Capture is cheap; rediscovery is expensive.

Phase 2 — Triage

Not everything deserves a full evaluation. Triage ruthlessly against three quick filters:

1. Is it real? Does it install and run at all, or is it a README promising a future product?
2. Is it relevant? Does it fit a category your audience actually cares about?
3. Is it differentiated? Is it meaningfully different from the three things you've already scored, or is it a clone?

Anything that passes triage goes into the "to evaluate" queue. Everything else gets a one-line note and a parking spot, so you never re-triage the same thing twice.

Phase 3 — Test

This is the phase people fake, and faking it is how bad roundups get written. You cannot score reliability, ease of setup, or problem fit by reading marketing copy. You have to run the thing.

# A minimal, honest test harness for an agent skill
# 1. Read the source BEFORE installing — this is the security review
cat path/to/SKILL.md
ls -la path/to/skill/   # what scripts ship with it?

# 2. Install into an isolated, throwaway environment
npx skills add owner/repo --skill the-skill

# 3. Time the path from install to first useful result
time ./run-first-real-task.sh

# 4. Run the same realistic task 5 times with varied inputs
for i in 1 2 3 4 5; do
  ./run-real-task.sh "test-input-$i" >> results.log 2>&1
done

# 5. Record what broke, what was slow, and what surprised you

Use a realistic task, not the toy example from the docs. Docs examples are chosen to succeed; your readers' real inputs are not. Note the setup time in minutes, count the failures, and write down anything that surprised you — surprises are where the most useful review notes live.

Phase 4 — Score and justify

Now, and only now, open the rubric. For each of the eight criteria, assign a 1-5 with a one-sentence justification. The justification is mandatory. If you can't write a sentence explaining a score, you don't understand the tool well enough to publish a number.

Compute the weighted total, then do a gut check: does the number match your felt experience of using the tool? If a tool scored 4.5 but you'd never actually reach for it, your rubric is missing a dimension — find it and add it. The rubric serves your judgment; it doesn't replace it.

Maintaining a living database

A roundup is a snapshot. A database is an asset. The difference is that the database keeps earning after you write it, because you can re-slice it into a dozen pieces of content and refresh it as the world moves.

Keep it boring and plain-text so it's diffable, greppable, and version-controlled. A single YAML or JSON file per tool, or one big structured file, both work. Here's a record shape that's served me well:

- id: superpowers
  name: "Superpowers"
  url: "https://github.com/obra/superpowers"
  category: "agent-skills"
  one_liner: "A broad library of composable agent skills."
  found_via: "ecosystem-hub"
  first_seen: "2026-03-02"
  last_reviewed: "2026-05-20"
  scores:
    problem_fit: 5
    ease_of_setup: 4
    reliability: 4
    documentation: 4
    maintenance: 5
    pricing: 5      # open source
    lock_in: 5      # plain markdown skills, fully portable
    security: 4
  weighted_total: 4.6
  notes: "Read SKILL.md files before enabling individual skills; breadth is the draw."
  disclosure: "none"   # no affiliate, no sponsorship
  status: "recommended"

Refreshing without rewriting everything

The killer feature of a database is staleness tracking. Add a last_reviewed date to every record and let your agent flag anything older than your refresh window — 90 days is a sane default for a fast-moving space. A refresh isn't a full re-evaluation; it's a quick check of the dimensions most likely to have changed:

• Is it still maintained? (commits, issue responsiveness)
• Did pricing change? (the most common silent shift)
• Did a breaking version ship? (re-run the setup test)
• Did a better alternative appear? (re-check differentiation)

Most refreshes take five minutes and change nothing. The few that change something are exactly the updates your readers most need, because they're the ones nobody else caught. This is your durable edge: anyone can write a roundup once; almost nobody keeps one correct.

Turning the database into content

Here's where curation pays you back. One well-maintained database becomes many pieces of content, each a different view of the same trustworthy data.

Roundups

The classic "best X for Y" article. Pull every tool in a category, sort by weighted score, and write a short, honest paragraph for each. Lead with who it's for and who it's not for — that "not for" sentence is what makes readers trust you, because it proves you're not just selling. End each entry with the one thing that would make you change your recommendation.

Comparison tables

When two or three tools are close, a table beats prose. Show the per-criterion scores side by side so readers can apply their own weights:

Criterion	Superpowers	GStack	GSD Core
Problem fit	5	4	5
Ease of setup	4	5	4
Reliability	4	4	5
Maintenance	5	4	4
Lock-in	5	4	5
Best for	breadth	quick start	disciplined workflows

The reader who cares most about setup speed reads one row; the reader who cares about lock-in reads another. A good table respects that different people have different weights — and it's honest precisely because it lets them disagree with your overall ranking.

Newsletters

The database is a newsletter engine. Each issue writes itself from three queries: what's new (records added since last issue), what changed (records whose scores or status moved on refresh), and what to watch (high-potential candidates still in evaluation). That's a useful, recurring email with almost no incremental writing — and "what changed" is the section no aggregator can replicate, because it requires having scored the thing the first time.

Disclosure, ethics, and affiliate transparency

If you're going to monetize curation — and you should be able to — your credibility depends entirely on a clean wall between recommendations and revenue. The rules are simple and non-negotiable:

• Score before you monetize. A tool earns its place by its rubric score, full stop. A sponsorship can buy visibility (a clearly-labeled placement) but never ranking. If money can move a tool up your list, you don't have a list, you have an ad network pretending to be one.
• Label everything. Affiliate links, sponsored sections, and free review units all get a plain-language disclosure where the reader actually sees it — not buried in a footer. "This link is an affiliate link; it doesn't affect the score" costs you nothing and buys you everything.
• Carry the disclosure field in the database itself. Make transparency structural, not an afterthought. Every record stores its disclosure status, so it's impossible to publish a recommendation without the disclosure traveling with it.
• Disclose your method, too. Publish the rubric and the weights. When readers can see how you judge, they trust the judgments even when they disagree. Hidden methodology is just authority cosplay.

The fastest way to destroy a curation business is to let one paid placement masquerade as an honest recommendation. The slowest, most durable way to build one is to disclose so thoroughly that readers never have to wonder.

Avoiding recency bias and hype

Two failure modes quietly wreck otherwise-good curation. Both come from the same root: confusing novelty and noise with quality.

Recency bias is the pull to rank the newest thing highest simply because it's new and you just read about it. New tools are exciting; they're also unproven, often unmaintained six months later, and frequently solving a problem a boring older tool already solved. The defense is structural: the rubric has no "newness" axis. A tool gets points for problem fit and reliability, both of which favor maturity. If something new genuinely scores higher, great — but it has to earn it on the same axes as everything else.

Hype bias is letting popularity leak into your scoring. You see 40k stars and a thousand retweets and your brain pre-decides the tool is good before you've run it. The defense is the workflow: you score from testing, not from sentiment. Read the source, run the task five times, write the justifications — by the time you reach a number, the hype has been replaced by evidence.

A practical trick: score blind when you can. Evaluate the tool against the rubric before you look at its star count or read its launch thread. Then check the popularity signals last, only as a weak maintenance input. You'll be amazed how often the crowd-favorite and the rubric-favorite are different tools — and how often the rubric is right.

A worked example: curating the agent-skills ecosystem

Let me make this concrete with a domain I actually curate: agent skills. The ecosystem is young, fast, and noisy — a perfect test of the method.

A curator working this space would track and score things like Superpowers (a broad, composable skill library), GStack (a batteries-included starter stack), GSD Core ("get stuff done" workflow skills), the AWS Agent Toolkit (cloud-integration skills), and guideline-style resources like the Karpathy guidelines and Matt Pocock's TypeScript-flavored agent material. Each goes through the same four phases: source, triage, test, score.

What does the rubric surface that a popularity ranking wouldn't?

• A library can be broad (high problem fit for "I need lots of skills") while being uneven (you must read each SKILL.md before enabling it — a real ease-of-setup and security cost). The rubric forces you to say both, instead of just "it's popular."
• A guideline document isn't a tool you install — so "ease of setup" and "reliability" don't apply the same way, and you'd adapt the rubric for that record type rather than forcing a bad fit. Knowing when to flex the rubric is part of the skill.
• A cloud-integration toolkit scores high on problem fit for its niche but high on lock-in by nature. That's not a flaw to hide; it's a tradeoff to name, so the reader who's already all-in on that cloud reads it differently than the reader who isn't.

The output isn't "here are the most-starred agent skills." It's "here's which skill to reach for given your constraints, here's exactly why, here's what would change my mind, and here's when I last checked." That's the difference between a leaderboard and curation — and it's why a well-maintained list in a noisy space is one of the most defensible things you can build.

Resources & links

Internal — start with the skills and guides that pair with curation:

• SEO Longform Writer skill — turn a curated database into roundups that actually rank.
• Productized AI Service skill — package your curation into a recurring offer.
• Context Engineering skill — keep your agent's research context clean across long evaluation sessions.
• Karpathy Guidelines skill — a guideline resource worth having in your tracked database.
• Most Popular AI Coding Skills guide — popularity data to read skeptically, exactly as discussed above.
• Make Money With AI in 2026 guide — the business models that sit on top of trusted curation.
• Write SEO Articles With AI guide — distribution for your roundups.
• The AI Curation learning track — the full path from rubric to revenue.

External — example repos worth curating in the agent-skills domain (read the SKILL.md and source before installing any of them):

• Superpowers — https://github.com/obra/superpowers
• GStack — https://github.com/garrytan/gstack
• Karpathy agent skills — https://github.com/multica-ai/andrej-karpathy-skills

Related skills on this site: Superpowers, GStack, GSD Core, Grill Me, Agent Toolkit AWS, Skilld, and Ship Skills With Package.