DEVX-167: docs: add dependency-graph, deployment-coordination, and skill-creation skills

Co-authored-by: emil User <emil.simeonov@tutanota.com>
2026-09-05 14:16:52 +00:00
parent c6bd4e8f63
commit 4638e334b5
4 changed files with 431 additions and 0 deletions
@@ -0,0 +1,142 @@
+# skill-creation
+
+How to create, validate, and maintain Devin skills. Skills must be
+clear, succinct, and actionable — no AI slop.
+
+## When to Invoke
+
+Invoke this skill when:
+- Creating a new skill
+- Amending an existing skill
+- Evaluating whether a skill is needed
+- Reviewing a PR that adds or modifies skills
+
+## Prerequisites
+
+- Skill directory: `.devin/skills/<skill-name>/SKILL.md`
+- Validator: `tests/test_skills.py` (infra) or reference to it
+- Tests: `tests/unit/test_skills_validation.py` (infra)
+
+## When to Create a Skill
+
+Create a skill when:
+- An agent struggles with a task repeatedly (branch hygiene, PR order)
+- A workflow has non-obvious ordering constraints (deployment coordination)
+- A task requires specific tool usage over raw commands (CI monitoring)
+- Multiple agents need shared context (dependency graph)
+
+Do NOT create a skill for:
+- One-off tasks (use a spec instead)
+- Tasks already covered by AGENTS.md
+- Tasks that are obvious from the Makefile or README
+- Tasks that change frequently (skills should be stable)
+
+## Skill Structure
+
+Every skill MUST have:
+
+```markdown
+# <skill-name>
+
+One-line description of what the skill does.
+
+## When to Invoke
+
+2-4 bullet points describing when to use this skill.
+
+## Prerequisites
+
+What must exist before using the skill (venv, .env, tools).
+
+## <Core Content>
+
+The actual guidance. Keep it actionable.
+
+## Verification (if applicable)
+
+How to verify the skill's guidance works.
+
+## Common Mistakes (if applicable)
+
+What agents get wrong without this skill.
+```
+
+## Quality Standards
+
+### Do
+- **Be specific.** Reference exact make targets, file paths, commands.
+- **Be concise.** Each section should be scannable in under 30 seconds.
+- **Be actionable.** Every paragraph should tell the agent what to DO.
+- **Use tables** for command reference, mappings, and comparisons.
+- **Use code blocks** for commands the agent should run.
+- **Link to other skills** when related (e.g., "See `dependency-graph` skill").
+
+### Don't
+- **No preamble.** Don't start with "This skill helps agents..." — just state what it does.
+- **No filler.** Don't repeat information from AGENTS.md or other skills.
+- **No vague advice.** "Be careful with branches" is useless. "Run `git branch --show-current` before every commit" is useful.
+- **No AI slop.** Don't write "In this comprehensive guide, we will explore..." — just give the guidance.
+- **No redundant sections.** If "Common Mistakes" would repeat "When to Invoke", skip it.
+- **No marketing.** Don't describe the skill as "powerful" or "comprehensive".
+
+## Scope Rules
+
+- **One skill per concern.** Don't mix branch hygiene with CI monitoring.
+- **Project-specific, not generic.** Skills reference this repo's make targets, file paths, and conventions — not abstract advice.
+- **Shared skills must be identical across repos.** Use `SHARED_SKILLS` in the validator to enforce this.
+- **Per-repo skills must reflect that repo's reality.** Don't copy infra-specific targets to sso-bridge.
+
+## Automated Validation
+
+Every skill must pass the validator (`tests/test_skills.py`). The validator checks:
+
+1. **Structure** — H1 title, "When to Invoke" section, "Prerequisites" section
+2. **Commands** — referenced `make <target>` commands exist in Makefile or devx.mak
+3. **Paths** — referenced file paths exist in the repo
+4. **Shared skills** — identical content across repos (SHA-256 comparison)
+5. **No drift** — no references to nonexistent commands or files
+
+Run the validator:
+```bash
+python3 tests/test_skills.py --repo infra --repo sso-bridge
+```
+
+## Effectiveness Evaluation
+
+### Static Checks (automated, CI)
+
+The validator runs in CI as part of `make pytest-cov`. A failing skill
+test blocks the PR. This catches:
+- Missing sections
+- Invalid commands
+- Broken file references
+- Cross-repo drift
+
+### Runtime Metrics (manual, periodic)
+
+Track these signals to evaluate skill effectiveness:
+- **Skill invocation frequency** — how often agents invoke the skill
+- **Success rate when invoked** — did the skill prevent the mistake it targets?
+- **Feedback issues** — agents create Gitea issues with `feedback` label when a skill is unclear or wrong
+- **Mistake recurrence** — if agents still make the mistake the skill targets, the skill needs improvement
+
+### Retrospective Review
+
+Periodically (monthly or after major incidents) review skills:
+1. List all skills and their last-modified dates
+2. Check for feedback issues tagged `skill-improvement`
+3. Verify referenced commands still exist (run validator)
+4. Remove skills that are no longer relevant
+5. Update skills where mistakes still recur
+6. Document lessons in this skill's "Common Mistakes" section
+
+## Creating a New Skill — Checklist
+
+- [ ] Identify the repeated struggle or non-obvious workflow
+- [ ] Check no existing skill covers it
+- [ ] Write the skill following the structure above
+- [ ] Run `python3 tests/test_skills.py` — must pass
+- [ ] Run `make pytest-cov` — must pass with 100% coverage
+- [ ] If shared across repos, copy identical content to each repo
+- [ ] Add the skill to `SHARED_SKILLS` in the validator if shared
+- [ ] Create PR, verify CI passes, merge