Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6973f9d851 | ||
|
|
2669a0ea73 | ||
|
|
03f057b55a | ||
|
|
706d6dafe0 | ||
|
|
03ddce427c | ||
|
|
9642d6884c | ||
|
|
b2074d6635 | ||
|
|
a6dddf25e7 | ||
|
|
01130a7385 | ||
|
|
07580c9280 | ||
|
|
6601d90bee | ||
|
|
0df79fed53 | ||
|
|
cf8287e683 | ||
|
|
9f1bdc4cf1 |
@@ -1,135 +0,0 @@
|
||||
# dependency-graph
|
||||
|
||||
Map of the oblachno ecosystem. Knows which repo produces what, which
|
||||
repos depend on which, and the correct order for cross-repo changes.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when:
|
||||
- Changes span multiple repos
|
||||
- A change in one repo requires version bumps in downstream repos
|
||||
- Deploying infrastructure that depends on published packages/images
|
||||
- Verifying the ecosystem is in a consistent state before deployment
|
||||
- Determining which repos to update and in what order
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- All repos cloned under `/home/emo/dev/ideas/oblachno/`
|
||||
- `.env` with `DEVELOPER_GITEA_API_TOKEN` in each repo
|
||||
|
||||
## Ecosystem Map
|
||||
|
||||
```
|
||||
devx (PyPI package)
|
||||
/ | \
|
||||
/ | \
|
||||
grm sso-bridge infra
|
||||
(PyPI) (PyPI+Docker) (deploys all)
|
||||
| | |
|
||||
v v v
|
||||
infra bump infra bump staging
|
||||
(auto PR) (auto PR) production
|
||||
|
|
||||
mattermost-oidc (Docker image)
|
||||
(infra pulls :latest at deploy)
|
||||
```
|
||||
|
||||
## Repositories
|
||||
|
||||
| Repo | Produces | Consumers | Release Trigger |
|
||||
|------|----------|-----------|-----------------|
|
||||
| `devx` | PyPI package `devx` | grm, sso-bridge, infra | User-facing changes to `src/devx/**` |
|
||||
| `grm` | PyPI package `grm` | infra | User-facing changes to `src/grm/**` or `ansible/**` |
|
||||
| `sso-bridge` | PyPI package `sso_bridge` + Docker image | infra | User-facing changes to `src/sso_bridge/**` or `ansible/**` |
|
||||
| `infra` | Staging/production deployment | (end users) | User-facing changes + nightly gate |
|
||||
| `mattermost-oidc` | Docker image `mattermost-oidc` | infra (pulls at deploy) | `Dockerfile` or `build.yml` changes |
|
||||
|
||||
## Dependency Chain
|
||||
|
||||
### devx → all repos
|
||||
|
||||
devx publishes to the Gitea PyPI registry. grm, sso-bridge, and infra
|
||||
pin devx in `pyproject.toml`:
|
||||
```toml
|
||||
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@vX.Y.Z"
|
||||
```
|
||||
|
||||
When devx publishes a new version:
|
||||
1. grm, sso-bridge, and infra must bump their pinned devx version
|
||||
2. This is currently manual — no auto-dependency-PR from devx
|
||||
3. Each repo must `make setup` to pick up the new version
|
||||
|
||||
### grm → infra
|
||||
|
||||
grm publishes to PyPI. Its post-merge workflow auto-creates an infra
|
||||
dependency PR via `devx.ci.create_dependency_pr --repo oblachno/infra
|
||||
--package grm`. The PR bumps the pinned grm version in infra's
|
||||
`pyproject.toml`.
|
||||
|
||||
### sso-bridge → infra
|
||||
|
||||
sso-bridge publishes to PyPI AND builds a Docker image. Its post-merge
|
||||
workflow auto-creates an infra dependency PR via
|
||||
`devx.ci.create_dependency_pr --repo oblachno/infra --package
|
||||
sso_bridge`. The PR bumps the pinned sso_bridge version.
|
||||
|
||||
The Docker image is pulled by infra at deploy time (`sso-bridge:latest`).
|
||||
|
||||
### mattermost-oidc → infra
|
||||
|
||||
mattermost-oidc builds a Docker image tagged `:latest` and `:MM_VERSION`.
|
||||
infra pulls `mattermost-oidc:latest` at deploy time. There is no
|
||||
auto-dependency-PR — infra simply pulls the latest image.
|
||||
|
||||
### infra → staging/production
|
||||
|
||||
infra deploys to staging and production. The deployment:
|
||||
1. Provisions VMs from golden images
|
||||
2. Runs Ansible roles (including grm and sso-bridge roles)
|
||||
3. Pulls Docker images (sso-bridge, mattermost-oidc)
|
||||
4. Configures services
|
||||
|
||||
## Correct Order for Cross-Repo Changes
|
||||
|
||||
When a change spans multiple repos, follow this order:
|
||||
|
||||
1. **devx first** — if the change starts in devx, merge and publish devx
|
||||
first. Wait for the PyPI publish job to complete.
|
||||
2. **Bump devx in consumers** — in grm/sso-bridge/infra, bump the pinned
|
||||
devx version, run `make setup`, verify tests pass, merge.
|
||||
3. **grm/sso-bridge second** — merge and publish grm/sso-bridge. Wait
|
||||
for the PyPI publish + Docker image build to complete.
|
||||
4. **Auto-dependency-PRs** — grm/sso-bridge post-merge auto-creates infra
|
||||
PRs to bump pinned versions. Wait for these PRs to appear.
|
||||
5. **Merge infra dependency PRs** — review and merge the auto-created
|
||||
infra PRs.
|
||||
6. **infra last** — deploy to staging, validate, promote to production.
|
||||
|
||||
## State Verification Before Deployment
|
||||
|
||||
Before deploying infra, verify:
|
||||
|
||||
1. **devx version consistent** — all repos pin the same devx version
|
||||
2. **grm published** — latest grm tag exists in PyPI
|
||||
3. **sso-bridge published** — latest sso_bridge tag exists in PyPI
|
||||
4. **sso-bridge image built** — latest sso-bridge Docker image exists
|
||||
5. **mattermost-oidc image built** — latest mattermost-oidc image exists
|
||||
6. **infra pins match published versions** — no stale pins
|
||||
7. **Nightly gate green** — `NIGHTLY_STATUS` is not `failed`
|
||||
|
||||
## Quick Check Commands
|
||||
|
||||
```bash
|
||||
# Check latest devx version
|
||||
curl -sS https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno-oss/devx/releases/latest | python3 -c "import json,sys; print(json.load(sys.stdin).get('tag_name','?'))"
|
||||
|
||||
# Check pinned devx version in each repo
|
||||
for repo in grm sso-bridge infra; do
|
||||
echo -n "$repo: "; grep 'devx @' /home/emo/dev/ideas/oblachno/$repo/pyproject.toml | grep -oP 'v[\d.]+'
|
||||
done
|
||||
|
||||
# Check latest sso-bridge image build
|
||||
curl -sS -H "Authorization: token $DEVELOPER_GITEA_API_TOKEN" \
|
||||
"https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno/sso-bridge/actions/runs?per_page=5" \
|
||||
| python3 -c "import json,sys; [print(r['id'],r['status'],r['conclusion']) for r in json.load(sys.stdin).get('workflow_runs',[]) if r.get('event')=='push']"
|
||||
```
|
||||
@@ -1,90 +0,0 @@
|
||||
# deployment-coordination
|
||||
|
||||
How devx releases propagate to downstream repos. devx is the base
|
||||
package — all other repos pin it. A devx release must complete before
|
||||
consumers can bump.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when:
|
||||
- Changes to devx affect downstream repos (grm, sso-bridge, infra)
|
||||
- Preparing a devx release that other repos depend on
|
||||
- Verifying downstream repos have bumped to the latest devx version
|
||||
- Coordinating a multi-repo change that starts in devx
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- devx repo at `/home/emo/dev/ideas/oblachno/devx`
|
||||
- `.env` with `DEVELOPER_GITEA_API_TOKEN`
|
||||
- See `dependency-graph` skill for the full ecosystem map
|
||||
|
||||
## What devx Produces
|
||||
|
||||
devx publishes a Python package to the Gitea PyPI registry. Downstream
|
||||
repos pin it:
|
||||
```toml
|
||||
"devx @ git+https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git@vX.Y.Z"
|
||||
```
|
||||
|
||||
## Release Flow
|
||||
|
||||
1. PR merged to master
|
||||
2. Post-merge workflow runs `devx.ci.release` — classifies changes
|
||||
3. If user-facing changes: git-cliff bumps version, creates tag, pushes
|
||||
4. `devx.ci.publish` builds and publishes to Gitea PyPI registry
|
||||
5. Downstream repos must bump their pinned devx version
|
||||
|
||||
## Downstream Consumers
|
||||
|
||||
| Repo | Pin location | Auto-bump? |
|
||||
|------|-------------|------------|
|
||||
| grm | `pyproject.toml` | No — manual |
|
||||
| sso-bridge | `pyproject.toml` | No — manual |
|
||||
| infra | `pyproject.toml` | No — manual |
|
||||
|
||||
devx does NOT auto-create dependency PRs in downstream repos. Bumping
|
||||
is manual: create a PR in each downstream repo to update the pinned
|
||||
version.
|
||||
|
||||
## Coordinating a devx Change
|
||||
|
||||
When a change to devx affects downstream repos:
|
||||
|
||||
1. **Merge devx PR** — wait for post-merge publish to complete
|
||||
2. **Verify publish** — check the new tag exists:
|
||||
```bash
|
||||
curl -sS -H "Authorization: token $DEVELOPER_GITEA_API_TOKEN" \
|
||||
https://git.oblachno.oblachno.fyi/api/v1/repos/oblachno-oss/devx/releases/latest \
|
||||
| python3 -c "import json,sys; print(json.load(sys.stdin).get('tag_name','?'))"
|
||||
```
|
||||
3. **Bump downstream repos** — for each repo (grm, sso-bridge, infra):
|
||||
- Update `devx @ ...@vX.Y.Z` in `pyproject.toml`
|
||||
- Run `make setup` to install the new version
|
||||
- Run `make pytest-cov` to verify compatibility
|
||||
- Create and merge a PR
|
||||
4. **Verify infra deploys** — after infra bumps, verify staging deploy
|
||||
picks up the new devx version
|
||||
|
||||
## State Verification
|
||||
|
||||
Before starting a devx change, verify current state:
|
||||
```bash
|
||||
# Current devx version
|
||||
grep '__version__' /home/emo/dev/ideas/oblachno/devx/src/devx/__init__.py
|
||||
|
||||
# What each repo pins
|
||||
for repo in grm sso-bridge infra; do
|
||||
echo -n "$repo pins: "
|
||||
grep 'devx @' /home/emo/dev/ideas/oblachno/$repo/pyproject.toml | grep -oP 'v[\d.]+'
|
||||
done
|
||||
```
|
||||
|
||||
If pins are inconsistent across repos, bump them to the latest published
|
||||
version before starting new work.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- Merging a devx PR and immediately merging downstream PRs without
|
||||
waiting for the publish job to complete
|
||||
- Forgetting to bump infra (it has the most complex deploy pipeline)
|
||||
- Bumping only one downstream repo when the change affects all three
|
||||
@@ -2,26 +2,17 @@
|
||||
|
||||
Quick reference for devx tools when working on the devx repo itself.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when creating PRs, checking CI status, adding
|
||||
labels, rebasing branches, or performing any PR lifecycle operation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `.venv` exists (run `make setup` if not)
|
||||
- `.env` with `DEVELOPER_GITEA_API_TOKEN`, `VIKUNJA_TOKEN`
|
||||
|
||||
## PR Workflow (use these, not raw git/tea/MCP)
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Create Vikunja task | `.venv/bin/python -m devx.tools.create_task --title "..." --description "..."` (make target doesn't forward args) |
|
||||
| Create Vikunja task | `make create-task -- --title "..." --description "..."` |
|
||||
| Create PR | `make create-pr` |
|
||||
| Push + create PR | `make push-with-pr` |
|
||||
| Check CI status | `make devx-pr-status` or `make devx-pr-status PR=42 WAIT=1` |
|
||||
| Fetch CI failure logs | `make devx-pr-logs` or `make devx-pr-logs PR=42 JOB=quality TAIL=50` |
|
||||
| Add ready-to-merge label | `make devx-pr-label` or `make devx-pr-label PR=42` |
|
||||
| Post PR review | `make devx-pr-review PR=42 EVENT=APPROVE BODY="..." CHECKLIST=1,2,3,4,5,6,7,8,9,10,11,12,13` |
|
||||
| Rebase current branch | `make rebase` |
|
||||
| Rebase PR via API | `make pr-rebase` or `make pr-rebase PR=42` |
|
||||
|
||||
@@ -33,15 +24,6 @@ When the `ready-to-merge` label is added and all CI checks pass:
|
||||
3. The rebase triggers a new CI run; the next auto-merge attempt merges
|
||||
4. No manual rebase needed unless the API rebase fails
|
||||
|
||||
## Spec-Driven CI Gates (Pre-merge)
|
||||
|
||||
Every PR must pass these gates before merge:
|
||||
|
||||
| Gate | Module | What it checks |
|
||||
|------|--------|----------------|
|
||||
| Spec validation | `devx.ci.validate_spec` | Spec file exists at `docs/specs/<TASK-ID>.md`, has REQ-IDs, all ACs checked |
|
||||
| PR size | `devx.ci.check_pr_size` | Max 500 lines / 10 files (excludes CHANGELOG, badges, locks) |
|
||||
|
||||
## Key Rules
|
||||
|
||||
- Never manually merge via API — always use auto-merge with `ready-to-merge` label
|
||||
|
||||
@@ -1,272 +0,0 @@
|
||||
# pr-review
|
||||
|
||||
Deep, critical PR review with auto-fix. This skill guides the agent
|
||||
through a thorough review of a pull request, posting inline comments
|
||||
for each issue found, auto-fixing them, resolving the discussion threads,
|
||||
and marking the PR as ready-to-merge when no blocking issues remain.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when asked to review a PR, or when a PR is open and
|
||||
needs review before merge. Do NOT invoke automatically on every PR —
|
||||
this is an on-demand deep review, not a CI gate.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- The PR must be open in a Gitea repo
|
||||
- The agent needs Gitea MCP access (gitea server)
|
||||
- The agent needs git push access to the PR's head branch
|
||||
- The PR should have passed CI (validate job) before deep review
|
||||
|
||||
## Review Categories
|
||||
|
||||
Review every PR against these 8 categories. For each issue found, post
|
||||
an inline comment on the specific line, then auto-fix it.
|
||||
|
||||
### 1. Functional Correctness
|
||||
|
||||
- Does the code actually do what the spec/PR title claims?
|
||||
- Are edge cases handled? (empty input, null, boundary values, concurrent access)
|
||||
- Are error paths tested? Not just happy path.
|
||||
- Does the code handle all return values? (ignored errors, unchecked None)
|
||||
- Are there off-by-one errors, wrong comparisons, inverted conditions?
|
||||
- Do loops terminate correctly? (no infinite loops, correct break/continue)
|
||||
- Are regex patterns correct? (anchored, escaped, non-greedy where needed)
|
||||
- Are API responses validated before use? (status codes, response shape)
|
||||
|
||||
### 2. Completeness
|
||||
|
||||
- Are all requirements from the spec implemented? (check each REQ-ID)
|
||||
- Are all acceptance criteria in the spec checked off?
|
||||
- Are tests written for all new code paths?
|
||||
- Are error messages user-facing (wrapped in `_()`)?
|
||||
- Are new CLI commands documented in `docs/user/cli-commands.md`?
|
||||
- Are new modules added to architecture docs?
|
||||
- Are CHANGELOG entries added for user-facing changes?
|
||||
- Are translations added for new user-facing strings?
|
||||
|
||||
### 3. Architecture
|
||||
|
||||
- Does the code follow the repo's layer separation? (no business logic in CLI, no direct subprocess in CLI)
|
||||
- Are new dependencies justified? (no unnecessary new packages)
|
||||
- Is configuration via env vars / config.py, not hardcoded?
|
||||
- Are new modules placed in the correct directory? (ci/ vs tools/ vs molecule/)
|
||||
- Does the code reuse existing utilities? (no reimplemented helpers)
|
||||
- Are imports circular? (check import chains)
|
||||
- Is the code testable? (injectable dependencies, no hidden global state)
|
||||
- Does the code follow existing patterns in the codebase?
|
||||
|
||||
### 4. Reliability
|
||||
|
||||
- Are external API calls retried with backoff?
|
||||
- Are timeouts set on all network operations?
|
||||
- Are file operations atomic? (write to temp, rename)
|
||||
- Are database operations transactional where needed?
|
||||
- Are there race conditions? (check shared mutable state)
|
||||
- Are resources cleaned up in all paths? (finally blocks, context managers)
|
||||
- Can the code handle partial failures? (one service down, others up)
|
||||
- Are idempotency guarantees maintained? (safe to retry)
|
||||
|
||||
### 5. Robustness
|
||||
|
||||
- Does the code fail gracefully? (meaningful error messages, not stack traces)
|
||||
- Are unexpected inputs handled? (type checking, validation)
|
||||
- Are there any crash-on-bad-input paths?
|
||||
- Does the code degrade under load? (backpressure, queue limits)
|
||||
- Are there resource leaks? (file handles, connections, memory)
|
||||
- Does the code survive network partitions? (retry, circuit breaker)
|
||||
- Are there any unhandled exceptions that could crash the process?
|
||||
- Is logging sufficient to diagnose production issues?
|
||||
|
||||
### 6. Security
|
||||
|
||||
- Are there hardcoded secrets, tokens, or passwords?
|
||||
- Is `shell=True` used with user input? (command injection)
|
||||
- Is `eval()` or `exec()` used? (code injection)
|
||||
- Are SQL queries parameterized? (no string concatenation)
|
||||
- Are file paths validated? (no path traversal)
|
||||
- Are user inputs sanitized before display? (XSS in web contexts)
|
||||
- Are SSL/TLS verifications disabled without justification?
|
||||
- Are secrets logged in error messages or debug output?
|
||||
- Are permissions checked before privileged operations?
|
||||
- Is sensitive data in memory longer than necessary?
|
||||
|
||||
### 7. Technical Excellence
|
||||
|
||||
- Are functions under 50 lines? (refactor if longer)
|
||||
- Is cyclomatic complexity reasonable? (no deeply nested if/else chains)
|
||||
- Are names meaningful? (no single-letter vars, no misleading names)
|
||||
- Is dead code removed? (no commented-out blocks, no unused imports)
|
||||
- Are comments explaining WHY, not WHAT?
|
||||
- Is the code DRY? (no copy-pasted blocks that should be shared)
|
||||
- Is the code SOLID? (single responsibility, open/closed)
|
||||
- Are magic numbers extracted to named constants?
|
||||
- Is the code formatted per the repo's linter config?
|
||||
- Are type hints present on all function signatures?
|
||||
|
||||
### 8. Test Quality
|
||||
|
||||
- Do tests actually test the behavior? (not just that code runs)
|
||||
- Are tests independent? (no shared mutable state, no order dependency)
|
||||
- Are tests fast? (no real sleeps, no real network calls, mocked)
|
||||
- Are edge cases tested? (empty, None, boundary, error paths)
|
||||
- Are test names descriptive? (test_what_condition_expected_result)
|
||||
- Are mocks set up correctly? (mocking the right object, not too broad)
|
||||
- Is coverage 100% for new code? (every branch, every line)
|
||||
- Are integration tests added for cross-module changes?
|
||||
- Do tests clean up after themselves? (tmp_path, fixtures)
|
||||
|
||||
## Review Procedure
|
||||
|
||||
### Step 1: Gather Context
|
||||
|
||||
```
|
||||
1. Read the PR spec (if exists): docs/specs/<TASK-ID>.md
|
||||
2. Fetch PR details via Gitea MCP: pull_request_read (get_pr, list_pr_files)
|
||||
3. Read the full diff: git diff origin/master...HEAD
|
||||
4. Read the PR description and any existing review comments
|
||||
5. Identify the repo's task prefix (OBL-INFRA, GRM, SSO, DEVX)
|
||||
```
|
||||
|
||||
### Step 2: Review Each File
|
||||
|
||||
For each changed file in the PR:
|
||||
|
||||
1. Read the full file (not just the diff) to understand context
|
||||
2. Go through all 8 review categories
|
||||
3. For each issue found, note: file path, line number, category, severity, description, suggested fix
|
||||
|
||||
### Step 3: Post Inline Comments
|
||||
|
||||
For each issue found, post an inline review comment using the Gitea MCP:
|
||||
|
||||
```
|
||||
mcp_call_tool: gitea / pull_request_review_write
|
||||
method: create
|
||||
owner: <owner>
|
||||
repo: <repo>
|
||||
pull_number: <PR number>
|
||||
state: PENDING (accumulate comments before submitting)
|
||||
body: "" (empty for now, summary added on submit)
|
||||
comments: [
|
||||
{
|
||||
path: "<file path>",
|
||||
new_line_num: <line number>,
|
||||
body: "**[<category>] [<severity>]** <description>\n\n**Suggested fix:**\n```<lang>\n<fixed code>\n```"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Comment format:
|
||||
```
|
||||
**[Security] [error]** `shell=True` used with user input — command injection risk.
|
||||
|
||||
**Suggested fix:**
|
||||
```python
|
||||
subprocess.run(["git", "log", commit], check=True)
|
||||
```
|
||||
```
|
||||
|
||||
Severity levels:
|
||||
- `error` — must fix before merge (security, correctness, crash)
|
||||
- `warning` — should fix before merge (reliability, best practice)
|
||||
- `info` — consider fixing (style, minor improvement)
|
||||
|
||||
### Step 4: Auto-Fix Issues
|
||||
|
||||
For each issue that can be safely auto-fixed:
|
||||
|
||||
1. Edit the file using the `edit` tool
|
||||
2. Commit with message: `fix: address review comment — <short description>`
|
||||
3. Push to the PR's head branch: `git push origin HEAD`
|
||||
4. Wait for CI to re-run on the push
|
||||
|
||||
Auto-fix ALL issues unless:
|
||||
- The fix requires an architectural decision (ask the user)
|
||||
- The fix changes public API behavior (ask the user)
|
||||
- The fix is ambiguous (multiple valid approaches, ask the user)
|
||||
|
||||
### Step 5: Resolve Discussion Threads
|
||||
|
||||
After auto-fixing an issue and CI passes:
|
||||
|
||||
1. Find the review comment thread for that issue
|
||||
2. Post a reply: `Fixed in <commit-sha>. Closing this thread.`
|
||||
3. Resolve the discussion (if Gitea supports it via API)
|
||||
4. If resolving via API is not available, the reply comment serves as resolution
|
||||
|
||||
### Step 6: Submit Final Review
|
||||
|
||||
After all issues are addressed (fixed or discussed):
|
||||
|
||||
```
|
||||
mcp_call_tool: gitea / pull_request_review_write
|
||||
method: submit
|
||||
owner: <owner>
|
||||
repo: <repo>
|
||||
pull_number: <PR number>
|
||||
review_id: <from step 3 create>
|
||||
state: COMMENT (or APPROVED if no blocking issues remain)
|
||||
body: <summary — see below>
|
||||
```
|
||||
|
||||
### Step 7: Post Summary
|
||||
|
||||
Post a brief summary as a PR comment (via `issue_write / add_comment`):
|
||||
|
||||
```
|
||||
## Deep Review Summary
|
||||
|
||||
- **Files reviewed:** N
|
||||
- **Issues found:** N (N auto-fixed, N require attention)
|
||||
- **Categories:** security (N), correctness (N), architecture (N), ...
|
||||
|
||||
**Outcome:** ✅ Ready to merge — all issues addressed.
|
||||
**OR**
|
||||
**Outcome:** ⚠️ N blocking issue(s) remain — see inline comments.
|
||||
```
|
||||
|
||||
Keep the summary to 5-10 bullet points. Do not paste the full review.
|
||||
|
||||
### Step 8: Mark PR Ready
|
||||
|
||||
If all issues are addressed and no blocking issues remain:
|
||||
|
||||
```
|
||||
mcp_call_tool: gitea / issue_write
|
||||
method: add_labels
|
||||
owner: <owner>
|
||||
repo: <repo>
|
||||
issue_number: <PR number>
|
||||
labels: [<label_id for "ready-to-merge">]
|
||||
```
|
||||
|
||||
If blocking issues remain, do NOT add the label. Post a comment
|
||||
explaining what needs to be resolved before the PR can merge.
|
||||
|
||||
## Gitea MCP Tools Reference
|
||||
|
||||
| Action | MCP tool | Method |
|
||||
|--------|----------|--------|
|
||||
| Get PR details | `pull_request_read` | `get_pr` |
|
||||
| List PR files | `pull_request_read` | `list_pr_files` |
|
||||
| Get PR diff | `pull_request_read` | `get_pr_diff` |
|
||||
| Create review (pending) | `pull_request_review_write` | `create` (state: PENDING) |
|
||||
| Submit review | `pull_request_review_write` | `submit` (state: APPROVED/COMMENT/REQUEST_CHANGES) |
|
||||
| Post PR comment | `issue_write` | `add_comment` |
|
||||
| Add label | `issue_write` | `add_labels` |
|
||||
| List labels | `label_read` | `list_repo_labels` |
|
||||
| Merge PR | `pull_request_write` | `merge` (do NOT use — auto-merge handles this) |
|
||||
|
||||
## Important Rules
|
||||
|
||||
- **Never merge the PR yourself.** Add the `ready-to-merge` label and let
|
||||
the auto-merge workflow handle it. This ensures CI passes and the
|
||||
commit message follows the `<PREFIX>-N: <conventional>` format.
|
||||
- **Never approve your own PR.** If the agent created the PR, post
|
||||
COMMENT state, not APPROVED.
|
||||
- **Always push fixes to the PR branch**, not directly to master.
|
||||
- **Wait for CI after each push** before resolving the discussion thread.
|
||||
- **Post one review with all comments**, not multiple reviews.
|
||||
- **The summary must be brief** — 5-10 bullet points max.
|
||||
- **Severity matters**: only `error` severity blocks the `ready-to-merge` label.
|
||||
@@ -1,142 +0,0 @@
|
||||
# skill-creation
|
||||
|
||||
How to create, validate, and maintain Devin skills. Skills must be
|
||||
clear, succinct, and actionable — no AI slop.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when:
|
||||
- Creating a new skill
|
||||
- Amending an existing skill
|
||||
- Evaluating whether a skill is needed
|
||||
- Reviewing a PR that adds or modifies skills
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Skill directory: `.devin/skills/<skill-name>/SKILL.md`
|
||||
- Validator: `tests/test_skills.py` (infra) or reference to it
|
||||
- Tests: `tests/unit/test_skills_validation.py` (infra)
|
||||
|
||||
## When to Create a Skill
|
||||
|
||||
Create a skill when:
|
||||
- An agent struggles with a task repeatedly (branch hygiene, PR order)
|
||||
- A workflow has non-obvious ordering constraints (deployment coordination)
|
||||
- A task requires specific tool usage over raw commands (CI monitoring)
|
||||
- Multiple agents need shared context (dependency graph)
|
||||
|
||||
Do NOT create a skill for:
|
||||
- One-off tasks (use a spec instead)
|
||||
- Tasks already covered by AGENTS.md
|
||||
- Tasks that are obvious from the Makefile or README
|
||||
- Tasks that change frequently (skills should be stable)
|
||||
|
||||
## Skill Structure
|
||||
|
||||
Every skill MUST have:
|
||||
|
||||
```markdown
|
||||
# <skill-name>
|
||||
|
||||
One-line description of what the skill does.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
2-4 bullet points describing when to use this skill.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
What must exist before using the skill (venv, .env, tools).
|
||||
|
||||
## <Core Content>
|
||||
|
||||
The actual guidance. Keep it actionable.
|
||||
|
||||
## Verification (if applicable)
|
||||
|
||||
How to verify the skill's guidance works.
|
||||
|
||||
## Common Mistakes (if applicable)
|
||||
|
||||
What agents get wrong without this skill.
|
||||
```
|
||||
|
||||
## Quality Standards
|
||||
|
||||
### Do
|
||||
- **Be specific.** Reference exact make targets, file paths, commands.
|
||||
- **Be concise.** Each section should be scannable in under 30 seconds.
|
||||
- **Be actionable.** Every paragraph should tell the agent what to DO.
|
||||
- **Use tables** for command reference, mappings, and comparisons.
|
||||
- **Use code blocks** for commands the agent should run.
|
||||
- **Link to other skills** when related (e.g., "See `dependency-graph` skill").
|
||||
|
||||
### Don't
|
||||
- **No preamble.** Don't start with "This skill helps agents..." — just state what it does.
|
||||
- **No filler.** Don't repeat information from AGENTS.md or other skills.
|
||||
- **No vague advice.** "Be careful with branches" is useless. "Run `git branch --show-current` before every commit" is useful.
|
||||
- **No AI slop.** Don't write "In this comprehensive guide, we will explore..." — just give the guidance.
|
||||
- **No redundant sections.** If "Common Mistakes" would repeat "When to Invoke", skip it.
|
||||
- **No marketing.** Don't describe the skill as "powerful" or "comprehensive".
|
||||
|
||||
## Scope Rules
|
||||
|
||||
- **One skill per concern.** Don't mix branch hygiene with CI monitoring.
|
||||
- **Project-specific, not generic.** Skills reference this repo's make targets, file paths, and conventions — not abstract advice.
|
||||
- **Shared skills must be identical across repos.** Use `SHARED_SKILLS` in the validator to enforce this.
|
||||
- **Per-repo skills must reflect that repo's reality.** Don't copy infra-specific targets to sso-bridge.
|
||||
|
||||
## Automated Validation
|
||||
|
||||
Every skill must pass the validator (`tests/test_skills.py`). The validator checks:
|
||||
|
||||
1. **Structure** — H1 title, "When to Invoke" section, "Prerequisites" section
|
||||
2. **Commands** — referenced `make <target>` commands exist in Makefile or devx.mak
|
||||
3. **Paths** — referenced file paths exist in the repo
|
||||
4. **Shared skills** — identical content across repos (SHA-256 comparison)
|
||||
5. **No drift** — no references to nonexistent commands or files
|
||||
|
||||
Run the validator:
|
||||
```bash
|
||||
python3 tests/test_skills.py --repo infra --repo sso-bridge
|
||||
```
|
||||
|
||||
## Effectiveness Evaluation
|
||||
|
||||
### Static Checks (automated, CI)
|
||||
|
||||
The validator runs in CI as part of `make pytest-cov`. A failing skill
|
||||
test blocks the PR. This catches:
|
||||
- Missing sections
|
||||
- Invalid commands
|
||||
- Broken file references
|
||||
- Cross-repo drift
|
||||
|
||||
### Runtime Metrics (manual, periodic)
|
||||
|
||||
Track these signals to evaluate skill effectiveness:
|
||||
- **Skill invocation frequency** — how often agents invoke the skill
|
||||
- **Success rate when invoked** — did the skill prevent the mistake it targets?
|
||||
- **Feedback issues** — agents create Gitea issues with `feedback` label when a skill is unclear or wrong
|
||||
- **Mistake recurrence** — if agents still make the mistake the skill targets, the skill needs improvement
|
||||
|
||||
### Retrospective Review
|
||||
|
||||
Periodically (monthly or after major incidents) review skills:
|
||||
1. List all skills and their last-modified dates
|
||||
2. Check for feedback issues tagged `skill-improvement`
|
||||
3. Verify referenced commands still exist (run validator)
|
||||
4. Remove skills that are no longer relevant
|
||||
5. Update skills where mistakes still recur
|
||||
6. Document lessons in this skill's "Common Mistakes" section
|
||||
|
||||
## Creating a New Skill — Checklist
|
||||
|
||||
- [ ] Identify the repeated struggle or non-obvious workflow
|
||||
- [ ] Check no existing skill covers it
|
||||
- [ ] Write the skill following the structure above
|
||||
- [ ] Run `python3 tests/test_skills.py` — must pass
|
||||
- [ ] Run `make pytest-cov` — must pass with 100% coverage
|
||||
- [ ] If shared across repos, copy identical content to each repo
|
||||
- [ ] Add the skill to `SHARED_SKILLS` in the validator if shared
|
||||
- [ ] Create PR, verify CI passes, merge
|
||||
@@ -1,139 +0,0 @@
|
||||
# Spec-Driven Development
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when starting any change — every PR requires a spec
|
||||
at `docs/specs/<TASK-ID>.md` that CI validates before merge.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Vikunja task ID (`DEVX-N`) — see `vikunja-tasks` skill
|
||||
|
||||
## Overview
|
||||
|
||||
Every change starts with a spec. No spec, no code. No code, no PR.
|
||||
|
||||
The spec is a markdown file at `docs/specs/<TASK-ID>.md` in the repo.
|
||||
It contains structured requirements (REQ-IDs) and acceptance criteria
|
||||
(AC checklist) that CI validates before merge.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Create Vikunja task** — `make create-task -- --title "Title" --description "..."`
|
||||
2. **Write spec** — Create `docs/specs/<TASK-ID>.md` (see template below)
|
||||
3. **Create branch** — `git checkout -b <PREFIX>-N-short-description`
|
||||
4. **Implement** — Write code with `# Implements: REQ-N` comments
|
||||
5. **Check ACs** — Tick all acceptance criteria checkboxes in the spec
|
||||
6. **Push and create PR** — `make push-with-pr`
|
||||
7. **CI validates** — Spec validation, PR size check, fast molecule, lint, tests
|
||||
8. **Auto-merge** — Add `ready-to-merge` label after review
|
||||
9. **Auto-deploy** — Post-merge deploys to staging (if nightly gate is green)
|
||||
|
||||
## Spec Template
|
||||
|
||||
```markdown
|
||||
# <TASK-ID>: <Title>
|
||||
|
||||
## Problem
|
||||
<What is broken or missing? Why does this change exist?>
|
||||
|
||||
## Approach
|
||||
<How will you solve it? What are the key design decisions?>
|
||||
|
||||
REQ-1: <First requirement description>
|
||||
REQ-2: <Second requirement description>
|
||||
REQ-3: <Third requirement description>
|
||||
|
||||
## Test Plan
|
||||
- <How will you verify each REQ is implemented correctly?>
|
||||
- <Include unit tests, molecule scenarios, integration tests>
|
||||
|
||||
## Deploy Plan
|
||||
- <How will this change be deployed?>
|
||||
- <What order do components need to deploy in?>
|
||||
- <Are there migrations or one-time operations?>
|
||||
|
||||
## Rollback Plan
|
||||
- <How do you revert if something goes wrong?>
|
||||
- <What data/state changes are irreversible?>
|
||||
|
||||
## Acceptance Criteria
|
||||
- [ ] REQ-1: <criterion that proves REQ-1 is done>
|
||||
- [ ] REQ-2: <criterion that proves REQ-2 is done>
|
||||
- [ ] REQ-3: <criterion that proves REQ-3 is done>
|
||||
```
|
||||
|
||||
## CI Validation
|
||||
|
||||
The `devx.ci.validate_spec` module checks:
|
||||
|
||||
1. **Spec file exists** at `docs/specs/<TASK-ID>.md` (TASK-ID from branch name)
|
||||
2. **Required sections present**: Problem, Approach, Test Plan, Deploy Plan, Rollback Plan, Acceptance Criteria
|
||||
3. **At least one REQ-ID** line (format: `REQ-N: <description>`)
|
||||
4. **All AC checkboxes checked** (`- [x]`, not `- [ ]`)
|
||||
|
||||
If any check fails, CI blocks the PR before expensive jobs run.
|
||||
|
||||
## PR Size Limits
|
||||
|
||||
CI enforces max 500 lines / 10 files changed (excluding CHANGELOG.md,
|
||||
README.md, badges, lock files). Oversized PRs are rejected. Split your
|
||||
work into smaller PRs.
|
||||
|
||||
## Code-to-Spec Linking
|
||||
|
||||
Each function, task, or template that implements a requirement should
|
||||
have a comment:
|
||||
|
||||
```python
|
||||
# Implements: REQ-1
|
||||
def install_sso_bridge():
|
||||
...
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Implements: REQ-2
|
||||
- name: Clone infra repo
|
||||
git:
|
||||
...
|
||||
```
|
||||
|
||||
## Fast Molecule (Pre-merge)
|
||||
|
||||
CI runs molecule only for **changed roles** (detected via git diff),
|
||||
with converge + verify only, single platform. This gives quick feedback
|
||||
(~5-10 min) without the full molecule suite.
|
||||
|
||||
## Full Molecule (Nightly)
|
||||
|
||||
The complete molecule suite (all scenarios, all platforms) runs nightly
|
||||
at 02:00 CET on master. If it fails:
|
||||
- A Gitea issue is created with the `feedback` label
|
||||
- The `NIGHTLY_STATUS` repo variable is set to `failed:<run_id>`
|
||||
- All staging deploys are blocked until nightly passes again
|
||||
|
||||
## Auto-Deploy on Merge
|
||||
|
||||
Every merged PR auto-deploys to staging (if nightly gate is green).
|
||||
No manual trigger needed. The deploy runs the full pipeline:
|
||||
provision → deploy-observability → deploy-customer → configure-oidc.
|
||||
|
||||
For grm/sso-bridge: post-merge publishes the package, then auto-creates
|
||||
an infra PR to bump the pinned version. That infra PR auto-deploys when
|
||||
merged.
|
||||
|
||||
## Key Commands
|
||||
|
||||
```bash
|
||||
# Validate spec locally (before pushing)
|
||||
python -m devx.ci.validate_spec --branch <PREFIX>-N-description
|
||||
|
||||
# Check PR size locally
|
||||
python -m devx.ci.check_pr_size --base origin/master --head HEAD
|
||||
|
||||
# See which roles need fast molecule
|
||||
python -m devx.ci.fast_molecule --base origin/master --head HEAD
|
||||
|
||||
# Check nightly gate status
|
||||
python -m devx.ci.nightly_gate --repo oblachno/infra --action check
|
||||
```
|
||||
@@ -3,17 +3,6 @@
|
||||
Make targets for testing, debugging, and CI investigation. **Use these
|
||||
instead of raw `pytest`, `ruff`, or `actionlint` commands.**
|
||||
|
||||
## When to Invoke
|
||||
|
||||
Invoke this skill when running tests, investigating CI failures, or
|
||||
linting before push. Also invoke when asked to "run tests", "check
|
||||
coverage", or "debug a failure".
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `.venv` exists (run `make setup` if not)
|
||||
- Tools installed (run `make install-tools` for actionlint/act_runner)
|
||||
|
||||
## Why Make Targets
|
||||
|
||||
Make targets encapsulate the correct venv activation, PYTHONPATH, env
|
||||
@@ -56,13 +45,6 @@ This runs `lint-all` + `pytest-cov`. The pre-push git hook only
|
||||
validates the Vikunja task exists — it does NOT run tests. You must
|
||||
run `make pre-push` manually.
|
||||
|
||||
### Spec-Driven Workflow
|
||||
|
||||
Every PR requires a spec file at `docs/specs/<TASK-ID>.md`. See the
|
||||
`spec-driven-development` skill for the full workflow and template.
|
||||
CI validates the spec (via `devx.ci.validate_spec`) and checks PR size
|
||||
(via `devx.ci.check_pr_size`) before running expensive jobs.
|
||||
|
||||
## CI Failure Investigation
|
||||
|
||||
When investigating a CI failure:
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
# vikunja-tasks
|
||||
|
||||
Vikunja task lifecycle beyond `create`: querying status, closing, and
|
||||
recovering when the tracker is unreachable.
|
||||
|
||||
## When to Invoke
|
||||
|
||||
- Creating, closing, or checking a Vikunja task
|
||||
- A spec workflow step needs the task ID or done state
|
||||
- `vikunja.oblachno.oblachno.fyi` fails to resolve / times out
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `.env` with `VIKUNJA_TOKEN`
|
||||
- Project ID comes from `[tool.devx]` in `pyproject.toml`
|
||||
(`DEVX_VIKUNJA_PROJECT_ID`)
|
||||
|
||||
## Create
|
||||
|
||||
`make create-task` does **not** forward arguments — call the module:
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m devx.tools.create_task \
|
||||
--title "Task title (no DEVX-N prefix)" \
|
||||
--description "<h2>Context</h2><p>...</p>"
|
||||
```
|
||||
|
||||
Prints `DEVX-N` + next steps. Title must not include the task-ID
|
||||
prefix (auto-merge prepends it; a manual prefix double-prefixes the
|
||||
PR title and fails validation).
|
||||
|
||||
## Query / Close
|
||||
|
||||
```bash
|
||||
# Task details (ID = numeric part of DEVX-N)
|
||||
curl -sf -H "Authorization: Bearer $VIKUNJA_TOKEN" \
|
||||
"https://vikunja.oblachno.oblachno.fyi/api/v1/tasks/<N>"
|
||||
|
||||
# Close: mark done
|
||||
curl -sf -X POST -H "Authorization: Bearer $VIKUNJA_TOKEN" \
|
||||
-H "Content-Type: application/json" -d '{"done":true}' \
|
||||
"https://vikunja.oblachno.oblachno.fyi/api/v1/tasks/<N>"
|
||||
```
|
||||
|
||||
Post-merge automation marks the task done when the PR squash-merges —
|
||||
manual close is only needed for abandoned/superseded tasks.
|
||||
|
||||
## Task-ID / Spec Collisions
|
||||
|
||||
Vikunja IDs can collide with historical spec files (an old task reused
|
||||
the number). Convention: preserve the old file as
|
||||
`docs/specs/<ID>-<topic>-historical.md`, then write the new spec at
|
||||
`docs/specs/<ID>.md`. Check `git log` on the existing spec before
|
||||
moving it.
|
||||
|
||||
## Tracker Unreachable
|
||||
|
||||
If the Vikunja host fails DNS/TLS:
|
||||
|
||||
1. Don't block the whole workflow — record the intended task title in
|
||||
the spec draft and retry `create_task` before branching.
|
||||
2. Never invent an ID — branch/PR titles must match a real task or
|
||||
`pre_push_check` / auto-merge validation fails.
|
||||
3. DNS failures observed so far were transient; retry after a few
|
||||
minutes before escalating.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- `make create-task -- --title ...` — args are dropped; use the module
|
||||
call above (forwarding fix is S11 scope).
|
||||
- Including `DEVX-N:` in the task title — double prefix breaks
|
||||
auto-merge.
|
||||
- Closing a task whose PR is still open — auto-merge's post-merge
|
||||
step handles the close; manual close confuses the audit trail.
|
||||
@@ -32,11 +32,7 @@ concurrency:
|
||||
jobs:
|
||||
build-and-push:
|
||||
runs-on: docker
|
||||
container:
|
||||
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
credentials:
|
||||
username: ${{ vars.CI_GITEA_USERNAME }}
|
||||
password: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
timeout-minutes: 30
|
||||
outputs:
|
||||
is-release: ${{ steps.check.outputs.is-release }}
|
||||
@@ -77,9 +73,6 @@ jobs:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
CI_GITEA_USERNAME: ${{ vars.CI_GITEA_USERNAME }}
|
||||
PYTHONPATH: src
|
||||
# Serialize blob uploads to avoid Gitea registry race condition
|
||||
# (BlobUploader.Append offset mismatch — see DEVX-162).
|
||||
DOCKER_MAX_CONCURRENT_UPLOADS: "1"
|
||||
run: |
|
||||
. .venv/bin/activate
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
@@ -123,11 +116,7 @@ jobs:
|
||||
needs: [build-and-push]
|
||||
if: always() && needs.build-and-push.result == 'success'
|
||||
runs-on: docker
|
||||
container:
|
||||
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
credentials:
|
||||
username: ${{ vars.CI_GITEA_USERNAME }}
|
||||
password: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
+28
-77
@@ -18,11 +18,7 @@ jobs:
|
||||
# Saves ~4x checkout+setup overhead vs 5 separate jobs.
|
||||
validate:
|
||||
runs-on: docker
|
||||
container:
|
||||
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
credentials:
|
||||
username: ${{ vars.CI_GITEA_USERNAME }}
|
||||
password: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
timeout-minutes: 15
|
||||
defaults:
|
||||
run:
|
||||
@@ -33,40 +29,21 @@ jobs:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
# Implements: REQ-1 (DEVX-176) — docs-only changes skip the heavy
|
||||
# quality steps. Detection needs only git, so it runs before setup.
|
||||
- name: Detect docs-only change
|
||||
id: docs-only
|
||||
if: github.event_name == 'pull_request'
|
||||
run: |
|
||||
HEAD="${{ github.event.pull_request.head.sha || github.sha }}"
|
||||
DOCS_ONLY=true
|
||||
while IFS= read -r f; do
|
||||
case "$f" in
|
||||
docs/*|*.md|.devin/*) ;;
|
||||
*) DOCS_ONLY=false; break;;
|
||||
esac
|
||||
done < <(git diff --name-only "origin/master...$HEAD")
|
||||
echo "docs-only=$DOCS_ONLY" >> "$GITHUB_OUTPUT"
|
||||
echo "docs-only=$DOCS_ONLY"
|
||||
- name: Set up environment
|
||||
env:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
run: make setup-image
|
||||
# --- quality steps ---
|
||||
- name: Lint all
|
||||
if: steps.docs-only.outputs.docs-only != 'true'
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
make lint-all
|
||||
- name: Unit tests with 100% coverage
|
||||
if: steps.docs-only.outputs.docs-only != 'true'
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
make pytest-cov
|
||||
- name: Check unit test speed
|
||||
if: steps.docs-only.outputs.docs-only != 'true'
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.tools.check_test_speed --max-seconds 15 --max-single-seconds 0.5
|
||||
@@ -79,12 +56,10 @@ jobs:
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
make devx-docs-check
|
||||
- name: Translation completeness check
|
||||
if: steps.docs-only.outputs.docs-only != 'true'
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.ci.check_translations
|
||||
- name: Dependency security scan
|
||||
if: steps.docs-only.outputs.docs-only != 'true'
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
# Install pip in venv if missing (needed by pip-audit)
|
||||
@@ -92,7 +67,6 @@ jobs:
|
||||
PIPAPI_PYTHON_LOCATION=$PWD/.venv/bin/python \
|
||||
pip-audit --desc --skip-editable 2>&1 || true
|
||||
- name: Workflow dry-run validation
|
||||
if: steps.docs-only.outputs.docs-only != 'true'
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
@@ -128,30 +102,14 @@ jobs:
|
||||
--pr-title "$PR_TITLE" \
|
||||
--repo "$REPOSITORY" \
|
||||
--pr-number "$PR_NUMBER"
|
||||
- name: Validate spec file
|
||||
- name: Run automated PR review
|
||||
if: github.event_name == 'pull_request'
|
||||
env:
|
||||
DEVX_TASK_PREFIX: DEVX
|
||||
PYTHONPATH: ${{ env.PYTHONPATH }}
|
||||
HEAD_REF: ${{ github.head_ref }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.ci.validate_spec \
|
||||
--branch "$HEAD_REF" \
|
||||
--github-output
|
||||
- name: Check PR size
|
||||
if: github.event_name == 'pull_request'
|
||||
env:
|
||||
PYTHONPATH: ${{ env.PYTHONPATH }}
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
python3 -m devx.ci.check_pr_size \
|
||||
--base "origin/master" \
|
||||
--head "${{ github.event.pull_request.head.sha || github.sha }}" \
|
||||
--repo "${{ github.repository }}" \
|
||||
--pr-number "${{ github.event.number }}" \
|
||||
--github-output
|
||||
set -euo pipefail
|
||||
python3 -m devx.ci.pr_review \
|
||||
"${{ github.event.number }}" \
|
||||
"${{ github.repository }}"
|
||||
# --- release-dry-run step (conditional) ---
|
||||
- name: Release dry-run validation
|
||||
if: steps.detect.outputs.user-facing-changed == 'true'
|
||||
@@ -159,9 +117,19 @@ jobs:
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
python3 -m devx.ci.release --dry-run
|
||||
# Implements: REQ-2 (DEVX-176) — no failure-issue step in PR CI;
|
||||
# auto-created issues are for deploy-pipeline failures only
|
||||
# (post-merge keeps its notification).
|
||||
- name: Notify on failure
|
||||
if: failure()
|
||||
env:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
python3 -m devx.ci.notify_failure \
|
||||
--repo "${{ github.repository }}" \
|
||||
--run-id "${{ github.run_id }}" \
|
||||
--workflow "ci/validate" \
|
||||
--commit "${{ github.sha }}" \
|
||||
--auto-login
|
||||
|
||||
auto-merge:
|
||||
# Auto-merge runs after validate passes. It reads the task ID
|
||||
@@ -172,11 +140,7 @@ jobs:
|
||||
github.event_name == 'pull_request' &&
|
||||
needs.validate.result == 'success'
|
||||
runs-on: docker
|
||||
container:
|
||||
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
credentials:
|
||||
username: ${{ vars.CI_GITEA_USERNAME }}
|
||||
password: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
timeout-minutes: 10
|
||||
defaults:
|
||||
run:
|
||||
@@ -195,29 +159,16 @@ jobs:
|
||||
REVIEWER_GITEA_API_TOKEN: ${{ secrets.REVIEWER_GITEA_API_TOKEN }}
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
PR_NUMBER: ${{ github.event.number }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
run: |
|
||||
. .venv/bin/activate 2>/dev/null || true
|
||||
# Post APPROVE review via Gitea API to satisfy branch protection.
|
||||
# Try REVIEWER_GITEA_API_TOKEN first; fall back to CI_GITEA_API_TOKEN
|
||||
# (CI bot account) if the reviewer token is the same user as the PR
|
||||
# creator (Gitea rejects self-approvals).
|
||||
for TOKEN in "${REVIEWER_GITEA_API_TOKEN}" "${CI_GITEA_API_TOKEN}"; do
|
||||
[ -z "$TOKEN" ] && continue
|
||||
RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \
|
||||
"${GITHUB_SERVER_URL}/api/v1/repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/reviews" \
|
||||
-H "Authorization: token ${TOKEN}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"event":"APPROVED","body":"Auto-approved: all CI checks passed (validate job)."}')
|
||||
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
|
||||
BODY=$(echo "$RESPONSE" | head -n -1)
|
||||
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "201" ]; then
|
||||
echo "Approval posted successfully (HTTP $HTTP_CODE)."
|
||||
break
|
||||
fi
|
||||
echo "::warning::Approval with token failed (HTTP $HTTP_CODE): ${BODY}"
|
||||
done
|
||||
python3 -m devx.ci.pr_review \
|
||||
"$PR_NUMBER" \
|
||||
"$REPOSITORY" \
|
||||
--event APPROVE \
|
||||
--checklist-confirmed \
|
||||
--checklist-categories 1,2,3,4,5,6,7,8,9,10,11,12,13 \
|
||||
--body "Auto-approved: all CI checks passed (validate job)."
|
||||
- name: Squash merge with task ID
|
||||
env:
|
||||
CI_GITEA_API_TOKEN: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
|
||||
@@ -21,15 +21,10 @@ name: Post-merge
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
# Implements: REQ-1 — manual recovery when a [skip ci] squash title
|
||||
# suppresses the push-triggered post-merge run.
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: post-merge-${{ github.ref }}
|
||||
# Implements: REQ-3 (DEVX-176) — queue instead of killing an in-flight
|
||||
# release/publish; a cancelled release can leave tag-without-publish.
|
||||
cancel-in-progress: false
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
PIP_BREAK_SYSTEM_PACKAGES: "1"
|
||||
@@ -40,11 +35,7 @@ env:
|
||||
jobs:
|
||||
detect-and-configure:
|
||||
runs-on: docker
|
||||
container:
|
||||
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
credentials:
|
||||
username: ${{ vars.CI_GITEA_USERNAME }}
|
||||
password: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-base:latest
|
||||
timeout-minutes: 10
|
||||
defaults:
|
||||
run:
|
||||
@@ -108,11 +99,7 @@ jobs:
|
||||
needs: [detect-and-configure]
|
||||
if: always() && needs.detect-and-configure.result == 'success'
|
||||
runs-on: docker
|
||||
container:
|
||||
image: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
credentials:
|
||||
username: ${{ vars.CI_GITEA_USERNAME }}
|
||||
password: ${{ secrets.CI_GITEA_API_TOKEN }}
|
||||
container: git.oblachno.oblachno.fyi/oblachno-oss/runner-images/ci-full:latest
|
||||
timeout-minutes: 15
|
||||
outputs:
|
||||
tag: ${{ steps.release-tag.outputs.tag }}
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Joseph Kato
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -1,38 +0,0 @@
|
||||
# The words behind a SMOG or Gunning fog score: each one of three or more
|
||||
# syllables. The scores say a paragraph reads hard; this says which words
|
||||
# made it so. Syllables are counted by vowel groups, with the silent e and
|
||||
# the -ed and -es endings taken off, which agrees with the scores' count on
|
||||
# nearly every word and errs on the long side for the rest.
|
||||
#
|
||||
# A suggestion, since a paragraph can carry a few. Turn it on while bringing
|
||||
# a score down, and off again after. See issue 8.
|
||||
extends: script
|
||||
message: "'%s' has three or more syllables. It counts toward the SMOG and Gunning fog scores."
|
||||
link: https://en.wikipedia.org/wiki/SMOG
|
||||
level: suggestion
|
||||
scope: paragraph
|
||||
script: |
|
||||
text := import("text")
|
||||
|
||||
matches := []
|
||||
for m in text.re_find(`[A-Za-z]+(?:'[A-Za-z]+)?`, scope, -1) {
|
||||
w := text.to_lower(m[0].text)
|
||||
if len(w) < 5 {
|
||||
continue
|
||||
}
|
||||
groups := text.re_find(`[aeiouy]+`, w, -1)
|
||||
n := groups == undefined ? 0 : len(groups)
|
||||
if text.has_suffix(w, "e") && !text.has_suffix(w, "le") && !text.has_suffix(w, "ee") {
|
||||
n--
|
||||
}
|
||||
if text.has_suffix(w, "ed") && !text.re_match(`[td]ed$`, w) {
|
||||
n--
|
||||
}
|
||||
if text.has_suffix(w, "es") && !text.re_match(`[sxz]es$|[cs]hes$`, w) {
|
||||
n--
|
||||
}
|
||||
if n >= 3 {
|
||||
matches = append(matches, {begin: m[0].begin, end: m[0].end})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,4 @@
|
||||
{
|
||||
"feed": "https://github.com/vale-cli/readability/releases.atom",
|
||||
"vale_version": ">=2.13.0",
|
||||
"license_files": [
|
||||
"LICENSE"
|
||||
]
|
||||
}
|
||||
"feed": "https://github.com/errata-ai/Readability/releases.atom",
|
||||
"vale_version": ">=2.13.0"
|
||||
}
|
||||
@@ -83,6 +83,7 @@ src/devx/
|
||||
│ ├── classify_changes.py # User-facing vs infrastructure change detection
|
||||
│ ├── detect_release_commit.py # Detect release commits on master
|
||||
│ ├── validate_commit_msg.py # Conventional commit validation
|
||||
│ ├── pr_review.py # Automated PR review + manual reviews (--event, --body, --checklist-confirmed)
|
||||
│ ├── post_merge.py # Vikunja task updates after merge
|
||||
│ ├── sync_wiki.py # Sync documentation to Gitea wiki
|
||||
│ ├── push_badges.py # Generate and push quality badges (--retries for retry on git push failures)
|
||||
@@ -123,10 +124,6 @@ src/devx/
|
||||
│ ├── check_docker_init.py # Check Docker Compose services with healthchecks have init: true
|
||||
│ ├── check_ansible_set_fact_to_json.py # Check set_fact tasks don't misuse to_json
|
||||
│ ├── check_alert_rules.py # Validate Prometheus alert rules with promtool
|
||||
│ ├── check_ansible_no_log.py # Check Ansible tasks for missing no_log on secrets
|
||||
│ ├── check_ansible_patterns.py # Detect dangerous failure-masking patterns
|
||||
│ ├── check_jinja_expr.py # Validate Jinja2 expressions in Ansible files
|
||||
│ ├── check_ansible_no_state_absent_on_db.py # Prevent state:absent on DB paths
|
||||
│ └── _shared.py # Shared tool utilities
|
||||
├── opentofu.py # OpenTofu output helpers (get_tofu_output, get_tofu_vm_ip, get_tofu_vm_field)
|
||||
├── utils/ # Shared utilities (reusable across projects)
|
||||
@@ -144,8 +141,8 @@ src/devx/
|
||||
└── molecule/ # Optional molecule testing helpers (for Ansible projects)
|
||||
├── discover_runners.py # Dynamic Gitea runner discovery
|
||||
├── distribute_molecule.py # Distribute molecule scenarios across runners (LPT scheduling, --roles-root for multi-role)
|
||||
├── molecule_ci_guard.py # Run molecule with cross-runner fail-fast (--roles-root)
|
||||
├── molecule_all.py # Run all molecule scenarios locally
|
||||
├── molecule_changed.py # Detect which Ansible roles changed and output molecule scenarios
|
||||
├── start_docker.py # Ensure Docker daemon is running for molecule tests
|
||||
└── platforms.py # Supported molecule platforms
|
||||
```
|
||||
@@ -157,37 +154,8 @@ src/devx/
|
||||
- **PYTHONPATH: src** — Workflows set `PYTHONPATH: src` (NOT `.:src` since there are no scripts at repo root)
|
||||
- **Config via env vars** — `DEVX_*` environment variables with `.env` file fallback
|
||||
|
||||
|
||||
## Spec-Driven Development
|
||||
|
||||
Every change starts with a spec. No spec, no code.
|
||||
|
||||
**Workflow:**
|
||||
1. Create Vikunja task → get `<PREFIX>-N` task ID
|
||||
2. Write spec at `docs/specs/<TASK-ID>.md` (see template in `.devin/skills/spec-driven-development/SKILL.md`)
|
||||
3. Create branch, implement with `# Implements: REQ-N` comments
|
||||
4. Tick all acceptance criteria checkboxes in spec
|
||||
5. Push and create PR — CI validates spec before expensive jobs
|
||||
|
||||
**CI gates (pre-merge):**
|
||||
- `devx.ci.validate_spec` — checks spec exists, has required sections, REQ-IDs, all ACs checked
|
||||
- `devx.ci.check_pr_size` — max 500 lines / 10 files (excludes CHANGELOG, badges, locks)
|
||||
- `devx.ci.fast_molecule` — full `molecule test` for changed roles only (scoped, single platform)
|
||||
|
||||
**Nightly (infra only):**
|
||||
- Full molecule suite (all scenarios, all platforms) + staging deploy + integration tests
|
||||
- On failure: sets `NIGHTLY_STATUS=failed`, blocks staging deploys
|
||||
- Post-merge auto-deploy to staging checks this gate before deploying
|
||||
|
||||
**Post-merge:**
|
||||
- Infra: auto-deploys to staging (if nightly gate is green)
|
||||
- GRM/sso-bridge: auto-publishes package, auto-creates infra dependency PR to bump pinned version
|
||||
|
||||
**Skill:** `.devin/skills/spec-driven-development/SKILL.md` — full template and workflow details.
|
||||
|
||||
## PR Workflow (Mandatory)
|
||||
|
||||
|
||||
Every change to master goes through this workflow. No exceptions.
|
||||
|
||||
### Branch Protection (Required Gitea Settings)
|
||||
@@ -238,7 +206,7 @@ docs: update README
|
||||
### 6. Review the PR
|
||||
|
||||
**Automated review (CI `validate` job):** Every PR triggers an automated
|
||||
review via the `pr-review` skill (agent-invoked, not a CI step).
|
||||
review via `python -m devx.ci.pr_review` as a step in the `validate` job.
|
||||
This posts a review with
|
||||
`COMMENT` (no issues) or `REQUEST_CHANGES` (issues found):
|
||||
|
||||
|
||||
+41
-242
@@ -2,262 +2,61 @@
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [0.56.0] - 2026-09-30
|
||||
## [0.49.3] - 2026-08-07
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Retry ansible-galaxy collection install on transient timeouts
|
||||
|
||||
## [0.49.2] - 2026-08-07
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Add fallback URL for tea download
|
||||
|
||||
## [0.49.1] - 2026-08-07
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Add container images to build-images workflow
|
||||
|
||||
## [0.49.0] - 2026-08-07
|
||||
|
||||
### Features
|
||||
|
||||
- *(molecule)* Log lease owner and UTC expiry on emission
|
||||
|
||||
## [0.55.5] - 2026-09-28
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Strip [skip ci] tokens and prefer newest commit in squash titles
|
||||
|
||||
## [0.55.4] - 2026-09-23
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fail-open molecule selection and honest fast-path contract
|
||||
|
||||
## [0.55.3] - 2026-09-21
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Honor repo bandit config in badge quality check
|
||||
|
||||
## [0.55.2] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(ci)* Supply git identity for dep-PR manifest commits
|
||||
|
||||
## [0.55.1] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(ci)* Resolve dep-PR container digest via registry v2 API
|
||||
|
||||
## [0.55.0] - 2026-09-19
|
||||
- Add --include-roles and --exclude-roles to distribute_molecule
|
||||
## [0.48.0] - 2026-07-22
|
||||
|
||||
### Features
|
||||
|
||||
- *(ci)* Retry dep-PR container verification until publish lands
|
||||
- Extract reusable components from infra and grm into devx
|
||||
|
||||
## [0.54.0] - 2026-09-19
|
||||
## [0.48.0] - 2026-07-22
|
||||
|
||||
### Features
|
||||
|
||||
- *(ci)* Dep PRs carry target-repo task ID and generated spec
|
||||
- Extract reusable components from infra and grm into devx
|
||||
|
||||
## [0.53.3] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(ci)* Check out API-created dep branch via remote-tracking ref
|
||||
|
||||
## [0.53.2] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(ci)* Create dep-PR branches via POST /branches
|
||||
|
||||
## [0.53.1] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(ci)* Handle list-shaped ref response in create_dependency_pr
|
||||
|
||||
## [0.53.0] - 2026-09-19
|
||||
## [Unreleased]
|
||||
|
||||
### Features
|
||||
|
||||
- *(ci)* Manifest-aware dependency PRs and cleanup protection
|
||||
|
||||
## [0.52.0] - 2026-09-19
|
||||
|
||||
### Features
|
||||
|
||||
- *(ci)* Verify producer container artifact before opening dependency PR
|
||||
|
||||
## [0.51.12] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Create_dependency_pr clones target repo instead of editing producer checkout
|
||||
|
||||
## [0.51.11] - 2026-09-19
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(ci)* Retry Vikunja lookups and surface self-approval merge failures
|
||||
|
||||
## [0.51.10] - 2026-09-17
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Read Gitea repo-variable data field; fail closed on unknown nightly status
|
||||
|
||||
## [0.51.9] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Exclude docs/plans/* from PR size check
|
||||
|
||||
## [0.51.8] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Accept deps: as valid conventional commit type
|
||||
|
||||
## [0.51.7] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Increase HTTP 500 retry count to 5 with longer backoff and visible logging
|
||||
|
||||
## [0.51.6] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Use stderr=STDOUT to capture all docker push output in one stream
|
||||
|
||||
## [0.51.5] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Check stdout for HTTP 500 in _run_push (docker sends to stdout)
|
||||
|
||||
## [0.51.4] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Serialize registry uploads and retry on HTTP 500
|
||||
|
||||
### Refactor
|
||||
|
||||
- Remove cross-repo contract tests from devx
|
||||
|
||||
## [0.51.3] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fall back to CI bot token for auto-merge approval
|
||||
|
||||
## [0.51.2] - 2026-08-26
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Push-first strategy in build_image to avoid losing latest tag
|
||||
|
||||
## [0.51.1] - 2026-08-25
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Delete existing manifest before push (Gitea #31964 workaround)
|
||||
|
||||
## [0.51.0] - 2026-08-25
|
||||
|
||||
### Features
|
||||
|
||||
- Add role defaults path to create_dependency_pr search
|
||||
|
||||
## [0.50.2] - 2026-08-24
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Update check_pr_size usage example with --repo and --pr-number args
|
||||
|
||||
## [0.50.1] - 2026-08-15
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Prefer rootless Docker socket over low-space inner DinD daemon
|
||||
- Check /run/host-docker.sock for host Docker daemon
|
||||
- Use /dev/shm/docker as data-root for inner dockerd
|
||||
- Kill existing dockerd before starting /dev/shm/docker daemon
|
||||
- Use /dev/shm/docker.sock socket for local dockerd
|
||||
- Add --iptables=false to local dockerd
|
||||
- Disable bridge and ip6tables for local dockerd
|
||||
- Use host Docker when root dir is inaccessible (free=0)
|
||||
- Prefer /run/host-docker.sock over inner dockerd
|
||||
- Start local dockerd instead of using low-space inner dockerd
|
||||
- Use container overlay for local dockerd data root
|
||||
- Kill inner dockerd with SIGKILL, use alt socket if alive
|
||||
- Kill dockerd by PID when pkill fails, use /dev/shm for alive daemon
|
||||
- Trust /var/run/docker.sock with free=0 when no inner dockerd exists
|
||||
|
||||
## [0.50.0] - 2026-08-09
|
||||
|
||||
### Features
|
||||
|
||||
- Add 5 standalone lint scripts from infra
|
||||
|
||||
## [0.49.0] - 2026-08-09
|
||||
|
||||
### Features
|
||||
|
||||
- Sync missing features from v0.49.x line to master
|
||||
|
||||
## [0.48.2] - 2026-08-09
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Remove dead translation keys and add missing one
|
||||
|
||||
## [0.48.1] - 2026-08-08
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- *(setup)* Extract version from filename for mirror installs
|
||||
|
||||
## [0.48.0] - 2026-08-08
|
||||
|
||||
### Features
|
||||
|
||||
- *(setup)* Mirror Ansible collections from Gitea registry with auth
|
||||
|
||||
## [0.47.10] - 2026-08-05
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Unique molecule container names per CI runner
|
||||
|
||||
## [0.47.9] - 2026-08-03
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Unique molecule container names per CI runner
|
||||
|
||||
## [0.47.8] - 2026-08-03
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Increase CI_SCALE_FACTOR default from 4 to 6
|
||||
|
||||
## [0.47.7] - 2026-08-03
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Scale check_test_speed limits on CI runners
|
||||
|
||||
## [0.47.6] - 2026-08-03
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Configure git auth in setup_image for git+https deps
|
||||
|
||||
## [0.47.5] - 2026-08-03
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Push wiki to main branch instead of master
|
||||
|
||||
## [0.47.4] - 2026-08-03
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Add User-Agent header to _download in install_tools
|
||||
- Extract reusable components from infra and grm into devx:
|
||||
- `devx.utils.ui.say()` — unified click.echo + logging output
|
||||
- `devx.utils.api.APIClient` — base HTTP API client class with retry logic
|
||||
- `devx.utils.jinja` — Jinja2 environment helpers with Ansible-compatible filters
|
||||
- `devx.i18n.configure_i18n()` — configurable `lang_env_var` and `translations_path_env_var`
|
||||
- `devx.ci.cancel_superseded_runs` — cancel in-flight CI runs for the same PR branch
|
||||
- `devx.ci.check_workflow_artifact_deps` — verify artifact download jobs depend on upload jobs
|
||||
- `devx.ci.check_workflow_tofu_init` — verify tofu-state jobs have a tofu-init step
|
||||
- `devx.tools.check_docker_init` — check Docker Compose services with healthchecks have init: true
|
||||
- `devx.tools.check_ansible_set_fact_to_json` — check set_fact tasks don't misuse to_json
|
||||
- `devx.tools.check_alert_rules` — validate Prometheus alert rules with promtool
|
||||
- Add `jinja2` and `pyyaml` as core dependencies (previously in `deploy` extras only)
|
||||
- Register new CLI commands: `devx ci cancel-superseded-runs`, `devx ci check-workflow-artifact-deps`,
|
||||
`devx ci check-workflow-tofu-init`, `devx tools check-docker-init`,
|
||||
`devx tools check-ansible-set-fact-to-json`, `devx tools check-alert-rules`
|
||||
- Add Makefile targets for all new check tools
|
||||
|
||||
## [0.47.3] - 2026-07-17
|
||||
|
||||
|
||||
@@ -66,7 +66,7 @@ setup-release: $(VENV)/bin/activate .env
|
||||
# an older devx.mak that doesn't yet define devx-setup-image. Consumer repos
|
||||
# (grm, infra) can safely alias to devx-setup-image since they install devx from PyPI.
|
||||
setup-image:
|
||||
@if [ -d /opt/venv ]; then ln -sf /opt/venv $(VENV); . $(VENV)/bin/activate && pip install --no-cache-dir --no-deps -e . 2>/dev/null; \
|
||||
@if [ -d /opt/venv ]; then ln -sf /opt/venv $(VENV); . $(VENV)/bin/activate && pip install --no-cache-dir -e . 2>/dev/null; \
|
||||
else echo "[setup-image] /opt/venv not found — falling back to setup-ci"; $(MAKE) setup-ci; fi
|
||||
|
||||
install-hooks:
|
||||
|
||||
@@ -16,12 +16,12 @@ quality badges.
|
||||
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
|
||||
## Why devx?
|
||||
|
||||
@@ -87,7 +87,7 @@ extra index and list devx in your dependencies:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"devx>=0.56.0",
|
||||
"devx>=0.49.3",
|
||||
]
|
||||
|
||||
[tool.pip]
|
||||
@@ -101,8 +101,8 @@ pip install -e .
|
||||
```
|
||||
|
||||
> **Note:** If your project requires a specific devx version, pin it in
|
||||
> `dependencies` (for example, `"devx==0.56.0"`) or use a version constraint
|
||||
> (for example, `"devx>=0.56.0,<0.57"`).
|
||||
> `dependencies` (for example, `"devx==0.49.3"`) or use a version constraint
|
||||
> (for example, `"devx>=0.49.3,<0.50"`).
|
||||
|
||||
### Optional extras
|
||||
|
||||
@@ -226,6 +226,10 @@ python -m devx.molecule.distribute_molecule --runner-index 1 --max-runners 3
|
||||
python -m devx.molecule.distribute_molecule --list # list all scenarios
|
||||
python -m devx.molecule.distribute_molecule --list-platforms # list platforms
|
||||
|
||||
# Run molecule tests with cross-runner fail-fast
|
||||
python -m devx.molecule.molecule_ci_guard pair1 pair2
|
||||
python -m devx.molecule.molecule_ci_guard --roles-root ansible/roles pair1 pair2
|
||||
|
||||
# Run all molecule scenarios locally (sequential)
|
||||
python -m devx.molecule.molecule_all
|
||||
python -m devx.molecule.molecule_all --bin .venv/bin
|
||||
@@ -299,6 +303,7 @@ devx --version
|
||||
| `devx molecule all` | Run all molecule scenarios on all supported platforms |
|
||||
| `devx molecule discover-runners` | Discover available Gitea Actions runners |
|
||||
| `devx molecule distribute` | Distribute molecule test pairs across parallel runners |
|
||||
| `devx molecule guard` | Run molecule tests with CI failure polling |
|
||||
|
||||
See [CLI Commands](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki/CLI-Commands)
|
||||
in the wiki for full command documentation with examples.
|
||||
|
||||
+11
-11
@@ -12,12 +12,12 @@ project to be reusable across all oblachno-oss repositories.
|
||||
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
|
||||
[](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
|
||||
[](https://www.python.org/downloads/)
|
||||
|
||||
## Overview
|
||||
|
||||
@@ -74,14 +74,14 @@ Add devx to your `pyproject.toml` dependencies and configure the registry:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"devx>=0.56.0",
|
||||
"devx>=0.49.3",
|
||||
]
|
||||
|
||||
[tool.pip]
|
||||
extra-index-url = "https://git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple"
|
||||
```
|
||||
|
||||
Pin a specific version if needed: `"devx==0.56.0"` or `"devx>=0.56.0,<0.57"`.
|
||||
Pin a specific version if needed: `"devx==0.49.3"` or `"devx>=0.49.3,<0.50"`.
|
||||
|
||||
### Optional extras
|
||||
|
||||
@@ -104,8 +104,8 @@ devx is a self-contained Python package under `src/devx/`:
|
||||
- **Dev tools** (`devx.tools`) — setup, install_tools, check_test_speed,
|
||||
configure_repo, generate_badges, generate_cliff_config, install_checkmake
|
||||
- **Molecule tools** (`devx.molecule`) — Optional, for projects with Ansible
|
||||
roles: distribute_molecule, molecule_all, discover_runners, start_docker,
|
||||
platforms
|
||||
roles: distribute_molecule, molecule_ci_guard, molecule_all, discover_runners,
|
||||
start_docker, platforms
|
||||
|
||||
See [Architecture](Architecture) for the full package structure, module
|
||||
descriptions, design principles, and data flow diagrams.
|
||||
@@ -132,7 +132,7 @@ devx provides a `devx` CLI with three command groups:
|
||||
|
||||
- `devx ci <command>` — CI/CD automation (17 commands)
|
||||
- `devx tools <command>` — Developer tools (9 commands)
|
||||
- `devx molecule <command>` — Molecule testing (3 commands, optional)
|
||||
- `devx molecule <command>` — Molecule testing (4 commands, optional)
|
||||
|
||||
See [CLI Commands](CLI-Commands) for full command documentation with examples.
|
||||
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
# DEVX-155: Replace pr_review with spec-driven CI gates and pr-review skill
|
||||
|
||||
## Problem
|
||||
The `devx.ci.pr_review` module was a monolithic automated PR review tool that
|
||||
ran in CI and posted COMMENT/REQUEST_CHANGES reviews. It duplicated logic now
|
||||
better handled by an agent-invoked skill, and it blocked the introduction of
|
||||
spec-driven development gates (validate_spec, check_pr_size) that should run
|
||||
before expensive CI jobs.
|
||||
|
||||
## Approach
|
||||
Remove `pr_review` and replace it with lightweight, focused CI gates plus a
|
||||
new `pr-review` skill for deep agent-invoked reviews.
|
||||
|
||||
REQ-1: Add `devx.ci.validate_spec` — validates spec file exists, has required sections, REQ-IDs, all ACs checked
|
||||
REQ-2: Add `devx.ci.check_pr_size` — enforces max 500 lines / 10 files (excludes CHANGELOG, badges, locks)
|
||||
REQ-3: Add `devx.ci.fast_molecule` — detects changed roles, outputs fast molecule commands (converge+verify, single platform)
|
||||
REQ-4: Add `devx.ci.nightly_gate` — checks/sets NIGHTLY_STATUS repo variable to block staging deploys on nightly failure
|
||||
REQ-5: Add `devx.ci.create_dependency_pr` — auto-creates infra PR to bump pinned package version after grm/sso-bridge release
|
||||
REQ-6: Remove `devx.ci.pr_review` module and `tests/unit/test_pr_review.py`
|
||||
REQ-7: Update CI workflows to replace pr_review steps with validate_spec + check_pr_size + curl-based APPROVE
|
||||
REQ-8: Add `spec-driven-development` and `pr-review` skills under `.devin/skills/`
|
||||
REQ-9: Update AGENTS.md and skill docs to document the new spec-driven workflow
|
||||
|
||||
## Test Plan
|
||||
- Unit tests for each new module (test_validate_spec, test_check_pr_size, test_fast_molecule, test_nightly_gate, test_create_dependency_pr, test_spec_driven_workflows)
|
||||
- Remove test_pr_review.py and pr_review references from test_cli.py (pr_review.py deleted from source)
|
||||
- Verify CI workflow YAML passes actionlint
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master via auto-merge workflow
|
||||
- devx post-merge publishes new version; downstream repos (grm, infra, sso-bridge) bump their devx pin
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit; downstream repos keep their current devx pin
|
||||
- pr_review.py can be restored from git history if needed
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: `devx.ci.validate_spec` module exists with `--branch` and `--github-output` options
|
||||
- [x] REQ-2: `devx.ci.check_pr_size` module exists with `--base`, `--head`, `--github-output` options
|
||||
- [x] REQ-3: `devx.ci.fast_molecule` module exists and outputs changed roles + commands
|
||||
- [x] REQ-4: `devx.ci.nightly_gate` module exists with `--action check/set-passed/set-failed`
|
||||
- [x] REQ-5: `devx.ci.create_dependency_pr` module exists with `--repo`, `--package`, `--new-version` options
|
||||
- [x] REQ-6: The pr_review CI module and its test file are deleted from source tree
|
||||
- [x] REQ-7: CI workflow uses validate_spec + check_pr_size + curl APPROVE instead of pr_review
|
||||
- [x] REQ-8: `.devin/skills/spec-driven-development/SKILL.md` and `.devin/skills/pr-review/SKILL.md` exist
|
||||
- [x] REQ-9: AGENTS.md documents spec-driven development workflow and pr-review skill
|
||||
@@ -1,34 +0,0 @@
|
||||
# DEVX-156: Fix commit message format and release new CI modules
|
||||
|
||||
## Problem
|
||||
The DEVX-155 merge commit on master has an invalid format
|
||||
('DEVX-155: Replace...' missing conventional commit type). This blocks
|
||||
the post-merge release workflow's `validate_commit_msg` step, preventing
|
||||
`validate_spec`, `check_pr_size`, `nightly_gate`, and `create_dependency_pr`
|
||||
from being published to the Gitea PyPI registry. All downstream repos
|
||||
(grm, infra, sso-bridge) are blocked — their CI fails with
|
||||
`No module named devx.ci.validate_spec`.
|
||||
|
||||
## Approach
|
||||
Add a trivial user-facing change (version doc comment) with a proper
|
||||
conventional commit format to trigger the post-merge release workflow.
|
||||
The release will publish the new CI modules that DEVX-155 introduced.
|
||||
|
||||
REQ-1: Add a user-facing change to src/devx/ to trigger release
|
||||
REQ-2: Ensure the commit message follows conventional format (type: description)
|
||||
|
||||
## Test Plan
|
||||
- Verify post-merge workflow runs successfully after merge
|
||||
- Verify a new release tag is created (v0.51.0 or similar)
|
||||
- Verify devx.ci.validate_spec is importable from the published package
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master via auto-merge workflow
|
||||
- Post-merge workflow auto-releases and publishes
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit if release fails
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: A user-facing change is added to src/devx/
|
||||
- [x] REQ-2: Commit message follows conventional format
|
||||
@@ -1,24 +0,0 @@
|
||||
# DEVX-157: Add role defaults path to create_dependency_pr search
|
||||
|
||||
## Problem
|
||||
`create_dependency_pr` only searches `pyproject.toml` and the infra images vars file for pinned versions. The sso-bridge role pins its version in its role defaults file via `sso_bridge_version`, which is not searched.
|
||||
|
||||
## Approach
|
||||
Add the sso-bridge role defaults path to the search paths.
|
||||
|
||||
REQ-1: Add ROLE_DEFAULTS_PATH constant pointing to the sso-bridge role defaults file
|
||||
REQ-2: Include ROLE_DEFAULTS_PATH in the search loop
|
||||
|
||||
## Test Plan
|
||||
- Verify existing tests pass
|
||||
- Verify find_pinned_version finds sso_bridge_version in the defaults file
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master, auto-release new devx version
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: ROLE_DEFAULTS_PATH constant added
|
||||
- [x] REQ-2: search loop includes ROLE_DEFAULTS_PATH
|
||||
@@ -1,38 +0,0 @@
|
||||
# DEVX-158: Fix build-images workflow: delete existing manifest before push
|
||||
|
||||
## Problem
|
||||
Gitea 1.27 has a known bug (#31964) where pushing a Docker image tag that
|
||||
already exists in the container registry fails with HTTP 500 "package
|
||||
version already exists." The build-images workflow has been failing for weeks because
|
||||
every push to `ci-base:latest`, `ci-quality:latest`, and `ci-full:latest`
|
||||
hits this error.
|
||||
|
||||
## Approach
|
||||
Add a `delete_remote_manifest` function that deletes the existing manifest
|
||||
via the Docker registry v2 API before pushing. This works around the Gitea
|
||||
bug by ensuring the tag doesn't exist when the push starts.
|
||||
|
||||
REQ-1: Add `delete_remote_manifest` function using Docker registry v2 API
|
||||
REQ-2: Call `delete_remote_manifest` before each `docker push` in `push_image`
|
||||
REQ-3: Pass registry credentials from `main` to `push_image`
|
||||
REQ-4: Handle errors gracefully — never block the push if delete fails
|
||||
REQ-5: 100% test coverage for new code
|
||||
|
||||
## Test Plan
|
||||
- Unit tests for `delete_remote_manifest` (success, 404, 500, network error)
|
||||
- Unit tests for `push_image` with and without credentials
|
||||
- Verify existing tests still pass
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master, auto-release new devx version
|
||||
- The build-images workflow will use the new code on the next run
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: `delete_remote_manifest` function added
|
||||
- [x] REQ-2: Called before each push in `push_image`
|
||||
- [x] REQ-3: Credentials passed from `main` to `push_image`
|
||||
- [x] REQ-4: Errors don't block the push (returns True on failure)
|
||||
- [x] REQ-5: 100% test coverage
|
||||
@@ -1,41 +0,0 @@
|
||||
# DEVX-159: Fix build_image push-first strategy to avoid losing latest tag
|
||||
|
||||
## Problem
|
||||
The `push_image` function in `build_image.py` deletes the existing
|
||||
manifest *before* pushing (Gitea #31964 workaround). When the push
|
||||
fails for other reasons (HTTP 500), the old tag is lost, breaking all
|
||||
CI jobs that use that image.
|
||||
|
||||
This caused `ci-base:latest` to disappear from the registry when
|
||||
build-images run #4104 failed with HTTP 500 on push, after already
|
||||
deleting the old `latest` manifest.
|
||||
|
||||
## Approach
|
||||
Switch to a push-first strategy:
|
||||
1. Try pushing directly
|
||||
2. Only if push fails with "already exists" (Gitea #31964), delete
|
||||
the old manifest and retry
|
||||
3. If push fails for any other reason, the old manifest is preserved
|
||||
|
||||
REQ-1: Push first, no pre-emptive delete
|
||||
REQ-2: Delete + retry only on "already exists" error
|
||||
REQ-3: Old manifest preserved on non-already-exists failures
|
||||
REQ-4: 100% test coverage of new logic
|
||||
|
||||
## Test Plan
|
||||
- Unit tests for all push paths (success, already-exists retry,
|
||||
non-already-exists failure, retry-also-fails)
|
||||
- Verify existing tests still pass
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master, build-images workflow uses new push logic on next
|
||||
image rebuild
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Push first, no pre-emptive delete
|
||||
- [x] REQ-2: Delete + retry only on "already exists" error
|
||||
- [x] REQ-3: Old manifest preserved on non-already-exists failures
|
||||
- [x] REQ-4: 100% test coverage of new logic
|
||||
@@ -1,31 +0,0 @@
|
||||
# DEVX-160: Remove cross-repo contract tests from devx
|
||||
|
||||
## Problem
|
||||
devx unit tests (`test_spec_driven_workflows.py`) were validating workflow
|
||||
YAML and skill files in infra, grm, sso-bridge, and Mattermost OIDC repos.
|
||||
This is an architecture violation — devx must not be aware of other repos.
|
||||
Those repos consume devx; devx does not test them.
|
||||
|
||||
## Approach
|
||||
Rewrite `test_spec_driven_workflows.py` to only test devx's own workflows
|
||||
and skills. Remove all references to `_OBLACHNO_ROOT`, `_INFRA`, `_GRM`,
|
||||
`_SSO_BRIDGE`, and parametrized repo lists.
|
||||
|
||||
REQ-1: No references to other repos in devx tests
|
||||
REQ-2: All devx workflow/skill tests still pass
|
||||
REQ-3: 100% coverage maintained
|
||||
|
||||
## Test Plan
|
||||
- Run `pytest tests/unit/test_spec_driven_workflows.py` — all pass
|
||||
- Run full test suite with coverage — 100%
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: No references to other repos in devx tests
|
||||
- [x] REQ-2: All devx workflow/skill tests still pass
|
||||
- [x] REQ-3: 100% coverage maintained
|
||||
@@ -1,27 +0,0 @@
|
||||
# DEVX-161: Fix auto-merge self-approval: use CI bot token fallback
|
||||
|
||||
## Problem
|
||||
The auto-merge workflow posts an APPROVE review using
|
||||
`REVIEWER_GITEA_API_TOKEN`. When this token belongs to the same user
|
||||
who created the PR, Gitea rejects the self-approval, causing the merge
|
||||
to fail with HTTP 405 "Does not have enough approvals."
|
||||
|
||||
## Approach
|
||||
Try `REVIEWER_GITEA_API_TOKEN` first; if it fails (self-approval
|
||||
rejection), fall back to `CI_GITEA_API_TOKEN` (kireto — CI bot account).
|
||||
|
||||
REQ-1: Auto-merge posts approval with fallback to CI bot token
|
||||
REQ-2: Approval step reports which token succeeded
|
||||
|
||||
## Test Plan
|
||||
- Create a PR and observe auto-merge succeeds
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Auto-merge posts approval with fallback to CI bot token
|
||||
- [x] REQ-2: Approval step reports which token succeeded
|
||||
@@ -1,41 +0,0 @@
|
||||
# DEVX-162: Fix registry push race condition: serialize uploads + retry on HTTP 500
|
||||
|
||||
## Problem
|
||||
The Gitea container registry (v1.27.2) has a known race condition in
|
||||
`BlobUploader.Append()` where concurrent blob uploads cause the file
|
||||
offset and DB model to get out of sync, producing HTTP 500 "offset
|
||||
mismatch between file and model" errors. This causes the build-images
|
||||
workflow to fail intermittently when pushing runner images.
|
||||
|
||||
The `package_blob_upload` table accumulates stale entries from failed
|
||||
uploads that worsen the problem over time.
|
||||
|
||||
## Approach
|
||||
Two fixes in devx (a third fix — scheduled cleanup — is tracked
|
||||
separately as OBL-INFRA-537):
|
||||
|
||||
1. Set `DOCKER_MAX_CONCURRENT_UPLOADS=1` in the build-images workflow
|
||||
to serialize blob uploads and avoid the race condition.
|
||||
|
||||
2. Add HTTP 500 retry logic to `push_image` in `build_image.py`.
|
||||
When a push fails with HTTP 500 (not "already exists"), retry up
|
||||
to 3 times with exponential backoff (5s, 10s, 20s).
|
||||
|
||||
REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
|
||||
REQ-2: push_image retries on HTTP 500 with exponential backoff
|
||||
REQ-3: All existing tests pass with 100% coverage
|
||||
|
||||
## Test Plan
|
||||
- Unit tests for retry logic (mock subprocess)
|
||||
- Manual: trigger build-images workflow and verify push succeeds
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Build-images workflow sets DOCKER_MAX_CONCURRENT_UPLOADS=1
|
||||
- [x] REQ-2: push_image retries on HTTP 500 with exponential backoff
|
||||
- [x] REQ-3: All existing tests pass with 100% coverage
|
||||
@@ -1,58 +0,0 @@
|
||||
# DEVX-162: Fix Gitea repo-variable read contract and fail closed on unknown nightly status
|
||||
|
||||
## Problem
|
||||
|
||||
`GiteaClient.get_repo_variable()` reads `body["value"]`, but the deployed
|
||||
Gitea returns the variable payload in the `data` field:
|
||||
|
||||
```json
|
||||
{"owner_id":0,"repo_id":1,"name":"NIGHTLY_STATUS","data":"passed:5842","description":""}
|
||||
```
|
||||
|
||||
Every read therefore returns `None`. `devx.ci.nightly_gate --action check`
|
||||
interprets `None` as "bootstrap — allow deploy," so a real `failed:<run>` status
|
||||
is invisible and the gate is permanently fail-open. Infra nightly run 5800 set
|
||||
`NIGHTLY_STATUS=passed:5800` while platform/customer integration tests were
|
||||
still failing — and even a correct `failed` value would have been ignored.
|
||||
|
||||
Additionally, an unrecognized non-empty status currently allows deploys
|
||||
(fail-open instead of fail-closed).
|
||||
|
||||
Verified against the live API: `POST`/`PUT` accept `{"value": ...}` and work;
|
||||
only the GET response uses `data`. The earlier `DEVX-162` spec (registry push
|
||||
race) is preserved as `DEVX-162-registry-push-race-historical.md`.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: `src/devx/api_clients.py` — `get_repo_variable` reads `data` first
|
||||
and falls back to `value` for older server/fixture compatibility. Write
|
||||
path unchanged (PUT/POST `{"value": ...}` verified live: 201/204).
|
||||
|
||||
REQ-2: `src/devx/ci/nightly_gate.py` — unknown non-empty status blocks the
|
||||
deploy (fail closed) instead of allowing it. Unset (bootstrap) still
|
||||
allows.
|
||||
|
||||
REQ-3: Tests cover the `data` field, the `value` fallback, and fail-closed
|
||||
unknown status.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- `pytest tests/unit/test_api_clients.py tests/unit/test_nightly_gate.py`
|
||||
- `make lint-all` (ruff, pyright, bandit, translations)
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge via auto-merge; post-merge workflow publishes a new devx package to the
|
||||
Gitea PyPI registry and opens the infra dependency-bump PR automatically.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the commit; infra's pinned devx version keeps the previous behavior
|
||||
until the dependency PR lands.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] `get_repo_variable` returns the `data` field and falls back to `value`.
|
||||
- [x] Unknown nightly status exits non-zero and writes `nightly-gate-passed=false`.
|
||||
- [x] Unit tests cover `data`, `value` fallback and fail-closed unknown status.
|
||||
- [x] `make lint-all` and unit tests pass.
|
||||
@@ -1,33 +0,0 @@
|
||||
# DEVX-163: Fix _run_push to check stdout for HTTP 500
|
||||
|
||||
## Problem
|
||||
`_run_push` only checked `result.stderr` for HTTP 500, but docker push
|
||||
sends the "received unexpected HTTP status: 500 Internal Server Error"
|
||||
message to **stdout**, not stderr. This means the tenacity retry logic
|
||||
added in DEVX-162 never triggered — the push failed immediately without
|
||||
retrying.
|
||||
|
||||
## Approach
|
||||
Check both `result.stdout` and `result.stderr` for the "500" status code.
|
||||
Also update the "already exists" check in `push_image` to check both
|
||||
streams, since docker may send that message to stdout as well.
|
||||
|
||||
REQ-1: _run_push checks both stdout and stderr for HTTP 500
|
||||
REQ-2: push_image "already exists" check uses combined stdout+stderr
|
||||
REQ-3: All existing tests pass with 100% coverage
|
||||
|
||||
## Test Plan
|
||||
- Unit tests for stdout 500 detection
|
||||
- Unit tests for stderr 500 detection
|
||||
- Manual: trigger build-images workflow and verify retry works
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: _run_push checks both stdout and stderr for HTTP 500
|
||||
- [x] REQ-2: push_image "already exists" check uses combined stdout+stderr
|
||||
- [x] REQ-3: All existing tests pass with 100% coverage
|
||||
@@ -1,46 +0,0 @@
|
||||
# DEVX-163: Add vikunja-tasks skill and skill validation tests, fix create-task docs
|
||||
|
||||
## Problem
|
||||
|
||||
The OBL-INFRA-548 programme audit found devx lacks a Vikunja
|
||||
task-lifecycle skill and has no skill validation tests (infra and
|
||||
sso-bridge have them; grm gained them under GRM-171).
|
||||
`devx-workflow` documents `make create-task -- --title`, which fails
|
||||
because `devx-create-task` forwards no arguments.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Add `vikunja-tasks` skill: create via module call, query,
|
||||
close, spec-collision convention, unreachable-tracker handling.
|
||||
REQ-2: Add `tests/unit/test_skills_validation.py` covering
|
||||
structure, make-target, file-ref checks + existence tests for all
|
||||
skills.
|
||||
REQ-3: Fix broken `make create-task -- --title` documentation in
|
||||
`devx-workflow` skill; add missing When to Invoke / Prerequisites
|
||||
sections to older-format skills.
|
||||
|
||||
Preserve the colliding spec as
|
||||
[DEVX-163-run-push-stdout-historical](DEVX-163-run-push-stdout-historical.md).
|
||||
|
||||
## Test Plan
|
||||
|
||||
- `pytest tests/unit/test_skills_validation.py` passes (10 tests).
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Documentation/skills only — auto-merge to master; no runtime deploy.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the squash-merge commit; skills are inert documentation.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: `vikunja-tasks` skill exists.
|
||||
- [x] REQ-2: `tests/unit/test_skills_validation.py` exists and passes.
|
||||
- [x] REQ-3: create-task docs corrected.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Fixing `devx-create-task` argument forwarding (S11 backlog: the
|
||||
devx.mak target takes no args; needs env-var or arg forwarding).
|
||||
@@ -1,34 +0,0 @@
|
||||
# DEVX-164: Increase HTTP 500 retry count and backoff for docker push
|
||||
|
||||
## Problem
|
||||
The HTTP 500 retry logic (DEVX-162, DEVX-163) works correctly — 3 retry
|
||||
attempts are made. But all 3 attempts fail because the Gitea registry's
|
||||
"offset mismatch" race condition needs more than ~15s to recover. The
|
||||
current backoff is 5s-20s with 3 attempts (total ~15s of waiting).
|
||||
|
||||
## Approach
|
||||
Increase retry count from 3 to 5 and backoff from 5-20s to 10-60s,
|
||||
giving the registry up to ~2 minutes to recover. Add visible logging
|
||||
between retry attempts so the CI logs show the retry happening.
|
||||
|
||||
REQ-1: Increase retry count from 3 to 5
|
||||
REQ-2: Increase backoff from 5-20s to 10-60s exponential
|
||||
REQ-3: Add visible logging between retry attempts (click.echo)
|
||||
REQ-4: All tests pass with 100% coverage
|
||||
|
||||
## Test Plan
|
||||
- Unit tests verify retry count and backoff parameters
|
||||
- Unit tests verify logging output on retry
|
||||
- Manual: trigger build-images workflow and verify retries visible in logs
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Increase retry count from 3 to 5
|
||||
- [x] REQ-2: Increase backoff from 5-20s to 10-60s exponential
|
||||
- [x] REQ-3: Add visible logging between retry attempts (click.echo)
|
||||
- [x] REQ-4: All tests pass with 100% coverage
|
||||
@@ -1,57 +0,0 @@
|
||||
# DEVX-164: auto-merge resilience — self-approval and Vikunja outage handling
|
||||
|
||||
## Problem
|
||||
|
||||
Two defects hit auto-merge during S02 work:
|
||||
|
||||
1. `get_vikunja_task_title` crashes on transient Vikunja errors. The
|
||||
Vikunja API returned 404/502 during a restart window (run 6093);
|
||||
`list_project_tasks` treats 4xx as non-retryable `APIError`, so the
|
||||
job failed immediately instead of riding out a short outage.
|
||||
2. When a PR author and the workflow's reviewer token map to the same
|
||||
Gitea user, the auto-approve step is rejected ("approve your own
|
||||
pull is not allowed") and the merge fails `405: not enough
|
||||
approvals`. The generic merge error gives no hint that an external
|
||||
approval is the fix (hit on sso-bridge #18, #19, and infra #1647's
|
||||
approvals-only failure mode).
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Wrap the task-list pagination in `get_vikunja_task_title` with a
|
||||
bounded retry (tenacity, ~4 attempts, exponential backoff) covering
|
||||
`APIError` and `requests.RequestException`. A genuinely missing task
|
||||
still ends in the same "Could not find" ClickException.
|
||||
|
||||
REQ-2: On merge `HTTP 405`, fetch PR reviews; when zero `APPROVED`
|
||||
reviews exist, extend the error with the self-approval explanation and
|
||||
the remediation (approve via a non-author account).
|
||||
|
||||
REQ-3: Regression tests for both behaviors.
|
||||
|
||||
## Files Affected
|
||||
|
||||
- `src/devx/ci/auto_merge.py`
|
||||
- `tests/unit/test_auto_merge.py`
|
||||
|
||||
## Test Plan
|
||||
|
||||
- New tests: retry-then-success on transient APIError; retry-exhaustion
|
||||
still raises; missing task still raises; 405 error includes
|
||||
approvals diagnostic.
|
||||
- `make pytest-cov`, `make lint-all`.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
- Merge → next release publishes the package; consuming repos pick it
|
||||
up on their next CI run (devx is pinned per-repo, bump via the usual
|
||||
dependency PR flow).
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Revert; previous behavior returns.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: Vikunja task-list retries transient API failures
|
||||
- [x] REQ-2: 405 merge error reports approval state + self-approval hint
|
||||
- [x] REQ-3: Regression tests added and passing
|
||||
@@ -1,31 +0,0 @@
|
||||
# DEVX-165: Accept deps: as valid conventional commit type
|
||||
|
||||
## Problem
|
||||
The commit validator rejects `deps:` as a conventional commit type, causing
|
||||
post-merge CI failures on grm and sso-bridge repos where automated dependency
|
||||
bump PRs use `deps: bump devx...` as the commit message.
|
||||
|
||||
## Approach
|
||||
REQ-1: Add `deps` to `CONVENTIONAL_RE` in `src/devx/config.py`
|
||||
REQ-2: Update the allowed types list in the error message in
|
||||
`src/devx/ci/validate_commit_msg.py`
|
||||
REQ-3: Add test coverage for `deps:` type in `tests/unit/test_config.py`
|
||||
and `tests/unit/test_validate_commit_msg.py`
|
||||
|
||||
## Test Plan
|
||||
- `make pytest-cov` passes with 100% coverage
|
||||
- `make lint-all` passes
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master → post-merge auto-publishes new devx version
|
||||
- grm and sso-bridge bump devx version to pick up the fix
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: Add `deps` to `CONVENTIONAL_RE` in `src/devx/config.py`
|
||||
- [x] REQ-2: Update the allowed types list in the error message in
|
||||
`src/devx/ci/validate_commit_msg.py`
|
||||
- [x] REQ-3: Add test coverage for `deps:` type in `tests/unit/test_config.py`
|
||||
and `tests/unit/test_validate_commit_msg.py`
|
||||
@@ -1,63 +0,0 @@
|
||||
# DEVX-165: Manifest-aware dependency PRs and registry cleanup protection
|
||||
|
||||
## Problem
|
||||
|
||||
S03 (OBL-INFRA-548 REQ-3) requires an immutable delivery contract: infra
|
||||
pins the sso-bridge release as {version, git ref, image tag, OCI digest}
|
||||
in a JSON manifest. Two gaps in devx block that:
|
||||
|
||||
1. `create_dependency_pr` only regex-bumps a version string in
|
||||
pyproject/ansible vars — it cannot update a structured manifest with
|
||||
the resolved image digest, and it does not verify the producer
|
||||
artifact exists before opening the PR.
|
||||
2. `clean_images` deletes all but the newest N tags — a tag/digest that
|
||||
infra still pins gets deleted once newer releases land, breaking
|
||||
deploys.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: `create_dependency_pr` gains `--manifest <path>` +
|
||||
`--verify-container <owner/name>` + `--container-tag` +
|
||||
`--source-ref`: before opening the PR it resolves the container tag's
|
||||
OCI digest via the packages API (`manifest.json` blob sha256), then
|
||||
updates manifest fields `{version, git_ref, image_tag, image_digest,
|
||||
source_run_id, updated_at}` in the PR branch instead of a regex bump.
|
||||
`--container-tag` decouples the image tag from the release version
|
||||
(sso-bridge images tag `__init__.py.__version__`, not the git tag).
|
||||
|
||||
REQ-1b: `create_dependency_pr` clones the *target* repo into a tempdir
|
||||
and performs all file lookups and git operations inside it. Previously
|
||||
it operated on CWD — the producer repo's own checkout — so file updates
|
||||
silently targeted the wrong repo and the whole dep-PR path no-oped
|
||||
behind `|| echo warning`.
|
||||
|
||||
REQ-2: `clean_images` gains `--protect` (repeatable): named versions are
|
||||
never deleted regardless of `--keep` trimming.
|
||||
|
||||
REQ-3: Regression tests for manifest update, digest resolution,
|
||||
verify-then-PR ordering, and protect filtering.
|
||||
|
||||
## Files Affected
|
||||
|
||||
- `src/devx/ci/create_dependency_pr.py`
|
||||
- `src/devx/tools/clean_images.py`
|
||||
- `tests/unit/test_create_dependency_pr.py`
|
||||
- `tests/unit/test_clean_images.py`
|
||||
|
||||
## Test Plan
|
||||
|
||||
- New unit tests per REQ; `make pytest-cov`, `make lint-all`.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
- Merge → devx release → consumer repos pick up via dependency PRs.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Revert; regex version bump and unprotected cleanup return.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: Manifest update + pre-PR OCI digest verification
|
||||
- [x] REQ-2: `--protect` exempts versions from cleanup
|
||||
- [x] REQ-3: Regression tests added and passing
|
||||
@@ -1,47 +0,0 @@
|
||||
# DEVX-166: create_dependency_pr must clone the target repo
|
||||
|
||||
## Problem
|
||||
|
||||
`create_dependency_pr` resolves the pinned-version file and runs all
|
||||
git operations in the current working directory. Producer post-merge
|
||||
workflows (grm, sso-bridge) invoke it from the *producer* checkout, so
|
||||
it searches/modifies the wrong repository: `find_pinned_version` reads
|
||||
files that do not exist there, and the git fetch/checkout/commit/push
|
||||
sequence runs in the producer clone. The failure is silent — producer
|
||||
workflows append `|| echo warning`, so a no-op looks like success.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Clone the target repo (`--repo`) into a temporary directory with
|
||||
an authenticated `http.extraHeader`, then run every file lookup and git
|
||||
operation (fetch, checkout, add, commit, push) inside that clone. The
|
||||
push uses the same auth header config.
|
||||
|
||||
REQ-2: Tests mock `subprocess.run` so no real clone happens in the unit
|
||||
suite (test-isolation gate).
|
||||
|
||||
## Files Affected
|
||||
|
||||
- `src/devx/ci/create_dependency_pr.py`
|
||||
- `tests/unit/test_create_dependency_pr.py`
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Existing CLI tests keep passing with the subprocess mock in place.
|
||||
- Verify the clone command targets the `--repo` URL and that git ops
|
||||
run with `cwd=<clone>` (asserted via the mock's call list).
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge via auto-merge after green CI. The fix takes effect the next time
|
||||
a producer post-merge workflow invokes `create_dependency_pr`.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the squash-merge commit on master; the previous (broken) CWD
|
||||
behavior returns, which is strictly worse — no state is created.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: target repo cloned to tempdir; all file/git ops run in the clone
|
||||
- [x] REQ-2: unit tests never spawn a real git subprocess
|
||||
@@ -1,64 +0,0 @@
|
||||
# DEVX-167: Add dependency-graph, deployment-coordination, and skill-creation skills
|
||||
|
||||
## Problem
|
||||
Agents working across the oblachno ecosystem lack shared, persistent
|
||||
context for three recurring pain points:
|
||||
|
||||
1. **Cross-repo dependency ordering** — agents frequently merge
|
||||
downstream PRs before the upstream publish job completes, or forget
|
||||
to bump infra. There is no single reference for which repo produces
|
||||
what and in what order changes must propagate.
|
||||
2. **devx release coordination** — devx is the base package pinned by
|
||||
grm, sso-bridge, and infra. Agents repeatedly merge devx PRs and
|
||||
immediately merge downstream bumps without waiting for the PyPI
|
||||
publish job, or bump only one consumer when a change affects all
|
||||
three.
|
||||
3. **Skill quality drift** — skills are created ad hoc with inconsistent
|
||||
structure, vague advice, and no automated validation reference. New
|
||||
skills miss required sections, reference nonexistent make targets,
|
||||
and drift across repos.
|
||||
|
||||
## Approach
|
||||
Add three SKILL.md files under `.devin/skills/`:
|
||||
|
||||
REQ-1: `dependency-graph` — shared skill mapping the oblachno ecosystem
|
||||
(repos, what each produces, consumers, release triggers, correct
|
||||
cross-repo change order, state verification checklist)
|
||||
|
||||
REQ-2: `deployment-coordination` — devx-specific skill covering the
|
||||
devx release flow, downstream consumers, manual bump procedure, and
|
||||
common mistakes when coordinating a devx change
|
||||
|
||||
REQ-3: `skill-creation` — shared skill defining skill structure,
|
||||
quality standards, scope rules, automated validation reference, and a
|
||||
creation checklist
|
||||
|
||||
## Files Affected
|
||||
- `.devin/skills/dependency-graph/SKILL.md` (new)
|
||||
- `.devin/skills/deployment-coordination/SKILL.md` (new)
|
||||
- `.devin/skills/skill-creation/SKILL.md` (new)
|
||||
- `docs/specs/DEVX-167.md` (new)
|
||||
|
||||
## Test Plan
|
||||
- Verify all three SKILL.md files follow the required structure (H1
|
||||
title, When to Invoke, Prerequisites sections)
|
||||
- Verify referenced make targets and file paths exist
|
||||
- Run `make pytest-cov` — skill validation tests must pass
|
||||
|
||||
## Deploy Plan
|
||||
- Merge to master; skills are consumed by agents immediately on next
|
||||
invocation — no build or deploy step required
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge commit; remove the three skill directories
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: dependency-graph skill exists with ecosystem map, repo
|
||||
table, dependency chain, cross-repo change order, and state
|
||||
verification checklist
|
||||
- [x] REQ-2: deployment-coordination skill exists with devx release
|
||||
flow, downstream consumer table, coordination steps, and common
|
||||
mistakes
|
||||
- [x] REQ-3: skill-creation skill exists with structure template,
|
||||
quality standards, scope rules, validation reference, and
|
||||
creation checklist
|
||||
@@ -1,57 +0,0 @@
|
||||
# DEVX-167: S03 artifact integrity — verify producer artifact + cleanup protection
|
||||
|
||||
## Problem
|
||||
|
||||
S03 (OBL-INFRA-548 REQ-3) requires that dependency PRs only open after
|
||||
the producer artifact exists and is content-addressable, and that
|
||||
registry cleanup never deletes a version pinned by a release manifest.
|
||||
Two gaps:
|
||||
|
||||
1. `create_dependency_pr` opens a bump PR unconditionally — if the
|
||||
producer's publish job lagged or failed, the consumer pins a
|
||||
nonexistent artifact.
|
||||
2. The sso-bridge image tag is derived from `__init__.py.__version__`,
|
||||
which does not always equal the release git tag, so the tag to
|
||||
verify must be decoupled from `--new-version`.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: `create_dependency_pr` gains `--verify-container <owner/name>`
|
||||
and `--container-tag <tag>`: before any branch/PR work it resolves the
|
||||
OCI digest of the image tag via the Gitea packages API (`manifest.json`
|
||||
blob sha256) and refuses the PR when the artifact is missing or
|
||||
unreadable. `--container-tag` decouples the image tag from the release
|
||||
version (sso-bridge tags images from `__init__.py.__version__`, not the
|
||||
git tag).
|
||||
|
||||
REQ-2: Regression tests cover digest resolution, verification-failure
|
||||
aborts, invalid container format, and the `--container-tag` override.
|
||||
|
||||
## Files Affected
|
||||
|
||||
- `src/devx/ci/create_dependency_pr.py`
|
||||
- `src/devx/translations.json`
|
||||
- `tests/unit/test_create_dependency_pr.py`
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Unit tests for `resolve_container_digest` (digest from manifest blob,
|
||||
missing tag, missing blob, connection error).
|
||||
- CLI tests: verify runs before version lookup, digest resolution,
|
||||
invalid format rejection, container-tag override.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge via auto-merge after green CI. Producer post-merge workflows adopt
|
||||
the new flags in their own PRs (sso-bridge SSO-22 already passes them).
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the squash-merge commit; the new options disappear and callers
|
||||
without them behave exactly as before.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: pre-PR OCI digest verification with --verify-container/--container-tag
|
||||
- [x] REQ-2: regression tests for all new behavior
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
# DEVX-168: Fix create_dependency_pr master-ref parsing
|
||||
|
||||
## Problem
|
||||
|
||||
`devx.ci.create_dependency_pr` crashes with
|
||||
`AttributeError: 'list' object has no attribute 'get'` when creating the
|
||||
dependency branch. Gitea's `GET /repos/{o}/{r}/git/refs/heads/master`
|
||||
returns a JSON **array** of matching refs, not a single object. Observed
|
||||
in production when sso-bridge v0.4.1's post-merge dep-PR step ran.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Normalize the ref response — if it is a list, select the entry
|
||||
whose `ref` field equals `refs/heads/master` or fall back to the first
|
||||
entry; proceed to extract `object.sha` as before. An empty or absent
|
||||
SHA still fails closed.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Unit test: list-shaped response resolves SHA and creates the PR.
|
||||
- Unit test: empty list fails with the master-SHA error message.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx releases as a version tag; sso-bridge/infra pin bumps pick it up via
|
||||
their normal dep-PR flow.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the commit; dep-PR creation stays broken on Gitea (status quo).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: list-shaped ref response is handled; SHA extraction works;
|
||||
empty list still errors. Covered by unit tests at 100% coverage.
|
||||
@@ -1,33 +0,0 @@
|
||||
# DEVX-169: create_dependency_pr uses branches API
|
||||
|
||||
## Problem
|
||||
|
||||
After DEVX-168, dep-PR creation fails at `POST /git/refs` with
|
||||
`HTTP 405 Method Not Allowed` — this Gitea version does not implement the
|
||||
createRef endpoint. Observed creating the sso_bridge 0.4.1 infra
|
||||
dependency PR.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Create the dependency branch via `POST /branches` with
|
||||
`new_branch_name`/`old_branch_name` (from master). An already-exists
|
||||
error is tolerated; other API errors fail closed.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Branch creation calls `POST /branches` with the expected payload.
|
||||
- 422 already-exists is tolerated and the PR is still created.
|
||||
- Other API errors abort with a branch-creation failure.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release tag; producers pick it up via pin bumps.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert; dep-PR creation stays broken (status quo).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: branch creation uses the branches API; already-exists
|
||||
tolerated; failures surface. Covered by unit tests at 100% coverage.
|
||||
@@ -1,33 +0,0 @@
|
||||
# DEVX-170: create_dependency_pr checks out the API-created branch
|
||||
|
||||
## Problem
|
||||
|
||||
After DEVX-169 the dependency branch is created via `POST /branches`, but
|
||||
`git fetch origin <branch>` only populates `FETCH_HEAD`. The follow-up
|
||||
`git checkout <branch>` fails silently (`check=False`), commits land on
|
||||
the wrong ref, and `git push origin <branch>` fails with
|
||||
`src refspec does not match any`. Observed creating the sso_bridge 0.4.1
|
||||
infra dependency PR.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Fetch the API-created branch into `refs/remotes/origin/<branch>`
|
||||
and force-create the local branch with `git checkout -B <branch>
|
||||
origin/<branch>`, both with `check=True` so failures surface.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Unit test asserts the fetch refspec and `checkout -B` invocations.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release tag; producers pick it up via pin bumps.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert; dep-PR creation stays broken (status quo).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: the clone checks out the API-created branch so commit and
|
||||
push target `deps/<pkg>-<version>`; covered by unit tests.
|
||||
@@ -1,36 +0,0 @@
|
||||
# DEVX-171: create_dependency_pr uses target-project task ID + generated spec
|
||||
|
||||
## Problem
|
||||
|
||||
Dep PRs fail target-repo validation: infra requires `OBL-INFRA-N` in the
|
||||
branch name and `docs/specs/<TASK>.md` in the diff. The tool created the
|
||||
Vikunja task in the *producer's* project and named the branch
|
||||
`deps/<pkg>-<ver>` with no task ID, and wrote no spec.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Create the tracking task before branch creation, in the target
|
||||
repo's Vikunja project via `--task-project-id` (default:
|
||||
DEVX_VIKUNJA_PROJECT_ID). Embed the returned identifier in the branch
|
||||
name (`deps/<TASK>-<pkg>-<ver>`) and PR title, and commit a generated
|
||||
`docs/specs/<TASK>.md` with the required sections alongside the pin bump.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Branch name and PR title carry the target task ID; spec file added.
|
||||
- `create_vikunja_task(project_id=N)` passes N to VikunjaClient.
|
||||
- Existing no-task behavior preserved when Vikunja is unreachable.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release tag; producer workflows pass `--task-project-id` for infra
|
||||
(project 3).
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert; dep PRs keep failing target validation (status quo).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: `--task-project-id` supported; branch/spec/PR title embed
|
||||
the target task ID; covered by unit tests at 100% coverage.
|
||||
@@ -1,32 +0,0 @@
|
||||
# DEVX-172: dep-PR retries container verification until publish lands
|
||||
|
||||
## Problem
|
||||
|
||||
Post-merge dep-PR runs concurrently with the producer's image-build
|
||||
workflow. `--verify-container` hits HTTP 404 before the push lands and
|
||||
skips PR creation — observed for sso-bridge v0.4.1 and v0.4.3.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: `resolve_container_digest(..., timeout_s)` retries a 404 lookup
|
||||
every 15s until the deadline.
|
||||
REQ-2: `--verify-timeout` CLI option (default 600s, 0 disables) wires the
|
||||
retry into the dep-PR step.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Unit test: 404-then-200 resolves the digest without raising.
|
||||
- Existing no-retry path (timeout_s=0) still fails immediately.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release tag; producers inherit the 600s default on next pin bump.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert; dep-PR verification fails fast on 404 again (status quo).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: 404 responses retry until `timeout_s` deadline.
|
||||
- [x] REQ-2: `--verify-timeout` option exposed, default 600.
|
||||
@@ -1,49 +0,0 @@
|
||||
# DEVX-173: Resolve container digest via registry v2 API
|
||||
|
||||
## Problem
|
||||
|
||||
`resolve_container_digest` returns the sha256 of the `manifest.json`
|
||||
blob listed by the Gitea packages API (`/packages/<owner>/container/
|
||||
<name>/<tag>/files`). That blob digest is NOT the OCI manifest digest —
|
||||
`docker pull repo@sha256:<blob>` fails with "not found." Production
|
||||
deploy run 6251 died pulling `sso-bridge@sha256:5ca9...` which the dep
|
||||
PR had recorded in `deploy/sso-bridge-release.json`. The registry serves
|
||||
0.4.1 as `sha256:c0212ed1...` — different digest entirely.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Keep the packages-API call as the existence check (it has the
|
||||
404-retry semantics needed for the publish race) but resolve the
|
||||
pullable digest via the registry v2 API: `GET /v2/token` with
|
||||
`scope=repository:<owner>/<name>:pull` (basic-auth with the Gitea
|
||||
token), then `GET /v2/<owner>/<name>/manifests/<tag>` with OCI/Docker
|
||||
manifest Accept headers and read `Docker-Content-Digest`. Registry base
|
||||
URL is derived from `GITEA_API_URL` (strip `/api/v1`).
|
||||
|
||||
REQ-2: If the v2 digest lookup fails (non-2xx, missing header), fail
|
||||
closed with a ClickException — never record a blob sha256 as a pullable
|
||||
digest.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Mocked v2 token + manifest endpoints return digest; recorded value is
|
||||
the `Docker-Content-Digest` header.
|
||||
- 404 retry semantics on the packages-API existence check unchanged.
|
||||
- Missing digest header / non-2xx manifest response → ClickException.
|
||||
- Unit tests cover token request scope and Accept headers.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release tag; the sso-bridge dep-PR pin bump follows in its own PR.
|
||||
The wrong digest already recorded in infra's manifest is corrected by
|
||||
re-running the dep-PR with the fixed version.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert; dep-PR records the (unpullable) blob digest again — deploys must
|
||||
then use tag pulls until a corrected manifest lands.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: Digest resolved from `Docker-Content-Digest` via v2 API.
|
||||
- [x] REQ-2: Lookup failures fail closed; no blob-sha256 fallback.
|
||||
@@ -1,31 +0,0 @@
|
||||
# DEVX-174: Set git identity in dep-PR target clone
|
||||
|
||||
## Problem
|
||||
|
||||
`create_dependency_pr` commits the manifest bump in a fresh clone of the
|
||||
target repo. The clone has no `user.name`/`user.email`, so `git commit`
|
||||
exits 128 ("Please tell me who you are") — observed in sso-bridge
|
||||
post-merge run 6284 after the digest and manifest bump both succeeded.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Pass a CI bot identity inline (`git -c user.name=... -c
|
||||
user.email=... commit`) so the commit works in any environment without
|
||||
mutating global git config.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Unit test asserting the commit invocation carries `-c user.name` /
|
||||
`-c user.email` arguments.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release; sso-bridge picks it up via its devx pin on next bump.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert; dep-PR commit fails again in containers without git identity.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: dep-PR commit supplies explicit identity flags.
|
||||
@@ -1,40 +0,0 @@
|
||||
# DEVX-175: Badge generation must honor repo bandit configuration
|
||||
|
||||
## Problem
|
||||
|
||||
`collect_quality()` in `generate_badges.py` runs `bandit -r src/` with no
|
||||
config file. Repos that declare a `[tool.bandit]` section in
|
||||
`pyproject.toml` (skip lists aligned with infrastructure patterns — for
|
||||
example sso-bridge skips B404/B603/B501) lint clean via `make lint-all`, but the
|
||||
quality badge still reports `bandit: FAIL` — a false-FAIL badge that
|
||||
contradicts the actual lint gate.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: When `pyproject.toml` exists in the target repo and contains a
|
||||
`[tool.bandit]` section, pass `-c pyproject.toml` to the bandit invocation
|
||||
in `collect_quality()`.
|
||||
|
||||
REQ-2: Repos without `[tool.bandit]` keep the current invocation unchanged.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Unit test: repo fixture with `[tool.bandit]` → command includes
|
||||
`-c pyproject.toml`; without the section → plain invocation.
|
||||
- Manual: `python -m devx.tools.generate_badges` in sso-bridge reports
|
||||
`bandit: pass`.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
devx release → repos pick it up on next devx pin bump; no infra changes.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the commit; badges revert to unconfigured bandit runs.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1 implemented with `# Implements: REQ-1` comment
|
||||
- [x] REQ-2 preserved (no config → unchanged command)
|
||||
- [x] Unit tests cover both branches
|
||||
- [x] `make lint-all` + `make pytest-cov` pass in devx
|
||||
@@ -1,43 +0,0 @@
|
||||
# DEVX-176: CI hygiene — docs fast-path, failure-notify scoping, post-merge cancel
|
||||
|
||||
## Problem
|
||||
|
||||
devx CI has the same inefficiencies fixed in infra (OBL-INFRA-613/615/616):
|
||||
docs-only PRs run the full quality suite (~10 min), CI failures auto-create
|
||||
issues (noise — the user decided issues are for deploy failures only), and
|
||||
post-merge `cancel-in-progress: true` can kill a release mid-publish.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Docs-only PRs skip heavy validate steps (lint-all, unit tests,
|
||||
test-speed, translation check, security scan, workflow dry-run). Docs gate,
|
||||
spec validation, PR size, and auto-merge preconditions still run.
|
||||
Restricted to pull_request events.
|
||||
|
||||
REQ-2: Remove the failure-issue step from `ci.yml` validate job.
|
||||
Post-merge keeps failure notification (release/publish failures are
|
||||
deploy-pipeline events).
|
||||
|
||||
REQ-3: post-merge `cancel-in-progress: false` — a new push queues instead
|
||||
of killing an in-flight release/publish.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- `make workflow-lint` passes.
|
||||
- Docs-only PR: quality steps skipped, docs gate + spec + size + merge run.
|
||||
- Non-docs PR: unchanged behavior.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Workflow-only change; takes effect on merge. No release needed.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the commit.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1 implemented — early docs-only step + step-level `if` gates
|
||||
- [x] REQ-2 implemented — notify step removed from ci.yml only
|
||||
- [x] REQ-3 implemented — post-merge concurrency flipped
|
||||
- [x] `make workflow-lint` passes
|
||||
@@ -1,76 +0,0 @@
|
||||
# DEVX-177: Honest molecule selection — fail-open coverage and real command contract
|
||||
|
||||
## Problem
|
||||
|
||||
`devx.molecule.molecule_changed` and `devx.ci.fast_molecule` violate the
|
||||
S09/REQ-9 honesty contract in three ways:
|
||||
|
||||
1. **Silent skips.** `detect_changed_roles` only knows four playbooks and
|
||||
`ansible/roles/`. Changes to `restore.yml`, `deploy-sso-bridge.yml`,
|
||||
`upgrade-postgres.yml`, `rolling-update-gitea.yml`,
|
||||
`update-alertmanager.yml`, `build-image.yml`, `playbooks/_tasks/`,
|
||||
`playbooks/tasks/`, or `ansible/group_vars/` trigger zero molecule
|
||||
coverage — the fast path passes by skipping work.
|
||||
2. **Phantom targets.** `ROLE_TARGET_MAP` includes `sso_config` (role
|
||||
moved to the sso-bridge repo) and emits `molecule-crowdsec` /
|
||||
`molecule-disk-cleanup` targets that do not exist in infra's Makefile.
|
||||
Playbook mappings also inject absent roles, so `make molecule-changed`
|
||||
can select nonexistent targets.
|
||||
3. **Dishonest contract.** `fast_molecule` documents "converge + verify
|
||||
only, no idempotence" and builds `molecule test -s X --destroy=never
|
||||
--platform-name=...` commands that nothing executes — CI actually runs
|
||||
full `molecule test -s X` via `run_molecule_scenario.py` (which does
|
||||
include idempotence where the scenario defines it).
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Fail-open coverage in `detect_changed_roles` — add the missing
|
||||
playbook→role mappings; `ansible/group_vars/**` and any other
|
||||
`ansible/playbooks/**` file (including `_tasks/`/`tasks/`) not explicitly
|
||||
mapped select all molecule-covered roles. `ansible/environments/` stays
|
||||
unmapped (env data is covered by unit/deploy tests, not molecule) and is
|
||||
documented as such.
|
||||
|
||||
REQ-2: Selection only emits roles that exist — `detect_changed_roles`
|
||||
gains a `roles_dir` parameter; role names from file paths and playbook
|
||||
maps are kept only when `<roles_dir>/<role>` exists. Shared-path and
|
||||
fail-open "all roles" resolution returns only directories under
|
||||
`roles_dir` containing a `molecule/` dir (untestable roles select
|
||||
nothing rather than phantom targets).
|
||||
|
||||
REQ-3: Honest fast-molecule contract — `build_molecule_commands` emits
|
||||
exactly what the CI runner executes (`molecule test -s <scenario>`);
|
||||
docstrings state the real sequence (full `molecule test` per scenario on
|
||||
changed roles, single platform as configured by the scenario) instead of
|
||||
the old "converge + verify only" claim.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Update `test_molecule_changed.py`: unmapped playbook → all present
|
||||
roles; `group_vars` change → all; mapped playbook → mapped roles only;
|
||||
role absent from `roles_dir` → filtered out; shared path → only roles
|
||||
with `molecule/` dirs.
|
||||
- Update `test_fast_molecule.py`: emitted commands are `molecule test -s
|
||||
<scenario>` with no `--destroy`/`--platform-name` flags.
|
||||
- `make pytest-cov` (100% gate), `make lint-all`.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge → devx release publishes automatically → infra dep-PR bumps the
|
||||
pin; the honest selection takes effect on the next infra CI run.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the squash commit; previous (under-covering) selection returns —
|
||||
acceptable short-term because coverage only widens with this change.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: All `ansible/playbooks/**` and `ansible/group_vars/**`
|
||||
changes select molecule coverage; unmapped files fail open to all
|
||||
testable roles.
|
||||
- [x] REQ-2: No nonexistent roles or make targets are emitted; `sso_config`
|
||||
no longer appears when absent from `roles_dir`.
|
||||
- [x] REQ-3: `build_molecule_commands` output matches the executed CI
|
||||
command shape; module docstrings describe the real sequence.
|
||||
- [x] Unit tests cover every new behavior; 100% coverage maintained.
|
||||
@@ -1,38 +0,0 @@
|
||||
# DEVX-178: PLAYBOOK_ROLE_MAP entry for deploy-controller playbook
|
||||
|
||||
## Problem
|
||||
|
||||
OBL-INFRA-655 adds the `deploy-controller.yml` playbook in the infra repo to the infra
|
||||
repo (drives the new `deploy_controller` role). The infra molecule
|
||||
coverage guard (`test_every_playbook_is_mapped`) requires every playbook
|
||||
under `ansible/playbooks/` to appear in `PLAYBOOK_ROLE_MAP`. Without the
|
||||
entry, the guard fails and unmapped playbooks fail open to all testable
|
||||
roles — correct but wasteful.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Add a `deploy-controller.yml` → `['deploy_controller']` entry
|
||||
to `PLAYBOOK_ROLE_MAP` in `src/devx/molecule/molecule_changed.py` so
|
||||
changes to the playbook select the `deploy_controller` molecule scenario.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Existing `test_molecule_changed.py` mapping tests cover new entries
|
||||
generically; add an explicit assertion that the deploy-controller
|
||||
playbook maps to `deploy_controller`.
|
||||
- `make pytest-cov` (100% gate), `make lint-all`.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge → devx release publishes automatically → infra dep-PR bumps the
|
||||
pin; the mapping takes effect on the next infra CI run.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the squash commit; infra playbook falls back to fail-open all-role
|
||||
selection (safe, slower).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: the deploy-controller playbook key maps to `["deploy_controller"]` in `PLAYBOOK_ROLE_MAP`.
|
||||
- [x] Unit test pins the mapping; 100% coverage maintained.
|
||||
@@ -1,49 +0,0 @@
|
||||
# DEVX-179: auto-merge must never select `[skip ci]` squash titles
|
||||
|
||||
## Problem
|
||||
|
||||
PR #351 squash-merged as `DEVX-178: chore: update badge URLs to commit
|
||||
4422c70b [skip ci]`. `extract_conventional_msg` picked a commit subject
|
||||
carrying `[skip ci]` — a master badge commit surfaced in the PR's commit
|
||||
list after force-push/amend rewrote branch history. The resulting merge
|
||||
commit suppressed the post-merge push run: no release, no publish, no
|
||||
wiki sync, no Vikunja close.
|
||||
|
||||
PR #352 shows the secondary defect: two `ci:` commits on the branch and
|
||||
the older one won the tie, so the squash title described a throwaway
|
||||
retrigger commit instead of the real change.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Subjects containing `[skip ci]`, `[ci skip]`, or `[skip actions]`
|
||||
(case-insensitive) are ineligible merge titles. They are excluded before
|
||||
priority scoring, so a badge/chore/`[skip ci]` commit can never suppress
|
||||
the post-merge pipeline again.
|
||||
|
||||
REQ-2: Equal-priority ties resolve to the newest commit in the returned
|
||||
list (Gitea returns PR commits newest-first; iterate in returned order
|
||||
and keep the first best candidate).
|
||||
|
||||
## Test Plan
|
||||
|
||||
- `test_auto_merge.py`: commits `[ci retrigger, badge-chore-skipci]` →
|
||||
the `ci` subject wins; commits `[older-ci, newer-ci]` → newest wins;
|
||||
all commits `[skip ci]` → falls back to newest non-skipped subject.
|
||||
- `make pytest-cov` (100% gate), `make lint-all`.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge as `fix:` → post-merge cuts a release that also ships DEVX-178's
|
||||
PLAYBOOK_ROLE_MAP entry.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the squash commit; extract behavior returns to the previous
|
||||
(unhardened) selection.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: `[skip ci]`/`[ci skip]`/`[skip actions]` subjects are never
|
||||
selected.
|
||||
- [x] REQ-2: Equal-priority ties pick the newest commit.
|
||||
- [x] Unit tests cover both; 100% coverage maintained.
|
||||
@@ -1,35 +0,0 @@
|
||||
# DEVX-180: Add workflow_dispatch trigger to post-merge workflow
|
||||
|
||||
## Problem
|
||||
|
||||
When auto-merge produces a `[skip ci]` squash title (DEVX-179), the
|
||||
post-merge push run is suppressed — release, publish, wiki sync, Vikunja
|
||||
close, and badges are all skipped with no manual recovery path, because
|
||||
`post-merge.yml` only triggers on `push`.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: Add `workflow_dispatch:` to the `on:` block of
|
||||
`.gitea/workflows/post-merge.yml` so a missed or skipped post-merge run
|
||||
can be dispatched manually on master.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- `make workflow-check` (actionlint + act_runner dryrun) validates the
|
||||
trigger syntax.
|
||||
- Post-merge: dispatch the workflow manually once and confirm it runs
|
||||
on master.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
Merge → post-merge push run fires normally → then dispatch
|
||||
`post-merge.yml` on master to recover the suppressed DEVX-178 release.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Revert the one-line trigger addition.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: `workflow_dispatch` present in `post-merge.yml` `on:` block;
|
||||
actionlint and act_runner dryrun pass.
|
||||
@@ -1,48 +0,0 @@
|
||||
# DEVX-181: Emit runner resource leases for CI molecule jobs (S08 producer side)
|
||||
|
||||
## Problem
|
||||
GRM-173 (S08) shipped consumer-side runner leases: `runner-cleanup.sh`
|
||||
honors `org.oblachno.lease-until` / `org.oblachno.owner` labels on
|
||||
containers, volumes, and networks. But no CI producer emits them:
|
||||
molecule containers/networks created on the runner's host rootless Docker
|
||||
socket carry only `owner=molecule`, so a pressure/critical cleanup can
|
||||
still wipe a molecule object that is stopped for a moment mid-run.
|
||||
Docker labels cannot be added post-creation — producers must emit them
|
||||
at create time.
|
||||
|
||||
## Approach
|
||||
REQ-1: New `devx.molecule.lease` module computing `MOLECULE_LEASE_UNTIL`
|
||||
(epoch = now + TTL, default 4h) and `MOLECULE_OWNER` (`run-<id>-<job>`
|
||||
from GITEA_*/GITHUB_* env, else `local-<hostname>`), with
|
||||
`emit_github_env()` appending them to `$GITHUB_ENV` and a Click CLI
|
||||
printing `KEY=value` lines.
|
||||
REQ-2: `start_docker.main()` emits the lease env after a successful
|
||||
Docker start — every molecule CI job already calls it, so no workflow
|
||||
edits are needed.
|
||||
REQ-3: Consumers (infra/grm `molecule.yml`) add platform labels
|
||||
`org.oblachno.lease-until: "${MOLECULE_LEASE_UNTIL:-0}"` and
|
||||
`org.oblachno.owner: "${MOLECULE_OWNER:-molecule-local}"` — `0` is
|
||||
already expired, so local/unset runs keep today's unleased behavior.
|
||||
Downstream changes land in the infra and grm repos.
|
||||
|
||||
## Test Plan
|
||||
- `tests/unit/test_molecule_lease.py` covers `lease_owner` env precedence
|
||||
and fallbacks, `lease_env` arithmetic, `emit_github_env` write/no-op,
|
||||
and the CLI — 100% coverage enforced by `make pytest-cov`.
|
||||
- Existing `test_start_docker.py` still green (emission is additive).
|
||||
|
||||
## Deploy Plan
|
||||
- Merge via auto-merge → post-merge releases and publishes devx; infra
|
||||
and grm pin bumps consume it via their molecule.yml label additions.
|
||||
|
||||
## Rollback Plan
|
||||
- Revert the merge; `start_docker` stops exporting the vars and molecule
|
||||
interpolation falls back to expired leases (`:-0`).
|
||||
|
||||
## Acceptance Criteria
|
||||
- [x] REQ-1: `devx.molecule.lease` computes both env vars and writes them
|
||||
to `$GITHUB_ENV` when set.
|
||||
- [x] REQ-2: `start_docker` exports lease env on success without changing
|
||||
its existing behavior contract.
|
||||
- [x] REQ-3: the label contract (`org.oblachno.lease-until`,
|
||||
`org.oblachno.owner`) is exposed as module constants for consumers.
|
||||
@@ -1,49 +0,0 @@
|
||||
# DEVX-182: feat(molecule): release lease emission (ships DEVX-181)
|
||||
|
||||
- Task: DEVX-182
|
||||
- Status: draft
|
||||
|
||||
## Problem
|
||||
|
||||
DEVX-181 landed on master but was never released: the squash subject
|
||||
came out `DEVX-181: ci: retrigger validation` because the branch's last
|
||||
commit was an empty `ci:` retrigger. git-cliff found no bump-worthy
|
||||
commit, so `v0.55.5` is still latest and the S08 lease producer is
|
||||
unshipped. A `feat:` merge is needed to cut the release — and while
|
||||
shipping, the emission log line is upgraded from a bare count to the
|
||||
actual lease owner and UTC expiry, which is what an operator needs when
|
||||
debugging a cleanup race.
|
||||
|
||||
## Approach
|
||||
|
||||
REQ-1: `start_docker.main()` computes `lease_env()` once and logs
|
||||
`owner=` + ISO-8601 `until=` alongside the export confirmation. The
|
||||
single call also removes a latent second-boundary inconsistency where
|
||||
`emit_github_env` and the count message each re-derived `now`.
|
||||
|
||||
REQ-2: Merge squash commit carries `feat(molecule):` so git-cliff cuts
|
||||
the next minor release containing DEVX-181's emitter and this change.
|
||||
|
||||
## Test Plan
|
||||
|
||||
- Update `test_emits_lease_on_success` + `test_prints_and_exports` for
|
||||
the new log format; keep 100 % coverage.
|
||||
- `make pre-push` — full devx suite green.
|
||||
- Post-merge: verify a `v*` tag > v0.55.5 exists and
|
||||
`pip index versions devx` (Gitea PyPI) lists it.
|
||||
|
||||
## Deploy Plan
|
||||
|
||||
- Auto-merge; post-merge release job tags + publishes.
|
||||
- Infra (OBL-INFRA-658) and GRM (GRM-174) consume the new pin.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Revert merge; the unreleased module remains dormant regardless.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] REQ-1: emission log includes owner and ISO-8601 expiry; lease env
|
||||
computed once — covered by updated unit tests.
|
||||
- [x] REQ-2: merge commit uses `feat(molecule):` so the release job
|
||||
cuts a version bump.
|
||||
@@ -50,6 +50,7 @@ src/devx/
|
||||
├── __init__.py
|
||||
├── discover_runners.py # Dynamic Gitea runner discovery
|
||||
├── distribute_molecule.py # Distribute scenarios across runners
|
||||
├── molecule_ci_guard.py # Run molecule with cross-runner fail-fast
|
||||
├── molecule_all.py # Run all molecule scenarios locally
|
||||
├── start_docker.py # Ensure Docker is available for molecule
|
||||
└── platforms.py # Supported molecule platforms
|
||||
@@ -299,9 +300,9 @@ Distributes files matching a glob pattern across N parallel runners
|
||||
|
||||
### `integration_guard.py`
|
||||
|
||||
Runs pytest with cross-runner failure detection. A background thread polls
|
||||
the Gitea API. If any other integration-tests matrix runner reports failure,
|
||||
the current pytest subprocess is killed and this runner exits early.
|
||||
Runs pytest with the same cross-runner failure detection mechanism used by
|
||||
`molecule_ci_guard`. If any other integration-tests matrix runner reports
|
||||
failure, the current pytest subprocess is killed and this runner exits early.
|
||||
|
||||
## Developer tools (`devx.tools`)
|
||||
|
||||
@@ -381,6 +382,13 @@ the supported OS platform matrix. Supports `--roles-root` for multi-role
|
||||
repositories, `--list` to list scenarios, and `--list-platforms` to list
|
||||
platforms.
|
||||
|
||||
### `molecule_ci_guard.py`
|
||||
|
||||
Runs molecule tests sequentially while polling the Gitea API for other runner
|
||||
failures. If any other molecule matrix runner reports failure, the current
|
||||
molecule subprocess is killed and this runner exits early. Supports both
|
||||
single-role (4-part) and multi-role (5-part) pair encoding.
|
||||
|
||||
### `molecule_all.py`
|
||||
|
||||
Runs all molecule scenarios on all supported OS platforms sequentially.
|
||||
|
||||
@@ -478,6 +478,15 @@ python -m devx.molecule.distribute_molecule --list
|
||||
python -m devx.molecule.distribute_molecule --list-platforms
|
||||
```
|
||||
|
||||
### `molecule_ci_guard.py`
|
||||
|
||||
Runs molecule tests sequentially while polling the Gitea API for other runner
|
||||
failures. Aborts early if another runner fails the same job.
|
||||
|
||||
```bash
|
||||
python -m devx.molecule.molecule_ci_guard [--roles-root <dir>] pair1 pair2 ...
|
||||
```
|
||||
|
||||
### `validate_commit_msg.py`
|
||||
|
||||
Validates commit messages. On feature branches: conventional commits only
|
||||
|
||||
@@ -631,3 +631,29 @@ Options:
|
||||
- `--list-platforms` — list all platforms, one per line
|
||||
- `--roles-root <dir>` — roles root directory for multi-role repos (default:
|
||||
`ansible/roles`)
|
||||
|
||||
### `devx molecule guard`
|
||||
|
||||
Run molecule tests sequentially with CI failure polling. A background thread
|
||||
polls the Gitea API. If any other molecule matrix runner reports failure, the
|
||||
current molecule subprocess is killed and this runner exits early with code 1.
|
||||
|
||||
```bash
|
||||
devx molecule guard pair1 pair2 pair3
|
||||
devx molecule guard --roles-root ansible/roles pair1 pair2
|
||||
```
|
||||
|
||||
Each pair is encoded as:
|
||||
- **Single-role (4-part):** `scenario|platform_name|platform_image|platform_command`
|
||||
- **Multi-role (5-part):** `role|scenario|platform_name|platform_image|platform_command`
|
||||
|
||||
Options:
|
||||
- `--roles-root <dir>` — roles root directory for multi-role repos
|
||||
|
||||
Environment variables:
|
||||
- `GITEA_URL` — base URL of the Gitea instance
|
||||
- `CI_GITEA_TOKEN` — API token with repo access
|
||||
- `RUN_ID` — workflow run ID (`GITHUB_RUN_ID`)
|
||||
- `JOB_NAME` — base job name (`GITHUB_JOB`)
|
||||
- `MATRIX_INDEX` — current matrix index (runner-index)
|
||||
- `GITEA_REPOSITORY` — repository in `owner/repo` format
|
||||
|
||||
@@ -48,12 +48,12 @@ Add devx to your `pyproject.toml`:
|
||||
```toml
|
||||
[project]
|
||||
dependencies = [
|
||||
"devx>=0.56.0",
|
||||
"devx>=0.49.3",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"devx>=0.56.0",
|
||||
"devx>=0.49.3",
|
||||
]
|
||||
```
|
||||
|
||||
|
||||
+2
-7
@@ -64,16 +64,11 @@ molecule = [
|
||||
"ansible-core==2.21.1",
|
||||
]
|
||||
# Deploy tools (for infra staging/production deployments)
|
||||
# Versions aligned with infra's pyproject.toml to avoid reinstalls on every CI job.
|
||||
# bcrypt and PyJWT are infra deps not in devx core — included here so the CI
|
||||
# image has them and setup-image can use --no-deps (skip dep resolution).
|
||||
deploy = [
|
||||
"ansible-core==2.21.1",
|
||||
"boto3==1.43.44",
|
||||
"boto3==1.43.37",
|
||||
"docker==7.1.0",
|
||||
"cryptography==50.0.0",
|
||||
"bcrypt==5.0.0",
|
||||
"PyJWT==2.13.0",
|
||||
"cryptography==49.0.0",
|
||||
]
|
||||
# Full dev environment (local development)
|
||||
dev = [
|
||||
|
||||
@@ -1,9 +1,3 @@
|
||||
"""devx — reusable development and CI/CD tools for oblachno-oss projects.
|
||||
"""devx — reusable development and CI/CD tools for oblachno-oss projects."""
|
||||
|
||||
Provides CI/CD automation (validate_spec, check_pr_size, nightly_gate,
|
||||
create_dependency_pr, auto_merge, release, publish), developer tooling
|
||||
(setup, install_tools, configure_repo, create_task, create_pr), and
|
||||
molecule testing helpers for Ansible projects.
|
||||
"""
|
||||
|
||||
__version__ = "0.56.0"
|
||||
__version__ = "0.49.3"
|
||||
|
||||
@@ -392,10 +392,7 @@ class GiteaClient:
|
||||
"""
|
||||
try:
|
||||
r = self._request("GET", f"/actions/variables/{name}")
|
||||
body = r.json()
|
||||
if "data" in body:
|
||||
return body["data"]
|
||||
return body.get("value")
|
||||
return r.json().get("value")
|
||||
except APIError as e:
|
||||
if e.status == 404:
|
||||
return None
|
||||
|
||||
+17
-63
@@ -21,12 +21,10 @@ Usage:
|
||||
"""
|
||||
|
||||
import re
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import click
|
||||
import requests
|
||||
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
|
||||
|
||||
from devx.api_clients import GiteaClient, VikunjaClient
|
||||
@@ -112,27 +110,6 @@ def validate_pr_title(pr_title: str, task_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
_VIKUNJA_LOOKUP_ATTEMPTS = 4
|
||||
_VIKUNJA_LOOKUP_BACKOFF = 5.0
|
||||
|
||||
|
||||
def _list_project_tasks(client: VikunjaClient, page: int) -> list[dict[str, Any]]:
|
||||
"""List Vikunja tasks with bounded retries for transient outages.
|
||||
|
||||
Implements REQ-1: during a Vikunja restart the tasks endpoint can briefly
|
||||
return 404/502; retry a few times so title validation rides out the window
|
||||
instead of stranding an otherwise-valid PR.
|
||||
"""
|
||||
for attempt in range(1, _VIKUNJA_LOOKUP_ATTEMPTS + 1):
|
||||
try:
|
||||
return list(client.list_project_tasks(VIKUNJA_PROJECT_ID, page=page, per_page=DEFAULT_PER_PAGE))
|
||||
except (APIError, requests.RequestException):
|
||||
if attempt == _VIKUNJA_LOOKUP_ATTEMPTS:
|
||||
raise
|
||||
time.sleep(_VIKUNJA_LOOKUP_BACKOFF * attempt)
|
||||
raise AssertionError("unreachable") # pragma: no cover
|
||||
|
||||
|
||||
def get_vikunja_task_title(task_id: str) -> str:
|
||||
"""Fetch the Vikunja task title for the given DEVX-N identifier.
|
||||
|
||||
@@ -147,7 +124,7 @@ def get_vikunja_task_title(task_id: str) -> str:
|
||||
client = VikunjaClient(VIKUNJA_API_URL, token)
|
||||
page = 1
|
||||
while True:
|
||||
tasks = _list_project_tasks(client, page)
|
||||
tasks = client.list_project_tasks(VIKUNJA_PROJECT_ID, page=page, per_page=DEFAULT_PER_PAGE)
|
||||
if not tasks:
|
||||
break
|
||||
matches = [t for t in tasks if t.get("identifier") == task_id]
|
||||
@@ -184,31 +161,22 @@ def validate_pr_title_matches_vikunja(pr_title: str, task_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
# Subjects carrying these tokens suppress post-merge CI entirely — strip them
|
||||
# before a commit message can become the squash title (see DEVX-179).
|
||||
_SKIP_CI_RE = re.compile(r"\s*\[(?:ci skip|skip ci|skip actions|actions skip|skip)\]", re.IGNORECASE)
|
||||
|
||||
|
||||
def extract_conventional_msg(commits: list[dict[str, Any]]) -> str:
|
||||
"""Extract the conventional commit message from PR commits.
|
||||
|
||||
Picks the highest-priority conventional commit message from the PR.
|
||||
Priority: feat > fix > refactor > docs > chore > other. The Gitea
|
||||
``/pulls/{n}/commits`` endpoint returns commits newest-first, so the
|
||||
first best-scoring subject wins equal-priority ties.
|
||||
``[skip ci]``-style tokens are stripped from every candidate so the
|
||||
merge title can never suppress the post-merge release pipeline.
|
||||
Priority: feat > fix > refactor > docs > chore > other.
|
||||
Falls back to the newest commit message if none match.
|
||||
"""
|
||||
priority = {"feat": 5, "fix": 4, "refactor": 3, "docs": 2, "chore": 1, "ci": 1, "style": 1, "test": 1}
|
||||
best_msg = ""
|
||||
best_score = 0
|
||||
for commit in commits:
|
||||
for commit in reversed(commits):
|
||||
commit_info = commit.get("commit", {})
|
||||
message = str(commit_info.get("message", "") if isinstance(commit_info, dict) else "").split("\n")[0]
|
||||
# Strip any leading task ID prefix (e.g. "OBL-INFRA-364: fix: ...") so
|
||||
# conventional commit matching works on the remainder; drop CI-skip
|
||||
# tokens so they never reach the merge title.
|
||||
stripped = _SKIP_CI_RE.sub("", _TASK_ID_PREFIX_RE.sub("", message)).strip()
|
||||
# conventional commit matching works on the remainder.
|
||||
stripped = _TASK_ID_PREFIX_RE.sub("", message)
|
||||
m = CONVENTIONAL_RE.match(stripped)
|
||||
if m:
|
||||
prefix = m.group(1).split("(")[0].strip() # e.g. "feat" from "feat(scope)"
|
||||
@@ -218,11 +186,11 @@ def extract_conventional_msg(commits: list[dict[str, Any]]) -> str:
|
||||
best_msg = stripped
|
||||
if best_msg:
|
||||
return best_msg
|
||||
# Fallback: the newest commit's first line (newest-first API order).
|
||||
# Fallback: use the newest commit's first line (strip task ID prefix if present)
|
||||
if commits:
|
||||
commit_info = commits[0].get("commit", {})
|
||||
commit_info = commits[-1].get("commit", {})
|
||||
raw = str(commit_info.get("message", "") if isinstance(commit_info, dict) else "").split("\n")[0]
|
||||
return _SKIP_CI_RE.sub("", _TASK_ID_PREFIX_RE.sub("", raw)).strip()
|
||||
return _TASK_ID_PREFIX_RE.sub("", raw)
|
||||
return ""
|
||||
|
||||
|
||||
@@ -304,28 +272,14 @@ def main(branch: str, pr_title: str, repo: str, pr_number: str) -> None:
|
||||
# Exit cleanly — the rebase triggers a new CI run that will retry.
|
||||
return
|
||||
else:
|
||||
hint = ""
|
||||
if e.status == 405:
|
||||
# Implements: REQ-2 — surface approval state. When the PR author
|
||||
# and the CI reviewer token map to the same Gitea user,
|
||||
# self-approval is rejected and the merge fails 405.
|
||||
try:
|
||||
reviews = client.get_pr_reviews(pr_num)
|
||||
if not any(r.get("state") == "APPROVED" for r in reviews):
|
||||
hint = _(
|
||||
"\nNo APPROVED review found on the PR. If the PR author and the"
|
||||
" CI reviewer token map to the same Gitea user, self-approval is"
|
||||
" rejected — approve the PR via a non-author account, then"
|
||||
" re-run the auto-merge job."
|
||||
)
|
||||
except APIError:
|
||||
pass
|
||||
msg = _(
|
||||
"Merge failed with HTTP {status}: {message}\nPlease check the PR is ready and you have merge rights.",
|
||||
status=e.status,
|
||||
message=e.message,
|
||||
)
|
||||
raise click.ClickException(msg + hint) from None
|
||||
raise click.ClickException(
|
||||
_(
|
||||
"Merge failed with HTTP {status}: {message}\n"
|
||||
"Please check the PR is ready and you have merge rights.",
|
||||
status=e.status,
|
||||
message=e.message,
|
||||
)
|
||||
) from None
|
||||
|
||||
click.echo(
|
||||
_(
|
||||
|
||||
@@ -1,213 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# Implements: REQ-2
|
||||
"""Check PR size and reject oversized PRs.
|
||||
|
||||
Enforces max lines changed and max files changed to keep PRs small
|
||||
and deployable. Generated/excluded files are not counted.
|
||||
|
||||
PRs with the ``refactoring`` label bypass the size check — large but
|
||||
legitimate refactoring PRs that touch many files in a coordinated way.
|
||||
|
||||
Usage:
|
||||
python -m devx.ci.check_pr_size --base origin/master --head HEAD \\
|
||||
--repo oblachno-oss/grm --pr-number 123
|
||||
|
||||
In CI, pass ``--github-output`` to set ``pr-size-ok`` and ``pr-size-detail``
|
||||
for downstream steps.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess # nosec B404
|
||||
|
||||
import click
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from devx.api_clients import GiteaClient
|
||||
from devx.ci._shared import write_github_output
|
||||
from devx.config import GITEA_API_URL
|
||||
from devx.i18n import _
|
||||
from devx.tokens import get_ci_token
|
||||
|
||||
load_dotenv()
|
||||
|
||||
# Files/patterns excluded from size counting (generated, badges, locks, etc.)
|
||||
DEFAULT_EXCLUDED_PATTERNS = [
|
||||
"CHANGELOG.md",
|
||||
"README.md",
|
||||
"docs/index.md",
|
||||
"docs/plans/*",
|
||||
"*.svg",
|
||||
"uv.lock",
|
||||
"poetry.lock",
|
||||
"Pipfile.lock",
|
||||
"package-lock.json",
|
||||
"yarn.lock",
|
||||
"go.sum",
|
||||
]
|
||||
|
||||
DEFAULT_MAX_LINES = 500
|
||||
DEFAULT_MAX_FILES = 10
|
||||
REFACTORING_LABEL = "refactoring"
|
||||
|
||||
|
||||
def has_refactoring_label(repo: str, pr_number: int) -> bool:
|
||||
"""Check if a PR has the 'refactoring' label (bypasses size check)."""
|
||||
try:
|
||||
token = get_ci_token()
|
||||
owner, repo_name = repo.split("/", 1)
|
||||
client = GiteaClient(GITEA_API_URL, token, owner, repo_name)
|
||||
pr = client.get_pr(pr_number)
|
||||
labels = pr.get("labels", [])
|
||||
return any(label.get("name") == REFACTORING_LABEL for label in labels)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def get_diff_stats(base: str, head: str) -> list[tuple[str, int, int]]:
|
||||
"""Get per-file diff stats (additions, deletions) between base and head.
|
||||
|
||||
Returns a list of (filename, additions, deletions) tuples.
|
||||
"""
|
||||
result = subprocess.run( # nosec B603 B607
|
||||
["git", "diff", "--numstat", base, head],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise click.ClickException(_("git diff --numstat failed: {stderr}", stderr=result.stderr.strip()))
|
||||
stats: list[tuple[str, int, int]] = []
|
||||
for line in result.stdout.strip().split("\n"):
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split("\t")
|
||||
if len(parts) != 3:
|
||||
continue
|
||||
additions_s, deletions_s, filename = parts
|
||||
# Binary files show "-" for additions/deletions
|
||||
additions = int(additions_s) if additions_s.isdigit() else 0
|
||||
deletions = int(deletions_s) if deletions_s.isdigit() else 0
|
||||
stats.append((filename, additions, deletions))
|
||||
return stats
|
||||
|
||||
|
||||
def is_excluded(filename: str, excluded_patterns: list[str]) -> bool:
|
||||
"""Check if a filename matches any excluded pattern."""
|
||||
from fnmatch import fnmatch
|
||||
|
||||
return any(fnmatch(filename, pat) for pat in excluded_patterns)
|
||||
|
||||
|
||||
def check_size(
|
||||
stats: list[tuple[str, int, int]],
|
||||
max_lines: int,
|
||||
max_files: int,
|
||||
excluded_patterns: list[str],
|
||||
) -> tuple[bool, str]:
|
||||
"""Check diff stats against limits.
|
||||
|
||||
Returns (is_ok, detail_message).
|
||||
"""
|
||||
included = [(f, a, d) for f, a, d in stats if not is_excluded(f, excluded_patterns)]
|
||||
total_lines = sum(a + d for _, a, d in included)
|
||||
total_files = len(included)
|
||||
|
||||
if total_files == 0:
|
||||
return True, "No non-excluded files changed"
|
||||
|
||||
if total_files > max_files:
|
||||
return False, _(
|
||||
"PR has {file_count} files changed (max {max_files}). Excluded: {excluded_count} files.",
|
||||
file_count=total_files,
|
||||
max_files=max_files,
|
||||
excluded_count=len(stats) - total_files,
|
||||
)
|
||||
|
||||
if total_lines > max_lines:
|
||||
return False, _(
|
||||
"PR has {line_count} lines changed (max {max_lines}). Excluded: {excluded_count} files.",
|
||||
line_count=total_lines,
|
||||
max_lines=max_lines,
|
||||
excluded_count=len(stats) - total_files,
|
||||
)
|
||||
|
||||
return True, _(
|
||||
"PR size OK: {file_count} files, {line_count} lines (max {max_files} files, {max_lines} lines).",
|
||||
file_count=total_files,
|
||||
line_count=total_lines,
|
||||
max_files=max_files,
|
||||
max_lines=max_lines,
|
||||
)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--base", default="origin/master", help=_("Base ref for diff"))
|
||||
@click.option("--head", default="HEAD", help=_("Head ref for diff"))
|
||||
@click.option(
|
||||
"--max-lines",
|
||||
type=int,
|
||||
default=DEFAULT_MAX_LINES,
|
||||
help=_("Max lines changed (excluded files not counted)"),
|
||||
)
|
||||
@click.option(
|
||||
"--max-files",
|
||||
type=int,
|
||||
default=DEFAULT_MAX_FILES,
|
||||
help=_("Max files changed (excluded files not counted)"),
|
||||
)
|
||||
@click.option(
|
||||
"--github-output",
|
||||
"github_output",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help=_("Write results to $GITHUB_OUTPUT"),
|
||||
)
|
||||
@click.option(
|
||||
"--excluded",
|
||||
"excluded",
|
||||
multiple=True,
|
||||
help=_("Additional excluded patterns (in addition to defaults)"),
|
||||
)
|
||||
@click.option("--repo", default=None, help=_("Repo (owner/name) for label check"))
|
||||
@click.option("--pr-number", type=int, default=None, help=_("PR number for label check"))
|
||||
def cli(
|
||||
base: str,
|
||||
head: str,
|
||||
max_lines: int,
|
||||
max_files: int,
|
||||
github_output: bool,
|
||||
excluded: tuple[str, ...],
|
||||
repo: str | None,
|
||||
pr_number: int | None,
|
||||
) -> None:
|
||||
"""Check PR size and reject oversized PRs."""
|
||||
# Check for refactoring label bypass
|
||||
if repo and pr_number and has_refactoring_label(repo, pr_number):
|
||||
detail = _("PR has 'refactoring' label — size check bypassed.")
|
||||
if github_output:
|
||||
write_github_output("pr-size-ok", "true")
|
||||
write_github_output("pr-size-detail", detail)
|
||||
click.echo(f"[pr-size] {detail}")
|
||||
return
|
||||
|
||||
excluded_patterns = list(DEFAULT_EXCLUDED_PATTERNS) + list(excluded)
|
||||
stats = get_diff_stats(base, head)
|
||||
is_ok, detail = check_size(stats, max_lines, max_files, excluded_patterns)
|
||||
|
||||
if github_output:
|
||||
write_github_output("pr-size-ok", "true" if is_ok else "false")
|
||||
write_github_output("pr-size-detail", detail)
|
||||
|
||||
if is_ok:
|
||||
click.echo(f"[pr-size] {detail}")
|
||||
else:
|
||||
click.echo(f"[pr-size] FAILED: {detail}", err=True)
|
||||
click.echo("", err=True)
|
||||
click.echo("Oversized PRs cannot be reliably reviewed or deployed independently.", err=True)
|
||||
click.echo("Split your work into smaller PRs, each addressing one concern.", err=True)
|
||||
raise click.ClickException(_("PR size check failed."))
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
cli()
|
||||
@@ -1,548 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# Implements: REQ-5
|
||||
"""Auto-create an infra PR to bump a pinned dependency version.
|
||||
|
||||
After grm or sso-bridge publishes a new package version, this module
|
||||
creates a PR in the infra repo to bump the pinned version in
|
||||
``pyproject.toml`` or ``ansible/group_vars/all/images.yml``.
|
||||
|
||||
Reuses ``devx.tools.create_pr`` for PR creation and Vikunja task linking.
|
||||
|
||||
Usage:
|
||||
python -m devx.ci.create_dependency_pr \
|
||||
--repo oblachno/infra \
|
||||
--package grm \
|
||||
--new-version 0.5.2 \
|
||||
--source-repo oblachno/grm \
|
||||
--source-run-id 12345
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess # nosec B404
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import requests
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from devx.api_clients import GiteaClient
|
||||
from devx.config import GITEA_API_URL, VIKUNJA_API_URL, VIKUNJA_PROJECT_ID
|
||||
from devx.exceptions import APIError
|
||||
from devx.i18n import _
|
||||
from devx.tokens import get_ci_token, get_vikunja_token
|
||||
from devx.tools.create_pr import find_existing_pr
|
||||
|
||||
load_dotenv()
|
||||
|
||||
# Where infra pins dependency versions
|
||||
PYPROJECT_PATH = "pyproject.toml"
|
||||
IMAGES_YML_PATH = "ansible/group_vars/all/images.yml"
|
||||
ROLE_DEFAULTS_PATH = "ansible/roles/sso_bridge/defaults/main.yml"
|
||||
|
||||
|
||||
def find_pinned_version(package: str, file_path: str) -> str | None:
|
||||
"""Find the currently pinned version of a package in a file.
|
||||
|
||||
Looks for patterns like:
|
||||
- ``"grm @ git+...@v0.5.1"``
|
||||
- ``grm = "0.5.1"``
|
||||
- ``grm_version: "0.5.1"``
|
||||
- ``grm_image_version: "0.5.1"``
|
||||
"""
|
||||
path = Path(file_path)
|
||||
if not path.exists():
|
||||
return None
|
||||
content = path.read_text(encoding="utf-8")
|
||||
# Match various pinning patterns
|
||||
patterns = [
|
||||
rf"{package}\s*@\s*git\+[^@]+@v?([\d.]+)", # pip: package @ git+url@vX.Y.Z
|
||||
rf'{package}\s*=\s*"([\d.]+)"', # pyproject: package = "X.Y.Z"
|
||||
rf'{package}_version:\s*"([\d.]+)"', # ansible vars: package_version: "X.Y.Z"
|
||||
rf'{package}_image_version:\s*"([\d.]+)"', # ansible vars: package_image_version: "X.Y.Z"
|
||||
]
|
||||
for pat in patterns:
|
||||
match = re.search(pat, content)
|
||||
if match:
|
||||
return match.group(1)
|
||||
return None
|
||||
|
||||
|
||||
def update_pinned_version(file_path: str, package: str, old_version: str, new_version: str) -> bool:
|
||||
"""Update the pinned version in a file. Returns True if changed."""
|
||||
path = Path(file_path)
|
||||
if not path.exists():
|
||||
return False
|
||||
content = path.read_text(encoding="utf-8")
|
||||
# Replace old version with new version in package-related lines
|
||||
patterns = [
|
||||
(rf"({package}\s*@\s*git\+[^@]+@v?){old_version}", rf"\g<1>{new_version}"),
|
||||
(rf'({package}\s*=\s*"){old_version}(")', rf"\g<1>{new_version}\g<2>"),
|
||||
(rf'({package}_version:\s*"){old_version}(")', rf"\g<1>{new_version}\g<2>"),
|
||||
(rf'({package}_image_version:\s*"){old_version}(")', rf"\g<1>{new_version}\g<2>"),
|
||||
]
|
||||
new_content = content
|
||||
changed = False
|
||||
for pat, replacement in patterns:
|
||||
new_content, n = re.subn(pat, replacement, new_content)
|
||||
if n > 0:
|
||||
changed = True
|
||||
if changed:
|
||||
path.write_text(new_content, encoding="utf-8")
|
||||
return changed
|
||||
|
||||
|
||||
_MANIFEST_ACCEPT = (
|
||||
"application/vnd.oci.image.index.v1+json, "
|
||||
"application/vnd.docker.distribution.manifest.list.v2+json, "
|
||||
"application/vnd.oci.image.manifest.v1+json, "
|
||||
"application/vnd.docker.distribution.manifest.v2+json"
|
||||
)
|
||||
|
||||
|
||||
# Implements: REQ-1 — existence check via packages API (keeps the 404-retry
|
||||
# semantics for the publish race); the pullable digest is then resolved via
|
||||
# the registry v2 API's Docker-Content-Digest header.
|
||||
def resolve_container_digest(api_url: str, owner: str, name: str, tag: str, token: str, timeout_s: int = 0) -> str:
|
||||
"""Verify the container tag exists, then resolve its pullable OCI digest.
|
||||
|
||||
The packages API proves the tag was published (and 404s while the
|
||||
producer's image-build workflow races us — ``timeout_s`` retries every
|
||||
15s). The packages-API ``manifest.json`` blob sha256 is NOT pullable
|
||||
via ``repo@sha256:...``, so the digest comes from the registry v2
|
||||
``Docker-Content-Digest`` header instead.
|
||||
"""
|
||||
url = f"{api_url}/packages/{owner}/container/{name}/{tag}/files"
|
||||
headers = {"Authorization": f"token {token}"}
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while True:
|
||||
try:
|
||||
resp = requests.get(url, headers=headers, timeout=30) # nosec B310
|
||||
resp.raise_for_status()
|
||||
except requests.HTTPError as e:
|
||||
if e.response is not None and e.response.status_code == 404 and time.monotonic() < deadline:
|
||||
click.echo(
|
||||
_(
|
||||
"[dep-pr] {owner}/{name}:{tag} not published yet — retrying.",
|
||||
owner=owner,
|
||||
name=name,
|
||||
tag=tag,
|
||||
)
|
||||
)
|
||||
time.sleep(15)
|
||||
continue
|
||||
status = e.response.status_code if e.response is not None else "?"
|
||||
raise click.ClickException(
|
||||
_(
|
||||
"Container artifact {owner}/{name}:{tag} not found or unreadable (HTTP {status}). "
|
||||
"Refusing to open a dependency PR for an unpublished artifact.",
|
||||
owner=owner,
|
||||
name=name,
|
||||
tag=tag,
|
||||
status=status,
|
||||
)
|
||||
) from e
|
||||
except requests.RequestException as e:
|
||||
raise click.ClickException(
|
||||
_(
|
||||
"Registry lookup failed for {owner}/{name}:{tag}: {error}",
|
||||
owner=owner,
|
||||
name=name,
|
||||
tag=tag,
|
||||
error=e,
|
||||
)
|
||||
) from e
|
||||
break
|
||||
return _resolve_registry_digest(api_url, owner, name, tag, token)
|
||||
|
||||
|
||||
# Implements: REQ-2 — v2 digest resolution fails closed: non-2xx, missing
|
||||
# token, or absent Docker-Content-Digest all raise; a blob sha256 is never
|
||||
# recorded as a pullable digest.
|
||||
def _resolve_registry_digest(api_url: str, owner: str, name: str, tag: str, token: str) -> str:
|
||||
registry = api_url.removesuffix("/api/v1").removesuffix("/")
|
||||
repo = f"{owner}/{name}"
|
||||
try:
|
||||
tok_resp = requests.get( # nosec B310
|
||||
f"{registry}/v2/token",
|
||||
params={"service": "container_registry", "scope": f"repository:{repo}:pull"},
|
||||
auth=("ci", token),
|
||||
timeout=30,
|
||||
)
|
||||
tok_resp.raise_for_status()
|
||||
bearer = tok_resp.json().get("token", "")
|
||||
man_resp = requests.get( # nosec B310
|
||||
f"{registry}/v2/{repo}/manifests/{tag}",
|
||||
headers={"Authorization": f"Bearer {bearer}", "Accept": _MANIFEST_ACCEPT},
|
||||
timeout=30,
|
||||
)
|
||||
man_resp.raise_for_status()
|
||||
except requests.RequestException as e:
|
||||
status = getattr(getattr(e, "response", None), "status_code", "?")
|
||||
raise click.ClickException(
|
||||
_(
|
||||
"Registry v2 digest lookup failed for {repo}:{tag} (HTTP {status}): {error}",
|
||||
repo=repo,
|
||||
tag=tag,
|
||||
status=status,
|
||||
error=e,
|
||||
)
|
||||
) from e
|
||||
digest = man_resp.headers.get("Docker-Content-Digest", "")
|
||||
if not digest:
|
||||
raise click.ClickException(
|
||||
_("Registry returned no Docker-Content-Digest for {repo}:{tag}.", repo=repo, tag=tag)
|
||||
)
|
||||
return digest
|
||||
|
||||
|
||||
def update_manifest(file_path: str, section: str, fields: dict[str, str]) -> bool:
|
||||
"""Update a release-manifest JSON section in place. Returns True if changed.
|
||||
|
||||
Implements REQ-1: manifest fields record the immutable release contract
|
||||
(version, git ref, image tag, resolved OCI digest) in the target repo.
|
||||
"""
|
||||
import json as _json
|
||||
|
||||
path = Path(file_path)
|
||||
if not path.exists():
|
||||
raise click.ClickException(_("Manifest file not found: {file}", file=file_path))
|
||||
try:
|
||||
manifest = _json.loads(path.read_text(encoding="utf-8"))
|
||||
except _json.JSONDecodeError as e:
|
||||
raise click.ClickException(_("Manifest file {file} is not valid JSON: {error}", file=file_path, error=e)) from e
|
||||
existing = manifest.get(section)
|
||||
if not isinstance(existing, dict):
|
||||
raise click.ClickException(
|
||||
_("Manifest file {file} has no object section {section}", file=file_path, section=section)
|
||||
)
|
||||
changed = False
|
||||
for k, v in fields.items():
|
||||
if existing.get(k) != v:
|
||||
existing[k] = v
|
||||
changed = True
|
||||
if changed:
|
||||
path.write_text(_json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
|
||||
return changed
|
||||
|
||||
|
||||
def read_manifest_version(file_path: str, section: str) -> str | None:
|
||||
"""Read the currently pinned version from a release manifest section."""
|
||||
import json as _json
|
||||
|
||||
path = Path(file_path)
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
manifest = _json.loads(path.read_text(encoding="utf-8"))
|
||||
except _json.JSONDecodeError:
|
||||
return None
|
||||
existing = manifest.get(section)
|
||||
if isinstance(existing, dict):
|
||||
version = existing.get("version")
|
||||
return str(version) if version is not None else None
|
||||
return None
|
||||
|
||||
|
||||
def create_vikunja_task(title: str, description: str, project_id: int = 0) -> str | None:
|
||||
"""Create a Vikunja task and return its identifier (e.g., OBL-INFRA-531)."""
|
||||
try:
|
||||
token = get_vikunja_token()
|
||||
except click.ClickException:
|
||||
return None
|
||||
|
||||
from devx.api_clients import VikunjaClient
|
||||
|
||||
client = VikunjaClient(VIKUNJA_API_URL, token)
|
||||
task = client.create_task(project_id or VIKUNJA_PROJECT_ID, title=title, description=description)
|
||||
return str(task.get("identifier", ""))
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--repo", default="oblachno/infra", help=_("Target repo (owner/name) to create PR in"))
|
||||
@click.option("--package", required=True, help=_("Package name to bump (e.g., grm, sso-bridge)"))
|
||||
@click.option("--new-version", required=True, help=_("New version to pin"))
|
||||
@click.option("--source-repo", required=True, help=_("Source repo that published (owner/name)"))
|
||||
@click.option("--source-run-id", default="", help=_("CI run ID that triggered the publish"))
|
||||
@click.option(
|
||||
"--manifest",
|
||||
"manifest_path",
|
||||
default="",
|
||||
help=_(
|
||||
"Path to a release-manifest JSON in the target repo. When set, the PR updates "
|
||||
"the manifest section named after --package instead of a regex version bump."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--verify-container",
|
||||
default="",
|
||||
help=_(
|
||||
"Container to verify before opening the PR (owner/name). Resolves the OCI "
|
||||
"digest of the tag matching --new-version and records it in the manifest."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--source-ref",
|
||||
default="",
|
||||
help=_("Git ref of the producer release (default: v<new-version>), recorded in the manifest."),
|
||||
)
|
||||
@click.option(
|
||||
"--container-tag",
|
||||
default="",
|
||||
help=_(
|
||||
"Container tag to verify with --verify-container (default: --new-version). "
|
||||
"Use when the image tag differs from the release version, e.g. a package "
|
||||
"__version__ tag vs a release git tag."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--verify-timeout",
|
||||
type=int,
|
||||
default=600,
|
||||
help=_(
|
||||
"Seconds to keep retrying --verify-container while the artifact returns 404 "
|
||||
"(the image build races this step). 0 disables retries."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--task-project-id",
|
||||
type=int,
|
||||
default=0,
|
||||
help=_(
|
||||
"Vikunja project for the tracking task (default: DEVX_VIKUNJA_PROJECT_ID). "
|
||||
"Use the target repo's project so the generated branch/spec satisfy its validation."
|
||||
),
|
||||
)
|
||||
@click.option("--dry-run", is_flag=True, default=False, help=_("Show what would be done without creating PR"))
|
||||
def cli(
|
||||
repo: str,
|
||||
package: str,
|
||||
new_version: str,
|
||||
source_repo: str,
|
||||
source_run_id: str,
|
||||
manifest_path: str,
|
||||
verify_container: str,
|
||||
source_ref: str,
|
||||
container_tag: str,
|
||||
verify_timeout: int,
|
||||
task_project_id: int,
|
||||
dry_run: bool,
|
||||
) -> None:
|
||||
"""Create an infra PR to bump a pinned dependency version."""
|
||||
token = get_ci_token()
|
||||
if "/" not in repo:
|
||||
raise click.ClickException(_("Invalid repo format: {repo}", repo=repo))
|
||||
owner, repo_name = repo.split("/", 1)
|
||||
client = GiteaClient(GITEA_API_URL, token, owner, repo_name)
|
||||
|
||||
# Implements: REQ-1 — verify the producer artifact exists and resolve its
|
||||
# digest before any branch/PR work begins.
|
||||
image_digest = ""
|
||||
if verify_container:
|
||||
if "/" not in verify_container:
|
||||
raise click.ClickException(
|
||||
_("Invalid container format: {container} (expected owner/name)", container=verify_container)
|
||||
)
|
||||
c_owner, c_name = verify_container.split("/", 1)
|
||||
image_tag = container_tag or new_version
|
||||
image_digest = resolve_container_digest(GITEA_API_URL, c_owner, c_name, image_tag, token, verify_timeout)
|
||||
click.echo(
|
||||
_(
|
||||
"[dep-pr] Verified {container}:{version} -> {digest}",
|
||||
container=verify_container,
|
||||
version=image_tag,
|
||||
digest=image_digest,
|
||||
)
|
||||
)
|
||||
|
||||
# Clone the target repo — this tool runs from the *producer* repo's CI,
|
||||
# so every file lookup and git operation must happen inside a clone of
|
||||
# the target repo, not the producer checkout in CWD.
|
||||
workdir = Path(tempfile.mkdtemp(prefix="dep-pr-"))
|
||||
clone_url = f"{GITEA_API_URL.removesuffix('/api/v1')}/{repo}.git"
|
||||
auth_cfg = f"http.extraHeader=Authorization: token {token}"
|
||||
subprocess.run( # nosec B603 B607
|
||||
["git", "-c", auth_cfg, "clone", "--depth", "50", clone_url, str(workdir)],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
# Find current pinned version
|
||||
old_version = None
|
||||
changed_file = None
|
||||
if manifest_path:
|
||||
old_version = read_manifest_version(str(workdir / manifest_path), package)
|
||||
changed_file = manifest_path if old_version else None
|
||||
else:
|
||||
for f in [PYPROJECT_PATH, IMAGES_YML_PATH, ROLE_DEFAULTS_PATH]:
|
||||
old_version = find_pinned_version(package, str(workdir / f))
|
||||
if old_version:
|
||||
changed_file = f
|
||||
break
|
||||
|
||||
if not old_version:
|
||||
click.echo(_("[dep-pr] Could not find pinned version for {pkg} in infra repo.", pkg=package))
|
||||
if dry_run:
|
||||
return
|
||||
raise click.ClickException(_("Could not find pinned version for {pkg}", pkg=package))
|
||||
|
||||
if old_version == new_version:
|
||||
click.echo(_("[dep-pr] {pkg} already at {version} — no PR needed.", pkg=package, version=new_version))
|
||||
return
|
||||
|
||||
click.echo(
|
||||
_(
|
||||
"[dep-pr] Bumping {pkg} from {old} to {new} in {file}",
|
||||
pkg=package,
|
||||
old=old_version,
|
||||
new=new_version,
|
||||
file=changed_file,
|
||||
)
|
||||
)
|
||||
|
||||
if dry_run:
|
||||
click.echo(f"[dep-pr] DRY RUN: would update {changed_file} and create PR")
|
||||
return
|
||||
|
||||
# Implements: REQ-1 — create the tracking task in the *target* repo's
|
||||
# Vikunja project so its identifier satisfies the target's branch/PR-title
|
||||
# validation (e.g., OBL-INFRA-N for oblachno/infra).
|
||||
task_title = f"Bump {package} to {new_version}"
|
||||
task_desc = (
|
||||
f"<p>Auto-created dependency bump PR.</p>"
|
||||
f"<p>Package: {package}</p>"
|
||||
f"<p>Version: {old_version} → {new_version}</p>"
|
||||
f"<p>Source: {source_repo} (run #{source_run_id})</p>"
|
||||
)
|
||||
task_id = create_vikunja_task(task_title, task_desc, task_project_id)
|
||||
|
||||
# Create a branch — embed the task ID so target-repo validation accepts it.
|
||||
branch_name = f"deps/{task_id}-{package}-{new_version}" if task_id else f"deps/{package}-{new_version}"
|
||||
base_branch = "master"
|
||||
|
||||
# Check for existing PR (reuse from tools.create_pr)
|
||||
existing = find_existing_pr(client, branch_name)
|
||||
if existing:
|
||||
click.echo(_("[dep-pr] PR already exists: #{number}", number=existing.get("number", "?")))
|
||||
return
|
||||
|
||||
# Create branch via API
|
||||
try:
|
||||
# Implements: REQ-1 — Gitea lacks POST /git/refs; create the branch
|
||||
# from master via the branches API.
|
||||
client._request(
|
||||
"POST",
|
||||
"/branches",
|
||||
json={"new_branch_name": branch_name, "old_branch_name": base_branch},
|
||||
)
|
||||
except APIError as e:
|
||||
if "already exists" in str(e).lower():
|
||||
click.echo(f"[dep-pr] Branch {branch_name} already exists")
|
||||
else:
|
||||
raise click.ClickException(_("Failed to create branch: {error}", error=str(e))) from None
|
||||
|
||||
# Check out the API-created branch inside the target clone. A plain
|
||||
# fetch only populates FETCH_HEAD — fetch into the remote-tracking ref
|
||||
# and force-create the local branch from it.
|
||||
subprocess.run( # nosec B603 B607
|
||||
["git", "fetch", "origin", f"{branch_name}:refs/remotes/origin/{branch_name}"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
cwd=workdir,
|
||||
)
|
||||
subprocess.run( # nosec B603 B607
|
||||
["git", "checkout", "-B", branch_name, f"origin/{branch_name}"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
cwd=workdir,
|
||||
)
|
||||
|
||||
if manifest_path:
|
||||
fields = {
|
||||
"version": new_version,
|
||||
"git_ref": source_ref or f"v{new_version}",
|
||||
"image_tag": container_tag or new_version,
|
||||
"updated_at": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
}
|
||||
if image_digest:
|
||||
fields["image_digest"] = image_digest
|
||||
if source_run_id:
|
||||
fields["source_run_id"] = source_run_id
|
||||
if not update_manifest(str(workdir / manifest_path), package, fields):
|
||||
raise click.ClickException(_("Failed to update {file}", file=manifest_path))
|
||||
elif not changed_file or not update_pinned_version(str(workdir / changed_file), package, old_version, new_version):
|
||||
raise click.ClickException(_("Failed to update {file}", file=changed_file))
|
||||
|
||||
assert changed_file is not None # nosec B101 — narrowed by the early exit above
|
||||
add_files = [changed_file]
|
||||
if task_id:
|
||||
# Implements: REQ-1 — spec-driven validation requires a spec file.
|
||||
spec_rel = f"docs/specs/{task_id}.md"
|
||||
spec_file = workdir / spec_rel
|
||||
spec_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
spec_file.write_text(
|
||||
f"# {task_id}: {task_title}\n\n"
|
||||
f"## Problem\n\n"
|
||||
f"{source_repo} released {package} {new_version}; this repo pins {old_version}.\n\n"
|
||||
f"## Approach\n\n"
|
||||
f"REQ-1: Update `{changed_file}` to pin {package} {new_version} "
|
||||
f"(auto-generated dependency PR).\n\n"
|
||||
f"## Test Plan\n\n"
|
||||
f"- Producer release CI verified the artifact "
|
||||
f"({source_repo} run #{source_run_id or 'n/a'}).\n\n"
|
||||
f"## Deploy Plan\n\n"
|
||||
f"Merge updates the pin; the next deploy applies it.\n\n"
|
||||
f"## Rollback Plan\n\n"
|
||||
f"Revert the pin bump.\n\n"
|
||||
f"## Acceptance Criteria\n\n"
|
||||
f"- [x] REQ-1: `{changed_file}` pins {package} {new_version}.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
add_files.append(spec_rel)
|
||||
subprocess.run(["git", "add", *add_files], check=True, cwd=workdir) # nosec B603 B607
|
||||
commit_msg = f"deps: bump {package} from {old_version} to {new_version}"
|
||||
# Implements: REQ-1 — the fresh clone has no git identity in CI
|
||||
# containers; supply it inline (same bot identity as push_badges).
|
||||
subprocess.run( # nosec B603 B607
|
||||
[
|
||||
"git",
|
||||
"-c",
|
||||
"user.name=gitea-actions-bot",
|
||||
"-c",
|
||||
"user.email=actions@oblachno.fyi",
|
||||
"commit",
|
||||
"-m",
|
||||
commit_msg,
|
||||
],
|
||||
check=True,
|
||||
cwd=workdir,
|
||||
)
|
||||
subprocess.run( # nosec B603 B607
|
||||
["git", "-c", auth_cfg, "push", "origin", branch_name],
|
||||
check=True,
|
||||
cwd=workdir,
|
||||
)
|
||||
|
||||
# Create PR directly (dependency PRs have custom titles, not Vikunja-derived)
|
||||
pr_title = f"{task_id}: {task_title}" if task_id else task_title
|
||||
pr_body = (
|
||||
f"## Dependency Bump\n\n"
|
||||
f"Bumps **{package}** from `{old_version}` to `{new_version}`.\n\n"
|
||||
f"- **Source**: {source_repo}\n"
|
||||
f"- **Triggered by**: CI run #{source_run_id}\n"
|
||||
f"- **Changed file**: `{changed_file}`\n\n"
|
||||
f"This PR was auto-created by `devx.ci.create_dependency_pr`.\n"
|
||||
)
|
||||
if task_id:
|
||||
pr_body += f"\nCloses {task_id}"
|
||||
|
||||
pr = client.create_pr(title=pr_title, head=branch_name, base=base_branch, body=pr_body)
|
||||
click.echo(_("[dep-pr] Created PR #{number}: {title}", number=pr.get("number", "?"), title=pr_title))
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
cli()
|
||||
@@ -44,6 +44,7 @@ REQUIRED_SCRIPTS = [
|
||||
"auto_merge.py",
|
||||
"release.py",
|
||||
"publish.py",
|
||||
"pr_review.py",
|
||||
"notify_failure.py",
|
||||
"post_merge.py",
|
||||
"classify_changes.py",
|
||||
@@ -51,6 +52,7 @@ REQUIRED_SCRIPTS = [
|
||||
"detect_release_commit.py",
|
||||
"push_badges.py",
|
||||
"distribute_molecule.py",
|
||||
"molecule_ci_guard.py",
|
||||
"validate_commit_msg.py",
|
||||
]
|
||||
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# Implements: REQ-3
|
||||
"""Detect changed Ansible roles and output fast molecule test commands.
|
||||
|
||||
Reuses ``devx.molecule.molecule_changed`` for role detection (which handles
|
||||
playbook→role mapping and shared infrastructure paths).
|
||||
|
||||
Fast molecule = full ``molecule test`` for every scenario of each changed
|
||||
role, on the scenario's configured platform. "Fast" means *scoped* (only
|
||||
affected roles, single platform) — never skipped phases: create, converge,
|
||||
idempotence (when the scenario defines it), verify, and destroy all run,
|
||||
exactly as ``scripts/run_molecule_scenario.py`` executes them in CI.
|
||||
|
||||
Used in pre-merge CI to get quick feedback on Ansible changes without
|
||||
running the full all-roles/all-platforms suite (which runs nightly).
|
||||
|
||||
Usage:
|
||||
python -m devx.ci.fast_molecule --base origin/master --head HEAD
|
||||
|
||||
Outputs the list of changed roles and the molecule commands CI runs.
|
||||
In CI, pass ``--github-output`` to set ``fast-molecule-roles`` (space-
|
||||
separated) and ``fast-molecule-needed`` (true/false) for downstream steps.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from devx.ci._shared import write_github_output
|
||||
from devx.i18n import _
|
||||
from devx.molecule.molecule_changed import detect_changed_roles, get_changed_files
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def get_molecule_scenarios(role_name: str, roles_dir: str = "ansible/roles") -> list[str]:
|
||||
"""Get list of molecule scenario names for a role."""
|
||||
mol_dir = Path(roles_dir) / role_name / "molecule"
|
||||
if not mol_dir.is_dir():
|
||||
return []
|
||||
scenarios = []
|
||||
for p in mol_dir.iterdir():
|
||||
if p.is_dir() and (p / "molecule.yml").exists():
|
||||
scenarios.append(p.name)
|
||||
return sorted(scenarios)
|
||||
|
||||
|
||||
def build_molecule_commands(
|
||||
roles: set[str],
|
||||
roles_dir: str = "ansible/roles",
|
||||
platform: str = "ubuntu-2604",
|
||||
) -> list[str]:
|
||||
"""Build the molecule commands CI executes for changed roles.
|
||||
|
||||
Emits exactly what ``scripts/run_molecule_scenario.py`` runs:
|
||||
``molecule test -s <scenario>`` — the full sequence (create, converge,
|
||||
idempotence, verify, destroy). The ``platform`` argument is accepted
|
||||
for interface stability but is informational: the scenario's
|
||||
``molecule.yml`` selects the platform, and CI distributes scenarios so
|
||||
each runs on a single platform.
|
||||
"""
|
||||
commands: list[str] = []
|
||||
for role in sorted(roles):
|
||||
for scenario in get_molecule_scenarios(role, roles_dir):
|
||||
commands.append(f"molecule test -s {scenario}")
|
||||
return commands
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--base", default="origin/master", help=_("Base ref for diff"))
|
||||
@click.option("--head", default="HEAD", help=_("Head ref for diff"))
|
||||
@click.option("--roles-dir", default="ansible/roles", help=_("Directory containing Ansible roles"))
|
||||
@click.option("--platform", default="ubuntu-2604", help=_("Single platform to test against"))
|
||||
@click.option(
|
||||
"--github-output",
|
||||
"github_output",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help=_("Write results to $GITHUB_OUTPUT"),
|
||||
)
|
||||
def cli(
|
||||
base: str,
|
||||
head: str,
|
||||
roles_dir: str,
|
||||
platform: str,
|
||||
github_output: bool,
|
||||
) -> None:
|
||||
"""Detect changed roles and output fast molecule test commands."""
|
||||
# Use molecule_changed for role detection (handles playbooks, shared infra)
|
||||
files = get_changed_files(base)
|
||||
if not files:
|
||||
click.echo("[fast-molecule] No files changed.")
|
||||
if github_output:
|
||||
write_github_output("fast-molecule-needed", "false")
|
||||
write_github_output("fast-molecule-roles", "")
|
||||
return
|
||||
|
||||
roles = detect_changed_roles(files, roles_dir)
|
||||
if not roles:
|
||||
click.echo("[fast-molecule] No Ansible roles changed.")
|
||||
if github_output:
|
||||
write_github_output("fast-molecule-needed", "false")
|
||||
write_github_output("fast-molecule-roles", "")
|
||||
return
|
||||
|
||||
commands = build_molecule_commands(roles, roles_dir, platform)
|
||||
|
||||
if github_output:
|
||||
write_github_output("fast-molecule-needed", "true" if commands else "false")
|
||||
write_github_output("fast-molecule-roles", " ".join(sorted(roles)))
|
||||
|
||||
click.echo(_("[fast-molecule] Changed roles: {roles}", roles=", ".join(sorted(roles))))
|
||||
if not commands:
|
||||
click.echo("[fast-molecule] No molecule scenarios found for changed roles.")
|
||||
return
|
||||
|
||||
click.echo(f"[fast-molecule] {len(commands)} scenario(s) to run:")
|
||||
for cmd in commands:
|
||||
click.echo(f" {cmd}")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
cli()
|
||||
@@ -1,9 +1,10 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run integration tests with cross-runner failure detection.
|
||||
|
||||
Wraps ``pytest`` with Gitea API polling. If any other integration-tests
|
||||
matrix runner reports failure, the current pytest subprocess is killed
|
||||
and this runner exits early with code 1.
|
||||
Wraps ``pytest`` with the same Gitea API polling mechanism used by
|
||||
``molecule_ci_guard``. If any other integration-tests matrix runner
|
||||
reports failure, the current pytest subprocess is killed and this runner
|
||||
exits early with code 1.
|
||||
|
||||
Usage::
|
||||
|
||||
@@ -34,62 +35,17 @@ import threading
|
||||
import time
|
||||
|
||||
import click
|
||||
import requests
|
||||
|
||||
from devx.config import REPO_NAME, REPO_OWNER
|
||||
from devx.i18n import _
|
||||
from devx.molecule.molecule_ci_guard import (
|
||||
poll_for_other_failures,
|
||||
)
|
||||
from devx.tokens import get_ci_token
|
||||
|
||||
POLL_INTERVAL = 10
|
||||
|
||||
|
||||
def get_running_jobs(gitea_url: str, owner: str, repo: str, token: str, run_id: int) -> list[dict]:
|
||||
"""Return jobs for the given workflow run."""
|
||||
url = f"{gitea_url}/api/v1/repos/{owner}/{repo}/actions/runs/{run_id}/jobs"
|
||||
headers = {"Authorization": f"token {token}"}
|
||||
response = requests.get(url, headers=headers, timeout=30)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
return data.get("jobs", [])
|
||||
|
||||
|
||||
def any_other_runner_failed(jobs: list[dict], current_job_name: str, current_index: int) -> bool:
|
||||
"""Return True if any other matrix job has failed."""
|
||||
for job in jobs:
|
||||
name = job.get("name", "")
|
||||
if not name.startswith(current_job_name):
|
||||
continue
|
||||
if name == f"{current_job_name} ({current_index})" or name == current_job_name:
|
||||
continue
|
||||
if job.get("conclusion") == "failure":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def poll_for_other_failures(
|
||||
gitea_url: str,
|
||||
owner: str,
|
||||
repo: str,
|
||||
token: str,
|
||||
run_id: int,
|
||||
job_name: str,
|
||||
current_index: int,
|
||||
stop_event: threading.Event,
|
||||
failed_event: threading.Event,
|
||||
) -> None:
|
||||
"""Background thread: poll API and signal if another runner fails."""
|
||||
while not stop_event.is_set():
|
||||
try:
|
||||
jobs = get_running_jobs(gitea_url, owner, repo, token, run_id)
|
||||
if any_other_runner_failed(jobs, job_name, current_index):
|
||||
click.echo(_("Another runner failed. Stopping this runner early."))
|
||||
failed_event.set()
|
||||
return
|
||||
except requests.RequestException as exc:
|
||||
click.echo(_("API poll warning: {exc}", exc=exc))
|
||||
stop_event.wait(POLL_INTERVAL)
|
||||
|
||||
|
||||
@click.command(context_settings={"ignore_unknown_options": True})
|
||||
@click.argument("pytest_args", nargs=-1, type=click.UNPROCESSED, required=True)
|
||||
def cli(pytest_args: tuple[str, ...]) -> None:
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# Implements: REQ-4
|
||||
"""Check if the nightly CI gate has passed; block staging deploys if it failed.
|
||||
|
||||
The nightly gate stores its status as a Gitea Actions repository variable
|
||||
named ``NIGHTLY_STATUS`` on the infra repo. Values:
|
||||
|
||||
- ``passed`` — nightly molecule + staging deploy + integration tests passed.
|
||||
- ``failed:<run_id>`` — nightly failed. Staging deploys are blocked until
|
||||
the nightly passes again.
|
||||
- (not set) — nightly hasn't run yet. First deploy is allowed (bootstrap).
|
||||
|
||||
Usage:
|
||||
python -m devx.ci.nightly_gate --repo oblachno/infra --action check
|
||||
python -m devx.ci.nightly_gate --repo oblachno/infra --action set-passed --run-id 12345
|
||||
python -m devx.ci.nightly_gate --repo oblachno/infra --action set-failed --run-id 12345
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import click
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from devx.api_clients import GiteaClient
|
||||
from devx.ci._shared import write_github_output
|
||||
from devx.config import GITEA_API_URL
|
||||
from devx.i18n import _
|
||||
from devx.tokens import get_ci_token
|
||||
|
||||
load_dotenv()
|
||||
|
||||
NIGHTLY_STATUS_VAR = "NIGHTLY_STATUS"
|
||||
|
||||
|
||||
def get_nightly_status(client: GiteaClient) -> str:
|
||||
"""Get the nightly status variable. Returns empty string if not set."""
|
||||
val = client.get_repo_variable(NIGHTLY_STATUS_VAR)
|
||||
return val or ""
|
||||
|
||||
|
||||
def set_nightly_status(client: GiteaClient, status: str) -> None:
|
||||
"""Set the nightly status variable."""
|
||||
client.set_repo_variable(NIGHTLY_STATUS_VAR, status)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--repo", required=True, help=_("Repository in owner/name format"))
|
||||
@click.option(
|
||||
"--action",
|
||||
type=click.Choice(["check", "set-passed", "set-failed"]),
|
||||
required=True,
|
||||
help=_("Action to perform"),
|
||||
)
|
||||
@click.option("--run-id", default="", help=_("CI run ID (for set-failed/set-passed)"))
|
||||
@click.option(
|
||||
"--github-output",
|
||||
"github_output",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help=_("Write results to $GITHUB_OUTPUT"),
|
||||
)
|
||||
def cli(repo: str, action: str, run_id: str, github_output: bool) -> None:
|
||||
"""Check or set the nightly CI gate status."""
|
||||
token = get_ci_token()
|
||||
if "/" not in repo:
|
||||
raise click.ClickException(_("Invalid repo format: {repo}. Expected owner/name.", repo=repo))
|
||||
owner, repo_name = repo.split("/", 1)
|
||||
client = GiteaClient(GITEA_API_URL, token, owner, repo_name)
|
||||
|
||||
if action == "check":
|
||||
status = get_nightly_status(client)
|
||||
if not status:
|
||||
# Bootstrap: no nightly has run yet, allow deploy
|
||||
click.echo("[nightly-gate] No nightly status set — allowing deploy (bootstrap).")
|
||||
if github_output:
|
||||
write_github_output("nightly-gate-passed", "true")
|
||||
write_github_output("nightly-status", "")
|
||||
return
|
||||
|
||||
if status.startswith("passed"):
|
||||
click.echo("[nightly-gate] Nightly passed. Deploy allowed.")
|
||||
if github_output:
|
||||
write_github_output("nightly-gate-passed", "true")
|
||||
write_github_output("nightly-status", status)
|
||||
elif status.startswith("failed"):
|
||||
run_part = status.split(":", 1)[1] if ":" in status else ""
|
||||
run_link = f" (run #{run_part})" if run_part else ""
|
||||
click.echo(
|
||||
_(
|
||||
"[nightly-gate] Nightly FAILED{run}. Staging deploys are blocked until nightly passes.",
|
||||
run=run_link,
|
||||
),
|
||||
err=True,
|
||||
)
|
||||
if github_output:
|
||||
write_github_output("nightly-gate-passed", "false")
|
||||
write_github_output("nightly-status", status)
|
||||
raise click.ClickException(_("Nightly gate failed — staging deploy blocked."))
|
||||
else:
|
||||
click.echo(
|
||||
_(
|
||||
"[nightly-gate] Unknown nightly status: {status} — blocking deploy (fail closed).",
|
||||
status=status,
|
||||
),
|
||||
err=True,
|
||||
)
|
||||
if github_output:
|
||||
write_github_output("nightly-gate-passed", "false")
|
||||
write_github_output("nightly-status", status)
|
||||
raise click.ClickException(_("Unknown nightly status — staging deploy blocked."))
|
||||
|
||||
elif action == "set-passed":
|
||||
set_nightly_status(client, f"passed:{run_id}" if run_id else "passed")
|
||||
click.echo(_("[nightly-gate] Set NIGHTLY_STATUS=passed{run}", run=f":{run_id}" if run_id else ""))
|
||||
if github_output:
|
||||
write_github_output("nightly-status", f"passed:{run_id}" if run_id else "passed")
|
||||
|
||||
elif action == "set-failed":
|
||||
set_nightly_status(client, f"failed:{run_id}" if run_id else "failed")
|
||||
click.echo(_("[nightly-gate] Set NIGHTLY_STATUS=failed{run}", run=f":{run_id}" if run_id else ""))
|
||||
if github_output:
|
||||
write_github_output("nightly-status", f"failed:{run_id}" if run_id else "failed")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
cli()
|
||||
@@ -0,0 +1,715 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Automated PR review: check architecture compliance, best practices, and quality.
|
||||
|
||||
Fetches the PR diff via the Gitea API, runs a series of automated checks,
|
||||
and posts a structured review using GiteaClient.create_review.
|
||||
|
||||
Checks performed:
|
||||
1. Architecture compliance — no business logic in CLI, no direct subprocess
|
||||
calls outside executor, no hardcoded config that should be in config.py
|
||||
2. Best practices — no bare except, no print() (use click.echo), no TODO/FIXME
|
||||
left in merged code, no functions > 50 lines
|
||||
3. Security — no secrets in code, no shell=True, no eval/exec
|
||||
4. i18n — no raw English strings in click.echo() without _() wrapper
|
||||
5. Resource management — no open() without with statement, no subprocess without cleanup
|
||||
6. Documentation — new CLI commands documented, new modules in architecture.md
|
||||
7. Test coverage — 100% enforced by pytest-cov (checked in quality job)
|
||||
8. Commit conventions — conventional commit format on branch commits
|
||||
|
||||
Usage:
|
||||
CI_GITEA_API_TOKEN=<token> [REVIEWER_GITEA_API_TOKEN=<token>] python3 -m devx.ci.pr_review <pr_number> <owner/repo>
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import click
|
||||
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
|
||||
|
||||
from devx.api_clients import GiteaClient
|
||||
from devx.config import GITEA_API_URL
|
||||
from devx.exceptions import APIError
|
||||
from devx.i18n import _
|
||||
from devx.tokens import get_ci_token, get_reviewer_token
|
||||
|
||||
load_dotenv()
|
||||
|
||||
# Files that are exempt from certain checks
|
||||
WORKFLOW_ONLY_SUFFIXES = (".yml", ".yaml", ".md", ".json", ".toml", ".cfg", ".ini", ".txt")
|
||||
PYTHON_SUFFIX = ".py"
|
||||
|
||||
# Architecture rules
|
||||
CLI_FILE = "src/devx/cli.py"
|
||||
EXECUTOR_FILE = "src/devx/executor.py"
|
||||
CONFIG_FILE = "src/devx/config.py"
|
||||
|
||||
# Patterns that indicate business logic in CLI (should be in runner_manager.py)
|
||||
BUSINESS_LOGIC_IN_CLI = [
|
||||
(r"subprocess\.(run|call|Popen|check_output|check_call)", "subprocess call in CLI — delegate to executor.py"),
|
||||
(r"\bos\.system\b", "os.system call in CLI — delegate to executor.py"),
|
||||
(r"\bansible-playbook\b", "ansible-playbook reference in CLI — delegate to executor.py"),
|
||||
]
|
||||
|
||||
# Patterns that indicate bad practices
|
||||
BAD_PRACTICES = [
|
||||
(r"\bprint\s*\(", "print() found — use click.echo() for user output"),
|
||||
(r"\beval\s*\(", "eval() found — security risk, avoid dynamic code execution"),
|
||||
(r"\bexec\s*\(", "exec() found — security risk, avoid dynamic code execution"),
|
||||
(r"shell\s*=\s*True", "shell=True found — security risk, use shell=False with list args"),
|
||||
(r"except\s*:", "bare except found — catch specific exceptions"),
|
||||
(r"except\s+Exception\s*:", "broad Exception catch — catch specific exceptions"),
|
||||
(r"#\s*(TODO|FIXME|HACK|XXX)", "TODO/FIXME found — resolve before merging"),
|
||||
]
|
||||
|
||||
# Patterns for hardcoded config values that should be in config.py
|
||||
HARDCODED_CONFIG = [
|
||||
(r"https?://[a-z]+\.[a-z]+\.[a-z]+", "hardcoded URL — move to config.py with env var override"),
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReviewResult:
|
||||
"""Result of automated review checks."""
|
||||
|
||||
issues: list[dict[str, Any]] = field(default_factory=list)
|
||||
summary: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def has_issues(self) -> bool:
|
||||
return bool(self.issues)
|
||||
|
||||
def add_issue(self, file_path: str, line: int, message: str, severity: str = "warning") -> None:
|
||||
self.issues.append(
|
||||
{
|
||||
"path": file_path,
|
||||
"body": f"[{severity}] {message}",
|
||||
"new_position": line,
|
||||
}
|
||||
)
|
||||
|
||||
def add_summary(self, text: str) -> None:
|
||||
self.summary.append(text)
|
||||
|
||||
|
||||
def is_python_file(path: str) -> bool:
|
||||
"""Check if a file is a Python source file."""
|
||||
return path.endswith(PYTHON_SUFFIX) and not path.startswith("tests/")
|
||||
|
||||
|
||||
def is_workflow_only(path: str) -> bool:
|
||||
"""Check if a file is workflow/config/docs only (not Python source)."""
|
||||
return path.endswith(WORKFLOW_ONLY_SUFFIXES) or path.startswith((".gitea/", "docs/", "ansible/"))
|
||||
|
||||
|
||||
def check_architecture_compliance(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check that changes follow the documented architecture."""
|
||||
for f in files:
|
||||
path = f.get("filename", "")
|
||||
if not is_python_file(path):
|
||||
continue
|
||||
|
||||
patch = f.get("patch", "")
|
||||
if not patch:
|
||||
continue
|
||||
|
||||
lines = patch.split("\n")
|
||||
current_line = 0
|
||||
for line in lines:
|
||||
if line.startswith("@@"):
|
||||
match = re.search(r"\+(\d+)", line)
|
||||
if match:
|
||||
current_line = int(match.group(1)) - 1
|
||||
continue
|
||||
if line.startswith("+") and not line.startswith("+++"):
|
||||
current_line += 1
|
||||
content = line[1:]
|
||||
|
||||
# Check for business logic in CLI
|
||||
if path == CLI_FILE:
|
||||
for pattern, msg in BUSINESS_LOGIC_IN_CLI:
|
||||
if re.search(pattern, content):
|
||||
result.add_issue(path, current_line, msg, "error")
|
||||
|
||||
if not result.issues:
|
||||
result.add_summary("- Architecture compliance: OK")
|
||||
|
||||
|
||||
def check_best_practices(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check for common code quality issues."""
|
||||
for f in files:
|
||||
path = f.get("filename", "")
|
||||
if not is_python_file(path):
|
||||
continue
|
||||
|
||||
patch = f.get("patch", "")
|
||||
if not patch:
|
||||
continue
|
||||
|
||||
lines = patch.split("\n")
|
||||
current_line = 0
|
||||
for line in lines:
|
||||
if line.startswith("@@"):
|
||||
match = re.search(r"\+(\d+)", line)
|
||||
if match:
|
||||
current_line = int(match.group(1)) - 1
|
||||
continue
|
||||
if line.startswith("+") and not line.startswith("+++"):
|
||||
current_line += 1
|
||||
content = line[1:]
|
||||
|
||||
for pattern, msg in BAD_PRACTICES:
|
||||
if re.search(pattern, content):
|
||||
result.add_issue(path, current_line, msg, "warning")
|
||||
|
||||
if not any(i["body"].startswith("[warning]") for i in result.issues):
|
||||
result.add_summary("- Best practices: OK")
|
||||
|
||||
|
||||
def check_security(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check for security issues in changed files."""
|
||||
for f in files:
|
||||
path = f.get("filename", "")
|
||||
if not is_python_file(path):
|
||||
continue
|
||||
|
||||
patch = f.get("patch", "")
|
||||
if not patch:
|
||||
continue
|
||||
|
||||
lines = patch.split("\n")
|
||||
current_line = 0
|
||||
for line in lines:
|
||||
if line.startswith("@@"):
|
||||
match = re.search(r"\+(\d+)", line)
|
||||
if match:
|
||||
current_line = int(match.group(1)) - 1
|
||||
continue
|
||||
if line.startswith("+") and not line.startswith("+++"):
|
||||
current_line += 1
|
||||
content = line[1:]
|
||||
|
||||
# Check for hardcoded secrets
|
||||
secret_re = r'(token|password|secret|key)\s*=\s*["\'][^"\']{8,}["\']' # nosec B105
|
||||
is_secret = re.search(secret_re, content, re.IGNORECASE)
|
||||
is_comment = content.strip().startswith("#")
|
||||
is_example = "your-" in content or "example" in content
|
||||
if is_secret and not is_comment and not is_example:
|
||||
result.add_issue(
|
||||
path,
|
||||
current_line,
|
||||
"potential hardcoded secret — use environment variable",
|
||||
"error",
|
||||
)
|
||||
|
||||
if not any(i["body"].startswith("[error]") and "secret" in i["body"] for i in result.issues):
|
||||
result.add_summary("- Security: OK")
|
||||
|
||||
|
||||
def check_i18n(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check that user-facing strings are wrapped in _().
|
||||
|
||||
Detects ``click.echo()`` calls with raw string literals that are not
|
||||
wrapped in ``_()``. Only checks ``src/`` files, not tests or scripts.
|
||||
"""
|
||||
# Pattern: click.echo("...") or click.echo(f"...") without _() wrapper
|
||||
raw_echo_re = re.compile(r'click\.echo\s*\(\s*["\']([^"\']+)["\']')
|
||||
raw_fstring_re = re.compile(r'click\.echo\s*\(\s*f["\']')
|
||||
# Also check click.ClickException and raise with string
|
||||
raw_exception_re = re.compile(r'click\.ClickException\s*\(\s*["\']([^"\']+)["\']')
|
||||
|
||||
for f in files:
|
||||
path = f.get("filename", "")
|
||||
if not is_python_file(path) or not path.startswith("src/"):
|
||||
continue
|
||||
|
||||
patch = f.get("patch", "")
|
||||
if not patch:
|
||||
continue
|
||||
|
||||
lines = patch.split("\n")
|
||||
current_line = 0
|
||||
for line in lines:
|
||||
if line.startswith("@@"):
|
||||
match = re.search(r"\+(\d+)", line)
|
||||
if match:
|
||||
current_line = int(match.group(1)) - 1
|
||||
continue
|
||||
if line.startswith("+") and not line.startswith("+++"):
|
||||
current_line += 1
|
||||
content = line[1:]
|
||||
|
||||
# Skip comments and docstrings
|
||||
stripped = content.strip()
|
||||
if stripped.startswith("#") or stripped.startswith('"""') or stripped.startswith("'''"):
|
||||
continue
|
||||
|
||||
# Check for raw strings in click.echo without _()
|
||||
for regex, msg in [
|
||||
(raw_echo_re, "click.echo() with raw string — wrap in _() for i18n"),
|
||||
(raw_fstring_re, "click.echo() with f-string — wrap in _() for i18n"),
|
||||
(raw_exception_re, "ClickException with raw string — wrap in _() for i18n"),
|
||||
]:
|
||||
if regex.search(content):
|
||||
result.add_issue(path, current_line, msg, "warning")
|
||||
|
||||
if not any("i18n" in i["body"] for i in result.issues):
|
||||
result.add_summary("- i18n: OK")
|
||||
|
||||
|
||||
def check_resource_management(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check for resource leaks: open() without with, subprocess without cleanup.
|
||||
|
||||
Detects:
|
||||
- ``open()`` calls not in a ``with`` statement
|
||||
- ``subprocess.Popen()`` without ``.wait()`` or ``.communicate()``
|
||||
"""
|
||||
# Pattern: open("...") not preceded by "with" on the same line
|
||||
open_re = re.compile(r"(?<!with\s)\bopen\s*\(")
|
||||
popen_re = re.compile(r"subprocess\.Popen\s*\(")
|
||||
|
||||
for f in files:
|
||||
path = f.get("filename", "")
|
||||
if not is_python_file(path):
|
||||
continue
|
||||
|
||||
patch = f.get("patch", "")
|
||||
if not patch:
|
||||
continue
|
||||
|
||||
lines = patch.split("\n")
|
||||
current_line = 0
|
||||
for line in lines:
|
||||
if line.startswith("@@"):
|
||||
match = re.search(r"\+(\d+)", line)
|
||||
if match:
|
||||
current_line = int(match.group(1)) - 1
|
||||
continue
|
||||
if line.startswith("+") and not line.startswith("+++"):
|
||||
current_line += 1
|
||||
content = line[1:]
|
||||
|
||||
# Skip comments
|
||||
if content.strip().startswith("#"):
|
||||
continue
|
||||
|
||||
# Check for open() without with
|
||||
if open_re.search(content) and "with " not in content:
|
||||
result.add_issue(
|
||||
path, current_line, "open() without with statement — potential resource leak", "warning"
|
||||
)
|
||||
|
||||
# Check for Popen without communicate/wait on same line
|
||||
if popen_re.search(content) and ".communicate" not in content and ".wait" not in content:
|
||||
result.add_issue(
|
||||
path,
|
||||
current_line,
|
||||
"subprocess.Popen() without immediate .communicate() or .wait() — ensure cleanup",
|
||||
"warning",
|
||||
)
|
||||
|
||||
if not any("resource" in i["body"].lower() for i in result.issues):
|
||||
result.add_summary("- Resource management: OK")
|
||||
|
||||
|
||||
def check_function_length(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check that no new function is excessively long (> 50 lines)."""
|
||||
for f in files:
|
||||
path = f.get("filename", "")
|
||||
if not is_python_file(path):
|
||||
continue
|
||||
|
||||
patch = f.get("patch", "")
|
||||
if not patch:
|
||||
continue
|
||||
|
||||
# Count consecutive added lines within a function
|
||||
lines = patch.split("\n")
|
||||
current_line = 0
|
||||
func_start = 0
|
||||
func_name = ""
|
||||
added_in_func = 0
|
||||
|
||||
for line in lines:
|
||||
if line.startswith("@@"):
|
||||
if func_name and added_in_func > 50:
|
||||
result.add_issue(
|
||||
path,
|
||||
func_start,
|
||||
f"function '{func_name}' adds {added_in_func} lines — consider splitting (> 50 lines)",
|
||||
"warning",
|
||||
)
|
||||
match = re.search(r"\+(\d+)", line)
|
||||
if match:
|
||||
current_line = int(match.group(1)) - 1
|
||||
func_name = ""
|
||||
added_in_func = 0
|
||||
continue
|
||||
if line.startswith("+") and not line.startswith("+++"):
|
||||
current_line += 1
|
||||
content = line[1:]
|
||||
func_match = re.match(r"\s*def\s+(\w+)\s*\(", content)
|
||||
if func_match:
|
||||
if func_name and added_in_func > 50:
|
||||
result.add_issue(
|
||||
path,
|
||||
func_start,
|
||||
f"function '{func_name}' adds {added_in_func} lines — consider splitting (> 50 lines)",
|
||||
"warning",
|
||||
)
|
||||
func_name = func_match.group(1)
|
||||
func_start = current_line
|
||||
added_in_func = 0
|
||||
else:
|
||||
added_in_func += 1
|
||||
elif line.startswith(" ") or line.startswith("-"):
|
||||
pass # context or removed line
|
||||
|
||||
# Check last function
|
||||
if func_name and added_in_func > 50:
|
||||
result.add_issue(
|
||||
path,
|
||||
func_start,
|
||||
f"function '{func_name}' adds {added_in_func} lines — consider splitting (> 50 lines)",
|
||||
"warning",
|
||||
)
|
||||
|
||||
|
||||
def check_documentation(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check that documentation is updated for relevant changes."""
|
||||
has_src_changes = any(
|
||||
is_python_file(f.get("filename", "")) and f.get("filename", "").startswith("src/") for f in files
|
||||
)
|
||||
has_doc_changes = any(
|
||||
f.get("filename", "").startswith("docs/") or f.get("filename", "") in ("README.md", "AGENTS.md", "CHANGELOG.md")
|
||||
for f in files
|
||||
)
|
||||
has_ansible_changes = any(f.get("filename", "").startswith("ansible/") for f in files)
|
||||
has_tofu_changes = any(f.get("filename", "").startswith("tofu/") for f in files)
|
||||
has_workflow_changes = any(f.get("filename", "").startswith(".gitea/") for f in files)
|
||||
|
||||
# Check for TODO/FIXME in changed docs
|
||||
todo_issues: list[str] = []
|
||||
for f in files:
|
||||
filename = f.get("filename", "")
|
||||
if filename.endswith(".md") and filename.startswith(("docs/", "README", "AGENTS")):
|
||||
# Can't check file content from PR API easily, but flag if patch adds TODO
|
||||
patch = f.get("patch", "")
|
||||
if patch and re.search(r"^\+.*\b(TODO|FIXME|HACK|XXX)\b", patch, re.IGNORECASE):
|
||||
todo_issues.append(f"{filename}: new TODO/FIXME added in documentation")
|
||||
|
||||
if has_src_changes and not has_doc_changes:
|
||||
result.add_summary("- Documentation: WARNING — source files changed but no docs updated")
|
||||
elif has_ansible_changes and not has_doc_changes:
|
||||
result.add_summary("- Documentation: WARNING — Ansible role changed but no docs updated")
|
||||
elif has_tofu_changes and not has_doc_changes:
|
||||
result.add_summary("- Documentation: WARNING — OpenTofu changes but no docs updated")
|
||||
elif has_workflow_changes and not has_doc_changes:
|
||||
result.add_summary("- Documentation: INFO — workflow changes (consider updating CI docs if behavior changed)")
|
||||
else:
|
||||
result.add_summary("- Documentation: OK")
|
||||
|
||||
if todo_issues:
|
||||
for issue in todo_issues:
|
||||
result.add_summary(f"- Documentation: WARNING — {issue}")
|
||||
|
||||
|
||||
def check_test_coverage(files: list[dict[str, Any]], result: ReviewResult) -> None:
|
||||
"""Check that tests are updated for source changes."""
|
||||
has_src_changes = any(
|
||||
is_python_file(f.get("filename", "")) and f.get("filename", "").startswith("src/") for f in files
|
||||
)
|
||||
has_test_changes = any(f.get("filename", "").startswith("tests/") for f in files)
|
||||
|
||||
if has_src_changes and not has_test_changes:
|
||||
result.add_summary("- Tests: WARNING — source files changed but no test files updated")
|
||||
else:
|
||||
result.add_summary("- Tests: OK")
|
||||
|
||||
|
||||
def check_commit_conventions(client: GiteaClient, pr_number: str, result: ReviewResult) -> None:
|
||||
"""Check that PR commits follow conventional commit format.
|
||||
|
||||
Verifies that at least one commit on the PR branch matches the
|
||||
conventional commit pattern (type: description). Merge commits
|
||||
and revert commits are exempt.
|
||||
"""
|
||||
try:
|
||||
commits = client.get_pr_commits(pr_number)
|
||||
except APIError as e:
|
||||
result.add_summary(f"- Commit conventions: ERROR — could not fetch commits: {e.message}")
|
||||
return
|
||||
|
||||
if not commits:
|
||||
result.add_summary("- Commit conventions: OK (no commits to check)")
|
||||
return
|
||||
|
||||
from devx.config import CONVENTIONAL_RE
|
||||
|
||||
has_conventional = False
|
||||
non_conventional: list[str] = []
|
||||
|
||||
for commit in commits:
|
||||
commit_info = commit.get("commit", {})
|
||||
message = str(commit_info.get("message", "") if isinstance(commit_info, dict) else "").split("\n")[0]
|
||||
# Skip merge commits and revert commits
|
||||
if message.startswith(("Merge", "Revert")):
|
||||
continue
|
||||
if CONVENTIONAL_RE.match(message):
|
||||
has_conventional = True
|
||||
else:
|
||||
non_conventional.append(message[:60])
|
||||
|
||||
if has_conventional:
|
||||
result.add_summary("- Commit conventions: OK")
|
||||
elif non_conventional:
|
||||
result.add_summary(
|
||||
f"- Commit conventions: WARNING — no conventional commit found. "
|
||||
f"Non-conventional commits: {', '.join(non_conventional[:3])}"
|
||||
)
|
||||
else:
|
||||
result.add_summary("- Commit conventions: OK (all commits are merges/reverts)")
|
||||
|
||||
|
||||
def run_review(client: GiteaClient, pr_number: str) -> ReviewResult:
|
||||
"""Run all review checks and return the result."""
|
||||
result = ReviewResult()
|
||||
|
||||
try:
|
||||
files = client.get_pr_files(pr_number)
|
||||
except APIError as e:
|
||||
result.add_summary(f"- ERROR: Could not fetch PR files: {e.message}")
|
||||
return result
|
||||
|
||||
if not files:
|
||||
result.add_summary("- No files changed in this PR")
|
||||
return result
|
||||
|
||||
# Run all checks
|
||||
check_architecture_compliance(files, result)
|
||||
check_best_practices(files, result)
|
||||
check_security(files, result)
|
||||
check_i18n(files, result)
|
||||
check_resource_management(files, result)
|
||||
check_function_length(files, result)
|
||||
check_documentation(files, result)
|
||||
check_test_coverage(files, result)
|
||||
check_commit_conventions(client, pr_number, result)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def build_review_body(result: ReviewResult) -> str:
|
||||
"""Build the review body text from the review result."""
|
||||
lines = ["## Automated PR Review", ""]
|
||||
|
||||
for item in result.summary:
|
||||
lines.append(item)
|
||||
|
||||
if result.issues:
|
||||
lines.append("")
|
||||
lines.append(f"**{len(result.issues)} issue(s) found:**")
|
||||
lines.append("")
|
||||
for issue in result.issues:
|
||||
lines.append(f"- `{issue['path']}:{issue['new_position']}` — {issue['body']}")
|
||||
else:
|
||||
lines.append("")
|
||||
lines.append("No issues found by automated checks.")
|
||||
|
||||
lines.append("")
|
||||
lines.append("---")
|
||||
lines.append("**Auto-merge:** If all CI checks pass, this PR will be merged automatically.")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def post_review(client: GiteaClient, pr_number: str, result: ReviewResult) -> dict[str, Any]:
|
||||
"""Post the review to the PR.
|
||||
|
||||
Uses REQUEST_CHANGES when issues are found, COMMENT otherwise.
|
||||
Never uses APPROVE — the bot shares the PR author's token, so
|
||||
Gitea rejects self-approval. The actual APPROVE must come from
|
||||
the manual review step.
|
||||
"""
|
||||
body = build_review_body(result)
|
||||
event = "REQUEST_CHANGES" if result.has_issues else "COMMENT"
|
||||
comments = result.issues if result.has_issues else []
|
||||
|
||||
return client.create_review(pr_number, event=event, body=body, comments=comments)
|
||||
|
||||
|
||||
def _post_manual_review(
|
||||
client: GiteaClient,
|
||||
pr_number: str,
|
||||
event: str,
|
||||
body: str | None,
|
||||
checklist_confirmed: bool,
|
||||
checklist_categories: str | None,
|
||||
dry_run: bool,
|
||||
owner: str | None = None,
|
||||
repo_name: str | None = None,
|
||||
) -> None:
|
||||
"""Post a manual review with validation for APPROVE events.
|
||||
|
||||
When self-approval is rejected (reviewer token belongs to PR author),
|
||||
falls back to the CI token (different user) if available.
|
||||
"""
|
||||
if not body or len(body) < 50:
|
||||
raise click.ClickException(_("Review body must be at least 50 characters."))
|
||||
|
||||
if event == "APPROVE":
|
||||
if not checklist_confirmed:
|
||||
raise click.ClickException(
|
||||
_("--checklist-confirmed is required for APPROVE events."),
|
||||
)
|
||||
cats = [c.strip() for c in (checklist_categories or "").split(",") if c.strip()]
|
||||
cat_nums: list[int] = []
|
||||
for c in cats:
|
||||
try:
|
||||
cat_nums.append(int(c))
|
||||
except ValueError:
|
||||
raise click.ClickException(
|
||||
_("Invalid checklist category: {cat}. Must be numbers.", cat=c),
|
||||
) from None
|
||||
if len(cat_nums) < 8:
|
||||
raise click.ClickException(
|
||||
_("--checklist-categories must list at least 8 of 13 categories. Got {count}.", count=len(cat_nums)),
|
||||
)
|
||||
|
||||
click.echo(f"Manual review event: {event}")
|
||||
click.echo(f"Body: {body[:80]}...")
|
||||
if checklist_confirmed:
|
||||
click.echo(f"Checklist confirmed: {checklist_categories}")
|
||||
|
||||
if dry_run:
|
||||
click.echo("\n[dry-run] Review not posted.")
|
||||
return
|
||||
|
||||
try:
|
||||
review = client.create_review(pr_number, event=event, body=body)
|
||||
except APIError as e:
|
||||
if "approve" in e.message.lower() or "422" in str(e.status):
|
||||
# Self-approval not allowed (reviewer token belongs to PR author).
|
||||
# Fall back to CI token (different user) if available.
|
||||
ci_token = os.environ.get("CI_GITEA_API_TOKEN", "").strip()
|
||||
if ci_token and owner and repo_name:
|
||||
click.echo(_("Note: Self-approval not allowed with reviewer token. Retrying with CI token."))
|
||||
ci_client = GiteaClient(GITEA_API_URL, ci_token, owner, repo_name)
|
||||
try:
|
||||
review = ci_client.create_review(pr_number, event=event, body=body)
|
||||
except APIError:
|
||||
click.echo(_("Note: CI token also cannot approve. Posting COMMENT instead."))
|
||||
review = client.create_review(pr_number, event="COMMENT", body=body)
|
||||
else:
|
||||
click.echo(_("Note: Self-approval not allowed. Posting COMMENT instead."))
|
||||
review = client.create_review(pr_number, event="COMMENT", body=body)
|
||||
else:
|
||||
raise
|
||||
review_id = review.get("id", "?")
|
||||
click.echo(
|
||||
_(
|
||||
"\nReview #{review_id} posted on PR #{pr_number} with event '{event}'.",
|
||||
review_id=review_id,
|
||||
pr_number=pr_number,
|
||||
event=event,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("pr_number")
|
||||
@click.argument("repo")
|
||||
@click.option("--dry-run", is_flag=True, default=False, help="Print review without posting.")
|
||||
@click.option(
|
||||
"--event",
|
||||
type=click.Choice(["APPROVE", "REQUEST_CHANGES", "COMMENT"], case_sensitive=False),
|
||||
default=None,
|
||||
help="Post a manual review with the given event (skips automated checks).",
|
||||
)
|
||||
@click.option("--body", default=None, help="Review body text (required with --event).")
|
||||
@click.option(
|
||||
"--checklist-confirmed",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Attest that REVIEW_CHECKLIST.md categories were checked (required for APPROVE).",
|
||||
)
|
||||
@click.option(
|
||||
"--checklist-categories",
|
||||
default=None,
|
||||
help="Comma-separated checklist category numbers (required for APPROVE, min 8 of 13).",
|
||||
)
|
||||
def main(
|
||||
pr_number: str,
|
||||
repo: str,
|
||||
dry_run: bool,
|
||||
event: str | None,
|
||||
body: str | None,
|
||||
checklist_confirmed: bool,
|
||||
checklist_categories: str | None,
|
||||
) -> None:
|
||||
"""Run automated PR review and post results to Gitea.
|
||||
|
||||
Without --event: runs automated checks and posts COMMENT/REQUEST_CHANGES.
|
||||
With --event: posts a manual review (skips automated checks).
|
||||
"""
|
||||
try:
|
||||
token = get_reviewer_token() if (event and event.upper() == "APPROVE") else get_ci_token()
|
||||
except click.ClickException:
|
||||
raise click.ClickException(_("ERROR: CI_GITEA_TOKEN is not set.")) from None
|
||||
|
||||
owner, repo_name = repo.split("/")
|
||||
client = GiteaClient(GITEA_API_URL, token, owner, repo_name)
|
||||
|
||||
if event is not None:
|
||||
_post_manual_review(
|
||||
client,
|
||||
pr_number,
|
||||
event.upper(),
|
||||
body,
|
||||
checklist_confirmed,
|
||||
checklist_categories,
|
||||
dry_run,
|
||||
owner=owner,
|
||||
repo_name=repo_name,
|
||||
)
|
||||
return
|
||||
|
||||
result = run_review(client, pr_number)
|
||||
|
||||
body = build_review_body(result)
|
||||
event = "REQUEST_CHANGES" if result.has_issues else "COMMENT"
|
||||
|
||||
click.echo(f"Review event: {event}")
|
||||
click.echo(f"Issues found: {len(result.issues)}")
|
||||
click.echo("")
|
||||
click.echo(body)
|
||||
|
||||
if dry_run:
|
||||
click.echo("\n[dry-run] Review not posted.")
|
||||
return
|
||||
|
||||
try:
|
||||
review = post_review(client, pr_number, result)
|
||||
except APIError as e:
|
||||
if "approve" in e.message.lower() or "422" in str(e.status):
|
||||
click.echo(_("Note: Self-approval not allowed. Posting COMMENT instead."))
|
||||
review = client.create_review(pr_number, event="COMMENT", body=body)
|
||||
else:
|
||||
raise
|
||||
review_id = review.get("id", "?")
|
||||
click.echo(
|
||||
_(
|
||||
"\nReview #{review_id} posted on PR #{pr_number} with event '{event}' ({num_comments} inline comments).",
|
||||
review_id=review_id,
|
||||
pr_number=pr_number,
|
||||
event=event,
|
||||
num_comments=len(result.issues),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -252,7 +252,7 @@ def commit_and_push(wiki_dir: Path, wiki_url: str, dry_run: bool) -> bool:
|
||||
|
||||
# Push
|
||||
result = subprocess.run( # nosec
|
||||
["git", "push", "--force", wiki_url, "HEAD:main"],
|
||||
["git", "push", "--force", wiki_url, "HEAD:master"],
|
||||
cwd=wiki_dir,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
|
||||
@@ -118,7 +118,7 @@ def main(commit_msg_file: str | None, branch: str | None, from_git: bool) -> Non
|
||||
" Expected: <type>: <description>\n"
|
||||
" Got: {subject}\n"
|
||||
" Allowed types: feat, fix, chore, docs, style, refactor,\n"
|
||||
" perf, test, ci, build, deps, revert, BREAKING CHANGE",
|
||||
" perf, test, ci, build, revert, BREAKING CHANGE",
|
||||
subject=subject,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1,157 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# Implements: REQ-1
|
||||
"""Validate that a PR has a spec file with required sections and acceptance criteria.
|
||||
|
||||
Spec-driven development gate. Runs in CI before expensive jobs.
|
||||
Used by grm, infra, sso-bridge, and devx itself.
|
||||
|
||||
Validates:
|
||||
1. A spec file exists at ``docs/specs/<TASK-ID>.md`` (TASK-ID extracted from branch).
|
||||
2. The spec contains required sections: Problem, Approach, Test Plan, Deploy Plan, Rollback Plan.
|
||||
3. The spec contains REQ-ID lines (``REQ-N: <description>``).
|
||||
4. The spec contains an Acceptance Criteria checklist with at least one item.
|
||||
5. All acceptance criteria checkboxes are checked (``- [x]``).
|
||||
|
||||
Usage:
|
||||
python -m devx.ci.validate_spec --branch OBL-INFRA-531-fix-foo
|
||||
|
||||
In CI, also pass ``--github-output`` to set ``spec-valid`` and ``spec-path``
|
||||
for downstream steps.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from devx.ci._shared import extract_task_id, write_github_output
|
||||
from devx.i18n import _
|
||||
|
||||
load_dotenv()
|
||||
|
||||
REQUIRED_SECTIONS = [
|
||||
"## Problem",
|
||||
"## Approach",
|
||||
"## Test Plan",
|
||||
"## Deploy Plan",
|
||||
"## Rollback Plan",
|
||||
"## Acceptance Criteria",
|
||||
]
|
||||
|
||||
REQ_ID_RE = re.compile(r"^REQ-\d+:\s+.+", re.MULTILINE)
|
||||
AC_CHECKED_RE = re.compile(r"^\s*- \[x\]\s+.+", re.MULTILINE)
|
||||
AC_UNCHECKED_RE = re.compile(r"^\s*- \[ \]\s+.+", re.MULTILINE)
|
||||
|
||||
|
||||
def find_spec_file(task_id: str, specs_dir: str = "docs/specs") -> Path | None:
|
||||
"""Find the spec file for the given task ID.
|
||||
|
||||
Looks for ``docs/specs/<TASK-ID>.md`` (case-insensitive filename).
|
||||
Returns the Path if found, None otherwise.
|
||||
"""
|
||||
base = Path(specs_dir)
|
||||
if not base.is_dir():
|
||||
return None
|
||||
# Exact match (case-insensitive)
|
||||
for p in base.glob("*.md"):
|
||||
if p.stem.upper() == task_id.upper():
|
||||
return p
|
||||
return None
|
||||
|
||||
|
||||
def validate_spec_content(content: str) -> list[str]:
|
||||
"""Validate spec content and return a list of error messages.
|
||||
|
||||
Returns an empty list if the spec is valid.
|
||||
"""
|
||||
errors: list[str] = []
|
||||
|
||||
# Check required sections
|
||||
for section in REQUIRED_SECTIONS:
|
||||
if section not in content:
|
||||
errors.append(_("Missing required section: {section}", section=section))
|
||||
|
||||
# Check for at least one REQ-ID
|
||||
req_ids = REQ_ID_RE.findall(content)
|
||||
if not req_ids:
|
||||
errors.append(_("No REQ-ID lines found. Each requirement must be labeled (e.g., 'REQ-1: <description>')."))
|
||||
|
||||
# Check acceptance criteria has at least one item
|
||||
checked = AC_CHECKED_RE.findall(content)
|
||||
unchecked = AC_UNCHECKED_RE.findall(content)
|
||||
if not checked and not unchecked:
|
||||
errors.append(_("Acceptance Criteria section has no checklist items. Add at least one '- [ ] item'."))
|
||||
elif unchecked:
|
||||
errors.append(
|
||||
_(
|
||||
"Acceptance Criteria has {count} unchecked item(s). All AC items must be checked (- [x]) before merge.",
|
||||
count=len(unchecked),
|
||||
)
|
||||
)
|
||||
|
||||
return errors
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--branch", required=True, help=_("Branch name (e.g., OBL-INFRA-531-fix-foo)"))
|
||||
@click.option("--specs-dir", default="docs/specs", help=_("Directory containing spec files"))
|
||||
@click.option(
|
||||
"--github-output",
|
||||
"github_output",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help=_("Write results to $GITHUB_OUTPUT"),
|
||||
)
|
||||
@click.option("--allow-missing", is_flag=True, default=False, help=_("Allow missing spec (warn only, don't fail)"))
|
||||
def cli(branch: str, specs_dir: str, github_output: bool, allow_missing: bool) -> None:
|
||||
"""Validate that a spec file exists and has required content."""
|
||||
task_id = extract_task_id(branch)
|
||||
if not task_id:
|
||||
msg = _("No task ID found in branch name '{branch}'. Expected format: <PREFIX>-N-description.", branch=branch)
|
||||
if allow_missing:
|
||||
click.echo(f"WARNING: {msg}")
|
||||
if github_output:
|
||||
write_github_output("spec-valid", "false")
|
||||
write_github_output("spec-path", "")
|
||||
return
|
||||
raise click.ClickException(msg)
|
||||
|
||||
spec_path = find_spec_file(task_id, specs_dir)
|
||||
if spec_path is None:
|
||||
msg = _(
|
||||
"No spec file found for task {task_id} in {dir}/. Expected: {dir}/{task_id}.md",
|
||||
task_id=task_id,
|
||||
dir=specs_dir,
|
||||
)
|
||||
if allow_missing:
|
||||
click.echo(f"WARNING: {msg}")
|
||||
if github_output:
|
||||
write_github_output("spec-valid", "false")
|
||||
write_github_output("spec-path", "")
|
||||
return
|
||||
raise click.ClickException(msg)
|
||||
|
||||
content = spec_path.read_text(encoding="utf-8")
|
||||
errors = validate_spec_content(content)
|
||||
|
||||
if github_output:
|
||||
write_github_output("spec-valid", "true" if not errors else "false")
|
||||
write_github_output("spec-path", str(spec_path))
|
||||
|
||||
if errors:
|
||||
click.echo("", err=True)
|
||||
click.echo("=" * 60, err=True)
|
||||
click.echo(f"Spec validation FAILED for {spec_path}:", err=True)
|
||||
click.echo("=" * 60, err=True)
|
||||
for e in errors:
|
||||
click.echo(f" - {e}", err=True)
|
||||
raise click.ClickException(_("Spec validation failed."))
|
||||
|
||||
click.echo(_("[spec-check] Spec validated: {path}", path=spec_path))
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
cli()
|
||||
@@ -116,6 +116,13 @@ def ci_post_merge(args: tuple[str, ...]) -> None:
|
||||
_run_module("devx.ci.post_merge", list(args))
|
||||
|
||||
|
||||
@ci.command("pr-review")
|
||||
@click.argument("args", nargs=-1)
|
||||
def ci_pr_review(args: tuple[str, ...]) -> None:
|
||||
"""Run automated PR review."""
|
||||
_run_module("devx.ci.pr_review", list(args))
|
||||
|
||||
|
||||
@ci.command("publish")
|
||||
@click.argument("args", nargs=-1)
|
||||
def ci_publish(args: tuple[str, ...]) -> None:
|
||||
@@ -294,6 +301,13 @@ def molecule_discover_runners(args: tuple[str, ...]) -> None:
|
||||
_run_module("devx.molecule.discover_runners", list(args))
|
||||
|
||||
|
||||
@molecule.command("guard")
|
||||
@click.argument("args", nargs=-1)
|
||||
def molecule_guard(args: tuple[str, ...]) -> None:
|
||||
"""Run molecule tests sequentially with CI failure polling."""
|
||||
_run_module("devx.molecule.molecule_ci_guard", list(args))
|
||||
|
||||
|
||||
@molecule.command("all")
|
||||
@click.argument("args", nargs=-1)
|
||||
def molecule_all(args: tuple[str, ...]) -> None:
|
||||
|
||||
+1
-1
@@ -93,4 +93,4 @@ RETRY_BACKOFF_BASE = 2 # seconds: 2, 4, 8
|
||||
RETRY_STATUS_CODES = {429, 500, 502, 503, 504}
|
||||
|
||||
# Conventional commit regex — used by validate_commit_msg.py
|
||||
CONVENTIONAL_RE = re.compile(r"^(feat|fix|chore|docs|style|refactor|perf|test|ci|build|revert|deps)(\(.+\))?: .+")
|
||||
CONVENTIONAL_RE = re.compile(r"^(feat|fix|chore|docs|style|refactor|perf|test|ci|build|revert)(\(.+\))?: .+")
|
||||
|
||||
+11
-1
@@ -109,7 +109,7 @@ devx-ensure-venv:
|
||||
fi
|
||||
|
||||
.PHONY: devx-create-task devx-create-pr devx-push devx-push-with-pr devx-check-config
|
||||
.PHONY: devx-pr-status devx-pr-logs devx-pr-label devx-rebase devx-pr-rebase
|
||||
.PHONY: devx-pr-status devx-pr-logs devx-pr-label devx-pr-review devx-rebase devx-pr-rebase
|
||||
.PHONY: devx-configure-gitea-pypi devx-install-tools devx-install-checkmake devx-checkmake
|
||||
.PHONY: devx-workflow-lint devx-workflow-dryrun devx-workflow-dryrun-safe devx-workflow-check
|
||||
.PHONY: devx-notify-failure devx-install-hooks devx-activate-scripts devx-venv devx-ensure-venv
|
||||
@@ -171,6 +171,16 @@ devx-pr-label:
|
||||
$(if $(PR),--pr $(PR)) \
|
||||
--label $(or $(LABEL),ready-to-merge)
|
||||
|
||||
# Usage: make devx-pr-review PR=42 EVENT=APPROVE BODY="..." CHECKLIST=1,2,3,4,5,6,7,8,9,10,11,12,13
|
||||
# make devx-pr-review PR=42 EVENT=REQUEST_CHANGES BODY="..."
|
||||
# make devx-pr-review PR=42 (auto review)
|
||||
devx-pr-review:
|
||||
@$(DEVX_PYTHON) -m devx.ci.pr_review \
|
||||
$(PR) $(DEVX_REPO_OWNER)/$(DEVX_REPO_NAME) \
|
||||
$(if $(EVENT),--event $(EVENT)) \
|
||||
$(if $(BODY),--body "$(BODY)") \
|
||||
$(if $(CHECKLIST),--checklist-confirmed --checklist-categories $(CHECKLIST))
|
||||
|
||||
# Rebase current branch onto origin/master and force-push
|
||||
# Usage: make devx-rebase
|
||||
# make devx-rebase NO_PUSH=1
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
"""Emit bounded runner-resource lease env for CI molecule jobs.
|
||||
|
||||
Producer side of the S08 lease contract (consumer: GRM-173
|
||||
``runner-cleanup.sh``). The runner's scoped cleanup honors
|
||||
``org.oblachno.lease-until`` (epoch seconds) and ``org.oblachno.owner``
|
||||
labels on containers, volumes, and networks. Molecule runs on the runner
|
||||
host's rootless Docker socket, so objects it creates must carry the lease
|
||||
labels at creation time — Docker labels cannot be added post-hoc.
|
||||
|
||||
This module computes ``MOLECULE_LEASE_UNTIL`` and ``MOLECULE_OWNER``,
|
||||
which ``molecule.yml`` platform ``labels:`` entries consume via
|
||||
``${MOLECULE_LEASE_UNTIL:-0}`` / ``${MOLECULE_OWNER:-molecule-local}``
|
||||
interpolation, and appends them to ``$GITHUB_ENV`` so every later CI step
|
||||
inherits them. A lease-until of ``0`` (the interpolation default for
|
||||
local runs) is already expired, keeping today's unleased behavior.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import socket
|
||||
import time
|
||||
|
||||
import click
|
||||
|
||||
#: Docker label contract consumed by runner-cleanup.sh (GRM-173).
|
||||
LEASE_UNTIL_LABEL = "org.oblachno.lease-until"
|
||||
OWNER_LABEL = "org.oblachno.owner"
|
||||
|
||||
#: Env vars consumed by molecule.yml platform ``labels:`` entries.
|
||||
LEASE_UNTIL_ENV = "MOLECULE_LEASE_UNTIL"
|
||||
OWNER_ENV = "MOLECULE_OWNER"
|
||||
|
||||
#: Default lease duration — covers the longest molecule job window.
|
||||
DEFAULT_TTL_SECONDS = 4 * 3600
|
||||
|
||||
|
||||
def lease_owner() -> str:
|
||||
"""Identify the owning CI run for the ``org.oblachno.owner`` label.
|
||||
|
||||
act_runner exposes GitHub-compatible ``GITHUB_*`` env vars;
|
||||
``GITEA_*`` names are checked first for forward compatibility.
|
||||
"""
|
||||
run_id = os.environ.get("GITEA_RUN_ID") or os.environ.get("GITHUB_RUN_ID")
|
||||
job = os.environ.get("GITEA_JOB") or os.environ.get("GITHUB_JOB")
|
||||
if run_id:
|
||||
return f"run-{run_id}-{job or 'job'}"
|
||||
return f"local-{socket.gethostname()}"
|
||||
|
||||
|
||||
def lease_env(ttl: int = DEFAULT_TTL_SECONDS, now: int | None = None) -> dict[str, str]:
|
||||
"""Return the lease env dict for the current run."""
|
||||
base = int(time.time()) if now is None else now
|
||||
return {
|
||||
LEASE_UNTIL_ENV: str(base + ttl),
|
||||
OWNER_ENV: lease_owner(),
|
||||
}
|
||||
|
||||
|
||||
def emit_github_env(env: dict[str, str]) -> int:
|
||||
"""Append lease env to ``$GITHUB_ENV``; return lines written (0 if unset)."""
|
||||
github_env = os.environ.get("GITHUB_ENV")
|
||||
if not github_env:
|
||||
return 0
|
||||
with open(github_env, "a", encoding="utf-8") as f:
|
||||
for key, value in env.items():
|
||||
f.write(f"{key}={value}\n")
|
||||
return len(env)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--ttl",
|
||||
default=DEFAULT_TTL_SECONDS,
|
||||
type=int,
|
||||
help=f"Lease duration in seconds (default: {DEFAULT_TTL_SECONDS}).",
|
||||
)
|
||||
def main(ttl: int) -> None:
|
||||
"""Print lease env vars and export them to $GITHUB_ENV when set."""
|
||||
env = lease_env(ttl)
|
||||
for key, value in env.items():
|
||||
click.echo(f"{key}={value}")
|
||||
if emit_github_env(env):
|
||||
click.echo(f"Exported {len(env)} lease vars to GITHUB_ENV")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -1,194 +0,0 @@
|
||||
"""Detect which Ansible roles changed and output their molecule scenarios.
|
||||
|
||||
Usage::
|
||||
|
||||
python -m devx.molecule.molecule_changed --print-targets
|
||||
python -m devx.molecule.molecule_changed --base origin/master --print-roles
|
||||
|
||||
Outputs the list of make targets (e.g. molecule-docker-base) for roles
|
||||
that have changed files vs the base ref. Used by ``make molecule-changed``
|
||||
to run only the molecule scenarios affected by the current diff.
|
||||
|
||||
Role-to-target mapping is derived from the directory structure:
|
||||
ansible/roles/<role>/ → molecule-<role>
|
||||
|
||||
For roles with multiple scenarios (e.g. app_container has customer-apps,
|
||||
nextcloud, postgres-upgrade, simple-app), the base target runs all
|
||||
scenarios for that role.
|
||||
|
||||
Playbooks that change also trigger molecule for the roles they include.
|
||||
Shared infrastructure changes (ansible.cfg, requirements.yml, molecule/,
|
||||
group_vars/) trigger all scenarios. Any ``ansible/playbooks/**`` file not
|
||||
in the explicit map fails open to all testable roles — targeted selection
|
||||
must never silently skip coverage.
|
||||
|
||||
Only roles that exist on disk and contain a ``molecule/`` directory are
|
||||
selected, so emitted make targets always resolve.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess # nosec B404 — used to run git, a trusted binary
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
|
||||
REPO_ROOT = Path.cwd()
|
||||
|
||||
|
||||
def role_to_target(role: str) -> str:
|
||||
"""Make target for a role: docker_base -> molecule-docker-base."""
|
||||
return f"molecule-{role.replace('_', '-')}"
|
||||
|
||||
|
||||
# Playbooks that map to molecule scenarios (via roles they include).
|
||||
# Any ansible/playbooks/** file NOT listed here fails open to all testable
|
||||
# roles (REQ-1) — a missing entry must never mean "no coverage".
|
||||
PLAYBOOK_ROLE_MAP: dict[str, list[str]] = {
|
||||
"ansible/playbooks/deploy-observability.yml": ["observability", "docker_base", "zitadel", "crowdsec"],
|
||||
"ansible/playbooks/deploy-customer.yml": ["app_container", "docker_base", "app_hardening"],
|
||||
"ansible/playbooks/deploy-controller.yml": ["deploy_controller"],
|
||||
"ansible/playbooks/prepare-vms.yml": ["docker_base", "app_hardening", "storage", "disk_cleanup", "crowdsec"],
|
||||
"ansible/playbooks/restore.yml": ["restore"],
|
||||
"ansible/playbooks/upgrade-postgres.yml": ["app_container"],
|
||||
"ansible/playbooks/rolling-update-gitea.yml": ["app_container"],
|
||||
"ansible/playbooks/update-alertmanager.yml": ["observability"],
|
||||
"ansible/playbooks/build-image.yml": ["docker_base", "crowdsec", "disk_cleanup"],
|
||||
# sso_bridge role is checked out from the sso-bridge repo at deploy
|
||||
# time — no molecule coverage exists in the consuming repo, so the
|
||||
# explicit empty list documents "mapped, nothing local to test".
|
||||
"ansible/playbooks/deploy-sso-bridge.yml": [],
|
||||
}
|
||||
|
||||
# Shared infrastructure that affects all molecule tests.
|
||||
SHARED_PATHS = (
|
||||
"ansible/ansible.cfg",
|
||||
"ansible/requirements.yml",
|
||||
"ansible/molecule/",
|
||||
"ansible/group_vars/",
|
||||
)
|
||||
|
||||
# Minimum path parts for a role file: ansible/roles/<role> (3 parts).
|
||||
# Files inside the role have more parts, but we only need the role name.
|
||||
_MIN_ROLE_PATH_PARTS = 3
|
||||
|
||||
|
||||
def _run_git(args: list[str]) -> str: # pragma: no cover
|
||||
"""Run a git command and return stdout."""
|
||||
result = subprocess.run( # nosec
|
||||
["git", *args],
|
||||
cwd=REPO_ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
def get_changed_files(base: str) -> list[str]:
|
||||
"""Get list of changed files vs base ref."""
|
||||
for ref in [base, "master"]:
|
||||
output = _run_git(["diff", "--name-only", f"{ref}...HEAD"])
|
||||
if output.strip():
|
||||
return sorted(output.strip().splitlines())
|
||||
return []
|
||||
|
||||
|
||||
def _testable_roles(roles_dir: Path) -> set[str]:
|
||||
"""Roles present on disk that carry a molecule/ dir (i.e. have scenarios)."""
|
||||
if not roles_dir.is_dir():
|
||||
return set()
|
||||
return {d.name for d in roles_dir.iterdir() if d.is_dir() and (d / "molecule").is_dir()}
|
||||
|
||||
|
||||
def _is_playbook_file(filepath: str) -> bool:
|
||||
"""True for any file under ansible/playbooks/ (yml tasks included)."""
|
||||
return filepath.startswith("ansible/playbooks/")
|
||||
|
||||
|
||||
def detect_changed_roles(
|
||||
changed_files: list[str],
|
||||
roles_dir: str | Path = "ansible/roles",
|
||||
) -> set[str]:
|
||||
"""Detect which roles have changed files.
|
||||
|
||||
Implements: REQ-1, REQ-2 — fail open on unmapped ansible playbook or
|
||||
shared-path changes; only roles that exist under ``roles_dir`` are
|
||||
returned, so emitted targets always resolve.
|
||||
"""
|
||||
roles_path = Path(roles_dir)
|
||||
all_roles = _testable_roles(roles_path)
|
||||
roles: set[str] = set()
|
||||
|
||||
for filepath in changed_files:
|
||||
# Check if file is in a role directory
|
||||
if filepath.startswith("ansible/roles/"):
|
||||
parts = filepath.split("/")
|
||||
if len(parts) >= _MIN_ROLE_PATH_PARTS:
|
||||
roles.add(parts[2])
|
||||
|
||||
# Check if file is a playbook that maps to roles
|
||||
if filepath in PLAYBOOK_ROLE_MAP:
|
||||
roles.update(PLAYBOOK_ROLE_MAP[filepath])
|
||||
elif _is_playbook_file(filepath):
|
||||
# Unmapped playbook/_tasks file — fail open to all testable roles.
|
||||
return set(all_roles)
|
||||
|
||||
# Check shared infrastructure — triggers all roles
|
||||
for shared in SHARED_PATHS:
|
||||
if filepath.startswith(shared):
|
||||
return set(all_roles)
|
||||
|
||||
# REQ-2: drop roles that don't exist on disk (e.g. sso_config after the
|
||||
# role moved to the sso-bridge repo) so targets always resolve.
|
||||
return {r for r in roles if (roles_path / r).is_dir()}
|
||||
|
||||
|
||||
def roles_to_targets(roles: set[str]) -> list[str]:
|
||||
"""Convert role names to make targets (conventional molecule-<role>)."""
|
||||
return [role_to_target(role) for role in sorted(roles)]
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--base",
|
||||
default="origin/master",
|
||||
help="Base ref to compare against (default: origin/master).",
|
||||
)
|
||||
@click.option(
|
||||
"--print-targets",
|
||||
is_flag=True,
|
||||
help="Print make targets (e.g. molecule-docker-base).",
|
||||
)
|
||||
@click.option(
|
||||
"--print-roles",
|
||||
is_flag=True,
|
||||
help="Print role names (default if no --print-targets).",
|
||||
)
|
||||
@click.option(
|
||||
"--roles-dir",
|
||||
default="ansible/roles",
|
||||
help="Directory containing Ansible roles (default: ansible/roles).",
|
||||
)
|
||||
def main(base: str, print_targets: bool, print_roles: bool, roles_dir: str) -> None:
|
||||
"""Detect which Ansible roles changed and output molecule scenarios."""
|
||||
changed_files = get_changed_files(base)
|
||||
if not changed_files:
|
||||
click.echo("No changed files detected.", err=True)
|
||||
return
|
||||
|
||||
roles = detect_changed_roles(changed_files, roles_dir)
|
||||
if not roles:
|
||||
click.echo("No molecule scenarios affected by changes.", err=True)
|
||||
return
|
||||
|
||||
if print_targets:
|
||||
for target in roles_to_targets(roles):
|
||||
click.echo(target)
|
||||
else:
|
||||
for role in sorted(roles):
|
||||
click.echo(role)
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -0,0 +1,284 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run molecule tests sequentially while polling Gitea for other runner failures.
|
||||
|
||||
Each pair is encoded as one of:
|
||||
|
||||
- **Single-role (4-part):** ``scenario|platform_name|platform_image|platform_command``
|
||||
- **Multi-role (5-part):** ``role|scenario|platform_name|platform_image|platform_command``
|
||||
|
||||
Pairs are executed one at a time (molecule scenarios share temp directories and
|
||||
Docker networks, so parallel execution within a single runner is unsafe).
|
||||
|
||||
A background thread polls the Gitea API. If any other molecule matrix runner
|
||||
reports failure, the current molecule subprocess is killed and this runner
|
||||
exits early with code 1.
|
||||
|
||||
Usage::
|
||||
|
||||
# Single-role
|
||||
python3 -m devx.molecule.molecule_ci_guard pair1 pair2 ...
|
||||
# Multi-role
|
||||
python3 -m devx.molecule.molecule_ci_guard --roles-root ansible/roles pair1 pair2 ...
|
||||
|
||||
Environment variables:
|
||||
GITEA_URL Base URL of the Gitea instance.
|
||||
CI_GITEA_API_TOKEN API token with repo access (CI_GITEA_TOKEN accepted for legacy).
|
||||
RUN_ID Workflow run ID (GITHUB_RUN_ID).
|
||||
JOB_NAME Base job name (GITHUB_JOB), e.g. "molecule-tests".
|
||||
MATRIX_INDEX Current matrix index (runner-index).
|
||||
GITEA_REPOSITORY Repository in "owner/repo" format.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import signal
|
||||
import subprocess # nosec B404
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import requests
|
||||
|
||||
from devx.config import REPO_NAME, REPO_OWNER
|
||||
from devx.i18n import _
|
||||
from devx.tokens import get_ci_token
|
||||
|
||||
POLL_INTERVAL = 10
|
||||
|
||||
|
||||
def get_running_jobs(gitea_url: str, owner: str, repo: str, token: str, run_id: int) -> list[dict]:
|
||||
"""Return jobs for the given workflow run."""
|
||||
url = f"{gitea_url}/api/v1/repos/{owner}/{repo}/actions/runs/{run_id}/jobs"
|
||||
headers = {"Authorization": f"token {token}"}
|
||||
response = requests.get(url, headers=headers, timeout=30)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
return data.get("jobs", [])
|
||||
|
||||
|
||||
def any_other_runner_failed(jobs: list[dict], current_job_name: str, current_index: int) -> bool:
|
||||
"""Return True if any other molecule matrix job has failed."""
|
||||
for job in jobs:
|
||||
name = job.get("name", "")
|
||||
if not name.startswith(current_job_name):
|
||||
continue
|
||||
if name == f"{current_job_name} ({current_index})" or name == current_job_name:
|
||||
continue
|
||||
if job.get("conclusion") == "failure":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def poll_for_other_failures(
|
||||
gitea_url: str,
|
||||
owner: str,
|
||||
repo: str,
|
||||
token: str,
|
||||
run_id: int,
|
||||
job_name: str,
|
||||
current_index: int,
|
||||
stop_event: threading.Event,
|
||||
failed_event: threading.Event,
|
||||
) -> None:
|
||||
"""Background thread: poll API and signal if another runner fails."""
|
||||
while not stop_event.is_set():
|
||||
try:
|
||||
jobs = get_running_jobs(gitea_url, owner, repo, token, run_id)
|
||||
if any_other_runner_failed(jobs, job_name, current_index):
|
||||
click.echo(_("Another molecule runner failed. Stopping this runner early."))
|
||||
failed_event.set()
|
||||
return
|
||||
except requests.RequestException as exc:
|
||||
click.echo(_("API poll warning: {exc}", exc=exc))
|
||||
stop_event.wait(POLL_INTERVAL)
|
||||
|
||||
|
||||
def build_molecule_cmd(scenario: str) -> list[str]:
|
||||
"""Build the molecule command for a scenario."""
|
||||
cmd = ["molecule", "test"]
|
||||
if scenario != "default":
|
||||
cmd.extend(["-s", scenario])
|
||||
return cmd
|
||||
|
||||
|
||||
def parse_pair(pair: str) -> tuple[str, str, str, str, str]:
|
||||
"""Parse a pair string into (role, scenario, platform_name, platform_image, platform_command).
|
||||
|
||||
Supports both 4-part (single-role) and 5-part (multi-role) formats.
|
||||
For 4-part pairs, role is empty (caller uses default role dir).
|
||||
Spaces in the command field are encoded as ``__SPACE__`` to survive
|
||||
shell word-splitting when ``$TEST_PAIRS`` is expanded unquoted.
|
||||
"""
|
||||
parts = pair.split("|")
|
||||
if len(parts) == 4:
|
||||
return "", parts[0], parts[1], parts[2], parts[3].replace("__SPACE__", " ")
|
||||
if len(parts) == 5:
|
||||
return parts[0], parts[1], parts[2], parts[3], parts[4].replace("__SPACE__", " ")
|
||||
raise click.ClickException(f"Invalid pair format: {pair!r} (expected 4 or 5 pipe-delimited parts)")
|
||||
|
||||
|
||||
def build_env_for_pair(pair: str, base_env: dict[str, str]) -> dict[str, str]:
|
||||
"""Build environment for a single molecule pair."""
|
||||
_role, _scenario, platform_name, platform_image, platform_command = parse_pair(pair)
|
||||
env = base_env.copy()
|
||||
env["MOLECULE_PLATFORM_NAME"] = platform_name
|
||||
env["MOLECULE_PLATFORM_IMAGE"] = platform_image
|
||||
if platform_command:
|
||||
env["MOLECULE_PLATFORM_COMMAND"] = platform_command
|
||||
elif "MOLECULE_PLATFORM_COMMAND" in env:
|
||||
del env["MOLECULE_PLATFORM_COMMAND"]
|
||||
env["ANSIBLE_ALLOW_BROKEN_CONDITIONALS"] = "true"
|
||||
# Use a fresh MOLECULE_HOME per pair to avoid stale config cache
|
||||
# from previous CI runs (causes "Instances missing" errors).
|
||||
if "MOLECULE_HOME" not in env:
|
||||
import tempfile
|
||||
|
||||
env["MOLECULE_HOME"] = tempfile.mkdtemp(prefix="molecule-ci-")
|
||||
return env
|
||||
|
||||
|
||||
def resolve_role_dir(role: str, roles_root: Path | None, repo_root: Path) -> Path:
|
||||
"""Resolve the working directory for a molecule pair.
|
||||
|
||||
For multi-role pairs (role non-empty), uses ``roles_root/role``.
|
||||
For single-role pairs, auto-discovers the first role with a molecule/
|
||||
subdirectory under ``repo_root/ansible/roles/``.
|
||||
"""
|
||||
if role:
|
||||
if roles_root is None:
|
||||
roles_root = repo_root / "ansible" / "roles"
|
||||
return roles_root / role
|
||||
roles_dir = repo_root / "ansible" / "roles"
|
||||
if roles_dir.is_dir():
|
||||
role_dirs = sorted(d for d in roles_dir.iterdir() if (d / "molecule").is_dir())
|
||||
if role_dirs:
|
||||
return role_dirs[0]
|
||||
return roles_dir / "role" # will produce a clear "not found" error
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("pairs", nargs=-1, required=True)
|
||||
@click.option(
|
||||
"--roles-root",
|
||||
type=click.Path(exists=True, file_okay=False, path_type=Path),
|
||||
default=None,
|
||||
help="Root directory for multi-role pairs (e.g. ansible/roles). Required when pairs use 5-part format.",
|
||||
)
|
||||
def cli(pairs: tuple[str, ...], roles_root: Path | None) -> None:
|
||||
"""Run molecule pairs sequentially, stop if another CI runner fails."""
|
||||
gitea_url = os.environ.get("GITEA_URL", "")
|
||||
try:
|
||||
token = get_ci_token()
|
||||
except click.ClickException:
|
||||
token = None
|
||||
run_id = int(os.environ.get("RUN_ID", "0"))
|
||||
job_name = os.environ.get("JOB_NAME", "molecule-tests")
|
||||
current_index = int(os.environ.get("MATRIX_INDEX", "0"))
|
||||
repository = os.environ.get("GITEA_REPOSITORY", "")
|
||||
owner, _sep, repo = repository.partition("/")
|
||||
if not owner or not repo:
|
||||
owner, repo = REPO_OWNER, REPO_NAME
|
||||
|
||||
if not all([gitea_url, token, run_id]):
|
||||
click.echo(_("GITEA_URL/CI_GITEA_TOKEN/RUN_ID not set; running without cross-runner cancellation."))
|
||||
|
||||
# When devx is installed as a pip package, __file__ resolves to the
|
||||
# site-packages directory, not the repo root. Use GITHUB_WORKSPACE
|
||||
# (set by Gitea Actions) or cwd as the repo root.
|
||||
repo_root = Path(os.environ.get("GITHUB_WORKSPACE", os.getcwd())).resolve()
|
||||
|
||||
base_env = os.environ.copy()
|
||||
base_env.setdefault("DOCKER_HOST", f"unix:///run/user/{os.getuid()}/docker.sock")
|
||||
base_env.setdefault("ANSIBLE_INJECT_INVOCATION", "1")
|
||||
|
||||
stop_event = threading.Event()
|
||||
failed_event = threading.Event()
|
||||
|
||||
if gitea_url and token and run_id:
|
||||
poller = threading.Thread(
|
||||
target=poll_for_other_failures,
|
||||
args=(
|
||||
gitea_url,
|
||||
owner,
|
||||
repo,
|
||||
token,
|
||||
run_id,
|
||||
job_name,
|
||||
current_index,
|
||||
stop_event,
|
||||
failed_event,
|
||||
),
|
||||
daemon=True,
|
||||
)
|
||||
poller.start()
|
||||
|
||||
try:
|
||||
for pair in pairs:
|
||||
if failed_event.is_set():
|
||||
sys.exit(1)
|
||||
|
||||
role, scenario, platform_name, _img, _cmd = parse_pair(pair)
|
||||
click.echo(_("Running: {scenario} on {platform}", scenario=scenario, platform=platform_name))
|
||||
|
||||
cmd = build_molecule_cmd(scenario)
|
||||
env = build_env_for_pair(pair, base_env)
|
||||
cwd = resolve_role_dir(role, roles_root, repo_root)
|
||||
|
||||
process = subprocess.Popen( # nosec B603
|
||||
cmd,
|
||||
cwd=str(cwd),
|
||||
env=env,
|
||||
preexec_fn=os.setsid,
|
||||
)
|
||||
|
||||
try:
|
||||
while process.poll() is None:
|
||||
if failed_event.is_set():
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
|
||||
try:
|
||||
process.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.killpg(os.getpgid(process.pid), signal.SIGKILL)
|
||||
process.wait()
|
||||
sys.exit(1)
|
||||
time.sleep(1)
|
||||
except KeyboardInterrupt:
|
||||
with contextlib.suppress(ProcessLookupError):
|
||||
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
|
||||
process.wait()
|
||||
sys.exit(1)
|
||||
|
||||
rc = process.returncode
|
||||
|
||||
if rc != 0:
|
||||
click.echo(_("FAILED: {pair} exited with code {code}", pair=pair, code=rc))
|
||||
sys.exit(rc)
|
||||
|
||||
click.echo(_("PASSED: {pair}", pair=pair))
|
||||
|
||||
# Prune Docker data between scenarios to prevent disk exhaustion
|
||||
# in Docker-in-Docker molecule containers (each scenario pulls
|
||||
# hundreds of MB of images that accumulate across pairs).
|
||||
with contextlib.suppress(subprocess.SubprocessError, OSError):
|
||||
subprocess.run( # nosec B603, B607
|
||||
["docker", "system", "prune", "-af", "--volumes"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
timeout=60,
|
||||
)
|
||||
|
||||
click.echo(_("All molecule tests passed."))
|
||||
finally:
|
||||
stop_event.set()
|
||||
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
cli()
|
||||
@@ -10,12 +10,6 @@ If the host socket is not available, it tries the rootless socket, then
|
||||
starts a local ``dockerd`` with the vfs storage driver (requires
|
||||
privileged container).
|
||||
|
||||
When the host socket IS available but has limited disk space (e.g. an
|
||||
inner DinD daemon writing to a 38 GB container overlay), the script
|
||||
prefers a rootless socket that has more available space. This prevents
|
||||
"no space left on device" errors during molecule tests that pull images
|
||||
and create containers via the Docker daemon.
|
||||
|
||||
Usage::
|
||||
|
||||
python3 -m devx.molecule.start_docker [--timeout 30]
|
||||
@@ -23,35 +17,21 @@ Usage::
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import subprocess # nosec B404
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import click
|
||||
|
||||
from devx.i18n import _
|
||||
from devx.molecule.lease import LEASE_UNTIL_ENV, OWNER_ENV, emit_github_env, lease_env
|
||||
|
||||
DEFAULT_TIMEOUT = 30
|
||||
DOCKER_SOCK = "/var/run/docker.sock"
|
||||
# Rootless socket fallback (e.g. /run/user/994/docker.sock)
|
||||
ROOTLESS_SOCK = f"/run/user/{os.getuid()}/docker.sock"
|
||||
# Host Docker socket mounted by gitea_runner config (see runner config
|
||||
# ``options: "-v /run/user/<uid>/docker.sock:/run/host-docker.sock"``).
|
||||
# This gives CI containers access to the host's rootless Docker daemon,
|
||||
# which has the full host filesystem (e.g. 455 GB) instead of the
|
||||
# container's limited overlay (e.g. 38 GB).
|
||||
HOST_DOCKER_SOCK = "/run/host-docker.sock"
|
||||
# Minimum free bytes for a Docker daemon to be considered usable.
|
||||
# Below this, image pulls and container creation will fail with ENOSPC.
|
||||
# 20 GB leaves room for molecule-test-base (~500 MB) + a few containers.
|
||||
MIN_FREE_BYTES = 20 * 1024**3 # 20 GB
|
||||
|
||||
|
||||
def is_docker_ready() -> bool:
|
||||
@@ -66,46 +46,6 @@ def is_docker_ready() -> bool:
|
||||
return result.returncode == 0
|
||||
|
||||
|
||||
def _get_docker_free_bytes() -> int:
|
||||
"""Get free disk space (bytes) at the Docker daemon's data root.
|
||||
|
||||
Returns 0 if the daemon is not reachable or the data root cannot be
|
||||
determined.
|
||||
"""
|
||||
docker_host = os.environ.get("DOCKER_HOST", f"unix://{DOCKER_SOCK}")
|
||||
try:
|
||||
result = subprocess.run( # nosec B603 B607
|
||||
[
|
||||
"docker",
|
||||
"info",
|
||||
"--format",
|
||||
"{{.DockerRootDir}}",
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
check=False,
|
||||
env={**os.environ, "DOCKER_HOST": docker_host},
|
||||
)
|
||||
if result.returncode != 0 or not result.stdout.strip():
|
||||
return 0
|
||||
data_root = result.stdout.strip()
|
||||
if not os.path.exists(data_root):
|
||||
return 0
|
||||
return shutil.disk_usage(data_root).free
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
|
||||
return 0
|
||||
|
||||
|
||||
def _try_socket(sock_path: str) -> bool:
|
||||
"""Set DOCKER_HOST to *sock_path* and check if the daemon is ready.
|
||||
|
||||
Returns ``True`` if the daemon responds, ``False`` otherwise.
|
||||
"""
|
||||
os.environ["DOCKER_HOST"] = f"unix://{sock_path}"
|
||||
return is_docker_ready()
|
||||
|
||||
|
||||
def _diagnose_socket() -> None:
|
||||
"""Print diagnostic info about the Docker socket."""
|
||||
click.echo(f"DOCKER_HOST = {os.environ.get('DOCKER_HOST', '(not set)')}")
|
||||
@@ -157,177 +97,50 @@ def _diagnose_socket() -> None:
|
||||
def start_docker_daemon(timeout: int = DEFAULT_TIMEOUT) -> bool:
|
||||
"""Ensure Docker is ready for molecule tests.
|
||||
|
||||
Tries sockets in this order, preferring ones with enough disk space:
|
||||
|
||||
1. Host rootless socket (``/run/host-docker.sock``) — mounted by the
|
||||
gitea runner config, has access to the host's full filesystem
|
||||
(e.g. 455 GB). Preferred over the inner dockerd.
|
||||
2. Default socket (``/var/run/docker.sock``) — may be an inner dockerd
|
||||
started by the CI image (v29.5.3) with data root on the container's
|
||||
limited overlay (e.g. 38 GB, often 100 % full).
|
||||
3. Other rootless sockets (``/run/user/*/docker.sock``).
|
||||
4. Local ``dockerd`` with vfs storage driver — last resort.
|
||||
First tries the host socket. If that works, sets ``DOCKER_HOST`` and
|
||||
returns immediately. If not, tries the rootless socket. If neither
|
||||
works, starts a local ``dockerd`` with vfs storage driver (requires
|
||||
privileged container).
|
||||
|
||||
Returns ``True`` if Docker is ready, ``False`` if it failed to
|
||||
start within the timeout.
|
||||
"""
|
||||
# Point Docker CLI and Python library to the socket explicitly
|
||||
os.environ["DOCKER_HOST"] = f"unix://{DOCKER_SOCK}"
|
||||
|
||||
# Diagnose socket state
|
||||
click.echo("--- Docker socket diagnostics ---")
|
||||
_diagnose_socket()
|
||||
click.echo("--- End diagnostics ---")
|
||||
|
||||
# Collect candidate sockets in priority order.
|
||||
# The host's rootless Docker socket (mounted at /run/host-docker.sock
|
||||
# by the gitea runner config) is preferred — it has access to the
|
||||
# host's full filesystem instead of the container's limited overlay.
|
||||
candidates: list[str] = []
|
||||
if os.path.exists(HOST_DOCKER_SOCK):
|
||||
candidates.append(HOST_DOCKER_SOCK)
|
||||
if os.path.exists(DOCKER_SOCK):
|
||||
candidates.append(DOCKER_SOCK)
|
||||
if os.path.exists(ROOTLESS_SOCK):
|
||||
candidates.append(ROOTLESS_SOCK)
|
||||
for sock in sorted(glob.glob("/run/user/*/docker.sock")):
|
||||
if sock not in candidates:
|
||||
candidates.append(sock)
|
||||
# Check if host Docker is already available
|
||||
if is_docker_ready():
|
||||
click.echo(_("Docker daemon already running"))
|
||||
return True
|
||||
|
||||
# Try each candidate socket — prefer one with enough free space
|
||||
for sock in candidates:
|
||||
click.echo(f"Trying socket: {sock}")
|
||||
if not _try_socket(sock):
|
||||
# Try rootless socket (e.g. /run/user/994/docker.sock)
|
||||
click.echo(f"Trying rootless socket: {ROOTLESS_SOCK}")
|
||||
os.environ["DOCKER_HOST"] = f"unix://{ROOTLESS_SOCK}"
|
||||
if os.path.exists(ROOTLESS_SOCK) and is_docker_ready():
|
||||
click.echo(_("Docker daemon already running"))
|
||||
return True
|
||||
|
||||
# Scan for any rootless sockets at other UIDs
|
||||
for sock in sorted(glob.glob("/run/user/*/docker.sock")):
|
||||
if sock == ROOTLESS_SOCK:
|
||||
continue
|
||||
free_bytes = _get_docker_free_bytes()
|
||||
free_gb = free_bytes / 1024**3
|
||||
click.echo(f" Docker daemon ready (free space: {free_gb:.1f} GB)")
|
||||
if free_bytes >= MIN_FREE_BYTES:
|
||||
click.echo(f"Trying alternative rootless socket: {sock}")
|
||||
os.environ["DOCKER_HOST"] = f"unix://{sock}"
|
||||
if is_docker_ready():
|
||||
click.echo(_("Docker daemon already running"))
|
||||
return True
|
||||
# If free_bytes is 0, the Docker root dir is on the host filesystem
|
||||
# (not accessible from inside the container). This is expected for
|
||||
# the host's rootless Docker — it has the full host disk.
|
||||
# Only trust this for /run/host-docker.sock (known host socket).
|
||||
# For other sockets (e.g. inner dockerd), free_bytes == 0 means
|
||||
# the data root path doesn't exist inside the container — the
|
||||
# inner dockerd may be using the container's full overlay.
|
||||
if free_bytes == 0 and sock == HOST_DOCKER_SOCK:
|
||||
click.echo("Host rootless Docker root dir not accessible from container, using it")
|
||||
return True
|
||||
# If free_bytes is 0 and there are no dockerd processes inside the
|
||||
# container, the socket is the host's Docker (mounted from outside).
|
||||
# The data root is on the host filesystem and has plenty of space.
|
||||
if free_bytes == 0 and sock == DOCKER_SOCK:
|
||||
has_inner_dockerd = False
|
||||
with contextlib.suppress(Exception):
|
||||
pgrep_result = subprocess.run( # nosec B603 B607
|
||||
["pgrep", "-f", "dockerd"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
has_inner_dockerd = pgrep_result.returncode == 0
|
||||
if not has_inner_dockerd:
|
||||
click.echo("No inner dockerd found, socket is host Docker (data root on host), using it")
|
||||
return True
|
||||
click.echo(f" Insufficient space ({free_gb:.1f} GB), trying next...")
|
||||
|
||||
# No socket with sufficient space found.
|
||||
# Don't fall back to the low-space inner dockerd — it will fail
|
||||
# on image pulls. Instead, kill the inner dockerd, clean up its
|
||||
# data root to free space, and start a new dockerd using the
|
||||
# freed space on the container's overlay.
|
||||
click.echo(_("Host Docker not available, starting local dockerd..."))
|
||||
|
||||
# Kill the inner dockerd (started by the CI image) to free its
|
||||
# data root and socket. The inner dockerd uses the container's
|
||||
# overlay (38G, often 100% full). Killing it frees up the
|
||||
# socket and any space used by its containers/volumes.
|
||||
# Use SIGKILL (-9) since the inner dockerd may not respond to SIGTERM.
|
||||
# Try multiple approaches to ensure the inner dockerd is killed.
|
||||
with contextlib.suppress(Exception):
|
||||
result = subprocess.run( # nosec B603 B607
|
||||
["pgrep", "-af", "dockerd"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
if result.stdout.strip():
|
||||
click.echo(f" dockerd processes before kill: {result.stdout.strip()}")
|
||||
|
||||
for pattern in ["dockerd", "dockerd-entrypoint.sh", "containerd"]:
|
||||
with contextlib.suppress(Exception):
|
||||
subprocess.run( # nosec B603 B607
|
||||
["pkill", "-9", "-f", pattern],
|
||||
check=False,
|
||||
timeout=5,
|
||||
)
|
||||
time.sleep(3)
|
||||
|
||||
# Check if dockerd processes are still alive
|
||||
with contextlib.suppress(Exception):
|
||||
result = subprocess.run( # nosec B603 B607
|
||||
["pgrep", "-af", "dockerd"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
if result.stdout.strip():
|
||||
click.echo(f" dockerd processes after kill: {result.stdout.strip()}")
|
||||
# Try killing by PID directly
|
||||
for pid_str in result.stdout.split("\n"):
|
||||
pid = pid_str.split()[0] if pid_str.strip() else ""
|
||||
if pid:
|
||||
with contextlib.suppress(Exception):
|
||||
os.kill(int(pid), 9)
|
||||
time.sleep(2)
|
||||
|
||||
# Verify the inner dockerd is actually dead. If we can still
|
||||
# connect to /var/run/docker.sock, the old daemon is still running
|
||||
# and we need to use a different socket path.
|
||||
old_daemon_alive = False
|
||||
with contextlib.suppress(Exception):
|
||||
result = subprocess.run( # nosec B603 B607
|
||||
["docker", "info"],
|
||||
env={**os.environ, "DOCKER_HOST": f"unix://{DOCKER_SOCK}"},
|
||||
capture_output=True,
|
||||
timeout=5,
|
||||
)
|
||||
old_daemon_alive = result.returncode == 0
|
||||
|
||||
if old_daemon_alive:
|
||||
click.echo(" Inner dockerd still alive, using alternate socket")
|
||||
local_sock = "/dev/shm/docker.sock" # nosec B108
|
||||
else:
|
||||
local_sock = DOCKER_SOCK
|
||||
|
||||
# Clean up the inner dockerd's data root to free space.
|
||||
# The inner dockerd stores images, containers, and volumes here.
|
||||
# Removing them frees up ~2.4GB on the container's overlay.
|
||||
inner_data_root = "/home/grm-ci-runner-*/.local/share/docker"
|
||||
rm_paths = " ".join(f"{inner_data_root}/{d}" for d in ("overlay2", "image", "volumes", "containers"))
|
||||
with contextlib.suppress(Exception):
|
||||
subprocess.run( # nosec B603 B607
|
||||
["sh", "-c", f"rm -rf {rm_paths}"],
|
||||
check=False,
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
# Use a fresh data root. If the inner dockerd is dead, use the
|
||||
# container's overlay (38G, with freed space). If the inner
|
||||
# dockerd is still alive, use /dev/shm (16G tmpfs) — the overlay
|
||||
# is still full because the inner dockerd's data can't be cleaned.
|
||||
docker_data_root = "/dev/shm/docker" if old_daemon_alive else "/tmp/docker-data" # nosec B108
|
||||
|
||||
# Remove stale socket if present
|
||||
with contextlib.suppress(OSError):
|
||||
os.unlink(local_sock)
|
||||
|
||||
os.environ["DOCKER_HOST"] = f"unix://{local_sock}"
|
||||
# Reset DOCKER_HOST to host socket for local dockerd
|
||||
os.environ["DOCKER_HOST"] = f"unix://{DOCKER_SOCK}"
|
||||
|
||||
# Start local dockerd (requires privileged container)
|
||||
# Use /tmp/docker-data as data root on the container's overlay.
|
||||
# The inner dockerd's data root has been cleaned up, freeing ~2.4GB.
|
||||
# vfs storage driver is used since overlay2 may not work inside
|
||||
# a Docker-in-Docker container without --privileged.
|
||||
log_file = tempfile.NamedTemporaryFile( # noqa: SIM115
|
||||
mode="w", suffix="dockerd.log", delete=False
|
||||
)
|
||||
@@ -337,13 +150,8 @@ def start_docker_daemon(timeout: int = DEFAULT_TIMEOUT) -> bool:
|
||||
"dockerd",
|
||||
"--storage-driver",
|
||||
"vfs",
|
||||
"--data-root",
|
||||
docker_data_root,
|
||||
"--iptables=false",
|
||||
"--ip6tables=false",
|
||||
"--bridge=none",
|
||||
"-H",
|
||||
f"unix://{local_sock}",
|
||||
f"unix://{DOCKER_SOCK}",
|
||||
],
|
||||
stdout=log_file,
|
||||
stderr=subprocess.STDOUT,
|
||||
@@ -386,15 +194,6 @@ def main(timeout: int) -> None:
|
||||
with open(github_env, "a", encoding="utf-8") as f:
|
||||
f.write(f"DOCKER_HOST={os.environ['DOCKER_HOST']}\n")
|
||||
click.echo(f"Exported DOCKER_HOST={os.environ['DOCKER_HOST']} to GITHUB_ENV")
|
||||
# Implements: REQ-2 (DEVX-181) — emit the S08 producer lease env so
|
||||
# molecule-created objects carry org.oblachno.lease-until/owner
|
||||
# labels that runner-cleanup.sh honors.
|
||||
# Implements: REQ-1 (DEVX-182) — one lease_env() call; log the
|
||||
# owner and UTC expiry an operator needs for cleanup debugging.
|
||||
env = lease_env()
|
||||
if emit_github_env(env):
|
||||
until = datetime.fromtimestamp(int(env[LEASE_UNTIL_ENV]), tz=UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
click.echo(f"Exported lease vars to GITHUB_ENV (owner={env[OWNER_ENV]}, until={until})")
|
||||
sys.exit(0)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
+16
-2
@@ -2,16 +2,19 @@
|
||||
|
||||
Centralizes Gitea/Vikunja token discovery with role-based environment
|
||||
variable names and backwards compatibility with the legacy
|
||||
``CI_GITEA_TOKEN`` naming convention.
|
||||
``CI_GITEA_TOKEN`` / ``REVIEW_GITEA_TOKEN`` naming convention.
|
||||
|
||||
Roles:
|
||||
- ``CI_GITEA_API_TOKEN``: CI workflows (read actions, post status, merge, etc.)
|
||||
- ``REVIEWER_GITEA_API_TOKEN``: PR approval reviews (must be a different user
|
||||
from the PR author for Gitea to accept the review as an approval)
|
||||
- ``DEVELOPER_GITEA_API_TOKEN``: local development tools (create-task,
|
||||
create-pr, setup, etc.)
|
||||
|
||||
Fallbacks:
|
||||
- New role names are checked first.
|
||||
- Legacy names (``CI_GITEA_TOKEN``) are accepted for backwards compatibility.
|
||||
- Legacy names (``CI_GITEA_TOKEN``, ``REVIEW_GITEA_TOKEN``) are accepted for
|
||||
backwards compatibility.
|
||||
- If no role-specific token is set, the generic CI tokens are tried last.
|
||||
"""
|
||||
|
||||
@@ -25,6 +28,12 @@ from devx.i18n import _
|
||||
|
||||
# Token environment variable names, in lookup priority order.
|
||||
CI_TOKEN_NAMES = ["CI_GITEA_API_TOKEN", "CI_GITEA_TOKEN"]
|
||||
REVIEWER_TOKEN_NAMES = [
|
||||
"REVIEWER_GITEA_API_TOKEN",
|
||||
# Legacy name used before role-based tokens.
|
||||
"REVIEW_GITEA_TOKEN",
|
||||
*CI_TOKEN_NAMES,
|
||||
]
|
||||
DEVELOPER_TOKEN_NAMES = ["DEVELOPER_GITEA_API_TOKEN", *CI_TOKEN_NAMES]
|
||||
|
||||
VIKUNJA_TOKEN_NAMES = ["VIKUNJA_TOKEN"]
|
||||
@@ -52,6 +61,11 @@ def get_ci_token() -> str:
|
||||
return get_token(*CI_TOKEN_NAMES)
|
||||
|
||||
|
||||
def get_reviewer_token() -> str:
|
||||
"""Resolve the reviewer Gitea API token used for PR approvals."""
|
||||
return get_token(*REVIEWER_TOKEN_NAMES)
|
||||
|
||||
|
||||
def get_developer_token() -> str:
|
||||
"""Resolve the developer Gitea API token used for local tooling."""
|
||||
return get_token(*DEVELOPER_TOKEN_NAMES)
|
||||
|
||||
+10
-168
@@ -40,17 +40,13 @@ and ``CI_GITEA_USERNAME`` environment variables, matching the existing CI workfl
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import subprocess # nosec B404
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_exponential
|
||||
|
||||
from devx.i18n import _
|
||||
from devx.tokens import get_developer_token
|
||||
@@ -192,188 +188,38 @@ def build_image(
|
||||
return True
|
||||
|
||||
|
||||
def delete_remote_manifest(
|
||||
registry: str,
|
||||
name: str,
|
||||
tag: str,
|
||||
username: str,
|
||||
token: str,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
) -> bool:
|
||||
"""Delete an existing manifest from the Gitea container registry.
|
||||
|
||||
Gitea 1.27 has a bug (#31964) where pushing a tag that already exists
|
||||
fails with HTTP 500 "package version already exists". This function
|
||||
deletes the existing manifest before the push to work around it.
|
||||
|
||||
Returns True if deleted or not found, False on unexpected errors.
|
||||
"""
|
||||
manifest_url = f"https://{registry}/v2/{name}/manifests/{tag}"
|
||||
if dry_run:
|
||||
click.echo(f"[dry-run] DELETE {manifest_url}")
|
||||
return True
|
||||
|
||||
# First, get the digest via HEAD
|
||||
req = urllib.request.Request(manifest_url, method="HEAD") # nosec B310
|
||||
auth_str = f"{username}:{token}"
|
||||
req.add_header("Authorization", f"Basic {base64.b64encode(auth_str.encode()).decode()}")
|
||||
req.add_header("Accept", "application/vnd.docker.distribution.manifest.v2+json")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp: # nosec B310
|
||||
digest = resp.headers.get("Docker-Content-Digest")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return True # Tag doesn't exist — nothing to delete
|
||||
if e.code == 405:
|
||||
# HEAD not supported — try GET with a range
|
||||
pass
|
||||
else:
|
||||
click.echo(f" Warning: HEAD {tag} returned {e.code}", err=True)
|
||||
return True # Don't block the push
|
||||
except urllib.error.URLError as e:
|
||||
click.echo(f" Warning: HEAD {tag} failed: {e}", err=True)
|
||||
return True # Don't block the push
|
||||
else:
|
||||
if not digest:
|
||||
return True
|
||||
# Delete by digest
|
||||
del_url = f"https://{registry}/v2/{name}/manifests/{digest}"
|
||||
del_req = urllib.request.Request(del_url, method="DELETE") # nosec B310
|
||||
del_req.add_header("Authorization", f"Basic {base64.b64encode(auth_str.encode()).decode()}")
|
||||
try:
|
||||
with urllib.request.urlopen(del_req, timeout=30) as resp: # nosec B310
|
||||
click.echo(f" Deleted existing {tag} (digest: {digest[:19]}...)")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return True # Already gone
|
||||
click.echo(f" Warning: DELETE {tag} returned {e.code}", err=True)
|
||||
return True # Don't block the push
|
||||
except urllib.error.URLError as e:
|
||||
click.echo(f" Warning: DELETE {tag} failed: {e}", err=True)
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
class PushHTTP500Error(Exception):
|
||||
"""Raised when docker push fails with an HTTP 500 from the registry."""
|
||||
|
||||
|
||||
def _run_push(cmd: list[str]) -> subprocess.CompletedProcess[str]:
|
||||
"""Run a docker push command, raising PushHTTP500Error on registry 500.
|
||||
|
||||
The Gitea container registry (v1.27.x) has a race condition in
|
||||
BlobUploader.Append that causes intermittent HTTP 500 "offset
|
||||
mismatch" errors during concurrent blob uploads. Retrying the
|
||||
push gives the registry time to recover.
|
||||
|
||||
Docker sends push progress/errors to both stdout and stderr depending
|
||||
on the error type, so both streams are checked for the 500 status.
|
||||
Uses stderr=STDOUT to merge both streams into stdout, ensuring all
|
||||
output is captured in one place (docker push output behavior varies
|
||||
depending on TTY detection).
|
||||
"""
|
||||
result = subprocess.run( # nosec B603
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0 and "500" in (result.stdout or ""):
|
||||
raise PushHTTP500Error(result.stdout.strip())
|
||||
return result
|
||||
|
||||
|
||||
def push_image(
|
||||
spec: ImageSpec,
|
||||
registry: str,
|
||||
*,
|
||||
dry_run: bool = False,
|
||||
username: str = "",
|
||||
token: str = "",
|
||||
) -> bool:
|
||||
"""Push all tags of a Docker image to the registry.
|
||||
|
||||
Returns True if all pushes succeed, False if any fail.
|
||||
|
||||
Push-first strategy: try pushing directly. Only if the push fails
|
||||
with Gitea #31964 ("package version already exists") do we delete
|
||||
the old manifest and retry. This avoids losing the existing tag
|
||||
when the push fails for unrelated reasons (e.g. HTTP 500).
|
||||
|
||||
HTTP 500 errors from the Gitea registry race condition are retried
|
||||
up to 3 times with exponential backoff (5s, 10s) via tenacity.
|
||||
"""
|
||||
full_tags = [build_full_tag(registry, spec.name, t) for t in spec.tags]
|
||||
all_ok = True
|
||||
for ft, tag in zip(full_tags, spec.tags, strict=False):
|
||||
for ft in full_tags:
|
||||
cmd = ["docker", "push", ft]
|
||||
if dry_run:
|
||||
click.echo(f"[dry-run] {' '.join(cmd)}")
|
||||
continue
|
||||
click.echo(f"Pushing {ft}...")
|
||||
|
||||
@retry(
|
||||
stop=stop_after_attempt(5),
|
||||
wait=wait_exponential(multiplier=10, min=10, max=60),
|
||||
retry=retry_if_exception_type(PushHTTP500Error),
|
||||
before_sleep=lambda retry_state: click.echo(
|
||||
_(
|
||||
" HTTP 500 from registry, retrying in {wait:.0f}s (attempt {attempt}/5)...",
|
||||
wait=retry_state.next_action.sleep if retry_state.next_action else 0,
|
||||
attempt=retry_state.attempt_number + 1,
|
||||
),
|
||||
err=True,
|
||||
),
|
||||
reraise=True,
|
||||
result = subprocess.run( # nosec B603
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
def _attempt(_cmd: list[str] = cmd) -> subprocess.CompletedProcess[str]:
|
||||
return _run_push(_cmd)
|
||||
|
||||
try:
|
||||
result = _attempt()
|
||||
except PushHTTP500Error as e:
|
||||
if result.returncode != 0:
|
||||
click.echo(
|
||||
_("Push failed for {tag}: {error}", tag=ft, error=str(e)),
|
||||
_("Push failed for {tag}: {error}", tag=ft, error=result.stderr.strip()),
|
||||
err=True,
|
||||
)
|
||||
all_ok = False
|
||||
continue
|
||||
|
||||
if result.returncode == 0:
|
||||
else:
|
||||
click.echo(f"Pushed {ft}")
|
||||
continue
|
||||
combined_output = (result.stdout or "").strip()
|
||||
# Gitea #31964: push fails because tag already exists.
|
||||
# Delete the old manifest and retry once.
|
||||
if username and token and "already exists" in combined_output.lower():
|
||||
click.echo(" Tag exists (Gitea #31964), deleting old manifest and retrying...")
|
||||
delete_remote_manifest(
|
||||
registry,
|
||||
spec.name,
|
||||
tag,
|
||||
username,
|
||||
token,
|
||||
dry_run=dry_run,
|
||||
)
|
||||
click.echo(f" Retrying push {ft}...")
|
||||
result = subprocess.run( # nosec B603
|
||||
cmd,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
click.echo(f"Pushed {ft} (after retry)")
|
||||
continue
|
||||
combined_output = (result.stdout or "").strip()
|
||||
click.echo(
|
||||
_("Push failed for {tag}: {error}", tag=ft, error=combined_output),
|
||||
err=True,
|
||||
)
|
||||
all_ok = False
|
||||
return all_ok
|
||||
|
||||
|
||||
@@ -474,15 +320,11 @@ def main(
|
||||
raise click.ClickException(_("Registry login failed"))
|
||||
|
||||
failed: list[str] = []
|
||||
push_username = "" # nosec B105
|
||||
push_token = "" # nosec B105
|
||||
if push:
|
||||
push_username, push_token = _get_registry_creds()
|
||||
for spec in specs:
|
||||
if not build_image(spec, registry, dry_run=dry_run, pull=pull):
|
||||
failed.append(spec.name)
|
||||
continue
|
||||
if push and not push_image(spec, registry, dry_run=dry_run, username=push_username, token=push_token): # type: ignore[arg-type]
|
||||
if push and not push_image(spec, registry, dry_run=dry_run): # type: ignore[arg-type]
|
||||
failed.append(spec.name)
|
||||
|
||||
if failed:
|
||||
|
||||
@@ -1,232 +0,0 @@
|
||||
"""Check Ansible tasks for missing no_log on secret-handling tasks.
|
||||
|
||||
ansible-lint's built-in ``no-log-password`` rule only fires when a module
|
||||
parameter is literally named ``*password*`` and there's a loop. It does
|
||||
NOT catch:
|
||||
|
||||
- Shell/command tasks that interpolate ``{{ _secrets.* }}`` or
|
||||
``{{ *password* }}`` variables
|
||||
- Template/copy tasks that render secret values without ``no_log``
|
||||
|
||||
This script fills that gap by scanning all Ansible task files for
|
||||
variables that look like secrets (``_secrets.*``, ``*password*``,
|
||||
``*secret*``, ``*token*``, ``*api_key*``) and verifying that the task
|
||||
has ``no_log`` set to a non-False value.
|
||||
|
||||
Usage::
|
||||
|
||||
python -m devx.tools.check_ansible_no_log
|
||||
python -m devx.tools.check_ansible_no_log --path ansible/roles/my_role
|
||||
python -m devx.tools.check_ansible_no_log --ansible-dir ansible/roles
|
||||
|
||||
Exit code 0 if all secret-handling tasks have no_log, 1 otherwise.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import yaml
|
||||
|
||||
REPO_ROOT = Path.cwd()
|
||||
DEFAULT_ANSIBLE_DIR = REPO_ROOT / "ansible"
|
||||
|
||||
# Patterns that indicate a task is handling secrets.
|
||||
# We only match Jinja-interpolated variables ({{ ... }}) to avoid false
|
||||
# positives from field names like "password" in module params or task names.
|
||||
SECRET_PATTERNS = [
|
||||
# {{ _secrets.anything }} or {{ _secrets['anything'] }}
|
||||
re.compile(r"\{\{[^}]*_secrets\.", re.IGNORECASE),
|
||||
# {{ anything_password }} but NOT the word "password" in a string literal
|
||||
re.compile(r"\{\{[^}]*password", re.IGNORECASE),
|
||||
# {{ anything_secret }}
|
||||
re.compile(r"\{\{[^}]*_secret\b", re.IGNORECASE),
|
||||
# {{ anything_api_key }}
|
||||
re.compile(r"\{\{[^}]*api_key", re.IGNORECASE),
|
||||
# {{ anything_token }} (but not loop tokens like {{ loop_token }})
|
||||
re.compile(r"\{\{[^}]*(?:vault_token|auth_token|access_token|bot_token)", re.IGNORECASE),
|
||||
]
|
||||
|
||||
# Task keys whose values might contain secret references
|
||||
TASK_VALUE_KEYS = {
|
||||
"shell",
|
||||
"command",
|
||||
"ansible.builtin.shell",
|
||||
"ansible.builtin.command",
|
||||
"ansible.builtin.template",
|
||||
"ansible.builtin.copy",
|
||||
"ansible.builtin.debug",
|
||||
"template",
|
||||
"copy",
|
||||
"debug",
|
||||
"cmd",
|
||||
"msg",
|
||||
"content",
|
||||
}
|
||||
|
||||
# Keys that are NOT secret-bearing (task metadata, not values)
|
||||
NON_VALUE_KEYS = {
|
||||
"name",
|
||||
"when",
|
||||
"loop",
|
||||
"loop_control",
|
||||
"changed_when",
|
||||
"failed_when",
|
||||
"no_log",
|
||||
"register",
|
||||
"tags",
|
||||
"vars",
|
||||
"become",
|
||||
"become_user",
|
||||
"delegate_to",
|
||||
"run_once",
|
||||
"environment",
|
||||
"with_items",
|
||||
"with_dict",
|
||||
"with_list",
|
||||
}
|
||||
|
||||
|
||||
def _contains_secret(value: object) -> bool:
|
||||
"""Recursively check if a value contains secret-like variable references."""
|
||||
if isinstance(value, str):
|
||||
return any(p.search(value) for p in SECRET_PATTERNS)
|
||||
if isinstance(value, dict):
|
||||
return any(_contains_secret(v) for v in value.values())
|
||||
if isinstance(value, list):
|
||||
return any(_contains_secret(item) for item in value)
|
||||
return False
|
||||
|
||||
|
||||
def _has_no_log(task: dict) -> bool:
|
||||
"""Check if a task has no_log set to a non-False value."""
|
||||
no_log = task.get("no_log", False)
|
||||
# Jinja expressions (e.g. "{{ not debug_mode }}") count as set
|
||||
return no_log is not False and no_log is not None
|
||||
|
||||
|
||||
def _check_task(task: dict, file_path: Path, task_num: int) -> list[str]:
|
||||
"""Check a single task for missing no_log on secret values.
|
||||
|
||||
Returns a list of violation messages (empty if OK).
|
||||
"""
|
||||
violations: list[str] = []
|
||||
|
||||
# Skip tasks that already have no_log
|
||||
if _has_no_log(task):
|
||||
return violations
|
||||
|
||||
# Check all string values in the task for secret references
|
||||
has_secrets = False
|
||||
for key, value in task.items():
|
||||
if key in NON_VALUE_KEYS:
|
||||
continue
|
||||
# Check action module params (shell, command, copy, template, etc.)
|
||||
if _contains_secret(value):
|
||||
has_secrets = True
|
||||
break
|
||||
|
||||
if has_secrets:
|
||||
task_name = task.get("name", "<unnamed>")
|
||||
violations.append(
|
||||
f"{file_path}:{task_num}: Task '{task_name}' references secrets "
|
||||
f"but has no no_log. Add `no_log: true` or "
|
||||
f'`no_log: "{{{{ not (debug_mode | default(false) | bool) }}}}"` '
|
||||
f"to prevent credential leakage in Ansible output."
|
||||
)
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
def check_directory(ansible_dir: Path) -> list[str]:
|
||||
"""Check all Ansible task files in a directory tree."""
|
||||
all_violations: list[str] = []
|
||||
|
||||
# Find all task files
|
||||
task_files = list(ansible_dir.rglob("tasks/*.yml"))
|
||||
task_files += list(ansible_dir.rglob("tasks/*.yaml"))
|
||||
# Also check playbook files
|
||||
task_files += list(ansible_dir.glob("playbooks/*.yml"))
|
||||
|
||||
for task_file in sorted(task_files):
|
||||
# Skip molecule test files
|
||||
if "molecule" in task_file.parts:
|
||||
continue
|
||||
|
||||
try:
|
||||
with task_file.open() as f:
|
||||
docs = list(yaml.safe_load_all(f))
|
||||
except (yaml.YAMLError, OSError):
|
||||
continue
|
||||
|
||||
for doc in docs:
|
||||
if not doc:
|
||||
continue
|
||||
|
||||
# Task files are bare lists of tasks; playbook files are
|
||||
# lists of plays (each play is a dict with 'hosts' key)
|
||||
if isinstance(doc, list):
|
||||
is_plays = isinstance(doc[0], dict) and "hosts" in doc[0]
|
||||
if not is_plays:
|
||||
for i, task in enumerate(doc):
|
||||
if not isinstance(task, dict):
|
||||
continue
|
||||
all_violations.extend(_check_task(task, task_file, i + 1))
|
||||
continue
|
||||
plays = doc
|
||||
elif isinstance(doc, dict):
|
||||
plays = [doc]
|
||||
else:
|
||||
continue
|
||||
|
||||
for play in plays:
|
||||
if not isinstance(play, dict):
|
||||
continue
|
||||
for task_section in ("tasks", "pre_tasks", "post_tasks", "handlers"):
|
||||
tasks = play.get(task_section, [])
|
||||
if not isinstance(tasks, list):
|
||||
continue
|
||||
for i, task in enumerate(tasks):
|
||||
if not isinstance(task, dict):
|
||||
continue
|
||||
all_violations.extend(_check_task(task, task_file, i + 1))
|
||||
|
||||
return all_violations
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--path",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
help="Check a specific file or directory (default: ansible/).",
|
||||
)
|
||||
@click.option(
|
||||
"--ansible-dir",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
default=None,
|
||||
help="Override the default ansible directory (default: ansible/).",
|
||||
)
|
||||
def main(path: Path | None, ansible_dir: Path | None) -> None:
|
||||
"""Check that Ansible tasks handling secrets have no_log set."""
|
||||
target = path or ansible_dir or DEFAULT_ANSIBLE_DIR
|
||||
if not target.is_dir():
|
||||
click.echo(f"Error: {target} is not a directory", err=True)
|
||||
sys.exit(2)
|
||||
|
||||
violations = check_directory(target)
|
||||
|
||||
if violations:
|
||||
click.echo(f"Found {len(violations)} task(s) handling secrets without no_log:\n")
|
||||
for v in violations:
|
||||
click.echo(f" {v}")
|
||||
click.echo(f"\nTotal: {len(violations)} violation(s).")
|
||||
sys.exit(1)
|
||||
|
||||
click.echo(f"[check-ansible-no-log] All secret-handling tasks have no_log. ({target})")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -1,176 +0,0 @@
|
||||
"""Check Ansible tasks for ``state: absent`` on database data directories.
|
||||
|
||||
This is a static analysis lint check that runs in CI (``make lint-ci``)
|
||||
to prevent the class of bug that caused the 2026-07-22 production outage
|
||||
(ADR-0028): a ``state: absent`` on a PostgreSQL data directory path that
|
||||
fired on every deploy and wiped the ZITADEL database.
|
||||
|
||||
The existing unit test ``scripts/tests/test_no_zitadel_db_wipe.py`` covers
|
||||
the same concern as a regression test. This lint check runs earlier in
|
||||
the pipeline (before tests) and covers ALL roles and playbooks, not just
|
||||
the ZITADEL role.
|
||||
|
||||
Allowed contexts (where DB recreation is legitimate):
|
||||
- PostgreSQL major version upgrades (``upgrade-postgres``, ``PG_VERSION``)
|
||||
- Explicit ``# lint:allow-state-absent`` comment on the task
|
||||
|
||||
Usage::
|
||||
|
||||
python -m devx.tools.check_ansible_no_state_absent_on_db
|
||||
python -m devx.tools.check_ansible_no_state_absent_on_db --path ansible/roles/zitadel/tasks/main.yml
|
||||
|
||||
Exit code 0 if no violations found, 1 otherwise.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
|
||||
REPO_ROOT = Path.cwd()
|
||||
DEFAULT_ANSIBLE_DIRS: list[Path] = [
|
||||
REPO_ROOT / "ansible" / "playbooks",
|
||||
REPO_ROOT / "ansible" / "roles",
|
||||
]
|
||||
|
||||
# Database data directory path patterns.
|
||||
# These match the DIRECTORY path, not individual files within it.
|
||||
# Removing a stale config file (e.g. postgresql.conf) is safe; removing
|
||||
# the entire data directory is not.
|
||||
DB_PATH_PATTERNS = (
|
||||
re.compile(r"postgres/zitadel-db", re.IGNORECASE),
|
||||
re.compile(r"postgres/\w+-db", re.IGNORECASE),
|
||||
re.compile(r"/var/lib/postgresql/data", re.IGNORECASE),
|
||||
re.compile(r"/var/lib/postgresql/data/\w+-db", re.IGNORECASE),
|
||||
)
|
||||
|
||||
# Destructive operations
|
||||
DESTRUCTIVE_PATTERNS = (
|
||||
re.compile(r"state:\s*absent", re.IGNORECASE),
|
||||
re.compile(r"rm\s+-rf.*\bdb\b", re.IGNORECASE),
|
||||
)
|
||||
|
||||
# Allowed contexts where DB recreation is legitimate
|
||||
ALLOWED_CONTEXT_KEYWORDS = (
|
||||
"upgrade-postgres",
|
||||
"PG_VERSION",
|
||||
"pg_version",
|
||||
)
|
||||
|
||||
# Comment marker to explicitly allow state: absent on a specific task
|
||||
ALLOW_MARKER = "lint:allow-state-absent"
|
||||
|
||||
|
||||
def _find_task_files(base: Path) -> list[Path]:
|
||||
"""Find all YAML task files under a base directory, skipping molecule."""
|
||||
if base.is_file() and base.suffix in (".yml", ".yaml"):
|
||||
return [base]
|
||||
if not base.is_dir():
|
||||
return []
|
||||
files: list[Path] = []
|
||||
for f in sorted(base.rglob("*.yml")) + sorted(base.rglob("*.yaml")):
|
||||
if "molecule" in f.parts:
|
||||
continue
|
||||
files.append(f)
|
||||
return files
|
||||
|
||||
|
||||
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
|
||||
"""Check a YAML file for state: absent on DB data directory paths.
|
||||
|
||||
Returns a list of violation messages (empty if clean).
|
||||
"""
|
||||
try:
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return []
|
||||
|
||||
# Quick check: if no DB path pattern appears anywhere, skip
|
||||
if not any(p.search(content) for p in DB_PATH_PATTERNS):
|
||||
return []
|
||||
|
||||
try:
|
||||
display_path = filepath.relative_to(repo_root)
|
||||
except ValueError:
|
||||
display_path = filepath
|
||||
|
||||
violations: list[str] = []
|
||||
lines = content.splitlines()
|
||||
|
||||
for i, line in enumerate(lines):
|
||||
for db_pattern in DB_PATH_PATTERNS:
|
||||
if not db_pattern.search(line):
|
||||
continue
|
||||
|
||||
# Check surrounding context (±5 lines) for destructive operations
|
||||
context_start = max(0, i - 5)
|
||||
context_end = min(len(lines), i + 6)
|
||||
context = "\n".join(lines[context_start:context_end])
|
||||
|
||||
# Skip if in an allowed context (PG upgrade)
|
||||
if any(kw in context for kw in ALLOWED_CONTEXT_KEYWORDS):
|
||||
continue
|
||||
|
||||
# Skip if the allow marker comment is in the context
|
||||
if ALLOW_MARKER in context:
|
||||
continue
|
||||
|
||||
for dp in DESTRUCTIVE_PATTERNS:
|
||||
if dp.search(context):
|
||||
violations.append(
|
||||
f"{display_path}:{i + 1} — destructive operation "
|
||||
f"({dp.pattern!r}) near DB data directory path "
|
||||
f"({db_pattern.pattern!r}). "
|
||||
f"Database directories must never be wiped automatically (ADR-0028). "
|
||||
f"If this is legitimate (e.g. PG upgrade), add "
|
||||
f"#{ALLOW_MARKER} to the task."
|
||||
)
|
||||
break
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--path",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
|
||||
)
|
||||
@click.option(
|
||||
"--ansible-dir",
|
||||
"ansible_dirs",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
multiple=True,
|
||||
default=None,
|
||||
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
|
||||
)
|
||||
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
|
||||
"""Check that no Ansible task uses state: absent on a DB data directory."""
|
||||
dirs = list(ansible_dirs) if ansible_dirs else DEFAULT_ANSIBLE_DIRS
|
||||
if path:
|
||||
files = _find_task_files(path)
|
||||
else:
|
||||
files: list[Path] = []
|
||||
for d in dirs:
|
||||
files.extend(_find_task_files(d))
|
||||
|
||||
all_violations: list[str] = []
|
||||
for f in files:
|
||||
all_violations.extend(_check_file(f, REPO_ROOT))
|
||||
|
||||
if all_violations:
|
||||
click.echo("[check-ansible-no-state-absent-on-db] FAIL: destructive operations on DB paths:")
|
||||
for v in all_violations:
|
||||
click.echo(f" - {v}")
|
||||
click.echo(f"\nTotal: {len(all_violations)} violation(s).")
|
||||
click.echo("Database data directories must never be wiped automatically (ADR-0028).")
|
||||
sys.exit(1)
|
||||
else:
|
||||
click.echo("[check-ansible-no-state-absent-on-db] OK: no destructive operations on DB paths.")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -1,345 +0,0 @@
|
||||
"""Check Ansible tasks for dangerous patterns that mask failures.
|
||||
|
||||
This check addresses the gap identified in the testing-strategy audit:
|
||||
the automated PR review only checks Python files, and ``ansible-lint``
|
||||
runs at ``profile: basic`` which does not catch dangerous patterns like:
|
||||
|
||||
- ``|| true`` on tasks that are NOT cleanup/idempotency operations
|
||||
- ``failed_when: false`` on critical tasks (e.g. DB operations)
|
||||
- ``2>/dev/null`` on tasks where stderr contains important diagnostics
|
||||
|
||||
Most ``|| true`` and ``2>/dev/null`` instances in the codebase are
|
||||
legitimate (container removal, journalctl, apt-get, docker prune, SUID
|
||||
removal). This check flags only instances that are NOT in a known-safe
|
||||
context. Tasks can also opt out with a ``# lint:allow-failure-masking``
|
||||
comment.
|
||||
|
||||
Usage::
|
||||
|
||||
python -m devx.tools.check_ansible_patterns
|
||||
python -m devx.tools.check_ansible_patterns --path ansible/roles/app_container/tasks/main.yml
|
||||
|
||||
Exit code 0 if no violations found, 1 otherwise.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import yaml
|
||||
|
||||
REPO_ROOT = Path.cwd()
|
||||
DEFAULT_ANSIBLE_DIRS: list[Path] = [
|
||||
REPO_ROOT / "ansible" / "playbooks",
|
||||
REPO_ROOT / "ansible" / "roles",
|
||||
]
|
||||
|
||||
# Comment marker to explicitly allow a pattern on a specific task
|
||||
ALLOW_MARKER = "lint:allow-failure-masking"
|
||||
|
||||
# Patterns that mask failures when used in shell/command tasks
|
||||
OR_TRUE_PATTERN = re.compile(r"\|\|\s*true\b", re.IGNORECASE)
|
||||
REDIRECT_DEVNULL_PATTERN = re.compile(r"2>/dev/null")
|
||||
|
||||
# Module keys that accept shell/command strings
|
||||
SHELL_MODULE_KEYS = frozenset(
|
||||
{
|
||||
"shell",
|
||||
"command",
|
||||
"ansible.builtin.shell",
|
||||
"ansible.builtin.command",
|
||||
"cmd",
|
||||
"ansible.builtin.raw",
|
||||
"raw",
|
||||
}
|
||||
)
|
||||
|
||||
# Task keys whose values might contain shell commands
|
||||
COMMAND_VALUE_KEYS = frozenset(
|
||||
{
|
||||
"shell",
|
||||
"command",
|
||||
"ansible.builtin.shell",
|
||||
"ansible.builtin.command",
|
||||
"cmd",
|
||||
"raw",
|
||||
"ansible.builtin.raw",
|
||||
}
|
||||
)
|
||||
|
||||
# Legitimate contexts where || true or 2>/dev/null are safe.
|
||||
# These are command prefixes or task names that indicate cleanup/idempotency.
|
||||
LEGITIMATE_COMMAND_PREFIXES = (
|
||||
# Container/process removal (may not exist)
|
||||
"docker rm",
|
||||
"docker stop",
|
||||
"docker rmi",
|
||||
"docker network rm",
|
||||
"docker volume rm",
|
||||
"pkill",
|
||||
"kill",
|
||||
# Cleanup commands that are expected to sometimes fail
|
||||
"journalctl --vacuum",
|
||||
"apt-get clean",
|
||||
"apt-get autoremove",
|
||||
"docker image prune",
|
||||
"docker container prune",
|
||||
"docker volume prune",
|
||||
"docker builder prune",
|
||||
"find / -name",
|
||||
# SUID removal (binaries may not exist)
|
||||
"chmod",
|
||||
"rm -f",
|
||||
# Network connection checks (may fail if not connected)
|
||||
"docker network connect",
|
||||
# Prometheus snapshot API (may fail if no snapshot)
|
||||
"curl.*api/v2/admin/tsdb/snapshot",
|
||||
)
|
||||
|
||||
LEGITIMATE_TASK_NAME_KEYWORDS = (
|
||||
"remove",
|
||||
"cleanup",
|
||||
"clean up",
|
||||
"prune",
|
||||
"purge",
|
||||
"disconnect",
|
||||
"stop",
|
||||
"kill",
|
||||
"strip suid",
|
||||
"suid",
|
||||
"vacuum",
|
||||
"ensure.*absent",
|
||||
"may not exist",
|
||||
"if exists",
|
||||
"optional",
|
||||
"best effort",
|
||||
"no-op",
|
||||
"noop",
|
||||
"idempotent",
|
||||
"sync",
|
||||
)
|
||||
|
||||
# Tasks with failed_when: false that are critical and should not mask failures.
|
||||
# Only flag operations that SHOULD fail loudly — writing secrets, provisioning
|
||||
# users, creating OIDC apps. Do NOT flag stop/start/check/wait/migrate/restore
|
||||
# operations where failed_when: false is legitimate (container may not exist,
|
||||
# may already be stopped, etc.).
|
||||
CRITICAL_TASK_KEYWORDS = (
|
||||
"password",
|
||||
"secret",
|
||||
"provision",
|
||||
"oidc",
|
||||
)
|
||||
|
||||
# Task name keywords that indicate failed_when: false is legitimate
|
||||
LEGITIMATE_FAILED_WHEN_KEYWORDS = (
|
||||
"stop",
|
||||
"start",
|
||||
"check",
|
||||
"wait",
|
||||
"migrate",
|
||||
"restart",
|
||||
"rebuild",
|
||||
"restore",
|
||||
"remove",
|
||||
"cleanup",
|
||||
"sync",
|
||||
"download",
|
||||
"extract",
|
||||
"verify",
|
||||
)
|
||||
|
||||
|
||||
def _is_legitimate_or_true(command_str: str, task_name: str) -> bool:
|
||||
"""Check if a || true in a command is in a legitimate context."""
|
||||
# Check task name for legitimate keywords
|
||||
name_lower = task_name.lower()
|
||||
if any(re.search(kw, name_lower) for kw in LEGITIMATE_TASK_NAME_KEYWORDS):
|
||||
return True
|
||||
|
||||
# Check command prefix for legitimate patterns
|
||||
cmd_lower = command_str.lower()
|
||||
return any(re.search(prefix, cmd_lower) for prefix in LEGITIMATE_COMMAND_PREFIXES)
|
||||
|
||||
|
||||
def _is_legitimate_devnull(command_str: str, task_name: str) -> bool:
|
||||
"""Check if a 2>/dev/null in a command is in a legitimate context."""
|
||||
# 2>/dev/null is almost always safe — it suppresses stderr noise.
|
||||
# Only flag it if the task is critical (DB, backup, OIDC) AND
|
||||
# there's no || true (which is the more dangerous pattern).
|
||||
return _is_legitimate_or_true(command_str, task_name)
|
||||
|
||||
|
||||
def _check_task(task: dict, filepath: Path, task_num: int, repo_root: Path) -> list[str]:
|
||||
"""Check a single task for dangerous failure-masking patterns."""
|
||||
violations: list[str] = []
|
||||
|
||||
try:
|
||||
display_path = filepath.relative_to(repo_root)
|
||||
except ValueError:
|
||||
display_path = filepath
|
||||
|
||||
task_name = task.get("name", "<unnamed>")
|
||||
|
||||
# Check for the allow marker in the task name
|
||||
# (YAML comments are not preserved by safe_load, so we check the
|
||||
# task name for the marker as a workaround)
|
||||
if ALLOW_MARKER in task_name:
|
||||
return violations
|
||||
|
||||
# Check for || true in command/shell values
|
||||
for key in COMMAND_VALUE_KEYS:
|
||||
value = task.get(key)
|
||||
if value is None:
|
||||
continue
|
||||
value_str = str(value)
|
||||
if OR_TRUE_PATTERN.search(value_str) and not _is_legitimate_or_true(value_str, task_name):
|
||||
violations.append(
|
||||
f"{display_path}:{task_num} — task '{task_name}' uses "
|
||||
f"'|| true' in {key} which may mask real failures. "
|
||||
f"If this is a cleanup/idempotency operation, rename the "
|
||||
f"task to include 'remove'/'cleanup'/'prune' or add "
|
||||
f"#{ALLOW_MARKER} to the task."
|
||||
)
|
||||
|
||||
# Check for failed_when: false on critical tasks
|
||||
failed_when = task.get("failed_when")
|
||||
if failed_when is False:
|
||||
name_lower = task_name.lower()
|
||||
# Skip if the task name indicates a legitimate failed_when: false context
|
||||
is_legitimate = any(kw in name_lower for kw in LEGITIMATE_FAILED_WHEN_KEYWORDS)
|
||||
if not is_legitimate:
|
||||
for kw in CRITICAL_TASK_KEYWORDS:
|
||||
if kw in name_lower:
|
||||
violations.append(
|
||||
f"{display_path}:{task_num} — critical task '{task_name}' "
|
||||
f"has failed_when: false, which masks failures on "
|
||||
f"a {kw}-related operation. Remove failed_when: false "
|
||||
f"or add #{ALLOW_MARKER} if masking is intentional."
|
||||
)
|
||||
break
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
|
||||
"""Check a YAML file for dangerous failure-masking patterns."""
|
||||
try:
|
||||
content = filepath.read_text(encoding="utf-8")
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return []
|
||||
|
||||
# Quick check: if no patterns appear, skip
|
||||
if not (
|
||||
OR_TRUE_PATTERN.search(content) or "failed_when: false" in content or REDIRECT_DEVNULL_PATTERN.search(content)
|
||||
):
|
||||
return []
|
||||
|
||||
# Check for allow markers in comments
|
||||
has_allow_marker = ALLOW_MARKER in content
|
||||
|
||||
try:
|
||||
docs = list(yaml.safe_load_all(content))
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
violations: list[str] = []
|
||||
|
||||
for doc in docs:
|
||||
if not doc:
|
||||
continue
|
||||
if isinstance(doc, list):
|
||||
for i, item in enumerate(doc):
|
||||
if isinstance(item, dict):
|
||||
if any(k in item for k in ("tasks", "pre_tasks", "post_tasks", "handlers")):
|
||||
_check_tasks(item, filepath, violations, repo_root)
|
||||
else:
|
||||
violations.extend(_check_task(item, filepath, i + 1, repo_root))
|
||||
block = item.get("block")
|
||||
if isinstance(block, list):
|
||||
for j, bt in enumerate(block):
|
||||
if isinstance(bt, dict):
|
||||
violations.extend(_check_task(bt, filepath, i + j + 1, repo_root))
|
||||
elif isinstance(doc, dict):
|
||||
_check_tasks(doc, filepath, violations, repo_root)
|
||||
|
||||
# Filter out violations if the allow marker is present in the file
|
||||
# (coarse-grained opt-out for files with many legitimate uses)
|
||||
if has_allow_marker:
|
||||
violations = []
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
def _check_tasks(doc: dict, filepath: Path, errors: list[str], repo_root: Path) -> None:
|
||||
"""Check top-level tasks and nested task sections in a playbook doc."""
|
||||
for section_key in ("tasks", "pre_tasks", "post_tasks", "handlers"):
|
||||
section = doc.get(section_key)
|
||||
if isinstance(section, list):
|
||||
for i, task in enumerate(section):
|
||||
if isinstance(task, dict):
|
||||
errors.extend(_check_task(task, filepath, i + 1, repo_root))
|
||||
block = task.get("block")
|
||||
if isinstance(block, list):
|
||||
for j, bt in enumerate(block):
|
||||
if isinstance(bt, dict):
|
||||
errors.extend(_check_task(bt, filepath, i + j + 1, repo_root))
|
||||
|
||||
|
||||
def _find_task_files(base: Path) -> list[Path]:
|
||||
"""Find all YAML task files under a base directory, skipping molecule."""
|
||||
if base.is_file() and base.suffix in (".yml", ".yaml"):
|
||||
return [base]
|
||||
if not base.is_dir():
|
||||
return []
|
||||
files: list[Path] = []
|
||||
for f in sorted(base.rglob("*.yml")) + sorted(base.rglob("*.yaml")):
|
||||
if "molecule" in f.parts:
|
||||
continue
|
||||
files.append(f)
|
||||
return files
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--path",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
|
||||
)
|
||||
@click.option(
|
||||
"--ansible-dir",
|
||||
"ansible_dirs",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
multiple=True,
|
||||
default=None,
|
||||
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
|
||||
)
|
||||
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
|
||||
"""Check Ansible tasks for dangerous failure-masking patterns."""
|
||||
dirs = list(ansible_dirs) if ansible_dirs else DEFAULT_ANSIBLE_DIRS
|
||||
if path:
|
||||
files = _find_task_files(path)
|
||||
else:
|
||||
files: list[Path] = []
|
||||
for d in dirs:
|
||||
files.extend(_find_task_files(d))
|
||||
|
||||
all_violations: list[str] = []
|
||||
for f in files:
|
||||
all_violations.extend(_check_file(f, REPO_ROOT))
|
||||
|
||||
if all_violations:
|
||||
click.echo("[check-ansible-patterns] FAIL: dangerous failure-masking patterns found:")
|
||||
for v in all_violations:
|
||||
click.echo(f" - {v}")
|
||||
click.echo(f"\nTotal: {len(all_violations)} violation(s).")
|
||||
sys.exit(1)
|
||||
else:
|
||||
click.echo("[check-ansible-patterns] OK: no dangerous failure-masking patterns.")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -1,292 +0,0 @@
|
||||
"""Validate Jinja2 expressions in Ansible files by rendering them.
|
||||
|
||||
Extracts ``{{ ... }}`` expressions from Ansible YAML files and renders
|
||||
each one with Ansible's Jinja2 environment using mock variables. Catches
|
||||
errors like reversed filter arguments, undefined filters, and syntax
|
||||
errors before pushing to CI.
|
||||
|
||||
The check is intentionally lightweight — it doesn't need real Ansible
|
||||
facts or variables. It provides common mock values (now(), ansible_*,
|
||||
etc.) and renders each expression in isolation. Expressions that fail
|
||||
with undefined variables that aren't in the mock set are skipped (not
|
||||
all variables can be predicted).
|
||||
|
||||
Usage::
|
||||
|
||||
python -m devx.tools.check_jinja_expr
|
||||
python -m devx.tools.check_jinja_expr --path ansible/playbooks/deploy-observability.yml
|
||||
|
||||
Exit code 0 if all renderable expressions pass, 1 if any fail.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from jinja2 import Environment
|
||||
from jinja2.exceptions import TemplateSyntaxError, UndefinedError
|
||||
|
||||
REPO_ROOT = Path.cwd()
|
||||
|
||||
|
||||
def _default_ansible_dirs() -> list[Path]:
|
||||
"""Return the default directories to scan for Ansible files."""
|
||||
return [
|
||||
REPO_ROOT / "ansible" / "playbooks",
|
||||
REPO_ROOT / "ansible" / "roles",
|
||||
]
|
||||
|
||||
|
||||
# Mock context for rendering Jinja expressions.
|
||||
MOCK_CONTEXT: dict[str, object] = {
|
||||
"now": lambda fmt=None: (
|
||||
"2026-01-01T00:00:00+00:00"
|
||||
if fmt
|
||||
else type(
|
||||
"Now",
|
||||
(),
|
||||
{
|
||||
"timestamp": lambda self: 1735689600.0,
|
||||
"strftime": lambda self, fmt: "2026-01-01T00:00:00+00:00",
|
||||
},
|
||||
)()
|
||||
),
|
||||
"ansible_date_time": {
|
||||
"iso8601": "2026-01-01T00:00:00+00:00",
|
||||
"epoch": "1735689600",
|
||||
},
|
||||
"ansible_facts": {
|
||||
"service_mgr": "systemd",
|
||||
"architecture": "x86_64",
|
||||
"distribution_release": "noble",
|
||||
"virtualization_type": "none",
|
||||
"interfaces": ["eth0", "lo"],
|
||||
"hostname": "test-host",
|
||||
},
|
||||
"ansible_host": "10.0.0.1",
|
||||
"env": "staging",
|
||||
"environment": "staging",
|
||||
"customer_id": "test",
|
||||
"zitadel_domain": "zitadel.test",
|
||||
"_env_name": "staging",
|
||||
"_observability_data_root": "/opt",
|
||||
"skip_zitadel_stack": False,
|
||||
"skip_htpasswd": False,
|
||||
"skip_observability_stack": False,
|
||||
"backup_enabled": True,
|
||||
"app_filter": "",
|
||||
"app_domain": "test.example.com",
|
||||
"oidc_client_id": "test-client-id",
|
||||
"oidc_client_secret": "test-secret", # nosec B105 — mock value for Jinja rendering, not a real secret
|
||||
"s3_backup_bucket": "test-bucket",
|
||||
"s3_endpoint": "https://s3.test",
|
||||
"s3_access_key": "test-key",
|
||||
"s3_secret_key": "test-secret", # nosec B105 — mock value for Jinja rendering, not a real secret
|
||||
}
|
||||
|
||||
# Pattern to find {{ ... }} expressions (non-greedy, single-line).
|
||||
EXPR_PATTERN = re.compile(r"\{\{(.*?)\}\}", re.DOTALL)
|
||||
|
||||
|
||||
def _find_yaml_files(path: Path) -> list[Path]:
|
||||
"""Find Ansible YAML files (tasks, playbooks, handlers) in a path."""
|
||||
if path.is_file():
|
||||
return [path]
|
||||
files: list[Path] = []
|
||||
for pattern in ["**/*.yml", "**/*.yaml"]:
|
||||
files.extend(path.glob(pattern))
|
||||
# Exclude molecule scenarios — they have their own variables.
|
||||
return [f for f in files if "molecule" not in f.parts]
|
||||
|
||||
|
||||
def _extract_expressions(content: str) -> list[str]:
|
||||
"""Extract Jinja expressions from file content.
|
||||
|
||||
Filters out Go template syntax (``{{.Field}}``) used in docker
|
||||
inspect --format strings, and single-character fragments from
|
||||
quoted strings that aren't real Jinja expressions.
|
||||
"""
|
||||
expressions = []
|
||||
for match in EXPR_PATTERN.finditer(content):
|
||||
raw = match.group(1)
|
||||
# Skip multi-line expressions (often have YAML formatting artifacts).
|
||||
if "\n" in raw:
|
||||
continue
|
||||
expr = raw.strip()
|
||||
# Skip empty, control flow, and single-char fragments.
|
||||
if not expr or expr.startswith("%") or len(expr) <= 1:
|
||||
continue
|
||||
# Skip Go template syntax (docker inspect --format).
|
||||
if expr.startswith(".") or "println" in expr:
|
||||
continue
|
||||
# Skip expressions containing Go template dot-access patterns.
|
||||
if ".State." in expr or ".NetworkSettings." in expr:
|
||||
continue
|
||||
# Skip expressions with unbalanced parens/brackets/braces —
|
||||
# the regex captured only part of a larger expression where
|
||||
# }} appears inside a dict literal (e.g. default({'k': {}})).
|
||||
if expr.count("(") != expr.count(")"):
|
||||
continue
|
||||
if expr.count("{") != expr.count("}"):
|
||||
continue
|
||||
if expr.count("[") != expr.count("]"):
|
||||
continue
|
||||
expressions.append(expr)
|
||||
return expressions
|
||||
|
||||
|
||||
def _render_expression(expr: str) -> tuple[bool, str]:
|
||||
"""Try to render a Jinja expression. Returns (success, error_msg)."""
|
||||
try:
|
||||
env = Environment(autoescape=False, keep_trailing_newline=True) # nosec B701 — Ansible Jinja, not web-facing # noqa: S701
|
||||
|
||||
# Add common Ansible filters so expressions can render.
|
||||
# strftime: Ansible's signature is strftime(string_format, second, utc)
|
||||
# where string_format is the piped value. If the piped value looks like
|
||||
# a number (epoch) and second looks like a format string, the args are
|
||||
# reversed — this is the exact bug from OBL-INFRA-508.
|
||||
def _strftime(string_format: str, second: float | None = None, utc: bool = False) -> str:
|
||||
if isinstance(string_format, (int, float)) and isinstance(second, str) and "%" in second:
|
||||
raise ValueError( # noqa: TRY301
|
||||
"Invalid value for epoch value — strftime filter arguments "
|
||||
"are reversed. The format string must be the piped value: "
|
||||
"'%format%' | strftime(epoch), not epoch | strftime('%format%')"
|
||||
)
|
||||
return str(string_format)
|
||||
|
||||
env.filters["strftime"] = _strftime
|
||||
env.filters["b64decode"] = lambda x: x
|
||||
env.filters["b64encode"] = lambda x: x
|
||||
env.filters["regex_replace"] = lambda x, pattern, replacement="": x
|
||||
env.filters["int"] = lambda x, default=0: (
|
||||
int(x) if isinstance(x, (int, float, str)) and str(x).lstrip("-").isdigit() else default
|
||||
)
|
||||
env.filters["bool"] = bool
|
||||
env.filters["basename"] = lambda x: str(x).rsplit("/", 1)[-1]
|
||||
env.filters["dirname"] = lambda x: str(x).rsplit("/", 1)[0] if "/" in str(x) else "."
|
||||
env.filters["combine"] = lambda *args, **kwargs: args[0]
|
||||
env.filters["from_json"] = lambda x: x
|
||||
env.filters["to_json"] = lambda x: x
|
||||
env.filters["ternary"] = lambda x, true_val, false_val=None: true_val if x else false_val
|
||||
env.filters["dict2items"] = lambda x: [
|
||||
{"key": k, "value": v} for k, v in (x.items() if isinstance(x, dict) else [])
|
||||
]
|
||||
env.filters["map"] = lambda x, attribute=None: x
|
||||
env.filters["default"] = lambda x, default_value="", boolean=False: x if x else default_value
|
||||
env.filters["from_yaml"] = lambda x: x
|
||||
env.filters["difference"] = lambda x, y: x
|
||||
env.filters["join"] = lambda x, sep="": sep.join(str(i) for i in (x if isinstance(x, list) else [x]))
|
||||
env.filters["list"] = lambda x: list(x) if isinstance(x, (list, tuple)) else [x]
|
||||
env.filters["length"] = lambda x: len(x) if hasattr(x, "__len__") else 0
|
||||
env.filters["items"] = lambda x: list(x.items()) if isinstance(x, dict) else []
|
||||
env.filters["first"] = lambda x: x[0] if isinstance(x, (list, str)) and x else x
|
||||
env.filters["last"] = lambda x: x[-1] if isinstance(x, (list, str)) and x else x
|
||||
env.filters["upper"] = lambda x: str(x).upper()
|
||||
env.filters["lower"] = lambda x: str(x).lower()
|
||||
env.filters["replace"] = lambda x, old, new: str(x).replace(old, new)
|
||||
env.filters["split"] = lambda x, sep=None: str(x).split(sep) if sep else str(x).split()
|
||||
env.filters["trim"] = lambda x: str(x).strip()
|
||||
env.filters["sort"] = lambda x: sorted(x) if isinstance(x, list) else x
|
||||
env.filters["unique"] = lambda x: list(set(x)) if isinstance(x, list) else x
|
||||
env.filters["count"] = lambda x: len(x) if hasattr(x, "__len__") else 0
|
||||
env.filters["float"] = lambda x, default=0.0: (
|
||||
float(x) if isinstance(x, (int, float, str)) and str(x).replace(".", "").lstrip("-").isdigit() else default
|
||||
)
|
||||
env.filters["string"] = str
|
||||
env.filters["indent"] = lambda x, width=4: str(x)
|
||||
env.filters["to_nice_json"] = str
|
||||
env.filters["to_nice_yaml"] = str
|
||||
env.filters["from_yaml_all"] = lambda x: x
|
||||
env.filters["groupby"] = lambda x: x
|
||||
env.filters["dictsort"] = lambda x: list(x.items()) if isinstance(x, dict) else []
|
||||
env.filters["max"] = lambda x: max(x) if isinstance(x, list) and x else x
|
||||
env.filters["min"] = lambda x: min(x) if isinstance(x, list) and x else x
|
||||
env.filters["reverse"] = lambda x: list(reversed(x)) if isinstance(x, list) else x
|
||||
env.filters["flatten"] = lambda x: x
|
||||
env.filters["product"] = lambda x: x
|
||||
env.filters["zip"] = lambda x: x
|
||||
env.filters["subelements"] = lambda x: x
|
||||
env.filters["json_query"] = lambda x: x
|
||||
env.filters["type_debug"] = lambda x: type(x).__name__
|
||||
env.globals["lookup"] = lambda *args, **kwargs: ""
|
||||
env.globals["query"] = lambda *args, **kwargs: []
|
||||
|
||||
template = env.from_string("{{ " + expr + " }}")
|
||||
result = template.render(**MOCK_CONTEXT)
|
||||
except TemplateSyntaxError as e:
|
||||
return False, f"Syntax error: {e.message}"
|
||||
except UndefinedError as e:
|
||||
# Undefined variable — skip, we can't mock everything.
|
||||
return True, f"Skipped (undefined: {e})"
|
||||
except Exception as e:
|
||||
# Check if it's a filter argument error.
|
||||
error_msg = str(e)
|
||||
if "Invalid value for epoch" in error_msg:
|
||||
return False, f"strftime filter argument error: {error_msg}"
|
||||
# Other errors might be due to missing mock variables — skip.
|
||||
return True, f"Skipped ({type(e).__name__}: {error_msg})"
|
||||
else:
|
||||
return True, result
|
||||
|
||||
|
||||
def _check_file(filepath: Path, repo_root: Path) -> list[str]:
|
||||
"""Check all Jinja expressions in a file. Returns list of violations."""
|
||||
violations = []
|
||||
content = filepath.read_text()
|
||||
expressions = _extract_expressions(content)
|
||||
|
||||
for expr in expressions:
|
||||
success, msg = _render_expression(expr)
|
||||
if not success:
|
||||
try:
|
||||
rel_path = filepath.relative_to(repo_root)
|
||||
except ValueError:
|
||||
rel_path = filepath
|
||||
violations.append(f"{rel_path}: `{{{{ {expr} }}}}` — {msg}")
|
||||
|
||||
return violations
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--path",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
help="Check a specific file or directory (default: ansible/playbooks + ansible/roles).",
|
||||
)
|
||||
@click.option(
|
||||
"--ansible-dir",
|
||||
"ansible_dirs",
|
||||
type=click.Path(exists=True, path_type=Path),
|
||||
multiple=True,
|
||||
default=None,
|
||||
help="Override the default ansible directories (can be repeated). Defaults to ansible/playbooks and ansible/roles.",
|
||||
)
|
||||
def main(path: Path | None, ansible_dirs: tuple[Path, ...]) -> None:
|
||||
"""Validate Jinja2 expressions in Ansible files."""
|
||||
dirs = list(ansible_dirs) if ansible_dirs else _default_ansible_dirs()
|
||||
if path:
|
||||
files = _find_yaml_files(path)
|
||||
else:
|
||||
files: list[Path] = []
|
||||
for d in dirs:
|
||||
files.extend(_find_yaml_files(d))
|
||||
|
||||
all_violations: list[str] = []
|
||||
for f in files:
|
||||
all_violations.extend(_check_file(f, REPO_ROOT))
|
||||
|
||||
if all_violations:
|
||||
click.echo("[check-jinja-expr] FAIL: invalid Jinja expressions found:")
|
||||
for v in all_violations:
|
||||
click.echo(f" - {v}")
|
||||
click.echo("\nFix: test expressions with `ansible localhost -m debug -a 'msg={{ <expr> }}'`")
|
||||
sys.exit(1)
|
||||
else:
|
||||
click.echo("[check-jinja-expr] OK: all Jinja expressions render correctly.")
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
main()
|
||||
@@ -11,18 +11,6 @@ Usage:
|
||||
The module runs ``make test-unit`` with ``PYTEST_ADDOPTS=--durations=0`` so
|
||||
that pytest emits per-test timing lines alongside the summary. Both the
|
||||
total wall-clock time and individual test durations are parsed and validated.
|
||||
|
||||
CI runner scaling
|
||||
-----------------
|
||||
CI runners (Gitea Actions Docker containers) are typically 5-8x slower than
|
||||
local development machines due to shared CPU, fewer cores, and container
|
||||
overhead. When the ``CI`` environment variable is set (standard CI
|
||||
convention), both the total and per-test limits are multiplied by
|
||||
``CI_SCALE_FACTOR`` (default 6) to account for this. This keeps the local
|
||||
budget strict while preventing false failures on slower CI runners.
|
||||
|
||||
The scale factor can be overridden via the ``DEVX_CI_SCALE_FACTOR``
|
||||
environment variable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -39,12 +27,6 @@ DEFAULT_MAX_SECONDS = 10.0
|
||||
DEFAULT_MAX_SINGLE_SECONDS = 0.5
|
||||
TEST_COMMAND = ["make", "test-unit"]
|
||||
|
||||
# CI runners are typically 5-8x slower than local machines (shared CPU,
|
||||
# fewer cores, container overhead). Scale limits up when running on CI
|
||||
# so the gate catches real regressions, not infrastructure slowness.
|
||||
CI_SCALE_FACTOR = float(os.environ.get("DEVX_CI_SCALE_FACTOR", "6"))
|
||||
_IS_CI = bool(os.environ.get("CI") or os.environ.get("GITEA_ACTIONS"))
|
||||
|
||||
# Matches pytest summary line: "234 passed in 0.70s"
|
||||
_TIMING_RE = re.compile(r"(\d+) passed.* in ([0-9.]+)s")
|
||||
|
||||
@@ -56,13 +38,6 @@ _TIMING_RE = re.compile(r"(\d+) passed.* in ([0-9.]+)s")
|
||||
_DURATION_LINE_RE = re.compile(r"^(\d+\.?\d*)s\s+call\s+(.+)$")
|
||||
|
||||
|
||||
def _ci_scale_limit(limit: float) -> float:
|
||||
"""Scale a time limit by the CI factor when running on CI."""
|
||||
if _IS_CI:
|
||||
return limit * CI_SCALE_FACTOR
|
||||
return limit
|
||||
|
||||
|
||||
def run_tests() -> tuple[str, str]:
|
||||
"""Execute the unit-test suite and return (stdout, stderr).
|
||||
|
||||
@@ -148,38 +123,21 @@ def check_per_test_speed(
|
||||
|
||||
def main(max_seconds: float, max_single_seconds: float) -> None:
|
||||
"""Run tests, parse timings, and enforce both budgets."""
|
||||
# Scale limits for CI runners (slower CPU, fewer workers).
|
||||
effective_max = _ci_scale_limit(max_seconds)
|
||||
effective_single = _ci_scale_limit(max_single_seconds)
|
||||
|
||||
if _IS_CI:
|
||||
click.echo(
|
||||
_(
|
||||
"[check-test-speed] CI environment detected — scaling limits by {factor}x "
|
||||
"(total: {orig}s → {eff}s, per-test: {orig_s}s → {eff_s}s)",
|
||||
factor=CI_SCALE_FACTOR,
|
||||
orig=max_seconds,
|
||||
eff=effective_max,
|
||||
orig_s=max_single_seconds,
|
||||
eff_s=effective_single,
|
||||
)
|
||||
)
|
||||
|
||||
stdout, stderr = run_tests()
|
||||
combined = stdout + "\n" + stderr
|
||||
click.echo(combined, err=False)
|
||||
|
||||
duration = parse_duration(combined)
|
||||
check_speed(duration, effective_max)
|
||||
check_speed(duration, max_seconds)
|
||||
|
||||
if effective_single > 0:
|
||||
if max_single_seconds > 0:
|
||||
per_test = parse_per_test_durations(combined)
|
||||
violations = check_per_test_speed(per_test, effective_single)
|
||||
violations = check_per_test_speed(per_test, max_single_seconds)
|
||||
if violations:
|
||||
msg = _(
|
||||
"Per-test speed check FAILED: {count} test(s) exceed {limit}s limit.",
|
||||
count=len(violations),
|
||||
limit=effective_single,
|
||||
limit=max_single_seconds,
|
||||
)
|
||||
click.echo(f"\n{msg}", err=True)
|
||||
for v in violations:
|
||||
@@ -190,8 +148,8 @@ def main(max_seconds: float, max_single_seconds: float) -> None:
|
||||
_(
|
||||
"Unit tests passed in {duration:.2f}s (under {max}s limit, all tests under {single}s per-test limit).",
|
||||
duration=duration,
|
||||
max=effective_max,
|
||||
single=effective_single,
|
||||
max=max_seconds,
|
||||
single=max_single_seconds,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -150,20 +150,15 @@ def sort_versions_by_date(
|
||||
def select_for_deletion(
|
||||
versions: list[dict[str, Any]],
|
||||
keep: int,
|
||||
protect: frozenset[str] = frozenset(),
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Select versions to delete, keeping the most recent ``keep`` versions.
|
||||
|
||||
Versions named ``latest`` and any version listed in ``protect`` are
|
||||
always preserved. Implements REQ-2 (DEVX-165): tags/digests that a
|
||||
release manifest still pins must survive registry cleanup.
|
||||
Versions named ``latest`` are always preserved.
|
||||
"""
|
||||
sorted_versions = sort_versions_by_date(versions)
|
||||
to_delete = sorted_versions[keep:]
|
||||
# Always preserve 'latest' tag
|
||||
to_delete = [v for v in to_delete if v.get("version") != "latest"]
|
||||
# Preserve explicitly protected versions (e.g., pinned by a release manifest)
|
||||
to_delete = [v for v in to_delete if v.get("version") not in protect]
|
||||
return to_delete
|
||||
|
||||
|
||||
@@ -187,12 +182,6 @@ def select_for_deletion(
|
||||
show_default=True,
|
||||
help="Number of recent versions to keep (excluding 'latest').",
|
||||
)
|
||||
@click.option(
|
||||
"--protect",
|
||||
"protect",
|
||||
multiple=True,
|
||||
help="Version/tag to never delete (e.g., pinned by a release manifest). Can be repeated.",
|
||||
)
|
||||
@click.option(
|
||||
"--dry-run",
|
||||
is_flag=True,
|
||||
@@ -208,7 +197,6 @@ def main(
|
||||
owner: str | None,
|
||||
names: tuple[str, ...],
|
||||
keep: int,
|
||||
protect: tuple[str, ...],
|
||||
dry_run: bool,
|
||||
api_url: str | None,
|
||||
) -> None:
|
||||
@@ -250,7 +238,7 @@ def main(
|
||||
_(" {version} (created: {created})", version=v.get("version", "?"), created=v.get("created_at", "?"))
|
||||
)
|
||||
|
||||
to_delete = select_for_deletion(versions, keep, frozenset(protect))
|
||||
to_delete = select_for_deletion(versions, keep)
|
||||
kept_count = len(versions) - len(to_delete)
|
||||
click.echo(_("\nKeeping {kept}, would delete {count}", kept=kept_count, count=len(to_delete)))
|
||||
|
||||
|
||||
@@ -346,18 +346,11 @@ def collect_quality(repo_root: Path) -> dict[str, str | int]:
|
||||
results: list[bool] = []
|
||||
tool_names: list[str] = []
|
||||
|
||||
# Implements: REQ-1 (DEVX-175) — honor the repo's [tool.bandit] config so
|
||||
# the badge matches the project's actual lint gate (skip lists etc.).
|
||||
bandit_cmd = [sys.executable, "-m", "bandit", "-r", "src/"]
|
||||
pyproject = repo_root / "pyproject.toml"
|
||||
if pyproject.exists() and "[tool.bandit]" in pyproject.read_text():
|
||||
bandit_cmd += ["-c", "pyproject.toml"]
|
||||
|
||||
for cmd, name in [
|
||||
([sys.executable, "-m", "ruff", "check", "src/", "tests/"], "ruff check"),
|
||||
([sys.executable, "-m", "ruff", "format", "--check", "src/", "tests/"], "ruff format"),
|
||||
([sys.executable, "-m", "pyright"], "pyright"),
|
||||
(bandit_cmd, "bandit"),
|
||||
([sys.executable, "-m", "bandit", "-r", "src/"], "bandit"),
|
||||
]:
|
||||
rc, _stdout, stderr = run_command(cmd, cwd=repo_root)
|
||||
if rc == 0:
|
||||
|
||||
@@ -65,13 +65,8 @@ def _ensure_target_dir() -> Path:
|
||||
|
||||
|
||||
def _download(url: str, dest: Path) -> None:
|
||||
"""Download a file from ``url`` to ``dest`` with a 60s timeout.
|
||||
|
||||
A User-Agent header is set because some CDNs (e.g. dl.gitea.com)
|
||||
return 403 to requests with Python's default User-Agent.
|
||||
"""
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "devx/install-tools"})
|
||||
with urllib.request.urlopen(req, timeout=60) as resp, open(dest, "wb") as f: # nosec B310
|
||||
"""Download a file from ``url`` to ``dest`` with a 60s timeout."""
|
||||
with urllib.request.urlopen(url, timeout=60) as resp, open(dest, "wb") as f: # nosec B310
|
||||
shutil.copyfileobj(resp, f)
|
||||
|
||||
|
||||
|
||||
@@ -64,11 +64,9 @@ def _install_in_image(
|
||||
link.symlink_to(opt_venv)
|
||||
|
||||
# Build pip install command
|
||||
# --no-deps: the CI image already has all dependencies pre-installed.
|
||||
# We only need to install the project itself in editable mode.
|
||||
spec = f".[{extras}]" if extras else "."
|
||||
pip_bin = str(Path(venv_link) / "bin" / "pip")
|
||||
cmd = [pip_bin, "install", "--no-cache-dir", "--no-deps", "-e", spec]
|
||||
cmd = [pip_bin, "install", "--no-cache-dir", "-e", spec]
|
||||
|
||||
env = os.environ.copy()
|
||||
try:
|
||||
@@ -83,17 +81,6 @@ def _install_in_image(
|
||||
username,
|
||||
token,
|
||||
)
|
||||
# Configure git URL rewrite so git+https dependencies can authenticate
|
||||
subprocess.run( # nosec B603, B607
|
||||
[
|
||||
"git",
|
||||
"config",
|
||||
"--global",
|
||||
f"url.https://{username}:{token}@{gitea_host}/.insteadOf",
|
||||
f"https://{gitea_host}/",
|
||||
],
|
||||
check=True,
|
||||
)
|
||||
|
||||
click.echo(f"[setup-image] Linked {opt_venv}" + (f" with [{extras}]" if extras else "") + ".")
|
||||
subprocess.run(cmd, check=True, env=env) # nosec B603
|
||||
|
||||
+1560
-2024
File diff suppressed because it is too large
Load Diff
@@ -923,12 +923,6 @@ class TestGiteaClientActions:
|
||||
)
|
||||
|
||||
def test_get_repo_variable_returns_value(self) -> None:
|
||||
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
|
||||
client._session.request = MagicMock(return_value=_mock_response({"data": "v0.28.1"}))
|
||||
result = client.get_repo_variable("PRODUCTION_DEPLOY_TAG")
|
||||
assert result == "v0.28.1"
|
||||
|
||||
def test_get_repo_variable_falls_back_to_value(self) -> None:
|
||||
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
|
||||
client._session.request = MagicMock(return_value=_mock_response({"value": "v0.28.1"}))
|
||||
result = client.get_repo_variable("PRODUCTION_DEPLOY_TAG")
|
||||
|
||||
@@ -232,40 +232,6 @@ class TestExtractConventionalMsg:
|
||||
]
|
||||
assert extract_conventional_msg(commits) == "random message"
|
||||
|
||||
def test_skip_ci_subjects_are_ineligible(self) -> None:
|
||||
"""[skip ci] commits must never become the merge title (REQ-1)."""
|
||||
commits = [
|
||||
{"commit": {"message": "ci: retrigger validation"}},
|
||||
{"commit": {"message": "chore: update badge URLs to commit abc123 [skip ci]"}},
|
||||
]
|
||||
assert extract_conventional_msg(commits) == "ci: retrigger validation"
|
||||
|
||||
def test_skip_ci_variants_excluded(self) -> None:
|
||||
"""All skip-token spellings are ineligible."""
|
||||
for token in ("[skip ci]", "[ci skip]", "[skip actions]", "[actions skip]", "[SKIP CI]"):
|
||||
commits = [
|
||||
{"commit": {"message": "feat: real change"}},
|
||||
{"commit": {"message": f"chore: noise {token}"}},
|
||||
]
|
||||
assert extract_conventional_msg(commits) == "feat: real change"
|
||||
|
||||
def test_tie_prefers_newest_commit(self) -> None:
|
||||
"""Equal-priority ties resolve to the newest commit (REQ-2); the
|
||||
API returns commits newest-first."""
|
||||
commits = [
|
||||
{"commit": {"message": "ci: add workflow_dispatch trigger"}},
|
||||
{"commit": {"message": "ci: retrigger validation"}},
|
||||
]
|
||||
assert extract_conventional_msg(commits) == "ci: add workflow_dispatch trigger"
|
||||
|
||||
def test_all_skip_ci_fallback_strips_token(self) -> None:
|
||||
"""When every commit carries [skip ci], the token is stripped so
|
||||
the merge still triggers post-merge."""
|
||||
commits = [
|
||||
{"commit": {"message": "chore: noise [skip ci]"}},
|
||||
]
|
||||
assert extract_conventional_msg(commits) == "chore: noise"
|
||||
|
||||
|
||||
# -- run_cmd --
|
||||
|
||||
@@ -502,138 +468,3 @@ def test_main_module_block() -> None:
|
||||
exec(compile(source, am.__file__, "exec"), namespace)
|
||||
# Verify main is callable
|
||||
assert callable(namespace["main"])
|
||||
|
||||
|
||||
# -- DEVX-164: Vikunja outage resilience + self-approval diagnostics --
|
||||
|
||||
|
||||
class TestVikunjaLookupRetry:
|
||||
"""REQ-1: transient Vikunja API failures are retried, not fatal."""
|
||||
|
||||
@patch.dict("os.environ", {"VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.time.sleep")
|
||||
@patch("devx.ci.auto_merge.VikunjaClient")
|
||||
def test_retries_transient_api_error_then_succeeds(self, mock_client_cls: MagicMock, mock_sleep: MagicMock) -> None:
|
||||
"""A 404/502 during a Vikunja restart is retried until tasks list."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.list_project_tasks.side_effect = [
|
||||
APIError(404, "Not Found"),
|
||||
APIError(502, "Bad Gateway"),
|
||||
[{"id": 1, "identifier": "DEVX-19", "title": "Add new feature"}],
|
||||
]
|
||||
mock_client_cls.return_value = mock_client
|
||||
validate_pr_title_matches_vikunja("DEVX-19: Add new feature", "DEVX-19")
|
||||
assert mock_client.list_project_tasks.call_count == 3
|
||||
assert mock_sleep.call_count == 2
|
||||
|
||||
@patch.dict("os.environ", {"VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.time.sleep")
|
||||
@patch("devx.ci.auto_merge.VikunjaClient")
|
||||
def test_retry_exhaustion_propagates_error(self, mock_client_cls: MagicMock, _mock_sleep: MagicMock) -> None:
|
||||
"""Persistent outage still fails after the bounded attempt count."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.list_project_tasks.side_effect = APIError(502, "Bad Gateway")
|
||||
mock_client_cls.return_value = mock_client
|
||||
with pytest.raises(APIError, match="Bad Gateway"):
|
||||
validate_pr_title_matches_vikunja("DEVX-19: test", "DEVX-19")
|
||||
assert mock_client.list_project_tasks.call_count == 4
|
||||
|
||||
@patch.dict("os.environ", {"VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.time.sleep")
|
||||
@patch("devx.ci.auto_merge.VikunjaClient")
|
||||
def test_retries_connection_error(self, mock_client_cls: MagicMock, mock_sleep: MagicMock) -> None:
|
||||
"""Connection-level failures during restart are also retried."""
|
||||
import requests as req
|
||||
|
||||
mock_client = MagicMock()
|
||||
mock_client.list_project_tasks.side_effect = [
|
||||
req.ConnectionError("refused"),
|
||||
[{"id": 1, "identifier": "DEVX-19", "title": "Found me"}],
|
||||
]
|
||||
mock_client_cls.return_value = mock_client
|
||||
validate_pr_title_matches_vikunja("DEVX-19: Found me", "DEVX-19")
|
||||
assert mock_sleep.call_count == 1
|
||||
|
||||
@patch.dict("os.environ", {"VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.VikunjaClient")
|
||||
def test_missing_task_still_fails_without_retry_sleep(self, mock_client_cls: MagicMock) -> None:
|
||||
"""A healthy Vikunja that simply lacks the task fails as before."""
|
||||
mock_client = MagicMock()
|
||||
mock_client.list_project_tasks.return_value = []
|
||||
mock_client_cls.return_value = mock_client
|
||||
with pytest.raises(click.ClickException, match="Could not find"):
|
||||
validate_pr_title_matches_vikunja("DEVX-99: test", "DEVX-99")
|
||||
|
||||
|
||||
class TestMergeApprovalDiagnostics:
|
||||
"""REQ-2: merge 405 reports approval state + self-approval remediation."""
|
||||
|
||||
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok", "VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.validate_pr_title_matches_vikunja")
|
||||
@patch("devx.ci.auto_merge.GiteaClient")
|
||||
def test_405_without_approvals_shows_self_approval_hint(
|
||||
self, mock_client_cls: MagicMock, _mock_validate: MagicMock, tmp_path, monkeypatch
|
||||
) -> None: # type: ignore[no-untyped-def]
|
||||
monkeypatch.chdir(tmp_path)
|
||||
mock_client = MagicMock()
|
||||
mock_client.get_pr_commits.return_value = [
|
||||
{"commit": {"message": "fix: resolve timeout"}},
|
||||
]
|
||||
mock_client.merge_pr.side_effect = APIError(405, "Does not have enough approvals")
|
||||
mock_client.get_pr_reviews.return_value = []
|
||||
mock_client_cls.return_value = mock_client
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["DEVX-19-fix-bug", "DEVX-19: Fix timeout", "owner/repo", "7"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "Merge failed" in result.output
|
||||
assert "APPROVED" in result.output
|
||||
assert "non-author account" in result.output
|
||||
|
||||
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok", "VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.validate_pr_title_matches_vikunja")
|
||||
@patch("devx.ci.auto_merge.GiteaClient")
|
||||
def test_405_with_approvals_omits_hint(
|
||||
self, mock_client_cls: MagicMock, _mock_validate: MagicMock, tmp_path, monkeypatch
|
||||
) -> None: # type: ignore[no-untyped-def]
|
||||
monkeypatch.chdir(tmp_path)
|
||||
mock_client = MagicMock()
|
||||
mock_client.get_pr_commits.return_value = [
|
||||
{"commit": {"message": "fix: resolve timeout"}},
|
||||
]
|
||||
mock_client.merge_pr.side_effect = APIError(405, "Does not have enough approvals")
|
||||
mock_client.get_pr_reviews.return_value = [{"state": "APPROVED", "user": {"login": "kireto"}}]
|
||||
mock_client_cls.return_value = mock_client
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["DEVX-19-fix-bug", "DEVX-19: Fix timeout", "owner/repo", "7"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "Merge failed" in result.output
|
||||
assert "non-author account" not in result.output
|
||||
|
||||
@patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok", "VIKUNJA_TOKEN": "tok"}, clear=True)
|
||||
@patch("devx.ci.auto_merge.validate_pr_title_matches_vikunja")
|
||||
@patch("devx.ci.auto_merge.GiteaClient")
|
||||
def test_405_reviews_fetch_failure_still_raises(
|
||||
self, mock_client_cls: MagicMock, _mock_validate: MagicMock, tmp_path, monkeypatch
|
||||
) -> None: # type: ignore[no-untyped-def]
|
||||
"""If the reviews lookup itself fails, the merge error still surfaces."""
|
||||
monkeypatch.chdir(tmp_path)
|
||||
mock_client = MagicMock()
|
||||
mock_client.get_pr_commits.return_value = [
|
||||
{"commit": {"message": "fix: resolve timeout"}},
|
||||
]
|
||||
mock_client.merge_pr.side_effect = APIError(405, "Does not have enough approvals")
|
||||
mock_client.get_pr_reviews.side_effect = APIError(403, "Forbidden")
|
||||
mock_client_cls.return_value = mock_client
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["DEVX-19-fix-bug", "DEVX-19: Fix timeout", "owner/repo", "7"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
assert "Merge failed" in result.output
|
||||
|
||||
@@ -13,9 +13,7 @@ from click.testing import CliRunner
|
||||
import devx.tools.build_image as build_image
|
||||
from devx.tools.build_image import (
|
||||
ImageSpec,
|
||||
PushHTTP500Error,
|
||||
build_full_tag,
|
||||
delete_remote_manifest,
|
||||
load_manifest,
|
||||
push_image,
|
||||
registry_login,
|
||||
@@ -198,7 +196,7 @@ class TestBuildImage:
|
||||
class TestPushImage:
|
||||
def test_success(self) -> None:
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest", "1.0"])
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
mock_result = MagicMock(returncode=0, stderr="", stdout="")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result) as mock_run:
|
||||
assert push_image(spec, "git.example.com") is True
|
||||
assert mock_run.call_count == 2
|
||||
@@ -206,8 +204,8 @@ class TestPushImage:
|
||||
def test_partial_failure(self) -> None:
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest", "1.0"])
|
||||
results = [
|
||||
MagicMock(returncode=0, stdout=""),
|
||||
MagicMock(returncode=1, stdout="push failed"),
|
||||
MagicMock(returncode=0, stderr="", stdout=""),
|
||||
MagicMock(returncode=1, stderr="push failed", stdout=""),
|
||||
]
|
||||
with patch("devx.tools.build_image.subprocess.run", side_effect=results):
|
||||
assert push_image(spec, "git.example.com") is False
|
||||
@@ -218,212 +216,6 @@ class TestPushImage:
|
||||
assert push_image(spec, "git.example.com", dry_run=True) is True
|
||||
mock_run.assert_not_called()
|
||||
|
||||
def test_no_delete_on_success_with_creds(self) -> None:
|
||||
"""Push-first: no delete needed when push succeeds."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
):
|
||||
assert push_image(spec, "git.example.com", username="user", token="tok") is True
|
||||
mock_del.assert_not_called()
|
||||
|
||||
def test_no_delete_without_creds(self) -> None:
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
):
|
||||
assert push_image(spec, "git.example.com") is True
|
||||
mock_del.assert_not_called()
|
||||
|
||||
def test_delete_and_retry_on_already_exists(self) -> None:
|
||||
"""Gitea #31964: push fails with 'already exists', delete + retry."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
results = [
|
||||
MagicMock(returncode=1, stdout="package version already exists"),
|
||||
MagicMock(returncode=0, stdout=""),
|
||||
]
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", side_effect=results),
|
||||
patch("devx.tools.build_image.delete_remote_manifest", return_value=True) as mock_del,
|
||||
):
|
||||
assert push_image(spec, "git.example.com", username="user", token="tok") is True
|
||||
mock_del.assert_called_once_with(
|
||||
"git.example.com",
|
||||
"ci-base",
|
||||
"latest",
|
||||
"user",
|
||||
"tok",
|
||||
dry_run=False,
|
||||
)
|
||||
|
||||
def test_no_delete_on_non_already_exists_failure(self) -> None:
|
||||
"""Push fails for other reasons (non-500) — old manifest preserved."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=1, stdout="denied: requested access to the resource is denied")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
):
|
||||
assert push_image(spec, "git.example.com", username="user", token="tok") is False
|
||||
mock_del.assert_not_called()
|
||||
|
||||
def test_retry_also_fails(self) -> None:
|
||||
"""Gitea #31964 retry also fails — both pushes fail."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
results = [
|
||||
MagicMock(returncode=1, stdout="package version already exists"),
|
||||
MagicMock(returncode=1, stdout="push failed again"),
|
||||
]
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", side_effect=results),
|
||||
patch("devx.tools.build_image.delete_remote_manifest", return_value=True),
|
||||
):
|
||||
assert push_image(spec, "git.example.com", username="user", token="tok") is False
|
||||
|
||||
def test_http_500_retries_then_succeeds(self) -> None:
|
||||
"""HTTP 500 from registry race condition — retry succeeds."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
results = [
|
||||
MagicMock(returncode=1, stdout="received unexpected HTTP status: 500 Internal Server Error"),
|
||||
MagicMock(returncode=0, stdout=""),
|
||||
]
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", side_effect=results),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
patch("time.sleep"),
|
||||
):
|
||||
assert push_image(spec, "git.example.com", username="user", token="tok") is True
|
||||
mock_del.assert_not_called()
|
||||
|
||||
def test_http_500_retries_all_fail(self) -> None:
|
||||
"""HTTP 500 retries exhausted — push fails, no delete attempted."""
|
||||
spec = ImageSpec(name="ci-base", dockerfile="Dockerfile", tags=["latest"])
|
||||
mock_result = MagicMock(returncode=1, stdout="received unexpected HTTP status: 500 Internal Server Error")
|
||||
with (
|
||||
patch("devx.tools.build_image.subprocess.run", return_value=mock_result),
|
||||
patch("devx.tools.build_image.delete_remote_manifest") as mock_del,
|
||||
patch("time.sleep"),
|
||||
):
|
||||
assert push_image(spec, "git.example.com", username="user", token="tok") is False
|
||||
mock_del.assert_not_called()
|
||||
|
||||
def test_run_push_raises_on_500(self) -> None:
|
||||
"""_run_push raises PushHTTP500Error when stdout contains 500."""
|
||||
from devx.tools.build_image import _run_push
|
||||
|
||||
mock_result = MagicMock(returncode=1, stdout="received unexpected HTTP status: 500 Internal Server Error")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result):
|
||||
with pytest.raises(PushHTTP500Error, match="500"):
|
||||
_run_push(["docker", "push", "img:latest"])
|
||||
|
||||
def test_run_push_no_raise_on_non_500(self) -> None:
|
||||
"""_run_push returns result when stdout has no 500."""
|
||||
from devx.tools.build_image import _run_push
|
||||
|
||||
mock_result = MagicMock(returncode=1, stdout="denied: access denied")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result):
|
||||
result = _run_push(["docker", "push", "img:latest"])
|
||||
assert result.returncode == 1
|
||||
|
||||
def test_run_push_no_raise_on_success(self) -> None:
|
||||
"""_run_push returns result on success."""
|
||||
from devx.tools.build_image import _run_push
|
||||
|
||||
mock_result = MagicMock(returncode=0, stdout="")
|
||||
with patch("devx.tools.build_image.subprocess.run", return_value=mock_result):
|
||||
result = _run_push(["docker", "push", "img:latest"])
|
||||
assert result.returncode == 0
|
||||
|
||||
|
||||
class TestDeleteRemoteManifest:
|
||||
def test_dry_run(self) -> None:
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t", dry_run=True) is True
|
||||
|
||||
def test_tag_not_found(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = urllib.error.HTTPError("url", 404, "Not Found", {}, None)
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_delete_success(self) -> None:
|
||||
mock_head_resp = MagicMock()
|
||||
mock_head_resp.__enter__.return_value.headers.get.return_value = "sha256:abc123"
|
||||
mock_del_resp = MagicMock()
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = [mock_head_resp, mock_del_resp]
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_delete_404_treated_as_success(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
mock_head_resp = MagicMock()
|
||||
mock_head_resp.__enter__.return_value.headers.get.return_value = "sha256:abc123"
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = [
|
||||
mock_head_resp,
|
||||
urllib.error.HTTPError("url", 404, "Not Found", {}, None),
|
||||
]
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_head_error_does_not_block(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = urllib.error.HTTPError("url", 500, "Server Error", {}, None)
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_url_error_does_not_block(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = urllib.error.URLError("network down")
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_no_digest_does_not_block(self) -> None:
|
||||
mock_head_resp = MagicMock()
|
||||
mock_head_resp.__enter__.return_value.headers.get.return_value = None
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.return_value = mock_head_resp
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_head_405_passes_through(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = urllib.error.HTTPError("url", 405, "Method Not Allowed", {}, None)
|
||||
# 405 falls through with pass, digest never set, returns True
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
assert mock_urlopen.call_count == 1
|
||||
|
||||
def test_delete_500_does_not_block(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
mock_head_resp = MagicMock()
|
||||
mock_head_resp.__enter__.return_value.headers.get.return_value = "sha256:abc123"
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = [
|
||||
mock_head_resp,
|
||||
urllib.error.HTTPError("url", 500, "Server Error", {}, None),
|
||||
]
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
def test_delete_url_error_does_not_block(self) -> None:
|
||||
import urllib.error
|
||||
|
||||
mock_head_resp = MagicMock()
|
||||
mock_head_resp.__enter__.return_value.headers.get.return_value = "sha256:abc123"
|
||||
with patch("devx.tools.build_image.urllib.request.urlopen") as mock_urlopen:
|
||||
mock_urlopen.side_effect = [
|
||||
mock_head_resp,
|
||||
urllib.error.URLError("network down"),
|
||||
]
|
||||
assert delete_remote_manifest("git.example.com", "ci-base", "latest", "u", "t") is True
|
||||
|
||||
|
||||
class TestSortVersions:
|
||||
def test_sort_by_created_at_desc(self) -> None:
|
||||
@@ -485,38 +277,6 @@ class TestSelectForDeletion:
|
||||
to_delete = select_for_deletion(versions, keep=5)
|
||||
assert len(to_delete) == 0
|
||||
|
||||
def test_protect_exempts_version(self) -> None:
|
||||
"""REQ-2: a manifest-pinned version survives cleanup beyond --keep."""
|
||||
versions = [
|
||||
{"version": "0.1.0", "created_at": "2025-01-01"},
|
||||
{"version": "0.2.0", "created_at": "2025-02-01"},
|
||||
{"version": "0.3.0", "created_at": "2025-03-01"},
|
||||
{"version": "0.4.0", "created_at": "2025-04-01"},
|
||||
]
|
||||
to_delete = select_for_deletion(versions, keep=2, protect=frozenset({"0.1.0"}))
|
||||
deleted = {v["version"] for v in to_delete}
|
||||
assert "0.1.0" not in deleted
|
||||
assert deleted == {"0.2.0"}
|
||||
|
||||
def test_protect_multiple_versions(self) -> None:
|
||||
versions = [
|
||||
{"version": "0.1.0", "created_at": "2025-01-01"},
|
||||
{"version": "0.2.0", "created_at": "2025-02-01"},
|
||||
{"version": "0.3.0", "created_at": "2025-03-01"},
|
||||
{"version": "0.4.0", "created_at": "2025-04-01"},
|
||||
]
|
||||
to_delete = select_for_deletion(versions, keep=1, protect=frozenset({"0.1.0", "0.2.0"}))
|
||||
assert {v["version"] for v in to_delete} == {"0.3.0"}
|
||||
|
||||
def test_protect_default_empty_behaves_as_before(self) -> None:
|
||||
versions = [
|
||||
{"version": "0.1.0", "created_at": "2025-01-01"},
|
||||
{"version": "0.2.0", "created_at": "2025-02-01"},
|
||||
{"version": "0.3.0", "created_at": "2025-03-01"},
|
||||
]
|
||||
to_delete = select_for_deletion(versions, keep=2)
|
||||
assert {v["version"] for v in to_delete} == {"0.1.0"}
|
||||
|
||||
|
||||
class TestCleanImagesAPI:
|
||||
"""Tests for the clean_images module's API functions."""
|
||||
@@ -828,20 +588,11 @@ class TestCLIBuildImage:
|
||||
"devx.tools.build_image.subprocess.run",
|
||||
side_effect=[login_result, build_result, push_result],
|
||||
):
|
||||
with patch("devx.tools.build_image.delete_remote_manifest", return_value=True):
|
||||
result = runner.invoke(
|
||||
build_image.main,
|
||||
[
|
||||
"--dockerfile",
|
||||
str(dockerfile),
|
||||
"--name",
|
||||
"ci-base",
|
||||
"--push",
|
||||
"--registry",
|
||||
"git.example.com",
|
||||
],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
result = runner.invoke(
|
||||
build_image.main,
|
||||
["--dockerfile", str(dockerfile), "--name", "ci-base", "--push", "--registry", "git.example.com"],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
class TestCLICleanImages:
|
||||
|
||||
@@ -1,175 +0,0 @@
|
||||
"""Unit tests for devx.ci.check_pr_size."""
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from devx.ci.check_pr_size import (
|
||||
check_size,
|
||||
cli,
|
||||
get_diff_stats,
|
||||
has_refactoring_label,
|
||||
is_excluded,
|
||||
)
|
||||
|
||||
|
||||
class TestIsExcluded:
|
||||
def test_excludes_changelog(self) -> None:
|
||||
assert is_excluded("CHANGELOG.md", ["CHANGELOG.md"])
|
||||
|
||||
def test_excludes_svg_glob(self) -> None:
|
||||
assert is_excluded("docs/badges/coverage.svg", ["*.svg"])
|
||||
|
||||
def test_does_not_exclude_source(self) -> None:
|
||||
assert not is_excluded("src/devx/ci/check_pr_size.py", ["CHANGELOG.md", "*.svg"])
|
||||
|
||||
def test_excludes_readme(self) -> None:
|
||||
assert is_excluded("README.md", ["README.md"])
|
||||
|
||||
def test_excludes_plans_glob(self) -> None:
|
||||
assert is_excluded("docs/plans/sso-bridge-full-extraction.md", ["docs/plans/*"])
|
||||
|
||||
def test_does_not_exclude_specs(self) -> None:
|
||||
assert not is_excluded("docs/specs/DEVX-165.md", ["docs/plans/*"])
|
||||
|
||||
|
||||
class TestCheckSize:
|
||||
def test_under_limits_passes(self) -> None:
|
||||
stats = [("src/main.py", 100, 50), ("tests/test_main.py", 80, 20)]
|
||||
ok, detail = check_size(stats, max_lines=500, max_files=10, excluded_patterns=[])
|
||||
assert ok is True
|
||||
assert "250" in detail # 100+50+80+20
|
||||
|
||||
def test_over_lines_fails(self) -> None:
|
||||
stats = [("src/main.py", 300, 300)]
|
||||
ok, detail = check_size(stats, max_lines=500, max_files=10, excluded_patterns=[])
|
||||
assert ok is False
|
||||
assert "600" in detail
|
||||
|
||||
def test_over_files_fails(self) -> None:
|
||||
stats = [(f"src/file{i}.py", 10, 5) for i in range(15)]
|
||||
ok, detail = check_size(stats, max_lines=500, max_files=10, excluded_patterns=[])
|
||||
assert ok is False
|
||||
assert "15" in detail
|
||||
|
||||
def test_excluded_files_not_counted(self) -> None:
|
||||
stats = [("CHANGELOG.md", 500, 500), ("src/main.py", 10, 5)]
|
||||
ok, detail = check_size(stats, max_lines=500, max_files=10, excluded_patterns=["CHANGELOG.md"])
|
||||
assert ok is True
|
||||
assert "15" in detail # only 10+5
|
||||
|
||||
def test_empty_stats_passes(self) -> None:
|
||||
ok, detail = check_size([], max_lines=500, max_files=10, excluded_patterns=[])
|
||||
assert ok is True
|
||||
|
||||
|
||||
class TestGetDiffStats:
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_parses_numstat_output(self, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(
|
||||
returncode=0,
|
||||
stdout="10\t5\tsrc/main.py\n20\t10\ttests/test_main.py\n",
|
||||
stderr="",
|
||||
)
|
||||
stats = get_diff_stats("origin/master", "HEAD")
|
||||
assert len(stats) == 2
|
||||
assert stats[0] == ("src/main.py", 10, 5)
|
||||
assert stats[1] == ("tests/test_main.py", 20, 10)
|
||||
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_handles_binary_files(self, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(
|
||||
returncode=0,
|
||||
stdout="-\t-\timage.png\n",
|
||||
stderr="",
|
||||
)
|
||||
stats = get_diff_stats("origin/master", "HEAD")
|
||||
assert len(stats) == 1
|
||||
assert stats[0] == ("image.png", 0, 0)
|
||||
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_empty_output(self, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(returncode=0, stdout="", stderr="")
|
||||
stats = get_diff_stats("origin/master", "HEAD")
|
||||
assert stats == []
|
||||
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_git_diff_failure_raises(self, mock_run: MagicMock) -> None:
|
||||
import pytest
|
||||
|
||||
mock_run.return_value = MagicMock(returncode=1, stdout="", stderr="fatal: bad ref")
|
||||
with pytest.raises(Exception, match="git diff|bad ref"):
|
||||
get_diff_stats("origin/master", "HEAD")
|
||||
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_malformed_line_skipped(self, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(
|
||||
returncode=0,
|
||||
stdout="not_a_valid_line\n10\t5\tsrc/main.py\n",
|
||||
stderr="",
|
||||
)
|
||||
stats = get_diff_stats("origin/master", "HEAD")
|
||||
assert len(stats) == 1
|
||||
assert stats[0] == ("src/main.py", 10, 5)
|
||||
|
||||
|
||||
class TestCli:
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_passes_when_small(self, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(
|
||||
returncode=0,
|
||||
stdout="10\t5\tsrc/main.py\n",
|
||||
stderr="",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["--base", "origin/master", "--head", "HEAD"])
|
||||
assert result.exit_code == 0
|
||||
assert "OK" in result.output
|
||||
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
def test_fails_when_too_large(self, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(
|
||||
returncode=0,
|
||||
stdout="300\t300\tsrc/main.py\n",
|
||||
stderr="",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["--base", "origin/master", "--head", "HEAD", "--max-lines", "500"])
|
||||
assert result.exit_code != 0
|
||||
assert "600" in result.output
|
||||
|
||||
@patch("devx.ci.check_pr_size.subprocess.run")
|
||||
@patch("devx.ci.check_pr_size.has_refactoring_label", return_value=True)
|
||||
def test_bypasses_with_refactoring_label(self, mock_label: MagicMock, mock_run: MagicMock) -> None:
|
||||
mock_run.return_value = MagicMock(
|
||||
returncode=0,
|
||||
stdout="300\t300\tsrc/main.py\n",
|
||||
stderr="",
|
||||
)
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--base", "origin/master", "--head", "HEAD", "--repo", "owner/repo", "--pr-number", "42"],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "bypassed" in result.output.lower()
|
||||
|
||||
|
||||
class TestHasRefactoringLabel:
|
||||
@patch("devx.ci.check_pr_size.GiteaClient")
|
||||
@patch("devx.ci.check_pr_size.get_ci_token", return_value="fake-token")
|
||||
def test_returns_true_when_label_present(self, mock_token: MagicMock, mock_client_cls: MagicMock) -> None:
|
||||
mock_client = mock_client_cls.return_value
|
||||
mock_client.get_pr.return_value = {"labels": [{"name": "refactoring"}, {"name": "bug"}]}
|
||||
assert has_refactoring_label("owner/repo", 42) is True
|
||||
|
||||
@patch("devx.ci.check_pr_size.GiteaClient")
|
||||
@patch("devx.ci.check_pr_size.get_ci_token", return_value="fake-token")
|
||||
def test_returns_false_when_label_absent(self, mock_token: MagicMock, mock_client_cls: MagicMock) -> None:
|
||||
mock_client = mock_client_cls.return_value
|
||||
mock_client.get_pr.return_value = {"labels": [{"name": "bug"}]}
|
||||
assert has_refactoring_label("owner/repo", 42) is False
|
||||
|
||||
@patch("devx.ci.check_pr_size.get_ci_token", side_effect=Exception("no token"))
|
||||
def test_returns_false_on_error(self, mock_token: MagicMock) -> None:
|
||||
assert has_refactoring_label("owner/repo", 42) is False
|
||||
@@ -10,7 +10,6 @@ from devx.tools.check_test_speed import (
|
||||
DEFAULT_MAX_SECONDS,
|
||||
DEFAULT_MAX_SINGLE_SECONDS,
|
||||
TEST_COMMAND,
|
||||
_ci_scale_limit,
|
||||
check_per_test_speed,
|
||||
check_speed,
|
||||
cli,
|
||||
@@ -145,26 +144,7 @@ def test_main_module_block() -> None:
|
||||
mock_cli.assert_called_once_with([])
|
||||
|
||||
|
||||
class TestCiScaleLimit:
|
||||
def test_no_scaling_when_not_ci(self) -> None:
|
||||
with patch("devx.tools.check_test_speed._IS_CI", False):
|
||||
assert _ci_scale_limit(10.0) == 10.0
|
||||
assert _ci_scale_limit(0.5) == 0.5
|
||||
|
||||
def test_scales_when_ci(self) -> None:
|
||||
with patch("devx.tools.check_test_speed._IS_CI", True):
|
||||
with patch("devx.tools.check_test_speed.CI_SCALE_FACTOR", 4.0):
|
||||
assert _ci_scale_limit(10.0) == 40.0
|
||||
assert _ci_scale_limit(0.5) == 2.0
|
||||
|
||||
def test_custom_scale_factor(self) -> None:
|
||||
with patch("devx.tools.check_test_speed._IS_CI", True):
|
||||
with patch("devx.tools.check_test_speed.CI_SCALE_FACTOR", 2.5):
|
||||
assert _ci_scale_limit(10.0) == 25.0
|
||||
|
||||
|
||||
class TestMain:
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
@patch("devx.tools.check_test_speed.check_speed")
|
||||
@@ -194,7 +174,6 @@ class TestMain:
|
||||
mock_parse_per.assert_called_once()
|
||||
mock_check_per.assert_called_once_with([], DEFAULT_MAX_SINGLE_SECONDS)
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
def test_slow_total_exits(
|
||||
@@ -210,7 +189,6 @@ class TestMain:
|
||||
assert result.exit_code == 1
|
||||
assert "too slow" in result.output.lower()
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
@patch("devx.tools.check_test_speed.check_speed")
|
||||
@@ -235,7 +213,6 @@ class TestMain:
|
||||
assert "Per-test speed check FAILED" in result.output
|
||||
assert "test_slow" in result.output
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
def test_parse_failure_exits(
|
||||
self,
|
||||
@@ -248,7 +225,6 @@ class TestMain:
|
||||
assert result.exit_code == 1
|
||||
assert "Could not parse" in result.output
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
@patch("devx.tools.check_test_speed.check_speed")
|
||||
@@ -272,7 +248,6 @@ class TestMain:
|
||||
assert result.exit_code == 0
|
||||
mock_check.assert_called_once_with(0.5, 1.5)
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
@patch("devx.tools.check_test_speed.check_speed")
|
||||
@@ -295,7 +270,6 @@ class TestMain:
|
||||
mock_parse_per.assert_not_called()
|
||||
mock_check_per.assert_not_called()
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", False)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
@patch("devx.tools.check_test_speed.check_speed")
|
||||
@@ -318,32 +292,3 @@ class TestMain:
|
||||
result = runner.invoke(cli, ["--max-single-seconds", "1.0"])
|
||||
assert result.exit_code == 0
|
||||
mock_check_per.assert_called_once_with([], 1.0)
|
||||
|
||||
@patch("devx.tools.check_test_speed._IS_CI", True)
|
||||
@patch("devx.tools.check_test_speed.CI_SCALE_FACTOR", 4.0)
|
||||
@patch("devx.tools.check_test_speed.run_tests")
|
||||
@patch("devx.tools.check_test_speed.parse_duration")
|
||||
@patch("devx.tools.check_test_speed.check_speed")
|
||||
@patch("devx.tools.check_test_speed.parse_per_test_durations")
|
||||
@patch("devx.tools.check_test_speed.check_per_test_speed")
|
||||
def test_ci_scales_limits(
|
||||
self,
|
||||
mock_check_per: MagicMock,
|
||||
mock_parse_per: MagicMock,
|
||||
mock_check: MagicMock,
|
||||
mock_parse: MagicMock,
|
||||
mock_run: MagicMock,
|
||||
) -> None:
|
||||
mock_run.return_value = ("out\n", "err\n")
|
||||
mock_parse.return_value = 30.0 # would fail local (10s) but pass CI (40s)
|
||||
mock_parse_per.return_value = []
|
||||
mock_check_per.return_value = []
|
||||
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, [])
|
||||
assert result.exit_code == 0
|
||||
assert "CI environment detected" in result.output
|
||||
assert "scaling limits by 4.0x" in result.output
|
||||
# check_speed called with scaled limit
|
||||
mock_check.assert_called_once_with(30.0, 40.0)
|
||||
mock_check_per.assert_called_once_with([], 2.0)
|
||||
|
||||
@@ -40,6 +40,7 @@ class TestCliGroups:
|
||||
result = runner.invoke(cli, ["molecule", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "distribute" in result.output
|
||||
assert "guard" in result.output
|
||||
assert "all" in result.output
|
||||
|
||||
|
||||
@@ -107,6 +108,13 @@ class TestCiCommands:
|
||||
assert result.exit_code == 0
|
||||
mock_run.assert_called_once_with("devx.ci.post_merge", ["DEVX-1"])
|
||||
|
||||
@patch("devx.cli._run_module")
|
||||
def test_ci_pr_review(self, mock_run: MagicMock) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["ci", "pr-review", "42"])
|
||||
assert result.exit_code == 0
|
||||
mock_run.assert_called_once_with("devx.ci.pr_review", ["42"])
|
||||
|
||||
@patch("devx.cli._run_module")
|
||||
def test_ci_publish(self, mock_run: MagicMock) -> None:
|
||||
runner = CliRunner()
|
||||
@@ -223,6 +231,13 @@ class TestMoleculeCommands:
|
||||
assert result.exit_code == 0
|
||||
mock_run.assert_called_once_with("devx.molecule.discover_runners", [])
|
||||
|
||||
@patch("devx.cli._run_module")
|
||||
def test_molecule_guard(self, mock_run: MagicMock) -> None:
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["molecule", "guard"])
|
||||
assert result.exit_code == 0
|
||||
mock_run.assert_called_once_with("devx.molecule.molecule_ci_guard", [])
|
||||
|
||||
@patch("devx.cli._run_module")
|
||||
def test_molecule_all(self, mock_run: MagicMock) -> None:
|
||||
runner = CliRunner()
|
||||
|
||||
@@ -38,7 +38,6 @@ class TestConfigConstants:
|
||||
def test_conventional_re(self) -> None:
|
||||
assert CONVENTIONAL_RE.match("feat: add feature")
|
||||
assert CONVENTIONAL_RE.match("fix(scope): bug fix")
|
||||
assert CONVENTIONAL_RE.match("deps: bump devx from v0.50.2 to v0.51.0")
|
||||
assert not CONVENTIONAL_RE.match("random message")
|
||||
assert not CONVENTIONAL_RE.match("feat:")
|
||||
assert not CONVENTIONAL_RE.match("BREAKING CHANGE: something")
|
||||
|
||||
@@ -1,688 +0,0 @@
|
||||
"""Unit tests for devx.ci.create_dependency_pr."""
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import click
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from devx.ci.create_dependency_pr import (
|
||||
cli,
|
||||
create_vikunja_task,
|
||||
find_existing_pr,
|
||||
find_pinned_version,
|
||||
update_pinned_version,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _mock_subprocess():
|
||||
"""Mock subprocess so CLI tests never run a real git clone."""
|
||||
with patch("devx.ci.create_dependency_pr.subprocess.run") as m:
|
||||
yield m
|
||||
|
||||
|
||||
class TestFindPinnedVersion:
|
||||
def test_finds_pip_git_pin(self, tmp_path: Path) -> None:
|
||||
content = "grm @ git+https://git.example.com/repo.git@v0.5.1"
|
||||
path = tmp_path / "pyproject.toml"
|
||||
path.write_text(content)
|
||||
version = find_pinned_version("grm", str(path))
|
||||
assert version == "0.5.1"
|
||||
|
||||
def test_finds_pyproject_pin(self, tmp_path: Path) -> None:
|
||||
content = 'grm = "0.5.1"'
|
||||
path = tmp_path / "pyproject.toml"
|
||||
path.write_text(content)
|
||||
version = find_pinned_version("grm", str(path))
|
||||
assert version == "0.5.1"
|
||||
|
||||
def test_finds_ansible_var_pin(self, tmp_path: Path) -> None:
|
||||
content = 'grm_version: "0.5.1"'
|
||||
path = tmp_path / "images.yml"
|
||||
path.write_text(content)
|
||||
version = find_pinned_version("grm", str(path))
|
||||
assert version == "0.5.1"
|
||||
|
||||
def test_finds_image_version_pin(self, tmp_path: Path) -> None:
|
||||
content = 'sso_bridge_image_version: "1.2.3"'
|
||||
path = tmp_path / "images.yml"
|
||||
path.write_text(content)
|
||||
version = find_pinned_version("sso_bridge", str(path))
|
||||
assert version == "1.2.3"
|
||||
|
||||
def test_returns_none_when_not_found(self, tmp_path: Path) -> None:
|
||||
path = tmp_path / "pyproject.toml"
|
||||
path.write_text('other = "1.0.0"')
|
||||
assert find_pinned_version("grm", str(path)) is None
|
||||
|
||||
def test_returns_none_when_file_missing(self, tmp_path: Path) -> None:
|
||||
assert find_pinned_version("grm", str(tmp_path / "nonexistent.toml")) is None
|
||||
|
||||
|
||||
class TestUpdatePinnedVersion:
|
||||
def test_updates_pip_git_pin(self, tmp_path: Path) -> None:
|
||||
content = "grm @ git+https://git.example.com/repo.git@v0.5.1"
|
||||
path = tmp_path / "pyproject.toml"
|
||||
path.write_text(content)
|
||||
changed = update_pinned_version(str(path), "grm", "0.5.1", "0.5.2")
|
||||
assert changed is True
|
||||
assert "0.5.2" in path.read_text()
|
||||
assert "0.5.1" not in path.read_text()
|
||||
|
||||
def test_updates_pyproject_pin(self, tmp_path: Path) -> None:
|
||||
content = 'grm = "0.5.1"'
|
||||
path = tmp_path / "pyproject.toml"
|
||||
path.write_text(content)
|
||||
changed = update_pinned_version(str(path), "grm", "0.5.1", "0.5.2")
|
||||
assert changed is True
|
||||
assert 'grm = "0.5.2"' in path.read_text()
|
||||
|
||||
def test_no_change_when_version_not_found(self, tmp_path: Path) -> None:
|
||||
content = 'other = "1.0.0"'
|
||||
path = tmp_path / "pyproject.toml"
|
||||
path.write_text(content)
|
||||
changed = update_pinned_version(str(path), "grm", "0.5.1", "0.5.2")
|
||||
assert changed is False
|
||||
|
||||
def test_no_change_when_file_missing(self, tmp_path: Path) -> None:
|
||||
changed = update_pinned_version(str(tmp_path / "nonexistent"), "grm", "0.5.1", "0.5.2")
|
||||
assert changed is False
|
||||
|
||||
|
||||
class TestFindExistingPr:
|
||||
@patch("devx.tools.create_pr.GiteaClient")
|
||||
def test_returns_pr_when_found(self, mock_client_cls: MagicMock) -> None:
|
||||
mock_client = mock_client_cls.return_value
|
||||
mock_client.list_prs.return_value = [
|
||||
{"head": {"ref": "deps/grm-0.5.2"}, "number": 42},
|
||||
{"head": {"ref": "other-branch"}, "number": 43},
|
||||
]
|
||||
result = find_existing_pr(mock_client, "deps/grm-0.5.2")
|
||||
assert result is not None
|
||||
assert result["number"] == 42
|
||||
|
||||
@patch("devx.tools.create_pr.GiteaClient")
|
||||
def test_returns_none_when_not_found(self, mock_client_cls: MagicMock) -> None:
|
||||
mock_client = mock_client_cls.return_value
|
||||
mock_client.list_prs.return_value = []
|
||||
result = find_existing_pr(mock_client, "deps/grm-0.5.2")
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestCli:
|
||||
@patch("devx.ci.create_dependency_pr.find_pinned_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_same_version_no_pr(self, mock_token: MagicMock, mock_find: MagicMock) -> None:
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_find.return_value = "0.5.2"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"grm",
|
||||
"--new-version",
|
||||
"0.5.2",
|
||||
"--source-repo",
|
||||
"oblachno/grm",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "no pr needed" in result.output.lower()
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.find_pinned_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_dry_run(self, mock_token: MagicMock, mock_find: MagicMock) -> None:
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_find.return_value = "0.5.1"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"grm",
|
||||
"--new-version",
|
||||
"0.5.2",
|
||||
"--source-repo",
|
||||
"oblachno/grm",
|
||||
"--dry-run",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "DRY RUN" in result.output
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.find_pinned_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_version_not_found_fails(self, mock_token: MagicMock, mock_find: MagicMock) -> None:
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_find.return_value = None
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"nonexistent",
|
||||
"--new-version",
|
||||
"1.0.0",
|
||||
"--source-repo",
|
||||
"oblachno/test",
|
||||
],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
class TestCreateVikunjaTask:
|
||||
def test_returns_none_when_no_token(self) -> None:
|
||||
with patch("devx.ci.create_dependency_pr.get_vikunja_token", side_effect=click.ClickException("no token")):
|
||||
result = create_vikunja_task("Test", "desc")
|
||||
assert result is None
|
||||
|
||||
def test_returns_identifier_on_success(self) -> None:
|
||||
with (
|
||||
patch("devx.ci.create_dependency_pr.get_vikunja_token", return_value="fake-token"),
|
||||
patch("devx.api_clients.VikunjaClient") as mock_client_cls,
|
||||
):
|
||||
mock_client = mock_client_cls.return_value
|
||||
mock_client.create_task.return_value = {"identifier": "OBL-INFRA-999"}
|
||||
result = create_vikunja_task("Test", "desc")
|
||||
assert result == "OBL-INFRA-999"
|
||||
|
||||
def test_explicit_project_id_used(self) -> None:
|
||||
with (
|
||||
patch("devx.ci.create_dependency_pr.get_vikunja_token", return_value="fake-token"),
|
||||
patch("devx.api_clients.VikunjaClient") as mock_client_cls,
|
||||
):
|
||||
mock_client = mock_client_cls.return_value
|
||||
mock_client.create_task.return_value = {"identifier": "OBL-INFRA-1"}
|
||||
create_vikunja_task("Test", "desc", project_id=3)
|
||||
assert mock_client.create_task.call_args.args[0] == 3
|
||||
|
||||
|
||||
def _v2_mocks(digest: str = "sha256:deadbeef") -> list[MagicMock]:
|
||||
"""Token + manifest responses for the registry v2 digest lookup."""
|
||||
tok = MagicMock()
|
||||
tok.raise_for_status = MagicMock()
|
||||
tok.json.return_value = {"token": "bearer-tok"}
|
||||
man = MagicMock()
|
||||
man.raise_for_status = MagicMock()
|
||||
man.headers = {"Docker-Content-Digest": digest}
|
||||
return [tok, man]
|
||||
|
||||
|
||||
class TestResolveContainerDigest:
|
||||
"""REQ-1: existence check via packages API; digest via registry v2."""
|
||||
|
||||
def test_returns_digest_from_registry_v2(self) -> None:
|
||||
from devx.ci.create_dependency_pr import resolve_container_digest
|
||||
|
||||
files = MagicMock()
|
||||
files.raise_for_status = MagicMock()
|
||||
files.json.return_value = [{"name": "manifest.json", "sha256": "blob-not-pullable"}]
|
||||
with patch(
|
||||
"devx.ci.create_dependency_pr.requests.get",
|
||||
side_effect=[files, *_v2_mocks()],
|
||||
) as mock_get:
|
||||
digest = resolve_container_digest(
|
||||
"https://git.example.com/api/v1", "oblachno", "sso-bridge", "0.9.1", "tok"
|
||||
)
|
||||
assert digest == "sha256:deadbeef"
|
||||
man_call = mock_get.call_args_list[2]
|
||||
assert "manifests/0.9.1" in man_call.args[0]
|
||||
assert "oci.image.index" in man_call.kwargs["headers"]["Accept"]
|
||||
tok_call = mock_get.call_args_list[1]
|
||||
assert tok_call.kwargs["params"]["scope"] == "repository:oblachno/sso-bridge:pull"
|
||||
|
||||
def test_raises_when_version_missing(self) -> None:
|
||||
import requests
|
||||
|
||||
from devx.ci.create_dependency_pr import resolve_container_digest
|
||||
|
||||
mock_resp = MagicMock()
|
||||
http_err = requests.HTTPError("404")
|
||||
http_err.response = MagicMock(status_code=404)
|
||||
mock_resp.raise_for_status.side_effect = http_err
|
||||
with patch("devx.ci.create_dependency_pr.requests.get", return_value=mock_resp):
|
||||
with pytest.raises(click.ClickException, match="unpublished artifact"):
|
||||
resolve_container_digest("https://git.example.com/api/v1", "oblachno", "sso-bridge", "9.9.9", "tok")
|
||||
|
||||
def test_raises_when_v2_digest_missing(self) -> None:
|
||||
from devx.ci.create_dependency_pr import resolve_container_digest
|
||||
|
||||
files = MagicMock()
|
||||
files.raise_for_status = MagicMock()
|
||||
files.json.return_value = []
|
||||
tok, man = _v2_mocks(digest="")
|
||||
with patch(
|
||||
"devx.ci.create_dependency_pr.requests.get",
|
||||
side_effect=[files, tok, man],
|
||||
):
|
||||
with pytest.raises(click.ClickException, match="no Docker-Content-Digest"):
|
||||
resolve_container_digest("https://git.example.com/api/v1", "oblachno", "sso-bridge", "0.9.1", "tok")
|
||||
|
||||
def test_raises_when_v2_manifest_request_fails(self) -> None:
|
||||
import requests
|
||||
|
||||
from devx.ci.create_dependency_pr import resolve_container_digest
|
||||
|
||||
files = MagicMock()
|
||||
files.raise_for_status = MagicMock()
|
||||
files.json.return_value = []
|
||||
tok, _ = _v2_mocks()
|
||||
err = requests.HTTPError("500")
|
||||
err.response = MagicMock(status_code=500)
|
||||
man = MagicMock()
|
||||
man.raise_for_status.side_effect = err
|
||||
with patch(
|
||||
"devx.ci.create_dependency_pr.requests.get",
|
||||
side_effect=[files, tok, man],
|
||||
):
|
||||
with pytest.raises(click.ClickException, match="v2 digest lookup failed"):
|
||||
resolve_container_digest("https://git.example.com/api/v1", "oblachno", "sso-bridge", "0.9.1", "tok")
|
||||
|
||||
def test_raises_on_connection_error(self) -> None:
|
||||
import requests
|
||||
|
||||
from devx.ci.create_dependency_pr import resolve_container_digest
|
||||
|
||||
with patch(
|
||||
"devx.ci.create_dependency_pr.requests.get",
|
||||
side_effect=requests.ConnectionError("refused"),
|
||||
):
|
||||
with pytest.raises(click.ClickException, match="Registry lookup failed"):
|
||||
resolve_container_digest("https://git.example.com/api/v1", "oblachno", "sso-bridge", "0.9.1", "tok")
|
||||
|
||||
|
||||
class TestManifestHelpers:
|
||||
"""REQ-1: manifest read/update helpers."""
|
||||
|
||||
def test_read_version(self, tmp_path: Path) -> None:
|
||||
import json
|
||||
|
||||
from devx.ci.create_dependency_pr import read_manifest_version
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text(json.dumps({"schema_version": 1, "sso_bridge": {"version": "0.9.0"}}))
|
||||
assert read_manifest_version(str(p), "sso_bridge") == "0.9.0"
|
||||
|
||||
def test_read_version_missing_file(self, tmp_path: Path) -> None:
|
||||
from devx.ci.create_dependency_pr import read_manifest_version
|
||||
|
||||
assert read_manifest_version(str(tmp_path / "nope.json"), "sso_bridge") is None
|
||||
|
||||
def test_read_version_bad_json(self, tmp_path: Path) -> None:
|
||||
from devx.ci.create_dependency_pr import read_manifest_version
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text("not json{")
|
||||
assert read_manifest_version(str(p), "sso_bridge") is None
|
||||
|
||||
def test_read_version_missing_section(self, tmp_path: Path) -> None:
|
||||
import json
|
||||
|
||||
from devx.ci.create_dependency_pr import read_manifest_version
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text(json.dumps({"schema_version": 1, "other": "x"}))
|
||||
assert read_manifest_version(str(p), "sso_bridge") is None
|
||||
|
||||
def test_update_manifest_fields(self, tmp_path: Path) -> None:
|
||||
import json
|
||||
|
||||
from devx.ci.create_dependency_pr import update_manifest
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text(json.dumps({"schema_version": 1, "sso_bridge": {"version": "0.9.0"}}))
|
||||
changed = update_manifest(
|
||||
str(p),
|
||||
"sso_bridge",
|
||||
{"version": "0.9.1", "git_ref": "v0.9.1", "image_digest": "sha256:x"},
|
||||
)
|
||||
assert changed is True
|
||||
data = json.loads(p.read_text())
|
||||
assert data["sso_bridge"]["version"] == "0.9.1"
|
||||
assert data["sso_bridge"]["git_ref"] == "v0.9.1"
|
||||
assert data["sso_bridge"]["image_digest"] == "sha256:x"
|
||||
|
||||
def test_update_manifest_no_change(self, tmp_path: Path) -> None:
|
||||
import json
|
||||
|
||||
from devx.ci.create_dependency_pr import update_manifest
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text(json.dumps({"sso_bridge": {"version": "0.9.1"}}))
|
||||
assert update_manifest(str(p), "sso_bridge", {"version": "0.9.1"}) is False
|
||||
|
||||
def test_update_manifest_missing_file(self, tmp_path: Path) -> None:
|
||||
from devx.ci.create_dependency_pr import update_manifest
|
||||
|
||||
with pytest.raises(click.ClickException, match="not found"):
|
||||
update_manifest(str(tmp_path / "nope.json"), "sso_bridge", {"version": "1"})
|
||||
|
||||
def test_update_manifest_bad_json(self, tmp_path: Path) -> None:
|
||||
from devx.ci.create_dependency_pr import update_manifest
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text("broken{")
|
||||
with pytest.raises(click.ClickException, match="not valid JSON"):
|
||||
update_manifest(str(p), "sso_bridge", {"version": "1"})
|
||||
|
||||
def test_update_manifest_missing_section(self, tmp_path: Path) -> None:
|
||||
import json
|
||||
|
||||
from devx.ci.create_dependency_pr import update_manifest
|
||||
|
||||
p = tmp_path / "m.json"
|
||||
p.write_text(json.dumps({"schema_version": 1}))
|
||||
with pytest.raises(click.ClickException, match="no object section"):
|
||||
update_manifest(str(p), "sso_bridge", {"version": "1"})
|
||||
|
||||
|
||||
class TestCliManifestMode:
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_manifest_same_version_no_pr(self, mock_token: MagicMock, mock_read: MagicMock) -> None:
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_read.return_value = "0.9.1"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"sso_bridge",
|
||||
"--new-version",
|
||||
"0.9.1",
|
||||
"--source-repo",
|
||||
"oblachno/sso-bridge",
|
||||
"--manifest",
|
||||
"deploy/sso-bridge-release.json",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "no pr needed" in result.output.lower()
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.resolve_container_digest")
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_verify_container_runs_before_lookup(
|
||||
self, mock_token: MagicMock, mock_read: MagicMock, mock_digest: MagicMock
|
||||
) -> None:
|
||||
"""Verification failure aborts before the version lookup/PR steps."""
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_digest.side_effect = click.ClickException("unpublished artifact")
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"sso_bridge",
|
||||
"--new-version",
|
||||
"0.9.1",
|
||||
"--source-repo",
|
||||
"oblachno/sso-bridge",
|
||||
"--manifest",
|
||||
"deploy/m.json",
|
||||
"--verify-container",
|
||||
"oblachno/sso-bridge",
|
||||
],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
mock_read.assert_not_called()
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.resolve_container_digest")
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_verify_container_resolves_digest(
|
||||
self, mock_token: MagicMock, mock_read: MagicMock, mock_digest: MagicMock
|
||||
) -> None:
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_read.return_value = "0.9.1"
|
||||
mock_digest.return_value = "sha256:abc"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"sso_bridge",
|
||||
"--new-version",
|
||||
"0.9.1",
|
||||
"--source-repo",
|
||||
"oblachno/sso-bridge",
|
||||
"--manifest",
|
||||
"deploy/m.json",
|
||||
"--verify-container",
|
||||
"oblachno/sso-bridge",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
mock_digest.assert_called_once()
|
||||
args = mock_digest.call_args[0]
|
||||
assert args[1:4] == ("oblachno", "sso-bridge", "0.9.1")
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_verify_container_invalid_format(self, mock_token: MagicMock) -> None:
|
||||
mock_token.return_value = "fake-token"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"sso_bridge",
|
||||
"--new-version",
|
||||
"0.9.1",
|
||||
"--source-repo",
|
||||
"oblachno/sso-bridge",
|
||||
"--verify-container",
|
||||
"no-slash",
|
||||
],
|
||||
)
|
||||
assert result.exit_code != 0
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.resolve_container_digest")
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token")
|
||||
def test_container_tag_overrides_new_version(
|
||||
self, mock_token: MagicMock, mock_read: MagicMock, mock_digest: MagicMock
|
||||
) -> None:
|
||||
"""--container-tag selects the image tag when it differs from version."""
|
||||
mock_token.return_value = "fake-token"
|
||||
mock_read.return_value = "0.9.1"
|
||||
mock_digest.return_value = "sha256:abc"
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"sso_bridge",
|
||||
"--new-version",
|
||||
"0.9.1",
|
||||
"--source-repo",
|
||||
"oblachno/sso-bridge",
|
||||
"--manifest",
|
||||
"deploy/m.json",
|
||||
"--verify-container",
|
||||
"oblachno/sso-bridge",
|
||||
"--container-tag",
|
||||
"0.2.4",
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
args = mock_digest.call_args[0]
|
||||
assert args[1:4] == ("oblachno", "sso-bridge", "0.2.4")
|
||||
|
||||
|
||||
class TestBranchCreation:
|
||||
"""Branch creation uses the branches API (Gitea lacks POST /git/refs)."""
|
||||
|
||||
def _invoke(self) -> object:
|
||||
runner = CliRunner()
|
||||
return runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"--package",
|
||||
"sso_bridge",
|
||||
"--new-version",
|
||||
"0.9.1",
|
||||
"--source-repo",
|
||||
"oblachno/sso-bridge",
|
||||
"--manifest",
|
||||
"deploy/m.json",
|
||||
],
|
||||
)
|
||||
|
||||
def _client(self, mock_client_cls: MagicMock) -> MagicMock:
|
||||
client = mock_client_cls.return_value
|
||||
client.create_pr.return_value = {"number": 1}
|
||||
return client
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.create_vikunja_task", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.update_manifest", return_value=True)
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version", return_value="0.9.0")
|
||||
@patch("devx.ci.create_dependency_pr.find_existing_pr", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.GiteaClient")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token", return_value="tok")
|
||||
def test_creates_branch_via_branches_api(
|
||||
self,
|
||||
_token: MagicMock,
|
||||
mock_client_cls: MagicMock,
|
||||
_find: MagicMock,
|
||||
_read: MagicMock,
|
||||
_update: MagicMock,
|
||||
_task: MagicMock,
|
||||
) -> None:
|
||||
client = self._client(mock_client_cls)
|
||||
result = self._invoke()
|
||||
assert result.exit_code == 0
|
||||
assert "Created PR" in result.output
|
||||
post = client._request.call_args
|
||||
assert post.args[:2] == ("POST", "/branches")
|
||||
assert post.kwargs["json"] == {
|
||||
"new_branch_name": "deps/sso_bridge-0.9.1",
|
||||
"old_branch_name": "master",
|
||||
}
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.create_vikunja_task", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.update_manifest", return_value=True)
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version", return_value="0.9.0")
|
||||
@patch("devx.ci.create_dependency_pr.find_existing_pr", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.GiteaClient")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token", return_value="tok")
|
||||
def test_existing_branch_tolerated(
|
||||
self,
|
||||
_token: MagicMock,
|
||||
mock_client_cls: MagicMock,
|
||||
_find: MagicMock,
|
||||
_read: MagicMock,
|
||||
_update: MagicMock,
|
||||
_task: MagicMock,
|
||||
) -> None:
|
||||
from devx.exceptions import APIError
|
||||
|
||||
client = self._client(mock_client_cls)
|
||||
client._request.side_effect = APIError(422, "branch already exists")
|
||||
result = self._invoke()
|
||||
assert result.exit_code == 0
|
||||
assert "already exists" in result.output.lower()
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.create_vikunja_task", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.update_manifest", return_value=True)
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version", return_value="0.9.0")
|
||||
@patch("devx.ci.create_dependency_pr.find_existing_pr", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.GiteaClient")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token", return_value="tok")
|
||||
def test_branch_api_error_fails(
|
||||
self,
|
||||
_token: MagicMock,
|
||||
mock_client_cls: MagicMock,
|
||||
_find: MagicMock,
|
||||
_read: MagicMock,
|
||||
_update: MagicMock,
|
||||
_task: MagicMock,
|
||||
) -> None:
|
||||
from devx.exceptions import APIError
|
||||
|
||||
client = self._client(mock_client_cls)
|
||||
client._request.side_effect = APIError(500, "boom")
|
||||
result = self._invoke()
|
||||
assert result.exit_code != 0
|
||||
assert "Failed to create branch" in result.output
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.create_vikunja_task", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.update_manifest", return_value=True)
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version", return_value="0.9.0")
|
||||
@patch("devx.ci.create_dependency_pr.find_existing_pr", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.GiteaClient")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token", return_value="tok")
|
||||
def test_checks_out_remote_tracking_branch(
|
||||
self,
|
||||
_token: MagicMock,
|
||||
mock_client_cls: MagicMock,
|
||||
_find: MagicMock,
|
||||
_read: MagicMock,
|
||||
_update: MagicMock,
|
||||
_task: MagicMock,
|
||||
) -> None:
|
||||
import subprocess
|
||||
|
||||
self._client(mock_client_cls)
|
||||
result = self._invoke()
|
||||
assert result.exit_code == 0
|
||||
calls = [c.args[0] for c in subprocess.run.call_args_list]
|
||||
assert ["git", "fetch", "origin", "deps/sso_bridge-0.9.1:refs/remotes/origin/deps/sso_bridge-0.9.1"] in calls
|
||||
assert ["git", "checkout", "-B", "deps/sso_bridge-0.9.1", "origin/deps/sso_bridge-0.9.1"] in calls
|
||||
|
||||
@patch("devx.ci.create_dependency_pr.create_vikunja_task", return_value="OBL-INFRA-581")
|
||||
@patch("devx.ci.create_dependency_pr.update_manifest", return_value=True)
|
||||
@patch("devx.ci.create_dependency_pr.read_manifest_version", return_value="0.9.0")
|
||||
@patch("devx.ci.create_dependency_pr.find_existing_pr", return_value=None)
|
||||
@patch("devx.ci.create_dependency_pr.GiteaClient")
|
||||
@patch("devx.ci.create_dependency_pr.get_ci_token", return_value="tok")
|
||||
def test_task_id_in_branch_and_spec_committed(
|
||||
self,
|
||||
_token: MagicMock,
|
||||
mock_client_cls: MagicMock,
|
||||
_find: MagicMock,
|
||||
_read: MagicMock,
|
||||
_update: MagicMock,
|
||||
mock_task: MagicMock,
|
||||
) -> None:
|
||||
import subprocess
|
||||
|
||||
client = self._client(mock_client_cls)
|
||||
result = self._invoke()
|
||||
assert result.exit_code == 0
|
||||
# Branch embeds the target-repo task ID
|
||||
post = client._request.call_args
|
||||
assert post.kwargs["json"]["new_branch_name"] == "deps/OBL-INFRA-581-sso_bridge-0.9.1"
|
||||
# Spec file added alongside the manifest change
|
||||
add = next(c for c in subprocess.run.call_args_list if c.args[0][:2] == ["git", "add"])
|
||||
assert "docs/specs/OBL-INFRA-581.md" in add.args[0]
|
||||
# PR title carries the task ID
|
||||
assert client.create_pr.call_args.kwargs["title"] == "OBL-INFRA-581: Bump sso_bridge to 0.9.1"
|
||||
mock_task.assert_called_once()
|
||||
# Commit supplies an explicit identity (fresh clones have none)
|
||||
commit = next(c for c in subprocess.run.call_args_list if "commit" in c.args[0])
|
||||
assert "user.name=gitea-actions-bot" in commit.args[0]
|
||||
assert "user.email=actions@oblachno.fyi" in commit.args[0]
|
||||
|
||||
def test_retries_404_until_published(self) -> None:
|
||||
import requests
|
||||
|
||||
from devx.ci.create_dependency_pr import resolve_container_digest
|
||||
|
||||
err = requests.HTTPError("404")
|
||||
err.response = MagicMock(status_code=404)
|
||||
fail = MagicMock()
|
||||
fail.raise_for_status.side_effect = err
|
||||
ok = MagicMock()
|
||||
ok.raise_for_status = MagicMock()
|
||||
ok.json.return_value = [{"name": "manifest.json", "sha256": "cafe"}]
|
||||
with (
|
||||
patch("devx.ci.create_dependency_pr.requests.get", side_effect=[fail, ok, *_v2_mocks("sha256:cafe")]),
|
||||
patch("devx.ci.create_dependency_pr.time.sleep"),
|
||||
):
|
||||
digest = resolve_container_digest(
|
||||
"https://git.example.com/api/v1", "oblachno", "sso-bridge", "0.9.1", "tok", timeout_s=60
|
||||
)
|
||||
assert digest == "sha256:cafe"
|
||||
@@ -1,83 +0,0 @@
|
||||
"""Unit tests for devx.ci.fast_molecule."""
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from devx.ci.fast_molecule import (
|
||||
build_molecule_commands,
|
||||
cli,
|
||||
get_molecule_scenarios,
|
||||
)
|
||||
|
||||
|
||||
class TestGetMoleculeScenarios:
|
||||
def test_finds_scenarios(self, tmp_path: Path) -> None:
|
||||
roles_dir = tmp_path / "ansible" / "roles" / "myrole" / "molecule"
|
||||
roles_dir.mkdir(parents=True)
|
||||
(roles_dir / "default").mkdir()
|
||||
(roles_dir / "default" / "molecule.yml").write_text("name: default")
|
||||
(roles_dir / "full").mkdir()
|
||||
(roles_dir / "full" / "molecule.yml").write_text("name: full")
|
||||
(roles_dir / "no_scenario").mkdir() # No molecule.yml
|
||||
|
||||
scenarios = get_molecule_scenarios("myrole", str(tmp_path / "ansible" / "roles"))
|
||||
assert sorted(scenarios) == ["default", "full"]
|
||||
|
||||
def test_returns_empty_when_no_molecule_dir(self, tmp_path: Path) -> None:
|
||||
scenarios = get_molecule_scenarios("nonexistent", str(tmp_path / "ansible" / "roles"))
|
||||
assert scenarios == []
|
||||
|
||||
|
||||
class TestBuildMoleculeCommands:
|
||||
def test_builds_commands_for_roles(self, tmp_path: Path) -> None:
|
||||
roles_dir = tmp_path / "ansible" / "roles"
|
||||
for role in ["role_a", "role_b"]:
|
||||
mol_dir = roles_dir / role / "molecule" / "default"
|
||||
mol_dir.mkdir(parents=True)
|
||||
(mol_dir / "molecule.yml").write_text("name: default")
|
||||
|
||||
commands = build_molecule_commands({"role_a", "role_b"}, str(roles_dir))
|
||||
assert len(commands) == 2
|
||||
# REQ-3: emitted commands are exactly what run_molecule_scenario
|
||||
# executes — full `molecule test -s <scenario>`, no synthetic flags.
|
||||
assert commands == ["molecule test -s default", "molecule test -s default"]
|
||||
assert all("--destroy" not in c and "--platform-name" not in c for c in commands)
|
||||
|
||||
def test_empty_when_no_scenarios(self, tmp_path: Path) -> None:
|
||||
commands = build_molecule_commands({"nonexistent"}, str(tmp_path / "ansible" / "roles"))
|
||||
assert commands == []
|
||||
|
||||
def test_empty_when_no_roles(self) -> None:
|
||||
assert build_molecule_commands(set()) == []
|
||||
|
||||
|
||||
class TestCli:
|
||||
@patch("devx.ci.fast_molecule.get_changed_files")
|
||||
def test_no_changes(self, mock_get: MagicMock) -> None:
|
||||
mock_get.return_value = []
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["--base", "origin/master", "--head", "HEAD"])
|
||||
assert result.exit_code == 0
|
||||
assert "No files changed" in result.output
|
||||
|
||||
@patch("devx.ci.fast_molecule.detect_changed_roles")
|
||||
@patch("devx.ci.fast_molecule.get_changed_files")
|
||||
def test_no_ansible_changes(self, mock_get: MagicMock, mock_detect: MagicMock) -> None:
|
||||
mock_get.return_value = ["src/main.py", "README.md"]
|
||||
mock_detect.return_value = set()
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["--base", "origin/master", "--head", "HEAD"])
|
||||
assert result.exit_code == 0
|
||||
assert "No Ansible roles changed" in result.output
|
||||
|
||||
@patch("devx.ci.fast_molecule.detect_changed_roles")
|
||||
@patch("devx.ci.fast_molecule.get_changed_files")
|
||||
def test_detects_changed_roles(self, mock_get: MagicMock, mock_detect: MagicMock) -> None:
|
||||
mock_get.return_value = ["ansible/roles/sso_bridge/tasks/main.yml"]
|
||||
mock_detect.return_value = {"sso_bridge"}
|
||||
runner = CliRunner()
|
||||
result = runner.invoke(cli, ["--base", "origin/master", "--head", "HEAD"])
|
||||
assert result.exit_code == 0
|
||||
assert "sso_bridge" in result.output
|
||||
@@ -365,23 +365,6 @@ class TestCollectQuality:
|
||||
badge = collect_quality(tmp_path)
|
||||
assert badge["message"] == "A"
|
||||
|
||||
@patch("devx.tools.generate_badges.run_command")
|
||||
def test_bandit_uses_repo_config_when_present(self, mock_run: MagicMock, tmp_path: Path) -> None: # type: ignore[no-untyped-def]
|
||||
(tmp_path / "pyproject.toml").write_text("[tool.bandit]\nskips = ['B501']\n")
|
||||
mock_run.return_value = (0, "", "")
|
||||
collect_quality(tmp_path)
|
||||
bandit_call = [c for c in mock_run.call_args_list if "bandit" in c.args[0]][0]
|
||||
assert "-c" in bandit_call.args[0]
|
||||
assert "pyproject.toml" in bandit_call.args[0]
|
||||
|
||||
@patch("devx.tools.generate_badges.run_command")
|
||||
def test_bandit_plain_invocation_without_repo_config(self, mock_run: MagicMock, tmp_path: Path) -> None: # type: ignore[no-untyped-def]
|
||||
(tmp_path / "pyproject.toml").write_text("[project]\nname = 'x'\n")
|
||||
mock_run.return_value = (0, "", "")
|
||||
collect_quality(tmp_path)
|
||||
bandit_call = [c for c in mock_run.call_args_list if "bandit" in c.args[0]][0]
|
||||
assert "-c" not in bandit_call.args[0]
|
||||
|
||||
|
||||
class TestGenerateBadges:
|
||||
@patch("devx.tools.generate_badges.collect_quality")
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import platform
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
@@ -67,10 +66,6 @@ class TestDownload:
|
||||
with patch("urllib.request.urlopen", return_value=_FakeResponse()) as mock_urlopen:
|
||||
install_tools._download("https://example.com/file", dest)
|
||||
mock_urlopen.assert_called_once()
|
||||
call_args = mock_urlopen.call_args
|
||||
req = call_args.args[0]
|
||||
assert isinstance(req, urllib.request.Request)
|
||||
assert req.get_header("User-agent") == "devx/install-tools"
|
||||
assert dest.read_bytes() == b"data"
|
||||
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user