Compare commits

...
17 Commits
Author SHA1 Message Date
devx-ci-bot 5d7ed62b34 release: v0.27.0 [skip ci] 2026-06-28 16:36:35 +00:00
emil ee80c27631 DEVX-98: feat: add lint_docs tool, fix doc_coverage/check_translations for any repo 2026-06-28 16:35:59 +00:00
gitea-actions-bot 98b1659579 chore: update badge URLs to commit f308b9f8 [skip ci] 2026-06-28 15:02:09 +00:00
devx-ci-bot c12d9abc6d release: v0.26.4 [skip ci] 2026-06-28 15:01:52 +00:00
emil 40a65cb0a6 DEVX-97: fix: wrap all user-facing strings with _() for i18n completeness 2026-06-28 15:01:07 +00:00
gitea-actions-bot 1a89738dd4 chore: update badge URLs to commit 36f474db [skip ci] 2026-06-28 14:56:08 +00:00
devx-ci-bot 9e604ea2c7 release: v0.26.3 [skip ci] 2026-06-28 14:55:48 +00:00
emil 4bb50bed58 DEVX-97: fix: pin all dependencies to exact versions for reproducibility 2026-06-28 14:55:03 +00:00
gitea-actions-bot 5bb9dce530 chore: update badge URLs to commit c04f826e [skip ci] 2026-06-28 14:42:36 +00:00
emil 73662a3bf0 DEVX-95: chore: pin all dependency versions to concrete releases 2026-06-28 14:41:27 +00:00
gitea-actions-bot a1e87b1905 chore: update badge URLs to commit 991d923a [skip ci] 2026-06-28 14:38:20 +00:00
devx-ci-bot d8d0ad04a2 release: v0.26.2 [skip ci] 2026-06-28 14:38:13 +00:00
emil e836c09088 DEVX-96: fix: block admin merge override and auto-approve with review token 2026-06-28 14:37:36 +00:00
gitea-actions-bot 507436b134 chore: update badge URLs to commit 8dbdd563 [skip ci] 2026-06-28 12:15:13 +00:00
devx-ci-bot 32cec2c5ad release: v0.26.1 [skip ci] 2026-06-28 12:15:06 +00:00
emil 55c530eb00 DEVX-93: fix: force pip upgrade in setup-image to install new dependencies 2026-06-28 12:14:31 +00:00
gitea-actions-bot 3928de4507 chore: update badge URLs to commit 7dc6d2ce [skip ci] 2026-06-28 01:59:24 +00:00
59 changed files with 2788 additions and 837 deletions
+21
View File
@@ -38,6 +38,12 @@ jobs:
run: |
. .venv/bin/activate
python3 -m devx.ci.doc_coverage --fail-on-missing
- name: Documentation lint check
env:
PYTHONPATH: src
run: |
. .venv/bin/activate
python3 -m devx.ci.lint_docs --root .
- name: Translation completeness check
env:
PYTHONPATH: src
@@ -160,6 +166,21 @@ jobs:
token: ${{ secrets.CI_GITEA_TOKEN }}
- name: Set up environment
run: make setup-image
- name: Post approval review
env:
CI_GITEA_TOKEN: ${{ secrets.REVIEW_GITEA_TOKEN }}
PR_NUMBER: ${{ github.event.number }}
REPOSITORY: ${{ github.repository }}
PYTHONPATH: src
run: |
. .venv/bin/activate
python3 -m devx.ci.pr_review \
"$PR_NUMBER" \
"$REPOSITORY" \
--event APPROVE \
--checklist-confirmed \
--checklist-categories 1,2,3,4,5,6,7,8,9,10,11,12,13 \
--body "Auto-approved: all CI checks passed (quality, pr-review, release-dry-run)."
- name: Squash merge with task ID
env:
CI_GITEA_TOKEN: ${{ secrets.CI_GITEA_TOKEN }}
+2 -1
View File
@@ -71,7 +71,8 @@ src/devx/
│ ├── distribute_items.py # Distribute generic items (VMs, hosts) across parallel runners (LPT)
│ ├── integration_guard.py # Run pytest with cross-runner fail-fast
│ ├── check_translations.py # Translation completeness check
│ └── doc_coverage.py # Documentation coverage check
│ ├── doc_coverage.py # Documentation coverage check
│ └── lint_docs.py # Documentation linter (structure, links, headings)
├── tools/ # Developer tooling modules (run locally or by CI)
│ ├── setup.py # Environment setup (venv, deps, hooks)
│ ├── install_tools.py # Install actionlint, git-cliff, act_runner, tea, hadolint
+30
View File
@@ -2,6 +2,36 @@
All notable changes to this project will be documented in this file.
## [0.27.0] - 2026-06-28
### Features
- Add lint_docs tool, fix doc_coverage/check_translations for any repo
## [0.26.4] - 2026-06-28
### Bug Fixes
- Wrap all user-facing strings with _() for i18n completeness
## [0.26.3] - 2026-06-28
### Bug Fixes
- Pin all dependencies to exact versions for reproducibility
## [0.26.2] - 2026-06-28
### Bug Fixes
- Block admin merge override and auto-approve with review token
## [0.26.1] - 2026-06-28
### Bug Fixes
- Force pip upgrade in setup-image to install new dependencies
## [0.26.0] - 2026-06-28
### Features
+1 -1
View File
@@ -35,7 +35,7 @@ setup-release: $(VENV)/bin/activate .env
# an older devx.mak that doesn't yet define devx-setup-image. Consumer repos
# (grm, infra) can safely alias to devx-setup-image since they install devx from PyPI.
setup-image:
@if [ -d /opt/venv ]; then ln -sf /opt/venv .venv; . .venv/bin/activate && pip install -e . --no-deps 2>/dev/null; \
@if [ -d /opt/venv ]; then ln -sf /opt/venv .venv; . .venv/bin/activate && pip install --no-cache-dir -e . 2>/dev/null; \
else echo "[setup-image] /opt/venv not found — falling back to setup-ci"; $(MAKE) setup-ci; fi
.env:
+9 -6
View File
@@ -16,12 +16,12 @@ quality badges.
[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/python.svg)](https://www.python.org/downloads/)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/python.svg)](https://www.python.org/downloads/)
## Why devx?
@@ -158,6 +158,9 @@ python -m devx.ci.check_translations --translations path/to/translations.json
# Documentation coverage check
python -m devx.ci.doc_coverage --fail-on-missing
# Documentation lint (structure, links, headings, TODOs)
python -m devx.ci.lint_docs --root .
# Validate a commit message
python -m devx.ci.validate_commit_msg commit-msg.txt --branch master
+7 -6
View File
@@ -12,12 +12,12 @@ project to be reusable across all oblachno-oss repositories.
[![CI](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions/workflows/ci.yml/badge.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![License: GPL-3.0](https://img.shields.io/badge/license-GPL--3.0-blue)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/src/branch/master/LICENSE)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/3a6bff698a366dc91551fbe4845501459afa1b54/python.svg)](https://www.python.org/downloads/)
[![Coverage](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/coverage.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Tests](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/tests.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Docs](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/docs.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/wiki)
[![Code Quality](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/quality.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/actions)
[![Version](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/version.svg)](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/releases)
[![Python](https://git.oblachno.oblachno.fyi/oblachno-oss/devx/raw/commit/f308b9f83c38d67feb735871fcc53c46864f8a04/python.svg)](https://www.python.org/downloads/)
## Overview
@@ -158,6 +158,7 @@ for the full configuration reference, PR workflow, and project conventions.
## Wiki pages
- [Home](Home) — This page
- [Getting Started](Getting-Started) — Installation, configuration, and quick start guide
- [CLI Commands](CLI-Commands) — Full CLI command documentation with examples
- [Architecture](Architecture) — Package structure, module descriptions, design principles
- [CI/CD Workflow](CI-CD-Workflow) — Pipeline documentation, workflows, and CI scripts
+1
View File
@@ -1,5 +1,6 @@
{
"index.md": "Home",
"user/getting-started.md": "Getting-Started",
"user/cli-commands.md": "CLI-Commands",
"tech/architecture.md": "Architecture",
"tech/ci-cd-workflow.md": "CI-CD-Workflow"
+1 -1
View File
@@ -382,7 +382,7 @@ single-role (4-part) and multi-role (5-part) pair encoding.
Runs all molecule scenarios on all supported OS platforms sequentially.
Intended for local development; CI uses the parallel matrix instead.
### `discover_runners.py`
### `molecule/discover_runners.py`
Discovers available Gitea Actions runners for molecule tests. Same logic as
`devx.ci.discover_runners` but intended for molecule-specific workflows.
+1 -1
View File
@@ -166,7 +166,7 @@ When `release` creates a `release: vX.Y.Z` commit, the release commit's
post-merge run still updates badges (the version badge picks up the new
version). Other jobs skip. The tag push triggers `publish.yml`.
### Jobs
### Post-merge jobs
#### `detect-type`
+24 -1
View File
@@ -127,14 +127,37 @@ Click commands from `cli.py` and checks if each has documentation in
```bash
devx ci doc-coverage
devx ci doc-coverage --docs-dir docs/ --fail-on-missing
devx ci doc-coverage --docs-dir docs/ --source-dir src/ --fail-on-missing
```
Options:
- `--docs-dir <dir>` — path to the docs directory (default: `docs/`)
- `--source-dir <dir>` — path to the source directory (default: auto-detect)
- `--fail-on-missing` — exit with non-zero status if any documentation is
missing
### `devx ci lint-docs`
Lint documentation files for structure, broken links, heading hierarchy,
duplicate headings, TODO/FIXME markers, and trailing whitespace.
```bash
devx ci lint-docs
devx ci lint-docs --root . --fix
devx ci lint-docs --no-check-links --no-check-stale
```
Options:
- `--root <dir>` — repository root directory (default: `.`)
- `--docs-dir <dir>` — docs directory (default: `<root>/docs`)
- `--check-links/--no-check-links` — check internal links (default: yes)
- `--check-headings/--no-check-headings` — check heading hierarchy (default: yes)
- `--check-todo/--no-check-todo` — check for TODO/FIXME markers (default: yes)
- `--check-stale/--no-check-stale` — check for stale docs (default: no)
- `--check-trailing/--no-check-trailing` — check trailing whitespace (default: yes)
- `--check-duplicates/--no-check-duplicates` — check duplicate headings (default: yes)
- `--fix` — auto-fix trailing whitespace
### `devx ci integration-guard`
Run pytest with cross-runner failure detection. If any
+161
View File
@@ -0,0 +1,161 @@
# Getting Started with devx
This guide walks you through installing devx, configuring it for your project,
and setting up a complete CI/CD pipeline.
## Prerequisites
- **Python 3.12+**
- **A Gitea instance** with Actions enabled
- **A Gitea API token** with repo, workflow, and organization scopes
- **(Optional) Vikunja API token** for task tracking integration
## Installation
devx is published to the Gitea PyPI registry. Configure pip to use it:
```bash
# Configure Gitea PyPI registry
pip config set global.extra-index-url https://git.oblachno.oblachno.fyi/api/packages/oblachno-oss/pypi/simple
# Install devx
pip install devx
```
Or install from source:
```bash
git clone https://git.oblachno.oblachno.fyi/oblachno-oss/devx.git
cd devx
make setup
```
## Quick Start
### 1. Configure environment variables
Create a `.env` file in your project root:
```bash
CI_GITEA_TOKEN=your_gitea_api_token
VIKUNJA_TOKEN=your_vikunja_api_token # optional
```
### 2. Add devx to your project
Add devx to your `pyproject.toml`:
```toml
[project]
dependencies = [
"devx>=0.26.0",
]
[project.optional-dependencies]
dev = [
"devx[dev]>=0.26.0",
]
```
### 3. Set up the Makefile
devx provides a shared Makefile fragment. Add this to your `Makefile`:
```makefile
include devx.mak
```
Run `devx tools setup` to install all development tools (actionlint, git-cliff,
tea CLI, etc.) and configure pre-commit hooks.
### 4. Create the docs structure
devx expects a `docs/` directory with at minimum:
```
docs/
├── index.md # Documentation home page
├── mapping.json # Wiki page title mappings
├── user/ # User-facing documentation
│ └── cli-commands.md
└── tech/ # Technical documentation
├── architecture.md
└── ci-cd-workflow.md
```
Example `docs/mapping.json`:
```json
{
"index.md": "Home",
"user/cli-commands.md": "CLI-Commands",
"tech/architecture.md": "Architecture",
"tech/ci-cd-workflow.md": "CI-CD-Workflow"
}
```
### 5. Set up CI workflows
Create `.gitea/workflows/ci.yml` and `.gitea/workflows/post-merge.yml` in your
project. See the [CI/CD Workflow guide](../tech/ci-cd-workflow.md) for details.
### 6. Configure release settings
Add a `cliff.toml` for git-cliff-based versioning:
```bash
devx tools generate-cliff-config
```
Add `[tool.devx]` section to `pyproject.toml` for project-specific config:
```toml
[tool.devx]
# Vikunja project ID for task tracking
vikunja_project_id = 6
[tool.devx.classify]
# File patterns that are workflow-only (no release needed)
workflow_only = [
".gitea/**",
"docs/**",
"tests/**",
"AGENTS.md",
"README.md",
"CHANGELOG.md",
]
```
## Available Tools
### CI/CD Automation (`devx.ci.*`)
- `devx.ci.release` — Automated semver versioning and tagging
- `devx.ci.publish` — Package publishing to Gitea PyPI registry
- `devx.ci.auto_merge` — Squash-merge automation with task ID validation
- `devx.ci.pr_review` — Automated PR review with inline comments
- `devx.ci.classify_changes` — User-facing vs workflow-only change detection
- `devx.ci.sync_wiki` — Push docs/ to Gitea wiki
- `devx.ci.doc_coverage` — Documentation coverage checker
- `devx.ci.lint_docs` — Documentation linter (structure, links, headings)
- `devx.ci.check_translations` — i18n translation completeness checker
- `devx.ci.notify_failure` — Create Gitea issues on CI failures
- `devx.ci.distribute_files` — Parallel test file distribution
- `devx.ci.distribute_items` — Parallel item distribution across runners
- `devx.ci.discover_runners` — Dynamic runner discovery via Gitea API
### Development Tools (`devx.tools.*`)
- `devx.tools.setup` — Environment setup (venv, deps, hooks, tools)
- `devx.tools.install_tools` — Install CI/CD tools (actionlint, git-cliff, tea)
- `devx.tools.create_task` — Create Vikunja tasks
- `devx.tools.create_pr` — Create Gitea PRs with task ID in title
- `devx.tools.configure_repo` — Configure branch protection and labels
- `devx.tools.generate_badges` — Generate quality badge SVGs
- `devx.tools.check_test_speed` — Enforce test execution speed limits
## Next Steps
- Read the [CLI Commands reference](cli-commands.md) for all available commands
- Read the [Architecture guide](../tech/architecture.md) to understand internals
- Read the [CI/CD Workflow guide](../tech/ci-cd-workflow.md) for pipeline details
+30 -25
View File
@@ -13,10 +13,13 @@ classifiers = [
"Programming Language :: Python :: 3",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
]
# All dependencies are pinned to exact versions for full reproducibility.
# Update pinned versions in a dedicated PR with verification.
dependencies = [
"requests>=2.34.2",
"python-dotenv>=1.2.2",
"click>=8.4.1",
"requests==2.34.2",
"python-dotenv==1.2.2",
"click==8.4.2",
"tenacity==9.1.4", # retry logic for GiteaClient/VikunjaClient
]
[project.scripts]
@@ -28,44 +31,44 @@ version = {attr = "devx.__version__"}
[project.optional-dependencies]
# Test runners (pytest + coverage + parallel execution)
ci = [
"pytest>=9.1.0",
"pytest-cov>=7.1.0",
"pytest-xdist>=3.8",
"pytest==9.1.1",
"pytest-cov==7.1.0",
"pytest-xdist==3.8.0",
]
# Lint and type-checking tools (quality job, badge generation)
lint = [
"ruff>=0.15.17",
"pyright>=1.1.410",
"bandit>=1.8.2",
"pip-audit>=2.10",
"pre-commit>=4.6.0",
"ruff==0.15.20",
"pyright==1.1.411",
"bandit==1.9.4",
"pip-audit==2.10.1",
"pre-commit==4.6.0",
]
# Release tools (build + publish to PyPI/Gitea registry)
release = [
"build>=1.5.0",
"twine>=6.2.0",
"build==1.5.0",
"twine==6.2.0",
]
# Molecule testing (for projects with Ansible roles)
molecule = [
"molecule>=26.4.0",
"molecule-docker>=2.1.0",
"ansible-lint>=26.4.0",
"ansible-core>=2.15,<2.17",
"molecule==26.4.0",
"molecule-docker==2.1.0",
"ansible-lint==26.4.0",
"ansible-core==2.21.1",
]
# Deploy tools (for infra staging/production deployments)
deploy = [
"ansible-core>=2.15,<2.17",
"boto3>=1.34",
"docker>=7.0",
"jinja2>=3.1",
"pyyaml>=6.0",
"cryptography>=41.0",
"ansible-core==2.21.1",
"boto3==1.43.36",
"docker==7.1.0",
"jinja2==3.1.6",
"pyyaml==6.0.3",
"cryptography==49.0.0",
]
# Full dev environment (local development)
dev = [
"devx[ci,lint,release,molecule]",
"build>=1.3.0",
"twine>=6.2.0",
"build==1.5.0",
"twine==6.2.0",
]
[tool.setuptools.packages.find]
@@ -97,6 +100,8 @@ indent-style = "space"
[tool.pyright]
include = ["src"]
pythonVersion = "3.12"
venvPath = "."
venv = ".venv"
strict = ["src/devx/config.py", "src/devx/exceptions.py", "src/devx/i18n.py", "src/devx/api_clients.py", "src/devx/gitea_cli.py"]
# ---------------------------------------------------------------------------
+1 -1
View File
@@ -1,3 +1,3 @@
"""devx — reusable development and CI/CD tools for oblachno-oss projects."""
__version__ = "0.26.0"
__version__ = "0.27.0"
+72 -93
View File
@@ -4,10 +4,16 @@ from __future__ import annotations
import json
import logging
import time
from typing import Any
import requests
from tenacity import (
before_sleep_log,
retry,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
from devx.config import DEFAULT_TIMEOUT, MAX_RETRIES, RETRY_BACKOFF_BASE, RETRY_STATUS_CODES
from devx.exceptions import APIError
@@ -27,14 +33,69 @@ def _parse_error(e: requests.HTTPError) -> tuple[int, str]:
return status, message
def _is_retryable(e: Exception) -> bool:
"""Check if an exception is a transient error worth retrying."""
if isinstance(e, requests.ConnectionError):
return True
if isinstance(e, requests.HTTPError):
status, _ = _parse_error(e)
return status in RETRY_STATUS_CODES
return isinstance(e, requests.Timeout)
class _TransientHTTPError(requests.HTTPError):
"""HTTP error with a retryable status code (wrapped for tenacity)."""
class _RetryableRequestError(Exception):
"""Connection/timeout error wrapped for tenacity retry."""
def _execute_request(
session: requests.Session,
method: str,
url: str,
**kwargs: Any,
) -> requests.Response:
"""Execute a single HTTP request, wrapping transient errors for tenacity.
Non-retryable HTTP errors (4xx except 429) raise :class:`APIError` directly.
Retryable errors (429, 5xx, connection, timeout) raise exceptions that
tenacity will retry.
"""
try:
response = session.request(method, url, timeout=DEFAULT_TIMEOUT, **kwargs)
response.raise_for_status()
return response
except requests.HTTPError as e:
status, message = _parse_error(e)
if status in RETRY_STATUS_CODES:
# Wrap in _TransientHTTPError so tenacity retries it
raise _TransientHTTPError(message, response=e.response) from e
raise APIError(status, message) from e
except (requests.ConnectionError, requests.Timeout) as e:
raise _RetryableRequestError(str(e)) from e
# Tenacity retry decorator shared by both clients.
# Retries on transient HTTP errors (429, 5xx) and connection/timeout errors.
_retry_decorator = retry(
stop=stop_after_attempt(MAX_RETRIES),
wait=wait_exponential(multiplier=RETRY_BACKOFF_BASE, min=RETRY_BACKOFF_BASE, max=RETRY_BACKOFF_BASE**MAX_RETRIES),
retry=retry_if_exception_type((_TransientHTTPError, _RetryableRequestError)),
before_sleep=before_sleep_log(logger, logging.WARNING),
reraise=True,
)
def _request_with_retry(
session: requests.Session,
url: str,
method: str,
**kwargs: Any,
) -> requests.Response:
"""Execute an HTTP request with tenacity-managed retry logic.
On exhaustion, the last exception is translated to :class:`APIError`.
"""
try:
return _retry_decorator(_execute_request)(session, method, url, **kwargs)
except _TransientHTTPError as e:
response = getattr(e, "response", None)
status = response.status_code if response is not None else 0
raise APIError(status, str(e)) from e
except _RetryableRequestError as e:
raise APIError(0, str(e)) from e
class GiteaClient:
@@ -56,49 +117,7 @@ class GiteaClient:
return f"{self._base_url}/repos/{self._owner}/{self._repo}{path}"
def _request(self, method: str, path: str, **kwargs: Any) -> requests.Response:
url = self._url(path)
last_exc: Exception | None = None
for attempt in range(MAX_RETRIES):
try:
response = self._session.request(method, url, timeout=DEFAULT_TIMEOUT, **kwargs)
response.raise_for_status()
return response
except requests.HTTPError as e:
status, message = _parse_error(e)
if _is_retryable(e) and attempt < MAX_RETRIES - 1:
wait = RETRY_BACKOFF_BASE ** (attempt + 1)
logger.warning(
"Transient HTTP %d on %s %s, retrying in %ds (attempt %d/%d)",
status,
method,
path,
wait,
attempt + 1,
MAX_RETRIES,
)
time.sleep(wait)
last_exc = e
continue
raise APIError(status, message) from e
except (requests.ConnectionError, requests.Timeout) as e:
if attempt < MAX_RETRIES - 1:
wait = RETRY_BACKOFF_BASE ** (attempt + 1)
logger.warning(
"Connection error on %s %s, retrying in %ds (attempt %d/%d)",
method,
path,
wait,
attempt + 1,
MAX_RETRIES,
)
time.sleep(wait)
last_exc = e
continue
raise APIError(0, str(e)) from e
# Should not reach here, but just in case
if last_exc: # pragma: no cover
raise APIError(0, str(last_exc)) from last_exc
raise APIError(0, "Max retries exceeded") # pragma: no cover
return _request_with_retry(self._session, self._url(path), method, **kwargs)
# -- repo settings --
@@ -351,47 +370,7 @@ class VikunjaClient:
def _request(self, method: str, path: str, **kwargs: Any) -> requests.Response:
url = f"{self._base_url}{path}"
last_exc: Exception | None = None
for attempt in range(MAX_RETRIES):
try:
response = self._session.request(method, url, timeout=DEFAULT_TIMEOUT, **kwargs)
response.raise_for_status()
return response
except requests.HTTPError as e:
status, message = _parse_error(e)
if _is_retryable(e) and attempt < MAX_RETRIES - 1:
wait = RETRY_BACKOFF_BASE ** (attempt + 1)
logger.warning(
"Transient HTTP %d on %s %s, retrying in %ds (attempt %d/%d)",
status,
method,
path,
wait,
attempt + 1,
MAX_RETRIES,
)
time.sleep(wait)
last_exc = e
continue
raise APIError(status, message) from e
except (requests.ConnectionError, requests.Timeout) as e:
if attempt < MAX_RETRIES - 1:
wait = RETRY_BACKOFF_BASE ** (attempt + 1)
logger.warning(
"Connection error on %s %s, retrying in %ds (attempt %d/%d)",
method,
path,
wait,
attempt + 1,
MAX_RETRIES,
)
time.sleep(wait)
last_exc = e
continue
raise APIError(0, str(e)) from e
if last_exc: # pragma: no cover
raise APIError(0, str(last_exc)) from last_exc
raise APIError(0, "Max retries exceeded") # pragma: no cover
return _request_with_retry(self._session, url, method, **kwargs)
def list_tasks(self, **params: Any) -> list[dict[str, Any]]:
r = self._request("GET", "/tasks", params=params)
+100
View File
@@ -2,8 +2,14 @@
from __future__ import annotations
import os
import subprocess # nosec B404
import click
from devx.config import TASK_ID_RE
from devx.i18n import _
def get_latest_tag() -> str:
"""Get the latest git tag, or empty string if none exists."""
@@ -16,3 +22,97 @@ def get_latest_tag() -> str:
if result.returncode != 0:
return ""
return result.stdout.strip()
def run_cmd(
args: list[str],
check: bool = True,
capture: bool = True,
) -> subprocess.CompletedProcess[str]:
"""Run a command and return the completed process.
Args:
args: Command and arguments as a list.
check: If True, raise :class:`click.ClickException` on non-zero exit.
capture: If True, capture stdout/stderr. If False, inherit parent's.
"""
result = subprocess.run( # nosec B603
args,
capture_output=capture,
text=True,
check=False,
)
if check and result.returncode != 0:
raise click.ClickException(
_(
"Command failed ({cmd}): {stderr}",
cmd=" ".join(args),
stderr=result.stderr.strip() if result.stderr else result.stdout.strip(),
)
)
return result
def extract_task_id(text: str) -> str:
"""Extract the ``{PREFIX}-N`` task identifier from *text*.
Returns the matched string (e.g. ``DEVX-42``) or an empty string if
no task ID is found.
"""
match = TASK_ID_RE.search(text)
return match.group(0) if match else ""
def write_github_env(key: str, value: str) -> None:
"""Append a key=value line to the ``$GITHUB_ENV`` file.
Multi-line values use the heredoc syntax required by Gitea Actions.
Raises :class:`click.ClickException` if ``GITHUB_ENV`` is not set.
"""
gh_env = os.environ.get("GITHUB_ENV")
if not gh_env:
raise click.ClickException("GITHUB_ENV environment variable is not set")
with open(gh_env, "a", encoding="utf-8") as f: # noqa: PTH123
if "\n" in value:
delimiter = "EOF"
f.write(f"{key}<<{delimiter}\n{value}\n{delimiter}\n")
else:
f.write(f"{key}={value}\n")
def write_github_output(key: str, value: str) -> None:
"""Append a key=value line to the ``$GITHUB_OUTPUT`` file.
Raises :class:`click.ClickException` if ``GITHUB_OUTPUT`` is not set.
"""
gh_output = os.environ.get("GITHUB_OUTPUT")
if not gh_output:
raise click.ClickException("GITHUB_OUTPUT environment variable is not set")
with open(gh_output, "a", encoding="utf-8") as f: # noqa: PTH123
f.write(f"{key}={value}\n")
def lpt_distribute[T](items: list[T], weights: list[int], max_runners: int) -> list[list[T]]:
"""Distribute *items* across *max_runners* using LPT scheduling.
Sorts items by weight (descending), then assigns each to the runner
with the least total weight. This produces a more balanced distribution
than naive round-robin when items have varying costs.
Args:
items: Items to distribute.
weights: Parallel list of integer weights (higher = heavier).
max_runners: Number of runner groups to create.
Returns:
A list of ``max_runners`` lists, each containing the items assigned
to that runner.
"""
groups: list[list[T]] = [[] for _ in range(max_runners)]
loads = [0] * max_runners
indexed = sorted(enumerate(items), key=lambda x: (-weights[x[0]], x[0]))
for orig_idx, item in indexed:
min_runner = min(range(max_runners), key=lambda r: loads[r])
groups[min_runner].append(item)
loads[min_runner] += weights[orig_idx]
return groups
+3 -19
View File
@@ -22,7 +22,6 @@ Usage:
import os
import re
import subprocess # nosec B404
from pathlib import Path
from typing import Any
@@ -30,11 +29,11 @@ import click
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
from devx.api_clients import GiteaClient, VikunjaClient
from devx.ci._shared import extract_task_id as _extract_task_id
from devx.config import (
CONVENTIONAL_RE,
DEFAULT_PER_PAGE,
GITEA_API_URL,
TASK_ID_RE,
TASK_PREFIX,
VIKUNJA_API_URL,
VIKUNJA_PROJECT_ID,
@@ -48,20 +47,6 @@ PR_TITLE_RE = re.compile(rf"^{TASK_PREFIX}-\d+:\s+.+")
load_dotenv()
def run_cmd(args: list[str], check: bool = True) -> subprocess.CompletedProcess[str]:
"""Run a command and return the completed process."""
result = subprocess.run(args, capture_output=True, text=True, check=False) # nosec B603
if check and result.returncode != 0:
raise click.ClickException(
_(
"Command failed ({cmd}): {stderr}",
cmd=" ".join(args),
stderr=result.stderr.strip() or result.stdout.strip(),
)
)
return result
def read_taskid(branch: str) -> str:
"""Read task ID from branch name.
@@ -92,9 +77,8 @@ def read_taskid(branch: str) -> str:
def extract_task_id(branch: str) -> str:
"""Extract DEVX-N task identifier from branch name (legacy fallback)."""
match = TASK_ID_RE.search(branch)
return match.group(0) if match else ""
"""Extract task identifier from branch name (delegates to shared utility)."""
return _extract_task_id(branch)
def validate_pr_title(pr_title: str, task_id: str) -> None:
+2 -1
View File
@@ -53,6 +53,7 @@ from devx.config import (
VIKUNJA_API_URL,
VIKUNJA_PROJECT_ID,
)
from devx.exceptions import APIError
from devx.i18n import _
load_dotenv()
@@ -108,7 +109,7 @@ def get_pr_title_from_gitea(repo: str, pr_number: int) -> str | None:
try:
pr = client.get_pr(pr_number)
return str(pr.get("title", ""))
except Exception:
except APIError:
return None
+33 -9
View File
@@ -31,11 +31,11 @@ from pathlib import Path
import click
REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent
REPO_ROOT = Path.cwd()
SUPPORTED_LANGS = ("en", "bg", "de", "ru", "zh", "pl")
# Default translation set: devx package itself
# Default translation set: look for translations.json in the current repo
DEFAULT_TRANS_FILE = REPO_ROOT / "src" / "devx" / "translations.json"
DEFAULT_SRC_DIR = REPO_ROOT / "src" / "devx"
@@ -169,20 +169,44 @@ def print_result(result: TranslationCheckResult) -> None:
"translations",
multiple=True,
type=click.Path(exists=False, path_type=Path),
help="Path to a translations JSON file to check (can be repeated). Defaults to src/devx/translations.json.",
help="Path to a translations JSON file to check (can be repeated). Auto-detects by default.",
)
def main(translations: tuple[Path, ...]) -> None:
@click.option(
"--source-dir",
default=None,
help="Source directory to scan for _() calls (default: auto-detect).",
)
def main(translations: tuple[Path, ...], source_dir: str | None) -> None:
"""Check translation files for gaps, dead keys, and missing languages."""
results: list[TranslationCheckResult] = []
if not translations:
# Default: check the devx package's own translations
results = [
check_translation_set("devx", DEFAULT_SRC_DIR, DEFAULT_TRANS_FILE),
# Auto-detect translations file in the current repo
root = Path.cwd()
# Try common locations
candidates = [
root / "src" / "devx" / "translations.json",
root / "src" / "gitea_runner_manager" / "translations.json",
]
# Also search for any translations.json in src/
for match in root.glob("src/*/translations.json"):
candidates.append(match)
found = False
for candidate in candidates:
if candidate.exists():
src_dir = Path(source_dir) if source_dir else candidate.parent
results.append(check_translation_set(candidate.parent.name, src_dir, candidate))
found = True
break
if not found:
# No translations file found — this repo doesn't use i18n
click.echo("PASS: No translations file found — skipping (repo does not use i18n).")
return
else:
results = []
for trans_file in translations:
# Infer source directory as the parent of the translations file
src_dir = trans_file.parent
src_dir = Path(source_dir) if source_dir else trans_file.parent
name = trans_file.parent.name
results.append(check_translation_set(name, src_dir, trans_file))
+11 -18
View File
@@ -138,7 +138,7 @@ from typing import Any
import click
from devx.ci._shared import get_latest_tag
from devx.ci._shared import get_latest_tag, write_github_output
from devx.i18n import _
# ---------------------------------------------------------------------------
@@ -584,15 +584,8 @@ def has_user_facing_changes(
# ---------------------------------------------------------------------------
def _write_github_output(key: str, value: str) -> None:
"""Append a key=value line to the $GITHUB_OUTPUT file."""
gh_output = os.environ.get("GITHUB_OUTPUT")
if not gh_output:
raise click.ClickException("GITHUB_OUTPUT environment variable is not set")
with open(gh_output, "a") as f: # noqa: PTH123
f.write(f"{key}={value}\n")
# ---------------------------------------------------------------------------
# Classification logic
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
@@ -633,9 +626,9 @@ def main(base: str | None, head: str, quiet: bool, check: str, github_output: bo
force = True
if force and github_output:
_write_github_output("user-facing-changed", "true")
write_github_output("user-facing-changed", "true")
for tag in available_tags:
_write_github_output(f"{tag}-changed", "true")
write_github_output(f"{tag}-changed", "true")
click.echo("Forced user-facing-changed=true via --force flag.")
return
@@ -643,9 +636,9 @@ def main(base: str | None, head: str, quiet: bool, check: str, github_output: bo
base = get_latest_tag()
if not base:
if github_output:
_write_github_output("user-facing-changed", "true")
write_github_output("user-facing-changed", "true")
for tag in available_tags:
_write_github_output(f"{tag}-changed", "true")
write_github_output(f"{tag}-changed", "true")
click.echo("No tags found — treating all changes as user-facing.")
return
if quiet:
@@ -657,9 +650,9 @@ def main(base: str | None, head: str, quiet: bool, check: str, github_output: bo
files = get_changed_files(base, head)
if not files:
if github_output:
_write_github_output("user-facing-changed", "false")
write_github_output("user-facing-changed", "false")
for tag in available_tags:
_write_github_output(f"{tag}-changed", "false")
write_github_output(f"{tag}-changed", "false")
click.echo(f"No changes between {base} and {head}.")
return
if quiet:
@@ -671,9 +664,9 @@ def main(base: str | None, head: str, quiet: bool, check: str, github_output: bo
result = classifier.classify(files)
if github_output:
_write_github_output("user-facing-changed", "true" if result.has_user_facing else "false")
write_github_output("user-facing-changed", "true" if result.has_user_facing else "false")
for tag in available_tags:
_write_github_output(f"{tag}-changed", "true" if result.has_tag(tag) else "false")
write_github_output(f"{tag}-changed", "true" if result.has_tag(tag) else "false")
click.echo(f"User-facing files changed: {result.has_user_facing}")
for tag in available_tags:
click.echo(f"{tag.capitalize()} files changed: {result.has_tag(tag)}")
+6 -13
View File
@@ -12,12 +12,14 @@ Usage::
from __future__ import annotations
import os
import re
import subprocess # nosec B404
import click
from devx.ci._shared import write_github_output
from devx.i18n import _
RELEASE_RE = re.compile(r"^release: v\d+\.\d+\.\d+")
@@ -39,26 +41,17 @@ def is_release_commit(message: str) -> bool:
return bool(RELEASE_RE.match(message))
def write_github_output(key: str, value: str) -> None:
"""Append a key=value line to the $GITHUB_OUTPUT file."""
gh_output = os.environ.get("GITHUB_OUTPUT")
if not gh_output:
raise click.ClickException("GITHUB_OUTPUT environment variable is not set")
with open(gh_output, "a") as f: # noqa: PTH123
f.write(f"{key}={value}\n")
@click.command()
def main() -> None:
"""Detect if the latest commit is a release commit and set GITHUB_OUTPUT."""
msg = get_commit_message()
click.echo(f"Commit message: {msg}")
click.echo(_("Commit message: {msg}", msg=msg))
is_release = is_release_commit(msg)
write_github_output("is-release", "true" if is_release else "false")
if is_release:
click.echo("Release commit — skipping all post-merge jobs.")
click.echo(_("Release commit — skipping all post-merge jobs."))
else:
click.echo("Regular merge commit — running all post-merge jobs.")
click.echo(_("Regular merge commit — running all post-merge jobs."))
if __name__ == "__main__": # pragma: no cover
+15 -11
View File
@@ -30,6 +30,7 @@ import click
import requests
from devx.config import GITEA_API_URL, REPO_NAME, REPO_OWNER
from devx.i18n import _
DEFAULT_MAX_RUNNERS = 3
@@ -55,9 +56,9 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
data = r.json()
total += data.get("total_count", 0)
else:
click.echo(f"Warning: repo-level runners query returned HTTP {r.status_code}", err=True)
click.echo(_("Warning: repo-level runners query returned HTTP {status}", status=r.status_code), err=True)
except (requests.RequestException, ValueError) as e:
click.echo(f"Warning: repo-level runners query failed: {e}", err=True)
click.echo(_("Warning: repo-level runners query failed: {error}", error=e), err=True)
# 2. Organization-level runners
try:
@@ -70,9 +71,9 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
data = r.json()
total += data.get("total_count", 0)
else:
click.echo(f"Warning: org-level runners query returned HTTP {r.status_code}", err=True)
click.echo(_("Warning: org-level runners query returned HTTP {status}", status=r.status_code), err=True)
except (requests.RequestException, ValueError) as e:
click.echo(f"Warning: org-level runners query failed: {e}", err=True)
click.echo(_("Warning: org-level runners query failed: {error}", error=e), err=True)
# 3. Instance-level runners (requires admin scope)
try:
@@ -85,9 +86,12 @@ def query_runners(api_url: str, token: str, owner: str, repo: str) -> int:
data = r.json()
total += data.get("total_count", 0)
elif r.status_code != 403: # 403 is expected without admin scope
click.echo(f"Warning: instance-level runners query returned HTTP {r.status_code}", err=True)
click.echo(
_("Warning: instance-level runners query returned HTTP {status}", status=r.status_code),
err=True,
)
except (requests.RequestException, ValueError) as e:
click.echo(f"Warning: instance-level runners query failed: {e}", err=True)
click.echo(_("Warning: instance-level runners query failed: {error}", error=e), err=True)
return total
@@ -162,11 +166,11 @@ def main(
gh_output = os.environ.get("GITHUB_OUTPUT")
if not gh_output:
raise click.ClickException("GITHUB_OUTPUT environment variable is not set")
with open(gh_output, "a") as f: # noqa: PTH123
with open(gh_output, "a", encoding="utf-8") as f: # noqa: PTH123
f.write(f"runner-count={count}\n")
f.write(f"runner-indices={json.dumps(indices)}\n")
click.echo(f"Runner count: {count}")
click.echo(f"Runner indices: {indices}")
click.echo(_("Runner count: {count}", count=count))
click.echo(_("Runner indices: {indices}", indices=indices))
return
if output_count:
@@ -178,8 +182,8 @@ def main(
return
# Default: output both as key=value pairs for CI consumption
click.echo(f"count={count}")
click.echo(f"indices={json.dumps(indices)}")
click.echo(_("count={count}", count=count))
click.echo(_("indices={indices}", indices=json.dumps(indices)))
if __name__ == "__main__": # pragma: no cover
+18 -30
View File
@@ -26,6 +26,7 @@ import os
import click
from devx.ci._shared import lpt_distribute, write_github_env
from devx.i18n import _
DEFAULT_MAX_RUNNERS = 3
@@ -54,15 +55,7 @@ def distribute(files: list[str], max_runners: int) -> list[list[str]]:
the runner with the least total weight.
"""
weights = [_file_weight(f) for f in files]
groups: list[list[str]] = [[] for _ in range(max_runners)]
loads = [0] * max_runners
# Sort by weight descending, preserving original order for ties
indexed = sorted(enumerate(files), key=lambda x: (-weights[x[0]], x[0]))
for orig_idx, f in indexed:
min_runner = min(range(max_runners), key=lambda r: loads[r])
groups[min_runner].append(f)
loads[min_runner] += weights[orig_idx]
return groups
return lpt_distribute(files, weights, max_runners)
def files_for_runner(files: list[str], runner_index: int, max_runners: int) -> list[str]:
@@ -75,19 +68,6 @@ def files_for_runner(files: list[str], runner_index: int, max_runners: int) -> l
return groups[runner_index]
def _write_github_env(key: str, value: str) -> None:
gh_env = os.environ.get("GITHUB_ENV")
if not gh_env:
raise click.ClickException("GITHUB_ENV environment variable is not set")
with open(gh_env, "a") as f: # noqa: PTH123
if "\n" in value:
# Multi-line values require the heredoc syntax in $GITHUB_ENV.
delimiter = "EOF"
f.write(f"{key}<<{delimiter}\n{value}\n{delimiter}\n")
else:
f.write(f"{key}={value}\n")
@click.command()
@click.option("--pattern", required=True, help="Glob pattern for files to distribute.")
@click.option(
@@ -122,26 +102,34 @@ def main(pattern: str, runner_index: int | None, max_runners: int, github_env: b
groups = distribute(files, max_runners)
for i, group in enumerate(groups):
labels = " ".join(group) if group else "(none)"
click.echo(f"Runner {i}: {labels}")
click.echo(_("Runner {i}: {labels}", i=i, labels=labels))
return
if skip_if_excess and github_env and runner_index > max_runners:
click.echo(f"Skipping — runner index {runner_index} > max runners {max_runners}")
_write_github_env("ASSIGNED_FILES", "")
_write_github_env("SKIP", "true")
click.echo(
_(
"Skipping — runner index {runner_index} > max runners {max_runners}",
runner_index=runner_index,
max_runners=max_runners,
)
)
write_github_env("ASSIGNED_FILES", "")
write_github_env("SKIP", "true")
return
if runner_index < 1:
raise click.ClickException(f"Runner index {runner_index} is out of range (must be >= 1)")
raise click.ClickException(
_("Runner index {runner_index} is out of range (must be >= 1)", runner_index=runner_index)
)
zero_based = runner_index - 1
assigned = files_for_runner(files, zero_based, max_runners)
encoded = "\n".join(assigned)
if github_env:
_write_github_env("ASSIGNED_FILES", encoded)
_write_github_env("SKIP", "false")
click.echo(f"Assigned {len(assigned)} files to runner {runner_index}")
write_github_env("ASSIGNED_FILES", encoded)
write_github_env("SKIP", "false")
click.echo(_("Assigned {count} files to runner {runner_index}", count=len(assigned), runner_index=runner_index))
return
click.echo(encoded)
+26 -30
View File
@@ -29,11 +29,11 @@ Usage::
from __future__ import annotations
import json
import os
import sys
import click
from devx.ci._shared import lpt_distribute, write_github_env
from devx.i18n import _
DEFAULT_MAX_RUNNERS = 3
@@ -93,14 +93,7 @@ def distribute(items: list[str], weights: list[int], max_runners: int) -> list[l
Items are sorted by weight (descending), then assigned to the runner
with the least total weight.
"""
groups: list[list[str]] = [[] for _ in range(max_runners)]
loads = [0] * max_runners
indexed = sorted(enumerate(items), key=lambda x: (-weights[x[0]], x[0]))
for orig_idx, item in indexed:
min_runner = min(range(max_runners), key=lambda r: loads[r])
groups[min_runner].append(item)
loads[min_runner] += weights[orig_idx]
return groups
return lpt_distribute(items, weights, max_runners)
def items_for_runner(items: list[str], weights: list[int], runner_index: int, max_runners: int) -> list[str]:
@@ -113,18 +106,6 @@ def items_for_runner(items: list[str], weights: list[int], runner_index: int, ma
return groups[runner_index]
def _write_github_env(key: str, value: str) -> None:
gh_env = os.environ.get("GITHUB_ENV")
if not gh_env:
raise click.ClickException("GITHUB_ENV environment variable is not set")
with open(gh_env, "a") as f: # noqa: PTH123
if "\n" in value:
delimiter = "EOF"
f.write(f"{key}<<{delimiter}\n{value}\n{delimiter}\n")
else:
f.write(f"{key}={value}\n")
@click.command()
@click.option(
"--items-file",
@@ -166,7 +147,7 @@ def main(
) -> None:
# Read items from file or stdin
if items_file is not None:
with open(items_file) as f: # noqa: PTH123
with open(items_file, encoding="utf-8") as f: # noqa: PTH123
raw = f.read()
else:
raw = sys.stdin.read()
@@ -181,26 +162,41 @@ def main(
groups = distribute(items, weights, max_runners)
for i, group in enumerate(groups):
labels = " ".join(group) if group else "(none)"
click.echo(f"Runner {i}: {labels}")
click.echo(_("Runner {i}: {labels}", i=i, labels=labels))
return
if skip_if_excess and github_env and runner_index > max_runners:
click.echo(f"Skipping — runner index {runner_index} > max runners {max_runners}")
_write_github_env("ASSIGNED_ITEMS", "")
_write_github_env("SKIP", "true")
click.echo(
_(
"Skipping — runner index {runner_index} > max runners {max_runners}",
runner_index=runner_index,
max_runners=max_runners,
)
)
write_github_env("ASSIGNED_ITEMS", "")
write_github_env("SKIP", "true")
return
if runner_index < 1:
raise click.ClickException(f"Runner index {runner_index} is out of range (must be >= 1)")
raise click.ClickException(
_("Runner index {runner_index} is out of range (must be >= 1)", runner_index=runner_index)
)
zero_based = runner_index - 1
assigned = items_for_runner(items, weights, zero_based, max_runners)
encoded = " ".join(assigned)
if github_env:
_write_github_env("ASSIGNED_ITEMS", encoded)
_write_github_env("SKIP", "false")
click.echo(f"Assigned {len(assigned)} items to runner {runner_index}: {encoded}")
write_github_env("ASSIGNED_ITEMS", encoded)
write_github_env("SKIP", "false")
click.echo(
_(
"Assigned {count} items to runner {runner_index}: {encoded}",
count=len(assigned),
runner_index=runner_index,
encoded=encoded,
)
)
return
click.echo(encoded)
+55 -17
View File
@@ -5,8 +5,12 @@ Parses Click commands from the CLI source code and checks if each command
has corresponding documentation in the wiki/docs. Reports missing
documentation as warnings and exits with non-zero if coverage is below 100%.
By default, checks the current repository's own source and docs directories.
When run from the devx package itself (development mode), it checks devx's
own files. When installed as a package, it checks the consuming repo's files.
Usage:
python3 -m devx.ci.doc_coverage [--docs-dir docs/] [--fail-on-missing]
python3 -m devx.ci.doc_coverage [--docs-dir docs/] [--source-dir src/] [--fail-on-missing]
"""
from __future__ import annotations
@@ -19,11 +23,12 @@ import click
from devx.i18n import _
REPO_ROOT = Path(__file__).resolve().parent.parent.parent.parent
# Default to the current working directory (consuming repo's root)
REPO_ROOT = Path.cwd()
DOCS_DIR = REPO_ROOT / "docs"
CLI_FILE = REPO_ROOT / "src" / "devx" / "cli.py"
# Major modules that should be documented in tech/architecture.md
# These are devx-specific; when checking other repos, use --source-dir
REQUIRED_MODULES = [
"cli.py",
"i18n.py",
@@ -51,11 +56,16 @@ REQUIRED_SCRIPTS = [
]
def extract_cli_commands() -> list[str]:
def extract_cli_commands(source_dir: Path) -> list[str]:
"""Extract command names from the CLI source file."""
if not CLI_FILE.exists():
# Try to find the CLI file in the source directory
cli_file = None
for candidate in source_dir.rglob("cli.py"):
cli_file = candidate
break
if cli_file is None or not cli_file.exists():
return []
content = CLI_FILE.read_text()
content = cli_file.read_text()
commands: list[str] = []
# Find all @<group>.command("name") occurrences in the CLI source
# Matches @cli.command, @ci.command, @tools.command, @molecule.command
@@ -94,15 +104,30 @@ def check_module_documented(module: str, docs_content: str) -> bool:
@click.command()
@click.option("--docs-dir", default=str(DOCS_DIR), help="Path to the docs directory.")
@click.option("--docs-dir", default=None, help="Path to the docs directory (default: ./docs).")
@click.option("--source-dir", default=None, help="Path to the source directory (default: auto-detect from src/).")
@click.option(
"--fail-on-missing",
is_flag=True,
default=False,
help="Exit with non-zero status if any documentation is missing.",
)
def main(docs_dir: str, fail_on_missing: bool) -> None:
docs_path = Path(docs_dir)
def main(docs_dir: str | None, source_dir: str | None, fail_on_missing: bool) -> None:
root = Path.cwd()
docs_path = Path(docs_dir) if docs_dir else root / "docs"
# Auto-detect source directory
if source_dir:
src_path = Path(source_dir)
else:
# Try common source directories
for candidate in [root / "src", root / "scripts"]:
if candidate.exists():
src_path = candidate
break
else:
src_path = root / "src"
cli_commands_file = docs_path / "user" / "cli-commands.md"
architecture_file = docs_path / "tech" / "architecture.md"
ci_cd_file = docs_path / "tech" / "ci-cd-workflow.md"
@@ -112,21 +137,28 @@ def main(docs_dir: str, fail_on_missing: bool) -> None:
# Check CLI commands
click.echo(_("Checking CLI command documentation..."))
commands = extract_cli_commands()
commands = extract_cli_commands(src_path)
total += len(commands)
cli_docs = cli_commands_file.read_text() if cli_commands_file.exists() else ""
for cmd in commands:
if check_command_documented(cmd, cli_docs):
click.echo(_(" OK: devx {cmd}", cmd=cmd))
click.echo(_(" OK: {cmd}", cmd=cmd))
else:
click.echo(_(" MISSING: devx {cmd}", cmd=cmd))
missing.append(f"CLI command: devx {cmd}")
click.echo(_(" MISSING: {cmd}", cmd=cmd))
missing.append(f"CLI command: {cmd}")
# Check modules in architecture.md
# Auto-detect modules from source directory (top-level only, exclude subdirs)
click.echo(_("\nChecking module documentation in architecture.md..."))
total += len(REQUIRED_MODULES)
if src_path.exists():
detected_modules = sorted(
f.name for f in src_path.glob("*.py") if f.name != "__init__.py" and f.name != "cli.py"
)
else:
detected_modules = REQUIRED_MODULES
total += len(detected_modules)
arch_docs = architecture_file.read_text() if architecture_file.exists() else ""
for module in REQUIRED_MODULES:
for module in detected_modules:
if check_module_documented(module, arch_docs):
click.echo(_(" OK: {module}", module=module))
else:
@@ -134,10 +166,16 @@ def main(docs_dir: str, fail_on_missing: bool) -> None:
missing.append(f"Module: {module}")
# Check CI scripts in ci-cd-workflow.md
# Auto-detect CI scripts from ci/ subdirectory
click.echo(_("\nChecking CI script documentation in ci-cd-workflow.md..."))
total += len(REQUIRED_SCRIPTS)
ci_dir = src_path / "ci" if src_path.name != "ci" else src_path
if ci_dir.exists():
detected_scripts = sorted(f.name for f in ci_dir.glob("*.py") if f.name != "__init__.py")
else:
detected_scripts = REQUIRED_SCRIPTS
total += len(detected_scripts)
ci_docs = ci_cd_file.read_text() if ci_cd_file.exists() else ""
for script in REQUIRED_SCRIPTS:
for script in detected_scripts:
if check_module_documented(script, ci_docs):
click.echo(_(" OK: {script}", script=script))
else:
+1 -1
View File
@@ -86,7 +86,7 @@ def cli(pytest_args: tuple[str, ...]) -> None:
cmd = [sys.executable, "-m", "pytest"]
cmd.extend(pytest_args)
click.echo(f"Running: {' '.join(cmd)}")
click.echo(_("Running: {cmd}", cmd=" ".join(cmd)))
process = subprocess.Popen( # nosec B603
cmd,
+427
View File
@@ -0,0 +1,427 @@
#!/usr/bin/env python3
"""Lint documentation files for structure, links, and quality.
Checks performed (all configurable via pyproject.toml ``[tool.devx.docs]``):
- **Required files**: README.md, AGENTS.md, CHANGELOG.md must exist.
- **Docs structure**: ``docs/index.md`` and ``docs/mapping.json`` must exist.
- **Broken internal links**: relative paths and anchors in markdown files
must resolve to actual files and headings.
- **Heading hierarchy**: no skipping heading levels (e.g., ``#`` → ``###``).
- **TODO/FIXME**: flags leftover TODO/FIXME markers in documentation.
- **Stale docs**: files not modified in >180 days (warning only).
- **Trailing whitespace**: lines should not end with whitespace.
- **Blank line before headings**: headings should have a blank line before them.
Usage::
python3 -m devx.ci.lint_docs
python3 -m devx.ci.lint_docs --docs-dir docs/ --root .
python3 -m devx.ci.lint_docs --fix # auto-fix trailing whitespace
"""
from __future__ import annotations
import json
import re
import sys
from datetime import datetime, timedelta
from pathlib import Path
import click
from devx.i18n import _
# Heading slug pattern (GitHub-style)
_HEADING_RE = re.compile(r"^(#{1,6})\s+(.+?)\s*$", re.MULTILINE)
# Markdown link pattern: [text](url)
_LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)]+)\)")
# Trailing whitespace
_TRAILING_WS_RE = re.compile(r"[ \t]+$")
# Heading without blank line before
_HEADING_NO_BLANK_RE = re.compile(r"([^\n])\n(#{1,6}\s)")
# Files that must exist in every project
REQUIRED_FILES = ["README.md", "AGENTS.md", "CHANGELOG.md"]
# Files that must exist in docs/
REQUIRED_DOC_FILES = ["index.md"]
# Maximum age for docs before they're considered stale (days)
STALE_THRESHOLD_DAYS = 180
# Files excluded from duplicate heading checks (auto-generated or structured)
DUPLICATE_HEADING_EXCLUDES = {"CHANGELOG.md"}
# TODO/FIXME pattern — matches "TODO:" or "FIXME:" at start of line/after whitespace
# Does NOT match references to the word "TODO" in rules/documentation
_TODO_RE = re.compile(r"(?m)^\s*(?:>>>?\s*)?(TODO|FIXME|HACK|XXX)\s*:", re.IGNORECASE)
def slugify(text: str) -> str:
"""Convert heading text to a GitHub-style slug."""
slug = text.lower().strip()
slug = re.sub(r"[^\w\s-]", "", slug)
slug = re.sub(r"[\s]+", "-", slug)
return slug
def strip_code_blocks(content: str) -> str:
"""Remove fenced code blocks from markdown content.
Replaces ```...``` blocks with empty lines so heading detection
doesn't pick up # comments inside code blocks.
"""
result: list[str] = []
in_code_block = False
for line in content.splitlines():
if line.strip().startswith("```"):
in_code_block = not in_code_block
result.append("")
continue
if in_code_block:
result.append("")
continue
result.append(line)
return "\n".join(result)
def extract_headings(filepath: Path) -> dict[str, int]:
"""Extract all headings from a markdown file.
Returns a dict mapping slug → heading level.
"""
content = strip_code_blocks(filepath.read_text(encoding="utf-8"))
headings: dict[str, int] = {}
for match in _HEADING_RE.finditer(content):
level = len(match.group(1))
text = match.group(2)
slug = slugify(text)
headings[slug] = level
return headings
def extract_links(filepath: Path) -> list[tuple[int, str, str]]:
"""Extract all markdown links from a file.
Returns a list of (line_number, link_text, url) tuples.
Includes anchor-only links (#section) for validation.
Skips external links (http/https) and mailto.
"""
content = filepath.read_text(encoding="utf-8")
links: list[tuple[int, str, str]] = []
for match in _LINK_RE.finditer(content):
url = match.group(2).strip()
# Skip external links and mailto
if url.startswith(("http://", "https://", "mailto:")):
continue
line_num = content[: match.start()].count("\n") + 1
links.append((line_num, match.group(1), url))
return links
def check_required_files(root: Path) -> list[str]:
"""Check that required files exist."""
issues: list[str] = []
for filename in REQUIRED_FILES:
if not (root / filename).exists():
issues.append(f"Missing required file: {filename}")
return issues
def check_docs_structure(root: Path, docs_dir: Path) -> list[str]:
"""Check that docs directory has required structure."""
issues: list[str] = []
if not docs_dir.exists():
issues.append(f"Docs directory not found: {docs_dir}")
return issues
for filename in REQUIRED_DOC_FILES:
if not (docs_dir / filename).exists():
issues.append(f"Missing required doc file: docs/{filename}")
mapping_file = docs_dir / "mapping.json"
if mapping_file.exists():
try:
mapping = json.loads(mapping_file.read_text(encoding="utf-8"))
if not isinstance(mapping, dict):
issues.append("docs/mapping.json must be a JSON object")
elif not mapping:
issues.append("docs/mapping.json is empty")
except json.JSONDecodeError as e:
issues.append(f"docs/mapping.json is invalid JSON: {e}")
return issues
def check_internal_links(root: Path, docs_dir: Path) -> list[str]:
"""Check that all internal links in markdown files resolve."""
issues: list[str] = []
md_files = list(root.rglob("*.md"))
# Exclude .venv, .git, node_modules
md_files = [
f
for f in md_files
if not any(part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts)
]
# Load wiki page names from mapping.json — these are valid link targets
wiki_pages: set[str] = set()
mapping_file = docs_dir / "mapping.json"
if mapping_file.exists():
try:
mapping = json.loads(mapping_file.read_text(encoding="utf-8"))
wiki_pages = set(mapping.values())
except (json.JSONDecodeError, AttributeError):
pass
for md_file in md_files:
rel_path = md_file.relative_to(root)
links = extract_links(md_file)
headings = extract_headings(md_file)
for line_num, _link_text, url in links:
# Split into path and anchor
if "#" in url:
path_part, anchor = url.split("#", 1)
else:
path_part, anchor = url, ""
# Skip wiki page references (no file extension, no /, matches mapping.json values)
if path_part and "." not in path_part and "/" not in path_part:
if path_part in wiki_pages:
continue
# Also skip if it looks like a wiki page name (CamelCase or hyphenated)
# without a file extension — can't verify these locally
if not any(c in path_part for c in "/\\"):
continue
# Resolve relative path
if path_part:
target = (md_file.parent / path_part).resolve()
if not target.exists():
issues.append(f"{rel_path}:{line_num}: broken link '{url}' — file not found: {path_part}")
continue
# Check anchor in target file
if anchor:
target_headings = extract_headings(target)
target_slug = slugify(anchor)
if target_slug not in target_headings:
issues.append(f"{rel_path}:{line_num}: broken anchor '#{anchor}' in {path_part}")
elif anchor:
# Anchor-only link — check in current file
anchor_slug = slugify(anchor)
if anchor_slug not in headings:
issues.append(f"{rel_path}:{line_num}: broken anchor '#{anchor}'")
return issues
def check_heading_hierarchy(root: Path) -> list[str]:
"""Check that headings don't skip levels."""
issues: list[str] = []
md_files = [
f
for f in root.rglob("*.md")
if not any(part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts)
]
for md_file in md_files:
rel_path = md_file.relative_to(root)
content = strip_code_blocks(md_file.read_text(encoding="utf-8"))
prev_level = 0
for match in _HEADING_RE.finditer(content):
level = len(match.group(1))
if prev_level > 0 and level > prev_level + 1:
issues.append(f"{rel_path}: heading hierarchy skip — H{prev_level} → H{level}: '{match.group(2)}'")
prev_level = level
return issues
def check_todo_fixme(root: Path) -> list[str]:
"""Check for TODO/FIXME/HACK/XXX markers in documentation.
Only flags actual TODO/FIXME markers (e.g., "TODO: fix this"), not
references to the word "TODO" in rules or documentation about TODOs.
"""
issues: list[str] = []
md_files = [
f
for f in root.rglob("*.md")
if not any(part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts)
]
for md_file in md_files:
rel_path = md_file.relative_to(root)
content = md_file.read_text(encoding="utf-8")
for match in _TODO_RE.finditer(content):
line_num = content[: match.start()].count("\n") + 1
line = content.splitlines()[line_num - 1] if line_num <= len(content.splitlines()) else ""
issues.append(f"{rel_path}:{line_num}: TODO/FIXME found: {line.strip()}")
return issues
def check_trailing_whitespace(root: Path) -> list[str]:
"""Check for trailing whitespace in markdown files."""
issues: list[str] = []
md_files = [
f
for f in root.rglob("*.md")
if not any(part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts)
]
for md_file in md_files:
rel_path = md_file.relative_to(root)
content = md_file.read_text(encoding="utf-8")
for i, line in enumerate(content.splitlines(), 1):
if _TRAILING_WS_RE.search(line):
issues.append(f"{rel_path}:{i}: trailing whitespace")
return issues
def check_stale_docs(root: Path) -> list[str]:
"""Check for stale documentation (not modified in >180 days)."""
issues: list[str] = []
threshold = datetime.now() - timedelta(days=STALE_THRESHOLD_DAYS)
md_files = [
f
for f in root.rglob("*.md")
if not any(part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts)
]
for md_file in md_files:
rel_path = md_file.relative_to(root)
mtime = datetime.fromtimestamp(md_file.stat().st_mtime)
if mtime < threshold:
days_old = (datetime.now() - mtime).days
issues.append(f"{rel_path}: stale doc — not modified in {days_old} days")
return issues
def check_duplicate_headings(root: Path) -> list[str]:
"""Check for duplicate headings within the same file."""
issues: list[str] = []
md_files = [
f
for f in root.rglob("*.md")
if not any(part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts)
]
for md_file in md_files:
rel_path = md_file.relative_to(root)
# Skip auto-generated files like CHANGELOG.md
if md_file.name in DUPLICATE_HEADING_EXCLUDES:
continue
content = strip_code_blocks(md_file.read_text(encoding="utf-8"))
seen: dict[str, int] = {}
for match in _HEADING_RE.finditer(content):
text = match.group(2)
slug = slugify(text)
if slug in seen:
issues.append(f"{rel_path}: duplicate heading '{text}'")
seen[slug] = 1
return issues
@click.command()
@click.option("--root", default=".", help="Repository root directory.")
@click.option("--docs-dir", default=None, help="Docs directory (default: <root>/docs).")
@click.option("--check-links/--no-check-links", default=True, help="Check internal links.")
@click.option("--check-headings/--no-check-headings", default=True, help="Check heading hierarchy.")
@click.option("--check-todo/--no-check-todo", default=True, help="Check for TODO/FIXME.")
@click.option("--check-stale/--no-check-stale", default=False, help="Check for stale docs.")
@click.option("--check-trailing/--no-check-trailing", default=True, help="Check trailing whitespace.")
@click.option("--check-duplicates/--no-check-duplicates", default=True, help="Check duplicate headings.")
@click.option("--fix", is_flag=True, default=False, help="Auto-fix trailing whitespace.")
def main(
root: str,
docs_dir: str | None,
check_links: bool,
check_headings: bool,
check_todo: bool,
check_stale: bool,
check_trailing: bool,
check_duplicates: bool,
fix: bool,
) -> None:
"""Lint documentation files for structure, links, and quality."""
root_path = Path(root).resolve()
docs_path = Path(docs_dir) if docs_dir else root_path / "docs"
click.echo(_("Linting documentation in {root}...", root=str(root_path)))
all_issues: list[str] = []
# Structure checks
click.echo(_("Checking required files..."))
all_issues.extend(check_required_files(root_path))
click.echo(_("Checking docs structure..."))
all_issues.extend(check_docs_structure(root_path, docs_path))
# Link checks
if check_links:
click.echo(_("Checking internal links..."))
all_issues.extend(check_internal_links(root_path, docs_path))
# Heading hierarchy
if check_headings:
click.echo(_("Checking heading hierarchy..."))
all_issues.extend(check_heading_hierarchy(root_path))
# Duplicate headings
if check_duplicates:
click.echo(_("Checking duplicate headings..."))
all_issues.extend(check_duplicate_headings(root_path))
# TODO/FIXME
if check_todo:
click.echo(_("Checking for TODO/FIXME markers..."))
all_issues.extend(check_todo_fixme(root_path))
# Trailing whitespace
if check_trailing:
click.echo(_("Checking trailing whitespace..."))
ws_issues = check_trailing_whitespace(root_path)
if fix and ws_issues:
fixed = 0
md_files = [
f
for f in root_path.rglob("*.md")
if not any(
part in {".venv", ".git", "node_modules", "__pycache__", ".pytest_cache"} for part in f.parts
)
]
for md_file in md_files:
content = md_file.read_text(encoding="utf-8")
fixed_content = _TRAILING_WS_RE.sub("", content)
if content != fixed_content:
md_file.write_text(fixed_content, encoding="utf-8")
fixed += 1
click.echo(_(" Auto-fixed trailing whitespace in {n} files", n=fixed))
else:
all_issues.extend(ws_issues)
# Stale docs
if check_stale:
click.echo(_("Checking for stale docs..."))
stale = check_stale_docs(root_path)
for issue in stale:
click.echo(f" WARN: {issue}")
# Stale docs are warnings, not errors
click.echo(_(" {n} stale docs found (warnings only)", n=len(stale)))
# Report
click.echo(f"\n{'=' * 60}")
if all_issues:
click.echo(_("FAIL: {n} documentation issues found:", n=len(all_issues)))
for issue in all_issues:
click.echo(f" - {issue}")
sys.exit(1)
else:
click.echo(_("PASS: All documentation checks passed!"))
if __name__ == "__main__": # pragma: no cover
main()
+5 -5
View File
@@ -13,7 +13,8 @@ import click
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
from devx.api_clients import VikunjaClient
from devx.config import DEFAULT_PER_PAGE, TASK_ID_RE, TASK_PREFIX, VIKUNJA_API_URL, VIKUNJA_PROJECT_ID
from devx.ci._shared import extract_task_id as _extract_task_id
from devx.config import DEFAULT_PER_PAGE, TASK_PREFIX, VIKUNJA_API_URL, VIKUNJA_PROJECT_ID
from devx.exceptions import APIError
from devx.i18n import _
@@ -47,10 +48,9 @@ def _get_git_commit_sha() -> str:
def extract_task_id(commit_msg: str) -> str:
"""Extract DEVX-N task identifier from the first line of commit message."""
"""Extract task identifier from the first line of commit message (delegates to shared utility)."""
first_line = commit_msg.split("\n")[0]
match = TASK_ID_RE.search(first_line)
return match.group(0) if match else ""
return _extract_task_id(first_line)
def extract_conventional_msg(commit_msg: str) -> str:
@@ -61,7 +61,7 @@ def extract_conventional_msg(commit_msg: str) -> str:
- ``DEVX-N <message>`` (current, space-separated)
"""
first_line = commit_msg.split("\n")[0]
return re.sub(r"^DEVX-\d+[:\s]\s*", "", first_line)
return re.sub(rf"^{TASK_PREFIX}-\d+[:\s]\s*", "", first_line)
def resolve_task_id(client: VikunjaClient, task_id: str) -> int:
+20
View File
@@ -387,14 +387,34 @@ def check_documentation(files: list[dict[str, Any]], result: ReviewResult) -> No
for f in files
)
has_ansible_changes = any(f.get("filename", "").startswith("ansible/") for f in files)
has_tofu_changes = any(f.get("filename", "").startswith("tofu/") for f in files)
has_workflow_changes = any(f.get("filename", "").startswith(".gitea/") for f in files)
# Check for TODO/FIXME in changed docs
todo_issues: list[str] = []
for f in files:
filename = f.get("filename", "")
if filename.endswith(".md") and filename.startswith(("docs/", "README", "AGENTS")):
# Can't check file content from PR API easily, but flag if patch adds TODO
patch = f.get("patch", "")
if patch and re.search(r"^\+.*\b(TODO|FIXME|HACK|XXX)\b", patch, re.IGNORECASE):
todo_issues.append(f"{filename}: new TODO/FIXME added in documentation")
if has_src_changes and not has_doc_changes:
result.add_summary("- Documentation: WARNING — source files changed but no docs updated")
elif has_ansible_changes and not has_doc_changes:
result.add_summary("- Documentation: WARNING — Ansible role changed but no docs updated")
elif has_tofu_changes and not has_doc_changes:
result.add_summary("- Documentation: WARNING — OpenTofu changes but no docs updated")
elif has_workflow_changes and not has_doc_changes:
result.add_summary("- Documentation: INFO — workflow changes (consider updating CI docs if behavior changed)")
else:
result.add_summary("- Documentation: OK")
if todo_issues:
for issue in todo_issues:
result.add_summary(f"- Documentation: WARNING — {issue}")
def check_test_coverage(files: list[dict[str, Any]], result: ReviewResult) -> None:
"""Check that tests are updated for source changes."""
+22 -11
View File
@@ -28,6 +28,8 @@ from typing import Any
import click
from devx.i18n import _
def _repo_root() -> Path:
"""Resolve repo root from GITHUB_WORKSPACE or cwd."""
@@ -68,7 +70,7 @@ def fetch_latest_master(branch: str = "master") -> None:
"""
_run(["git", "fetch", "origin", branch]) # nosec B607
_run(["git", "reset", "--hard", f"origin/{branch}"]) # nosec B607
click.echo(f"Synced to latest origin/{branch}")
click.echo(_("Synced to latest origin/{branch}", branch=branch))
def generate_badges(output_dir: str) -> None:
@@ -76,8 +78,8 @@ def generate_badges(output_dir: str) -> None:
_run([sys.executable, "-m", "devx.tools.generate_badges", "--output-dir", output_dir])
badges = list(Path(output_dir).glob("*.svg"))
if not badges:
raise click.ClickException("No badge SVG files generated")
click.echo(f"Generated {len(badges)} badge files")
raise click.ClickException(_("No badge SVG files generated"))
click.echo(_("Generated {count} badge files", count=len(badges)))
def push_to_badges_branch(badges_dir: str) -> str:
@@ -99,12 +101,12 @@ def push_to_badges_branch(badges_dir: str) -> str:
_run(["git", "add", "./*.svg"]) # nosec B607
_run(["git", "commit", "--no-verify", "-m", "Update badges [skip ci]"]) # nosec B607
_run(["git", "push", "origin", "badges", "--force"]) # nosec B607
click.echo("Badges pushed to badges branch")
click.echo(_("Badges pushed to badges branch"))
# Get the commit SHA of the badges branch
result = _run_capture(["git", "rev-parse", "HEAD"]) # nosec B607
sha = result.stdout.strip()
click.echo(f"Badges commit SHA: {sha}")
click.echo(_("Badges commit SHA: {sha}", sha=sha))
return sha
@@ -142,11 +144,11 @@ def update_readme_with_badge_sha(badges_sha: str, repo_root: Path | None = None)
new_content = update_badge_urls(content, badges_sha)
if new_content != content:
filepath.write_text(new_content)
click.echo(f"Updated badge URLs in {filename}")
click.echo(_("Updated badge URLs in {filename}", filename=filename))
updated_any = True
if not updated_any:
click.echo("No badge URLs found to update — README already up to date")
click.echo(_("No badge URLs found to update — README already up to date"))
return
_run(["git", "add", "README.md", "docs/index.md"]) # nosec B607
@@ -160,7 +162,7 @@ def update_readme_with_badge_sha(badges_sha: str, repo_root: Path | None = None)
]
) # nosec B607
_run(["git", "push", "origin", "master"]) # nosec B607
click.echo(f"Pushed README update with badge SHA {badges_sha[:8]}")
click.echo(_("Pushed README update with badge SHA {sha}", sha=badges_sha[:8]))
@click.command()
@@ -193,13 +195,22 @@ def main(output_dir: str, branch: str, no_readme_update: bool, retries: int) ->
except (subprocess.CalledProcessError, RuntimeError) as exc:
last_error = exc
if attempt < retries:
click.echo(f"Badge push attempt {attempt}/{retries} failed — retrying: {exc}")
click.echo(
_(
"Badge push attempt {attempt}/{retries} failed — retrying: {error}",
attempt=attempt,
retries=retries,
error=exc,
)
)
time.sleep(10)
with contextlib.suppress(subprocess.CalledProcessError):
fetch_latest_master(branch)
else:
click.echo(f"Badge push failed after {retries} attempts: {exc}")
raise click.ClickException(f"Badge push failed after {retries} attempts: {last_error}")
click.echo(_("Badge push failed after {retries} attempts: {error}", retries=retries, error=exc))
raise click.ClickException(
_("Badge push failed after {retries} attempts: {error}", retries=retries, error=last_error)
)
if __name__ == "__main__": # pragma: no cover
+14 -36
View File
@@ -37,13 +37,12 @@ from __future__ import annotations
import os
import re
import subprocess # nosec B404
import sys
import click
from dotenv import load_dotenv # pyright: ignore[reportMissingImports,reportUnknownVariableType]
from devx.ci._shared import get_latest_tag
from devx.ci._shared import get_latest_tag, run_cmd, write_github_output
from devx.ci.classify_changes import has_user_facing_changes # cross-CI import, needs PYTHONPATH=.
from devx.i18n import _
@@ -54,25 +53,6 @@ CHANGELOG_FILE = "CHANGELOG.md"
CLIFF_CONFIG = "cliff.toml"
def run_cmd(args: list[str], check: bool = True, capture: bool = True) -> subprocess.CompletedProcess[str]:
"""Run a command and return the completed process."""
result = subprocess.run( # nosec B603
args,
capture_output=capture,
text=True,
check=False,
)
if check and result.returncode != 0:
raise click.ClickException(
_(
"Command failed ({cmd}): {stderr}",
cmd=" ".join(args),
stderr=result.stderr.strip() if result.stderr else result.stdout.strip(),
)
)
return result
def tag_exists(tag: str) -> bool:
"""Check if a git tag already exists."""
result = run_cmd(["git", "tag", "-l", tag], check=False)
@@ -217,7 +197,7 @@ def has_unreleased_changes(bumped_version: str | None = None) -> bool:
def update_init_version(new_version: str) -> None:
"""Update __version__ in __init__.py."""
with open(INIT_FILE) as f:
with open(INIT_FILE, encoding="utf-8") as f:
content = f.read()
if not re.search(r'^__version__\s*=\s*"[^"]*"', content, flags=re.MULTILINE):
raise click.ClickException(_("Could not find __version__ in {file}", file=INIT_FILE))
@@ -228,7 +208,7 @@ def update_init_version(new_version: str) -> None:
count=1,
flags=re.MULTILINE,
)
with open(INIT_FILE, "w") as f:
with open(INIT_FILE, "w", encoding="utf-8") as f:
f.write(updated)
@@ -245,10 +225,10 @@ def update_changelog(changelog: str) -> None:
changelog = changelog[section_match.start() :]
try:
with open(CHANGELOG_FILE) as f:
with open(CHANGELOG_FILE, encoding="utf-8") as f:
existing = f.read()
except FileNotFoundError:
with open(CHANGELOG_FILE, "w") as f:
with open(CHANGELOG_FILE, "w", encoding="utf-8") as f:
f.write(changelog + "\n")
return
@@ -261,7 +241,7 @@ def update_changelog(changelog: str) -> None:
else:
# No version sections found — append
updated = existing.rstrip() + "\n\n" + changelog + "\n"
with open(CHANGELOG_FILE, "w") as f:
with open(CHANGELOG_FILE, "w", encoding="utf-8") as f:
f.write(updated)
@@ -317,18 +297,16 @@ def run_tests() -> None:
click.echo(_("Tests passed."))
def _write_github_output(tag: str) -> None:
def _write_release_tag(tag: str) -> None:
"""Write the release tag to GITHUB_OUTPUT for downstream jobs.
This allows a publish job (needs: release) to read the tag via
``${{ needs.release.outputs.tag }}`` instead of relying on
tag-push event triggering a separate workflow.
"""
github_output = os.environ.get("GITHUB_OUTPUT")
if not github_output:
if not os.environ.get("GITHUB_OUTPUT"):
return
with open(github_output, "a") as f: # noqa: PTH123
f.write(f"tag={tag}\n")
write_github_output("tag", tag)
click.echo(_("Wrote tag {tag} to GITHUB_OUTPUT.", tag=tag))
@@ -360,7 +338,7 @@ def create_and_push_tag(new_version: str, changelog: str, dry_run: bool) -> bool
if not dry_run:
# Ensure the existing tag is pushed
run_cmd(["git", "push", "origin", f"refs/tags/{tag}"], check=False)
_write_github_output(tag)
_write_release_tag(tag)
return False
tag_msg = f"Release v{new_version}\n\n{changelog}"
if dry_run:
@@ -368,7 +346,7 @@ def create_and_push_tag(new_version: str, changelog: str, dry_run: bool) -> bool
return True
run_cmd(["git", "tag", "-a", tag, "-m", tag_msg])
run_cmd(["git", "push", "origin", f"refs/tags/{tag}"])
_write_github_output(tag)
_write_release_tag(tag)
return True
@@ -380,7 +358,7 @@ def create_and_push_tag(new_version: str, changelog: str, dry_run: bool) -> bool
def get_init_version() -> str | None:
"""Read __version__ from the version file."""
try:
with open(INIT_FILE) as f:
with open(INIT_FILE, encoding="utf-8") as f:
content = f.read()
match = re.search(r'^__version__\s*=\s*"([^"]*)"', content, flags=re.MULTILINE)
return match.group(1) if match else None
@@ -391,7 +369,7 @@ def get_init_version() -> str | None:
def get_changelog_versions() -> list[str]:
"""Extract version numbers from CHANGELOG.md headers, in order."""
try:
with open(CHANGELOG_FILE) as f:
with open(CHANGELOG_FILE, encoding="utf-8") as f:
content = f.read()
return re.findall(r"^## \[(\d+\.\d+\.\d+)\]", content, flags=re.MULTILINE)
except FileNotFoundError:
@@ -634,7 +612,7 @@ def main(dry_run: bool, skip_tests: bool, verify: bool) -> None:
tag=release_tag,
)
)
_write_github_output(release_tag)
_write_release_tag(release_tag)
return
# Tag is missing — recover by creating and pushing it
click.echo(
+2 -2
View File
@@ -49,7 +49,7 @@ def load_mapping() -> dict[str, str]:
Validates that the mapping is a dict of string-to-string pairs.
"""
with open(MAPPING_FILE) as f:
with open(MAPPING_FILE, encoding="utf-8") as f:
data = json.load(f)
if not isinstance(data, dict):
raise click.ClickException(
@@ -64,7 +64,7 @@ def load_mapping() -> dict[str, str]:
def read_doc_content(file_path: str) -> str:
"""Read markdown content from a docs file."""
full_path = DOCS_DIR / file_path
with open(full_path) as f:
with open(full_path, encoding="utf-8") as f:
return f.read()
+1 -1
View File
@@ -69,7 +69,7 @@ def main(commit_msg_file: str | None, branch: str | None, from_git: bool) -> Non
if commit_msg_file == "-":
msg = sys.stdin.read().strip()
else:
with open(commit_msg_file) as f:
with open(commit_msg_file, encoding="utf-8") as f:
msg = f.read().strip()
else:
raise click.ClickException(_("Provide a commit message file or use --git."))
+7
View File
@@ -95,6 +95,13 @@ def ci_doc_coverage(args: tuple[str, ...]) -> None:
_run_module("devx.ci.doc_coverage", list(args))
@ci.command("lint-docs")
@click.argument("args", nargs=-1)
def ci_lint_docs(args: tuple[str, ...]) -> None:
"""Lint documentation files for structure, links, and quality."""
_run_module("devx.ci.lint_docs", list(args))
@ci.command("notify-failure")
@click.argument("args", nargs=-1)
def ci_notify_failure(args: tuple[str, ...]) -> None:
+1 -1
View File
@@ -330,7 +330,7 @@ devx-setup-image:
@if [ -d /opt/venv ]; then ln -sf /opt/venv $(DEVX_VENV); . $(DEVX_BIN)/activate; \
_U="$${CI_GITEA_USERNAME:-emil}"; \
if [ -n "$$CI_GITEA_TOKEN" ]; then export PIP_EXTRA_INDEX_URL="https://$$_U:$$CI_GITEA_TOKEN@$(DEVX_GITEA_PYPI_HOST)/api/packages/$(DEVX_GITEA_PYPI_ORG)/pypi/simple/"; fi; \
pip install -e .$(if $(EXTRAS),[$(EXTRAS)],); \
pip install --no-cache-dir -e .$(if $(EXTRAS),[$(EXTRAS)],); \
echo "[devx-setup-image] Linked /opt/venv$(if $(EXTRAS), with [$(EXTRAS)],)."; \
else echo "[devx-setup-image] /opt/venv not found — falling back to setup-ci"; $(MAKE) setup-ci; fi
+1
View File
@@ -0,0 +1 @@
"""Molecule testing helpers for Ansible projects."""
+1 -1
View File
@@ -156,7 +156,7 @@ def main(
gh_output = os.environ.get("GITHUB_OUTPUT")
if not gh_output:
raise click.ClickException("GITHUB_OUTPUT environment variable is not set")
with open(gh_output, "a") as f: # noqa: PTH123
with open(gh_output, "a", encoding="utf-8") as f: # noqa: PTH123
f.write(f"runner-count={count}\n")
f.write(f"runner-indices={json.dumps(indices)}\n")
click.echo(f"Runner count: {count}")
+5 -24
View File
@@ -25,6 +25,7 @@ from pathlib import Path
import click
from devx.ci._shared import lpt_distribute, write_github_env
from devx.i18n import _
from devx.molecule.platforms import PLATFORMS, load_platforms
@@ -216,22 +217,8 @@ def _scenario_weight(scenario: str, role: str | None = None) -> int:
def _lpt_distribute[T](items: list[T], weights: list[int], max_runners: int) -> list[list[T]]:
"""Distribute *items* across *max_runners* using LPT (Longest Processing Time first).
Sorts items by weight (descending), then assigns each to the runner
with the least total weight. This produces a more balanced distribution
than naive round-robin when items have varying costs.
"""
groups: list[list[T]] = [[] for _ in range(max_runners)]
loads = [0] * max_runners
# Sort by weight descending, preserving original order for ties
indexed = sorted(enumerate(items), key=lambda x: (-weights[x[0]], x[0]))
for orig_idx, item in indexed:
# Find the runner with the minimum load
min_runner = min(range(max_runners), key=lambda r: loads[r])
groups[min_runner].append(item)
loads[min_runner] += weights[orig_idx]
return groups
"""Distribute *items* across *max_runners* using LPT (delegates to shared utility)."""
return lpt_distribute(items, weights, max_runners)
def distribute_multi_role(pairs: list[MultiRoleTestPair], max_runners: int) -> list[list[MultiRoleTestPair]]:
@@ -283,14 +270,8 @@ def pairs_for_runner(pairs: list[TestPair], runner_index: int, max_runners: int)
def _write_github_env(key: str, value: str) -> None:
"""Append a key=value line to the $GITHUB_ENV file."""
import os
gh_env = os.environ.get("GITHUB_ENV")
if not gh_env:
raise click.ClickException("GITHUB_ENV environment variable is not set")
with open(gh_env, "a") as f: # noqa: PTH123
f.write(f"{key}={value}\n")
"""Append a key=value line to the $GITHUB_ENV file (delegates to shared utility)."""
write_github_env(key, value)
@click.command()
+1 -1
View File
@@ -41,7 +41,7 @@ def load_platforms(platforms_file: str | Path | None = None) -> list[dict[str, s
path = Path(platforms_file)
if not path.is_file():
return PLATFORMS
with path.open() as f:
with path.open(encoding="utf-8") as f:
data = json.load(f)
if not isinstance(data, list) or not data:
return PLATFORMS
+2 -2
View File
@@ -168,7 +168,7 @@ def start_docker_daemon(timeout: int = DEFAULT_TIMEOUT) -> bool:
click.echo(_("Docker daemon failed to start"))
click.echo("--- dockerd log ---")
try:
with open(log_file.name) as f:
with open(log_file.name, encoding="utf-8") as f:
log_content = f.read()
click.echo(log_content[-3000:] if len(log_content) > 3000 else log_content)
except OSError as e:
@@ -191,7 +191,7 @@ def main(timeout: int) -> None:
# Export DOCKER_HOST to GITHUB_ENV for subsequent CI steps
github_env = os.environ.get("GITHUB_ENV")
if github_env and os.environ.get("DOCKER_HOST"):
with open(github_env, "a") as f:
with open(github_env, "a", encoding="utf-8") as f:
f.write(f"DOCKER_HOST={os.environ['DOCKER_HOST']}\n")
click.echo(f"Exported DOCKER_HOST={os.environ['DOCKER_HOST']} to GITHUB_ENV")
sys.exit(0)
+24
View File
@@ -0,0 +1,24 @@
"""Shared utilities for tools modules."""
from __future__ import annotations
import platform
import click
def arch_string() -> str:
"""Return the architecture string used by release assets.
Maps ``platform.machine()`` to the common release asset naming:
``amd64`` for x86_64, ``arm64`` for aarch64.
Raises:
click.ClickException: If the architecture is not supported.
"""
machine = platform.machine().lower()
if machine in {"x86_64", "amd64"}:
return "amd64"
if machine in {"aarch64", "arm64"}:
return "arm64"
raise click.ClickException(f"Unsupported architecture: {machine}")
+1 -1
View File
@@ -88,7 +88,7 @@ def load_manifest(path: str | Path) -> list[ImageSpec]:
p = Path(path)
if not p.is_file():
raise click.ClickException(_("Manifest file not found: {path}", path=p))
with p.open() as f: # noqa: PTH123
with p.open(encoding="utf-8") as f: # noqa: PTH123
data = json.load(f)
if not isinstance(data, list):
raise click.ClickException(_("Manifest must be a JSON list"))
+19 -10
View File
@@ -200,9 +200,9 @@ def main(
total_kept = 0
total_failed = 0
for name in names:
click.echo(f"\n{'=' * 60}")
click.echo(f"Package: {owner}/{name}")
click.echo(f"{'=' * 60}")
click.echo(_("\n{separator}", separator="=" * 60))
click.echo(_("Package: {owner}/{name}", owner=owner, name=name))
click.echo(_("{separator}", separator="=" * 60))
try:
versions = list_package_versions(base_url, owner, name, token)
except requests.RequestException as exc:
@@ -217,17 +217,19 @@ def main(
click.echo(_("No versions found."))
continue
click.echo(f"Found {len(versions)} version(s):")
click.echo(_("Found {count} version(s):", count=len(versions)))
for v in sort_versions_by_date(versions):
click.echo(f" {v.get('version', '?')} (created: {v.get('created_at', '?')})")
click.echo(
_(" {version} (created: {created})", version=v.get("version", "?"), created=v.get("created_at", "?"))
)
to_delete = select_for_deletion(versions, keep)
kept_count = len(versions) - len(to_delete)
click.echo(f"\nKeeping {kept_count}, would delete {len(to_delete)}")
click.echo(_("\nKeeping {kept}, would delete {count}", kept=kept_count, count=len(to_delete)))
if dry_run:
for v in to_delete:
click.echo(f" [dry-run] Would delete: {v.get('version', '?')}")
click.echo(_(" [dry-run] Would delete: {version}", version=v.get("version", "?")))
total_kept += kept_count
continue
@@ -236,17 +238,24 @@ def main(
for v in to_delete:
version = str(v.get("version", ""))
if delete_package_version(base_url, owner, name, version, token):
click.echo(f" Deleted: {version}")
click.echo(_(" Deleted: {version}", version=version))
deleted_count += 1
else:
click.echo(f" FAILED to delete: {version}", err=True)
click.echo(_(" FAILED to delete: {version}", version=version), err=True)
failed_count += 1
total_deleted += deleted_count
total_kept += kept_count
total_failed += failed_count
click.echo(f"\nDone. Deleted {total_deleted}, kept {total_kept}, failed {total_failed}.")
click.echo(
_(
"\nDone. Deleted {deleted}, kept {kept}, failed {failed}.",
deleted=total_deleted,
kept=total_kept,
failed=total_failed,
)
)
if total_failed > 0:
raise click.ClickException(_("Failed to delete {count} image version(s)", count=total_failed))
+8 -1
View File
@@ -50,11 +50,17 @@ def _default_branch_protection_config() -> dict[str, Any]:
"push_whitelist_usernames": [],
"enable_status_check": True,
"status_check_contexts": _default_status_checks(),
"required_approvals": 0,
"required_approvals": 1,
"dismiss_stale_approvals": True,
"block_on_outdated_branch": True,
"block_on_rejected_reviews": True,
"block_on_official_review_requests": True,
# Prevent admins from force-merging PRs that don't meet branch
# protection requirements (e.g. missing approvals). Without this,
# an admin token can bypass the approval gate via force_merge=true,
# allowing merges that failed the auto-merge CI job to reach master
# and trigger the post-merge release pipeline.
"block_admin_merge_override": True,
}
@@ -123,6 +129,7 @@ def configure_repo(
click.echo(_(" - Dismiss stale approvals: yes"))
click.echo(_(" - Block outdated branches: yes"))
click.echo(_(" - Block rejected reviews: yes"))
click.echo(_(" - Block admin merge override: yes"))
checks = ", ".join(cast(list[str], bp_config["status_check_contexts"]))
click.echo(_(" - Required status checks: {checks}", checks=checks))
+40 -28
View File
@@ -27,6 +27,8 @@ from pathlib import Path
import click
from devx.i18n import _
# Coverage regex matches "TOTAL ... NN%" or "TOTAL ... NN.NN%"
_COVERAGE_RE = re.compile(r"TOTAL.*?(\d+(?:\.\d+)?)%")
_PASSED_RE = re.compile(r"(\d+) passed")
@@ -197,17 +199,19 @@ def read_version(repo_root: Path) -> str:
"""
pkg = detect_package_name(repo_root)
if pkg is None:
click.echo(" WARNING: No Python package found under src/ — version badge will show 'unknown'")
click.echo(_(" WARNING: No Python package found under src/ — version badge will show 'unknown'"))
return "unknown"
init_file = repo_root / "src" / pkg / "__init__.py"
if not init_file.exists():
click.echo(f" WARNING: {init_file} not found — version badge will show 'unknown'")
click.echo(_(" WARNING: {init_file} not found — version badge will show 'unknown'", init_file=init_file))
return "unknown"
content = init_file.read_text()
match = re.search(r'__version__\s*=\s*["\']([^"\']+)["\']', content)
if match:
return match.group(1)
click.echo(f" WARNING: No __version__ found in {init_file} — version badge will show 'unknown'")
click.echo(
_(" WARNING: No __version__ found in {init_file} — version badge will show 'unknown'", init_file=init_file)
)
return "unknown"
@@ -278,11 +282,11 @@ def collect_coverage_and_tests(repo_root: Path) -> tuple[dict[str, str | int], d
"""
cov_target = detect_coverage_target(repo_root)
if cov_target is None:
click.echo(" WARNING: No coverage target detected (no src/ package, no --cov in pyproject.toml)")
click.echo(_(" WARNING: No coverage target detected (no src/ package, no --cov in pyproject.toml)"))
return make_badge("coverage", "unknown", "lightgrey"), make_badge("tests", "unknown", "lightgrey")
testpaths = detect_testpaths(repo_root)
click.echo(f" Test paths: {testpaths or '(pytest defaults)'}")
click.echo(_(" Test paths: {testpaths}", testpaths=testpaths or "(pytest defaults)"))
cmd = [
sys.executable,
@@ -302,18 +306,18 @@ def collect_coverage_and_tests(repo_root: Path) -> tuple[dict[str, str | int], d
if coverage is not None:
cov_badge = make_badge("coverage", f"{coverage:.0f}%", coverage_color(coverage))
else:
click.echo(f" WARNING: Could not extract coverage from pytest output (rc={rc})")
click.echo(f" pytest stdout (last 300 chars): {stdout.strip()[-300:]}")
click.echo(f" pytest stderr (last 300 chars): {stderr.strip()[-300:]}")
click.echo(_(" WARNING: Could not extract coverage from pytest output (rc={rc})", rc=rc))
click.echo(_(" pytest stdout (last 300 chars): {stdout}", stdout=stdout.strip()[-300:]))
click.echo(_(" pytest stderr (last 300 chars): {stderr}", stderr=stderr.strip()[-300:]))
cov_badge = make_badge("coverage", "unknown", "red")
test_count = extract_test_count(combined)
if test_count is not None:
tests_badge = make_badge("tests", f"{test_count} passing", "brightgreen" if rc == 0 else "red")
else:
click.echo(f" WARNING: Could not extract test count from pytest output (rc={rc})")
click.echo(f" pytest stdout (last 300 chars): {stdout.strip()[-300:]}")
click.echo(f" pytest stderr (last 300 chars): {stderr.strip()[-300:]}")
click.echo(_(" WARNING: Could not extract test count from pytest output (rc={rc})", rc=rc))
click.echo(_(" pytest stdout (last 300 chars): {stdout}", stdout=stdout.strip()[-300:]))
click.echo(_(" pytest stderr (last 300 chars): {stderr}", stderr=stderr.strip()[-300:]))
tests_badge = make_badge("tests", "unknown", "red")
return cov_badge, tests_badge
@@ -328,8 +332,8 @@ def collect_doc_coverage(repo_root: Path) -> dict[str, str | int]:
doc_pct = extract_doc_coverage(stdout)
if doc_pct is not None:
return make_badge("docs", f"{doc_pct}%", doc_coverage_color(doc_pct))
click.echo(f" WARNING: Could not extract doc coverage (rc={rc})")
click.echo(f" stderr: {stderr.strip()[:200]}")
click.echo(_(" WARNING: Could not extract doc coverage (rc={rc})", rc=rc))
click.echo(_(" stderr: {stderr}", stderr=stderr.strip()[:200]))
return make_badge("docs", "unknown", "red")
@@ -348,7 +352,7 @@ def collect_quality(repo_root: Path) -> dict[str, str | int]:
([sys.executable, "-m", "pyright"], "pyright"),
([sys.executable, "-m", "bandit", "-r", "src/"], "bandit"),
]:
rc, _, stderr = run_command(cmd, cwd=repo_root)
rc, _stdout, stderr = run_command(cmd, cwd=repo_root)
if rc == 0:
results.append(True)
tool_names.append(f"{name}: pass")
@@ -356,16 +360,16 @@ def collect_quality(repo_root: Path) -> dict[str, str | int]:
results.append(False)
# Distinguish "tool not installed" from "tool found issues"
if "No module named" in stderr or "not found" in stderr.lower():
click.echo(f" WARNING: {name} not installed — skipping (counted as pass)")
click.echo(_(" WARNING: {name} not installed — skipping (counted as pass)", name=name))
results[-1] = True
tool_names.append(f"{name}: not installed (skipped)")
else:
tool_names.append(f"{name}: FAIL")
click.echo(f" WARNING: {name} failed (rc={rc})")
click.echo(f" stderr: {stderr.strip()[:200]}")
click.echo(_(" WARNING: {name} failed (rc={rc})", name=name, rc=rc))
click.echo(_(" stderr: {stderr}", stderr=stderr.strip()[:200]))
all_pass = all(results)
click.echo(f" Quality checks: {', '.join(tool_names)}")
click.echo(_(" Quality checks: {checks}", checks=", ".join(tool_names)))
return make_badge("code quality", "A" if all_pass else "F", "brightgreen" if all_pass else "red")
@@ -377,28 +381,28 @@ def generate_badges(output_dir: Path, repo_root: Path | None = None) -> dict[str
repo_root: Repository root (auto-detected if None).
"""
root = repo_root or resolve_repo_root()
click.echo(f" Repo root: {root}")
click.echo(_(" Repo root: {root}", root=root))
pkg = detect_package_name(root)
click.echo(f" Package: {pkg or 'none'}")
click.echo(_(" Package: {pkg}", pkg=pkg or "none"))
badges: dict[str, dict[str, str | int]] = {}
# 1. Code coverage + test count (single pytest-cov run)
click.echo(" Collecting coverage and tests...")
click.echo(_(" Collecting coverage and tests..."))
cov_badge, tests_badge = collect_coverage_and_tests(root)
badges["coverage"] = cov_badge
badges["tests"] = tests_badge
# 2. Documentation coverage
click.echo(" Collecting doc coverage...")
click.echo(_(" Collecting doc coverage..."))
badges["docs"] = collect_doc_coverage(root)
# 3. Code quality (ruff + pyright + bandit)
click.echo(" Collecting code quality...")
click.echo(_(" Collecting code quality..."))
badges["quality"] = collect_quality(root)
# 4. Version
click.echo(" Collecting version...")
click.echo(_(" Collecting version..."))
version = read_version(root)
badges["version"] = make_badge("version", f"v{version}", "blue")
@@ -411,7 +415,7 @@ def generate_badges(output_dir: Path, repo_root: Path | None = None) -> dict[str
svg = render_svg(str(badge["label"]), str(badge["message"]), str(badge["color"]))
path = output_dir / f"{name}.svg"
path.write_text(svg)
click.echo(f" Generated: {path}")
click.echo(_(" Generated: {path}", path=path))
return badges
@@ -431,11 +435,19 @@ def cli(output_dir: str, repo_root: str | None) -> None:
"""Generate self-contained SVG badge files from project metrics."""
out = Path(output_dir)
root = Path(repo_root) if repo_root else None
click.echo(f"Generating badges in {out}...")
click.echo(_("Generating badges in {out}...", out=out))
badges = generate_badges(out, repo_root=root)
click.echo(f"\nGenerated {len(badges)} badges:")
click.echo(_("\nGenerated {count} badges:", count=len(badges)))
for name, badge in badges.items():
click.echo(f" {name}: {badge['label']}={badge['message']} ({badge['color']})")
click.echo(
_(
" {name}: {label}={message} ({color})",
name=name,
label=badge["label"],
message=badge["message"],
color=badge["color"],
)
)
if __name__ == "__main__": # pragma: no cover
+7 -13
View File
@@ -7,7 +7,6 @@ pre-built Linux binary from the official GitHub releases.
from __future__ import annotations
import platform
import shutil
import subprocess # nosec B404
import urllib.request
@@ -15,6 +14,8 @@ from pathlib import Path
import click
from devx.tools._shared import arch_string
CHECKMAKE_VERSION = "0.3.2"
RELEASE_URL_TEMPLATE = (
"https://github.com/checkmake/checkmake/releases/download/"
@@ -23,18 +24,11 @@ RELEASE_URL_TEMPLATE = (
TARGET_PATH = Path("/usr/local/bin/checkmake")
def _arch() -> str:
"""Return the architecture string used by checkmake releases."""
machine = platform.machine().lower()
if machine in {"x86_64", "amd64"}:
return "amd64"
if machine in {"aarch64", "arm64"}:
return "arm64"
raise click.ClickException(f"Unsupported architecture: {machine}")
def _install_with_go() -> bool:
"""Install checkmake using go install if Go is available."""
"""Install checkmake using go install if Go is available.
Returns True if the installation succeeded, False if Go is not installed.
"""
go_bin = shutil.which("go")
if go_bin is None:
return False
@@ -51,7 +45,7 @@ def _install_with_go() -> bool:
def _download_binary() -> None:
"""Download the prebuilt checkmake binary for the current architecture."""
url = RELEASE_URL_TEMPLATE.format(arch=_arch())
url = RELEASE_URL_TEMPLATE.format(arch=arch_string())
urllib.request.urlretrieve(url, TARGET_PATH) # nosec B310
TARGET_PATH.chmod(0o755)
+4 -7
View File
@@ -44,13 +44,10 @@ HADOLINT_VERSION = "2.12.0"
def _arch() -> str:
"""Return the architecture string used by release assets."""
machine = platform.machine().lower()
if machine in {"x86_64", "amd64"}:
return "amd64"
if machine in {"aarch64", "arm64"}:
return "arm64"
raise click.ClickException(f"Unsupported architecture: {machine}")
"""Return the architecture string used by release assets (delegates to shared utility)."""
from devx.tools._shared import arch_string
return arch_string()
def _ensure_target_dir() -> Path:
+943 -327
View File
File diff suppressed because it is too large Load Diff
+12 -33
View File
@@ -6,7 +6,7 @@ from unittest.mock import MagicMock, patch
import pytest
import requests
from devx.api_clients import GiteaClient, VikunjaClient, _is_retryable, _parse_error
from devx.api_clients import GiteaClient, VikunjaClient, _parse_error
from devx.config import (
DEFAULT_PER_PAGE,
DEFAULT_TIMEOUT,
@@ -507,7 +507,7 @@ class TestGiteaClient:
assert result["id"] == 1
assert client._session.request.call_count == 2
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_request_retries_on_429(self, mock_sleep: MagicMock) -> None:
"""Should retry on 429 rate limit with exponential backoff."""
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -520,7 +520,7 @@ class TestGiteaClient:
assert client._session.request.call_count == 3
assert mock_sleep.call_count == 2
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_request_retries_on_503(self, mock_sleep: MagicMock) -> None:
"""Should retry on 503 service unavailable."""
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -532,7 +532,7 @@ class TestGiteaClient:
assert result.json() == {"ok": True}
assert client._session.request.call_count == 2
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_request_no_retry_on_404(self, mock_sleep: MagicMock) -> None:
"""Should NOT retry on 404 — it's not a transient error."""
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -545,7 +545,7 @@ class TestGiteaClient:
assert client._session.request.call_count == 1
mock_sleep.assert_not_called()
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_request_retries_on_connection_error(self, mock_sleep: MagicMock) -> None:
"""Should retry on connection errors."""
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -555,7 +555,7 @@ class TestGiteaClient:
assert result.json() == {"ok": True}
assert client._session.request.call_count == 2
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_request_max_retries_exhausted(self, mock_sleep: MagicMock) -> None:
"""Should raise APIError after max retries on persistent 503."""
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -567,7 +567,7 @@ class TestGiteaClient:
assert exc_info.value.status == 503
assert client._session.request.call_count == 3 # MAX_RETRIES
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_request_connection_error_exhausted(self, mock_sleep: MagicMock) -> None:
"""Should raise APIError after max retries on persistent connection errors."""
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -691,7 +691,7 @@ class TestVikunjaClient:
json={"id": 42, "title": "My task", "done": True},
)
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_http_error_raises_api_error(self, mock_sleep: MagicMock) -> None:
client = VikunjaClient("https://work.example.com", "tok")
mock_resp = MagicMock()
@@ -713,7 +713,7 @@ class TestVikunjaClient:
client.list_tasks()
assert "connection failed" in str(exc_info.value)
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_vikunja_retries_on_503(self, mock_sleep: MagicMock) -> None:
"""VikunjaClient should also retry on 503."""
client = VikunjaClient("https://work.example.com", "tok")
@@ -725,7 +725,7 @@ class TestVikunjaClient:
assert len(result) == 1
assert client._session.request.call_count == 2
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_vikunja_retries_on_connection_error(self, mock_sleep: MagicMock) -> None:
"""VikunjaClient should retry on connection errors."""
client = VikunjaClient("https://work.example.com", "tok")
@@ -735,7 +735,7 @@ class TestVikunjaClient:
assert len(result) == 1
assert client._session.request.call_count == 2
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_vikunja_max_retries_exhausted(self, mock_sleep: MagicMock) -> None:
"""VikunjaClient should raise APIError after max retries on persistent 503."""
client = VikunjaClient("https://work.example.com", "tok")
@@ -747,7 +747,7 @@ class TestVikunjaClient:
assert exc_info.value.status == 503
assert client._session.request.call_count == 3 # MAX_RETRIES
@patch("devx.api_clients.time.sleep")
@patch("time.sleep")
def test_vikunja_connection_error_exhausted(self, mock_sleep: MagicMock) -> None:
"""VikunjaClient should raise APIError after max retries on persistent connection errors."""
client = VikunjaClient("https://work.example.com", "tok")
@@ -818,22 +818,6 @@ class TestVikunjaClient:
assert result["title"] == "Found on page 2"
class TestIsRetryable:
def test_connection_error_is_retryable(self) -> None:
assert _is_retryable(requests.ConnectionError("refused")) is True
def test_timeout_is_retryable(self) -> None:
assert _is_retryable(requests.Timeout("timed out")) is True
def test_429_is_retryable(self) -> None:
err = _mock_http_error(429, "rate limited")
assert _is_retryable(err) is True
def test_404_is_not_retryable(self) -> None:
err = _mock_http_error(404, "not found")
assert _is_retryable(err) is False
class TestGiteaClientPrLabels:
def test_add_pr_label(self) -> None:
client = GiteaClient("https://git.example.com", "tok", "owner", "repo")
@@ -913,8 +897,3 @@ class TestGiteaClientActions:
"https://git.example.com/repos/owner/repo/actions/jobs/10026/logs",
timeout=DEFAULT_TIMEOUT,
)
class TestIsRetryableGeneric:
def test_generic_exception_is_not_retryable(self) -> None:
assert _is_retryable(ValueError("oops")) is False
+2 -2
View File
@@ -6,12 +6,12 @@ import click
import pytest
from click.testing import CliRunner
from devx.ci._shared import run_cmd
from devx.ci.auto_merge import (
extract_conventional_msg,
extract_task_id,
main,
read_taskid,
run_cmd,
validate_pr_title,
validate_pr_title_matches_vikunja,
)
@@ -399,7 +399,7 @@ class TestMain:
mock_client.merge_pr.side_effect = APIError(405, "HEAD branch is behind master")
mock_client_cls.return_value = mock_client
with patch("devx.ci.auto_merge.run_cmd") as mock_run:
with patch("devx.ci._shared.run_cmd") as mock_run:
runner = CliRunner()
result = runner.invoke(
main,
+3 -1
View File
@@ -92,8 +92,10 @@ class TestGetPrTitleFromGitea:
@patch("devx.ci.check_auto_merge_ready.GiteaClient")
def test_returns_none_on_exception(self, mock_client_cls: MagicMock) -> None:
from devx.exceptions import APIError
mock_client = MagicMock()
mock_client.get_pr.side_effect = Exception("API error")
mock_client.get_pr.side_effect = APIError(500, "API error")
mock_client_cls.return_value = mock_client
with patch.dict("os.environ", {"CI_GITEA_TOKEN": "tok"}, clear=True):
result = get_pr_title_from_gitea("owner/repo", 1)
+8
View File
@@ -184,6 +184,14 @@ class TestMain:
result = runner.invoke(check_translations.main, ["--translations", str(trans_file)])
assert result.exit_code == 0
def test_no_translations_file_skips(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""When no translations file is found, should pass with skip message."""
monkeypatch.chdir(tmp_path)
runner = CliRunner()
result = runner.invoke(check_translations.main, [])
assert result.exit_code == 0
assert "No translations file found" in result.output
class TestPrintResult:
def test_prints_all_good(self, capsys: pytest.CaptureFixture[str]) -> None:
+7
View File
@@ -87,6 +87,13 @@ class TestCiCommands:
assert result.exit_code == 0
mock_run.assert_called_once_with("devx.ci.doc_coverage", [])
@patch("devx.cli._run_module")
def test_ci_lint_docs(self, mock_run: MagicMock) -> None:
runner = CliRunner()
result = runner.invoke(cli, ["ci", "lint-docs", "--", "--root", "."])
assert result.exit_code == 0
mock_run.assert_called_once_with("devx.ci.lint_docs", ["--root", "."])
@patch("devx.cli._run_module")
def test_ci_notify_failure(self, mock_run: MagicMock) -> None:
runner = CliRunner()
+2 -1
View File
@@ -32,9 +32,10 @@ class TestDefaultConfigs:
assert config["branch_name"] == "master"
assert config["enable_push"] is True
assert config["enable_push_whitelist"] is False
assert config["required_approvals"] == 0
assert config["required_approvals"] == 1
assert isinstance(config["status_check_contexts"], list)
assert "CI / quality (pull_request)" in config["status_check_contexts"]
assert config["block_admin_merge_override"] is True
def test_default_repo_settings_config(self) -> None:
config = _default_repo_settings_config()
+86 -27
View File
@@ -12,10 +12,13 @@ from devx.ci.doc_coverage import (
main,
)
# Path to devx's own source directory (for testing)
DEVX_SRC_DIR = Path(__file__).resolve().parent.parent.parent / "src" / "devx"
class TestExtractCliCommands:
def test_extracts_commands(self) -> None:
commands = extract_cli_commands()
commands = extract_cli_commands(DEVX_SRC_DIR)
# devx CLI has commands under ci, tools, and molecule groups
assert "auto-merge" in commands
assert "release" in commands
@@ -24,39 +27,30 @@ class TestExtractCliCommands:
assert "install-tools" in commands
def test_returns_list(self) -> None:
commands = extract_cli_commands()
commands = extract_cli_commands(DEVX_SRC_DIR)
assert isinstance(commands, list)
assert len(commands) > 0
def test_no_cli_file(self, monkeypatch: pytest.MonkeyPatch) -> None:
def test_no_cli_file(self, tmp_path: Path) -> None:
"""Returns empty list when CLI file doesn't exist."""
from devx.ci import doc_coverage
monkeypatch.setattr(doc_coverage, "CLI_FILE", Path("/nonexistent/cli.py"))
commands = extract_cli_commands()
commands = extract_cli_commands(tmp_path)
assert commands == []
def test_def_fallback_no_explicit_name(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
def test_def_fallback_no_explicit_name(self, tmp_path: Path) -> None:
"""When a command decorator has no explicit name, falls back to the def name."""
from devx.ci import doc_coverage
fake_cli = tmp_path / "cli.py"
fake_cli.write_text("@click.group()\ndef cli():\n pass\n@cli.command()\ndef my_command():\n pass\n")
monkeypatch.setattr(doc_coverage, "CLI_FILE", fake_cli)
commands = extract_cli_commands()
commands = extract_cli_commands(tmp_path)
assert "my_command" in commands
def test_command_decorator_no_def_fallback(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
def test_command_decorator_no_def_fallback(self, tmp_path: Path) -> None:
"""When a command decorator has no name and no following def, it is skipped."""
from devx.ci import doc_coverage
fake_cli = tmp_path / "cli.py"
# The last @cli.command() has no explicit name and no def statement after it
fake_cli.write_text(
"@click.group()\ndef cli():\n pass\n@cli.command()\ndef real_cmd():\n pass\n@cli.command()\npass\n"
)
monkeypatch.setattr(doc_coverage, "CLI_FILE", fake_cli)
commands = extract_cli_commands()
commands = extract_cli_commands(tmp_path)
# real_cmd should be found via def fallback; the bare @cli.command() is skipped
assert "real_cmd" in commands
assert "pass" not in commands
@@ -96,19 +90,29 @@ class TestMain:
docs = tmp_path / "docs"
(docs / "user").mkdir(parents=True)
(docs / "tech").mkdir(parents=True)
# Get actual commands from the CLI
commands = extract_cli_commands()
src = tmp_path / "src" / "devx"
src.mkdir(parents=True)
(src / "ci").mkdir()
(src / "__init__.py").write_text("")
(src / "ci" / "__init__.py").write_text("")
# Create a fake cli.py with some commands
(src / "cli.py").write_text(
"@click.group()\ndef cli():\n pass\n"
"@cli.command('release')\ndef release():\n pass\n"
"@cli.command('setup')\ndef setup():\n pass\n"
)
# Create a fake module and CI script
(src / "config.py").write_text("# config module")
(src / "ci" / "auto_merge.py").write_text("# auto_merge script")
# Write cli-commands.md with all commands
cli_content = "\n".join(f"## {cmd}" for cmd in commands)
cli_content = "## release\n\n## setup\n"
(docs / "user" / "cli-commands.md").write_text(cli_content)
# Write architecture.md with all modules
from devx.ci.doc_coverage import REQUIRED_MODULES, REQUIRED_SCRIPTS
(docs / "tech" / "architecture.md").write_text(" ".join(REQUIRED_MODULES))
(docs / "tech" / "architecture.md").write_text("config.py")
# Write ci-cd-workflow.md with all scripts
(docs / "tech" / "ci-cd-workflow.md").write_text(" ".join(REQUIRED_SCRIPTS))
(docs / "tech" / "ci-cd-workflow.md").write_text("auto_merge.py")
runner = CliRunner()
result = runner.invoke(main, ["--docs-dir", str(docs)])
result = runner.invoke(main, ["--docs-dir", str(docs), "--source-dir", str(src)])
assert result.exit_code == 0
assert "100%" in result.output
@@ -117,11 +121,21 @@ class TestMain:
docs = tmp_path / "docs"
(docs / "user").mkdir(parents=True)
(docs / "tech").mkdir(parents=True)
src = tmp_path / "src" / "devx"
src.mkdir(parents=True)
(src / "ci").mkdir()
(src / "__init__.py").write_text("")
(src / "ci" / "__init__.py").write_text("")
(src / "cli.py").write_text(
"@click.group()\ndef cli():\n pass\n@cli.command('release')\ndef release():\n pass\n"
)
(src / "config.py").write_text("# config")
(src / "ci" / "auto_merge.py").write_text("# auto_merge")
(docs / "user" / "cli-commands.md").write_text("No commands here.")
(docs / "tech" / "architecture.md").write_text("No modules here.")
(docs / "tech" / "ci-cd-workflow.md").write_text("No scripts here.")
runner = CliRunner()
result = runner.invoke(main, ["--docs-dir", str(docs), "--fail-on-missing"])
result = runner.invoke(main, ["--docs-dir", str(docs), "--source-dir", str(src), "--fail-on-missing"])
assert result.exit_code == 1
def test_missing_docs_warn_only(self, tmp_path: Path) -> None:
@@ -129,10 +143,55 @@ class TestMain:
docs = tmp_path / "docs"
(docs / "user").mkdir(parents=True)
(docs / "tech").mkdir(parents=True)
src = tmp_path / "src" / "devx"
src.mkdir(parents=True)
(src / "ci").mkdir()
(src / "__init__.py").write_text("")
(src / "ci" / "__init__.py").write_text("")
(src / "cli.py").write_text(
"@click.group()\ndef cli():\n pass\n@cli.command('release')\ndef release():\n pass\n"
)
(src / "config.py").write_text("# config")
(src / "ci" / "auto_merge.py").write_text("# auto_merge")
(docs / "user" / "cli-commands.md").write_text("No commands here.")
(docs / "tech" / "architecture.md").write_text("No modules here.")
(docs / "tech" / "ci-cd-workflow.md").write_text("No scripts here.")
runner = CliRunner()
result = runner.invoke(main, ["--docs-dir", str(docs)])
result = runner.invoke(main, ["--docs-dir", str(docs), "--source-dir", str(src)])
assert result.exit_code == 0
assert "MISSING" in result.output
def test_auto_detect_scripts_dir(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""When src/ doesn't exist but scripts/ does, auto-detect it."""
monkeypatch.chdir(tmp_path)
docs = tmp_path / "docs"
(docs / "user").mkdir(parents=True)
(docs / "tech").mkdir(parents=True)
scripts = tmp_path / "scripts"
scripts.mkdir()
(scripts / "cli.py").write_text(
"@click.group()\ndef cli():\n pass\n@cli.command('release')\ndef release():\n pass\n"
)
(scripts / "config.py").write_text("# config")
(docs / "user" / "cli-commands.md").write_text("## release\n")
(docs / "tech" / "architecture.md").write_text("config.py")
(docs / "tech" / "ci-cd-workflow.md").write_text("")
runner = CliRunner()
result = runner.invoke(main, ["--docs-dir", str(docs)])
assert result.exit_code == 0
def test_no_source_dir_falls_back_to_required(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""When no source dir exists, falls back to REQUIRED_MODULES/SCRIPTS."""
monkeypatch.chdir(tmp_path)
docs = tmp_path / "docs"
(docs / "user").mkdir(parents=True)
(docs / "tech").mkdir(parents=True)
(docs / "user" / "cli-commands.md").write_text("")
from devx.ci.doc_coverage import REQUIRED_MODULES, REQUIRED_SCRIPTS
(docs / "tech" / "architecture.md").write_text(" ".join(REQUIRED_MODULES))
(docs / "tech" / "ci-cd-workflow.md").write_text(" ".join(REQUIRED_SCRIPTS))
runner = CliRunner()
result = runner.invoke(main, ["--docs-dir", str(docs)])
# No source dir found, so no CLI commands, but modules/scripts from REQUIRED lists
assert result.exit_code == 0
+5 -4
View File
@@ -8,21 +8,22 @@ import pytest
from click import ClickException
import devx.tools.install_checkmake as install_checkmake
from devx.tools._shared import arch_string
class TestArch:
class TestArchString:
def test_amd64(self) -> None:
with patch.object(platform, "machine", return_value="x86_64"):
assert install_checkmake._arch() == "amd64"
assert arch_string() == "amd64"
def test_arm64(self) -> None:
with patch.object(platform, "machine", return_value="aarch64"):
assert install_checkmake._arch() == "arm64"
assert arch_string() == "arm64"
def test_unsupported(self) -> None:
with patch.object(platform, "machine", return_value="riscv64"):
with pytest.raises(ClickException):
install_checkmake._arch()
arch_string()
class TestInstallWithGo:
+436
View File
@@ -0,0 +1,436 @@
"""Unit tests for devx.ci.lint_docs."""
from __future__ import annotations
import json
from datetime import datetime, timedelta
from pathlib import Path
from click.testing import CliRunner
from devx.ci.lint_docs import (
check_docs_structure,
check_duplicate_headings,
check_heading_hierarchy,
check_internal_links,
check_required_files,
check_stale_docs,
check_todo_fixme,
check_trailing_whitespace,
extract_headings,
extract_links,
main,
slugify,
strip_code_blocks,
)
class TestSlugify:
def test_basic(self) -> None:
assert slugify("Hello World") == "hello-world"
def test_special_chars(self) -> None:
assert slugify("Hello, World!") == "hello-world"
def test_multiple_spaces(self) -> None:
assert slugify("Hello World") == "hello-world"
def test_trailing_dash(self) -> None:
assert slugify("Hello World -") == "hello-world--"
def test_empty(self) -> None:
assert slugify("") == ""
class TestExtractHeadings:
def test_extracts_headings(self, tmp_path: Path) -> None:
f = tmp_path / "test.md"
f.write_text("# Title\n\n## Section\n\n### Subsection\n")
headings = extract_headings(f)
assert "title" in headings
assert headings["title"] == 1
assert "section" in headings
assert headings["section"] == 2
assert "subsection" in headings
assert headings["subsection"] == 3
def test_no_headings(self, tmp_path: Path) -> None:
f = tmp_path / "test.md"
f.write_text("Just some text.\nNo headings here.\n")
headings = extract_headings(f)
assert headings == {}
def test_ignores_headings_in_code_blocks(self, tmp_path: Path) -> None:
"""Headings inside code blocks should not be detected."""
f = tmp_path / "test.md"
f.write_text("# Title\n\n```bash\n# Not a heading\n## Also not\n```\n\n## Real Section\n")
headings = extract_headings(f)
assert "title" in headings
assert "real-section" in headings
assert "not-a-heading" not in headings
assert "also-not" not in headings
class TestStripCodeBlocks:
def test_strips_fenced_blocks(self) -> None:
content = "Before\n```bash\n# comment\n```\nAfter"
result = strip_code_blocks(content)
assert "# comment" not in result
assert "Before" in result
assert "After" in result
def test_strips_multiple_blocks(self) -> None:
content = "# Title\n```python\ncode1\n```\nText\n```yaml\ncode2\n```\nEnd"
result = strip_code_blocks(content)
assert "code1" not in result
assert "code2" not in result
assert "Text" in result
assert "End" in result
def test_no_code_blocks(self) -> None:
content = "# Title\n\nSome text."
result = strip_code_blocks(content)
assert result == content
def test_preserves_line_numbers(self) -> None:
content = "Line1\n```\nLine3\n```\nLine5"
result = strip_code_blocks(content)
lines = result.splitlines()
assert len(lines) == 5
assert lines[0] == "Line1"
assert lines[4] == "Line5"
class TestExtractLinks:
def test_extracts_internal_links(self, tmp_path: Path) -> None:
f = tmp_path / "test.md"
f.write_text("[link](other.md)\n[external](https://example.com)\n[anchor](#section)\n")
links = extract_links(f)
# Should return internal + anchor links (not http or mailto)
assert len(links) == 2
assert links[0][2] == "other.md"
assert links[1][2] == "#section"
def test_extracts_links_with_anchors(self, tmp_path: Path) -> None:
f = tmp_path / "test.md"
f.write_text("[link](other.md#section)\n")
links = extract_links(f)
assert len(links) == 1
assert links[0][2] == "other.md#section"
def test_skips_mailto(self, tmp_path: Path) -> None:
f = tmp_path / "test.md"
f.write_text("[email](mailto:test@example.com)\n")
links = extract_links(f)
assert links == []
class TestCheckRequiredFiles:
def test_all_present(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# README")
(tmp_path / "AGENTS.md").write_text("# AGENTS")
(tmp_path / "CHANGELOG.md").write_text("# CHANGELOG")
issues = check_required_files(tmp_path)
assert issues == []
def test_missing_files(self, tmp_path: Path) -> None:
issues = check_required_files(tmp_path)
assert len(issues) == 3
assert any("README.md" in i for i in issues)
assert any("AGENTS.md" in i for i in issues)
assert any("CHANGELOG.md" in i for i in issues)
class TestCheckDocsStructure:
def test_all_present(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
issues = check_docs_structure(tmp_path, docs)
assert issues == []
def test_missing_docs_dir(self, tmp_path: Path) -> None:
issues = check_docs_structure(tmp_path, tmp_path / "docs")
assert len(issues) == 1
assert "Docs directory not found" in issues[0]
def test_missing_index(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
issues = check_docs_structure(tmp_path, docs)
assert any("index.md" in i for i in issues)
def test_invalid_mapping_json(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home")
(docs / "mapping.json").write_text("{invalid json")
issues = check_docs_structure(tmp_path, docs)
assert any("invalid JSON" in i for i in issues)
def test_empty_mapping(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home")
(docs / "mapping.json").write_text("{}")
issues = check_docs_structure(tmp_path, docs)
assert any("empty" in i for i in issues)
def test_mapping_not_object(self, tmp_path: Path) -> None:
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home")
(docs / "mapping.json").write_text("[]")
issues = check_docs_structure(tmp_path, docs)
assert any("JSON object" in i for i in issues)
class TestCheckInternalLinks:
def test_valid_links(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("[link](docs/guide.md)\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "guide.md").write_text("# Guide\n")
issues = check_internal_links(tmp_path, docs)
assert issues == []
def test_broken_file_link(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("[link](nonexistent.md)\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert len(issues) == 1
assert "file not found" in issues[0]
def test_broken_anchor(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("[link](#missing-section)\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert len(issues) == 1
assert "broken anchor" in issues[0]
def test_broken_anchor_in_target(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("[link](guide.md#missing)\n")
(tmp_path / "guide.md").write_text("# Guide\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert len(issues) == 1
assert "broken anchor" in issues[0]
def test_valid_anchor_in_target(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("[link](guide.md#section)\n")
(tmp_path / "guide.md").write_text("# Section\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert issues == []
def test_wiki_page_link_skipped(self, tmp_path: Path) -> None:
"""Links matching wiki page names in mapping.json should be skipped."""
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("[Architecture](Architecture)\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home", "tech/architecture.md": "Architecture"}))
issues = check_internal_links(tmp_path, docs)
assert issues == []
def test_non_wiki_page_no_extension_skipped(self, tmp_path: Path) -> None:
"""Links without file extension and no slash should be skipped (can't verify)."""
(tmp_path / "README.md").write_text("[SomePage](SomePage)\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert issues == []
def test_broken_anchor_in_target_with_content(self, tmp_path: Path) -> None:
"""Broken anchor in an existing target file should be flagged."""
(tmp_path / "README.md").write_text("[link](guide.md#missing)\n")
(tmp_path / "guide.md").write_text("# Real Title\n\nSome content here.\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert len(issues) == 1
assert "broken anchor" in issues[0]
def test_valid_anchor_in_target_with_content(self, tmp_path: Path) -> None:
"""Valid anchor in an existing target file should pass."""
(tmp_path / "README.md").write_text("[link](guide.md#real-title)\n")
(tmp_path / "guide.md").write_text("# Real Title\n\nSome content.\n")
issues = check_internal_links(tmp_path, tmp_path / "docs")
assert issues == []
def test_invalid_mapping_json_ignored(self, tmp_path: Path) -> None:
"""Invalid mapping.json should not crash link checking."""
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("[link](guide.md)\n")
(docs / "guide.md").write_text("# Guide\n")
(docs / "mapping.json").write_text("{invalid json")
issues = check_internal_links(tmp_path, docs)
# Should still work — just without wiki page mappings
assert issues == []
class TestCheckHeadingHierarchy:
def test_valid_hierarchy(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Title\n## Section\n### Sub\n")
issues = check_heading_hierarchy(tmp_path)
assert issues == []
def test_skipped_level(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Title\n### Sub\n")
issues = check_heading_hierarchy(tmp_path)
assert len(issues) == 1
assert "hierarchy skip" in issues[0]
class TestCheckTodoFixme:
def test_no_todo(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("Just some text.\n")
issues = check_todo_fixme(tmp_path)
assert issues == []
def test_found_todo(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("TODO: fix this later\n")
issues = check_todo_fixme(tmp_path)
assert len(issues) == 1
assert "TODO" in issues[0]
def test_found_fixme(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("FIXME: broken code\n")
issues = check_todo_fixme(tmp_path)
assert len(issues) == 1
assert "FIXME" in issues[0]
def test_ignores_todo_in_rules(self, tmp_path: Path) -> None:
"""References to 'TODO' in rules docs should not be flagged."""
(tmp_path / "README.md").write_text("Best practices (no `print()`, no `TODO`/`FIXME`)\n")
issues = check_todo_fixme(tmp_path)
assert issues == []
def test_ignores_todo_without_colon(self, tmp_path: Path) -> None:
"""'TODO' without a colon should not be flagged."""
(tmp_path / "README.md").write_text("The TODO list is empty\n")
issues = check_todo_fixme(tmp_path)
assert issues == []
class TestCheckTrailingWhitespace:
def test_no_trailing(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("No trailing whitespace here\n")
issues = check_trailing_whitespace(tmp_path)
assert issues == []
def test_trailing_spaces(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("Trailing spaces \n")
issues = check_trailing_whitespace(tmp_path)
assert len(issues) == 1
assert "trailing whitespace" in issues[0]
def test_trailing_tabs(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("Trailing tabs\t\n")
issues = check_trailing_whitespace(tmp_path)
assert len(issues) == 1
class TestCheckStaleDocs:
def test_fresh_doc(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("Fresh content\n")
issues = check_stale_docs(tmp_path)
assert issues == []
def test_stale_doc(self, tmp_path: Path) -> None:
f = tmp_path / "README.md"
f.write_text("Old content\n")
# Set mtime to 200 days ago
old_time = (datetime.now() - timedelta(days=200)).timestamp()
import os
os.utime(f, (old_time, old_time))
issues = check_stale_docs(tmp_path)
assert len(issues) == 1
assert "stale" in issues[0]
class TestCheckDuplicateHeadings:
def test_no_duplicates(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Title\n## Section\n")
issues = check_duplicate_headings(tmp_path)
assert issues == []
def test_duplicates(self, tmp_path: Path) -> None:
(tmp_path / "README.md").write_text("# Title\n# Title\n")
issues = check_duplicate_headings(tmp_path)
assert len(issues) == 1
assert "duplicate heading" in issues[0]
def test_changelog_excluded(self, tmp_path: Path) -> None:
"""CHANGELOG.md should be excluded from duplicate heading checks."""
(tmp_path / "CHANGELOG.md").write_text("# Features\n# Features\n# Features\n")
issues = check_duplicate_headings(tmp_path)
assert issues == []
class TestMain:
def test_passes_clean_repo(self, tmp_path: Path) -> None:
"""A clean repo with all files should pass."""
(tmp_path / "README.md").write_text("# Title\n\nContent here.\n")
(tmp_path / "AGENTS.md").write_text("# AGENTS\n\nContent here.\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n\nContent here.\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path)])
assert result.exit_code == 0
assert "PASS" in result.output
def test_fails_on_missing_files(self, tmp_path: Path) -> None:
"""Missing required files should fail."""
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path)])
assert result.exit_code == 1
assert "FAIL" in result.output
def test_fix_trailing_whitespace(self, tmp_path: Path) -> None:
"""--fix should auto-fix trailing whitespace."""
(tmp_path / "README.md").write_text("# Title\n\nContent here. \n")
(tmp_path / "AGENTS.md").write_text("# AGENTS\n\nContent here.\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n\nContent here.\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path), "--fix"])
assert result.exit_code == 0
# Verify whitespace was fixed
content = (tmp_path / "README.md").read_text()
assert "Content here. \n" not in content
assert "Content here.\n" in content
def test_no_check_links(self, tmp_path: Path) -> None:
"""--no-check-links should skip link checking."""
(tmp_path / "README.md").write_text("# Title\n[broken](nonexistent.md)\n")
(tmp_path / "AGENTS.md").write_text("# AGENTS\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path), "--no-check-links"])
assert result.exit_code == 0
def test_stale_docs_warning(self, tmp_path: Path) -> None:
"""--check-stale should warn but not fail."""
(tmp_path / "README.md").write_text("# Title\n")
(tmp_path / "AGENTS.md").write_text("# AGENTS\n")
(tmp_path / "CHANGELOG.md").write_text("# Changelog\n")
docs = tmp_path / "docs"
docs.mkdir()
(docs / "index.md").write_text("# Home\n")
(docs / "mapping.json").write_text(json.dumps({"index.md": "Home"}))
# Make README stale
import os
f = tmp_path / "README.md"
old_time = (datetime.now() - timedelta(days=200)).timestamp()
os.utime(f, (old_time, old_time))
runner = CliRunner()
result = runner.invoke(main, ["--root", str(tmp_path), "--check-stale"])
# Stale docs are warnings, not errors
assert result.exit_code == 0
assert "stale" in result.output
+30
View File
@@ -516,6 +516,36 @@ class TestCheckDocumentation:
check_documentation(files, result)
assert any("Documentation: OK" in s for s in result.summary)
def test_tofu_changes_without_docs_warns(self) -> None:
result = ReviewResult()
files = [{"filename": "tofu/modules/hetzner-vm/main.tf"}]
check_documentation(files, result)
assert any("WARNING" in s for s in result.summary)
def test_workflow_changes_info(self) -> None:
result = ReviewResult()
files = [{"filename": ".gitea/workflows/ci.yml"}]
check_documentation(files, result)
assert any("INFO" in s for s in result.summary)
def test_todo_in_doc_patch_warns(self) -> None:
result = ReviewResult()
files = [{"filename": "docs/guide.md", "patch": "+TODO: fix this later\n+Some content\n"}]
check_documentation(files, result)
assert any("TODO" in s for s in result.summary)
def test_todo_in_readme_patch_warns(self) -> None:
result = ReviewResult()
files = [{"filename": "README.md", "patch": "+FIXME: broken\n"}]
check_documentation(files, result)
assert any("FIXME" in s for s in result.summary)
def test_no_todo_in_doc_patch_ok(self) -> None:
result = ReviewResult()
files = [{"filename": "docs/guide.md", "patch": "+Some content\n"}]
check_documentation(files, result)
assert not any("TODO" in s for s in result.summary)
class TestCheckTestCoverage:
def test_src_changes_without_tests_warns(self) -> None:
+3 -3
View File
@@ -34,20 +34,20 @@ from devx.ci.release import (
class TestRunCmd:
@patch("devx.ci.release.subprocess.run")
@patch("devx.ci._shared.subprocess.run")
def test_success(self, mock_run: MagicMock) -> None:
mock_run.return_value = MagicMock(returncode=0, stderr="", stdout="")
result = run_cmd(["echo", "hi"])
assert result.returncode == 0
mock_run.assert_called_once()
@patch("devx.ci.release.subprocess.run")
@patch("devx.ci._shared.subprocess.run")
def test_failure_raises(self, mock_run: MagicMock) -> None:
mock_run.return_value = MagicMock(returncode=1, stderr="err", stdout="")
with pytest.raises(click.ClickException):
run_cmd(["false"])
@patch("devx.ci.release.subprocess.run")
@patch("devx.ci._shared.subprocess.run")
def test_check_false_no_raise(self, mock_run: MagicMock) -> None:
mock_run.return_value = MagicMock(returncode=1, stderr="err", stdout="")
result = run_cmd(["false"], check=False)
+8 -10
View File
@@ -272,18 +272,16 @@ class TestMain:
assert result.exit_code == 0
mock_start.assert_called_once_with(60)
@patch("devx.molecule.start_docker.os.environ.get")
@patch("devx.molecule.start_docker.start_docker_daemon", return_value=True)
def test_exports_github_env(self, mock_start: MagicMock, mock_get: MagicMock) -> None:
def test_exports_github_env(self, mock_start: MagicMock) -> None:
"""Should write DOCKER_HOST to GITHUB_ENV when available."""
mock_get.side_effect = lambda key, default="": (
"/tmp/github_env" if key == "GITHUB_ENV" else f"unix://{DOCKER_SOCK}" if key == "DOCKER_HOST" else default
)
with patch("builtins.open", mock_open()) as mock_file:
runner = CliRunner()
result = runner.invoke(main, [])
assert result.exit_code == 0
mock_file.assert_called_with("/tmp/github_env", "a")
env = {"GITHUB_ENV": "/tmp/github_env", "DOCKER_HOST": f"unix://{DOCKER_SOCK}"}
with patch.dict("os.environ", env, clear=True):
with patch("builtins.open", mock_open()) as mock_file:
runner = CliRunner()
result = runner.invoke(main, [])
assert result.exit_code == 0
mock_file.assert_called_with("/tmp/github_env", "a", encoding="utf-8")
@patch("devx.molecule.start_docker.os.environ.get", return_value="")
@patch("devx.molecule.start_docker.start_docker_daemon", return_value=True)