DEVX-62: feat: weighted LPT distribution, workflow fixes, decouple vikunja/sync-wiki from release

This commit is contained in:
emil
2026-06-26 17:57:03 +00:00
parent e271c79e93
commit 41c631d5f5
8 changed files with 301 additions and 95 deletions
+29 -5
View File
@@ -66,7 +66,7 @@ src/devx/
│ ├── sync_wiki.py # Sync documentation to Gitea wiki
│ ├── push_badges.py # Generate and push quality badges (--retries for retry on git push failures)
│ ├── notify_failure.py # Create Gitea issues on CI failures (--auto-login)
│ ├── distribute_files.py # Distribute files across parallel runners
│ ├── distribute_files.py # Distribute files across parallel runners (LPT scheduling)
│ ├── integration_guard.py # Run pytest with cross-runner fail-fast
│ ├── check_translations.py # Translation completeness check
│ └── doc_coverage.py # Documentation coverage check
@@ -79,7 +79,7 @@ src/devx/
├── opentofu.py # OpenTofu output helpers (get_tofu_output, get_tofu_vm_ip, get_tofu_vm_field)
└── molecule/ # Optional molecule testing helpers (for Ansible projects)
├── discover_runners.py # Dynamic Gitea runner discovery
├── distribute_molecule.py # Distribute molecule scenarios across runners (--roles-root for multi-role)
├── distribute_molecule.py # Distribute molecule scenarios across runners (LPT scheduling, --roles-root for multi-role)
├── molecule_ci_guard.py # Run molecule with cross-runner fail-fast (--roles-root)
├── molecule_all.py # Run all molecule scenarios locally
└── platforms.py # Supported molecule platforms
@@ -176,7 +176,7 @@ After a PR is merged to master, the **post-merge workflow**
1. **detect-type** — Checks if the commit is a regular merge or a
release commit (`release: vX.Y.Z`). All subsequent jobs skip for
release commits.
release commits (except badges).
2. **release** — Runs `python -m devx.ci.release` which:
- Checks for user-facing changes via `python -m devx.ci.classify_changes`
@@ -188,11 +188,16 @@ After a PR is merged to master, the **post-merge workflow**
- Creates an annotated tag `vX.Y.Z` on the release commit
- Pushes both the commit and tag to master
3. **sync-wiki** — Syncs documentation to the Gitea wiki.
3. **sync-wiki** — Syncs documentation to the Gitea wiki. Runs for ALL
non-release commits (not just when release succeeds), so docs-only
changes still update the wiki.
4. **badges** — Generates and pushes quality badge SVGs to the `badges` branch.
Uses `if: always()` so it runs on every push, including release commits.
5. **vikunja** — Marks the corresponding Vikunja task as done.
5. **vikunja** — Marks the corresponding Vikunja task as done. Runs for ALL
non-release commits (not just when release succeeds), so infrastructure-only
changes still update the task tracker.
The tag push triggers the **publish workflow** (`.gitea/workflows/publish.yml`)
which builds and publishes the package to the Gitea PyPI registry.
@@ -310,6 +315,25 @@ auto-merge:
(needs.molecule-tests.result == 'success' || needs.molecule-tests.result == 'skipped')
```
### LPT Test Distribution Algorithm
`distribute_molecule` and `distribute_files` use **LPT (Longest Processing
Time first)** scheduling instead of naive round-robin. This produces a more
balanced distribution when test items have varying costs:
1. **Weight estimation**: Each item is assigned a weight:
- Molecule scenarios: heuristic by name (`nextcloud`=10, `gitea`=8,
`binary`=2, default=3). See `_SCENARIO_WEIGHTS` in
`distribute_molecule.py`.
- Integration test files: weight by file size in bytes (as a proxy
for test runtime).
2. **LPT assignment**: Items are sorted by weight (descending), then
each is assigned to the runner with the least total weight.
This ensures heavy scenarios (e.g. `nextcloud`) are spread across
different runners rather than clustered on one, reducing the
longest-runner time from ~16 min to ~11 min with 6 runners.
## Config System
devx uses environment variables with `.env` file fallback for configuration.