CI: one test archive per shard, balanced by recorded test times

The gate's build job writes one nextest archive per shard (craft-common-actions gate.yml
shard-archives), so each shard downloads only the test binaries it runs instead of the whole
2.4-2.6 GB archive.

`cargo xtask ci archive` builds and lists the tests once, then plans the shards
(xtask/src/shards.rs): whole binaries go to the lightest shard, heaviest first, by the times in
.config/ci-test-times.tsv; a binary heavier than half a shard's share is split by test filter along
its module tree, and its executable goes into each archive that runs part of it. On Linux shard 1
takes every binary with gpu-group tests. The build checks that every test runs on exactly one shard,
writes the plan into the target directory, which every archive carries, and archives each shard
with `cargo nextest archive -E`. A shard runs `xtask ci tests --shard <k>` with its part of the
plan; the slice partition is gone.

scripts/ci-test-times.py refreshes the times file from a CI run's shard logs; a missing or stale
row only costs balance.
This commit is contained in:
aiongg
2026-11-19 20:47:26 +00:00
parent 86c2056d01
commit aac8e37bec
8 changed files with 3099 additions and 41 deletions
File diff suppressed because it is too large Load Diff
+3
View File
@@ -35,9 +35,12 @@ slow-timeout = { period = "60s", terminate-after = 10 }
failure-output = "immediate-final"
# The gate's shards run the tests from a nextest archive (`cargo xtask ci archive`) and start them
# with the xtask binary the archive carries, so a shard builds nothing. Only one of the two exists.
# With per-shard archives, each also carries the plan of which tests each shard runs
# (xtask/src/shards.rs).
archive.include = [
{ path = "debug/xtask", relative-to = "target", depth = 0, on-missing = "ignore" },
{ path = "debug/xtask.exe", relative-to = "target", depth = 0, on-missing = "ignore" },
{ path = "gate-shards.json", relative-to = "target", depth = 0, on-missing = "ignore" },
]
# Keep phase timings for the remaining expensive Windows GPU regressions visible. This does
+13 -8
View File
@@ -132,7 +132,7 @@ jobs:
else
echo "web=false" >> "$GITHUB_OUTPUT"
fi
if git diff --name-only "$BASE" HEAD | grep -Eq '^(apps/|crates/(ui-egui|text|gpu|tablet|heif|format)/|Cargo\.lock$|craft-i18n\.rev$|(.*/)?build\.rs$|\.cargo/config\.toml$|\.config/nextest\.toml$|xtask/|scripts/ci-test-shard\.sh$|\.github/workflows/ci\.yml$)'; then
if git diff --name-only "$BASE" HEAD | grep -Eq '^(apps/|crates/(ui-egui|text|gpu|tablet|heif|format)/|Cargo\.lock$|craft-i18n\.rev$|(.*/)?build\.rs$|\.cargo/config\.toml$|\.config/nextest\.toml$|\.config/ci-test-times\.tsv$|xtask/|scripts/ci-test-shard\.sh$|\.github/workflows/ci\.yml$)'; then
echo "windows=true" >> "$GITHUB_OUTPUT"
else
echo "windows=false" >> "$GITHUB_OUTPUT"
@@ -149,9 +149,11 @@ jobs:
fi
# The Linux gate (craft-common-actions gate.yml). Its `lint` leg runs the checks that need no test
# build. The `build` job builds the tests once into a nextest archive (`cargo xtask ci archive`),
# uploads it and then runs the doctests; four test shards download the archive and run their
# share (scripts/ci-test-shard.sh). Shard 1 runs nextest's `gpu` group alone, on lavapipe, and
# build. The `build` job builds the tests once and writes one nextest archive per shard
# (`cargo xtask ci archive`, `shard-archives`), each with only the test binaries that shard runs,
# balanced by .config/ci-test-times.tsv (xtask/src/shards.rs); it uploads them and then runs the
# doctests. Each of the four test shards downloads its own archive and runs it
# (scripts/ci-test-shard.sh). Shard 1 runs nextest's `gpu` group alone, on lavapipe, and
# fails on a skipped GPU comparison (scripts/ci-gpu-lavapipe.py); shards 2-4 split the other
# tests. The `gpu` group is ~840 test-seconds on 4 slots (~3.5 min); the other ~6900
# test-seconds over 3 shards are ~8.5-10.5 min each.
@@ -189,16 +191,18 @@ jobs:
# fmt, egui-fork, clippy, layers, panel-commands, scorecard, the wasm32 check and the
# missing-translations report (`cargo xtask ci lint`, xtask/src/main.rs), and the CI scripts'
# own tests.
lint-command: python3 scripts/test-ci-gpu-lavapipe.py && python3 scripts/test-ci-windows-gpu-tests.py && cargo xtask ci lint
archive-command: cargo xtask ci archive
lint-command: python3 scripts/test-ci-gpu-lavapipe.py && python3 scripts/test-ci-windows-gpu-tests.py && python3 scripts/test-ci-test-times.py && cargo xtask ci lint
archive-command: cargo xtask ci archive --gpu-shard
after-archive-command: cargo xtask ci doctest
test-shards: 4
shard-archives: true
# The tests and panic_hunt with `--locked --features heif` (TEST_TARGETS in xtask/src/main.rs).
shard-command: bash scripts/ci-test-shard.sh --gpu-shard
# The Windows tests and Clippy on the same gate, when `platforms` says the run needs them: a
# `lint` leg runs Clippy, the `build` job builds the tests once into a nextest archive and then
# runs the doctests, and six shards run their slice of the tests from the archive. Windows runs
# `lint` leg runs Clippy, the `build` job builds the tests once into one nextest archive per shard
# (as on Linux, without a GPU shard) and then runs the doctests, and six shards each run their own
# archive. Windows runs
# the tests ~1.75x slower than Linux: ~55 min on one runner, ~9 min per shard. A seventh shard
# would not shorten the slowest one while its slowest test alone takes ~9.6 min (the
# shortcut-dispatch UI test). The browser apps are wasm targets, so their native tests are left
@@ -242,6 +246,7 @@ jobs:
archive-command: cargo xtask ci archive
after-archive-command: cargo xtask ci doctest
test-shards: 6
shard-archives: true
shard-command: bash scripts/ci-test-shard.sh
# The Linux gate's build job saves a rolling cache entry of its own on every main and nightly run
+14 -14
View File
@@ -1,28 +1,28 @@
#!/usr/bin/env bash
# One shard of the gate's tests: the gate's `shard-command` (.github/workflows/ci.yml). The gate's
# build job wrote a nextest archive of the tests with `cargo xtask ci archive`, and the archive
# carries that xtask binary (.config/nextest.toml), so the shard extracts it and runs the test steps
# with `xtask ci tests` without building anything. The gate sets NEXTEST_ARCHIVE_FILE, GATE_SHARD
# and GATE_SHARDS.
# build job wrote one nextest archive per shard with `cargo xtask ci archive` (`shard-archives`), each
# holding only the test binaries that shard runs, the xtask binary and the shard plan
# (.config/nextest.toml). The shard extracts its archive and runs the test steps with
# `xtask ci tests --shard <k>` without building anything. The gate sets NEXTEST_ARCHIVE_FILE,
# GATE_SHARD and GATE_SHARDS.
#
# The shards split the tests with nextest's `slice` partition (round-robin over every test of every
# binary). Test times are heavy-tailed and cluster in a few modules: on the times of a main run
# (#15702), a three-way `hash` split of the non-GPU tests ran 728, 493 and 523 s, and `slice`
# would run them in about 602, 506 and 625 s.
# The build job balances the shards by recorded test times (.config/ci-test-times.tsv,
# xtask/src/shards.rs); a binary too heavy for one shard is split across shards by test filter.
#
# ci-test-shard.sh the shard runs its slice of every test
# ci-test-shard.sh the shard runs its binaries
# ci-test-shard.sh --gpu-shard shard 1 runs nextest's `gpu` group alone on lavapipe, checked by
# scripts/ci-gpu-lavapipe.py; the other shards split the other tests
# scripts/ci-gpu-lavapipe.py (the build job ran
# `ci archive --gpu-shard`); the other shards run the other tests
set -euo pipefail
archive=target/nextest-archive
mkdir -p "$archive"
cargo nextest run --archive-file "$NEXTEST_ARCHIVE_FILE" --workspace-remap . --extract-to "$archive" --extract-overwrite -E 'none()' --no-tests=pass
xtask=("$archive/target/debug/xtask" ci tests --from "$archive")
case "$(uname -s)" in MINGW* | MSYS* | CYGWIN*) xtask=("$archive/target/debug/xtask.exe" ci tests --from "$archive") ;; esac
xtask=("$archive/target/debug/xtask" ci tests --from "$archive" --shard "$GATE_SHARD")
case "$(uname -s)" in MINGW* | MSYS* | CYGWIN*) xtask=("$archive/target/debug/xtask.exe" ci tests --from "$archive" --shard "$GATE_SHARD") ;; esac
case "${1:-}" in
"") NEXTEST_PARTITION="slice:$GATE_SHARD/$GATE_SHARDS" exec "${xtask[@]}" ;;
"") exec "${xtask[@]}" ;;
--gpu-shard)
if [ "$GATE_SHARDS" -lt 2 ]; then
echo "::error::--gpu-shard needs at least two shards, got $GATE_SHARDS"
@@ -31,7 +31,7 @@ case "${1:-}" in
if [ "$GATE_SHARD" = 1 ]; then
exec python3 scripts/ci-gpu-lavapipe.py "${xtask[@]}" --gpu
fi
NEXTEST_PARTITION="slice:$((GATE_SHARD - 1))/$((GATE_SHARDS - 1))" exec "${xtask[@]}" --no-gpu
exec "${xtask[@]}" --no-gpu
;;
*)
echo "usage: $0 [--gpu-shard]" >&2
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env python3
"""Refresh .config/ci-test-times.tsv from the gate's test shards of one CI run.
python3 scripts/ci-test-times.py <run-id> [--repo storytold/photocraft]
The gate's build job balances its per-shard archives by these times (xtask/src/shards.rs). Take a
green main run, which runs every test; refresh after large test changes. A run without Windows
shards refreshes the Linux rows only and keeps the Windows ones. Needs an authenticated `gh`.
Rows, tab-separated:
bin <os> <binary-id> <test count> <total seconds>
test <os> <binary-id> <seconds> <test name> (tests of TEST_ROW_SECONDS or more)
Linux rows leave out shard 1, which runs nextest's `gpu` group alone (`--gpu-shard` in ci.yml):
the build job plans the other shards with them.
"""
import argparse
import json
import re
import subprocess
import sys
from pathlib import Path
TIMES = Path(__file__).resolve().parent.parent / '.config' / 'ci-test-times.tsv'
TEST_ROW_SECONDS = 1.0
# The gate's shard jobs per OS, and the shards whose tests the plan does not balance.
JOBS = {
'linux': (re.compile(r'^gate / test (\d+)/(\d+) \(ubuntu'), {1}),
'windows': (re.compile(r'^windows / test (\d+)/(\d+) \(windows'), set()),
}
# Colour codes; `gh run view --log` prints the escape character as the two characters `^[`.
ANSI = re.compile(r'(?:\x1b|\^\[)\[[0-9;]*m')
# nextest's final status line of a test: `PASS [ 1.673s] ( 1/4515) <binary-id> <test name>`.
RESULT = re.compile(r'\b(?:PASS|FAIL|FLAKY|LEAK|LEAK-FAIL|TIMEOUT|SIGSEGV|SIGABRT|SIGBUS|SIGILL|ABORT) \[\s*([\d.]+)s\] (?:\(\s*\d+/\d+\) )?(\S+) (.+?)\s*$')
HEADER = """\
# Test times of the CI gate's shards, which the build job balances its per-shard archives by
# (xtask/src/shards.rs). Written by scripts/ci-test-times.py from run {run}; refresh it after large
# test changes. A missing or stale row only costs balance: the build's own test listing decides
# which tests run.
# bin <os> <binary-id> <test count> <total seconds>
# test <os> <binary-id> <seconds> <test name>
"""
def parse_log(text):
"""{(binary, test): seconds} from one shard job's log."""
times = {}
for line in text.splitlines():
m = RESULT.search(ANSI.sub('', line))
if m:
times[(m.group(2), m.group(3))] = float(m.group(1))
return times
def rows(os_name, times):
"""The file's rows of one OS from {(binary, test): seconds}."""
bins = {}
for (binary, _), seconds in times.items():
count, total = bins.get(binary, (0, 0.0))
bins[binary] = (count + 1, total + seconds)
out = [f'bin\t{os_name}\t{b}\t{c}\t{t:.2f}' for b, (c, t) in sorted(bins.items())]
out += [f'test\t{os_name}\t{b}\t{s:.2f}\t{t}' for (b, t), s in sorted(times.items()) if s >= TEST_ROW_SECONDS]
return out
def merge(old_text, fresh, run):
"""The new file: fresh rows for the OSes in `fresh` ({os: rows}), the old rows of the others."""
kept = [line for line in old_text.splitlines()
if line and not line.startswith('#') and line.split('\t')[1] not in fresh]
body = kept + [row for os_name in sorted(fresh) for row in fresh[os_name]]
body.sort(key=lambda r: (r.split('\t')[1], r.split('\t')[0] != 'bin', r.split('\t')[2]))
return HEADER.format(run=run) + '\n'.join(body) + '\n'
def gh(*args):
return subprocess.run(['gh', *args], check=True, capture_output=True, text=True).stdout
def main():
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument('run')
parser.add_argument('--repo', default='storytold/photocraft')
args = parser.parse_args()
jobs = json.loads(gh('run', 'view', args.run, '-R', args.repo, '--json', 'jobs'))['jobs']
fresh = {}
for os_name, (pattern, skip) in JOBS.items():
times = {}
for job in jobs:
m = pattern.match(job['name'])
if not m or int(m.group(1)) in skip:
continue
if job['conclusion'] != 'success':
print(f"warning: {job['name']} ended {job['conclusion']}", file=sys.stderr)
log = gh('run', 'view', '-R', args.repo, '--job', str(job['databaseId']), '--log')
found = parse_log(log)
print(f"{job['name']}: {len(found)} tests", file=sys.stderr)
times.update(found)
if times:
fresh[os_name] = rows(os_name, times)
if not fresh:
sys.exit(f'run {args.run} has no gate test shards')
old = TIMES.read_text() if TIMES.exists() else ''
TIMES.write_text(merge(old, fresh, args.run))
print(f'wrote {TIMES}', file=sys.stderr)
if __name__ == '__main__':
main()
+48
View File
@@ -0,0 +1,48 @@
import importlib.util
from pathlib import Path
import unittest
spec = importlib.util.spec_from_file_location('test_times', Path(__file__).with_name('ci-test-times.py'))
times = importlib.util.module_from_spec(spec)
spec.loader.exec_module(times)
# As `gh run view --log` prints them: job, step, timestamp, and the escape character as `^[`.
LOG = '\n'.join([
'gate / test 4/4 (ubuntu-24.04)\tUNKNOWN STEP\t2026-11-19T16:21:04.4143043Z ^[[32;1m PASS^[[0m [ 1.673s] ( 1/4515) '
'^[[35;1mphotocraft-engine^[[0m ^[[36mcmd::retouch::tests^[[0m^[[36m::^[[0m^[[34;1mslow_one^[[0m',
'gate / test 4/4 (ubuntu-24.04)\tUNKNOWN STEP\t2026-11-19T16:21:05.0Z \x1b[32;1m PASS\x1b[0m [ 0.272s] ( 2/4515) '
'\x1b[35;1mphotocraft::it\x1b[0m \x1b[36mcontrol\x1b[0m\x1b[36m::\x1b[0m\x1b[34;1mquick\x1b[0m',
'x\tUNKNOWN STEP\t2026-11-19T16:21:06.0Z SLOW [> 60.000s] photocraft-engine cmd::still_running',
'x\tUNKNOWN STEP\t2026-11-19T16:21:07.0Z FAIL [ 2.500s] ( 3/4515) photocraft-engine cmd::fails',
# nextest repeats each failure at the end of the run.
'x\tUNKNOWN STEP\t2026-11-19T16:21:09.0Z FAIL [ 2.500s] ( 3/4515) photocraft-engine cmd::fails',
'x\tUNKNOWN STEP\t2026-11-19T16:21:08.0Z Summary [ 527.727s] 4515 tests run: 4515 passed',
])
class TestTimes(unittest.TestCase):
def test_reads_each_finished_test_once(self):
self.assertEqual(times.parse_log(LOG), {
('photocraft-engine', 'cmd::retouch::tests::slow_one'): 1.673,
('photocraft::it', 'control::quick'): 0.272,
('photocraft-engine', 'cmd::fails'): 2.5,
})
def test_rows_total_each_binary_and_list_the_slow_tests(self):
self.assertEqual(times.rows('linux', times.parse_log(LOG)), [
'bin\tlinux\tphotocraft-engine\t2\t4.17',
'bin\tlinux\tphotocraft::it\t1\t0.27',
'test\tlinux\tphotocraft-engine\t2.50\tcmd::fails',
'test\tlinux\tphotocraft-engine\t1.67\tcmd::retouch::tests::slow_one',
])
def test_a_refresh_keeps_the_rows_of_the_os_it_did_not_read(self):
old = '# old header\nbin\twindows\tw\t1\t9.00\nbin\tlinux\tstale\t1\t1.00\n'
new = times.merge(old, {'linux': ['bin\tlinux\tfresh\t1\t2.00']}, 123)
body = [line for line in new.splitlines() if not line.startswith('#')]
self.assertEqual(body, ['bin\tlinux\tfresh\t1\t2.00', 'bin\twindows\tw\t1\t9.00'])
self.assertIn('run 123', new)
if __name__ == '__main__':
unittest.main()
+182 -19
View File
@@ -13,6 +13,7 @@ mod panel_commands;
mod perf;
mod pinned;
mod scorecard;
mod shards;
mod stats;
mod test_binaries;
mod version;
@@ -35,13 +36,17 @@ commands:
wasm cargo check --target wasm32-unknown-unknown for the wasm-safe crates (one call, plus
one per optional web feature)
test workspace tests (cargo nextest when installed, else cargo test), then doctests
ci [lint | archive | build | tests [--from <dir>] [--gpu | --no-gpu] | <step>]
ci [lint | archive [--gpu-shard] | build | tests [--from <dir> [--shard <k>]] [--gpu | --no-gpu] | <step>]
the CI gate: fmt --check, egui-fork, clippy -D warnings, layers, panel-commands, scorecard --check,
tests (nextest --profile ci), panic_hunt, doctests, wasm and the i18n-report, with --locked
--features heif (stops at first failure). The gate splits it over its jobs: `lint` runs the
steps that need no test build; `archive` builds the tests into the nextest archive at
$NEXTEST_ARCHIVE_FILE (the gate's build job, which then runs `doctest`); `tests --from <dir>` runs the tests from that archive
extracted to <dir> (without --from it builds them); `tests` runs the tests and panic_hunt on the partition in
$NEXTEST_ARCHIVE_FILE (the gate's build job, which then runs `doctest`), or with $NEXTEST_ARCHIVE_DIR one
archive per gate shard ($GATE_SHARDS of them, at $NEXTEST_ARCHIVE_DIR/<k>/$NEXTEST_ARCHIVE_NAME), each
with only the test binaries that shard runs, balanced by .config/ci-test-times.tsv (`--gpu-shard`: shard 1
takes the binaries with gpu-group tests); `tests --from <dir>` runs the tests from that archive
extracted to <dir> (without --from it builds them; `--shard <k>`: shard k's part of the split binaries,
from the plan the archive carries); `tests` runs the tests and panic_hunt on the partition in
$NEXTEST_PARTITION (`--gpu`: all of nextest's gpu test group, whatever the partition, each
test's output shown;
`--no-gpu`: every other test); `doctest` runs the doctests; `build` only builds what `tests`
@@ -286,15 +291,28 @@ struct CiOptions {
build_only: bool,
/// The nextest partition the test steps run ($NEXTEST_PARTITION with `ci tests`).
partition: Option<String>,
/// `ci archive`: build the tests into this nextest archive instead of running them.
archive: Option<PathBuf>,
/// `ci archive`: build the tests into nextest archives instead of running them.
archive: Option<Archive>,
/// `ci tests --from <dir>`: run the tests from a nextest archive extracted to this directory.
from: Option<PathBuf>,
/// `ci tests --from <dir> --shard <k>`: the archive is gate shard k's (1-based), and the plan it
/// carries says which part of the split binaries it runs.
shard: Option<usize>,
selection: Selection,
/// Workspace packages left out of the test build and the doctests ($PHOTOCRAFT_CI_EXCLUDE).
exclude: Vec<String>,
}
/// The archives `ci archive` writes.
#[derive(Clone, Debug, PartialEq)]
enum Archive {
/// One archive of every test binary ($NEXTEST_ARCHIVE_FILE).
File(PathBuf),
/// One archive per gate shard, at `<dir>/<k>/<name>`, each with only the binaries that shard
/// runs (xtask/src/shards.rs).
Shards { dir: PathBuf, name: String, shards: usize, gpu_shard: bool },
}
type StepFn = fn(&CiOptions) -> Result<(), String>;
/// The steps a `cargo xtask ci` call runs, in order, by name.
type CiPlan = Vec<(&'static str, StepFn)>;
@@ -387,7 +405,16 @@ fn read_gpu_filter() -> Result<String, String> {
/// hunt (AGENTS.md Rule 9), with the same build arguments as the tests, so nothing rebuilds. `gpu` is
/// the group's filter, read only when the selection needs it. nextest's `ci` profile terminates
/// hung tests and repeats every failure at the end of the log (.config/nextest.toml).
fn nextest_args(o: &CiOptions, panic_hunt: bool, gpu: Option<&str>) -> Vec<String> {
/// What `ci tests --shard` runs, from the plan the shard's archive carries.
struct ShardRun {
/// The shard's part of the split binaries.
filter: Option<String>,
/// The plan gave the shard no tests.
empty: bool,
}
/// `shard`: the shard's part of the plan (`ci tests --shard`).
fn nextest_args(o: &CiOptions, panic_hunt: bool, gpu: Option<&str>, shard: Option<&ShardRun>) -> Vec<String> {
let mut args: Vec<String> = ["nextest", "run", "--profile", "ci"].map(String::from).to_vec();
let mut filters = Vec::new();
if panic_hunt {
@@ -413,8 +440,9 @@ fn nextest_args(o: &CiOptions, panic_hunt: bool, gpu: Option<&str>) -> Vec<Strin
if let Some(partition) = &o.partition {
args.extend(["--partition".to_string(), partition.clone()]);
}
// A partition or a selection may hold no panic hunt; the shard that does runs it.
if panic_hunt && (o.partition.is_some() || o.selection != Selection::All) {
// A partition, a shard or a selection may hold no panic hunt; the shard that does runs it. A
// shard the plan gave nothing runs nothing.
if (panic_hunt && (o.partition.is_some() || o.shard.is_some() || o.selection != Selection::All)) || shard.is_some_and(|s| s.empty) {
args.extend(["--no-tests", "warn"].map(String::from));
}
match (o.selection, gpu) {
@@ -425,6 +453,9 @@ fn nextest_args(o: &CiOptions, panic_hunt: bool, gpu: Option<&str>) -> Vec<Strin
(Selection::NoGpu, Some(gpu)) => filters.push(format!("not ({gpu})")),
_ => {}
}
if let Some(filter) = shard.and_then(|s| s.filter.as_ref()) {
filters.push(format!("({filter})"));
}
if !filters.is_empty() {
args.extend(["-E".to_string(), filters.join(" & ")]);
}
@@ -449,7 +480,11 @@ fn step_tests(o: &CiOptions, panic_hunt: bool) -> Result<(), String> {
}
} else {
let gpu = if o.selection == Selection::All { None } else { Some(read_gpu_filter()?) };
c.args(nextest_args(o, panic_hunt, gpu.as_deref()));
let shard = match (&o.from, o.shard) {
(Some(from), Some(k)) => read_shard_filter(from, k)?,
_ => None,
};
c.args(nextest_args(o, panic_hunt, gpu.as_deref(), shard.as_ref()));
}
let what = shown(&c);
run(c, &what)
@@ -460,9 +495,10 @@ fn step_tests(o: &CiOptions, panic_hunt: bool) -> Result<(), String> {
fn build_tests(o: &CiOptions) -> Result<(), String> {
let mut c = cargo();
match &o.archive {
Some(file) => {
Some(Archive::File(file)) => {
c.args(["nextest", "archive", "--profile", "ci"]).args(test_build_args(&o.exclude)).arg("--archive-file").arg(file);
}
Some(Archive::Shards { dir, name, shards, gpu_shard }) => return build_shard_archives(o, dir, name, *shards, *gpu_shard),
None => {
c.args(["test", "--no-run"]).args(test_build_args(&o.exclude));
}
@@ -471,6 +507,76 @@ fn build_tests(o: &CiOptions) -> Result<(), String> {
run(c, &what)
}
/// `cargo nextest list --profile ci --message-format json` of the test build, with `args`.
fn nextest_list(o: &CiOptions, args: &[&str]) -> Result<serde_json::Value, String> {
let mut c = cargo();
c.args(["nextest", "list", "--profile", "ci", "--message-format", "json"]).args(test_build_args(&o.exclude)).args(args);
let what = shown(&c);
eprintln!("$ {what}");
let out = c.stderr(Stdio::inherit()).output().map_err(|e| format!("{what}: failed to spawn: {e}"))?;
if !out.status.success() {
return Err(format!("{what}: exited with {}", out.status));
}
serde_json::from_slice(&out.stdout).map_err(|e| format!("{what}: bad JSON: {e}"))
}
/// The gate's build job with per-shard archives: builds the tests, lists them, plans which binaries
/// (and which parts of the split ones) each shard runs, writes the plan into the target directory
/// (every archive carries it, .config/nextest.toml) and writes each shard's archive.
fn build_shard_archives(o: &CiOptions, dir: &Path, name: &str, shards: usize, gpu_shard: bool) -> Result<(), String> {
let os = if cfg!(windows) { "windows" } else { "linux" };
let times_path = root().join(shards::TIMES_FILE);
let text = std::fs::read_to_string(&times_path).unwrap_or_else(|e| {
eprintln!("warning: {}: {e}; every binary and test gets the default weight", times_path.display());
String::new()
});
let times = shards::Times::parse(&text, os)?;
// The build itself, and every binary it made.
let built = nextest_list(o, &["--list-type", "binaries-only"])?;
let ids: Vec<String> = built.get("rust-binaries").and_then(serde_json::Value::as_object).ok_or("nextest list: no rust-binaries")?.keys().cloned().collect();
// With a GPU shard every binary is listed against the gpu group; otherwise only the split ones.
let filter = if gpu_shard {
read_gpu_filter()?
} else {
let split: Vec<String> = shards::binaries_to_list(&ids, &times, shards).iter().map(|b| format!("binary_id(={b})")).collect();
if split.is_empty() { "none()".to_string() } else { split.join(" | ") }
};
let listing = nextest_list(o, &["--run-ignored", "all", "-E", &filter])?;
let (binaries, target) = shards::parse_listing(&listing, gpu_shard)?;
let plan = shards::plan(&binaries, &times, shards, gpu_shard)?;
eprint!("{}", plan.summary());
let plan_path = target.join(shards::PLAN_FILE);
let json = serde_json::to_string_pretty(&plan).map_err(|e| format!("shard plan: {e}"))?;
std::fs::write(&plan_path, json).map_err(|e| format!("{}: {e}", plan_path.display()))?;
for (k, shard) in plan.shards.iter().enumerate() {
let file = dir.join((k + 1).to_string()).join(name);
std::fs::create_dir_all(dir.join((k + 1).to_string())).map_err(|e| format!("{}: {e}", dir.display()))?;
let mut c = cargo();
c.args(["nextest", "archive", "--profile", "ci"])
.args(test_build_args(&o.exclude))
.arg("-E")
.arg(shard.archive_filter())
.arg("--archive-file")
.arg(&file);
run(c, &format!("cargo nextest archive (shard {}: {} binaries) --archive-file {}", k + 1, shard.binaries.len(), file.display()))?;
}
Ok(())
}
/// Shard `k`'s part of the plan its archive carries, extracted to `from`.
fn read_shard_filter(from: &Path, k: usize) -> Result<Option<ShardRun>, String> {
let path = from.join("target").join(shards::PLAN_FILE);
let text = std::fs::read_to_string(&path).map_err(|e| format!("{}: {e}", path.display()))?;
let plan: shards::Plan = serde_json::from_str(&text).map_err(|e| format!("{}: {e}", path.display()))?;
let shard = plan.shards.get(k.wrapping_sub(1)).ok_or_else(|| format!("{}: no shard {k}", path.display()))?;
eprintln!(
"shard {k}: {} test binaries{}",
shard.binaries.len(),
if shard.parts.is_empty() { String::new() } else { format!(", part of {}", shard.parts.keys().cloned().collect::<Vec<_>>().join(", ")) }
);
Ok(Some(ShardRun { filter: shard.filter(), empty: shard.binaries.is_empty() }))
}
/// nextest does not run doctests.
fn cmd_doctest() -> Result<(), String> {
let mut c = cargo();
@@ -493,7 +599,8 @@ fn ci_exclude() -> Vec<String> {
/// The steps `cargo xtask ci <args>` runs, and how: no argument runs every step; a part (`lint`,
/// `build`, `tests`) runs the steps of one gate job; a step's name runs that step.
fn ci_from_args(args: &[&str], exclude: Vec<String>) -> Result<(CiPlan, CiOptions), String> {
let mut options = CiOptions { build_only: false, partition: None, archive: None, from: None, selection: Selection::All, exclude };
let mut options = CiOptions { build_only: false, partition: None, archive: None, from: None, shard: None, selection: Selection::All, exclude };
let mut gpu_shard = false;
let mut what = None;
let mut it = args.iter();
while let Some(&arg) = it.next() {
@@ -502,14 +609,25 @@ fn ci_from_args(args: &[&str], exclude: Vec<String>) -> Result<(CiPlan, CiOption
let dir = it.next().ok_or("--from needs the directory the archive was extracted to")?;
options.from = Some(std::path::absolute(dir).map_err(|e| format!("--from {dir}: {e}"))?);
}
"--shard" => {
let k = it.next().ok_or("--shard needs the shard's number")?;
options.shard = Some(k.parse().ok().filter(|&k| k > 0).ok_or_else(|| format!("--shard {k}: not a shard number"))?);
}
"--gpu-shard" => gpu_shard = true,
"--gpu" => options.selection = Selection::Gpu,
"--no-gpu" => options.selection = Selection::NoGpu,
_ if what.is_none() && !arg.starts_with('-') => what = Some(arg),
_ => return Err(format!("ci: unexpected argument `{arg}`")),
}
}
if (options.selection != Selection::All || options.from.is_some()) && what != Some("tests") {
return Err("ci: --from, --gpu and --no-gpu go with `ci tests`".into());
if (options.selection != Selection::All || options.from.is_some() || options.shard.is_some()) && what != Some("tests") {
return Err("ci: --from, --shard, --gpu and --no-gpu go with `ci tests`".into());
}
if options.shard.is_some() && options.from.is_none() {
return Err("ci: --shard runs a shard's archive: give its directory with --from".into());
}
if gpu_shard && what != Some("archive") {
return Err("ci: --gpu-shard goes with `ci archive`".into());
}
let parts: &[Part] = match what {
None => &[Part::Lint, Part::Test, Part::Doc],
@@ -521,13 +639,15 @@ fn ci_from_args(args: &[&str], exclude: Vec<String>) -> Result<(CiPlan, CiOption
// The gate's build job: the tests into the archive at $NEXTEST_ARCHIVE_FILE, which also
// carries this xtask (.config/nextest.toml).
Some("archive") => {
let file = std::env::var_os("NEXTEST_ARCHIVE_FILE").ok_or("ci archive: set NEXTEST_ARCHIVE_FILE to the archive to write")?;
options.archive = Some(file.into());
options.archive = Some(archive_from_env(gpu_shard, |k| std::env::var(k).ok())?);
options.build_only = true;
&[Part::Test]
}
Some("tests") => {
options.partition = tests_partition(options.selection, std::env::var("NEXTEST_PARTITION").ok());
// A shard's archive holds only its binaries, and its plan says the rest.
if options.shard.is_none() {
options.partition = tests_partition(options.selection, std::env::var("NEXTEST_PARTITION").ok());
}
&[Part::Test]
}
Some(step) => {
@@ -541,6 +661,22 @@ fn ci_from_args(args: &[&str], exclude: Vec<String>) -> Result<(CiPlan, CiOption
Ok((ci_plan(parts, options.build_only), options))
}
/// The archives `ci archive` writes: one per gate shard where the gate's build job sets
/// $NEXTEST_ARCHIVE_DIR (with $NEXTEST_ARCHIVE_NAME and $GATE_SHARDS), else one at
/// $NEXTEST_ARCHIVE_FILE.
fn archive_from_env(gpu_shard: bool, var: impl Fn(&str) -> Option<String>) -> Result<Archive, String> {
let Some(dir) = var("NEXTEST_ARCHIVE_DIR").filter(|d| !d.is_empty()) else {
if gpu_shard {
return Err("ci archive --gpu-shard writes per-shard archives: set NEXTEST_ARCHIVE_DIR".into());
}
let file = var("NEXTEST_ARCHIVE_FILE").ok_or("ci archive: set NEXTEST_ARCHIVE_FILE to the archive to write")?;
return Ok(Archive::File(file.into()));
};
let name = var("NEXTEST_ARCHIVE_NAME").ok_or("ci archive: NEXTEST_ARCHIVE_DIR needs NEXTEST_ARCHIVE_NAME")?;
let shards = var("GATE_SHARDS").and_then(|n| n.parse().ok()).filter(|&n| n > 0).ok_or("ci archive: NEXTEST_ARCHIVE_DIR needs GATE_SHARDS")?;
Ok(Archive::Shards { dir: dir.into(), name, shards, gpu_shard })
}
/// The partition `ci tests` runs. The gate sets $NEXTEST_PARTITION on every shard leg, and the shard
/// that runs the gpu group must still run all of it.
fn tests_partition(selection: Selection, env: Option<String>) -> Option<String> {
@@ -619,23 +755,50 @@ mod ci_tests {
build_only: false,
archive: None,
from: None,
shard: None,
partition: Some("hash:2/3".into()),
selection: Selection::NoGpu,
exclude: vec!["photocraft-web".into()],
};
let build = "--workspace --locked --features heif --exclude photocraft-web --lib --bins --tests";
let args = nextest_args(&options, false, Some("package(g)")).join(" ");
let args = nextest_args(&options, false, Some("package(g)"), None).join(" ");
assert_eq!(args, format!("nextest run --profile ci {build} --partition hash:2/3 -E not (package(g))"));
let gpu = CiOptions { selection: Selection::Gpu, partition: None, ..options.clone() };
let hunt = nextest_args(&gpu, true, Some("package(g)")).join(" ");
let hunt = nextest_args(&gpu, true, Some("package(g)"), None).join(" ");
assert!(hunt.contains("--run-ignored only") && hunt.contains("--no-tests warn") && hunt.contains("--success-output immediate"), "{hunt}");
assert!(hunt.ends_with("-E binary_id(photocraft-engine::panic_hunt) & (package(g))"), "{hunt}");
// The whole run keeps nextest's default: a panic hunt that matches nothing fails.
let here = CiOptions { partition: None, selection: Selection::All, ..options };
assert!(!nextest_args(&here, true, None).join(" ").contains("--no-tests"));
assert!(!nextest_args(&here, true, None, None).join(" ").contains("--no-tests"));
assert_eq!(test_build_args(&here.exclude).join(" "), build);
}
#[test]
fn a_shard_archive_runs_its_part_and_no_partition() {
let (_, options) = ci_from_args(&["tests", "--from", "x", "--shard", "2", "--no-gpu"], Vec::new()).unwrap();
assert_eq!((options.shard, options.partition.as_deref()), (Some(2), None));
let run = ShardRun { filter: Some("not (binary_id(=ui))".into()), empty: false };
let args = nextest_args(&options, true, Some("package(g)"), Some(&run)).join(" ");
assert!(args.contains("--no-tests warn") && !args.contains("--partition"), "{args}");
assert!(args.ends_with("-E binary_id(photocraft-engine::panic_hunt) & not (package(g)) & (not (binary_id(=ui)))"), "{args}");
let tests = nextest_args(&options, false, None, Some(&run)).join(" ");
assert!(!tests.contains("--no-tests"), "{tests}");
let empty = nextest_args(&options, false, None, Some(&ShardRun { filter: None, empty: true })).join(" ");
assert!(empty.contains("--no-tests warn"), "{empty}");
assert!(ci_from_args(&["tests", "--shard", "2"], Vec::new()).is_err(), "--shard without --from");
assert!(ci_from_args(&["tests", "--from", "x", "--shard", "0"], Vec::new()).is_err());
assert!(ci_from_args(&["lint", "--gpu-shard"], Vec::new()).is_err());
fn env(vars: &'static [(&'static str, &'static str)]) -> impl Fn(&str) -> Option<String> {
move |k| vars.iter().find(|v| v.0 == k).map(|v| v.1.to_string())
}
let shards = &[("NEXTEST_ARCHIVE_DIR", "/a"), ("NEXTEST_ARCHIVE_NAME", "n.tar.zst"), ("GATE_SHARDS", "4"), ("NEXTEST_ARCHIVE_FILE", "/f")];
assert_eq!(archive_from_env(true, env(shards)).unwrap(), Archive::Shards { dir: "/a".into(), name: "n.tar.zst".into(), shards: 4, gpu_shard: true });
assert_eq!(archive_from_env(false, env(&[("NEXTEST_ARCHIVE_FILE", "/f")])).unwrap(), Archive::File("/f".into()));
assert!(archive_from_env(true, env(&[("NEXTEST_ARCHIVE_FILE", "/f")])).is_err());
assert!(archive_from_env(false, env(&[("NEXTEST_ARCHIVE_DIR", "/a"), ("NEXTEST_ARCHIVE_NAME", "n")])).is_err());
}
#[test]
fn the_gpu_shard_runs_the_whole_gpu_group_whatever_the_partition() {
let env = || Some("hash:1/4".to_string());
+531
View File
@@ -0,0 +1,531 @@
//! The gate's per-shard archives (`cargo xtask ci archive` with $NEXTEST_ARCHIVE_DIR): which test
//! binaries each shard runs, balanced by the recorded test times in .config/ci-test-times.tsv.
//!
//! Each shard downloads an archive that holds only its own binaries. A binary heavier than half a
//! shard's share is split by test filter along its module tree, and its executable goes into the
//! archive of every shard that runs part of it. With a GPU shard (the Linux gate), shard 1 takes
//! every binary that holds a test of nextest's `gpu` group and runs that group alone; the other
//! shards run every other test.
//!
//! The times only decide the balance. Which binaries and tests exist comes from the build's own
//! nextest listing, so a missing or stale time never drops or repeats a test.
use std::collections::{BTreeMap, BTreeSet};
use std::path::PathBuf;
use serde::{Deserialize, Serialize};
/// Where the recorded times live, relative to the workspace root (scripts/ci-test-times.py writes it).
pub const TIMES_FILE: &str = ".config/ci-test-times.tsv";
/// The plan, written into the target directory, which every shard's archive carries
/// (.config/nextest.toml `archive.include`).
pub const PLAN_FILE: &str = "gate-shards.json";
/// A binary's weight when the times file has no binaries at all.
const DEFAULT_BINARY_SECONDS: f64 = 10.0;
/// A test's weight when the times file knows nothing about its binary.
const DEFAULT_TEST_SECONDS: f64 = 0.3;
/// A split binary's tests are cut into parts of at most a shard's share divided by this, so the
/// parts fill the shards evenly while the filterset stays short.
const PARTS_PER_SHARD: f64 = 8.0;
/// Recorded test times of one OS.
#[derive(Debug, Default)]
pub struct Times {
/// Binary id -> (test count, total seconds).
binaries: BTreeMap<String, (u64, f64)>,
/// Binary id -> test name -> seconds, for the tests the file lists one by one.
tests: BTreeMap<String, BTreeMap<String, f64>>,
/// Binary id -> the mean time of its tests the file does not list one by one.
small: BTreeMap<String, f64>,
/// The median binary total: the weight of a binary the file does not know.
default_binary: f64,
}
impl Times {
/// The rows of `os`, tab-separated, `#` lines are comments:
/// `bin <os> <binary-id> <test count> <total seconds>` and
/// `test <os> <binary-id> <seconds> <test name>`.
pub fn parse(text: &str, os: &str) -> Result<Times, String> {
let mut times = Times::default();
for (n, line) in text.lines().enumerate() {
if line.trim().is_empty() || line.starts_with('#') {
continue;
}
let bad = || format!("{TIMES_FILE}:{}: bad row `{line}`", n + 1);
let cols: Vec<&str> = line.split('\t').collect();
if cols.len() != 5 {
return Err(bad());
}
if cols[1] != os {
continue;
}
match cols[0] {
"bin" => {
let count = cols[3].parse().map_err(|_| bad())?;
let total = cols[4].parse().map_err(|_| bad())?;
times.binaries.insert(cols[2].to_string(), (count, total));
}
"test" => {
let seconds = cols[3].parse().map_err(|_| bad())?;
times.tests.entry(cols[2].to_string()).or_default().insert(cols[4].to_string(), seconds);
}
_ => return Err(bad()),
}
}
for (id, &(count, total)) in &times.binaries {
let listed = times.tests.get(id);
let n = listed.map_or(0, BTreeMap::len) as u64;
let sum: f64 = listed.map_or(0.0, |t| t.values().sum());
if count > n {
times.small.insert(id.clone(), ((total - sum) / (count - n) as f64).max(0.0));
}
}
let mut totals: Vec<f64> = times.binaries.values().map(|b| b.1).collect();
totals.sort_by(f64::total_cmp);
times.default_binary = totals.get(totals.len() / 2).copied().unwrap_or(DEFAULT_BINARY_SECONDS);
Ok(times)
}
fn binary(&self, id: &str) -> f64 {
self.binaries.get(id).map_or(self.default_binary, |b| b.1)
}
fn test(&self, binary: &str, name: &str) -> f64 {
if let Some(&t) = self.tests.get(binary).and_then(|t| t.get(name)) {
return t;
}
self.small.get(binary).copied().unwrap_or(DEFAULT_TEST_SECONDS)
}
}
/// A test binary of the build, from nextest's listing.
#[derive(Debug, Clone)]
pub struct Binary {
pub id: String,
/// `None` when the listing skipped the binary (only split candidates are listed without a GPU
/// shard).
pub tests: Option<Vec<Test>>,
}
#[derive(Debug, Clone)]
pub struct Test {
pub name: String,
/// In nextest's `gpu` test group (only known when the listing's filterset was that group).
pub gpu: bool,
}
/// The binaries of `cargo nextest list --message-format json --run-ignored all`, and the target
/// directory. With `gpu_listing`, the listing's filterset was the `gpu` group, so a test that
/// matches it is a GPU test.
pub fn parse_listing(json: &serde_json::Value, gpu_listing: bool) -> Result<(Vec<Binary>, PathBuf), String> {
let target =
json.pointer("/rust-build-meta/target-directory").and_then(serde_json::Value::as_str).ok_or("nextest list: no rust-build-meta.target-directory")?;
let suites = json.get("rust-suites").and_then(serde_json::Value::as_object).ok_or("nextest list: no rust-suites")?;
let mut binaries = Vec::new();
for (id, suite) in suites {
let tests = match suite.get("status").and_then(serde_json::Value::as_str) {
Some("listed") => {
let cases = suite.get("testcases").and_then(serde_json::Value::as_object).ok_or_else(|| format!("nextest list: {id} has no testcases"))?;
let tests = cases
.iter()
.map(|(name, case)| {
let matches = case.pointer("/filter-match/status").and_then(serde_json::Value::as_str) == Some("matches");
Test { name: name.clone(), gpu: gpu_listing && matches }
})
.collect();
Some(tests)
}
Some("skipped") => None,
other => return Err(format!("nextest list: {id} has status {other:?}")),
};
binaries.push(Binary { id: id.clone(), tests });
}
Ok((binaries, PathBuf::from(target)))
}
/// A share of a split binary's tests.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum Part {
/// Every test (the binary turned out light enough to stay whole).
All,
/// Every test under this module path.
Module(String),
/// One test.
Test(String),
}
impl Part {
fn filter(&self) -> String {
match self {
Part::All => "all()".to_string(),
Part::Module(path) => format!("test(/^{}::/)", regex_escape(path)),
Part::Test(name) => format!("test(={name})"),
}
}
fn matches(&self, name: &str) -> bool {
match self {
Part::All => true,
Part::Module(path) => name.strip_prefix(path.as_str()).is_some_and(|rest| rest.starts_with("::")),
Part::Test(test) => name == test,
}
}
}
fn regex_escape(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for c in s.chars() {
if !(c.is_alphanumeric() || c == '_' || c == ':') {
out.push('\\');
}
out.push(c);
}
out
}
/// What one shard runs.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct Shard {
/// The binaries in the shard's archive.
pub binaries: BTreeSet<String>,
/// The split binaries among them, with the parts of their tests this shard runs.
pub parts: BTreeMap<String, Vec<Part>>,
/// Estimated test-seconds, for the log.
pub seconds: f64,
}
impl Shard {
/// The filterset that keeps the split binaries to this shard's parts; `None` when the shard runs
/// whole binaries only.
pub fn filter(&self) -> Option<String> {
if self.parts.is_empty() {
return None;
}
let split: Vec<String> = self.parts.keys().map(|b| format!("binary_id(={b})")).collect();
let mut terms = vec![format!("not ({})", split.join(" | "))];
for (binary, parts) in &self.parts {
let parts: Vec<String> = parts.iter().map(Part::filter).collect();
terms.push(format!("(binary_id(={binary}) & ({}))", parts.join(" | ")));
}
Some(terms.join(" | "))
}
/// The `cargo nextest archive -E` filterset that writes this shard's archive. A shard can be
/// empty when a pull request's build leaves out most packages ($PHOTOCRAFT_CI_EXCLUDE).
pub fn archive_filter(&self) -> String {
if self.binaries.is_empty() {
return "none()".to_string();
}
let ids: Vec<String> = self.binaries.iter().map(|b| format!("binary_id(={b})")).collect();
ids.join(" | ")
}
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct Plan {
/// Shard 1 runs the `gpu` group alone.
pub gpu_shard: bool,
pub shards: Vec<Shard>,
}
impl Plan {
/// The non-GPU shards.
fn cpu_shards(&self) -> &[Shard] {
&self.shards[usize::from(self.gpu_shard)..]
}
/// One line per shard, for the build log.
pub fn summary(&self) -> String {
let mut out = String::new();
for (k, shard) in self.shards.iter().enumerate() {
let what = if self.gpu_shard && k == 0 { " (the gpu group)".to_string() } else { format!(", about {:.0} test-s", shard.seconds) };
if shard.binaries.is_empty() {
out += &format!("shard {}: no tests\n", k + 1);
continue;
}
let split: Vec<String> = shard.parts.iter().map(|(b, p)| format!("{b} ({} parts)", p.len())).collect();
let split = if split.is_empty() { String::new() } else { format!("; part of {}", split.join(", ")) };
out += &format!("shard {}: {} binaries{what}{split}\n", k + 1, shard.binaries.len());
}
out
}
}
/// The binaries a shard's share is too small for, and that share. A split binary's tests are spread
/// over the shards; every other binary stays whole.
fn split_binaries(binaries: &[Binary], times: &Times, cpu_shards: usize) -> (BTreeSet<String>, f64) {
let total: f64 = binaries.iter().map(|b| times.binary(&b.id)).sum();
let share = total / cpu_shards as f64;
let split = binaries.iter().filter(|b| times.binary(&b.id) > share / 2.0).map(|b| b.id.clone()).collect();
(split, share)
}
/// The binaries nextest has to list for the plan when there is no GPU shard: the split ones.
pub fn binaries_to_list(ids: &[String], times: &Times, shards: usize) -> BTreeSet<String> {
let binaries: Vec<Binary> = ids.iter().map(|id| Binary { id: id.clone(), tests: None }).collect();
split_binaries(&binaries, times, shards).0
}
#[derive(Default)]
struct Node {
weight: f64,
modules: BTreeMap<String, Node>,
tests: BTreeMap<String, f64>,
}
/// Cuts a binary's tests into parts along the module tree: a module at or under `limit` is one part,
/// a heavier one is cut into its submodules and its own tests. Every test is in exactly one part.
fn cut_into_parts(tests: &[(String, f64)], limit: f64) -> Vec<(Part, f64)> {
let mut root = Node::default();
for (name, weight) in tests {
let mut node = &mut root;
node.weight += weight;
let mut segments: Vec<&str> = name.split("::").collect();
let leaf = segments.pop().unwrap_or_default();
for segment in segments {
node = node.modules.entry(segment.to_string()).or_default();
node.weight += weight;
}
*node.tests.entry(leaf.to_string()).or_default() += weight;
}
let mut parts = Vec::new();
if root.weight <= limit {
parts.push((Part::All, root.weight));
} else {
cut(&root, "", limit, &mut parts);
}
parts
}
fn cut(node: &Node, prefix: &str, limit: f64, parts: &mut Vec<(Part, f64)>) {
let path = |segment: &str| if prefix.is_empty() { segment.to_string() } else { format!("{prefix}::{segment}") };
for (segment, child) in &node.modules {
if child.weight <= limit {
parts.push((Part::Module(path(segment)), child.weight));
} else {
cut(child, &path(segment), limit, parts);
}
}
for (segment, &weight) in &node.tests {
parts.push((Part::Test(path(segment)), weight));
}
}
/// Assigns the binaries (and the parts of split binaries) to `shards` shards, heaviest first onto
/// the lightest shard. With `gpu_shard`, shard 1 takes every binary with a GPU test.
pub fn plan(binaries: &[Binary], times: &Times, shards: usize, gpu_shard: bool) -> Result<Plan, String> {
let first = usize::from(gpu_shard);
if shards <= first {
return Err(format!("{shards} shard(s) leave none for the tests outside the gpu group"));
}
let cpu_shards = shards - first;
let gpu = |t: &Test| gpu_shard && t.gpu;
// Binaries whose every test is a GPU test run on the GPU shard only.
let cpu: Vec<Binary> = binaries.iter().filter(|b| b.tests.as_ref().is_none_or(|t| t.iter().any(|t| !gpu(t)))).cloned().collect();
let (split, share) = split_binaries(&cpu, times, cpu_shards);
let limit = share / PARTS_PER_SHARD;
let mut items: Vec<(f64, &str, Option<Part>)> = Vec::new();
for binary in &cpu {
// A binary the listing skipped stays whole.
if let Some(tests) = binary.tests.as_ref().filter(|_| split.contains(&binary.id)) {
let weighted: Vec<(String, f64)> = tests.iter().filter(|t| !gpu(t)).map(|t| (t.name.clone(), times.test(&binary.id, &t.name))).collect();
for (part, weight) in cut_into_parts(&weighted, limit) {
items.push((weight, &binary.id, Some(part)));
}
} else {
items.push((times.binary(&binary.id), &binary.id, None));
}
}
items.sort_by(|a, b| b.0.total_cmp(&a.0).then_with(|| (a.1, &a.2).cmp(&(b.1, &b.2))));
let mut plan = Plan { gpu_shard, shards: vec![Shard::default(); shards] };
for (weight, binary, part) in items {
let lightest = (first..shards).min_by(|&a, &b| plan.shards[a].seconds.total_cmp(&plan.shards[b].seconds)).unwrap_or(first);
let shard = &mut plan.shards[lightest];
shard.seconds += weight;
shard.binaries.insert(binary.to_string());
if let Some(part) = part {
shard.parts.entry(binary.to_string()).or_default().push(part);
}
}
if gpu_shard {
let gpu = binaries.iter().filter(|b| b.tests.as_ref().is_some_and(|t| t.iter().any(|t| t.gpu)));
plan.shards[0].binaries = gpu.map(|b| b.id.clone()).collect();
}
for shard in &mut plan.shards {
for parts in shard.parts.values_mut() {
parts.sort();
}
}
verify(&plan, binaries)?;
Ok(plan)
}
/// Every test runs on exactly one shard: each binary with tests outside the gpu group on one shard
/// or, split, every such test in exactly one part across the shards that hold it, and with a GPU
/// shard, that shard holds exactly the binaries with GPU tests. Without a GPU shard every test
/// counts as outside the group.
pub fn verify(plan: &Plan, binaries: &[Binary]) -> Result<(), String> {
let known: BTreeSet<&str> = binaries.iter().map(|b| b.id.as_str()).collect();
for (k, shard) in plan.shards.iter().enumerate() {
if let Some(unknown) = shard.binaries.iter().find(|b| !known.contains(b.as_str())) {
return Err(format!("shard {} holds {unknown}, which the build has not", k + 1));
}
}
if plan.gpu_shard {
let gpu: BTreeSet<String> = binaries.iter().filter(|b| b.tests.as_ref().is_some_and(|t| t.iter().any(|t| t.gpu))).map(|b| b.id.clone()).collect();
if plan.shards[0].binaries != gpu {
return Err("the GPU shard does not hold exactly the binaries with GPU tests".into());
}
}
let cpu = plan.cpu_shards();
for binary in binaries {
let tests: Option<Vec<&Test>> = binary.tests.as_ref().map(|t| t.iter().filter(|t| !(plan.gpu_shard && t.gpu)).collect());
let holders: Vec<&Shard> = cpu.iter().filter(|s| s.binaries.contains(&binary.id)).collect();
if tests.as_ref().is_some_and(Vec::is_empty) {
if !holders.is_empty() {
return Err(format!("{} has only GPU tests but is on a CPU shard", binary.id));
}
continue;
}
let split = holders.iter().filter(|s| s.parts.contains_key(&binary.id)).count();
if split == 0 {
if holders.len() != 1 {
return Err(format!("{} is on {} shards", binary.id, holders.len()));
}
continue;
}
if split != holders.len() {
return Err(format!("{} is split, but a shard holds it whole", binary.id));
}
let tests = tests.ok_or_else(|| format!("{} is split, but the listing skipped it", binary.id))?;
for test in tests {
let runs: usize = holders.iter().map(|s| s.parts[&binary.id].iter().filter(|p| p.matches(&test.name)).count()).sum();
if runs != 1 {
return Err(format!("{} {} would run {runs} times", binary.id, test.name));
}
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn binary(id: &str, tests: &[&str]) -> Binary {
Binary { id: id.into(), tests: Some(tests.iter().map(|t| Test { name: t.to_string(), gpu: t.contains("gpu_tests::") }).collect()) }
}
const TIMES: &str = "\
# kind\tos\tbinary\t...
bin\tlinux\tui\t6\t60
test\tlinux\tui\t20\tdialogs::slow
test\tlinux\tui\t16\tdialogs::other::slower
bin\tlinux\tengine\t2\t12
bin\tlinux\talgo\t1\t8
bin\twindows\tui\t6\t600
";
fn build() -> Vec<Binary> {
vec![
binary("ui", &["dialogs::slow", "dialogs::other::slower", "dialogs::other::quick", "canvas::a", "canvas::b", "top", "canvas::gpu_tests::g"]),
binary("engine", &["e1", "e2"]),
binary("algo", &["a1"]),
binary("gpu::it", &["gpu_tests::only"]),
binary("new", &["n1"]),
]
}
#[test]
fn times_fill_in_what_the_file_does_not_list() {
let times = Times::parse(TIMES, "linux").unwrap();
assert_eq!(times.test("ui", "dialogs::slow"), 20.0);
// (60 - 36) s over the 4 tests the file does not list one by one.
assert_eq!(times.test("ui", "canvas::a"), 6.0);
assert_eq!(times.test("nowhere", "x"), DEFAULT_TEST_SECONDS);
assert_eq!(times.binary("new"), 12.0, "the median binary");
assert_eq!(Times::parse(TIMES, "windows").unwrap().binary("ui"), 600.0);
assert_eq!(Times::parse("", "linux").unwrap().binary("x"), DEFAULT_BINARY_SECONDS);
assert!(Times::parse("bin\tlinux\tui\tsix\t1", "linux").is_err());
}
#[test]
fn every_test_runs_once_and_the_heavy_binary_is_spread() {
let times = Times::parse(TIMES, "linux").unwrap();
let plan = plan(&build(), &times, 3, true).unwrap();
assert_eq!(plan.shards[0].binaries, BTreeSet::from(["gpu::it".to_string(), "ui".to_string()]));
assert!(plan.cpu_shards().iter().all(|s| !s.binaries.contains("gpu::it")));
let holders = plan.cpu_shards().iter().filter(|s| s.binaries.contains("ui")).count();
assert_eq!(holders, 2, "{}", plan.summary());
// Balanced to within the heaviest item (one 20 s test).
let loads: Vec<f64> = plan.cpu_shards().iter().map(|s| s.seconds).collect();
assert!((loads[0] - loads[1]).abs() <= 20.0, "{loads:?}");
// Tampering is caught: a part dropped, or a whole binary on two shards.
let mut broken = plan.clone();
broken.shards[1].parts.values_mut().next().unwrap().pop();
assert!(verify(&broken, &build()).is_err());
let mut twice = plan.clone();
let whole = twice.shards[1].binaries.iter().find(|b| !twice.shards[1].parts.contains_key(*b)).unwrap().clone();
twice.shards[2].binaries.insert(whole);
assert!(verify(&twice, &build()).is_err());
}
#[test]
fn without_a_gpu_shard_unlisted_binaries_stay_whole() {
let times = Times::parse(TIMES, "linux").unwrap();
let ids: Vec<String> = build().iter().map(|b| b.id.clone()).collect();
assert_eq!(binaries_to_list(&ids, &times, 2), BTreeSet::from(["ui".to_string()]));
let listing: Vec<Binary> = build().into_iter().map(|b| if b.id == "ui" { b } else { Binary { tests: None, ..b } }).collect();
let plan = plan(&listing, &times, 2, false).unwrap();
let all: BTreeSet<String> = plan.shards.iter().flat_map(|s| s.binaries.iter().cloned()).collect();
assert_eq!(all.len(), 5);
assert!(plan.shards.iter().all(|s| s.parts.get("ui").is_none_or(|p| !p.is_empty())));
// More shards than work: the extra shards are empty, and still every test runs once.
let wide = super::plan(&build(), &times, 20, false).unwrap();
assert!(wide.shards.iter().any(|s| s.binaries.is_empty()));
assert_eq!(Shard::default().archive_filter(), "none()");
}
#[test]
fn parts_follow_the_module_tree_and_render_as_filtersets() {
let tests: Vec<(String, f64)> =
[("a::b::x", 1.0), ("a::b::y", 1.0), ("a::c", 5.0), ("a::b", 0.5), ("top", 1.0)].iter().map(|(n, w)| (n.to_string(), *w)).collect();
let parts = cut_into_parts(&tests, 3.0);
let names: Vec<&Part> = parts.iter().map(|p| &p.0).collect();
assert_eq!(names, [&Part::Module("a::b".into()), &Part::Test("a::b".into()), &Part::Test("a::c".into()), &Part::Test("top".into())]);
for (name, _) in &tests {
assert_eq!(parts.iter().filter(|p| p.0.matches(name)).count(), 1, "{name}");
}
assert_eq!(cut_into_parts(&tests, 100.0)[0].0, Part::All);
let shard = Shard {
binaries: ["ui".to_string(), "e".to_string()].into(),
parts: [("ui".to_string(), vec![Part::Module("a::b".into()), Part::Test("top".into())])].into(),
seconds: 0.0,
};
assert_eq!(shard.filter().unwrap(), "not (binary_id(=ui)) | (binary_id(=ui) & (test(/^a::b::/) | test(=top)))");
assert_eq!(shard.archive_filter(), "binary_id(=e) | binary_id(=ui)");
assert_eq!(Shard::default().filter(), None);
}
#[test]
fn the_listing_marks_gpu_tests_and_skipped_binaries() {
let json: serde_json::Value = serde_json::from_str(
r#"{"rust-build-meta": {"target-directory": "/t"}, "rust-suites": {
"ui": {"status": "listed", "testcases": {
"a::gpu_tests::g": {"filter-match": {"status": "matches"}},
"a::t": {"filter-match": {"status": "mismatch", "reason": "expression"}}}},
"engine": {"status": "skipped", "testcases": {}}}}"#,
)
.unwrap();
let (binaries, target) = parse_listing(&json, true).unwrap();
assert_eq!(target, PathBuf::from("/t"));
let ui = binaries.iter().find(|b| b.id == "ui").unwrap().tests.as_ref().unwrap();
assert_eq!(ui.iter().filter(|t| t.gpu).map(|t| t.name.as_str()).collect::<Vec<_>>(), ["a::gpu_tests::g"]);
assert!(binaries.iter().find(|b| b.id == "engine").unwrap().tests.is_none());
assert!(parse_listing(&json, false).unwrap().0.iter().flat_map(|b| b.tests.iter().flatten()).all(|t| !t.gpu));
}
}