Audit 001 · continuous integration
A green job that ran no tests
In BerriAI/litellm at commit d947ec1f, the step that runs the unit-test shard has three separate paths that call exit 0 without pytest having executed a single test. I ran the file's own shell block to confirm it. It does.
The finding
The step is id: tests at line 1072. It reads a per-shard test-path from the build matrix, filters that list down to the paths that actually exist on disk, and then hands the survivors to pytest. Three times, it decides not to fail:
1 · The shard selection is empty lines 1087–1090
selection="${TEST_PATH}"
if [ -z "${selection// /}" ]; then
echo "shard selection is empty; nothing to run"
exit 02 · No path in the selection exists lines 1106–1109
if [ "${existing_paths}" -eq 0 ]; then
echo "No path in the selection exists (${selection}); nothing to run"
exit 0This is the one that matters most. A directory rename, a shard whose path went stale, or a typo in the matrix leaves existing_paths at zero — and the job goes green. The only trace is a ::warning:: annotation in a log nobody opens on a passing run.
3 · pytest collected zero tests lines 1132–1135
if [ "$status" -eq 5 ]; then
echo "pytest collected no tests from ${selection}; passing"
exit 0Exit code 5 is pytest's documented code for “no tests were collected”. Here it is translated into success, with the word passing.
Reproduce it in 60 seconds
The script below does not re-implement the workflow's logic. It extracts the run: block of the id: tests step from the file, executes it verbatim under bash -e, and replaces uv with a stub that reports what the real runner reported. Two scenarios, both of which should end red:
| # | Scenario | Should be | Is |
|---|---|---|---|
| A | the shard names a directory that no longer exists | red | green (rc 0) |
| B | the runner returns 5 — zero tests collected | red | green (rc 0) |
git clone --depth 1 https://github.com/BerriAI/litellm /tmp/litellm-001
cd /tmp/litellm-001 && git checkout d947ec1f7d672883afbf8b79574be9ac8a44b43d
curl -O https://alexisromtz-lab.github.io/la-auditora/portafolio/auditoria-001-litellm/reproduce.py
python3 reproduce.py .github/workflows/test-unit.yml ; echo "rc=$?"Exit code 1 means the step went green without running tests. The script returns 0 when a step fails closed, which is the control: patching the two guards to exit 1 makes it return 0. An instrument that cannot say “no finding” cannot say “finding” either.
Why the automated scanner came back indeterminate
The structural analyser I run first was pointed at the same commit and declined to give a verdict. Its output, verbatim:
commit_sha_medido d947ec1f7d672883afbf8b79574be9ac8a44b43d
inv1 UNVERIFIED
selector_dinamico true (selector_dinamico_pytest: true)
total_disco 4229 test files
alcanzables_literal 4220
huerfanos_candidatos 9
loc_total_tests 1,970,420
loc_huerfana 1,196
loc_fail_open 0
bsi_orfandad 0.0006 <- upper bound, not a verdict
bsi_permisividad 0.0
clasificacion indeterminado: inv1 UNVERIFIEDIt returned “indeterminate” for a reason, and that reason is the audit. The analyser establishes the test universe from literal evidence in the CI files — a file is “reachable” if some workflow names its path. Here it found 4,220 of 4,229 reachable that way, and then declared the invariant unverified, because it had detected a dynamic selector: the shard paths are not written in the workflow, they are assembled from the build matrix at runtime.
So it refused to count orphans it could not falsify. Correct behaviour. But it leaves the actual question untouched: what happens when that runtime selector resolves to nothing? That is not a question about static reachability, it is a question about a branch in a shell script — and a branch in a shell script can simply be read and executed.
That is what this audit did. The 9 orphan candidates stay on the record as candidates: the dynamic selector may well reach them through the matrix, and without the run log I will not call them orphans. bsi 0.0006 is an upper bound on orphaned test volume, not a cleanliness score.
What this does not prove
It does not prove this has ever happened. I did not read the GitHub Actions history for this repository. I am asserting that the code path exists and behaves as described at the pinned commit — not that a green run was wrong on any given date. Doing that needs the Actions API over a declared window, which I did not measure.
It does not claim negligence. All three paths print an annotation, and one of them prints the word passing. That looks like a deliberate decision to tolerate empty shards, not an oversight. The finding is about what the signal can distinguish, not about intent.
It covers one step of one workflow. litellm has 40 workflow files. I did not audit the other 39, and nothing here generalises to them.
No patch is proposed here. This page reports a measurement. A fix is a separate conversation with the maintainers.
What I did not find — and looked for
A report that only lists hits is not a report. These are the checks that came back clean at the same commit, each measured directly on the clone:
| Check | Result |
|---|---|
remote uses: references pinned to a full 40-hex SHA | 184 / 184 — no floating tags. 27 distinct actions, 0 mutable refs. (Plus 72 references to local composite actions, which need no pin.) |
continue-on-error: true | 4 occurrences, all on codecov/codecov-action upload steps, none on a test runner |
pytest-rerunfailures / pytest-retry enabled on shared runners | disabled at lines 1530 and 1525 (-p no:rerunfailures -p no:pytest-retry) |
| step timeouts | present on the test steps (timeout-minutes from the matrix) |
| workflow permissions | permissions: {} on the jobs checked |
184 of 184 remote actions pinned, across 40 workflow files, is better than almost every repository I have measured. The organisation that wrote this file spends real effort on its supply chain. Which is exactly why the three empty-shard exits are worth reporting: they sit in a file that is otherwise careful.
One adjacent observation, offered as context and not as a finding: the five Codecov uploads also set fail_ci_if_error: false, so coverage is uploaded but cannot fail the build either. That is a normal and defensible choice; it is listed here so nobody thinks the coverage gate is load-bearing.
Need this on your repository?
The audit is the product, and the receipt is the point. Every finding ships as a pinned commit, the blob hash of the file it lives in, the exact command that reproduces it, and a declared list of what was not measured.