Skip to content

ci: run the suite and the gate through kindkit's workflow - #78

Merged
CameronBrooks11 merged 5 commits into
mainfrom
ci/adopt-kindkit-workflow
Oct 8, 2026
Merged

CameronBrooks11 merged 5 commits into
mainfrom
ci/adopt-kindkit-workflow

Conversation

@CameronBrooks11

@CameronBrooks11 CameronBrooks11 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Follows #77, the kindkit pin bump, which is merged. This branch has
main merged into it (no rebase); the diff against main is this change
alone.

What

rowspec CI now calls kindkit's reusable workflow, following kindkit's
README section "Adopting it where conformance is already a required
check".

  • New job kind in .github/workflows/check.yml. It calls
    kindspec/kindkit/.github/workflows/kind.yml@8e21b558c85d5612009f3a1d69a3a68bc0191637
    with:

    fixture-root:          conformance/cases
    suite:                 cd conformance && uv run python run_cases.py
    second-implementation: cd conformance && uv run python run_cases.py rowspec_alt.table
    mutants:               cd conformance && uv run python mutants.py
    

    The job grants contents: read and actions: read, which the workflow
    needs to find its own commit. It reports as kind / conformance.

  • The conformance job keeps its name. Its required check keeps
    reporting, so the existing ruleset still applies. It drops the just conform, just mutants and just conform-alt steps, which are now the
    workflow's gates. It keeps everything the workflow does not cover:
    just check, the extra-absent assertion, just test without the extra,
    and the example recipe.

  • xlsx-extra and action-selftest.yml are unchanged.

  • No rowspec gate code changed. mutants.py calls kindkit.gate(),
    and at 8e21b55 that call writes KINDKIT_GATE_REPORT itself. Its probe
    goes through probe_command, which now carries the runner's root
    through. run_cases.py is built on kindkit.cli, which reads
    KINDKIT_REPORT_JSON.

just test still runs the suite and the gate itself, through
tests/test_conformance.py, as it did before. So the conformance job
still exercises them, but under pytest rather than as gate steps.

The workflow ref and the dev pin stay in step

kindkit's commit now appears in two places: uses: in check.yml, and
the dev dependency group in pyproject.toml. The workflow fetches
kindkit's tools/ at the first. The runner and the gate those tools drive
are the package pinned by the second. If the two differ, the gates check a
report contract that the installed kit may not keep.

tests/test_pins.py::test_the_kindkit_workflow_runs_at_the_commit_the_package_is_pinned_to
finds every uses: of a kindspec/kindkit workflow and requires each one
to name the dev group's commit. It also fails if it finds no such uses:
at all, so it cannot pass vacuously. Comments in pyproject.toml,
check.yml and AGENTS.md all say to move the two together.

The owner and the repository are matched case-insensitively, because
GitHub treats KindSpec/KindKit as the same repository. The commit must
still be lowercase hex. Review found the gap: with the original
case-sensitive pattern, a second job calling
KindSpec/KindKit/.github/workflows/kind.yml@bf716e3... beside the correct
ref passed (1 passed).

The test was shown to fail. Each mutation was applied to the file,
hash-checked to confirm it changed, and then reverted. This is the sweep
after the fix:

mixed-case-second-ref-old-sha: killed        second job, KindSpec/KindKit@bf716e3...
mixed-case-second-ref-correct-sha: SURVIVED  control: KindSpec/KindKit@8e21b55..., which is correct
mixed-case-only-ref-old-sha: killed          the only ref, KindSpec/KindKit@bf716e3...
uppercase-sha: killed                        kind.yml@8E21B558...
workflow-ref-old-sha: killed                 uses: ...kind.yml@bf716e3...
workflow-ref-branch: killed                  uses: ...kind.yml@main
pyproject-pin-old: killed                    kindkit @ git+...@bf716e3...
no-kind-job: killed                          no kindkit uses: at all

CI, green and red

Green on the current head (3ce2477): run
37715625047,
with every context green, kind / conformance included. Green on the first push: run
37713862672.
Every step of every job is green, kind / conformance included. From the
kind / conformance job log (job 113105742193):

kindkit at 8e21b558c85d5612009f3a1d69a3a68bc0191637
0 invalid case(s) across 410 case(s) in 1 tree(s)
0 failure(s) across all 410 case(s) under 'conformance/cases'      # reference
0 failure(s) across all 410 case(s) under 'conformance/cases'      # rowspec_alt.table
73 killed, 0 survived, 2 equivalent, 0 stale, 0 broken
the gate killed 73 mutant(s) of '/home/runner/work/rowspec/rowspec/reference/rowspec/table.py' over all 410 case(s)

The gate figures are the same as just mutants gives on main. The
source line names reference/rowspec/table.py. The workflow does not
check which file was mutated (kindkit README), so a reviewer has to.

Red on a throwaway branch: run
37713891424.
tmp/red-demo-broken-case was this branch with one expectation broken:
eval/aggregate-over-an-exponent-name-is-a-broken-reference wanted
#REF!(1e4) instead of #REF!(1e3). Its throwaway PR was #79, which is
now closed, and the branch is deleted. Steps of kind / conformance (job
113105834937):

the case tree follows the convention                     success
the reference implementation passes every case in tree   failure
the second implementation passes every case in tree      failure
the suite must be able to fail                           success

with

  FAIL eval/aggregate-over-an-exponent-name-is-a-broken-reference  o: wanted '#REF!(1e4)', got '#REF!(1e3)'
1 failure(s) across all 410 case(s) under 'conformance/cases'

The mutation step stays green there by design. The gate counts kills
against the baseline's failing set, and that run logged baseline-fail:
for the broken case. So a wrong expectation is the suite steps' job to
catch, not the gate's. The conformance job also went red, at just test.

Required checks: one ruleset edit, immediately before merge

conformance, xlsx-extra, passes-on-a-good-tree and
fails-on-a-broken-total all still report on this branch. So the PR is
mergeable under today's ruleset. But kind / conformance would gate
nothing until it is required (kindspec/.github AGENTS.md §3.2: a new
workflow is not required by default).

The owner's edit: immediately before merging this PR, and not in it,
add kind / conformance (integration 15368, GitHub Actions) to the
required status checks of ruleset 22723542. Keep all four existing
contexts. This is an addition, not a swap: the conformance job still
exists and still reports. So no context goes unreported on this branch,
before or after the edit.

gh api repos/kindspec/rowspec/rulesets/22723542 > rs.json
jq '(.rules[] | select(.type=="required_status_checks") | .parameters.required_status_checks)
      += [{"context":"kind / conformance","integration_id":15368}]
    | {name, target, enforcement, conditions, bypass_actors, rules}' rs.json > rs-new.json
diff <(jq -S '{name,target,enforcement,conditions,bypass_actors,rules}' rs.json) <(jq -S . rs-new.json)   # one added context, nothing else
gh api -X PUT repos/kindspec/rowspec/rulesets/22723542 --input rs-new.json

The diff line was run against the live ruleset, without the PUT. It
shows exactly the one added context. After the edit, any other open branch
that does not have this change sits at Expected for kind / conformance
until it takes it, and strict makes it do that anyway.

Order: #77 is merged and this branch is up to date with main. The
owner adds the required check immediately before merging this PR, then
merges it.

Not changed here

Refs kindspec/kindkit#4
Refs kindspec/kindkit#3

kindkit's reusable conformance workflow reads the runner report's
`root` key, KINDKIT_REPORT_JSON and KINDKIT_GATE_REPORT, none of which
exist at bf716e3 (v0.1.0). kindkit has no newer tag, so the pin moves
to the commit on its main that added the workflow.

No rowspec code changes. Re-measured: check, conform, conform-alt and
mutants print identical output; test differs only in wall time.

Refs kindspec/kindkit#4
Refs kindspec/kindkit#3
A new `kind` job calls kindkit's reusable kind.yml at 8e21b55, the
commit the dev group pins. It validates the case tree, runs the suite
on both implementations against conformance/cases, and judges the
mutation gate by its own report. It reports as `kind / conformance`.

The `conformance` job drops `just conform`, `just conform-alt` and
`just mutants`, and keeps lint, `just test` without the extra, and the
example recipe. Its name, and so its required check, is unchanged.

tests/test_pins.py now fails if the workflow's kindkit commit and the
package pin differ.

Refs kindspec/kindkit#4
Refs kindspec/kindkit#3
@CameronBrooks11
CameronBrooks11 marked this pull request as ready for review October 8, 2026 01:42
GitHub owner and repository names are case-insensitive, so a stale
ref spelled KindSpec/KindKit called the same workflow but was never
compared with the pin, and the test passed beside a correct ref.
@CameronBrooks11
CameronBrooks11 merged commit ed0844b into main Oct 8, 2026
8 checks passed
@CameronBrooks11
CameronBrooks11 deleted the ci/adopt-kindkit-workflow branch October 8, 2026 02:03
CameronBrooks11 added a commit that referenced this pull request Oct 8, 2026
## What

Prepares the 0.3.0 release. No code changes.

- The version is `0.3.0` in `pyproject.toml`, in
`reference/rowspec/__init__.py`'s `__version__`, and in the action's
default install pin (`rowspec==0.3.0`). `uv.lock` changes only rowspec's
own version line.
- The README's action example points at `@v0.3.0`.
- The changelog's unreleased section becomes `[0.3.0] — 2026-10-08`,
with compare links and an empty `[Unreleased]` above it.
- The ROADMAP's "Where it is" figures are re-measured (below), and a
short "Shipped in 0.3.0" section sits above "Shipped in 0.2.0".

**Not tagged.** The tag is the owner's call after review.

## Why 0.3.0

Under SemVer's pre-1.0 convention a minor bump carries behaviour
changes, and this release has three that a consumer can observe:

- **`rowspec check` output changes on CSV input**: a lone `CR` now
produces a warning, and `--strict` refuses it (#59). A pipeline running
`--strict` on a file with a lone `CR` goes from green to red.
- **The action has a new output**, `eval-outcome` (#72).
- **`conformance/run_cases.py` exits 2, not 1**, when the implementation
will not import (#62). Anyone scripting against the runner's exit code
sees a different value.

A patch release (0.2.1) would claim no observable behaviour changed,
which is not true here.

## The changelog against `git log v0.2.0..origin/main`

All 19 commits in `git log --oneline v0.2.0..origin/main` were checked
against the section. The existing entries covered #52/#72, #68, #78,
#77, #64, #60, #76 and #57. I added these, which consumers can see but
the section did not mention:

- **Added**: the action's `eval-outcome` output (#72). It came in with
the post-release workflow and was missing from the changelog.
- **Changed**: the runner's exit 2 on an unimportable implementation
(#62); the action's `setup-python` pinned to a commit (#71); the release
gate running `just test`/`just test-xlsx` and checking `__version__`
(#69, #70); SPDX headers and the new `tests/` and
`conformance/reserved/` `LICENSE` files (#74); the `reference/` boundary
check seeing dynamic imports (#65).
- **Licensing (#74), stated in the changelog entry**:
`conformance/reserved/` (was MIT, from `conformance/LICENSE`) and
`docs/ci/rowspec-check.yml` (was CC-BY-4.0) are now CC0-1.0. The root
`LICENSE` now names Apache-2.0 OR MIT for `action.yml`,
`.github/workflows/` and the `justfile`, which it did not list before.
Both relicensings move to more permissive terms. `git diff 228b61e~1
228b61e -- LICENSE docs/LICENSE` shows all three.

The section's opening note says what is left out. #58 and #75 are
documentation-only, and the note covers them as 0.2.0's did. #66, #67
and #73 are dev and CI changes with no effect outside this repository,
and the note now names them.

I checked the new claims before writing them:
- `cd conformance && uv run python run_cases.py no_such_impl` prints the
`HARD FAILURE:` line and exits `2`.
- `grep -rhn 'uses:' .github/workflows action.yml | grep -vE
'@[0-9a-f]{40}'` matches only local `./` refs and comments.

## Re-measured figures

Run on this branch, in the order listed. Each ROADMAP figure now has the
command that produced it beside it.

| figure | command |
| --- | --- |
| `All checks passed!` | `just check` |
| 410 conformance cases | `find conformance/cases -name expect.json \|
wc -l` |
| `0 failure(s) across 410 case(s)` | `just conform` |
| `0 failure(s) across 410 case(s)` (second impl) | `just conform-alt` |
| `73 killed, 0 survived, 2 equivalent, 0 stale, 0 broken` | `just
mutants` |
| 75 distinct mutations | `just mutants \| grep -ciE '^
(killed\|equiv\|survived\|stale\|broken\|bogus) '` |
| `73 passed, 1 skipped` | `just test` |
| `9 passed` | `just test-xlsx` |

Two ROADMAP rows changed:
- `just test` went from 52 passed to 73 passed. The new tests are
`test_pins.py`, `test_licensing.py` and `test_tooling.py`, plus
additions to existing files since v0.2.0.
- The "75 distinct mutations" row's right-hand column was prose ("the
gate refuses two that mutate identically"). It now gives the command,
with the refusal kept as a parenthetical.

The `just test-xlsx` row is new.

The count command matches case-insensitively and includes `bogus`,
because kindkit prints the failing verdicts in uppercase (`SURVIVED`,
`STALE`, `BROKEN`, `BOGUS`). It was checked against a sample of all six
per-mutant line shapes and kindkit's summary lines:
```
new regex, 6 verdict lines:  6
new regex, 5 summary lines: 0
old regex, 6 verdict lines:  2
```
It still gives `75` on this tree.

## Dry run of `release.yml`'s build job

I built from `git archive HEAD` of this commit, unpacked into an empty
directory, so the build saw only tracked files. The commands are the
workflow's.

- `uv build` produced `rowspec-0.3.0.tar.gz` and
`rowspec-0.3.0-py3-none-any.whl`.
- **Tag check**, with `GITHUB_REF_NAME=v0.3.0`: `tag v0.3.0 matches
built version 0.3.0`, exit 0. With `GITHUB_REF_NAME=v0.2.0` as a
negative control: `::error::tag v0.2.0 does not match built version
0.3.0`, exit 1.
- **Action pin check**: `action.yml pins rowspec==0.3.0, matching the
release`.
- **Wheel smoke step**: I extracted its `run:` script from the YAML with
`yaml.safe_load` and ran it with `bash -e` from the build directory.
  ```
wheel version: __version__ and metadata agree on 0.3.0
(…/uvcache/archive-v0/…/site-packages/rowspec/__init__.py)
  wheel CLI: eval refuses a broken total and annotates it
  ```
The printed `rowspec.__file__` is under uv's ephemeral environment, not
the checkout. The version assertion therefore ran against the wheel.
- **METADATA**: `Version: 0.3.0`, `License-Expression: Apache-2.0 OR
MIT`, `Requires-Python: >=3.11`, `Provides-Extra: xlsx`, `Requires-Dist:
openpyxl>=3.1; extra == 'xlsx'`. Besides its `.dist-info`, the wheel
contains only `rowspec/` and `rowspec_xlsx/`.
- **A fresh venv with the wheel installed** (`uv venv` + `uv pip install
<wheel>`):
  - `rowspec eval` on a table whose total is `#REF!(/0)` exits 1.
- `rowspec check` on the same table exits 0 (`0 refused`). That is the
reason the action runs both commands.
- `rowspec eval docs/example` exits 0 (`grand = 133.5`); `rowspec check
docs/example` exits 0.
- A CSV with a lone `CR` gives a warning and exits 0 under `check`, and
is refused with exit 1 under `check --strict`.
- **A second fresh venv with `<wheel>[xlsx]`**: `rowspec_xlsx` imports
from the venv, with openpyxl 3.1.5.

## After merge

Tag `v0.3.0` on the squash commit. `release.yml` then runs the gate,
builds, publishes through the trusted publisher, and runs
`action-published.yml` against the published wheel. This is the first
release in which that job runs (#52); before now it has only run on
branches pushed to probe it. If it goes red, the tag and the upload are
already final, and the fix is a patch release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant