Skip to content

tests/vim: finish #71 - behaviour tests for every settings area - #158

Open
harleypig wants to merge 7 commits into
masterfrom
test/finish-71
Open

harleypig wants to merge 7 commits into
masterfrom
test/finish-71

Conversation

@harleypig

@harleypig harleypig commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

Finishes the behaviour tests for our own settings, area by area. Each
commit covers one area of #71's checklist. Tests only: no settings file
changes, and no test found a bug.

Closes #71

Areas: which tests cover each

  • Options (plugin/set.vim, plugin/xdg-settings.vim):
    set.bats (every section of set.vim, read after full startup),
    old-files.bats (new: the 90-day and 180-day cleanup, everything else
    left alone, the backup and viminfo options), xdg.bats, tidy.bats
  • Mappings (plugin/maps.vim, plugin/help.vim): maps.bats,
    tidy.bats, maps-rest.bats (new: <Leader>tt, Ctrl-L, <Leader>fb),
    help.bats (new: help in a right-hand vertical split, its buffer-local
    Enter and Ctrl-[), quickfix.bats (the <Leader>q* mappings)
  • Quickfix (autoload/quickfixtoggle.vim): quickfix.bats (new:
    opens and returns focus, a second call closes, a lone location list is
    not read as quickfix, :QuickfixToggle, <Leader>qf, <Leader>q] and
    <Leader>q[)
  • Display: plugin/diagnostic-colors.vim in colors.bats and
    diagnostic-colors.bats (new: the palette survives a colour scheme
    loading); plugin/gutter.vim in tidy.bats and gutter.bats (new:
    startup values, the sign toggle's branches with numbers off and
    relative-only); after/plugin/statusline.vim in statusline.bats,
    colors.bats and statusline-parts.bats (new: each branch of
    YASL_BufferInfo() and YASL_Warnings(), YASL_FileSize() at 0, 1,
    1023, 1024, 1 MiB - 1 and 1 MiB, the assembled statusline).
    plugin/colors.vim and after/diff.vim were removed in Remove inert code: tools/, checkfiletype, diff, bats, virtual text, mustache #124 (after/diff.vim is never loaded: remove, move, or note #98), so
    there is nothing of theirs to test.
  • Linting (early and late plugin/ale.vim): ale-settings.bats,
    ale-linters.bats, load.bats, ale-values.bats (new: early values
    read as ALE's plugin is about to load; the late global linters and
    fixers exactly; both still whole after every ftplugin's filetype has
    been opened, Opening a Python or bats buffer wipes json's ALE fixer setting #77)
  • Filetype detection (plugin/checkfiletype.vim): removed in
    Remove inert code: tools/, checkfiletype, diff, bats, virtual text, mustache #124 after checkfiletype.vim and the git ftplugin are inert: remove, make work, or note #97 found it inert, so there is nothing to test. The Ansible
    detection tests in load.bats cover what detects filetypes now.
  • Per-language settings (after/ftplugin/): hcl, java, python,
    sh, terraform, vim and yaml in ftplugins.bats (new); json in
    load.bats; markdown in ale-settings.bats and intellisense.bats;
    linter names in ale-linters.bats; sh's shfmt command in
    sh-ftplugin.bats; python's and yaml's other cases in tidy.bats and
    load.bats
  • Help tags (after/plugin/helptags.vim): helptags.bats

Two lines are left untested, because silent Ex mode cannot show them:

  • set.vim's t_ti= and t_te=. Vim in -es loads no termcap, so both
    read empty whatever set.vim does. set.bats's header says so.
  • help.vim's resize to at most 80 columns. In -es no window is wider
    than 80 columns, whatever columns is set to, so the resize never has
    anything to do. help.bats's header says so.

Test plan

  • bats tests/vim with a fresh, empty TMPDIR: 146 tests, all ok, and
    the directory was still empty afterwards.
  • Every new test was seen red. For each one, the line it guards in the
    settings file was changed (a value, a boundary, a branch condition, a
    mapping target, or the line deleted), the file was run, and the
    settings file was put back. Each change turned the expected test red.
    The exception is in the next section.
  • CI (bats tests/vim on Ubuntu 24.04's Vim 9.1.0016) failed on the
    first push because of two ways CI's Vim differs from the local one. It
    has no +clipboard, so set.vim's clipboard line does nothing there.
    Its own syntax/sh.vim also writes the errors a silent! hides into an
    active redirect. The last commit fixes both: the clipboard check runs
    only where exists('+clipboard'), and errors are read from the message
    history, as load.bats reads them. CI is now green.

Testing evaluation

Done by the agent that opened this PR, to the qa-tester's standard. No
team-managed: sentinel is declared.

  1. Verdict: this PR is the tests. The settings code is unchanged.
  2. Automatable: yes, all of it, in bats under tests/vim/.
  3. Falsifiers: each test was seen red against a change to the line it
    guards (see Test plan). One case does not have that falsifier:
    terraform's commentstring, foldmethod and b:ale_fixers. Deleting
    them from terraform.vim leaves the test green, because
    vim-polyglot's terraform ftplugin runs runtime! ftplugin/hcl.vim,
    which also sources our after/ftplugin/hcl.vim. The test asserts what a
    terraform buffer ends up with and says where the values come from. The
    tflint linter does come only from terraform.vim, and deleting it turns
    the test red.
  4. Disposition: written in this PR.
  5. Entry-point reach: mappings were driven through :normal on their
    keys, the toggle through its function, its command and its mapping, and
    the early ALE values through an autocmd that fires as ALE's own plugin
    file loads.

Manual verification

  1. From a checkout of this branch with submodules initialised
    (git submodule update --init) and BATS_LIB_PATH pointing at
    bats-support and bats-assert.
  2. Run bats tests/vim.
  3. Success: every line starts with ok, the last test is number 146, and
    the exit status is 0.
  4. Failure: change a value one of the new tests guards, for example
    textwidth=78 to textwidth=80 in the late plugin/set.vim. Run
    bats tests/vim/set.bats and not ok 6 wrapping: … appears.

What happened: step 2 printed 1..146, every line ok, exit status
0. Step 4 printed not ok 6 wrapping: format options, backspace over everything, 78 columns, and after set.vim was restored the file ran
green.

Review

The diff was reviewed by its author: each test was checked to read what
the settings file sets rather than what Vim or a plugin does with it
(.claude/TESTS.md), to fail on a broken setting, and not to depend on
the local Vim. Tests that open sh or java buffers read errors from the
message history, as load.bats does. They do not read v:errmsg or the
redirect, because CI's Vim fills both from its own syntax/sh.vim.

opt-out: no review skill ran. The diff adds tests and changes no
settings, and each test was checked by breaking the line it guards and
seeing it go red, which covers what a diff review would look for here.

Completion report

  1. The functionality works: met. The suite passes locally, and CI is
    watched on this PR.
  2. The access model is defined: not applicable. These are tests of a
    local editor configuration.
  3. Permissions are implemented: not applicable, for the same reason.
  4. Audit logging is implemented: not applicable, for the same reason.
  5. Validation is complete: not applicable. There is no input boundary.
  6. Errors are handled: met. Each test checks that Vim raised no error.
  7. Testing is complete: met for every area. The two lines silent Ex mode
    cannot exercise are named above.
  8. Documentation exists: met. Each new file's header says what it covers
    and which other files cover the rest. .claude/TESTS.md needs no
    change.
  9. The AI Completion Report has been produced: this section.

Co-Authored-By: AI Agent noreply@harleypig.com

harleypig and others added 6 commits November 4, 2026 02:20
set.bats now checks each section of the late set.vim after a full
startup, one test per section; the Terminal Settings section is left
out, since Vim in silent Ex mode loads no termcap and t_ti and t_te read
empty whatever set.vim does.

old-files.bats covers what xdg-settings.vim does beyond where it points
Vim: the 90-day cleanup in the backup, undo, swap and view directories,
the 180-day viminfo limit, everything else under the state directory
left alone, and the backup and viminfo options. Each test was seen red
against a changed age limit, pattern or option.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
help.bats checks what the late help.vim does to a help window: it moves
to a vertical split on the right, and its buffer alone gets Enter and
Ctrl-[ for following a link and going back. The resize to 80 columns is
left out: in silent Ex mode no window is wider than 80, so it never has
anything to do.

maps-rest.bats covers the maps.vim mappings no other file did:
<Leader>tt's terminal split, Ctrl-L clearing search highlighting, and
<Leader>fb reaching ALE's fixer. Each test was seen red against a changed
mapping or autocmd.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
quickfix.bats checks the late quickfixtoggle#toggle(): it opens the
quickfix window and returns the cursor to the window it was called from,
closes the window on a second call, and treats an open location list as
no quickfix window. :QuickfixToggle and <Leader>qf both reach it, and
<Leader>q] and <Leader>q[ step through the list. Each test was seen red
against a broken toggle or a remapped key.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
statusline-parts.bats calls the statusline's functions directly: every
branch of YASL_BufferInfo() and YASL_Warnings(), YASL_FileSize() at 0,
1, 1023, 1024, 1 MiB - 1 and 1 MiB, and the statusline that joins them.

gutter.bats checks gutter.vim's startup values and the sign toggle's
branches tidy.bats did not reach: signs back in a column of their own
with numbers off, and in the number column with only relative numbers.

diagnostic-colors.bats checks the palette is applied again when a colour
scheme loads.

The area's colors.vim and after/diff.vim were removed in #124, so there
is nothing of theirs left to test. Each test was seen red against a
changed branch, boundary or setting.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
ale-values.bats reads the early ale.vim's values from an autocmd that
fires as ALE's own plugin file is about to load, so a value set too late
reads as unset rather than as ALE's default. It also checks the late
ale.vim's global linters and fixers exactly, and that opening a buffer
of every filetype we have an ftplugin for, the list taken from the
after/ftplugin directory, leaves them whole (#77 - one ftplugin wiped
another's fixers). Each test was seen red against a changed or moved
value, or an ftplugin that overwrote the globals.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
ftplugins.bats reads what each after/ftplugin file not already covered
in full sets in a buffer of its filetype: hcl's and terraform's comments,
folds, fixer and linter; java's empty linter list, with and without
google-java-format; python's two-space indent and its isort and yapf
fixers in order; sh's bash syntax, fix on save and full shfmt options;
vim's continuation indent; and yaml's yamllint added to a list the
buffer already has, and given to an Ansible buffer. json and markdown
were already covered by load.bats, ale-settings.bats and
intellisense.bats.

Each test was seen red against a changed setting, except terraform's
comments, folds and fixer: vim-polyglot's terraform ftplugin runs our
hcl.vim too, so a terraform buffer has them whether terraform.vim sets
them or not. The test says so.

Errors are read from the messages that reached the user rather than
v:errmsg, which CI's Vim sets from its own syntax/sh.vim.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
@harleypig harleypig added this to the Pre-improvement milestone Nov 4, 2026
CI's Vim showed two differences from the local Vim.

It is built without +clipboard, so set.vim's clipboard line does
nothing there and the option reads empty. set.bats now checks for
unnamed only where exists('+clipboard') says the option works.

Its own syntax/sh.vim writes the errors that a silent! hides to an
active redirect, so ftplugins.bats and ale-values.bats saw errors that
were not ours. They now read errors from the message history, as
load.bats does. An error one of our ftplugins raises still turns them
red.

Refs #71

Co-Authored-By: AI Agent <noreply@harleypig.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests: behaviour tests for our own settings, area by area

1 participant