Repository navigation
Feat/api compat validation - #65
Merged
Merged
Conversation
… them SnippetCompilationTests only checked that each snippet parsed as valid C#. That catches a missing brace and little else: a snippet calling a method the API does not have parses perfectly and still fails to build for anyone who copies it. It now builds a Roslyn compilation per snippet against the actual MESL.SQLRace.API assemblies, referencing both TRUSTED_PLATFORM_ASSEMBLIES and this assembly's own output directory so the same code works on a developer machine and on a CI runner. Compilation is metadata-only, so no SQL Race licence is involved and the suite runs on a plain GitHub runner, which makes it the one suite CI can execute. SqlRaceApiVersion parameterises the package reference, so the snippets can be compiled against a chosen build rather than whatever restores: dotnet test -p:SqlRaceApiVersion=<version> CI runs the suite advisory-only to begin with, while the failures it reports are worked through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Compiling a snippet proves it still builds. It does not prove it still returns the right numbers, and it says nothing at all about the Python and MATLAB snippets, where a rename or a changed signature fails silently at runtime for pythonnet and NET.addAssembly callers. run-examples-e2e.ps1 executes the runnable snippets in all three languages against the deterministic fixture and diffs their stdout against tests/golden/<language>/<name>.txt. Values that legitimately vary between runs - GUIDs, timestamps, absolute paths, elapsed times - are normalised first, so what is asserted is the data: parameter names, sample counts, computed values. Which snippets run, their arguments, and why the rest are skipped now live in scripts/snippet-manifest.psd1 rather than inline in the script, so adding a snippet is a data change and the skips stay in one visible place. A snippet that runs with no golden file reports NOGOLD rather than passing, and skips are listed in the summary - a snippet silently missing from the run would otherwise look identical to one that passed. SnippetRunner gained a SnippetFile property that swaps the Compile item instead of copying each snippet over Program.cs, which used to leave the working tree dirty for the duration of a run and lost the original if the run was interrupted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A MATLAB startup costs the better part of a minute, so a missing install or an unavailable licence seat produced twenty-odd identical failures over a quarter of an hour. One checkout up front turns that into one clear message in forty seconds, and reports the snippets as skipped rather than failed - they were never run, so calling them failures is wrong. The message names MLM_LICENSE_FILE and points at the two things that actually go wrong on a build agent: no route to the FlexLM server, or no free concurrent seat.
…ippet wedging the run Three changes, all in service of running the examples against a SQL Race build before it is published to anyone. AdditionalPackageSource takes a folder holding a .nupkg and passes it as RestoreAdditionalProjectSources. That is deliberately not "dotnet nuget add source": the repo nuget.config starts with <clear />, which discards any source added at user level. This is what lets a build validate the package it just produced without publishing it first. SqlRaceApiVersion moves to Directory.Build.props so it applies to every project at once. Setting it on the test project alone left SqlRace.Examples.Core and FunctionLibrary still resolving Version="*" from the feed, so a candidate run would have mixed two different SQL Race builds and proved nothing. Verified: all three projects now resolve the same pinned version. Snippets run through Start-Process with a hard timeout and are killed as a process tree if they exceed it. A snippet that hangs rather than fails used to wedge the whole run, and because the hung process keeps a handle on the SQLite fixture, every snippet after it failed too. TIMEOUT is reported separately from FAIL - the snippet is hung, not wrong. Reset-Fixture also retries, since SQLite handles are not always released the moment a process exits. Two things that had to be right for the process launch to work at all: Start-Process -PassThru releases the native handle on exit unless Handle is touched first, leaving ExitCode empty; and the redirected output has to be read as UTF-8, or the box-drawing characters in the snippets' table rules come back as mojibake and every golden comparison fails. Verified end to end against a package present only in a local folder: C# 24 pass / 1 known failure / 8 skipped, Python 5 pass / 2 known failures - identical to the published-package run.
Two things the first CI run of the validation pipeline exposed. The test project pinned Microsoft.CodeAnalysis.CSharp 4.8.0, and restore failed with NU1605 because the candidate API required 4.9.2. That floor is not fixed: the SQL Race build rewrites its nuspec dependency versions to whatever the solution consumed, so it moves with their dependency graph. Dropping the reference to inherit the API's version fails the other way - an older release asks for 4.3.1, which has no LanguageVersion.CSharp12. 4.* resolves at or above whatever the API asks for, so a downgrade cannot happen and the language version the snippets are written in is always understood. Verified against the old API: resolves 4.14.0, builds, 34 pass and 3 fail as before. The Python leg produced seven identical "The system cannot find the file specified" failures on the agent, which says nothing about the actual cause - Python is not installed there. Test-PythonAvailable mirrors the MATLAB preflight: it reports the snippets as skipped with a reason, and separately distinguishes Python missing from pythonnet being unusable, since a missing cffi fails inside pythonnet.load() with an error that reads like a SQL Race fault. Verified both branches.
fdl-function-basic, dotnet-function-basic and dotnet-function-with-pda used a Functions API that does not exist in any released build: SetFdlCode, SetOutputParameter, AddInputParameter, SetImplementation, a two-argument Build and an enumerable FunctionBuildResults. None of the three compiled against a released MESL.SQLRace.API, which is what the new snippet compilation suite reports. Rewritten against MESL.SqlRace.Domain.Functions as it is today: - CreateFunctionDefinition takes the builder's UniqueId, not a type id plus output name, so FdlFunctionConstants/DotNetFunctionConstants are now imported from their own namespaces. - FDL source goes on the implementation definition, cast to IFdlFunctionImplementationDefinition; a .NET implementation is linked by assigning DotNetFunctionImplementationDefinition.Function. - Inputs go on InputParameterIdentifiers, outputs on OutputParameterDefinitions as FunctionOutputParameterDefinition. - Build takes the definition alone and returns FunctionBuildResults, whose Errors carry ErrorText. - IDotNetFunction is Initialize(IFunctionManager) plus Execute(IExecutionContext) over a batch, not a per-sample Execute with input/output dictionaries. The .NET snippets now follow projects/csharp/FunctionLibrary, so the snippet and the buildable project demonstrate the same pattern. FDL output parameters are published as "<Name>:<ApplicationName>", and a built function's outputs are added to sessions as they load, so the FDL snippet builds before it loads and reads "TemperatureFahrenheit:Functions". The compilation step in CI no longer needs continue-on-error, and the manifest's skip reasons for these three are updated to the real ones. Verified: 37/37 SnippetCompilation tests pass against 2.1.26050.4, 2.1.26169.4 and 2.1.26212.6. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MSBuild stops at the first Directory.Build.props it finds walking up from a project, so the repo-root props - which define SqlRaceApiVersion - did not reach the projects under projects/csharp. Their package reference is Version="$(SqlRaceApiVersion)", so it expanded to an empty version range. NuGet resolves that to the oldest MESL.SQLRace.API available rather than treating it as an error, which left FunctionLibrary and SqlRace.Examples.Core on a different build from the test project. That is the mixed-build case SqlRaceApiVersion exists to avoid, and it would make a compatibility run report on the wrong build. It showed up as an NU1604 restore error only because projects/csharp also sets TreatWarningsAsErrors. With the import, all three projects resolve the same build: dotnet test -p:SqlRaceApiVersion=2.1.26169.4 -> 2.1.26169.4 for FunctionLibrary, SqlRace.Examples.Core and the tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This repository is public, so its comments, docs and CI configuration are read by the same people who read the snippets. A few of them were written for an internal audience. - tests/README.md pointed at build pipeline files by path in another repository. Replaced with the part a reader here can use: that these suites run against SQL Race builds before they are published. - A prerelease build number was the worked example for -SqlRaceApiVersion in five places. It is now a <candidate-version> placeholder, which also does not go stale. - A few comments referred to the reader in the third person. Reworded to address them directly. No behaviour change; the snippets themselves were already clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both failed in the pipeline run. They fail for unrelated reasons. 04_extract_events.py used five members that do not exist. EventCollection exposes a public EventDefinitions property that is assigned null in the constructor and never populated anywhere in the API, so session.Events. EventDefinitions is always null; the populated collection is session. EventDefinitions. The rest were wrong names: Name -> Description, Values -> RawData, Timestamp -> TimeStamp, EventDefinitionName -> EventDefinitionKey plus GroupName. Python binds by name at runtime, so none of this was visible until the snippet was actually executed. event-definitions-and-data.cs got two things wrong. An EventDefinition's fourth argument lists conversion function names, not field labels, and each must be registered with AddConversion first - Commit() drops any definition whose conversions are missing, logging an error rather than throwing, so the failure surfaced later as "event definition not found" from AddEventData. Separately, events are stored in time segments, so a session with no channel data has StartTime 0 and discards every event without raising anything. The snippet now writes a parameter and 250 samples first. Verified by compiling and running both against a local build: the C# snippet emits all four events in time order, stably across three runs. Events are also sorted explicitly, because they come back in segment order. The Python snippet runs clean but prints an empty table: events do not survive a session reload - parameters do - so a loaded fixture session can never carry any. It will report NOGOLD until that is resolved.
It no longer crashes, but it verifies nothing: event data does not survive a session reload, so the loaded fixture session it is pointed at can never carry any. The loop body never runs, which leaves most of the member names just corrected unexercised, and the golden would be an empty table. A skip with a stated reason is more honest than a pass over nothing. Making it demonstrate anything means having it create its own session with events, which is a rewrite of a published example rather than a fix.
The snippet has never passed, so no golden was ever recorded and it would have reported NOGOLD - ran, but nothing verified - even after the fix. Generated with the harness against the last published release, 2.1.26212.6, and confirmed stable: two further runs compared clean. All four events appear in time order with their raw data, and the per-run session GUID normalises away.
It assumed every parameter shared the first one's timestamps. The fixture is deliberately mixed-rate, so Speed:Shaft's 500 values went into a 1000-row index and pandas refused. Slicing to len(df) truncates but cannot extend. Assigning through pd.Series aligns on index and pads the short column with NaN. That alone would make the snippet quietly wrong rather than broken: columns are matched by position, so Speed:Shaft's 50 Hz samples sit against Temperature's 10 Hz timestamps and every one of its values lands on the wrong time. head(10) never reaches the padding, so nothing on screen hints at it. The snippet now says so in its output when the counts differ, and points at multi-rate-alignment.cs, which builds a common timestamp array properly. Golden generated against 2.1.26212.6 and confirmed stable over two further runs.
TimOliverMAT
reviewed
Sep 3, 2026
TimOliverMAT
reviewed
Sep 3, 2026
Three vacuous goldens: query-with-composite-filter, 05-query-sessions-by-metadata and 03_query_sessions all recorded "no sessions found". They passed while proving nothing, which is worse than failing. Three independent causes, all now fixed. The fixture session was named "Verified Test Session" while every query snippet filters on Identifier containing "Example", so it could never match. Renamed to "Verified Example Session". Its recording date was fixed at 2025-06-15, outside the "last 30 days" window the composite filter asks for. Only the 10:00 time-of-day matters - that is what makes Session.StartTime equal StartTimeNs - so the date is now today and the window is satisfied. query-with-composite-filter also had two bugs of its own. It filtered on "TimeOfRecording", which is the property name on the result rather than a filter field; the field is SessionStartDateTime. An unrecognised name falls through to being treated as a session metadata item and quietly matches nothing. It also passed the date as ISO 8601 "O", while ScalarFilter parses with TryParseExact against ScalarFilter.DateFormat, so any other format silently fails. It now uses SessionFieldIdentifiers and the DateTime overload, which formats itself. Also widened the date normalisation in run-examples-e2e.ps1. pythonnet renders DateTime with the current culture, so 03_query_sessions prints "9/3/2026 10:00:00 AM" - single-digit month and day and an AM/PM suffix, none of which the old pattern matched. That golden would have passed today and failed tomorrow. Verified: 25 C# and 6 Python snippets pass against the regenerated goldens, and the four that differed were reviewed individually before regenerating rather than being rewritten wholesale.
I was wrong that event data does not survive a session reload, and the skip on 04_extract_events rested on that. It does survive: writing a definition and two events, then reloading, returns both with correct timestamps and raw data. My earlier test committed the parameters in one configuration set and the event definition in a second. Only the first configuration set committed to a session survives a reload - the note about that was already in this file, about parameters. Everything that must persist has to go in one commit. So BuildBatch now takes an optional hook to add conversions and event definitions to the same configuration, and the fixture raises three events at +2s, +5s and +8s carrying the temperature ramp's value at that moment against a fixed threshold - still derivable by hand, like every other value in here. 04_extract_events is unskipped and now exercises the whole snippet rather than an empty loop. Its output matches the closed form exactly: 40.0, 70.0 and 100.0 for Temperature(200), Temperature(500) and Temperature(800). Verified nothing else moved: 25 C# and 7 Python snippets pass, with no golden other than the new one needing to change.
TimOliverMAT
reviewed
Sep 4, 2026
| Lap 3 0 | ||
| Lap 4 0 | ||
| Lap 5 0 | ||
| Total events: 0 |
There was a problem hiding this comment.
There still seems to be an issue with this test
It recorded "Total events: 0" - a pass over nothing. None of the eight scenario files under notebooks/scenarios/data/ contain a single event definition, so the snippet is reporting empty data honestly and the golden asserted nothing. Without it the snippet reports NOGOLD, which is the accurate state: it runs, and nothing is verified. Fixing it properly means adding event data to motorsport-lap-analysis.ssn2. Retrofitting works - tested on a copy, two definitions and three events survive a reload - but that is a change to committed binary data with no generator script behind it, which belongs in its own change rather than this one. Correcting something I got wrong earlier: the other MATLAB goldens are fine. The scenario .ssn2 files are committed and tracked, so "Turbine Inspection Week 1" and the rest are reproducible repo data, not sessions left on a dev machine.
TimOliverMAT
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.