Worse than a coin flip. Pooled AUROC 0.434, with the entire 95% confidence interval sitting below 0.5.

An AUROC below chance is not a weak model. It is a correctly-learned model wired backwards. Trust went up when a camera was degraded. Had I reported accuracy instead of AUROC, or skipped the confidence interval, this would have looked like a mediocre component rather than an inverted one, and I would have spent weeks tuning it instead of fixing the sign.

Each of the 82 held-out frames is replicated across 5 synthetic fault types and 6 cameras, giving one binary label per camera-frame: degraded or clean. The scorer's continuous trust output is the score, and AUROC is computed over that pooled set. The 95% interval comes from 1,000 bootstrap resamples over frames, not over camera-frame pairs, because pairs drawn from the same frame are not independent and resampling them would give a narrower interval than the evidence supports.

fix v2 differs from fix v1 by stratifying the loss per fault type and optimising the worst-case type rather than the pooled mean, which is why the gain is larger than the pooled number alone suggests: glare and rain streaks were dragging the average.

0.764 is a passing score, not a good one. Three things are open:

• The faults are synthetic. A real degraded lens is not a Gaussian blur, and I have no measurement on real degradation because I have no labelled real fault data.
• 82 frames is small. The CI is honest about that, but a wider held-out set would move the interval more than another loss function would.
• The per-fault-type breakdown is in the artifacts but not on this page. Occlusion is the weakest type and I have not published that number, which is the next thing to do.

Because I do not have one I would defend. AI-text detection scores are extremely sensitive to the evaluation set: a detector trained and tested on the same generator family reports numbers that collapse the moment it meets a model it has not seen, and almost every published figure in this space quietly benefits from that.

Reporting an in-distribution accuracy here would put a big number on the card and mean nothing. The number worth publishing is accuracy against text from a generator absent from training, and until I have run that, this card shows test coverage and stops.

The symptom was not a crash. It was a ranker that scored plausibly and ranked badly.

A LightGBM model consumes a feature vector by position. When a feature is added to the extraction code and an older model is loaded against the new extractor, every feature after the insertion point shifts by one. The model still returns confident floats. Nothing raises. Ranking quality degrades in a way that looks like a bad model rather than a broken contract.

The fix is that inference no longer trusts positional order: each model carries the feature names it was trained on, inference reconciles the live extractor against that list, and a mismatch fails loudly at load time rather than silently at ranking time.

NDCG@10 on a held-out query set. The pipeline is built and the reranker trains, but I have no relevance judgements I did not generate myself, and an NDCG computed against labels I invented would measure my labelling, not my ranking.

That is the honest state of it. The missing measurement is the reason this is a systems project on my profile rather than a ranking-quality one.

A learned policy would have demoed better. It would also have been the wrong choice.

The output of this system is a decision about whether a person is fit to keep driving. When that decision is wrong, someone has to explain why it fired, and "the network weighted the features that way" is not an explanation anyone can act on, audit, or fix in the field. A state machine with explicit thresholds and hysteresis can be read, argued with, and adjusted by someone who is not me.

Hysteresis specifically: without it, a signal sitting on a threshold produces alert chatter, and an alarm that cries wolf is worse than no alarm because people learn to dismiss it.

Worth stating plainly rather than letting a joint project imply solo work. The perception stack, the PointPillars implementation and the BEV encoder are the parts I own. The physiological signal pipeline and the fusion logic were built jointly. Ask on the repo if you need a finer breakdown than that; I would rather answer than have you guess.

Not "95%".

19/20 with a Wilson 95% interval is roughly [0.764, 0.997]. The point estimate says 95 and the interval says the true pass rate could plausibly be 76, which is a completely different system. Reporting the bare percentage takes a result compatible with "fails one in four times" and prints it as near-perfect.

This is the entire reason the framework computes Wilson rather than the textbook normal approximation, which misbehaves badly exactly here: at proportions near 1 with small n, it happily produces upper bounds above 1.0.

Two reasons, one good and one not.

The good one: a deterministic stub means the test suite is reproducible and free, so CI can run the whole thing on every commit without an API key or a bill, and a failing test means the harness broke rather than a model drifted.

The one that costs me: it also means this repo currently contains zero findings about any real model. It is a well-built instrument that has not been pointed at anything. That is the weakest item on this profile, and the fix is not more framework, it is running it against two real models and publishing the intervals.

No. 0.595, against a target I set myself at 0.70.

The tempting move here is to quietly lower the target to 0.55, call it met, and publish a green number. Nobody would ever know. That is exactly the move this project exists to argue against, so the target stays at 0.70 and the card says missed.

The more useful number is the third one below. Context precision 0.437 says the retriever is handing the generator mostly irrelevant chunks, which means faithfulness is low because retrieval is bad, not because generation is unfaithful. Fixing the prompt would have moved nothing.

The judge model returns a short verdict paragraph containing a score. The parser was supposed to extract that score. It was instead counting sentences in the response and dividing by itself, which produces 1.0 for any non-empty answer, then clamping to [0, 1], which hides the arithmetic entirely.

Why it survived: a harness that always reports 1.0 looks like a working system. Every test passed. The suite had no case where a correct implementation and a broken one had to disagree.

The regression test now includes a deliberately ungrounded answer to a control question whose supporting document was removed from the index. The old parser scores it 1.0. The fixed parser scores it near zero. The test asserts the score is below 0.3, so it fails against the old code, which is the only property that makes a regression test worth having.

It was an accepted-but-unused parameter, which is its own small lesson: the function signature promised a behaviour the body never delivered, and nothing in the type system or the tests noticed, because an ignored argument is valid code.

It now compares the generated answer against the retrieved context before returning, and refuses to answer rather than answering from model knowledge when the overlap is below threshold. On the control question, the system now declines instead of confidently answering from parametric memory. Declining is the correct behaviour for a tutor, and it is the behaviour the original evaluation claimed to already have.

Because the two errors are not comparable and averaging them pretends they are.

A false positive routes a routine message to a nurse who spends a minute on it. A false negative sends someone with a real emergency an automated reply. A single F1 score treats those as trading against each other at a fixed rate, and any optimiser handed that objective will happily buy precision with recall, because that is what improves the number.

So urgent recall is not in the objective at all. It is a floor: below 0.92 the run fails and no other result is reported. Everything else is optimised inside that constraint. The Brier score sits alongside it because a model can hit the recall floor while being wildly overconfident, and a confidence number nobody can trust is worse than no confidence number.

Every message this was trained and evaluated on is synthetic. That is a real constraint, not a footnote.

Synthetic patient messages are written by someone who knows what category they belong to, so the category is legible in the wording in a way real messages are not. Real ones bury the urgent detail in the third sentence after two paragraphs about a pharmacy. A model that performs well here has demonstrated it can learn the mapping, not that it can triage.

Which is why the measured recall is not published on this card. The threshold is real, the harness is real, and the number would be measuring the wrong thing.

0.991 ROC-AUC is the highest figure on this profile and the least informative one.

Kvasir v2 is a curated benchmark: clean frames, balanced-ish classes, consistent capture conditions, and a fixed set of pathologies. Strong performance on it says the model learned a benchmark that many models learn. It says nothing about frames with motion blur, poor insufflation, or a pathology outside the eight classes, which is most of real endoscopy.

Cohen's kappa is on the card next to accuracy for the same reason: kappa corrects for agreement that would happen by chance given the class distribution, and on an imbalanced medical set the gap between accuracy and kappa is where the honest reading lives.

The number that would matter is performance on frames from a different scope, hospital and operator. I do not have that data, so I have not made that claim.

Partly to understand it. Forward hooks capture the last convolutional activations, backward hooks capture the gradients flowing into them, the channel weights are the spatially pooled gradients, and the map is the ReLU of their weighted sum. Writing it out is a few dozen lines and removes any doubt about which layer and which gradient a library chose for you.

The practical reason: a dual-head model has two heads to attribute from, and the interesting question is whether the binary head and the 8-class head are looking at the same region. When they disagree, the confident binary call is worth distrusting. That comparison is awkward to express through a library API built around one output.

A RAG system that cites is not the same as a RAG system whose citations are real. The common failure is a fluent answer carrying a citation to a chunk that does not support it, which is worse than no citation, because the citation is what convinces the reader to stop checking.

Enforcement here means a cited span has to trace back to retrieved context before the answer is returned. An answer that cites nothing retrievable does not ship with a citation attached to it.

The policy-injection blocklist is the adjacent problem: retrieved text is untrusted input, and a document containing instructions is still a document. Content coming out of the index is treated as data to quote, never as instructions to follow.

This card describes a voice RAG assistant. The repo is called rag-adversarial-robustness-eval-harness. Those read as two different projects, and if you noticed the mismatch before I pointed it out, your instinct was right to.

The honest history is that the repo started as the robustness harness and grew the voice pipeline around it without ever being renamed. Renaming it is on the list. Flagging it here costs less than having you find it and wonder what else on this page is loosely attached to what it points at.