fix(pipeline): surface reactive fallback failures and retry them next run - #841
Merged
Conversation
… run - Print stage failures in the console reporter when no spinner is running, so a writer flush, provenance write or reactive-fallback error reported outside a stage leaves a trace instead of being dropped - Adopt the dump’s change fingerprint only after the fallback re-run completes; a throwing reset or re-run keeps the endpoint’s fingerprint so the dataset is retried instead of frozen out until the dump changes - Document the fingerprint behaviour of the fallback in the pipeline reference
…st the dump - Adopt the dump’s fingerprint only when the fallback re-run succeeds; a stage failure against the dump is caught inside the re-run rather than thrown, and must not freeze the dataset out either - State in the docs that the stored dump fingerprint only skips a run in which the endpoint fails probing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #804
When the reactive dump fallback imported the dump but then failed before or during the re-run, two things went wrong at once. The error never reached the log, because
ConsoleReporter.stageFailedonly printed while a spinner was active, and the fallback catch block fires just after the import spinner has finished. And the dataset was recordedfailedunder the dump’s fingerprint, so a following run in which the endpoint failed probing skipped it as unchanged until the dump itself changed.Changes
@lde/pipeline-console-reporter:stageFailednow prints an error line when no spinner is running, the wayimportFailedalready did. This covers the'write','provenance'and'reactive-dump-fallback'failures the pipeline reports outside any stage.@lde/pipeline: the dump’s change fingerprint is adopted only after the fallback re-run has succeeded. IfrunWriter.resetor the re-run throws, or a stage fails against the dump, the record keeps the endpoint’s fingerprint (null), so the dataset is retried on the next run instead of frozen out.sparqlWithImportFallbackentry in the pipeline reference now states this fingerprint behaviour, including that the stored dump fingerprint only skips a run in which the endpoint fails probing; when the endpoint passes probing, it is tried again first.Not included
resetfailure in@lde/search-typesense’s in-placedropWrittenThisRun(point 3 of the issue). With the reporter change it will show up in the log on the next occurrence, which is what is needed to diagnose it.runWriter.flushthrows transiently is recordedfailedunder the dump’s fingerprint. That is pre-existing and deserves its own issue, since the general fix (record infrastructure failures under anullfingerprint, or haveshouldReprocessconsultstatus) changes the skip-unchanged contract.