Add Blob Versioning Support to Azurite - #2735
Rodolfo Orozco Vasquez (rorozcov) wants to merge 92 commits into
Conversation
…eate blob. No testing yet. All theoretical
…tore. Now local testing needed. Saving progress
…bs and api changes
…tests with versioning enabled. Must add versioning related checks
There was a problem hiding this comment.
🔵 Needs a closer look
The new continuation-token encoding for version-aware pagination is ambiguous for certain valid blob names and can break paging behavior, so it should be made robust before merge.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
src/blob/persistence/PageWithDelimiter.ts:226
- Continuation tokens for name+timestamp mode are currently built by joining
[name, timestamp]with the hard-coded__version_marker__separator. If a blob name contains that substring, the token becomes ambiguous andmarker.split(VERSIONING_MARKER)(e.g., in LokiBlobMetadataStore.listBlobs) will fail and return 400, breaking pagination for valid blob names. Consider encoding the marker tuple as an unambiguous format (e.g., JSON string or base64 of JSON) and accepting the legacy joined format for backward compatibility.
src/blob/persistence/LokiBlobMetadataStore.ts:166 - PR description mentions version IDs using JavaScript millisecond precision rather than Azure's 7 fractional digits, but the implementation converts timestamps to 7 fractional digits and even uses a per-blob sub-millisecond counter (see generateVersionId). Please align the PR description (and any external docs) with the implemented 7-digit RFC3339 versionId format so expectations match runtime behavior.
- Files reviewed: 59/61 changed files
- Comments generated: 0 new
- Review effort level: Lite
Akanksha Jain (@jainakanksha-msft) Your comments have been addressed. I am merging main into my branch again. I am also working on the pagination continuation token and wanted your input. Initially, I made it very simple to keep it backwards compatible with how azurite used to do it. However, it is technically opaque from Azure's point of view so I am thinking of making it opaque as well rather than azurite's current use of the blob name as the continuation token and my PR's use of blobname__Version_marker__timestamp. Let me know what you think. If not once I merge main we should be good to go |
There was a problem hiding this comment.
🟡 Changes recommended
Continuation token encoding in PageWithDelimiter is ambiguous for certain blob names and can break listing pagination, and there is also a mislabeled test tag in the Azurite parity suite.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 59/61 changed files
- Comments generated: 2
- Review effort level: Lite
Co-authored-by: rorozcov <44987991+rorozcov@users.noreply.github.com>
Use base64url-encoded, versioned JSON containing: type BlobListMarkerV1 = { |
Co-authored-by: rorozcov <44987991+rorozcov@users.noreply.github.com>
…d id Co-authored-by: rorozcov <44987991+rorozcov@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
There are documentation/test labeling issues in the current diff that should be corrected before merge, in addition to the overall size/surface area of the change.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
tests/blob/apis/versioning.azurite.parity.test.ts:185
- This test is in the Azurite parity suite but is tagged with "Indicee Production Account (@production)" in the name, which can break test filtering / categorization. Rename the tag to "Marko Nikic (@azurite)" (or drop it) to match the rest of this file.
- Files reviewed: 59/61 changed files
- Comments generated: 1
- Review effort level: Lite
|
Copilot Fix the code for all comments in this review thread. When a review comment includes a suggested change, apply the suggestion exactly. Do not make changes beyond what is described in the linked review thread. |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
Review effort: Lite
Findings: 3
Open (9)
this.latestMarker.join(PageWithDelimiter.VERSIONING_MARKER),…isCurrentVersionis being forwarded as-is, which can emitfalseinto the response. Azure’s… · NewisCurrentVersionis being forwarded as-is, which can emitfalseinto the response. Azure’s… · NewparseAccountModelFlagsintroduces non-trivial parsing behavior (single vs multi-account modes;… · New@production, which is…
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved review findings remain, including critical persistence and access-log regressions.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
Open (5)
Resolved since last review (9)
this.latestMarker.join(PageWithDelimiter.VERSIONING_MARKER),…parseAccountModelFlagsintroduces non-trivial parsing behavior (single vs multi-account modes;…isCurrentVersionis being forwarded as-is, which can emitfalseinto the response. Azure’s…isCurrentVersionis being forwarded as-is, which can emitfalseinto the response. Azure’s…@production, which is…
| accessTierChangeTime: date | ||
| }, | ||
| snapshot: "", | ||
| isCommitted: true, |
| // accessTierInferred | ||
| }, | ||
| snapshot: "", | ||
| isCommitted: true, |
| ); | ||
|
|
||
| const blobServerFactory = new BlobServerFactory(); | ||
| this.server = await blobServerFactory.createServer(env, accountModelStore); |
| if (!configuration.accountModelStore) { | ||
| throw new Error("Account model store must be provided in BlobConfiguration"); | ||
| } |
| if ( | ||
| versionId !== undefined && | ||
| versionId !== "" && | ||
| !/^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{7}Z$/.test(versionId) | ||
| ) { |
Preserve account configuration and version history while adopting upstream lease validation ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical and moderate correctness and compatibility findings remain, and validation was not executed.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 5
Open (7)
| account: sourceAccount, | ||
| container: sourceContainer, | ||
| blob: sourceBlob, | ||
| snapshot | ||
| snapshot: snapshot, | ||
| versionId: versionId |
| public async fill( | ||
| reader: (offset: number) => Promise<BlobType[]>, | ||
| namer: (item: BlobType) => string, | ||
| markerFunc: (item: BlobType) => BlobListMarkerTuple, |
Title: Add Blob Versioning Support to Azurite
Summary
Adds Azure Blob Storage versioning support to Azurite, enabling applications to create, list, retrieve, and delete previous blob versions during local development.
Closes #665.
Account model
This work introduces the account model abstraction designed in collaboration with the Azurite team. The account model provides a central place for account-scoped feature configuration so additional Azure Storage capabilities can be added without coupling them directly to server startup or individual blob handlers.
Blob versioning is currently the first and only feature backed by this abstraction. The design supports Azurite's multi-account mode, allowing each configured account to enable or disable versioning independently. Accounts without an explicit versioning setting retain the existing non-versioned behavior for backward compatibility.
Motivation
Blob versioning automatically preserves previous blob states after modification or deletion. Without emulator support, applications that depend on version-aware workflows cannot be tested locally before deployment to Azure.
Implementation
AccountModelsetting for enabling blob versioning.versionId.includeVersionswith version-aware continuation markers.Configuration
Blob versioning can be configured using:
--accountConfigFilePathfor JSON configuration files.--accountConfigAsJsonfor inline JSON configuration.Configured accounts must also be present in
AZURITE_ACCOUNTSfor authentication.Behavior
Limitations
This change does not currently support:
Validation