3.1: breaking API cleanup, batch handlers, pre-created Blob containers (#51)

docs/v3-design.md

@@ -277,14 +277,15 @@ Previews publish from `v3.0.0-preview.N` tags with the existing `nuget.yml`; nug
 
 ## 3.1 (issue #51)
 
-Issue #51 collected a review of 3.0 against Azure Functions queue triggers, the Service Bus processor, MassTransit, Wolverine, Rebus and SQS libraries. 3.1 delivered all of it except the 4.0 candidates (breaking changes) and the items the review chose to skip. The decisions that shape the code:
+Issue #51 collected a review of 3.0 against Azure Functions queue triggers, the Service Bus processor, MassTransit, Wolverine, Rebus and SQS libraries. 3.1 delivered all of it, the 4.0 candidates included (see below), except the items the review chose to skip. The decisions that shape the code:
 
 - **One envelope format, with forward-compatibility rules.** 3.0's trace-context wrapper became the general envelope (`MessageEnvelope`): trace context, headers, compression, a reference to a body kept elsewhere, and the dead-letter record, each optional. A reader skips properties it does not know; anything that changes how the body is read uses a new `encoding` value, a new body kind or a new version, which an older reader refuses (the message gets `Error` and is dead-lettered intact) instead of reading it as a plain message, which System.Text.Json would turn into an object with empty fields. The marker must still be the first property, so an ordinary payload is never mistaken for an envelope.
 - **One queue implementation over two transports.** `StorageQueue` now sits on `IQueueTransport`, implemented over the SDK client and over the in-memory queue. Encoding, size checks, dead-lettering, replay and telemetry exist once, and a test on in-memory queues runs the production path. This replaced `InMemoryStorageQueue` and `StorageQueueCommon`.
-- **One receive loop per processor.** It waits for a free handler slot, takes every other free slot (up to 32), receives that many, and starts each delivery on its own task, which arms lease renewal first. This keeps F4 (no message waits behind another's handler) and adds "never hold a message no handler is free for". A non-empty receive goes straight back for more, which also fixes the slow pick-up after an idle spell. Batch handlers remain a possible follow-on.
+- **One receive loop per processor.** It waits for a free handler slot, takes every other free slot (up to 32), receives that many, and starts each delivery on its own task, which arms lease renewal first. This keeps F4 (no message waits behind another's handler) and adds "never hold a message no handler is free for". A non-empty receive goes straight back for more, which also fixes the slow pick-up after an idle spell.
 - **Typed views share one lease.** `QueueMessage<T>`'s receipt and settlement moved to an internal `MessageLease`, so a router, a CloudEvent adapter or a lambda handler can hand the application a message of its own type that settles the same delivery the processor tracks.
 - **Compression lives in the envelope, not in a serializer wrapper.** The issue suggested a serializer wrapper; putting it in the envelope means every 3.1 receiver decompresses without configuration, the same way it reads trace context.
 - **Dead-letter categories are a bounded set** (`max_dequeue`, `deserialization`, `handler`, `unroutable`) because they are a metric tag and a replay filter.
 - **Configuration binding is split.** Processor options use the source-generated binder (trim-safe). Client settings keep reflection binding, because the generator cannot create or fill `QueueClientOptions`, which would silently drop `"ClientOptions": { "MessageEncoding": "Base64" }`; those overloads are marked as not trim-safe.
-- **Behaviour change taken:** the default serializer reads property names case-insensitively. Writing camelCase by default stays a 4.0 candidate.
+- **Batch handlers share the processor; only delivery differs.** `QueueProcessorBase<TMessage>` holds the lifecycle, the receive loop, pausing, status and the per-message steps (lease renewal, reading the body, the poison checks, the error policy, settlement). A slot is one delivery: one message for `QueueProcessor`, one batch of up to `MaxBatchSize` for `QueueBatchProcessor`, and one receive fills as many free slots as fit in 32 messages. A batch arms every lease first, reads every body inside the delivery, dead-letters the unreadable and over-delivered individually and leaves them out, then calls the handler once in one scope. Return completes what the handler left unsettled; throw runs the error policy per unsettled message. Options share `QueueProcessorOptionsBase`; the batch side names its concurrency `MaxConcurrentBatches` rather than reusing `MaxConcurrentMessages` with a different meaning. Middleware is per message (`QueueMessageContext.Message`), so it does not run for batches rather than run with a design that would not wrap the bulk call. Status and the active-handlers metric count messages for both, so saturation reads the same. One span per batch, linked to every sender's context and parented to none of them.
+- **The 4.0 candidates moved into 3.1.** The issue held back four breaking changes for 4.0: the library's own `SendResult` and `QueueStatus` records instead of the SDK's `SendReceipt` and `QueueProperties` (so a fake needs no `QueuesModelFactory`), `RenewLockAsync` renamed `RenewVisibilityAsync` to match the option, `QueueClientSettingsBuilder` renamed `QueueServiceBuilder` so it no longer reads like a sibling of `IQueueClientBuilder`, and camelCase output from the default serializer (`JsonSerializerDefaults.Web`, which also reads names case-insensitively). 3.0 had shipped only just before with negligible uptake, so they were made as clean changes in 3.1, with no aliases or `[Obsolete]` forwarders, rather than carried until a major version. The README lists them as breaking changes from 3.0.
 - **Separate packages** for Blob storage (`AzureStorage.QueueService.Blobs`, the claim-check store) and health checks (`AzureStorage.QueueService.HealthChecks`), so the core takes neither dependency. Both build on public seams in the core: `ILargeMessageStore` and `IQueueProcessorController`.

src/AzureStorage.QueueService.Blobs/BlobStorageQueueClientBuilderExtensions.cs

@@ -28,7 +28,8 @@ namespace AzureStorage.QueueService;
 /// margin (and long enough to replay dead-lettered messages).
 /// </para>
 /// <para>
-/// The identity needs the <c>Storage Blob Data Contributor</c> role on the container or the account.
+/// The identity needs the <c>Storage Blob Data Contributor</c> role on the container or the account. One that is not
+/// allowed to create containers needs the container created ahead of time and <c>createContainerIfNotExists: false</c>.
 /// </para>
 /// </remarks>
 public static class BlobStorageQueueClientBuilderExtensions
@@ -38,36 +39,45 @@ public static class BlobStorageQueueClientBuilderExtensions
 
     /// <summary>
     /// Keeps bodies too big for a queue message as blobs in <paramref name="container"/>, which is created on the first
-    /// large send if it does not exist.
+    /// large send if it does not exist (unless <paramref name="createContainerIfNotExists"/> is false).
     /// </summary>
     /// <param name="builder">The client to attach the store to.</param>
     /// <param name="container">The container to keep bodies in, with the credential to reach it.</param>
     /// <param name="thresholdBytes">
     /// Also send bodies of at least this many bytes to Blob storage even when they would fit on the queue. Null (the
     /// default) sends only what would otherwise be over the 64 KiB limit.
     /// </param>
+    /// <param name="createContainerIfNotExists">
+    /// True (the default) creates the container on the first large send if it does not exist. Pass false when the
+    /// container is created ahead of time and the identity is not allowed to create containers.
+    /// </param>
     /// <returns>The same builder, for chaining.</returns>
-    public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClientBuilder builder, BlobContainerClient container, int? thresholdBytes = null)
+    public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClientBuilder builder, BlobContainerClient container, int? thresholdBytes = null, bool createContainerIfNotExists = true)
     {
         ArgumentNullException.ThrowIfNull(builder);
         ArgumentNullException.ThrowIfNull(container);
-        return builder.UseBlobStorageForLargeMessages(_ => container, thresholdBytes);
+        return builder.UseBlobStorageForLargeMessages(_ => container, thresholdBytes, createContainerIfNotExists);
     }
 
     /// <summary>
     /// Keeps bodies too big for a queue message as blobs in the container <paramref name="createContainer"/> returns. It
     /// is called once, from the application's services, the first time the client is used, so the container client can
     /// come from dependency injection (for example one registered with <c>Microsoft.Extensions.Azure</c>). The container
-    /// is created on the first large send if it does not exist.
+    /// is created on the first large send if it does not exist (unless <paramref name="createContainerIfNotExists"/> is
+    /// false).
     /// </summary>
     /// <param name="builder">The client to attach the store to.</param>
     /// <param name="createContainer">Returns the container to keep bodies in, with the credential to reach it.</param>
     /// <param name="thresholdBytes">
     /// Also send bodies of at least this many bytes to Blob storage even when they would fit on the queue. Null (the
     /// default) sends only what would otherwise be over the 64 KiB limit.
     /// </param>
+    /// <param name="createContainerIfNotExists">
+    /// True (the default) creates the container on the first large send if it does not exist. Pass false when the
+    /// container is created ahead of time and the identity is not allowed to create containers.
+    /// </param>
     /// <returns>The same builder, for chaining.</returns>
-    public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClientBuilder builder, Func<IServiceProvider, BlobContainerClient> createContainer, int? thresholdBytes = null)
+    public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClientBuilder builder, Func<IServiceProvider, BlobContainerClient> createContainer, int? thresholdBytes = null, bool createContainerIfNotExists = true)
     {
         ArgumentNullException.ThrowIfNull(builder);
         ArgumentNullException.ThrowIfNull(createContainer);
@@ -79,15 +89,15 @@ public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClie
             {
                 var container = createContainer(provider)
                     ?? throw new InvalidOperationException($"{Describe(clientName)} keeps large messages in Blob storage, but the delegate passed to UseBlobStorageForLargeMessages returned no container.");
-                return new BlobLargeMessageStore(container, createContainerIfNotExists: true, provider.GetService<TimeProvider>());
+                return new BlobLargeMessageStore(container, createContainerIfNotExists, provider.GetService<TimeProvider>());
             },
             thresholdBytes);
     }
 
     /// <summary>
     /// Keeps bodies too big for a queue message as blobs in the container <paramref name="containerName"/>, in the same
     /// storage account as the queue and reached with the same credentials. The container is created on the first large
-    /// send if it does not exist.
+    /// send if it does not exist (unless <paramref name="createContainerIfNotExists"/> is false).
     /// </summary>
     /// <remarks>
     /// The Blob endpoint is worked out from the client's settings the first time the client is used: a connection
@@ -108,9 +118,13 @@ public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClie
     /// Also send bodies of at least this many bytes to Blob storage even when they would fit on the queue. Null (the
     /// default) sends only what would otherwise be over the 64 KiB limit.
     /// </param>
+    /// <param name="createContainerIfNotExists">
+    /// True (the default) creates the container on the first large send if it does not exist. Pass false when the
+    /// container is created ahead of time and the identity is not allowed to create containers.
+    /// </param>
     /// <returns>The same builder, for chaining.</returns>
     /// <exception cref="ArgumentException"><paramref name="containerName"/> is not a valid container name.</exception>
-    public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClientBuilder builder, string containerName = DefaultContainerName, int? thresholdBytes = null)
+    public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClientBuilder builder, string containerName = DefaultContainerName, int? thresholdBytes = null, bool createContainerIfNotExists = true)
     {
         ArgumentNullException.ThrowIfNull(builder);
         ArgumentNullException.ThrowIfNull(containerName);
@@ -123,7 +137,8 @@ public static IQueueClientBuilder UseBlobStorageForLargeMessages(this IQueueClie
                 var settings = provider.GetRequiredService<IOptionsMonitor<QueueClientSettings>>().Get(clientName);
                 return BlobEndpoints.ServiceClientFor(settings, Describe(clientName)).GetBlobContainerClient(containerName);
             },
-            thresholdBytes);
+            thresholdBytes,
+            createContainerIfNotExists);
     }
 
     private static string Describe(string clientName) =>

src/AzureStorage.QueueService.Blobs/README.md

@@ -27,7 +27,7 @@ builder.Services.AddAzureStorageQueueClient(queues =>
 });
 ```
 
-The first form works out the Blob endpoint from the queue client's own settings: a connection string is used as is, `https://myaccount.queue.core.windows.net/...` becomes `https://myaccount.blob.core.windows.net/`, and the storage emulator works too. For a custom domain, use the second or third form. Every form creates the container the first time a large message is sent, and takes an optional `thresholdBytes` to send smaller bodies to Blob storage as well.
+The first form works out the Blob endpoint from the queue client's own settings: a connection string is used as is, `https://myaccount.queue.core.windows.net/...` becomes `https://myaccount.blob.core.windows.net/`, and the storage emulator works too. For a custom domain, use the second or third form. Every form creates the container the first time a large message is sent (pass `createContainerIfNotExists: false` if it already exists), and takes an optional `thresholdBytes` to send smaller bodies to Blob storage as well.
 
 Large messages travel in the library's envelope, which only 3.1 and later read: upgrade every consumer of the queue before turning this on.
 
@@ -37,7 +37,7 @@ Most blobs are deleted automatically, but some are not: a message that expires o
 
 ## Permissions
 
-The identity the client connects with needs **Storage Blob Data Contributor** on the container (or the storage account) to create the container and to write, read and delete blobs. A connection string with the account key needs nothing more. If the container is created ahead of time, create the store yourself with `new BlobLargeMessageStore(container, createContainerIfNotExists: false)` and attach it with `UseLargeMessageStore(store)`.
+The identity the client connects with needs **Storage Blob Data Contributor** on the container (or the storage account) to create the container and to write, read and delete blobs. A connection string with the account key needs nothing more. An identity that isn't allowed to create containers needs the container created ahead of time, and `createContainerIfNotExists: false` on the registration, for example `.UseBlobStorageForLargeMessages("order-bodies", createContainerIfNotExists: false)`.
 
 ## Security
 

src/AzureStorage.QueueService/ICloudEventHandler.cs

@@ -12,7 +12,7 @@ namespace AzureStorage.QueueService;
 /// Everything <see cref="IQueueMessageHandler{TMessage}"/> says about settlement, retries and cancellation applies:
 /// return and the message is completed, throw and it is retried, or settle it yourself through the message. An
 /// event with no data arrives with <c>data</c> null. With
-/// <see cref="QueueProcessorOptions.DeadLetterOnDeserializationFailure"/> off, an envelope or data that could not be
+/// <see cref="QueueProcessorOptionsBase.DeadLetterOnDeserializationFailure"/> off, an envelope or data that could not be
 /// read arrives with <c>data</c> null and <see cref="QueueMessage{T}.Error"/> set.
 /// </remarks>
 /// <typeparam name="TData">

src/AzureStorage.QueueService/IQueueBatchHandler.cs

@@ -0,0 +1,35 @@
+namespace AzureStorage.QueueService;
+
+/// <summary>
+/// Handles a batch of messages for a processor registered with <c>AddBatchProcessor&lt;TMessage, THandler&gt;()</c>: up
+/// to <see cref="QueueBatchProcessorOptions.MaxBatchSize"/> messages from one receive, in one call, so the handler can
+/// write them to a database or call an API in bulk. A new instance is resolved from a new dependency-injection scope for
+/// every batch.
+/// </summary>
+/// <remarks>
+/// <para>
+/// Every message in the batch is still its own message: its own lease renewal, dequeue count, retry delay and
+/// settlement. Return normally and the processor completes every message you did not settle yourself. Throw and the
+/// error policy (<c>DeadLetterOn&lt;TException&gt;()</c>, then <see cref="QueueProcessorOptionsBase.OnHandlerError"/>) is
+/// applied to each message you had not settled, which otherwise is abandoned and retried after
+/// <see cref="QueueProcessorOptionsBase.RetryDelay"/> for its own dequeue count. Settle individual messages with
+/// <see cref="QueueMessage{T}.CompleteAsync"/>, <see cref="QueueMessage{T}.AbandonAsync"/> or
+/// <see cref="QueueMessage{T}.DeadLetterAsync(string?, CancellationToken)"/> (for example, dead-letter one bad item and
+/// return) and the processor leaves those alone.
+/// </para>
+/// <para>
+/// Messages whose body cannot be read, and messages delivered more than
+/// <see cref="QueueProcessorOptionsBase.MaxDequeueCount"/> times, are dead-lettered on their own and left out, so the
+/// batch can be smaller than what was received; when none are left the handler is not called. With
+/// <see cref="QueueProcessorOptionsBase.DeadLetterOnDeserializationFailure"/> off, an unreadable message stays in the
+/// batch with <see cref="QueueMessage{T}.Body"/> null and <see cref="QueueMessage{T}.Error"/> set. Middleware does not
+/// run for batches. The token is cancelled when the host shuts down.
+/// </para>
+/// </remarks>
+public interface IQueueBatchHandler<TMessage>
+{
+    /// <summary>Handles one batch.</summary>
+    /// <param name="messages">At least one message, each with its body read as <typeparamref name="TMessage"/>, its identity, and the operations that settle it.</param>
+    /// <param name="cancellationToken">Cancelled when the host shuts down and the drain timeout has passed.</param>
+    ValueTask HandleAsync(IReadOnlyList<QueueMessage<TMessage>> messages, CancellationToken cancellationToken);
+}

src/AzureStorage.QueueService/IQueueMessageHandler.cs

@@ -9,7 +9,7 @@ namespace AzureStorage.QueueService;
 /// Return normally and the processor completes the message; throw and it is abandoned and retried later; or settle
 /// it yourself with <see cref="QueueMessage{T}.CompleteAsync"/>, <see cref="QueueMessage{T}.AbandonAsync"/> or
 /// <see cref="QueueMessage{T}.DeadLetterAsync(string?, CancellationToken)"/> and the processor leaves it alone. With
-/// <see cref="QueueProcessorOptions.DeadLetterOnDeserializationFailure"/> off, <see cref="QueueMessage{T}.Body"/>
+/// <see cref="QueueProcessorOptionsBase.DeadLetterOnDeserializationFailure"/> off, <see cref="QueueMessage{T}.Body"/>
 /// can be null with <see cref="QueueMessage{T}.Error"/> set. The token is cancelled when the host shuts down.
 /// </remarks>
 public interface IQueueMessageHandler<TMessage>