feat(dictation): fall back to the buffered endpoint past the live 120 s cap

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## [4.43.0]
+
+- `client.dictation` now handles dictations past the live endpoint's 120 s cap. When `transcribeLive()` or `openLive()` fails with 413 `audio_too_large`, the SDK keeps reading the audio until it ends (the stream or iterable ends, or `close()` / `result()` on a session), posts the whole recording once to the buffered endpoint `POST /v1/transcribe` (up to 20 minutes or 256 MB, same config, file name and content type), and resolves with that result exactly as a live one, `final_text` included. To have the recording to send, an in-memory copy of every uploaded audio byte is kept from the first one, about 32 KB per second at 16 kHz mono (about 38 MB at 20 minutes); a copy past 256 MB is dropped and the call throws `DictationError` (413, `audio_too_large`). A fallback result waits for the whole recording to upload and be transcribed, so it returns more slowly than a live one
+- Add `fallbackToBuffered` to `DictationLiveOptions` (default `true`). Set it to `false` to keep no copy and get the live 413 as before. Only a 413 with `audio_too_large` falls back; every other failure is thrown unchanged, a failed buffered request throws its own error, and `abort()` or an abort `signal` drops everything with no fallback request. Once the fallback engages, `timeout` no longer cuts off the rest of the recording, and the buffered request gets a fresh `timeout` deadline
+
 ## [4.42.0]
 
 - Add Text-to-Speech: `client.tts.synthesize(request)` returns one text's complete audio (`TextToSpeechResponse`: `audio` plus the response headers as `requestId`, `region`, `model`, `voiceId`, `encoding`, `sampleRate`, `billedCharacters`, `audioDurationMs`, ...), `client.tts.stream(request)` returns it as the API generates it (`TextToSpeechStream`, an async iterable over a web `ReadableStream`), and `client.tts.session(params)` opens a WebSocket `TextToSpeechSession` for voice agents: `connect()`, `generate()` / `flush()` / `audio()`, `speak()`, `cancel()`, `keepAlive()`, `close()`, with `on("audio" | "flushDone" | "wordBoundaries" | "cancelled" | "termination" | "error" | "close", ...)`. Requests take the API's fields (`text`, `voice_id`, `model_id`, `language`, `sample_rate`, `encoding`, `container`, `bit_rate`); failures throw `TextToSpeechError` (`status`, `errorCode`, `param`, `requestId`, `retryable`, `retryAfter`, `errors`, `closeCode`, `body`). New client option `ttsBaseUrl` (default `https://tts.us.assemblyai.com`); `client.tts.createTemporaryToken()` mints a token for browser sessions. Text sent while `connect()` is pending is held and sent the moment the socket opens, so it need not wait for `Begin`

CLAUDE.md

@@ -328,7 +328,8 @@ transcript when the audio ends.
 iterable, or a web `ReadableStream<Uint8Array>` — or audio already held whole: a local file path
 (Node/Bun/Deno), a data URL, raw bytes (`Uint8Array`/`ArrayBuffer`), or a Blob/File, which is
 sent as a single chunk over the same connection. **Not** a URL — that throws, pointing at
-`client.transcripts`. Audio must be WAV or raw 16-bit PCM, ≤120 s per request.
+`client.transcripts`. Audio must be WAV or raw 16-bit PCM. The live upload takes ≤120 s; longer
+dictations (≤20 min) fall back to the buffered endpoint by default (below).
 
 ```typescript
 await client.dictation.warm(); // open the connection ahead of time (does not validate the key)
@@ -386,15 +387,28 @@ timestamps), `confidence`, `llm_response` (`string | null`, the rewrite when the
 429/503. Auth, rate-limit, size and capacity failures can surface part-way through the upload
 rather than only at the end.
 
+**Past 120 s — buffered fallback** (`DictationLiveOptions.fallbackToBuffered`, default `true`, applies
+to `transcribeLive()` and `openLive()`): the SDK keeps an in-memory copy of every audio byte it
+uploads. When the live request fails with 413 `audio_too_large`, it keeps reading the source until
+it ends (stream/iterable ends, or `close()`/`result()`), then posts the whole copy once to `POST
+/v1/transcribe` (≤20 min / 256 MB, same config, file name and content type) and returns that result
+like a live one, `final_text` included. A failed buffered request throws its own error; any other
+live failure is thrown unchanged. `abort()`/`signal` drop everything, with no fallback request. The
+live deadline stops applying once the fallback engages; the buffered request gets a fresh `timeout`.
+Trade-offs: a fallback result waits for the whole recording to upload and be transcribed (slower
+than live), and the copy costs ~32 KB/s at 16 kHz mono (~38 MB at 20 min); a copy past 256 MB is
+dropped and the call throws `DictationError` (413, `audio_too_large`). `fallbackToBuffered: false`
+keeps no copy and throws the live 413 as before.
+
 **Pre-warming**: `await client.dictation.warm()` issues a `GET /v1/warm` to take DNS + TCP + TLS
 off the critical path. It returns `true` once the socket is open — any HTTP response counts, even
 a non-200 one, because it does not validate the key — and `false` on a transport failure. Call it
 shortly before the request; the pooled connection idles out after a few seconds.
 
-**Client-side timeout**: third argument, `DictationLiveOptions` — `{ timeout, signal }`.
-`timeout` (default 300 000 ms) is a **total** deadline from the start of the request, spanning the
-upload, the transcription and the LLM pass; the service caps audio at 120 s. `signal` cancels a
-`transcribeLive()` from outside; sessions manage their own.
+**Client-side timeout**: third argument, `DictationLiveOptions` — `{ timeout, signal,
+fallbackToBuffered }`. `timeout` (default 300 000 ms) is a **total** deadline from the start of the
+request, spanning the upload, the transcription and the LLM pass; the live upload caps audio at
+120 s. `signal` cancels a `transcribeLive()` from outside; sessions manage their own.
 
 **Exports** from the package root: `DictationTranscriber`, `DictationLiveSession`,
 `DictationError`, and the `DictationAudioInput`, `DictationConfig`, `DictationLiveOptions`,
@@ -464,7 +478,7 @@ Gotchas: `stream()` gives up a body that sends no audio for `readTimeout` ms (30
 - **Streaming uses universal-3-5-pro** as the speech model
 - **Never expose API keys client-side** — use temporary auth tokens for browser streaming
 - **Sync timeout is a total deadline** — `transcribe()`, `transcribeLive()` and `openLive()` all default to a 180 s deadline spanning the whole request (upload plus transcription), and the sync API still caps audio at 120 s
-- **Dictation timeout is a total 300 s deadline** — `client.dictation` spans the upload, the transcription and the LLM pass in one deadline; the service caps audio at 120 s and accepts WAV or raw 16-bit PCM only
+- **Dictation timeout is a total 300 s deadline** — `client.dictation` spans the upload, the transcription and the LLM pass in one deadline; the live upload caps audio at 120 s (longer dictations fall back to the buffered endpoint, up to 20 min, unless `fallbackToBuffered: false`) and accepts WAV or raw 16-bit PCM only
 - **Node >= 18 required**
 - **Only runtime dependency**: ws (WebSocket library)
 - **Multi-runtime support**: Works in Node.js, Deno, Bun, Cloudflare Workers, and browsers

README.md

@@ -557,8 +557,9 @@ console.log(result.final_text, result.session_id);
 
 `result.final_text` is the one to show the user: the LLM rewrite when an
 `llm_instruction` ran, the raw transcript otherwise. `result.text` always holds
-the raw transcript. Audio must be WAV or raw 16-bit PCM, up to 120 seconds per
-request.
+the raw transcript. Audio must be WAV or raw 16-bit PCM. The live upload takes
+up to 120 seconds; a longer dictation, up to 20 minutes, is sent again whole
+(see "Dictations longer than 2 minutes" below).
 
 > This is not the streaming API. `client.streaming` returns words while the
 > speaker is still talking; dictation returns one finished transcript when the
@@ -638,8 +639,8 @@ A URL is rejected with an error pointing at `client.transcripts`, which is where
 URL ingestion lives.
 
 `options.timeout` (default 300 000 ms) is a **total** deadline from the start of
-the request, spanning the upload, the transcription and the LLM pass; the
-service caps the audio itself at 120 seconds. `options.signal` drops the request
+the request, spanning the upload, the transcription and the LLM pass; the live
+upload caps the audio itself at 120 seconds. `options.signal` drops the request
 from outside, the way `abort()` does for a session.
 
 Caveats, for both entry points:
@@ -654,6 +655,50 @@ Caveats, for both entry points:
 
 </details>
 
+<details>
+<summary>Dictations longer than 2 minutes</summary>
+
+The live upload takes up to 120 seconds of audio. Past that, the dictation API
+refuses it with a 413 (`audio_too_large`), and by default the SDK falls back to
+the buffered endpoint, `POST /v1/transcribe`, which takes up to 20 minutes or
+256 MB:
+
+1. From the first byte, the SDK keeps an in-memory copy of the audio it uploads.
+2. When the live request is refused for its length, the SDK keeps reading your
+   audio until it ends: the stream or iterable ends, or you call `close()` /
+   `result()` on a session.
+3. It posts the whole recording once, with the same config, and resolves with
+   that transcript exactly as it would a live one.
+
+Nothing changes in your code: `transcribeLive()` and `openLive()` work the same
+for a 10 second note and a 15 minute one. Two trade-offs come with it:
+
+- **Latency.** After a fallback, the result waits for the whole recording to
+  upload and be transcribed, so a long dictation returns more slowly than a live
+  one.
+- **Memory.** The copy costs about 32 KB per second of 16 kHz mono 16-bit PCM,
+  about 38 MB at 20 minutes. A copy that grows past 256 MB is dropped, and the
+  call throws a `DictationError` (413, `audio_too_large`).
+
+Only a 413 with `audio_too_large` falls back; every other failure is thrown as
+it is. If the buffered request fails, its own error is thrown (a 413 past 20
+minutes, say). `abort()` or an abort `signal` drops everything with no fallback
+request. `options.timeout` stops applying to the live request once the fallback
+engages (the rest of the recording takes as long as it takes), and the buffered
+request gets a fresh `timeout` of its own.
+
+To keep no copy and get the 413 instead, turn it off:
+
+```typescript
+const result = await client.dictation.transcribeLive(
+  micStream,
+  { sample_rate: 16_000, channels: 1 },
+  { fallbackToBuffered: false },
+);
+```
+
+</details>
+
 <details>
 <summary>Rewrite the transcript with an LLM</summary>
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "assemblyai",
-  "version": "4.42.0",
+  "version": "4.43.0",
   "description": "The AssemblyAI JavaScript SDK provides an easy-to-use interface for interacting with the AssemblyAI API, which supports async and real-time transcription and text-to-speech.",
   "engines": {
     "node": ">=18"

src/services/dictation/live.ts

@@ -12,9 +12,11 @@ import { LiveSession } from "../../utils/live";
  * iterables, and audio held whole, go to `client.dictation.transcribeLive()`
  * instead.
  *
- * Everything `transcribeLive()` says about the 120 s audio cap, the
- * server-side silence abort and errors surfacing mid-upload applies here
- * unchanged.
+ * Everything `transcribeLive()` says about the 120 s audio cap and the
+ * buffered fallback past it, the server-side silence abort and errors
+ * surfacing mid-upload applies here unchanged. When the fallback engages,
+ * the session keeps taking writes until `close()` or `result()`, then sends
+ * the whole recording at once; `abort()` drops it with no fallback request.
  *
  * @example
  * ```ts

src/services/dictation/recording.ts

@@ -0,0 +1,187 @@
+import { DictationError } from "../../utils/errors/dictation";
+
+/**
+ * The most audio the buffered fallback keeps in memory: the size cap of the
+ * buffered endpoint, `POST /v1/transcribe` (256 MB). A copy past it could
+ * never be accepted there, so it is dropped rather than grown further.
+ */
+export const maxBufferedAudioBytes = 256 * 1024 * 1024;
+
+/**
+ * The error a fallback throws when the recording outgrows
+ * `maxBufferedAudioBytes`: the same status and code the buffered endpoint
+ * itself answers with for audio over its size cap.
+ */
+function overflowError(): DictationError {
+  return new DictationError(
+    `audio exceeds ${maxBufferedAudioBytes} bytes (256 MB), the most the ` +
+      "buffered dictation fallback keeps",
+    413,
+    "audio_too_large",
+  );
+}
+
+/**
+ * Rejects with the signal's reason once it aborts; never settles otherwise.
+ * The listener is removed by `dispose()`.
+ */
+function abortion(signal: AbortSignal): {
+  promise: Promise<never>;
+  dispose: () => void;
+} {
+  let dispose = () => undefined as void;
+  const promise = new Promise<never>((_, reject) => {
+    const onAbort = () =>
+      reject(signal.reason ?? new Error("The operation was aborted."));
+    signal.addEventListener("abort", onAbort, { once: true });
+    dispose = () => signal.removeEventListener("abort", onAbort);
+  });
+  // Only ever observed through Promise.race; never an unhandled rejection.
+  promise.catch(() => undefined);
+  return { promise, dispose };
+}
+
+/**
+ * The caller's audio source, with a copy kept of every chunk it produces.
+ *
+ * The live upload reads the source through `chunks`; the copy holds the same
+ * bytes from the first one on, so that when the live request is refused for
+ * being too long the whole recording can be sent again in one buffered
+ * request. Only the audio is kept, never the multipart framing.
+ *
+ * The source has exactly one reader at a time: a chunk the live upload asked
+ * for but gave up waiting on (when its body is ended early) is still
+ * recorded, and is the first one `drain()` sees.
+ * @internal
+ */
+export class AudioRecording {
+  private readonly iterator: AsyncIterator<Uint8Array>;
+  private recorded: Uint8Array[] = [];
+  private recordedBytes = 0;
+  private pending: Promise<IteratorResult<Uint8Array>> | undefined;
+  private ended = false;
+  private overflowed = false;
+  private released = false;
+
+  /**
+   * The source as the live upload reads it. Ending this view early (the body
+   * does so when the server answers mid-upload) leaves the source itself
+   * open: whether it is released or drained is decided once the response is
+   * known, by `release()` or `drain()`.
+   */
+  readonly chunks: AsyncIterable<Uint8Array>;
+
+  /**
+   * @param source - The caller's audio, already coerced to bytes.
+   */
+  constructor(source: AsyncIterable<Uint8Array>) {
+    this.iterator = source[Symbol.asyncIterator]();
+    this.chunks = {
+      [Symbol.asyncIterator]: () => ({
+        next: () => this.next(),
+        return: async () => ({ done: true as const, value: undefined }),
+      }),
+    };
+  }
+
+  /**
+   * The next chunk from the source, recorded before it is handed out. Calls
+   * made while one is in flight share it, so no chunk is read twice or lost.
+   */
+  private next(): Promise<IteratorResult<Uint8Array>> {
+    if (this.ended) return Promise.resolve({ done: true, value: undefined });
+    if (!this.pending) {
+      this.pending = this.iterator.next().then(
+        (result) => {
+          this.pending = undefined;
+          if (result.done) this.ended = true;
+          else this.record(result.value);
+          return result;
+        },
+        (error: unknown) => {
+          this.pending = undefined;
+          this.ended = true;
+          throw error;
+        },
+      );
+    }
+    return this.pending;
+  }
+
+  private record(chunk: Uint8Array): void {
+    if (chunk.byteLength === 0 || this.released) return;
+    this.recordedBytes += chunk.byteLength;
+    if (this.overflowed) return;
+    if (this.recordedBytes > maxBufferedAudioBytes) {
+      // Past the buffered endpoint's cap the copy is useless; let it go now
+      // rather than holding hundreds of megabytes until the request ends.
+      this.overflowed = true;
+      this.recorded = [];
+      return;
+    }
+    // Keep a private copy: a producer may reuse its buffer for the next chunk.
+    this.recorded.push(chunk.slice());
+  }
+
+  /**
+   * Read the source to its end, recording every chunk: the rest of the
+   * recording, after the live request was refused. Called once the live
+   * upload has stopped reading.
+   * @param signal - Aborts the drain; the source is released and the copy
+   * dropped.
+   * @returns A promise that resolves once the source has ended.
+   * @throws DictationError (413, `audio_too_large`) when the copy passes
+   * `maxBufferedAudioBytes`.
+   * @throws Whatever the source throws, unchanged, or the signal's reason.
+   */
+  async drain(signal?: AbortSignal): Promise<void> {
+    const aborted = signal ? abortion(signal) : undefined;
+    try {
+      for (;;) {
+        if (signal?.aborted) {
+          throw signal.reason ?? new Error("The operation was aborted.");
+        }
+        if (this.overflowed) throw overflowError();
+        const next = this.next();
+        const result = aborted
+          ? await Promise.race([next, aborted.promise])
+          : await next;
+        if (result.done) break;
+      }
+      // An abort can end the source (a session's abort() ends its queue)
+      // in the same breath as it fires the signal; it is still an abort.
+      if (signal?.aborted) {
+        throw signal.reason ?? new Error("The operation was aborted.");
+      }
+      if (this.overflowed) throw overflowError();
+    } catch (error) {
+      this.release();
+      throw error;
+    } finally {
+      aborted?.dispose();
+    }
+  }
+
+  /**
+   * Hand over the recorded audio, in order, and forget it here.
+   * @returns The recorded chunks.
+   * @throws DictationError (413, `audio_too_large`) when the copy passed
+   * `maxBufferedAudioBytes`.
+   */
+  take(): Uint8Array[] {
+    if (this.overflowed) throw overflowError();
+    const recorded = this.recorded;
+    this.recorded = [];
+    return recorded;
+  }
+
+  /**
+   * Let go of the source and drop the copy. A source still producing is
+   * stopped, as the live upload would have stopped it.
+   */
+  release(): void {
+    this.released = true;
+    this.recorded = [];
+    void this.iterator.return?.()?.catch(() => undefined);
+  }
+}