feat(dialog): a popup inside the dialog owns Tab and Escape while it holds focus

.changeset/dialog-inner-popups.md

@@ -0,0 +1,15 @@
+---
+'@dunky.dev/dom-dialog': patch
+---
+
+A popup opened inside the dialog owns Tab and Escape while it holds focus.
+
+A select menu, a popover, a menu, or a combobox list inside a modal dialog —
+from any library, as long as it carries ARIA popup semantics — is not a layer
+the stack knows, so the dialog kept treating itself as topmost: its
+capture-phase Escape closed the dialog together with the popup, and its focus
+trap cancelled Tab inside the popup and pulled focus back into the window. Both
+now stand down while such a popup holds focus and resume once focus is back in
+the window, so one Escape closes the popup and the next reaches the dialog. A
+control whose popup is expanded (a combobox input) hands over Escape only; Tab
+is how its popup is left, so the trap keeps it.

.changeset/overlay-popup-focus.md

@@ -0,0 +1,19 @@
+---
+'@dunky.dev/dom-overlay': minor
+---
+
+Two queries tell a layer when a popup it cannot see owns the keyboard:
+`foreignPopupHoldsFocus(id)` and `expandedPopupControlHoldsFocus(id)`.
+
+A popup can hold focus inside a layer without ever registering in the stack —
+a third-party listbox or menu, a combobox's list. The stack still names the
+layer topmost, so the layer would keep answering Escape and trapping Tab under
+the popup. The queries read the real owner from ARIA instead: focus in an
+element with a popup role (`aria-haspopup`'s values) that is neither the
+layer's window nor a registered layer, or on a control inside the window whose
+popup is expanded (`aria-expanded` with `aria-haspopup`). No cooperation from
+the popup is required.
+
+```ts
+enabled: () => isTopmostLayer(id) && !foreignPopupHoldsFocus(id)
+```

packages/core/dialog/SPEC.md

@@ -150,6 +150,12 @@ Per the [APG modal-dialog keyboard interaction](https://www.w3.org/WAI/ARIA/apg/
   always the cycle's last stop, wherever it renders — the dismissal
   affordance follows the content instead of interrupting it. Focus never tabs
   out of the dialog.
+- **Inner popups**: a popup opened inside the dialog that is not itself a
+  dialog — a select menu, a popover, a menu, a combobox list, from this
+  library or any other — owns the keyboard while it holds focus: Tab stays in
+  it and Escape closes it alone. The dialog's trap and Escape resume once
+  focus is back in the window. The dialog recognizes the popup by its ARIA
+  popup semantics, so no cooperation from the popup is needed.
 - **No focusables**: Tab is a no-op; focus stays on the dialog window.
 - **On close**: focus returns to the element focused before opening (normally
   the Trigger).
@@ -170,7 +176,8 @@ stack of dialogs only the topmost one exists until it closes.
 - **Escape**: lands only on the topmost dialog, subject to that dialog's own
   dismissal settings and veto. Its reach is that dialog's escape scope: one
   layer (the default — the stack unwinds one layer per press) or the whole
-  stack.
+  stack. An inner popup (select menu, popover, menu, combobox list) counts as
+  a layer of its own: one press closes it, the next reaches the dialog.
 - **Outside press**: pressing around the topmost dialog is an outside
   interaction for that dialog alone, following its own dismissal settings; the
   dialogs beneath are unaffected.

packages/dom/components/dialog/SPEC.md

@@ -44,6 +44,14 @@ answer wherever focus is. Only the topmost layer answers, and it offers the
 consumer's `onEscapeKeyDown` a veto through `preventDefault` before it moves
 the machine.
 
+A popup inside the dialog that never joined the stack — a listbox or menu from
+another library, a control whose popup is expanded — answers before the dialog
+while it holds focus: the listener stands down until focus is back in the
+window, so one press closes the popup and the next reaches the dialog. The
+stack cannot name such a layer, so the answer is read from ARIA through the
+overlay util's focus queries (`foreignPopupHoldsFocus`,
+`expandedPopupControlHoldsFocus`).
+
 How far an allowed Escape reaches is that dialog's `escapeScope`: itself, so a
 nested stack unwinds one dialog per press, or the whole stack at once. Either
 way the dialog that received the press is the only one that gates or vetoes
@@ -129,7 +137,10 @@ dialog's to answer:
 
 `dialogTrapOptions` is the trap configuration the substrate hands to its
 `trapFocus` wrapper: a modal dialog traps while it is topmost, and the Close
-part is the cycle's last stop wherever it renders.
+part is the cycle's last stop wherever it renders. The trap stands down while
+an unregistered popup inside the dialog holds focus — Tab is the popup's for
+as long as it does. A control whose popup is expanded keeps the trap: focus is
+still in the window, and Tab is how such a popup is left.
 
 ## API
 

packages/dom/components/dialog/src/effects.ts

@@ -1,5 +1,10 @@
 import { dialogEffects, type DialogEffect } from '@dunky.dev/dialog'
-import { isTopmostLayer, layersBelow } from '@dunky.dev/dom-overlay'
+import {
+  expandedPopupControlHoldsFocus,
+  foreignPopupHoldsFocus,
+  isTopmostLayer,
+  layersBelow,
+} from '@dunky.dev/dom-overlay'
 
 // Escape is a document-level concern, not a part's — it must work wherever
 // focus is.
@@ -9,7 +14,12 @@ const trackEscape: DialogEffect = [
       if (event.key !== 'Escape' || !machine.matches('open')) return
       // Only the topmost dialog answers Escape — a nested stack closes one
       // layer at a time, unless this dialog's scope is the whole stack.
-      if (!isTopmostLayer(machine.context.id)) return
+      const { id } = machine.context
+      if (!isTopmostLayer(id)) return
+      // A popup inside the dialog that never joined the stack answers first
+      // while it holds focus — this listener runs in the capture phase, so
+      // it would otherwise close the dialog under the popup.
+      if (foreignPopupHoldsFocus(id) || expandedPopupControlHoldsFocus(id)) return
       props.onEscapeKeyDown?.(event)
       if (event.defaultPrevented) return
       // Read the stack before the send: closing this layer releases it, and

packages/dom/components/dialog/src/focus-trap.ts

@@ -1,6 +1,6 @@
 import type { DialogMachine } from '@dunky.dev/dialog'
 import type { TrapFocusOptions } from '@dunky.dev/dom-focus-trap'
-import { isTopmostLayer } from '@dunky.dev/dom-overlay'
+import { foreignPopupHoldsFocus, isTopmostLayer } from '@dunky.dev/dom-overlay'
 
 /**
  * The trap configuration for a dialog window, for whichever hook the substrate
@@ -11,8 +11,12 @@ import { isTopmostLayer } from '@dunky.dev/dom-overlay'
 export function dialogTrapOptions(machine: DialogMachine, closeId: () => string): TrapFocusOptions {
   return {
     // Only a modal dialog traps, and only while topmost — a nested dialog
-    // owns focus while open.
-    enabled: () => machine.context.modal && isTopmostLayer(machine.context.id),
+    // owns focus while open, and so does a popup inside the dialog that holds
+    // it without having joined the stack.
+    enabled: () =>
+      machine.context.modal &&
+      isTopmostLayer(machine.context.id) &&
+      !foreignPopupHoldsFocus(machine.context.id),
     // The Close part is the cycle's last stop wherever it renders (core
     // SPEC); found by its derived id.
     last: () => document.getElementById(closeId()),

packages/dom/components/dialog/tests/dialog.test.ts

@@ -124,6 +124,36 @@ describe('domDialogEffects — Escape', () => {
     pressEscape()
     expect(service.matches('open')).toBe(true)
   })
+
+  // A popup inside the dialog owns Escape while it holds focus — read from
+  // ARIA, so a popup that never registers in the stack counts too.
+  it('stands down while a popup inside the dialog holds focus', () => {
+    const service = build({ defaultOpen: true })
+    const content = mountLayer(
+      'dlg',
+      1,
+      '<ul role="listbox"><li id="option" role="option" tabindex="0">a</li></ul>',
+    )
+    ;(content.querySelector('#option') as HTMLElement).focus()
+    armEscape(service)
+
+    pressEscape()
+    expect(service.matches('open')).toBe(true)
+  })
+
+  it('stands down while a control with an expanded popup holds focus', () => {
+    const service = build({ defaultOpen: true })
+    const content = mountLayer(
+      'dlg',
+      1,
+      '<input id="combo" role="combobox" aria-haspopup="listbox" aria-expanded="true" />',
+    )
+    ;(content.querySelector('#combo') as HTMLElement).focus()
+    armEscape(service)
+
+    pressEscape()
+    expect(service.matches('open')).toBe(true)
+  })
 })
 
 describe('openDialogLayer', () => {
@@ -440,6 +470,21 @@ describe('dialogTrapOptions', () => {
     expect(enabled?.()).toBe(false)
   })
 
+  it('stands down while a popup inside the dialog holds focus', () => {
+    const service = build({ defaultOpen: true })
+    const content = mountLayer(
+      'dlg',
+      1,
+      '<ul role="listbox"><li id="option" role="option" tabindex="0">a</li></ul>',
+    )
+    const { enabled } = dialogTrapOptions(service, () => 'dlg-close')
+
+    ;(content.querySelector('#option') as HTMLElement).focus()
+    expect(enabled?.()).toBe(false)
+    content.focus()
+    expect(enabled?.()).toBe(true)
+  })
+
   it('resolves Close as the cycle’s last stop, wherever it renders', () => {
     const service = build({ defaultOpen: true })
     mountLayer('dlg', 1, '<button id="dlg-close">close</button>')

packages/dom/utils/overlay/SPEC.md

@@ -54,6 +54,30 @@ against each other.
 - The backdrop is resolved through a getter, not a snapshot: a re-hide (a
   layer above closing) sees the element current at that moment.
 
+### Popups the stack never sees
+
+A popup can hold focus inside a layer without registering — a listbox or menu
+from another library, a combobox's list. The stack still names the layer
+topmost, so the layer would keep answering Escape and trapping Tab under the
+popup. Two queries read the real owner from ARIA instead, and a layer consults
+them to stand down:
+
+- `foreignPopupHoldsFocus(id)` — focus sits in a popup that is neither the
+  layer's own window nor a registered layer: an element with a popup role
+  (`aria-haspopup`'s values — listbox, menu, tree, grid, dialog), wherever it
+  renders, or anything outside the window that is in no popup role at all —
+  the page is inert while a modal layer is open, so whatever holds focus out
+  there is a layer. Focus on the body doesn't count: the layer re-enters from
+  there. Registered layers never count as foreign — a layer beneath is inert,
+  and focus reported there re-enters the topmost layer's trap.
+- `expandedPopupControlHoldsFocus(id)` — focus sits on a control inside the
+  window whose popup is expanded (`aria-expanded="true"` with `aria-haspopup`),
+  the way a combobox keeps focus on its input while its list is open.
+  `aria-expanded` alone is a disclosure, which has no popup to close.
+
+No cooperation from the popup is required — any well-formed ARIA popup works.
+A popup that does register is simply topmost, and the stack answers as usual.
+
 ### Initial focus
 
 The strict rule is only that focus moves into the overlay: an overlay that
@@ -97,6 +121,8 @@ again — but keeps painting until its exit visual finishes:
 | `Layer`                                          | `OverlayLayer` + `element`, `modal`, an optional `backdrop` getter, and an optional `dismiss`.                                    |
 | `isTopmostLayer(id)`                             | Whether the layer owns Escape and the focus trap right now.                                                                       |
 | `layersBelow(id)`                                | The layers stacked beneath, topmost first — the unwinding order for a stack-scoped dismissal.                                     |
+| `foreignPopupHoldsFocus(id)`                     | Whether focus sits in a popup that is neither the layer's window nor a registered layer.                                          |
+| `expandedPopupControlHoldsFocus(id)`             | Whether focus sits on a control inside the layer's window whose popup is expanded.                                                |
 | `getInitialFocus(content, designated?)`          | The element to focus on open: `designated`, else first form field, else the overlay window — each step filtered for renderedness. |
 | `hideExitingLayer(content, boundary, backdrop?)` | Inerts the still-painting layer for the exit window; returns the undo.                                                            |
 | `watchExitAnimation(element, onComplete)`        | Reports the exit visual's end once; returns the cancel.                                                                           |

packages/dom/utils/overlay/src/index.ts

@@ -1,4 +1,5 @@
 export { registerLayer, isTopmostLayer, layersBelow, type Layer } from './stack'
+export { foreignPopupHoldsFocus, expandedPopupControlHoldsFocus } from './popup-focus'
 export { getInitialFocus } from './get-initial-focus'
 export { watchExitAnimation } from './watch-exit-animation'
 export { hideExitingLayer } from './hide-exiting-layer'

packages/dom/utils/overlay/src/popup-focus.ts

@@ -0,0 +1,46 @@
+import { getLayer, layerContaining } from './stack'
+
+// A popup can hold focus inside a layer without ever joining the stack — a
+// listbox or menu from another library, a combobox's list — so the stack still
+// names the layer topmost while the popup owns the keyboard. Ownership is read
+// from ARIA instead: the popup roles (`aria-haspopup`'s values) mark the layer
+// focus sits in, and the stack tells a registered layer from a popup that
+// never registered. No cooperation is required — any well-formed ARIA popup
+// works.
+const POPUP_SELECTOR =
+  '[role="listbox"], [role="menu"], [role="tree"], [role="grid"], [role="dialog"], [role="alertdialog"]'
+
+/**
+ * Whether focus sits in a popup that is neither the layer's own window nor a
+ * registered layer — wherever it renders, inside the window or portalled
+ * beside it. Outside every registered window and in no popup role counts too:
+ * the page is inert while a modal layer is open, so whatever holds focus out
+ * there is a layer. The body (focus in browser chrome, or nowhere) doesn't: the
+ * layer re-enters from there.
+ */
+export function foreignPopupHoldsFocus(id: string): boolean {
+  const layer = getLayer(id)
+  if (layer === undefined) return false
+  const active = document.activeElement
+  if (active === null || active === document.body) return false
+  const popup = active.closest(POPUP_SELECTOR)
+  if (popup === null) return layerContaining(active) === undefined
+  return popup !== layer.element && layerContaining(popup)?.element !== popup
+}
+
+/**
+ * Whether focus sits on a control inside the layer's window whose popup is
+ * expanded — a combobox keeps focus on its input while its list is open — so
+ * the popup owns Escape. `aria-expanded` alone isn't enough: a disclosure or
+ * accordion trigger is expanded too and has no popup to close, so
+ * `aria-haspopup` must name one.
+ */
+export function expandedPopupControlHoldsFocus(id: string): boolean {
+  const layer = getLayer(id)
+  if (layer === undefined) return false
+  const active = document.activeElement
+  if (active === null || !layer.element.contains(active)) return false
+  if (active.getAttribute('aria-expanded') !== 'true') return false
+  const popup = active.getAttribute('aria-haspopup')
+  return popup !== null && popup !== 'false'
+}

packages/dom/utils/overlay/src/stack.ts

@@ -102,3 +102,19 @@ export function isTopmostLayer(id: string): boolean {
 export function layersBelow(id: string): Layer[] {
   return getStore().stack.below(id)
 }
+
+export function getLayer(id: string): Layer | undefined {
+  for (const layer of getStore().stack.ordered()) {
+    if (layer.id === id) return layer
+  }
+  return undefined
+}
+
+// The registered layer whose window holds `node` — what tells a sibling in
+// the stack from a popup that never registered.
+export function layerContaining(node: Node): Layer | undefined {
+  for (const layer of getStore().stack.ordered()) {
+    if (layer.element.contains(node)) return layer
+  }
+  return undefined
+}