Track execution contexts across navigations and wait for a frame's next one

docs/architecture/transport-and-protocol.md

@@ -101,6 +101,8 @@ Three kinds of target get a session of their own:
 
 **Execution context IDs are numbered per session.** Two renderers will both hand out context 1, so the map from context to frame is keyed by session and context together. A frame's context ID is only meaningful on the session that owns that frame, and every call site pairs the two.
 
+**A frame's context ID is replaced on every navigation.** `Runtime.executionContextCreated` records the main world of the frame named in its `auxData` (an isolated world, with `isDefault: false`, never replaces it), `Runtime.executionContextDestroyed` drops the entry with that ID from the session that reported it, and `Runtime.executionContextsCleared` drops every entry that session reported. The three are read from one channel per session, so a navigation's teardown is always handled before the context that replaces it. They do not share a channel with `Page.loadEventFired`, so a navigation can complete while the old context is still recorded or before the new one is. Commands that name a context cover both cases. A frame with nothing recorded waits for its next context, bounded by the caller's timeout or 30 seconds, and fails at once with `MotusTargetClosedException` if the frame is detached or the page closes; the main frame instead leaves the ID out, which the browser resolves to the main frame's current main world. A command refused with "Cannot find context with specified id" is sent once more against whatever has been recorded since. The browser looks the context up before running anything, so the refused attempt never ran and the retry cannot run a script twice. BiDi has no counterpart: script there is addressed to a browsing context and runs in whatever realm it currently holds, so there is no ID to track or wait for.
+
 ### Routing a frame to its session
 
 `Page.SessionFor(IFrame?)` is the single lookup. A frame that owns a session is routed to it; every other frame, including a same-process child of an out-of-process frame, falls through to the page's own session. That fallback is what keeps every existing round trip byte-identical.

docs/guides/frames-and-iframes.md

@@ -90,7 +90,9 @@ if (frame.IsDetached)
 }
 ```
 
-`IsDetached` is how a removed frame is told apart from a selector that simply did not match. A held `IFrame` answers rather than failing obscurely several calls later.
+`IsDetached` is how a removed frame is told apart from a selector that simply did not match. A held `IFrame` answers rather than failing obscurely several calls later. Evaluating, reading content or resolving a locator through a detached frame throws `MotusTargetClosedException` straight away, and so does a call that was already waiting on the frame when it went.
+
+An evaluation, a locator or a content read issued the moment a frame finishes navigating runs in the new document. A frame's main world is replaced on every navigation and the browser reports the new one separately from the end of the navigation, so for a frame other than the main frame such a call waits for the new world to be reported, for up to 30 seconds or the call's own timeout, rather than running against the old document or in the main frame by mistake.
 
 ---
 
@@ -159,7 +161,7 @@ Each of these costs real time to rediscover.
 
 **The endpoint's target list is not a list of a page's frames.** `http://127.0.0.1:9222/json/list` reports one entry per CDP target, and only a frame the browser put in its own process has a target of its own. So the list shows those, typed `iframe` rather than `page`, and shows nothing at all for every frame the page's own renderer hosts. A document with two frames, one cross-origin and one `srcdoc`, produces exactly one `iframe` entry. Counting entries there will undercount the frames on the page, and filtering for `page` entries will miss all of them. Motus builds `page.Frames` from auto-attach and the frame tree instead, which covers both kinds, so read frames from there and treat the target list as a view of processes rather than of documents.
 
-**A frame changing process is not a frame going away, but a frame whose parent navigated is.** The browser announces a frame moved into another renderer process as a detach carrying the reason `swap`, and says the same for a document it puts into the back-forward cache. Motus tells both apart from a real removal: the frame stays in `page.Frames`, `IsDetached` stays false, no `FrameDetached` arrives, and the `IFrame` being held keeps working across the move. Navigation is the other case. Committing a new document in a frame drops the frames that frame hosted, because the new document reports its own, so a child frame is rediscovered after its parent navigates and comes back as a different `IFrame`, even when it loads the same URL. What that leaves to watch for is the handle from before. It reports `IsDetached`, and checking that is worth the line, because a frame the page no longer lists has no execution context of its own to aim at and an evaluation through it falls back to the page's, quietly answering for the main document rather than failing. Read the frame from `page.Frames` again after a navigation and wait for it as under [Traversing](#traversing), since for a moment after the parent commits its new document the subtree is empty.
+**A frame changing process is not a frame going away, but a frame whose parent navigated is.** The browser announces a frame moved into another renderer process as a detach carrying the reason `swap`, and says the same for a document it puts into the back-forward cache. Motus tells both apart from a real removal: the frame stays in `page.Frames`, `IsDetached` stays false, no `FrameDetached` arrives, and the `IFrame` being held keeps working across the move. Navigation is the other case. Committing a new document in a frame drops the frames that frame hosted, because the new document reports its own, so a child frame is rediscovered after its parent navigates and comes back as a different `IFrame`, even when it loads the same URL. What that leaves to watch for is the handle from before. It reports `IsDetached`, and anything run through it throws `MotusTargetClosedException` rather than answering for some other document. Read the frame from `page.Frames` again after a navigation and wait for it as under [Traversing](#traversing), since for a moment after the parent commits its new document the subtree is empty.
 
 **Application globals are absent from isolated worlds.** An isolated world shares the frame's document and nothing else, so the DOM is fully present while everything the application put on `window` is not. An expression that reads a framework handle returns `undefined` there and the failure looks like the application not having loaded. Use `ExecutionWorld.Main`, which is the default, whenever you are reading application state.
 

src/Motus.Mcp/Snapshot/FrameCollector.cs

@@ -32,6 +32,10 @@ internal static class FrameCollector
     private const string SelfIndexExpression =
         "(() => { const p = window.parent; for (let i = 0; i < p.length; i++) { if (p[i] === window) return i; } return -1; })()";
 
+    // How long the frames of one document get to say which of their parent's contexts they are.
+    // A frame with a document answers in milliseconds.
+    private static readonly TimeSpan SelfIndexAllowance = TimeSpan.FromSeconds(1);
+
     // Which child browsing context this element holds, asked of the document the element lives in.
     private const string HostIndexExpression =
         "el => { const w = el.ownerDocument.defaultView; for (let i = 0; i < w.length; i++) { if (w[i] === el.contentWindow) return i; } return -1; }";
@@ -182,24 +186,32 @@ void Walk(IFrame frame)
     /// Frames that cannot answer, and any index two frames both claim, are left out, so a reading
     /// that is not trustworthy falls through to the order the engine gives instead of guessing.
     /// </summary>
+    /// <remarks>
+    /// A frame that has not loaded a document yet, such as a lazy iframe still below the fold, has
+    /// no context to evaluate in, and an evaluation there waits for one until its own timeout. So
+    /// every frame is asked at once and the answers are read after <see cref="SelfIndexAllowance"/>
+    /// at most: a frame that cannot answer costs the snapshot that allowance once, rather than a
+    /// full evaluation timeout for each such frame.
+    /// </remarks>
     private static async Task<IReadOnlyDictionary<int, IFrame>> ByContextIndexAsync(
         IReadOnlyList<IFrame> children)
     {
         var byContext = new Dictionary<int, IFrame>();
         var duplicates = new HashSet<int>();
 
-        foreach (var child in children)
+        var asks = children.Select(AskSelfIndex).ToList();
+        await Task.WhenAny(Task.WhenAll(asks), Task.Delay(SelfIndexAllowance)).ConfigureAwait(false);
+
+        for (var i = 0; i < children.Count; i++)
         {
-            int position;
-            try
-            {
-                position = await child.EvaluateAsync<int>(SelfIndexExpression).ConfigureAwait(false);
-            }
-            catch (Exception)
-            {
+            var child = children[i];
+            var ask = asks[i];
+
+            // Still waiting, or failed: either way this frame can't say where it belongs.
+            if (!ask.IsCompletedSuccessfully)
                 continue;
-            }
 
+            var position = ask.Result;
             if (position < 0)
                 continue;
 
@@ -213,6 +225,19 @@ private static async Task<IReadOnlyDictionary<int, IFrame>> ByContextIndexAsync(
         return byContext;
     }
 
+    // Never faults, so an answer that arrives after the allowance has nobody left to observe it.
+    private static async Task<int> AskSelfIndex(IFrame child)
+    {
+        try
+        {
+            return await child.EvaluateAsync<int>(SelfIndexExpression).ConfigureAwait(false);
+        }
+        catch (Exception)
+        {
+            return -1;
+        }
+    }
+
     /// <summary>
     /// The backend node ids of the frame elements in a document, in document order.
     /// </summary>

src/Motus/Page/Page.Evaluation.cs

@@ -15,20 +15,27 @@ public async Task<T> EvaluateAsync<T>(string expression, object? arg = null)
 
     public async Task<IJSHandle> EvaluateHandleAsync(string expression, object? arg = null)
     {
-        var contextId = _mainFrameId is not null
-            ? GetExecutionContextId(_mainFrameId)
-            : null;
-
-        var result = await _session.SendAsync(
-            "Runtime.evaluate",
-            new RuntimeEvaluateParams(
-                Expression: WrapExpression(expression, arg),
-                ReturnByValue: false,
-                AwaitPromise: true,
-                ContextId: contextId),
-            CdpJsonContext.Default.RuntimeEvaluateParams,
-            CdpJsonContext.Default.RuntimeEvaluateResult,
-            CancellationToken.None).ConfigureAwait(false);
+        var wrapped = WrapExpression(expression, arg);
+
+        async Task<(IMotusSession Session, RuntimeEvaluateResult Result)> Evaluate(ExecutionTarget target)
+        {
+            var evaluated = await target.Session.SendAsync(
+                "Runtime.evaluate",
+                new RuntimeEvaluateParams(
+                    Expression: wrapped,
+                    ReturnByValue: false,
+                    AwaitPromise: true,
+                    ContextId: target.ContextId),
+                CdpJsonContext.Default.RuntimeEvaluateParams,
+                CdpJsonContext.Default.RuntimeEvaluateResult,
+                CancellationToken.None).ConfigureAwait(false);
+
+            return (target.Session, evaluated);
+        }
+
+        var (session, result) = _mainFrameId is not null
+            ? await RunInFrameAsync(_mainFrameId, Evaluate).ConfigureAwait(false)
+            : await Evaluate(new ExecutionTarget(_session, null)).ConfigureAwait(false);
 
         if (result.ExceptionDetails is not null)
             throw new InvalidOperationException(
@@ -38,7 +45,7 @@ public async Task<IJSHandle> EvaluateHandleAsync(string expression, object? arg
             throw new InvalidOperationException(
                 "Expression did not return an object. Use EvaluateAsync<T> for primitive values.");
 
-        return new JsHandle(_session, result.Result.ObjectId);
+        return new JsHandle(session, result.Result.ObjectId);
     }
 
     public async Task<T> WaitForFunctionAsync<T>(string expression, object? arg = null, double? timeout = null)

src/Motus/Page/Page.Events.cs

@@ -97,10 +97,11 @@ internal void StartFrameStructureEventPump(IMotusSession session, CancellationTo
             CdpJsonContext.Default.PageFrameStoppedLoadingEvent,
             evt => FrameStoppedLoading?.Invoke(evt.FrameId), ct);
 
-        _ = PumpEventsAsync(session,
-            "Runtime.executionContextCreated",
-            CdpJsonContext.Default.RuntimeExecutionContextCreatedEvent,
-            OnExecutionContextCreated, ct);
+        // Contexts are made and torn down by the same navigations, so their events are read
+        // together for the same reason as the frame tree's. BiDi has no counterpart to listen for
+        // (see PumpExecutionContextEventsAsync).
+        if (session is not BiDiSession)
+            _ = PumpExecutionContextEventsAsync(session, ct);
 
         _ = PumpEventsAsync(session,
             "Runtime.consoleAPICalled",
@@ -291,6 +292,10 @@ private void DetachFrame(string frameId)
             return;
 
         frame.MarkDetached();
+
+        // Anyone waiting for this frame's next context would otherwise wait out the clock.
+        SignalExecutionContextWaiters(frameId);
+
         FrameDetached?.Invoke(this, frame);
     }
 
@@ -310,34 +315,6 @@ private void RecordFrameOwnership(string frameId, IMotusSession source, bool isP
             _frameIdToSession[frameId] = source;
     }
 
-    private void OnExecutionContextCreated(RuntimeExecutionContextCreatedEvent evt)
-    {
-        var ctx = evt.Context;
-
-        // Extract frameId from auxData if present
-        string? frameId = null;
-        var isDefaultWorld = true;
-        if (ctx.AuxData is JsonElement aux && aux.ValueKind == JsonValueKind.Object)
-        {
-            if (aux.TryGetProperty("frameId", out var fid))
-                frameId = fid.GetString();
-
-            // A frame has one main world and any number of others beside it. Only the main world
-            // is the frame's context: an isolated world reports the same frame id, and recording
-            // it here would send every later main-world evaluation into the isolated world, where
-            // the page's own globals do not exist. That failure is silent, so it is guarded at the
-            // one place contexts are recorded rather than at each reader.
-            if (aux.TryGetProperty("isDefault", out var isDefault)
-                && isDefault.ValueKind is JsonValueKind.False)
-            {
-                isDefaultWorld = false;
-            }
-        }
-
-        if (frameId is not null && isDefaultWorld)
-            _frameIdToExecutionContext[frameId] = ctx.Id;
-    }
-
     private void OnConsoleApiCalled(RuntimeConsoleApiCalledEvent evt)
     {
         var text = string.Join(" ", evt.Args.Select(a =>