Fix OG Viewer failing to load pages over the 6 MiB fetch cap

.github/extensions/og-preview/README.md

@@ -11,6 +11,10 @@ platforms, alongside a raw OpenGraph metadata view and a diagnostics checklist.
 - **Raw metadata** — every `og:*`, `twitter:*`, and other `<meta>` tag, grouped,
   with per-value quick **copy buttons** and a one-click **Copy JSON**.
 - **Diagnostics** — checks for the required/recommended OpenGraph tags.
+- **Prompt composer** — a **Prompt** tab that turns the page into a
+  ready-to-paste AI prompt: a full JSON snapshot of its SEO and social metadata
+  plus task modules you can mix and match. See
+  [Prompt composer](#prompt-composer).
 - **Collapsible page-info footer** — final URL, HTTP status, tag count, and
   diagnostics summary; expanded/collapsed state is remembered.
 - **Quick examples** — one-tap chip to preview `aspire.dev`.
@@ -33,12 +37,18 @@ port) that serves the static UI from `ui/` and a JSON API:
 | Route | Purpose |
 | --- | --- |
 | `GET /` | Renderer page (auto-loads `?u=<url>`) |
+| `GET /build-prompt.mjs` | Prompt composer module, shared with `get_prompt` |
 | `GET /api/fetch?u=` | Fetch + parse the target, return metadata JSON |
 | `GET /api/img?u=` | Image proxy fallback |
 | `GET /events` | Server-Sent Events; agent-driven loads are pushed here |
 
 The target page is fetched and parsed server-side (no external dependencies),
-which sidesteps browser CORS and lets it reach `localhost`.
+which sidesteps browser CORS and lets it reach `localhost`. Page and resource
+fetches are capped at 6 MiB: an oversized HTML page is truncated rather than
+rejected (its metadata lives in the head), while oversized scripts, styles,
+images, and other files fail to load. The Browse tab runs the whole document,
+and large API reference pages put their interactive content after several MB of
+inline data, so the page it loads gets a 16 MiB cap; past that it is cut off.
 
 ## Network access and isolation
 
@@ -79,6 +89,84 @@ node --test .github\extensions\og-preview\tests\*.test.mjs
   and return its resolved preview fields.
 - **`get_metadata`** `{ url }` *(canvas action)* — fetch + parse a URL and return
   all raw metadata as JSON, without opening the canvas.
+- **`get_prompt`** `{ url, modules?, instructions?, includeAgentReadiness? }`
+  *(canvas action)* — build the composed prompt and its JSON payload for a URL,
+  without loading the page into the canvas. It returns the `prompt`, the JSON
+  `payload`, the final `url`, and the `modules` used. See
+  [Prompt composer](#prompt-composer).
+
+## Prompt composer
+
+The **Prompt** tab (and the `get_prompt` action) turns the loaded page into one
+prompt for any AI assistant: a short context block, the tasks you pick, your own
+notes, and a JSON snapshot of the page's SEO and social metadata. Page content is
+untrusted, so the prompt tells the assistant to treat the JSON as data and never
+follow instructions found inside it.
+
+In the tab you can:
+
+- **Pick tasks** — the selection only changes the task text; the JSON is always
+  the same. Audit and Fix head tags are on by default.
+- **Add instructions** — optional notes (up to 8,000 characters) that are added
+  to the prompt. Your tasks and notes are remembered in the browser.
+- **Check the size** — the prompt's length in characters and an estimated token
+  count.
+- **Copy prompt** or **Copy JSON** — the JSON is the same snapshot that ends the
+  prompt.
+- **Open in Copilot** or **Create issue** — shown when the page links to its
+  source GitHub repository (for example, an "Edit this page" link). They send
+  the current prompt to a new Copilot session or to an issue in that repository,
+  like the Diagnostics tab's actions.
+
+The agent-readiness checks run after the page loads and join the snapshot when
+they finish; until then `agentReadiness` is `null`.
+
+| Task module | Tab label | What it asks for |
+| --- | --- | --- |
+| `audit` | Audit | Find SEO and social-sharing issues |
+| `fix-tags` | Fix head tags | Exact markup for the issues found |
+| `copy` | Rewrite copy | Better title and description options |
+| `social-image` | Share image | Review `og:image` and the Twitter card |
+| `structured-data` | Structured data | Review and fix JSON-LD |
+| `agent-readiness` | Agent readiness | AI crawler and agent discovery |
+| `implement` | Implement | Make the edits in the source repo |
+| `verify` | Verify | Re-check the page after the change |
+
+`get_prompt` takes `url` plus optional `modules` (the ids above; defaults to
+`audit` and `fix-tags`, and an empty list returns only the context and the JSON),
+`instructions` (up to 8,000 characters), and `includeAgentReadiness` (defaults to
+`true`; set it to `false` to skip about 17 extra requests).
+
+The JSON payload is versioned with `schemaVersion` and bounded. Whatever was cut
+is named in `truncated`, so a clipped value is never mistaken for a complete one:
+
+- Every string outside JSON-LD `data` is clipped at 1,000 characters (meta tag
+  values at 500, tag names at 100).
+- Lists are capped: 200 other meta tags, 60 Open Graph and 60 Twitter tags, 20
+  icons, 50 alternate links, 5 `h1` headings, and 10 JSON-LD blocks.
+- A JSON-LD block's `data` is included when it is at most 6,000 characters and 8
+  levels deep, and while the included blocks total at most 20,000 characters. Any
+  other valid block is marked `dataOmitted: true` and is judged by its `types`
+  and `size` alone.
+
+The text above the JSON names only the URL you asked for. Redirect targets, the
+canonical URL, and the detected repository appear only inside the JSON, which the
+prompt marks as untrusted.
+
+A page that pushes every limit at once can still produce a large prompt. Copying
+always works, but **Open in Copilot** and **Create issue** requests are limited to
+512 KB, and a GitHub issue body to 65,536 characters.
+
+| Field | Contents |
+| --- | --- |
+| `url` | Input and final URL, whether it redirected, canonical URL, origin, hostname |
+| `http` | Status, content type, `bodyTruncated`, and the `x-robots-tag`, `link`, `content-language`, and `last-modified` headers |
+| `seo` | Title and description with their lengths, `lang`, robots (meta, googlebot, `x-robots-tag`), viewport, `h1` headings, hreflang `alternates`, JSON-LD `structuredData` (each block's types and size, plus its data when small enough) |
+| `social` | The `resolved` card fields plus the raw `openGraph` and `twitter` tags |
+| `otherMeta`, `icons` | Every other meta tag as found, and the declared icons |
+| `diagnostics` | The same checks as the Diagnostics tab |
+| `agentReadiness` | The agent-readiness checks, or `null` when they weren't run |
+| `repository`, `document` | The detected source repository (or `null`) and the tag count |
 
 ## Files
 
@@ -87,6 +175,7 @@ og-preview/
   extension.mjs        wiring: server, routes, canvas declaration + actions
   lib/http-fetch.mjs   dependency-free http/https fetch (redirects, timeout)
   lib/parse-og.mjs     meta-tag parser -> resolved fields, groups, diagnostics
+  lib/build-prompt.mjs JSON payload + composable prompt (server and canvas UI)
   ui/index.html        renderer markup
   ui/styles.css        platform-styled cards + app-theme chrome
   ui/app.js            client logic (fetch, render, tabs, SSE)

.github/extensions/og-preview/extension.mjs

@@ -4,9 +4,10 @@
 // raw-metadata and diagnostics view.
 //
 // Each open canvas instance gets its own loopback HTTP server (ephemeral port)
-// that serves the static UI and a small JSON API. The canvas also exposes two
-// agent-callable actions: `preview_url` (drive the open canvas to a URL) and
-// `get_metadata` (fetch + parse without a canvas, returning raw metadata).
+// that serves the static UI and a small JSON API. The canvas also exposes three
+// agent-callable actions: `preview_url` (drive the open canvas to a URL),
+// `get_metadata` (fetch + parse without a canvas, returning raw metadata) and
+// `get_prompt` (a composed AI prompt with a full JSON payload for a URL).
 
 import { createServer } from "node:http";
 import { readFile } from "node:fs/promises";
@@ -19,8 +20,21 @@ import { fetchUrl, normalizeUrl, authorizePreview } from "./lib/http-fetch.mjs";
 import { createAccess, requestAccess } from "./lib/request-access.mjs";
 import { parseMetadata } from "./lib/parse-og.mjs";
 import { checkAgentReadiness } from "./lib/agent-readiness.mjs";
+import {
+    DEFAULT_MODULE_IDS,
+    MAX_INSTRUCTIONS_CHARS,
+    PROMPT_MODULES,
+    buildPayload,
+    composePrompt,
+    normalizeModuleIds,
+} from "./lib/build-prompt.mjs";
 
 const UI_DIR = new URL("./ui/", import.meta.url);
+// The prompt builder is shared with the UI, which loads it from this folder.
+const LIB_DIR = new URL("./lib/", import.meta.url);
+
+// Response headers that matter for SEO; forwarded into the prompt payload.
+const SEO_HEADERS = ["x-robots-tag", "link", "content-language", "last-modified"];
 
 // GitHub "blob" (and "raw") web URLs render an HTML page, not the file itself.
 // Rewrite them to raw.githubusercontent.com so /api/raw fetches the actual
@@ -41,10 +55,16 @@ function githubBlobToRaw(value) {
     return value;
 }
 
+// The browse frame runs the whole document, so it gets more room than a metadata
+// read: API reference pages inline several MB of search data ahead of their
+// interactive content, which a 6 MiB cut would drop.
+const BROWSE_DOC_MAX_BYTES = 16 * 1024 * 1024;
+
 const CONTENT_TYPES = {
     ".html": "text/html; charset=utf-8",
     ".css": "text/css; charset=utf-8",
     ".js": "text/javascript; charset=utf-8",
+    ".mjs": "text/javascript; charset=utf-8",
 };
 
 // instanceId -> { instanceId, server, url, currentUrl, titleKey, clients:Set<res> }
@@ -63,9 +83,9 @@ function log(message, level = "info") {
     }
 }
 
-async function serveAsset(res, name) {
+async function serveAsset(res, name, dir = UI_DIR) {
     try {
-        const buf = await readFile(fileURLToPath(new URL(name, UI_DIR)));
+        const buf = await readFile(fileURLToPath(new URL(name, dir)));
         res.setHeader("Content-Type", CONTENT_TYPES[extname(name)] || "application/octet-stream");
         res.end(buf);
     } catch {
@@ -77,7 +97,8 @@ async function serveAsset(res, name) {
 /** Fetch a target URL and parse its OpenGraph metadata. */
 async function loadMetadata(rawUrl, policy) {
     const target = normalizeUrl(rawUrl);
-    const result = await fetchUrl(target, policy);
+    // Metadata lives in the document head, so an oversized page is cut, not rejected.
+    const result = await fetchUrl(target, { ...policy, onLimit: "truncate" });
     if (result.status >= 400) {
         throw new Error(`Target responded with HTTP ${result.status}.`);
     }
@@ -88,6 +109,18 @@ async function loadMetadata(rawUrl, policy) {
     const data = parseMetadata(html, result.url);
     data.requestedUrl = result.url;
     data.httpStatus = result.status;
+    data.truncated = result.truncated;
+    data.http = {
+        inputUrl: target,
+        status: result.status,
+        contentType: result.contentType || "",
+        headers: Object.fromEntries(
+            SEO_HEADERS.filter((name) => result.headers?.[name] != null).map((name) => [
+                name,
+                String(result.headers[name]),
+            ]),
+        ),
+    };
     return data;
 }
 
@@ -457,6 +490,9 @@ async function handleRequest(entry, req, res) {
     if (path === "/styles.css" || path === "/app.js") {
         return serveAsset(res, path.slice(1));
     }
+    if (path === "/build-prompt.mjs") {
+        return serveAsset(res, "build-prompt.mjs", LIB_DIR);
+    }
 
     const access = requestAccess(entry, req, reqUrl);
     if (!access) return sendJson(res, 403, { error: "Preview access denied." });
@@ -666,13 +702,17 @@ async function handleRequest(entry, req, res) {
                 ...policy,
                 accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
                 timeoutMs: 15000,
+                maxBytes: BROWSE_DOC_MAX_BYTES,
+                onLimit: "truncate",
             });
             const ct = r.contentType || "";
             const isHtml = !ct || /text\/html|application\/xhtml\+xml|\/xml|text\/plain/i.test(ct);
             res.setHeader("Cache-Control", "no-store");
             res.setHeader("Access-Control-Allow-Origin", "*");
             const appOrigin = proxyOrigin;
             if (!isHtml) {
+                // A cut-off image or PDF would be corrupt; only documents may be partial.
+                if (r.truncated) throw new Error("Response exceeded the size limit.");
                 // Serve non-HTML targets (images, PDFs, …) verbatim so links to
                 // them still render inside the browse frame.
                 res.statusCode = r.status >= 400 ? r.status : 200;
@@ -835,6 +875,72 @@ const ogCanvas = createCanvas({
                 };
             },
         },
+        {
+            name: "get_prompt",
+            description:
+                "Build a ready-to-use AI prompt for a URL: a full JSON payload of its SEO and social metadata (URL and redirects, HTTP headers, title, description, canonical, robots, hreflang, structured data, Open Graph, Twitter, diagnostics, agent readiness) wrapped in composable task prompts. Works without the canvas being open. Supports localhost.",
+            inputSchema: {
+                type: "object",
+                properties: {
+                    url: { type: "string", description: "URL to inspect." },
+                    modules: {
+                        type: "array",
+                        items: { type: "string", enum: PROMPT_MODULES.map((module) => module.id) },
+                        description: `Task modules to include. Defaults to: ${DEFAULT_MODULE_IDS.join(", ")}. An empty list returns only the context and the JSON payload.`,
+                    },
+                    instructions: {
+                        type: "string",
+                        maxLength: MAX_INSTRUCTIONS_CHARS,
+                        description: "Extra instructions to include in the prompt.",
+                    },
+                    includeAgentReadiness: {
+                        type: "boolean",
+                        description: "Also run the agent-readiness checks (about 17 extra requests). Defaults to true.",
+                    },
+                },
+                required: ["url"],
+                additionalProperties: false,
+            },
+            handler: async (ctx) => {
+                const { url, modules, instructions, includeAgentReadiness } = ctx.input ?? {};
+                if (!url) throw new CanvasError("invalid_input", "An 'url' value is required.");
+                if (modules !== undefined && !Array.isArray(modules)) {
+                    throw new CanvasError("invalid_input", "'modules' must be an array of module ids.");
+                }
+                const selection = normalizeModuleIds(modules ?? DEFAULT_MODULE_IDS);
+                if (selection.unknown.length) {
+                    const unknown = selection.unknown.map((id) => String(id).slice(0, 40)).join(", ");
+                    const valid = PROMPT_MODULES.map((module) => module.id).join(", ");
+                    throw new CanvasError("invalid_input", `Unknown prompt module(s): ${unknown}. Valid modules: ${valid}.`);
+                }
+                if (
+                    instructions !== undefined &&
+                    (typeof instructions !== "string" || instructions.length > MAX_INSTRUCTIONS_CHARS)
+                ) {
+                    throw new CanvasError(
+                        "invalid_input",
+                        `'instructions' must be a string of at most ${MAX_INSTRUCTIONS_CHARS} characters.`,
+                    );
+                }
+                const policy = await authorizePreview(url);
+                const data = await loadMetadata(url, policy);
+                let agentReadiness = null;
+                if (includeAgentReadiness !== false) {
+                    try {
+                        agentReadiness = await checkAgentReadiness(data.requestedUrl, policy);
+                    } catch {
+                        /* optional: the payload reports agentReadiness as null when the probes didn't run */
+                    }
+                }
+                const payload = buildPayload({ data, agentReadiness });
+                return {
+                    url: payload.url.final,
+                    modules: selection.ids,
+                    prompt: composePrompt({ payload, moduleIds: selection.ids, instructions }),
+                    payload,
+                };
+            },
+        },
     ],
     open: async (ctx) => {
         let entry = instances.get(ctx.instanceId);

.github/extensions/og-preview/lib/agent-readiness.mjs

@@ -33,6 +33,9 @@ async function probe(url, opts = {}) {
             timeoutMs: 7000,
             maxRedirects: 4,
             maxBytes: 256 * 1024,
+            // Presence, headers, and the leading bytes are what the checks read, so
+            // an oversized response is cut short rather than reported as missing.
+            onLimit: "truncate",
             ...opts,
         });
         return {
@@ -42,6 +45,7 @@ async function probe(url, opts = {}) {
             contentType: r.contentType || "",
             body: r.body,
             bytes: r.body ? r.body.length : 0,
+            truncated: Boolean(r.truncated),
             finalUrl: r.url,
         };
     } catch {
@@ -67,10 +71,10 @@ function isJsonish(r) {
     return head.startsWith("{") || head.startsWith("[");
 }
 
-function fmtBytes(n) {
+function fmtBytes(n, truncated = false) {
     if (!n) return "0 B";
-    if (n < 1024) return `${n} B`;
-    return `${(n / 1024).toFixed(n < 10240 ? 1 : 0)} KB`;
+    const size = n < 1024 ? `${n} B` : `${(n / 1024).toFixed(n < 10240 ? 1 : 0)} KB`;
+    return truncated ? `${size}+` : size;
 }
 
 function missDetail(r) {
@@ -172,7 +176,7 @@ export async function checkAgentReadiness(rawUrl, policy = {}) {
                     label: "robots.txt",
                     status: scored(hasRobots, true),
                     detail: hasRobots
-                        ? `Found · ${fmtBytes(robots.bytes)}${sitemapInRobots ? " · declares Sitemap" : ""}`
+                        ? `Found · ${fmtBytes(robots.bytes, robots.truncated)}${sitemapInRobots ? " · declares Sitemap" : ""}`
                         : missDetail(robots),
                 },
                 {
@@ -202,7 +206,7 @@ export async function checkAgentReadiness(rawUrl, policy = {}) {
                     label: "llms.txt",
                     status: scored(hasLlms, true),
                     detail: hasLlms
-                        ? `Found · ${fmtBytes(llms.bytes)}${hasLlmsFull ? " · llms-full.txt too" : ""}`
+                        ? `Found · ${fmtBytes(llms.bytes, llms.truncated)}${hasLlmsFull ? " · llms-full.txt too" : ""}`
                         : missDetail(llms),
                 },
                 {