Console contract, rag-sec-tutorial, and removal of the one-click RAG packs

README.md

@@ -14,27 +14,25 @@
 
 A lab is a **pack**: ordered steps that are safe to re-run. `launch` builds it and resumes after a failure. `destroy` removes only what the pack created.
 
-- **Databricks labs.** RAG over real SEC 10-K filings, and a text-to-SQL agent, in your own Free Edition workspace.
+- **Databricks labs.** A step-by-step RAG tutorial over real SEC 10-K filings (notebooks you run), and a text-to-SQL agent, in your own Free Edition workspace.
 - **Web IDE per learner.** Private browser VS Code with the RAG lab preinstalled, on Modal, EC2 or local Docker.
 - **AWS sandboxes.** Cohort IAM with Terraform, plus plan-then-apply cleanup of shared accounts.
-- **Plain English.** `labagent "deploy rag-sec"` shows the command it understood and asks first (local Ollama, any OpenAI-compatible endpoint).
+- **Plain English.** `labagent "deploy rag-tutorial"` shows the command it understood and asks first (local Ollama, any OpenAI-compatible endpoint).
 
 ## Quick start
 
 ```bash
 pip install 'labagent[rag]'            # Python 3.9+; sign in with: databricks auth login
 labagent packs                         # list the labs
-labagent launch rag-sec-filings --name demo
-labagent destroy rag-sec-filings --name demo
+labagent launch rag-sec-tutorial --name demo     # uploads the notebooks to your workspace; you run them
+labagent destroy rag-sec-tutorial --name demo    # removes the uploaded folder
 ```
 
 ## Labs
 
 | Pack | What you get | Runs on |
 |---|---|---|
-| `rag-sec-filings` | RAG over a pinned SEC 10-K dataset: hybrid search, filters, citations, a quality gate before deploy | Databricks Free Edition |
-| `rag-sec-filings-live` | Same, fetching fresh filings from EDGAR | Databricks Free Edition |
-| `rag-lakebase` | Small demo corpus to smoke-test your setup | Databricks Free Edition |
+| `rag-sec-tutorial` | Nine notebooks that build RAG on a pinned SEC 10-K dataset one step at a time: embeddings, pgvector, hybrid search, filters, citations, an evaluation gate, and the deployed app. Each ends with a check cell | Databricks Free Edition |
 | `db-agent` | [db-agent](https://github.com/db-agent/db-agent) text-to-SQL over a Databricks sample dataset, pinned commit | Databricks Free Edition |
 | `rag-ide` | A private browser VS Code per learner with the RAG lab and notebooks | Modal, EC2 or local Docker |
 

docs/job-contract.md

@@ -8,14 +8,15 @@
 | Field | Type | Notes |
 |---|---|---|
 | `run_id` | string | Echoed back in the result |
-| `command` | `provision` `inventory` `sweep` `reset` `offboard` `audit` `cleanup` | Anything else is refused |
+| `command` | `provision` `inventory` `sweep` `reset` `offboard` `audit` `cleanup` `launch` `destroy` | Anything else is refused |
 | `mode` | `plan` `apply` `dry-run` | `sweep` needs `plan` or `apply`; `provision` supports `dry-run` |
 | `cohort` | `{id, created_date}` | `created_date` is `dd-mmm-yyyy` and matches the `date` tag |
 | `learners` | `[{learner_id, username, active?}]` | Roster. Held in memory and on the runner's ephemeral disk only |
 | `plan` | object | Required for `sweep` + `apply`: the exact plan returned by a previous `plan` run |
 | `approved_hash` | string | The plan hash a human approved. Apply is refused on mismatch |
 | `secrets_pubkey_pem` | PEM | RSA public key. Required for `provision` and `reset` (passwords are RSA-OAEP SHA-256 encrypted to it) |
 | `ruleset` | object | For `audit` and `cleanup` (plan). See docs/account-cleanup.md |
+| `pack`, `name`, `params` | string, string, object | For `launch` and `destroy`: a pack id or alias, the instance name (`[a-z0-9-]`, 31 characters at most) and its parameters. `image` and `profile` are refused. These commands need no AWS account allow list: the pack brings its own credentials |
 | `options` | object | `all_cohorts`, `force`, `learner_id` (reset), `only_learner_id` (sweep a single learner), `password_reset_required` (provision/reset; default from config), `regions` (audit/cleanup; `["*"]` = all enabled regions) |
 
 ## Result (labagent to orchestrator)
@@ -33,6 +34,8 @@
 }
 ```
 
+For `launch`, `secrets[]` holds one `ide_password` per learner, encrypted to `secrets_pubkey_pem` (also when the launch fails part way, so passwords already handed out are not lost). `summary.outputs` carries URLs and hosts only.
+
 Exit code: 0 ok, 1 partial (something is `pending` or `failed`: re-run), 2 failed (nothing was done or the input was refused).
 
 ## Rules the engine enforces

docs/packs.md

@@ -5,9 +5,10 @@ credentials, or in your CI. Add a lab by writing a `Pack` subclass; the framewor
 
 ```bash
 labagent packs                                   # list
-labagent launch rag-lakebase --name demo --dry-run
-labagent launch rag-lakebase --name demo         # re-run to resume after a failure
-labagent destroy rag-lakebase --name demo        # removes only what this pack created
+labagent packs --json                            # catalog for dashboards: params, platforms, cost tier
+labagent launch rag-sec-tutorial --name demo --dry-run
+labagent launch rag-sec-tutorial --name demo         # re-run to resume after a failure
+labagent destroy rag-sec-tutorial --name demo        # removes only what this pack created
 ```
 
 ## Writing a pack
@@ -27,45 +28,50 @@ Ship it in your own package and register it: `[project.entry-points."labagent.pa
 Rules: `check` is a read-only "already done" test; step outputs are non-secret; tokens stay in memory (`ctx.mem`).
 State is a 0600 file at `~/.labagent/packs/<pack>/<name>.json`.
 
-## RAG packs (Databricks Lakebase)
+## RAG tutorial (Databricks Lakebase)
 
-One reusable pipeline, pluggable data sources. Each stage is idempotent, and `evaluate` gates the deploy:
+`rag-sec-tutorial` builds nothing itself. It uploads nine notebooks and the app code into your workspace, and you run them in order.
+Each notebook is short, safe to re-run, and ends with a check cell that stops with a clear message if the step did not work.
 
-`preflight, project, locate, build (fetch, parse, chunk), schema, load (embed), index, evaluate, app`
-
-| Pack | Source | Use |
-|---|---|---|
-| `rag-sec-filings` | Pinned snapshot of real SEC 10-Ks ([dataset release](https://github.com/becloudready/labagent-datasets/releases/tag/sec-10k-2026.10)), split into Items. No SEC access or contact needed | Financial research |
-| `rag-sec-filings-live` | The same corpus downloaded fresh from EDGAR at launch (needs a contact for the SEC) | Newer filings, other tickers |
-| `rag-lakebase` | 25 invented passages | Smoke test only |
+| Notebook | You |
+|---|---|
+| `00_setup` | Check your model endpoints, create the Lakebase project |
+| `01_data` | Download the pinned SEC 10-K dataset (checked against a SHA-256) and explore it |
+| `02_embeddings` | Call the embedding model, compare similarity, time batches |
+| `03_pgvector` | Create the table, load the filings in two stages (everyone but Apple, then Apple) to see why embedding decides what search can find |
+| `04_search` | Add HNSW and full-text indexes, run vector, keyword and hybrid (reciprocal rank fusion) search, add filters |
+| `05_answer_and_eval` | Generate cited answers, try a question with no answer, score retrieval against a 0.6 gate |
+| `06_deploy_app` | Create the Databricks App, give its own identity a read-only role, deploy |
+| `99_cleanup` | Remove the app, the schema and the database project |
 
 ```bash
 pip install 'labagent[rag]'      # needs the Databricks CLI, then: databricks auth login
-labagent launch rag-sec-filings --name sec --param "user_agent=Full Name name@example.com" \
-    --param tickers=AAPL,JPM,PFE --param filings=2
+labagent launch rag-sec-tutorial --name demo
 ```
 
-What makes it more than a toy: section-aware chunking, hybrid retrieval (pgvector + full-text, rank-fused), metadata filters
-(`{"ticker": "AAPL"}`), numbered citations with the filing URL, and an evaluation set that must reach `min_hit_rate`
-(default 0.6) before the app is deployed. The snapshot is downloaded from a GitHub release and verified against a SHA-256 committed in
-`labagent/packs/rag/sources/datasets.json`; a mismatch is refused. The live pack follows EDGAR rules (declared contact, under 10 requests
-per second, cached locally). Filings are never stored in this repo.
+It prints a link to the uploaded folder. Open it, start with `00_setup`, and finish with `99_cleanup`. The notebooks run on serverless
+compute, need no cluster, and download the dataset themselves. Free Edition allows one Lakebase project and three apps, and stops apps
+after 24 hours. `labagent destroy rag-sec-tutorial --name demo` removes only the uploaded folder: run `99_cleanup` first to remove what the
+notebooks created.
 
-Free Edition allows one Lakebase project: pass `--param project=<existing>` to share it. Each source uses its own schema, and
-`destroy` drops only that schema (and the project only if labagent created it). Apps stop after 24 hours.
+The one-command versions of this lab (`rag-lakebase`, `rag-sec-filings`, `rag-sec-filings-live`) were removed in favour of the tutorial.
+Anyone who types one of those names gets a message pointing here. They are still in the `v0.2.1` release and git tag.
 
-Add a source by subclassing `labagent.packs.rag.sources.Source` (`chunks()` and `eval_set()`) and registering it under the
-`labagent.rag_sources` entry point.
+Add a data source by subclassing `labagent.packs.rag.sources.Source` and registering it under the `labagent.rag_sources` entry point.
 
 ## Publishing a dataset snapshot (maintainers)
 
 ```bash
-labagent dataset --chunks ~/.labagent/cache/rag-sec-filings-live/<name>.jsonl --name sec-10k --version 2026.11 --out out/
+labagent dataset --chunks chunks.jsonl --name sec-10k --version 2026.11 --out out/
 gh release create sec-10k-2026.11 out/* --repo becloudready/labagent-datasets
 ```
 
-The build is byte-for-byte reproducible. Then add the release URL and the printed `sha256` to `datasets.json` in a pull request.
-Data limitations (for example, no Item 7 for JPM and CVX) are listed in the dataset README.
+`chunks.jsonl` has one JSON object per line: `id`, `title`, `content`, `meta`. The build is byte-for-byte reproducible. Then add the release
+URL and the printed `sha256` to `labagent/packs/rag/sources/datasets.json` in a pull request. Data limitations (for example, no Item 7 for
+JPM and CVX) are listed in the dataset README.
+
+The SEC chunks were produced by the `rag-sec-filings-live` pack, which no longer exists. The parser it used is still in the code
+(`labagent.packs.rag.sources.sec.SecFilings`) but nothing calls it any more. To rebuild the SEC dataset from EDGAR, use the `v0.2.1` tag.
 
 ## db-agent (text-to-SQL on Databricks)
 
@@ -91,8 +97,8 @@ Steps: `preflight` (login, SQL warehouse, sample data, serving endpoints), `fetc
 ## Plain English
 
 ```bash
-labagent "Run and deploy rag-sec in databricks"      # shows the command, asks, then runs it
-labagent "tear down rag-sec" --print-only            # just show the command
+labagent "Run and deploy rag-tutorial in databricks"      # shows the command, asks, then runs it
+labagent "tear down rag-tutorial" --print-only            # just show the command
 labagent ask what can you deploy                     # 'ask' lets you skip the quotes
 ```
 
@@ -103,7 +109,7 @@ A remote endpoint is announced before your sentence is sent to it.
 
 The model never runs anything, and it is not trusted for details:
 - The instance name (`called x`) and parameters (`profile DEFAULT`, `tickers AAPL,JPM`) are read from your sentence by exact rules, so they cannot be invented.
-- A pack you typed (`rag-sec`, or any id or alias) wins over the model's pick. If the model guessed, the pack needs a distinctive word from your sentence.
+- A pack you typed (`rag-tutorial`, or any id or alias) wins over the model's pick. If the model guessed, the pack needs a distinctive word from your sentence.
 - Requests that ask to create and delete in one sentence are refused. A delete needs a delete word, and you confirm by typing the instance name.
 - Without a terminal, nothing runs unless you pass `--yes`.
 

examples/01_rag_starter_lab.py

@@ -5,7 +5,7 @@
     python ../examples/01_rag_starter_lab.py
 
 Set GROQ_API_KEY in .env for real answers; without any key you get a clearly marked mock answer.
-This is the laptop version of the labagent Databricks packs (`labagent launch rag-sec-filings`).
+This is the laptop version of the labagent Databricks packs (`labagent launch rag-sec-tutorial`).
 """
 from pathlib import Path
 

labagent/ai/rag/adapters.py

@@ -2,7 +2,7 @@
 
 Shipped: ChromaAdapter (default, embedded, free), MemoryAdapter (no dependencies), QdrantAdapter (remote or local).
 Pinecone and pgvector fit the same four methods. For managed pgvector on Lakebase use the pack instead:
-`labagent launch rag-sec-filings`.
+`labagent launch rag-sec-tutorial`.
 """
 import math
 import uuid

labagent/ai/rag/client.py

@@ -5,7 +5,7 @@
     print(client.query("What is a vector database?").answer)
 
 How it connects to labagent: `labagent rag init` scaffolds a project that uses this client, `labagent rag serve` shows
-it in a browser, and the Databricks packs (`labagent launch rag-sec-filings`) deploy the same pipeline to Lakebase.
+it in a browser, and the Databricks tutorial (`labagent launch rag-sec-tutorial`) builds the same pipeline on Lakebase.
 """
 import asyncio
 import re

labagent/ai/rag/samples.py

@@ -183,7 +183,7 @@ def code(src): return {"cell_type": "code", "metadata": {}, "execution_count": N
            "2. Set `top_k=1` and ask a question that needs two documents.\n3. Call `rag.reset()`, re-ingest with `chunk_chars=100` in `RAGSettings`, and compare answers.\n"),
         code("print(rag.query('What is the lunch menu?').answer)\n"),
         md("## Next\n\nOpen the visual playground in a terminal with `labagent rag playground` and compare scores side by side. "
-           "For a managed version on Databricks Lakebase, see `labagent launch rag-sec-filings`.\n"),
+           "For a managed version on Databricks Lakebase, see `labagent launch rag-sec-tutorial`.\n"),
     ]
     for i, c in enumerate(cells):
         c["id"] = f"cell{i:02d}"          # nbformat 4.5 requires cell ids
@@ -258,7 +258,7 @@ def code(src): return {"cell_type": "code", "metadata": {}, "execution_count": N
         md("## 9. What it cost, and where next\n\nEmbeddings ran on this machine, so they were free. Answers cost fractions of a cent on Groq's free tier. "
            "The compute you are using is billed per second only while this IDE runs and stops when it goes idle.\n\n"
            "**Next:** the same pipeline runs managed on Databricks Lakebase (Postgres with pgvector) and Databricks Apps:\n\n"
-           "```\npip install 'labagent[rag]'\nlabagent launch rag-sec-filings --name demo\n```\n"),
+           "```\npip install 'labagent[rag]'\nlabagent launch rag-sec-tutorial --name demo\n```\n"),
     ]
     for i, c in enumerate(cells):
         c["id"] = f"cell{i:02d}"          # nbformat 4.5 requires cell ids

labagent/cli.py

@@ -12,7 +12,7 @@
 from .config import STATE_BACKEND_DIR, load_config
 from .context import Ctx, GuardError
 from .envelope import Envelope
-from .run import COMMANDS, run_job
+from .run import COMMANDS, PACK_COMMANDS, run_job, run_pack_job
 
 
 def _emit(env: Envelope, output: str) -> None:
@@ -28,6 +28,9 @@ def _packs_main(args) -> int:
     from .packs import registry
     from .packs.base import run_pack
     if args.cmd == "packs":
+        if args.json:
+            print(json.dumps({"packs": [pk.describe() for _, pk in sorted(registry.all_packs().items())]}, indent=2))
+            return 0
         for pid, pk in sorted(registry.all_packs().items()):
             print(f"{pid:16} {pk.summary}")
         return 0
@@ -50,10 +53,10 @@ def _packs_main(args) -> int:
 
 
 def _nl_main(argv) -> int:
-    """labagent "deploy rag-sec on databricks": translate with a small model, confirm, then run the real command."""
+    """labagent "deploy rag-tutorial on databricks": translate with a small model, confirm, then run the real command."""
     from . import nl
     from .packs import registry
-    q = argparse.ArgumentParser(prog="labagent", description="Plain-English front end: labagent \"deploy rag-sec on databricks\"")
+    q = argparse.ArgumentParser(prog="labagent", description="Plain-English front end: labagent \"deploy rag-tutorial on databricks\"")
     q.add_argument("text", nargs="+")
     q.add_argument("--endpoint", help=f"OpenAI-compatible base URL (default {nl.DEFAULT_URL}, or LABAGENT_LLM_URL)")
     q.add_argument("--model", help=f"Model (default {nl.DEFAULT_MODEL}, or LABAGENT_LLM_MODEL)")
@@ -101,11 +104,12 @@ def main(argv=None) -> int:
     r.add_argument("--job", required=True, help="Path to job JSON (contains roster: keep it out of git and logs)")
     r.add_argument("--output", default="result.json", help="Result envelope path, or - for stdout")
     r.add_argument("--ruleset", help="Path to a ruleset JSON for audit/cleanup (merged into the job)")
-    sub.add_parser("packs", help="List available lab packs")
+    pk = sub.add_parser("packs", help="List available lab packs")
+    pk.add_argument("--json", action="store_true", help="Machine-readable catalog with parameters, platforms and cost tier (used by the console)")
     from .ai.rag import cli as rag_cli
     rag_cli.add_parser(sub)
     dp = sub.add_parser("dataset", help="Package a chunk file as a versioned, checksummed snapshot (maintainers)")
-    dp.add_argument("--chunks", required=True, help="JSONL of chunks, e.g. ~/.labagent/cache/rag-sec-filings/<name>.jsonl")
+    dp.add_argument("--chunks", required=True, help="JSONL of chunks: one object per line with id, title, content and meta")
     dp.add_argument("--name", required=True)
     dp.add_argument("--version", required=True)
     dp.add_argument("--out", required=True, help="Output directory")
@@ -140,8 +144,12 @@ def main(argv=None) -> int:
         print("\n".join(sorted(COMMANDS)))
         return 0
     try:
-        cfg = load_config(args.config)
         job = json.loads(Path(args.job).read_text())
+        if job.get("command") in PACK_COMMANDS:       # lab packs bring their own credentials: no AWS account guard here
+            env = run_pack_job(job)
+            _emit(env, args.output)
+            return env.exit_code
+        cfg = load_config(args.config)
         session = boto3.Session(profile_name=args.profile, region_name=cfg.region)
         if getattr(args, "ruleset", None) and "ruleset" not in job:
             job["ruleset"] = json.loads(Path(args.ruleset).read_text())