[wu-sheng]

Pull conversations from another asz

A person judging an agent's conversations often wants them on their own machine, while asz collects
them somewhere else. This adds a pull between two asz installations: one serves its conversations,
and the other copies the ones a person picks, then judges and reads them locally.

What a person does

• asz token create makes a token on the serving side. With remote_server.tokens_file set, asz view
  and asz server answer pulls on their address, and every request there, the page too, needs a token.
  TLS and client certificates are optional.
• On the pulling side, remotes: names the servers. asz pull -list lists what each holds, with filters
  and pages. asz pull REMOTE/CONVERSATION copies one conversation's head as a snapshot.
• asz judge -remote-conversation REMOTE/CONVERSATION[:TAG] judges a snapshot, and -pull pulls first.
• asz view lists pulled conversations beside the root's own, each title opened by [remote, NAME].

Deployment

flowchart LR
  subgraph prod["Where the agents run, in a private network"]
    A["agents"] -->|"traces, sent over the network"| S["asz server<br/>collects and parses<br/>into its storage root<br/>page and pull routes on one address"]
    A -->|"transcripts and file changes"| D[("shared volume")]
    D -->|"collected"| S
  end
  subgraph me["Your machine"]
    P["asz pull -list<br/>asz pull prod/CONVERSATION"] --> R["snapshots under<br/>_remote/prod/CONVERSATION/TAG/"]
    R --> J["asz judge -remote-conversation"]
    R --> V["asz view<br/>rows titled [remote, prod]"]
  end
  P -->|"HTTPS, encrypted by TLS<br/>Authorization: Bearer TOKEN<br/>a client certificate, when asked"| S

Loading

One pull:

sequenceDiagram
  participant C as asz pull
  participant S as asz server
  C->>S: GET /pull/v1/conversations (the listing)
  C->>S: GET .../manifest
  S-->>C: 202 freezing, polled until 200: the manifest, a snapshot tag, a lease
  C->>S: POST .../snapshots/TAG/archives (the files not held here)
  S-->>C: 202 building, polled until 200: size and digest
  C->>S: GET .../archives/ID/data, Range of at most 8 MiB, If-Match
  S-->>C: 206, range after range, resumed where a connection broke
  C->>S: POST .../snapshots/TAG/done (the lease released)
  Note over C: checks the evidence, then publishes the snapshot here

Loading

Configuration

The server, with tokens and TLS (client_ca_file also requires client certificates):

remote_server:
  tokens_file: /etc/asz/tokens          # one line per token: a name and its SHA-256
  tls:
    cert_file: /etc/asz/server.crt
    key_file: /etc/asz/server.key
    client_ca_file: ""                  # set to require a client certificate this CA signed
  max_concurrent_per_token: 2
  snapshot_ttl: 1h

asz token create -file /etc/asz/tokens alice   # prints the token once
asz server -config asz.yaml 0.0.0.0:8788       # or asz view: the page and the pull routes on one address

The client:

remote_snapshot_ttl: 720h
remotes:
  - name: prod
    url: https://asz.example.internal:8788
    token_file: ~/.asz/prod.token       # or token_env; exactly one
    ca_file: /etc/asz/ca.crt            # trust this CA for this remote, in place of the system's
    client_cert_file: ""                # both, for a server with client_ca_file
    client_key_file: ""
    server_name: ""                     # when the certificate names another host than the url

• Every request to a server with a tokens file needs Authorization: Bearer TOKEN, the page too. The
  server keeps only SHA-256 hashes, and deleting a line revokes that token within five seconds.
• Without TLS, tokens travel in clear text, and asz warns when the address is not a loopback one.
• The client always checks the server's certificate, follows no redirect, and uses only an http://
  proxy.

How it works

• The server freezes a conversation's head into a snapshot it serves for snapshot_ttl after its last
  use. A client takes a lease, asks for the files it does not hold as one tar.gz archive, and fetches it
  in ranges of at most 8 MiB that resume where a connection broke.
• The client keeps each pull as an immutable snapshot under _remote/REMOTE/CONVERSATION/TAG/, reuses
  what its newest valid snapshot holds, and checks the evidence against the digests its rounds carry
  before it publishes. Snapshots, and the judge runs in them, expire after remote_snapshot_ttl.
• Every write is crash-safe: built privately, synced, then published by a rename; an interrupted
  pull, sweep or server transition is recovered by the next one. A fault model tests each step against
  a power loss at that step.

The protocol is documented in docs/en/formats/asz-pull.md, setup in docs/en/setup/remote-pull.md.

Testing

• Unit and fault tests for the server, the client, the formats and storage. Every guard was checked by
  removing it and seeing its test fail.
• An end-to-end test runs servers in Docker Compose behind TLS and a shared cache, and checks token
  authentication and revocation (make e2e-pull).
• The CI unit-test step gets a 30-minute limit: one client test waits the two minutes a metadata answer
  has.

No production deployment has been measured: speed, compression and memory at production size are not
known yet.

Follow-up

• Wider tests across both durability models through HTTP, proxies, and several remotes at once.
• Measurements on a production-size root.