streamcore-cli

English | 简体中文

Command-line tool for StreamCoreAI. Currently supports RAG document ingestion — parsing, chunking, embedding, and uploading documents to your vector store.

Ingestion for streamcore-server — realtime media infrastructure for AI-powered applications. If this is useful, a star on the server repo is the one that helps.

Install

Prebuilt binaries for macOS and Linux, both architectures, are on the releases page:

curl -sL https://github.com/streamcoreai/streamcore-cli/releases/latest/download/streamcore-cli_$(uname -s)_$(uname -m).tar.gz | tar xz
./streamcore-cli version

With a Go toolchain:

go install github.com/streamcoreai/streamcore-cli@latest

# or from source
git clone https://github.com/streamcoreai/streamcore-cli
cd streamcore-cli && go build -o streamcore-cli .

Getting started

Run the interactive setup wizard to configure your RAG provider and credentials:

streamcore-cli setup

This walks you through:

1. Choosing a vector store provider (pgvector or Supabase)
2. Entering your OpenAI API key and picking an embedding model
3. Entering provider-specific credentials (connection string, Supabase URL, etc.)

The config is saved to ~/.streamcore/config.toml. If you've already run setup, it pre-fills your existing values so you can update individual fields.

Setup then creates the table, sized for the embedding model you picked, so there is no SQL to run by hand. Supabase projects need their direct Postgres connection string for this — Project Settings → Database — because PostgREST can insert rows but cannot run DDL. Leave it blank and the SQL is printed for the dashboard's editor instead.

Commands

setup — Interactive configuration wizard

streamcore-cli setup

version — Print the version

streamcore-cli version

ingest — Upload documents to your RAG vector store

# Ingest one or more files
streamcore-cli ingest docs/faq.pdf product-catalog.xlsx notes.md

# Override provider or point to a specific config
streamcore-cli ingest --provider supabase --config ./my-config.toml data.csv

# Control chunk size and overlap
streamcore-cli ingest --chunk-size 256 --chunk-overlap 32 manual.docx

Supported file formats

Format	Extensions
Plain text	.txt
Markdown	.md, .markdown
CSV	.csv
PDF	.pdf
Word	.docx
Excel	.xlsx

Flags

Flag	Default	Description
--config	~/.streamcore/config.toml	Path to config file
--provider	from config	Override RAG provider (pgvector, supabase)
--chunk-size	512	Target chunk size in words
--chunk-overlap	64	Overlap between chunks in words

Configuration

The CLI stores its config at ~/.streamcore/config.toml. Run streamcore-cli setup to create or update it interactively.

The config file is looked up in this order:

1. Explicit --config path
2. ~/.streamcore/config.toml
3. ./config.toml (local override)
4. ../server/config.toml (fallback to server config if in monorepo)

The format is standard TOML:

[openai]
api_key = "sk-..."

[rag]
provider = "supabase"       # "pgvector" or "supabase"
embedding_model = "text-embedding-3-small"
top_k = 3

[pgvector]
connection_string = "postgres://user:pass@localhost:5432/mydb"
table = "documents"

[supabase]
url = "https://xxx.supabase.co"
api_key = "your-service-role-key"
table = "documents"

You can also set OPENAI_API_KEY as an environment variable instead of putting it in the config file.

The config is compatible with the StreamCoreAI server's config.toml — if you're running both in the same repo, the CLI can fall back to the server's config so you don't configure things twice.

Database setup

streamcore-cli setup creates all of this. The SQL is here for anyone who would rather run it themselves, and because the server reads what these tables hold — the same DDL is documented in streamcore-server.

Substitute the vector width your embedding model produces:

embedding_model	Column type
text-embedding-3-small (default)	vector(1536)
text-embedding-ada-002	vector(1536)
text-embedding-3-large	vector(3072)

pgvector

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE IF NOT EXISTS documents (
    id SERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding vector(1536) NOT NULL,
    embedding_model TEXT NOT NULL,
    source TEXT,
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS documents_embedding_model_idx ON documents (embedding_model);

Supabase

Everything above, plus the RPC the server calls at query time and policies letting your key read and write:

CREATE OR REPLACE FUNCTION match_documents(
    query_embedding vector(1536),
    match_count int DEFAULT 3
)
RETURNS TABLE (content text, similarity float)
LANGUAGE plpgsql AS $$
BEGIN
    RETURN QUERY
    SELECT d.content, 1 - (d.embedding <=> query_embedding) AS similarity
    FROM documents d
    ORDER BY d.embedding <=> query_embedding
    LIMIT match_count;
END;
$$;

ALTER TABLE documents ENABLE ROW LEVEL SECURITY;

-- SELECT for the server's queries, INSERT for this tool.
CREATE POLICY "Allow read access to documents"
ON documents FOR SELECT TO authenticated, anon USING (true);

CREATE POLICY "Allow insert access to documents"
ON documents FOR INSERT TO authenticated, anon WITH CHECK (true);

CREATE POLICY "Allow update access to documents"
ON documents FOR UPDATE TO authenticated, anon USING (true);

The embedding model is part of the schema

A vector store only makes sense against the model that built it. Two failures follow from that, and ingest checks for both before writing anything:

• Wrong width. text-embedding-3-large produces 3072 dimensions and will not fit a vector(1536) column. Postgres catches this one on its own, mid-ingest.
• Right width, wrong model. ada-002 and 3-small are both 1536. Mixing them raises no error anywhere — the table fills up, queries return, and the chunks that come back are the wrong ones. This is what the embedding_model column exists to catch.

The server runs the same two checks when it boots and refuses to start on a mismatch.

If you ingested before embedding_model existed, migrate:

ALTER TABLE documents ADD COLUMN embedding_model TEXT;
UPDATE documents SET embedding_model = '<the model you ingested with>';
ALTER TABLE documents ALTER COLUMN embedding_model SET NOT NULL;
CREATE INDEX IF NOT EXISTS documents_embedding_model_idx ON documents (embedding_model);

Only you know which model wrote those rows, which is why the backfill is a placeholder. If you no longer know, re-ingest.

How it works

1. Verify — Checks the target table once, before any file is opened: the vector column's width, and whether existing rows were embedded with a different model. Both are fatal.
2. Parse — Extracts plain text from the input file. Word documents are parsed from their underlying XML, Excel files are converted to field-value pairs per row, CSVs use the header row as field names.
3. Chunk — Splits the text into overlapping chunks, breaking on paragraph and sentence boundaries rather than cutting mid-sentence.
4. Embed — Sends each chunk to the OpenAI embeddings API to get a vector.
5. Store — Inserts the chunk content, embedding, embedding model, and source filename into your vector store.

The source column stores the original filename, which can be useful for filtering or attribution.

Project structure

streamcore-cli/
├── main.go                     # CLI entry point, config loading
└── internal/
    ├── schema/
    │   └── schema.go           # Table DDL and the model/width contract with the server
    ├── setup/
    │   ├── setup.go            # Interactive TUI setup wizard
    │   └── provision.go        # Runs the DDL so setup leaves a working store
    ├── parser/                 # File format parsers
    │   ├── parser.go           # Format dispatcher
    │   ├── text.go             # .txt, .md
    │   ├── csv.go              # .csv
    │   ├── pdf.go              # .pdf
    │   ├── docx.go             # .docx (pure Go, no cgo)
    │   └── xlsx.go             # .xlsx
    ├── chunker/
    │   └── chunker.go          # Text splitting with overlap
    ├── embed/
    │   └── embed.go            # OpenAI embeddings client
    └── store/
        ├── store.go            # Store interface
        ├── pgvector.go         # pgvector (PostgreSQL) backend
        └── supabase.go         # Supabase REST API backend