From ea437efe51e197414fef6223b1414987750049d5 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 15 Sep 2026 15:36:45 +0000
Subject: [PATCH 1/5] Add repository overview docs

Co-authored-by: bio-boris <1258634+bio-boris@users.noreply.github.com>
---
 docs/developer/index.md               |   5 ++
 docs/developer/repository_overview.md | 106 ++++++++++++++++++++++++++
 mkdocs.yml                            |   1 +
 3 files changed, 112 insertions(+)
 create mode 100644 docs/developer/repository_overview.md

diff --git a/docs/developer/index.md b/docs/developer/index.md
index 908dafe6..30fbf8bd 100644
--- a/docs/developer/index.md
+++ b/docs/developer/index.md
@@ -2,6 +2,11 @@
 
 This guide contains technical information about the Data Transfer Service (DTS).
 
+## Start Here
+
+* [Repository Overview](repository_overview.md): purpose, architecture,
+  extension points, and the major directories in this repository
+
 ## Architecture Decision Records
 
 [Here](adrs/index.md) we've recorded design decisions in a succinct format that
diff --git a/docs/developer/repository_overview.md b/docs/developer/repository_overview.md
new file mode 100644
index 00000000..0f758294
--- /dev/null
+++ b/docs/developer/repository_overview.md
@@ -0,0 +1,106 @@
+# Repository Overview
+
+The Data Transfer Service (DTS) is a Go service that gives clients one API for
+searching remote data holdings, collecting file metadata, and requesting file
+transfers between participating systems.
+
+The core design assumption in this repository is that DTS must work with a
+highly heterogeneous, continuously changing set of unrelated systems. Rather
+than requiring a single storage model or transfer stack, DTS isolates
+system-specific behavior behind pluggable database and endpoint interfaces and
+coordinates transfers through a shared workflow.
+
+## Purpose
+
+DTS sits between clients and external data providers. It is responsible for:
+
+* authenticating requests and associating them with users
+* exposing REST endpoints for search, metadata lookup, transfer creation, and
+  transfer status
+* translating those requests into calls against configured source and
+  destination systems
+* coordinating staging, transfer, manifest generation, finalization, and
+  persistence of transfer state
+
+In practice, this means the service can present a stable API even when source
+databases, destination databases, and transfer providers all have different
+capabilities and conventions.
+
+## High-level architecture
+
+DTS is organized into a few layers:
+
+1. **Service/API layer** (`main.go`, `services/`): starts the process, loads
+   configuration, exposes the HTTP API, and validates user requests.
+2. **Workflow/orchestration layer** (`transfers/`): owns transfer lifecycle
+   management, persistence, status updates, and coordination between staging,
+   moving, and finalization.
+3. **Integration layer** (`databases/`, `endpoints/`, `auth/`): adapts
+   database-specific, endpoint-specific, and authentication-specific behavior to
+   common interfaces.
+4. **Support packages** (`config/`, `journal/`, `credit/`): provide
+   configuration loading, transfer journaling, and metadata/provenance support.
+
+This separation lets the API and workflow logic stay mostly stable while new
+systems are introduced through integration packages and configuration.
+
+## Transfer lifecycle
+
+A typical transfer moves through these steps:
+
+1. The API receives a search, metadata, or transfer request.
+2. The service authenticates the caller and validates the request.
+3. The transfer dispatcher records the request and determines whether files must
+   be staged first.
+4. If staging is needed, the stager polls the source database until files are
+   ready.
+5. The mover asks the configured transfer endpoint provider to move files to the
+   destination.
+6. The manifestor writes transfer metadata and places it with the payload.
+7. The destination database finalizes the transfer, and the journal records the
+   outcome.
+
+State is persisted so transfers can survive service restarts.
+
+## Extensibility model
+
+The main extensibility points are:
+
+* **Databases** implement search, metadata lookup, optional staging, user
+  federation, and finalization behavior for a participating system.
+* **Endpoints** implement the actual file movement mechanism, such as Globus,
+  S3, or local filesystem transfers.
+* **Configuration** binds named databases and endpoints to concrete provider
+  implementations and credentials.
+
+This model is what allows DTS to adapt to unrelated systems with different
+identifiers, metadata schemas, staging requirements, authentication needs, and
+transfer technologies.
+
+## Repository layout
+
+Key areas of the repository are:
+
+* `/main.go`: process startup and shutdown
+* `/services`: REST API handlers and transport-facing types
+* `/transfers`: dispatcher, store, stager, mover, manifestor, and transfer
+  state management
+* `/databases`: shared database interface plus concrete integrations such as
+  JDP, KBase, NMDC, and S3-backed databases
+* `/endpoints`: shared endpoint interface plus concrete transfer providers
+* `/auth`: request authentication and user identity lookup
+* `/config`: YAML configuration loading and validation
+* `/journal`: transfer record persistence
+* `/docs`: MkDocs site for operators, integrators, and developers
+* `/integration`, `/dtstest`: integration-test support and test helpers
+
+## Why the architecture looks this way
+
+The repository favors orchestration over deep normalization. DTS does not try to
+force every external system into the same internal storage or execution model.
+Instead, it defines a narrow common contract for the capabilities it needs and
+lets each integration satisfy that contract in its own way.
+
+That approach keeps the service practical for environments where participating
+systems are independently managed, evolve at different speeds, and may only
+share a small set of interoperable behaviors.
diff --git a/mkdocs.yml b/mkdocs.yml
index cd9de71b..94d9f66d 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -19,6 +19,7 @@ nav:
     - 'Glossary': 'integration/glossary.md'
   - 'Developer Guide':
     - 'Overview': 'developer/index.md'
+    - 'Repository Overview': 'developer/repository_overview.md'
     - 'Architecture Decision Records':
       - 'developer/adrs/index.md'
       - 'Transmitting Metadata': 'developer/adrs/transmit-metadata.md'

From 0e0e1ae681b0b2fd5e83d05a1f9b07605dafdb60 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Tue, 15 Sep 2026 15:54:41 +0000
Subject: [PATCH 2/5] Deepen repository overview

Co-authored-by: bio-boris <1258634+bio-boris@users.noreply.github.com>
---
 docs/developer/index.md               |   3 +-
 docs/developer/repository_overview.md | 182 +++++++++++++++++++++++---
 2 files changed, 169 insertions(+), 16 deletions(-)

diff --git a/docs/developer/index.md b/docs/developer/index.md
index 30fbf8bd..b0b27299 100644
--- a/docs/developer/index.md
+++ b/docs/developer/index.md
@@ -5,7 +5,8 @@ This guide contains technical information about the Data Transfer Service (DTS).
 ## Start Here
 
 * [Repository Overview](repository_overview.md): purpose, architecture,
-  extension points, and the major directories in this repository
+  architectural drivers, runtime model, extension points, and the major
+  directories in this repository
 
 ## Architecture Decision Records
 
diff --git a/docs/developer/repository_overview.md b/docs/developer/repository_overview.md
index 0f758294..37562dcd 100644
--- a/docs/developer/repository_overview.md
+++ b/docs/developer/repository_overview.md
@@ -6,9 +6,9 @@ transfers between participating systems.
 
 The core design assumption in this repository is that DTS must work with a
 highly heterogeneous, continuously changing set of unrelated systems. Rather
-than requiring a single storage model or transfer stack, DTS isolates
-system-specific behavior behind pluggable database and endpoint interfaces and
-coordinates transfers through a shared workflow.
+than requiring a single storage model, metadata shape, identity model, or
+transfer stack, DTS isolates system-specific behavior behind pluggable database
+and endpoint interfaces and coordinates transfers through a shared workflow.
 
 ## Purpose
 
@@ -26,6 +26,33 @@ In practice, this means the service can present a stable API even when source
 databases, destination databases, and transfer providers all have different
 capabilities and conventions.
 
+At a conceptual level, DTS is neither a storage platform nor a transfer engine
+in its own right. It is an orchestration service. It accepts a user-facing
+request, identifies which participating systems must be involved, and drives
+those systems through a normalized lifecycle that is simple enough for clients
+but flexible enough for operators and integrators.
+
+## Architectural drivers
+
+Several design pressures explain why the codebase looks the way it does:
+
+* **Heterogeneity is expected, not exceptional.** Participating systems may use
+  different identifiers, metadata structures, staging workflows, trust models,
+  and transfer technologies.
+* **Integrations evolve independently.** DTS assumes external systems are owned
+  by different organizations and may change on different release cycles.
+* **The interoperable contract is intentionally narrow.** DTS only standardizes
+  the capabilities it truly needs: search, metadata lookup, staging, transport,
+  finalization, and user mapping.
+* **Transfers are long-running.** The service must continue work outside the
+  lifetime of a single HTTP request and recover from restarts.
+* **Provenance matters alongside payload delivery.** A completed transfer is not
+  just copied files; it also includes manifest and metadata production.
+
+These drivers lead the repository to emphasize interfaces, orchestration
+components, and persisted state rather than a single deep domain model shared by
+all integrations.
+
 ## High-level architecture
 
 DTS is organized into a few layers:
@@ -44,23 +71,101 @@ DTS is organized into a few layers:
 This separation lets the API and workflow logic stay mostly stable while new
 systems are introduced through integration packages and configuration.
 
+Another useful lens is to think about two different kinds of stability:
+
+* the **northbound** surface is stable and client-oriented: a small REST API,
+  predictable transfer states, and a uniform request model
+* the **southbound** surface is adaptable and integration-oriented: provider
+  implementations, credentials, endpoint details, and local policy
+
+The repository exists to keep those two surfaces decoupled.
+
+## Core runtime model
+
+At runtime, the service follows a coordinator pattern:
+
+* `main.go` loads configuration, initializes logging, creates the service, and
+  owns process startup and graceful shutdown.
+* `services/` exposes the HTTP API and translates requests into transfer-domain
+  operations.
+* `transfers/` runs the asynchronous transfer subsystem and owns the long-lived
+  state machine for each transfer.
+
+Inside `transfers/`, several cooperating components divide responsibility:
+
+* the **dispatcher** accepts requests, creates transfer records, and decides
+  which step comes next
+* the **store** is the system of record for transfer specifications,
+  descriptors, payload size, and status
+* the **stager** handles source-side preparation when files are not immediately
+  transferable
+* the **mover** delegates payload copy to endpoint providers and monitors
+  progress
+* the **manifestor** builds and transfers the manifest that describes the
+  payload and its metadata
+
+These components communicate through channels and background goroutines. That is
+an important architectural choice: DTS treats transfers as asynchronous
+workflows rather than request-scoped operations.
+
 ## Transfer lifecycle
 
 A typical transfer moves through these steps:
 
 1. The API receives a search, metadata, or transfer request.
 2. The service authenticates the caller and validates the request.
-3. The transfer dispatcher records the request and determines whether files must
-   be staged first.
-4. If staging is needed, the stager polls the source database until files are
-   ready.
-5. The mover asks the configured transfer endpoint provider to move files to the
-   destination.
-6. The manifestor writes transfer metadata and places it with the payload.
-7. The destination database finalizes the transfer, and the journal records the
-   outcome.
-
-State is persisted so transfers can survive service restarts.
+3. The dispatcher records the transfer specification and inspects the source and
+   destination requirements.
+4. If staging is required, the stager requests source-side preparation and polls
+   until the files are ready.
+5. The mover groups payload files by source endpoint and asks the relevant
+   endpoint provider to move them to the destination.
+6. The manifestor generates a Frictionless-style manifest describing the
+   transfer and places that manifest with the payload.
+7. The destination-side finalization logic completes any system-specific
+   post-transfer processing.
+8. The journal records the outcome, and completed state remains available until
+   retention rules remove it.
+
+This lifecycle is deliberately broken into stages because different source and
+destination systems become ready at different times. Some payloads can be moved
+immediately; others must first be restored from archival storage, staged onto a
+filesystem, or associated with a destination-specific import step. DTS handles
+that variation by advancing transfers through coarse-grained states instead of
+assuming one synchronous copy operation.
+
+## Abstraction boundaries
+
+Two interfaces carry most of the repository's heterogeneity.
+
+### Database integrations
+
+The `databases.Database` interface represents a participating data system from
+the perspective of DTS. A database implementation answers questions such as:
+
+* what files match a search query?
+* what descriptors and metadata correspond to a set of file IDs?
+* which transfer endpoints are valid for this system?
+* do files need staging, and if so, what is the staging status?
+* how is an ORCID mapped to a local user?
+* what finalization work is needed after transfer?
+
+This interface deliberately combines search/discovery concerns with
+transfer-preparation concerns because real participating systems often expose
+both through the same service boundary.
+
+### Endpoint integrations
+
+The `endpoints.Endpoint` interface represents a transport-capable location. An
+endpoint implementation knows how to:
+
+* determine whether files are staged and valid at that location
+* start a transfer to another endpoint
+* poll for transfer status
+* cancel a transfer
+
+This keeps transport mechanics separate from data-system semantics. A database
+can say *what* must move, while an endpoint can say *how* it moves.
 
 ## Extensibility model
 
@@ -71,12 +176,46 @@ The main extensibility points are:
 * **Endpoints** implement the actual file movement mechanism, such as Globus,
   S3, or local filesystem transfers.
 * **Configuration** binds named databases and endpoints to concrete provider
-  implementations and credentials.
+  implementations, credentials, and deployment-specific settings.
 
 This model is what allows DTS to adapt to unrelated systems with different
 identifiers, metadata schemas, staging requirements, authentication needs, and
 transfer technologies.
 
+The configuration layer is especially important. DTS does not hardcode a single
+instance of a database or endpoint; it binds deployment-specific names to
+provider implementations and credentials. That makes the same codebase usable in
+environments where the same provider type may appear with different identities,
+roots, permissions, or organizational rules.
+
+The repository also supports a limited notion of custom destinations. That is a
+useful design clue: DTS is optimized for configured named systems, but it still
+allows controlled escape hatches when a transfer target cannot be modeled as a
+first-class database integration.
+
+## Authentication and trust model
+
+Authentication is separated from transfer orchestration. Requests arrive with a
+bearer token, and DTS resolves that token to a user identity with an ORCID. The
+service can do that through its local encrypted token file or by falling back to
+KBase authentication.
+
+That identity is then propagated into transfer logic so source and destination
+integrations can apply their own local authorization or federation rules. This
+matches the broader architectural pattern of the repository: DTS normalizes just
+enough identity information to coordinate work, but it does not try to replace
+each participating system's local trust model.
+
+## Persistence and recovery
+
+Long-running orchestration only works if state survives interruption. DTS
+persists transfer-related state under its configured data directory and restores
+that state when the service starts again. Database save/load hooks, store
+records, and the transfer journal all contribute to that recovery model.
+
+Architecturally, this means DTS is not just a thin wrapper over live provider
+calls. It is a durable coordinator with memory of in-flight and completed work.
+
 ## Repository layout
 
 Key areas of the repository are:
@@ -94,6 +233,13 @@ Key areas of the repository are:
 * `/docs`: MkDocs site for operators, integrators, and developers
 * `/integration`, `/dtstest`: integration-test support and test helpers
 
+The documentation structure mirrors the repository's main audiences:
+
+* **administrators** need deployment and configuration guidance
+* **integrators** need to understand the contracts their systems must satisfy
+* **developers** need to understand the internal orchestration and extension
+  model
+
 ## Why the architecture looks this way
 
 The repository favors orchestration over deep normalization. DTS does not try to
@@ -104,3 +250,9 @@ lets each integration satisfy that contract in its own way.
 That approach keeps the service practical for environments where participating
 systems are independently managed, evolve at different speeds, and may only
 share a small set of interoperable behaviors.
+
+The tradeoff is that some complexity is intentionally pushed to the edges of the
+system. Database and endpoint adapters absorb the messiness of real-world
+heterogeneity so that the client API and transfer lifecycle can stay coherent.
+That is the central architectural idea to keep in mind when working anywhere in
+this repository.

From c3192df56b8dd3c7ae9e3b8320422af544e3591f Mon Sep 17 00:00:00 2001
From: "Jeffrey N. Johnson" <jeff@cohere-llc.com>
Date: Mon, 28 Sep 2026 15:06:31 -0700
Subject: [PATCH 3/5] Filling in short summaries for each package.

---
 docs/developer/auth.md         | 11 ++++++++++-
 docs/developer/config.md       |  5 ++++-
 docs/developer/credit.md       |  4 +++-
 docs/developer/databases.md    | 33 ++++++++++++++++++++++++++++++++-
 docs/developer/endpoints.md    |  8 +++++++-
 docs/developer/frictionless.md |  4 ----
 docs/developer/index.md        |  3 ---
 docs/developer/services.md     |  5 ++++-
 docs/developer/tasks.md        |  5 -----
 docs/developer/transfers.md    | 21 +++++++++++++++++++++
 10 files changed, 81 insertions(+), 18 deletions(-)
 delete mode 100644 docs/developer/frictionless.md
 delete mode 100644 docs/developer/tasks.md
 create mode 100644 docs/developer/transfers.md

diff --git a/docs/developer/auth.md b/docs/developer/auth.md
index dbc602d7..924ab6bb 100644
--- a/docs/developer/auth.md
+++ b/docs/developer/auth.md
@@ -1,3 +1,12 @@
 # The `auth` Package
 
-**TODO: stuff goes here.**
+This package includes representations of users and credentials, along with a handful of
+authentication capabilities:
+
+* a proxy for the KBase auth server, which is used to authenticate users with KBase tokens
+* a proxy for the KBase Minio Management Service (MMS), which maps a KBase tokens to any additional
+  associated user credentials, like S3 for the KBase Lakehouse environment
+* a standalone DTS authenticator that reads data from an encrypted file on the local file system
+
+## Selected Topics
+
diff --git a/docs/developer/config.md b/docs/developer/config.md
index aea6358b..f35a88f6 100644
--- a/docs/developer/config.md
+++ b/docs/developer/config.md
@@ -1,3 +1,6 @@
 # The `config` Package
 
-**TODO: stuff goes here.**
+This package defines data structures that store configuration information for the various components
+of the DTS. Each of these data structures can be unmarshalled from YAML data, and can also be
+transmitted as a `map[string]any` using the [mapstructure](https://github.com/mitchellh/mapstructure)
+package.
diff --git a/docs/developer/credit.md b/docs/developer/credit.md
index 422e33e6..142dd175 100644
--- a/docs/developer/credit.md
+++ b/docs/developer/credit.md
@@ -1,4 +1,6 @@
 # The `credit` Package
 
-**TODO: stuff goes here.**
+This package contains code generated from the LinkML spec for the
+[KBase Credit Metadata Schema](https://kbase.github.io/credit_engine/). Credit information is stored
+in this schema in `credit` fields within a DTS file manifest included with every transfer payload.
 
diff --git a/docs/developer/databases.md b/docs/developer/databases.md
index bc639fa3..df0612bd 100644
--- a/docs/developer/databases.md
+++ b/docs/developer/databases.md
@@ -1,4 +1,35 @@
 # The `databases` Package
 
-**TODO: stuff goes here.**
+This package contains data structures and interfaces defining the functions of data sources for the
+DTS. Specific implementations of these data source interfaces live in subpackages within this
+package. Supported data sources are:
 
+* `jdp`: [The JGI Data Portal](https://data.jgi.doe.gov/)
+* `kbase`: [The KBase Workspace Servіce](https://kbase.us/services/ws/docs/index.html)
+* `kbase_lakehouse`: [The KBase Data Lakehouse](https://hub.berdl.kbase.us/hub/)
+* `nmdc`: [The DOE National Microbiome Data Collaborative](https://www.genomicscience.energy.gov/nmdc/)
+
+## Selected Topics
+
+### KBase User Federation
+
+The `Database` interface includes a `LocalUser` method that maps an [ORCID](https://orcid.org/) to
+a local username for the data source. This is needed to determine the final destination of the files
+in a transfer payload.
+
+For KBase-related data sources, this method is implemented by a simple
+[user federation goroutine](https://github.com/kbase/dts/blob/main/databases/kbase/user_federation.go)
+that reads a spreadsheet file (`kbase_user_orcids.csv`) from the DTS local data directory. The DTS
+reads this file on startup and then at the top of each hour, in case it has been updated. The file
+itself has two columns, which can appear in either possible order:
+
+* `username`: the local username for each KBase user
+* `orcid`: the ORCID corresponding to the local user in the same record
+
+Obviously, it would be good for this mapping to be performed automatically by a system of record.
+We use this file currently to allow the DTS to perform a transfer on behalf of a user different from
+the one whose credentials were used to authorize the request. This capability is used in the KBase
+WSS data source. The KBase Lakehouse data source, by contrast, requires that any user requesting a
+transfer provide their own KBase token. If this assumption is adopted throughout the DTS, user
+federation can be performed at the time of request authentication, eliminating the need for this
+kind of machinery.
diff --git a/docs/developer/endpoints.md b/docs/developer/endpoints.md
index 9fd9d723..b2cac6cb 100644
--- a/docs/developer/endpoints.md
+++ b/docs/developer/endpoints.md
@@ -1,3 +1,9 @@
 # The `endpoints` Package
 
-**TODO: stuff goes here.**
+This package contains interfaces and data structures representing transfer endpoints used by services
+like [Globus](https://www.globus.org/) and [Amazon S3](https://aws.amazon.com/s3/). These endpoints
+are used to initiate file transfers, check their status, and cancel them as needed.
+
+Currently, there are implementations for Globus, S3, and "local" endpoints. The bulk of the logic in
+these implementations is related to their providers, with logic for elementary operations kept
+separate from DTS business logic where practical.
diff --git a/docs/developer/frictionless.md b/docs/developer/frictionless.md
deleted file mode 100644
index 10b144b5..00000000
--- a/docs/developer/frictionless.md
+++ /dev/null
@@ -1,4 +0,0 @@
-# The `frictionless` Package
-
-**TODO: stuff goes here.**
-
diff --git a/docs/developer/index.md b/docs/developer/index.md
index b0b27299..09292970 100644
--- a/docs/developer/index.md
+++ b/docs/developer/index.md
@@ -29,9 +29,6 @@ the Data Transfer Service.
   integration of DTS with database providers
 * [endpoints](endpoints.md): defines endpoint types for file transfer
   providers used by DTS, such as [Globus](https://globus.org)
-* [frictionless](frictionless.md): defines [data structures](https://frictionlessdata.io/)
-  that describe data for [individual files](https://specs.frictionlessdata.io/data-resource/)
-  and [packages containing multiple files](https://specs.frictionlessdata.io/data-package/)
 * [services](services.md): defines types that implement the REST endpoints
   provided by the DTS
 * [tasks](tasks.md): implements the "heart" of the DTS, which creates and
diff --git a/docs/developer/services.md b/docs/developer/services.md
index ea971358..af7a6610 100644
--- a/docs/developer/services.md
+++ b/docs/developer/services.md
@@ -1,4 +1,7 @@
 # The `services` Package
 
-**TODO: stuff goes here.**
+This package implements the server portion of the DTS that exposes its API, dispatching requests via
+Goroutines. The server uses [Huma](https://huma.rocks/) to service requests. Documentation for each
+API endpoint is automatically generated from annotated input and output fields in the implementing
+function.
 
diff --git a/docs/developer/tasks.md b/docs/developer/tasks.md
deleted file mode 100644
index 24b9483e..00000000
--- a/docs/developer/tasks.md
+++ /dev/null
@@ -1,5 +0,0 @@
-# The `tasks` Package
-
-**TODO: stuff goes here.**
-
-
diff --git a/docs/developer/transfers.md b/docs/developer/transfers.md
new file mode 100644
index 00000000..08ae36b9
--- /dev/null
+++ b/docs/developer/transfers.md
@@ -0,0 +1,21 @@
+# The `transfers` Package
+
+This package contains data structures and functions that represent the lifecycle of every transfer
+in the DTS. This lifecycle is implemented by several concurrent goroutines in the idiom of 
+[communicating sequential processes](https://en.wikipedia.org/wiki/Communicating_sequential_processes).
+An interaction diagram of the lifecycle is shown below.
+
+![DTS transfer orchestration](https://github.com/kbase/dts/blob/main/transfers/orchestration.png)
+
+The important processes in this lifecycle are
+
+* the **dispatcher**, which accepts user requests forwarded from the REST API and orchestrates a
+  response involving one or more of the other processes
+* the **store**, which keeps records of transfers and their current status, allowing other
+  processes to create, update, and fetch these records
+* the **stager**, which handles the process of "staging" files for transfer (e. g. unarchiving them
+  from tape so they sit at the right location on disk)
+* the **mover**, which initiates transfers for sets of staged files, monitoring their status and
+  updating it within the store
+* the **manifestor**, which generates and transfers a JSON manifest file after each successful
+  file transfer

From b8a091e2a4dd08019ec940089d4392c65e51754a Mon Sep 17 00:00:00 2001
From: "Jeffrey N. Johnson" <jeff@cohere-llc.com>
Date: Mon, 28 Sep 2026 17:09:47 -0700
Subject: [PATCH 4/5] Adding technical remarks.

---
 docs/developer/auth.md      | 16 +++++++++++++++-
 docs/developer/config.md    | 12 ++++++++++++
 docs/developer/endpoints.md | 27 +++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)

diff --git a/docs/developer/auth.md b/docs/developer/auth.md
index 924ab6bb..c243bf3d 100644
--- a/docs/developer/auth.md
+++ b/docs/developer/auth.md
@@ -8,5 +8,19 @@ authentication capabilities:
   associated user credentials, like S3 for the KBase Lakehouse environment
 * a standalone DTS authenticator that reads data from an encrypted file on the local file system
 
-## Selected Topics
+## A Few Remarks
 
+The DTS standalone authenticator was created to allow users without KBase dev tokens to request
+transfers in the early days of the project. Given that KBase (in its old and new incarnations) is
+the only destination for DTS file transfers so far, it seems likely that this standalone
+authenticator is not strictly necessary. But it's pretty simple.
+
+**Who gets to request a file transfer?** Every resource-burdened request to the DTS must have an
+`Authorization` header with a valid KBase or DTS token. In a file transfer to the KBase WSS, this
+token need not belong to the user to whom the files are transferred -- each transfer accepts an
+ORCID that is mapped to a local user on the destination system, so an authenticated user can
+request file transfers on behalf of other users.
+
+In the KBase Lakehouse, the authenticated user is the only one authorized to receive the transferred
+files. Technically, this is because the KBase auth server passes this user's token to the MMS and
+returns the user's S3 credentials, which are needed to complete the transfer.
diff --git a/docs/developer/config.md b/docs/developer/config.md
index f35a88f6..05c4ac3c 100644
--- a/docs/developer/config.md
+++ b/docs/developer/config.md
@@ -4,3 +4,15 @@ This package defines data structures that store configuration information for th
 of the DTS. Each of these data structures can be unmarshalled from YAML data, and can also be
 transmitted as a `map[string]any` using the [mapstructure](https://github.com/mitchellh/mapstructure)
 package.
+
+## Selected Topics
+
+### `mapstructure`
+
+The `mapstructure` package was introduced to make it easier for other packages to accept independent
+configuration objects to aid in separate testing. Unfortunately,
+
+* how `mapstructure` works in practice is not necessarily how one expects, so the abstraction
+  is little more [leaky](https://en.wikipedia.org/wiki/Leaky_abstraction) than one might expect
+* the [repository](https://github.com/mitchellh/mapstructure) has been archived, which is not
+  something one likes to see in a component of "sustainable" software
diff --git a/docs/developer/endpoints.md b/docs/developer/endpoints.md
index b2cac6cb..e17d93cc 100644
--- a/docs/developer/endpoints.md
+++ b/docs/developer/endpoints.md
@@ -7,3 +7,30 @@ are used to initiate file transfers, check their status, and cancel them as need
 Currently, there are implementations for Globus, S3, and "local" endpoints. The bulk of the logic in
 these implementations is related to their providers, with logic for elementary operations kept
 separate from DTS business logic where practical.
+
+## Selected Topics
+
+### Globus Manifest Uploads
+
+The DTS generates a file manifest for each transferred payload, uploading it to the destination with
+the rest of its files. The generation and transfer of this manifest is a little different from the
+rest of the file transfer process, because the DTS writes this file itself, which means it needs
+access to a local file system. How, then, does the DTS upload the manifest? If the local file system
+exists within a Globus collection or share, the manifest can be transferred like any other file. But
+this isn't always possible.
+
+In cases where the DTS doesn't have local access to a file system that belongs to an existing
+Globus share, the DTS queries the destination endpoint to see if it supports
+[HTTPS access](https://docs.globus.org/globus-connect-server/v5.4/https-access-collections/). If
+HTTPS access is available, the DTS uses it to upload the manifest directly to the endpoint.
+
+### Premium Globus Connectors (e.g. S3)
+
+Globus offers several [premium storage connectors](https://www.globus.org/connectors) that allow
+file transfers between Globus endpoints and those maintained by other providers. We use a Globus
+S3 connector to transfer files to the KBase Data Lakehouse environment.
+
+How does the connector determine whether a user is authorized to transfer to one of these special
+endpoints? The Globus endpoint uses the [Globus Connect Manager Server API](https://docs.globus.org/globus-connect-server/v5.4/api/)
+to register the [user's corresponding credentials](https://docs.globus.org/globus-connect-server/v5.4/api/openapi_User_Credentials/)
+on the Connector endpoint, and then Globus seamlessly handles the file transfer.

From 403a07bb7f06fafa9d9e29b1de1e79ed3385a27e Mon Sep 17 00:00:00 2001
From: "Jeffrey N. Johnson" <jeff@cohere-llc.com>
Date: Wed, 30 Sep 2026 10:41:04 -0700
Subject: [PATCH 5/5] Documented a few more details.

---
 docs/admin/deployment.md    |  6 ++++++
 docs/admin/index.md         |  1 +
 docs/admin/kbase_wss.md     | 17 +++++++++++++++++
 docs/developer/auth.md      |  4 ++++
 docs/developer/databases.md |  8 +++++++-
 docs/developer/transfers.md |  9 +++++++++
 6 files changed, 44 insertions(+), 1 deletion(-)
 create mode 100644 docs/admin/kbase_wss.md

diff --git a/docs/admin/deployment.md b/docs/admin/deployment.md
index e1e627e4..8c399287 100644
--- a/docs/admin/deployment.md
+++ b/docs/admin/deployment.md
@@ -107,6 +107,12 @@ navigate to the `dts` deployment.
 
 That's it! You've now updated the service with new features and bugfixes.
 
+## Deploying to Argonne's Rancher2 Environment
+
+The DTS is automatically deployed to ANL's Rancher2 environment for the KBase Data Lakehouse
+by a [GitHub workflow](https://github.com/kbase/dts/blob/main/.github/workflows/build.yml)
+that invokes a standard KBase process.
+
 ## Maintaining the DTS Data Directory
 
 The DTS uses its data directory to manage its own state, as well as its
diff --git a/docs/admin/index.md b/docs/admin/index.md
index 3115cb4c..a7f346e5 100644
--- a/docs/admin/index.md
+++ b/docs/admin/index.md
@@ -11,4 +11,5 @@ services it needs to perform data transfers.
 * [Deploying DTS via Docker](deployment.md)
 * [Configuring DTS](config.md)
 * [Granting DTS Access to a Globus Endpoint](globus.md)
+* [The KBase Workspace Service (WSS) Environment](kbase_wss.md)
 
diff --git a/docs/admin/kbase_wss.md b/docs/admin/kbase_wss.md
new file mode 100644
index 00000000..b3b29b98
--- /dev/null
+++ b/docs/admin/kbase_wss.md
@@ -0,0 +1,17 @@
+# The KBase Workspace Service (WSS) Environment
+
+The Data Transfer Service (DTS) can transfer files to a user's staging folder within the 
+legacy KBase Workspace Service environment using the `kbase` database as a destination.
+In this environment, an authorized DTS user can request a transfer on behalf of another KBase
+user.
+
+The way this works behind the scenes is that the files are not transferred directly to the
+authorized user or to the user receiving the files. Instead, the files are transferred to
+a "holding pen". There, a file watcher process sits, observing the folders created and
+populated by the DTS. When a `manifest.json` file is completely written to a DTS transfer
+folder (name: `dts-<transfer-id>`), the watcher moves the files to their final destination.
+
+If this watcher process is not running, the files remain in this holding pen. This can be
+confusing to users performing transfers. If a user doesn't see the files they requested but
+sees that the transfer haѕ been completed (via a status check, for example), checking the
+status of the watcher process is a good first step.
diff --git a/docs/developer/auth.md b/docs/developer/auth.md
index c243bf3d..c20e0a05 100644
--- a/docs/developer/auth.md
+++ b/docs/developer/auth.md
@@ -24,3 +24,7 @@ request file transfers on behalf of other users.
 In the KBase Lakehouse, the authenticated user is the only one authorized to receive the transferred
 files. Technically, this is because the KBase auth server passes this user's token to the MMS and
 returns the user's S3 credentials, which are needed to complete the transfer.
+
+A user's S3 credentials are stored in a set of named credentials under the `auth.User`'s
+`ConnectionCredentials` field. The keys in this field are providers for whom credentials are
+stored (`s3`, `globus`, etc), and the values are credentials with IDs, usernameѕ, and secrets.
diff --git a/docs/developer/databases.md b/docs/developer/databases.md
index df0612bd..7c793f9b 100644
--- a/docs/developer/databases.md
+++ b/docs/developer/databases.md
@@ -21,10 +21,11 @@ For KBase-related data sources, this method is implemented by a simple
 [user federation goroutine](https://github.com/kbase/dts/blob/main/databases/kbase/user_federation.go)
 that reads a spreadsheet file (`kbase_user_orcids.csv`) from the DTS local data directory. The DTS
 reads this file on startup and then at the top of each hour, in case it has been updated. The file
-itself has two columns, which can appear in either possible order:
+itself has two or three columns, which can appear in either possible order:
 
 * `username`: the local username for each KBase user
 * `orcid`: the ORCID corresponding to the local user in the same record
+* `globusid`: optionally, the Globus ID corresponding to the local user in the same record
 
 Obviously, it would be good for this mapping to be performed automatically by a system of record.
 We use this file currently to allow the DTS to perform a transfer on behalf of a user different from
@@ -33,3 +34,8 @@ WSS data source. The KBase Lakehouse data source, by contrast, requires that any
 transfer provide their own KBase token. If this assumption is adopted throughout the DTS, user
 federation can be performed at the time of request authentication, eliminating the need for this
 kind of machinery.
+
+Why the Globus ID? A KBase user that wishes to perform transfers to the KBase Data Lakehouse
+via Globus must provide both S3 credentials and a valid Globus ID in order to use the Globus
+S3 connector. If a KBase user has no associated Globus ID, they will not be able to perform
+transfers directly to the lakehouse.
diff --git a/docs/developer/transfers.md b/docs/developer/transfers.md
index 08ae36b9..6f132fb8 100644
--- a/docs/developer/transfers.md
+++ b/docs/developer/transfers.md
@@ -19,3 +19,12 @@ The important processes in this lifecycle are
   updating it within the store
 * the **manifestor**, which generates and transfers a JSON manifest file after each successful
   file transfer
+
+## A Few Remarks
+
+In order to transmit a user's S3 credentials to Globus to transfer files into the KBase Data
+Lakehouse, we inserted a hack into the store process. In the hack, we check to see whether the
+destination database is the KBase Data Lakehouse, and if so, we read the Globus ID from the
+KBase user federation system and insert it into the user's `ConnectionCredentials` field where it
+can be read by the Globus endpoint downstream in order to authorize the transfer via the Globus
+S3 connector.