Add repository overview docs

docs/admin/deployment.md

@@ -107,6 +107,12 @@ navigate to the `dts` deployment.
 
 That's it! You've now updated the service with new features and bugfixes.
 
+## Deploying to Argonne's Rancher2 Environment
+
+The DTS is automatically deployed to ANL's Rancher2 environment for the KBase Data Lakehouse
+by a [GitHub workflow](https://github.com/kbase/dts/blob/main/.github/workflows/build.yml)
+that invokes a standard KBase process.
+
 ## Maintaining the DTS Data Directory
 
 The DTS uses its data directory to manage its own state, as well as its

docs/admin/index.md

@@ -11,4 +11,5 @@ services it needs to perform data transfers.
 * [Deploying DTS via Docker](deployment.md)
 * [Configuring DTS](config.md)
 * [Granting DTS Access to a Globus Endpoint](globus.md)
+* [The KBase Workspace Service (WSS) Environment](kbase_wss.md)
 

docs/admin/kbase_wss.md

@@ -0,0 +1,17 @@
+# The KBase Workspace Service (WSS) Environment
+
+The Data Transfer Service (DTS) can transfer files to a user's staging folder within the 
+legacy KBase Workspace Service environment using the `kbase` database as a destination.
+In this environment, an authorized DTS user can request a transfer on behalf of another KBase
+user.
+
+The way this works behind the scenes is that the files are not transferred directly to the
+authorized user or to the user receiving the files. Instead, the files are transferred to
+a "holding pen". There, a file watcher process sits, observing the folders created and
+populated by the DTS. When a `manifest.json` file is completely written to a DTS transfer
+folder (name: `dts-<transfer-id>`), the watcher moves the files to their final destination.
+
+If this watcher process is not running, the files remain in this holding pen. This can be
+confusing to users performing transfers. If a user doesn't see the files they requested but
+sees that the transfer haѕ been completed (via a status check, for example), checking the
+status of the watcher process is a good first step.

docs/developer/auth.md

@@ -1,3 +1,30 @@
 # The `auth` Package
 
-**TODO: stuff goes here.**
+This package includes representations of users and credentials, along with a handful of
+authentication capabilities:
+
+* a proxy for the KBase auth server, which is used to authenticate users with KBase tokens
+* a proxy for the KBase Minio Management Service (MMS), which maps a KBase tokens to any additional
+  associated user credentials, like S3 for the KBase Lakehouse environment
+* a standalone DTS authenticator that reads data from an encrypted file on the local file system
+
+## A Few Remarks
+
+The DTS standalone authenticator was created to allow users without KBase dev tokens to request
+transfers in the early days of the project. Given that KBase (in its old and new incarnations) is
+the only destination for DTS file transfers so far, it seems likely that this standalone
+authenticator is not strictly necessary. But it's pretty simple.
+
+**Who gets to request a file transfer?** Every resource-burdened request to the DTS must have an
+`Authorization` header with a valid KBase or DTS token. In a file transfer to the KBase WSS, this
+token need not belong to the user to whom the files are transferred -- each transfer accepts an
+ORCID that is mapped to a local user on the destination system, so an authenticated user can
+request file transfers on behalf of other users.
+
+In the KBase Lakehouse, the authenticated user is the only one authorized to receive the transferred
+files. Technically, this is because the KBase auth server passes this user's token to the MMS and
+returns the user's S3 credentials, which are needed to complete the transfer.
+
+A user's S3 credentials are stored in a set of named credentials under the `auth.User`'s
+`ConnectionCredentials` field. The keys in this field are providers for whom credentials are
+stored (`s3`, `globus`, etc), and the values are credentials with IDs, usernameѕ, and secrets.

docs/developer/config.md

@@ -1,3 +1,18 @@
 # The `config` Package
 
-**TODO: stuff goes here.**
+This package defines data structures that store configuration information for the various components
+of the DTS. Each of these data structures can be unmarshalled from YAML data, and can also be
+transmitted as a `map[string]any` using the [mapstructure](https://github.com/mitchellh/mapstructure)
+package.
+
+## Selected Topics
+
+### `mapstructure`
+
+The `mapstructure` package was introduced to make it easier for other packages to accept independent
+configuration objects to aid in separate testing. Unfortunately,
+
+* how `mapstructure` works in practice is not necessarily how one expects, so the abstraction
+  is little more [leaky](https://en.wikipedia.org/wiki/Leaky_abstraction) than one might expect
+* the [repository](https://github.com/mitchellh/mapstructure) has been archived, which is not
+  something one likes to see in a component of "sustainable" software

docs/developer/credit.md

@@ -1,4 +1,6 @@
 # The `credit` Package
 
-**TODO: stuff goes here.**
+This package contains code generated from the LinkML spec for the
+[KBase Credit Metadata Schema](https://kbase.github.io/credit_engine/). Credit information is stored
+in this schema in `credit` fields within a DTS file manifest included with every transfer payload.
 

docs/developer/databases.md

@@ -1,4 +1,41 @@
 # The `databases` Package
 
-**TODO: stuff goes here.**
+This package contains data structures and interfaces defining the functions of data sources for the
+DTS. Specific implementations of these data source interfaces live in subpackages within this
+package. Supported data sources are:
 
+* `jdp`: [The JGI Data Portal](https://data.jgi.doe.gov/)
+* `kbase`: [The KBase Workspace Servіce](https://kbase.us/services/ws/docs/index.html)
+* `kbase_lakehouse`: [The KBase Data Lakehouse](https://hub.berdl.kbase.us/hub/)
+* `nmdc`: [The DOE National Microbiome Data Collaborative](https://www.genomicscience.energy.gov/nmdc/)
+
+## Selected Topics
+
+### KBase User Federation
+
+The `Database` interface includes a `LocalUser` method that maps an [ORCID](https://orcid.org/) to
+a local username for the data source. This is needed to determine the final destination of the files
+in a transfer payload.
+
+For KBase-related data sources, this method is implemented by a simple
+[user federation goroutine](https://github.com/kbase/dts/blob/main/databases/kbase/user_federation.go)
+that reads a spreadsheet file (`kbase_user_orcids.csv`) from the DTS local data directory. The DTS
+reads this file on startup and then at the top of each hour, in case it has been updated. The file
+itself has two or three columns, which can appear in either possible order:
+
+* `username`: the local username for each KBase user
+* `orcid`: the ORCID corresponding to the local user in the same record
+* `globusid`: optionally, the Globus ID corresponding to the local user in the same record
+
+Obviously, it would be good for this mapping to be performed automatically by a system of record.
+We use this file currently to allow the DTS to perform a transfer on behalf of a user different from
+the one whose credentials were used to authorize the request. This capability is used in the KBase
+WSS data source. The KBase Lakehouse data source, by contrast, requires that any user requesting a
+transfer provide their own KBase token. If this assumption is adopted throughout the DTS, user
+federation can be performed at the time of request authentication, eliminating the need for this
+kind of machinery.
+
+Why the Globus ID? A KBase user that wishes to perform transfers to the KBase Data Lakehouse
+via Globus must provide both S3 credentials and a valid Globus ID in order to use the Globus
+S3 connector. If a KBase user has no associated Globus ID, they will not be able to perform
+transfers directly to the lakehouse.

docs/developer/endpoints.md

@@ -1,3 +1,36 @@
 # The `endpoints` Package
 
-**TODO: stuff goes here.**
+This package contains interfaces and data structures representing transfer endpoints used by services
+like [Globus](https://www.globus.org/) and [Amazon S3](https://aws.amazon.com/s3/). These endpoints
+are used to initiate file transfers, check their status, and cancel them as needed.
+
+Currently, there are implementations for Globus, S3, and "local" endpoints. The bulk of the logic in
+these implementations is related to their providers, with logic for elementary operations kept
+separate from DTS business logic where practical.
+
+## Selected Topics
+
+### Globus Manifest Uploads
+
+The DTS generates a file manifest for each transferred payload, uploading it to the destination with
+the rest of its files. The generation and transfer of this manifest is a little different from the
+rest of the file transfer process, because the DTS writes this file itself, which means it needs
+access to a local file system. How, then, does the DTS upload the manifest? If the local file system
+exists within a Globus collection or share, the manifest can be transferred like any other file. But
+this isn't always possible.
+
+In cases where the DTS doesn't have local access to a file system that belongs to an existing
+Globus share, the DTS queries the destination endpoint to see if it supports
+[HTTPS access](https://docs.globus.org/globus-connect-server/v5.4/https-access-collections/). If
+HTTPS access is available, the DTS uses it to upload the manifest directly to the endpoint.
+
+### Premium Globus Connectors (e.g. S3)
+
+Globus offers several [premium storage connectors](https://www.globus.org/connectors) that allow
+file transfers between Globus endpoints and those maintained by other providers. We use a Globus
+S3 connector to transfer files to the KBase Data Lakehouse environment.
+
+How does the connector determine whether a user is authorized to transfer to one of these special
+endpoints? The Globus endpoint uses the [Globus Connect Manager Server API](https://docs.globus.org/globus-connect-server/v5.4/api/)
+to register the [user's corresponding credentials](https://docs.globus.org/globus-connect-server/v5.4/api/openapi_User_Credentials/)
+on the Connector endpoint, and then Globus seamlessly handles the file transfer.

docs/developer/index.md

@@ -2,6 +2,12 @@
 
 This guide contains technical information about the Data Transfer Service (DTS).
 
+## Start Here
+
+* [Repository Overview](repository_overview.md): purpose, architecture,
+  architectural drivers, runtime model, extension points, and the major
+  directories in this repository
+
 ## Architecture Decision Records
 
 [Here](adrs/index.md) we've recorded design decisions in a succinct format that
@@ -23,9 +29,6 @@ the Data Transfer Service.
   integration of DTS with database providers
 * [endpoints](endpoints.md): defines endpoint types for file transfer
   providers used by DTS, such as [Globus](https://globus.org)
-* [frictionless](frictionless.md): defines [data structures](https://frictionlessdata.io/)
-  that describe data for [individual files](https://specs.frictionlessdata.io/data-resource/)
-  and [packages containing multiple files](https://specs.frictionlessdata.io/data-package/)
 * [services](services.md): defines types that implement the REST endpoints
   provided by the DTS
 * [tasks](tasks.md): implements the "heart" of the DTS, which creates and