diff --git a/docs/admin/deployment.md b/docs/admin/deployment.md index e1e627e4..8c399287 100644 --- a/docs/admin/deployment.md +++ b/docs/admin/deployment.md @@ -107,6 +107,12 @@ navigate to the `dts` deployment. That's it! You've now updated the service with new features and bugfixes. +## Deploying to Argonne's Rancher2 Environment + +The DTS is automatically deployed to ANL's Rancher2 environment for the KBase Data Lakehouse +by a [GitHub workflow](https://github.com/kbase/dts/blob/main/.github/workflows/build.yml) +that invokes a standard KBase process. + ## Maintaining the DTS Data Directory The DTS uses its data directory to manage its own state, as well as its diff --git a/docs/admin/index.md b/docs/admin/index.md index 3115cb4c..a7f346e5 100644 --- a/docs/admin/index.md +++ b/docs/admin/index.md @@ -11,4 +11,5 @@ services it needs to perform data transfers. * [Deploying DTS via Docker](deployment.md) * [Configuring DTS](config.md) * [Granting DTS Access to a Globus Endpoint](globus.md) +* [The KBase Workspace Service (WSS) Environment](kbase_wss.md) diff --git a/docs/admin/kbase_wss.md b/docs/admin/kbase_wss.md new file mode 100644 index 00000000..b3b29b98 --- /dev/null +++ b/docs/admin/kbase_wss.md @@ -0,0 +1,17 @@ +# The KBase Workspace Service (WSS) Environment + +The Data Transfer Service (DTS) can transfer files to a user's staging folder within the +legacy KBase Workspace Service environment using the `kbase` database as a destination. +In this environment, an authorized DTS user can request a transfer on behalf of another KBase +user. + +The way this works behind the scenes is that the files are not transferred directly to the +authorized user or to the user receiving the files. Instead, the files are transferred to +a "holding pen". There, a file watcher process sits, observing the folders created and +populated by the DTS. When a `manifest.json` file is completely written to a DTS transfer +folder (name: `dts-`), the watcher moves the files to their final destination. + +If this watcher process is not running, the files remain in this holding pen. This can be +confusing to users performing transfers. If a user doesn't see the files they requested but +sees that the transfer haѕ been completed (via a status check, for example), checking the +status of the watcher process is a good first step. diff --git a/docs/developer/auth.md b/docs/developer/auth.md index dbc602d7..c20e0a05 100644 --- a/docs/developer/auth.md +++ b/docs/developer/auth.md @@ -1,3 +1,30 @@ # The `auth` Package -**TODO: stuff goes here.** +This package includes representations of users and credentials, along with a handful of +authentication capabilities: + +* a proxy for the KBase auth server, which is used to authenticate users with KBase tokens +* a proxy for the KBase Minio Management Service (MMS), which maps a KBase tokens to any additional + associated user credentials, like S3 for the KBase Lakehouse environment +* a standalone DTS authenticator that reads data from an encrypted file on the local file system + +## A Few Remarks + +The DTS standalone authenticator was created to allow users without KBase dev tokens to request +transfers in the early days of the project. Given that KBase (in its old and new incarnations) is +the only destination for DTS file transfers so far, it seems likely that this standalone +authenticator is not strictly necessary. But it's pretty simple. + +**Who gets to request a file transfer?** Every resource-burdened request to the DTS must have an +`Authorization` header with a valid KBase or DTS token. In a file transfer to the KBase WSS, this +token need not belong to the user to whom the files are transferred -- each transfer accepts an +ORCID that is mapped to a local user on the destination system, so an authenticated user can +request file transfers on behalf of other users. + +In the KBase Lakehouse, the authenticated user is the only one authorized to receive the transferred +files. Technically, this is because the KBase auth server passes this user's token to the MMS and +returns the user's S3 credentials, which are needed to complete the transfer. + +A user's S3 credentials are stored in a set of named credentials under the `auth.User`'s +`ConnectionCredentials` field. The keys in this field are providers for whom credentials are +stored (`s3`, `globus`, etc), and the values are credentials with IDs, usernameѕ, and secrets. diff --git a/docs/developer/config.md b/docs/developer/config.md index aea6358b..05c4ac3c 100644 --- a/docs/developer/config.md +++ b/docs/developer/config.md @@ -1,3 +1,18 @@ # The `config` Package -**TODO: stuff goes here.** +This package defines data structures that store configuration information for the various components +of the DTS. Each of these data structures can be unmarshalled from YAML data, and can also be +transmitted as a `map[string]any` using the [mapstructure](https://github.com/mitchellh/mapstructure) +package. + +## Selected Topics + +### `mapstructure` + +The `mapstructure` package was introduced to make it easier for other packages to accept independent +configuration objects to aid in separate testing. Unfortunately, + +* how `mapstructure` works in practice is not necessarily how one expects, so the abstraction + is little more [leaky](https://en.wikipedia.org/wiki/Leaky_abstraction) than one might expect +* the [repository](https://github.com/mitchellh/mapstructure) has been archived, which is not + something one likes to see in a component of "sustainable" software diff --git a/docs/developer/credit.md b/docs/developer/credit.md index 422e33e6..142dd175 100644 --- a/docs/developer/credit.md +++ b/docs/developer/credit.md @@ -1,4 +1,6 @@ # The `credit` Package -**TODO: stuff goes here.** +This package contains code generated from the LinkML spec for the +[KBase Credit Metadata Schema](https://kbase.github.io/credit_engine/). Credit information is stored +in this schema in `credit` fields within a DTS file manifest included with every transfer payload. diff --git a/docs/developer/databases.md b/docs/developer/databases.md index bc639fa3..7c793f9b 100644 --- a/docs/developer/databases.md +++ b/docs/developer/databases.md @@ -1,4 +1,41 @@ # The `databases` Package -**TODO: stuff goes here.** +This package contains data structures and interfaces defining the functions of data sources for the +DTS. Specific implementations of these data source interfaces live in subpackages within this +package. Supported data sources are: +* `jdp`: [The JGI Data Portal](https://data.jgi.doe.gov/) +* `kbase`: [The KBase Workspace Servіce](https://kbase.us/services/ws/docs/index.html) +* `kbase_lakehouse`: [The KBase Data Lakehouse](https://hub.berdl.kbase.us/hub/) +* `nmdc`: [The DOE National Microbiome Data Collaborative](https://www.genomicscience.energy.gov/nmdc/) + +## Selected Topics + +### KBase User Federation + +The `Database` interface includes a `LocalUser` method that maps an [ORCID](https://orcid.org/) to +a local username for the data source. This is needed to determine the final destination of the files +in a transfer payload. + +For KBase-related data sources, this method is implemented by a simple +[user federation goroutine](https://github.com/kbase/dts/blob/main/databases/kbase/user_federation.go) +that reads a spreadsheet file (`kbase_user_orcids.csv`) from the DTS local data directory. The DTS +reads this file on startup and then at the top of each hour, in case it has been updated. The file +itself has two or three columns, which can appear in either possible order: + +* `username`: the local username for each KBase user +* `orcid`: the ORCID corresponding to the local user in the same record +* `globusid`: optionally, the Globus ID corresponding to the local user in the same record + +Obviously, it would be good for this mapping to be performed automatically by a system of record. +We use this file currently to allow the DTS to perform a transfer on behalf of a user different from +the one whose credentials were used to authorize the request. This capability is used in the KBase +WSS data source. The KBase Lakehouse data source, by contrast, requires that any user requesting a +transfer provide their own KBase token. If this assumption is adopted throughout the DTS, user +federation can be performed at the time of request authentication, eliminating the need for this +kind of machinery. + +Why the Globus ID? A KBase user that wishes to perform transfers to the KBase Data Lakehouse +via Globus must provide both S3 credentials and a valid Globus ID in order to use the Globus +S3 connector. If a KBase user has no associated Globus ID, they will not be able to perform +transfers directly to the lakehouse. diff --git a/docs/developer/endpoints.md b/docs/developer/endpoints.md index 9fd9d723..e17d93cc 100644 --- a/docs/developer/endpoints.md +++ b/docs/developer/endpoints.md @@ -1,3 +1,36 @@ # The `endpoints` Package -**TODO: stuff goes here.** +This package contains interfaces and data structures representing transfer endpoints used by services +like [Globus](https://www.globus.org/) and [Amazon S3](https://aws.amazon.com/s3/). These endpoints +are used to initiate file transfers, check their status, and cancel them as needed. + +Currently, there are implementations for Globus, S3, and "local" endpoints. The bulk of the logic in +these implementations is related to their providers, with logic for elementary operations kept +separate from DTS business logic where practical. + +## Selected Topics + +### Globus Manifest Uploads + +The DTS generates a file manifest for each transferred payload, uploading it to the destination with +the rest of its files. The generation and transfer of this manifest is a little different from the +rest of the file transfer process, because the DTS writes this file itself, which means it needs +access to a local file system. How, then, does the DTS upload the manifest? If the local file system +exists within a Globus collection or share, the manifest can be transferred like any other file. But +this isn't always possible. + +In cases where the DTS doesn't have local access to a file system that belongs to an existing +Globus share, the DTS queries the destination endpoint to see if it supports +[HTTPS access](https://docs.globus.org/globus-connect-server/v5.4/https-access-collections/). If +HTTPS access is available, the DTS uses it to upload the manifest directly to the endpoint. + +### Premium Globus Connectors (e.g. S3) + +Globus offers several [premium storage connectors](https://www.globus.org/connectors) that allow +file transfers between Globus endpoints and those maintained by other providers. We use a Globus +S3 connector to transfer files to the KBase Data Lakehouse environment. + +How does the connector determine whether a user is authorized to transfer to one of these special +endpoints? The Globus endpoint uses the [Globus Connect Manager Server API](https://docs.globus.org/globus-connect-server/v5.4/api/) +to register the [user's corresponding credentials](https://docs.globus.org/globus-connect-server/v5.4/api/openapi_User_Credentials/) +on the Connector endpoint, and then Globus seamlessly handles the file transfer. diff --git a/docs/developer/frictionless.md b/docs/developer/frictionless.md deleted file mode 100644 index 10b144b5..00000000 --- a/docs/developer/frictionless.md +++ /dev/null @@ -1,4 +0,0 @@ -# The `frictionless` Package - -**TODO: stuff goes here.** - diff --git a/docs/developer/index.md b/docs/developer/index.md index 908dafe6..09292970 100644 --- a/docs/developer/index.md +++ b/docs/developer/index.md @@ -2,6 +2,12 @@ This guide contains technical information about the Data Transfer Service (DTS). +## Start Here + +* [Repository Overview](repository_overview.md): purpose, architecture, + architectural drivers, runtime model, extension points, and the major + directories in this repository + ## Architecture Decision Records [Here](adrs/index.md) we've recorded design decisions in a succinct format that @@ -23,9 +29,6 @@ the Data Transfer Service. integration of DTS with database providers * [endpoints](endpoints.md): defines endpoint types for file transfer providers used by DTS, such as [Globus](https://globus.org) -* [frictionless](frictionless.md): defines [data structures](https://frictionlessdata.io/) - that describe data for [individual files](https://specs.frictionlessdata.io/data-resource/) - and [packages containing multiple files](https://specs.frictionlessdata.io/data-package/) * [services](services.md): defines types that implement the REST endpoints provided by the DTS * [tasks](tasks.md): implements the "heart" of the DTS, which creates and diff --git a/docs/developer/repository_overview.md b/docs/developer/repository_overview.md new file mode 100644 index 00000000..37562dcd --- /dev/null +++ b/docs/developer/repository_overview.md @@ -0,0 +1,258 @@ +# Repository Overview + +The Data Transfer Service (DTS) is a Go service that gives clients one API for +searching remote data holdings, collecting file metadata, and requesting file +transfers between participating systems. + +The core design assumption in this repository is that DTS must work with a +highly heterogeneous, continuously changing set of unrelated systems. Rather +than requiring a single storage model, metadata shape, identity model, or +transfer stack, DTS isolates system-specific behavior behind pluggable database +and endpoint interfaces and coordinates transfers through a shared workflow. + +## Purpose + +DTS sits between clients and external data providers. It is responsible for: + +* authenticating requests and associating them with users +* exposing REST endpoints for search, metadata lookup, transfer creation, and + transfer status +* translating those requests into calls against configured source and + destination systems +* coordinating staging, transfer, manifest generation, finalization, and + persistence of transfer state + +In practice, this means the service can present a stable API even when source +databases, destination databases, and transfer providers all have different +capabilities and conventions. + +At a conceptual level, DTS is neither a storage platform nor a transfer engine +in its own right. It is an orchestration service. It accepts a user-facing +request, identifies which participating systems must be involved, and drives +those systems through a normalized lifecycle that is simple enough for clients +but flexible enough for operators and integrators. + +## Architectural drivers + +Several design pressures explain why the codebase looks the way it does: + +* **Heterogeneity is expected, not exceptional.** Participating systems may use + different identifiers, metadata structures, staging workflows, trust models, + and transfer technologies. +* **Integrations evolve independently.** DTS assumes external systems are owned + by different organizations and may change on different release cycles. +* **The interoperable contract is intentionally narrow.** DTS only standardizes + the capabilities it truly needs: search, metadata lookup, staging, transport, + finalization, and user mapping. +* **Transfers are long-running.** The service must continue work outside the + lifetime of a single HTTP request and recover from restarts. +* **Provenance matters alongside payload delivery.** A completed transfer is not + just copied files; it also includes manifest and metadata production. + +These drivers lead the repository to emphasize interfaces, orchestration +components, and persisted state rather than a single deep domain model shared by +all integrations. + +## High-level architecture + +DTS is organized into a few layers: + +1. **Service/API layer** (`main.go`, `services/`): starts the process, loads + configuration, exposes the HTTP API, and validates user requests. +2. **Workflow/orchestration layer** (`transfers/`): owns transfer lifecycle + management, persistence, status updates, and coordination between staging, + moving, and finalization. +3. **Integration layer** (`databases/`, `endpoints/`, `auth/`): adapts + database-specific, endpoint-specific, and authentication-specific behavior to + common interfaces. +4. **Support packages** (`config/`, `journal/`, `credit/`): provide + configuration loading, transfer journaling, and metadata/provenance support. + +This separation lets the API and workflow logic stay mostly stable while new +systems are introduced through integration packages and configuration. + +Another useful lens is to think about two different kinds of stability: + +* the **northbound** surface is stable and client-oriented: a small REST API, + predictable transfer states, and a uniform request model +* the **southbound** surface is adaptable and integration-oriented: provider + implementations, credentials, endpoint details, and local policy + +The repository exists to keep those two surfaces decoupled. + +## Core runtime model + +At runtime, the service follows a coordinator pattern: + +* `main.go` loads configuration, initializes logging, creates the service, and + owns process startup and graceful shutdown. +* `services/` exposes the HTTP API and translates requests into transfer-domain + operations. +* `transfers/` runs the asynchronous transfer subsystem and owns the long-lived + state machine for each transfer. + +Inside `transfers/`, several cooperating components divide responsibility: + +* the **dispatcher** accepts requests, creates transfer records, and decides + which step comes next +* the **store** is the system of record for transfer specifications, + descriptors, payload size, and status +* the **stager** handles source-side preparation when files are not immediately + transferable +* the **mover** delegates payload copy to endpoint providers and monitors + progress +* the **manifestor** builds and transfers the manifest that describes the + payload and its metadata + +These components communicate through channels and background goroutines. That is +an important architectural choice: DTS treats transfers as asynchronous +workflows rather than request-scoped operations. + +## Transfer lifecycle + +A typical transfer moves through these steps: + +1. The API receives a search, metadata, or transfer request. +2. The service authenticates the caller and validates the request. +3. The dispatcher records the transfer specification and inspects the source and + destination requirements. +4. If staging is required, the stager requests source-side preparation and polls + until the files are ready. +5. The mover groups payload files by source endpoint and asks the relevant + endpoint provider to move them to the destination. +6. The manifestor generates a Frictionless-style manifest describing the + transfer and places that manifest with the payload. +7. The destination-side finalization logic completes any system-specific + post-transfer processing. +8. The journal records the outcome, and completed state remains available until + retention rules remove it. + +This lifecycle is deliberately broken into stages because different source and +destination systems become ready at different times. Some payloads can be moved +immediately; others must first be restored from archival storage, staged onto a +filesystem, or associated with a destination-specific import step. DTS handles +that variation by advancing transfers through coarse-grained states instead of +assuming one synchronous copy operation. + +## Abstraction boundaries + +Two interfaces carry most of the repository's heterogeneity. + +### Database integrations + +The `databases.Database` interface represents a participating data system from +the perspective of DTS. A database implementation answers questions such as: + +* what files match a search query? +* what descriptors and metadata correspond to a set of file IDs? +* which transfer endpoints are valid for this system? +* do files need staging, and if so, what is the staging status? +* how is an ORCID mapped to a local user? +* what finalization work is needed after transfer? + +This interface deliberately combines search/discovery concerns with +transfer-preparation concerns because real participating systems often expose +both through the same service boundary. + +### Endpoint integrations + +The `endpoints.Endpoint` interface represents a transport-capable location. An +endpoint implementation knows how to: + +* determine whether files are staged and valid at that location +* start a transfer to another endpoint +* poll for transfer status +* cancel a transfer + +This keeps transport mechanics separate from data-system semantics. A database +can say *what* must move, while an endpoint can say *how* it moves. + +## Extensibility model + +The main extensibility points are: + +* **Databases** implement search, metadata lookup, optional staging, user + federation, and finalization behavior for a participating system. +* **Endpoints** implement the actual file movement mechanism, such as Globus, + S3, or local filesystem transfers. +* **Configuration** binds named databases and endpoints to concrete provider + implementations, credentials, and deployment-specific settings. + +This model is what allows DTS to adapt to unrelated systems with different +identifiers, metadata schemas, staging requirements, authentication needs, and +transfer technologies. + +The configuration layer is especially important. DTS does not hardcode a single +instance of a database or endpoint; it binds deployment-specific names to +provider implementations and credentials. That makes the same codebase usable in +environments where the same provider type may appear with different identities, +roots, permissions, or organizational rules. + +The repository also supports a limited notion of custom destinations. That is a +useful design clue: DTS is optimized for configured named systems, but it still +allows controlled escape hatches when a transfer target cannot be modeled as a +first-class database integration. + +## Authentication and trust model + +Authentication is separated from transfer orchestration. Requests arrive with a +bearer token, and DTS resolves that token to a user identity with an ORCID. The +service can do that through its local encrypted token file or by falling back to +KBase authentication. + +That identity is then propagated into transfer logic so source and destination +integrations can apply their own local authorization or federation rules. This +matches the broader architectural pattern of the repository: DTS normalizes just +enough identity information to coordinate work, but it does not try to replace +each participating system's local trust model. + +## Persistence and recovery + +Long-running orchestration only works if state survives interruption. DTS +persists transfer-related state under its configured data directory and restores +that state when the service starts again. Database save/load hooks, store +records, and the transfer journal all contribute to that recovery model. + +Architecturally, this means DTS is not just a thin wrapper over live provider +calls. It is a durable coordinator with memory of in-flight and completed work. + +## Repository layout + +Key areas of the repository are: + +* `/main.go`: process startup and shutdown +* `/services`: REST API handlers and transport-facing types +* `/transfers`: dispatcher, store, stager, mover, manifestor, and transfer + state management +* `/databases`: shared database interface plus concrete integrations such as + JDP, KBase, NMDC, and S3-backed databases +* `/endpoints`: shared endpoint interface plus concrete transfer providers +* `/auth`: request authentication and user identity lookup +* `/config`: YAML configuration loading and validation +* `/journal`: transfer record persistence +* `/docs`: MkDocs site for operators, integrators, and developers +* `/integration`, `/dtstest`: integration-test support and test helpers + +The documentation structure mirrors the repository's main audiences: + +* **administrators** need deployment and configuration guidance +* **integrators** need to understand the contracts their systems must satisfy +* **developers** need to understand the internal orchestration and extension + model + +## Why the architecture looks this way + +The repository favors orchestration over deep normalization. DTS does not try to +force every external system into the same internal storage or execution model. +Instead, it defines a narrow common contract for the capabilities it needs and +lets each integration satisfy that contract in its own way. + +That approach keeps the service practical for environments where participating +systems are independently managed, evolve at different speeds, and may only +share a small set of interoperable behaviors. + +The tradeoff is that some complexity is intentionally pushed to the edges of the +system. Database and endpoint adapters absorb the messiness of real-world +heterogeneity so that the client API and transfer lifecycle can stay coherent. +That is the central architectural idea to keep in mind when working anywhere in +this repository. diff --git a/docs/developer/services.md b/docs/developer/services.md index ea971358..af7a6610 100644 --- a/docs/developer/services.md +++ b/docs/developer/services.md @@ -1,4 +1,7 @@ # The `services` Package -**TODO: stuff goes here.** +This package implements the server portion of the DTS that exposes its API, dispatching requests via +Goroutines. The server uses [Huma](https://huma.rocks/) to service requests. Documentation for each +API endpoint is automatically generated from annotated input and output fields in the implementing +function. diff --git a/docs/developer/tasks.md b/docs/developer/tasks.md deleted file mode 100644 index 24b9483e..00000000 --- a/docs/developer/tasks.md +++ /dev/null @@ -1,5 +0,0 @@ -# The `tasks` Package - -**TODO: stuff goes here.** - - diff --git a/docs/developer/transfers.md b/docs/developer/transfers.md new file mode 100644 index 00000000..6f132fb8 --- /dev/null +++ b/docs/developer/transfers.md @@ -0,0 +1,30 @@ +# The `transfers` Package + +This package contains data structures and functions that represent the lifecycle of every transfer +in the DTS. This lifecycle is implemented by several concurrent goroutines in the idiom of +[communicating sequential processes](https://en.wikipedia.org/wiki/Communicating_sequential_processes). +An interaction diagram of the lifecycle is shown below. + +![DTS transfer orchestration](https://github.com/kbase/dts/blob/main/transfers/orchestration.png) + +The important processes in this lifecycle are + +* the **dispatcher**, which accepts user requests forwarded from the REST API and orchestrates a + response involving one or more of the other processes +* the **store**, which keeps records of transfers and their current status, allowing other + processes to create, update, and fetch these records +* the **stager**, which handles the process of "staging" files for transfer (e. g. unarchiving them + from tape so they sit at the right location on disk) +* the **mover**, which initiates transfers for sets of staged files, monitoring their status and + updating it within the store +* the **manifestor**, which generates and transfers a JSON manifest file after each successful + file transfer + +## A Few Remarks + +In order to transmit a user's S3 credentials to Globus to transfer files into the KBase Data +Lakehouse, we inserted a hack into the store process. In the hack, we check to see whether the +destination database is the KBase Data Lakehouse, and if so, we read the Globus ID from the +KBase user federation system and insert it into the user's `ConnectionCredentials` field where it +can be read by the Globus endpoint downstream in order to authorize the transfer via the Globus +S3 connector. diff --git a/mkdocs.yml b/mkdocs.yml index cd9de71b..94d9f66d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -19,6 +19,7 @@ nav: - 'Glossary': 'integration/glossary.md' - 'Developer Guide': - 'Overview': 'developer/index.md' + - 'Repository Overview': 'developer/repository_overview.md' - 'Architecture Decision Records': - 'developer/adrs/index.md' - 'Transmitting Metadata': 'developer/adrs/transmit-metadata.md'