Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
986 lines (794 loc) · 43.6 KB

ARCHITECTURE.md

File metadata and controls

986 lines (794 loc) · 43.6 KB
Raw

bssh Architecture Documentation

Overview

bssh (Broadcast SSH / Backend.AI SSH) is a high-performance parallel SSH command execution tool with SSH-compatible interface. This document provides a high-level architecture overview. For detailed component documentation, see docs/architecture/.

Core Capabilities

  • Parallel command execution across multiple nodes
  • SSH-compatible command-line interface (drop-in replacement)
  • SSH port forwarding (-L, -R, -D/SOCKS proxy)
  • SSH jump host support (-J)
  • SSH configuration file parsing (-F)
  • Interactive PTY sessions with single/multiplex modes
  • SFTP file transfers (upload/download) with bounded pipelined streaming
  • Backend.AI cluster auto-detection
  • pdsh compatibility mode

System Architecture

        ┌─────────────────────────────────────────────────────────┐
        │                     CLI Interface                       │
        │                       (main.rs)                         │
        │        (-L, -R, -D, -J, -F, -t/T, SSH-compatible)       │
        └────────────────────────────┬────────────────────────────┘
                                     │
        ┌─────────────┬──────────────┼──────────────┬─────────────┐
        ▼             ▼              ▼              ▼             ▼
┌──────────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌──────────┐
│   Commands   │ │  Config   │ │  Utils    │ │Forwarding │ │   Jump   │
│   Module     │ │  Manager  │ │  Module   │ │  Manager  │ │   Host   │
│ (commands/*) │ │(config.rs)│ │ (utils/*) │ │(forward/*)│ │ (jump/*) │
└──────┬───────┘ └─────┬─────┘ └───────────┘ └───┬───────┘ └───┬──────┘
       │               │                         │             │
       │               ▼                         │             │
       │       ┌──────────────┐                  │             │
       │       │ SSH Config   │                  │             │
       │       │    Parser    │                  │             │
       │       │(ssh_config/*)│                  │             │
       │       └──────────────┘                  │             │
       ▼                                         ▼             ▼
┌──────────────┐                         ┌──────────────┐ ┌──────────────────┐
│   Executor   │◄────────────────────────┤     Node     │ │ Port Forwarders  │
│  (Parallel)  │                         │    Parser    │ │  (L/R/D modes)   │
│(executor.rs) │                         │  (node.rs)   │ │    + Tunnels     │
└──────┬───────┘                         └──────────────┘ └────────┬─────────┘
       │                                                           │
       ├──────────┬────────────┬───────────────────────────────────┘
       ▼          ▼            ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│   SSH    │ │   SSH    │ │   SSH    │
│  Client  │ │  Client  │ │  Client  │
│ (russh)  │ │ (russh)  │ │ (russh)  │
└──────────┘ └──────────┘ └──────────┘

Component Summary

CLI Interface

Documentation: docs/architecture/cli-interface.md

The CLI system provides an SSH-compatible command-line interface with multiple operation modes:

  • Native bssh mode: Cluster-based parallel execution
  • SSH compatibility mode: Drop-in SSH replacement for single-host operations
  • pdsh compatibility mode: Compatible with pdsh command-line syntax

Key features:

  • clap v4 with derive macros for type-safe argument parsing
  • Backend.AI cluster auto-detection
  • Hostlist expression support (pdsh-compatible)
  • Mode detection based on binary name, environment, or flags

Configuration Management

Documentation: docs/architecture/configuration.md

Hierarchical configuration system with multiple sources:

  1. Backend.AI environment variables (auto-detection)
  2. Current directory (./config.yaml)
  3. XDG config directory (~/.config/bssh/config.yaml)
  4. CLI specified path (via --config flag)

Features:

  • YAML format for human readability
  • Environment variable expansion (${VAR} syntax)
  • SSH configuration file integration
  • Platform-specific paths via XDG Base Directory specification

Parallel Executor

Documentation: docs/architecture/executor.md

Tokio-based async executor for concurrent command execution:

  • Semaphore-based concurrency limiting
  • Two-stage signal handling (default) or batch mode
  • Fail-fast mode for early termination on errors
  • Real-time progress visualization
  • Stream mode for live output

SSH Client

Documentation: docs/architecture/ssh-client.md

Built on russh and russh-sftp with custom tokio_client wrapper:

  • Connection management with russh
  • Multiple authentication methods (agent, key file, password)
  • Host key verification (known_hosts support)
  • Command execution with streaming output
  • SFTP file transfers (upload/download)
  • Connection timeout handling
  • Configurable SSH keepalive (ServerAliveInterval, ServerAliveCountMax)
  • Address family selection (-4/-6, AddressFamily); see Address Family Preference

Terminal User Interface (TUI)

Documentation: docs/architecture/tui.md

Interactive terminal interface for real-time command monitoring:

  • Multiple views (JobList, JobDetail, Logs, System)
  • Keyboard navigation and command palette
  • Progress parsing from command output
  • Real-time log streaming
  • Clean shutdown handling

Interactive Mode

Documentation: docs/architecture/interactive-mode.md

PTY-based interactive SSH sessions:

  • Single-host mode: Direct PTY connection to one host
  • Multiplex mode: Broadcast input to multiple hosts
  • Terminal escape sequence handling
  • Raw mode terminal management
  • Signal propagation (Ctrl+C, window resize)

SSH Configuration Parser

Documentation: docs/architecture/ssh-config-parser.md

OpenSSH-compatible configuration file parser:

  • Include directive support with recursion limits
  • Match directive (Host, LocalUser)
  • All standard SSH options
  • Configuration caching for performance
  • Override chain resolution

SSH Jump Host Support

Documentation: docs/architecture/ssh-jump-hosts.md

ProxyJump (-J) support for bastion hosts:

  • Multiple jump host chains
  • IPv6 and custom port support
  • Authentication through jump hosts
  • Integration with all bssh operations
  • Automatic tunnel management

SSH Port Forwarding

Documentation: docs/architecture/ssh-port-forwarding.md

Full port forwarding support:

  • Local forwarding (-L): Forward local port to remote
  • Remote forwarding (-R): Forward remote port to local
  • Dynamic forwarding (-D): SOCKS proxy mode
  • Multiple forwarding rules
  • Automatic port allocation

Exit Code Strategy

Documentation: docs/architecture/exit-code-strategy.md

MPI-compatible exit code handling:

  • MainRank (default): Returns main rank's exit code
  • RequireAllSuccess: Returns 0 only if all nodes succeed
  • MainRankWithFailureCheck: Hybrid mode for detailed diagnostics
  • Automatic main rank detection (Backend.AI integration)
  • Preserves actual exit codes (SIGSEGV=139, OOM=137, etc.)

ping is the exception: it runs no user command, so no remote status exists to forward and MainRank does not apply. It reuses RequireAllSuccess for the 0/1 boundary and reports 255 when no host answered or when bssh failed before connecting, following OpenSSH's "ssh itself failed" convention. ping_nodes returns a PingOutcome tally rather than Result<()>, dispatch_command returns Result<i32>, and main::dispatch_and_exit is the single place that converts a nonzero command-level code into the process exit status.

Shared Module

Common utilities for code reuse between bssh client and server implementations:

  • Validation: Input validation for usernames, hostnames, paths with security checks
  • Rate Limiting: Generic token bucket rate limiter for connection/auth throttling
  • Authentication Types: Common auth result types and user info structures
  • Error Types: Shared error types for validation, auth, connection, and rate limiting

The security and jump::rate_limiter modules re-export from shared for backward compatibility.

Server Security Module

Security features for the SSH server (src/server/security/):

  • AuthRateLimiter: Fail2ban-like authentication rate limiting

    • Tracks failed authentication attempts per IP address
    • Automatic banning after exceeding configurable threshold
    • Time-windowed failure counting (failures outside window not counted)
    • Configurable ban duration with automatic expiration
    • IP whitelist for exempting trusted addresses from banning
    • Memory-safe with configurable maximum tracked IPs
    • Automatic cleanup of expired records via background task
    • Thread-safe async implementation with Arc<RwLock<>>

  • IpAccessControl: IP-based connection filtering

    • Whitelist mode: Only allow connections from specified CIDR ranges
    • Blacklist mode: Block connections from specified CIDR ranges
    • Blacklist takes priority over whitelist (blocked IPs are always denied)
    • Support for both IPv4 and IPv6 addresses and CIDR notation
    • Dynamic updates: Add/remove rules at runtime via SharedIpAccessControl
    • Early rejection at connection level before handler creation
    • Thread-safe with fail-closed behavior on lock contention
    • Configuration via allowed_ips and blocked_ips in server config

File Transfer Filter Module

Policy-based filtering infrastructure for SFTP and SCP file transfer operations (src/server/filter/):

Structure:

  • mod.rs - TransferFilter trait, Operation enum, FilterResult enum, NoOpFilter
  • policy.rs - FilterPolicy engine, FilterRule, Matcher trait, SharedFilterPolicy
  • path.rs - Path-based matchers: PrefixMatcher, ExactMatcher, ComponentMatcher, ExtensionMatcher
  • pattern.rs - Pattern-based matchers: GlobMatcher, RegexMatcher, CombinedMatcher, NotMatcher

Key Components:

  • Operation: Enum representing file operations

    • Upload, Download, Delete, Rename
    • CreateDir, ListDir, Stat, SetStat
    • Symlink, ReadLink

  • FilterResult: Actions to take on matched operations

    • Allow - Permit the operation (default)
    • Deny - Block the operation
    • Log - Allow but log for auditing

  • TransferFilter Trait: Interface for custom filter implementations

    • check(path, operation, user) - Check single path operations
    • check_with_dest(src, dest, operation, user) - Check two-path operations (rename, symlink)
    • is_enabled() - Check if filtering is active

  • FilterPolicy: First-match-wins rule evaluation engine

    • Ordered rule evaluation
    • Configurable default action
    • Enable/disable filtering
    • Create from YAML configuration via from_config()

  • FilterRule: Combines matcher, action, and optional constraints

    • Path pattern matcher
    • Per-operation restrictions
    • Per-user restrictions
    • Named rules for debugging

Built-in Matchers:

Matcher Purpose Example
GlobMatcher Wildcard patterns *.key, *.pem
RegexMatcher Full regex support (?i)\.exe$
PrefixMatcher Directory tree matching /etc/
ExactMatcher Specific file matching /etc/shadow
ComponentMatcher Path component matching .git, .ssh
ExtensionMatcher File extension matching exe, key
CombinedMatcher OR-combine matchers Multiple patterns
NotMatcher Invert matcher results Exclude patterns

Security Features:

  • normalize_path() function for path traversal prevention
  • ReDoS protection via regex size limits
  • Case-insensitive extension matching

Usage Example:

use bssh::server::filter::{FilterPolicy, FilterResult, Operation};
use bssh::server::filter::pattern::GlobMatcher;
use bssh::server::filter::policy::FilterRule;
use std::path::Path;

// Create policy that blocks *.key files
let policy = FilterPolicy::new()
    .with_default(FilterResult::Allow)
    .add_rule(FilterRule::new(
        Box::new(GlobMatcher::new("*.key").unwrap()),
        FilterResult::Deny,
    ));

// Check if operation is allowed
let result = policy.check(
    Path::new("/etc/secret.key"),
    Operation::Download,
    "alice"
);
assert_eq!(result, FilterResult::Deny);

Configuration (YAML):

filter:
  enabled: true
  default_action: allow
  rules:
    - name: block-sensitive-keys
      pattern: "*.{key,pem}"
      action: deny
      operations:
        - download
        - upload
    - name: block-hidden-dirs
      path_prefix: "/home"
      pattern: ".*"
      action: deny
      users:
        - guest

Audit Logging Module

Comprehensive audit logging infrastructure for the SSH server (src/server/audit/):

Structure:

  • mod.rs - AuditManager for collecting and distributing audit events
  • event.rs - AuditEvent type definitions and builder pattern
  • exporter.rs - AuditExporter trait and NullExporter implementation
  • file.rs - FileExporter for JSON Lines output with rotation support

Key Components:

  • AuditEvent: Represents discrete auditable actions with fields for:

    • Unique event ID (UUID v4)
    • Timestamp (UTC)
    • Event type, session ID, username, client IP
    • File paths, bytes transferred, operation result
    • Protocol and additional details

  • EventType: Categorizes security and operational events:

    • Authentication: AuthSuccess, AuthFailure, AuthRateLimited
    • Sessions: SessionStart, SessionEnd
    • Commands: CommandExecuted, CommandBlocked
    • File operations: FileOpenRead, FileOpenWrite, FileRead, FileWrite, FileClose, FileUploaded, FileDownloaded, FileDeleted, FileRenamed
    • Directory operations: DirectoryCreated, DirectoryDeleted, DirectoryListed
    • Filters: TransferDenied, TransferAllowed
    • Security: IpBlocked, IpUnblocked, SuspiciousActivity

  • EventResult: Operation outcomes (Success, Failure, Denied, Error)

  • AuditExporter Trait: Interface for audit event destinations

    • export() - Export single event
    • export_batch() - Export multiple events (optimizable)
    • flush() - Ensure pending events are written
    • close() - Clean up resources

  • NullExporter: No-op exporter for testing and disabled audit logging

  • FileExporter: File-based exporter writing events in JSON Lines format

    • Append mode to preserve existing data
    • Optional log rotation based on file size (RotateConfig)
    • Optional gzip compression for rotated files
    • Thread-safe using async Mutex
    • Async I/O using tokio
    • Automatic parent directory creation
    • Restrictive file permissions (0o600 on Unix)

  • AuditManager: Central manager with async processing

    • Background worker for non-blocking event processing
    • Configurable buffering (buffer size, batch size)
    • Periodic flush intervals
    • Multiple exporter support
    • Graceful shutdown with event flush

Configuration:

let config = AuditConfig::new()
    .with_enabled(true)
    .with_buffer_size(1000)
    .with_batch_size(100)
    .with_flush_interval(5);

File Exporter Usage:

use bssh::server::audit::file::{FileExporter, RotateConfig};
use std::path::Path;

// Simple file exporter
let exporter = FileExporter::new(Path::new("/var/log/audit.log"))?;

// With rotation (50 MB, 10 backups, gzip compression)
let rotate_config = RotateConfig::new()
    .with_max_size(50 * 1024 * 1024)
    .with_max_backups(10)
    .with_compress(true);

let exporter = FileExporter::new(Path::new("/var/log/audit.log"))?
    .with_rotation(rotate_config);

Output Format (JSON Lines - one JSON object per line):

{"id":"uuid","timestamp":"2024-01-15T10:30:00Z","event_type":"file_uploaded","session_id":"sess-001","user":"admin","client_ip":"192.168.1.100","path":"/data/report.pdf","bytes":1048576,"result":"success","protocol":"sftp"}

  • OtelExporter: OpenTelemetry exporter for distributed tracing and observability

    • OTLP/gRPC protocol support using tonic
    • Event to LogRecord mapping with proper attribute conversion
    • Severity level mapping based on event types and results
    • Resource attributes including service.name and service.version
    • Graceful shutdown and flush methods
    • TLS support for secure audit data transmission

  • LogstashExporter: Logstash exporter for ELK stack integration

    • TCP connection with JSON Lines protocol (newline-delimited JSON)
    • Optional TLS encryption for secure transmission
    • Automatic reconnection on connection failure
    • Batch support for efficient event transmission
    • Connection timeout handling (default: 10 seconds)
    • Configurable host and port

OtelExporter Usage:

use bssh::server::audit::otel::OtelExporter;
use bssh::server::audit::exporter::AuditExporter;
use bssh::server::audit::event::{AuditEvent, EventType};

// Create exporter with OTLP endpoint
let exporter = OtelExporter::new("http://localhost:4317")?;

// Export an audit event
let event = AuditEvent::new(
    EventType::AuthSuccess,
    "alice".to_string(),
    "session-123".to_string(),
);
exporter.export(event).await?;

// Graceful shutdown
exporter.close().await?;

LogstashExporter Usage:

use bssh::server::audit::logstash::LogstashExporter;
use bssh::server::audit::exporter::AuditExporter;
use bssh::server::audit::event::{AuditEvent, EventType};

// Create exporter (unencrypted by default)
let exporter = LogstashExporter::new("logstash.example.com", 5044)?
    .with_tls(true);  // Enable TLS for production

// Export an audit event
let event = AuditEvent::new(
    EventType::AuthSuccess,
    "alice".to_string(),
    "session-123".to_string(),
);
exporter.export(event).await?;

// Graceful shutdown
exporter.close().await?;

Server CLI Binary

Binary: bssh-server

The bssh-server binary provides a command-line interface for managing and operating the SSH server:

Subcommands:

  • run - Start the SSH server (default when no subcommand specified)
  • gen-config - Generate a configuration file template with secure defaults
  • hash-password - Hash passwords for configuration using Argon2id (recommended)
  • check-config - Validate configuration files and display settings
  • gen-host-key - Generate SSH host keys (Ed25519 or RSA)
  • version - Show version and build information

Global Options:

  • -c, --config <FILE> - Configuration file path
  • -b, --bind-address <ADDR> - Override bind address
  • -p, --port <PORT> - Override listen port
  • -k, --host-key <FILE> - Host key file(s) (can be repeated)
  • -v, --verbose - Verbosity level (repeatable: -v, -vv, -vvv)
  • -D, --foreground - Run in foreground (don't daemonize)
  • --pid-file <FILE> - PID file path

Usage Examples:

# Generate configuration template
bssh-server gen-config -o /etc/bssh/server.yaml

# Generate Ed25519 host key (recommended)
bssh-server gen-host-key -t ed25519 -o /etc/bssh/ssh_host_ed25519_key

# Generate RSA host key (for compatibility)
bssh-server gen-host-key -t rsa -o /etc/bssh/ssh_host_rsa_key --bits 4096

# Hash a password for configuration
bssh-server hash-password

# Validate configuration
bssh-server check-config -c /etc/bssh/server.yaml

# Start server with configuration file
bssh-server -c /etc/bssh/server.yaml

# Start server with CLI overrides
bssh-server -c /etc/bssh/server.yaml -p 2222 -b 0.0.0.0 -k /path/to/key

SSH Server Module

Documentation: docs/architecture/server-configuration.md

SSH server implementation using the russh library for accepting incoming connections:

Structure (src/server/):

  • mod.rs - BsshServer struct and russh::server::Server trait implementation
  • config/mod.rs - Module exports and backward compatibility layer
  • config/types.rs - Comprehensive configuration types with serde
  • config/loader.rs - Config loader with validation and environment overrides
  • handler.rs - SshHandler implementing russh::server::Handler trait
  • session.rs - Session state management (SessionManager, SessionInfo, ChannelState)
  • exec.rs - Command execution for SSH exec requests
  • sftp.rs - SFTP subsystem handler with path traversal prevention
  • scp.rs - SCP protocol handler with sink/source modes
  • auth/ - Authentication provider infrastructure
  • audit/ - Audit logging infrastructure (event types, exporters, manager)

Key Components:

  • BsshServer: Main server struct managing the SSH server lifecycle

    • Accepts connections on configured address
    • Loads host keys from OpenSSH format files
    • Configures russh with authentication settings
    • Creates shared rate limiter for authentication attempts

  • Server Configuration System: Dual configuration system for flexibility

    • Builder API (ServerConfig): Programmatic configuration for embedded use
    • File-Based (ServerFileConfig): YAML configuration with environment overrides
    • Configuration precedence: CLI > Environment > File > Defaults
    • Configuration validation at startup (host keys, CIDR ranges, paths)
    • Support for BSSH_* environment variable overrides

  • ServerConfig: Configuration options with builder pattern

    • Host key paths and listen address
    • Connection limits and timeouts
    • Authentication method toggles (password, publickey, keyboard-interactive)
    • Public key authentication configuration (authorized_keys location)
    • Command execution configuration (shell, timeout, allowed/blocked commands)

  • ServerFileConfig: Comprehensive YAML file configuration

    • Server settings (bind address, port, host keys, keepalive)
    • Authentication (public key, password with inline or file-based users)
    • Shell configuration (default shell, environment, command timeout)
    • SFTP/SCP enablement with optional chroot
    • File transfer filtering rules
    • Audit logging (file, OpenTelemetry, Logstash exporters)
    • Security settings (auth attempts, bans, session limits, IP allowlist/blocklist)

  • SshHandler: Per-connection handler for SSH protocol events

    • Public key authentication via AuthProvider trait
    • Rate limiting for authentication attempts (token bucket)
    • Auth rate limiting with ban support (fail2ban-like)
    • Channel operations (open, close, EOF, data)
    • PTY, exec, shell, and subsystem request handling
    • Command execution with stdout/stderr streaming

  • PTY Module (src/server/pty.rs): Pseudo-terminal management for interactive sessions

    • PTY master/slave pair creation using POSIX APIs via nix crate
    • Window size management with TIOCSWINSZ ioctl
    • Async I/O for PTY master file descriptor using tokio's AsyncFd
    • Configuration management (terminal type, dimensions, pixel sizes)
    • Implements AsyncRead and AsyncWrite for PTY I/O

  • Shell Session Module (src/server/shell.rs): Interactive shell session handler

    • Shell process spawning with login shell configuration (-l flag)
    • Terminal environment setup (TERM, HOME, USER, SHELL, PATH)
    • Bidirectional I/O forwarding between SSH channel and PTY master
    • Window resize event handling forwarded to PTY
    • Proper session cleanup on disconnect (SIGHUP to shell, process termination)
    • Controlling terminal setup via TIOCSCTTY ioctl

  • CommandExecutor: Executes commands requested by SSH clients

    • Shell-based command execution with -c flag
    • Environment variable configuration (HOME, USER, SHELL, PATH)
    • Stdout/stderr streaming to SSH channel
    • Command timeout with graceful process termination
    • Command allow/block list validation for security
    • Exit code propagation to client

  • SessionManager: Tracks active sessions with configurable capacity

    • Session creation and cleanup
    • Idle session management
    • Authentication state tracking

  • SftpHandler: SFTP subsystem handler (src/server/sftp.rs)

    • Implements russh_sftp::server::Handler trait for file transfer operations
    • Advertises and serves limits@openssh.com so clients negotiate the server's packet, read, write, and handle ceilings before bulk transfers
    • Path traversal prevention with chroot-like isolation
    • File operations: open, read, write, close
    • Directory operations: opendir, readdir, mkdir, rmdir
    • Attribute operations: stat, lstat, fstat, setstat, fsetstat
    • Path operations: realpath, rename, remove, readlink, symlink
    • Symlink validation ensures targets remain within root directory
    • Handle limit enforcement to prevent resource exhaustion
    • Read size capping to prevent memory exhaustion
    • Pipelined downloads tolerate legal short READ replies by re-requesting the missing byte range before ordered reassembly, and clamp advertised transfer ceilings to the negotiated packet payload budget

  • ScpHandler: SCP protocol handler (src/server/scp.rs)

    • Implements SCP server protocol for file transfers via the scp command
    • Sink mode (-t flag): receives files from client (upload)
    • Source mode (-f flag): sends files to client (download)
    • Recursive transfer support (-r flag) for directories
    • Time preservation (-p flag) for file modification times
    • Security features:
      • Path traversal prevention with normalized path resolution
      • Symlink escape prevention via canonicalization
      • Filename validation (rejects /, .., .)
      • File size limit (10 GB maximum)
      • Mode permission masking (strips setuid/setgid/sticky bits)
      • Line length limits to prevent DoS via buffer exhaustion
    • Automatic SCP command detection in exec_request handler
    • Configurable via scp_enabled setting

Server Authentication Module

The authentication subsystem (src/server/auth/) provides extensible authentication for the SSH server:

Structure:

  • mod.rs - Module exports and re-exports
  • provider.rs - AuthProvider trait definition
  • publickey.rs - PublicKeyVerifier implementation
  • password.rs - PasswordVerifier implementation with Argon2id hashing
  • composite.rs - CompositeAuthProvider combining multiple auth methods

AuthProvider Trait:

The AuthProvider trait defines the interface for all authentication backends:

#[async_trait]
pub trait AuthProvider: Send + Sync {
    async fn verify_publickey(&self, username: &str, key: &PublicKey) -> Result<AuthResult>;
    async fn verify_password(&self, username: &str, password: &str) -> Result<AuthResult>;
    async fn get_user_info(&self, username: &str) -> Result<Option<UserInfo>>;
    async fn user_exists(&self, username: &str) -> Result<bool>;
}

PublicKeyVerifier:

Implements public key authentication by parsing OpenSSH authorized_keys files:

  • Key file location modes:

    • Directory mode: {dir}/{username}/authorized_keys
    • Pattern mode: /home/{user}/.ssh/authorized_keys

  • Supported key types:

    • ssh-ed25519, ssh-ed448
    • ssh-rsa, ssh-dss
    • ecdsa-sha2-nistp256/384/521
    • Security keys (sk-ssh-ed25519, sk-ecdsa-sha2-nistp256)

  • Key options parsing:

    • command="..." - Force specific command
    • from="..." - Restrict source addresses
    • no-pty, no-port-forwarding, no-agent-forwarding, no-X11-forwarding
    • environment="..." - Set environment variables

PasswordVerifier:

Implements password authentication with secure password hashing:

  • Argon2id hashing: Uses the OWASP-recommended password hashing algorithm

    • Memory cost: 19 MiB
    • Time cost: 2 iterations
    • Parallelism: 1

  • User configuration:

    • External YAML file with user definitions
    • Inline users in server configuration
    • User attributes: name, password_hash, shell, home, env

  • Security features:

    • Timing attack mitigation with constant-time verification
    • Minimum verification time (100ms) regardless of user existence
    • Dummy hash verification for non-existent users
    • Secure memory cleanup using zeroize crate
    • User enumeration protection

  • Hash compatibility:

    • Argon2id (recommended, generated by hash-password command)
    • bcrypt (supported for backward compatibility)

CompositeAuthProvider:

Combines multiple authentication methods into a single provider:

  • Delegates to PublicKeyVerifier for public key auth
  • Delegates to PasswordVerifier for password auth
  • Prioritizes password verifier for user info (more detailed)
  • Supports hot-reloading of password users via reload_password_users()

Security Features:

  • Username validation: Prevents path traversal attacks (e.g., ../etc/passwd)
  • File permission checks (Unix): Rejects world/group-writable files and symlinks
  • Symlink protection: Uses symlink_metadata() to detect and reject symlinks
  • Parent directory validation: Checks parent directory permissions
  • Rate limiting: Token bucket rate limiter for authentication attempts
  • Timing attack mitigation: Constant-time behavior in password verification and user_exists() check
  • Secure memory handling: Password strings cleared from memory after use via zeroize
  • Comprehensive logging: All authentication attempts are logged

Data Flow

Command Execution Flow

User Input → CLI Parser → Mode Detection → Node Resolution
                                                 ↓
                                         Configuration Loading
                                                 ↓
                                         SSH Config Parsing
                                                 ↓
                                    Jump Host Chain Creation
                                                 ↓
                                      Parallel Executor Setup
                                                 ↓
                            ┌────────────────────┴─────────────────┐
                            ▼                                      ▼
                    Connection Pool                       Task Spawning
                            ↓                                      ↓
                    Per-Node Execution                   Semaphore Control
                            ↓                                      ↓
                    Command/Transfer                      Result Collection
                            ↓                                      ↓
                    Output Streaming                     Exit Code Strategy
                            └────────────────────┬─────────────────┘
                                                 ▼
                                          User Output

Address Family Preference

The OpenSSH-compatible -4 / -6 flags and the ssh_config AddressFamily keyword share one representation, AddressFamily in src/ssh/tokio_client/address_family.rs:

pub enum AddressFamily { Any, V4, V6 }

Resolution. AddressFamily::resolve(ipv4_flag, ipv6_flag, config_value) implements the OpenSSH precedence rule: command line flag, then config keyword, then the any default. AddressFamily::from_config_value accepts any | inet | inet6 case-insensitively and warns (via tracing) rather than failing on an unrecognized value, so a configuration file OpenSSH would tolerate does not become a hard error. The dispatcher builds an SshConnectionConfigResolver from CLI overrides, YAML defaults, and the parsed ssh_config; exec, ping, upload, and download resolve that object for each target node so per-host Host blocks apply to the actual connection target.

Threading. The preference rides on SshConnectionConfig, the struct every connection path already carries. Client::connect_with_ssh_config passes it to connect_with_config_inner, which resolves the target, filters the candidate list, and connects. Any returns the resolver's list untouched, which is what keeps the unflagged path byte-for-byte identical to the previous behavior. JumpHostChain receives the same resolver and resolves each jump hop against that hop's own host name instead of inheriting the destination's settings. The SFTP paths (upload / download) now receive the full resolved SshConnectionConfig, so AddressFamily, Compression, ServerAliveInterval, and ServerAliveCountMax follow the same per-host path as exec and ping. ForwardingConfig carries its own copy of the resolved address family for the forwarding-target filter, since forwarders run detached from the connect config.

Scope. The constraint is a hard filter where bssh opens the socket. For forwarding targets the unforced path preserves server-side resolution, while a forced family switches to locally resolved numeric addresses so the family request has an observable effect:

Path Behavior
Direct connect (exec, interactive, ping, SFTP) Hard filter in connect_with_config_inner
First jump hop Hard filter (shares the direct connect path)
-L / -D listener Selects the implicit bind address (::1 / :: under -6); an explicit bind address wins
-L / SOCKS5 -D target With Any, sends the requested hostname in direct-tcpip and lets the remote sshd resolve it. With V4 / V6, resolves locally, filters the candidate list, and sends the matching numeric address
SOCKS4 -D target Unfiltered; SOCKS4 carries a literal IPv4 destination by protocol definition
Jump hops past the first, and the destination behind a chain Uses the same direct-tcpip model as -L/SOCKS5 -D targets: hostname with Any, locally filtered numeric address with V4 / V6. The family also selects the best-effort address recorded for host key verification diagnostics
-R listener Not constrained; the server binds it
bssh-server Out of scope; separate CLI

Failure mode. A forced family with no matching resolved address is a hard failure with no fallback to the other family, matching OpenSSH. It surfaces as the dedicated Error::NoAddressForFamily { host, family } variant, rendered as no IPv6 address found for <host>, which replaces the generic could not resolve to any addresses so a family mismatch is distinguishable from a genuine resolution failure.

Error Handling Strategy

  • Connection errors: Retry with exponential backoff
  • Authentication failures: Immediate failure with clear diagnostics
  • Command execution errors: Captured with exit codes
  • Timeout handling: Configurable per-connection and per-command
  • Signal handling: Clean shutdown on Ctrl+C with two-stage confirmation

Test Environment-Variable Mutation Pattern (EnvGuard)

Several test suites must temporarily set or remove process-wide environment variables (e.g. BACKENDAI_CLUSTER_HOSTS, HOME, SSH_AUTH_SOCK). Under Rust 2024 edition, std::env::set_var and std::env::remove_var are marked unsafe because concurrent mutation of the environment is undefined behaviour at the libc level on glibc, musl, and macOS. EnvGuard centralises all such mutations in src/test_helpers/env_guard.rs.

Soundness contract: every test that constructs an EnvGuard MUST be annotated with #[serial_test::serial]. Every other test in the same crate binary that reads or mutates the same variable MUST also carry #[serial] (or a matching #[serial(key)] group). Note that #[serial] only serializes against other #[serial] / #[parallel] tests — unannotated tests may still run concurrently with serial ones and would race on environment reads. This is not an EnvGuard limitation; it is an inherent constraint of the libc environment-variable API.

use serial_test::serial;
use crate::test_helpers::EnvGuard;

#[test]
#[serial]
fn my_test() {
    let _host = EnvGuard::set("BACKENDAI_CLUSTER_HOSTS", "node1,node2");
    // Variable is automatically restored when `_host` drops at end of scope.
}

Integration tests access the same struct via a #[path]-based re-export in tests/common/mod.rs, which avoids making EnvGuard part of the public bssh crate API while keeping a single source of truth. When adding a new integration-test binary that needs EnvGuard, add mod common; at the top of that file and use common::EnvGuard.

Use #[serial(key)] (a named group) when two sets of tests touch different, non-overlapping variables and can therefore run concurrently with each other but not with themselves; omit the key (plain #[serial]) when in doubt.

Security Model

Authentication

  • SSH agent authentication (auto-detection)
  • Private key files with passphrase support
  • Password authentication (discouraged in production)
  • Public key authentication preferred

Host Verification

  • known_hosts file verification
  • Three modes: Yes (strict), No (insecure), AcceptNew (recommended)
  • Per-host configuration support
  • Host key fingerprint display

OpenSSH host certificates

russh 0.63 widened client::Handler::check_server_key from &PublicKey to &PublicKeyOrCertificate, so the callback can now also receive an OpenSSH host certificate. bssh does not participate in that scheme:

  • It never advertises certificate host key algorithms. Both Preferred overrides (src/ssh/tokio_client/connection.rs for the client, src/server/mod.rs for the server) change only compression and inherit host_key_certificates from Preferred::DEFAULT, which is empty. A server therefore cannot negotiate a certificate with bssh.
  • If a peer sends one regardless, ClientHandler::check_server_key refuses it and returns ServerCheckFailed. bssh verifies no CA signatures (the @cert-authority scan in src/ssh/tokio_client/host_verification.rs only warns and falls back to TOFU), so the key inside a certificate has never been vouched for by anything bssh trusts. Matching it against known_hosts would answer a different question than the one the certificate poses.
  • ServerCheckMethod::NoCheck still accepts, because there the operator has turned host verification off outright.

server::Config likewise gained a certificates field; bssh's construction ends in ..Default::default(), so it stays empty and the server keeps presenting a plain host key.

Data Protection

  • No credential logging
  • Secure memory handling for passphrases
  • Encrypted SSH transport (via russh)
  • Connection timeout enforcement

Network Security

  • Jump host support for bastion architectures
  • Port forwarding for secure tunneling
  • SSH config directive support for security policies

Release Signing and Notarization (macOS)

Official macOS release binaries (bssh, bssh-server, bssh-keygen) are signed with a "Developer ID Application" certificate and notarized through Apple's notarytool. This is handled by two composite actions in .github/actions/, mirrored from continuum-router (originally from backend.ai-go):

  • macos-signing-setup extracts the Developer ID p12 to a PEM (via openssl pkcs12 -legacy, selected by capability since Apple's LibreSSL lacks the option), rejects any p12 that holds no Developer ID Application certificate, and installs rcodesign (keychain-free signing).
  • macos-sign-notarize-binary signs a staged copy with the hardened runtime and a pinned reverse-DNS identifier (BUNDLE_ID base, with -server / -keygen suffixes), asserts the resulting authority, runtime flag, and identifier, packages the flat zip with ditto, submits it to notarytool --wait, and gates on status: Accepted. Bare Mach-O binaries cannot be stapled, so Gatekeeper resolves the ticket online.

Rationale: releases up to v2.4.1 were signed with an "Apple Distribution" certificate (an App Store submission identity without the Developer ID leaf extension) and never notarized. When that certificate was revoked, macOS killed installed binaries on launch and deleted them as malware. The authority assertion exists so a wrong certificate fails the release instead of shipping quietly.

Required release credentials (GitHub packaging environment): APPLE_CERTIFICATE (base64 Developer ID Application p12), APPLE_CERTIFICATE_PASSWORD, APPLE_ID, APPLE_TEAM_ID, APPLE_PASSWORD (app-specific password), and the BUNDLE_ID variable (com.lablup.bssh).

Dependencies and Licensing

Core Dependencies

  • tokio - Async runtime
  • russh / russh-sftp - SSH protocol implementation
  • clap - CLI argument parsing
  • serde / serde_yaml - Configuration serialization
  • tracing / tracing-subscriber - Structured logging
  • anyhow / thiserror - Error handling

License

See LICENSE file for licensing information.

Appendix

Performance Tuning

  • Parallelism: Adjust --parallel flag (default: 10)
  • Connection timeout: Use --connect-timeout (default: 30s)
  • Command timeout: Use --timeout (default: 5min)
  • Keepalive: Configurable via --server-alive-interval (default: 60s) and --server-alive-count-max (default: 3)
    • Interval of 0 disables keepalive
    • Connection is considered dead after interval * (count_max + 1) seconds without response
    • Equivalent to OpenSSH ServerAliveInterval and ServerAliveCountMax options

Configuration Schema

See docs/architecture/configuration.md for complete YAML schema and examples.

Exit Codes

  • 0: Success (all nodes, or main rank succeeded)
  • 1: General failure
  • 130: Terminated by SIGINT (Ctrl+C)
  • 255: Generated by ping as a bssh-level signal when no host was reachable, or bssh failed before attempting any connection. The default MainRank strategy can also produce 255 by forwarding a remote command's exit status verbatim
  • Other: Preserved from main rank (SIGSEGV=139, OOM=137, etc.)

See docs/architecture/exit-code-strategy.md for detailed strategy documentation.

Further Reading

For detailed component documentation, see: