Cloud Training Operations Guide¶

This document covers production deployment and operations for SimpleTuner's cloud training feature, with a focus on complete integration with existing DevOps infrastructure.

Network Architecture¶

Outbound Connections¶

The server makes outbound HTTPS connections to configured cloud providers. Each provider has its own API endpoints and requirements.

Provider-specific network details: - Replicate API Endpoints

Inbound Connections¶

Firewall Rules¶

Firewall requirements depend on your configured provider(s).

Provider-specific firewall rules: - Replicate Firewall Configuration

Webhook IP Allowlisting¶

For enhanced security, you can restrict webhook delivery to specific IP ranges. When configured, webhooks from IPs outside the allowlist are rejected with a 403 Forbidden response.

Configuration via API:

# Set allowed IPs for a provider's webhooks
curl -X PUT http://localhost:8080/api/cloud/providers/{provider} \
  -H "Content-Type: application/json" \
  -d '{
    "webhook_allowed_ips": ["10.0.0.0/8", "192.168.0.0/16"]
  }'

Configuration via Web UI:

1. Navigate to Cloud tab → Advanced Configuration
2. In the "Webhook Security" section, add IP ranges
3. Use CIDR notation (e.g., 10.0.0.0/8) or single IPs (1.2.3.4/32)

IP Format:

Provider-specific webhook IPs: - Replicate Webhook IPs

Behavior:

Audit Logging:

Rejected webhooks are logged to the audit trail:

curl "http://localhost:8080/api/audit?event_type=webhook_rejected&limit=100"

Proxy Configuration¶

Environment Variables¶

# HTTP/HTTPS proxy
export HTTPS_PROXY="http://proxy.corp.example.com:8080"
export HTTP_PROXY="http://proxy.corp.example.com:8080"

# Custom CA bundle for corporate CAs
export SIMPLETUNER_CA_BUNDLE="/etc/pki/tls/certs/ca-bundle.crt"

# Disable SSL verification (NOT recommended for production)
export SIMPLETUNER_SSL_VERIFY="false"

# HTTP timeout (seconds)
export SIMPLETUNER_HTTP_TIMEOUT="60"

Via Provider Config¶

# Via API
PUT /api/cloud/providers/{provider}
{
    "ssl_verify": true,
    "ssl_ca_bundle": "/etc/pki/tls/certs/corporate-ca.crt",
    "proxy_url": "http://proxy:8080",
    "http_timeout": 60.0
}

Via Web UI (Advanced Configuration)¶

The Cloud tab includes an Advanced Configuration panel for network settings:

SSL Verification Bypass¶

Disabling SSL verification requires explicit acknowledgment due to security implications:

1. Click the SSL Verification toggle to disable
2. A confirmation dialog appears: "Disabling SSL verification is a security risk. Only do this if you have a self-signed certificate or are behind a corporate proxy. Continue?"
3. Click "OK" to confirm and save the setting

The acknowledgment is session-scoped. Subsequent toggles within the same session won't require re-confirmation.

Corporate Proxy Configuration¶

For environments using HTTP proxies:

1. Navigate to the Cloud tab → Advanced Configuration
2. Enter the proxy URL (e.g., http://proxy.corp.example.com:8080)
3. Optionally set a custom CA bundle if your proxy performs TLS inspection
4. Adjust the HTTP timeout if your proxy adds latency

Settings are saved immediately when changed and apply to all subsequent provider API calls.

Health Monitoring¶

Endpoints¶

Health Check Response¶

{
  "status": "healthy",
  "version": "1.0.0",
  "uptime_seconds": 3600.5,
  "timestamp": "2024-01-15T10:30:00Z",
  "components": [
    {
      "name": "database",
      "status": "healthy",
      "latency_ms": 1.2,
      "message": "SQLite database accessible"
    },
    {
      "name": "secrets",
      "status": "healthy",
      "message": "API token configured"
    }
  ]
}

Include provider API checks (adds latency):

GET /api/cloud/health?include_providers=true

Prometheus Metrics¶

Scrape endpoint: /api/cloud/metrics/prometheus

Enabling Prometheus Export¶

Prometheus export is disabled by default. Enable it via the Metrics tab in the Admin panel or via API:

curl -X PUT http://localhost:8080/api/cloud/metrics/config \
  -H "Content-Type: application/json" \
  -d '{"prometheus_enabled": true, "prometheus_categories": ["jobs", "http", "health"]}'

Metric Categories¶

Metrics are organized into categories that can be individually enabled:

Configuration Templates¶

Quick-start templates for common use cases:

curl -X POST http://localhost:8080/api/cloud/metrics/config/templates/standard

Available Metrics¶

# Server uptime
simpletuner_uptime_seconds 3600.5

# Job metrics
simpletuner_jobs_total 150
simpletuner_jobs_by_status{status="completed"} 120
simpletuner_jobs_by_status{status="failed"} 10
simpletuner_jobs_by_status{status="running"} 5
simpletuner_jobs_active 8
simpletuner_cost_usd_total 450.25
simpletuner_job_duration_seconds_avg 1800.5

# HTTP metrics
simpletuner_http_requests_total{endpoint="POST_/api/cloud/jobs/submit"} 50
simpletuner_http_errors_total{endpoint_status="POST_/api/cloud/jobs/submit_500"} 2
simpletuner_http_request_latency_ms_avg{endpoint="POST_/api/cloud/jobs/submit"} 250.5

# Rate limiting
simpletuner_rate_limit_violations_total 15
simpletuner_rate_limit_tracked_clients 42

# Approvals
simpletuner_approval_requests_pending 3
simpletuner_approval_requests_by_status{status="approved"} 25

# Audit
simpletuner_audit_log_entries_total 1500
simpletuner_audit_log_entries_24h 120

# Circuit breakers (per provider)
simpletuner_circuit_breaker_state{provider="..."} 0
simpletuner_circuit_breaker_failures_total{provider="..."} 5

# Provider status (per provider)
simpletuner_cost_limit_percent_used{provider="..."} 45.5
simpletuner_credit_balance_usd{provider="..."} 150.00

Prometheus Configuration¶

scrape_configs:
  - job_name: 'simpletuner'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: '/api/cloud/metrics/prometheus'
    scrape_interval: 30s

Preview Metrics Output¶

Preview what will be exported without affecting configuration:

curl "http://localhost:8080/api/cloud/metrics/config/preview?categories=jobs&categories=health"

Rate Limiting¶

Overview¶

SimpleTuner includes built-in rate limiting to protect against abuse and ensure fair resource usage. Rate limits are applied per-IP with configurable rules for different endpoints.

Configuration¶

Rate limiting can be configured via environment variables:

# Default rate limit for unmatched endpoints
export RATE_LIMIT_CALLS=100      # Requests per period
export RATE_LIMIT_PERIOD=60      # Period in seconds

# Set to 0 to disable rate limiting entirely
export RATE_LIMIT_CALLS=0

Default Rate Limit Rules¶

Different endpoints have different rate limits based on sensitivity:

Excluded Paths¶

The following paths are excluded from rate limiting:

• /health - Health checks
• /api/events/stream - SSE connections
• /static/ - Static files
• /api/cloud/hints - UI hints (not security-sensitive)
• /api/users/me - Current user check
• /api/cloud/providers - Provider list

Response Headers¶

All responses include rate limit headers:

X-RateLimit-Limit: 100        # Maximum requests allowed
X-RateLimit-Remaining: 95     # Requests remaining in period
X-RateLimit-Reset: 1705320000 # Unix timestamp when limit resets

HTTP/1.1 429 Too Many Requests
Retry-After: 45
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1705320045

{"detail": "Rate limit exceeded. Please try again later."}

Client IP Detection¶

The middleware properly handles proxy headers:

1. X-Forwarded-For - Standard proxy header (first IP is the client)
2. X-Real-IP - Nginx proxy header
3. Direct connection IP - Fallback

Rate limits are bypassed for localhost (127.0.0.1, ::1) in development.

Audit Logging¶

Rate limit violations are logged to the audit trail with: - Client IP address - Requested endpoint - HTTP method - User-Agent header

Query audit logs for rate limit events:

curl "http://localhost:8080/api/audit?event_type=rate_limited&limit=100"

Custom Rate Limit Rules¶

from simpletuner.simpletuner_sdk.server.middleware.security_middleware import (
    RateLimitMiddleware,
)

# Custom rules: (pattern, calls, period, methods)
custom_rules = [
    (r"^/api/cloud/expensive$", 5, 300, ["POST"]),  # 5 per 5 minutes
    (r"^/api/cloud/public$", 1000, 60, None),       # 1000 per minute for all methods
]

app.add_middleware(
    RateLimitMiddleware,
    calls=100,           # Default limit
    period=60,           # Default period
    rules=custom_rules,  # Custom rules
    enable_audit=True,   # Log violations
)

Distributed Rate Limiting (Async Rate Limiter)¶

For multi-worker deployments, SimpleTuner provides a distributed rate limiter that uses the configured state backend (SQLite, Redis, PostgreSQL, or MySQL) to share rate limit state across workers.

from simpletuner.simpletuner_sdk.server.services.cloud.container import get_rate_limiter

# Create a rate limiter with sliding window
limiter = await get_rate_limiter(
    max_requests=100,    # Maximum requests in window
    window_seconds=60,   # Window duration
    key_prefix="api",    # Optional prefix for keys
)

# Check if a request should be allowed
allowed = await limiter.check("user:123")
if not allowed:
    raise RateLimitExceeded()

# Or use context manager for automatic tracking
async with limiter.track("user:123") as allowed:
    if not allowed:
        return Response(status_code=429)
    # Process request...

Sliding Window Algorithm:

The rate limiter uses a sliding window algorithm that provides smoother rate limiting than fixed windows:

Time:     |----60s window----|
Requests: [x x x x x][x x x]
          ↑ expired  ↑ counted

• Requests are timestamped when they arrive
• Only requests within the window are counted
• Older requests expire and are pruned automatically
• No "burst at window boundary" problem

Backend-Specific Behavior:

from simpletuner.simpletuner_sdk.server.routes.cloud._shared import (
    webhook_rate_limiter,  # 100 req/min for webhooks
    s3_rate_limiter,       # 50 req/min for S3 uploads
)

# Use in route handlers
@router.post("/webhooks/{provider}")
async def handle_webhook(request: Request):
    client_ip = request.client.host
    if not await webhook_rate_limiter.check(client_ip):
        raise HTTPException(status_code=429, detail="Rate limit exceeded")
    # Process webhook...

# Get current usage for a key
usage = await limiter.get_usage("user:123")
print(f"Requests in window: {usage['count']}/{usage['limit']}")
print(f"Window resets in: {usage['reset_in_seconds']}s")

Storage Endpoint (S3-Compatible)¶

Overview¶

SimpleTuner provides an S3-compatible endpoint for uploading training outputs (checkpoints, samples, logs) from cloud training jobs back to your local machine. This enables cloud training jobs to "call home" with results.

Architecture¶

┌─────────────────────┐          ┌─────────────────────┐
│   Cloud Training    │          │   Local SimpleTuner │
│   Job               │ ──────── │   Server            │
│                     │   HTTPS  │                     │
│   Uploads outputs   │          │ /api/cloud/storage/*│
│   via S3 protocol   │          │                     │
└─────────────────────┘          └─────────────────────┘
                                         │
                                         ▼
                                ┌─────────────────────┐
                                │   Local Filesystem  │
                                │   ~/.simpletuner/   │
                                │   outputs/{job_id}/ │
                                └─────────────────────┘

Requirements¶

For cloud jobs to upload to your local server, you need:

1. Public HTTPS endpoint - Cloud providers can't reach localhost
2. SSL certificate - Most providers require HTTPS
3. Firewall access - Allow inbound connections on your chosen port

Option 1: Cloudflared Tunnel (Recommended)¶

Cloudflare Tunnel provides a secure tunnel without opening firewall ports.

# Install cloudflared
# macOS
brew install cloudflared

# Linux
curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared
chmod +x cloudflared
sudo mv cloudflared /usr/local/bin/

# Create a tunnel (requires Cloudflare account)
cloudflared tunnel login
cloudflared tunnel create simpletuner

# Get your tunnel ID
cloudflared tunnel list

tunnel: YOUR_TUNNEL_ID
credentials-file: ~/.cloudflared/YOUR_TUNNEL_ID.json

ingress:
  - hostname: simpletuner.yourdomain.com
    service: http://localhost:8001
  - service: http_status:404

# Start the tunnel
cloudflared tunnel run simpletuner

# Or run as a service
sudo cloudflared service install

# Set the public URL for S3 uploads
export SIMPLETUNER_PUBLIC_URL="https://simpletuner.yourdomain.com"

Option 2: ngrok¶

ngrok provides quick tunnels for development.

# Install ngrok
# macOS
brew install ngrok

# Linux
curl -s https://ngrok-agent.s3.amazonaws.com/ngrok.asc | sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null
echo "deb https://ngrok-agent.s3.amazonaws.com buster main" | sudo tee /etc/apt/sources.list.d/ngrok.list
sudo apt update && sudo apt install ngrok

# Authenticate (requires ngrok account)
ngrok config add-authtoken YOUR_TOKEN

# Start ngrok tunnel to SimpleTuner port
ngrok http 8001

# Note the HTTPS URL from the output:
# Forwarding: https://abc123.ngrok.io -> http://localhost:8001

export SIMPLETUNER_PUBLIC_URL="https://abc123.ngrok.io"

Option 3: Direct Public IP¶

# Allow inbound HTTPS
sudo ufw allow 8001/tcp

# Or with iptables
sudo iptables -A INPUT -p tcp --dport 8001 -j ACCEPT

# Install certbot
sudo apt install certbot

# Get certificate (requires DNS pointing to your IP)
sudo certbot certonly --standalone -d simpletuner.yourdomain.com

# Configure SimpleTuner
export SIMPLETUNER_SSL_CERT="/etc/letsencrypt/live/simpletuner.yourdomain.com/fullchain.pem"
export SIMPLETUNER_SSL_KEY="/etc/letsencrypt/live/simpletuner.yourdomain.com/privkey.pem"
export SIMPLETUNER_PUBLIC_URL="https://simpletuner.yourdomain.com:8001"

Storage Endpoint Configuration¶

Configure the S3 endpoint behavior via provider settings:

curl -X PUT http://localhost:8001/api/cloud/providers/{provider} \
  -H "Content-Type: application/json" \
  -d '{
    "s3_endpoint_enabled": true,
    "s3_public_url": "https://simpletuner.yourdomain.com",
    "s3_output_path": "~/.simpletuner/outputs"
  }'

Or via the Cloud tab → Advanced Configuration.

Upload Authentication¶

S3 uploads are authenticated using short-lived upload tokens:

1. When a job is submitted, a unique upload token is generated
2. The token is passed to the cloud job as an environment variable
3. The job uses the token as the S3 access key when uploading
4. Tokens expire after the job completes or is cancelled

Supported S3 Operations¶

Troubleshooting Storage Uploads¶

Uploads failing with "Unauthorized": - Verify the upload token is being passed correctly - Check that the job ID matches the token - Ensure the job is still in an active state (not completed/cancelled)

Uploads timing out: - Check your tunnel is running (cloudflared tunnel run or ngrok http) - Verify the public URL is accessible from the internet - Test with: curl -I https://your-public-url/api/cloud/health

SSL certificate errors: - For ngrok/cloudflared, SSL is handled automatically - For direct connections, ensure your certificate is valid - Check intermediate certificates are included in the chain

# Test local connectivity
curl http://localhost:8001/api/cloud/health

# Test from external (if direct IP)
curl https://your-public-ip:8001/api/cloud/health

View upload progress:

# Check current uploads
curl http://localhost:8001/api/cloud/jobs/{job_id}

# Response includes upload_progress

Structured Logging¶

Configuration¶

# Log level: DEBUG, INFO, WARNING, ERROR
export SIMPLETUNER_LOG_LEVEL="INFO"

# Format: "json" or "text"
export SIMPLETUNER_LOG_FORMAT="json"

# Optional file output
export SIMPLETUNER_LOG_FILE="/var/log/simpletuner/cloud.log"

JSON Log Format¶

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "INFO",
  "logger": "simpletuner.cloud.jobs",
  "message": "Job submitted",
  "correlation_id": "abc123def456",
  "source": {
    "file": "jobs.py",
    "line": 350,
    "function": "submit_job"
  },
  "extra": {
    "job_id": "xyz789",
    "provider": "..."
  }
}

Programmatic Configuration¶

from simpletuner.simpletuner_sdk.server.services.cloud.structured_logging import (
    configure_structured_logging,
    get_logger,
    LogContext,
)

# Configure logging
configure_structured_logging(
    level="INFO",
    json_output=True,
    log_file="/var/log/simpletuner/cloud.log",
)

# Get a logger
logger = get_logger("mymodule")

# Log with context
with LogContext(job_id="abc123", provider="..."):
    logger.info("Processing job")  # Includes job_id and provider

Backup and Restore¶

Database Location¶

The SQLite database is stored at:

~/.simpletuner/config/cloud/jobs.db

With WAL files:

~/.simpletuner/config/cloud/jobs.db-wal
~/.simpletuner/config/cloud/jobs.db-shm

Command-Line Backup¶

# Simple copy (stop server first for consistency)
cp ~/.simpletuner/config/cloud/jobs.db /backup/jobs_$(date +%Y%m%d).db

# Online backup with sqlite3
sqlite3 ~/.simpletuner/config/cloud/jobs.db ".backup /backup/jobs.db"

Programmatic Backup¶

from simpletuner.simpletuner_sdk.server.services.cloud import JobStore

store = JobStore()

# Create timestamped backup
backup_path = store.backup()
print(f"Backup created: {backup_path}")

# Custom backup path
backup_path = store.backup("/backup/custom_backup.db")

# List available backups
backups = store.list_backups()
for b in backups:
    print(f"  {b.name}: {b.stat().st_size / 1024:.1f} KB")

# Get database info
info = store.get_database_info()
print(f"Database: {info['size_mb']} MB, {info['job_count']} jobs")

Restore¶

from pathlib import Path
from simpletuner.simpletuner_sdk.server.services.cloud import JobStore

store = JobStore()

# WARNING: This overwrites the current database!
success = store.restore(Path("/backup/jobs_backup_20240115_103000.db"))

Automated Backup Script¶

#!/bin/bash
# /etc/cron.daily/simpletuner-backup

BACKUP_DIR="/backup/simpletuner"
RETENTION_DAYS=30
DB_PATH="$HOME/.simpletuner/config/cloud/jobs.db"

mkdir -p "$BACKUP_DIR"

# Create backup
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/jobs_backup_$TIMESTAMP.db"

sqlite3 "$DB_PATH" ".backup '$BACKUP_FILE'"

# Compress
gzip "$BACKUP_FILE"

# Remove old backups
find "$BACKUP_DIR" -name "jobs_backup_*.db.gz" -mtime +$RETENTION_DAYS -delete

echo "Backup created: ${BACKUP_FILE}.gz"

Secrets Management¶

See SECRETS_AND_CACHE.md for detailed documentation on secrets providers.

Supported Providers¶

1. Environment Variables (default)
2. File-based Secrets (~/.simpletuner/secrets.json or YAML)
3. AWS Secrets Manager (requires pip install boto3)
4. HashiCorp Vault (requires pip install hvac)

Provider Priority¶

Secrets are resolved in order: 1. Environment variables (highest priority - allows overrides) 2. File-based secrets 3. AWS Secrets Manager 4. HashiCorp Vault

Troubleshooting¶

Connection Issues¶

Proxy not working:

# Test proxy connectivity
curl -x http://proxy:8080 https://your-provider-api-endpoint

# Check environment
env | grep -i proxy

SSL certificate errors:

# Test with custom CA
curl --cacert /path/to/ca.crt https://your-provider-api-endpoint

# Verify CA bundle
openssl verify -CAfile /path/to/ca.crt server.crt

Provider-specific troubleshooting: - Replicate Troubleshooting

Database Issues¶

Locked database:

# Check for open connections
fuser ~/.simpletuner/config/cloud/jobs.db

# Force WAL checkpoint
sqlite3 ~/.simpletuner/config/cloud/jobs.db "PRAGMA wal_checkpoint(TRUNCATE)"

Corrupted database:

# Check integrity
sqlite3 ~/.simpletuner/config/cloud/jobs.db "PRAGMA integrity_check"

# Recover (creates new database from good pages)
sqlite3 ~/.simpletuner/config/cloud/jobs.db ".recover" | sqlite3 jobs_recovered.db

Health Check Failures¶

# Test health endpoint
curl -s http://localhost:8080/api/cloud/health | jq .

# Check with provider checks included
curl -s 'http://localhost:8080/api/cloud/health?include_providers=true' | jq .