Fix linting issues and refactor for improved code quality

Allow toggling HTTP interaction recording via CLI, environment, and Web UI
Improve visibility of multiple devices in recordings by adding original value comments to .http files
2026-09-07 15:07:17 +00:00 · 2026-02-14 12:39:39 +01:00 · 2026-02-14 12:39:39 +01:00 · 2026-02-14 12:39:39 +01:00 · 2026-02-14 12:39:39 +01:00 · 2026-02-14 00:14:29 +01:00
26 changed files with 1989 additions and 580 deletions
@@ -1,6 +1,6 @@
-# Bose SoundTouch API Client
+# Bose SoundTouch Toolkit

-A comprehensive Go library and CLI tool for controlling Bose SoundTouch devices via their Web API.
+A comprehensive solution for controlling and preserving Bose SoundTouch devices, including a Go library, CLI tool, and a local service for cloud emulation.

 [![Go Reference](https://pkg.go.dev/badge/github.com/gesellix/bose-soundtouch.svg)](https://pkg.go.dev/github.com/gesellix/bose-soundtouch)
 [![Go Report Card](https://goreportcard.com/badge/github.com/gesellix/bose-soundtouch)](https://goreportcard.com/report/github.com/gesellix/bose-soundtouch)
@@ -21,7 +21,8 @@ A comprehensive Go library and CLI tool for controlling Bose SoundTouch devices
 - 🖥️ **CLI Tool**: Comprehensive command-line interface
 - 🌐 **SoundTouch Service**: Emulate Bose cloud services for offline device operation
 - 🔧 **Service Migration**: Migrate devices to use local services instead of Bose cloud
- 📊 **Traffic Analysis**: Proxy and log device communications for debugging
+- 📊 **Traffic Analysis**: Proxy and log device communications
+- 📝 **HTTP Recording**: Persist interactions as re-playable `.http` files
 - 🔒 **Production Ready**: Extensive testing with real SoundTouch hardware
 - 🌐 **Cross-Platform**: Windows, macOS, Linux support

@@ -42,165 +43,47 @@ go get github.com/gesellix/bose-soundtouch

 ### CLI Usage

-#### Discover Devices
+Find SoundTouch devices on your network:
 ```bash
-# Find SoundTouch devices on your network
 soundtouch-cli discover devices
 ```

-# Control a Device
+Control a device (replace `192.168.1.100` with your speaker's IP):
 ```bash
-# Basic device information
-soundtouch-cli --host 192.168.1.100 info get
+# Basic information
+soundtouch-cli --host 192.168.1.100 info

 # Media controls
 soundtouch-cli --host 192.168.1.100 play start
 soundtouch-cli --host 192.168.1.100 volume set --level 50
-soundtouch-cli --host 192.168.1.100 source select --source SPOTIFY

 # Preset management
 soundtouch-cli --host 192.168.1.100 preset list
-soundtouch-cli --host 192.168.1.100 preset store-current --slot 1
-soundtouch-cli --host 192.168.1.100 preset select --slot 1
-
-# Browse and discover content
-soundtouch-cli --host 192.168.1.100 browse tunein
-soundtouch-cli --host 192.168.1.100 station search-tunein --query "jazz"
-soundtouch-cli --host 192.168.1.100 station add --source TUNEIN --token <token> --name "Jazz Radio"
-
-# Speaker notifications (ST-10 only)
-soundtouch-cli --host 192.168.1.100 speaker tts --text "Welcome home" --app-key YOUR_KEY
-soundtouch-cli --host 192.168.1.100 speaker url --url "https://example.com/doorbell.mp3" --app-key YOUR_KEY
-soundtouch-cli --host 192.168.1.100 speaker beep
-
-# Real-time monitoring
-soundtouch-cli --host 192.168.1.100 events subscribe
 ```

-### SoundTouch Service
+For full CLI documentation, see [docs/CLI-REFERENCE.md](docs/CLI-REFERENCE.md).

-The `soundtouch-service` is a local server that emulates Bose's cloud services, enabling offline operation and custom integrations. This is particularly valuable as Bose has announced the discontinuation of cloud support in May 2026.
+### SoundTouch Service (Cloud Shutdown Protection)

-#### Key Features
+The `soundtouch-service` is a local server that emulates Bose's cloud services. This is critical for keeping your speakers functional after the **Bose Cloud Shutdown in May 2026**.

- **🏠 Local Service Emulation**: Complete BMX (Bose Media eXchange) and Marge service implementation
- **🔧 Device Migration**: Seamlessly migrate devices from Bose cloud to local services  
- **📊 Traffic Proxying**: Inspect and log all device communications for debugging
- **🌐 Web Management UI**: Browser-based interface for device management
- **💾 Persistent Data**: Store device configurations, presets, and usage statistics
- **🔍 Auto-Discovery**: Automatically detect and configure SoundTouch devices
- **🔒 Offline Operation**: Continue using full device functionality without internet
-
-#### Quick Start
+#### Key Features:
+- **🏠 Local Emulation**: BMX and Marge service implementation
+- **🔧 Device Migration**: Seamlessly transition devices to local control
+- **🌐 Web Management UI**: Easy browser-based setup and management
+- **💾 Persistent Data**: Store presets, recents, and sources locally
+- **📝 HTTP Recording**: Persist all interactions as re-playable `.http` files

+#### Quick Start:
 ```bash
-# Install the service
-go install github.com/gesellix/bose-soundtouch/cmd/soundtouch-service@latest
-
-# Start with default settings (http://localhost:8000, proxying to http://localhost:8001)
+# Start the service
 soundtouch-service
-
-# Or configure with environment variables
-PORT=9000 PYTHON_BACKEND_URL=http://your-python-backend:8001 DATA_DIR=/my/data soundtouch-service
 ```
+Open `http://localhost:8000` in your browser to manage your devices.

-#### Running with Docker
+For a comprehensive guide on transitioning your system, see the [Bose Cloud Shutdown: Survival Guide](docs/CLOUD-SHUTDOWN-GUIDE.md).

-You can also run the SoundTouch service using Docker or Docker Compose.
-
-> **Note for macOS and Windows users**: The `--net host` option is only supported on Linux. On macOS and Windows, service discovery (mDNS, UPnP) will not work automatically within the container. You will need to manually enter your device's IP address in the management UI, and the service will communicate with it directly.
-
-##### Using Docker
-
-**Linux (with host networking for discovery):**
-```bash
-docker run -d \
-  --name soundtouch-service \
-  --network host \
-  -v $(pwd)/data:/app/data \
-  ghcr.io/gesellix/bose-soundtouch:latest
-```
-
-**macOS / Windows (with port mapping):**
-```bash
-docker run --rm -it \
-  -p 8000:8000 -p 8443:8443 \
-  -v $(pwd)/data:/app/data \
-  --env SERVER_URL=http://soundtouch.local:8000 \
-  --env HTTPS_SERVER_URL=https://soundtouch.local:8443 \
-  ghcr.io/gesellix/bose-soundtouch:latest
-```
-
-> **Note**: The hostnames configured via `SERVER_URL` and `HTTPS_SERVER_URL` are automatically added as Subject Alternative Names (SAN) to the generated TLS certificate, ensuring valid SSL connections.
-
-Alternatively, without explicit server URLs:
-```bash
-docker run -d \
-  --name soundtouch-service \
-  -p 8000:8000 \
-  -v $(pwd)/data:/app/data \
-  ghcr.io/gesellix/bose-soundtouch:latest
-```
-
-##### Using Docker Compose
-
-Create a `docker-compose.yml` file:
-
-```yaml
-services:
-  soundtouch-service:
-    image: ghcr.io/gesellix/bose-soundtouch:latest
-    container_name: soundtouch-service
-    # Linux users: use host networking for device discovery
-    # network_mode: host
-    # macOS/Windows users: use port mapping (discovery will be manual)
-    ports:
-      - "8000:8000"
-      - "8443:8443"
-    environment:
-      - PORT=8000
-      - SERVER_URL=http://soundtouch.local:8000
-      - HTTPS_SERVER_URL=https://soundtouch.local:8443
-      - DATA_DIR=/app/data
-    volumes:
-      - ./data:/app/data
-    restart: unless-stopped
-```
-
-> **Note**: Hostnames from `SERVER_URL` and `HTTPS_SERVER_URL` are automatically included in the TLS certificate's Subject Alternative Names (SAN).
-
-And run:
-
-```bash
-docker-compose up -d
-```
-
-> **Note**: `--network host` is required for device discovery via UPnP and mDNS to work correctly within the container.
-
-#### Device Migration Example
-
-```bash
-# 1. Start the service
-soundtouch-service
-
-# 2. Open web UI at http://localhost:8000
-# 3. Discover your devices
-# 4. Click "Migrate" to configure devices to use local services
-
-# Or use the API directly:
-curl -X POST http://localhost:8000/setup/migrate/192.168.1.100
-```
-
-#### Service Endpoints
-
- **Web UI**: `http://localhost:8000/` - Device management interface
- **Discovery**: `GET /setup/devices` - List discovered devices
- **Migration**: `POST /setup/migrate/{deviceIP}` - Switch device to local services
- **BMX Services**: `/bmx/*` - Music service emulation (TuneIn, etc.)
- **Marge Services**: `/marge/*` - Account and device management
- **Proxy**: `/proxy/*` - Traffic inspection and debugging
-
-See [docs/SOUNDTOUCH-SERVICE.md](docs/SOUNDTOUCH-SERVICE.md) for detailed configuration and API reference.
+Detailed service configuration and Docker instructions can be found in [docs/SOUNDTOUCH-SERVICE.md](docs/SOUNDTOUCH-SERVICE.md).

 ### Library Usage

@@ -5,6 +5,8 @@ package main
 import (
 	"context"
 	"crypto/tls"
+	"flag"
+	"fmt"
 	"log"
 	"net/http"
 	"net/http/httputil"
@@ -28,14 +30,27 @@ func main() {
 	ds := initDataStore(config.dataDir)
 	cm := initCertificateManager(config.dataDir)
 	sm := setup.NewManager(config.serverURL, ds, cm)
-	server := handlers.NewServer(ds, sm, config.serverURL, config.redact, config.logBody)
+	server := handlers.NewServer(ds, sm, config.serverURL, config.redact, config.logBody, config.record)
+
+	recorder := proxy.NewRecorder(config.dataDir)
+	recorder.Redact = config.redact
+	patternsPath := filepath.Join(config.dataDir, "patterns.json")
+
+	patterns, err := proxy.LoadPatterns(patternsPath)
+	if err == nil && len(patterns) > 0 {
+		recorder.Patterns = patterns
+	} else if err != nil {
+		log.Printf("Warning: Failed to load patterns from %s: %v", patternsPath, err)
+	}
+
+	server.SetRecorder(recorder)

 	tlsConfig, err := cm.GetServerTLSConfig(config.domains)
 	if err != nil {
 		log.Printf("Warning: Failed to setup TLS: %v", err)
 	}

-	pyProxy := setupPythonProxy(config.targetURL, config.redact, config.logBody)
+	pyProxy := setupPythonProxy(config.targetURL, config.redact, config.logBody, recorder, server)

 	startDeviceDiscovery(server)

@@ -61,61 +76,32 @@ type serviceConfig struct {
 	httpsAddr      string
 	redact         bool
 	logBody        bool
+	record         bool
 	domains        []string
 }

 func loadConfig() serviceConfig {
-	port := os.Getenv("PORT")
-	if port == "" {
-		port = "8000"
+	// Define flags
+	fPort, fBindAddr, fTargetURL, fDataDir, fServerURL, fHTTPSPort, fHTTPSServerURL, fRedact, fLogBody, fRecord := defineFlags()
+
+	flag.Usage = func() {
+		fmt.Fprintf(os.Stderr, "Usage of soundtouch-service:\n")
+		flag.PrintDefaults()
+		fmt.Fprintf(os.Stderr, "\nConfiguration can also be set via environment variables.\n")
 	}

-	bindAddr := os.Getenv("BIND_ADDR")
+	flag.Parse()
+
+	port := getEnvOrDefault("PORT", *fPort, "8000")
+	bindAddr := getEnvOrDefault("BIND_ADDR", *fBindAddr, "")

 	addr := bindAddr + ":" + port
 	if bindAddr == "" {
 		addr = ":" + port
 	}

-	targetURL := os.Getenv("PYTHON_BACKEND_URL")
-	if targetURL == "" {
-		targetURL = "http://localhost:8001"
-	}
-
-	dataDir := os.Getenv("DATA_DIR")
-	if dataDir == "" {
-		dataDir = "data"
-	}
-
-	serverURL := os.Getenv("SERVER_URL")
-	if serverURL == "" {
-		hostname, _ := os.Hostname()
-		if hostname == "" {
-			hostname = "localhost"
-		}
-
-		serverURL = "http://" + strings.ToLower(hostname) + ":" + port
-	}
-
-	httpsPort := os.Getenv("HTTPS_PORT")
-	if httpsPort == "" {
-		httpsPort = "8443"
-	}
-
-	httpsAddr := bindAddr + ":" + httpsPort
-	if bindAddr == "" {
-		httpsAddr = ":" + httpsPort
-	}
-
-	httpsServerURL := os.Getenv("HTTPS_SERVER_URL")
-	if httpsServerURL == "" {
-		hostname, _ := os.Hostname()
-		if hostname == "" {
-			hostname = "localhost"
-		}
-
-		httpsServerURL = "https://" + strings.ToLower(hostname) + ":" + httpsPort
-	}
+	targetURL := getEnvOrDefault("PYTHON_BACKEND_URL", *fTargetURL, "http://localhost:8001")
+	dataDir := getEnvOrDefault("DATA_DIR", *fDataDir, "data")

 	hostname, _ := os.Hostname()
 	if hostname == "" {
@@ -124,6 +110,79 @@ func loadConfig() serviceConfig {

 	hostname = strings.ToLower(hostname)

+	serverURL := getEnvOrDefault("SERVER_URL", *fServerURL, "")
+	if serverURL == "" {
+		serverURL = "http://" + hostname + ":" + port
+	}
+
+	httpsPort := getEnvOrDefault("HTTPS_PORT", *fHTTPSPort, "8443")
+
+	httpsAddr := bindAddr + ":" + httpsPort
+	if bindAddr == "" {
+		httpsAddr = ":" + httpsPort
+	}
+
+	httpsServerURL := getEnvOrDefault("HTTPS_SERVER_URL", *fHTTPSServerURL, "")
+	if httpsServerURL == "" {
+		httpsServerURL = "https://" + hostname + ":" + httpsPort
+	}
+
+	domains := getDomains(serverURL, httpsServerURL, hostname)
+
+	redactVal := getEnvOrDefault("REDACT_PROXY_LOGS", *fRedact, "")
+	redact := redactVal != "false"
+
+	logBodyVal := getEnvOrDefault("LOG_PROXY_BODY", *fLogBody, "")
+	logBody := logBodyVal == "true"
+
+	recordVal := getEnvOrDefault("RECORD_INTERACTIONS", *fRecord, "")
+	record := recordVal != "false"
+
+	return serviceConfig{
+		port:           port,
+		bindAddr:       bindAddr,
+		addr:           addr,
+		targetURL:      targetURL,
+		dataDir:        dataDir,
+		serverURL:      serverURL,
+		httpsServerURL: httpsServerURL,
+		httpsAddr:      httpsAddr,
+		redact:         redact,
+		logBody:        logBody,
+		record:         record,
+		domains:        domains,
+	}
+}
+
+func defineFlags() (port, bind, target, data, server, httpsPort, httpsServer, redact, logBody, record *string) {
+	port = flag.String("port", "", "Port to bind the service to (env: PORT)")
+	bind = flag.String("bind", "", "Network interface to bind to (env: BIND_ADDR)")
+	target = flag.String("target-url", "", "URL for Python-based service components (env: PYTHON_BACKEND_URL)")
+	data = flag.String("data-dir", "", "Directory for persistent data (env: DATA_DIR)")
+	server = flag.String("server-url", "", "External URL of this service (env: SERVER_URL)")
+	httpsPort = flag.String("https-port", "", "HTTPS port to bind the service to (env: HTTPS_PORT)")
+	httpsServer = flag.String("https-server-url", "", "External HTTPS URL (env: HTTPS_SERVER_URL)")
+	redact = flag.String("redact-logs", "", "Redact sensitive data in proxy logs (true/false, env: REDACT_PROXY_LOGS)")
+	logBody = flag.String("log-bodies", "", "Log full request/response bodies (true/false, env: LOG_PROXY_BODY)")
+	record = flag.String("record-interactions", "", "Record HTTP interactions to disk (true/false, env: RECORD_INTERACTIONS)")
+
+	return
+}
+
+func getEnvOrDefault(envKey, flagVal, defaultVal string) string {
+	if flagVal != "" {
+		return flagVal
+	}
+
+	val := os.Getenv(envKey)
+	if val != "" {
+		return val
+	}
+
+	return defaultVal
+}
+
+func getDomains(serverURL, httpsServerURL, hostname string) []string {
 	domainsMap := map[string]bool{
 		"streaming.bose.com":  true,
 		"updates.bose.com":    true,
@@ -149,19 +208,7 @@ func loadConfig() serviceConfig {
 		domains = append(domains, d)
 	}

-	return serviceConfig{
-		port:           port,
-		bindAddr:       bindAddr,
-		addr:           addr,
-		targetURL:      targetURL,
-		dataDir:        dataDir,
-		serverURL:      serverURL,
-		httpsServerURL: httpsServerURL,
-		httpsAddr:      httpsAddr,
-		redact:         os.Getenv("REDACT_PROXY_LOGS") != "false",
-		logBody:        os.Getenv("LOG_PROXY_BODY") == "true",
-		domains:        domains,
-	}
+	return domains
 }

 func initDataStore(dataDir string) *datastore.DataStore {
@@ -182,7 +229,7 @@ func initCertificateManager(dataDir string) *certmanager.CertificateManager {
 	return cm
 }

-func setupPythonProxy(targetURL string, redact, logBody bool) *httputil.ReverseProxy {
+func setupPythonProxy(targetURL string, redact, logBody bool, recorder *proxy.Recorder, server *handlers.Server) *httputil.ReverseProxy {
 	target, err := url.Parse(targetURL)
 	if err != nil {
 		log.Fatalf("Failed to parse target URL: %v", err)
@@ -197,6 +244,8 @@ func setupPythonProxy(targetURL string, redact, logBody bool) *httputil.ReverseP

 		currentLp := proxy.NewLoggingProxy(target.String(), redact)
 		currentLp.LogBody = logBody
+		currentLp.RecordEnabled = server.GetRecordEnabled()
+		currentLp.SetRecorder(recorder)
 		currentLp.LogResponse(res)

 		return nil
@@ -208,6 +257,8 @@ func setupPythonProxy(targetURL string, redact, logBody bool) *httputil.ReverseP

 		currentLp := proxy.NewLoggingProxy(target.String(), redact)
 		currentLp.LogBody = logBody
+		currentLp.RecordEnabled = server.GetRecordEnabled()
+		currentLp.SetRecorder(recorder)
 		currentLp.LogRequest(req)
 	}

@@ -227,6 +278,7 @@ func setupRouter(server *handlers.Server, pyProxy *httputil.ReverseProxy) *chi.M
 	r := chi.NewRouter()
 	r.Use(middleware.Logger)
 	r.Use(middleware.Recoverer)
+	r.Use(server.RecordMiddleware)

 	r.Get("/", server.HandleRoot)
 	r.Get("/health", server.HandleHealth)
@@ -270,6 +322,7 @@ func setupRouter(server *handlers.Server, pyProxy *httputil.ReverseProxy) *chi.M

 	r.Route("/setup", func(r chi.Router) {
 		r.Get("/devices", server.HandleListDiscoveredDevices)
+		r.Post("/devices", server.HandleAddManualDevice)
 		r.Post("/discover", server.HandleTriggerDiscovery)
 		r.Get("/discovery-status", server.HandleGetDiscoveryStatus)
 		r.Get("/settings", server.HandleGetSettings)
@@ -280,6 +333,7 @@ func setupRouter(server *handlers.Server, pyProxy *httputil.ReverseProxy) *chi.M
 		r.Post("/ensure-remote-services/{deviceIP}", server.HandleEnsureRemoteServices)
 		r.Post("/remove-remote-services/{deviceIP}", server.HandleRemoveRemoteServices)
 		r.Post("/backup/{deviceIP}", server.HandleBackupConfig)
+		r.Post("/sync/{deviceIP}", server.HandleInitialSync)
 		r.Post("/test-connection/{deviceIP}", server.HandleTestConnection)
 		r.Post("/test-hosts/{deviceIP}", server.HandleTestHostsRedirection)
 		r.Get("/ca.crt", server.HandleGetCACert)
@@ -1,2 +1,3 @@
 certs/
 default/
+interactions/
@@ -0,0 +1,17 @@
+[
+  {
+    "name": "IPv4",
+    "regexp": "^\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}$",
+    "replacement": "{ip}"
+  },
+  {
+    "name": "DeviceID",
+    "regexp": "^[A-F0-9]{12}$",
+    "replacement": "{deviceId}"
+  },
+  {
+    "name": "AccountID",
+    "regexp": "^\\d{1,10}$",
+    "replacement": "{accountId}"
+  }
+]
@@ -0,0 +1,81 @@
+### Bose Cloud Shutdown: Survival Guide for SoundTouch
+
+With Bose's announcement of discontinuing cloud support for SoundTouch devices in May 2026, this project provides the necessary tools to keep your speakers fully functional using a local emulation service.
+
+This guide explains how to set up the `soundtouch-service` to run your devices independently of Bose's servers.
+
+---
+
+### Supported Use Cases
+
+1.  **Local Service Emulation**: The service emulates Bose's BMX (Bose Media eXchange) and Marge services, which handle content registries, presets, recents, and software update checks.
+2.  **Traffic Redirection**: Tools are provided to redirect your speakers to this local service instead of `*.bose.com`.
+3.  **Offline Operation**: Once redirected, the speakers function without needing to reach Bose's servers.
+4.  **Preset & Recent Management**: Captures and stores presets and "recently played" items locally.
+
+---
+
+### Setup Steps
+
+To set up your SoundTouch system for local-only operation, follow these steps:
+
+#### 1. Install and Start the Service
+Run the `soundtouch-service` on a machine that is always on (like a Raspberry Pi or a NAS) within your local network.
+
+```bash
+# Install the service
+go install github.com/gesellix/bose-soundtouch/cmd/soundtouch-service@latest
+
+# Start the service (defaults to http://localhost:8000)
+soundtouch-service
+```
+
+#### 2. Access the Management UI
+Open your web browser and navigate to the service's web interface:
+`http://<your-server-ip>:8000/`, e.g. `http://localhost:8000/`
+
+*Note: The service also supports a `/web/` path for management.*
+
+#### 3. Enable SSH on Your Speakers
+To migrate your speakers, the service needs SSH access. You can enable it by:
+1. Creating an empty file named `remote_services` on a USB stick.
+2. Inserting the USB stick into the SoundTouch speaker's service port.
+3. Rebooting the speaker.
+Once enabled, you can log in as `root` (no password).
+
+#### 4. Discover and Sync Device Data
+The web interface handles the entire process in a guided flow across four tabs:
+
+*   **Step 1: Devices**: The service automatically scans for SoundTouch devices on your network. If a device is not found, you can manually add its IP address.
+*   **Step 2: Data Sync**: Select your device and click "Start Sync". This will automatically fetch your presets, recents, and configured sources from the speaker and store them in the local `data/` directory.
+*   **Step 3: Migration**: Choose your redirection method (XML Recommended) and click "Confirm Migration & Reboot".
+*   **Step 4: Settings**: Configure global server URLs and proxy behavior (logging, redaction).
+
+#### 5. Verify Your Local Data
+Once migrated, your speaker will use the data captured during the Sync step.
+*   The service stores data in the `data/` directory, organized by device serial number (e.g., `data/default/devices/<SERIAL>/`).
+*   **Automatic Capture**: As you use the device (changing presets, playing new music), the service continues to "learn" and update your local files.
+
+---
+
+### Comparison with other implementations (soundcork)
+Our implementation (`soundtouch-service`) is largely compatible with the Python-based `soundcork` project but offers several advantages:
+- **Web UI**: Integrated management interface for discovery and migration.
+- **Surgical Migration**: Uses XML-based redirection by default, which is less invasive than `/etc/hosts`.
+- **Automated SSL**: Handles Root CA injection automatically for secure communication.
+- **Proxy Support**: Can proxy requests to original Bose servers while "learning" your configuration.
+
+---
+
+### Alternative: DNS Redirection (No SSH)
+If you prefer not to modify your speakers via SSH, you can use a local DNS server (like Pi-hole, AdGuard Home, or Unbound) to point the following domains to your local server's IP:
+
+*   `bmx.bose.com`
+*   `streaming.bose.com`
+*   `updates.bose.com`
+*   `stats.bose.com`
+*   `content.api.bose.io`
+
+*Note: DNS redirection for HTTPS services requires the speakers to trust your local service's SSL certificate. The SSH-based migration handles this automatically by injecting the CA.*
+
+---
@@ -11,6 +11,7 @@ The service provides:
 - **📊 Traffic Proxying**: Inspect and log all device communications for debugging
 - **🌐 Web Management UI**: Browser-based interface for device management
 - **💾 Persistent Data**: Store device configurations, presets, and usage statistics
+- **📝 HTTP Recording**: Persist all interactions as re-playable `.http` files
 - **🔍 Auto-Discovery**: Automatically detect and configure SoundTouch devices
 - **🔒 Offline Operation**: Continue using full device functionality without internet

@@ -49,10 +50,64 @@ cd Bose-SoundTouch
 go build -o soundtouch-service ./cmd/soundtouch-service
 ```

-### Docker (coming soon)
+### Docker Support
+
+You can run the SoundTouch service using Docker or Docker Compose. 
+
+> **Note for macOS and Windows users**: The `--net host` option is only supported on Linux. On macOS and Windows, service discovery (mDNS, UPnP) will not work automatically within the container. You will need to manually enter your device's IP address in the management UI, and the service will communicate with it directly.
+
+#### Using Docker
+
+**Linux (with host networking for discovery):**
 ```bash
-# Docker support planned for future release
-docker run -p 8000:8000 gesellix/soundtouch-service
+docker run -d \
+  --name soundtouch-service \
+  --network host \
+  -v $(pwd)/data:/app/data \
+  ghcr.io/gesellix/bose-soundtouch:latest
+```
+
+**macOS / Windows (with port mapping):**
+```bash
+docker run --rm -it \
+  -p 8000:8000 -p 8443:8443 \
+  -v $(pwd)/data:/app/data \
+  --env SERVER_URL=http://soundtouch.local:8000 \
+  --env HTTPS_SERVER_URL=https://soundtouch.local:8443 \
+  ghcr.io/gesellix/bose-soundtouch:latest
+```
+
+> **Note**: The hostnames configured via `SERVER_URL` and `HTTPS_SERVER_URL` are automatically added as Subject Alternative Names (SAN) to the generated TLS certificate, ensuring valid SSL connections.
+
+#### Using Docker Compose
+
+Create a `docker-compose.yml` file:
+
+```yaml
+services:
+  soundtouch-service:
+    image: ghcr.io/gesellix/bose-soundtouch:latest
+    container_name: soundtouch-service
+    # Linux users: use host networking for device discovery
+    # network_mode: host
+    # macOS/Windows users: use port mapping (discovery will be manual)
+    ports:
+      - "8000:8000"
+      - "8443:8443"
+    environment:
+      - PORT=8000
+      - SERVER_URL=http://soundtouch.local:8000
+      - HTTPS_SERVER_URL=https://soundtouch.local:8443
+      - DATA_DIR=/app/data
+    volumes:
+      - ./data:/app/data
+    restart: unless-stopped
+```
+
+And run:
+
+```bash
+docker-compose up -d
 ```

 ## Quick Start
@@ -80,15 +135,18 @@ Use the web interface or API to migrate devices from Bose cloud services to your

 The service can be configured via environment variables or command-line flags:

-| Variable | Flag | Description | Default |
-|----------|------|-------------|---------|
-| `PORT` | `--port` | Port to bind the service to | `8000` |
-| `BIND_ADDR` | `--bind` | Network interface to bind to | all (ipv4 and ipv6) |
-| `DATA_DIR` | `--data-dir` | Directory for persistent data | `./data` |
-| `SERVER_URL` | `--server-url` | External URL of this service | `http://<hostname>:8000` |
-| `REDACT_PROXY_LOGS` | `--redact-logs` | Redact sensitive data in proxy logs | `true` |
-| `LOG_PROXY_BODY` | `--log-bodies` | Log full request/response bodies | `false` |
-| `DISCOVERY_INTERVAL` | `--discovery-interval` | Device discovery interval | `5m` |
+| Variable              | Flag                    | Description                                      | Default                   |
+|-----------------------|-------------------------|--------------------------------------------------|---------------------------|
+| `PORT`                | `--port`                | Port to bind the service to                      | `8000`                    |
+| `BIND_ADDR`           | `--bind`                | Network interface to bind to                     | all (ipv4 and ipv6)       |
+| `DATA_DIR`            | `--data-dir`            | Directory for persistent data                    | `./data`                  |
+| `SERVER_URL`          | `--server-url`          | External URL of this service                     | `http://<hostname>:8000`  |
+| `HTTPS_SERVER_URL`    | `--https-server-url`    | External HTTPS URL                               | `https://<hostname>:8443` |
+| `PYTHON_BACKEND_URL`  |                         | URL for Python-based service components (legacy) |                           |
+| `REDACT_PROXY_LOGS`   | `--redact-logs`         | Redact sensitive data in proxy logs              | `true`                    |
+| `LOG_PROXY_BODY`      | `--log-bodies`          | Log full request/response bodies                 | `false`                   |
+| `RECORD_INTERACTIONS` | `--record-interactions` | Record HTTP interactions to disk                 | `true`                    |
+| `DISCOVERY_INTERVAL`  | `--discovery-interval`  | Device discovery interval                        | `5m`                      |

 ### Configuration Examples

@@ -309,6 +367,48 @@ The web management interface provides a comprehensive dashboard for managing you
 3. **Troubleshooting**: Use the debug tools to diagnose device connectivity issues
 4. **Log Analysis**: Enable detailed logging for development and troubleshooting

+## HTTP Interaction Recording
+
+The service automatically records all HTTP interactions (both those handled locally and those proxied upstream) as `.http` files. These files are compatible with the [IntelliJ IDEA HTTP Client](https://www.jetbrains.com/help/idea/exploring-http-syntax.html).
+
+### Key Features
+
+- **Session Grouping**: All interactions from a single server session are stored in a dedicated directory named `{timestamp}-{pid}`.
+- **Chronological Order**: Files are prefixed with a sequential number (e.g., `0001-`, `0002-`) to preserve the exact order of requests across the entire session.
+- **Path-Based Structure**: Recordings are organized into subdirectories based on their URL path for better discoverability.
+- **Automatic Sanitization**: Variable path segments like IP addresses, Device IDs, and Account IDs are automatically identified and replaced with placeholders (e.g., `{{ip}}`, `{{deviceId}}`). The original values are preserved as comments at the top of the recorded `.http` files for easy identification.
+- **Re-playability**: An `http-client.env.json` file is generated for each session, allowing you to re-play the recorded requests immediately in IntelliJ IDEA.
+
+### Configuration
+
+#### Redaction
+
+By default, the service redacts sensitive information from the recorded `.http` files, including:
+- `Authorization` headers
+- `Cookie` headers
+- `X-Bose-Token` headers
+
+This behavior is controlled by the `--redact-logs` flag or the `REDACT_PROXY_LOGS` environment variable.
+
+#### Custom Patterns
+
+The service uses regex patterns to identify variable segments in URL paths. These patterns are loaded from `data/patterns.json`. You can add custom patterns to this file to support additional variable segments:
+
+```json
+[
+  {
+    "name": "MyVariable",
+    "regexp": "^[0-9]{5}$",
+    "replacement": "{myVar}"
+  }
+]
+```
+
+Variables found via these patterns will be:
+1. Used as directory names in the `interactions/` folder.
+2. Parameterized as `{{myVar}}` within the `.http` files.
+3. Added to the `http-client.env.json` file with their actual values.
+
 ## Persistent Data

 ### Data Directory Structure
@@ -327,6 +427,15 @@ data/
 │       ├── Sources.xml
 │       ├── Presets.xml
 │       └── Recents.xml
+├── interactions/
+│   └── {SESSION_ID}/
+│       ├── self/
+│       │   └── {PATH}/
+│       │       └── {SEQ}-{TIME}-{METHOD}.http
+│       ├── upstream/
+│       │   └── {PATH}/
+│       │       └── {SEQ}-{TIME}-{METHOD}.http
+│       └── http-client.env.json
 ├── stats/
 │   ├── usage/
 │   │   └── *.json
@@ -355,6 +464,14 @@ data/
 #### Events (`events/`)
 - **device_events_*.log**: Device event history and debugging logs

+#### HTTP Interactions (`interactions/`)
+- **{SESSION_ID}/**: A unique directory per server run (format: `YYYYMMDD-HHMMSS-PID`).
+- **self/**: Requests handled directly by the service.
+- **upstream/**: Requests proxied to external Bose services.
+- **{PATH}/**: Nested subdirectories reflecting the URL path (sanitized).
+- **http-client.env.json**: IntelliJ IDEA HTTP Client environment file with session variables.
+- **{SEQ}-{TIME}-{METHOD}.http**: Individual interaction recordings in standard HTTP Client format.
+
 ### Data Management

 #### Backup Strategy
@@ -383,6 +500,29 @@ find data/events/ -name "*.log" -mtime +30 -delete
 find data/stats/ -name "*.json" -mtime +90 -delete
 ```

+## API Endpoints
+
+### Management UI
+- **URL**: `http://localhost:8000/` or `http://localhost:8000/web/`
+- **Description**: Browser-based guided flow for discovery, data sync, and migration.
+
+### Setup API
+- `GET /setup/devices`: List all known (auto-discovered and manual) devices.
+- `POST /setup/devices`: Manually add a device by IP.
+- `POST /setup/discover`: Trigger a new network discovery scan.
+- `GET /setup/discovery-status`: Check if a scan is currently in progress.
+- `POST /setup/sync/{deviceIP}`: Fetch presets, recents, and sources from a device.
+- `GET /setup/summary/{deviceIP}`: Get a detailed migration readiness summary.
+- `POST /setup/migrate/{deviceIP}`: Migrate a device using the specified method (XML/Hosts).
+- `GET /setup/ca.crt`: Download the Root CA certificate for manual installation.
+
+### Emulated Services
+- `/bmx/registry/v1/services`: BMX service registry.
+- `/bmx/tunein/v1/*`: TuneIn radio emulation.
+- `/marge/accounts/*`: Account and device management.
+- `/marge/updates/soundtouch`: Software update emulation.
+- `/proxy/*`: Logging proxy for original Bose services.
+
 ## Troubleshooting

 ### Common Issues
@@ -160,12 +160,19 @@ type ServiceRecent struct {

 // ConfiguredSource represents a configured media source with authentication details.
 type ConfiguredSource struct {
-	DisplayName      string `json:"display_name" xml:"sourcename"`
-	ID               string `json:"id" xml:"id,attr"`
-	Secret           string `json:"secret" xml:"credential"`
-	SecretType       string `json:"secret_type" xml:"credential_type,attr"`
-	SourceKeyType    string `json:"source_key_type" xml:"sourceproviderid"`
-	SourceKeyAccount string `json:"source_key_account" xml:"username"`
+	DisplayName string `json:"display_name" xml:"displayName,attr"`
+	ID          string `json:"id" xml:"id,attr"`
+	Secret      string `json:"secret" xml:"secret,attr"`
+	SecretType  string `json:"secret_type" xml:"secretType,attr"`
+	SourceKey   struct {
+		Type    string `xml:"type,attr"`
+		Account string `xml:"account,attr"`
+	} `json:"source_key" xml:"sourceKey"`
+
+	// Legacy fields for backward compatibility in code if needed,
+	// though it's better to update the code to use SourceKey.
+	SourceKeyType    string `json:"source_key_type" xml:"-"`
+	SourceKeyAccount string `json:"source_key_account" xml:"-"`
 }

 // ServiceDeviceInfo represents information about a SoundTouch device.
@@ -177,6 +184,7 @@ type ServiceDeviceInfo struct {
 	FirmwareVersion     string `json:"firmware_version" xml:"softwareVersion"`
 	IPAddress           string `json:"ip_address" xml:"ipAddress"`
 	Name                string `json:"name" xml:"name"`
+	DiscoveryMethod     string `json:"discovery_method,omitempty"`
 }

 // CustomerSupportDevice represents device information for customer support purposes.
@@ -132,7 +132,9 @@ func (ds *DataStore) ListAllDevices() ([]models.ServiceDeviceInfo, error) {
 			}

 			accDevices := ds.listDevicesInAccount(dir, acc.Name())
-			for _, info := range accDevices {
+			for i := range accDevices {
+				info := accDevices[i]
+
 				key := info.DeviceID
 				if key == "" {
 					key = info.IPAddress
@@ -219,6 +221,7 @@ func (ds *DataStore) parseDeviceInfoFile(path string) (*models.ServiceDeviceInfo
 			Type      string `xml:"type,attr"`
 			IPAddress string `xml:"ipAddress"`
 		} `xml:"networkInfo"`
+		DiscoveryMethod string `xml:"discoveryMethod"`
 	}

 	if err := xml.Unmarshal(data, &info); err != nil {
@@ -226,9 +229,10 @@ func (ds *DataStore) parseDeviceInfoFile(path string) (*models.ServiceDeviceInfo
 	}

 	deviceInfo := &models.ServiceDeviceInfo{
-		DeviceID:    info.DeviceID,
-		ProductCode: fmt.Sprintf("%s %s", info.Type, info.ModuleType),
-		Name:        info.Name,
+		DeviceID:        info.DeviceID,
+		ProductCode:     fmt.Sprintf("%s %s", info.Type, info.ModuleType),
+		Name:            info.Name,
+		DiscoveryMethod: info.DiscoveryMethod,
 	}

 	for _, comp := range info.Components {
@@ -250,9 +254,9 @@ func (ds *DataStore) parseDeviceInfoFile(path string) (*models.ServiceDeviceInfo
 	return deviceInfo, nil
 }

-// GetPresets retrieves all presets for the specified account.
-func (ds *DataStore) GetPresets(account string) ([]models.ServicePreset, error) {
-	path := filepath.Join(ds.AccountDir(account), constants.PresetsFile)
+// GetPresets retrieves all presets for the specified account and device.
+func (ds *DataStore) GetPresets(account, device string) ([]models.ServicePreset, error) {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.PresetsFile)

 	data, err := os.ReadFile(path)
 	if err != nil {
@@ -303,9 +307,9 @@ func (ds *DataStore) GetPresets(account string) ([]models.ServicePreset, error)
 	return presets, nil
 }

-// SavePresets saves the preset list for the specified account.
-func (ds *DataStore) SavePresets(account string, presets []models.ServicePreset) error {
-	path := filepath.Join(ds.AccountDir(account), constants.PresetsFile)
+// SavePresets saves the preset list for the specified account and device.
+func (ds *DataStore) SavePresets(account, device string, presets []models.ServicePreset) error {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.PresetsFile)

 	type PresetXML struct {
 		ID          string `xml:"id,attr"`
@@ -357,9 +361,9 @@ func (ds *DataStore) SavePresets(account string, presets []models.ServicePreset)
 	return os.WriteFile(path, append(header, data...), 0644)
 }

-// GetRecents retrieves all recent items for the specified account.
-func (ds *DataStore) GetRecents(account string) ([]models.ServiceRecent, error) {
-	path := filepath.Join(ds.AccountDir(account), constants.RecentsFile)
+// GetRecents retrieves all recent items for the specified account and device.
+func (ds *DataStore) GetRecents(account, device string) ([]models.ServiceRecent, error) {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.RecentsFile)

 	data, err := os.ReadFile(path)
 	if err != nil {
@@ -410,9 +414,9 @@ func (ds *DataStore) GetRecents(account string) ([]models.ServiceRecent, error)
 	return recents, nil
 }

-// SaveRecents saves the recent items list for the specified account.
-func (ds *DataStore) SaveRecents(account string, recents []models.ServiceRecent) error {
-	path := filepath.Join(ds.AccountDir(account), constants.RecentsFile)
+// SaveRecents saves the recent items list for the specified account and device.
+func (ds *DataStore) SaveRecents(account, device string, recents []models.ServiceRecent) error {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.RecentsFile)

 	type RecentXML struct {
 		ID          string `xml:"id,attr"`
@@ -494,13 +498,14 @@ func (ds *DataStore) SaveDeviceInfo(account, device string, info *models.Service
 	}

 	type InfoXML struct {
-		XMLName     xml.Name         `xml:"info"`
-		DeviceID    string           `xml:"deviceID,attr"`
-		Name        string           `xml:"name"`
-		Type        string           `xml:"type"`
-		ModuleType  string           `xml:"moduleType"`
-		Components  []ComponentXML   `xml:"components>component"`
-		NetworkInfo []NetworkInfoXML `xml:"networkInfo"`
+		XMLName         xml.Name         `xml:"info"`
+		DeviceID        string           `xml:"deviceID,attr"`
+		Name            string           `xml:"name"`
+		Type            string           `xml:"type"`
+		ModuleType      string           `xml:"moduleType"`
+		Components      []ComponentXML   `xml:"components>component"`
+		NetworkInfo     []NetworkInfoXML `xml:"networkInfo"`
+		DiscoveryMethod string           `xml:"discoveryMethod,omitempty"`
 	}

 	// Parsing product code back to type and moduleType (best effort)
@@ -539,6 +544,7 @@ func (ds *DataStore) SaveDeviceInfo(account, device string, info *models.Service
 				IPAddress: info.IPAddress,
 			},
 		},
+		DiscoveryMethod: info.DiscoveryMethod,
 	}

 	data, err := xml.MarshalIndent(ix, "", "    ")
@@ -557,9 +563,9 @@ func (ds *DataStore) RemoveDevice(account, device string) error {
 	return os.RemoveAll(dir)
 }

-// GetConfiguredSources retrieves all configured sources for the specified account.
-func (ds *DataStore) GetConfiguredSources(account string) ([]models.ConfiguredSource, error) {
-	path := filepath.Join(ds.AccountDir(account), constants.SourcesFile)
+// GetConfiguredSources retrieves all configured sources for the specified account and device.
+func (ds *DataStore) GetConfiguredSources(account, device string) ([]models.ConfiguredSource, error) {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.SourcesFile)

 	data, err := os.ReadFile(path)
 	if err != nil {
@@ -567,81 +573,52 @@ func (ds *DataStore) GetConfiguredSources(account string) ([]models.ConfiguredSo
 	}

 	var sourcesWrap struct {
-		Sources []struct {
-			DisplayName string `xml:"displayName,attr"`
-			ID          string `xml:"id,attr"`
-			Secret      string `xml:"secret,attr"`
-			SecretType  string `xml:"secretType,attr"`
-			SourceKey   struct {
-				Account string `xml:"account,attr"`
-				Type    string `xml:"type,attr"`
-			} `xml:"sourceKey"`
-		} `xml:"source"`
+		Sources []models.ConfiguredSource `xml:"source"`
 	}

 	if err := xml.Unmarshal(data, &sourcesWrap); err != nil {
 		return nil, fmt.Errorf("malformed sources XML at %s: %w", path, err)
 	}

-	var sources []models.ConfiguredSource
-
-	lastID := 100001
-
-	for _, s := range sourcesWrap.Sources {
-		id := s.ID
-		if id == "" {
-			id = strconv.Itoa(lastID)
-			lastID++
+	for i := range sourcesWrap.Sources {
+		s := &sourcesWrap.Sources[i]
+		if s.ID == "" {
+			s.ID = strconv.Itoa(100001 + i)
 		}
-
-		sources = append(sources, models.ConfiguredSource{
-			DisplayName:      s.DisplayName,
-			ID:               id,
-			Secret:           s.Secret,
-			SecretType:       s.SecretType,
-			SourceKeyType:    s.SourceKey.Type,
-			SourceKeyAccount: s.SourceKey.Account,
-		})
+		// Sync legacy fields
+		s.SourceKeyType = s.SourceKey.Type
+		s.SourceKeyAccount = s.SourceKey.Account
 	}

-	return sources, nil
+	return sourcesWrap.Sources, nil
 }

-// SaveConfiguredSources saves the configured sources list for the specified account.
-func (ds *DataStore) SaveConfiguredSources(account string, sources []models.ConfiguredSource) error {
-	path := filepath.Join(ds.AccountDir(account), constants.SourcesFile)
+// SaveConfiguredSources saves the configured sources list for the specified account and device.
+func (ds *DataStore) SaveConfiguredSources(account, device string, sources []models.ConfiguredSource) error {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.SourcesFile)
 	if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
 		return err
 	}

-	type sourceXML struct {
-		DisplayName string `xml:"displayName,attr"`
-		ID          string `xml:"id,attr"`
-		Secret      string `xml:"secret,attr"`
-		SecretType  string `xml:"secretType,attr"`
-		SourceKey   struct {
-			Account string `xml:"account,attr"`
-			Type    string `xml:"type,attr"`
-		} `xml:"sourceKey"`
-	}
-
 	type sourcesWrap struct {
-		XMLName xml.Name    `xml:"sources"`
-		Sources []sourceXML `xml:"source"`
+		XMLName xml.Name                  `xml:"sources"`
+		Sources []models.ConfiguredSource `xml:"source"`
 	}

-	wrap := sourcesWrap{}
-
-	for _, s := range sources {
-		sx := sourceXML{
-			DisplayName: s.DisplayName,
-			ID:          s.ID,
-			Secret:      s.Secret,
-			SecretType:  s.SecretType,
+	// Ensure SourceKey is populated from legacy fields if necessary before saving
+	for i := range sources {
+		s := &sources[i]
+		if s.SourceKey.Type == "" && s.SourceKeyType != "" {
+			s.SourceKey.Type = s.SourceKeyType
 		}
-		sx.SourceKey.Account = s.SourceKeyAccount
-		sx.SourceKey.Type = s.SourceKeyType
-		wrap.Sources = append(wrap.Sources, sx)
+
+		if s.SourceKey.Account == "" && s.SourceKeyAccount != "" {
+			s.SourceKey.Account = s.SourceKeyAccount
+		}
+	}
+
+	wrap := sourcesWrap{
+		Sources: sources,
 	}

 	data, err := xml.MarshalIndent(wrap, "", "    ")
@@ -675,9 +652,9 @@ func (ds *DataStore) Initialize() error {
 	return nil
 }

-// GetETagForPresets returns the ETag (modification time) for the presets file.
-func (ds *DataStore) GetETagForPresets(account string) int64 {
-	path := filepath.Join(ds.AccountDir(account), constants.PresetsFile)
+// GetETagForPresets returns the ETag (modification time) for the presets file for a specific device.
+func (ds *DataStore) GetETagForPresets(account, device string) int64 {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.PresetsFile)

 	info, err := os.Stat(path)
 	if err != nil {
@@ -687,9 +664,9 @@ func (ds *DataStore) GetETagForPresets(account string) int64 {
 	return info.ModTime().UnixNano() / int64(time.Millisecond)
 }

-// GetETagForSources returns the ETag (modification time) for the sources file.
-func (ds *DataStore) GetETagForSources(account string) int64 {
-	path := filepath.Join(ds.AccountDir(account), constants.SourcesFile)
+// GetETagForSources returns the ETag (modification time) for the sources file for a specific device.
+func (ds *DataStore) GetETagForSources(account, device string) int64 {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.SourcesFile)

 	info, err := os.Stat(path)
 	if err != nil {
@@ -699,9 +676,9 @@ func (ds *DataStore) GetETagForSources(account string) int64 {
 	return info.ModTime().UnixNano() / int64(time.Millisecond)
 }

-// GetETagForRecents returns the ETag (modification time) for the recents file.
-func (ds *DataStore) GetETagForRecents(account string) int64 {
-	path := filepath.Join(ds.AccountDir(account), constants.RecentsFile)
+// GetETagForRecents returns the ETag (modification time) for the recents file for a specific device.
+func (ds *DataStore) GetETagForRecents(account, device string) int64 {
+	path := filepath.Join(ds.AccountDeviceDir(account, device), constants.RecentsFile)

 	info, err := os.Stat(path)
 	if err != nil {
@@ -711,11 +688,11 @@ func (ds *DataStore) GetETagForRecents(account string) int64 {
 	return info.ModTime().UnixNano() / int64(time.Millisecond)
 }

-// GetETagForAccount returns the highest ETag among presets, sources, and recents for the account.
-func (ds *DataStore) GetETagForAccount(account string) int64 {
-	e1 := ds.GetETagForPresets(account)
-	e2 := ds.GetETagForSources(account)
-	e3 := ds.GetETagForRecents(account)
+// GetETagForAccount returns the highest ETag among presets, sources, and recents for the account and device.
+func (ds *DataStore) GetETagForAccount(account, device string) int64 {
+	e1 := ds.GetETagForPresets(account, device)
+	e2 := ds.GetETagForSources(account, device)
+	e3 := ds.GetETagForRecents(account, device)

 	maxETag := e1
 	if e2 > maxETag {
@@ -49,12 +49,12 @@ func TestDataStore(t *testing.T) {
 		},
 	}

-	err = ds.SavePresets(account, presets)
+	err = ds.SavePresets(account, device, presets)
 	if err != nil {
 		t.Errorf("SavePresets failed: %v", err)
 	}

-	loadedPresets, err := ds.GetPresets(account)
+	loadedPresets, err := ds.GetPresets(account, device)
 	if err != nil {
 		t.Errorf("GetPresets failed: %v", err)
 	}
@@ -72,12 +72,12 @@ func TestDataStore(t *testing.T) {
 		},
 	}

-	err = ds.SaveRecents(account, recents)
+	err = ds.SaveRecents(account, device, recents)
 	if err != nil {
 		t.Errorf("SaveRecents failed: %v", err)
 	}

-	loadedRecents, err := ds.GetRecents(account)
+	loadedRecents, err := ds.GetRecents(account, device)
 	if err != nil {
 		t.Errorf("GetRecents failed: %v", err)
 	}
@@ -294,29 +294,37 @@ func TestConfiguredSources(t *testing.T) {

 	sources := []models.ConfiguredSource{
 		{
-			DisplayName:      "Source 1",
-			ID:               "101",
-			Secret:           "secret1",
-			SecretType:       "type1",
+			DisplayName: "Source 1",
+			ID:          "101",
+			Secret:      "secret1",
+			SecretType:  "type1",
+			SourceKey: struct {
+				Type    string `xml:"type,attr"`
+				Account string `xml:"account,attr"`
+			}{Type: "TUNEIN", Account: "user1"},
 			SourceKeyType:    "TUNEIN",
 			SourceKeyAccount: "user1",
 		},
 		{
-			DisplayName:      "Source 2",
-			ID:               "102",
-			Secret:           "secret2",
-			SecretType:       "type2",
+			DisplayName: "Source 2",
+			ID:          "102",
+			Secret:      "secret2",
+			SecretType:  "type2",
+			SourceKey: struct {
+				Type    string `xml:"type,attr"`
+				Account string `xml:"account,attr"`
+			}{Type: "PANDORA", Account: "user2"},
 			SourceKeyType:    "PANDORA",
 			SourceKeyAccount: "user2",
 		},
 	}

-	err := ds.SaveConfiguredSources(account, sources)
+	err := ds.SaveConfiguredSources(account, "any", sources)
 	if err != nil {
 		t.Fatalf("SaveConfiguredSources failed: %v", err)
 	}

-	loadedSources, err := ds.GetConfiguredSources(account)
+	loadedSources, err := ds.GetConfiguredSources(account, "any")
 	if err != nil {
 		t.Fatalf("GetConfiguredSources failed: %v", err)
 	}
@@ -343,12 +351,12 @@ func TestConfiguredSources(t *testing.T) {
 		},
 	}

-	err = ds.SaveConfiguredSources(account, sources2)
+	err = ds.SaveConfiguredSources(account, "any", sources2)
 	if err != nil {
 		t.Fatal(err)
 	}

-	loadedSources2, err := ds.GetConfiguredSources(account)
+	loadedSources2, err := ds.GetConfiguredSources(account, "any")
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -23,17 +23,19 @@ func TestMargeETags(t *testing.T) {
 	ds := datastore.NewDataStore(tempDir)

 	account := "12345"
+	deviceID := "DEV1"
 	accountDir := filepath.Join(tempDir, account)
-	_ = os.MkdirAll(accountDir, 0755)
+	deviceDir := filepath.Join(accountDir, "devices", deviceID)
+	_ = os.MkdirAll(deviceDir, 0755)

 	// Create some initial data
-	presetsFile := filepath.Join(accountDir, "Presets.xml")
+	presetsFile := filepath.Join(deviceDir, "Presets.xml")
 	_ = os.WriteFile(presetsFile, []byte("<presets/>"), 0644)

-	sourcesFile := filepath.Join(accountDir, "Sources.xml")
+	sourcesFile := filepath.Join(deviceDir, "Sources.xml")
 	_ = os.WriteFile(sourcesFile, []byte("<sources/>"), 0644)

-	recentsFile := filepath.Join(accountDir, "Recents.xml")
+	recentsFile := filepath.Join(deviceDir, "Recents.xml")
 	_ = os.WriteFile(recentsFile, []byte("<recents/>"), 0644)

 	// Ensure devices directory exists for AccountFull
@@ -36,7 +36,9 @@ func (s *Server) HandleMargeSourceProviders(w http.ResponseWriter, r *http.Reque
 func (s *Server) HandleMargeAccountFull(w http.ResponseWriter, r *http.Request) {
 	account := chi.URLParam(r, "account")

-	etag := strconv.FormatInt(s.ds.GetETagForAccount(account), 10)
+	device := r.URL.Query().Get("device")
+
+	etag := strconv.FormatInt(s.ds.GetETagForAccount(account, device), 10)
 	if r.Header.Get("If-None-Match") == etag {
 		w.WriteHeader(http.StatusNotModified)
 		return
@@ -79,14 +81,15 @@ func (s *Server) HandleMargeSoftwareUpdate(w http.ResponseWriter, r *http.Reques
 // HandleMargePresets returns the Marge presets for a device.
 func (s *Server) HandleMargePresets(w http.ResponseWriter, r *http.Request) {
 	account := chi.URLParam(r, "account")
+	device := chi.URLParam(r, "device")

-	etag := strconv.FormatInt(s.ds.GetETagForPresets(account), 10)
+	etag := strconv.FormatInt(s.ds.GetETagForPresets(account, device), 10)
 	if r.Header.Get("If-None-Match") == etag {
 		w.WriteHeader(http.StatusNotModified)
 		return
 	}

-	data, err := marge.PresetsToXML(s.ds, account)
+	data, err := marge.PresetsToXML(s.ds, account, device)
 	if err != nil {
 		http.Error(w, err.Error(), http.StatusInternalServerError)
 		return
@@ -102,7 +105,7 @@ func (s *Server) HandleMargeUpdatePreset(w http.ResponseWriter, r *http.Request)
 	account := chi.URLParam(r, "account")
 	device := chi.URLParam(r, "device")

-	etag := strconv.FormatInt(s.ds.GetETagForPresets(account), 10)
+	etag := strconv.FormatInt(s.ds.GetETagForPresets(account, device), 10)
 	w.Header()["ETag"] = []string{etag}

 	presetNumberStr := chi.URLParam(r, "presetNumber")
@@ -134,7 +137,7 @@ func (s *Server) HandleMargeAddRecent(w http.ResponseWriter, r *http.Request) {
 	account := chi.URLParam(r, "account")
 	device := chi.URLParam(r, "device")

-	etag := strconv.FormatInt(s.ds.GetETagForRecents(account), 10)
+	etag := strconv.FormatInt(s.ds.GetETagForRecents(account, device), 10)
 	w.Header()["ETag"] = []string{etag}

 	body, err := io.ReadAll(r.Body)
@@ -135,12 +135,14 @@ func TestMargePresets(t *testing.T) {
 	ds := datastore.NewDataStore(tempDir)

 	account := "12345"
+	deviceID := "any"

 	accountDir := filepath.Join(tempDir, account)
-	err = os.MkdirAll(accountDir, 0755)
+	deviceDir := filepath.Join(accountDir, "devices", deviceID)
+	err = os.MkdirAll(deviceDir, 0755)

 	if err != nil {
-		t.Fatalf("Failed to create account dir: %v", err)
+		t.Fatalf("Failed to create device dir: %v", err)
 	}

 	r, _ := setupRouter("http://localhost:8001", ds)
@@ -149,24 +151,17 @@ func TestMargePresets(t *testing.T) {
 	defer ts.Close()

 	// Mock Sources.xml and Presets.xml
-	if err := os.WriteFile(filepath.Join(accountDir, "Sources.xml"), []byte(`
+	if err := os.WriteFile(filepath.Join(deviceDir, "Sources.xml"), []byte(`
 		<sources>
-			<source id="123" type="Audio">
-				<createdOn>2012-09-19T12:43:00.000+00:00</createdOn>
-				<credential type="token"></credential>
-				<name>TUNEIN</name>
-				<sourceproviderid>1</sourceproviderid>
-				<sourcename>TUNEIN</sourcename>
-				<sourcesettings></sourcesettings>
-				<updatedOn>2012-09-19T12:43:00.000+00:00</updatedOn>
-				<username></username>
+			<source id="123" displayName="TUNEIN" secret="" secretType="Audio">
+				<sourceKey type="TUNEIN" account=""/>
 			</source>
 		</sources>
 	`), 0644); err != nil {
 		t.Fatalf("Failed to write Sources.xml: %v", err)
 	}

-	if err := os.WriteFile(filepath.Join(accountDir, "Presets.xml"), []byte(`
+	if err := os.WriteFile(filepath.Join(deviceDir, "Presets.xml"), []byte(`
 		<presets>
 			<preset id="1">
 				<ContentItem source="TUNEIN" type="station" location="/station/s123" sourceAccount="" isPresetable="true">
@@ -211,26 +206,28 @@ func TestMargeUpdatePreset(t *testing.T) {
 	ds := datastore.NewDataStore(tempDir)

 	account := "12345"
+	deviceID := "DEV1"

 	accountDir := filepath.Join(tempDir, account)
-	err = os.MkdirAll(accountDir, 0755)
+	deviceDir := filepath.Join(accountDir, "devices", deviceID)
+	err = os.MkdirAll(deviceDir, 0755)

 	if err != nil {
-		t.Fatalf("Failed to create account dir: %v", err)
+		t.Fatalf("Failed to create device dir: %v", err)
 	}

 	// Mock Sources.xml
-	if err := os.WriteFile(filepath.Join(accountDir, "Sources.xml"), []byte(`
+	if err := os.WriteFile(filepath.Join(deviceDir, "Sources.xml"), []byte(`
 		<sources>
-			<source id="SRC1" type="Audio">
-				<sourcename>TUNEIN</sourcename>
+			<source id="SRC1" displayName="TUNEIN" secret="" secretType="Audio">
+				<sourceKey type="TUNEIN" account=""/>
 			</source>
 		</sources>
 	`), 0644); err != nil {
 		t.Fatalf("Failed to write Sources.xml: %v", err)
 	}

-	if err := os.WriteFile(filepath.Join(accountDir, "Presets.xml"), []byte(`<presets></presets>`), 0644); err != nil {
+	if err := os.WriteFile(filepath.Join(deviceDir, "Presets.xml"), []byte(`<presets></presets>`), 0644); err != nil {
 		t.Fatalf("Failed to write Presets.xml: %v", err)
 	}

@@ -248,7 +245,7 @@ func TestMargeUpdatePreset(t *testing.T) {
 			<containerArt>http://example.com/new.jpg</containerArt>
 		</preset>`

-	res, err := http.Post(ts.URL+"/marge/accounts/"+account+"/devices/DEV1/presets/1", "application/xml", strings.NewReader(payload))
+	res, err := http.Post(ts.URL+"/marge/accounts/"+account+"/devices/"+deviceID+"/presets/1", "application/xml", strings.NewReader(payload))
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -261,7 +258,7 @@ func TestMargeUpdatePreset(t *testing.T) {
 	}

 	// Verify file was saved
-	presetData, _ := os.ReadFile(filepath.Join(accountDir, "Presets.xml"))
+	presetData, _ := os.ReadFile(filepath.Join(deviceDir, "Presets.xml"))
 	if !strings.Contains(string(presetData), "New Preset") {
 		t.Error("Preset was not saved to datastore")
 	}
@@ -278,26 +275,28 @@ func TestMargeDeviceInfo(t *testing.T) {
 	ds := datastore.NewDataStore(tempDir)

 	account := "12345"
+	deviceID := "DEV1"

 	accountDir := filepath.Join(tempDir, account)
-	err = os.MkdirAll(accountDir, 0755)
+	deviceDir := filepath.Join(accountDir, "devices", deviceID)
+	err = os.MkdirAll(deviceDir, 0755)

 	if err != nil {
-		t.Fatalf("Failed to create account dir: %v", err)
+		t.Fatalf("Failed to create device dir: %v", err)
 	}

 	// Mock Sources.xml
-	if err := os.WriteFile(filepath.Join(accountDir, "Sources.xml"), []byte(`
+	if err := os.WriteFile(filepath.Join(deviceDir, "Sources.xml"), []byte(`
 		<sources>
-			<source id="SRC1" type="Audio">
-				<sourcename>TUNEIN</sourcename>
+			<source id="SRC1" displayName="TUNEIN" secret="" secretType="Audio">
+				<sourceKey type="TUNEIN" account=""/>
 			</source>
 		</sources>
 	`), 0644); err != nil {
 		t.Fatalf("Failed to write Sources.xml: %v", err)
 	}

-	if err := os.WriteFile(filepath.Join(accountDir, "Recents.xml"), []byte(`<recents></recents>`), 0644); err != nil {
+	if err := os.WriteFile(filepath.Join(deviceDir, "Recents.xml"), []byte(`<recents></recents>`), 0644); err != nil {
 		t.Fatalf("Failed to write Recents.xml: %v", err)
 	}

@@ -314,7 +313,7 @@ func TestMargeDeviceInfo(t *testing.T) {
 			<contentItemType>station</contentItemType>
 		</recent>`

-	res, err := http.Post(ts.URL+"/marge/accounts/"+account+"/devices/DEV1/recents", "application/xml", strings.NewReader(payload))
+	res, err := http.Post(ts.URL+"/marge/accounts/"+account+"/devices/"+deviceID+"/recents", "application/xml", strings.NewReader(payload))
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -326,7 +325,7 @@ func TestMargeDeviceInfo(t *testing.T) {
 	}

 	// Verify file was saved
-	recentData, _ := os.ReadFile(filepath.Join(accountDir, "Recents.xml"))
+	recentData, _ := os.ReadFile(filepath.Join(deviceDir, "Recents.xml"))
 	if !strings.Contains(string(recentData), "Recent Station") {
 		t.Error("Recent was not saved to datastore")
 	}
@@ -35,6 +35,8 @@ func (s *Server) HandleProxyRequest(w http.ResponseWriter, r *http.Request) {

 	lp := proxy.NewLoggingProxy(target.String(), s.proxyRedact)
 	lp.LogBody = s.proxyLogBody
+	lp.RecordEnabled = s.recordEnabled
+	lp.SetRecorder(s.recorder)

 	proxy := httputil.NewSingleHostReverseProxy(target)
 	// Update director to set the correct host and path
@@ -1,10 +1,12 @@
 package handlers

 import (
+	"context"
 	"encoding/json"
 	"net/http"
 	"os"

+	"github.com/gesellix/bose-soundtouch/pkg/models"
 	"github.com/gesellix/bose-soundtouch/pkg/service/setup"
 	"github.com/go-chi/chi/v5"
 )
@@ -25,9 +27,53 @@ func (s *Server) HandleListDiscoveredDevices(w http.ResponseWriter, _ *http.Requ
 	}
 }

+// HandleAddManualDevice adds a device manually by IP.
+func (s *Server) HandleAddManualDevice(w http.ResponseWriter, r *http.Request) {
+	var body struct {
+		IP string `json:"ip"`
+	}
+	if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
+		http.Error(w, "Invalid request body", http.StatusBadRequest)
+		return
+	}
+
+	if body.IP == "" {
+		http.Error(w, "IP address is required", http.StatusBadRequest)
+		return
+	}
+
+	// Try to get live info
+	liveInfo, err := s.sm.GetLiveDeviceInfo(body.IP)
+	if err != nil {
+		// Even if we can't get live info, we might want to add it?
+		// But usually we need at least the serial for proper account management.
+		http.Error(w, "Failed to reach device at "+body.IP+": "+err.Error(), http.StatusBadGateway)
+		return
+	}
+
+	// Reuse handleDiscoveredDevice logic via a fake models.DiscoveredDevice
+	d := models.DiscoveredDevice{
+		Name:            liveInfo.Name,
+		Host:            body.IP,
+		ModelID:         liveInfo.Type,
+		SerialNo:        liveInfo.SerialNumber,
+		DiscoveryMethod: "manual",
+	}
+
+	s.handleDiscoveredDevice(d)
+
+	w.Header().Set("Content-Type", "application/json")
+
+	if err := json.NewEncoder(w).Encode(map[string]bool{"ok": true}); err != nil {
+		http.Error(w, "Failed to encode response", http.StatusInternalServerError)
+		return
+	}
+}
+
 // HandleTriggerDiscovery triggers a new device discovery scan.
-func (s *Server) HandleTriggerDiscovery(w http.ResponseWriter, r *http.Request) {
-	go s.DiscoverDevices(r.Context())
+func (s *Server) HandleTriggerDiscovery(w http.ResponseWriter, _ *http.Request) {
+	//nolint:contextcheck
+	go s.DiscoverDevices(context.Background())

 	w.WriteHeader(http.StatusAccepted)
 	_, _ = w.Write([]byte(`{"status": "Discovery started"}`))
@@ -305,6 +351,7 @@ func (s *Server) HandleGetProxySettings(w http.ResponseWriter, _ *http.Request)
 	if err := json.NewEncoder(w).Encode(map[string]bool{
 		"redact":   s.proxyRedact,
 		"log_body": s.proxyLogBody,
+		"record":   s.recordEnabled,
 	}); err != nil {
 		http.Error(w, "Failed to encode response", http.StatusInternalServerError)
 		return
@@ -331,6 +378,7 @@ func (s *Server) HandleUpdateProxySettings(w http.ResponseWriter, r *http.Reques
 	var settings struct {
 		Redact  bool `json:"redact"`
 		LogBody bool `json:"log_body"`
+		Record  bool `json:"record"`
 	}
 	if err := json.NewDecoder(r.Body).Decode(&settings); err != nil {
 		http.Error(w, err.Error(), http.StatusBadRequest)
@@ -339,6 +387,7 @@ func (s *Server) HandleUpdateProxySettings(w http.ResponseWriter, r *http.Reques

 	s.proxyRedact = settings.Redact
 	s.proxyLogBody = settings.LogBody
+	s.recordEnabled = settings.Record

 	w.Header().Set("Content-Type", "application/json")

@@ -388,6 +437,23 @@ func (s *Server) HandleTestHostsRedirection(w http.ResponseWriter, r *http.Reque
 	}
 }

+// HandleInitialSync fetches presets, recents and sources from the device and saves them to the datastore.
+func (s *Server) HandleInitialSync(w http.ResponseWriter, r *http.Request) {
+	deviceIP := chi.URLParam(r, "deviceIP")
+	if deviceIP == "" {
+		http.Error(w, "Missing deviceIP", http.StatusBadRequest)
+		return
+	}
+
+	if err := s.sm.SyncDeviceData(deviceIP); err != nil {
+		http.Error(w, err.Error(), http.StatusInternalServerError)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+	_, _ = w.Write([]byte(`{"ok": true}`))
+}
+
 // HandleTestConnection performs a connection check from the device to the server.
 func (s *Server) HandleTestConnection(w http.ResponseWriter, r *http.Request) {
 	deviceIP := chi.URLParam(r, "deviceIP")
@@ -0,0 +1,99 @@
+package handlers
+
+import (
+	"bufio"
+	"bytes"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+)
+
+// RecordMiddleware returns a middleware that records "self" requests and responses.
+func (s *Server) RecordMiddleware(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if s.recorder == nil || !s.recordEnabled {
+			next.ServeHTTP(w, r)
+			return
+		}
+
+		// Buffer the request body if it exists
+		var reqBody []byte
+
+		if r.Body != nil {
+			var err error
+
+			reqBody, err = io.ReadAll(r.Body)
+			if err == nil {
+				r.Body = io.NopCloser(bytes.NewBuffer(reqBody))
+			}
+		}
+
+		// wrap ResponseWriter to capture the response
+		rw := &responseWriter{
+			ResponseWriter: w,
+			body:           &bytes.Buffer{},
+		}
+
+		next.ServeHTTP(rw, r)
+
+		// Create a response object for the recorder
+		res := rw.getRecordedResponse(r)
+		if res.Body != nil {
+			defer func() { _ = res.Body.Close() }()
+		}
+
+		// Put back the original request body for recording
+		r.Body = io.NopCloser(bytes.NewBuffer(reqBody))
+
+		_ = s.recorder.Record("self", r, res)
+	})
+}
+
+type responseWriter struct {
+	http.ResponseWriter
+	statusCode int
+	body       *bytes.Buffer
+}
+
+func (rw *responseWriter) Header() http.Header {
+	return rw.ResponseWriter.Header()
+}
+
+func (rw *responseWriter) WriteHeader(code int) {
+	rw.statusCode = code
+	rw.ResponseWriter.WriteHeader(code)
+}
+
+func (rw *responseWriter) Write(b []byte) (int, error) {
+	rw.body.Write(b)
+	return rw.ResponseWriter.Write(b)
+}
+
+func (rw *responseWriter) getRecordedResponse(r *http.Request) *http.Response {
+	statusCode := rw.statusCode
+	if statusCode == 0 {
+		statusCode = http.StatusOK
+	}
+
+	return &http.Response{
+		StatusCode: statusCode,
+		Header:     rw.ResponseWriter.Header(),
+		Body:       io.NopCloser(bytes.NewBuffer(rw.body.Bytes())),
+		Request:    r,
+	}
+}
+
+func (rw *responseWriter) Flush() {
+	if f, ok := rw.ResponseWriter.(http.Flusher); ok {
+		f.Flush()
+	}
+}
+
+func (rw *responseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error) {
+	if h, ok := rw.ResponseWriter.(http.Hijacker); ok {
+		return h.Hijack()
+	}
+
+	return nil, nil, fmt.Errorf("ResponseWriter does not support Hijacker")
+}
@@ -8,32 +8,46 @@ import (
 	"github.com/gesellix/bose-soundtouch/pkg/discovery"
 	"github.com/gesellix/bose-soundtouch/pkg/models"
 	"github.com/gesellix/bose-soundtouch/pkg/service/datastore"
+	"github.com/gesellix/bose-soundtouch/pkg/service/proxy"
 	"github.com/gesellix/bose-soundtouch/pkg/service/setup"
 )

 // Server handles HTTP requests for the SoundTouch service.
 type Server struct {
-	ds           *datastore.DataStore
-	sm           *setup.Manager
-	serverURL    string
-	proxyURL     string
-	discovering  bool
-	proxyRedact  bool
-	proxyLogBody bool
+	ds            *datastore.DataStore
+	sm            *setup.Manager
+	serverURL     string
+	proxyURL      string
+	discovering   bool
+	proxyRedact   bool
+	proxyLogBody  bool
+	recordEnabled bool
+	recorder      *proxy.Recorder
 }

 // NewServer creates a new SoundTouch service server.
-func NewServer(ds *datastore.DataStore, sm *setup.Manager, serverURL string, proxyRedact, proxyLogBody bool) *Server {
+func NewServer(ds *datastore.DataStore, sm *setup.Manager, serverURL string, proxyRedact, proxyLogBody, recordEnabled bool) *Server {
 	return &Server{
-		ds:           ds,
-		sm:           sm,
-		serverURL:    serverURL,
-		proxyURL:     serverURL,
-		proxyRedact:  proxyRedact,
-		proxyLogBody: proxyLogBody,
+		ds:            ds,
+		sm:            sm,
+		serverURL:     serverURL,
+		proxyURL:      serverURL,
+		proxyRedact:   proxyRedact,
+		proxyLogBody:  proxyLogBody,
+		recordEnabled: recordEnabled,
 	}
 }

+// SetRecorder sets the recorder for the server.
+func (s *Server) SetRecorder(r *proxy.Recorder) {
+	s.recorder = r
+}
+
+// GetRecordEnabled returns whether recording is enabled.
+func (s *Server) GetRecordEnabled() bool {
+	return s.recordEnabled
+}
+
 // DiscoverDevices starts a background device discovery process.
 //
 //nolint:contextcheck
@@ -44,16 +58,18 @@ func (s *Server) DiscoverDevices(ctx context.Context) {

 	log.Println("Scanning for Bose devices...")

+	// Use background context if none provided or if it's likely a request context
 	if ctx == nil {
-		var cancel context.CancelFunc
-
-		ctx, cancel = context.WithTimeout(context.Background(), 10*time.Second)
-		defer cancel()
+		ctx = context.Background()
 	}

+	// Always wrap in a timeout to prevent hanging forever
+	discoveryCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
+	defer cancel()
+
 	svc := discovery.NewService(10 * time.Second)

-	devices, err := svc.DiscoverDevices(ctx)
+	devices, err := svc.DiscoverDevices(discoveryCtx)
 	if err != nil {
 		log.Printf("Discovery error: %v", err)
 		return
@@ -94,6 +110,7 @@ func (s *Server) handleDiscoveredDevice(d models.DiscoveredDevice) {
 		DeviceSerialNumber: d.SerialNo,
 		ProductCode:        d.ModelID,
 		FirmwareVersion:    "0.0.0", // Unknown from discovery
+		DiscoveryMethod:    d.DiscoveryMethod,
 	}

 	// If we had an IP-based entry and now have a Serial, clean up the IP-based entry
@@ -109,7 +126,8 @@ func (s *Server) handleDiscoveredDevice(d models.DiscoveredDevice) {

 func (s *Server) findExistingDeviceID(d models.DiscoveredDevice) string {
 	allDevices, _ := s.ds.ListAllDevices()
-	for _, known := range allDevices {
+	for i := range allDevices {
+		known := allDevices[i]
 		if d.SerialNo != "" && (known.DeviceID == d.SerialNo || known.DeviceSerialNumber == d.SerialNo) {
 			if known.DeviceID != "" {
 				return known.DeviceID
@@ -9,3 +9,28 @@ pre { background-color: #eee; padding: 10px; overflow-x: auto; font-size: 12px;
 .diff-container { display: flex; gap: 10px; }
 .diff-pane { flex: 1; min-width: 0; }
 .config-header { font-weight: bold; margin-bottom: 5px; display: block; }
+
+/* Tabs */
+.tabs { margin-top: 20px; }
+.tab-buttons { display: flex; border-bottom: 1px solid #ddd; margin-bottom: 20px; }
+.tab-btn { background: #f8f8f8; border: 1px solid #ddd; border-bottom: none; padding: 10px 20px; margin-right: 5px; border-top-left-radius: 4px; border-top-right-radius: 4px; }
+.tab-btn:hover { background: #eee; }
+.tab-btn.active { background: white; border-bottom: 2px solid #2196F3; font-weight: bold; }
+.tab-content { display: none; padding: 10px; }
+.tab-content.active { display: block; }
+
+.device-selection {
+    margin-bottom: 20px;
+    padding: 10px;
+    background-color: #f0f7ff;
+    border-radius: 4px;
+    border: 1px solid #d0e0f0;
+}
+.device-selection label {
+    font-weight: bold;
+    margin-right: 10px;
+}
+.device-selection select {
+    padding: 5px;
+    min-width: 250px;
+}
@@ -8,149 +8,194 @@
 </head>
 <body>
    <h1>Soundcork Management</h1>
-    <h2>Discovered Devices <span id="discovery-indicator" style="font-size: 0.5em; vertical-align: middle; display: none;">🔍 Scanning...</span></h2>
-    <div id="device-list">Loading devices...</div>

-    <div id="manual-entry" style="margin-top: 20px; border-top: 1px solid #eee; padding-top: 10px;">
-        <h3>Manual Entry</h3>
-        <input type="text" id="manual-ip" placeholder="Device IP (e.g. 192.168.1.100)">
-        <button onclick="showSummary(document.getElementById('manual-ip').value)">Check Migration</button>
-
-        <h3 style="margin-top: 20px;">Settings</h3>
-        <div style="margin-bottom: 10px;">
-            <label for="target-domain">Target Domain:</label>
-            <input type="text" id="target-domain" placeholder="http://localhost:8000" style="width: 300px;">
-            <span style="font-size: 0.8em; color: #666;">(This URL will be used for standard services)</span>
+    <div class="tabs">
+        <div class="tab-buttons">
+            <button class="tab-btn active" onclick="openTab(event, 'tab-devices')">1. Devices</button>
+            <button class="tab-btn" onclick="openTab(event, 'tab-sync')">2. Data Sync</button>
+            <button class="tab-btn" onclick="openTab(event, 'tab-migration')">3. Migration</button>
+            <button class="tab-btn" onclick="openTab(event, 'tab-settings')">4. Settings</button>
        </div>
-        <div style="margin-bottom: 10px;">
-            <label for="proxy-domain">Proxy Domain:</label>
-            <input type="text" id="proxy-domain" placeholder="http://localhost:8000" style="width: 300px;">
-            <span style="font-size: 0.8em; color: #666;">(This URL will be used to proxy upstream Bose services)</span>
-        </div>
-        <div style="margin-bottom: 10px;">
-            Proxy Logging:
-            <label><input type="checkbox" id="proxy-redact" onchange="updateProxySettings()"> Redact Sensitive Headers</label>
-            <label style="margin-left: 15px;"><input type="checkbox" id="proxy-log-body" onchange="updateProxySettings()"> Log Bodies</label>
-        </div>
-    </div>

-    <div id="status" class="status"></div>
-
-    <div id="migration-summary" class="summary-box">
-        <h3>Migration Summary for <span id="summary-ip"></span></h3>
-        <p>SSH Connection: <span id="ssh-status"></span></p>
-        <p id="original-config-status" style="display: none;">Backup: ✅ Found .original config at <code>/opt/Bose/etc/SoundTouchSdkPrivateCfg.xml.original</code> <button onclick="toggleOriginalConfig()">Show Original Config</button></p>
-        <p id="no-original-config-status" style="display: none;">Backup: ❌ Not found <button id="backup-config-btn">Backup Config Now</button></p>
-        <p>Remote Services Enabled: <span id="remote-services-status"></span> <span id="remote-services-found" style="font-size: 0.8em; color: #666;"></span></p>
-        <p>Local Root CA Trusted: <span id="ca-trust-status"></span> <button id="trust-ca-btn" style="display: none; background-color: #607D8B; color: white; border: none; padding: 2px 8px; font-size: 0.8em; margin-left: 10px;">Trust CA Now</button></p>
-
-        <div id="connection-test" style="margin: 15px 0; padding: 10px; border: 1px solid #ddd; background-color: #eefbff;">
-            <strong>HTTPS Connection Test:</strong><br>
-            <span style="font-size: 0.85em; color: #555;">Verify the device can reach the server over HTTPS.</span>
-            <div style="margin-top: 10px;">
-                URL: <code id="test-url"></code>
+        <!-- Tab 1: Devices -->
+        <div id="tab-devices" class="tab-content active">
+            <h2>Known Devices <span id="discovery-indicator" style="font-size: 0.5em; vertical-align: middle; display: none;">🔍 Scanning...</span></h2>
+            <div id="device-list">Loading devices...</div>
+            <div style="margin-top: 20px;">
+                <button onclick="triggerDiscovery()">Scan Again</button>
+                <input type="text" id="add-manual-ip" placeholder="Manual IP (e.g. 192.168.1.100)" style="margin-left: 20px; padding: 4px;">
+                <button onclick="addManualDevice()">Add Device</button>
            </div>
-            <div style="margin-top: 10px;">
-                <button id="test-connection-explicit-btn" style="background-color: #607D8B; color: white; border: none; padding: 5px 10px; font-size: 0.9em;">Test with Explicit CA.crt</button>
-                <button id="test-connection-trusted-btn" style="background-color: #607D8B; color: white; border: none; padding: 5px 10px; font-size: 0.9em;">Test with Shared Trust Store</button>
-            </div>
-            <div id="test-result" style="margin-top: 10px; display: none; padding: 10px; border-radius: 4px; font-family: monospace; white-space: pre-wrap; font-size: 0.85em; max-height: 200px; overflow-y: auto;"></div>
        </div>

-        <div id="hosts-redirection-test" style="margin: 15px 0; padding: 10px; border: 1px solid #ddd; background-color: #fff4e6; display: none;">
-            <strong>Preliminary /etc/hosts Test:</strong><br>
-            <span style="font-size: 0.85em; color: #555;">Verify the device's /etc/hosts mechanism before full migration.</span>
-            <div style="margin-top: 10px;">
-                Domain: <code>custom-test-api.bose.fake</code>
+        <!-- Tab 2: Data Sync -->
+        <div id="tab-sync" class="tab-content">
+            <h2>Initial Data Sync</h2>
+            <p>Before migrating, fetch your presets, recents, and configured sources from the device to ensure they are available locally.</p>
+            <div class="device-selection">
+                <label for="sync-device-list">Device:</label>
+                <select id="sync-device-list">
+                    <option value="">-- Select a device --</option>
+                </select>
+                <button id="sync-now-btn">Start Sync</button>
            </div>
-            <div style="margin-top: 10px;">
-                <button id="test-hosts-btn" style="background-color: #FF9800; color: white; border: none; padding: 5px 10px; font-size: 0.9em;">Test Hosts Redirection</button>
+            <div id="sync-status" class="status"></div>
+            <div id="sync-results" style="margin-top: 20px; display: none;">
+                <h3>Sync Results</h3>
+                <div id="sync-log" style="font-family: monospace; background: #f4f4f4; padding: 10px; border-radius: 4px; max-height: 300px; overflow-y: auto;"></div>
            </div>
-            <div id="hosts-test-result" style="margin-top: 10px; display: none; padding: 10px; border-radius: 4px; font-family: monospace; white-space: pre-wrap; font-size: 0.85em; max-height: 200px; overflow-y: auto;"></div>
        </div>

-        <div style="margin: 15px 0; padding: 10px; border: 1px solid #ddd; background-color: #f9f9f9;">
-            <label for="migration-method"><strong>Migration Method:</strong></label>
-            <select id="migration-method" onchange="toggleMigrationMethod()">
-                <option value="xml">XML Configuration (Recommended - redirects specific services)</option>
-                <option value="hosts">/etc/hosts + Root CA (Advanced - global redirection)</option>
-            </select>
-        </div>
-
-        <div id="original-config-pane" style="display: none; margin-bottom: 20px;">
-            <span class="config-header">Original Config (Backup)</span>
-            <pre id="original-config-content"></pre>
-        </div>
-
-        <div id="service-options" style="margin-bottom: 20px; display: none;">
-            <h4>Service Implementations</h4>
-            <table>
-                <tr><th>Service</th><th>Original URL</th><th>Implementation</th></tr>
-                <tr>
-                    <td>Marge (Streaming)</td>
-                    <td id="orig-marge">loading...</td>
-                    <td>
-                        <select id="opt-marge" onchange="refreshSummary()">
-                            <option value="soundcork">Soundcork (Go/Python)</option>
-                            <option value="original">Original (Proxy via soundcork-go)</option>
-                        </select>
-                    </td>
-                </tr>
-                <tr>
-                    <td>Stats</td>
-                    <td id="orig-stats">loading...</td>
-                    <td>
-                        <select id="opt-stats" onchange="refreshSummary()">
-                            <option value="soundcork">Soundcork (Go/Python)</option>
-                            <option value="original">Original (Proxy via soundcork-go)</option>
-                        </select>
-                    </td>
-                </tr>
-                <tr>
-                    <td>Software Update</td>
-                    <td id="orig-sw_update">loading...</td>
-                    <td>
-                        <select id="opt-sw_update" onchange="refreshSummary()">
-                            <option value="soundcork">Soundcork (Go/Python)</option>
-                            <option value="original">Original (Proxy via soundcork-go)</option>
-                        </select>
-                    </td>
-                </tr>
-                <tr>
-                    <td>BMX (Registry)</td>
-                    <td id="orig-bmx">loading...</td>
-                    <td>
-                        <select id="opt-bmx" onchange="refreshSummary()">
-                            <option value="soundcork">Soundcork (Go/Python)</option>
-                            <option value="original">Original (Proxy via soundcork-go)</option>
-                        </select>
-                    </td>
-                </tr>
-            </table>
-        </div>
-
-        <div class="diff-container">
-            <div id="xml-diff-pane" class="diff-pane">
-                <span class="config-header">Current Config (on Speaker)</span>
-                <pre id="current-config"></pre>
+        <!-- Tab 3: Migration -->
+        <div id="tab-migration" class="tab-content">
+            <h2>Device Migration</h2>
+            <div class="device-selection">
+                <label for="migration-device-list">Device:</label>
+                <select id="migration-device-list" onchange="showSummary(this.value)">
+                    <option value="">-- Select a device --</option>
+                </select>
            </div>
-            <div id="planned-xml-pane" class="diff-pane">
-                <span class="config-header">Planned Config (Soundcork)</span>
-                <pre id="planned-config"></pre>
-            </div>
-            <div id="planned-hosts-pane" class="diff-pane" style="display: none;">
-                <span class="config-header">Planned /etc/hosts Entries</span>
-                <pre id="planned-hosts"></pre>
-                <div style="margin-top: 10px; font-size: 0.9em; color: #666;">
-                    <strong>Note:</strong> This method also injects the local Root CA into <code>/etc/pki/tls/certs/ca-bundle.crt</code> to enable secure HTTPS communication.
+
+            <div id="status" class="status"></div>
+
+            <div id="migration-summary" class="summary-box" style="display: none;">
+                <h3>Migration Summary for <span id="summary-ip"></span></h3>
+                <p>SSH Connection: <span id="ssh-status"></span></p>
+                <p id="original-config-status" style="display: none;">Backup: ✅ Found .original config at <code>/opt/Bose/etc/SoundTouchSdkPrivateCfg.xml.original</code> <button onclick="toggleOriginalConfig()">Show Original Config</button></p>
+                <p id="no-original-config-status" style="display: none;">Backup: ❌ Not found <button id="backup-config-btn">Backup Config Now</button></p>
+                <p>Remote Services Enabled: <span id="remote-services-status"></span> <span id="remote-services-found" style="font-size: 0.8em; color: #666;"></span></p>
+                <p>Local Root CA Trusted: <span id="ca-trust-status"></span> <button id="trust-ca-btn" style="display: none; background-color: #607D8B; color: white; border: none; padding: 2px 8px; font-size: 0.8em; margin-left: 10px;">Trust CA Now</button></p>
+
+                <div id="connection-test" style="margin: 15px 0; padding: 10px; border: 1px solid #ddd; background-color: #eefbff;">
+                    <strong>HTTPS Connection Test:</strong><br>
+                    <span style="font-size: 0.85em; color: #555;">Verify the device can reach the server over HTTPS.</span>
+                    <div style="margin-top: 10px;">
+                        URL: <code id="test-url"></code>
+                    </div>
+                    <div style="margin-top: 10px;">
+                        <button id="test-connection-explicit-btn" style="background-color: #607D8B; color: white; border: none; padding: 5px 10px; font-size: 0.9em;">Test with Explicit CA.crt</button>
+                        <button id="test-connection-trusted-btn" style="background-color: #607D8B; color: white; border: none; padding: 5px 10px; font-size: 0.9em;">Test with Shared Trust Store</button>
+                    </div>
+                    <div id="test-result" style="margin-top: 10px; display: none; padding: 10px; border-radius: 4px; font-family: monospace; white-space: pre-wrap; font-size: 0.85em; max-height: 200px; overflow-y: auto;"></div>
+                </div>
+
+                <div id="hosts-redirection-test" style="margin: 15px 0; padding: 10px; border: 1px solid #ddd; background-color: #fff4e6; display: none;">
+                    <strong>Preliminary /etc/hosts Test:</strong><br>
+                    <span style="font-size: 0.85em; color: #555;">Verify the device's /etc/hosts mechanism before full migration.</span>
+                    <div style="margin-top: 10px;">
+                        Domain: <code>custom-test-api.bose.fake</code>
+                    </div>
+                    <div style="margin-top: 10px;">
+                        <button id="test-hosts-btn" style="background-color: #FF9800; color: white; border: none; padding: 5px 10px; font-size: 0.9em;">Test Hosts Redirection</button>
+                    </div>
+                    <div id="hosts-test-result" style="margin-top: 10px; display: none; padding: 10px; border-radius: 4px; font-family: monospace; white-space: pre-wrap; font-size: 0.85em; max-height: 200px; overflow-y: auto;"></div>
+                </div>
+
+                <div style="margin: 15px 0; padding: 10px; border: 1px solid #ddd; background-color: #f9f9f9;">
+                    <label for="migration-method"><strong>Migration Method:</strong></label>
+                    <select id="migration-method" onchange="toggleMigrationMethod()">
+                        <option value="xml">XML Configuration (Recommended - redirects specific services)</option>
+                        <option value="hosts">/etc/hosts + Root CA (Advanced - global redirection)</option>
+                    </select>
+                </div>
+
+                <div id="original-config-pane" style="display: none; margin-bottom: 20px;">
+                    <span class="config-header">Original Config (Backup)</span>
+                    <pre id="original-config-content"></pre>
+                </div>
+
+                <div id="service-options" style="margin-bottom: 20px; display: none;">
+                    <h4>Service Implementations</h4>
+                    <table>
+                        <tr><th>Service</th><th>Original URL</th><th>Implementation</th></tr>
+                        <tr>
+                            <td>Marge (Streaming)</td>
+                            <td id="orig-marge">loading...</td>
+                            <td>
+                                <select id="opt-marge" onchange="refreshSummary()">
+                                    <option value="soundcork">Soundcork (Go/Python)</option>
+                                    <option value="original">Original (Proxy via soundcork-go)</option>
+                                </select>
+                            </td>
+                        </tr>
+                        <tr>
+                            <td>Stats</td>
+                            <td id="orig-stats">loading...</td>
+                            <td>
+                                <select id="opt-stats" onchange="refreshSummary()">
+                                    <option value="soundcork">Soundcork (Go/Python)</option>
+                                    <option value="original">Original (Proxy via soundcork-go)</option>
+                                </select>
+                            </td>
+                        </tr>
+                        <tr>
+                            <td>Software Update</td>
+                            <td id="orig-sw_update">loading...</td>
+                            <td>
+                                <select id="opt-sw_update" onchange="refreshSummary()">
+                                    <option value="soundcork">Soundcork (Go/Python)</option>
+                                    <option value="original">Original (Proxy via soundcork-go)</option>
+                                </select>
+                            </td>
+                        </tr>
+                        <tr>
+                            <td>BMX (Registry)</td>
+                            <td id="orig-bmx">loading...</td>
+                            <td>
+                                <select id="opt-bmx" onchange="refreshSummary()">
+                                    <option value="soundcork">Soundcork (Go/Python)</option>
+                                    <option value="original">Original (Proxy via soundcork-go)</option>
+                                </select>
+                            </td>
+                        </tr>
+                    </table>
+                </div>
+
+                <div class="diff-container">
+                    <div id="xml-diff-pane" class="diff-pane">
+                        <span class="config-header">Current Config (on Speaker)</span>
+                        <pre id="current-config"></pre>
+                    </div>
+                    <div id="planned-xml-pane" class="diff-pane">
+                        <span class="config-header">Planned Config (Soundcork)</span>
+                        <pre id="planned-config"></pre>
+                    </div>
+                    <div id="planned-hosts-pane" class="diff-pane" style="display: none;">
+                        <span class="config-header">Planned /etc/hosts Entries</span>
+                        <pre id="planned-hosts"></pre>
+                        <div style="margin-top: 10px; font-size: 0.9em; color: #666;">
+                            <strong>Note:</strong> This method also injects the local Root CA into <code>/etc/pki/tls/certs/ca-bundle.crt</code> to enable secure HTTPS communication.
+                        </div>
+                    </div>
+                </div>
+                <div style="margin-top: 15px;">
+                    <button id="confirm-migrate-btn" style="background-color: #4CAF50; color: white; border: none; padding: 10px 20px;">Confirm Migration & Reboot</button>
+                    <button id="ensure-remote-btn" style="background-color: #2196F3; color: white; border: none; padding: 10px 20px;">Enable Persistent Remote Services</button>
+                    <button id="remove-remote-btn" style="background-color: #f44336; color: white; border: none; padding: 10px 20px;">Remove Persistent Remote Services</button>
+                    <button onclick="document.getElementById('migration-summary').style.display='none'" style="padding: 10px 20px;">Cancel</button>
                </div>
            </div>
        </div>
-        <div style="margin-top: 15px;">
-            <button id="confirm-migrate-btn" style="background-color: #4CAF50; color: white; border: none; padding: 10px 20px;">Confirm Migration & Reboot</button>
-            <button id="ensure-remote-btn" style="background-color: #2196F3; color: white; border: none; padding: 10px 20px;">Enable Persistent Remote Services</button>
-            <button id="remove-remote-btn" style="background-color: #f44336; color: white; border: none; padding: 10px 20px;">Remove Persistent Remote Services</button>
-            <button onclick="document.getElementById('migration-summary').style.display='none'" style="padding: 10px 20px;">Cancel</button>
+
+        <!-- Tab 4: Settings -->
+        <div id="tab-settings" class="tab-content">
+            <h2>System Settings</h2>
+            <div style="margin-bottom: 20px;">
+                <label for="target-domain">Target Domain:</label>
+                <input type="text" id="target-domain" placeholder="http://localhost:8000" style="width: 300px;">
+                <span style="font-size: 0.8em; color: #666;">(This URL will be used for standard services)</span>
+            </div>
+            <div style="margin-bottom: 20px;">
+                <label for="proxy-domain">Proxy Domain:</label>
+                <input type="text" id="proxy-domain" placeholder="http://localhost:8000" style="width: 300px;">
+                <span style="font-size: 0.8em; color: #666;">(This URL will be used to proxy upstream Bose services)</span>
+            </div>
+            <div style="margin-bottom: 20px;">
+                Proxy Logging:
+                <label><input type="checkbox" id="proxy-redact" onchange="updateProxySettings()"> Redact Sensitive Headers</label>
+                <label style="margin-left: 15px;"><input type="checkbox" id="proxy-log-body" onchange="updateProxySettings()"> Log Bodies</label>
+                <label style="margin-left: 15px;"><input type="checkbox" id="proxy-record" onchange="updateProxySettings()"> Record Interactions</label>
+            </div>
        </div>
    </div>

@@ -20,6 +20,7 @@ async function fetchProxySettings() {
        const settings = await response.json();
        document.getElementById('proxy-redact').checked = settings.redact;
        document.getElementById('proxy-log-body').checked = settings.log_body;
+        document.getElementById('proxy-record').checked = settings.record;
    } catch (error) {
        console.error('Failed to fetch proxy settings', error);
    }
@@ -28,7 +29,8 @@ async function fetchProxySettings() {
 async function updateProxySettings() {
    const settings = {
        redact: document.getElementById('proxy-redact').checked,
-        log_body: document.getElementById('proxy-log-body').checked
+        log_body: document.getElementById('proxy-log-body').checked,
+        record: document.getElementById('proxy-record').checked
    };
    try {
        await fetch('/setup/proxy-settings', {
@@ -46,12 +48,22 @@ async function fetchDevices() {
        const response = await fetch('/setup/devices');
        const devices = await response.json();
        const container = document.getElementById('device-list');
+        const syncSelector = document.getElementById('sync-device-list');
+        const migrationSelector = document.getElementById('migration-device-list');

        if (devices.length === 0) {
-            container.innerHTML = 'No devices found.';
+            container.innerHTML = 'No devices known yet.';
        } else {
-            let html = '<table><tr><th>Name</th><th>IP Address</th><th>Model</th><th>Serial Number</th><th>Firmware</th><th>Action</th></tr>';
+            let html = '<table><tr><th>Name</th><th>IP Address</th><th>Model</th><th>Serial Number</th><th>Firmware</th><th>Method</th><th>Action</th></tr>';
+
+            // Clear and repopulate selectors
+            const currentSyncVal = syncSelector.value;
+            const currentMigrationVal = migrationSelector.value;
+            syncSelector.innerHTML = '<option value="">-- Select a device --</option>';
+            migrationSelector.innerHTML = '<option value="">-- Select a device --</option>';
+
            devices.forEach(d => {
+                const methodLabel = d.discovery_method === 'manual' ? '👤 Manual' : '🔍 Auto';
                html += `
                    <tr id="device-row-${d.ip_address.replace(/\./g, '-')}">
                        <td class="col-name">${d.name}</td>
@@ -59,13 +71,30 @@ async function fetchDevices() {
                        <td class="col-model">${d.product_code}</td>
                        <td class="col-serial">${d.device_serial_number}</td>
                        <td class="col-firmware">${d.firmware_version || '0.0.0'}</td>
-                        <td><button onclick="showSummary('${d.ip_address}')">Prepare Migration</button></td>
+                        <td class="col-method">${methodLabel}</td>
+                        <td>
+                            <button onclick="prepareSync('${d.ip_address}')">Sync Data</button>
+                            <button onclick="prepareMigration('${d.ip_address}')">Migrate</button>
+                        </td>
                    </tr>
                `;
+
+                const optSync = document.createElement('option');
+                optSync.value = d.ip_address;
+                optSync.textContent = `${d.name} (${d.ip_address})`;
+                syncSelector.appendChild(optSync);
+
+                const optMigrate = document.createElement('option');
+                optMigrate.value = d.ip_address;
+                optMigrate.textContent = `${d.name} (${d.ip_address})`;
+                migrationSelector.appendChild(optMigrate);
            });
            html += '</table>';
            container.innerHTML = html;

+            if (currentSyncVal) syncSelector.value = currentSyncVal;
+            if (currentMigrationVal) migrationSelector.value = currentMigrationVal;
+
            // Asynchronously fetch live info for each device
            devices.forEach(d => updateDeviceInfo(d.ip_address));
        }
@@ -74,15 +103,125 @@ async function fetchDevices() {
    }
 }

+function prepareSync(ip) {
+    document.getElementById('sync-device-list').value = ip;
+    openTab(null, 'tab-sync');
+}
+
+function prepareMigration(ip) {
+    document.getElementById('migration-device-list').value = ip;
+    openTab(null, 'tab-migration');
+    showSummary(ip);
+}
+
+function openTab(evt, tabId) {
+    const tabcontents = document.getElementsByClassName("tab-content");
+    for (let i = 0; i < tabcontents.length; i++) {
+        tabcontents[i].className = tabcontents[i].className.replace(" active", "");
+    }
+
+    const tablinks = document.getElementsByClassName("tab-btn");
+    for (let i = 0; i < tablinks.length; i++) {
+        tablinks[i].className = tablinks[i].className.replace(" active", "");
+    }
+
+    const content = document.getElementById(tabId);
+    if (content) {
+        content.className += " active";
+    }
+
+    if (evt) {
+        evt.currentTarget.className += " active";
+    } else {
+        // Find the button that corresponds to the tabId and activate it
+        for (let i = 0; i < tablinks.length; i++) {
+            const onclick = tablinks[i].getAttribute('onclick');
+            if (onclick && onclick.includes(tabId)) {
+                tablinks[i].className += " active";
+                break;
+            }
+        }
+    }
+}
+
+async function startSync() {
+    const ip = document.getElementById('sync-device-list').value;
+    if (!ip) {
+        alert('Please select a device first');
+        return;
+    }
+
+    const status = document.getElementById('sync-status');
+    const results = document.getElementById('sync-results');
+    const log = document.getElementById('sync-log');
+
+    status.style.display = 'block';
+    status.style.backgroundColor = '#eef';
+    status.textContent = 'Syncing data from ' + ip + '...';
+    results.style.display = 'none';
+    log.innerHTML = '';
+
+    try {
+        const response = await fetch('/setup/sync/' + ip, { method: 'POST' });
+        if (response.ok) {
+            status.style.backgroundColor = '#dfd';
+            status.textContent = '✅ Sync completed successfully!';
+            results.style.display = 'block';
+            log.innerHTML = 'Data fetched and saved to local datastore.\nPresets: OK\nRecents: OK\nSources: OK';
+        } else {
+            const err = await response.text();
+            throw new Error(err);
+        }
+    } catch (error) {
+        status.style.backgroundColor = '#fdd';
+        status.textContent = '❌ Sync failed: ' + error.message;
+    }
+}
+
+document.addEventListener('DOMContentLoaded', () => {
+    fetchSettings();
+    fetchDevices();
+    triggerDiscovery();
+
+    document.getElementById('sync-now-btn').onclick = startSync;
+});
+
+
+async function addManualDevice() {
+    const ip = document.getElementById('add-manual-ip').value.trim();
+    if (!ip) {
+        alert('Please enter an IP address');
+        return;
+    }
+
+    try {
+        const response = await fetch('/setup/devices', {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/json' },
+            body: JSON.stringify({ ip: ip })
+        });
+
+        if (response.ok) {
+            document.getElementById('add-manual-ip').value = '';
+            fetchDevices();
+        } else {
+            const err = await response.text();
+            alert('Failed to add device: ' + err);
+        }
+    } catch (error) {
+        alert('Error adding device: ' + error.message);
+    }
+}
+
 async function triggerDiscovery() {
    const indicator = document.getElementById('discovery-indicator');
-    indicator.style.display = 'inline';
+    if (indicator) indicator.style.display = 'inline';
    try {
        await fetch('/setup/discover', { method: 'POST' });
        pollDiscoveryStatus();
    } catch (error) {
        console.error('Failed to trigger discovery', error);
-        indicator.style.display = 'none';
+        if (indicator) indicator.style.display = 'none';
    }
 }

@@ -94,12 +233,12 @@ async function pollDiscoveryStatus() {
        if (data.discovering) {
            setTimeout(pollDiscoveryStatus, 2000);
        } else {
-            indicator.style.display = 'none';
+            if (indicator) indicator.style.display = 'none';
            fetchDevices();
        }
    } catch (error) {
        console.error('Failed to check discovery status', error);
-        indicator.style.display = 'none';
+        if (indicator) indicator.style.display = 'none';
    }
 }

@@ -112,10 +251,17 @@ async function updateDeviceInfo(ip) {
        const rowId = 'device-row-' + ip.replace(/\./g, '-');
        const row = document.getElementById(rowId);
        if (row) {
-            if (info.name) row.querySelector('.col-name').innerText = info.name;
-            if (info.type) row.querySelector('.col-model').innerText = info.type;
-            if (info.serialNumber) row.querySelector('.col-serial').innerText = info.serialNumber;
-            if (info.softwareVersion) row.querySelector('.col-firmware').innerText = info.softwareVersion;
+            const nameEl = row.querySelector('.col-name');
+            if (nameEl && info.name) nameEl.innerText = info.name;
+
+            const modelEl = row.querySelector('.col-model');
+            if (modelEl && info.type) modelEl.innerText = info.type;
+
+            const serialEl = row.querySelector('.col-serial');
+            if (serialEl && info.serialNumber) serialEl.innerText = info.serialNumber;
+
+            const firmwareEl = row.querySelector('.col-firmware');
+            if (firmwareEl && info.softwareVersion) firmwareEl.innerText = info.softwareVersion;
        }
    } catch (error) {
        console.warn('Failed to fetch live info for ' + ip, error);
@@ -124,7 +270,7 @@ async function updateDeviceInfo(ip) {

 async function showSummary(ip) {
    if (!ip) {
-        alert('Please enter a valid IP address.');
+        document.getElementById('migration-summary').style.display = 'none';
        return;
    }
    const targetUrl = document.getElementById('target-domain').value;
@@ -162,10 +308,17 @@ async function showSummary(ip) {
        const rowId = 'device-row-' + ip.replace(/\./g, '-');
        const row = document.getElementById(rowId);
        if (row) {
-            if (summary.device_name) row.querySelector('.col-name').innerText = summary.device_name;
-            if (summary.device_model) row.querySelector('.col-model').innerText = summary.device_model;
-            if (summary.device_serial) row.querySelector('.col-serial').innerText = summary.device_serial;
-            if (summary.firmware_version) row.querySelector('.col-firmware').innerText = summary.firmware_version;
+            const nameEl = row.querySelector('.col-name');
+            if (nameEl && summary.device_name) nameEl.innerText = summary.device_name;
+
+            const modelEl = row.querySelector('.col-model');
+            if (modelEl && summary.device_model) modelEl.innerText = summary.device_model;
+
+            const serialEl = row.querySelector('.col-serial');
+            if (serialEl && summary.device_serial) serialEl.innerText = summary.device_serial;
+
+            const firmwareEl = row.querySelector('.col-firmware');
+            if (firmwareEl && summary.firmware_version) firmwareEl.innerText = summary.firmware_version;
        }

        document.getElementById('ssh-status').innerText = summary.ssh_success ? '✅ Success' : '❌ Failed';
@@ -112,13 +112,13 @@ func GetConfiguredSourceXML(cs models.ConfiguredSource) string {
 }

 // PresetsToXML converts account presets to XML format for Marge responses.
-func PresetsToXML(ds *datastore.DataStore, account string) ([]byte, error) {
-	presets, err := ds.GetPresets(account)
+func PresetsToXML(ds *datastore.DataStore, account, device string) ([]byte, error) {
+	presets, err := ds.GetPresets(account, device)
 	if err != nil {
 		return nil, err
 	}

-	sources, err := ds.GetConfiguredSources(account)
+	sources, err := ds.GetConfiguredSources(account, device)
 	if err != nil {
 		return nil, err
 	}
@@ -135,7 +135,8 @@ func PresetsToXML(ds *datastore.DataStore, account string) ([]byte, error) {
 		res += fmt.Sprintf(`<name>%s</name>`, p.Name)

 		// Content Item Source
-		for _, s := range sources {
+		for j := range sources {
+			s := sources[j]
 			if s.ID == p.SourceID || (s.SourceKeyType == p.Source && s.SourceKeyAccount == p.SourceAccount) {
 				res += GetConfiguredSourceXML(s)
 				break
@@ -152,13 +153,13 @@ func PresetsToXML(ds *datastore.DataStore, account string) ([]byte, error) {
 }

 // RecentsToXML converts account recent items to XML format for Marge responses.
-func RecentsToXML(ds *datastore.DataStore, account string) ([]byte, error) {
-	recents, err := ds.GetRecents(account)
+func RecentsToXML(ds *datastore.DataStore, account, device string) ([]byte, error) {
+	recents, err := ds.GetRecents(account, device)
 	if err != nil {
 		return nil, err
 	}

-	sources, err := ds.GetConfiguredSources(account)
+	sources, err := ds.GetConfiguredSources(account, device)
 	if err != nil {
 		return nil, err
 	}
@@ -181,7 +182,8 @@ func RecentsToXML(ds *datastore.DataStore, account string) ([]byte, error) {
 		res += fmt.Sprintf(`<name>%s</name>`, r.Name)

 		// Content Item Source
-		for _, s := range sources {
+		for j := range sources {
+			s := sources[j]
 			if s.ID == r.SourceID || (s.SourceKeyType == r.Source && s.SourceKeyAccount == r.SourceAccount) {
 				res += GetConfiguredSourceXML(s)
 				break
@@ -240,12 +242,12 @@ func AccountFullToXML(ds *datastore.DataStore, account string) ([]byte, error) {
 		res += fmt.Sprintf(`<ipaddress>%s</ipaddress>`, info.IPAddress)
 		res += fmt.Sprintf(`<name>%s</name>`, info.Name)

-		presets, _ := PresetsToXML(ds, account)
+		presets, _ := PresetsToXML(ds, account, deviceID)
 		if len(presets) > len(xml.Header) {
 			res += string(presets[len(xml.Header):]) // strip header
 		}

-		recents, _ := RecentsToXML(ds, account)
+		recents, _ := RecentsToXML(ds, account, deviceID)
 		if len(recents) > len(xml.Header) {
 			res += string(recents[len(xml.Header):]) // strip header
 		}
@@ -257,11 +259,11 @@ func AccountFullToXML(ds *datastore.DataStore, account string) ([]byte, error) {
 	res += ProviderSettingsToXML(account)

 	if lastDeviceID != "" {
-		sources, _ := ds.GetConfiguredSources(account)
+		sources, _ := ds.GetConfiguredSources(account, lastDeviceID)

 		res += `<sources>`
-		for _, s := range sources {
-			res += GetConfiguredSourceXML(s)
+		for j := range sources {
+			res += GetConfiguredSourceXML(sources[j])
 		}

 		res += `</sources>`
@@ -273,13 +275,13 @@ func AccountFullToXML(ds *datastore.DataStore, account string) ([]byte, error) {
 }

 // UpdatePreset updates or creates a preset for the specified account and device.
-func UpdatePreset(ds *datastore.DataStore, account, _ string, presetNumber int, sourceXML []byte) ([]byte, error) {
-	sources, err := ds.GetConfiguredSources(account)
+func UpdatePreset(ds *datastore.DataStore, account, device string, presetNumber int, sourceXML []byte) ([]byte, error) {
+	sources, err := ds.GetConfiguredSources(account, device)
 	if err != nil {
 		return nil, err
 	}

-	presets, err := ds.GetPresets(account)
+	presets, err := ds.GetPresets(account, device)
 	if err != nil {
 		return nil, err
 	}
@@ -297,9 +299,9 @@ func UpdatePreset(ds *datastore.DataStore, account, _ string, presetNumber int,

 	var matchingSrc *models.ConfiguredSource

-	for _, s := range sources {
-		if s.ID == newPresetElem.SourceID {
-			matchingSrc = &s
+	for i := range sources {
+		if sources[i].ID == newPresetElem.SourceID {
+			matchingSrc = &sources[i]
 			break
 		}
 	}
@@ -331,7 +333,7 @@ func UpdatePreset(ds *datastore.DataStore, account, _ string, presetNumber int,

 	presets[presetNumber-1] = presetObj

-	if err := ds.SavePresets(account, presets); err != nil {
+	if err := ds.SavePresets(account, device, presets); err != nil {
 		return nil, err
 	}

@@ -351,12 +353,12 @@ func UpdatePreset(ds *datastore.DataStore, account, _ string, presetNumber int,

 // AddRecent adds or updates a recent item for the specified account and device.
 func AddRecent(ds *datastore.DataStore, account, device string, sourceXML []byte) ([]byte, error) {
-	sources, err := ds.GetConfiguredSources(account)
+	sources, err := ds.GetConfiguredSources(account, device)
 	if err != nil {
 		return nil, err
 	}

-	recents, err := ds.GetRecents(account)
+	recents, err := ds.GetRecents(account, device)
 	if err != nil {
 		return nil, err
 	}
@@ -407,7 +409,7 @@ func AddRecent(ds *datastore.DataStore, account, device string, sourceXML []byte
 		}
 	}

-	if err := ds.SaveRecents(account, recents); err != nil {
+	if err := ds.SaveRecents(account, device, recents); err != nil {
 		return nil, err
 	}

@@ -415,9 +417,9 @@ func AddRecent(ds *datastore.DataStore, account, device string, sourceXML []byte
 }

 func findMatchingSource(sources []models.ConfiguredSource, sourceID string) *models.ConfiguredSource {
-	for _, s := range sources {
-		if s.ID == sourceID {
-			return &s
+	for i := range sources {
+		if sources[i].ID == sourceID {
+			return &sources[i]
 		}
 	}

@@ -30,8 +30,8 @@ func TestMargeXML(t *testing.T) {
 	_ = ds.SaveDeviceInfo(account, device, info)

 	// Save empty presets/recents to avoid index out of range when stripping header
-	_ = ds.SavePresets(account, []models.ServicePreset{})
-	_ = ds.SaveRecents(account, []models.ServiceRecent{})
+	_ = ds.SavePresets(account, device, []models.ServicePreset{})
+	_ = ds.SaveRecents(account, device, []models.ServiceRecent{})

 	// Test SourceProvidersToXML
 	xmlData, err := SourceProvidersToXML()
@@ -78,17 +78,20 @@ func TestAddRecent_TimestampPreservation(t *testing.T) {

 	// 1. Setup configured sources
 	// We need a Sources.xml file in the account directory
-	sourcesPath := ds.AccountDir(account)
-	_ = os.MkdirAll(sourcesPath, 0755)
-	_ = ds.SaveConfiguredSources(account, []models.ConfiguredSource{
-		{
-			ID:               "101",
-			DisplayName:      "Test Source",
-			SourceKeyType:    "TUNEIN",
-			SourceKeyAccount: "test-user",
-		},
-	})
-	_ = ds.SaveRecents(account, []models.ServiceRecent{})
+	deviceDir := ds.AccountDeviceDir(account, device)
+	_ = os.MkdirAll(deviceDir, 0755)
+	src := models.ConfiguredSource{
+		ID:          "101",
+		DisplayName: "Test Source",
+		SecretType:  "Audio",
+	}
+	src.SourceKey.Type = "TUNEIN"
+	src.SourceKey.Account = "test-user"
+	src.SourceKeyType = "TUNEIN"
+	src.SourceKeyAccount = "test-user"
+
+	_ = ds.SaveConfiguredSources(account, device, []models.ConfiguredSource{src})
+	_ = ds.SaveRecents(account, device, []models.ServiceRecent{})

 	// 2. Add an initial recent
 	sourceXML := []byte(`
@@ -104,7 +107,7 @@ func TestAddRecent_TimestampPreservation(t *testing.T) {
 		t.Fatalf("AddRecent failed: %v", err)
 	}

-	recents, _ := ds.GetRecents(account)
+	recents, _ := ds.GetRecents(account, device)
 	if len(recents) != 1 {
 		t.Fatalf("Expected 1 recent, got %d", len(recents))
 	}
@@ -126,7 +129,7 @@ func TestAddRecent_TimestampPreservation(t *testing.T) {
 		t.Errorf("Expected preserved DateStr in createdOn, got XML: %s", string(respXML))
 	}

-	recents, _ = ds.GetRecents(account)
+	recents, _ = ds.GetRecents(account, device)
 	if len(recents) != 1 {
 		t.Errorf("Expected still 1 recent, got %d", len(recents))
 	}
@@ -0,0 +1,71 @@
+package proxy
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"regexp"
+)
+
+// PathPattern defines a regex and its replacement for sanitizing URL paths.
+type PathPattern struct {
+	Name        string `json:"name"`
+	Regexp      string `json:"regexp"`
+	Replacement string `json:"replacement"`
+	compiled    *regexp.Regexp
+}
+
+// PathPatterns is a collection of PathPattern.
+type PathPatterns []PathPattern
+
+// LoadPatterns loads path patterns from a JSON file.
+func LoadPatterns(path string) (PathPatterns, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return PathPatterns{}, nil
+		}
+
+		return nil, err
+	}
+
+	var patterns PathPatterns
+	if err := json.Unmarshal(data, &patterns); err != nil {
+		return nil, err
+	}
+
+	for i := range patterns {
+		re, err := regexp.Compile(patterns[i].Regexp)
+		if err != nil {
+			return nil, fmt.Errorf("invalid regex in pattern %s: %w", patterns[i].Name, err)
+		}
+
+		patterns[i].compiled = re
+	}
+
+	return patterns, nil
+}
+
+// Sanitize sanitizes a segment using the configured patterns.
+func (pp PathPatterns) Sanitize(segment string) (string, string) {
+	for _, p := range pp {
+		if p.compiled != nil && p.compiled.MatchString(segment) {
+			return p.Replacement, p.Replacement
+		}
+	}
+
+	return segment, ""
+}
+
+// DefaultPatterns returns the default set of path patterns.
+func DefaultPatterns() PathPatterns {
+	p := PathPattern{
+		Name:        "IPv4",
+		Regexp:      `^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$`,
+		Replacement: "{ip}",
+	}
+	re, _ := regexp.Compile(p.Regexp)
+	p.compiled = re
+
+	return PathPatterns{p}
+}
@@ -20,10 +20,12 @@ var sensitiveHeaders = []string{

 // LoggingProxy wraps a ReverseProxy to provide instrumentation.
 type LoggingProxy struct {
-	Proxy       *httputil.ReverseProxy
-	Redact      bool
-	LogBody     bool
-	MaxBodySize int64
+	Proxy         *httputil.ReverseProxy
+	Redact        bool
+	LogBody       bool
+	RecordEnabled bool
+	MaxBodySize   int64
+	Recorder      *Recorder
 }

 // NewLoggingProxy creates a lightweight logger for HTTP requests/responses.
@@ -36,6 +38,11 @@ func NewLoggingProxy(_ string, redact bool) *LoggingProxy {
 	}
 }

+// SetRecorder sets the recorder for the proxy.
+func (lp *LoggingProxy) SetRecorder(r *Recorder) {
+	lp.Recorder = r
+}
+
 // LogRequest prints an abbreviated request with optional header/body redaction.
 func (lp *LoggingProxy) LogRequest(r *http.Request) {
 	headers := formatHeaders(r.Header, lp.Redact)
@@ -82,6 +89,10 @@ func (lp *LoggingProxy) LogResponse(r *http.Response) {
 	}

 	log.Printf("[PROXY_RES] %d %s\n  Headers:\n%s\n  Body: %s", r.StatusCode, r.Request.URL.String(), headers, bodyStr)
+
+	if lp.Recorder != nil && lp.RecordEnabled {
+		_ = lp.Recorder.Record("upstream", r.Request, r)
+	}
 }

 func formatHeaders(h http.Header, redact bool) string {
@@ -0,0 +1,223 @@
+package proxy
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+// Recorder handles persisting HTTP interactions as .http files.
+type Recorder struct {
+	BaseDir    string
+	SessionID  string
+	SessionDir string
+	Patterns   PathPatterns
+	Redact     bool
+	counter    uint64
+	variables  map[string]string
+	mu         sync.Mutex
+}
+
+// NewRecorder creates a new HTTP interaction recorder.
+func NewRecorder(baseDir string) *Recorder {
+	sessionID := time.Now().Format("20060102-150405") + "-" + fmt.Sprintf("%d", os.Getpid())
+
+	return &Recorder{
+		BaseDir:   baseDir,
+		SessionID: sessionID,
+		Patterns:  DefaultPatterns(),
+		variables: make(map[string]string),
+	}
+}
+
+// Record persists a request and response to a .http file in the specified category (e.g., "self" or "upstream").
+func (r *Recorder) Record(category string, req *http.Request, res *http.Response) error {
+	if r.BaseDir == "" {
+		return nil
+	}
+
+	sanitizedSegments, replacements := r.getSanitizedSegments(req.URL.Path)
+	dir := r.getRecordingDir(category, sanitizedSegments)
+
+	if err := os.MkdirAll(dir, 0755); err != nil {
+		return fmt.Errorf("failed to create directory %s: %w", dir, err)
+	}
+
+	path := r.getRecordingPath(dir, req.Method)
+
+	var buf bytes.Buffer
+
+	r.writeRequest(&buf, req, replacements)
+
+	if res != nil {
+		r.writeResponse(&buf, res)
+	}
+
+	if err := os.WriteFile(path, buf.Bytes(), 0644); err != nil {
+		return err
+	}
+
+	return r.updateEnvFile(replacements)
+}
+
+func (r *Recorder) getSanitizedSegments(path string) ([]string, map[string]string) {
+	pathSegments := strings.Split(strings.Trim(path, "/"), "/")
+	sanitizedSegments := make([]string, 0, len(pathSegments))
+	replacements := make(map[string]string)
+
+	for _, segment := range pathSegments {
+		if segment == "" {
+			continue
+		}
+
+		sanitized, replacement := r.Patterns.Sanitize(segment)
+		sanitizedSegments = append(sanitizedSegments, sanitized)
+
+		if replacement != "" {
+			replacements[segment] = replacement
+		}
+	}
+
+	return sanitizedSegments, replacements
+}
+
+func (r *Recorder) getRecordingDir(category string, sanitizedSegments []string) string {
+	subDir := "root"
+	if len(sanitizedSegments) > 0 {
+		subDir = filepath.Join(sanitizedSegments...)
+	}
+
+	return filepath.Join(r.BaseDir, "interactions", r.SessionID, category, subDir)
+}
+
+func (r *Recorder) getRecordingPath(dir, method string) string {
+	timestamp := time.Now().Format("15-04-05.000")
+	count := atomic.AddUint64(&r.counter, 1)
+	filename := fmt.Sprintf("%04d-%s-%s.http", count, timestamp, method)
+
+	return filepath.Join(dir, filename)
+}
+
+func (r *Recorder) writeRequest(buf *bytes.Buffer, req *http.Request, replacements map[string]string) {
+	displayURL := req.URL.String()
+	for orig, repl := range replacements {
+		displayURL = strings.ReplaceAll(displayURL, orig, "{{"+strings.Trim(repl, "{}")+"}}")
+	}
+
+	fmt.Fprintf(buf, "### %s %s\n", req.Method, displayURL)
+
+	for orig, repl := range replacements {
+		key := strings.Trim(repl, "{}")
+		fmt.Fprintf(buf, "// %s: %s\n", key, orig)
+	}
+
+	fmt.Fprintf(buf, "%s %s\n", req.Method, displayURL)
+
+	for k, vv := range req.Header {
+		if r.Redact && isSensitive(k) {
+			fmt.Fprintf(buf, "%s: [REDACTED]\n", k)
+			continue
+		}
+
+		for _, v := range vv {
+			val := v
+			for orig, repl := range replacements {
+				val = strings.ReplaceAll(val, orig, "{{"+strings.Trim(repl, "{}")+"}}")
+			}
+
+			fmt.Fprintf(buf, "%s: %s\n", k, val)
+		}
+	}
+
+	buf.WriteString("\n")
+
+	if req.Body != nil {
+		bodyBytes, err := io.ReadAll(req.Body)
+		if err == nil {
+			req.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
+			buf.Write(bodyBytes)
+			buf.WriteString("\n")
+		}
+	}
+}
+
+func (r *Recorder) writeResponse(buf *bytes.Buffer, res *http.Response) {
+	buf.WriteString("\n")
+	buf.WriteString("> {% \n")
+	fmt.Fprintf(buf, "    // Response: %d %s\n", res.StatusCode, http.StatusText(res.StatusCode))
+	buf.WriteString("    // Headers:\n")
+
+	for k, vv := range res.Header {
+		if r.Redact && isSensitive(k) {
+			fmt.Fprintf(buf, "    // %s: [REDACTED]\n", k)
+			continue
+		}
+
+		for _, v := range vv {
+			fmt.Fprintf(buf, "    // %s: %s\n", k, v)
+		}
+	}
+
+	buf.WriteString("%}\n")
+
+	if res.Body != nil {
+		bodyBytes, err := io.ReadAll(res.Body)
+		if err == nil {
+			res.Body = io.NopCloser(bytes.NewBuffer(bodyBytes))
+
+			contentType := res.Header.Get("Content-Type")
+			if strings.Contains(contentType, "xml") || strings.Contains(contentType, "json") || strings.Contains(contentType, "text") {
+				buf.WriteString("\n/*\n")
+				buf.Write(bodyBytes)
+				buf.WriteString("\n*/\n")
+			} else {
+				fmt.Fprintf(buf, "\n// [Binary response body: %d bytes]\n", len(bodyBytes))
+			}
+		}
+	}
+}
+
+func (r *Recorder) updateEnvFile(newVars map[string]string) error {
+	if len(newVars) == 0 {
+		return nil
+	}
+
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	changed := false
+
+	for orig, repl := range newVars {
+		key := strings.Trim(repl, "{}")
+		if r.variables[key] != orig {
+			r.variables[key] = orig
+			changed = true
+		}
+	}
+
+	if !changed {
+		return nil
+	}
+
+	envFile := filepath.Join(r.BaseDir, "interactions", r.SessionID, "http-client.env.json")
+
+	// Create the structure: {"session": {"key": "val"}}
+	content := map[string]map[string]string{
+		"session": r.variables,
+	}
+
+	data, err := json.MarshalIndent(content, "", "  ")
+	if err != nil {
+		return err
+	}
+
+	return os.WriteFile(envFile, data, 0644)
+}
@@ -0,0 +1,329 @@
+package proxy
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/url"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"testing"
+)
+
+func TestRecorder_Record_Structure(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "recorder-test")
+	if err != nil {
+		t.Fatalf("failed to create temp dir: %v", err)
+	}
+	defer os.RemoveAll(tmpDir)
+
+	r := NewRecorder(tmpDir)
+
+	tests := []struct {
+		name     string
+		category string
+		path     string
+		expected string // Expected subdirectory after interactions/{sessionID}/{category}/
+	}{
+		{
+			name:     "root_path",
+			category: "self",
+			path:     "/",
+			expected: "root",
+		},
+		{
+			name:     "simple_path",
+			category: "self",
+			path:     "/setup/info",
+			expected: "setup/info",
+		},
+		{
+			name:     "path_with_ip",
+			category: "self",
+			path:     "/setup/info/192.168.178.35",
+			expected: "setup/info/{ip}",
+		},
+		{
+			name:     "upstream_path",
+			category: "upstream",
+			path:     "/v1/playback/station/s123",
+			expected: "v1/playback/station/s123",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			req := &http.Request{
+				Method: "GET",
+				URL: &url.URL{
+					Path: tt.path,
+				},
+				Header: make(http.Header),
+			}
+
+			err := r.Record(tt.category, req, nil)
+			if err != nil {
+				t.Fatalf("Record failed: %v", err)
+			}
+
+			expectedDir := filepath.Join(tmpDir, "interactions", r.SessionID, tt.category, tt.expected)
+			if _, err := os.Stat(expectedDir); os.IsNotExist(err) {
+				t.Errorf("Expected directory %s does not exist", expectedDir)
+			}
+
+			// Check if file was created
+			files, _ := os.ReadDir(expectedDir)
+			if len(files) == 0 {
+				t.Errorf("No files created in %s", expectedDir)
+			}
+			for _, f := range files {
+				if !strings.Contains(f.Name(), "-GET.http") {
+					t.Errorf("Unexpected filename: %s", f.Name())
+				}
+				// Verify prefix is 4 digits
+				if len(f.Name()) < 5 || !isDigit(f.Name()[0]) || !isDigit(f.Name()[1]) || !isDigit(f.Name()[2]) || !isDigit(f.Name()[3]) || f.Name()[4] != '-' {
+					t.Errorf("Filename %s does not have correct 0000- prefix", f.Name())
+				}
+			}
+		})
+	}
+}
+
+func TestRecorder_Record_Sanitization(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "recorder-sanitization-test")
+	if err != nil {
+		t.Fatalf("failed to create temp dir: %v", err)
+	}
+	defer os.RemoveAll(tmpDir)
+
+	r := NewRecorder(tmpDir)
+	// Add a custom pattern
+	r.Patterns = append(r.Patterns, PathPattern{
+		Name:        "DeviceID",
+		Regexp:      `^A81B\w{8}$`,
+		Replacement: "{deviceId}",
+	})
+	// Re-compile
+	for i := range r.Patterns {
+		re, _ := regexp.Compile(r.Patterns[i].Regexp)
+		r.Patterns[i].compiled = re
+	}
+
+	req := &http.Request{
+		Method: "GET",
+		URL: &url.URL{
+			Path: "/info/192.168.178.35/A81B6A536A98",
+		},
+		Header: make(http.Header),
+	}
+	req.Header.Set("X-Device", "A81B6A536A98")
+
+	err = r.Record("self", req, nil)
+	if err != nil {
+		t.Fatalf("Record failed: %v", err)
+	}
+
+	expectedDir := filepath.Join(tmpDir, "interactions", r.SessionID, "self", "info", "{ip}", "{deviceId}")
+	if _, err := os.Stat(expectedDir); os.IsNotExist(err) {
+		t.Errorf("Expected directory %s does not exist", expectedDir)
+	}
+
+	files, _ := os.ReadDir(expectedDir)
+	if len(files) == 0 {
+		t.Fatalf("No files created in %s", expectedDir)
+	}
+
+	content, _ := os.ReadFile(filepath.Join(expectedDir, files[0].Name()))
+	contentStr := string(content)
+
+	if !strings.Contains(contentStr, "### GET /info/{{ip}}/{{deviceId}}") {
+		t.Errorf("Expected sanitized comment in .http file, got:\n%s", contentStr)
+	}
+	if !strings.Contains(contentStr, "GET /info/{{ip}}/{{deviceId}}") {
+		t.Errorf("Expected sanitized URL in .http file, got:\n%s", contentStr)
+	}
+	if !strings.Contains(contentStr, "X-Device: {{deviceId}}") {
+		t.Errorf("Expected sanitized Header in .http file, got:\n%s", contentStr)
+	}
+}
+
+func TestRecorder_Record_Sanitization_Account(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "recorder-sanitization-account-test")
+	if err != nil {
+		t.Fatalf("failed to create temp dir: %v", err)
+	}
+	defer os.RemoveAll(tmpDir)
+
+	r := NewRecorder(tmpDir)
+	// Add AccountID pattern
+	r.Patterns = append(r.Patterns, PathPattern{
+		Name:        "AccountID",
+		Regexp:      `^\d{1,10}$`,
+		Replacement: "{accountId}",
+	})
+	// Re-compile
+	for i := range r.Patterns {
+		re, _ := regexp.Compile(r.Patterns[i].Regexp)
+		r.Patterns[i].compiled = re
+	}
+
+	req := &http.Request{
+		Method: "GET",
+		URL: &url.URL{
+			Path: "/marge/accounts/12345/full",
+		},
+		Header: make(http.Header),
+	}
+
+	err = r.Record("self", req, nil)
+	if err != nil {
+		t.Fatalf("Record failed: %v", err)
+	}
+
+	expectedDir := filepath.Join(tmpDir, "interactions", r.SessionID, "self", "marge", "accounts", "{accountId}", "full")
+	if _, err := os.Stat(expectedDir); os.IsNotExist(err) {
+		t.Errorf("Expected directory %s does not exist", expectedDir)
+	}
+
+	files, _ := os.ReadDir(expectedDir)
+	if len(files) == 0 {
+		t.Fatalf("No files created in %s", expectedDir)
+	}
+
+	content, _ := os.ReadFile(filepath.Join(expectedDir, files[0].Name()))
+	contentStr := string(content)
+
+	if !strings.Contains(contentStr, "### GET /marge/accounts/{{accountId}}/full") {
+		t.Errorf("Expected sanitized comment in .http file, got:\n%s", contentStr)
+	}
+	if !strings.Contains(contentStr, "GET /marge/accounts/{{accountId}}/full") {
+		t.Errorf("Expected sanitized URL in .http file, got:\n%s", contentStr)
+	}
+	if !strings.Contains(contentStr, "// accountId: 12345") {
+		t.Errorf("Expected accountId comment in .http file, got:\n%s", contentStr)
+	}
+}
+
+func TestRecorder_Record_Redaction(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "recorder-redaction-test")
+	if err != nil {
+		t.Fatalf("failed to create temp dir: %v", err)
+	}
+	defer os.RemoveAll(tmpDir)
+
+	r := NewRecorder(tmpDir)
+	r.Redact = true
+
+	req := &http.Request{
+		Method: "GET",
+		URL: &url.URL{
+			Path: "/test",
+		},
+		Header: make(http.Header),
+	}
+	req.Header.Set("Authorization", "Bearer sensitive-token")
+	req.Header.Set("Cookie", "session=secret")
+	req.Header.Set("X-Normal", "public-info")
+
+	err = r.Record("self", req, nil)
+	if err != nil {
+		t.Fatalf("Record failed: %v", err)
+	}
+
+	expectedDir := filepath.Join(tmpDir, "interactions", r.SessionID, "self", "test")
+	files, _ := os.ReadDir(expectedDir)
+	content, _ := os.ReadFile(filepath.Join(expectedDir, files[0].Name()))
+	contentStr := string(content)
+
+	if !strings.Contains(contentStr, "Authorization: [REDACTED]") {
+		t.Errorf("Expected Authorization header to be redacted, got:\n%s", contentStr)
+	}
+	if strings.Contains(contentStr, "sensitive-token") {
+		t.Errorf("Sensitive token still present in content:\n%s", contentStr)
+	}
+	if !strings.Contains(contentStr, "Cookie: [REDACTED]") {
+		t.Errorf("Expected Cookie header to be redacted, got:\n%s", contentStr)
+	}
+	if !strings.Contains(contentStr, "X-Normal: public-info") {
+		t.Errorf("Expected normal header to be present, got:\n%s", contentStr)
+	}
+}
+
+func isDigit(c byte) bool {
+	return c >= '0' && c <= '9'
+}
+
+func TestRecorder_IncreasingPrefix(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "recorder-prefix-test")
+	if err != nil {
+		t.Fatalf("failed to create temp dir: %v", err)
+	}
+	defer os.RemoveAll(tmpDir)
+
+	r := NewRecorder(tmpDir)
+	req := &http.Request{
+		Method: "GET",
+		URL: &url.URL{
+			Path: "/test",
+		},
+		Header: make(http.Header),
+	}
+
+	for i := 1; i <= 3; i++ {
+		err := r.Record("self", req, nil)
+		if err != nil {
+			t.Fatalf("Record failed: %v", err)
+		}
+	}
+
+	expectedDir := filepath.Join(tmpDir, "interactions", r.SessionID, "self", "test")
+	files, _ := os.ReadDir(expectedDir)
+	if len(files) != 3 {
+		t.Fatalf("Expected 3 files, got %d", len(files))
+	}
+
+	expectedPrefixes := []string{"0001-", "0002-", "0003-"}
+	for i, f := range files {
+		if !strings.HasPrefix(f.Name(), expectedPrefixes[i]) {
+			t.Errorf("File %d: expected prefix %s, got %s", i, expectedPrefixes[i], f.Name())
+		}
+	}
+}
+
+func TestRecorder_EnvFile(t *testing.T) {
+	tmpDir, err := os.MkdirTemp("", "recorder-env-test")
+	if err != nil {
+		t.Fatalf("failed to create temp dir: %v", err)
+	}
+	defer os.RemoveAll(tmpDir)
+
+	r := NewRecorder(tmpDir)
+	req := &http.Request{
+		Method: "GET",
+		URL: &url.URL{
+			Path: "/info/192.168.178.35",
+		},
+		Header: make(http.Header),
+	}
+
+	err = r.Record("self", req, nil)
+	if err != nil {
+		t.Fatalf("Record failed: %v", err)
+	}
+
+	envFile := filepath.Join(tmpDir, "interactions", r.SessionID, "http-client.env.json")
+	if _, err := os.Stat(envFile); os.IsNotExist(err) {
+		t.Fatalf("Expected env file %s does not exist", envFile)
+	}
+
+	data, _ := os.ReadFile(envFile)
+	var content map[string]map[string]string
+	if err := json.Unmarshal(data, &content); err != nil {
+		t.Fatalf("Failed to unmarshal env file: %v", err)
+	}
+
+	if content["session"]["ip"] != "192.168.178.35" {
+		t.Errorf("Expected ip to be 192.168.178.35, got %s", content["session"]["ip"])
+	}
+}
@@ -8,8 +8,11 @@ import (
 	"net/http"
 	"net/url"
 	"os"
+	"strconv"
 	"strings"

+	"github.com/gesellix/bose-soundtouch/pkg/models"
+
 	"github.com/gesellix/bose-soundtouch/pkg/service/certmanager"
 	"github.com/gesellix/bose-soundtouch/pkg/service/datastore"
 	"github.com/gesellix/bose-soundtouch/pkg/service/ssh"
@@ -247,7 +250,8 @@ func (m *Manager) populateDeviceInfo(summary *MigrationSummary, deviceIP string)
 	if m.DataStore != nil {
 		devices, err := m.DataStore.ListAllDevices()
 		if err == nil {
-			for _, d := range devices {
+			for i := range devices {
+				d := devices[i]
 				if d.IPAddress != deviceIP {
 					continue
 				}
@@ -984,3 +988,188 @@ func (m *Manager) resolveIP(host string, client SSHClient) string {

 	return ips[0].String()
 }
+
+// SyncDeviceData fetches presets, recents and sources from the device and saves them to the datastore.
+func (m *Manager) SyncDeviceData(deviceIP string) error {
+	// 1. Fetch info to get Serial Number (account identifier)
+	info, err := m.GetLiveDeviceInfo(deviceIP)
+	if err != nil {
+		return fmt.Errorf("failed to get device info: %w", err)
+	}
+
+	accountID := "default"
+
+	deviceID := info.SerialNumber
+	if deviceID == "" {
+		deviceID = deviceIP
+	}
+
+	// 2. Fetch Presets from :8090
+	m.syncPresets(deviceIP, accountID, deviceID)
+
+	// 3. Fetch Recents from :8090
+	m.syncRecents(deviceIP, accountID, deviceID)
+
+	// 4. Fetch Sources
+	m.syncSources(deviceIP, accountID, deviceID)
+
+	return nil
+}
+
+func (m *Manager) syncPresets(deviceIP, accountID, deviceID string) {
+	presetsURL := fmt.Sprintf("http://%s:8090/presets", deviceIP)
+	if _, _, splitErr := net.SplitHostPort(deviceIP); splitErr == nil {
+		presetsURL = fmt.Sprintf("http://%s/presets", deviceIP)
+	}
+
+	resp, err := http.Get(presetsURL)
+	if err != nil {
+		return
+	}
+
+	defer func() { _ = resp.Body.Close() }()
+
+	var ps models.Presets
+	if decodeErr := xml.NewDecoder(resp.Body).Decode(&ps); decodeErr != nil {
+		return
+	}
+
+	var servicePresets []models.ServicePreset
+
+	for _, p := range ps.Preset {
+		if p.ContentItem == nil {
+			continue
+		}
+
+		createdOn := ""
+		if p.CreatedOn != nil {
+			createdOn = strconv.FormatInt(*p.CreatedOn, 10)
+		}
+
+		updatedOn := ""
+		if p.UpdatedOn != nil {
+			updatedOn = strconv.FormatInt(*p.UpdatedOn, 10)
+		}
+
+		servicePresets = append(servicePresets, models.ServicePreset{
+			ServiceContentItem: models.ServiceContentItem{
+				ID:            strconv.Itoa(p.ID),
+				Name:          p.ContentItem.ItemName,
+				Source:        p.ContentItem.Source,
+				Type:          p.ContentItem.Type,
+				Location:      p.ContentItem.Location,
+				SourceAccount: p.ContentItem.SourceAccount,
+				SourceID:      "", // Preset doesn't have SourceID in ContentItem usually
+				IsPresetable:  strconv.FormatBool(p.ContentItem.IsPresetable),
+			},
+			ContainerArt: p.ContentItem.ContainerArt,
+			CreatedOn:    createdOn,
+			UpdatedOn:    updatedOn,
+		})
+	}
+
+	_ = m.DataStore.SavePresets(accountID, deviceID, servicePresets)
+}
+
+func (m *Manager) syncRecents(deviceIP, accountID, deviceID string) {
+	recentsURL := fmt.Sprintf("http://%s:8090/recents", deviceIP)
+	if _, _, splitErr := net.SplitHostPort(deviceIP); splitErr == nil {
+		recentsURL = fmt.Sprintf("http://%s/recents", deviceIP)
+	}
+
+	resp, err := http.Get(recentsURL)
+	if err != nil {
+		return
+	}
+
+	defer func() { _ = resp.Body.Close() }()
+
+	var rr models.RecentsResponse
+	if decodeErr := xml.NewDecoder(resp.Body).Decode(&rr); decodeErr != nil {
+		return
+	}
+
+	var serviceRecents []models.ServiceRecent
+
+	for _, r := range rr.Items {
+		if r.ContentItem == nil {
+			continue
+		}
+
+		serviceRecents = append(serviceRecents, models.ServiceRecent{
+			ServiceContentItem: models.ServiceContentItem{
+				ID:            r.ID,
+				Name:          r.ContentItem.ItemName,
+				Source:        r.ContentItem.Source,
+				Type:          r.ContentItem.Type,
+				Location:      r.ContentItem.Location,
+				SourceAccount: r.ContentItem.SourceAccount,
+				SourceID:      "", // RecentsResponseItem doesn't have SourceID usually
+				IsPresetable:  strconv.FormatBool(r.ContentItem.IsPresetable),
+			},
+			DeviceID:     r.DeviceID,
+			UtcTime:      strconv.FormatInt(r.UTCTime, 10),
+			ContainerArt: r.ContentItem.ContainerArt,
+		})
+	}
+
+	_ = m.DataStore.SaveRecents(accountID, deviceID, serviceRecents)
+}
+
+func (m *Manager) syncSources(deviceIP, accountID, deviceID string) {
+	client := m.NewSSH(deviceIP)
+
+	sourcesXML, err := client.Run("cat /mnt/nv/BoseApp-Persistence/1/Sources.xml")
+	if err == nil && sourcesXML != "" {
+		var srs struct {
+			Sources []models.ConfiguredSource `xml:"source"`
+		}
+		if xmlErr := xml.Unmarshal([]byte(sourcesXML), &srs); xmlErr == nil {
+			// After unmarshaling from SSH, ensure legacy fields are synced for internal use
+			for i := range srs.Sources {
+				s := &srs.Sources[i]
+				s.SourceKeyType = s.SourceKey.Type
+				s.SourceKeyAccount = s.SourceKey.Account
+			}
+
+			_ = m.DataStore.SaveConfiguredSources(accountID, deviceID, srs.Sources)
+
+			return
+		}
+	}
+
+	// Fallback to :8090/sources
+	sourcesURL := fmt.Sprintf("http://%s:8090/sources", deviceIP)
+	if _, _, splitErr := net.SplitHostPort(deviceIP); splitErr == nil {
+		sourcesURL = fmt.Sprintf("http://%s/sources", deviceIP)
+	}
+
+	resp, err := http.Get(sourcesURL)
+	if err != nil {
+		return
+	}
+
+	defer func() { _ = resp.Body.Close() }()
+
+	var srs models.Sources
+	if decodeErr := xml.NewDecoder(resp.Body).Decode(&srs); decodeErr == nil {
+		var configuredSources []models.ConfiguredSource
+
+		for _, s := range srs.SourceItem {
+			cs := models.ConfiguredSource{
+				DisplayName: s.DisplayName,
+				ID:          s.Source,
+				SecretType:  string(s.Status),
+			}
+			cs.SourceKey.Type = s.Source
+			cs.SourceKey.Account = s.SourceAccount
+			// Also set legacy fields for now
+			cs.SourceKeyType = s.Source
+			cs.SourceKeyAccount = s.SourceAccount
+
+			configuredSources = append(configuredSources, cs)
+		}
+
+		_ = m.DataStore.SaveConfiguredSources(accountID, deviceID, configuredSources)
+	}
+}
Author	SHA1	Message	Date
Tobias Gesellchen	dcf2e29c16	Fix linting issues and refactor for improved code quality	2026-02-14 12:39:39 +01:00
Tobias Gesellchen	9be1c7d588	Allow toggling HTTP interaction recording via CLI, environment, and Web UI	2026-02-14 12:39:39 +01:00
Tobias Gesellchen	133c07fefa	Improve visibility of multiple devices in recordings by adding original value comments to .http files	2026-02-14 12:39:39 +01:00
Tobias Gesellchen	ef90b4e848	Improve structure and re-usability of HTTP interaction recordings	2026-02-14 12:39:39 +01:00
Tobias Gesellchen	c8ef1a9de4	Update README and documentation to reflect Toolkit expansion and Docker support	2026-02-14 00:14:29 +01:00
Tobias Gesellchen	e47fa4c92c	Complete local Bose SoundTouch emulation service with guided migration UI	2026-02-14 00:14:29 +01:00