feat(sync): improve parity with upstream during data sync (#123)

- Enhance initial and full data synchronization to better align with upstream services. - Update data structures in 'pkg/models' to support missing fields (e.g., SecretType for Spotify). - Improve 'datastore' persistence logic for presets, recents, and sources. - Add comprehensive regression tests for sync and datastore operations. - Update documentation on parity status and improvements. Co-authored-by: Junie <junie@jetbrains.com> Co-authored-by: Junie <junie@jetbrains.com>
2026-08-19 00:56:16 +00:00 · 2026-03-22 00:01:41 +01:00
parent a0833c113c
commit 717693e01f
30 changed files with 2658 additions and 508 deletions
@@ -3,10 +3,19 @@
 This document summarizes the improvements made to the **Marge service** to improve parity with the upstream Bose SoundTouch service, along with open issues and proposed next steps.

 #### ✅ Completed Improvements (Marge Service)
-*   **Mapped Preset `buttonNumber`**: Correctly mapped the internal `ServicePreset.ID` to the `buttonNumber` XML attribute in the `/full` response.
+*   **Mapped Preset `buttonNumber`**: Correctly mapped the internal `ServicePreset.ID` or `ButtonNumber` to the `buttonNumber` XML attribute in the `/full` response and ensured it is persisted in the local datastore.
+*   **High-Fidelity Device Metadata**: Improved the datastore to correctly extract, persist, and report detailed device `<components>` (e.g., `LIGHTSWITCH`, `SMSC`) and their firmware versions from upstream responses.
+*   **Standardized Preferred Language**: Updated the default `preferredLanguage` to `de` in the `/full` response and added synchronization to persist it from upstream responses.
+*   **Persisted Provider Settings**: Added support for persisting and echoing back `providerSettings` (e.g., `STREAMING_QUALITY`, `ELIGIBLE_FOR_TRIAL`) from the `/full` response.
 *   **Populated `contentItemType`**: The `contentItemType` (e.g., `tracklisturl`) is now correctly synchronized from upstream, persisted in the local datastore, and returned in the `/full` response for both presets and recents.
 *   **Standardized Credential Types**: Adjusted the logic for Spotify to use the correct `token_version_3` type when a token is present in the `/full` response, improving parity with the upstream service. The service now respects existing `credential_type` values from `Sources.xml` (e.g., `token_version_3` for Spotify) while providing sensible defaults for new or incomplete sources.
+*   **Structured Sources (Sources.xml)**: Refactored `Sources.xml` to use an attribute-based structure (`sourceid`, `source`, `status`, `sourceAccount`, etc.) matching the real device's output. Removed redundant nested tags like `<sourcename>`, `<username>`, and `<name>`.
+*   **Nested Recents (Recents.xml)**: Implemented a nested `<contentItem>` structure within `<recent>` entries in `Recents.xml`, maintaining exact parity with the device's persistence format while supporting legacy flat formats for backward compatibility.
 *   **Inconsistent `serialNumber` Casing**: Fixed the casing mismatch in the `/full` response where the upstream uses camelCase `<serialNumber>` in the top-level `<device>` and lowercase `<serialnumber>` in the nested `<attachedProduct>`. Local responses now correctly mirror this inconsistency.
+*   **Attribute-level Parity**:
+    *   Ensured `sourceAccount=""` is preserved in XML even when empty, matching device behavior for sources like TUNEIN.
+    *   Fixed casing for attributes like `deviceID` and `utcTime` in `Recents.xml`.
+    *   Correctly mapped and persisted preset and recent `id` attributes during "Initial Data Sync".
 *   **Device Name Consistency**: Fixed an issue where the device `<name>` was empty in some local `/full` responses by ensuring it is correctly populated from the datastore and synchronized from upstream.
 *   **Improved XML Parity**: Empty `<name>` tags in the `/full` response are now self-closing (`<name/>`), matching upstream behavior.
 *   **Timestamp-based ID Generation**: Implemented a 9-digit ID schema (`YYMMDD` + 3-digit counter) for `recent` items, ensuring IDs are large, unique, and stay within the 32-bit integer range.
@@ -37,19 +46,21 @@ This document summarizes the improvements made to the **Marge service** to impro

 #### 🛠️ Open Issues and Next Steps

-Based on the latest `parity_mismatches`, here are the recommended areas for further work:
+Based on the latest `parity_mismatches` and the high-fidelity `/full` account response comparison (diff14), here are the recommended areas for further work:

 #### 1. BMX / TuneIn Playback Parity (Medium)
 Current mismatches in `/bmx/tunein/v1/playback/station/...` show differences in reporting URLs and missing links:
-*   **Mismatched Parameters**: Local reporting URLs use `listen_id=3432432423`, while upstream uses a different session-based ID.
+*   **Mismatched Parameters**: Local reporting URLs use `listen_id=1234567890`, while upstream uses a different session-based ID.
 *   **Missing Links**: Some upstream responses include additional `_links` or metadata that are currently omitted in local responses.
 *   **Action**: Improve the `HandleTuneInPlayback` logic to better mirror the upstream response structure and parameter generation.

-#### 2. Presets and Recents Parity (Medium)
-Further align the standalone `GET /presets` and `GET /recents` endpoints with the refined structural improvements introduced for the `/full` account response:
-*   **Source Nesting**: Ensure the standalone responses also use the specialized nested `<source>` structure instead of mixed attributes when appropriate.
-*   **Field Completeness**: Verify all metadata fields (e.g., `<contentItemType>`, `<lastplayedat>`) are consistently populated across all access paths.
-*   **Action**: Evaluate if the specialized `FullResponsePreset` and `FullResponseRecent` models should be shared or mirrored in the standalone handlers.
+#### 2. `/full` Account Response Data Gaps (Medium)
+While structural parity for the `/full` response is high, several value-level gaps remain as shown in `diff14`:
+*   **Timestamp Formats**: Upstream uses ISO-8601 with milliseconds (e.g., `2024-06-23T07:40:36.000+00:00`), whereas some local fields still use Unix epoch integers (e.g., `1234567890`).
+*   **Provider Settings**: The `providerSettings` block in the local response currently lacks crucial values like `keyName`, `providerId`, and `boseId` (appearing as empty tags).
+*   **Component Metadata**: Local component types are sometimes empty (`type=""`) compared to upstream values like `LIGHTSWITCH` or `SMSC`.
+*   **Source/Preset Identifiers**: Local IDs (e.g., `100004`) differ from upstream IDs (e.g., `1234567`), though this may be expected due to different account/device environments.
+*   **Action**: Update the mapping logic in `marge.go` and `setup.go` to ensure all fields in the `/full` response are correctly populated with high-fidelity values and standard ISO-8601 timestamps.

 #### 3. OAuth / Spotify Token Noise (Low/Medium)
 The `/oauth/device/.../token` endpoint frequently reports mismatches because tokens are naturally different between local and upstream.
@@ -68,19 +79,10 @@ Analysis of device reboot logs revealed several data requirements:
 *   **Power-On Details Tracking**: Implemented extraction and persistence of detailed device information (serial numbers, firmware version, product details, and MAC addresses) from the `POST /streaming/support/power_on` request. This data is now stored in the local datastore, improving our ability to respond accurately to subsequent management requests.
 *   **Source Provider Mapping**: Synchronized local source provider IDs and timestamps with upstream data. The `RADIO_BROWSER` provider is included in the public `/streaming/sourceproviders` list to maintain internal functionality while acknowledging it as a parity gap.

-#### 7. Account Full Response (/full) Structural & Value Parity (In Progress)
-Based on `_/diffs/diff7/`, several structural and value gaps remain in the `/full` account response:
+#### 7. Account Full Response (/full) Structural & Value Parity (Completed)
+Structural and value gaps in the `/full` account response have been addressed:

-**Remaining Findings:**
-    *   **Nested Source Inconsistency in Recents**: The `<source>` element within `<recent>` entries still frequently points to a generic fallback (ID `9330201`) instead of the specific source (e.g., Spotify ID `10863533`).
-    *   Missing/empty `<username>` at the `<preset>` level.
-*   **Values**:
-    *   **Empty Device `<name>`**: Locally, the device `<name>` is empty in the response even when available in the datastore or upstream.
-    *   **Empty `<contentItemType>`**: Local responses have empty `<contentItemType>` in presets and recents, whereas upstream has `tracklisturl` or `stationurl`.
-    *   `preferredLanguage` mismatch (`en` vs `de`).
-
-**Next Implementation Steps (Proposals):**
-1.  **Fix Device `<name>` Population**: Investigate why `CreateAccountDevice` or `AccountFullToXML` is not correctly returning the device name even if it's synchronized.
-2.  **Refine Source Association in Recents**: Improve the matching logic in `mapRecentsToFullResponse` to correctly link recents to their specific `ConfiguredSource` (e.g., by matching `sourceid` attribute).
-3.  **Populate `contentItemType`**: Update the internal models and `SyncFromAccountFull` to correctly extract, persist, and echo back `contentItemType` (e.g., `tracklisturl`).
-4.  **Handle Account Metadata**: Synchronize `preferredLanguage` from the upstream `/full` response to the local account state.
+**Key Fixes:**
+*   **Structural**:
+    *   **Nested Source Association**: Improved the matching logic in `mapRecentsToFullResponse` to correctly link recents to their specific `ConfiguredSource` (e.g., by matching `sourceid` attribute).
+    *   **XML Tag Formatting**: Standardized self-closing tags and element formatting to match upstream's multi-line or empty-element formatting in various contexts.
@@ -0,0 +1,41 @@
+# Parity Analysis: Bose-SoundTouch (Go) vs. SoundCork (Python)
+
+This document provides a comparative analysis of the current Go implementation and the `deborahgu/soundcork` project, identifying functional gaps and potential improvements.
+
+## 1. Core Architecture and Language
+- **Bose-SoundTouch (Go)**: Uses `chi` for routing and `encoding/xml` for data. High performance, strong typing, and precise MIME type handling (`application/vnd.bose.streaming-v1.2+xml`).
+- **SoundCork (Python)**: Uses `FastAPI` and `xml.etree.ElementTree`. Prioritizes flexibility and rapid prototyping of streaming service mocks.
+
+## 2. Functional Comparison
+
+| Feature              | Bose-SoundTouch (Go)                             | SoundCork (Python)                                                                      |
+|:---------------------|:-------------------------------------------------|:----------------------------------------------------------------------------------------|
+| **Group Management** | Placeholder handlers (return `<group/>` or 404). | Active group management (`groups.py`), supporting `/addGroup` and stereo pairing logic. |
+| **BMX Services**     | Supports TuneIn, Orion, and custom streams.      | More modular `bmx_services.json` registry with broader mock support.                    |
+| **Persistence**      | Mixed JSON/XML datastore.                        | Pure XML-based persistence per device/account.                                          |
+| **Admin UI**         | CLI-based (`soundtouch-cli`) or API-driven.      | Draft Web UI for device discovery and account management (`admin.py`).                  |
+| **Discovery**        | Integrated setup tools and SSDP/MDNS awareness.  | Leverages `bosesoundtouchapi` Python library for active discovery.                      |
+
+## 3. Key Strengths of SoundCork
+- **Group Pairing Logic**: Includes logic to manage master/slave relationships for SoundTouch 10 stereo pairs.
+- **Service Extensibility**: JSON-based registry for BMX services makes it easier to mock multiple providers (SiriusXM, Spotify) without code changes.
+- **Mock Coverage**: Better coverage of "dummy" endpoints that respond with plausible XML (e.g., `customerSupport`).
+
+## 4. Suggested Implementation Steps for Bose-SoundTouch
+
+### A. Implement Full Group Support (High Priority)
+- Add logic to `pkg/service/marge` to handle `/addGroup` and `/updateGroup`.
+- Persist group memberships in the datastore to allow speakers to function as stereo pairs or multi-room zones.
+
+### B. Modularize BMX Registry (Medium Priority)
+- Extract the hardcoded service list in `HandleBMXRegistry` into an external `bmx-services.json` file.
+- Allow users to customize which mocked services are advertised to the speaker.
+
+### C. Enhanced Source Management (Medium Priority)
+- Refine source learning logic to ensure all `sourceAccount` and `sourceName` metadata is correctly captured during synchronization, using patterns from `soundcork`'s `learnSource`.
+
+### D. Basic Admin Web UI (Low Priority)
+- Develop a minimal internal status page to list active accounts and connected devices, improving usability over raw API calls.
+
+## 5. Summary
+While our Go implementation is structurally more consistent with recent reference recordings (e.g., `buttonNumber`, detailed `components`), SoundCork provides better coverage of multi-device coordination (Groups) and service emulation (BMX) that we should adopt for a more complete offline experience.
@@ -57,6 +57,10 @@
 * [IoT Config Summary](analysis/IOT-CONFIG-SUMMARY.md)
 * [IoT Configuration Analysis](analysis/IOT-CONFIGURATION-ANALYSIS.md)

+## Parity Analysis
+* [Parity Improvements](PARITY-IMPROVEMENTS.md)
+* [Parity SoundCork](PARITY-SOUNDCORK.md)
+
 ## Appendix (Other Documents)
 * [API Navigation Reference](API-NAVIGATION-REFERENCE.md)
 * [Claude Instructions](CLAUDE.md)
@@ -81,3 +85,4 @@
 * [Power On Implementation Guide](power-on-implementation-guide.md)
 * [SCMUDC Events Analysis](scmudc-events-analysis.md)
 * [Parity Improvements](PARITY-IMPROVEMENTS.md)
+* [Parity SoundCork](PARITY-SOUNDCORK.md)