### Overview of Recent Improvements and Next Steps This document summarizes the improvements made to the **Marge service** to improve parity with the upstream Bose SoundTouch service, along with open issues and proposed next steps. #### ✅ Completed Improvements (Marge Service) * **Mapped Preset `buttonNumber`**: Correctly mapped the internal `ServicePreset.ID` or `ButtonNumber` to the `buttonNumber` XML attribute in the `/full` response and ensured it is persisted in the local datastore. * **High-Fidelity Device Metadata**: Improved the datastore to correctly extract, persist, and report detailed device `` (e.g., `LIGHTSWITCH`, `SMSC`) and their firmware versions from upstream responses. * **Standardized Preferred Language**: Updated the default `preferredLanguage` to `de` in the `/full` response and added synchronization to persist it from upstream responses. * **Persisted Provider Settings**: Added support for persisting and echoing back `providerSettings` (e.g., `STREAMING_QUALITY`, `ELIGIBLE_FOR_TRIAL`) from the `/full` response. * **Populated `contentItemType`**: The `contentItemType` (e.g., `tracklisturl`) is now correctly synchronized from upstream, persisted in the local datastore, and returned in the `/full` response for both presets and recents. * **Standardized Credential Types**: Adjusted the logic for Spotify to use the correct `token_version_3` type when a token is present in the `/full` response, improving parity with the upstream service. The service now respects existing `credential_type` values from `Sources.xml` (e.g., `token_version_3` for Spotify) while providing sensible defaults for new or incomplete sources. * **Structured Sources (Sources.xml)**: Refactored `Sources.xml` to use an attribute-based structure (`sourceid`, `source`, `status`, `sourceAccount`, etc.) matching the real device's output. Removed redundant nested tags like ``, ``, and ``. * **Nested Recents (Recents.xml)**: Implemented a nested `` structure within `` entries in `Recents.xml`, maintaining exact parity with the device's persistence format while supporting legacy flat formats for backward compatibility. * **Inconsistent `serialNumber` Casing**: Fixed the casing mismatch in the `/full` response where the upstream uses camelCase `` in the top-level `` and lowercase `` in the nested ``. Local responses now correctly mirror this inconsistency. * **Attribute-level Parity**: * Ensured `sourceAccount=""` is preserved in XML even when empty, matching device behavior for sources like TUNEIN. * Fixed casing for attributes like `deviceID` and `utcTime` in `Recents.xml`. * Correctly mapped and persisted preset and recent `id` attributes during "Initial Data Sync". * **Device Name Consistency**: Fixed an issue where the device `` was empty in some local `/full` responses by ensuring it is correctly populated from the datastore and synchronized from upstream. * **Improved XML Parity**: Empty `` tags in the `/full` response are now self-closing (``), matching upstream behavior. * **Timestamp-based ID Generation**: Implemented a 9-digit ID schema (`YYMMDD` + 3-digit counter) for `recent` items, ensuring IDs are large, unique, and stay within the 32-bit integer range. * **Automatic Source Learning**: The service now extracts and persists full metadata (credentials, provider IDs, and custom names) from incoming `POST /recent` requests. This improves parity for subsequent `GET /recents` calls. * **Source Provider Mapping**: Synchronized local source provider IDs and timestamps with upstream data. The `RADIO_BROWSER` provider is included in the public `/streaming/sourceproviders` list to maintain internal functionality while acknowledging it as a parity gap. * **Credential Preservation**: Improved `AddRecent` to correctly extract and echo back base64 tokens/credentials provided in the incoming request, improving source learning. * **XML Formatting Parity**: * Added `standalone="yes"` to the XML declaration for all Marge responses, including `recent`, `presets`, `full account`, `software update`, and `sourceproviders`. * Enforced self-closing `` tags for parity. * Standardized date formatting to UTC with milliseconds (`.000+00:00`). * Fixed casing for `/streaming/sourceproviders`: Root element is ``, but child elements are `` (all lowercase), matching upstream behavior. * Implemented structured XML marshaling with consistent 2-space indentation for recents and source providers. * **Improved TuneIn Parity**: Fixed TuneIn source mapping to use ID `25` and ensuring `sourcename` is empty in responses, matching upstream behavior for station playback. * **High-Fidelity Full Account Sync**: Refactored the `/streaming/account/{accountId}/full` response to match the upstream structure. This includes: * **Mapped Preset `buttonNumber`**: Correctly mapped the internal `ServicePreset.ID` to the `buttonNumber` XML attribute in the `/full` response. * **Structured XML Marshaling**: Replaced manual string concatenation with structured Go models and `xml.Marshal` for the entire response. * **Specific Response Models**: Introduced `FullResponseSource`, `FullResponsePreset`, and `FullResponseRecent` to accurately reflect the upstream structure where `` is a child element, rather than a set of attributes. * **Correct Nesting**: Ensured that `` and `` correctly nest their associated `` details, resolving previous data omissions. * **Device Identity**: Added `` and `` to both the top-level `` and its ``, ensuring consistent device identification. * **Field-Level Parity**: Mapped missing fields like `` and `` to match upstream expectations. * **Improved Source Matching**: Enhanced internal logic to correctly link presets and recents to their configured sources based on multiple identifiers (ID, Key, or Type). * **Verified Parity Mismatch Fixes**: Comprehensive reproduction tests (`TestParityMismatchReproduction_V2` and `TestParityMismatchReproduction_V3`) now confirm parity for identified mismatches in `POST /recent` and `GET /recents`, including credentials and source-specific metadata. * **Unified Response Logic**: Refactored the code so that both `POST /recent` and `GET /recents` use the same formatting functions, guaranteeing consistency. * **Robust Parity Detection**: Updated the local parity checker to be whitespace-insensitive for XML bodies, significantly reducing noise from minor indentation or newline differences. * **Maintainable XML Generation**: Reduced cyclomatic complexity and code duplication in `marge.go` by extracting focused helper functions for mapping internal data to response-specific XML models. --- #### 🛠️ Open Issues and Next Steps Based on the latest `parity_mismatches` and the high-fidelity `/full` account response comparison (diff14), here are the recommended areas for further work: #### 1. BMX / TuneIn Playback Parity (Medium) Current mismatches in `/bmx/tunein/v1/playback/station/...` show differences in reporting URLs and missing links: * **Mismatched Parameters**: Local reporting URLs use `listen_id=1234567890`, while upstream uses a different session-based ID. * **Missing Links**: Some upstream responses include additional `_links` or metadata that are currently omitted in local responses. * **Action**: Improve the `HandleTuneInPlayback` logic to better mirror the upstream response structure and parameter generation. #### 2. `/full` Account Response Data Gaps (Medium) While structural parity for the `/full` response is high, several value-level gaps remain as shown in `diff14`: * **Timestamp Formats**: Upstream uses ISO-8601 with milliseconds (e.g., `2024-06-23T07:40:36.000+00:00`), whereas some local fields still use Unix epoch integers (e.g., `1234567890`). * **Provider Settings**: The `providerSettings` block in the local response currently lacks crucial values like `keyName`, `providerId`, and `boseId` (appearing as empty tags). * **Component Metadata**: Local component types are sometimes empty (`type=""`) compared to upstream values like `LIGHTSWITCH` or `SMSC`. * **Source/Preset Identifiers**: Local IDs (e.g., `100004`) differ from upstream IDs (e.g., `1234567`), though this may be expected due to different account/device environments. * **Action**: Update the mapping logic in `marge.go` and `setup.go` to ensure all fields in the `/full` response are correctly populated with high-fidelity values and standard ISO-8601 timestamps. #### 3. OAuth / Spotify Token Noise (Low/Medium) The `/oauth/device/.../token` endpoint frequently reports mismatches because tokens are naturally different between local and upstream. * **The Issue**: This creates "noise" in your parity reports that isn't actually a bug. * **Action**: Update the parity detection logic (or the handler) to selectively ignore the `access_token` field while still verifying that the rest of the JSON structure (expires_in, scope, token_type) matches. #### 4. Large IDs for Other Models (Medium) While we fixed IDs for `recents`, other models like `presets` or `sources` might still use small auto-incrementing integers. * **Action**: Evaluate if other endpoints should also transition to the timestamp-based ID schema to further reduce diff noise. #### 5. Improved Data Persistence (Continuous) Continue the "learning" approach for other services. For example, if we see a new `sourceproviderid` in a Spotify or TuneIn request, we should ensure it is stored and reused. #### 6. Local Reboot & Device State Management (Continuous) Analysis of device reboot logs revealed several data requirements: * **Power-On Details Tracking**: Implemented extraction and persistence of detailed device information (serial numbers, firmware version, product details, and MAC addresses) from the `POST /streaming/support/power_on` request. This data is now stored in the local datastore, improving our ability to respond accurately to subsequent management requests. * **Source Provider Mapping**: Synchronized local source provider IDs and timestamps with upstream data. The `RADIO_BROWSER` provider is included in the public `/streaming/sourceproviders` list to maintain internal functionality while acknowledging it as a parity gap. #### 7. Account Full Response (/full) Structural & Value Parity (Completed) Structural and value gaps in the `/full` account response have been addressed: **Key Fixes:** * **Structural**: * **Nested Source Association**: Improved the matching logic in `mapRecentsToFullResponse` to correctly link recents to their specific `ConfiguredSource` (e.g., by matching `sourceid` attribute). * **XML Tag Formatting**: Standardized self-closing tags and element formatting to match upstream's multi-line or empty-element formatting in various contexts.