### Overview of Recent Improvements and Next Steps This document summarizes the improvements made to the **Marge service** to improve parity with the upstream Bose SoundTouch service, along with open issues and proposed next steps. #### ✅ Completed Improvements (Marge Service) * **Mapped Preset `buttonNumber`**: Correctly mapped the internal `ServicePreset.ID` to the `buttonNumber` XML attribute in the `/full` response. * **Populated `contentItemType`**: The `contentItemType` (e.g., `tracklisturl`) is now correctly synchronized from upstream, persisted in the local datastore, and returned in the `/full` response for both presets and recents. * **Standardized Credential Types**: Adjusted the logic for Spotify to use the correct `token_version_3` type when a token is present in the `/full` response, improving parity with the upstream service. The service now respects existing `credential_type` values from `Sources.xml` (e.g., `token_version_3` for Spotify) while providing sensible defaults for new or incomplete sources. * **Inconsistent `serialNumber` Casing**: Fixed the casing mismatch in the `/full` response where the upstream uses camelCase `` in the top-level `` and lowercase `` in the nested ``. Local responses now correctly mirror this inconsistency. * **Device Name Consistency**: Fixed an issue where the device `` was empty in some local `/full` responses by ensuring it is correctly populated from the datastore and synchronized from upstream. * **Improved XML Parity**: Empty `` tags in the `/full` response are now self-closing (``), matching upstream behavior. * **Timestamp-based ID Generation**: Implemented a 9-digit ID schema (`YYMMDD` + 3-digit counter) for `recent` items, ensuring IDs are large, unique, and stay within the 32-bit integer range. * **Automatic Source Learning**: The service now extracts and persists full metadata (credentials, provider IDs, and custom names) from incoming `POST /recent` requests. This improves parity for subsequent `GET /recents` calls. * **Source Provider Mapping**: Synchronized local source provider IDs and timestamps with upstream data. The `RADIO_BROWSER` provider is included in the public `/streaming/sourceproviders` list to maintain internal functionality while acknowledging it as a parity gap. * **Credential Preservation**: Improved `AddRecent` to correctly extract and echo back base64 tokens/credentials provided in the incoming request, improving source learning. * **XML Formatting Parity**: * Added `standalone="yes"` to the XML declaration for all Marge responses, including `recent`, `presets`, `full account`, `software update`, and `sourceproviders`. * Enforced self-closing `` tags for parity. * Standardized date formatting to UTC with milliseconds (`.000+00:00`). * Fixed casing for `/streaming/sourceproviders`: Root element is ``, but child elements are `` (all lowercase), matching upstream behavior. * Implemented structured XML marshaling with consistent 2-space indentation for recents and source providers. * **Improved TuneIn Parity**: Fixed TuneIn source mapping to use ID `25` and ensuring `sourcename` is empty in responses, matching upstream behavior for station playback. * **High-Fidelity Full Account Sync**: Refactored the `/streaming/account/{accountId}/full` response to match the upstream structure. This includes: * **Mapped Preset `buttonNumber`**: Correctly mapped the internal `ServicePreset.ID` to the `buttonNumber` XML attribute in the `/full` response. * **Structured XML Marshaling**: Replaced manual string concatenation with structured Go models and `xml.Marshal` for the entire response. * **Specific Response Models**: Introduced `FullResponseSource`, `FullResponsePreset`, and `FullResponseRecent` to accurately reflect the upstream structure where `` is a child element, rather than a set of attributes. * **Correct Nesting**: Ensured that `` and `` correctly nest their associated `` details, resolving previous data omissions. * **Device Identity**: Added `` and `` to both the top-level `` and its ``, ensuring consistent device identification. * **Field-Level Parity**: Mapped missing fields like `` and `` to match upstream expectations. * **Improved Source Matching**: Enhanced internal logic to correctly link presets and recents to their configured sources based on multiple identifiers (ID, Key, or Type). * **Verified Parity Mismatch Fixes**: Comprehensive reproduction tests (`TestParityMismatchReproduction_V2` and `TestParityMismatchReproduction_V3`) now confirm parity for identified mismatches in `POST /recent` and `GET /recents`, including credentials and source-specific metadata. * **Unified Response Logic**: Refactored the code so that both `POST /recent` and `GET /recents` use the same formatting functions, guaranteeing consistency. * **Robust Parity Detection**: Updated the local parity checker to be whitespace-insensitive for XML bodies, significantly reducing noise from minor indentation or newline differences. * **Maintainable XML Generation**: Reduced cyclomatic complexity and code duplication in `marge.go` by extracting focused helper functions for mapping internal data to response-specific XML models. --- #### 🛠️ Open Issues and Next Steps Based on the latest `parity_mismatches`, here are the recommended areas for further work: #### 1. BMX / TuneIn Playback Parity (Medium) Current mismatches in `/bmx/tunein/v1/playback/station/...` show differences in reporting URLs and missing links: * **Mismatched Parameters**: Local reporting URLs use `listen_id=3432432423`, while upstream uses a different session-based ID. * **Missing Links**: Some upstream responses include additional `_links` or metadata that are currently omitted in local responses. * **Action**: Improve the `HandleTuneInPlayback` logic to better mirror the upstream response structure and parameter generation. #### 2. Presets and Recents Parity (Medium) Further align the standalone `GET /presets` and `GET /recents` endpoints with the refined structural improvements introduced for the `/full` account response: * **Source Nesting**: Ensure the standalone responses also use the specialized nested `` structure instead of mixed attributes when appropriate. * **Field Completeness**: Verify all metadata fields (e.g., ``, ``) are consistently populated across all access paths. * **Action**: Evaluate if the specialized `FullResponsePreset` and `FullResponseRecent` models should be shared or mirrored in the standalone handlers. #### 3. OAuth / Spotify Token Noise (Low/Medium) The `/oauth/device/.../token` endpoint frequently reports mismatches because tokens are naturally different between local and upstream. * **The Issue**: This creates "noise" in your parity reports that isn't actually a bug. * **Action**: Update the parity detection logic (or the handler) to selectively ignore the `access_token` field while still verifying that the rest of the JSON structure (expires_in, scope, token_type) matches. #### 4. Large IDs for Other Models (Medium) While we fixed IDs for `recents`, other models like `presets` or `sources` might still use small auto-incrementing integers. * **Action**: Evaluate if other endpoints should also transition to the timestamp-based ID schema to further reduce diff noise. #### 5. Improved Data Persistence (Continuous) Continue the "learning" approach for other services. For example, if we see a new `sourceproviderid` in a Spotify or TuneIn request, we should ensure it is stored and reused. #### 6. Local Reboot & Device State Management (Continuous) Analysis of device reboot logs revealed several data requirements: * **Power-On Details Tracking**: Implemented extraction and persistence of detailed device information (serial numbers, firmware version, product details, and MAC addresses) from the `POST /streaming/support/power_on` request. This data is now stored in the local datastore, improving our ability to respond accurately to subsequent management requests. * **Source Provider Mapping**: Synchronized local source provider IDs and timestamps with upstream data. The `RADIO_BROWSER` provider is included in the public `/streaming/sourceproviders` list to maintain internal functionality while acknowledging it as a parity gap. #### 7. Account Full Response (/full) Structural & Value Parity (In Progress) Based on `_/diffs/diff7/`, several structural and value gaps remain in the `/full` account response: **Remaining Findings:** * **Nested Source Inconsistency in Recents**: The `` element within `` entries still frequently points to a generic fallback (ID `9330201`) instead of the specific source (e.g., Spotify ID `10863533`). * Missing/empty `` at the `` level. * **Values**: * **Empty Device ``**: Locally, the device `` is empty in the response even when available in the datastore or upstream. * **Empty ``**: Local responses have empty `` in presets and recents, whereas upstream has `tracklisturl` or `stationurl`. * `preferredLanguage` mismatch (`en` vs `de`). **Next Implementation Steps (Proposals):** 1. **Fix Device `` Population**: Investigate why `CreateAccountDevice` or `AccountFullToXML` is not correctly returning the device name even if it's synchronized. 2. **Refine Source Association in Recents**: Improve the matching logic in `mapRecentsToFullResponse` to correctly link recents to their specific `ConfiguredSource` (e.g., by matching `sourceid` attribute). 3. **Populate `contentItemType`**: Update the internal models and `SyncFromAccountFull` to correctly extract, persist, and echo back `contentItemType` (e.g., `tracklisturl`). 4. **Handle Account Metadata**: Synchronize `preferredLanguage` from the upstream `/full` response to the local account state.