57 lines
5.5 KiB
Markdown
57 lines
5.5 KiB
Markdown
# Lumi Companion transcription (experimental)
|
|
|
|
## Delivered foundation
|
|
|
|
`lumi_transcription` is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.
|
|
|
|
The implemented transport path is:
|
|
|
|
```text
|
|
OBS bridge boundary -> same-user companion IPC -> bounded companion queue
|
|
-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
|
|
-> supervised TranscriptionProvider -> stabilized revision-aware caption
|
|
-> WebSocket -> companion delivery adapter -> OBS native caption boundary
|
|
```
|
|
|
|
The repository contains a pinned native whisper.cpp rolling-window worker and a CPU build/protocol smoke test. It has not yet been benchmarked with a model on the target RTX 3060, packaged as the Windows CUDA runtime, or validated end to end. The repository also does not contain a complete signed OBS bridge installer. Accordingly, this branch does not claim live Twitch acceptance.
|
|
|
|
## Trust and privacy
|
|
|
|
- Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
|
|
- Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
|
|
- Device HTTP and WebSocket credentials require HTTPS/WSS. HTTP/WS is accepted only when an administrator downloads the package through an explicit loopback URL and Companion returns to that exact loopback origin, including its port. There is no environment-variable bypass.
|
|
- The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
|
|
- Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
|
|
- Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.
|
|
|
|
## Pairing and installation milestone
|
|
|
|
1. Serve Lumi through HTTPS and enable `lumi_transcription` under Admin > Plugins. Local development may use HTTP only from the exact `localhost` or loopback URL used to download Companion.
|
|
2. Use **Download Companion** in Admin or Plugins > Transcription. Lumi downloads and checksum-verifies the pinned Windows artifact, adds a short-lived single-use pairing file, and returns a private ZIP.
|
|
3. On the Windows streaming computer, extract the complete ZIP and start `Lumi.Companion.App.exe` within 15 minutes. Companion discovers the adjacent pairing file, exchanges it once, stores the credential with current-user DPAPI, and removes the pairing file after success.
|
|
4. Install a pinned runtime/model only after explicit confirmation. `small.en` is recommended; `small.en-q5_1` and `base.en` are fallbacks. Every artifact is checksum-verified before install.
|
|
5. Configure `LUMI_TRANSCRIPTION_WORKER` with the supervised streaming-worker executable once that worker is built for the target host.
|
|
|
|
The Avalonia tray UI, self-contained paired ZIP, shared Lumi settings shell, Admin summary, and host-side model download/load controls now exist. The ZIP is not code-signed, so Windows may warn. The normal signed installer, managed bridge install/repair, live OBS source enumeration, and measured model benchmark wizard are not complete yet.
|
|
|
|
## Operation and recovery
|
|
|
|
Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.
|
|
|
|
Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.
|
|
|
|
## Diagnostics
|
|
|
|
Use the compact Lumi Companion section on Admin for connection, device, inference, session, and package health. Open the shared-shell Transcription settings page for provider, model, device, and log details. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in `companion/docs/performance-acceptance-template.md`.
|
|
|
|
## Known limitations
|
|
|
|
- The managed .NET solution and self-contained Windows x64 application have been compiled in the development environment; the OBS SDK integration still requires its target toolchain and acceptance host.
|
|
- No real whisper.cpp streaming worker has been integrated or benchmarked.
|
|
- The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
|
|
- Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
|
|
- The downloadable bundle is an unsigned experimental self-contained ZIP, not the final signed installer.
|
|
- Bridge repair, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.
|
|
|
|
See `docs/adr/0001-companion-transcription-boundaries.md`, `protocol/companion-protocol-v1.md`, and `companion/docs/obs-native-caption-compatibility-spike.md`.
|