57 lines
5.7 KiB
Markdown
57 lines
5.7 KiB
Markdown
# Lumi Companion transcription (experimental)
|
|
|
|
## Delivered foundation
|
|
|
|
`lumi_transcription` is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.
|
|
|
|
The implemented transport path is:
|
|
|
|
```text
|
|
OBS bridge boundary -> same-user companion IPC -> bounded companion queue
|
|
-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
|
|
-> supervised TranscriptionProvider -> stabilized revision-aware caption
|
|
-> WebSocket -> companion delivery adapter -> OBS native caption boundary
|
|
```
|
|
|
|
The repository contains a pinned native whisper.cpp rolling-window worker and a CPU build/protocol smoke test. It has not yet been benchmarked with a model on the target RTX 3060, packaged as the Windows CUDA runtime, or validated end to end. The repository also does not contain a complete signed OBS bridge installer. Accordingly, this branch does not claim live Twitch acceptance.
|
|
|
|
## Trust and privacy
|
|
|
|
- Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
|
|
- Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
|
|
- Device HTTP and WebSocket credentials require HTTPS/WSS. HTTP/WS is accepted only when an administrator downloads the package through an explicit loopback URL and Companion returns to that exact loopback origin, including its port. There is no environment-variable bypass.
|
|
- The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
|
|
- Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
|
|
- Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.
|
|
|
|
## Pairing and installation milestone
|
|
|
|
1. Serve Lumi through HTTPS and enable `lumi_transcription` under Admin > Plugins. Local development may use HTTP only from the exact `localhost` or loopback URL used to download Companion.
|
|
2. Use **Download Companion** in Admin or Plugins > Transcription. Lumi downloads and checksum-verifies the pinned Windows artifact, adds a short-lived single-use pairing file, and returns a private ZIP.
|
|
3. On the Windows streaming computer, extract the complete ZIP and start `Lumi.Companion-Setup.exe` within 15 minutes. Setup installs the app under the current user's Local AppData, imports the adjacent pairing file, and launches Companion. Companion exchanges the pairing token once, stores the credential with current-user DPAPI, and removes the imported pairing file after success.
|
|
4. Install a pinned runtime/model only after explicit confirmation. `small.en` is recommended; `small.en-q5_1` and `base.en` are fallbacks. Every artifact is checksum-verified before install.
|
|
5. Build the supervised worker with `plugins/lumi_transcription/scripts/build-worker.ps1`. Lumi discovers the installed CUDA worker first and the CPU worker second. `LUMI_TRANSCRIPTION_WORKER` remains an explicit override for nonstandard installations.
|
|
|
|
The Avalonia tray UI, per-user Windows installer, shared Lumi settings shell, Admin summary, host-side model controls, managed OBS bridge install/repair, live source enumeration, and measured benchmark UI now exist. The installer is not code-signed yet, so Windows may warn.
|
|
|
|
## Operation and recovery
|
|
|
|
Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.
|
|
|
|
Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.
|
|
|
|
## Diagnostics
|
|
|
|
Use the compact Lumi Companion section on Admin for connection, device, inference, session, and package health. Open the shared-shell Transcription settings page for provider, model, device, and log details. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in `companion/docs/performance-acceptance-template.md`.
|
|
|
|
## Known limitations
|
|
|
|
- The managed .NET solution and self-contained Windows x64 application have been compiled in the development environment; the OBS SDK integration still requires its target toolchain and acceptance host.
|
|
- No real whisper.cpp streaming worker has been integrated or benchmarked.
|
|
- The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
|
|
- Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
|
|
- The downloadable bundle contains an unsigned experimental per-user installer; production code signing is still required.
|
|
- Bridge repair, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.
|
|
|
|
See `docs/adr/0001-companion-transcription-boundaries.md`, `protocol/companion-protocol-v1.md`, and `companion/docs/obs-native-caption-compatibility-spike.md`.
|