57 lines
4.7 KiB
Markdown
57 lines
4.7 KiB
Markdown
# Lumi Companion transcription (experimental)
|
|
|
|
## Delivered foundation
|
|
|
|
`lumi_transcription` is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.
|
|
|
|
The implemented transport path is:
|
|
|
|
```text
|
|
OBS bridge boundary -> same-user companion IPC -> bounded companion queue
|
|
-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
|
|
-> supervised TranscriptionProvider -> stabilized revision-aware caption
|
|
-> WebSocket -> companion delivery adapter -> OBS native caption boundary
|
|
```
|
|
|
|
The repository does not yet contain a validated whisper streaming worker or a complete signed OBS bridge installer. Accordingly, the admin page reports inference setup as required and this branch does not claim live Twitch acceptance.
|
|
|
|
## Trust and privacy
|
|
|
|
- Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
|
|
- Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
|
|
- Device HTTP and WebSocket credentials require HTTPS/WSS. Insecure transport is available only from localhost when `LUMI_COMPANION_DEV_ALLOW_INSECURE=1` is explicitly set.
|
|
- The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
|
|
- Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
|
|
- Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.
|
|
|
|
## Pairing and installation milestone
|
|
|
|
1. Serve Lumi through HTTPS and enable `lumi_transcription` under Admin > Plugins.
|
|
2. Open Plugins > Transcription and create a pairing package. It is a bootstrap package for the generic companion, not a credential that should be shared or committed.
|
|
3. On the Windows streaming computer, build the .NET 8 projects under `companion/` and run `Lumi.Companion.App pair <package>` for the current protocol milestone.
|
|
4. Install a pinned runtime/model only after explicit confirmation. `small.en` is recommended; `small.en-q5_1` and `base.en` are fallbacks. Every artifact is checksum-verified before install.
|
|
5. Configure `LUMI_TRANSCRIPTION_WORKER` with the supervised streaming-worker executable once that worker is built for the target host.
|
|
|
|
The normal signed installer, Avalonia tray UI, managed bridge install/repair, source selector, and model benchmark wizard are not complete yet.
|
|
|
|
## Operation and recovery
|
|
|
|
Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.
|
|
|
|
Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.
|
|
|
|
## Diagnostics
|
|
|
|
Use the Transcription admin page for provider, model, device, and log health. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in `companion/docs/performance-acceptance-template.md`.
|
|
|
|
## Known limitations
|
|
|
|
- The .NET SDK and OBS SDK are not available in the development environment used for this milestone, so those projects have not been compiled here.
|
|
- No real whisper.cpp streaming worker has been integrated or benchmarked.
|
|
- The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
|
|
- Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
|
|
- The companion shell is a protocol/bootstrap executable, not yet the Avalonia tray application.
|
|
- Installer signing, bridge repair, auto-start, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.
|
|
|
|
See `docs/adr/0001-companion-transcription-boundaries.md`, `protocol/companion-protocol-v1.md`, and `companion/docs/obs-native-caption-compatibility-spike.md`.
|