Rolfsvaag_Datateknikk/Lumi
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity
29cd4a0365
Lumi/docs/lumi-companion-transcription.md
Franz Rolfsvaag 29cd4a0365 Add experimental Companion and OBS bridge boundaries
2026-07-22 11:02:04 +02:00

4.7 KiB
Raw Blame History

Lumi Companion transcription (experimental)

Delivered foundation

lumi_transcription is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.

The implemented transport path is:

OBS bridge boundary -> same-user companion IPC -> bounded companion queue
-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
-> supervised TranscriptionProvider -> stabilized revision-aware caption
-> WebSocket -> companion delivery adapter -> OBS native caption boundary

The repository does not yet contain a validated whisper streaming worker or a complete signed OBS bridge installer. Accordingly, the admin page reports inference setup as required and this branch does not claim live Twitch acceptance.

Trust and privacy

  • Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
  • Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
  • Device HTTP and WebSocket credentials require HTTPS/WSS. Insecure transport is available only from localhost when LUMI_COMPANION_DEV_ALLOW_INSECURE=1 is explicitly set.
  • The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
  • Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
  • Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.

Pairing and installation milestone

  1. Serve Lumi through HTTPS and enable lumi_transcription under Admin > Plugins.
  2. Open Plugins > Transcription and create a pairing package. It is a bootstrap package for the generic companion, not a credential that should be shared or committed.
  3. On the Windows streaming computer, build the .NET 8 projects under companion/ and run Lumi.Companion.App pair <package> for the current protocol milestone.
  4. Install a pinned runtime/model only after explicit confirmation. small.en is recommended; small.en-q5_1 and base.en are fallbacks. Every artifact is checksum-verified before install.
  5. Configure LUMI_TRANSCRIPTION_WORKER with the supervised streaming-worker executable once that worker is built for the target host.

The normal signed installer, Avalonia tray UI, managed bridge install/repair, source selector, and model benchmark wizard are not complete yet.

Operation and recovery

Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.

Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.

Diagnostics

Use the Transcription admin page for provider, model, device, and log health. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in companion/docs/performance-acceptance-template.md.

Known limitations

  • The .NET SDK and OBS SDK are not available in the development environment used for this milestone, so those projects have not been compiled here.
  • No real whisper.cpp streaming worker has been integrated or benchmarked.
  • The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
  • Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
  • The companion shell is a protocol/bootstrap executable, not yet the Avalonia tray application.
  • Installer signing, bridge repair, auto-start, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.

See docs/adr/0001-companion-transcription-boundaries.md, protocol/companion-protocol-v1.md, and companion/docs/obs-native-caption-compatibility-spike.md.