Rolfsvaag_Datateknikk/Lumi
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity
655547938a
Lumi/docs/lumi-companion-transcription.md

5.3 KiB
Raw Blame History

Lumi Companion transcription

lumi_transcription is a separately disableable Lumi plugin. It owns paired devices, session state, model/runtime artifacts, worker supervision, caption stabilization, delivery, diagnostics, tests, and WebUI routes. Core supplies the shared Companion authentication and versioned WebSocket boundary.

Runtime path

selected OBS source -> same-user Companion IPC -> bounded audio queue
-> authenticated TLS WebSocket -> bounded Lumi session
-> supervised server-hosted whisper.cpp worker -> stabilized caption revisions
-> Companion -> OBS native caption boundary

Speech recognition never runs on the streaming computer. Raw audio is kept in bounded memory and is not logged or written to disk. Lumi prefers the published CUDA worker on an NVIDIA RTX 3060 or newer and retains the CPU worker as a portable fallback.

Install and pair

  1. Enable Lumi Transcription under Admin > Plugins.
  2. Use Download Companion from the Admin dashboard or Transcription page.
  3. Extract the private ZIP and run Lumi.Companion-Setup.exe within 15 minutes.
  4. Complete the per-user setup and install or repair the managed OBS integration while OBS is closed.
  5. Select the OBS microphone in Companion.
  6. Run the short voice-free full-path check. A successful result remains valid until a relevant source, integration, model, or host setting changes.
  7. Use the dedicated transcription test when measuring real speech accuracy, confidence, latency, and input level.

The Companion Transcription page and tray menu expose a persistent Generate and include captions switch. When it is off, Companion does not queue or send microphone audio for recognition, does not start a live transcription session, and removes caption inclusion from a private stream test. The video-only private test remains available. Path and benchmark tests remain disabled until captions are turned back on.

Production device HTTP and WebSocket traffic requires HTTPS/WSS. Plain HTTP/WebSocket is allowed only for a device paired from the exact matching loopback Lumi origin and only while both sides remain on loopback.

Lumi recognizes HTTPS terminated by a reverse proxy on loopback or an isolated private proxy network. Forwarded protocol headers received directly from public addresses are ignored, so only a trusted proxy can mark a request as HTTPS.

Models and workers

The Admin page installs checksum-pinned model and worker artifacts only after explicit confirmation. small.en is the recommended model; small.en-q5_1 and base.en remain available as lower-resource alternatives. The worker is supervised, restarts are bounded, and an explicit LUMI_TRANSCRIPTION_WORKER override remains available for nonstandard installations.

The native worker source is under plugins/lumi_transcription/backend/transcription/worker-native. Release workers are built with plugins/lumi_transcription/scripts/build-worker.ps1.

Operation and recovery

  • Live sessions require an active OBS stream. Tests use simulated delivery and never publish captions to Twitch.
  • Disconnects pause delivery and retain the session/model for a bounded grace period so a short reconnect can resume safely. Companion retries a paired Lumi host silently with bounded backoff and resumes its authenticated runtime state after reconnecting.
  • Manual benchmark completion waits for the final confidence-bearing caption; silence ends a test after ten seconds.
  • Device revocation takes effect on the next authenticated request or connection.
  • Disabling the plugin unregisters routes, closes clients and sessions, stops the worker, and leaves unrelated plugins running.

The Admin summary reports Companion, device, inference, and session health. Detailed benchmark records remain inspectable for one hour and include word/phrase latency and confidence views.

OBS Bridge and Stream Testing

The same paired Companion identity and same-user OBS Bridge are also used for private Stream Testing. Lumi issues the expiring RTMP/RTMPS destination from the validated pairing hostname; plugins do not re-authenticate and Companion does not invent or accept another host. Before changing OBS, Companion stores the complete service snapshot in its DPAPI-protected recovery record. Every terminal path uses that record until OBS confirms the exact service has been restored.

MediaMTX is installed, verified, configured, and supervised by Lumi Core on the server. It is not shipped in Companion and does not run inference or transcode media. See stream-testing.md for transport, firewall, managed-runtime, playback, and recovery details.

Verification

npm run verify:transcription
dotnet build companion/Lumi.Companion.sln -c Release -p:EnableWindowsTargeting=true

Target-machine acceptance can be recorded with companion/docs/performance-acceptance-template.md.

Remaining release limitation

The Windows installer is checksum-pinned but not yet code-signed, so Windows may show an unknown-publisher warning. The release includes its licence, privacy notice, third-party notices, and corresponding source for the GPL-licensed OBS bridge.

See docs/adr/0001-companion-transcription-boundaries.md, protocol/companion-protocol-v1.md, and companion/docs/obs-native-caption-compatibility-spike.md.