Rolfsvaag_Datateknikk/Lumi
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity
37dda538b3
Lumi/docs/lumi-companion-transcription.md
Franz Rolfsvaag 37dda538b3 Fix local Companion package and worker setup
2026-07-22 14:34:29 +02:00

5.7 KiB
Raw Blame History

Lumi Companion transcription (experimental)

Delivered foundation

lumi_transcription is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.

The implemented transport path is:

OBS bridge boundary -> same-user companion IPC -> bounded companion queue
-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
-> supervised TranscriptionProvider -> stabilized revision-aware caption
-> WebSocket -> companion delivery adapter -> OBS native caption boundary

The repository contains a pinned native whisper.cpp rolling-window worker and a CPU build/protocol smoke test. It has not yet been benchmarked with a model on the target RTX 3060, packaged as the Windows CUDA runtime, or validated end to end. The repository also does not contain a complete signed OBS bridge installer. Accordingly, this branch does not claim live Twitch acceptance.

Trust and privacy

  • Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
  • Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
  • Device HTTP and WebSocket credentials require HTTPS/WSS. HTTP/WS is accepted only when an administrator downloads the package through an explicit loopback URL and Companion returns to that exact loopback origin, including its port. There is no environment-variable bypass.
  • The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
  • Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
  • Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.

Pairing and installation milestone

  1. Serve Lumi through HTTPS and enable lumi_transcription under Admin > Plugins. Local development may use HTTP only from the exact localhost or loopback URL used to download Companion.
  2. Use Download Companion in Admin or Plugins > Transcription. Lumi downloads and checksum-verifies the pinned Windows artifact, adds a short-lived single-use pairing file, and returns a private ZIP.
  3. On the Windows streaming computer, extract the complete ZIP and start Lumi.Companion.App.exe within 15 minutes. Companion discovers the adjacent pairing file, exchanges it once, stores the credential with current-user DPAPI, and removes the pairing file after success.
  4. Install a pinned runtime/model only after explicit confirmation. small.en is recommended; small.en-q5_1 and base.en are fallbacks. Every artifact is checksum-verified before install.
  5. Build the supervised worker with plugins/lumi_transcription/scripts/build-worker.ps1. Lumi discovers the installed CUDA worker first and the CPU worker second. LUMI_TRANSCRIPTION_WORKER remains an explicit override for nonstandard installations.

The Avalonia tray UI, self-contained paired ZIP, shared Lumi settings shell, Admin summary, and host-side model download/load controls now exist. The ZIP is not code-signed, so Windows may warn. The normal signed installer, managed bridge install/repair, live OBS source enumeration, and measured model benchmark wizard are not complete yet.

Operation and recovery

Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.

Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.

Diagnostics

Use the compact Lumi Companion section on Admin for connection, device, inference, session, and package health. Open the shared-shell Transcription settings page for provider, model, device, and log details. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in companion/docs/performance-acceptance-template.md.

Known limitations

  • The managed .NET solution and self-contained Windows x64 application have been compiled in the development environment; the OBS SDK integration still requires its target toolchain and acceptance host.
  • No real whisper.cpp streaming worker has been integrated or benchmarked.
  • The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
  • Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
  • The downloadable bundle is an unsigned experimental self-contained ZIP, not the final signed installer.
  • Bridge repair, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.

See docs/adr/0001-companion-transcription-boundaries.md, protocol/companion-protocol-v1.md, and companion/docs/obs-native-caption-compatibility-spike.md.