Rolfsvaag_Datateknikk/Lumi
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 22 Wiki Activity
2f31390602
Lumi/docs/lumi-companion-transcription.md
Franz Rolfsvaag 2f31390602 Build Lumi Companion shell and whisper worker
2026-07-22 11:39:05 +02:00

4.9 KiB
Raw Blame History

Lumi Companion transcription (experimental)

Delivered foundation

lumi_transcription is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.

The implemented transport path is:

OBS bridge boundary -> same-user companion IPC -> bounded companion queue
-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
-> supervised TranscriptionProvider -> stabilized revision-aware caption
-> WebSocket -> companion delivery adapter -> OBS native caption boundary

The repository contains a pinned native whisper.cpp rolling-window worker and a CPU build/protocol smoke test. It has not yet been benchmarked with a model on the target RTX 3060, packaged as the Windows CUDA runtime, or validated end to end. The repository also does not contain a complete signed OBS bridge installer. Accordingly, this branch does not claim live Twitch acceptance.

Trust and privacy

  • Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
  • Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
  • Device HTTP and WebSocket credentials require HTTPS/WSS. Insecure transport is available only from localhost when LUMI_COMPANION_DEV_ALLOW_INSECURE=1 is explicitly set.
  • The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
  • Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
  • Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.

Pairing and installation milestone

  1. Serve Lumi through HTTPS and enable lumi_transcription under Admin > Plugins.
  2. Open Plugins > Transcription and create a pairing package. It is a bootstrap package for the generic companion, not a credential that should be shared or committed.
  3. On the Windows streaming computer, build companion/Lumi.Companion.sln, start Lumi.Companion.App, and choose the package from the Overview or Connection page.
  4. Install a pinned runtime/model only after explicit confirmation. small.en is recommended; small.en-q5_1 and base.en are fallbacks. Every artifact is checksum-verified before install.
  5. Configure LUMI_TRANSCRIPTION_WORKER with the supervised streaming-worker executable once that worker is built for the target host.

The Avalonia tray UI and host-side model download/load controls now exist. The normal signed installer, managed bridge install/repair, live OBS source enumeration, and measured model benchmark wizard are not complete yet.

Operation and recovery

Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.

Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.

Diagnostics

Use the Transcription admin page for provider, model, device, and log health. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in companion/docs/performance-acceptance-template.md.

Known limitations

  • The .NET SDK and OBS SDK are not available in the development environment used for this milestone, so those projects have not been compiled here.
  • No real whisper.cpp streaming worker has been integrated or benchmarked.
  • The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
  • Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
  • The companion shell is a protocol/bootstrap executable, not yet the Avalonia tray application.
  • Installer signing, bridge repair, auto-start, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.

See docs/adr/0001-companion-transcription-boundaries.md, protocol/companion-protocol-v1.md, and companion/docs/obs-native-caption-compatibility-spike.md.