From 29cd4a0365240571c6641f974f82eb2802d185ec Mon Sep 17 00:00:00 2001 From: Franz Rolfsvaag Date: Wed, 22 Jul 2026 11:02:04 +0200 Subject: [PATCH] Add experimental Companion and OBS bridge boundaries --- README.md | 5 + TODO.md | 22 +++++ companion/README.md | 9 ++ .../obs-native-caption-compatibility-spike.md | 14 +++ .../docs/performance-acceptance-template.md | 23 +++++ companion/native/obs-bridge/CMakeLists.txt | 10 ++ .../obs-bridge/include/bounded_spsc_queue.hpp | 32 +++++++ companion/native/obs-bridge/src/plugin.cpp | 58 ++++++++++++ .../Lumi.Companion.Transcription.csproj | 1 + .../ObsBridgePipe.cs | 73 +++++++++++++++ .../src/Lumi.Companion.App/CompanionState.cs | 10 ++ .../Lumi.Companion.App.csproj | 1 + companion/src/Lumi.Companion.App/Program.cs | 33 +++++++ .../Lumi.Companion.Core/CompanionSocket.cs | 93 +++++++++++++++++++ .../Lumi.Companion.Core.csproj | 1 + .../src/Lumi.Companion.Core/PairingClient.cs | 30 ++++++ .../SecureCredentialStore.cs | 33 +++++++ .../Lumi.Companion.PluginHost.csproj | 1 + .../Lumi.Companion.PluginHost/PluginWorker.cs | 23 +++++ .../Lumi.Companion.Protocol.csproj | 1 + .../src/Lumi.Companion.Protocol/ProtocolV1.cs | 66 +++++++++++++ ...0001-companion-transcription-boundaries.md | 13 +++ docs/lumi-companion-transcription.md | 56 +++++++++++ 23 files changed, 608 insertions(+) create mode 100644 companion/README.md create mode 100644 companion/docs/obs-native-caption-compatibility-spike.md create mode 100644 companion/docs/performance-acceptance-template.md create mode 100644 companion/native/obs-bridge/CMakeLists.txt create mode 100644 companion/native/obs-bridge/include/bounded_spsc_queue.hpp create mode 100644 companion/native/obs-bridge/src/plugin.cpp create mode 100644 companion/plugins/Lumi.Companion.Transcription/Lumi.Companion.Transcription.csproj create mode 100644 companion/plugins/Lumi.Companion.Transcription/ObsBridgePipe.cs create mode 100644 companion/src/Lumi.Companion.App/CompanionState.cs create mode 100644 companion/src/Lumi.Companion.App/Lumi.Companion.App.csproj create mode 100644 companion/src/Lumi.Companion.App/Program.cs create mode 100644 companion/src/Lumi.Companion.Core/CompanionSocket.cs create mode 100644 companion/src/Lumi.Companion.Core/Lumi.Companion.Core.csproj create mode 100644 companion/src/Lumi.Companion.Core/PairingClient.cs create mode 100644 companion/src/Lumi.Companion.Core/SecureCredentialStore.cs create mode 100644 companion/src/Lumi.Companion.PluginHost/Lumi.Companion.PluginHost.csproj create mode 100644 companion/src/Lumi.Companion.PluginHost/PluginWorker.cs create mode 100644 companion/src/Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj create mode 100644 companion/src/Lumi.Companion.Protocol/ProtocolV1.cs create mode 100644 docs/adr/0001-companion-transcription-boundaries.md create mode 100644 docs/lumi-companion-transcription.md diff --git a/README.md b/README.md index 3099904..a1195f7 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,11 @@ You can set these in `.env` or change role IDs in **Admin → Settings**. Use **Admin → Plugins** to install, enable, update, or uninstall plugins. You can also create a local plugin from the WebUI. +The experimental `experimental-companion` branch includes the independent Lumi +Companion transcription foundation. Its current scope, trust boundaries, setup, +and unverified target-machine work are documented in +[`docs/lumi-companion-transcription.md`](docs/lumi-companion-transcription.md). + ## Updates and recovery Use **Admin → Updates** for version-aware core and plugin updates. Lumi reads diff --git a/TODO.md b/TODO.md index d9df07f..c59bf3b 100644 --- a/TODO.md +++ b/TODO.md @@ -2,6 +2,28 @@ This file tracks larger Lumi work that cannot safely be completed in one pass. Keep pending work under the relevant category and move completed items to the Done section with a short note. +## Lumi Companion transcription — experimental-companion (2026-07-22) + +Foundation implemented locally: independent `lumi_transcription` plugin; generic +core WebSocket-upgrade capability; single-use pairing and revocable devices; +TLS-only device transport; bounded PCM/session queues; revision-safe settings; +provider/delivery interfaces; worker supervision; caption stabilization; pinned +whisper.cpp/model manifests; short-lived JSONL diagnostics; protocol schemas; +and .NET/native companion boundaries with focused verification. + +Release-blocking work remains: + +- Build the real streaming whisper.cpp worker, install it through the manifest, + and benchmark `small.en`, quantized `small.en`, and `base.en` on the RTX 3060. +- Complete the Avalonia tray application, signed installer, DPAPI migration UX, + managed OBS bridge install/repair/uninstall, and single-instance/live-quit flow. +- Complete native selected-source capture, nested Program-scene evaluation, + resampling/IPC, bridge health, and source rename/missing recovery. +- Run the OBS 31+/Twitch compatibility spike and prove toggleable closed captions; + tune replacement/display duration from player behavior without open-caption fallback. +- Add setup/test/conflict-resolution UI and live settings synchronization to the + companion, then run the target-topology failure and performance matrix. + ## Remaining DesignMotionHQ UX work — experimental-ux checkpoint (2026-07-22) The shared UX foundation and representative settings, navigation, theme, command, diff --git a/companion/README.md b/companion/README.md new file mode 100644 index 0000000..7124691 --- /dev/null +++ b/companion/README.md @@ -0,0 +1,9 @@ +# Lumi Companion (experimental transcription milestone) + +This directory is the single Lumi Companion product boundary. The current milestone supplies the versioned protocol client, Windows credential protection, bounded outbound transport, plugin-process supervision, and same-user OBS bridge IPC. The Avalonia tray shell, installer, and signed native bridge package remain target-machine work and are not represented as complete. + +Build prerequisites: .NET 8 SDK on Windows x64. The native bridge additionally requires CMake, Visual Studio C++ tools, and the OBS 31+ SDK. + +The app accepts a `.lumi-pairing.json` bootstrap package. It exchanges the embedded token once and stores the returned device credential with Windows DPAPI. Do not commit bootstrap packages, credentials, generated installers, build output, or logs. + +The companion never performs ASR in this MVP. Audio is normalized to 16 kHz mono signed 16-bit PCM, held in bounded memory, and sent to the paired Lumi host over TLS WebSockets. The OBS bridge talks only to the companion over a same-user named pipe. diff --git a/companion/docs/obs-native-caption-compatibility-spike.md b/companion/docs/obs-native-caption-compatibility-spike.md new file mode 100644 index 0000000..ac36f92 --- /dev/null +++ b/companion/docs/obs-native-caption-compatibility-spike.md @@ -0,0 +1,14 @@ +# OBS 31+ native caption compatibility spike + +Implementation source inspection confirms OBS 31 exposes `obs_output_output_caption_text2` and `obs_frontend_get_streaming_output`. The bridge calls the caption API only from its IPC worker against the active streaming output; it never calls it from an audio/render callback. + +This is not Twitch acceptance evidence. The following target-topology test remains mandatory before live delivery can be labelled ready: + +1. Build and install the signed bridge through the companion on Windows 11 with OBS 31+. +2. Start a private Twitch stream using the production service/output configuration. +3. Submit incrementally revised text with 1.5–3 second display durations. +4. Verify the Twitch web player and at least one mobile client expose a user-toggleable CC control. +5. Record whether OBS/Twitch queue, replace, or drop revisions and tune the adapter to prevent stale caption backlog. +6. Capture OBS CPU, render/encoder missed frames, audio dropouts, bridge queue drops, and network throughput before/after. + +Until this passes, the companion must allow the exact simulated delivery stream in Test mode but refuse to describe Twitch closed captions as verified. It must never substitute a baked-in open-caption overlay. diff --git a/companion/docs/performance-acceptance-template.md b/companion/docs/performance-acceptance-template.md new file mode 100644 index 0000000..92f26ff --- /dev/null +++ b/companion/docs/performance-acceptance-template.md @@ -0,0 +1,23 @@ +# Transcription target-topology acceptance report + +Status: not run. Do not replace blank measurements with estimates. + +Topology: Windows 11 streaming PC / OBS version ___ / Windows Server 2022 Lumi host / RTX 3060 driver ___ / wired LAN / Twitch test URL or evidence reference ___. + +| Measurement | Before | Live transcription | Result | +| --- | ---: | ---: | --- | +| Companion CPU / working set | ___ | ___ | ___ | +| OBS CPU | ___ | ___ | ___ | +| OBS render missed frames | ___ | ___ | ___ | +| OBS encoder missed frames | ___ | ___ | ___ | +| Audio dropouts | ___ | ___ | ___ | +| Bridge queue drops | ___ | ___ | ___ | +| Network throughput | ___ | ___ | ___ | +| Whisper CPU / GPU / VRAM / RAM | n/a | ___ | ___ | +| Model load time | n/a | ___ | ___ | +| Decode latency / real-time factor | n/a | ___ | ___ | +| First stable caption p50 / p95 | n/a | ___ | ___ | +| End-to-end caption p50 / p95 | n/a | ___ | ___ | +| Maximum queue depth / backlog growth | n/a | ___ | ___ | + +Confirm: test-mode real path ___; Twitch player toggleable CC ___; stable sustained speech ___; no stale dump after network flap ___; 30-second stream grace/resume ___; raw audio absent from disk ___; fallback activation evidence ___. diff --git a/companion/native/obs-bridge/CMakeLists.txt b/companion/native/obs-bridge/CMakeLists.txt new file mode 100644 index 0000000..9753dcd --- /dev/null +++ b/companion/native/obs-bridge/CMakeLists.txt @@ -0,0 +1,10 @@ +cmake_minimum_required(VERSION 3.28) +project(lumi-obs-bridge VERSION 0.1.0 LANGUAGES CXX) +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +find_package(libobs REQUIRED) +find_package(obs-frontend-api REQUIRED) +add_library(lumi-obs-bridge MODULE src/plugin.cpp) +target_include_directories(lumi-obs-bridge PRIVATE include) +target_link_libraries(lumi-obs-bridge PRIVATE OBS::libobs OBS::obs-frontend-api) +set_target_properties(lumi-obs-bridge PROPERTIES PREFIX "") diff --git a/companion/native/obs-bridge/include/bounded_spsc_queue.hpp b/companion/native/obs-bridge/include/bounded_spsc_queue.hpp new file mode 100644 index 0000000..7afb47b --- /dev/null +++ b/companion/native/obs-bridge/include/bounded_spsc_queue.hpp @@ -0,0 +1,32 @@ +#pragma once +#include +#include +#include +#include + +namespace lumi { +template class bounded_spsc_queue { + static_assert(Capacity > 1); + std::array values_{}; + alignas(64) std::atomic head_{0}; + alignas(64) std::atomic tail_{0}; + std::atomic dropped_{0}; +public: + bool try_push(T value) noexcept { + const auto head = head_.load(std::memory_order_relaxed); + const auto next = (head + 1) % Capacity; + if (next == tail_.load(std::memory_order_acquire)) { dropped_.fetch_add(1, std::memory_order_relaxed); return false; } + values_[head] = std::move(value); + head_.store(next, std::memory_order_release); + return true; + } + std::optional try_pop() noexcept { + const auto tail = tail_.load(std::memory_order_relaxed); + if (tail == head_.load(std::memory_order_acquire)) return std::nullopt; + T value = std::move(values_[tail]); + tail_.store((tail + 1) % Capacity, std::memory_order_release); + return value; + } + std::uint64_t dropped() const noexcept { return dropped_.load(std::memory_order_relaxed); } +}; +} diff --git a/companion/native/obs-bridge/src/plugin.cpp b/companion/native/obs-bridge/src/plugin.cpp new file mode 100644 index 0000000..852c25c --- /dev/null +++ b/companion/native/obs-bridge/src/plugin.cpp @@ -0,0 +1,58 @@ +#include +#include +#include +#include + +OBS_DECLARE_MODULE() +OBS_MODULE_USE_DEFAULT_LOCALE("lumi-obs-bridge", "en-US") + +const char *obs_module_description(void) +{ + return "Companion-managed Lumi audio and native-caption bridge"; +} + +static bool enumerate_source(void *, obs_source_t *source) +{ + if (!source) return true; + blog(LOG_DEBUG, "[Lumi Companion] OBS source available: uuid=%s name=%s active=%s", + obs_source_get_uuid(source), obs_source_get_name(source), obs_source_active(source) ? "true" : "false"); + return true; +} + +static void frontend_event(enum obs_frontend_event event, void *) +{ + if (event == OBS_FRONTEND_EVENT_STREAMING_STARTED || event == OBS_FRONTEND_EVENT_STREAMING_STOPPED || + event == OBS_FRONTEND_EVENT_RECORDING_STARTED || event == OBS_FRONTEND_EVENT_RECORDING_STOPPED) { + blog(LOG_INFO, "[Lumi Companion] OBS state changed: streaming=%s recording=%s", + obs_frontend_streaming_active() ? "true" : "false", obs_frontend_recording_active() ? "true" : "false"); + } +} + +// Called only by the bridge IPC worker, never by an OBS audio/render callback. +// Returning false keeps delivery failure isolated from the active OBS output. +static bool output_caption(const std::string &text, double display_seconds) +{ + if (!obs_frontend_streaming_active() || text.empty()) return false; + obs_output_t *output = obs_frontend_get_streaming_output(); + if (!output) return false; + obs_output_output_caption_text2(output, text.c_str(), display_seconds); + obs_output_release(output); + return true; +} + +bool obs_module_load(void) +{ + const char *version = obs_get_version_string(); + const int major = version ? std::atoi(version) : 0; + if (major < 31) blog(LOG_WARNING, "[Lumi Companion] OBS %s is outside the supported 31+ range", version ? version : "unknown"); + obs_frontend_add_event_callback(frontend_event, nullptr); + obs_enum_sources(enumerate_source, nullptr); + blog(LOG_INFO, "[Lumi Companion] bridge loaded; waiting for the companion-managed IPC transport"); + return true; +} + +void obs_module_unload(void) +{ + obs_frontend_remove_event_callback(frontend_event, nullptr); + blog(LOG_INFO, "[Lumi Companion] bridge unloaded"); +} diff --git a/companion/plugins/Lumi.Companion.Transcription/Lumi.Companion.Transcription.csproj b/companion/plugins/Lumi.Companion.Transcription/Lumi.Companion.Transcription.csproj new file mode 100644 index 0000000..892b685 --- /dev/null +++ b/companion/plugins/Lumi.Companion.Transcription/Lumi.Companion.Transcription.csproj @@ -0,0 +1 @@ +net8.0-windowsenableenable diff --git a/companion/plugins/Lumi.Companion.Transcription/ObsBridgePipe.cs b/companion/plugins/Lumi.Companion.Transcription/ObsBridgePipe.cs new file mode 100644 index 0000000..bff2baf --- /dev/null +++ b/companion/plugins/Lumi.Companion.Transcription/ObsBridgePipe.cs @@ -0,0 +1,73 @@ +using System.Buffers.Binary; +using System.IO.Pipes; +using System.Text.Json; +using Lumi.Companion.Protocol; + +namespace Lumi.Companion.Transcription; + +public sealed class ObsBridgePipe : IAsyncDisposable +{ + private static readonly HashSet AllowedTypes = ["hello", "source_list", "source_state", "obs_state", "health"]; + private readonly string _pipeName; + private readonly Func _onMessage; + private readonly Func, Task> _onAudio; + private readonly CancellationTokenSource _lifetime = new(); + private Task? _loop; + + public ObsBridgePipe(string userSidHash, Func onMessage, Func, Task> onAudio) + { + _pipeName = $"Lumi.Companion.ObsBridge.v1.{userSidHash}"; + _onMessage = onMessage; _onAudio = onAudio; + } + public void Start() => _loop ??= Task.Run(() => ListenAsync(_lifetime.Token)); + private async Task ListenAsync(CancellationToken cancellationToken) + { + while (!cancellationToken.IsCancellationRequested) + { + await using var pipe = new NamedPipeServerStream(_pipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Byte, + PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly, 65536, 65536); + await pipe.WaitForConnectionAsync(cancellationToken); + try { await ReadConnectionAsync(pipe, cancellationToken); } + catch (Exception) when (!cancellationToken.IsCancellationRequested) { /* reconnect without taking down the shell */ } + } + } + private async Task ReadConnectionAsync(Stream stream, CancellationToken cancellationToken) + { + var lengthBytes = new byte[4]; + while (!cancellationToken.IsCancellationRequested) + { + await ReadExactlyAsync(stream, lengthBytes, cancellationToken); + var length = BinaryPrimitives.ReadUInt32LittleEndian(lengthBytes); + if (length is 0 or > ProtocolV1.MaxJsonBytes) throw new InvalidDataException("OBS bridge message exceeds its limit."); + var message = new byte[(int)length]; + await ReadExactlyAsync(stream, message, cancellationToken); + if (length >= 4 && message.AsSpan(0, 4).SequenceEqual("LACP"u8)) + { + if (length > ProtocolV1.AudioHeaderBytes + ProtocolV1.MaxAudioPayloadBytes) throw new InvalidDataException("OBS audio frame exceeds its limit."); + await _onAudio(message); continue; + } + using var json = JsonDocument.Parse(message); + var type = json.RootElement.TryGetProperty("type", out var property) ? property.GetString() : null; + if (type is null || !AllowedTypes.Contains(type)) throw new InvalidDataException("OBS bridge message type is not allowed."); + await _onMessage(json.RootElement.Clone()); + } + } + public static async Task WriteAsync(Stream stream, object message, CancellationToken cancellationToken) + { + var body = JsonSerializer.SerializeToUtf8Bytes(message, ProtocolV1.JsonOptions); + if (body.Length > ProtocolV1.MaxJsonBytes) throw new InvalidDataException("Companion IPC message exceeds its limit."); + var prefix = new byte[4]; BinaryPrimitives.WriteUInt32LittleEndian(prefix, (uint)body.Length); + await stream.WriteAsync(prefix, cancellationToken); await stream.WriteAsync(body, cancellationToken); await stream.FlushAsync(cancellationToken); + } + private static async Task ReadExactlyAsync(Stream stream, Memory buffer, CancellationToken cancellationToken) + { + var read = 0; + while (read < buffer.Length) + { + var count = await stream.ReadAsync(buffer[read..], cancellationToken); + if (count == 0) throw new EndOfStreamException(); + read += count; + } + } + public async ValueTask DisposeAsync() { _lifetime.Cancel(); if (_loop is not null) try { await _loop; } catch (OperationCanceledException) { } _lifetime.Dispose(); } +} diff --git a/companion/src/Lumi.Companion.App/CompanionState.cs b/companion/src/Lumi.Companion.App/CompanionState.cs new file mode 100644 index 0000000..1204ffc --- /dev/null +++ b/companion/src/Lumi.Companion.App/CompanionState.cs @@ -0,0 +1,10 @@ +namespace Lumi.Companion.App; + +public enum TrayHealth { Ready, Operating, Degraded, Failed } +public sealed record CompanionState(bool ObsStreaming, bool ObsRecording, bool TestRunning, TrayHealth Health) +{ + public bool RequiresQuitConfirmation => ObsStreaming || ObsRecording; + public string QuitWarning => ObsStreaming + ? "OBS is streaming. Quitting Lumi Companion will stop transcription and closed captions, but it will not stop the OBS stream." + : ObsRecording ? "OBS is recording. Quitting Lumi Companion will stop active companion features." : string.Empty; +} diff --git a/companion/src/Lumi.Companion.App/Lumi.Companion.App.csproj b/companion/src/Lumi.Companion.App/Lumi.Companion.App.csproj new file mode 100644 index 0000000..fc376c1 --- /dev/null +++ b/companion/src/Lumi.Companion.App/Lumi.Companion.App.csproj @@ -0,0 +1 @@ +Exenet8.0-windowsenableenable diff --git a/companion/src/Lumi.Companion.App/Program.cs b/companion/src/Lumi.Companion.App/Program.cs new file mode 100644 index 0000000..6c2d2ee --- /dev/null +++ b/companion/src/Lumi.Companion.App/Program.cs @@ -0,0 +1,33 @@ +using Lumi.Companion.Core; + +if (!OperatingSystem.IsWindows()) +{ + Console.Error.WriteLine("The first Lumi Companion build supports Windows x64 only."); + return 2; +} + +var appData = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Lumi", "Companion"); +var credentialStore = new SecureCredentialStore(appData); +if (args is ["pair", var packagePath]) +{ + using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) }; + var client = new PairingClient(http); + var credential = await client.PairAsync(packagePath, new + { + install_id = credentialStore.GetOrCreateInstallId(), name = Environment.MachineName, + companion_version = "0.1.0-experimental.1", os = Environment.OSVersion.VersionString, + architecture = System.Runtime.InteropServices.RuntimeInformation.OSArchitecture.ToString() + }, CancellationToken.None); + credentialStore.Save(credential); + Console.WriteLine("Lumi Companion paired successfully. The device credential is protected for the current Windows user."); + return 0; +} + +if (credentialStore.Load() is null) +{ + Console.WriteLine("Lumi Companion is not paired. Run: Lumi.Companion.App pair "); + return 1; +} + +Console.WriteLine("Lumi Companion protocol foundation is installed. The Avalonia tray UI and managed OBS bridge installer are not included in this milestone."); +return 0; diff --git a/companion/src/Lumi.Companion.Core/CompanionSocket.cs b/companion/src/Lumi.Companion.Core/CompanionSocket.cs new file mode 100644 index 0000000..94abd77 --- /dev/null +++ b/companion/src/Lumi.Companion.Core/CompanionSocket.cs @@ -0,0 +1,93 @@ +using System.Net.WebSockets; +using System.Threading.Channels; +using Lumi.Companion.Protocol; + +namespace Lumi.Companion.Core; + +public sealed class CompanionSocket : IAsyncDisposable +{ + private readonly Channel> _audio = Channel.CreateBounded>(new BoundedChannelOptions(250) { FullMode = BoundedChannelFullMode.Wait, SingleReader = true }); + private ClientWebSocket? _socket; + private CancellationTokenSource? _lifetime; + private readonly SemaphoreSlim _sendLock = new(1, 1); + private long _droppedFrames; + public long DroppedFrames => Interlocked.Read(ref _droppedFrames); + public Guid SessionId { get; private set; } + public event Func? MessageReceived; + + public async Task ConnectAsync(DeviceCredential credential, string companionVersion, string pluginVersion, string? obsVersion, CancellationToken cancellationToken) + { + _socket = new ClientWebSocket(); + _socket.Options.SetRequestHeader("Authorization", $"LumiDevice {credential.DeviceId}.{credential.DeviceSecret}"); + var host = new Uri(credential.Host); + var uri = new UriBuilder(host) { Scheme = host.Scheme == "https" ? "wss" : "ws", Path = "/plugins/lumi_transcription/live" }.Uri; + await _socket.ConnectAsync(uri, cancellationToken); + var hello = ProtocolV1.EncodeEnvelope("hello", new { companion_version = companionVersion, plugin_version = pluginVersion, obs_version = obsVersion, capabilities = credential.Capabilities, audio = new { codec = "pcm_s16le", sample_rate = 16000, channels = 1, bits = 16 } }); + await _socket.SendAsync(hello, WebSocketMessageType.Text, true, cancellationToken); + var acknowledgement = await ReceiveMessageAsync(_socket, cancellationToken); + if (acknowledgement.Type != "hello_ack" || acknowledgement.Version != 1 || acknowledgement.SessionId is null) + throw new InvalidDataException("Lumi returned an incompatible companion handshake."); + SessionId = acknowledgement.SessionId.Value; + _lifetime = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + _ = Task.Run(() => SendAudioAsync(_lifetime.Token), _lifetime.Token); + _ = Task.Run(() => ReceiveAsync(_lifetime.Token), _lifetime.Token); + } + + public bool QueueAudio(AudioFrame frame) + { + var encoded = ProtocolV1.EncodeAudio(frame); + if (_audio.Writer.TryWrite(encoded)) return true; + _audio.Reader.TryRead(out _); + Interlocked.Increment(ref _droppedFrames); + return _audio.Writer.TryWrite(encoded); + } + public async Task SendAsync(string type, object payload, Guid? sessionId, CancellationToken cancellationToken) + { + if (_socket?.State != WebSocketState.Open) throw new InvalidOperationException("Lumi is not connected."); + await SendLockedAsync(ProtocolV1.EncodeEnvelope(type, payload, sessionId), WebSocketMessageType.Text, cancellationToken); + } + private async Task SendAudioAsync(CancellationToken cancellationToken) + { + await foreach (var frame in _audio.Reader.ReadAllAsync(cancellationToken)) + { + if (_socket?.State != WebSocketState.Open) continue; + await SendLockedAsync(frame, WebSocketMessageType.Binary, cancellationToken); + } + } + private async Task SendLockedAsync(ReadOnlyMemory body, WebSocketMessageType type, CancellationToken cancellationToken) + { + await _sendLock.WaitAsync(cancellationToken); + try { if (_socket?.State == WebSocketState.Open) await _socket.SendAsync(body, type, true, cancellationToken); } + finally { _sendLock.Release(); } + } + private async Task ReceiveAsync(CancellationToken cancellationToken) + { + while (_socket?.State == WebSocketState.Open && !cancellationToken.IsCancellationRequested) + { + var message = await ReceiveMessageAsync(_socket, cancellationToken); + if (MessageReceived is { } handler) await handler(message); + } + } + private static async Task ReceiveMessageAsync(ClientWebSocket socket, CancellationToken cancellationToken) + { + using var body = new MemoryStream(); + var buffer = new byte[8192]; + WebSocketReceiveResult result; + do + { + result = await socket.ReceiveAsync(new ArraySegment(buffer), cancellationToken); + if (result.MessageType == WebSocketMessageType.Close) throw new EndOfStreamException("Lumi closed the companion connection."); + body.Write(buffer, 0, result.Count); + if (body.Length > ProtocolV1.MaxJsonBytes) throw new InvalidDataException("Lumi message exceeds the protocol limit."); + } while (!result.EndOfMessage); + if (result.MessageType != WebSocketMessageType.Text) throw new InvalidDataException("Unexpected binary message from Lumi."); + return System.Text.Json.JsonSerializer.Deserialize(body.ToArray(), ProtocolV1.JsonOptions) + ?? throw new InvalidDataException("Lumi message is invalid."); + } + public async ValueTask DisposeAsync() + { + _lifetime?.Cancel(); + if (_socket?.State == WebSocketState.Open) await _socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "companion_exit", CancellationToken.None); + _socket?.Dispose(); _lifetime?.Dispose(); _sendLock.Dispose(); + } +} diff --git a/companion/src/Lumi.Companion.Core/Lumi.Companion.Core.csproj b/companion/src/Lumi.Companion.Core/Lumi.Companion.Core.csproj new file mode 100644 index 0000000..04f1273 --- /dev/null +++ b/companion/src/Lumi.Companion.Core/Lumi.Companion.Core.csproj @@ -0,0 +1 @@ +net8.0-windowsenableenable diff --git a/companion/src/Lumi.Companion.Core/PairingClient.cs b/companion/src/Lumi.Companion.Core/PairingClient.cs new file mode 100644 index 0000000..5709184 --- /dev/null +++ b/companion/src/Lumi.Companion.Core/PairingClient.cs @@ -0,0 +1,30 @@ +using System.Net.Http.Json; +using System.Text.Json; +using Lumi.Companion.Protocol; + +namespace Lumi.Companion.Core; + +public sealed class PairingClient(HttpClient http) +{ + public async Task PairAsync(string packagePath, object device, CancellationToken cancellationToken) + { + var bootstrap = JsonSerializer.Deserialize(await File.ReadAllBytesAsync(packagePath, cancellationToken), ProtocolV1.JsonOptions) + ?? throw new InvalidDataException("Pairing package is invalid."); + if (bootstrap.Format != "lumi-companion-bootstrap-v1" || bootstrap.ProtocolVersion != 1) + throw new InvalidDataException("Pairing package is incompatible with this companion."); + if (DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() >= bootstrap.ExpiresAt) + throw new InvalidDataException("Pairing package has expired. Download a new package from Lumi."); + var exchangeUri = new Uri(bootstrap.ExchangeUrl); + var insecureDev = Environment.GetEnvironmentVariable("LUMI_COMPANION_DEV_ALLOW_INSECURE") == "1" && exchangeUri.IsLoopback; + if (!exchangeUri.Scheme.Equals("https", StringComparison.OrdinalIgnoreCase) && !insecureDev) + throw new InvalidDataException("Pairing requires an HTTPS Lumi host."); + using var response = await http.PostAsJsonAsync(bootstrap.ExchangeUrl, new { token = bootstrap.Token, device }, ProtocolV1.JsonOptions, cancellationToken); + var body = await response.Content.ReadAsStringAsync(cancellationToken); + if (!response.IsSuccessStatusCode) throw new InvalidOperationException($"Pairing failed: {body}"); + using var json = JsonDocument.Parse(body); + return new DeviceCredential( + json.RootElement.GetProperty("device_id").GetString()!, json.RootElement.GetProperty("device_secret").GetString()!, + json.RootElement.GetProperty("host").GetString()!, json.RootElement.GetProperty("capabilities").EnumerateArray().Select(item => item.GetString()!).ToArray(), + json.RootElement.GetProperty("protocol_version").GetInt32()); + } +} diff --git a/companion/src/Lumi.Companion.Core/SecureCredentialStore.cs b/companion/src/Lumi.Companion.Core/SecureCredentialStore.cs new file mode 100644 index 0000000..ca6e71d --- /dev/null +++ b/companion/src/Lumi.Companion.Core/SecureCredentialStore.cs @@ -0,0 +1,33 @@ +using System.Security.Cryptography; +using System.Text.Json; +using Lumi.Companion.Protocol; + +namespace Lumi.Companion.Core; + +public sealed class SecureCredentialStore(string root) +{ + private readonly string _path = Path.Combine(root, "device.credential"); + private readonly string _installIdPath = Path.Combine(root, "install.id"); + public void Save(DeviceCredential credential) + { + Directory.CreateDirectory(Path.GetDirectoryName(_path)!); + var clear = JsonSerializer.SerializeToUtf8Bytes(credential, ProtocolV1.JsonOptions); + var protectedBytes = ProtectedData.Protect(clear, "Lumi.Companion.Device.v1"u8.ToArray(), DataProtectionScope.CurrentUser); + var temporary = $"{_path}.{Environment.ProcessId}.tmp"; + try { File.WriteAllBytes(temporary, protectedBytes); File.Move(temporary, _path, true); } + finally { File.Delete(temporary); } + } + public DeviceCredential? Load() + { + if (!File.Exists(_path)) return null; + var clear = ProtectedData.Unprotect(File.ReadAllBytes(_path), "Lumi.Companion.Device.v1"u8.ToArray(), DataProtectionScope.CurrentUser); + return JsonSerializer.Deserialize(clear, ProtocolV1.JsonOptions); + } + public void Remove() => File.Delete(_path); + public string GetOrCreateInstallId() + { + Directory.CreateDirectory(Path.GetDirectoryName(_installIdPath)!); + if (File.Exists(_installIdPath) && Guid.TryParse(File.ReadAllText(_installIdPath).Trim(), out var existing)) return existing.ToString(); + var created = Guid.NewGuid().ToString(); File.WriteAllText(_installIdPath, created); return created; + } +} diff --git a/companion/src/Lumi.Companion.PluginHost/Lumi.Companion.PluginHost.csproj b/companion/src/Lumi.Companion.PluginHost/Lumi.Companion.PluginHost.csproj new file mode 100644 index 0000000..169ce85 --- /dev/null +++ b/companion/src/Lumi.Companion.PluginHost/Lumi.Companion.PluginHost.csproj @@ -0,0 +1 @@ +net8.0-windowsenableenable diff --git a/companion/src/Lumi.Companion.PluginHost/PluginWorker.cs b/companion/src/Lumi.Companion.PluginHost/PluginWorker.cs new file mode 100644 index 0000000..f211886 --- /dev/null +++ b/companion/src/Lumi.Companion.PluginHost/PluginWorker.cs @@ -0,0 +1,23 @@ +using System.Diagnostics; + +namespace Lumi.Companion.PluginHost; + +public sealed class PluginWorker(string executable, string arguments) : IAsyncDisposable +{ + private Process? _process; + public void Start() + { + if (_process is { HasExited: false }) return; + _process = Process.Start(new ProcessStartInfo(executable, arguments) { UseShellExecute = false, CreateNoWindow = true, RedirectStandardError = true, RedirectStandardOutput = true }) + ?? throw new InvalidOperationException("Companion plugin worker could not start."); + } + public bool Healthy => _process is { HasExited: false }; + public async ValueTask DisposeAsync() + { + if (_process is not { HasExited: false }) return; + _process.CloseMainWindow(); + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(2)); + try { await _process.WaitForExitAsync(timeout.Token); } catch (OperationCanceledException) { _process.Kill(entireProcessTree: true); } + _process.Dispose(); + } +} diff --git a/companion/src/Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj b/companion/src/Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj new file mode 100644 index 0000000..b5baf32 --- /dev/null +++ b/companion/src/Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj @@ -0,0 +1 @@ +net8.0enableenable diff --git a/companion/src/Lumi.Companion.Protocol/ProtocolV1.cs b/companion/src/Lumi.Companion.Protocol/ProtocolV1.cs new file mode 100644 index 0000000..5c5671d --- /dev/null +++ b/companion/src/Lumi.Companion.Protocol/ProtocolV1.cs @@ -0,0 +1,66 @@ +using System.Buffers.Binary; +using System.Text.Json; +using System.Text.Json.Serialization; + +namespace Lumi.Companion.Protocol; + +public static class ProtocolV1 +{ + public const int Version = 1; + public const int AudioHeaderBytes = 64; + public const int MaxAudioPayloadBytes = 6400; + public const int MaxJsonBytes = 64 * 1024; + + public static byte[] EncodeAudio(AudioFrame frame) + { + if (frame.Pcm.Length > MaxAudioPayloadBytes || frame.Pcm.Length % 2 != 0) + throw new ArgumentOutOfRangeException(nameof(frame), "PCM payload must be even and at most 6,400 bytes."); + var output = new byte[AudioHeaderBytes + frame.Pcm.Length]; + "LACP"u8.CopyTo(output); + output[4] = Version; + output[5] = (byte)((frame.Active ? 1 : 0) | (frame.Muted ? 2 : 0)); + BinaryPrimitives.WriteUInt16LittleEndian(output.AsSpan(6), AudioHeaderBytes); + BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(8), frame.Sequence); + BinaryPrimitives.WriteUInt64LittleEndian(output.AsSpan(12), frame.CaptureTimestampUs); + frame.SessionId.TryWriteBytes(output.AsSpan(20), bigEndian: true, out _); + frame.SourceUuid.TryWriteBytes(output.AsSpan(36), bigEndian: true, out _); + BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(52), 16000); + BinaryPrimitives.WriteUInt16LittleEndian(output.AsSpan(56), 1); + BinaryPrimitives.WriteUInt16LittleEndian(output.AsSpan(58), 16); + BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(60), (uint)frame.Pcm.Length); + frame.Pcm.Span.CopyTo(output.AsSpan(AudioHeaderBytes)); + return output; + } + + public static byte[] EncodeEnvelope(string type, object payload, Guid? sessionId = null) => + JsonSerializer.SerializeToUtf8Bytes(new Envelope(Version, type, Guid.NewGuid(), DateTimeOffset.UtcNow, sessionId, payload), JsonOptions); + + public static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web) + { + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull + }; +} + +public sealed record AudioFrame(Guid SessionId, Guid SourceUuid, uint Sequence, ulong CaptureTimestampUs, bool Active, bool Muted, ReadOnlyMemory Pcm); +public sealed record Envelope( + [property: JsonPropertyName("version")] int Version, + [property: JsonPropertyName("type")] string Type, + [property: JsonPropertyName("id")] Guid Id, + [property: JsonPropertyName("sent_at")] DateTimeOffset SentAt, + [property: JsonPropertyName("session_id")] Guid? SessionId, + [property: JsonPropertyName("payload")] object Payload); +public sealed record ServerEnvelope( + [property: JsonPropertyName("version")] int Version, + [property: JsonPropertyName("type")] string Type, + [property: JsonPropertyName("id")] Guid Id, + [property: JsonPropertyName("sent_at")] DateTimeOffset SentAt, + [property: JsonPropertyName("session_id")] Guid? SessionId, + [property: JsonPropertyName("payload")] JsonElement Payload); +public sealed record PairingBootstrap( + [property: JsonPropertyName("format")] string Format, + [property: JsonPropertyName("token")] string Token, + [property: JsonPropertyName("host")] string Host, + [property: JsonPropertyName("exchange_url")] string ExchangeUrl, + [property: JsonPropertyName("expires_at")] long ExpiresAt, + [property: JsonPropertyName("protocol_version")] int ProtocolVersion); +public sealed record DeviceCredential(string DeviceId, string DeviceSecret, string Host, string[] Capabilities, int ProtocolVersion); diff --git a/docs/adr/0001-companion-transcription-boundaries.md b/docs/adr/0001-companion-transcription-boundaries.md new file mode 100644 index 0000000..32cbb10 --- /dev/null +++ b/docs/adr/0001-companion-transcription-boundaries.md @@ -0,0 +1,13 @@ +# ADR 0001: Companion transcription boundaries + +Status: accepted for the experimental MVP branch. + +The MVP uses server-hosted whisper.cpp behind `TranscriptionProvider` and returns captions through `CaptionDeliveryAdapter`. The first concrete boundaries are `WhisperCppServerProvider` and companion-managed OBS native delivery. Session and UI code depend on those interfaces rather than engine, location, or platform details. + +Deferred provider implementations are local companion inference and remote providers such as Qwen ASR. Qwen will be a separate provider; it will not impersonate whisper.cpp. Deferred delivery implementations include other streaming platforms and optional open-caption rendering. Open captions are never an implicit fallback for failed Twitch closed captions. + +OBS source identity is UUID-based. Multiple tracks, independent internal caption events, primary-track overlap policy, and delivery-enabled fields exist in the server model even though the first usable configuration targets one microphone. Protocol codec negotiation exists even though v1 accepts only PCM. + +The companion plugin host is process-oriented so one future integration can fail independently. Official package signature enforcement and Dev Mode unsigned-package handling are deferred with the installer/plugin packaging work. Offline authorization retains the low/normal/severe setting boundary; server inference still stops when Lumi is unreachable. + +Core owns only a reusable WebSocket upgrade registration mechanism. Removing or disabling `lumi_transcription` must not leave transcription routes, timers, sockets, workers, or global capabilities active. diff --git a/docs/lumi-companion-transcription.md b/docs/lumi-companion-transcription.md new file mode 100644 index 0000000..1c9b38e --- /dev/null +++ b/docs/lumi-companion-transcription.md @@ -0,0 +1,56 @@ +# Lumi Companion transcription (experimental) + +## Delivered foundation + +`lumi_transcription` is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown. + +The implemented transport path is: + +```text +OBS bridge boundary -> same-user companion IPC -> bounded companion queue +-> authenticated TLS WebSocket -> bounded per-source Lumi buffer +-> supervised TranscriptionProvider -> stabilized revision-aware caption +-> WebSocket -> companion delivery adapter -> OBS native caption boundary +``` + +The repository does not yet contain a validated whisper streaming worker or a complete signed OBS bridge installer. Accordingly, the admin page reports inference setup as required and this branch does not claim live Twitch acceptance. + +## Trust and privacy + +- Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest. +- Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI. +- Device HTTP and WebSocket credentials require HTTPS/WSS. Insecure transport is available only from localhost when `LUMI_COMPANION_DEV_ALLOW_INSECURE=1` is explicitly set. +- The OBS bridge never receives the Lumi credential and never connects to Lumi directly. +- Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference. +- Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics. + +## Pairing and installation milestone + +1. Serve Lumi through HTTPS and enable `lumi_transcription` under Admin > Plugins. +2. Open Plugins > Transcription and create a pairing package. It is a bootstrap package for the generic companion, not a credential that should be shared or committed. +3. On the Windows streaming computer, build the .NET 8 projects under `companion/` and run `Lumi.Companion.App pair ` for the current protocol milestone. +4. Install a pinned runtime/model only after explicit confirmation. `small.en` is recommended; `small.en-q5_1` and `base.en` are fallbacks. Every artifact is checksum-verified before install. +5. Configure `LUMI_TRANSCRIPTION_WORKER` with the supervised streaming-worker executable once that worker is built for the target host. + +The normal signed installer, Avalonia tray UI, managed bridge install/repair, source selector, and model benchmark wizard are not complete yet. + +## Operation and recovery + +Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery. + +Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational. + +## Diagnostics + +Use the Transcription admin page for provider, model, device, and log health. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in `companion/docs/performance-acceptance-template.md`. + +## Known limitations + +- The .NET SDK and OBS SDK are not available in the development environment used for this milestone, so those projects have not been compiled here. +- No real whisper.cpp streaming worker has been integrated or benchmarked. +- The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback. +- Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence. +- The companion shell is a protocol/bootstrap executable, not yet the Avalonia tray application. +- Installer signing, bridge repair, auto-start, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending. + +See `docs/adr/0001-companion-transcription-boundaries.md`, `protocol/companion-protocol-v1.md`, and `companion/docs/obs-native-caption-compatibility-spike.md`.