Add experimental Companion and OBS bridge boundaries

README.md

@@ -55,6 +55,11 @@ You can set these in `.env` or change role IDs in **Admin → Settings**.
 Use **Admin → Plugins** to install, enable, update, or uninstall plugins.
 You can also create a local plugin from the WebUI.
 
+The experimental `experimental-companion` branch includes the independent Lumi
+Companion transcription foundation. Its current scope, trust boundaries, setup,
+and unverified target-machine work are documented in
+[`docs/lumi-companion-transcription.md`](docs/lumi-companion-transcription.md).
+
 ## Updates and recovery
 
 Use **Admin → Updates** for version-aware core and plugin updates. Lumi reads

TODO.md

@@ -2,6 +2,28 @@
 
 This file tracks larger Lumi work that cannot safely be completed in one pass. Keep pending work under the relevant category and move completed items to the Done section with a short note.
 
+## Lumi Companion transcription — experimental-companion (2026-07-22)
+
+Foundation implemented locally: independent `lumi_transcription` plugin; generic
+core WebSocket-upgrade capability; single-use pairing and revocable devices;
+TLS-only device transport; bounded PCM/session queues; revision-safe settings;
+provider/delivery interfaces; worker supervision; caption stabilization; pinned
+whisper.cpp/model manifests; short-lived JSONL diagnostics; protocol schemas;
+and .NET/native companion boundaries with focused verification.
+
+Release-blocking work remains:
+
+- Build the real streaming whisper.cpp worker, install it through the manifest,
+  and benchmark `small.en`, quantized `small.en`, and `base.en` on the RTX 3060.
+- Complete the Avalonia tray application, signed installer, DPAPI migration UX,
+  managed OBS bridge install/repair/uninstall, and single-instance/live-quit flow.
+- Complete native selected-source capture, nested Program-scene evaluation,
+  resampling/IPC, bridge health, and source rename/missing recovery.
+- Run the OBS 31+/Twitch compatibility spike and prove toggleable closed captions;
+  tune replacement/display duration from player behavior without open-caption fallback.
+- Add setup/test/conflict-resolution UI and live settings synchronization to the
+  companion, then run the target-topology failure and performance matrix.
+
 ## Remaining DesignMotionHQ UX work — experimental-ux checkpoint (2026-07-22)
 
 The shared UX foundation and representative settings, navigation, theme, command,

companion/README.md

@@ -0,0 +1,9 @@
+# Lumi Companion (experimental transcription milestone)
+
+This directory is the single Lumi Companion product boundary. The current milestone supplies the versioned protocol client, Windows credential protection, bounded outbound transport, plugin-process supervision, and same-user OBS bridge IPC. The Avalonia tray shell, installer, and signed native bridge package remain target-machine work and are not represented as complete.
+
+Build prerequisites: .NET 8 SDK on Windows x64. The native bridge additionally requires CMake, Visual Studio C++ tools, and the OBS 31+ SDK.
+
+The app accepts a `.lumi-pairing.json` bootstrap package. It exchanges the embedded token once and stores the returned device credential with Windows DPAPI. Do not commit bootstrap packages, credentials, generated installers, build output, or logs.
+
+The companion never performs ASR in this MVP. Audio is normalized to 16 kHz mono signed 16-bit PCM, held in bounded memory, and sent to the paired Lumi host over TLS WebSockets. The OBS bridge talks only to the companion over a same-user named pipe.

companion/docs/obs-native-caption-compatibility-spike.md

@@ -0,0 +1,14 @@
+# OBS 31+ native caption compatibility spike
+
+Implementation source inspection confirms OBS 31 exposes `obs_output_output_caption_text2` and `obs_frontend_get_streaming_output`. The bridge calls the caption API only from its IPC worker against the active streaming output; it never calls it from an audio/render callback.
+
+This is not Twitch acceptance evidence. The following target-topology test remains mandatory before live delivery can be labelled ready:
+
+1. Build and install the signed bridge through the companion on Windows 11 with OBS 31+.
+2. Start a private Twitch stream using the production service/output configuration.
+3. Submit incrementally revised text with 1.5–3 second display durations.
+4. Verify the Twitch web player and at least one mobile client expose a user-toggleable CC control.
+5. Record whether OBS/Twitch queue, replace, or drop revisions and tune the adapter to prevent stale caption backlog.
+6. Capture OBS CPU, render/encoder missed frames, audio dropouts, bridge queue drops, and network throughput before/after.
+
+Until this passes, the companion must allow the exact simulated delivery stream in Test mode but refuse to describe Twitch closed captions as verified. It must never substitute a baked-in open-caption overlay.

companion/docs/performance-acceptance-template.md

@@ -0,0 +1,23 @@
+# Transcription target-topology acceptance report
+
+Status: not run. Do not replace blank measurements with estimates.
+
+Topology: Windows 11 streaming PC / OBS version ___ / Windows Server 2022 Lumi host / RTX 3060 driver ___ / wired LAN / Twitch test URL or evidence reference ___.
+
+| Measurement | Before | Live transcription | Result |
+| --- | ---: | ---: | --- |
+| Companion CPU / working set | ___ | ___ | ___ |
+| OBS CPU | ___ | ___ | ___ |
+| OBS render missed frames | ___ | ___ | ___ |
+| OBS encoder missed frames | ___ | ___ | ___ |
+| Audio dropouts | ___ | ___ | ___ |
+| Bridge queue drops | ___ | ___ | ___ |
+| Network throughput | ___ | ___ | ___ |
+| Whisper CPU / GPU / VRAM / RAM | n/a | ___ | ___ |
+| Model load time | n/a | ___ | ___ |
+| Decode latency / real-time factor | n/a | ___ | ___ |
+| First stable caption p50 / p95 | n/a | ___ | ___ |
+| End-to-end caption p50 / p95 | n/a | ___ | ___ |
+| Maximum queue depth / backlog growth | n/a | ___ | ___ |
+
+Confirm: test-mode real path ___; Twitch player toggleable CC ___; stable sustained speech ___; no stale dump after network flap ___; 30-second stream grace/resume ___; raw audio absent from disk ___; fallback activation evidence ___.

companion/native/obs-bridge/CMakeLists.txt

@@ -0,0 +1,10 @@
+cmake_minimum_required(VERSION 3.28)
+project(lumi-obs-bridge VERSION 0.1.0 LANGUAGES CXX)
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+find_package(libobs REQUIRED)
+find_package(obs-frontend-api REQUIRED)
+add_library(lumi-obs-bridge MODULE src/plugin.cpp)
+target_include_directories(lumi-obs-bridge PRIVATE include)
+target_link_libraries(lumi-obs-bridge PRIVATE OBS::libobs OBS::obs-frontend-api)
+set_target_properties(lumi-obs-bridge PROPERTIES PREFIX "")

companion/native/obs-bridge/include/bounded_spsc_queue.hpp

@@ -0,0 +1,32 @@
+#pragma once
+#include <array>
+#include <atomic>
+#include <cstddef>
+#include <optional>
+
+namespace lumi {
+template <typename T, std::size_t Capacity> class bounded_spsc_queue {
+    static_assert(Capacity > 1);
+    std::array<T, Capacity> values_{};
+    alignas(64) std::atomic<std::size_t> head_{0};
+    alignas(64) std::atomic<std::size_t> tail_{0};
+    std::atomic<std::uint64_t> dropped_{0};
+public:
+    bool try_push(T value) noexcept {
+        const auto head = head_.load(std::memory_order_relaxed);
+        const auto next = (head + 1) % Capacity;
+        if (next == tail_.load(std::memory_order_acquire)) { dropped_.fetch_add(1, std::memory_order_relaxed); return false; }
+        values_[head] = std::move(value);
+        head_.store(next, std::memory_order_release);
+        return true;
+    }
+    std::optional<T> try_pop() noexcept {
+        const auto tail = tail_.load(std::memory_order_relaxed);
+        if (tail == head_.load(std::memory_order_acquire)) return std::nullopt;
+        T value = std::move(values_[tail]);
+        tail_.store((tail + 1) % Capacity, std::memory_order_release);
+        return value;
+    }
+    std::uint64_t dropped() const noexcept { return dropped_.load(std::memory_order_relaxed); }
+};
+}

companion/native/obs-bridge/src/plugin.cpp

@@ -0,0 +1,58 @@
+#include <obs-module.h>
+#include <obs-frontend-api.h>
+#include <cstdlib>
+#include <string>
+
+OBS_DECLARE_MODULE()
+OBS_MODULE_USE_DEFAULT_LOCALE("lumi-obs-bridge", "en-US")
+
+const char *obs_module_description(void)
+{
+    return "Companion-managed Lumi audio and native-caption bridge";
+}
+
+static bool enumerate_source(void *, obs_source_t *source)
+{
+    if (!source) return true;
+    blog(LOG_DEBUG, "[Lumi Companion] OBS source available: uuid=%s name=%s active=%s",
+         obs_source_get_uuid(source), obs_source_get_name(source), obs_source_active(source) ? "true" : "false");
+    return true;
+}
+
+static void frontend_event(enum obs_frontend_event event, void *)
+{
+    if (event == OBS_FRONTEND_EVENT_STREAMING_STARTED || event == OBS_FRONTEND_EVENT_STREAMING_STOPPED ||
+        event == OBS_FRONTEND_EVENT_RECORDING_STARTED || event == OBS_FRONTEND_EVENT_RECORDING_STOPPED) {
+        blog(LOG_INFO, "[Lumi Companion] OBS state changed: streaming=%s recording=%s",
+             obs_frontend_streaming_active() ? "true" : "false", obs_frontend_recording_active() ? "true" : "false");
+    }
+}
+
+// Called only by the bridge IPC worker, never by an OBS audio/render callback.
+// Returning false keeps delivery failure isolated from the active OBS output.
+static bool output_caption(const std::string &text, double display_seconds)
+{
+    if (!obs_frontend_streaming_active() || text.empty()) return false;
+    obs_output_t *output = obs_frontend_get_streaming_output();
+    if (!output) return false;
+    obs_output_output_caption_text2(output, text.c_str(), display_seconds);
+    obs_output_release(output);
+    return true;
+}
+
+bool obs_module_load(void)
+{
+    const char *version = obs_get_version_string();
+    const int major = version ? std::atoi(version) : 0;
+    if (major < 31) blog(LOG_WARNING, "[Lumi Companion] OBS %s is outside the supported 31+ range", version ? version : "unknown");
+    obs_frontend_add_event_callback(frontend_event, nullptr);
+    obs_enum_sources(enumerate_source, nullptr);
+    blog(LOG_INFO, "[Lumi Companion] bridge loaded; waiting for the companion-managed IPC transport");
+    return true;
+}
+
+void obs_module_unload(void)
+{
+    obs_frontend_remove_event_callback(frontend_event, nullptr);
+    blog(LOG_INFO, "[Lumi Companion] bridge unloaded");
+}

companion/plugins/Lumi.Companion.Transcription/Lumi.Companion.Transcription.csproj

@@ -0,0 +1 @@
+<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><TargetFramework>net8.0-windows</TargetFramework><Nullable>enable</Nullable><ImplicitUsings>enable</ImplicitUsings></PropertyGroup><ItemGroup><ProjectReference Include="../../src/Lumi.Companion.Core/Lumi.Companion.Core.csproj" /><ProjectReference Include="../../src/Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj" /></ItemGroup></Project>

companion/plugins/Lumi.Companion.Transcription/ObsBridgePipe.cs

@@ -0,0 +1,73 @@
+using System.Buffers.Binary;
+using System.IO.Pipes;
+using System.Text.Json;
+using Lumi.Companion.Protocol;
+
+namespace Lumi.Companion.Transcription;
+
+public sealed class ObsBridgePipe : IAsyncDisposable
+{
+    private static readonly HashSet<string> AllowedTypes = ["hello", "source_list", "source_state", "obs_state", "health"];
+    private readonly string _pipeName;
+    private readonly Func<JsonElement, Task> _onMessage;
+    private readonly Func<ReadOnlyMemory<byte>, Task> _onAudio;
+    private readonly CancellationTokenSource _lifetime = new();
+    private Task? _loop;
+
+    public ObsBridgePipe(string userSidHash, Func<JsonElement, Task> onMessage, Func<ReadOnlyMemory<byte>, Task> onAudio)
+    {
+        _pipeName = $"Lumi.Companion.ObsBridge.v1.{userSidHash}";
+        _onMessage = onMessage; _onAudio = onAudio;
+    }
+    public void Start() => _loop ??= Task.Run(() => ListenAsync(_lifetime.Token));
+    private async Task ListenAsync(CancellationToken cancellationToken)
+    {
+        while (!cancellationToken.IsCancellationRequested)
+        {
+            await using var pipe = new NamedPipeServerStream(_pipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Byte,
+                PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly, 65536, 65536);
+            await pipe.WaitForConnectionAsync(cancellationToken);
+            try { await ReadConnectionAsync(pipe, cancellationToken); }
+            catch (Exception) when (!cancellationToken.IsCancellationRequested) { /* reconnect without taking down the shell */ }
+        }
+    }
+    private async Task ReadConnectionAsync(Stream stream, CancellationToken cancellationToken)
+    {
+        var lengthBytes = new byte[4];
+        while (!cancellationToken.IsCancellationRequested)
+        {
+            await ReadExactlyAsync(stream, lengthBytes, cancellationToken);
+            var length = BinaryPrimitives.ReadUInt32LittleEndian(lengthBytes);
+            if (length is 0 or > ProtocolV1.MaxJsonBytes) throw new InvalidDataException("OBS bridge message exceeds its limit.");
+            var message = new byte[(int)length];
+            await ReadExactlyAsync(stream, message, cancellationToken);
+            if (length >= 4 && message.AsSpan(0, 4).SequenceEqual("LACP"u8))
+            {
+                if (length > ProtocolV1.AudioHeaderBytes + ProtocolV1.MaxAudioPayloadBytes) throw new InvalidDataException("OBS audio frame exceeds its limit.");
+                await _onAudio(message); continue;
+            }
+            using var json = JsonDocument.Parse(message);
+            var type = json.RootElement.TryGetProperty("type", out var property) ? property.GetString() : null;
+            if (type is null || !AllowedTypes.Contains(type)) throw new InvalidDataException("OBS bridge message type is not allowed.");
+            await _onMessage(json.RootElement.Clone());
+        }
+    }
+    public static async Task WriteAsync(Stream stream, object message, CancellationToken cancellationToken)
+    {
+        var body = JsonSerializer.SerializeToUtf8Bytes(message, ProtocolV1.JsonOptions);
+        if (body.Length > ProtocolV1.MaxJsonBytes) throw new InvalidDataException("Companion IPC message exceeds its limit.");
+        var prefix = new byte[4]; BinaryPrimitives.WriteUInt32LittleEndian(prefix, (uint)body.Length);
+        await stream.WriteAsync(prefix, cancellationToken); await stream.WriteAsync(body, cancellationToken); await stream.FlushAsync(cancellationToken);
+    }
+    private static async Task ReadExactlyAsync(Stream stream, Memory<byte> buffer, CancellationToken cancellationToken)
+    {
+        var read = 0;
+        while (read < buffer.Length)
+        {
+            var count = await stream.ReadAsync(buffer[read..], cancellationToken);
+            if (count == 0) throw new EndOfStreamException();
+            read += count;
+        }
+    }
+    public async ValueTask DisposeAsync() { _lifetime.Cancel(); if (_loop is not null) try { await _loop; } catch (OperationCanceledException) { } _lifetime.Dispose(); }
+}

companion/src/Lumi.Companion.App/CompanionState.cs

@@ -0,0 +1,10 @@
+namespace Lumi.Companion.App;
+
+public enum TrayHealth { Ready, Operating, Degraded, Failed }
+public sealed record CompanionState(bool ObsStreaming, bool ObsRecording, bool TestRunning, TrayHealth Health)
+{
+    public bool RequiresQuitConfirmation => ObsStreaming || ObsRecording;
+    public string QuitWarning => ObsStreaming
+        ? "OBS is streaming. Quitting Lumi Companion will stop transcription and closed captions, but it will not stop the OBS stream."
+        : ObsRecording ? "OBS is recording. Quitting Lumi Companion will stop active companion features." : string.Empty;
+}

companion/src/Lumi.Companion.App/Lumi.Companion.App.csproj

@@ -0,0 +1 @@
+<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><OutputType>Exe</OutputType><TargetFramework>net8.0-windows</TargetFramework><Nullable>enable</Nullable><ImplicitUsings>enable</ImplicitUsings></PropertyGroup><ItemGroup><ProjectReference Include="../Lumi.Companion.Core/Lumi.Companion.Core.csproj" /><ProjectReference Include="../Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj" /></ItemGroup></Project>

companion/src/Lumi.Companion.App/Program.cs

@@ -0,0 +1,33 @@
+using Lumi.Companion.Core;
+
+if (!OperatingSystem.IsWindows())
+{
+    Console.Error.WriteLine("The first Lumi Companion build supports Windows x64 only.");
+    return 2;
+}
+
+var appData = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "Lumi", "Companion");
+var credentialStore = new SecureCredentialStore(appData);
+if (args is ["pair", var packagePath])
+{
+    using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };
+    var client = new PairingClient(http);
+    var credential = await client.PairAsync(packagePath, new
+    {
+        install_id = credentialStore.GetOrCreateInstallId(), name = Environment.MachineName,
+        companion_version = "0.1.0-experimental.1", os = Environment.OSVersion.VersionString,
+        architecture = System.Runtime.InteropServices.RuntimeInformation.OSArchitecture.ToString()
+    }, CancellationToken.None);
+    credentialStore.Save(credential);
+    Console.WriteLine("Lumi Companion paired successfully. The device credential is protected for the current Windows user.");
+    return 0;
+}
+
+if (credentialStore.Load() is null)
+{
+    Console.WriteLine("Lumi Companion is not paired. Run: Lumi.Companion.App pair <package.lumi-pairing.json>");
+    return 1;
+}
+
+Console.WriteLine("Lumi Companion protocol foundation is installed. The Avalonia tray UI and managed OBS bridge installer are not included in this milestone.");
+return 0;

companion/src/Lumi.Companion.Core/CompanionSocket.cs

@@ -0,0 +1,93 @@
+using System.Net.WebSockets;
+using System.Threading.Channels;
+using Lumi.Companion.Protocol;
+
+namespace Lumi.Companion.Core;
+
+public sealed class CompanionSocket : IAsyncDisposable
+{
+    private readonly Channel<ReadOnlyMemory<byte>> _audio = Channel.CreateBounded<ReadOnlyMemory<byte>>(new BoundedChannelOptions(250) { FullMode = BoundedChannelFullMode.Wait, SingleReader = true });
+    private ClientWebSocket? _socket;
+    private CancellationTokenSource? _lifetime;
+    private readonly SemaphoreSlim _sendLock = new(1, 1);
+    private long _droppedFrames;
+    public long DroppedFrames => Interlocked.Read(ref _droppedFrames);
+    public Guid SessionId { get; private set; }
+    public event Func<ServerEnvelope, Task>? MessageReceived;
+
+    public async Task ConnectAsync(DeviceCredential credential, string companionVersion, string pluginVersion, string? obsVersion, CancellationToken cancellationToken)
+    {
+        _socket = new ClientWebSocket();
+        _socket.Options.SetRequestHeader("Authorization", $"LumiDevice {credential.DeviceId}.{credential.DeviceSecret}");
+        var host = new Uri(credential.Host);
+        var uri = new UriBuilder(host) { Scheme = host.Scheme == "https" ? "wss" : "ws", Path = "/plugins/lumi_transcription/live" }.Uri;
+        await _socket.ConnectAsync(uri, cancellationToken);
+        var hello = ProtocolV1.EncodeEnvelope("hello", new { companion_version = companionVersion, plugin_version = pluginVersion, obs_version = obsVersion, capabilities = credential.Capabilities, audio = new { codec = "pcm_s16le", sample_rate = 16000, channels = 1, bits = 16 } });
+        await _socket.SendAsync(hello, WebSocketMessageType.Text, true, cancellationToken);
+        var acknowledgement = await ReceiveMessageAsync(_socket, cancellationToken);
+        if (acknowledgement.Type != "hello_ack" || acknowledgement.Version != 1 || acknowledgement.SessionId is null)
+            throw new InvalidDataException("Lumi returned an incompatible companion handshake.");
+        SessionId = acknowledgement.SessionId.Value;
+        _lifetime = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
+        _ = Task.Run(() => SendAudioAsync(_lifetime.Token), _lifetime.Token);
+        _ = Task.Run(() => ReceiveAsync(_lifetime.Token), _lifetime.Token);
+    }
+
+    public bool QueueAudio(AudioFrame frame)
+    {
+        var encoded = ProtocolV1.EncodeAudio(frame);
+        if (_audio.Writer.TryWrite(encoded)) return true;
+        _audio.Reader.TryRead(out _);
+        Interlocked.Increment(ref _droppedFrames);
+        return _audio.Writer.TryWrite(encoded);
+    }
+    public async Task SendAsync(string type, object payload, Guid? sessionId, CancellationToken cancellationToken)
+    {
+        if (_socket?.State != WebSocketState.Open) throw new InvalidOperationException("Lumi is not connected.");
+        await SendLockedAsync(ProtocolV1.EncodeEnvelope(type, payload, sessionId), WebSocketMessageType.Text, cancellationToken);
+    }
+    private async Task SendAudioAsync(CancellationToken cancellationToken)
+    {
+        await foreach (var frame in _audio.Reader.ReadAllAsync(cancellationToken))
+        {
+            if (_socket?.State != WebSocketState.Open) continue;
+            await SendLockedAsync(frame, WebSocketMessageType.Binary, cancellationToken);
+        }
+    }
+    private async Task SendLockedAsync(ReadOnlyMemory<byte> body, WebSocketMessageType type, CancellationToken cancellationToken)
+    {
+        await _sendLock.WaitAsync(cancellationToken);
+        try { if (_socket?.State == WebSocketState.Open) await _socket.SendAsync(body, type, true, cancellationToken); }
+        finally { _sendLock.Release(); }
+    }
+    private async Task ReceiveAsync(CancellationToken cancellationToken)
+    {
+        while (_socket?.State == WebSocketState.Open && !cancellationToken.IsCancellationRequested)
+        {
+            var message = await ReceiveMessageAsync(_socket, cancellationToken);
+            if (MessageReceived is { } handler) await handler(message);
+        }
+    }
+    private static async Task<ServerEnvelope> ReceiveMessageAsync(ClientWebSocket socket, CancellationToken cancellationToken)
+    {
+        using var body = new MemoryStream();
+        var buffer = new byte[8192];
+        WebSocketReceiveResult result;
+        do
+        {
+            result = await socket.ReceiveAsync(new ArraySegment<byte>(buffer), cancellationToken);
+            if (result.MessageType == WebSocketMessageType.Close) throw new EndOfStreamException("Lumi closed the companion connection.");
+            body.Write(buffer, 0, result.Count);
+            if (body.Length > ProtocolV1.MaxJsonBytes) throw new InvalidDataException("Lumi message exceeds the protocol limit.");
+        } while (!result.EndOfMessage);
+        if (result.MessageType != WebSocketMessageType.Text) throw new InvalidDataException("Unexpected binary message from Lumi.");
+        return System.Text.Json.JsonSerializer.Deserialize<ServerEnvelope>(body.ToArray(), ProtocolV1.JsonOptions)
+            ?? throw new InvalidDataException("Lumi message is invalid.");
+    }
+    public async ValueTask DisposeAsync()
+    {
+        _lifetime?.Cancel();
+        if (_socket?.State == WebSocketState.Open) await _socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "companion_exit", CancellationToken.None);
+        _socket?.Dispose(); _lifetime?.Dispose(); _sendLock.Dispose();
+    }
+}

companion/src/Lumi.Companion.Core/Lumi.Companion.Core.csproj

@@ -0,0 +1 @@
+<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><TargetFramework>net8.0-windows</TargetFramework><Nullable>enable</Nullable><ImplicitUsings>enable</ImplicitUsings></PropertyGroup><ItemGroup><ProjectReference Include="../Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj" /><PackageReference Include="System.Security.Cryptography.ProtectedData" Version="8.0.0" /></ItemGroup></Project>

companion/src/Lumi.Companion.Core/PairingClient.cs

@@ -0,0 +1,30 @@
+using System.Net.Http.Json;
+using System.Text.Json;
+using Lumi.Companion.Protocol;
+
+namespace Lumi.Companion.Core;
+
+public sealed class PairingClient(HttpClient http)
+{
+    public async Task<DeviceCredential> PairAsync(string packagePath, object device, CancellationToken cancellationToken)
+    {
+        var bootstrap = JsonSerializer.Deserialize<PairingBootstrap>(await File.ReadAllBytesAsync(packagePath, cancellationToken), ProtocolV1.JsonOptions)
+            ?? throw new InvalidDataException("Pairing package is invalid.");
+        if (bootstrap.Format != "lumi-companion-bootstrap-v1" || bootstrap.ProtocolVersion != 1)
+            throw new InvalidDataException("Pairing package is incompatible with this companion.");
+        if (DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() >= bootstrap.ExpiresAt)
+            throw new InvalidDataException("Pairing package has expired. Download a new package from Lumi.");
+        var exchangeUri = new Uri(bootstrap.ExchangeUrl);
+        var insecureDev = Environment.GetEnvironmentVariable("LUMI_COMPANION_DEV_ALLOW_INSECURE") == "1" && exchangeUri.IsLoopback;
+        if (!exchangeUri.Scheme.Equals("https", StringComparison.OrdinalIgnoreCase) && !insecureDev)
+            throw new InvalidDataException("Pairing requires an HTTPS Lumi host.");
+        using var response = await http.PostAsJsonAsync(bootstrap.ExchangeUrl, new { token = bootstrap.Token, device }, ProtocolV1.JsonOptions, cancellationToken);
+        var body = await response.Content.ReadAsStringAsync(cancellationToken);
+        if (!response.IsSuccessStatusCode) throw new InvalidOperationException($"Pairing failed: {body}");
+        using var json = JsonDocument.Parse(body);
+        return new DeviceCredential(
+            json.RootElement.GetProperty("device_id").GetString()!, json.RootElement.GetProperty("device_secret").GetString()!,
+            json.RootElement.GetProperty("host").GetString()!, json.RootElement.GetProperty("capabilities").EnumerateArray().Select(item => item.GetString()!).ToArray(),
+            json.RootElement.GetProperty("protocol_version").GetInt32());
+    }
+}

companion/src/Lumi.Companion.Core/SecureCredentialStore.cs

@@ -0,0 +1,33 @@
+using System.Security.Cryptography;
+using System.Text.Json;
+using Lumi.Companion.Protocol;
+
+namespace Lumi.Companion.Core;
+
+public sealed class SecureCredentialStore(string root)
+{
+    private readonly string _path = Path.Combine(root, "device.credential");
+    private readonly string _installIdPath = Path.Combine(root, "install.id");
+    public void Save(DeviceCredential credential)
+    {
+        Directory.CreateDirectory(Path.GetDirectoryName(_path)!);
+        var clear = JsonSerializer.SerializeToUtf8Bytes(credential, ProtocolV1.JsonOptions);
+        var protectedBytes = ProtectedData.Protect(clear, "Lumi.Companion.Device.v1"u8.ToArray(), DataProtectionScope.CurrentUser);
+        var temporary = $"{_path}.{Environment.ProcessId}.tmp";
+        try { File.WriteAllBytes(temporary, protectedBytes); File.Move(temporary, _path, true); }
+        finally { File.Delete(temporary); }
+    }
+    public DeviceCredential? Load()
+    {
+        if (!File.Exists(_path)) return null;
+        var clear = ProtectedData.Unprotect(File.ReadAllBytes(_path), "Lumi.Companion.Device.v1"u8.ToArray(), DataProtectionScope.CurrentUser);
+        return JsonSerializer.Deserialize<DeviceCredential>(clear, ProtocolV1.JsonOptions);
+    }
+    public void Remove() => File.Delete(_path);
+    public string GetOrCreateInstallId()
+    {
+        Directory.CreateDirectory(Path.GetDirectoryName(_installIdPath)!);
+        if (File.Exists(_installIdPath) && Guid.TryParse(File.ReadAllText(_installIdPath).Trim(), out var existing)) return existing.ToString();
+        var created = Guid.NewGuid().ToString(); File.WriteAllText(_installIdPath, created); return created;
+    }
+}

companion/src/Lumi.Companion.PluginHost/Lumi.Companion.PluginHost.csproj

@@ -0,0 +1 @@
+<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><TargetFramework>net8.0-windows</TargetFramework><Nullable>enable</Nullable><ImplicitUsings>enable</ImplicitUsings></PropertyGroup></Project>

companion/src/Lumi.Companion.PluginHost/PluginWorker.cs

@@ -0,0 +1,23 @@
+using System.Diagnostics;
+
+namespace Lumi.Companion.PluginHost;
+
+public sealed class PluginWorker(string executable, string arguments) : IAsyncDisposable
+{
+    private Process? _process;
+    public void Start()
+    {
+        if (_process is { HasExited: false }) return;
+        _process = Process.Start(new ProcessStartInfo(executable, arguments) { UseShellExecute = false, CreateNoWindow = true, RedirectStandardError = true, RedirectStandardOutput = true })
+            ?? throw new InvalidOperationException("Companion plugin worker could not start.");
+    }
+    public bool Healthy => _process is { HasExited: false };
+    public async ValueTask DisposeAsync()
+    {
+        if (_process is not { HasExited: false }) return;
+        _process.CloseMainWindow();
+        using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(2));
+        try { await _process.WaitForExitAsync(timeout.Token); } catch (OperationCanceledException) { _process.Kill(entireProcessTree: true); }
+        _process.Dispose();
+    }
+}

companion/src/Lumi.Companion.Protocol/Lumi.Companion.Protocol.csproj

@@ -0,0 +1 @@
+<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><TargetFramework>net8.0</TargetFramework><Nullable>enable</Nullable><ImplicitUsings>enable</ImplicitUsings></PropertyGroup></Project>

companion/src/Lumi.Companion.Protocol/ProtocolV1.cs

@@ -0,0 +1,66 @@
+using System.Buffers.Binary;
+using System.Text.Json;
+using System.Text.Json.Serialization;
+
+namespace Lumi.Companion.Protocol;
+
+public static class ProtocolV1
+{
+    public const int Version = 1;
+    public const int AudioHeaderBytes = 64;
+    public const int MaxAudioPayloadBytes = 6400;
+    public const int MaxJsonBytes = 64 * 1024;
+
+    public static byte[] EncodeAudio(AudioFrame frame)
+    {
+        if (frame.Pcm.Length > MaxAudioPayloadBytes || frame.Pcm.Length % 2 != 0)
+            throw new ArgumentOutOfRangeException(nameof(frame), "PCM payload must be even and at most 6,400 bytes.");
+        var output = new byte[AudioHeaderBytes + frame.Pcm.Length];
+        "LACP"u8.CopyTo(output);
+        output[4] = Version;
+        output[5] = (byte)((frame.Active ? 1 : 0) | (frame.Muted ? 2 : 0));
+        BinaryPrimitives.WriteUInt16LittleEndian(output.AsSpan(6), AudioHeaderBytes);
+        BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(8), frame.Sequence);
+        BinaryPrimitives.WriteUInt64LittleEndian(output.AsSpan(12), frame.CaptureTimestampUs);
+        frame.SessionId.TryWriteBytes(output.AsSpan(20), bigEndian: true, out _);
+        frame.SourceUuid.TryWriteBytes(output.AsSpan(36), bigEndian: true, out _);
+        BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(52), 16000);
+        BinaryPrimitives.WriteUInt16LittleEndian(output.AsSpan(56), 1);
+        BinaryPrimitives.WriteUInt16LittleEndian(output.AsSpan(58), 16);
+        BinaryPrimitives.WriteUInt32LittleEndian(output.AsSpan(60), (uint)frame.Pcm.Length);
+        frame.Pcm.Span.CopyTo(output.AsSpan(AudioHeaderBytes));
+        return output;
+    }
+
+    public static byte[] EncodeEnvelope(string type, object payload, Guid? sessionId = null) =>
+        JsonSerializer.SerializeToUtf8Bytes(new Envelope(Version, type, Guid.NewGuid(), DateTimeOffset.UtcNow, sessionId, payload), JsonOptions);
+
+    public static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web)
+    {
+        DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
+    };
+}
+
+public sealed record AudioFrame(Guid SessionId, Guid SourceUuid, uint Sequence, ulong CaptureTimestampUs, bool Active, bool Muted, ReadOnlyMemory<byte> Pcm);
+public sealed record Envelope(
+    [property: JsonPropertyName("version")] int Version,
+    [property: JsonPropertyName("type")] string Type,
+    [property: JsonPropertyName("id")] Guid Id,
+    [property: JsonPropertyName("sent_at")] DateTimeOffset SentAt,
+    [property: JsonPropertyName("session_id")] Guid? SessionId,
+    [property: JsonPropertyName("payload")] object Payload);
+public sealed record ServerEnvelope(
+    [property: JsonPropertyName("version")] int Version,
+    [property: JsonPropertyName("type")] string Type,
+    [property: JsonPropertyName("id")] Guid Id,
+    [property: JsonPropertyName("sent_at")] DateTimeOffset SentAt,
+    [property: JsonPropertyName("session_id")] Guid? SessionId,
+    [property: JsonPropertyName("payload")] JsonElement Payload);
+public sealed record PairingBootstrap(
+    [property: JsonPropertyName("format")] string Format,
+    [property: JsonPropertyName("token")] string Token,
+    [property: JsonPropertyName("host")] string Host,
+    [property: JsonPropertyName("exchange_url")] string ExchangeUrl,
+    [property: JsonPropertyName("expires_at")] long ExpiresAt,
+    [property: JsonPropertyName("protocol_version")] int ProtocolVersion);
+public sealed record DeviceCredential(string DeviceId, string DeviceSecret, string Host, string[] Capabilities, int ProtocolVersion);

docs/adr/0001-companion-transcription-boundaries.md

@@ -0,0 +1,13 @@
+# ADR 0001: Companion transcription boundaries
+
+Status: accepted for the experimental MVP branch.
+
+The MVP uses server-hosted whisper.cpp behind `TranscriptionProvider` and returns captions through `CaptionDeliveryAdapter`. The first concrete boundaries are `WhisperCppServerProvider` and companion-managed OBS native delivery. Session and UI code depend on those interfaces rather than engine, location, or platform details.
+
+Deferred provider implementations are local companion inference and remote providers such as Qwen ASR. Qwen will be a separate provider; it will not impersonate whisper.cpp. Deferred delivery implementations include other streaming platforms and optional open-caption rendering. Open captions are never an implicit fallback for failed Twitch closed captions.
+
+OBS source identity is UUID-based. Multiple tracks, independent internal caption events, primary-track overlap policy, and delivery-enabled fields exist in the server model even though the first usable configuration targets one microphone. Protocol codec negotiation exists even though v1 accepts only PCM.
+
+The companion plugin host is process-oriented so one future integration can fail independently. Official package signature enforcement and Dev Mode unsigned-package handling are deferred with the installer/plugin packaging work. Offline authorization retains the low/normal/severe setting boundary; server inference still stops when Lumi is unreachable.
+
+Core owns only a reusable WebSocket upgrade registration mechanism. Removing or disabling `lumi_transcription` must not leave transcription routes, timers, sockets, workers, or global capabilities active.

docs/lumi-companion-transcription.md

@@ -0,0 +1,56 @@
+# Lumi Companion transcription (experimental)
+
+## Delivered foundation
+
+`lumi_transcription` is an independently disableable Lumi plugin. Core contains only a generic HTTP upgrade registry so plugins can attach versioned WebSocket transports. The plugin owns pairing/devices, session state, settings revisions, model/runtime manifests, worker supervision, caption stabilization, delivery, logs, routes, and shutdown.
+
+The implemented transport path is:
+
+```text
+OBS bridge boundary -> same-user companion IPC -> bounded companion queue
+-> authenticated TLS WebSocket -> bounded per-source Lumi buffer
+-> supervised TranscriptionProvider -> stabilized revision-aware caption
+-> WebSocket -> companion delivery adapter -> OBS native caption boundary
+```
+
+The repository does not yet contain a validated whisper streaming worker or a complete signed OBS bridge installer. Accordingly, the admin page reports inference setup as required and this branch does not claim live Twitch acceptance.
+
+## Trust and privacy
+
+- Pairing packages contain a cryptographically random credential that expires after 15 minutes and can be activated once. Lumi stores only its SHA-256 digest.
+- Activation returns a revocable device secret once. The Windows companion stores it with current-user DPAPI.
+- Device HTTP and WebSocket credentials require HTTPS/WSS. Insecure transport is available only from localhost when `LUMI_COMPANION_DEV_ALLOW_INSECURE=1` is explicitly set.
+- The OBS bridge never receives the Lumi credential and never connects to Lumi directly.
+- Audio frames are capped at 200 ms and recovery buffers at five seconds. Old or excess frames are dropped; capture paths never block for inference.
+- Raw audio is never logged or written to disk. JSON Lines diagnostics default to seven days and 256 MiB. Caption text can be disabled in diagnostics.
+
+## Pairing and installation milestone
+
+1. Serve Lumi through HTTPS and enable `lumi_transcription` under Admin > Plugins.
+2. Open Plugins > Transcription and create a pairing package. It is a bootstrap package for the generic companion, not a credential that should be shared or committed.
+3. On the Windows streaming computer, build the .NET 8 projects under `companion/` and run `Lumi.Companion.App pair <package>` for the current protocol milestone.
+4. Install a pinned runtime/model only after explicit confirmation. `small.en` is recommended; `small.en-q5_1` and `base.en` are fallbacks. Every artifact is checksum-verified before install.
+5. Configure `LUMI_TRANSCRIPTION_WORKER` with the supervised streaming-worker executable once that worker is built for the target host.
+
+The normal signed installer, Avalonia tray UI, managed bridge install/repair, source selector, and model benchmark wizard are not complete yet.
+
+## Operation and recovery
+
+Live start is rejected unless OBS reports an active stream. Test mode is allowed without streaming and sends captions only to the simulated delivery output. If a live stream or connection ends, delivery pauses and the session/model receive a 30-second grace period. Reconnection can resume that session; expiry finalizes and stops it. Recording without streaming does not create third-party delivery.
+
+Device and capability revocation take effect on the next authenticated request/connection. Worker crashes are bounded to three restart attempts per minute. Disabling the plugin unregisters the WebSocket route, closes clients/sessions, stops the worker, clears timers, and leaves unrelated plugins operational.
+
+## Diagnostics
+
+Use the Transcription admin page for provider, model, device, and log health. Recovery errors distinguish missing setup, unavailable inference, source inactivity, protocol incompatibility, and revoked access. The target-machine test must additionally record latency, resource, OBS missed-frame, audio-dropout, queue-drop, and network metrics using the acceptance template in `companion/docs/performance-acceptance-template.md`.
+
+## Known limitations
+
+- The .NET SDK and OBS SDK are not available in the development environment used for this milestone, so those projects have not been compiled here.
+- No real whisper.cpp streaming worker has been integrated or benchmarked.
+- The bridge skeleton does not yet run its named-pipe worker or selected-source audio callback.
+- Twitch toggleable caption behavior has not been tested; native API presence is not acceptance evidence.
+- The companion shell is a protocol/bootstrap executable, not yet the Avalonia tray application.
+- Installer signing, bridge repair, auto-start, source discovery/nested Program-scene evaluation, benchmark UX, and conflict-resolution UI remain pending.
+
+See `docs/adr/0001-companion-transcription-boundaries.md`, `protocol/companion-protocol-v1.md`, and `companion/docs/obs-native-caption-compatibility-spike.md`.