fix: diagnose production plugin updates

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Lumi changelog
 
+## 0.2.2
+
+- Added stage-specific plugin update failures beside the affected plugin and persisted the production failure stage in update state.
+- Validated the selected repository plugin before creating a rollback snapshot and removed the generic, context-free failed result.
+- Added an optional admin-controlled production diagnostics endpoint with fixed read-only checks, one-time hashed access keys, HTTPS enforcement, redaction, rate limiting, and audited access.
+
 ## 0.2.1
 
 - Fixed repository update checks being handled twice and incorrectly ending in a failed button state after a successful result.

TODO.md

@@ -677,6 +677,8 @@ This section is for Lumi chat answer feedback and AI Improvement Center work, no
 
 ## Done
 
+- 2026-07-18: Added production-stage plugin update diagnostics in core 0.2.2: selected plugin source is verified before snapshotting, failures record their exact stage and target in update state, and the affected plugin row displays the server error directly.
+- 2026-07-18: Added opt-in production diagnostics in core 0.2.2: administrators can issue or revoke a one-time access key for an HTTPS-only, rate-limited endpoint with five fixed read-only checks, recursive secret/path redaction, and audited access; no arbitrary command, SQL, file, URL, or write capability is exposed.
 - 2026-07-18: Released the follow-up core 0.2.1 update-page fix: update checks now have one client-side handler, successful results use clear module-specific wording, and exact core/plugin version controls are collapsed and list only the selected module’s versions.
 - 2026-07-18: Audited the 0.1.9-to-0.2.0 update boundary, removed core's direct dependency on OKF plugin files, made OBS WebSocket optional, added locked runtime dependency repair, corrected 1.2.0 to 0.2.0, added immutable release metadata and exact-version core/plugin installs, and produced a checksummed data-preserving core repair patch.
 

docs/production-diagnostics.md

@@ -0,0 +1,60 @@
+# Production diagnostics
+
+Lumi includes an optional, disabled-by-default diagnostics endpoint for investigating problems that only occur on a production installation. It is intended for trusted maintainers who need runtime evidence without being given shell, database, file, or administrative access.
+
+## Security model
+
+- Only an administrator can enable access, replace the key, or revoke it.
+- The generated bearer key is shown once. Lumi stores its SHA-256 hash, a short display prefix, and its creation time—not the usable key.
+- Remote requests require HTTPS. Requests over plain HTTP are accepted only from loopback. Lumi trusts forwarded HTTPS headers only from a reverse proxy on the same machine.
+- Disabled endpoints and invalid keys return `404`, and valid keys are limited to 20 requests per minute.
+- The request can select only one fixed, allowlisted check. It cannot provide commands, SQL, paths, URLs, module names, or code.
+- Results recursively redact credential-like fields, authorization values, secret query parameters, local paths, and email addresses. Output size and nesting are bounded.
+- Successful and failed authorized requests are recorded in Lumi's normal audit logs. The endpoint does not change application or plugin data; the audit record is its only routine write.
+
+This is intentionally visible to Lumi administrators under **Admin → Diagnostics**. It is not a hidden support account or backdoor.
+
+## Enable and connect
+
+1. Open **Admin → Diagnostics**.
+2. Choose **Create access key** and complete the timed confirmation.
+3. Copy the one-time key to the trusted diagnostic computer. Store it outside the repository or in Lumi's ignored `.secrets` directory.
+4. Send a JSON `POST` to `/api/diagnostics/v1/run` using the production HTTPS address:
+
+```sh
+curl -X POST "https://your-lumi-host/api/diagnostics/v1/run" \
+  -H "Authorization: Bearer $LUMI_DIAGNOSTICS_KEY" \
+  -H "Content-Type: application/json" \
+  --data '{"check":"update_state"}'
+```
+
+For repository maintainers, save the key and address in the ignored `.secrets/production-diagnostics.json` file:
+
+```json
+{
+  "base_url": "https://your-lumi-host",
+  "key": "lumi_diag_replace-with-the-one-time-key"
+}
+```
+
+The bundled client then runs an allowlisted check and prints its redacted JSON result:
+
+```sh
+npm run diagnostics:production -- update_state
+```
+
+`LUMI_DIAGNOSTICS_CONFIG` can point to a key file outside the repository. `LUMI_DIAGNOSTICS_URL` and `LUMI_DIAGNOSTICS_KEY` are also supported for ephemeral environments.
+
+The supported checks are:
+
+- `system_health`: runtime, database, dependency, disk-space, and recovery status.
+- `update_state`: the last update status and stage, recovery state, and snapshot summary.
+- `plugins`: plugin manifest and registry versions without plugin data.
+- `recent_errors`: recent warnings and errors after redaction.
+- `benchmark`: a bounded read-only database, plugin scan, and JSON timing workload.
+
+Replace or revoke the key from the same page. Replacement immediately invalidates the previous key.
+
+## Scope and limitations
+
+The endpoint deliberately does not expose arbitrary benchmarks or generic diagnostic commands. New checks must be implemented and reviewed in core before they can run. For incident response, start with `update_state`, `system_health`, and `recent_errors`; their request IDs can be correlated with the audit list on the admin page.

docs/updates.md

@@ -160,3 +160,15 @@ Admin update actions publish Server-Sent Events through
 Core update success returns a five-second in-page notice before refresh/restart.
 Plugin update success updates progress for the affected plugin action without a
 whole-page refresh, then restarts Lumi so the selected plugin code is loaded.
+
+Plugin failures are shown directly beside the affected plugin instead of only
+changing the action button label. The error identifies the failed stage:
+repository metadata, selected repository version, recovery preparation,
+rollback snapshot, file replacement, or installed-plugin verification. The
+same stage and error are persisted in `data/update-state.json`; a later success
+clears the stale failure.
+
+When a problem occurs only on production, the optional **Admin > Diagnostics**
+page can expose the redacted `update_state`, `system_health`, and
+`recent_errors` checks to a trusted maintainer without shell or write access.
+See [Production diagnostics](production-diagnostics.md).

knowledge/core/lumi-core.md

@@ -14,7 +14,7 @@ editable: false
 Lumi is the core web UI and bot runtime.
 ## Runtime
 Package: lumi-bot
-Version: 0.2.1
+Version: 0.2.2
 ## Routes
 - GET /api/events
 - POST /api/destructive-confirmations

package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "lumi-bot",
-  "version": "0.2.1",
+  "version": "0.2.2",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "lumi-bot",
-      "version": "0.2.1",
+      "version": "0.2.2",
       "dependencies": {
         "adm-zip": "^0.5.12",
         "better-sqlite3": "^11.5.0",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "lumi-bot",
-  "version": "0.2.1",
+  "version": "0.2.2",
   "private": true,
   "type": "commonjs",
   "scripts": {
@@ -11,6 +11,8 @@
     "verify:safe-files": "node scripts/verify-safe-files.js",
     "verify:uploads": "node scripts/verify-upload-security.js",
     "verify:updates": "node scripts/verify-release-metadata.js && node scripts/verify-update-system.js",
+    "verify:diagnostics": "node scripts/verify-production-diagnostics.js",
+    "diagnostics:production": "node scripts/production-diagnostics-client.js",
     "build:repair-patch": "node scripts/build-core-repair-patch.js && node scripts/verify-core-repair-patch.js",
     "verify:web-auth": "node scripts/verify-web-auth.js",
     "verify:destructive-actions": "node scripts/verify-destructive-actions.js",

release-index.json

@@ -2,6 +2,36 @@
   "schema_version": 1,
   "channel": "stable",
   "releases": [
+    {
+      "version": "0.2.2",
+      "ref": "refs/tags/v0.2.2",
+      "released_at": "2026-07-18",
+      "installable": true,
+      "rollback_safe": true,
+      "replaces_versions": [
+        "1.2.0"
+      ],
+      "data_policy": "preserve",
+      "dependency_policy": "sync_on_restart",
+      "migration_notes": "Adds production-stage plugin update diagnostics, validates repository plugin files before changing live files, and provides optional secured read-only production diagnostics.",
+      "plugins": {
+        "auto-vc": "0.1.6",
+        "birthday": "0.1.3",
+        "economy-framework": "0.2.10",
+        "economy-games": "0.1.7",
+        "expression-interaction": "0.2.1",
+        "lumi_ai": "0.8.2",
+        "moderation": "0.1.5",
+        "okf": "0.1.1",
+        "quotes": "0.1.2",
+        "sample-plugin": "0.1.0",
+        "throne_wishlist": "0.1.2",
+        "welcome_messages": "0.1.1"
+      },
+      "tools": {
+        "lumi_ai_web_search": "0.1.1"
+      }
+    },
     {
       "version": "0.2.1",
       "ref": "refs/tags/v0.2.1",

scripts/build-core-repair-patch.js

@@ -4,7 +4,7 @@ const path = require("path");
 const AdmZip = require("adm-zip");
 
 const root = path.join(__dirname, "..");
-const destination = path.join(root, "dist", "lumi-core-1.2.0-to-0.2.1-repair.zip");
+const destination = path.join(root, "dist", "lumi-core-1.2.0-to-0.2.2-repair.zip");
 const files = [
   "CHANGELOG.md",
   "README.md",
@@ -14,6 +14,7 @@ const files = [
   "run.js",
   "update-manifest.json",
   "docs/updates.md",
+  "docs/production-diagnostics.md",
   "docs/update-audit-0.2.0.md",
   "knowledge/core/lumi-core.md",
   "scripts/verify-release-metadata.js",
@@ -21,16 +22,21 @@ const files = [
   "scripts/build-core-repair-patch.js",
   "scripts/verify-core-repair-patch.js",
   "scripts/verify-update-system.js",
+  "scripts/verify-production-diagnostics.js",
+  "scripts/production-diagnostics-client.js",
   "scripts/verify-webui.js",
   "src/services/dependency-manager.js",
   "src/services/overlay-connectors.js",
   "src/services/repo-update.js",
+  "src/services/production-diagnostics.js",
+  "src/services/settings.js",
   "src/services/update-index.js",
   "src/services/update-manager.js",
   "src/services/update-repository.js",
   "src/web/server.js",
   "src/web/public/app.js",
-  "src/web/views/admin-updates.ejs"
+  "src/web/views/admin-updates.ejs",
+  "src/web/views/admin-diagnostics.ejs"
 ];
 
 const hashes = {};
@@ -47,10 +53,10 @@ for (const relativePath of files) {
 
 const manifest = {
   schema_version: 1,
-  name: "Lumi core 0.2.1 repair",
+  name: "Lumi core 0.2.2 repair",
   target: "core",
-  from_versions: ["0.1.9", "1.2.0", "0.2.0", "0.2.1"],
+  from_versions: ["0.1.9", "1.2.0", "0.2.0", "0.2.1", "0.2.2"],
-  to_version: "0.2.1",
+  to_version: "0.2.2",
   data_policy: "preserve",
   dependency_policy: "sync_on_restart",
   created_at: new Date().toISOString(),

scripts/production-diagnostics-client.js

@@ -0,0 +1,66 @@
+const fs = require("fs");
+const path = require("path");
+
+const root = path.join(__dirname, "..");
+const allowedChecks = new Set(["system_health", "update_state", "plugins", "recent_errors", "benchmark"]);
+const check = String(process.argv[2] || "system_health").trim();
+if (!allowedChecks.has(check)) {
+  console.error(`Choose one of: ${Array.from(allowedChecks).join(", ")}`);
+  process.exit(2);
+}
+
+const configPath = path.resolve(process.env.LUMI_DIAGNOSTICS_CONFIG || path.join(root, ".secrets", "production-diagnostics.json"));
+let fileConfig = {};
+try {
+  fileConfig = JSON.parse(fs.readFileSync(configPath, "utf8"));
+} catch (error) {
+  if (!process.env.LUMI_DIAGNOSTICS_URL || !process.env.LUMI_DIAGNOSTICS_KEY) {
+    console.error(`Diagnostics configuration was not found or is invalid: ${configPath}`);
+    process.exit(2);
+  }
+}
+
+const baseUrl = String(process.env.LUMI_DIAGNOSTICS_URL || fileConfig.base_url || "").trim();
+const key = String(process.env.LUMI_DIAGNOSTICS_KEY || fileConfig.key || "").trim();
+let endpoint;
+try {
+  const parsed = new URL(baseUrl);
+  const isLoopback = ["localhost", "127.0.0.1", "::1", "[::1]"].includes(parsed.hostname);
+  if (parsed.protocol !== "https:" && !(parsed.protocol === "http:" && isLoopback)) {
+    throw new Error("Production diagnostics require HTTPS (HTTP is allowed only for loopback).");
+  }
+  if (parsed.username || parsed.password) throw new Error("Do not put credentials in the diagnostics URL.");
+  endpoint = new URL("/api/diagnostics/v1/run", parsed);
+} catch (error) {
+  console.error(error.message || "Configure a valid production diagnostics URL.");
+  process.exit(2);
+}
+if (!key.startsWith("lumi_diag_")) {
+  console.error("The production diagnostics key is missing or invalid.");
+  process.exit(2);
+}
+
+const controller = new AbortController();
+const timeout = setTimeout(() => controller.abort(), 15000);
+fetch(endpoint, {
+  method: "POST",
+  headers: {
+    authorization: `Bearer ${key}`,
+    "content-type": "application/json",
+    accept: "application/json"
+  },
+  body: JSON.stringify({ check }),
+  signal: controller.signal
+})
+  .then(async (response) => {
+    const body = await response.json().catch(() => null);
+    if (!response.ok || body?.ok !== true) {
+      throw new Error(body?.error || `Diagnostics request failed with HTTP ${response.status}.`);
+    }
+    console.log(JSON.stringify(body, null, 2));
+  })
+  .catch((error) => {
+    console.error(error.name === "AbortError" ? "Diagnostics request timed out." : error.message);
+    process.exitCode = 1;
+  })
+  .finally(() => clearTimeout(timeout));

scripts/verify-all.js

@@ -11,6 +11,7 @@ const checks = [
   "scripts/verify-placeholders.js",
   "scripts/verify-release-metadata.js",
   "scripts/verify-update-system.js",
+  "scripts/verify-production-diagnostics.js",
   "scripts/verify-plugin-update-preserves-data.js",
   "scripts/verify-safe-files.js",
   "scripts/verify-upload-security.js",

scripts/verify-core-repair-patch.js

@@ -7,7 +7,7 @@ const AdmZip = require("adm-zip");
 const { verifyPatchPackage } = require("../src/services/update-manager");
 
 const root = path.join(__dirname, "..");
-const archivePath = path.join(root, "dist", "lumi-core-1.2.0-to-0.2.1-repair.zip");
+const archivePath = path.join(root, "dist", "lumi-core-1.2.0-to-0.2.2-repair.zip");
 assert.equal(fs.existsSync(archivePath), true, "build the repair patch first");
 const zip = new AdmZip(archivePath);
 const entries = zip.getEntries().filter((entry) => !entry.isDirectory);
@@ -15,9 +15,9 @@ const names = new Set(entries.map((entry) => entry.entryName.replace(/\\/g, "/")
 assert.equal(names.has("patch-manifest.json"), true);
 const manifest = JSON.parse(zip.readAsText("patch-manifest.json"));
 assert.equal(manifest.target, "core");
-assert.equal(manifest.to_version, "0.2.1");
+assert.equal(manifest.to_version, "0.2.2");
 assert.equal(manifest.data_policy, "preserve");
-assert.deepEqual(manifest.from_versions, ["0.1.9", "1.2.0", "0.2.0", "0.2.1"]);
+assert.deepEqual(manifest.from_versions, ["0.1.9", "1.2.0", "0.2.0", "0.2.1", "0.2.2"]);
 
 const forbidden = /^(?:data|plugins|node_modules|config|storage|uploads|logs|database|databases|knowledge\/(?:community|corrections))(?:\/|$)|^\.env(?:\.|$)|^\.secrets$/;
 for (const entry of entries) {
@@ -29,7 +29,7 @@ for (const [relativePath, expected] of Object.entries(manifest.files)) {
   assert.equal(actual, expected, `${relativePath} checksum`);
 }
 assert.equal(Object.keys(manifest.files).length + 1, entries.length, "every repair file must be checksummed");
-assert.equal(JSON.parse(zip.readAsText("package.json")).version, "0.2.1");
+assert.equal(JSON.parse(zip.readAsText("package.json")).version, "0.2.2");
 
 const simulation = fs.mkdtempSync(path.join(os.tmpdir(), "lumi-repair-simulation-"));
 try {
@@ -47,12 +47,14 @@ try {
     fs.writeFileSync(target, `preserve:${sentinel}`);
   }
   zip.extractAllTo(simulation, true);
-  assert.equal(verifyPatchPackage(simulation).to_version, "0.2.1");
+  assert.equal(verifyPatchPackage(simulation).to_version, "0.2.2");
   for (const sentinel of sentinels) {
     assert.equal(fs.readFileSync(path.join(simulation, sentinel), "utf8"), `preserve:${sentinel}`);
   }
-  assert.equal(JSON.parse(fs.readFileSync(path.join(simulation, "package.json"), "utf8")).version, "0.2.1");
+  assert.equal(JSON.parse(fs.readFileSync(path.join(simulation, "package.json"), "utf8")).version, "0.2.2");
   assert.equal(fs.existsSync(path.join(simulation, "src", "services", "dependency-manager.js")), true);
+  assert.equal(fs.existsSync(path.join(simulation, "src", "services", "production-diagnostics.js")), true);
+  assert.equal(fs.existsSync(path.join(simulation, "src", "web", "views", "admin-diagnostics.ejs")), true);
 } finally {
   fs.rmSync(simulation, { recursive: true, force: true });
 }

scripts/verify-production-diagnostics.js

@@ -0,0 +1,90 @@
+const assert = require("assert");
+const fs = require("fs");
+const path = require("path");
+
+const root = path.join(__dirname, "..");
+require("../src/services/db").migrate();
+const diagnostics = require("../src/services/production-diagnostics");
+
+assert.deepEqual(Object.keys(diagnostics.CHECKS), [
+  "system_health",
+  "update_state",
+  "plugins",
+  "recent_errors",
+  "benchmark"
+]);
+
+const key = "lumi_diag_verification-key_123";
+const hash = diagnostics.hashAccessKey(key);
+assert.match(hash, /^[a-f0-9]{64}$/);
+assert.equal(diagnostics.verifyAccessKey(key, hash), true);
+assert.equal(diagnostics.verifyAccessKey(`${key}x`, hash), false);
+assert.equal(diagnostics.verifyAccessKey("wrong-prefix", hash), false);
+assert.equal(diagnostics.isSecureDiagnosticRequest({ secure: true, ip: "203.0.113.10" }), true);
+assert.equal(diagnostics.isSecureDiagnosticRequest({ secure: false, ip: "127.0.0.1" }), true);
+assert.equal(diagnostics.isSecureDiagnosticRequest({ secure: false, ip: "203.0.113.10" }), false);
+
+const redacted = diagnostics.redactDiagnosticValue({
+  token: "top-secret",
+  message: `authorization=Bearer-value ${key} /mnt/c/private/lumi/file.js user@example.com ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefgh`,
+  nested: { password: "also-secret", url: "https://example.test/?api_key=secret-value&ok=1" }
+});
+const redactedJson = JSON.stringify(redacted);
+for (const forbidden of ["top-secret", "also-secret", key, "/mnt/c/private", "user@example.com", "secret-value", "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefgh"]) {
+  assert.equal(redactedJson.includes(forbidden), false, `diagnostic output leaked ${forbidden}`);
+}
+assert.match(redactedJson, /redacted|diagnostics key|local path|email/i);
+
+const fingerprint = `verify-${Date.now()}`;
+for (let index = 0; index < diagnostics.MAX_REQUESTS_PER_MINUTE; index += 1) {
+  const rate = diagnostics.consumeRateLimit(fingerprint, 120000);
+  assert.equal(rate.allowed, true);
+}
+assert.equal(diagnostics.consumeRateLimit(fingerprint, 120000).allowed, false);
+assert.equal(diagnostics.consumeRateLimit(fingerprint, 180000).allowed, true);
+
+for (const check of Object.keys(diagnostics.CHECKS)) {
+  const result = diagnostics.runDiagnosticCheck(check);
+  assert.equal(result.schema_version, 1);
+  assert.equal(result.check, check);
+  assert.equal(typeof result.duration_ms, "number");
+  assert.doesNotMatch(JSON.stringify(result), /\blumi_diag_[A-Za-z0-9_-]+\b/);
+}
+assert.throws(() => diagnostics.runDiagnosticCheck("shell"), /Unknown diagnostic check/);
+
+const serviceSource = fs.readFileSync(path.join(root, "src", "services", "production-diagnostics.js"), "utf8");
+for (const forbidden of [
+  /child_process/,
+  /\bexec(?:Sync)?\s*\(/,
+  /\bspawn(?:Sync)?\s*\(/,
+  /\beval\s*\(/,
+  /new\s+Function\s*\(/,
+  /\bfetch\s*\(/,
+  /require\(["'](?:https?|net|tls|dgram)["']\)/
+]) {
+  assert.doesNotMatch(serviceSource, forbidden, "diagnostics must not expose execution or network primitives");
+}
+
+const serverSource = fs.readFileSync(path.join(root, "src", "web", "server.js"), "utf8");
+const endpointIndex = serverSource.indexOf('app.post("/api/diagnostics/v1/run"');
+const configuredIndex = serverSource.indexOf("app.use(requireConfigured)");
+assert(endpointIndex > 0 && endpointIndex < configuredIndex, "diagnostics endpoint must remain available for production recovery");
+assert.match(serverSource, /authenticateDiagnosticsRequest\(req\)/);
+assert.match(serverSource, /app\.set\("trust proxy", "loopback"\)/);
+assert.match(serverSource, /app\.get\("\/admin\/diagnostics", requireRole\("admin"\)/);
+assert.match(serverSource, /app\.post\("\/admin\/diagnostics\/access\/renew", requireRole\("admin"\)/);
+assert.match(serverSource, /app\.post\("\/admin\/diagnostics\/access\/revoke", requireRole\("admin"\)/);
+
+const viewSource = fs.readFileSync(path.join(root, "src", "web", "views", "admin-diagnostics.ejs"), "utf8");
+assert.match(viewSource, /no remote control/i);
+assert.match(viewSource, /data-confirm-mode="modal"/);
+assert.match(viewSource, /\.secrets/);
+assert.match(viewSource, /\/api\/diagnostics\/v1\/run/);
+
+const clientSource = fs.readFileSync(path.join(root, "scripts", "production-diagnostics-client.js"), "utf8");
+assert.match(clientSource, /\.secrets["'], "production-diagnostics\.json"/);
+assert.match(clientSource, /method: "POST"/);
+assert.match(clientSource, /new URL\("\/api\/diagnostics\/v1\/run"/);
+assert.doesNotMatch(clientSource, /console\.(?:log|error)\(\s*key\s*\)/, "client must not print its access key");
+
+console.log("Production diagnostics verification passed: fixed read-only checks, redaction, key validation, rate limiting, HTTPS proxy trust, and admin-only controls.");

scripts/verify-release-metadata.js

@@ -4,8 +4,8 @@ const path = require("path");
 const { findSafeTarget } = require("../src/services/versioning");
 
 const root = path.join(__dirname, "..");
-const releaseVersion = "0.2.1";
+const releaseVersion = "0.2.2";
-const previousCoreVersion = "0.2.0";
+const previousCoreVersion = "0.2.1";
 const earliestCompatibleCoreVersion = "0.1.9";
 const changedPlugins = {
   "auto-vc": { from: "0.1.5", to: "0.1.6", knowledge: "auto-vc" },
@@ -52,7 +52,7 @@ assert.equal(releaseIndex.releases[0].version, releaseVersion);
 assert.equal(releaseIndex.releases[0].ref, `refs/tags/v${releaseVersion}`);
 assert.equal(releaseIndex.releases[1].version, previousCoreVersion);
 assert.equal(releaseIndex.releases[1].ref, `refs/tags/v${previousCoreVersion}`);
-assert.equal(releaseIndex.releases[2].version, earliestCompatibleCoreVersion);
+assert.equal(releaseIndex.releases.at(-1).version, earliestCompatibleCoreVersion);
 assert.equal(hasVersionHeading(readText("CHANGELOG.md"), releaseVersion), true);
 assert.match(readText("knowledge/core/lumi-core.md"), new RegExp(`^Version: ${escapeRegex(releaseVersion)}$`, "m"));
 
@@ -87,4 +87,4 @@ assert.equal(webSearch.minimum_lumi_version, "0.2.0");
 assert.equal(webSearch.minimum_lumi_ai_version, changedPlugins.lumi_ai.to);
 assert.equal(hasVersionHeading(readText("plugins/lumi_ai_web_search/CHANGELOG.md"), webSearch.version), true);
 
-console.log("Release metadata verification passed: core 0.2.1 and 11 changed plugin/tool packages.");
+console.log("Release metadata verification passed: core 0.2.2 and 11 changed plugin/tool packages.");

scripts/verify-update-system.js

@@ -16,7 +16,7 @@ function readJson(relativePath) {
 
 const releaseIndex = readJson("release-index.json");
 const releaseVersions = releaseIndex.releases.map((release) => release.version);
-assert.deepEqual(releaseVersions, ["0.2.1", "0.2.0", "0.1.9"]);
+assert.deepEqual(releaseVersions, ["0.2.2", "0.2.1", "0.2.0", "0.1.9"]);
 assert.equal(new Set(releaseVersions).size, releaseVersions.length, "release versions must be unique");
 for (const release of releaseIndex.releases) {
   assert.equal(normalizeRepositoryRef(release.ref), release.ref);
@@ -37,6 +37,7 @@ for (const [toolId, version] of Object.entries(currentRelease.tools)) {
 const baseTarget = {
   current_version: "0.2.0",
   available_versions: [
+    { version: "0.2.2", ref: "refs/tags/v0.2.2", rollback_safe: true },
     { version: "0.2.1", ref: "refs/tags/v0.2.1", rollback_safe: true },
     { version: "0.2.0", ref: "refs/tags/v0.2.0", rollback_safe: true },
     { version: "0.1.9", ref: "refs/tags/v0.1.9", rollback_safe: true }
@@ -61,7 +62,7 @@ const corrected = buildStatus({
   channel: "stable"
 });
 assert.equal(corrected.version_correction, true);
-assert.equal(corrected.safe_target_version, "0.2.1");
+assert.equal(corrected.safe_target_version, "0.2.2");
 assert.equal(corrected.update_available, true);
 assert.equal(corrected.blocked, false);
 
@@ -73,6 +74,12 @@ for (const required of ["data", "plugins", "node_modules", "knowledge/community"
 const serverSource = fs.readFileSync(path.join(root, "src", "web", "server.js"), "utf8");
 assert.doesNotMatch(serverSource, /require\([^\n]*plugins\/okf\/backend/);
 assert.match(serverSource, /global\.lumiFrameworks\?\.okf\?\.saveCorrection/);
+const repoUpdateSource = fs.readFileSync(path.join(root, "src", "services", "repo-update.js"), "utf8");
+assert.match(repoUpdateSource, /Plugin update failed while \$\{stage\}/);
+assert.match(repoUpdateSource, /last_update_stage: stage/);
+assert.match(repoUpdateSource, /stage = "preparing the selected repository version"[\s\S]*verifyPluginFiles\(pluginId, managed\.path, target\.safe_target_version\)[\s\S]*stage = "creating the rollback snapshot"/);
+assert.match(repoUpdateSource, /stage = "replacing plugin files"[\s\S]*applyPluginFiles\(path\.join\(managed\.path, "plugins", pluginId\)/);
+assert.match(repoUpdateSource, /last_update_stage: "complete",\s*last_error: null/);
 const connectorSource = fs.readFileSync(path.join(root, "src", "services", "overlay-connectors.js"), "utf8");
 assert.match(connectorSource, /try\s*{[\s\S]*require\("obs-websocket-js"\)/);
 

scripts/verify-webui.js

@@ -205,7 +205,9 @@ function verifySharedUpdateActions() {
   assert(serverSource.includes('`Core version ${status.core.safe_target_version} is available.`'));
   assert(serverSource.includes('`${plugin.name} version ${plugin.safe_target_version} is available.`'));
   assert(updates.includes("data-update-check-form"));
+  assert(updates.includes("data-update-inline-result"));
   assert(updates.includes('class="lumi-expandable-settings update-version-picker"'));
+  assert(appSource.includes('submitter.textContent = "Update failed"'));
   assert(!updates.includes("reinstall current"));
   assert(!updates.includes(" · core <%= release.core_version %>"));
 }

src/services/production-diagnostics.js

@@ -0,0 +1,316 @@
+const crypto = require("crypto");
+const fs = require("fs");
+const path = require("path");
+const { performance } = require("perf_hooks");
+const { db } = require("./db");
+const { dependencyIssues } = require("./dependency-manager");
+const { listLogs, log } = require("./logger");
+const { getPlugins, scanPluginDirectories } = require("./plugins");
+const { readRecoveryMarker } = require("./recovery-mode");
+const { getSetting, setSetting } = require("./settings");
+const { readUpdateState } = require("./update-repository");
+
+const repoRoot = path.join(__dirname, "..", "..");
+const packageJson = require(path.join(repoRoot, "package.json"));
+const TOKEN_PREFIX = "lumi_diag_";
+const MAX_REQUESTS_PER_MINUTE = 20;
+const requestWindows = new Map();
+const CHECKS = Object.freeze({
+  system_health: "Runtime, database, dependency, disk-space, and recovery health.",
+  update_state: "Latest local update state, recovery marker, and snapshot summary.",
+  plugins: "Installed plugin manifests and registry versions without plugin data.",
+  recent_errors: "Recent warning/error records with secrets and local paths redacted.",
+  benchmark: "Bounded read-only database, plugin-scan, and serialization timing."
+});
+
+function diagnosticsAccessStatus() {
+  return {
+    enabled: getSetting("production_diagnostics_enabled", false) === true,
+    configured: Boolean(getSetting("production_diagnostics_key_hash", "")),
+    key_prefix: String(getSetting("production_diagnostics_key_prefix", "") || ""),
+    created_at: getSetting("production_diagnostics_key_created_at", null),
+    checks: Object.entries(CHECKS).map(([id, description]) => ({ id, description }))
+  };
+}
+
+function issueDiagnosticsAccessKey() {
+  const key = `${TOKEN_PREFIX}${crypto.randomBytes(32).toString("base64url")}`;
+  setSetting("production_diagnostics_key_hash", hashAccessKey(key));
+  setSetting("production_diagnostics_key_prefix", `${key.slice(0, TOKEN_PREFIX.length + 8)}…`);
+  setSetting("production_diagnostics_key_created_at", new Date().toISOString());
+  setSetting("production_diagnostics_enabled", true);
+  log("warn", "Production diagnostics access key rotated", { key_prefix: `${key.slice(0, TOKEN_PREFIX.length + 8)}…` });
+  return key;
+}
+
+function revokeDiagnosticsAccess() {
+  setSetting("production_diagnostics_enabled", false);
+  setSetting("production_diagnostics_key_hash", "");
+  setSetting("production_diagnostics_key_prefix", "");
+  setSetting("production_diagnostics_key_created_at", null);
+  requestWindows.clear();
+  log("warn", "Production diagnostics access revoked");
+}
+
+function authenticateDiagnosticsRequest(req, now = Date.now()) {
+  if (!diagnosticsAccessStatus().enabled) return { allowed: false, status: 404, reason: "disabled" };
+  if (!isSecureDiagnosticRequest(req)) return { allowed: false, status: 404, reason: "secure_transport_required" };
+  const authorization = String(req.get?.("authorization") || "");
+  const match = authorization.match(/^Bearer\s+([^\s]+)$/i);
+  const candidate = match?.[1] || "";
+  const storedHash = String(getSetting("production_diagnostics_key_hash", "") || "");
+  if (!verifyAccessKey(candidate, storedHash)) return { allowed: false, status: 404, reason: "invalid_key" };
+  const fingerprint = hashAccessKey(candidate).slice(0, 20);
+  const rate = consumeRateLimit(fingerprint, now);
+  if (!rate.allowed) return { allowed: false, status: 429, reason: "rate_limited", retry_after_seconds: rate.retry_after_seconds };
+  return { allowed: true, fingerprint, remaining: rate.remaining };
+}
+
+function runDiagnosticCheck(checkId) {
+  const check = String(checkId || "").trim();
+  if (!Object.hasOwn(CHECKS, check)) throw new Error("Unknown diagnostic check.");
+  const started = performance.now();
+  let result;
+  if (check === "system_health") result = systemHealth();
+  else if (check === "update_state") result = updateState();
+  else if (check === "plugins") result = pluginInventory();
+  else if (check === "recent_errors") result = recentErrors();
+  else result = boundedBenchmark();
+  return redactDiagnosticValue({
+    schema_version: 1,
+    check,
+    generated_at: new Date().toISOString(),
+    duration_ms: round(performance.now() - started),
+    result
+  });
+}
+
+function auditDiagnosticRequest(values = {}) {
+  log(values.ok === false ? "warn" : "info", "Production diagnostics request", {
+    request_id: values.request_id,
+    check: values.check,
+    ok: values.ok !== false,
+    key_fingerprint: values.fingerprint || null,
+    reason: values.reason || null,
+    duration_ms: values.duration_ms || null
+  });
+}
+
+function recentDiagnosticAudit(limit = 30) {
+  return listLogs({ limit: 250 })
+    .filter((entry) => ["Production diagnostics request", "Production diagnostics access key rotated", "Production diagnostics access revoked"].includes(entry.message))
+    .slice(0, Math.max(1, Math.min(100, Number(limit) || 30)))
+    .map((entry) => redactDiagnosticValue(entry));
+}
+
+function systemHealth() {
+  let database = { ok: false };
+  try {
+    database = { ok: db.prepare("SELECT 1 AS ok").get()?.ok === 1 };
+  } catch (error) {
+    database = { ok: false, error: error.message };
+  }
+  let disk = null;
+  try {
+    if (typeof fs.statfsSync !== "function") throw new Error("Disk statistics are unavailable on this Node.js version.");
+    const stats = fs.statfsSync(repoRoot);
+    disk = {
+      available_bytes: Number(stats.bavail) * Number(stats.bsize),
+      total_bytes: Number(stats.blocks) * Number(stats.bsize)
+    };
+  } catch {
+    disk = { available: false };
+  }
+  const dependencies = dependencyIssues(repoRoot);
+  return {
+    core_version: packageJson.version,
+    node_version: process.version,
+    platform: process.platform,
+    architecture: process.arch,
+    uptime_seconds: Math.floor(process.uptime()),
+    memory: process.memoryUsage(),
+    database,
+    disk,
+    dependencies: {
+      ready: dependencies.filter((item) => !item.optional).length === 0,
+      issues: dependencies
+    },
+    recovery: summarizeRecovery(readRecoveryMarker())
+  };
+}
+
+function updateState() {
+  const state = readUpdateState();
+  const index = readJson(path.join(repoRoot, "data", "snapshots", "index.json"), []);
+  const snapshots = Array.isArray(index) ? index : [];
+  return {
+    core_version: packageJson.version,
+    last_update: {
+      status: state.last_update_status || null,
+      target_kind: state.last_target_kind || null,
+      target_id: state.last_target_id || null,
+      target_version: state.last_target_version || null,
+      stage: state.last_update_stage || null,
+      error: state.last_error || null,
+      updated_at: state.last_update_at || state.updated_at || null,
+      source_ref: state.branch || null,
+      commit: state.commit || null
+    },
+    recovery: summarizeRecovery(readRecoveryMarker()),
+    snapshots: {
+      available: snapshots.filter((entry) => entry.status === "available").length,
+      latest: snapshots
+        .filter((entry) => entry.status === "available")
+        .sort((left, right) => Number(right.createdAt) - Number(left.createdAt))
+        .slice(0, 10)
+        .map((entry) => ({
+          type: entry.type,
+          plugin_id: entry.pluginId || null,
+          from_version: entry.from_version || null,
+          to_version: entry.to_version || null,
+          created_at: entry.createdAt || null,
+          storage_bytes: entry.storage_bytes || null
+        }))
+    }
+  };
+}
+
+function pluginInventory() {
+  const registry = new Map(getPlugins().map((plugin) => [plugin.id, plugin]));
+  return scanPluginDirectories().map((plugin) => ({
+    id: plugin.id,
+    name: plugin.name,
+    manifest_version: plugin.version,
+    registry_version: registry.get(plugin.id)?.version || null,
+    enabled: Boolean(registry.get(plugin.id)?.enabled),
+    version_matches_registry: !registry.has(plugin.id) || registry.get(plugin.id)?.version === plugin.version
+  })).sort((left, right) => left.id.localeCompare(right.id));
+}
+
+function recentErrors() {
+  return listLogs({ limit: 50, levels: ["warn", "error"] }).map((entry) => ({
+    level: entry.level,
+    message: entry.message,
+    details: entry.details,
+    created_at: entry.created_at
+  }));
+}
+
+function boundedBenchmark() {
+  const databaseStarted = performance.now();
+  for (let index = 0; index < 100; index += 1) db.prepare("SELECT 1 AS ok").get();
+  const databaseMs = performance.now() - databaseStarted;
+  const pluginStarted = performance.now();
+  let pluginCount = 0;
+  for (let index = 0; index < 5; index += 1) pluginCount = scanPluginDirectories().length;
+  const pluginMs = performance.now() - pluginStarted;
+  const serializationStarted = performance.now();
+  const sample = Array.from({ length: 250 }, (_, index) => ({ index, status: "ok", version: packageJson.version }));
+  for (let index = 0; index < 20; index += 1) JSON.stringify(sample);
+  const serializationMs = performance.now() - serializationStarted;
+  return {
+    fixed_workload: true,
+    database_select_100_ms: round(databaseMs),
+    plugin_scan_5_ms: round(pluginMs),
+    plugin_count: pluginCount,
+    json_serialize_20_ms: round(serializationMs)
+  };
+}
+
+function redactDiagnosticValue(value, key = "", depth = 0) {
+  if (depth > 8) return "[truncated]";
+  if (/(?:token|secret|password|passwd|authorization|cookie|session|credential|private.?key|api.?key)/i.test(key)) return "[redacted]";
+  if (typeof value === "string") return redactString(value).slice(0, 4000);
+  if (typeof value === "bigint") return Number(value);
+  if (Array.isArray(value)) return value.slice(0, 100).map((item) => redactDiagnosticValue(item, key, depth + 1));
+  if (value && typeof value === "object") {
+    return Object.fromEntries(Object.entries(value).slice(0, 100).map(([childKey, child]) => [childKey, redactDiagnosticValue(child, childKey, depth + 1)]));
+  }
+  return value;
+}
+
+function redactString(value) {
+  return String(value)
+    .replace(/\blumi_diag_[A-Za-z0-9_-]+\b/g, "[diagnostics key]")
+    .replace(/(https?:\/\/)[^/@\s:]+:[^/@\s]+@/gi, "$1[credentials]@")
+    .replace(/\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\b/g, "[signed token]")
+    .replace(/\b([a-z0-9_-]*(?:token|secret|session|password|passwd|api[_-]?key|auth)[a-z0-9_-]*)\s*[:=]\s*[^,\s;]+/gi, "$1=[redacted]")
+    .replace(/\b(?:bearer|basic)\s+[a-z0-9._~+/=-]+/gi, "[auth header]")
+    .replace(/([?&](?:token|secret|session|password|api_key|apikey|auth|code)=)[^&#\s]+/gi, "$1[redacted]")
+    .replace(/(?:[A-Za-z]:\\|\\\\)[^\s<>\"|?*]+/g, "[local path]")
+    .replace(/\/(?:home|Users|mnt|var|tmp|opt|root)\/[^\s)\]}>]+/g, "[local path]")
+    .replace(/[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi, "[email]")
+    .replace(/[A-Za-z0-9+_=-]{40,}/g, (candidate) => /[A-Za-z]/.test(candidate) && /\d/.test(candidate) ? "[high-entropy value]" : candidate);
+}
+
+function hashAccessKey(value) {
+  return crypto.createHash("sha256").update(String(value || "")).digest("hex");
+}
+
+function verifyAccessKey(candidate, expectedHash) {
+  if (!String(candidate || "").startsWith(TOKEN_PREFIX) || !/^[a-f0-9]{64}$/i.test(String(expectedHash || ""))) return false;
+  const actual = Buffer.from(hashAccessKey(candidate), "hex");
+  const expected = Buffer.from(expectedHash, "hex");
+  return actual.length === expected.length && crypto.timingSafeEqual(actual, expected);
+}
+
+function consumeRateLimit(fingerprint, now = Date.now()) {
+  const windowStart = now - (now % 60000);
+  const current = requestWindows.get(fingerprint);
+  const state = !current || current.window_start !== windowStart ? { window_start: windowStart, count: 0 } : current;
+  state.count += 1;
+  requestWindows.set(fingerprint, state);
+  if (requestWindows.size > 1000) {
+    for (const [key, value] of requestWindows.entries()) if (value.window_start < windowStart) requestWindows.delete(key);
+  }
+  return {
+    allowed: state.count <= MAX_REQUESTS_PER_MINUTE,
+    remaining: Math.max(0, MAX_REQUESTS_PER_MINUTE - state.count),
+    retry_after_seconds: Math.max(1, Math.ceil((windowStart + 60000 - now) / 1000))
+  };
+}
+
+function isSecureDiagnosticRequest(req) {
+  if (req.secure === true) return true;
+  const address = String(req.ip || req.socket?.remoteAddress || "");
+  return ["127.0.0.1", "::1", "::ffff:127.0.0.1"].includes(address);
+}
+
+function summarizeRecovery(marker) {
+  if (!marker) return { active: false };
+  return {
+    active: ["pending", "applying", "verifying", "failed", "stale"].includes(marker.status),
+    status: marker.status || null,
+    target_kind: marker.target_kind || null,
+    target_id: marker.target_id || null,
+    from_version: marker.from_version || null,
+    to_version: marker.to_version || null,
+    error: marker.error || null,
+    updated_at: marker.updated_at || null
+  };
+}
+
+function readJson(filePath, fallback) {
+  try { return JSON.parse(fs.readFileSync(filePath, "utf8")); } catch { return fallback; }
+}
+
+function round(value) {
+  return Math.round(Number(value) * 100) / 100;
+}
+
+module.exports = {
+  CHECKS,
+  MAX_REQUESTS_PER_MINUTE,
+  auditDiagnosticRequest,
+  authenticateDiagnosticsRequest,
+  consumeRateLimit,
+  diagnosticsAccessStatus,
+  hashAccessKey,
+  issueDiagnosticsAccessKey,
+  isSecureDiagnosticRequest,
+  recentDiagnosticAudit,
+  redactDiagnosticValue,
+  revokeDiagnosticsAccess,
+  runDiagnosticCheck,
+  verifyAccessKey
+};

src/services/repo-update.js

@@ -65,17 +65,6 @@ function verifyPluginFiles(pluginId, rootPath = repoRoot, expectedVersion = null
   return metadata;
 }
 
-function applyPluginFromRepositorySnapshot(remote, repositoryRef, pluginId, expectedVersion) {
-  const managed = ensureManagedRepo(remote, repositoryRef);
-  const pluginRoot = path.join(managed.path, "plugins", pluginId);
-  if (!fs.existsSync(path.join(pluginRoot, "plugin.json"))) {
-    throw new Error(`Plugin ${pluginId} was not found in ${repositoryRef}.`);
-  }
-  verifyPluginFiles(pluginId, managed.path, expectedVersion);
-  applyPluginFiles(pluginRoot, pluginId, { preserveData: true });
-  return managed;
-}
-
 function targetForRequestedVersion(baseTarget, requestedVersion, label) {
   if (!requestedVersion) return baseTarget;
   const version = parseSemver(requestedVersion)?.raw;
@@ -166,6 +155,8 @@ async function applyCoreUpdate({ source = "stable", remote = null, version = nul
         branch: managed.branch,
         last_update_at: new Date().toISOString(),
         last_update_status: "complete",
+        last_update_stage: "complete",
+        last_error: null,
         last_snapshot_id: record.id,
         last_target_kind: "core",
         last_target_version: target.safe_target_version
@@ -206,28 +197,41 @@ async function applyCoreUpdate({ source = "stable", remote = null, version = nul
 
 async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote = null, version = null, publish } = {}) {
   return withOperation(`plugin:${pluginId}`, async () => {
-    const status = getUpdateStatus({ source, remote });
+    let status = null;
-    const statusTarget = status.plugins.find((plugin) => plugin.id === pluginId);
+    let target = null;
-    if (!statusTarget) throw new Error("Plugin was not found in the local or repository catalog.");
+    let marker = null;
-    const target = targetForRequestedVersion(statusTarget, version, "Plugin version");
+    let managed = null;
-    if (target.blocked) throw new Error(target.blocked_reason || "Plugin update is blocked.");
-    if (!target.update_available) throw new Error("No safe plugin update target is available.");
-    const marker = createRecoveryMarker({
-      target_kind: "plugin",
-      target_id: pluginId,
-      from_version: target.current_version,
-      to_version: target.safe_target_version,
-      source_branch: target.source_branch,
-      update_method: target.installed === false ? "git_install" : "git",
-      rollback_safe: target.rollback_safe,
-      major_crossing: target.major_crossing
-    });
     let snapshot = null;
     let snapshotRecord = null;
+    let stage = "checking repository metadata";
     try {
+      status = getUpdateStatus({ source, remote });
+      const statusTarget = status.plugins.find((plugin) => plugin.id === pluginId);
+      if (!statusTarget) throw new Error("Plugin was not found in the local or repository catalog.");
+      target = targetForRequestedVersion(statusTarget, version, "Plugin version");
+      if (target.blocked) throw new Error(target.blocked_reason || "Plugin update is blocked.");
+      if (!target.update_available) throw new Error("No plugin update target is available.");
       emitProgress(publish, "update:queued", { target: "plugin", plugin_id: pluginId });
       emitProgress(publish, "update:metadata", target);
+
+      stage = "preparing the selected repository version";
+      managed = ensureManagedRepo(status.remote, target.source_branch);
+      verifyPluginFiles(pluginId, managed.path, target.safe_target_version);
+
+      stage = "preparing recovery";
+      marker = createRecoveryMarker({
+        target_kind: "plugin",
+        target_id: pluginId,
+        from_version: target.current_version,
+        to_version: target.safe_target_version,
+        source_branch: target.source_branch,
+        update_method: target.installed === false ? "git_install" : "git",
+        rollback_safe: target.rollback_safe,
+        major_crossing: target.major_crossing
+      });
       updateRecoveryMarker({ status: "applying" });
+
+      stage = "creating the rollback snapshot";
       snapshot = await createSnapshot({
         type: "plugin",
         pluginId,
@@ -249,14 +253,11 @@ async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote =
       emitProgress(publish, "update:snapshot", { target: "plugin", plugin_id: pluginId, snapshot_id: snapshotRecord.id });
       emitProgress(publish, "update:download", { target: "plugin", plugin_id: pluginId, branch: target.source_branch });
       emitProgress(publish, "update:apply", { target: "plugin", plugin_id: pluginId });
-      const managed = applyPluginFromRepositorySnapshot(
+      stage = "replacing plugin files";
-        status.remote,
+      applyPluginFiles(path.join(managed.path, "plugins", pluginId), pluginId, { preserveData: true });
-        target.source_branch,
-        pluginId,
-        target.safe_target_version
-      );
       updateRecoveryMarker({ status: "verifying" });
       emitProgress(publish, "update:verify", { target: "plugin", plugin_id: pluginId });
+      stage = "verifying the installed plugin";
       verifyPluginFiles(pluginId, repoRoot, target.safe_target_version);
       syncPluginRegistry();
       const record = snapshotRecord;
@@ -266,6 +267,8 @@ async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote =
         branch: managed.branch,
         last_update_at: new Date().toISOString(),
         last_update_status: "complete",
+        last_update_stage: "complete",
+        last_error: null,
         last_snapshot_id: record.id,
         last_target_kind: "plugin",
         last_target_id: pluginId,
@@ -275,6 +278,9 @@ async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote =
       emitProgress(publish, "update:complete", { target: "plugin", plugin_id: pluginId, snapshot_id: record.id });
       return { status: "complete", restart_required: true, snapshot: record, target };
     } catch (error) {
+      const originalMessage = error?.message || String(error || "Unknown update error.");
+      error.message = `Plugin update failed while ${stage}: ${originalMessage}`;
+      const recoveryDiagnostics = [];
       if (snapshotRecord) {
         try {
           restoreSnapshot(snapshotRecord.id, {
@@ -284,23 +290,36 @@ async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote =
             allowUnsafeMajorRollback: true
           });
         } catch (restoreError) {
-          markRecoveryMarkerFailed(restoreError);
+          recoveryDiagnostics.push(`automatic restore failed: ${restoreError.message}`);
         }
       } else if (snapshot) {
         try {
           discardSnapshot(snapshot);
-        } catch {
+        } catch (discardError) {
-          // Ignore cleanup failures.
+          recoveryDiagnostics.push(`incomplete snapshot cleanup failed: ${discardError.message}`);
         }
       }
-      markRecoveryMarkerFailed(error);
+      if (marker) {
-      writeUpdateState({
+        try {
-        last_update_at: new Date().toISOString(),
+          markRecoveryMarkerFailed(error);
-        last_update_status: "failed",
+        } catch (markerError) {
-        last_target_kind: "plugin",
+          recoveryDiagnostics.push(`recovery marker update failed: ${markerError.message}`);
-        last_target_id: pluginId,
+        }
-        last_error: error.message
+      }
-      });
+      if (recoveryDiagnostics.length) error.message += ` Recovery diagnostics: ${recoveryDiagnostics.join("; ")}`;
+      try {
+        writeUpdateState({
+          last_update_at: new Date().toISOString(),
+          last_update_status: "failed",
+          last_target_kind: "plugin",
+          last_target_id: pluginId,
+          last_target_version: target?.safe_target_version || null,
+          last_update_stage: stage,
+          last_error: error.message
+        });
+      } catch (stateError) {
+        error.message += ` Update-state diagnostics could not be saved: ${stateError.message}`;
+      }
       emitProgress(publish, "update:failed", { target: "plugin", plugin_id: pluginId, error: error.message });
       throw error;
     }

src/services/settings.js

@@ -68,6 +68,7 @@ function ensureDefaults() {
     discord_redirect_uri: envString("DISCORD_REDIRECT_URI", ""),
     auto_update_enabled: envBoolean("AUTO_UPDATE_ENABLED", false),
     auto_update_interval_minutes: envNumber("AUTO_UPDATE_INTERVAL_MINUTES", 60),
+    production_diagnostics_enabled: false,
     git_remote: envString("GIT_REMOTE", "origin"),
     git_branch: envString("GIT_BRANCH", "main"),
     bot_avatar_url: null,

src/web/public/app.js

@@ -1470,7 +1470,8 @@
       row.textContent = message;
       updateLog.prepend(row);
     }
-    const inlineResult = form?.closest(".card")?.querySelector("[data-update-inline-result]");
+    const resultScope = form?.closest(".plugin-update-row, [data-update-target='core'], .card");
+    const inlineResult = resultScope?.querySelector("[data-update-inline-result]");
     if (inlineResult) {
       inlineResult.className = `hint ${level === "danger" ? "status-danger" : level === "success" ? "status-success" : ""}`.trim();
       inlineResult.textContent = message;
@@ -1564,7 +1565,7 @@
           }
         } catch (error) {
           if (isStateButton) window.LumiStateButton?.error?.(submitter);
-          else if (submitter) submitter.textContent = "Failed";
+          else if (submitter) submitter.textContent = "Update failed";
           appendUpdateLog(error.message, "danger", form);
         } finally {
           if (!isStateButton && submitter) {

src/web/server.js

@@ -106,6 +106,15 @@ const {
   clearRecoveryMarker,
   updateRecoveryMarker
 } = require("../services/recovery-mode");
+const {
+  auditDiagnosticRequest,
+  authenticateDiagnosticsRequest,
+  diagnosticsAccessStatus,
+  issueDiagnosticsAccessKey,
+  recentDiagnosticAudit,
+  revokeDiagnosticsAccess,
+  runDiagnosticCheck
+} = require("../services/production-diagnostics");
 const {
   generateCommandPreview,
   previewParts
@@ -2876,6 +2885,10 @@ async function verifyYouTubeSettings(settings) {
 
 function createWebServer({ loadPlugins, discordClient }) {
   const app = express();
+  // Only trust forwarding headers from a reverse proxy on this machine. This
+  // lets the diagnostics endpoint recognize HTTPS without trusting arbitrary
+  // client-supplied X-Forwarded-Proto headers.
+  app.set("trust proxy", "loopback");
   const webhooks = createWebhookService();
   placeholders.registerCorePlaceholders();
   placeholders.registerPlatformPlaceholders({
@@ -3182,6 +3195,43 @@ function createWebServer({ loadPlugins, discordClient }) {
   };
 
   registerPublicOverlayRoutes(app);
+  app.post("/api/diagnostics/v1/run", (req, res) => {
+    res.set("Cache-Control", "no-store");
+    res.set("Pragma", "no-cache");
+    const authentication = authenticateDiagnosticsRequest(req);
+    if (!authentication.allowed) {
+      if (authentication.status === 429) {
+        res.set("Retry-After", String(authentication.retry_after_seconds));
+        return res.status(429).json({ ok: false, error: "Too many diagnostic requests. Try again shortly." });
+      }
+      return res.status(404).json({ error: "Not found." });
+    }
+    const requestId = crypto.randomUUID();
+    const startedAt = Date.now();
+    const check = String(req.body?.check || "").trim();
+    res.set("X-RateLimit-Remaining", String(authentication.remaining));
+    try {
+      const diagnostic = runDiagnosticCheck(check);
+      auditDiagnosticRequest({
+        request_id: requestId,
+        check,
+        ok: true,
+        fingerprint: authentication.fingerprint,
+        duration_ms: Date.now() - startedAt
+      });
+      return res.json({ ok: true, request_id: requestId, diagnostic });
+    } catch (error) {
+      auditDiagnosticRequest({
+        request_id: requestId,
+        check,
+        ok: false,
+        fingerprint: authentication.fingerprint,
+        reason: error.message,
+        duration_ms: Date.now() - startedAt
+      });
+      return res.status(400).json({ ok: false, request_id: requestId, error: error.message });
+    }
+  });
   app.use(requireConfigured);
   app.get("/api/events", requireAuth, subscribeWebEvents);
   app.post("/api/destructive-confirmations", requireAuth, (req, res) => {
@@ -5719,6 +5769,62 @@ function createWebServer({ loadPlugins, discordClient }) {
     res.redirect("/admin/theming");
   });
 
+  const renderDiagnosticsAdmin = (res, values = {}) => {
+    res.set("Cache-Control", "no-store");
+    res.render("admin-diagnostics", {
+      title: "Production diagnostics",
+      diagnosticsAccess: diagnosticsAccessStatus(),
+      diagnosticAudit: recentDiagnosticAudit(),
+      issuedKey: null,
+      diagnosticResult: null,
+      diagnosticError: null,
+      selectedCheck: "system_health",
+      ...values
+    });
+  };
+
+  app.get("/admin/diagnostics", requireRole("admin"), (_req, res) => {
+    renderDiagnosticsAdmin(res);
+  });
+
+  app.post("/admin/diagnostics/run", requireRole("admin"), (req, res) => {
+    const requestId = crypto.randomUUID();
+    const check = String(req.body.check || "").trim();
+    const startedAt = Date.now();
+    try {
+      const diagnosticResult = runDiagnosticCheck(check);
+      auditDiagnosticRequest({
+        request_id: requestId,
+        check,
+        ok: true,
+        fingerprint: `admin:${req.session.user.id}`,
+        duration_ms: Date.now() - startedAt
+      });
+      renderDiagnosticsAdmin(res, { diagnosticResult, selectedCheck: check });
+    } catch (error) {
+      auditDiagnosticRequest({
+        request_id: requestId,
+        check,
+        ok: false,
+        fingerprint: `admin:${req.session.user.id}`,
+        reason: error.message,
+        duration_ms: Date.now() - startedAt
+      });
+      renderDiagnosticsAdmin(res, { diagnosticError: error.message, selectedCheck: check });
+    }
+  });
+
+  app.post("/admin/diagnostics/access/renew", requireRole("admin"), (_req, res) => {
+    const issuedKey = issueDiagnosticsAccessKey();
+    renderDiagnosticsAdmin(res, { issuedKey });
+  });
+
+  app.post("/admin/diagnostics/access/revoke", requireRole("admin"), (req, res) => {
+    revokeDiagnosticsAccess();
+    setFlash(req, "success", "Production diagnostics access revoked.");
+    res.redirect("/admin/diagnostics");
+  });
+
   app.get("/admin/logs", requireRole("admin"), (req, res) => {
     const range = parseLogRange(req.query.range);
     const limit = parseLogLimit(req.query.limit);
@@ -6992,6 +7098,7 @@ function collectNavItems(user, pluginNav, currentPath) {
       section: "admin"
     },
     { label: "Feedback review", path: "/admin/feedback", role: "admin", section: "admin" },
+    { label: "Diagnostics", path: "/admin/diagnostics", role: "admin", section: "admin" },
     { label: "Logs", path: "/admin/logs", role: "admin", section: "admin" },
     { label: "Updates", path: "/admin/updates", role: "admin", section: "admin" },
     {
@@ -7252,6 +7359,7 @@ function getDefaultNavIcon(item) {
   if (pathName === "/admin/navigation") return "settings";
   if (pathName === "/admin/theming") return "theming";
   if (pathName === "/admin/privileges") return "privileges";
+  if (pathName === "/admin/diagnostics") return "admin";
   if (pathName === "/admin/logs") return "logs";
   if (pathName === "/admin/updates") return "updates";
   if (pathName === "/admin/commands") return "commands";

src/web/views/admin-diagnostics.ejs

@@ -0,0 +1,111 @@
+<%- include("partials/layout-top", { title }) %>
+
+<section class="card">
+  <%- include("partials/page-header", {
+    eyebrow: "Maintenance",
+    pageTitle: "Production diagnostics",
+    description: "Run safe health checks locally or grant temporary read-only diagnostic access for production troubleshooting."
+  }) %>
+
+  <div class="callout info">
+    <strong>Insight only — no remote control</strong>
+    <p>This service has no shell, file browser, SQL input, URL fetching, or write actions. It only runs the fixed checks listed below, redacts secrets and local paths, rate-limits requests, and records access in Lumi's logs.</p>
+  </div>
+
+  <div class="update-meta-grid">
+    <div><span>Remote access</span><strong><%= diagnosticsAccess.enabled ? "Enabled" : "Disabled" %></strong></div>
+    <div><span>Access key</span><strong><%= diagnosticsAccess.configured ? diagnosticsAccess.key_prefix : "Not created" %></strong></div>
+    <div><span>Created</span><strong><%= diagnosticsAccess.created_at ? new Date(diagnosticsAccess.created_at).toLocaleString() : "Never" %></strong></div>
+    <div><span>Endpoint</span><strong><code>/api/diagnostics/v1/run</code></strong></div>
+  </div>
+
+  <% if (issuedKey) { %>
+  <div class="callout success">
+    <strong>Save this key now</strong>
+    <p>It is shown once. Store it on the computer that will run diagnostics, preferably in the ignored <code>.secrets</code> directory. Lumi stores only its hash.</p>
+    <div class="input-action-row">
+      <input value="<%= issuedKey %>" readonly aria-label="New production diagnostics key" />
+      <button class="button subtle" type="button" data-copy="<%= issuedKey %>"><span data-copy-label>Copy key</span></button>
+    </div>
+  </div>
+  <% } %>
+
+  <div class="inline-actions">
+    <form method="post" action="/admin/diagnostics/access/renew" data-confirm-mode="modal" data-confirm-title="<%= diagnosticsAccess.configured ? "Replace diagnostics key" : "Enable remote diagnostics" %>" data-confirm-text="<%= diagnosticsAccess.configured ? "Replace the current key? Existing diagnostic clients will immediately lose access." : "Create a one-time key and enable the fixed read-only diagnostic endpoint?" %>" data-confirm-label="<%= diagnosticsAccess.configured ? "Replace key" : "Enable diagnostics" %>">
+      <button class="button" type="submit"><%= diagnosticsAccess.configured ? "Replace access key" : "Create access key" %></button>
+    </form>
+    <% if (diagnosticsAccess.enabled || diagnosticsAccess.configured) { %>
+    <form method="post" action="/admin/diagnostics/access/revoke" data-confirm-mode="modal" data-confirm-title="Revoke diagnostics access" data-confirm-text="Disable the endpoint and permanently invalidate the current diagnostics key?" data-confirm-label="Revoke access">
+      <button class="button danger" type="submit">Revoke access</button>
+    </form>
+    <% } %>
+  </div>
+</section>
+
+<section class="card">
+  <fieldset>
+    <legend>Available checks</legend>
+    <p class="hint">These are the only operations the remote endpoint accepts. Running one here uses the same redaction and output format.</p>
+    <form method="post" action="/admin/diagnostics/run" class="input-action-row">
+      <label class="field">
+        <span>Diagnostic check</span>
+        <select name="check" required>
+          <% diagnosticsAccess.checks.forEach((check) => { %>
+          <option value="<%= check.id %>" <%= selectedCheck === check.id ? "selected" : "" %>><%= check.id.replaceAll("_", " ") %> — <%= check.description %></option>
+          <% }) %>
+        </select>
+      </label>
+      <button class="button subtle" type="submit">Run check</button>
+    </form>
+  </fieldset>
+
+  <% if (diagnosticError) { %>
+  <div class="callout danger"><strong>Check failed</strong><p><%= diagnosticError %></p></div>
+  <% } %>
+  <% if (diagnosticResult) { %>
+  <details class="lumi-expandable-settings" open>
+    <summary><span><strong>Diagnostic result</strong><span class="hint"><%= diagnosticResult.check %> · <%= diagnosticResult.duration_ms %> ms</span></span></summary>
+    <div class="lumi-expandable-body">
+      <pre class="log-details"><%= JSON.stringify(diagnosticResult, null, 2) %></pre>
+    </div>
+  </details>
+  <% } %>
+</section>
+
+<section class="card">
+  <fieldset>
+    <legend>Connect from a trusted computer</legend>
+    <p>Use the public HTTPS address for this Lumi installation. Plain HTTP is accepted only from the same machine.</p>
+    <pre class="log-details">curl -X POST "https://your-lumi-host/api/diagnostics/v1/run" \
+  -H "Authorization: Bearer $LUMI_DIAGNOSTICS_KEY" \
+  -H "Content-Type: application/json" \
+  --data '{"check":"update_state"}'</pre>
+    <p class="hint">Keep the key out of shell history, tickets, chat, and logs. A local environment variable or an ignored <code>.secrets</code> file is safer than placing it directly in a command.</p>
+    <p class="hint">Repository maintainers can save <code>{ "base_url": "https://your-lumi-host", "key": "…" }</code> in <code>.secrets/production-diagnostics.json</code>, then run <code>npm run diagnostics:production -- update_state</code>.</p>
+  </fieldset>
+</section>
+
+<section class="card">
+  <fieldset>
+    <legend>Recent diagnostic access</legend>
+    <% if (!diagnosticAudit.length) { %>
+    <p class="hint">No diagnostic access has been recorded yet.</p>
+    <% } else { %>
+    <div class="log-window">
+      <% diagnosticAudit.forEach((entry) => { %>
+      <details class="log-entry level-<%= entry.level %>">
+        <summary>
+          <span class="log-marker" aria-hidden="true"></span>
+          <span class="log-message"><%= entry.message %></span>
+          <span class="log-level-pill"><%= entry.level %></span>
+          <span class="log-time"><%= new Date(entry.created_at).toLocaleString() %></span>
+        </summary>
+        <pre class="log-details"><%= entry.details || "No additional details." %></pre>
+      </details>
+      <% }) %>
+    </div>
+    <% } %>
+  </fieldset>
+</section>
+
+<%- include("partials/layout-bottom") %>

src/web/views/admin-updates.ejs

@@ -145,6 +145,7 @@
         </form>
         <% } %>
       </div>
+      <p class="hint" data-update-inline-result aria-live="polite"></p>
       <% if (core.available_versions?.length) { %>
       <details class="lumi-expandable-settings update-version-picker">
         <summary>
@@ -284,6 +285,7 @@
             </form>
             <% } %>
           </div>
+          <p class="hint" data-update-inline-result aria-live="polite"></p>
           <% if (plugin.available_versions?.length) { %>
           <details class="lumi-expandable-settings update-version-picker">
             <summary>

update-manifest.json

@@ -1,6 +1,6 @@
 {
   "name": "Lumi Core",
-  "version": "0.2.1",
+  "version": "0.2.2",
   "channel": "stable",
   "released_at": "2026-07-18",
   "compatible_from": "0.1.9",
@@ -8,7 +8,7 @@
   "replaces_versions": [
     "1.2.0"
   ],
-  "migration_notes": "Includes the 1.2.0 version correction and improves update-page behavior. Lumi synchronizes runtime dependencies on restart and preserves settings, databases, plugin data, community knowledge, AI models, runtimes, uploads, logs, and secrets.",
+  "migration_notes": "Includes the 1.2.0 version correction, production plugin-update diagnostics, and an optional secured read-only production diagnostics endpoint. Lumi synchronizes runtime dependencies on restart and preserves settings, databases, plugin data, community knowledge, AI models, runtimes, uploads, logs, and secrets.",
   "rollback_safe": true,
   "requirements": [
     "Node.js 18 or newer"
@@ -37,6 +37,18 @@
       ],
       "rollback_safe": true,
       "migration_notes": "Update-page behavior and wording fixes only; preserved local data is not replaced."
+    },
+    {
+      "version": "0.2.2",
+      "channel": "stable",
+      "released_at": "2026-07-18",
+      "compatible_from": "0.1.9",
+      "migration_kind": "patch",
+      "replaces_versions": [
+        "1.2.0"
+      ],
+      "rollback_safe": true,
+      "migration_notes": "Adds stage-specific plugin update preflight/error reporting and an optional secured read-only production diagnostics endpoint; preserved local data is not replaced."
     }
   ]
 }