From 46187137e6b545b5c13a8201a67bddaacab4bf8b Mon Sep 17 00:00:00 2001 From: Franz Rolfsvaag Date: Sat, 18 Jul 2026 15:48:54 +0200 Subject: [PATCH] fix: repair 0.2.0 updates and version installs --- .gitignore | 1 + CHANGELOG.md | 4 +- README.md | 6 ++ TODO.md | 4 +- docs/update-audit-0.2.0.md | 57 ++++++++++ docs/updates.md | 31 +++++- knowledge/core/lumi-core.md | 2 +- package-lock.json | 22 ++-- package.json | 10 +- plugins/lumi_ai_web_search/CHANGELOG.md | 2 +- plugins/lumi_ai_web_search/tool_info.json | 4 +- release-index.json | 62 +++++++++++ run.js | 16 +++ scripts/build-core-repair-patch.js | 60 +++++++++++ scripts/verify-all.js | 1 + scripts/verify-core-repair-patch.js | 60 +++++++++++ scripts/verify-release-metadata.js | 10 +- scripts/verify-update-system.js | 124 ++++++++++++++++++++++ src/services/dependency-manager.js | 122 +++++++++++++++++++++ src/services/overlay-connectors.js | 17 ++- src/services/repo-update.js | 86 ++++++++++++--- src/services/update-index.js | 83 +++++++++++++-- src/services/update-manager.js | 44 +++++++- src/services/update-repository.js | 36 ++++++- src/web/server.js | 13 ++- src/web/views/admin-updates.ejs | 36 +++++++ update-manifest.json | 20 ++-- 27 files changed, 869 insertions(+), 64 deletions(-) create mode 100644 docs/update-audit-0.2.0.md create mode 100644 release-index.json create mode 100644 scripts/build-core-repair-patch.js create mode 100644 scripts/verify-core-repair-patch.js create mode 100644 scripts/verify-update-system.js create mode 100644 src/services/dependency-manager.js diff --git a/.gitignore b/.gitignore index 389426c..395c94f 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,7 @@ plugins/*/data/** *.sqlite *.sqlite-* npm-debug.log +/dist/ security-audit-*.json security-audit-*.md taskfile.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index 5bb9ccd..0f4b102 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Lumi changelog -## 1.2.0 +## 0.2.0 - Added the OBS overlay management and Browser Source system, including scenes, live updates, visual editing, multimedia, external web sources, and OBS bridge extension points. - Added the role-gated Open Knowledge Foundation plugin with file-backed community/correction knowledge, generated reference material, administration workflows, and Lumi AI retrieval. @@ -9,3 +9,5 @@ - Improved update safety with compressed snapshots, model/runtime exclusions, retention controls, transactional replacement, rollback recovery, and preserved local data. - Added the OBS, OKF, command-list, update, settings, feedback, and repository-wide responsive UI/UX improvements included in this release. - Added focused verification for WebUI, authentication, feedback, placeholders, uploads, updates, OKF, Lumi AI, destructive actions, overlays, and webhooks. +- Corrected the mistakenly published `1.2.0` version to `0.2.0` and added a data-preserving repair path for affected installations. +- Repaired core-only upgrades that failed when the independently managed OKF plugin or optional OBS package was absent, and added exact tagged core/plugin installs with locked dependency synchronization. diff --git a/README.md b/README.md index 5bc2063..d6bc1a4 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,12 @@ and a compressed database backup without duplicating AI models or other preserved local data; age and per-target retention are configurable on the Updates page. +Admins may also install or repair a specific core or plugin release. These +choices use immutable version tags and verify the selected version before files +are replaced. `npm start` and `npm run run` launch the update-aware wrapper, +which repairs required packages from the lockfile after a core update while +allowing unavailable optional integrations to remain disabled. + Recovery mode can be started with `LUMI_SAFE_MODE=1 npm run run`, `node run.js --safe-mode`, or `data/recovery/safe-mode.flag`. See [`docs/updates.md`](docs/updates.md) and diff --git a/TODO.md b/TODO.md index 760a71f..6281aa3 100644 --- a/TODO.md +++ b/TODO.md @@ -677,9 +677,11 @@ This section is for Lumi chat answer feedback and AI Improvement Center work, no ## Done +- 2026-07-18: Audited the 0.1.9-to-0.2.0 update boundary, removed core's direct dependency on OKF plugin files, made OBS WebSocket optional, added locked runtime dependency repair, corrected 1.2.0 to 0.2.0, added immutable release metadata and exact-version core/plugin installs, and produced a checksummed data-preserving core repair patch. + ### 2026-07-18 -- 2026-07-18: Published Lumi Core 1.2.0 release metadata with a safe compatibility bridge from 0.1.9, patch-bumped all changed plugin/tool packages, added core and package-specific changelogs, synchronized generated OKF version references, and added release-metadata verification. +- 2026-07-18: Prepared Lumi Core 0.2.0 release metadata with a safe path from 0.1.9 and a version-correction path from the mistakenly published 1.2.0, patch-bumped all changed plugin/tool packages, added core and package-specific changelogs, synchronized generated OKF version references, and added release-metadata verification. - 2026-07-18: Fixed community OKF discovery end-to-end: Community Knowledge now lists and opens visible file-backed community entries alongside general entries, keeps admin-only files role-filtered, tolerates a single-character name typo such as `jejee`/`Jejeee`, and carries the earlier user subject into explicit OKF follow-ups without treating previous assistant text as evidence. - 2026-07-18: Made Lumi AI OKF-first: role-filtered OKF retrieval now precedes normal gate routing, the fast model can answer sufficient local facts, verified link questions return exact OKF URLs, early context is reused instead of searched twice, and Web Search is unavailable whenever the fast gate confirms OKF is sufficient. - 2026-07-18: Improved fast OKF reply quality: compact evidence now preserves useful roles, relationships, aliases, activities, and channel names; one-word or incomplete identity replies are rejected; grounded terse drafts receive one bounded fast-model rewrite; and relationship follow-ups must explicitly answer the requested relationship before bypassing the main model. diff --git a/docs/update-audit-0.2.0.md b/docs/update-audit-0.2.0.md new file mode 100644 index 0000000..724d4bb --- /dev/null +++ b/docs/update-audit-0.2.0.md @@ -0,0 +1,57 @@ +# 0.2.0 Update Compatibility Audit + +This audit compares pre-merge `main` (`1173b8a`, core 0.1.9) with the initially +published merge (`c917615`, incorrectly numbered 1.2.0). The merge changed 191 +files: 75 core/runtime files, 85 plugin files, 20 generated knowledge files, and +11 other project files. + +## Confirmed Upgrade Failures + +1. **Core imported an independently preserved plugin.** `src/web/server.js` + loaded `plugins/okf/backend/file_knowledge` during startup. Core-only updates + deliberately preserve the whole `plugins/` directory, so a 0.1.9 install did + not receive the new OKF plugin file and crashed before the web server started. +2. **A newly added package was required before dependencies were updated.** The + OBS connector loaded `obs-websocket-js` at module initialization. Core updates + preserve `node_modules/` and the old updater did not reconcile it with the new + lockfile, producing another startup crash. +3. **Core and plugin delivery were not coordinated.** The merged repository had + patch-bumped plugins, but a core update copied none of them by design. There + was no historical release catalog or exact-version plugin installer to let an + administrator bring an old installation to a known release set. +4. **The release number was wrong.** 1.2.0 was a mistaken major-version jump from + 0.1.9. Normal semantic-version ordering would reject the corrected 0.2.0 as a + downgrade for an already affected install. + +No unresolved static relative imports remain in the repaired tree. Plugin data +remains isolated under each plugin's `data/` directory, and current plugin +manifests match the 0.2.0 release catalog. + +## Repairs + +- Core now uses the OKF capability registered at + `global.lumiFrameworks.okf.saveCorrection`; if OKF is absent, only that action + shows an install/enable message. +- OBS WebSocket is an optional dependency and is loaded behind a guarded + connector boundary. Its absence cannot stop the overlay or web server. +- `run.js` reconciles required direct dependencies with `package-lock.json` + before each application start. Optional integration failure is degraded, not + fatal. +- `release-index.json` maps core, plugin, and Lumi AI tool versions to immutable + release tags. Admins can install or reinstall an exact core/plugin version. +- `update-manifest.json` declares 0.2.0 as the supported replacement for 1.2.0, + allowing affected installations to repair themselves despite semver ordering. +- Repository checkout and post-copy verification both confirm the requested + version. Plugin updates preserve plugin data and now restart before reporting + the selected code as active. + +## Data Safety + +The repair does not replace `data/`, `plugins/`, `node_modules/`, configuration, +storage, uploads, logs, databases, secrets, environment files, +`knowledge/community/`, or `knowledge/corrections/`. It creates a compressed +rollback snapshot first. The repair ZIP contains only managed core files, a +preservation manifest, and SHA-256 hashes; it contains no user or plugin data. + +The snapshot database copy is a safety backup, not a migration or replacement. +The repair introduces no destructive database migration. diff --git a/docs/updates.md b/docs/updates.md index f697663..7791d94 100644 --- a/docs/updates.md +++ b/docs/updates.md @@ -22,6 +22,13 @@ install. If the latest version is unsafe, the UI shows the safe target as the actionable version and lists why the absolute latest is not selected. +Admins can also choose an exact core or plugin version. Installable releases +come from `release-index.json` and resolve to immutable `refs/tags/vX.Y.Z` +references. Lumi checks the version inside the selected tag before replacing +live code. Reinstalling the current version is supported as a repair action; +downgrades show an additional warning because preserved newer data is not +automatically migrated backward. + Updates to or from an unversioned install or target remain available so an admin can recover directly from the repository. They are treated as manual repo updates: the action button stays enabled, but the UI shows warnings and the @@ -33,6 +40,7 @@ safety cannot be verified from metadata. The updater looks for these files on the source branch: - `update-manifest.json` or `lumi.manifest.json` +- `release-index.json` - `CHANGELOG.md` or `changelog.json` - `plugins//plugin.json` - `plugins//CHANGELOG.md` or `changelog.json` @@ -47,6 +55,9 @@ Changed plugins use their `plugin.json` version and a matching plugin `CHANGELOG.md`; Lumi AI tools use `tool_info.json` and their own changelog. `scripts/verify-release-metadata.js` checks those relationships, compatibility metadata, and generated OKF version references before publication. +`scripts/verify-update-system.js` additionally checks the release catalog, +immutable refs, exact-version selection, preserved paths, dependency recovery, +and optional core/plugin boundaries. Stable checks read `main` by default. The newest `experimental-*` branch is used only when explicitly selected on the updates page. @@ -113,6 +124,24 @@ Plugin code is prepared in a staging directory and swapped only after the new files are ready; plugin data is moved into the replacement as part of that transaction. +Core updates intentionally do not copy or delete the `plugins/` directory. +Core code therefore accesses optional plugin capabilities through the registered +framework API instead of importing files from a plugin directory. Plugins can +be installed or changed independently from their version picker. + +## Runtime Dependencies + +Start Lumi through `run.js` (the `npm start` command does this). Before starting +the application, the wrapper compares direct runtime packages with +`package-lock.json`. Missing or mismatched required packages are repaired with a +locked production install. This covers dependencies added by a core update even +though `node_modules/` is preserved during file replacement. + +Optional integrations, including `obs-websocket-js`, are loaded defensively. A +failed optional install is recorded and the rest of Lumi starts normally; the +affected connector reports that it is unavailable. Required dependency failure +opens recovery mode instead of repeatedly starting a partially installed core. + ## ZIP Fallback Core and plugin ZIP updates remain available, but they are hidden under @@ -130,4 +159,4 @@ Admin update actions publish Server-Sent Events through Core update success returns a five-second in-page notice before refresh/restart. Plugin update success updates progress for the affected plugin action without a -whole-page refresh. +whole-page refresh, then restarts Lumi so the selected plugin code is loaded. diff --git a/knowledge/core/lumi-core.md b/knowledge/core/lumi-core.md index b212215..e9f453f 100644 --- a/knowledge/core/lumi-core.md +++ b/knowledge/core/lumi-core.md @@ -14,7 +14,7 @@ editable: false Lumi is the core web UI and bot runtime. ## Runtime Package: lumi-bot -Version: 1.2.0 +Version: 0.2.0 ## Routes - GET /api/events - POST /api/destructive-confirmations diff --git a/package-lock.json b/package-lock.json index e6d8e3c..7d468f2 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "lumi-bot", - "version": "1.2.0", + "version": "0.2.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "lumi-bot", - "version": "1.2.0", + "version": "0.2.0", "dependencies": { "adm-zip": "^0.5.12", "better-sqlite3": "^11.5.0", @@ -16,11 +16,13 @@ "express": "^4.19.2", "express-session": "^1.18.1", "multer": "^2.2.0", - "obs-websocket-js": "^5.0.8", "tmi.js": "^1.8.5" }, "engines": { "node": ">=18" + }, + "optionalDependencies": { + "obs-websocket-js": "^5.0.8" } }, "node_modules/@discordjs/builders": { @@ -61,6 +63,7 @@ "resolved": "https://registry.npmjs.org/@msgpack/msgpack/-/msgpack-2.8.0.tgz", "integrity": "sha512-h9u4u/jiIRKbq25PM+zymTyW6bhTzELvOoUd+AvYriWOAKpLGnIamaET3pnHYoI5iYphAHBI4ayx0MehR+VVPQ==", "license": "ISC", + "optional": true, "engines": { "node": ">= 10" } @@ -413,7 +416,8 @@ "version": "4.2.0", "resolved": "https://registry.npmjs.org/crypto-js/-/crypto-js-4.2.0.tgz", "integrity": "sha512-KALDyEYgpY+Rlob/iriUtjV6d5Eq+Y191A5g4UqLAi8CyGP9N1+FdVbkc1SxKc2r4YAYqG8JzO2KGL+AizD70Q==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/date-fns": { "version": "2.16.1", @@ -643,7 +647,8 @@ "version": "5.0.4", "resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-5.0.4.tgz", "integrity": "sha512-mlsTRyGaPBjPedk6Bvw+aqbsXDtoAyAzm5MO7JgU+yVRyMQ5O8bD4Kcci7BS85f93veegeCPkL8R4GLClnjLFw==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/expand-template": { "version": "2.0.3", @@ -979,6 +984,7 @@ "resolved": "https://registry.npmjs.org/isomorphic-ws/-/isomorphic-ws-5.0.0.tgz", "integrity": "sha512-muId7Zzn9ywDsyXgTIafTry2sV3nySZeUDe6YedVd1Hvuuep5AsIlqK+XefWpYTyJG5e503F2xIuT2lcU6rCSw==", "license": "MIT", + "optional": true, "peerDependencies": { "ws": "*" } @@ -1203,6 +1209,7 @@ "resolved": "https://registry.npmjs.org/obs-websocket-js/-/obs-websocket-js-5.0.8.tgz", "integrity": "sha512-QDnQJMr5wuCoYugK02ggZ1/cvESs4KJDEK+UhGg0Ry35jnY8tK1Xr3KoPjKyoD6sd8G+WdtIdzuY+yUyPtogQQ==", "license": "MIT", + "optional": true, "dependencies": { "@msgpack/msgpack": "^2.7.1", "crypto-js": "^4.1.1", @@ -1221,6 +1228,7 @@ "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", "license": "MIT", + "optional": true, "dependencies": { "ms": "^2.1.3" }, @@ -1237,7 +1245,8 @@ "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" + "license": "MIT", + "optional": true }, "node_modules/on-finished": { "version": "2.4.1", @@ -1743,6 +1752,7 @@ "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-3.13.1.tgz", "integrity": "sha512-tLq3bSNx+xSpwvAJnzrK0Ep5CLNWjvFTOp71URMaAEWBfRb9nnJiBoUe0tF8bI4ZFO3omgBR6NvnbzVUT3Ly4g==", "license": "(MIT OR CC0-1.0)", + "optional": true, "engines": { "node": ">=14.16" }, diff --git a/package.json b/package.json index fcccefd..ea0c388 100644 --- a/package.json +++ b/package.json @@ -1,15 +1,17 @@ { "name": "lumi-bot", - "version": "1.2.0", + "version": "0.2.0", "private": true, "type": "commonjs", "scripts": { - "start": "node src/main.js", + "start": "node run.js", "run": "node run.js", "verify:all": "node scripts/verify-all.js", "verify:preflight": "node scripts/verify-preflight.js", "verify:safe-files": "node scripts/verify-safe-files.js", "verify:uploads": "node scripts/verify-upload-security.js", + "verify:updates": "node scripts/verify-release-metadata.js && node scripts/verify-update-system.js", + "build:repair-patch": "node scripts/build-core-repair-patch.js && node scripts/verify-core-repair-patch.js", "verify:web-auth": "node scripts/verify-web-auth.js", "verify:destructive-actions": "node scripts/verify-destructive-actions.js", "verify:overlays": "node scripts/verify-overlays.js", @@ -28,7 +30,9 @@ "express": "^4.19.2", "express-session": "^1.18.1", "multer": "^2.2.0", - "obs-websocket-js": "^5.0.8", "tmi.js": "^1.8.5" + }, + "optionalDependencies": { + "obs-websocket-js": "^5.0.8" } } diff --git a/plugins/lumi_ai_web_search/CHANGELOG.md b/plugins/lumi_ai_web_search/CHANGELOG.md index cdb8f88..5842720 100644 --- a/plugins/lumi_ai_web_search/CHANGELOG.md +++ b/plugins/lumi_ai_web_search/CHANGELOG.md @@ -2,5 +2,5 @@ ## 0.1.1 -- Aligned compatibility metadata with Lumi 1.2.0 and Lumi AI 0.8.2. +- Aligned compatibility metadata with Lumi 0.2.0 and Lumi AI 0.8.2. - Added integration verification for OKF-first tool exposure and grounded source handling. diff --git a/plugins/lumi_ai_web_search/tool_info.json b/plugins/lumi_ai_web_search/tool_info.json index e14d51a..178cc71 100644 --- a/plugins/lumi_ai_web_search/tool_info.json +++ b/plugins/lumi_ai_web_search/tool_info.json @@ -75,7 +75,7 @@ ], "status_file": "data/status.json", "dependencies": [], - "minimum_lumi_version": "1.2.0", + "minimum_lumi_version": "0.2.0", "minimum_lumi_ai_version": "0.8.2", "required_plugins": [ "core", @@ -93,7 +93,7 @@ "preserve_on_update": [ "data" ], - "update_notes": "Aligns Web Search with Lumi 1.2.0 and Lumi AI 0.8.2 OKF-first routing, source grounding, and tool-exposure verification.", + "update_notes": "Aligns Web Search with Lumi 0.2.0 and Lumi AI 0.8.2 OKF-first routing, source grounding, and tool-exposure verification.", "author": "Lumi", "homepage": "https://git.rolfsvaag.no/Rolfsvaag_Datateknikk/Lumi", "repository_path": "plugins/lumi_ai_web_search", diff --git a/release-index.json b/release-index.json new file mode 100644 index 0000000..6ba2995 --- /dev/null +++ b/release-index.json @@ -0,0 +1,62 @@ +{ + "schema_version": 1, + "channel": "stable", + "releases": [ + { + "version": "0.2.0", + "ref": "refs/tags/v0.2.0", + "released_at": "2026-07-18", + "installable": true, + "rollback_safe": true, + "replaces_versions": [ + "1.2.0" + ], + "data_policy": "preserve", + "dependency_policy": "sync_on_restart", + "migration_notes": "Corrects the mistakenly published 1.2.0 version, repairs optional dependency loading, and synchronizes runtime packages without replacing local data.", + "plugins": { + "auto-vc": "0.1.6", + "birthday": "0.1.3", + "economy-framework": "0.2.10", + "economy-games": "0.1.7", + "expression-interaction": "0.2.1", + "lumi_ai": "0.8.2", + "moderation": "0.1.5", + "okf": "0.1.1", + "quotes": "0.1.2", + "sample-plugin": "0.1.0", + "throne_wishlist": "0.1.2", + "welcome_messages": "0.1.1" + }, + "tools": { + "lumi_ai_web_search": "0.1.1" + } + }, + { + "version": "0.1.9", + "ref": "refs/tags/v0.1.9", + "released_at": "2026-06-17", + "installable": true, + "rollback_safe": true, + "data_policy": "preserve", + "dependency_policy": "sync_on_restart", + "migration_notes": "Historical pre-0.2.0 release. Installing older code preserves local data, but newer database changes are not reversed.", + "plugins": { + "auto-vc": "0.1.5", + "birthday": "0.1.2", + "economy-framework": "0.2.9", + "economy-games": "0.1.7", + "expression-interaction": "0.2.0", + "lumi_ai": "0.8.1", + "moderation": "0.1.4", + "quotes": "0.1.1", + "sample-plugin": "0.1.0", + "throne_wishlist": "0.1.1", + "welcome_messages": "0.1.0" + }, + "tools": { + "lumi_ai_web_search": "0.1.0" + } + } + ] +} diff --git a/run.js b/run.js index 4964a52..1f10d32 100644 --- a/run.js +++ b/run.js @@ -1,5 +1,6 @@ const path = require("path"); const { spawn } = require("child_process"); +const { ensureRuntimeDependencies } = require("./src/services/dependency-manager"); const entry = path.join(__dirname, "src", "main.js"); const safeModeEntry = path.join(__dirname, "safe-mode.js"); @@ -30,6 +31,21 @@ function startSafeMode() { } function startChild() { + if (process.env.LUMI_SKIP_DEPENDENCY_SYNC !== "1") { + try { + const dependencyResult = ensureRuntimeDependencies({ rootPath: __dirname }); + if (dependencyResult.installed) { + console.log(`Lumi synchronized ${dependencyResult.issues.length} changed or missing runtime dependencies.`); + } + if (dependencyResult.optional_failed || dependencyResult.unresolved?.length) { + console.warn("Lumi started without one or more optional integrations. Their settings pages will explain what is unavailable."); + } + } catch (error) { + console.error(error.message); + startSafeMode(); + return; + } + } const child = spawn(process.execPath, [entry], { stdio: "inherit", env: { ...process.env, BOT_WRAPPER: "1" } diff --git a/scripts/build-core-repair-patch.js b/scripts/build-core-repair-patch.js new file mode 100644 index 0000000..36a9b23 --- /dev/null +++ b/scripts/build-core-repair-patch.js @@ -0,0 +1,60 @@ +const crypto = require("crypto"); +const fs = require("fs"); +const path = require("path"); +const AdmZip = require("adm-zip"); + +const root = path.join(__dirname, ".."); +const destination = path.join(root, "dist", "lumi-core-1.2.0-to-0.2.0-repair.zip"); +const files = [ + "CHANGELOG.md", + "README.md", + "package.json", + "package-lock.json", + "release-index.json", + "run.js", + "update-manifest.json", + "docs/updates.md", + "docs/update-audit-0.2.0.md", + "knowledge/core/lumi-core.md", + "scripts/verify-release-metadata.js", + "scripts/verify-all.js", + "scripts/build-core-repair-patch.js", + "scripts/verify-core-repair-patch.js", + "scripts/verify-update-system.js", + "src/services/dependency-manager.js", + "src/services/overlay-connectors.js", + "src/services/repo-update.js", + "src/services/update-index.js", + "src/services/update-manager.js", + "src/services/update-repository.js", + "src/web/server.js", + "src/web/views/admin-updates.ejs" +]; + +const hashes = {}; +const zip = new AdmZip(); +for (const relativePath of files) { + const filePath = path.join(root, relativePath); + if (!fs.statSync(filePath, { throwIfNoEntry: false })?.isFile()) { + throw new Error(`Repair patch input is missing: ${relativePath}`); + } + const contents = fs.readFileSync(filePath); + hashes[relativePath] = crypto.createHash("sha256").update(contents).digest("hex"); + zip.addFile(relativePath.replace(/\\/g, "/"), contents); +} + +const manifest = { + schema_version: 1, + name: "Lumi core 0.2.0 repair", + target: "core", + from_versions: ["0.1.9", "1.2.0", "0.2.0"], + to_version: "0.2.0", + data_policy: "preserve", + dependency_policy: "sync_on_restart", + created_at: new Date().toISOString(), + files: hashes +}; +zip.addFile("patch-manifest.json", Buffer.from(`${JSON.stringify(manifest, null, 2)}\n`)); +fs.mkdirSync(path.dirname(destination), { recursive: true }); +zip.writeZip(destination); +console.log(destination); diff --git a/scripts/verify-all.js b/scripts/verify-all.js index dd28c21..07d0b59 100644 --- a/scripts/verify-all.js +++ b/scripts/verify-all.js @@ -10,6 +10,7 @@ const checks = [ "scripts/verify-feedback-system.js", "scripts/verify-placeholders.js", "scripts/verify-release-metadata.js", + "scripts/verify-update-system.js", "scripts/verify-plugin-update-preserves-data.js", "scripts/verify-safe-files.js", "scripts/verify-upload-security.js", diff --git a/scripts/verify-core-repair-patch.js b/scripts/verify-core-repair-patch.js new file mode 100644 index 0000000..30eef8f --- /dev/null +++ b/scripts/verify-core-repair-patch.js @@ -0,0 +1,60 @@ +const assert = require("assert"); +const crypto = require("crypto"); +const fs = require("fs"); +const os = require("os"); +const path = require("path"); +const AdmZip = require("adm-zip"); +const { verifyPatchPackage } = require("../src/services/update-manager"); + +const root = path.join(__dirname, ".."); +const archivePath = path.join(root, "dist", "lumi-core-1.2.0-to-0.2.0-repair.zip"); +assert.equal(fs.existsSync(archivePath), true, "build the repair patch first"); +const zip = new AdmZip(archivePath); +const entries = zip.getEntries().filter((entry) => !entry.isDirectory); +const names = new Set(entries.map((entry) => entry.entryName.replace(/\\/g, "/"))); +assert.equal(names.has("patch-manifest.json"), true); +const manifest = JSON.parse(zip.readAsText("patch-manifest.json")); +assert.equal(manifest.target, "core"); +assert.equal(manifest.to_version, "0.2.0"); +assert.equal(manifest.data_policy, "preserve"); +assert.deepEqual(manifest.from_versions, ["0.1.9", "1.2.0", "0.2.0"]); + +const forbidden = /^(?:data|plugins|node_modules|config|storage|uploads|logs|database|databases|knowledge\/(?:community|corrections))(?:\/|$)|^\.env(?:\.|$)|^\.secrets$/; +for (const entry of entries) { + assert.doesNotMatch(entry.entryName, forbidden, `repair includes preserved data path ${entry.entryName}`); +} +for (const [relativePath, expected] of Object.entries(manifest.files)) { + assert.equal(names.has(relativePath), true, `${relativePath} is listed but missing`); + const actual = crypto.createHash("sha256").update(zip.readFile(relativePath)).digest("hex"); + assert.equal(actual, expected, `${relativePath} checksum`); +} +assert.equal(Object.keys(manifest.files).length + 1, entries.length, "every repair file must be checksummed"); +assert.equal(JSON.parse(zip.readAsText("package.json")).version, "0.2.0"); + +const simulation = fs.mkdtempSync(path.join(os.tmpdir(), "lumi-repair-simulation-")); +try { + const sentinels = [ + "data/app.db", + "plugins/okf/data/local.db", + "node_modules/local-only/package.json", + "knowledge/community/people.md", + "knowledge/corrections/fix.md", + ".env" + ]; + for (const sentinel of sentinels) { + const target = path.join(simulation, sentinel); + fs.mkdirSync(path.dirname(target), { recursive: true }); + fs.writeFileSync(target, `preserve:${sentinel}`); + } + zip.extractAllTo(simulation, true); + assert.equal(verifyPatchPackage(simulation).to_version, "0.2.0"); + for (const sentinel of sentinels) { + assert.equal(fs.readFileSync(path.join(simulation, sentinel), "utf8"), `preserve:${sentinel}`); + } + assert.equal(JSON.parse(fs.readFileSync(path.join(simulation, "package.json"), "utf8")).version, "0.2.0"); + assert.equal(fs.existsSync(path.join(simulation, "src", "services", "dependency-manager.js")), true); +} finally { + fs.rmSync(simulation, { recursive: true, force: true }); +} + +console.log(`Core repair patch verification passed (${entries.length} files, no preserved data payloads).`); diff --git a/scripts/verify-release-metadata.js b/scripts/verify-release-metadata.js index 0dd32cd..13a5b42 100644 --- a/scripts/verify-release-metadata.js +++ b/scripts/verify-release-metadata.js @@ -4,7 +4,7 @@ const path = require("path"); const { findSafeTarget } = require("../src/services/versioning"); const root = path.join(__dirname, ".."); -const releaseVersion = "1.2.0"; +const releaseVersion = "0.2.0"; const previousCoreVersion = "0.1.9"; const changedPlugins = { "auto-vc": { from: "0.1.5", to: "0.1.6", knowledge: "auto-vc" }, @@ -38,6 +38,7 @@ function hasVersionHeading(markdown, version) { const packageJson = readJson("package.json"); const packageLock = readJson("package-lock.json"); const coreManifest = readJson("update-manifest.json"); +const releaseIndex = readJson("release-index.json"); assert.equal(packageJson.version, releaseVersion); assert.equal(packageLock.version, releaseVersion); assert.equal(packageLock.packages?.[""]?.version, releaseVersion); @@ -45,6 +46,11 @@ assert.equal(coreManifest.version, releaseVersion); assert.equal(coreManifest.channel, "stable"); assert.equal(coreManifest.compatible_from, previousCoreVersion); assert.equal(coreManifest.rollback_safe, true); +assert.deepEqual(coreManifest.replaces_versions, ["1.2.0"]); +assert.equal(releaseIndex.releases[0].version, releaseVersion); +assert.equal(releaseIndex.releases[0].ref, `refs/tags/v${releaseVersion}`); +assert.equal(releaseIndex.releases[1].version, previousCoreVersion); +assert.equal(releaseIndex.releases[1].ref, `refs/tags/v${previousCoreVersion}`); assert.equal(hasVersionHeading(readText("CHANGELOG.md"), releaseVersion), true); assert.match(readText("knowledge/core/lumi-core.md"), new RegExp(`^Version: ${escapeRegex(releaseVersion)}$`, "m")); @@ -79,4 +85,4 @@ assert.equal(webSearch.minimum_lumi_version, releaseVersion); assert.equal(webSearch.minimum_lumi_ai_version, changedPlugins.lumi_ai.to); assert.equal(hasVersionHeading(readText("plugins/lumi_ai_web_search/CHANGELOG.md"), webSearch.version), true); -console.log("Release metadata verification passed: core 1.2.0 and 11 changed plugin/tool packages."); +console.log("Release metadata verification passed: core 0.2.0 and 11 changed plugin/tool packages."); diff --git a/scripts/verify-update-system.js b/scripts/verify-update-system.js new file mode 100644 index 0000000..1d3fca2 --- /dev/null +++ b/scripts/verify-update-system.js @@ -0,0 +1,124 @@ +const assert = require("assert"); +const fs = require("fs"); +const os = require("os"); +const path = require("path"); + +const root = path.join(__dirname, ".."); +const { dependencyIssues, ensureRuntimeDependencies } = require("../src/services/dependency-manager"); +const { targetForRequestedVersion } = require("../src/services/repo-update"); +const { buildStatus } = require("../src/services/update-index"); +const { normalizeRepositoryRef } = require("../src/services/update-repository"); +const { buildCorePreservePaths } = require("../src/services/update-manager"); + +function readJson(relativePath) { + return JSON.parse(fs.readFileSync(path.join(root, relativePath), "utf8")); +} + +const releaseIndex = readJson("release-index.json"); +const releaseVersions = releaseIndex.releases.map((release) => release.version); +assert.deepEqual(releaseVersions, ["0.2.0", "0.1.9"]); +assert.equal(new Set(releaseVersions).size, releaseVersions.length, "release versions must be unique"); +for (const release of releaseIndex.releases) { + assert.equal(normalizeRepositoryRef(release.ref), release.ref); + assert.match(release.ref, /^refs\/tags\/v\d+\.\d+\.\d+$/); + assert.equal(release.data_policy, "preserve"); +} + +const currentRelease = releaseIndex.releases[0]; +assert.equal(currentRelease.version, readJson("package.json").version); +assert.deepEqual(currentRelease.replaces_versions, ["1.2.0"]); +for (const [pluginId, version] of Object.entries(currentRelease.plugins)) { + assert.equal(readJson(`plugins/${pluginId}/plugin.json`).version, version, `${pluginId} release catalog version`); +} +for (const [toolId, version] of Object.entries(currentRelease.tools)) { + assert.equal(readJson(`plugins/${toolId}/tool_info.json`).version, version, `${toolId} release catalog version`); +} + +const baseTarget = { + current_version: "0.2.0", + available_versions: [ + { version: "0.2.0", ref: "refs/tags/v0.2.0", rollback_safe: true }, + { version: "0.1.9", ref: "refs/tags/v0.1.9", rollback_safe: true } + ], + warnings: [] +}; +const selected = targetForRequestedVersion(baseTarget, "0.1.9", "Core version"); +assert.equal(selected.safe_target_version, "0.1.9"); +assert.equal(selected.source_branch, "refs/tags/v0.1.9"); +assert.equal(selected.update_available, true); +assert.equal(selected.requires_manual_confirmation, true); +assert.throws(() => targetForRequestedVersion(baseTarget, "9.9.9", "Core version"), /not listed/); +assert.throws(() => normalizeRepositoryRef("--upload-pack=bad"), /Invalid/); +assert.throws(() => normalizeRepositoryRef("refs/tags/../main"), /Invalid/); + +const corrected = buildStatus({ + kind: "core", + currentVersion: "1.2.0", + manifest: readJson("update-manifest.json"), + changelog: [], + sourceBranch: "main", + channel: "stable" +}); +assert.equal(corrected.version_correction, true); +assert.equal(corrected.safe_target_version, "0.2.0"); +assert.equal(corrected.update_available, true); +assert.equal(corrected.blocked, false); + +const preserve = buildCorePreservePaths(root); +for (const required of ["data", "plugins", "node_modules", "knowledge/community", "knowledge/corrections", ".env"]) { + assert.equal(preserve.has(required), true, `${required} must survive core updates`); +} + +const serverSource = fs.readFileSync(path.join(root, "src", "web", "server.js"), "utf8"); +assert.doesNotMatch(serverSource, /require\([^\n]*plugins\/okf\/backend/); +assert.match(serverSource, /global\.lumiFrameworks\?\.okf\?\.saveCorrection/); +const connectorSource = fs.readFileSync(path.join(root, "src", "services", "overlay-connectors.js"), "utf8"); +assert.match(connectorSource, /try\s*{[\s\S]*require\("obs-websocket-js"\)/); + +const fixture = fs.mkdtempSync(path.join(os.tmpdir(), "lumi-dependencies-")); +try { + fs.writeFileSync(path.join(fixture, "package.json"), JSON.stringify({ dependencies: { example: "^1.0.0" } })); + fs.writeFileSync(path.join(fixture, "package-lock.json"), JSON.stringify({ + lockfileVersion: 3, + packages: { "": {}, "node_modules/example": { version: "1.2.3" } } + })); + assert.deepEqual(dependencyIssues(fixture), [{ dependency: "example", reason: "missing", expected: "1.2.3", optional: false }]); + let invocation = null; + const result = ensureRuntimeDependencies({ + rootPath: fixture, + platform: "linux", + runner(command, args) { + invocation = { command, args }; + const packageDir = path.join(fixture, "node_modules", "example"); + fs.mkdirSync(packageDir, { recursive: true }); + fs.writeFileSync(path.join(packageDir, "package.json"), JSON.stringify({ version: "1.2.3" })); + return { status: 0, stdout: "", stderr: "" }; + } + }); + assert.equal(result.ready, true); + assert.equal(result.installed, true); + assert.equal(invocation.args.includes("ci"), true); + assert.deepEqual(dependencyIssues(fixture), []); +} finally { + fs.rmSync(fixture, { recursive: true, force: true }); +} + +const optionalFixture = fs.mkdtempSync(path.join(os.tmpdir(), "lumi-optional-dependency-")); +try { + fs.writeFileSync(path.join(optionalFixture, "package.json"), JSON.stringify({ optionalDependencies: { optional: "^1.0.0" } })); + fs.writeFileSync(path.join(optionalFixture, "package-lock.json"), JSON.stringify({ + lockfileVersion: 3, + packages: { "": {}, "node_modules/optional": { version: "1.0.0", optional: true } } + })); + const degraded = ensureRuntimeDependencies({ + rootPath: optionalFixture, + platform: "linux", + runner: () => ({ status: 1, stderr: "offline" }) + }); + assert.equal(degraded.ready, true); + assert.equal(degraded.optional_failed, true); +} finally { + fs.rmSync(optionalFixture, { recursive: true, force: true }); +} + +console.log("Update system verification passed: immutable refs, exact versions, preserved data, dependency repair, and optional boundaries."); diff --git a/src/services/dependency-manager.js b/src/services/dependency-manager.js new file mode 100644 index 0000000..b65f7dd --- /dev/null +++ b/src/services/dependency-manager.js @@ -0,0 +1,122 @@ +const crypto = require("crypto"); +const fs = require("fs"); +const path = require("path"); +const { spawnSync } = require("child_process"); + +function readJson(filePath, fallback = null) { + try { + return JSON.parse(fs.readFileSync(filePath, "utf8")); + } catch { + return fallback; + } +} + +function dependencyFingerprint(rootPath) { + const packageJson = fs.readFileSync(path.join(rootPath, "package.json")); + const lockPath = path.join(rootPath, "package-lock.json"); + const lock = fs.existsSync(lockPath) ? fs.readFileSync(lockPath) : Buffer.from(""); + return crypto.createHash("sha256").update(packageJson).update(lock).digest("hex"); +} + +function dependencyIssues(rootPath) { + const manifest = readJson(path.join(rootPath, "package.json"), {}); + const lock = readJson(path.join(rootPath, "package-lock.json"), {}); + const issues = []; + const dependencies = new Map([ + ...Object.keys(manifest.dependencies || {}).map((name) => [name, false]), + ...Object.keys(manifest.optionalDependencies || {}).map((name) => [name, true]) + ]); + for (const [name, optional] of [...dependencies.entries()].sort(([left], [right]) => left.localeCompare(right))) { + const installedPath = path.join(rootPath, "node_modules", ...name.split("/"), "package.json"); + const installed = readJson(installedPath); + const expected = lock.packages?.[`node_modules/${name}`]?.version || null; + if (!installed?.version) { + issues.push({ dependency: name, reason: "missing", expected, optional }); + } else if (expected && installed.version !== expected) { + issues.push({ dependency: name, reason: "version_mismatch", expected, installed: installed.version, optional }); + } + } + return issues; +} + +function writeState(rootPath, values) { + const statePath = path.join(rootPath, "data", "update-dependencies.json"); + fs.mkdirSync(path.dirname(statePath), { recursive: true }); + const temporary = `${statePath}.${process.pid}.tmp`; + fs.writeFileSync(temporary, `${JSON.stringify(values, null, 2)}\n`); + fs.renameSync(temporary, statePath); + return statePath; +} + +function npmInvocation(options = {}) { + if (process.env.npm_execpath && fs.existsSync(process.env.npm_execpath)) { + return { command: process.execPath, args: [process.env.npm_execpath] }; + } + return { + command: options.platform === "win32" || process.platform === "win32" ? "npm.cmd" : "npm", + args: [] + }; +} + +function ensureRuntimeDependencies(options = {}) { + const rootPath = path.resolve(options.rootPath || path.join(__dirname, "..", "..")); + const before = dependencyIssues(rootPath); + const fingerprint = dependencyFingerprint(rootPath); + if (!before.length) { + writeState(rootPath, { + schema_version: 1, + fingerprint, + status: "ready", + checked_at: new Date().toISOString() + }); + return { ready: true, installed: false, issues: [] }; + } + if (options.install === false) return { ready: false, installed: false, issues: before }; + + const invocation = npmInvocation(options); + const installArgs = fs.existsSync(path.join(rootPath, "package-lock.json")) + ? ["ci", "--omit=dev", "--no-audit", "--no-fund"] + : ["install", "--omit=dev", "--no-audit", "--no-fund"]; + const runner = options.runner || spawnSync; + const result = runner(invocation.command, [...invocation.args, ...installArgs], { + cwd: rootPath, + encoding: "utf8", + timeout: Math.max(60000, Number(options.timeoutMs) || 600000), + windowsHide: true + }); + if (result?.error || result?.status !== 0) { + const detail = String(result?.stderr || result?.stdout || result?.error?.message || "npm failed").trim().slice(-4000); + if (before.every((issue) => issue.optional)) { + writeState(rootPath, { + schema_version: 1, + fingerprint, + status: "ready_without_optional_dependencies", + checked_at: new Date().toISOString(), + unresolved: before, + warning: detail + }); + return { ready: true, installed: false, optional_failed: true, issues: before, warning: detail }; + } + throw new Error(`Dependency synchronization failed. ${detail}`); + } + const after = dependencyIssues(rootPath); + const requiredAfter = after.filter((issue) => !issue.optional); + if (requiredAfter.length) { + throw new Error(`Dependency synchronization finished with unresolved packages: ${requiredAfter.map((item) => item.dependency).join(", ")}.`); + } + writeState(rootPath, { + schema_version: 1, + fingerprint, + status: after.length ? "ready_without_optional_dependencies" : "ready", + installed_at: new Date().toISOString(), + resolved: before.filter((issue) => !after.some((remaining) => remaining.dependency === issue.dependency)), + unresolved: after + }); + return { ready: true, installed: true, issues: before, unresolved: after }; +} + +module.exports = { + dependencyFingerprint, + dependencyIssues, + ensureRuntimeDependencies +}; diff --git a/src/services/overlay-connectors.js b/src/services/overlay-connectors.js index fd2bf6e..8bcb6ab 100644 --- a/src/services/overlay-connectors.js +++ b/src/services/overlay-connectors.js @@ -1,4 +1,11 @@ -const { OBSWebSocket, EventSubscription } = require("obs-websocket-js"); +let OBSWebSocket = null; +let EventSubscription = {}; +let obsWebSocketLoadError = null; +try { + ({ OBSWebSocket, EventSubscription } = require("obs-websocket-js")); +} catch (error) { + obsWebSocketLoadError = error; +} const { publishWebEvent } = require("./web-events"); const { getObsSettings, @@ -50,6 +57,9 @@ function listOverlayConnectorProviders() { class LocalObsWebSocketConnector { constructor(settings) { + if (!OBSWebSocket) { + throw new Error("OBS WebSocket support is unavailable until Lumi dependencies are synchronized and the server is restarted."); + } this.settings = settings; this.client = new OBSWebSocket(); this.state = "disconnected"; @@ -374,7 +384,10 @@ class ObsBrowserBridgeConnector { registerOverlayConnectorProvider({ id: "local_obs_websocket", label: "OBS WebSocket", - description: "Connect from Lumi to OBS on the streaming computer or private local network.", + description: OBSWebSocket + ? "Connect from Lumi to OBS on the streaming computer or private local network." + : `Unavailable because obs-websocket-js could not load${obsWebSocketLoadError?.code === "MODULE_NOT_FOUND" ? "; restart Lumi through run.js to synchronize dependencies" : ""}.`, + available: Boolean(OBSWebSocket), create: (settings) => new LocalObsWebSocketConnector(settings) }); diff --git a/src/services/repo-update.js b/src/services/repo-update.js index acb88f9..cbd1a9b 100644 --- a/src/services/repo-update.js +++ b/src/services/repo-update.js @@ -14,6 +14,7 @@ const { ensureManagedRepo, writeUpdateState } = require("./update-repository"); +const { compareSemver, parseSemver } = require("./versioning"); const { createRecoveryMarker, updateRecoveryMarker, @@ -40,36 +41,78 @@ function emitProgress(publish, event, payload) { } } -function verifyCoreFiles() { +function verifyCoreFiles(rootPath = repoRoot, expectedVersion = null) { for (const file of ["package.json", "src/main.js", "src/web/server.js"]) { - if (!fs.existsSync(path.join(repoRoot, file))) { + if (!fs.existsSync(path.join(rootPath, file))) { throw new Error(`Core verification failed: ${file} is missing.`); } } + const actualVersion = JSON.parse(fs.readFileSync(path.join(rootPath, "package.json"), "utf8")).version; + if (expectedVersion && actualVersion !== expectedVersion) { + throw new Error(`Core verification failed: requested ${expectedVersion}, but repository ref contains ${actualVersion || "no version"}.`); + } } -function verifyPluginFiles(pluginId) { - const manifest = path.join(repoRoot, "plugins", pluginId, "plugin.json"); +function verifyPluginFiles(pluginId, rootPath = repoRoot, expectedVersion = null) { + const manifest = path.join(rootPath, "plugins", pluginId, "plugin.json"); if (!fs.existsSync(manifest)) { throw new Error(`Plugin verification failed: ${pluginId}/plugin.json is missing.`); } - JSON.parse(fs.readFileSync(manifest, "utf8")); + const metadata = JSON.parse(fs.readFileSync(manifest, "utf8")); + if (expectedVersion && metadata.version !== expectedVersion) { + throw new Error(`Plugin verification failed: requested ${expectedVersion}, but repository ref contains ${metadata.version || "no version"}.`); + } + return metadata; } -function applyPluginFromRepositorySnapshot(remote, branch, pluginId) { - const managed = ensureManagedRepo(remote, branch); +function applyPluginFromRepositorySnapshot(remote, repositoryRef, pluginId, expectedVersion) { + const managed = ensureManagedRepo(remote, repositoryRef); const pluginRoot = path.join(managed.path, "plugins", pluginId); if (!fs.existsSync(path.join(pluginRoot, "plugin.json"))) { - throw new Error(`Plugin ${pluginId} was not found in ${branch}.`); + throw new Error(`Plugin ${pluginId} was not found in ${repositoryRef}.`); } + verifyPluginFiles(pluginId, managed.path, expectedVersion); applyPluginFiles(pluginRoot, pluginId, { preserveData: true }); return managed; } -async function applyCoreUpdate({ source = "stable", remote = null, publish } = {}) { +function targetForRequestedVersion(baseTarget, requestedVersion, label) { + if (!requestedVersion) return baseTarget; + const version = parseSemver(requestedVersion)?.raw; + if (!version) throw new Error("Choose a valid version in x.y.z format."); + const selected = (baseTarget.available_versions || []).find((item) => item.version === version); + if (!selected) throw new Error(`${label} ${version} is not listed as an installable repository release.`); + const comparison = compareSemver(version, baseTarget.current_version); + const direction = comparison < 0 ? "downgrade" : comparison === 0 ? "repair reinstall" : "update"; + return { + ...baseTarget, + blocked: false, + blocked_reason: null, + update_available: true, + safe_target_version: version, + source_branch: selected.ref, + raw_target: selected, + rollback_safe: selected.rollback_safe !== false, + migration_notes: selected.migration_notes || baseTarget.migration_notes || "", + major_crossing: parseSemver(version)?.major !== parseSemver(baseTarget.current_version)?.major, + requires_manual_confirmation: true, + requested_version: version, + version_description: `${baseTarget.current_version} -> ${version} (${direction})`, + warnings: [ + ...(baseTarget.warnings || []), + comparison < 0 + ? "You selected an older code version. Lumi data is preserved, but an older release may not understand newer database changes." + : comparison === 0 + ? "This reinstalls the selected release to repair its managed files." + : `You selected ${version} instead of the recommended target.` + ] + }; +} + +async function applyCoreUpdate({ source = "stable", remote = null, version = null, publish } = {}) { return withOperation("core", async () => { const status = getUpdateStatus({ source, remote }); - const target = status.core; + const target = targetForRequestedVersion(status.core, version, "Core version"); if (target.blocked) throw new Error(target.blocked_reason || "Core update is blocked."); if (!target.update_available) throw new Error("No safe core update target is available."); const marker = createRecoveryMarker({ @@ -110,11 +153,12 @@ async function applyCoreUpdate({ source = "stable", remote = null, publish } = { emitProgress(publish, "update:recovery_marker", { target: "core", marker_id: marker.id }); emitProgress(publish, "update:download", { target: "core", branch: target.source_branch }); const managed = ensureManagedRepo(status.remote, target.source_branch); + verifyCoreFiles(managed.path, target.safe_target_version); emitProgress(publish, "update:apply", { target: "core" }); applyCoreFilesFromDirectory(managed.path); updateRecoveryMarker({ status: "verifying" }); emitProgress(publish, "update:verify", { target: "core" }); - verifyCoreFiles(); + verifyCoreFiles(repoRoot, target.safe_target_version); const record = snapshotRecord; markRecoveryMarkerComplete({ snapshot_id: record.id }); writeUpdateState({ @@ -160,11 +204,12 @@ async function applyCoreUpdate({ source = "stable", remote = null, publish } = { }); } -async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote = null, publish } = {}) { +async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote = null, version = null, publish } = {}) { return withOperation(`plugin:${pluginId}`, async () => { const status = getUpdateStatus({ source, remote }); - const target = status.plugins.find((plugin) => plugin.id === pluginId); - if (!target) throw new Error("Plugin is not installed."); + const statusTarget = status.plugins.find((plugin) => plugin.id === pluginId); + if (!statusTarget) throw new Error("Plugin was not found in the local or repository catalog."); + const target = targetForRequestedVersion(statusTarget, version, "Plugin version"); if (target.blocked) throw new Error(target.blocked_reason || "Plugin update is blocked."); if (!target.update_available) throw new Error("No safe plugin update target is available."); const marker = createRecoveryMarker({ @@ -204,10 +249,15 @@ async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote = emitProgress(publish, "update:snapshot", { target: "plugin", plugin_id: pluginId, snapshot_id: snapshotRecord.id }); emitProgress(publish, "update:download", { target: "plugin", plugin_id: pluginId, branch: target.source_branch }); emitProgress(publish, "update:apply", { target: "plugin", plugin_id: pluginId }); - const managed = applyPluginFromRepositorySnapshot(status.remote, target.source_branch, pluginId); + const managed = applyPluginFromRepositorySnapshot( + status.remote, + target.source_branch, + pluginId, + target.safe_target_version + ); updateRecoveryMarker({ status: "verifying" }); emitProgress(publish, "update:verify", { target: "plugin", plugin_id: pluginId }); - verifyPluginFiles(pluginId); + verifyPluginFiles(pluginId, repoRoot, target.safe_target_version); syncPluginRegistry(); const record = snapshotRecord; markRecoveryMarkerComplete({ snapshot_id: record.id }); @@ -221,8 +271,9 @@ async function applyPluginUpdateFromRepo(pluginId, { source = "stable", remote = last_target_id: pluginId, last_target_version: target.safe_target_version }); + emitProgress(publish, "update:restart_required", { target: "plugin", plugin_id: pluginId }); emitProgress(publish, "update:complete", { target: "plugin", plugin_id: pluginId, snapshot_id: record.id }); - return { status: "complete", restart_required: false, snapshot: record, target }; + return { status: "complete", restart_required: true, snapshot: record, target }; } catch (error) { if (snapshotRecord) { try { @@ -289,6 +340,7 @@ function disablePluginForRecovery(pluginId, publish) { module.exports = { applyCoreUpdate, applyPluginUpdateFromRepo, + targetForRequestedVersion, revertCoreSnapshot, revertPluginSnapshot, disablePluginForRecovery diff --git a/src/services/update-index.js b/src/services/update-index.js index 952feee..908586b 100644 --- a/src/services/update-index.js +++ b/src/services/update-index.js @@ -14,6 +14,7 @@ const { safeModeStatus } = require("./recovery-mode"); const { createMetadataReader, normalizeRepositoryTarget, + normalizeRepositoryRef, resolveSourceBranch } = require("./update-repository"); @@ -87,6 +88,50 @@ function coreManifest(reader) { {}; } +function releaseCatalog(reader) { + const raw = readJsonFromReader(reader, "release-index.json"); + const releases = Array.isArray(raw?.releases) ? raw.releases : []; + return releases.map((release) => { + const parsed = parseSemver(release?.version); + if (!parsed || release?.installable === false) return null; + try { + return { + ...release, + version: parsed.raw, + ref: normalizeRepositoryRef(release.ref) + }; + } catch { + return null; + } + }).filter(Boolean).sort((left, right) => compareSemver(right.version, left.version)); +} + +function availableVersionOptions(releases, kind, id = null) { + const seen = new Set(); + const options = []; + for (const release of releases) { + const version = kind === "core" + ? release.version + : kind === "tool" + ? release.tools?.[id] + : release.plugins?.[id]; + if (!parseSemver(version) || seen.has(version)) continue; + seen.add(version); + options.push({ + version: parseSemver(version).raw, + ref: release.ref, + core_version: release.version, + released_at: release.released_at || null, + rollback_safe: release.rollback_safe !== false, + data_policy: release.data_policy || "preserve", + dependency_policy: release.dependency_policy || "sync_on_restart", + migration_notes: release.migration_notes || "", + replaces_versions: Array.isArray(release.replaces_versions) ? release.replaces_versions : [] + }); + } + return options; +} + function localCoreVersion() { return readLocalJson("package.json")?.version || "0.0.0"; } @@ -247,7 +292,18 @@ function buildStatus({ kind, id, name, currentVersion, manifest, changelog, sour rollback_safe: false, unversioned: true }; - const targetResult = unversionedUpdate + const replacementTarget = versions.find((entry) => + Array.isArray(entry.replaces_versions) && entry.replaces_versions.includes(currentVersion) + ); + const targetResult = replacementTarget + ? { + target: replacementTarget, + latest: latest || replacementTarget, + blocked: false, + replacement: true, + warning: `Version ${currentVersion} was published with an incorrect version number; install ${replacementTarget.version} as its supported replacement.` + } + : unversionedUpdate ? { target: unversionedTarget, latest: latest || unversionedTarget, @@ -257,7 +313,9 @@ function buildStatus({ kind, id, name, currentVersion, manifest, changelog, sour : findSafeTarget(currentVersion, versions); const target = targetResult.target; const range = target && !unversionedUpdate - ? collectChangelogRange(currentVersion, target.version, changelog.length ? changelog : versions) + ? targetResult.replacement + ? (changelog.length ? changelog : versions).filter((entry) => entry.version === target.version) + : collectChangelogRange(currentVersion, target.version, changelog.length ? changelog : versions) : changelog; const warnings = []; const dangers = []; @@ -273,9 +331,7 @@ function buildStatus({ kind, id, name, currentVersion, manifest, changelog, sour current_version: currentVersion, latest_available_version: latest?.version || rawVersion || currentVersion, safe_target_version: target?.version || null, - update_available: Boolean( - target && (unversionedUpdate || compareSemver(target.version, currentVersion) > 0) - ), + update_available: Boolean(target && (targetResult.replacement || unversionedUpdate || compareSemver(target.version, currentVersion) > 0)), blocked: Boolean(targetResult.blocked), blocked_reason: targetResult.reason || null, source_branch: sourceBranch, @@ -289,7 +345,8 @@ function buildStatus({ kind, id, name, currentVersion, manifest, changelog, sour warnings, dangers, unversioned_update: unversionedUpdate, - requires_manual_confirmation: unversionedUpdate, + requires_manual_confirmation: unversionedUpdate || Boolean(targetResult.replacement), + version_correction: Boolean(targetResult.replacement), requirements: target?.requirements || manifest?.requirements || [], migration_notes: target?.migration_notes || manifest?.migration_notes || "", rollback_safe: target?.rollback_safe !== false, @@ -306,6 +363,7 @@ function getUpdateStatus(options = {}) { const reader = createMetadataReader(configuredRemote, sourceBranch); const remote = reader.repository || configuredRemote; try { + const releases = releaseCatalog(reader); const core = buildStatus({ kind: "core", currentVersion: localCoreVersion(), @@ -314,6 +372,7 @@ function getUpdateStatus(options = {}) { sourceBranch, channel: requestedSource === "experimental" ? "experimental" : "stable" }); + core.available_versions = availableVersionOptions(releases, "core"); const registry = pluginRegistry(); const remoteDirs = new Set(reader.listPluginDirs()); const remotePluginDirs = new Set(); @@ -355,7 +414,7 @@ function getUpdateStatus(options = {}) { const manifest = readJsonFromReader(reader, `${basePath}/plugin.json`) || {}; const changelog = changelogEntries(reader, basePath); const dbPlugin = registry.get(plugin.id); - return buildStatus({ + const status = buildStatus({ kind: "plugin", id: plugin.id, name: plugin.name, @@ -366,6 +425,8 @@ function getUpdateStatus(options = {}) { channel: requestedSource === "experimental" ? "experimental" : "stable", installed: plugin.installed !== false }); + status.available_versions = availableVersionOptions(releases, "plugin", plugin.id); + return status; }); const localTools = localToolCandidates(); for (const [toolId, tool] of localTools.entries()) { @@ -378,13 +439,15 @@ function getUpdateStatus(options = {}) { ...Array.from(remoteTools.keys()) ])).sort((a, b) => a.localeCompare(b)).map((toolId) => { const localTool = localTools.get(toolId); - return buildToolStatus({ + const status = buildToolStatus({ id: toolId, localMetadata: localTool?.metadata || null, remoteMetadata: remoteTools.get(toolId) || null, sourceBranch, installed: localTool?.installed === true }); + status.available_versions = availableVersionOptions(releases, "tool", toolId); + return status; }); const lumiAiPlugin = plugins.find((plugin) => plugin.id === "lumi_ai"); if (lumiAiPlugin) { @@ -402,6 +465,7 @@ function getUpdateStatus(options = {}) { source_branch: sourceBranch, requested_source: requestedSource, remote, + release_catalog: releases, core, plugins, ai_tools: aiTools, @@ -426,6 +490,9 @@ function getUpdateStatus(options = {}) { } module.exports = { + availableVersionOptions, + buildStatus, getUpdateStatus, + releaseCatalog, resolveSourceBranch }; diff --git a/src/services/update-manager.js b/src/services/update-manager.js index bc55760..ffea08c 100644 --- a/src/services/update-manager.js +++ b/src/services/update-manager.js @@ -350,6 +350,39 @@ function verifyPatchPackage(rootPath) { if (!hasAnyFiles(rootPath)) { throw new Error("Patch archive is empty."); } + const manifestPath = path.join(rootPath, "patch-manifest.json"); + if (!fs.existsSync(manifestPath)) return null; + const manifest = JSON.parse(fs.readFileSync(manifestPath, "utf8")); + if (manifest.target !== "core" || manifest.data_policy !== "preserve") { + throw new Error("Patch manifest must target core and preserve local data."); + } + const currentVersion = JSON.parse(fs.readFileSync(path.join(repoRoot, "package.json"), "utf8")).version; + if (Array.isArray(manifest.from_versions) && !manifest.from_versions.includes(currentVersion)) { + throw new Error(`This patch supports core ${manifest.from_versions.join(", ")}, not ${currentVersion}.`); + } + const files = manifest.files && typeof manifest.files === "object" ? manifest.files : {}; + if (!Object.keys(files).length) throw new Error("Patch manifest does not list any files."); + const preservePaths = buildCorePreservePaths(repoRoot); + for (const [relativePath, expectedHash] of Object.entries(files)) { + const normalized = normalizeRelative(relativePath); + if (!normalized || normalized.startsWith("/") || normalized.includes("../") || isRelativePathIgnored(normalized, preservePaths)) { + throw new Error(`Patch manifest contains an unsafe or preserved path: ${relativePath}`); + } + const filePath = path.resolve(rootPath, normalized); + if (!filePath.startsWith(path.resolve(rootPath) + path.sep) || !fs.statSync(filePath, { throwIfNoEntry: false })?.isFile()) { + throw new Error(`Patch file is missing: ${relativePath}`); + } + const actualHash = crypto.createHash("sha256").update(fs.readFileSync(filePath)).digest("hex"); + if (actualHash !== expectedHash) throw new Error(`Patch file checksum failed: ${relativePath}`); + } + const packageFile = path.join(rootPath, "package.json"); + if (manifest.to_version && fs.existsSync(packageFile)) { + const packageVersion = JSON.parse(fs.readFileSync(packageFile, "utf8")).version; + if (packageVersion !== manifest.to_version) { + throw new Error(`Patch targets ${manifest.to_version}, but package.json contains ${packageVersion}.`); + } + } + return manifest; } function verifyPluginPackage(rootPath) { @@ -597,6 +630,7 @@ function applyCoreFilesFromDirectory(rootPath) { function applyCorePatch(rootPath) { const preservePaths = buildCorePreservePaths(repoRoot); + preservePaths.add("patch-manifest.json"); copyDirectory( rootPath, repoRoot, @@ -712,8 +746,9 @@ async function applyBotUpdate(zipPath, options = {}) { const mode = options.mode === "patch" ? "patch" : "full"; const rootPath = mode === "patch" ? resolvePatchRoot(tempDir) : resolveZipRoot(tempDir); + let patchManifest = null; if (mode === "patch") { - verifyPatchPackage(rootPath); + patchManifest = verifyPatchPackage(rootPath); } else { verifyBotPackage(rootPath); } @@ -721,8 +756,8 @@ async function applyBotUpdate(zipPath, options = {}) { const marker = createRecoveryMarker({ target_kind: "core", target_id: "core", - from_version: options.metadata?.from_version || null, - to_version: options.metadata?.to_version || null, + from_version: options.metadata?.from_version || (patchManifest ? JSON.parse(fs.readFileSync(path.join(repoRoot, "package.json"), "utf8")).version : null), + to_version: options.metadata?.to_version || patchManifest?.to_version || null, source_branch: options.metadata?.source_branch || "manual_zip", update_method: options.mode === "patch" ? "zip_patch" : "zip", rollback_safe: options.metadata?.rollback_safe !== false, @@ -735,6 +770,8 @@ async function applyBotUpdate(zipPath, options = {}) { target_kind: "core", update_method: options.mode === "patch" ? "zip_patch" : "zip", ...options.metadata, + from_version: options.metadata?.from_version || (patchManifest ? JSON.parse(fs.readFileSync(path.join(repoRoot, "package.json"), "utf8")).version : null), + to_version: options.metadata?.to_version || patchManifest?.to_version || null, recovery_marker_id: marker.id } }); @@ -937,6 +974,7 @@ module.exports = { discardSnapshot, applyCoreFilesFromDirectory, applyPluginFiles, + verifyPatchPackage, resetPluginCode, resetDirectoryForFullUpdate, buildCorePreservePaths, diff --git a/src/services/update-repository.js b/src/services/update-repository.js index 98f5f2d..0b4a41c 100644 --- a/src/services/update-repository.js +++ b/src/services/update-repository.js @@ -159,8 +159,26 @@ function resetManagedRepo(repository) { }); } +function normalizeRepositoryRef(value) { + const reference = String(value || "main").trim(); + if (!reference || reference.length > 180 || reference.startsWith("-") || reference.includes("..") || reference.includes("@{")) { + throw new Error("Invalid repository version reference."); + } + if (!/^(?:refs\/(?:heads|tags)\/)?[A-Za-z0-9][A-Za-z0-9._/-]*$/.test(reference)) { + throw new Error("Invalid repository version reference."); + } + return reference.replace(/^origin\//, ""); +} + +function managedTargetForRef(reference) { + if (reference.startsWith("refs/tags/")) return reference; + if (reference.startsWith("refs/heads/")) return `refs/remotes/origin/${reference.slice("refs/heads/".length)}`; + return `refs/remotes/origin/${reference}`; +} + function ensureManagedRepo(remote, branch) { const repository = resolveRepositoryRemote(remote); + const reference = normalizeRepositoryRef(branch); migrateZipEraLayout(repository); fs.mkdirSync(updateCacheDir, { recursive: true }); @@ -177,15 +195,20 @@ function ensureManagedRepo(remote, branch) { } } - runGit(["fetch", "--prune", "origin"], { + runGit(["fetch", "--prune", "--tags", "origin"], { cwd: managedRepoDir, timeout: 300000 }); - runGit(["checkout", "-B", branch, `origin/${branch}`], { + const target = managedTargetForRef(reference); + runGit(["rev-parse", "--verify", `${target}^{commit}`], { + cwd: managedRepoDir, + timeout: 60000 + }); + runGit(["checkout", "--detach", target], { cwd: managedRepoDir, timeout: 120000 }); - runGit(["reset", "--hard", `origin/${branch}`], { + runGit(["reset", "--hard", target], { cwd: managedRepoDir, timeout: 120000 }); @@ -196,14 +219,16 @@ function ensureManagedRepo(remote, branch) { writeUpdateState({ remote: repository, - branch, + branch: reference, + commit: runGit(["rev-parse", "HEAD"], { cwd: managedRepoDir }), managed_repo_path: managedRepoDir, last_fetch_at: new Date().toISOString(), live_install_git_repository: isGitRepository(repoRoot) }); return { repository, - branch, + branch: reference, + commit: runGit(["rev-parse", "HEAD"], { cwd: managedRepoDir }), path: managedRepoDir }; } @@ -249,6 +274,7 @@ module.exports = { isRepositoryUrl, isGitRepository, normalizeRepositoryTarget, + normalizeRepositoryRef, resolveRepositoryRemote, resolveSourceBranch, migrateZipEraLayout, diff --git a/src/web/server.js b/src/web/server.js index 4c290a2..e3cc390 100644 --- a/src/web/server.js +++ b/src/web/server.js @@ -156,7 +156,6 @@ const { supportFeedback, unmergeFeedback } = require("../services/feedback"); -const { saveCorrectionKnowledgeFile } = require("../../plugins/okf/backend/file_knowledge"); function ensureSessionSecret() { let secret = getSetting("session_secret"); @@ -5819,7 +5818,11 @@ function createWebServer({ loadPlugins, discordClient }) { app.post("/admin/feedback/:id/create-okf-correction", requireRole("admin"), (req, res) => { try { const feedback = getFeedbackForAdmin(req.params.id); - const correction = saveCorrectionKnowledgeFile(process.cwd(), buildFeedbackCorrectionPayload(feedback, req.body)); + const saveCorrection = global.lumiFrameworks?.okf?.saveCorrection; + if (typeof saveCorrection !== "function") { + throw new Error("Install and enable the OKF Knowledge plugin before creating an OKF correction."); + } + const correction = saveCorrection(buildFeedbackCorrectionPayload(feedback, req.body), req.session.user); if (!correction) { throw new Error("Correction OKF file could not be created."); } @@ -6605,6 +6608,7 @@ function createWebServer({ loadPlugins, discordClient }) { try { const result = await applyCoreUpdate({ source: updateSourceFrom(req), + version: req.body.version || null, publish: publishWebEvent }); sendUpdateResult(req, res, { @@ -6761,12 +6765,15 @@ function createWebServer({ loadPlugins, discordClient }) { try { const result = await applyPluginUpdateFromRepo(req.params.id, { source: updateSourceFrom(req), + version: req.body.version || null, publish: publishWebEvent }); sendUpdateResult(req, res, { ...result, - message: "Plugin update applied." + refresh_after_ms: 5000, + message: "Plugin update applied. Lumi will restart after the confirmation notice." }); + scheduleRestartAfterNotice(); } catch (error) { sendUpdateError(req, res, error); } diff --git a/src/web/views/admin-updates.ejs b/src/web/views/admin-updates.ejs index dec494d..871795e 100644 --- a/src/web/views/admin-updates.ejs +++ b/src/web/views/admin-updates.ejs @@ -145,6 +145,24 @@ <% } %> + <% if (core.available_versions?.length) { %> +
+ Install a specific core version +
+ + + +
+

Versions are resolved from immutable repository tags and checked before any live files are replaced. Local databases, settings, plugins, models, uploads, and knowledge remain in place.

+
+ <% } %> <% } %> @@ -262,6 +280,24 @@ <% } %> + <% if (plugin.available_versions?.length) { %> +
+ Install a specific plugin version +
+ + + +
+

The plugin’s data folder is retained. Lumi restarts so the selected plugin code is actually loaded.

+
+ <% } %> <% }) %> diff --git a/update-manifest.json b/update-manifest.json index 0330c91..cd75863 100644 --- a/update-manifest.json +++ b/update-manifest.json @@ -1,26 +1,30 @@ { "name": "Lumi Core", - "version": "1.2.0", + "version": "0.2.0", "channel": "stable", "released_at": "2026-07-18", "compatible_from": "0.1.9", - "compatibility_bridge": true, - "migration_kind": "compatibility_bridge", - "migration_notes": "Lumi performs required database and generated-knowledge migrations automatically. Local configuration, plugin data, community knowledge, AI models, runtimes, uploads, logs, and secrets remain preserved.", + "migration_kind": "version_correction", + "replaces_versions": [ + "1.2.0" + ], + "migration_notes": "Corrects the mistakenly published 1.2.0 version to 0.2.0. Lumi synchronizes runtime dependencies on restart and preserves settings, databases, plugin data, community knowledge, AI models, runtimes, uploads, logs, and secrets.", "rollback_safe": true, "requirements": [ "Node.js 18 or newer" ], "versions": [ { - "version": "1.2.0", + "version": "0.2.0", "channel": "stable", "released_at": "2026-07-18", "compatible_from": "0.1.9", - "compatibility_bridge": true, - "migration_kind": "compatibility_bridge", + "migration_kind": "version_correction", + "replaces_versions": [ + "1.2.0" + ], "rollback_safe": true, - "migration_notes": "Automatic migrations only; preserved local data is not replaced." + "migration_notes": "Automatic dependency synchronization and version correction only; preserved local data is not replaced." } ] }