diff --git a/CHANGELOG.md b/CHANGELOG.md index b10d7c5..169ca8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Lumi changelog +## 0.2.11 + +- Added structured component, activity, event, and request metadata across Core, plugins, platforms, commands, webhooks, diagnostics, and WebUI administrator actions. +- Added recursive credential redaction, bounded log payloads, automatic age/count retention, and a scoped logger supplied to plugins. +- Upgraded Admin Logs with full server-side search and filters, filtered totals, live entries, clearer metadata, configurable retention, and text or JSON Lines downloads. + ## 0.2.10 - Fixed local Windows startup under Node.js 24 by invoking npm through its JavaScript CLI instead of directly spawning `npm.cmd`. diff --git a/README.md b/README.md index a0d1e32..9c7c8a4 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,14 @@ Before starting Lumi, you can check the runtime and native dependency with `npm run verify:preflight`. Run the complete syntax and focused test suite with `npm run verify:all`; it stops at the first failure and prints the failing check. +## Logs + +Administrators can review structured Core, plugin, platform, command, WebUI, +security, and integration activity under **Admin → Logs**. Logs support live +updates, component/activity filters, server-side search, redacted details, +request IDs, retention limits, and text or JSON Lines downloads. Plugin and +core logging conventions are documented in [`docs/logging.md`](docs/logging.md). + You can also seed local configuration with a `.env` file. Use `.env.example` as the template; `.env` is ignored by git. diff --git a/TODO.md b/TODO.md index 4813bf2..e0a8ee4 100644 --- a/TODO.md +++ b/TODO.md @@ -677,6 +677,7 @@ This section is for Lumi chat answer feedback and AI Improvement Center work, no ## Done +- 2026-07-18: Reworked shared logging repo-wide with structured component/category/event/request metadata, recursive secret redaction, bounded details, automatic retention, scoped plugin loggers, platform/command/webhook/lifecycle/admin HTTP auditing, live `/admin/logs` updates, complete server-side filters and summaries, and text/JSONL exports. - 2026-07-18: Fixed core 0.2.10 local Windows startup under Node.js 24 by resolving npm's JavaScript CLI through the active Node installation, retaining a command-shell fallback, and verifying automatic dependency repair through the real launcher on port 3000. - 2026-07-18: Fixed core 0.2.9 and Lumi AI 0.8.5 forms whose `action` or `method` fields could shadow the form endpoint, added a shared clobber-safe resolver, and migrated the affected feedback/settings requests to it. - 2026-07-18: Fixed Lumi AI 0.8.4 Improvement Center state changes for no-change, dismiss, edit, and restore actions with an explicit authenticated JSON contract, actionable errors, safe diagnostics, and focused regression coverage. diff --git a/docs/logging.md b/docs/logging.md new file mode 100644 index 0000000..8a4743f --- /dev/null +++ b/docs/logging.md @@ -0,0 +1,46 @@ +# Logging + +Lumi stores core, plugin, integration, command, and administrator activity in the shared SQLite `logs` table. Administrators review it at **Admin → Logs**. The page supports severity, component, activity type, range, full server-side search, and entry-count filters. New matching entries arrive through Lumi's authenticated event stream without a page refresh. + +Each entry may include: + +- `level`: `debug`, `info`, `warn`, or `error`. +- `source`: the component that produced the entry, such as `core:web`, `core:commands`, `platform:twitch`, or `plugin:lumi_ai`. +- `category`: a broad activity type such as `lifecycle`, `audit`, `http`, `command`, `integration`, `security`, or `plugin`. +- `event`: a stable machine-readable event name. +- `request_id`: the matching HTTP or diagnostic request identifier when available. +- A short message and optional structured details. + +## Writing logs + +Existing `log(level, message, details)` calls remain supported. New core services should use a scoped logger: + +```js +const { createLogger } = require("./logger"); +const logger = createLogger("core:example", { category: "integration" }); + +logger.info("Example connected", { endpoint: "primary" }, { event: "example_ready" }); +logger.error("Example request failed", error, { event: "example_failed" }); +``` + +Every loaded plugin receives a logger scoped to `plugin:` in its `init` context: + +```js +init({ logger }) { + logger.info("Plugin feature ready", { mode: "automatic" }, { event: "feature_ready" }); +} +``` + +Use `debug` for high-volume successful activity, `info` for meaningful state changes, `warn` for degraded or rejected work, and `error` for failed work that needs attention. Do not log full chat messages, request bodies, credentials, session data, third-party response bodies, or user access tokens. + +The shared logger recursively redacts credential-shaped object keys, authorization headers, and common secret query parameters before storage. Redaction is a final safety boundary, not permission to pass known secrets to the logger. Messages and details are length-bounded so a malformed response cannot grow the database without limit. + +## Administrator actions and requests + +Mutating WebUI requests produce an `audit` entry with method, matched route, result status, duration, role, user ID, and a request ID. Failed and unusually slow requests are recorded as `http` warnings or errors. Query strings and submitted bodies are not included. Webhook, platform, diagnostics, plugin lifecycle, command failure, startup, and shutdown paths use dedicated sources and events. + +## Retention and downloads + +By default Lumi retains logs for 30 days and keeps at most 100,000 entries. Administrators can change both limits under **Log storage** on the logs page. Cleanup runs at startup and immediately after retention settings are saved. It only removes rows from the shared logs table. + +Filtered downloads are available as readable text or JSON Lines. JSON Lines preserves all structured fields for external diagnostics without changing Lumi's database. diff --git a/docs/lumi-ui.md b/docs/lumi-ui.md index a2b0190..e482039 100644 --- a/docs/lumi-ui.md +++ b/docs/lumi-ui.md @@ -182,10 +182,13 @@ uptime, memory, plugin counts, content counts, and recent log severity totals. The dashboard renders lightweight SVG graphs using Lumi tokens and does not add a frontend framework dependency. -The logs page keeps server-side range/severity/limit filters and adds a labeled -responsive filter bar with search, reset, refresh, and download actions. Search -filters the loaded entries client-side; changing range, severity, or limit -reloads the same `/admin/logs` route with query parameters. +The logs page provides server-side range, severity, component, activity type, +search, and limit filters in a responsive control bar. Summary totals reflect +the complete filtered result rather than only the visible page. Entries expose +their source, category, stable event name, and request ID, and matching events +arrive live through the existing authenticated event stream. Downloads retain +the active component/activity/search filters and support readable text or JSON +Lines. Retention controls remain collapsed until needed. ## Updates And Local-Only Files diff --git a/knowledge/core/lumi-core.md b/knowledge/core/lumi-core.md index 71f68e1..6ba8300 100644 --- a/knowledge/core/lumi-core.md +++ b/knowledge/core/lumi-core.md @@ -14,7 +14,7 @@ editable: false Lumi is the core web UI and bot runtime. ## Runtime Package: lumi-bot -Version: 0.2.10 +Version: 0.2.11 ## Routes - GET /api/events - POST /api/destructive-confirmations @@ -91,6 +91,7 @@ Version: 0.2.10 - POST /admin/theming/custom/:id/delete - POST /admin/theming - GET /admin/logs +- POST /admin/logs/retention - GET /admin/logs/download - GET /admin/feedback - POST /admin/feedback/export @@ -815,16 +816,25 @@ Version: 0.2.10 ### GET /admin/logs - Purpose: Displays, downloads, or manages application logs. -- Inputs: query: `level`, `limit`, `range` +- Inputs: query: `category`, `level`, `limit`, `q`, `range`, `source` - Response format: HTML page rendered from an EJS view - Access: admin access expected - Side effects: Usually read-only. - Limits/notes: Generated from static route source analysis; confirm exact behavior in the handler before changing integrations. +### POST /admin/logs/retention + +- Purpose: Displays, downloads, or manages application logs. +- Inputs: body: `max_age_days`, `max_entries` +- Response format: HTTP redirect after handling the request +- Access: admin access expected; logged-in session required or used +- Side effects: writes or mutates server-side state +- Limits/notes: Generated from static route source analysis; confirm exact behavior in the handler before changing integrations. Most non-API POST routes are browser form submissions and usually redirect after completion. + ### GET /admin/logs/download - Purpose: Displays, downloads, or manages application logs. -- Inputs: query: `level`, `limit`, `range` +- Inputs: query: `category`, `format`, `level`, `limit`, `q`, `range`, `source` - Response format: plain or HTML response - Access: admin access expected - Side effects: writes or mutates server-side state diff --git a/package-lock.json b/package-lock.json index 7244e75..8db439a 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "lumi-bot", - "version": "0.2.10", + "version": "0.2.11", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "lumi-bot", - "version": "0.2.10", + "version": "0.2.11", "dependencies": { "adm-zip": "^0.5.12", "better-sqlite3": "^11.5.0", diff --git a/package.json b/package.json index 81c96b1..1ed62c5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "lumi-bot", - "version": "0.2.10", + "version": "0.2.11", "private": true, "type": "commonjs", "scripts": { diff --git a/release-index.json b/release-index.json index 270f4cc..9e1c07d 100644 --- a/release-index.json +++ b/release-index.json @@ -2,6 +2,36 @@ "schema_version": 1, "channel": "stable", "releases": [ + { + "version": "0.2.11", + "ref": "refs/tags/v0.2.11", + "released_at": "2026-07-18", + "installable": true, + "rollback_safe": true, + "replaces_versions": [ + "1.2.0" + ], + "data_policy": "preserve", + "dependency_policy": "sync_on_restart", + "migration_notes": "Adds structured, redacted, bounded logging and new optional log metadata columns. Existing logs and all settings, databases, plugin data, models, uploads, feedback, and secrets are preserved.", + "plugins": { + "auto-vc": "0.1.6", + "birthday": "0.1.3", + "economy-framework": "0.2.10", + "economy-games": "0.1.7", + "expression-interaction": "0.2.1", + "lumi_ai": "0.8.5", + "moderation": "0.1.5", + "okf": "0.1.1", + "quotes": "0.1.2", + "sample-plugin": "0.1.0", + "throne_wishlist": "0.1.2", + "welcome_messages": "0.1.1" + }, + "tools": { + "lumi_ai_web_search": "0.1.1" + } + }, { "version": "0.2.10", "ref": "refs/tags/v0.2.10", diff --git a/scripts/verify-all.js b/scripts/verify-all.js index a3923a9..3492c97 100644 --- a/scripts/verify-all.js +++ b/scripts/verify-all.js @@ -8,6 +8,7 @@ const checks = [ "scripts/verify-webui.js", "scripts/verify-web-auth.js", "scripts/verify-feedback-system.js", + "scripts/verify-logging.js", "scripts/verify-placeholders.js", "scripts/verify-release-metadata.js", "scripts/verify-update-system.js", diff --git a/scripts/verify-logging.js b/scripts/verify-logging.js new file mode 100644 index 0000000..f891657 --- /dev/null +++ b/scripts/verify-logging.js @@ -0,0 +1,85 @@ +const assert = require("assert"); +const fs = require("fs"); +const path = require("path"); + +const root = path.join(__dirname, ".."); +const sandbox = fs.mkdtempSync(path.join(root, ".tmp-lumi-logging-")); +const serviceDir = path.join(sandbox, "src", "services"); +fs.mkdirSync(serviceDir, { recursive: true }); +for (const file of ["db.js", "logger.js", "web-events.js"]) { + fs.copyFileSync(path.join(root, "src", "services", file), path.join(serviceDir, file)); +} + +let database; +try { + database = require(path.join(serviceDir, "db.js")); + database.db.exec(`CREATE TABLE logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + level TEXT NOT NULL, + message TEXT NOT NULL, + details TEXT, + created_at INTEGER NOT NULL + )`); + database.migrate(); + const logger = require(path.join(serviceDir, "logger.js")); + const { db } = database; + const columns = db.prepare("PRAGMA table_info(logs)").all().map((column) => column.name); + for (const column of ["source", "category", "event", "request_id"]) { + assert(columns.includes(column), `logs.${column} migration`); + } + + logger.log("info", "Legacy-compatible entry", { status: "ready" }); + const pluginLog = logger.createLogger("plugin:example", { category: "plugin" }); + pluginLog.warn("Example warning", { + access_token: "must-not-be-stored", + nested: { password: "also-secret", safe: "visible" }, + url: "https://example.com/run?token=hidden&mode=safe" + }, { event: "example_warning", requestId: "request-123" }); + logger.withLogContext({ source: "core:test", category: "verification", event: "context_entry" }, () => { + logger.log("debug", "Context-aware entry", "authorization=private-value"); + }); + + const all = logger.listLogs({ limit: 20 }); + assert.equal(all.length, 3); + const warning = all.find((entry) => entry.event === "example_warning"); + assert.equal(warning.source, "plugin:example"); + assert.equal(warning.category, "plugin"); + assert.equal(warning.request_id, "request-123"); + assert(warning.details.includes("[REDACTED]")); + assert.equal(warning.details.includes("must-not-be-stored"), false); + assert.equal(warning.details.includes("also-secret"), false); + assert.equal(warning.details.includes("token=hidden"), false); + assert(warning.details.includes("visible")); + + assert.equal(logger.listLogs({ sources: ["plugin:example"] }).length, 1); + assert.equal(logger.listLogs({ categories: ["verification"] }).length, 1); + assert.equal(logger.listLogs({ search: "request-123" }).length, 1); + assert.equal(logger.listLogs({ levels: ["error"] }).length, 0); + const summary = logger.summarizeLogs({}); + assert.equal(summary.total, 3); + assert.equal(summary.levels.warn, 1); + assert(logger.listLogFacets().sources.some((item) => item.value === "plugin:example" && item.count === 1)); + + db.prepare("UPDATE logs SET created_at = ? WHERE message = ?").run( + Date.now() - 45 * 24 * 60 * 60 * 1000, + "Legacy-compatible entry" + ); + const cleanup = logger.cleanupLogs({ maxAgeDays: 30, maxEntries: 1000 }); + assert.equal(cleanup.expired, 1); + assert.equal(logger.listLogs({ search: "Legacy-compatible" }).length, 0); + + const logView = fs.readFileSync(path.join(root, "src", "web", "views", "admin-logs.ejs"), "utf8"); + assert.match(logView, /data-log-source/); + assert.match(logView, /data-log-category/); + assert.match(logView, /data-log-live-status/); + assert.match(logView, /name="format"/); + const serverSource = fs.readFileSync(path.join(root, "src", "web", "server.js"), "utf8"); + assert.match(serverSource, /app\.post\("\/admin\/logs\/retention"/); + assert.match(serverSource, /summarizeLogs\(query\)/); + assert.match(serverSource, /"admin_action"/); + + console.log("Logging verification passed: structured scope, redaction, search, summaries, retention, admin audit, and live UI wiring."); +} finally { + try { database?.db?.close(); } catch {} + fs.rmSync(sandbox, { recursive: true, force: true }); +} diff --git a/scripts/verify-release-metadata.js b/scripts/verify-release-metadata.js index 757a89c..550ad0d 100644 --- a/scripts/verify-release-metadata.js +++ b/scripts/verify-release-metadata.js @@ -4,8 +4,8 @@ const path = require("path"); const { findSafeTarget } = require("../src/services/versioning"); const root = path.join(__dirname, ".."); -const releaseVersion = "0.2.10"; -const previousCoreVersion = "0.2.9"; +const releaseVersion = "0.2.11"; +const previousCoreVersion = "0.2.10"; const earliestCompatibleCoreVersion = "0.1.9"; const changedPlugins = { "auto-vc": { from: "0.1.5", to: "0.1.6", knowledge: "auto-vc" }, @@ -87,4 +87,4 @@ assert.equal(webSearch.minimum_lumi_version, "0.2.0"); assert.equal(webSearch.minimum_lumi_ai_version, "0.8.2"); assert.equal(hasVersionHeading(readText("plugins/lumi_ai_web_search/CHANGELOG.md"), webSearch.version), true); -console.log("Release metadata verification passed: core 0.2.10, Lumi AI 0.8.5, and synchronized package metadata."); +console.log("Release metadata verification passed: core 0.2.11, Lumi AI 0.8.5, and synchronized package metadata."); diff --git a/scripts/verify-update-system.js b/scripts/verify-update-system.js index 60d7cf3..740b456 100644 --- a/scripts/verify-update-system.js +++ b/scripts/verify-update-system.js @@ -16,7 +16,7 @@ function readJson(relativePath) { const releaseIndex = readJson("release-index.json"); const releaseVersions = releaseIndex.releases.map((release) => release.version); -assert.deepEqual(releaseVersions, ["0.2.10", "0.2.9", "0.2.8", "0.2.7", "0.2.6", "0.2.5", "0.2.4", "0.2.3", "0.2.2", "0.2.1", "0.2.0", "0.1.9"]); +assert.deepEqual(releaseVersions, ["0.2.11", "0.2.10", "0.2.9", "0.2.8", "0.2.7", "0.2.6", "0.2.5", "0.2.4", "0.2.3", "0.2.2", "0.2.1", "0.2.0", "0.1.9"]); assert.equal(new Set(releaseVersions).size, releaseVersions.length, "release versions must be unique"); for (const release of releaseIndex.releases) { assert.equal(normalizeRepositoryRef(release.ref), release.ref); @@ -37,6 +37,7 @@ for (const [toolId, version] of Object.entries(currentRelease.tools)) { const baseTarget = { current_version: "0.2.4", available_versions: [ + { version: "0.2.11", ref: "refs/tags/v0.2.11", rollback_safe: true }, { version: "0.2.10", ref: "refs/tags/v0.2.10", rollback_safe: true }, { version: "0.2.9", ref: "refs/tags/v0.2.9", rollback_safe: true }, { version: "0.2.8", ref: "refs/tags/v0.2.8", rollback_safe: true }, @@ -70,7 +71,7 @@ const corrected = buildStatus({ channel: "stable" }); assert.equal(corrected.version_correction, true); -assert.equal(corrected.safe_target_version, "0.2.10"); +assert.equal(corrected.safe_target_version, "0.2.11"); assert.equal(corrected.update_available, true); assert.equal(corrected.blocked, false); diff --git a/src/main.js b/src/main.js index 94bb48a..88bbf4f 100644 --- a/src/main.js +++ b/src/main.js @@ -28,6 +28,17 @@ async function main() { ensureDefaults(); registerCorePlaceholders(); logger.hookConsole(); + const runtimeLog = logger.createLogger("core:runtime", { category: "lifecycle" }); + const logCleanup = logger.cleanupLogs({ + maxAgeDays: getSetting("log_retention_days", logger.DEFAULT_MAX_AGE_DAYS), + maxEntries: getSetting("log_retention_max_entries", logger.DEFAULT_MAX_ENTRIES) + }); + runtimeLog.info("Lumi startup initiated", { + version: require("../package.json").version, + node: process.version, + platform: process.platform, + log_entries_removed: logCleanup.removed + }, { event: "startup" }); try { cleanupSnapshots(); } catch (error) { @@ -95,7 +106,9 @@ async function main() { const port = Number(process.env.PORT || 3000); app.listen(port, () => { - console.log(`WebUI listening on http://localhost:${port}`); + runtimeLog.run({ event: "web_ready" }, () => { + console.log(`WebUI listening on http://localhost:${port}`, { port }); + }); }); const autoUpdateEnabled = getSetting("auto_update_enabled", false); @@ -127,6 +140,7 @@ async function main() { return; } shuttingDown = true; + runtimeLog.info("Lumi shutdown started", null, { event: "shutdown" }); await overlayConnectorManager.stop(); await stopPlugins(); await stopBot(); diff --git a/src/services/command-router.js b/src/services/command-router.js index f3c9253..761d4bb 100644 --- a/src/services/command-router.js +++ b/src/services/command-router.js @@ -8,6 +8,9 @@ const { const { normalizeRandomReplies, selectRandomReply } = require("./command-random"); const { getEnabledPlatformIds, normalizePlatformSelection } = require("./platforms"); const placeholders = require("./placeholders"); +const { createLogger } = require("./logger"); + +const commandLog = createLogger("core:commands", { category: "command" }); function createCommandRouter({ settings }) { const commandMap = new Map(); @@ -101,6 +104,11 @@ function createCommandRouter({ settings }) { }); if (customHandled) { incrementCommands(user.id); + commandLog.debug("Custom command completed", { + trigger, + platform, + user_id: user.id + }, { event: "custom_command_completed" }); return true; } @@ -112,15 +120,33 @@ function createCommandRouter({ settings }) { await safeReply(reply, result); recordCommandUsage(handler.commandId); incrementCommands(user.id); + commandLog.debug("Command completed", { + command_id: handler.commandId, + trigger, + platform, + user_id: user.id + }, { event: "command_completed" }); return true; } if (result === true) { recordCommandUsage(handler.commandId); incrementCommands(user.id); + commandLog.debug("Command completed", { + command_id: handler.commandId, + trigger, + platform, + user_id: user.id + }, { event: "command_completed" }); return true; } } catch (error) { - console.error("Command handler failed", error); + commandLog.error("Command handler failed", { + command_id: handler.commandId, + trigger, + platform, + user_id: user.id, + error + }, { event: "command_failed" }); await safeReply(reply, "Command failed to execute."); return true; } @@ -214,7 +240,12 @@ async function handleCustomCommand({ trigger, platform, ctx, raw, reply }) { recordCommandUsage(`custom:${trigger}`); return true; } catch (error) { - console.error("Failed to reply to command", error); + commandLog.error("Custom command failed", { + trigger, + platform, + user_id: ctx.user.id, + error + }, { event: "custom_command_failed" }); await safeReply(reply, "Command failed to execute."); return true; } @@ -251,7 +282,7 @@ async function safeReply(reply, content) { try { await reply(content); } catch (error) { - console.error("Command reply failed", error); + commandLog.error("Command reply failed", error, { event: "command_reply_failed" }); } } diff --git a/src/services/db.js b/src/services/db.js index 3dafa82..3a3df9e 100644 --- a/src/services/db.js +++ b/src/services/db.js @@ -150,6 +150,10 @@ function migrate() { level TEXT NOT NULL, message TEXT NOT NULL, details TEXT, + source TEXT NOT NULL DEFAULT 'core', + category TEXT NOT NULL DEFAULT 'general', + event TEXT, + request_id TEXT, created_at INTEGER NOT NULL ); @@ -412,6 +416,25 @@ function migrate() { db.exec("ALTER TABLE overlays ADD COLUMN canvas_height INTEGER NOT NULL DEFAULT 1080"); } + const logColumns = db + .prepare("PRAGMA table_info(logs)") + .all() + .map((column) => column.name); + if (!logColumns.includes("source")) { + db.exec("ALTER TABLE logs ADD COLUMN source TEXT NOT NULL DEFAULT 'core'"); + } + if (!logColumns.includes("category")) { + db.exec("ALTER TABLE logs ADD COLUMN category TEXT NOT NULL DEFAULT 'general'"); + } + if (!logColumns.includes("event")) { + db.exec("ALTER TABLE logs ADD COLUMN event TEXT"); + } + if (!logColumns.includes("request_id")) { + db.exec("ALTER TABLE logs ADD COLUMN request_id TEXT"); + } + db.exec("CREATE INDEX IF NOT EXISTS logs_source_created_at_idx ON logs (source, created_at)"); + db.exec("CREATE INDEX IF NOT EXISTS logs_category_created_at_idx ON logs (category, created_at)"); + migrateLegacyUsers(); } diff --git a/src/services/discord.js b/src/services/discord.js index 1f75639..462e485 100644 --- a/src/services/discord.js +++ b/src/services/discord.js @@ -6,7 +6,10 @@ const Intents = discord.Intents; const Partials = discord.Partials; const { getSetting, setSetting } = require("./settings"); const { incrementMessages } = require("./stats"); -const { ensureUserForIdentity } = require("./users"); +const { ensureUserForIdentity } = require("./users"); +const { createLogger } = require("./logger"); + +const discordLog = createLogger("platform:discord", { category: "integration" }); let client = null; @@ -29,10 +32,10 @@ async function startBot({ commandRouter } = {}) { if (intents.length) { options.intents = intents; } - console.log("Discord bot starting with intents", { + discordLog.info("Discord bot starting", { intents, guildMembers: Boolean(resolveIntent("GuildMembers", "GUILD_MEMBERS")) - }); + }, { event: "platform_starting" }); if (Partials?.Channel) { options.partials = [Partials.Channel]; } @@ -40,7 +43,7 @@ async function startBot({ commandRouter } = {}) { client = new Client(options); client.on("ready", () => { - console.log(`Discord bot ready: ${client.user?.tag}`); + discordLog.info("Discord bot ready", { account: client.user?.tag || null }, { event: "platform_ready" }); const avatarUrl = getBotAvatarUrl(client.user); if (avatarUrl) { setSetting("bot_avatar_url", avatarUrl); @@ -86,7 +89,7 @@ async function startBot({ commandRouter } = {}) { try { await message.reply(content); } catch (error) { - console.error("Discord command reply failed", error); + discordLog.error("Discord command reply failed", error, { event: "reply_failed" }); } } }); diff --git a/src/services/logger.js b/src/services/logger.js index c5ad3a0..d80543d 100644 --- a/src/services/logger.js +++ b/src/services/logger.js @@ -1,143 +1,321 @@ +const { AsyncLocalStorage } = require("async_hooks"); const util = require("util"); const { db } = require("./db"); const LEVELS = new Set(["debug", "info", "warn", "error"]); +const DEFAULT_MAX_AGE_DAYS = 30; +const DEFAULT_MAX_ENTRIES = 100000; +const MAX_MESSAGE_LENGTH = 1000; +const MAX_DETAILS_LENGTH = 64 * 1024; +const contextStorage = new AsyncLocalStorage(); let consoleHooked = false; function log(level, ...args) { + return writeLog(level, args, contextStorage.getStore() || {}); +} + +function createLogger(source, defaults = {}) { + const base = normalizeMetadata({ ...defaults, source }); + const emit = (level, message, details, metadata = {}) => { + const overrides = compactMetadata(normalizeMetadata(metadata)); + return writeLog(level, details === undefined ? [message] : [message, details], { + ...base, + ...overrides + }); + }; + return Object.freeze({ + log: emit, + debug: (message, details, metadata) => emit("debug", message, details, metadata), + info: (message, details, metadata) => emit("info", message, details, metadata), + warn: (message, details, metadata) => emit("warn", message, details, metadata), + error: (message, details, metadata) => emit("error", message, details, metadata), + child: (childSource, childDefaults = {}) => createLogger( + [base.source, normalizeLabel(childSource, "")].filter(Boolean).join(":"), + { ...base, ...childDefaults } + ), + run: (metadata, callback) => withLogContext({ ...base, ...metadata }, callback) + }); +} + +function withLogContext(metadata, callback) { + const parent = contextStorage.getStore() || {}; + return contextStorage.run({ ...parent, ...compactMetadata(normalizeMetadata(metadata)) }, callback); +} + +function writeLog(level, args, metadata = {}) { const safeLevel = LEVELS.has(level) ? level : "info"; const entry = normalizeArgs(args); + const normalized = normalizeMetadata(metadata); const createdAt = Date.now(); try { - db.prepare( - "INSERT INTO logs (level, message, details, created_at) VALUES (?, ?, ?, ?)" - ).run(safeLevel, entry.message, entry.details, createdAt); + const result = db.prepare( + "INSERT INTO logs (level, message, details, source, category, event, request_id, created_at) " + + "VALUES (?, ?, ?, ?, ?, ?, ?, ?)" + ).run( + safeLevel, + truncate(entry.message, MAX_MESSAGE_LENGTH), + truncate(entry.details, MAX_DETAILS_LENGTH), + normalized.source || "core", + normalized.category || "general", + normalized.event || null, + normalized.requestId || null, + createdAt + ); + const stored = { + id: Number(result.lastInsertRowid), + level: safeLevel, + message: truncate(entry.message, MAX_MESSAGE_LENGTH), + details: truncate(entry.details, MAX_DETAILS_LENGTH), + source: normalized.source || "core", + category: normalized.category || "general", + event: normalized.event || null, + request_id: normalized.requestId || null, + created_at: createdAt + }; + publishLogEvent(stored); + return stored; } catch { - // Avoid throwing from logger. + return null; } } function listLogs(options = {}) { - const limit = - Number.isFinite(options.limit) && options.limit !== null - ? Math.max(1, options.limit) - : null; - const sinceMs = - Number.isFinite(options.sinceMs) && options.sinceMs > 0 - ? options.sinceMs - : null; - const levels = Array.isArray(options.levels) - ? options.levels.filter((level) => LEVELS.has(level)) - : []; + const query = buildLogQuery(options); + let sql = + "SELECT id, level, message, details, source, category, event, request_id, created_at FROM logs"; + if (query.clauses.length) sql += ` WHERE ${query.clauses.join(" AND ")}`; + sql += " ORDER BY created_at DESC, id DESC"; + if (query.limit) { + sql += " LIMIT ?"; + query.params.push(query.limit); + } + return db.prepare(sql).all(...query.params); +} +function summarizeLogs(options = {}) { + const query = buildLogQuery({ ...options, limit: null }); + let sql = "SELECT level, COUNT(*) AS count FROM logs"; + if (query.clauses.length) sql += ` WHERE ${query.clauses.join(" AND ")}`; + sql += " GROUP BY level"; + const levels = { error: 0, warn: 0, info: 0, debug: 0 }; + let total = 0; + for (const row of db.prepare(sql).all(...query.params)) { + if (levels[row.level] !== undefined) levels[row.level] = Number(row.count) || 0; + total += Number(row.count) || 0; + } + return { total, levels }; +} + +function listLogFacets(options = {}) { + const sinceMs = Number.isFinite(options.sinceMs) && options.sinceMs > 0 ? options.sinceMs : null; + const where = sinceMs ? " WHERE created_at >= ?" : ""; + const params = sinceMs ? [sinceMs] : []; + const facetRows = (column) => { + const connector = sinceMs ? " AND" : " WHERE"; + return db.prepare( + `SELECT ${column} AS value, COUNT(*) AS count FROM logs${where}${connector} ${column} IS NOT NULL AND ${column} != '' ` + + `GROUP BY ${column} ORDER BY count DESC, ${column} ASC` + ).all(...params).map((row) => ({ value: row.value, count: Number(row.count) || 0 })); + }; + return { sources: facetRows("source"), categories: facetRows("category") }; +} + +function cleanupLogs(options = {}) { + const maxAgeDays = clampNumber(options.maxAgeDays, 1, 3650, DEFAULT_MAX_AGE_DAYS); + const maxEntries = clampNumber(options.maxEntries, 1000, 1000000, DEFAULT_MAX_ENTRIES); + const cutoff = Date.now() - maxAgeDays * 24 * 60 * 60 * 1000; + const expired = db.prepare("DELETE FROM logs WHERE created_at < ?").run(cutoff).changes; + const overflow = db.prepare( + "DELETE FROM logs WHERE id NOT IN (SELECT id FROM logs ORDER BY created_at DESC, id DESC LIMIT ?)" + ).run(maxEntries).changes; + return { removed: expired + overflow, expired, overflow, maxAgeDays, maxEntries }; +} + +function hookConsole() { + if (consoleHooked) return; + consoleHooked = true; + + const original = { + debug: console.debug || console.log, + log: console.log, + info: console.info || console.log, + warn: console.warn || console.log, + error: console.error || console.log + }; + const capture = (level, method, args) => { + const context = contextStorage.getStore() || {}; + writeLog(level, args, { + source: context.source || "console", + category: context.category || "runtime", + event: context.event, + requestId: context.requestId + }); + original[method].apply(console, args); + }; + console.debug = (...args) => capture("debug", "debug", args); + console.log = (...args) => capture("info", "log", args); + console.info = (...args) => capture("info", "info", args); + console.warn = (...args) => capture("warn", "warn", args); + console.error = (...args) => capture("error", "error", args); +} + +function buildLogQuery(options = {}) { + const limit = Number.isFinite(options.limit) && options.limit !== null + ? Math.max(1, Math.floor(options.limit)) + : null; + const sinceMs = Number.isFinite(options.sinceMs) && options.sinceMs > 0 ? options.sinceMs : null; + const levels = normalizeList(options.levels, LEVELS); + const sources = normalizeList(options.sources); + const categories = normalizeList(options.categories); + const search = String(options.search || "").trim().slice(0, 200); const clauses = []; const params = []; if (sinceMs) { clauses.push("created_at >= ?"); params.push(sinceMs); } - if (levels.length) { - clauses.push(`level IN (${levels.map(() => "?").join(",")})`); - params.push(...levels); + appendInFilter(clauses, params, "level", levels); + appendInFilter(clauses, params, "source", sources); + appendInFilter(clauses, params, "category", categories); + if (search) { + const needle = `%${escapeLike(search)}%`; + clauses.push( + "(message LIKE ? ESCAPE '\\' OR details LIKE ? ESCAPE '\\' OR source LIKE ? ESCAPE '\\' " + + "OR category LIKE ? ESCAPE '\\' OR event LIKE ? ESCAPE '\\' OR request_id LIKE ? ESCAPE '\\')" + ); + params.push(needle, needle, needle, needle, needle, needle); } - - let query = - "SELECT id, level, message, details, created_at FROM logs"; - if (clauses.length) { - query += ` WHERE ${clauses.join(" AND ")}`; - } - query += " ORDER BY created_at DESC"; - if (limit) { - query += " LIMIT ?"; - params.push(limit); - } - - return db.prepare(query).all(...params); + return { clauses, params, limit }; } -function hookConsole() { - if (consoleHooked) { - return; - } - consoleHooked = true; +function appendInFilter(clauses, params, column, values) { + if (!values.length) return; + clauses.push(`${column} IN (${values.map(() => "?").join(",")})`); + params.push(...values); +} - const original = { - log: console.log, - info: console.info || console.log, - warn: console.warn || console.log, - error: console.error || console.log - }; +function normalizeList(values, allowed = null) { + const list = Array.isArray(values) ? values : values ? [values] : []; + return [...new Set(list.map((value) => normalizeLabel(value, "")).filter((value) => + value && (!allowed || allowed.has(value)) + ))]; +} - console.log = (...args) => { - log("info", ...args); - original.log.apply(console, args); - }; - console.info = (...args) => { - log("info", ...args); - original.info.apply(console, args); - }; - console.warn = (...args) => { - log("warn", ...args); - original.warn.apply(console, args); - }; - console.error = (...args) => { - log("error", ...args); - original.error.apply(console, args); +function normalizeMetadata(metadata = {}) { + return { + source: normalizeLabel(metadata.source, ""), + category: normalizeLabel(metadata.category, ""), + event: normalizeLabel(metadata.event, ""), + requestId: normalizeLabel(metadata.requestId || metadata.request_id, "") }; } +function compactMetadata(metadata) { + return Object.fromEntries(Object.entries(metadata).filter(([, value]) => value)); +} + +function normalizeLabel(value, fallback) { + const normalized = String(value || "").trim().toLowerCase().replace(/[^a-z0-9_.:-]+/g, "-").replace(/^-+|-+$/g, ""); + return normalized.slice(0, 80) || fallback; +} + function normalizeArgs(args) { - if (!args || args.length === 0) { - return { message: "Log entry", details: "" }; - } + if (!args || args.length === 0) return { message: "Log entry", details: "" }; let message = ""; const detailParts = []; - const first = args[0]; if (first instanceof Error) { - message = first.message || "Error"; - detailParts.push(first.stack || String(first)); + message = redactText(first.message || "Error"); + detailParts.push(redactText(first.stack || String(first))); } else { message = formatArg(first); } - for (const arg of args.slice(1)) { if (arg instanceof Error) { - detailParts.push(arg.stack || arg.message || String(arg)); - if (!message) { - message = arg.message || "Error"; - } + detailParts.push(redactText(arg.stack || arg.message || String(arg))); + if (!message) message = redactText(arg.message || "Error"); } else { detailParts.push(formatArg(arg)); } } - - if (!message) { - message = "Log entry"; - } - return { - message, + message: message || "Log entry", details: detailParts.filter(Boolean).join("\n") }; } function formatArg(value) { - if (typeof value === "string") { - return value; - } - if (value instanceof Error) { - return value.stack || value.message || String(value); - } - return util.inspect(value, { - depth: 4, - maxArrayLength: 50, + if (typeof value === "string") return redactText(value); + if (value instanceof Error) return redactText(value.stack || value.message || String(value)); + return redactText(util.inspect(redactValue(value), { + depth: 6, + maxArrayLength: 100, + maxStringLength: 4000, breakLength: 120 - }); + })); +} + +function redactValue(value, seen = new WeakSet()) { + if (value === null || value === undefined) return value; + if (typeof value === "string") return redactText(value); + if (typeof value !== "object") return value; + if (value instanceof Error) return redactText(value.stack || value.message || String(value)); + if (seen.has(value)) return "[Circular]"; + seen.add(value); + if (Array.isArray(value)) return value.map((item) => redactValue(item, seen)); + const output = {}; + for (const [key, item] of Object.entries(value)) { + output[key] = isSensitiveKey(key) ? "[REDACTED]" : redactValue(item, seen); + } + return output; +} + +function isSensitiveKey(key) { + return /(?:^|[_-])(authorization|cookie|password|passwd|secret|token|api[_-]?key|access[_-]?token|refresh[_-]?token|client[_-]?secret|private[_-]?key)(?:$|[_-])/i.test(String(key)); +} + +function redactText(value) { + return String(value || "") + .replace(/\b(Bearer|Basic)\s+[A-Za-z0-9._~+/=-]+/gi, "$1 [REDACTED]") + .replace(/([?&](?:token|key|secret|password|authorization)=)[^&#\s]+/gi, "$1[REDACTED]") + .replace(/\b((?:api[_-]?key|access[_-]?token|refresh[_-]?token|client[_-]?secret|password|passwd|secret|authorization)\s*[:=]\s*)[^\s,;}]+/gi, "$1[REDACTED]"); +} + +function truncate(value, maxLength) { + const text = String(value || ""); + if (text.length <= maxLength) return text; + return `${text.slice(0, maxLength)}\n[truncated]`; +} + +function escapeLike(value) { + return String(value).replace(/[\\%_]/g, (character) => `\\${character}`); +} + +function clampNumber(value, minimum, maximum, fallback) { + const parsed = Number(value); + if (!Number.isFinite(parsed)) return fallback; + return Math.min(maximum, Math.max(minimum, Math.floor(parsed))); +} + +function publishLogEvent(entry) { + try { + const { publishWebEvent } = require("./web-events"); + publishWebEvent("log:created", { ...entry, details: truncate(entry.details, 4000) }, { role: "admin" }); + } catch { + // Logging must never fail because live delivery is unavailable. + } } module.exports = { - log, + DEFAULT_MAX_AGE_DAYS, + DEFAULT_MAX_ENTRIES, + cleanupLogs, + createLogger, + hookConsole, + listLogFacets, listLogs, - hookConsole + log, + redactValue, + summarizeLogs, + withLogContext }; diff --git a/src/services/plugins.js b/src/services/plugins.js index 658690a..acad078 100644 --- a/src/services/plugins.js +++ b/src/services/plugins.js @@ -3,6 +3,9 @@ const fs = require("fs"); const { spawnSync } = require("child_process"); const { db } = require("./db"); const placeholders = require("./placeholders"); +const { createLogger } = require("./logger"); + +const pluginLifecycleLog = createLogger("core:plugins", { category: "lifecycle" }); const pluginsDir = path.join(__dirname, "..", "..", "plugins"); const cleanupHandlers = []; @@ -180,11 +183,15 @@ function setPluginEnabled(id, enabled) { Date.now(), id ); + pluginLifecycleLog.info(`Plugin ${enabled ? "enabled" : "disabled"}`, { plugin_id: id }, { + event: enabled ? "plugin_enabled" : "plugin_disabled" + }); } function removePlugin(id) { db.prepare("DELETE FROM plugins WHERE id = ?").run(id); db.prepare("DELETE FROM plugin_settings WHERE plugin_id = ?").run(id); + pluginLifecycleLog.warn("Plugin registry entry removed", { plugin_id: id }, { event: "plugin_removed" }); } function clearPluginCache(pluginPath) { @@ -223,30 +230,35 @@ function loadEnabled({ continue; } clearPluginCache(plugin.dir); + const pluginLog = createLogger(`plugin:${plugin.id}`, { category: "plugin" }); try { - const mod = require(mainPath); - if (mod && typeof mod.init === "function") { - const cleanup = mod.init({ - app, - discordClient, - twitchClient, - youtubeClient, - settings, - web, - webhooks, - db, - plugin, - commandRouter, - placeholders - }); - if (typeof cleanup === "function") { - cleanupHandlers.push({ id: plugin.id, cleanup }); - } else if (cleanup && typeof cleanup.stop === "function") { - cleanupHandlers.push({ id: plugin.id, cleanup: () => cleanup.stop() }); + pluginLog.run({ event: "plugin_load" }, () => { + const mod = require(mainPath); + if (mod && typeof mod.init === "function") { + const cleanup = mod.init({ + app, + discordClient, + twitchClient, + youtubeClient, + settings, + web, + webhooks, + db, + plugin, + commandRouter, + placeholders, + logger: pluginLog + }); + if (typeof cleanup === "function") { + cleanupHandlers.push({ id: plugin.id, cleanup }); + } else if (cleanup && typeof cleanup.stop === "function") { + cleanupHandlers.push({ id: plugin.id, cleanup: () => cleanup.stop() }); + } } - } + }); + pluginLog.info("Plugin loaded", { version: plugin.version }, { event: "plugin_loaded" }); } catch (error) { - console.error(`Plugin ${plugin.id} failed to load`, error); + pluginLog.error("Plugin failed to load", error, { event: "plugin_load_failed" }); } } } @@ -256,8 +268,10 @@ async function stopPlugins() { for (const handler of handlers) { try { await handler.cleanup(); + pluginLifecycleLog.info("Plugin stopped", { plugin_id: handler.id }, { event: "plugin_stopped" }); } catch (error) { - console.error(`Plugin ${handler.id} failed to stop`, error); + createLogger(`plugin:${handler.id}`, { category: "lifecycle" }) + .error("Plugin failed to stop", error, { event: "plugin_stop_failed" }); } } } @@ -284,6 +298,7 @@ function installFromGit(url, targetFolder) { if (result.status !== 0) { throw new Error(result.stderr || "Git clone failed."); } + pluginLifecycleLog.info("Plugin cloned from repository", { plugin_id: folderName }, { event: "plugin_installed" }); return targetPath; } @@ -295,6 +310,7 @@ function updatePluginFromGit(pluginPath) { if (result.status !== 0) { throw new Error(result.stderr || "Git pull failed."); } + pluginLifecycleLog.info("Plugin repository updated", { plugin_id: path.basename(pluginPath) }, { event: "plugin_updated" }); return result.stdout; } @@ -337,6 +353,7 @@ function createLocalPlugin({ id, name, description }) { } };\n`; fs.writeFileSync(mainPath, starter, "utf8"); + pluginLifecycleLog.info("Local plugin created", { plugin_id: safeId }, { event: "plugin_created" }); return pluginDir; } diff --git a/src/services/production-diagnostics.js b/src/services/production-diagnostics.js index f5827dc..a800273 100644 --- a/src/services/production-diagnostics.js +++ b/src/services/production-diagnostics.js @@ -4,7 +4,7 @@ const path = require("path"); const { performance } = require("perf_hooks"); const { db } = require("./db"); const { dependencyIssues } = require("./dependency-manager"); -const { listLogs, log } = require("./logger"); +const { createLogger, listLogs } = require("./logger"); const { getPlugins, scanPluginDirectories } = require("./plugins"); const { readRecoveryMarker } = require("./recovery-mode"); const { getSetting, setSetting } = require("./settings"); @@ -15,6 +15,7 @@ const packageJson = require(path.join(repoRoot, "package.json")); const TOKEN_PREFIX = "lumi_diag_"; const MAX_REQUESTS_PER_MINUTE = 20; const requestWindows = new Map(); +const diagnosticsLog = createLogger("core:diagnostics", { category: "security" }); const CHECKS = Object.freeze({ system_health: "Runtime, database, dependency, disk-space, and recovery health.", update_state: "Latest local update state, recovery marker, and snapshot summary.", @@ -39,7 +40,7 @@ function issueDiagnosticsAccessKey() { setSetting("production_diagnostics_key_prefix", `${key.slice(0, TOKEN_PREFIX.length + 8)}…`); setSetting("production_diagnostics_key_created_at", new Date().toISOString()); setSetting("production_diagnostics_enabled", true); - log("warn", "Production diagnostics access key rotated", { key_prefix: `${key.slice(0, TOKEN_PREFIX.length + 8)}…` }); + diagnosticsLog.warn("Production diagnostics access key rotated", { key_prefix: `${key.slice(0, TOKEN_PREFIX.length + 8)}…` }, { event: "access_rotated" }); return key; } @@ -49,7 +50,7 @@ function revokeDiagnosticsAccess() { setSetting("production_diagnostics_key_prefix", ""); setSetting("production_diagnostics_key_created_at", null); requestWindows.clear(); - log("warn", "Production diagnostics access revoked"); + diagnosticsLog.warn("Production diagnostics access revoked", null, { event: "access_revoked" }); } function authenticateDiagnosticsRequest(req, now = Date.now()) { @@ -86,14 +87,14 @@ function runDiagnosticCheck(checkId) { } function auditDiagnosticRequest(values = {}) { - log(values.ok === false ? "warn" : "info", "Production diagnostics request", { + diagnosticsLog.log(values.ok === false ? "warn" : "info", "Production diagnostics request", { request_id: values.request_id, check: values.check, ok: values.ok !== false, key_fingerprint: values.fingerprint || null, reason: values.reason || null, duration_ms: values.duration_ms || null - }); + }, { event: "diagnostic_request", requestId: values.request_id }); } function recentDiagnosticAudit(limit = 30) { diff --git a/src/services/settings.js b/src/services/settings.js index 80a49b8..09a93cb 100644 --- a/src/services/settings.js +++ b/src/services/settings.js @@ -69,6 +69,8 @@ function ensureDefaults() { auto_update_enabled: envBoolean("AUTO_UPDATE_ENABLED", false), auto_update_interval_minutes: envNumber("AUTO_UPDATE_INTERVAL_MINUTES", 60), production_diagnostics_enabled: false, + log_retention_days: 30, + log_retention_max_entries: 100000, git_remote: envString("GIT_REMOTE", "origin"), git_branch: envString("GIT_BRANCH", "main"), bot_avatar_url: null, diff --git a/src/services/twitch.js b/src/services/twitch.js index d623991..8878cd3 100644 --- a/src/services/twitch.js +++ b/src/services/twitch.js @@ -2,6 +2,9 @@ const tmi = require("tmi.js"); const { getSetting } = require("./settings"); const { incrementMessages } = require("./stats"); const { ensureUserForIdentity } = require("./users"); +const { createLogger } = require("./logger"); + +const twitchLog = createLogger("platform:twitch", { category: "integration" }); let client = null; @@ -32,7 +35,7 @@ async function startTwitchBot({ commandRouter } = {}) { }); client.on("connected", (address, port) => { - console.log(`Twitch bot connected to ${address}:${port}`); + twitchLog.info("Twitch bot connected", { address, port, channels }, { event: "platform_ready" }); }); client.on("message", async (channel, tags, message, self) => { @@ -66,7 +69,7 @@ async function startTwitchBot({ commandRouter } = {}) { try { await client.say(channel, content); } catch (error) { - console.error("Twitch command reply failed", error); + twitchLog.error("Twitch command reply failed", error, { event: "reply_failed" }); } } }); diff --git a/src/services/webhooks.js b/src/services/webhooks.js index 867de6f..5fac4d4 100644 --- a/src/services/webhooks.js +++ b/src/services/webhooks.js @@ -1,6 +1,8 @@ const crypto = require("crypto"); const express = require("express"); -const { log } = require("./logger"); +const { createLogger } = require("./logger"); + +const webhookLog = createLogger("core:webhooks", { category: "integration" }); function createWebhookService({ limit = "256kb" } = {}) { const endpoints = new Map(); @@ -60,14 +62,14 @@ function createWebhookService({ limit = "256kb" } = {}) { } return sendHandlerResult(res, await endpoint.handler(context)); } catch (error) { - log("error", "Webhook handler failed", { + webhookLog.error("Webhook handler failed", { pluginId: endpoint.pluginId, endpointId: endpoint.endpointId, namespace, slug, message: error?.message || String(error), stack: error?.stack || "" - }); + }, { event: "inbound_webhook_failed" }); if (!res.headersSent) { return res.status(500).json({ error: "Webhook processing failed." }); } @@ -76,11 +78,11 @@ function createWebhookService({ limit = "256kb" } = {}) { }); router.use((error, req, res, _next) => { const status = error?.type === "entity.too.large" ? 413 : 400; - log("warn", "Webhook request rejected", { + webhookLog.warn("Webhook request rejected", { path: req.path, status, message: error?.message || String(error) - }); + }, { event: "inbound_webhook_rejected" }); res.status(status).json({ error: status === 413 ? "Webhook payload is too large." : "Invalid webhook request." }); @@ -124,11 +126,11 @@ function createWebhookService({ limit = "256kb" } = {}) { endpointKeysByPlugin.set(safePluginId, new Map()); } endpointKeysByPlugin.get(safePluginId).set(safeEndpointId, key); - log("info", "Webhook endpoint registered", { + webhookLog.info("Webhook endpoint registered", { pluginId: safePluginId, endpointId: safeEndpointId, path: `/webhooks/${key}` - }); + }, { event: "webhook_registered" }); return { namespace: safeNamespace, slug: safeSlug, path: `/webhooks/${key}` }; } @@ -143,7 +145,7 @@ function createWebhookService({ limit = "256kb" } = {}) { if (!pluginEndpoints.size) { endpointKeysByPlugin.delete((pluginId || "").toString()); } - log("debug", "Webhook endpoint unregistered", { pluginId, endpointId }); + webhookLog.debug("Webhook endpoint unregistered", { pluginId, endpointId }, { event: "webhook_unregistered" }); return true; } @@ -229,12 +231,12 @@ async function sendWebhook({ }; if (response.ok || attempt === attempts) { if (!response.ok) { - log("warn", "Outbound webhook returned an error", { + webhookLog.warn("Outbound webhook returned an error", { pluginId: pluginId || null, url: redactUrl(url), status: response.status, attempt - }); + }, { event: "outbound_webhook_error" }); } return result; } @@ -242,12 +244,12 @@ async function sendWebhook({ lastError = error; lastDurationMs = Date.now() - startedAt; if (attempt === attempts) { - log("error", "Outbound webhook failed", { + webhookLog.error("Outbound webhook failed", { pluginId: pluginId || null, url: redactUrl(url), attempt, message: error?.message || String(error) - }); + }, { event: "outbound_webhook_failed" }); } } finally { clearTimeout(timer); diff --git a/src/services/youtube.js b/src/services/youtube.js index 95f0f14..e0e138c 100644 --- a/src/services/youtube.js +++ b/src/services/youtube.js @@ -1,6 +1,9 @@ const { getSetting, setSetting } = require("./settings"); const { incrementMessages } = require("./stats"); const { ensureUserForIdentity } = require("./users"); +const { createLogger } = require("./logger"); + +const youtubeLog = createLogger("platform:youtube", { category: "integration" }); let client = null; let pollTimer = null; @@ -43,7 +46,7 @@ async function startYouTubeBot({ commandRouter } = {}) { try { await hydrateBotChannel(state); } catch (error) { - console.error("YouTube bot failed to load channel details", error); + youtubeLog.error("YouTube bot failed to load channel details", error, { event: "channel_load_failed" }); } schedulePoll(state, 1000); @@ -75,7 +78,7 @@ function schedulePoll(state, delayMs) { pollTimer = setTimeout(() => { pollTimer = null; pollLiveChat(state).catch((error) => { - console.error("YouTube chat poll failed", error); + youtubeLog.error("YouTube chat poll failed", error, { event: "poll_failed" }); schedulePoll(state, 10000); }); }, delayMs); @@ -154,7 +157,7 @@ async function handleChatItem(state, liveChatId, item) { try { await sendChatMessage(state, liveChatId, content); } catch (error) { - console.error("YouTube command reply failed", error); + youtubeLog.error("YouTube command reply failed", error, { event: "reply_failed" }); } } }); diff --git a/src/web/public/app.js b/src/web/public/app.js index ada10fc..6256f09 100644 --- a/src/web/public/app.js +++ b/src/web/public/app.js @@ -1001,16 +1001,18 @@ const logList = document.querySelector("[data-log-list]"); if (logList) { - const entries = Array.from(logList.querySelectorAll("[data-log-entry]")); const searchInput = document.querySelector("[data-log-search]"); const levelSelect = document.querySelector("[data-log-level]"); + const sourceSelect = document.querySelector("[data-log-source]"); + const categorySelect = document.querySelector("[data-log-category]"); const rangeSelect = document.querySelector("[data-log-range]"); const limitSelect = document.querySelector("[data-log-limit]"); + const liveStatus = document.querySelector("[data-log-live-status] .status-indicator"); const applyLogFilters = () => { const term = (searchInput?.value || "").trim().toLowerCase(); - entries.forEach((entry) => { + Array.from(logList.querySelectorAll("[data-log-entry]")).forEach((entry) => { const haystack = (entry.dataset.search || entry.textContent || "") .toLowerCase() .trim(); @@ -1025,19 +1027,97 @@ applyLogFilters(); const reloadLogView = () => { - const url = new URL(window.location.href); - const rangeValue = rangeSelect?.value || "all"; - const levelValue = levelSelect?.value || "all"; - const limitValue = limitSelect?.value || "50"; - url.searchParams.set("range", rangeValue); - url.searchParams.set("level", levelValue); - url.searchParams.set("limit", limitValue); - window.location.assign(url.toString()); + document.querySelector("[data-log-filter-form]")?.requestSubmit(); }; levelSelect?.addEventListener("change", reloadLogView); + sourceSelect?.addEventListener("change", reloadLogView); + categorySelect?.addEventListener("change", reloadLogView); rangeSelect?.addEventListener("change", reloadLogView); limitSelect?.addEventListener("change", reloadLogView); + + const matchesLiveFilters = (entry) => { + if (levelSelect?.value !== "all" && entry.level !== levelSelect?.value) return false; + if (sourceSelect?.value !== "all" && entry.source !== sourceSelect?.value) return false; + if (categorySelect?.value !== "all" && entry.category !== categorySelect?.value) return false; + const range = rangeSelect?.value || "all"; + if (range !== "all" && Number(entry.created_at) < Date.now() - Number(range)) return false; + const term = (searchInput?.value || "").trim().toLowerCase(); + const haystack = [entry.message, entry.details, entry.source, entry.category, entry.event, entry.request_id] + .filter(Boolean).join(" ").toLowerCase(); + return !term || haystack.includes(term); + }; + + const createLiveLogEntry = (entry) => { + const details = document.createElement("details"); + details.className = `log-entry level-${entry.level}`; + details.dataset.logEntry = ""; + details.dataset.level = entry.level; + details.dataset.source = entry.source || "core"; + details.dataset.category = entry.category || "general"; + details.dataset.timestamp = String(entry.created_at || Date.now()); + details.dataset.search = [entry.message, entry.details, entry.source, entry.category, entry.event, entry.request_id] + .filter(Boolean).join(" ").toLowerCase(); + const summary = document.createElement("summary"); + const marker = document.createElement("span"); + marker.className = "log-marker"; + marker.setAttribute("aria-hidden", "true"); + const message = document.createElement("span"); + message.className = "log-message"; + message.textContent = entry.message || "Log entry"; + const scope = document.createElement("span"); + scope.className = "log-scope-pill"; + scope.textContent = entry.source || "core"; + const level = document.createElement("span"); + level.className = "log-level-pill"; + level.textContent = entry.level || "info"; + const time = document.createElement("span"); + time.className = "log-time"; + time.textContent = new Date(entry.created_at || Date.now()).toLocaleString(); + summary.append(marker, message, scope, level, time); + const meta = document.createElement("div"); + meta.className = "log-entry-meta"; + const values = [ + ["Activity", entry.category || "general"], + ["Event", entry.event], + ["Request ID", entry.request_id] + ]; + values.forEach(([label, value]) => { + if (!value) return; + const item = document.createElement("span"); + const strong = document.createElement("strong"); + strong.textContent = `${label}:`; + item.append(strong, ` ${value}`); + meta.append(item); + }); + const body = entry.details ? document.createElement("pre") : document.createElement("div"); + body.className = entry.details ? "log-details" : "log-details empty"; + body.textContent = entry.details || "No additional details."; + details.append(summary, meta, body); + return details; + }; + + window.addEventListener("lumi:log-created", (event) => { + const entry = event.detail || {}; + if (!entry.id || !matchesLiveFilters(entry)) return; + logList.querySelector("[data-log-empty]")?.remove(); + logList.prepend(createLiveLogEntry(entry)); + const limit = Number(logList.dataset.logLimitValue || 50); + const entries = Array.from(logList.querySelectorAll("[data-log-entry]")); + entries.slice(limit).forEach((item) => item.remove()); + const total = document.querySelector("[data-log-total]"); + if (total) total.textContent = String(Number(total.textContent || 0) + 1); + const levelCount = document.querySelector(`[data-log-count="${entry.level}"]`); + if (levelCount) levelCount.textContent = String(Number(levelCount.textContent || 0) + 1); + if (liveStatus) liveStatus.textContent = "Live · newest entry added"; + }); + window.addEventListener("lumi:event-status", (event) => { + if (!liveStatus) return; + const connected = event.detail?.status === "connected"; + liveStatus.textContent = connected ? "Live updates connected" : "Live updates reconnecting"; + liveStatus.classList.toggle("status-success", connected); + liveStatus.classList.toggle("status-warning", !connected); + }); } const logModal = document.querySelector("[data-log-modal]"); diff --git a/src/web/public/lumi-interactions.js b/src/web/public/lumi-interactions.js index 860e304..1e337e1 100644 --- a/src/web/public/lumi-interactions.js +++ b/src/web/public/lumi-interactions.js @@ -188,12 +188,19 @@ stream.addEventListener("server:warning", (event) => showEventNotice(readEvent(event), "warning")); stream.addEventListener("server:status", (event) => { const data = readEvent(event); - if (data.status === "connected") document.body.dataset.eventStream = "connected"; + if (data.status === "connected") { + document.body.dataset.eventStream = "connected"; + window.dispatchEvent(new CustomEvent("lumi:event-status", { detail: { status: "connected" } })); + } }); stream.addEventListener("ai:model_status", (event) => showEventNotice(readEvent(event), "danger")); stream.addEventListener("data:new_available", (event) => showRefreshPrompt(readEvent(event))); + stream.addEventListener("log:created", (event) => { + window.dispatchEvent(new CustomEvent("lumi:log-created", { detail: readEvent(event) })); + }); stream.onerror = () => { document.body.dataset.eventStream = "disconnected"; + window.dispatchEvent(new CustomEvent("lumi:event-status", { detail: { status: "disconnected" } })); }; } diff --git a/src/web/public/styles.css b/src/web/public/styles.css index 2ccaa10..fd195aa 100644 --- a/src/web/public/styles.css +++ b/src/web/public/styles.css @@ -1039,6 +1039,46 @@ body { border: 1px solid var(--border); } +.log-scope-pill { + max-width: 15rem; + padding: 4px 8px; + border: 1px solid color-mix(in srgb, var(--sea) 45%, var(--border)); + border-radius: 999px; + background: color-mix(in srgb, var(--sea) 12%, var(--surface-3)); + color: var(--ink-soft); + font-size: 0.75rem; + font-weight: 700; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.log-entry-meta { + display: flex; + flex-wrap: wrap; + gap: 8px 18px; + margin: 10px 2px 0; + color: var(--ink-soft); + font-size: 0.82rem; +} + +.log-live-status { + display: flex; + align-items: center; + justify-content: space-between; + gap: 12px; + min-height: 2.75rem; + margin-top: 14px; +} + +.log-summary-grid { + margin: 18px 0; +} + +.log-storage-settings { + margin-top: 16px; +} + .log-time { font-size: 0.85rem; color: var(--ink-soft); @@ -1063,6 +1103,22 @@ body { color: var(--ink-soft); } +@media (max-width: 760px) { + .log-entry summary { + align-items: flex-start; + flex-wrap: wrap; + } + + .log-message { + flex-basis: calc(100% - 22px); + } + + .log-time { + width: 100%; + padding-left: 20px; + } +} + .identity-list { list-style: none; padding: 0; diff --git a/src/web/server.js b/src/web/server.js index 86162a1..44f0cf7 100644 --- a/src/web/server.js +++ b/src/web/server.js @@ -51,7 +51,17 @@ const { getLeaderboardSections, getTopCommandOptions } = require("../services/top"); -const { log, listLogs } = require("../services/logger"); +const { + DEFAULT_MAX_AGE_DAYS, + DEFAULT_MAX_ENTRIES, + cleanupLogs, + createLogger, + listLogFacets, + listLogs, + log, + summarizeLogs, + withLogContext +} = require("../services/logger"); const { createWebhookService } = require("../services/webhooks"); const { getPlatformStatus, @@ -1344,6 +1354,7 @@ const DASHBOARD_SCOPES = { "7d": 7 * 24 * 60 * 60 * 1000 }; const memorySamples = []; +const webLog = createLogger("core:web", { category: "http" }); function normalizeLogLevel(value) { const normalized = (value || "").toString().trim().toLowerCase(); @@ -1358,6 +1369,14 @@ function parseLogLevels(value) { return raw.map(normalizeLogLevel).filter(Boolean); } +function normalizeLogFacet(value) { + return String(value || "").trim().toLowerCase().replace(/[^a-z0-9_.:-]+/g, "-").slice(0, 80); +} + +function parseLogSearch(value) { + return String(value || "").trim().slice(0, 200); +} + function parseLogRange(value) { if (value === undefined || value === null || value === "") { return { rangeMs: DEFAULT_LOG_RANGE_MS, rangeValue: `${DEFAULT_LOG_RANGE_MS}` }; @@ -2977,6 +2996,44 @@ function createWebServer({ loadPlugins, discordClient }) { ); app.use(express.urlencoded({ extended: false })); app.use(express.json({ limit: "1mb" })); + app.use((req, res, next) => { + const requestId = crypto.randomUUID(); + const startedAt = Date.now(); + const method = String(req.method || "GET").toUpperCase(); + const requestPath = String(req.path || "/"); + const pluginMatch = requestPath.match(/^\/plugins\/([^/]+)/); + const source = pluginMatch ? `plugin:${normalizeLogFacet(pluginMatch[1])}` : "core:web"; + res.setHeader("X-Request-Id", requestId); + return withLogContext({ source, category: "http", requestId }, () => { + res.on("finish", () => { + if (["/api/events", "/admin/updates/events"].includes(requestPath)) return; + const durationMs = Date.now() - startedAt; + const status = Number(res.statusCode) || 0; + const mutating = !["GET", "HEAD", "OPTIONS"].includes(method); + const slow = durationMs >= 2000; + if (!mutating && status < 400 && !slow) return; + const level = status >= 500 ? "error" : status >= 400 || slow ? "warn" : "info"; + const category = mutating && status < 400 ? "audit" : "http"; + const routePath = typeof req.route?.path === "string" + ? `${req.baseUrl || ""}${req.route.path}` + : requestPath.replace(/\b[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\b/gi, ":id"); + webLog.log(level, "HTTP request completed", { + method, + path: routePath, + status, + duration_ms: durationMs, + user_id: req.session?.user?.id || null, + role: req.session?.user?.isAdmin ? "admin" : req.session?.user?.isMod ? "mod" : req.session?.user ? "user" : "anonymous" + }, { + source, + category, + event: status >= 400 ? "http_error" : slow ? "slow_request" : "admin_action", + requestId + }); + }); + next(); + }); + }); app.use((_req, res, next) => { res.setHeader("X-Content-Type-Options", "nosniff"); next(); @@ -5833,35 +5890,90 @@ function createWebServer({ loadPlugins, discordClient }) { const range = parseLogRange(req.query.range); const limit = parseLogLimit(req.query.limit); const levelValue = normalizeLogLevel(req.query.level) || "all"; + const sourceValue = normalizeLogFacet(req.query.source) || "all"; + const categoryValue = normalizeLogFacet(req.query.category) || "all"; + const search = parseLogSearch(req.query.q); const levels = levelValue === "all" ? [] : [levelValue]; const sinceMs = range.rangeMs ? Date.now() - range.rangeMs : null; - const logs = listLogs({ limit: limit.limit, sinceMs, levels }); + const query = { + limit: limit.limit, + sinceMs, + levels, + sources: sourceValue === "all" ? [] : [sourceValue], + categories: categoryValue === "all" ? [] : [categoryValue], + search + }; + const logs = listLogs(query); res.render("admin-logs", { title: "Logs", logs, + logSummary: summarizeLogs(query), + logFacets: listLogFacets({ sinceMs }), + logRetention: { + maxAgeDays: getSetting("log_retention_days", DEFAULT_MAX_AGE_DAYS), + maxEntries: getSetting("log_retention_max_entries", DEFAULT_MAX_ENTRIES) + }, logFilters: { range: range.rangeValue, level: levelValue, - limit: limit.limitValue + limit: limit.limitValue, + source: sourceValue, + category: categoryValue, + search } }); }); + app.post("/admin/logs/retention", requireRole("admin"), (req, res) => { + const cleanup = cleanupLogs({ + maxAgeDays: req.body.max_age_days, + maxEntries: req.body.max_entries + }); + setSetting("log_retention_days", cleanup.maxAgeDays); + setSetting("log_retention_max_entries", cleanup.maxEntries); + webLog.info("Log retention updated", { + max_age_days: cleanup.maxAgeDays, + max_entries: cleanup.maxEntries, + removed: cleanup.removed, + user_id: req.session.user.id + }, { category: "audit", event: "log_retention_updated" }); + setFlash(req, "success", cleanup.removed + ? `Log storage updated and ${cleanup.removed} old entr${cleanup.removed === 1 ? "y was" : "ies were"} removed.` + : "Log storage settings updated."); + res.redirect("/admin/logs"); + }); + app.get("/admin/logs/download", requireRole("admin"), (req, res) => { const range = parseLogRange(req.query.range); const limit = parseLogLimit(req.query.limit, { allowAll: true }); const levels = parseLogLevels(req.query.level); + const source = normalizeLogFacet(req.query.source); + const category = normalizeLogFacet(req.query.category); + const search = parseLogSearch(req.query.q); const sinceMs = range.rangeMs ? Date.now() - range.rangeMs : null; - const logs = listLogs({ limit: limit.limit, sinceMs, levels }); + const logs = listLogs({ + limit: limit.limit, + sinceMs, + levels, + sources: source && source !== "all" ? [source] : [], + categories: category && category !== "all" ? [category] : [], + search + }); const stamp = new Date().toISOString().replace(/[:.]/g, "-"); + const format = req.query.format === "jsonl" ? "jsonl" : "txt"; res.setHeader( "Content-Disposition", - `attachment; filename="lumi-logs-${stamp}.txt"` + `attachment; filename="lumi-logs-${stamp}.${format}"` ); - res.setHeader("Content-Type", "text/plain; charset=utf-8"); + res.setHeader("Content-Type", format === "jsonl" ? "application/x-ndjson; charset=utf-8" : "text/plain; charset=utf-8"); + if (format === "jsonl") { + return res.send(logs.map((entry) => JSON.stringify(entry)).join("\n") + (logs.length ? "\n" : "")); + } const lines = logs.map((log) => { const timestamp = new Date(log.created_at).toISOString(); - const header = `${timestamp} [${log.level.toUpperCase()}] ${log.message}`; + const scope = [log.source, log.category, log.event].filter(Boolean).join(" / "); + const request = log.request_id ? ` request=${log.request_id}` : ""; + const header = `${timestamp} [${log.level.toUpperCase()}] [${scope}]${request} ${log.message}`; if (log.details) { return `${header}\n${log.details}\n`; } diff --git a/src/web/views/admin-logs.ejs b/src/web/views/admin-logs.ejs index 52e90e5..a78b9f5 100644 --- a/src/web/views/admin-logs.ejs +++ b/src/web/views/admin-logs.ejs @@ -1,35 +1,75 @@ <%- include("partials/layout-top", { title }) %> -<% const filters = logFilters || { range: '86400000', level: 'all', limit: '50' }; %> +<% + const filters = logFilters || { range: '86400000', level: 'all', limit: '50', source: 'all', category: 'all', search: '' }; + const summary = logSummary || { total: 0, levels: { error: 0, warn: 0, info: 0, debug: 0 } }; + const facets = logFacets || { sources: [], categories: [] }; + const retention = logRetention || { maxAgeDays: 30, maxEntries: 100000 }; +%>
-
-
-

Logs

-

Core system logs with severity, timestamps, and details.

-
-
- -
+