diff --git a/README.md b/README.md index f62f964..3ae59e6 100644 --- a/README.md +++ b/README.md @@ -147,11 +147,46 @@ Unrepresentable characters are rejected rather than accepted as replacement `?` bytes. Configure source UTF-8 and the target execution code page separately; this does not permit new strings absent from the original executable. +### Repository synchronization before verification + +When `server.json` sets `pushRemote`, the single worker fetches that remote's +branch matching its checked-out branch at startup and before admitting each +code/maintenance verification cohort. Existing verification, maintenance, merge +and remote-check jobs drain first; new jobs do not start during synchronization. +No remote is guessed when `pushRemote` is unset. + +Only a fast-forward is automatic. Local-ahead commits are retained. Divergence +reports both full commit IDs and requires a maintainer to reconcile the histories; +there is no automatic reset, rebase or merge commit. Dirty tracked, staged and +untracked files block synchronization, and ignored files cannot be overwritten by +incoming tracked files. Preserve local edits before retrying. + +Fetch/authentication failures and unsafe states leave batches queued with an +explicit `result.repositorySync.error`; merge and remote-check admission pauses. +The worker retries at its next pump and removes the diagnostic after a successful +sync, so operator repairs do not require resubmission or a restart. Fetch is +noninteractive and bounded to two minutes. + +After synchronization the worker reloads `data/decomp.json`, its selected +compile-command file and `data/toolchain.lock.json` from one committed snapshot, +updates the shared project object and invalidates checks cached under old settings. +The separate web process adopts only the published snapshot, never partially +updated working-tree configuration. Direct uncommitted configuration changes are +not loaded this way; commit them, or restart web and worker when no remote is used. + + ## Building td `node tools/build-td/build.mjs` (Node 22.21) writes `dist/td-windows-x64.exe`, `dist/td-linux-x64` and `SHA256SUMS`; copy them to `/downloads` on the server. +These download names are mutable: the server sends `Cache-Control: no-store` on `/download/` +responses, including the manifest and compressed binaries. Self-updates revalidate the manifest, +request the binary with `?sha256=` to bypass an older unversioned CDN entry, and +still verify its SHA-256 before replacing the installed executable. When deploying this policy +behind a CDN, purge existing `/download/` cache entries once: new origin headers do not evict +already-cached responses used by older clients. + ## Tests `npm test` uses synthetic PE/COFF files, so it doesn't need a compiler. @@ -160,7 +195,8 @@ does not permit new strings absent from the original executable. Set `"globalRules": true` in the target repository's `decomp.json` to enable the source-level global gate. It is opt-in; deploying this server does not enable it -for other projects. Restart web and worker after changing project configuration. +for other projects. Committed configuration is refreshed by repository synchronization +when `pushRemote` is configured; otherwise restart web and worker after changing it. The verifier applies the payload over current main before checking it, and the merger checks again before committing. Both ordinary and maintenance batches use @@ -195,6 +231,38 @@ build. Behavior-changing function work must use the normal task/review path. `td maintenance symbol --size N --reason "evidence"` corrects metadata; `--retire` is allowed only after main no longer annotates that global ID. +From a contributor workspace, a maintainer can correct a decorated/linker name +without changing a symbol's ID, address, task state, match state or confirmations: + +```sh +td maintenance symbol fn_00123 --decorated '?CorrectName@@YAXXZ' --reason "original-binary evidence" +td maintenance symbol fn_00123 --swap-decorated-with fn_00456 --reason "call targets show the names were reversed" +``` + +The maintained HTTP route is `POST /api/admin/symbols/:id`. Send either +`{"decorated":"?CorrectName@@YAXXZ","reason":"original-binary evidence"}` or +`{"swapDecoratedWith":"fn_00456","reason":"call-target evidence"}`. MCP exposes +the same fields through `td_correct_symbol`, with an additional `id` argument. +Both clients refresh their symbol snapshots after a successful correction. + +`decorated` must be a string of 1-2048 visible ASCII characters, with no whitespace +or control characters; it is never coerced, trimmed or truncated. C-linkage names +are supported as well as C++ decorated names. Null and empty values are rejected. +The evidence `reason` must be a nonblank string of at most 500 characters. +Unknown request fields are rejected. Existing `size`, `name` and `retire` fields +remain available; a swap cannot be combined with other corrections. + +A name already mapped to another ID returns HTTP 409 without applying any part +of the request. A swap requires two distinct, existing, unambiguous mappings +whose symbols have the same kind; it exchanges only their decorated names in +one database transaction. Do not +clear a mapping or use a temporary name as an intermediate step. Each affected +ID receives a `symbol_corrected` event containing the acting user, `old` and +`new` values, and the evidence reason (plus the other ID for a swap). Audit +records and corrections commit together. Symbol snapshot caches in every web +process and target version are invalidated by the committed correction, so an +immediate conditional refresh observes the new names rather than a stale 304. + Neither an unchanged-code fingerprint nor a successful unit compile proves that new global storage reproduces the original initializer/destructor behavior. Maintenance authors must preserve those semantics; an incomplete object layout diff --git a/deploy/rodecomp/compose.yaml b/deploy/rodecomp/compose.yaml index 25657ed..01b778f 100644 --- a/deploy/rodecomp/compose.yaml +++ b/deploy/rodecomp/compose.yaml @@ -42,7 +42,10 @@ services: USER_GID: "1000" FORGEJO__server__ROOT_URL: "https://clank.rodecomp.online/git/" FORGEJO__server__DOMAIN: "clank.rodecomp.online" - FORGEJO__server__DISABLE_SSH: "true" + FORGEJO__server__DISABLE_SSH: "false" + FORGEJO__server__SSH_PORT: "22" + FORGEJO__server__SSH_LISTEN_PORT: "22" + FORGEJO__server__SSH_DOMAIN: "192.168.1.77" FORGEJO__server__OFFLINE_MODE: "true" FORGEJO__server__LANDING_PAGE: "explore" FORGEJO__database__DB_TYPE: "sqlite3" @@ -56,6 +59,8 @@ services: FORGEJO__actions__ENABLED: "false" FORGEJO__packages__ENABLED: "false" FORGEJO__repository__DISABLED_REPO_UNITS: "repo.pulls,repo.actions,repo.packages,repo.releases,repo.projects" + ports: + - "22:22" volumes: - forgejo:/data restart: unless-stopped diff --git a/src/client/cli.js b/src/client/cli.js index 19db240..c5c279b 100644 --- a/src/client/cli.js +++ b/src/client/cli.js @@ -51,6 +51,7 @@ Maintenance (maintainers: structural changes with no function tasks) td maintenance submit --files a.cpp,b.h --notes T verified like a batch; merges only when no function's compiled code changes td maintenance symbol [--size N] [--name N] [--retire] --reason T correct the symbol database + [--decorated N | --swap-decorated-with ID] correct a decorated name or atomically swap two Peer review (attempts that are not byte-identical; judge by reading, not by tools) td peer list attempts waiting for review @@ -120,6 +121,8 @@ async function main() { files: { type: 'string' }, reason: { type: 'string' }, size: { type: 'string' }, + decorated: { type: 'string' }, + 'swap-decorated-with': { type: 'string' }, retire: { type: 'boolean', default: false }, json: { type: 'boolean', default: false }, free: { type: 'boolean', default: false }, @@ -218,11 +221,16 @@ connect your AI with: td connect --workspace "${ws.dir}"`); return print(await ops.submitMaintenance(ws, files, { notes: values.notes })); } if (sub === 'symbol') { - if (!id || !values.reason) throw new Error('usage: td maintenance symbol [--size N] [--name N] [--retire] --reason "evidence"'); - const body = { reason: values.reason, ...(values.size && { size: Number(values.size) }), ...(values.name && { name: values.name }), ...(values.retire && { retire: true }) }; - const r = await client.post(`/api/admin/symbols/${id}`, body); - await ops.refreshSymbols(ws); - return print(r); + if (!id || !values.reason) throw new Error('usage: td maintenance symbol [--size N] [--name N] [--retire] [--decorated N | --swap-decorated-with ID] --reason "evidence"'); + const body = { + reason: values.reason, + ...(values.size !== undefined && { size: Number(values.size) }), + ...(values.name !== undefined && { name: values.name }), + ...(values.retire && { retire: true }), + ...(values.decorated !== undefined && { decorated: values.decorated }), + ...(values['swap-decorated-with'] !== undefined && { swapDecoratedWith: values['swap-decorated-with'] }), + }; + return print(await ops.correctSymbol(ws, id, body)); } throw new Error('usage: td maintenance submit|symbol ...'); } diff --git a/src/client/mcp.js b/src/client/mcp.js index 8dd51b5..97fd1e1 100644 --- a/src/client/mcp.js +++ b/src/client/mcp.js @@ -105,6 +105,25 @@ const TOOLS = [ return r; }, }, + { + name: 'td_correct_symbol', + description: 'Maintainers only: correct symbol metadata from original-binary evidence, recording old/new values and the reason. Set decorated to replace one mapping, or swapDecoratedWith to atomically exchange two existing names of symbols with the same kind (no other correction fields with a swap). IDs, addresses, function state and confirmations remain unchanged for name corrections. Refreshes the workspace symbol snapshot.', + inputSchema: { + type: 'object', + additionalProperties: false, + properties: { + id: str('stable symbol ID to correct'), + reason: { type: 'string', minLength: 1, maxLength: 500, description: 'original-binary evidence for this correction' }, + size: { type: 'integer', minimum: 1, description: 'correct size in the current target version' }, + name: { type: ['string', 'null'], maxLength: 200, description: 'human-readable name; null clears it' }, + decorated: { type: 'string', minLength: 1, maxLength: 2048, pattern: '^[!-~]+$', description: 'exact decorated name; visible ASCII only, no whitespace or truncation' }, + swapDecoratedWith: { type: 'string', pattern: '^(fn|gl|vt|st)_[0-9]+$', description: 'other stable ID of the same symbol kind whose decorated name is exchanged with this one' }, + retire: { type: 'boolean', description: 'retire a global only after main no longer annotates it' }, + }, + required: ['id', 'reason'], + }, + run: async (ws, { id, ...correction }) => ops.correctSymbol(ws, id, correction), + }, { name: 'td_check', description: diff --git a/src/client/ops.js b/src/client/ops.js index 4447de0..aaae108 100644 --- a/src/client/ops.js +++ b/src/client/ops.js @@ -319,6 +319,13 @@ export async function submitMaintenance(ws, files, { notes } = {}) { return api(ws).post('/api/maintenance', { baseCommit: ws.head(), files: contents, notes }); } +/** Correct symbol metadata and immediately replace the workspace's cached name mappings. */ +export async function correctSymbol(ws, id, correction) { + const result = await api(ws).post(`/api/admin/symbols/${encodeURIComponent(id)}`, correction); + await refreshSymbols(ws); + return result; +} + /** Annotated functions whose definition differs from HEAD in the changed (or named) files under allowed paths. */ function rewrittenFunctions(ws, allowed, files) { const out = new Set(); diff --git a/src/client/update.js b/src/client/update.js index 1b4b93a..a20e78e 100644 --- a/src/client/update.js +++ b/src/client/update.js @@ -82,7 +82,7 @@ export async function maybeUpdate({ server, force = false, log = (m) => process. if (!force && state.checkedAt && Date.now() - state.checkedAt < CHECK_EVERY_MS) return null; state.checkedAt = Date.now(); - const res = await fetch(`${base}/download/SHA256SUMS`, { signal: AbortSignal.timeout(10_000) }); + const res = await fetch(`${base}/download/SHA256SUMS`, { cache: 'no-store', signal: AbortSignal.timeout(10_000) }); if (!res.ok) return writeState(state), null; const want = new RegExp(`^([0-9a-f]{64})\\s+${ASSET.replace('.', '\\.')}$`, 'm').exec(await res.text())?.[1]; if (!want || want === (await ownHash(exe, state))) return writeState(state), null; @@ -91,7 +91,8 @@ export async function maybeUpdate({ server, force = false, log = (m) => process. // Slow connections take as long as they need; only a download that stalls is given up. const stall = new AbortController(); let timer = setTimeout(() => stall.abort(), STALL_MS); - const bin = await fetch(`${base}/download/${ASSET}`, { signal: stall.signal }); + // A new manifest must not reuse a CDN's cached copy of the previous build. + const bin = await fetch(`${base}/download/${ASSET}?sha256=${want}`, { signal: stall.signal }); if (!bin.ok) return clearTimeout(timer), writeState(state), null; const parts = []; for await (const chunk of bin.body) { diff --git a/src/server/app.js b/src/server/app.js index eac3731..b51bf63 100644 --- a/src/server/app.js +++ b/src/server/app.js @@ -67,8 +67,12 @@ export function openServer(dataDir, { log = console } = {}) { const config = loadConfig(dataDir); const store = new Store(path.join(config.dataDir, 'td.sqlite')); const repo = new Repo(path.join(config.dataDir, 'repo')); - const project = loadProject(repo.dir); - config.envDigest = loadToolchainLock(repo.dir)[project.toolchain]?.clSha256 ?? null; + // Web and worker share Git, not JS objects. Never read half of a working-tree FF. + const projectHead = config.pushRemote ? store.getMeta('repository_project_head') ?? repo.head() : null; + const readProjectFile = projectHead ? (file) => repo.showMany(projectHead, [file]).get(file) : undefined; + const project = loadProject(repo.dir, readProjectFile); + config.toolchainLock = loadToolchainLock(repo.dir, readProjectFile); + config.envDigest = config.toolchainLock[project.toolchain]?.clSha256 ?? null; let peCache = null; const getPe = () => { @@ -94,6 +98,7 @@ export function openServer(dataDir, { log = console } = {}) { const peers = new PeerReviews({ store, project, trust }); const verifier = new Verifier({ store, config, repo, project, tasks, getPe, peers, trust, log }); verifier.regions = regions; + verifier.projectHead = projectHead; const merger = new Merger({ store, config, repo, project, tasks, verifier, getPe, peers, log }); const accounts = new Accounts(store); const originals = new Originals({ store }); @@ -102,6 +107,8 @@ export function openServer(dataDir, { log = console } = {}) { const maintenance = new Maintenance({ store, repo, project, verifier, merger, log }); verifier.maintenance = maintenance; const checks = new CheckQueue({ store, repo, verifier, config, log }); + verifier.merger = merger; + verifier.checks = checks; const credits = new Credits({ store }); return { config, store, repo, project, tasks, regions, quality, trust, credits, verifier, merger, maintenance, getPe, accounts, peers, originals, ghidra, checks }; } @@ -176,7 +183,7 @@ export function createApp(ctx) { gitWeb: config.gitWebUrl ?? null, version: v ? { id: v.id, label: v.label ?? v.id, sha256: v.sha256, size: v.size, download: config.distributeTarget ? `${config.publicUrl}/api/target/${encodeURIComponent(v.id)}` : null } : null, toolchain: project.toolchain, - toolchainLock: loadToolchainLock(repo.dir), + toolchainLock: config.pushRemote ? config.toolchainLock : loadToolchainLock(repo.dir), limits: { ...DEFAULT_LIMITS, ...project.lintLimits }, matchPolicy: project.matchPolicy, }; @@ -200,12 +207,12 @@ export function createApp(ctx) { router.on('GET', '/api/symbols', ({ req, res, query }) => { const v = query.get('version') ? store.get('SELECT * FROM versions WHERE id = ?', query.get('version')) : store.currentVersion(); if (!v) throw new HttpError(404, 'unknown version'); - // Building it takes a while; a minute-old table is fine for agents. + // A correction revision invalidates cached snapshots in every web process and version. const { body, etag } = cached(`symbols:${v.id}`, 60_000, () => { const rows = [...store.symbolMap(v.id).values()].map((s) => [s.id, s.kind, s.addr, s.size, s.name, s.decorated, s.library ? 1 : 0, s.unit]); const json = JSON.stringify({ version: v.id, columns: ['id', 'kind', 'addr', 'size', 'name', 'decorated', 'library', 'unit'], symbols: rows }); return { body: json, etag: `"${crypto.createHash('sha1').update(json).digest('hex')}"` }; - }); + }, store.getMeta('symbolCorrectionsRevision')); const headers = { etag, 'cache-control': 'no-cache', 'x-content-type-options': 'nosniff' }; if (req.headers['if-none-match'] === etag) { res.writeHead(304, headers); @@ -220,11 +227,11 @@ export function createApp(ctx) { const ACTIVE_MS = 15 * 60_000; // Public numbers every open page polls: computed at most once per `ttl` and shared by everyone. const shared = new Map(); - const cached = (key, ttl, compute) => { + const cached = (key, ttl, compute, revision = null) => { const hit = shared.get(key); - if (hit && Date.now() - hit.at < ttl) return hit.value; + if (hit && hit.revision === revision && Date.now() - hit.at < ttl) return hit.value; const value = compute(); - shared.set(key, { at: Date.now(), value }); + shared.set(key, { at: Date.now(), value, revision }); return value; }; @@ -859,32 +866,71 @@ export function createApp(ctx) { return { id: r.symbol.id, kind: r.symbol.kind, created: r.created, annotation: `// ${{ function: 'FUNCTION', global: 'GLOBAL', vtable: 'VTABLE', string: 'STRING' }[kind]}: ${r.symbol.id}` }; }); - // Corrections to the symbol database a maintainer makes from evidence in the original: a size, - // a name, or a global that is not one (an address inside another object, registered by mistake). + // Evidence-backed metadata corrections never change stable IDs or function match/review state. + // Decorated names can be replaced or exchanged atomically when two mappings were reversed. // Retiring removes its address in the current version, so nothing resolves to it any more; its // ID stays reserved. Only symbols no file on main annotates may be retired: a maintenance batch // removes those annotations first. router.on('POST', '/api/admin/symbols/:id', async ({ req, params }) => { const user = auth(req, { role: 'maintainer' }); const body = await readJson(req, 4096); - const sym = store.get('SELECT * FROM symbols WHERE id = ?', String(params.id)); - if (!sym) throw new HttpError(404, 'no such symbol'); - const v = store.currentVersion(); - const reason = String(body.reason ?? '').trim().slice(0, 500); - if (!reason) throw new HttpError(400, 'reason is required: the evidence for the correction'); - const changes = {}; - store.tx(() => { + if (!body || typeof body !== 'object' || Array.isArray(body)) throw new HttpError(400, 'body must be an object'); + const allowed = ['reason', 'size', 'name', 'retire', 'decorated', 'swapDecoratedWith']; + if (Object.keys(body).some((key) => !allowed.includes(key))) throw new HttpError(400, 'unknown symbol correction field'); + if (typeof body.reason !== 'string' || !body.reason.trim() || body.reason.length > 500) throw new HttpError(400, 'reason must be 1-500 characters: the evidence for the correction'); + const reason = body.reason.trim(); + const validDecorated = (value) => typeof value === 'string' && value.length > 0 && value.length <= 2048 && !/[^\x21-\x7e]/.test(value); + if (body.decorated !== undefined && !validDecorated(body.decorated)) throw new HttpError(400, 'decorated must be 1-2048 visible ASCII characters without whitespace'); + if (body.retire !== undefined && typeof body.retire !== 'boolean') throw new HttpError(400, 'retire must be a boolean'); + if (body.swapDecoratedWith !== undefined) { + if (typeof body.swapDecoratedWith !== 'string' || !/^(fn|gl|vt|st)_[0-9]+$/.test(body.swapDecoratedWith) || body.swapDecoratedWith === params.id) throw new HttpError(400, 'swapDecoratedWith must be a different symbol ID'); + if (['size', 'name', 'retire', 'decorated'].some((key) => body[key] !== undefined)) throw new HttpError(400, 'swapDecoratedWith cannot be combined with other corrections'); + } + const result = store.tx(() => { + const sym = store.get('SELECT * FROM symbols WHERE id = ?', params.id); + if (!sym) throw new HttpError(404, 'no such symbol'); + const assertUnique = (decorated, ownerId) => { + const duplicate = store.get('SELECT id FROM symbols WHERE decorated = ? AND id != ? LIMIT 1', decorated, ownerId); + if (duplicate) throw new HttpError(409, `decorated name is already mapped to ${duplicate.id}; use swapDecoratedWith to exchange two mappings`); + }; + if (body.swapDecoratedWith !== undefined) { + const other = store.get('SELECT * FROM symbols WHERE id = ?', body.swapDecoratedWith); + if (!other) throw new HttpError(404, 'no such swap symbol'); + if (sym.kind !== other.kind) throw new HttpError(409, 'a swap requires symbols of the same kind'); + if (!validDecorated(sym.decorated) || !validDecorated(other.decorated) || sym.decorated === other.decorated) throw new HttpError(409, 'a swap requires two distinct existing decorated names'); + assertUnique(sym.decorated, sym.id); + assertUnique(other.decorated, other.id); + store.run('UPDATE symbols SET decorated = CASE id WHEN ? THEN ? ELSE ? END WHERE id IN (?, ?)', sym.id, other.decorated, sym.decorated, sym.id, other.id); + for (const [from, to] of [[sym, other], [other, sym]]) { + store.event('symbol_corrected', user.id, { id: from.id, decorated: to.decorated, old: { decorated: from.decorated }, new: { decorated: to.decorated }, swapDecoratedWith: to.id, reason }); + } + store.setMeta('symbolCorrectionsRevision', Number(store.getMeta('symbolCorrectionsRevision') ?? 0) + 1); + return { id: sym.id, decorated: other.decorated, swapped: { id: other.id, decorated: sym.decorated } }; + } + const v = store.currentVersion(); + const changes = {}; + const old = {}; if (body.size !== undefined) { const size = Number(body.size); if (!Number.isSafeInteger(size) || size <= 0) throw new HttpError(400, 'size must be a positive integer'); + const address = v && store.get('SELECT size FROM symbol_addresses WHERE symbol_id = ? AND version_id = ?', sym.id, v.id); + if (!address) throw new HttpError(409, 'symbol has no address in the current version'); + old.size = address.size; store.run('UPDATE symbol_addresses SET size = ? WHERE symbol_id = ? AND version_id = ?', size, sym.id, v.id); changes.size = size; } if (body.name !== undefined) { const name = body.name === null ? null : String(body.name).slice(0, 200); + old.name = sym.name; store.run('UPDATE symbols SET name = ? WHERE id = ?', name, sym.id); changes.name = name; } + if (body.decorated !== undefined) { + assertUnique(body.decorated, sym.id); + old.decorated = sym.decorated; + store.run('UPDATE symbols SET decorated = ? WHERE id = ?', body.decorated, sym.id); + changes.decorated = body.decorated; + } if (body.retire === true) { if (sym.kind !== 'global') throw new HttpError(400, 'only globals are retired this way'); let annotated = ''; @@ -892,13 +938,17 @@ export function createApp(ctx) { annotated = repo.git(['grep', '-l', '-w', '-e', `GLOBAL: ${sym.id}`, 'HEAD', '--', '.']).trim(); } catch {} if (annotated) throw new HttpError(409, `${sym.id} is still annotated on main (${annotated.split('\n').map((l) => l.replace(/^HEAD:/, '')).join(', ')}); remove those annotations with a maintenance batch first`); + if (!v) throw new HttpError(409, 'no current version'); + old.retired = !store.get('SELECT 1 FROM symbol_addresses WHERE symbol_id = ? AND version_id = ?', sym.id, v.id); store.run('DELETE FROM symbol_addresses WHERE symbol_id = ? AND version_id = ?', sym.id, v.id); changes.retired = true; } - if (!Object.keys(changes).length) throw new HttpError(400, 'nothing to change: give size, name or retire: true'); - store.event('symbol_corrected', user.id, { id: sym.id, ...changes, reason }); + if (!Object.keys(changes).length) throw new HttpError(400, 'nothing to change: give size, name, decorated, swapDecoratedWith or retire: true'); + store.event('symbol_corrected', user.id, { id: sym.id, ...changes, old, new: changes, reason }); + store.setMeta('symbolCorrectionsRevision', Number(store.getMeta('symbolCorrectionsRevision') ?? 0) + 1); + return { id: sym.id, ...changes }; }); - return { id: sym.id, ...changes }; + return result; }); // --- the original function (assembly, pseudocode) for contributors without a local Ghidra. @@ -1211,6 +1261,7 @@ export function createApp(ctx) { const handler = async (req, res) => { const url = new URL(req.url, 'http://localhost'); try { + if (ctx.runsWorker === false) ctx.verifier?.refreshPublishedProject(); if ((req.method === 'GET' || req.method === 'HEAD') && url.pathname === '/discord') { res.writeHead(302, { location: 'https://discord.gg/5fDwn7tVNj', 'cache-control': 'no-store' }); return res.end(); @@ -1224,7 +1275,7 @@ export function createApp(ctx) { res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8', 'cache-control': 'no-cache', 'x-content-type-options': 'nosniff' }); return res.end(script); } - if (req.method === 'GET' && url.pathname.startsWith('/download/')) return serveDownload(url.pathname.slice(10), req, res); + if ((req.method === 'GET' || req.method === 'HEAD') && url.pathname.startsWith('/download/')) return serveDownload(url.pathname.slice(10), req, res); if (req.method === 'GET' && !url.pathname.startsWith('/api/')) return serveStatic(url.pathname, res); const m = router.match(req.method, url.pathname); if (!m) throw new HttpError(404, 'not found'); @@ -1284,6 +1335,8 @@ function withUpdateNote(req, result) { } function serveDownload(name, req, res) { + // Published names are overwritten on every release, including SHA256SUMS and gzip copies. + res.setHeader('cache-control', 'no-store'); const file = DOWNLOADS.has(name) && downloadsRoot ? path.join(downloadsRoot, name) : null; if (!file || !fs.existsSync(file)) { res.writeHead(404, { 'content-type': 'text/plain' }); @@ -1302,6 +1355,7 @@ function serveDownload(name, req, res) { vary: 'accept-encoding', ...(zipped ? { 'content-encoding': 'gzip' } : {}), }); + if (req.method === 'HEAD') return res.end(); fs.createReadStream(sent).pipe(res); } diff --git a/src/server/checks.js b/src/server/checks.js index aa9d45b..92e61a2 100644 --- a/src/server/checks.js +++ b/src/server/checks.js @@ -115,7 +115,7 @@ export class CheckQueue { } pump() { - if (this.stopped) return; + if (this.stopped || this.verifier.repositoryBlocked) return; const max = this.config.remoteCheckConcurrency ?? 2; while (this.running < max) { const job = this.store.tx(() => { diff --git a/src/server/git.js b/src/server/git.js index 1cce793..26e364b 100644 --- a/src/server/git.js +++ b/src/server/git.js @@ -13,7 +13,7 @@ export class Repo { git(args, opts = {}) { try { - return execFileSync('git', args, { cwd: this.dir, encoding: opts.encoding === undefined ? 'utf8' : opts.encoding, maxBuffer: 64 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'], input: opts.input }); + return execFileSync('git', args, { cwd: this.dir, encoding: opts.encoding === undefined ? 'utf8' : opts.encoding, maxBuffer: 64 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'], input: opts.input, timeout: opts.timeout, env: opts.env }); } catch (e) { // git's own explanation, not just "Command failed". const why = `${e.stderr ?? ''}${e.stdout ?? ''}`.trim(); @@ -28,10 +28,19 @@ export class Repo { isAncestor(commit) { try { - this.git(['merge-base', '--is-ancestor', commit, 'HEAD']); - return true; + return this.ancestorOf(commit, 'HEAD'); } catch { - return false; + return false; // a submitted base may not exist locally + } + } + + ancestorOf(commit, descendant) { + try { + this.git(['merge-base', '--is-ancestor', commit, descendant]); + return true; + } catch (e) { + if (e.status === 1) return false; + throw e; } } @@ -82,6 +91,37 @@ export class Repo { return this.git(['status', '--porcelain']).trim() === ''; } + /** Fetches the branch we push, without discarding local commits or working-tree edits. */ + sync(remote) { + const local = this.head(); + let branch; + try { + branch = this.git(['symbolic-ref', '--quiet', 'HEAD']).trim(); + } catch (e) { + throw new Error(`repository sync refused at local HEAD ${local}: no checked-out branch; check out the branch to synchronize`, { cause: e }); + } + const target = `${remote} ${branch}`; + try { + // Explicit refspec: never trust a stale tracking ref or fetch a differently named upstream. + this.git(['fetch', '--no-tags', '--', remote, branch], { timeout: 120_000, env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GCM_INTERACTIVE: 'never' } }); + const upstream = this.git(['rev-parse', '--verify', 'FETCH_HEAD^{commit}']).trim(); + const result = { remote, branch, before: local, head: local, remoteHead: upstream }; + if (!this.isClean()) { + throw new Error(`dirty server working tree at local HEAD ${local}, remote ${upstream}; preserve and commit or move the edits before retrying; no working-tree files were changed`); + } + if (local === upstream) return { ...result, state: 'current' }; + if (this.ancestorOf(upstream, local)) return { ...result, state: 'ahead' }; + if (!this.ancestorOf(local, upstream)) { + throw new Error(`diverged: local HEAD ${local}, remote ${upstream}; a maintainer must reconcile these histories explicitly and retry; no reset, rebase or merge was performed`); + } + // --no-overwrite-ignore also preserves ignored local files that become tracked upstream. + this.git(['-c', 'merge.autoStash=false', 'merge', '--ff-only', '--no-autostash', '--no-overwrite-ignore', upstream]); + return { ...result, state: 'fast-forward', head: this.head() }; + } catch (e) { + throw new Error(`repository sync from ${target} failed (local HEAD ${local}): ${e.message}`, { cause: e }); + } + } + /** * Three-way merges submitted files (written against `base`) onto HEAD. * @returns {{files: Record, conflicts: string[]}} diff --git a/src/server/maintenance.js b/src/server/maintenance.js index b6ff3fc..6307551 100644 --- a/src/server/maintenance.js +++ b/src/server/maintenance.js @@ -50,6 +50,8 @@ export class Maintenance { } async verifyAndMerge(batch) { + // The verifier pump synchronized the repository at the idle cohort boundary, before this + // batch entered a lane. Never fetch here: another batch may already be compiling. const files = JSON.parse(batch.files_json); const back = (returned, details = {}) => ({ state: 'returned', result: { returned, ...details } }); const sourceDir = this.project.sourceDirs[0]; diff --git a/src/server/merge.js b/src/server/merge.js index 0714271..180db36 100644 --- a/src/server/merge.js +++ b/src/server/merge.js @@ -200,7 +200,7 @@ export class Merger { this.settleInactive(); } const started = []; - while (!this.stopped && this.running < MERGE_SLOTS) { + while (!this.stopped && !this.verifier.repositoryBlocked && this.running < MERGE_SLOTS) { const pick = this.next(this.runningBig > 0); if (!pick) break; const big = (pick.batch.cost ?? 0) > BIG_BATCH; @@ -470,6 +470,7 @@ export class Merger { this.log.error(`push to ${this.config.pushRemote} failed: ${e.message}`); } } + this.store.setMeta('repository_project_head', commit); this.onMerged?.(commit); } diff --git a/src/server/verify.js b/src/server/verify.js index a00e467..16c9e71 100644 --- a/src/server/verify.js +++ b/src/server/verify.js @@ -8,7 +8,7 @@ import { freeTextErrorsAcross, executionCharset } from '../shared/freetext.js'; import { checkUnits } from '../shared/check.js'; import { reconcile } from '../shared/reconcile.js'; import { STATUS, sameOriginal } from '../shared/compare.js'; -import { listSources, isUnit, unitSettings } from '../shared/project.js'; +import { listSources, isUnit, unitSettings, loadProject, loadToolchainLock } from '../shared/project.js'; import { changedNames, unitsMentioning } from './affected.js'; import { undefinedDeclarations } from './undefined.js'; import { sourceIncluders } from './includers.js'; @@ -56,6 +56,7 @@ export class Verifier { start() { this.stopped = false; + this.needsSync = !!this.config.pushRemote; this.store.run(`UPDATE batches SET state = 'queued' WHERE state = 'verifying'`); this.store.run(`UPDATE attempts SET state = 'queued' WHERE state = 'verifying'`); // Queue order needs every batch's size (batches from before sizes were recorded). @@ -79,8 +80,77 @@ export class Verifier { this.stopped = true; // finishing verifications must not start new ones } + // A queued batch drains the existing cohort, including merge and remote-check lanes. Git + // and project reload are synchronous, so no job can start between the idle check and FF. + get syncPending() { + return !!this.config.pushRemote && (this.needsSync || !!this.store.get(`SELECT 1 FROM batches WHERE state = 'queued' AND kind IN ('code', 'maintenance') LIMIT 1`)); + } + + get repositoryBlocked() { + return this.syncPending || !!this.syncError; + } + + /** Read one immutable commit, including a newly selected compile-command path. */ + refreshProject(head = this.repo.head(), { invalidateChecks = true } = {}) { + if (this.projectHead === head) return false; + const readFile = (file) => this.repo.showMany(head, [file]).get(file); + const project = loadProject(this.repo.dir, readFile); + const lock = loadToolchainLock(this.repo.dir, readFile); + const digest = lock[project.toolchain]?.clSha256 ?? null; + const changed = JSON.stringify(project) !== JSON.stringify(this.project) || JSON.stringify(lock) !== JSON.stringify(this.config.toolchainLock); + if (changed) { + // Every component holds this same object; remove obsolete optional settings as well. + for (const key of Object.keys(this.project)) delete this.project[key]; + Object.assign(this.project, project); + this.config.envDigest = digest; + this.config.toolchainLock = lock; + this.baselineAt?.clear(); + this.trust?.forget(); + // Compile results from old flags, roots or policy cannot be reused by mergeCheck. + if (invalidateChecks) this.store.run('DELETE FROM batch_checks'); + } + this.projectHead = head; + return changed; + } + + refreshPublishedProject() { + if (!this.config.pushRemote) return; + const head = this.store.getMeta('repository_project_head'); + if (head) this.refreshProject(head, { invalidateChecks: false }); + } + + syncRepository() { + if (!this.config.pushRemote) { + this.syncError = null; + this.needsSync = false; + return; + } + if (this.running || this.merger?.busy || this.checks?.running) throw new Error('repository sync requires idle verification, merge and remote-check lanes'); + const result = this.repo.sync(this.config.pushRemote); + this.refreshProject(); + this.store.setMeta('repository_project_head', result.head); + this.syncError = null; + this.needsSync = false; + this.store.run(`UPDATE batches SET result_json = json_remove(result_json, '$.repositorySync') WHERE state = 'queued' AND json_extract(result_json, '$.repositorySync') IS NOT NULL`); + if (result.state === 'fast-forward') this.log.info(`repository fast-forwarded from ${result.before} to ${result.head} (${result.remote} ${result.branch})`); + return result; + } + pump() { if (this.stopped) return; + if (this.repositoryBlocked) { + if (this.running || this.merger?.busy || this.checks?.running) return; + try { + this.syncRepository(); + } catch (e) { + if (this.syncError !== e.message) this.log.error(e.message); + this.syncError = e.message; + // Infrastructure failures leave submissions queued for a clean retry, with the actual + // diagnostic visible in the batch result instead of a misleading bad-base rejection. + this.store.run(`UPDATE batches SET result_json = json_set(COALESCE(result_json, '{}'), '$.repositorySync', json(?)) WHERE state = 'queued' AND kind IN ('code', 'maintenance')`, JSON.stringify({ error: e.message })); + return; + } + } const max = this.config.verifyConcurrency ?? 3; while (this.running < max) { // Verification compiles only the submitted units, so every batch is small here; the units diff --git a/src/shared/project.js b/src/shared/project.js index a125d19..268a9b1 100644 --- a/src/shared/project.js +++ b/src/shared/project.js @@ -26,13 +26,16 @@ export const DEFAULT_PROJECT = { const UNIT_EXT = new Set(['.c', '.cpp', '.cc', '.cxx']); const SOURCE_EXT = new Set([...UNIT_EXT, '.h', '.hpp', '.inl']); -export function loadProject(repoDir) { - const file = path.join(repoDir, 'data', 'decomp.json'); - const cfg = fs.existsSync(file) ? JSON.parse(fs.readFileSync(file, 'utf8')) : {}; +export function loadProject(repoDir, readFile = (file) => readProjectFile(repoDir, file)) { + const cfg = JSON.parse(readFile('data/decomp.json') ?? '{}'); const project = { ...DEFAULT_PROJECT, ...cfg, matchPolicy: { ...DEFAULT_PROJECT.matchPolicy, ...(cfg.matchPolicy ?? {}) } }; // Optional exact per-unit command lines: { env, shapes: [[args]], units: { file: shapeIndex } }. // Args may use ${SRC} (repo root) and ${TC} (toolchain root); the runner fills them in. - if (cfg.compileCommands) project.compile = JSON.parse(fs.readFileSync(path.join(repoDir, cfg.compileCommands), 'utf8')); + if (cfg.compileCommands) { + const commands = readFile(cfg.compileCommands); + if (commands === null) throw new Error(`missing compile commands: ${cfg.compileCommands}`); + project.compile = JSON.parse(commands); + } return project; } @@ -71,9 +74,13 @@ export function addUnitsToCompileCommands(project, text, files) { return { text: JSON.stringify(doc, null, 1), added, units: doc.units }; } -export function loadToolchainLock(repoDir) { - const file = path.join(repoDir, 'data', 'toolchain.lock.json'); - return fs.existsSync(file) ? JSON.parse(fs.readFileSync(file, 'utf8')) : {}; +export function loadToolchainLock(repoDir, readFile = (file) => readProjectFile(repoDir, file)) { + return JSON.parse(readFile('data/toolchain.lock.json') ?? '{}'); +} + +function readProjectFile(repoDir, file) { + const absolute = path.join(repoDir, file); + return fs.existsSync(absolute) ? fs.readFileSync(absolute, 'utf8') : null; } /** Compiler flags and toolchain for one translation unit. */ diff --git a/test/repo-sync.test.js b/test/repo-sync.test.js new file mode 100644 index 0000000..7178a66 --- /dev/null +++ b/test/repo-sync.test.js @@ -0,0 +1,322 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { execFileSync } from 'node:child_process'; +import { Repo } from '../src/server/git.js'; +import { openServer, createApp } from '../src/server/app.js'; +import { now } from '../src/server/db.js'; +import { fixture as binaryFixture } from './helpers.js'; + +const quiet = { info() {}, error() {} }; + +function repositories(t) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'td-repo-sync-')); + const cleanup = []; + t.after(async () => { + for (const close of cleanup.reverse()) await close(); + fs.rmSync(root, { recursive: true, force: true }); + }); + const remote = path.join(root, 'remote.git'); + const producer = path.join(root, 'producer'); + const dataDir = path.join(root, 'server'); + const local = path.join(dataDir, 'repo'); + const git = (dir, ...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', stdio: 'pipe' }).trim(); + fs.mkdirSync(dataDir); + git(root, 'init', '--bare', '-q', '-b', 'main', remote); + git(root, 'clone', '-q', remote, producer); + const commit = (dir, files, message = 'fixture change') => { + for (const [file, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, file)), { recursive: true }); + fs.writeFileSync(path.join(dir, file), typeof content === 'string' ? content : JSON.stringify(content)); + } + git(dir, 'add', '-A'); + git(dir, '-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'commit', '-q', '-m', message); + return git(dir, 'rev-parse', 'HEAD'); + }; + const base = commit(producer, { + 'data/decomp.json': { name: 'before', sourceDirs: ['src'], includeDirs: ['include'], headerGlobs: ['include/**'], obsoleteSetting: true }, + 'src/base.cpp': 'int base_value;\n', + '.gitattributes': '* text eol=lf\n', + '.gitignore': 'ignored.txt\n', + }); + git(producer, 'push', '-q', 'origin', 'HEAD'); + git(root, 'clone', '-q', remote, local); + git(local, 'remote', 'rename', 'origin', 'publish'); + git(local, 'remote', 'add', 'origin', path.join(root, 'wrong-remote.git')); + const advance = (files) => { + const head = commit(producer, files); + git(producer, 'push', '-q', 'origin', 'HEAD'); + return head; + }; + return { root, remote, producer, dataDir, local, git, commit, base, advance, cleanup, repo: new Repo(local) }; +} + +function serverFixture(t, repos) { + fs.writeFileSync(path.join(repos.dataDir, 'server.json'), JSON.stringify({ pushRemote: 'publish', verifyConcurrency: 2, publicUrl: 'http://127.0.0.1', listen: { host: '127.0.0.1', port: 0 } })); + const ctx = openServer(repos.dataDir, { log: quiet }); + const exe = path.join(repos.dataDir, 'target.exe'); + const pe = binaryFixture().pe; + fs.writeFileSync(exe, pe); + ctx.store.run(`INSERT INTO versions (id, sha256, file, size, is_current, created_at) VALUES ('test', 'test', ?, ?, 1, ?)`, exe, pe.length, now()); + const user = Number(ctx.store.run(`INSERT INTO users (name, role, status, created_at) VALUES ('maintainer', 'maintainer', 'active', ?)`, now()).lastInsertRowid); + repos.cleanup.push(() => { ctx.verifier.stop(); ctx.store.db.close(); }); + const batch = (kind, base = repos.base, files = { 'forbidden.txt': 'invalid\n' }) => { + const id = Number(ctx.store.run(`INSERT INTO batches (user_id, base_commit, files_json, state, kind, created_at) VALUES (?, ?, ?, 'queued', ?, ?)`, user, base, JSON.stringify(files), kind, now()).lastInsertRowid); + if (kind === 'code') { + ctx.store.run(`INSERT OR IGNORE INTO symbols (id, kind, unit, created_at) VALUES ('fn_1', 'function', 'src/base.cpp', ?)`, now()); + const task = Number(ctx.store.run(`INSERT INTO tasks (symbol_id, state, updated_at) VALUES ('fn_1', 'verifying', ?)`, now()).lastInsertRowid); + ctx.store.run(`INSERT INTO attempts (task_id, user_id, base_commit, files_json, state, batch_id, created_at) VALUES (?, ?, ?, ?, 'queued', ?, ?)`, task, user, base, JSON.stringify(files), id, now()); + } + return id; + }; + return { ctx, batch, user }; +} + +async function drained(verifier) { + for (let i = 0; i < 100 && verifier.running; i++) await new Promise((resolve) => setTimeout(resolve, 10)); + assert.equal(verifier.running, 0, 'verification finishes'); +} + +test('sync fetches pushRemote and fast-forwards its current branch, preserving local-ahead work', (t) => { + const r = repositories(t); + const remoteHead = r.advance({ 'src/remote.cpp': 'int remote_value;\n' }); + assert.deepEqual(r.repo.sync('publish'), { remote: 'publish', branch: 'refs/heads/main', before: r.base, head: remoteHead, remoteHead, state: 'fast-forward' }); + assert.equal(fs.readFileSync(path.join(r.local, 'src/remote.cpp'), 'utf8'), 'int remote_value;\n'); + assert.equal(r.repo.sync('publish').state, 'current'); + const ahead = r.commit(r.local, { 'src/local.cpp': 'int local_value;\n' }); + assert.equal(r.repo.sync('publish').state, 'ahead'); + assert.equal(r.repo.head(), ahead); + assert.equal(r.repo.show('HEAD', 'src/local.cpp'), 'int local_value;\n'); +}); + +test('divergence gives both commits and leaves both histories and worktree intact', (t) => { + const r = repositories(t); + const local = r.commit(r.local, { 'src/local.cpp': 'int local_value;\n' }); + const remote = r.advance({ 'src/remote.cpp': 'int remote_value;\n' }); + assert.throws(() => r.repo.sync('publish'), (e) => /diverged/.test(e.message) && e.message.includes(local) && e.message.includes(remote) && /maintainer.*reconcile/.test(e.message)); + assert.equal(r.repo.head(), local); + assert.equal(r.repo.show('HEAD', 'src/local.cpp'), 'int local_value;\n'); + assert.equal(r.repo.show(remote, 'src/remote.cpp'), 'int remote_value;\n'); + assert.equal(r.repo.isClean(), true); +}); + +test('dirty tracked, staged and untracked files are preserved, as are ignored files upstream would track', (t) => { + const r = repositories(t); + r.advance({ 'src/base.cpp': 'int upstream_value;\n', 'new.txt': 'remote\n' }); + fs.writeFileSync(path.join(r.local, 'src/base.cpp'), 'int operator_edit;\n'); + fs.writeFileSync(path.join(r.local, 'new.txt'), 'local\n'); + r.git(r.local, 'add', 'src/base.cpp'); + const index = r.git(r.local, 'show', ':src/base.cpp'); + assert.throws(() => r.repo.sync('publish'), /dirty server working tree/); + assert.equal(r.repo.head(), r.base); + assert.equal(r.git(r.local, 'show', ':src/base.cpp'), index); + assert.equal(fs.readFileSync(path.join(r.local, 'new.txt'), 'utf8'), 'local\n'); +}); + +test('ignored untracked content is not overwritten by a fast-forward', (t) => { + const r = repositories(t); + fs.writeFileSync(path.join(r.local, 'ignored.txt'), 'operator data\n'); + fs.writeFileSync(path.join(r.producer, 'ignored.txt'), 'remote data\n'); + r.git(r.producer, 'add', '-f', 'ignored.txt'); + r.advance({ 'src/remote.cpp': 'int remote_value;\n' }); + assert.throws(() => r.repo.sync('publish'), /overwritten|untracked/i); + assert.equal(r.repo.head(), r.base); + assert.equal(fs.readFileSync(path.join(r.local, 'ignored.txt'), 'utf8'), 'operator data\n'); +}); + +test('failed fetch never reuses FETCH_HEAD or reports success', (t) => { + const r = repositories(t); + r.repo.sync('publish'); + r.git(r.local, 'remote', 'set-url', 'publish', path.join(r.root, 'unavailable.git')); + assert.throws(() => r.repo.sync('publish'), /repository sync.*failed.*local HEAD/s); + assert.equal(r.repo.head(), r.base); +}); + +test('code and maintenance admission waits for every active lane and retries dirty sync safely', async (t) => { + const r = repositories(t); + const { ctx, batch } = serverFixture(t, r); + const remote = r.advance({ 'src/remote.cpp': 'int remote_value;\n' }); + const code = batch('code', remote); + const maintenance = batch('maintenance', remote); + for (const lane of [ctx.verifier, ctx.merger, ctx.checks]) { + lane.running = 1; + ctx.verifier.pump(); + assert.equal(r.repo.head(), r.base); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', code).state, 'queued'); + lane.running = 0; + } + fs.writeFileSync(path.join(r.local, 'operator.txt'), 'preserve me\n'); + ctx.verifier.pump(); + for (const id of [code, maintenance]) { + const row = ctx.store.get('SELECT state, result_json FROM batches WHERE id = ?', id); + assert.equal(row.state, 'queued'); + assert.match(JSON.parse(row.result_json).repositorySync.error, /dirty server working tree/); + } + assert.equal(ctx.verifier.repositoryBlocked, true); + assert.equal(fs.readFileSync(path.join(r.local, 'operator.txt'), 'utf8'), 'preserve me\n'); + fs.renameSync(path.join(r.local, 'operator.txt'), path.join(r.root, 'preserved.txt')); + ctx.verifier.pump(); + await drained(ctx.verifier); + assert.equal(r.repo.head(), remote); + assert.equal(ctx.verifier.repositoryBlocked, false); + for (const id of [code, maintenance]) { + const row = ctx.store.get('SELECT state, result_json FROM batches WHERE id = ?', id); + assert.equal(row.state, 'returned'); + assert.match(row.result_json, /static gate/); + assert.equal(JSON.parse(row.result_json).repositorySync, undefined); + } +}); + +test('sync failure blocks new merge/check work and an operator reconciliation resumes queued work', async (t) => { + const r = repositories(t); + const { ctx, batch, user } = serverFixture(t, r); + const queued = batch('maintenance'); + const approved = batch('code'); + ctx.store.run(`UPDATE batches SET state = 'open' WHERE id = ?`, approved); + ctx.store.run(`UPDATE attempts SET state = 'approved' WHERE batch_id = ?`, approved); + const checkId = Number(ctx.store.run(`INSERT INTO check_jobs (user_id, files_hash, files_json, state, created_at) VALUES (?, 'test', '{}', 'queued', ?)`, user, now()).lastInsertRowid); + const local = r.commit(r.local, { 'src/local.cpp': 'int local_value;\n' }); + const remote = r.advance({ 'src/remote.cpp': 'int remote_value;\n' }); + ctx.verifier.pump(); + const error = JSON.parse(ctx.store.get('SELECT result_json FROM batches WHERE id = ?', queued).result_json).repositorySync.error; + assert.ok(error.includes(local) && error.includes(remote)); + await ctx.merger.pump(); + ctx.checks.pump(); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', approved).state, 'open'); + assert.equal(ctx.store.get('SELECT state FROM check_jobs WHERE id = ?', checkId).state, 'queued'); + assert.equal(r.repo.head(), local); + // Deliberate operator reconciliation, not an automatic reset/rebase by sync. + r.git(r.producer, 'fetch', '-q', r.local, 'main'); + r.git(r.producer, '-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'merge', '-q', '--no-edit', 'FETCH_HEAD'); + r.git(r.producer, 'push', '-q', 'origin', 'HEAD'); + ctx.verifier.pump(); + await drained(ctx.verifier); + assert.equal(ctx.verifier.repositoryBlocked, false); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', queued).state, 'returned'); + ctx.checks.pump(); + for (let i = 0; i < 100 && ctx.checks.running; i++) await new Promise((resolve) => setTimeout(resolve, 10)); + assert.equal(ctx.store.get('SELECT state FROM check_jobs WHERE id = ?', checkId).state, 'done'); + assert.equal(ctx.repo.show('HEAD', 'src/local.cpp'), 'int local_value;\n'); + assert.equal(ctx.repo.show('HEAD', 'src/remote.cpp'), 'int remote_value;\n'); +}); + +test('concurrent batches retain their snapshot until maintenance and code both finish', async (t) => { + const r = repositories(t); + const { ctx, batch } = serverFixture(t, r); + ctx.config.verifyConcurrency = 2; + const releases = []; + const hold = () => new Promise((resolve) => releases.push(resolve)); + const code = batch('code'); + const maintenance = batch('maintenance'); + ctx.verifier.verifyBatch = async (b) => { await hold(); ctx.store.run(`UPDATE batches SET state = 'returned' WHERE id = ?`, b.id); }; + ctx.maintenance.run = async (id) => { await hold(); ctx.store.run(`UPDATE batches SET state = 'returned' WHERE id = ?`, id); }; + ctx.verifier.pump(); + assert.equal(ctx.verifier.running, 2); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', code).state, 'verifying'); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', maintenance).state, 'verifying'); + const remote = r.advance({ 'src/cohort.cpp': 'int cohort_value;\n' }); + const next = batch('maintenance'); + ctx.verifier.pump(); + releases.shift()(); + await new Promise((resolve) => setImmediate(resolve)); + assert.equal(ctx.verifier.running, 1); + assert.equal(r.repo.head(), r.base); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', next).state, 'queued'); + releases.shift()(); + await new Promise((resolve) => setImmediate(resolve)); + assert.equal(r.repo.head(), remote); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', next).state, 'verifying'); + releases.shift()(); + await drained(ctx.verifier); +}); + +test('FF reloads compile-command paths and invalidates old checks before maintenance uses new roots', async (t) => { + const r = repositories(t); + const { ctx, batch } = serverFixture(t, r); + const id = batch('maintenance', r.base, { 'fresh/empty.h': '' }); + ctx.store.run(`INSERT INTO batch_checks (batch_id, head, check_json) VALUES (?, ?, '{}')`, id, r.base); + const shared = ctx.project; + const remote = r.advance({ + 'data/decomp.json': { name: 'after', sourceDirs: ['newsrc'], includeDirs: ['fresh'], headerGlobs: ['fresh/**'], toolchain: 'newtool', compileCommands: 'data/new-commands.json' }, + 'data/new-commands.json': { shapes: [['/c']], units: {} }, + 'data/toolchain.lock.json': { newtool: { clSha256: 'new-digest' } }, + }); + ctx.verifier.pump(); + await drained(ctx.verifier); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', id).state, 'merged'); + assert.equal(ctx.repo.isAncestor(remote), true); + assert.equal(ctx.repo.show('HEAD', 'fresh/empty.h'), ''); + assert.equal(ctx.project, shared); + assert.equal(ctx.maintenance.project, shared); + assert.equal(ctx.tasks.project, shared); + assert.equal(ctx.project.name, 'after'); + assert.equal(ctx.project.obsoleteSetting, undefined); + assert.deepEqual(ctx.project.compile, { shapes: [['/c']], units: {} }); + assert.equal(ctx.config.envDigest, 'new-digest'); + assert.equal(ctx.store.get('SELECT COUNT(*) AS n FROM batch_checks').n, 0); +}); + +test('split web serves only published config and lock snapshots, even during an unpublished FF', async (t) => { + const r = repositories(t); + const { ctx } = serverFixture(t, r); + ctx.verifier.syncRepository(); + const web = openServer(r.dataDir, { log: quiet }); + web.runsWorker = false; + const server = createApp(web); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + r.cleanup.push(async () => { await new Promise((resolve) => server.close(resolve)); web.store.db.close(); }); + const info = async () => { + const response = await fetch(`http://127.0.0.1:${server.address().port}/api/info`); + assert.equal(response.status, 200); + return response.json(); + }; + r.advance({ 'data/decomp.json': { name: 'published', toolchain: 'newtool' }, 'data/toolchain.lock.json': { newtool: { clSha256: 'published-digest' } } }); + r.repo.sync('publish'); // Repo moved, but the worker has not reloaded/published project data yet. + let response = await info(); + assert.equal(response.project, 'before'); + assert.equal(response.toolchain, 'msvc2010'); + assert.deepEqual(response.toolchainLock, {}); + fs.writeFileSync(path.join(r.local, 'data/decomp.json'), '{partial worktree write'); + ctx.verifier.refreshProject(); + ctx.store.setMeta('repository_project_head', r.repo.head()); + response = await info(); + assert.equal(response.project, 'published'); + assert.equal(response.toolchain, 'newtool'); + assert.deepEqual(response.toolchainLock, { newtool: { clSha256: 'published-digest' } }); + assert.equal(web.config.envDigest, 'published-digest'); +}); + +test('startup refreshes an idle worker even with no queued batches', (t) => { + const r = repositories(t); + const { ctx } = serverFixture(t, r); + const remote = r.advance({ 'data/decomp.json': { name: 'startup-refreshed' } }); + ctx.verifier.start(); + ctx.verifier.stop(); + assert.equal(r.repo.head(), remote); + assert.equal(ctx.project.name, 'startup-refreshed'); + assert.equal(ctx.store.getMeta('repository_project_head'), remote); +}); + +test('missing incoming compile commands block verification without publishing stale project data', async (t) => { + const r = repositories(t); + const { ctx, batch } = serverFixture(t, r); + ctx.verifier.syncRepository(); + const id = batch('maintenance'); + const remote = r.advance({ 'data/decomp.json': { name: 'broken', compileCommands: 'data/commands.json' } }); + ctx.verifier.pump(); + assert.equal(r.repo.head(), remote); + assert.equal(ctx.project.name, 'before'); + assert.equal(ctx.store.getMeta('repository_project_head'), r.base); + const row = ctx.store.get('SELECT state, result_json FROM batches WHERE id = ?', id); + assert.equal(row.state, 'queued'); + assert.match(JSON.parse(row.result_json).repositorySync.error, /missing compile commands: data\/commands\.json/); + const repaired = r.advance({ 'data/commands.json': { shapes: [['/c']], units: {} } }); + ctx.verifier.pump(); + await drained(ctx.verifier); + assert.equal(ctx.project.name, 'broken'); + assert.equal(ctx.store.getMeta('repository_project_head'), repaired); + assert.equal(ctx.verifier.repositoryBlocked, false); + assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', id).state, 'returned'); +}); diff --git a/test/symbol-corrections.test.js b/test/symbol-corrections.test.js new file mode 100644 index 0000000..ba21568 --- /dev/null +++ b/test/symbol-corrections.test.js @@ -0,0 +1,246 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { Store, hashToken, now } from '../src/server/db.js'; +import { Accounts } from '../src/server/accounts.js'; +import { createApp, DEFAULT_CONFIG } from '../src/server/app.js'; +import { Workspace } from '../src/client/workspace.js'; +import { correctSymbol, refreshSymbols } from '../src/client/ops.js'; + +async function fixture(t) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'td-symbol-corrections-')); + const store = new Store(path.join(dir, 'td.sqlite')); + const accounts = new Accounts(store); + for (const [id, current] of [['current', 1], ['previous', 0]]) { + store.run('INSERT INTO versions (id, sha256, file, size, is_current, created_at) VALUES (?, ?, ?, 1, ?, ?)', id, 'fixture', 'unused.exe', current, now()); + } + const symbols = ['?First@@YAXXZ', '?Second@@YAXXZ', null].map((decorated, i) => { + const { symbol } = store.ensureSymbol({ kind: 'function', versionId: 'current', addr: 0x401000 + i * 0x100, size: 16, name: `Function${i}`, unit: 'src/functions.cpp' }); + store.run('UPDATE symbols SET decorated = ? WHERE id = ?', decorated, symbol.id); + store.run('INSERT INTO symbol_addresses (symbol_id, version_id, addr, size) VALUES (?, ?, ?, ?)', symbol.id, 'previous', 0x501000 + i * 0x100, 12); + store.run("INSERT INTO func_status (symbol_id, file, status, score, commit_hash, updated_at, confirmed, code_hash) VALUES (?, 'src/functions.cpp', 'equivalent', 0.9, 'original-commit', 123, 1, 'original-code')", symbol.id); + store.run("INSERT INTO tasks (symbol_id, state, updated_at) VALUES (?, 'done', 123)", symbol.id); + return symbol.id; + }); + const user = (role) => { + const id = Number(store.run("INSERT INTO users (name, role, status, created_at) VALUES (?, ?, 'active', ?)", role, role, now()).lastInsertRowid); + const token = accounts.createApiToken(id, 'test').token; + const session = `tds_${role}`; + store.run('INSERT INTO sessions (user_id, token_hash, created_at, expires_at) VALUES (?, ?, ?, ?)', id, hashToken(session), now(), now() + 60_000); + return { id, token, cookie: `td_session=${session}` }; + }; + const maintainer = user('maintainer'); + const contributor = user('contributor'); + const config = { ...DEFAULT_CONFIG, dataDir: dir }; + const server = createApp({ store, accounts, config, project: {}, runsWorker: false }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + t.after(async () => { + await new Promise((resolve) => server.close(resolve)); + store.db.close(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + const base = `http://127.0.0.1:${server.address().port}`; + const post = async (id, body, auth = { token: maintainer.token }) => { + const res = await fetch(`${base}/api/admin/symbols/${id}`, { + method: 'POST', + headers: { 'content-type': 'application/json', ...(auth.token && { authorization: `Bearer ${auth.token}` }), ...(auth.cookie && { cookie: auth.cookie }), ...(auth.csrf && { 'x-td-csrf': '1' }) }, + body: JSON.stringify(body), + }); + return { status: res.status, body: await res.json() }; + }; + const snapshot = async (version = 'current', etag) => { + const res = await fetch(`${base}/api/symbols?version=${version}`, { headers: etag ? { 'if-none-match': etag } : {} }); + if (res.status === 304) return { status: 304 }; + const body = await res.json(); + return { status: res.status, etag: res.headers.get('etag'), symbols: new Map(body.symbols.map((row) => [row[0], Object.fromEntries(body.columns.map((c, i) => [c, row[i]]))])) }; + }; + const state = () => Object.fromEntries(['symbols', 'symbol_addresses', 'func_status', 'tasks'].map((table) => [table, store.all(`SELECT * FROM ${table} ORDER BY rowid`).map((row) => ({ ...row }))])); + const events = () => store.all("SELECT user_id, type, data FROM events WHERE type = 'symbol_corrected' ORDER BY id").map((event) => ({ ...event, data: JSON.parse(event.data) })); + return { store, symbols, maintainer, contributor, post, snapshot, state, events, base, dir }; +} + +test('decorated corrections require maintainer authorization and session CSRF', async (t) => { + const f = await fixture(t); + const before = f.state(); + const body = { decorated: '?Corrected@@YAXXZ', reason: 'Original export and disassembly identify this function.' }; + for (const [auth, status] of [[{}, 401], [{ token: 'td_invalid' }, 401], [{ token: f.contributor.token }, 403], [{ cookie: f.maintainer.cookie }, 403]]) { + assert.equal((await f.post(f.symbols[0], body, auth)).status, status); + } + assert.deepEqual(f.state(), before); + assert.deepEqual(f.events(), []); + assert.equal((await f.post(f.symbols[0], body, { cookie: f.maintainer.cookie, csrf: true })).status, 200); +}); + +test('decorated corrections reject malformed values without coercion, truncation or partial writes', async (t) => { + const f = await fixture(t); + const before = f.state(); + for (const decorated of [null, 0, true, {}, [], '', ' ', ' ?Name@@YAXXZ', '?Name@@YAXXZ ', 'bad\nname', 'bad\n', 'bad\r', 'bad\u0000name', 'bad\u007fname', 'caf\u00e9', 'x'.repeat(2049)]) { + assert.equal((await f.post(f.symbols[0], { name: 'Must not persist', size: 99, decorated, reason: 'Evidence' })).status, 400, JSON.stringify(decorated)); + } + for (const body of [null, [], 'wrong', { decorated: '?New@@YAXXZ' }, { decorated: '?New@@YAXXZ', reason: {} }, { decorated: '?New@@YAXXZ', reason: ' ' }, { decorated: '?New@@YAXXZ', reason: 'x'.repeat(501) }, { decorated: '?New@@YAXXZ', reason: 'Evidence', addr: 123 }]) { + assert.equal((await f.post(f.symbols[0], body)).status, 400, JSON.stringify(body)); + } + assert.deepEqual(f.state(), before); + assert.deepEqual(f.events(), []); +}); + +test('duplicate decorated names roll back every requested correction', async (t) => { + const f = await fixture(t); + const before = f.state(); + const result = await f.post(f.symbols[0], { decorated: '?Second@@YAXXZ', name: 'Must not persist', size: 99, reason: 'Evidence' }); + assert.equal(result.status, 409); + assert.match(result.body.error, new RegExp(f.symbols[1])); + assert.deepEqual(f.state(), before); + assert.deepEqual(f.events(), []); +}); + +test('audited decorated corrections preserve identity and confirmation and invalidate every version snapshot', async (t) => { + const f = await fixture(t); + const before = f.state(); + const snapshots = await Promise.all(['current', 'previous'].map((v) => f.snapshot(v))); + const reason = 'Original export and disassembly identify this function.'; + const result = await f.post(f.symbols[0], { decorated: '?Corrected@@YAXXZ', reason }); + assert.equal(result.status, 200, JSON.stringify(result.body)); + assert.deepEqual(result.body, { id: f.symbols[0], decorated: '?Corrected@@YAXXZ' }); + const expected = structuredClone(before); + expected.symbols[0].decorated = '?Corrected@@YAXXZ'; + assert.deepEqual(f.state(), expected); + assert.deepEqual(f.events(), [{ user_id: f.maintainer.id, type: 'symbol_corrected', data: { id: f.symbols[0], decorated: '?Corrected@@YAXXZ', old: { decorated: '?First@@YAXXZ' }, new: { decorated: '?Corrected@@YAXXZ' }, reason } }]); + for (const [i, version] of ['current', 'previous'].entries()) { + const snapshot = await f.snapshot(version, snapshots[i].etag); + assert.equal(snapshot.status, 200); + assert.notEqual(snapshot.etag, snapshots[i].etag); + assert.equal(snapshot.symbols.get(f.symbols[0]).decorated, '?Corrected@@YAXXZ'); + assert.equal((await f.snapshot(version, snapshot.etag)).status, 304); + } +}); + +test('atomic decorated swaps audit both directions and preserve addresses and function state', async (t) => { + const f = await fixture(t); + const before = f.state(); + const cached = await f.snapshot(); + const reason = 'Original call targets show the two recovered names were reversed.'; + const result = await f.post(f.symbols[0], { swapDecoratedWith: f.symbols[1], reason }); + assert.equal(result.status, 200, JSON.stringify(result.body)); + assert.deepEqual(result.body, { id: f.symbols[0], decorated: '?Second@@YAXXZ', swapped: { id: f.symbols[1], decorated: '?First@@YAXXZ' } }); + const expected = structuredClone(before); + [expected.symbols[0].decorated, expected.symbols[1].decorated] = [expected.symbols[1].decorated, expected.symbols[0].decorated]; + assert.deepEqual(f.state(), expected); + const events = f.events(); + assert.deepEqual(events.map((e) => [e.user_id, e.data.id, e.data.old.decorated, e.data.new.decorated, e.data.swapDecoratedWith, e.data.reason]), [ + [f.maintainer.id, f.symbols[0], '?First@@YAXXZ', '?Second@@YAXXZ', f.symbols[1], reason], + [f.maintainer.id, f.symbols[1], '?Second@@YAXXZ', '?First@@YAXXZ', f.symbols[0], reason], + ]); + const fresh = await f.snapshot('current', cached.etag); + assert.equal(fresh.status, 200); + assert.deepEqual(f.symbols.slice(0, 2).map((id) => fresh.symbols.get(id).decorated), ['?Second@@YAXXZ', '?First@@YAXXZ']); +}); + +test('unsafe swaps reject missing, ambiguous or conflicting mappings transactionally', async (t) => { + const f = await fixture(t); + const before = f.state(); + for (const [body, status] of [ + [{ swapDecoratedWith: f.symbols[0] }, 400], + [{ swapDecoratedWith: f.symbols[2] }, 409], + [{ swapDecoratedWith: 'fn_99999' }, 404], + [{ swapDecoratedWith: null }, 400], + [{ swapDecoratedWith: 123 }, 400], + [{ swapDecoratedWith: f.symbols[1], decorated: '?Other@@YAXXZ' }, 400], + [{ swapDecoratedWith: f.symbols[1], name: 'Other' }, 400], + [{ swapDecoratedWith: f.symbols[1], size: 99 }, 400], + [{ swapDecoratedWith: f.symbols[1], retire: true }, 400], + ]) { + assert.equal((await f.post(f.symbols[0], { ...body, reason: 'Evidence' })).status, status, JSON.stringify(body)); + } + assert.deepEqual(f.state(), before); + f.store.run('UPDATE symbols SET decorated = ? WHERE id = ?', '?First@@YAXXZ', f.symbols[2]); + const ambiguous = f.state(); + assert.equal((await f.post(f.symbols[0], { swapDecoratedWith: f.symbols[1], reason: 'Evidence' })).status, 409); + assert.deepEqual(f.state(), ambiguous); + assert.deepEqual(f.events(), []); +}); + +test('decorated swaps reject function-vtable exchanges without audit or state changes', async (t) => { + const f = await fixture(t); + const { symbol: vtable } = f.store.ensureSymbol({ kind: 'vtable', versionId: 'current', addr: 0x405000, size: 16, name: 'Widget vtable' }); + f.store.run('UPDATE symbols SET decorated = ? WHERE id = ?', '??_7Widget@@6B@', vtable.id); + const before = f.state(); + const revision = f.store.getMeta('symbolCorrectionsRevision'); + const snapshot = await f.snapshot(); + const result = await f.post(f.symbols[0], { swapDecoratedWith: vtable.id, reason: 'Mistaken cross-kind correction' }); + assert.equal(result.status, 409); + assert.deepEqual(f.state(), before); + assert.deepEqual(f.events(), []); + assert.equal(f.store.getMeta('symbolCorrectionsRevision'), revision); + assert.equal((await f.snapshot('current', snapshot.etag)).status, 304); +}); + +test('competing corrections cannot bind one decorated name to two IDs', async (t) => { + const f = await fixture(t); + const results = await Promise.all(f.symbols.slice(0, 2).map((id) => f.post(id, { decorated: '?Contested@@YAXXZ', reason: 'Evidence' }))); + assert.deepEqual(results.map((r) => r.status).sort(), [200, 409]); + const winner = results.find((r) => r.status === 200).body.id; + assert.deepEqual(f.store.all('SELECT id FROM symbols WHERE decorated = ?', '?Contested@@YAXXZ').map((r) => r.id), [winner]); + assert.deepEqual(f.events().map((e) => e.data.id), [winner]); +}); + +test('an audit insert failure rolls back both halves of a decorated swap', async (t) => { + const f = await fixture(t); + const before = f.state(); + f.store.db.exec(`CREATE TRIGGER fail_second_audit BEFORE INSERT ON events WHEN NEW.type = 'symbol_corrected' AND json_extract(NEW.data, '$.id') = '${f.symbols[1]}' BEGIN SELECT RAISE(ABORT, 'audit unavailable'); END`); + assert.equal((await f.post(f.symbols[0], { swapDecoratedWith: f.symbols[1], reason: 'Evidence' })).status, 500); + assert.deepEqual(f.state(), before); + assert.deepEqual(f.events(), []); +}); + +test('decorated correction cache invalidation reaches another web process database connection', async (t) => { + const f = await fixture(t); + const otherStore = new Store(path.join(f.dir, 'td.sqlite')); + const otherServer = createApp({ store: otherStore, accounts: new Accounts(otherStore), config: { ...DEFAULT_CONFIG, dataDir: f.dir }, project: {}, runsWorker: false }); + await new Promise((resolve) => otherServer.listen(0, '127.0.0.1', resolve)); + try { + const url = `http://127.0.0.1:${otherServer.address().port}/api/symbols`; + const cached = await fetch(url); + const etag = cached.headers.get('etag'); + await cached.json(); + assert.equal((await f.post(f.symbols[0], { decorated: '?CrossProcess@@YAXXZ', reason: 'Evidence' })).status, 200); + const changed = await fetch(url, { headers: { 'if-none-match': etag } }); + assert.equal(changed.status, 200); + const snapshot = await changed.json(); + assert.equal(snapshot.symbols.find((row) => row[0] === f.symbols[0])[snapshot.columns.indexOf('decorated')], '?CrossProcess@@YAXXZ'); + } finally { + await new Promise((resolve) => otherServer.close(resolve)); + otherStore.db.close(); + } +}); + +test('decorated names retain C linkage and maximum-length spellings exactly', async (t) => { + const f = await fixture(t); + for (const decorated of ['_Function@4', '@Function@4', 'Function', '?'.repeat(2048)]) { + const result = await f.post(f.symbols[2], { decorated, reason: 'Original symbol spelling' }); + assert.equal(result.status, 200, JSON.stringify(result.body)); + assert.equal(f.store.get('SELECT decorated FROM symbols WHERE id = ?', f.symbols[2]).decorated, decorated); + } +}); + +test('client corrections and swaps replace cached and persisted decorated-name lookups', async (t) => { + const f = await fixture(t); + const ws = new Workspace(path.join(f.dir, 'workspace')); + ws.config = { server: f.base, token: f.maintainer.token }; + ws.save(); + await refreshSymbols(ws); + assert.equal(ws.findSymbol('?First@@YAXXZ').id, f.symbols[0]); + await correctSymbol(ws, f.symbols[0], { decorated: '?ClientCorrected@@YAXXZ', reason: 'Evidence' }); + assert.equal(ws.findSymbol('?First@@YAXXZ'), null); + assert.equal(ws.nameIndex.get('?ClientCorrected@@YAXXZ'), f.symbols[0]); + assert.equal(new Workspace(ws.dir).findSymbol('?ClientCorrected@@YAXXZ').id, f.symbols[0]); + await correctSymbol(ws, f.symbols[0], { swapDecoratedWith: f.symbols[1], reason: 'Evidence' }); + assert.equal(ws.nameIndex.get('?ClientCorrected@@YAXXZ'), f.symbols[1]); + assert.equal(ws.nameIndex.get('?Second@@YAXXZ'), f.symbols[0]); + const persisted = fs.readFileSync(ws.symbolsFile, 'utf8'); + await assert.rejects(correctSymbol(ws, f.symbols[0], { decorated: '?ClientCorrected@@YAXXZ', reason: 'Evidence' }), (error) => error.status === 409); + assert.equal(fs.readFileSync(ws.symbolsFile, 'utf8'), persisted); + assert.equal(new Workspace(ws.dir).findSymbol('?ClientCorrected@@YAXXZ').id, f.symbols[1]); +}); diff --git a/test/update.test.js b/test/update.test.js new file mode 100644 index 0000000..48c4a92 --- /dev/null +++ b/test/update.test.js @@ -0,0 +1,122 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import http from 'node:http'; +import crypto from 'node:crypto'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import { gzipSync } from 'node:zlib'; +import { createApp, DEFAULT_CONFIG } from '../src/server/app.js'; + +const exec = promisify(execFile); +const ASSET = process.platform === 'win32' ? 'td-windows-x64.exe' : 'td-linux-x64'; +const hash = (data) => crypto.createHash('sha256').update(data).digest('hex'); +const installed = Buffer.from('installed td build'); +const published = Buffer.from('newly published td build'); + +async function listen(t, server, dir) { + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + t.after(async () => { + await new Promise((resolve) => server.close(resolve)); + fs.rmSync(dir, { recursive: true, force: true }); + }); + return `http://127.0.0.1:${server.address().port}`; +} + +async function runUpdater(dir, server) { + // Isolate the SEA gate and executable/home paths, not the updater's HTTP, hashing or writes. + const script = ` + import { createRequire } from 'node:module'; + import path from 'node:path'; + const [dir, server, updater] = process.argv.slice(1); + const require = createRequire(import.meta.url); + require('node:sea').isSea = () => true; + require('node:os').homedir = () => dir; + process.execPath = path.join(dir, 'td'); + const { maybeUpdate } = await import(updater); + const logs = []; + const result = await maybeUpdate({ server, force: true, log: (message) => logs.push(message) }); + console.log(JSON.stringify({ result, logs })); + `; + const { stdout } = await exec(process.execPath, ['--input-type=module', '-e', script, dir, server, new URL('../src/client/update.js', import.meta.url).href], { + env: { ...process.env, TD_NO_UPDATE: '0' }, + timeout: 15_000, + }); + return JSON.parse(stdout); +} + +async function cachedDownloads(t, corrupt = false) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'td-update-')); + const exe = path.join(dir, 'td'); + fs.writeFileSync(exe, installed); + const requests = []; + const server = http.createServer((req, res) => { + requests.push(req.url); + const url = new URL(req.url, 'http://localhost'); + if (url.pathname === '/download/SHA256SUMS') { + // A cache serves the old manifest unless the client requests revalidation. + const revalidate = /(?:no-cache|no-store|max-age=0)/.test(req.headers['cache-control'] ?? ''); + return res.end(`${hash(revalidate ? published : installed)} ${ASSET}\n`); + } + if (url.pathname === `/download/${ASSET}`) { + // The mutable URL is stale; only a new content-keyed URL reaches the new build. + const keyed = url.searchParams.get('sha256') === hash(published); + return res.end(keyed && !corrupt ? published : installed); + } + res.writeHead(404); + res.end(); + }); + return { dir, exe, requests, base: await listen(t, server, dir) }; +} + +test('self-update revalidates the manifest and bypasses a stale unversioned binary', async (t) => { + const f = await cachedDownloads(t); + const { result } = await runUpdater(f.dir, f.base); + assert.equal(typeof result, 'string', 'the downloaded update was installed'); + assert.deepEqual(fs.readFileSync(f.exe), published); + assert.deepEqual(f.requests, ['/download/SHA256SUMS', `/download/${ASSET}?sha256=${hash(published)}`]); + assert.equal(fs.existsSync(f.exe + '.new'), false); +}); + +test('self-update rejects mismatched bytes even at the expected SHA256 URL', async (t) => { + const f = await cachedDownloads(t, true); + const { result } = await runUpdater(f.dir, f.base); + assert.deepEqual(f.requests, ['/download/SHA256SUMS', `/download/${ASSET}?sha256=${hash(published)}`]); + assert.equal(result, null); + assert.deepEqual(fs.readFileSync(f.exe), installed, 'checksum failure must preserve the installed executable'); + assert.equal(fs.existsSync(f.exe + '.new'), false); + assert.equal(fs.existsSync(f.exe + '.old'), false); +}); + +test('mutable downloads are not stored by caches for GET or HEAD, including gzip and missing assets', async (t) => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'td-download-')); + const downloads = path.join(dir, 'downloads'); + fs.mkdirSync(downloads); + fs.writeFileSync(path.join(downloads, ASSET), published); + const zipped = gzipSync(published); + fs.writeFileSync(path.join(downloads, ASSET + '.gz'), zipped); + const manifest = `${hash(published)} ${ASSET}\n`; + fs.writeFileSync(path.join(downloads, 'SHA256SUMS'), manifest); + const server = createApp({ config: { ...DEFAULT_CONFIG, dataDir: dir }, project: {}, runsWorker: false }); + const base = await listen(t, server, dir); + for (const method of ['GET', 'HEAD']) { + for (const [asset, encoding, body, length] of [ + [ASSET, 'gzip', published, zipped.length], + [`${ASSET}?sha256=${hash(published)}`, 'identity', published, published.length], + ['SHA256SUMS', 'identity', Buffer.from(manifest), Buffer.byteLength(manifest)], + ]) { + const res = await fetch(`${base}/download/${asset}`, { method, headers: { 'accept-encoding': encoding } }); + assert.equal(res.status, 200); + assert.equal(res.headers.get('cache-control'), 'no-store'); + assert.equal(res.headers.get('content-encoding'), encoding === 'gzip' ? 'gzip' : null); + assert.equal(Number(res.headers.get('content-length')), length); + assert.deepEqual(Buffer.from(await res.arrayBuffer()), method === 'HEAD' ? Buffer.alloc(0) : body); + } + const missing = await fetch(`${base}/download/missing`, { method }); + assert.equal(missing.status, 404); + assert.equal(missing.headers.get('cache-control'), 'no-store'); + await missing.arrayBuffer(); + } +});