Bring repository in line with the deployed server
Commits the changes already running in production (image rodecomp/server:adminreview-20261028), which were never pushed: - Repository sync before verification: with pushRemote set, the worker fast-forwards its checkout at idle cohort boundaries, blocks merge/check lanes while a sync is pending or failed, and reloads data/decomp.json, the compile commands and data/toolchain.lock.json from one committed snapshot; the web process adopts only the published snapshot (git.js, verify.js, merge.js, checks.js, maintenance.js, project.js, app.js). - Symbol corrections: POST /api/admin/symbols/:id accepts decorated and swapDecoratedWith with strict validation, records old/new values, and invalidates cached symbol snapshots; td maintenance symbol --decorated/--swap-decorated-with and MCP td_correct_symbol (cli.js, mcp.js, ops.js, app.js). - Downloads: /download/ responses are no-store and answer HEAD; self-update revalidates the manifest and fetches the binary with ?sha256= to bypass stale CDN entries (update.js, app.js). - deploy/rodecomp/compose.yaml: Forgejo SSH enabled on port 22 (SSH_DOMAIN 192.168.1.77), as running. Tests: repo-sync, symbol-corrections and update suites added; npm test passes (164).
This commit is contained in:
ragnarok-clanker
parent
57ea62ef8c
commit
d7c825abc0
16 changed files with 1016 additions and 44 deletions
70
README.md
70
README.md
|
|
@ -147,11 +147,46 @@ Unrepresentable characters are rejected rather than accepted as replacement `?`
|
|||
bytes. Configure source UTF-8 and the target execution code page separately; this
|
||||
does not permit new strings absent from the original executable.
|
||||
|
||||
### Repository synchronization before verification
|
||||
|
||||
When `server.json` sets `pushRemote`, the single worker fetches that remote's
|
||||
branch matching its checked-out branch at startup and before admitting each
|
||||
code/maintenance verification cohort. Existing verification, maintenance, merge
|
||||
and remote-check jobs drain first; new jobs do not start during synchronization.
|
||||
No remote is guessed when `pushRemote` is unset.
|
||||
|
||||
Only a fast-forward is automatic. Local-ahead commits are retained. Divergence
|
||||
reports both full commit IDs and requires a maintainer to reconcile the histories;
|
||||
there is no automatic reset, rebase or merge commit. Dirty tracked, staged and
|
||||
untracked files block synchronization, and ignored files cannot be overwritten by
|
||||
incoming tracked files. Preserve local edits before retrying.
|
||||
|
||||
Fetch/authentication failures and unsafe states leave batches queued with an
|
||||
explicit `result.repositorySync.error`; merge and remote-check admission pauses.
|
||||
The worker retries at its next pump and removes the diagnostic after a successful
|
||||
sync, so operator repairs do not require resubmission or a restart. Fetch is
|
||||
noninteractive and bounded to two minutes.
|
||||
|
||||
After synchronization the worker reloads `data/decomp.json`, its selected
|
||||
compile-command file and `data/toolchain.lock.json` from one committed snapshot,
|
||||
updates the shared project object and invalidates checks cached under old settings.
|
||||
The separate web process adopts only the published snapshot, never partially
|
||||
updated working-tree configuration. Direct uncommitted configuration changes are
|
||||
not loaded this way; commit them, or restart web and worker when no remote is used.
|
||||
|
||||
|
||||
## Building td
|
||||
|
||||
`node tools/build-td/build.mjs` (Node 22.21) writes `dist/td-windows-x64.exe`, `dist/td-linux-x64` and
|
||||
`SHA256SUMS`; copy them to `<dataDir>/downloads` on the server.
|
||||
|
||||
These download names are mutable: the server sends `Cache-Control: no-store` on `/download/`
|
||||
responses, including the manifest and compressed binaries. Self-updates revalidate the manifest,
|
||||
request the binary with `?sha256=<expected hash>` to bypass an older unversioned CDN entry, and
|
||||
still verify its SHA-256 before replacing the installed executable. When deploying this policy
|
||||
behind a CDN, purge existing `/download/` cache entries once: new origin headers do not evict
|
||||
already-cached responses used by older clients.
|
||||
|
||||
## Tests
|
||||
|
||||
`npm test` uses synthetic PE/COFF files, so it doesn't need a compiler.
|
||||
|
|
@ -160,7 +195,8 @@ does not permit new strings absent from the original executable.
|
|||
|
||||
Set `"globalRules": true` in the target repository's `decomp.json` to enable the
|
||||
source-level global gate. It is opt-in; deploying this server does not enable it
|
||||
for other projects. Restart web and worker after changing project configuration.
|
||||
for other projects. Committed configuration is refreshed by repository synchronization
|
||||
when `pushRemote` is configured; otherwise restart web and worker after changing it.
|
||||
|
||||
The verifier applies the payload over current main before checking it, and the
|
||||
merger checks again before committing. Both ordinary and maintenance batches use
|
||||
|
|
@ -195,6 +231,38 @@ build. Behavior-changing function work must use the normal task/review path.
|
|||
`td maintenance symbol <id> --size N --reason "evidence"` corrects metadata;
|
||||
`--retire` is allowed only after main no longer annotates that global ID.
|
||||
|
||||
From a contributor workspace, a maintainer can correct a decorated/linker name
|
||||
without changing a symbol's ID, address, task state, match state or confirmations:
|
||||
|
||||
```sh
|
||||
td maintenance symbol fn_00123 --decorated '?CorrectName@@YAXXZ' --reason "original-binary evidence"
|
||||
td maintenance symbol fn_00123 --swap-decorated-with fn_00456 --reason "call targets show the names were reversed"
|
||||
```
|
||||
|
||||
The maintained HTTP route is `POST /api/admin/symbols/:id`. Send either
|
||||
`{"decorated":"?CorrectName@@YAXXZ","reason":"original-binary evidence"}` or
|
||||
`{"swapDecoratedWith":"fn_00456","reason":"call-target evidence"}`. MCP exposes
|
||||
the same fields through `td_correct_symbol`, with an additional `id` argument.
|
||||
Both clients refresh their symbol snapshots after a successful correction.
|
||||
|
||||
`decorated` must be a string of 1-2048 visible ASCII characters, with no whitespace
|
||||
or control characters; it is never coerced, trimmed or truncated. C-linkage names
|
||||
are supported as well as C++ decorated names. Null and empty values are rejected.
|
||||
The evidence `reason` must be a nonblank string of at most 500 characters.
|
||||
Unknown request fields are rejected. Existing `size`, `name` and `retire` fields
|
||||
remain available; a swap cannot be combined with other corrections.
|
||||
|
||||
A name already mapped to another ID returns HTTP 409 without applying any part
|
||||
of the request. A swap requires two distinct, existing, unambiguous mappings
|
||||
whose symbols have the same kind; it exchanges only their decorated names in
|
||||
one database transaction. Do not
|
||||
clear a mapping or use a temporary name as an intermediate step. Each affected
|
||||
ID receives a `symbol_corrected` event containing the acting user, `old` and
|
||||
`new` values, and the evidence reason (plus the other ID for a swap). Audit
|
||||
records and corrections commit together. Symbol snapshot caches in every web
|
||||
process and target version are invalidated by the committed correction, so an
|
||||
immediate conditional refresh observes the new names rather than a stale 304.
|
||||
|
||||
Neither an unchanged-code fingerprint nor a successful unit compile proves that
|
||||
new global storage reproduces the original initializer/destructor behavior.
|
||||
Maintenance authors must preserve those semantics; an incomplete object layout
|
||||
|
|
|
|||
|
|
@ -42,7 +42,10 @@ services:
|
|||
USER_GID: "1000"
|
||||
FORGEJO__server__ROOT_URL: "https://clank.rodecomp.online/git/"
|
||||
FORGEJO__server__DOMAIN: "clank.rodecomp.online"
|
||||
FORGEJO__server__DISABLE_SSH: "true"
|
||||
FORGEJO__server__DISABLE_SSH: "false"
|
||||
FORGEJO__server__SSH_PORT: "22"
|
||||
FORGEJO__server__SSH_LISTEN_PORT: "22"
|
||||
FORGEJO__server__SSH_DOMAIN: "192.168.1.77"
|
||||
FORGEJO__server__OFFLINE_MODE: "true"
|
||||
FORGEJO__server__LANDING_PAGE: "explore"
|
||||
FORGEJO__database__DB_TYPE: "sqlite3"
|
||||
|
|
@ -56,6 +59,8 @@ services:
|
|||
FORGEJO__actions__ENABLED: "false"
|
||||
FORGEJO__packages__ENABLED: "false"
|
||||
FORGEJO__repository__DISABLED_REPO_UNITS: "repo.pulls,repo.actions,repo.packages,repo.releases,repo.projects"
|
||||
ports:
|
||||
- "22:22"
|
||||
volumes:
|
||||
- forgejo:/data
|
||||
restart: unless-stopped
|
||||
|
|
|
|||
|
|
@ -51,6 +51,7 @@ Maintenance (maintainers: structural changes with no function tasks)
|
|||
td maintenance submit --files a.cpp,b.h --notes T verified like a batch; merges only when no
|
||||
function's compiled code changes
|
||||
td maintenance symbol <id> [--size N] [--name N] [--retire] --reason T correct the symbol database
|
||||
[--decorated N | --swap-decorated-with ID] correct a decorated name or atomically swap two
|
||||
|
||||
Peer review (attempts that are not byte-identical; judge by reading, not by tools)
|
||||
td peer list attempts waiting for review
|
||||
|
|
@ -120,6 +121,8 @@ async function main() {
|
|||
files: { type: 'string' },
|
||||
reason: { type: 'string' },
|
||||
size: { type: 'string' },
|
||||
decorated: { type: 'string' },
|
||||
'swap-decorated-with': { type: 'string' },
|
||||
retire: { type: 'boolean', default: false },
|
||||
json: { type: 'boolean', default: false },
|
||||
free: { type: 'boolean', default: false },
|
||||
|
|
@ -218,11 +221,16 @@ connect your AI with: td connect --workspace "${ws.dir}"`);
|
|||
return print(await ops.submitMaintenance(ws, files, { notes: values.notes }));
|
||||
}
|
||||
if (sub === 'symbol') {
|
||||
if (!id || !values.reason) throw new Error('usage: td maintenance symbol <id> [--size N] [--name N] [--retire] --reason "evidence"');
|
||||
const body = { reason: values.reason, ...(values.size && { size: Number(values.size) }), ...(values.name && { name: values.name }), ...(values.retire && { retire: true }) };
|
||||
const r = await client.post(`/api/admin/symbols/${id}`, body);
|
||||
await ops.refreshSymbols(ws);
|
||||
return print(r);
|
||||
if (!id || !values.reason) throw new Error('usage: td maintenance symbol <id> [--size N] [--name N] [--retire] [--decorated N | --swap-decorated-with ID] --reason "evidence"');
|
||||
const body = {
|
||||
reason: values.reason,
|
||||
...(values.size !== undefined && { size: Number(values.size) }),
|
||||
...(values.name !== undefined && { name: values.name }),
|
||||
...(values.retire && { retire: true }),
|
||||
...(values.decorated !== undefined && { decorated: values.decorated }),
|
||||
...(values['swap-decorated-with'] !== undefined && { swapDecoratedWith: values['swap-decorated-with'] }),
|
||||
};
|
||||
return print(await ops.correctSymbol(ws, id, body));
|
||||
}
|
||||
throw new Error('usage: td maintenance submit|symbol ...');
|
||||
}
|
||||
|
|
|
|||
|
|
@ -105,6 +105,25 @@ const TOOLS = [
|
|||
return r;
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'td_correct_symbol',
|
||||
description: 'Maintainers only: correct symbol metadata from original-binary evidence, recording old/new values and the reason. Set decorated to replace one mapping, or swapDecoratedWith to atomically exchange two existing names of symbols with the same kind (no other correction fields with a swap). IDs, addresses, function state and confirmations remain unchanged for name corrections. Refreshes the workspace symbol snapshot.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
properties: {
|
||||
id: str('stable symbol ID to correct'),
|
||||
reason: { type: 'string', minLength: 1, maxLength: 500, description: 'original-binary evidence for this correction' },
|
||||
size: { type: 'integer', minimum: 1, description: 'correct size in the current target version' },
|
||||
name: { type: ['string', 'null'], maxLength: 200, description: 'human-readable name; null clears it' },
|
||||
decorated: { type: 'string', minLength: 1, maxLength: 2048, pattern: '^[!-~]+$', description: 'exact decorated name; visible ASCII only, no whitespace or truncation' },
|
||||
swapDecoratedWith: { type: 'string', pattern: '^(fn|gl|vt|st)_[0-9]+$', description: 'other stable ID of the same symbol kind whose decorated name is exchanged with this one' },
|
||||
retire: { type: 'boolean', description: 'retire a global only after main no longer annotates it' },
|
||||
},
|
||||
required: ['id', 'reason'],
|
||||
},
|
||||
run: async (ws, { id, ...correction }) => ops.correctSymbol(ws, id, correction),
|
||||
},
|
||||
{
|
||||
name: 'td_check',
|
||||
description:
|
||||
|
|
|
|||
|
|
@ -319,6 +319,13 @@ export async function submitMaintenance(ws, files, { notes } = {}) {
|
|||
return api(ws).post('/api/maintenance', { baseCommit: ws.head(), files: contents, notes });
|
||||
}
|
||||
|
||||
/** Correct symbol metadata and immediately replace the workspace's cached name mappings. */
|
||||
export async function correctSymbol(ws, id, correction) {
|
||||
const result = await api(ws).post(`/api/admin/symbols/${encodeURIComponent(id)}`, correction);
|
||||
await refreshSymbols(ws);
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Annotated functions whose definition differs from HEAD in the changed (or named) files under allowed paths. */
|
||||
function rewrittenFunctions(ws, allowed, files) {
|
||||
const out = new Set();
|
||||
|
|
|
|||
|
|
@ -82,7 +82,7 @@ export async function maybeUpdate({ server, force = false, log = (m) => process.
|
|||
if (!force && state.checkedAt && Date.now() - state.checkedAt < CHECK_EVERY_MS) return null;
|
||||
state.checkedAt = Date.now();
|
||||
|
||||
const res = await fetch(`${base}/download/SHA256SUMS`, { signal: AbortSignal.timeout(10_000) });
|
||||
const res = await fetch(`${base}/download/SHA256SUMS`, { cache: 'no-store', signal: AbortSignal.timeout(10_000) });
|
||||
if (!res.ok) return writeState(state), null;
|
||||
const want = new RegExp(`^([0-9a-f]{64})\\s+${ASSET.replace('.', '\\.')}$`, 'm').exec(await res.text())?.[1];
|
||||
if (!want || want === (await ownHash(exe, state))) return writeState(state), null;
|
||||
|
|
@ -91,7 +91,8 @@ export async function maybeUpdate({ server, force = false, log = (m) => process.
|
|||
// Slow connections take as long as they need; only a download that stalls is given up.
|
||||
const stall = new AbortController();
|
||||
let timer = setTimeout(() => stall.abort(), STALL_MS);
|
||||
const bin = await fetch(`${base}/download/${ASSET}`, { signal: stall.signal });
|
||||
// A new manifest must not reuse a CDN's cached copy of the previous build.
|
||||
const bin = await fetch(`${base}/download/${ASSET}?sha256=${want}`, { signal: stall.signal });
|
||||
if (!bin.ok) return clearTimeout(timer), writeState(state), null;
|
||||
const parts = [];
|
||||
for await (const chunk of bin.body) {
|
||||
|
|
|
|||
|
|
@ -67,8 +67,12 @@ export function openServer(dataDir, { log = console } = {}) {
|
|||
const config = loadConfig(dataDir);
|
||||
const store = new Store(path.join(config.dataDir, 'td.sqlite'));
|
||||
const repo = new Repo(path.join(config.dataDir, 'repo'));
|
||||
const project = loadProject(repo.dir);
|
||||
config.envDigest = loadToolchainLock(repo.dir)[project.toolchain]?.clSha256 ?? null;
|
||||
// Web and worker share Git, not JS objects. Never read half of a working-tree FF.
|
||||
const projectHead = config.pushRemote ? store.getMeta('repository_project_head') ?? repo.head() : null;
|
||||
const readProjectFile = projectHead ? (file) => repo.showMany(projectHead, [file]).get(file) : undefined;
|
||||
const project = loadProject(repo.dir, readProjectFile);
|
||||
config.toolchainLock = loadToolchainLock(repo.dir, readProjectFile);
|
||||
config.envDigest = config.toolchainLock[project.toolchain]?.clSha256 ?? null;
|
||||
|
||||
let peCache = null;
|
||||
const getPe = () => {
|
||||
|
|
@ -94,6 +98,7 @@ export function openServer(dataDir, { log = console } = {}) {
|
|||
const peers = new PeerReviews({ store, project, trust });
|
||||
const verifier = new Verifier({ store, config, repo, project, tasks, getPe, peers, trust, log });
|
||||
verifier.regions = regions;
|
||||
verifier.projectHead = projectHead;
|
||||
const merger = new Merger({ store, config, repo, project, tasks, verifier, getPe, peers, log });
|
||||
const accounts = new Accounts(store);
|
||||
const originals = new Originals({ store });
|
||||
|
|
@ -102,6 +107,8 @@ export function openServer(dataDir, { log = console } = {}) {
|
|||
const maintenance = new Maintenance({ store, repo, project, verifier, merger, log });
|
||||
verifier.maintenance = maintenance;
|
||||
const checks = new CheckQueue({ store, repo, verifier, config, log });
|
||||
verifier.merger = merger;
|
||||
verifier.checks = checks;
|
||||
const credits = new Credits({ store });
|
||||
return { config, store, repo, project, tasks, regions, quality, trust, credits, verifier, merger, maintenance, getPe, accounts, peers, originals, ghidra, checks };
|
||||
}
|
||||
|
|
@ -176,7 +183,7 @@ export function createApp(ctx) {
|
|||
gitWeb: config.gitWebUrl ?? null,
|
||||
version: v ? { id: v.id, label: v.label ?? v.id, sha256: v.sha256, size: v.size, download: config.distributeTarget ? `${config.publicUrl}/api/target/${encodeURIComponent(v.id)}` : null } : null,
|
||||
toolchain: project.toolchain,
|
||||
toolchainLock: loadToolchainLock(repo.dir),
|
||||
toolchainLock: config.pushRemote ? config.toolchainLock : loadToolchainLock(repo.dir),
|
||||
limits: { ...DEFAULT_LIMITS, ...project.lintLimits },
|
||||
matchPolicy: project.matchPolicy,
|
||||
};
|
||||
|
|
@ -200,12 +207,12 @@ export function createApp(ctx) {
|
|||
router.on('GET', '/api/symbols', ({ req, res, query }) => {
|
||||
const v = query.get('version') ? store.get('SELECT * FROM versions WHERE id = ?', query.get('version')) : store.currentVersion();
|
||||
if (!v) throw new HttpError(404, 'unknown version');
|
||||
// Building it takes a while; a minute-old table is fine for agents.
|
||||
// A correction revision invalidates cached snapshots in every web process and version.
|
||||
const { body, etag } = cached(`symbols:${v.id}`, 60_000, () => {
|
||||
const rows = [...store.symbolMap(v.id).values()].map((s) => [s.id, s.kind, s.addr, s.size, s.name, s.decorated, s.library ? 1 : 0, s.unit]);
|
||||
const json = JSON.stringify({ version: v.id, columns: ['id', 'kind', 'addr', 'size', 'name', 'decorated', 'library', 'unit'], symbols: rows });
|
||||
return { body: json, etag: `"${crypto.createHash('sha1').update(json).digest('hex')}"` };
|
||||
});
|
||||
}, store.getMeta('symbolCorrectionsRevision'));
|
||||
const headers = { etag, 'cache-control': 'no-cache', 'x-content-type-options': 'nosniff' };
|
||||
if (req.headers['if-none-match'] === etag) {
|
||||
res.writeHead(304, headers);
|
||||
|
|
@ -220,11 +227,11 @@ export function createApp(ctx) {
|
|||
const ACTIVE_MS = 15 * 60_000;
|
||||
// Public numbers every open page polls: computed at most once per `ttl` and shared by everyone.
|
||||
const shared = new Map();
|
||||
const cached = (key, ttl, compute) => {
|
||||
const cached = (key, ttl, compute, revision = null) => {
|
||||
const hit = shared.get(key);
|
||||
if (hit && Date.now() - hit.at < ttl) return hit.value;
|
||||
if (hit && hit.revision === revision && Date.now() - hit.at < ttl) return hit.value;
|
||||
const value = compute();
|
||||
shared.set(key, { at: Date.now(), value });
|
||||
shared.set(key, { at: Date.now(), value, revision });
|
||||
return value;
|
||||
};
|
||||
|
||||
|
|
@ -859,32 +866,71 @@ export function createApp(ctx) {
|
|||
return { id: r.symbol.id, kind: r.symbol.kind, created: r.created, annotation: `// ${{ function: 'FUNCTION', global: 'GLOBAL', vtable: 'VTABLE', string: 'STRING' }[kind]}: ${r.symbol.id}` };
|
||||
});
|
||||
|
||||
// Corrections to the symbol database a maintainer makes from evidence in the original: a size,
|
||||
// a name, or a global that is not one (an address inside another object, registered by mistake).
|
||||
// Evidence-backed metadata corrections never change stable IDs or function match/review state.
|
||||
// Decorated names can be replaced or exchanged atomically when two mappings were reversed.
|
||||
// Retiring removes its address in the current version, so nothing resolves to it any more; its
|
||||
// ID stays reserved. Only symbols no file on main annotates may be retired: a maintenance batch
|
||||
// removes those annotations first.
|
||||
router.on('POST', '/api/admin/symbols/:id', async ({ req, params }) => {
|
||||
const user = auth(req, { role: 'maintainer' });
|
||||
const body = await readJson(req, 4096);
|
||||
const sym = store.get('SELECT * FROM symbols WHERE id = ?', String(params.id));
|
||||
if (!body || typeof body !== 'object' || Array.isArray(body)) throw new HttpError(400, 'body must be an object');
|
||||
const allowed = ['reason', 'size', 'name', 'retire', 'decorated', 'swapDecoratedWith'];
|
||||
if (Object.keys(body).some((key) => !allowed.includes(key))) throw new HttpError(400, 'unknown symbol correction field');
|
||||
if (typeof body.reason !== 'string' || !body.reason.trim() || body.reason.length > 500) throw new HttpError(400, 'reason must be 1-500 characters: the evidence for the correction');
|
||||
const reason = body.reason.trim();
|
||||
const validDecorated = (value) => typeof value === 'string' && value.length > 0 && value.length <= 2048 && !/[^\x21-\x7e]/.test(value);
|
||||
if (body.decorated !== undefined && !validDecorated(body.decorated)) throw new HttpError(400, 'decorated must be 1-2048 visible ASCII characters without whitespace');
|
||||
if (body.retire !== undefined && typeof body.retire !== 'boolean') throw new HttpError(400, 'retire must be a boolean');
|
||||
if (body.swapDecoratedWith !== undefined) {
|
||||
if (typeof body.swapDecoratedWith !== 'string' || !/^(fn|gl|vt|st)_[0-9]+$/.test(body.swapDecoratedWith) || body.swapDecoratedWith === params.id) throw new HttpError(400, 'swapDecoratedWith must be a different symbol ID');
|
||||
if (['size', 'name', 'retire', 'decorated'].some((key) => body[key] !== undefined)) throw new HttpError(400, 'swapDecoratedWith cannot be combined with other corrections');
|
||||
}
|
||||
const result = store.tx(() => {
|
||||
const sym = store.get('SELECT * FROM symbols WHERE id = ?', params.id);
|
||||
if (!sym) throw new HttpError(404, 'no such symbol');
|
||||
const assertUnique = (decorated, ownerId) => {
|
||||
const duplicate = store.get('SELECT id FROM symbols WHERE decorated = ? AND id != ? LIMIT 1', decorated, ownerId);
|
||||
if (duplicate) throw new HttpError(409, `decorated name is already mapped to ${duplicate.id}; use swapDecoratedWith to exchange two mappings`);
|
||||
};
|
||||
if (body.swapDecoratedWith !== undefined) {
|
||||
const other = store.get('SELECT * FROM symbols WHERE id = ?', body.swapDecoratedWith);
|
||||
if (!other) throw new HttpError(404, 'no such swap symbol');
|
||||
if (sym.kind !== other.kind) throw new HttpError(409, 'a swap requires symbols of the same kind');
|
||||
if (!validDecorated(sym.decorated) || !validDecorated(other.decorated) || sym.decorated === other.decorated) throw new HttpError(409, 'a swap requires two distinct existing decorated names');
|
||||
assertUnique(sym.decorated, sym.id);
|
||||
assertUnique(other.decorated, other.id);
|
||||
store.run('UPDATE symbols SET decorated = CASE id WHEN ? THEN ? ELSE ? END WHERE id IN (?, ?)', sym.id, other.decorated, sym.decorated, sym.id, other.id);
|
||||
for (const [from, to] of [[sym, other], [other, sym]]) {
|
||||
store.event('symbol_corrected', user.id, { id: from.id, decorated: to.decorated, old: { decorated: from.decorated }, new: { decorated: to.decorated }, swapDecoratedWith: to.id, reason });
|
||||
}
|
||||
store.setMeta('symbolCorrectionsRevision', Number(store.getMeta('symbolCorrectionsRevision') ?? 0) + 1);
|
||||
return { id: sym.id, decorated: other.decorated, swapped: { id: other.id, decorated: sym.decorated } };
|
||||
}
|
||||
const v = store.currentVersion();
|
||||
const reason = String(body.reason ?? '').trim().slice(0, 500);
|
||||
if (!reason) throw new HttpError(400, 'reason is required: the evidence for the correction');
|
||||
const changes = {};
|
||||
store.tx(() => {
|
||||
const old = {};
|
||||
if (body.size !== undefined) {
|
||||
const size = Number(body.size);
|
||||
if (!Number.isSafeInteger(size) || size <= 0) throw new HttpError(400, 'size must be a positive integer');
|
||||
const address = v && store.get('SELECT size FROM symbol_addresses WHERE symbol_id = ? AND version_id = ?', sym.id, v.id);
|
||||
if (!address) throw new HttpError(409, 'symbol has no address in the current version');
|
||||
old.size = address.size;
|
||||
store.run('UPDATE symbol_addresses SET size = ? WHERE symbol_id = ? AND version_id = ?', size, sym.id, v.id);
|
||||
changes.size = size;
|
||||
}
|
||||
if (body.name !== undefined) {
|
||||
const name = body.name === null ? null : String(body.name).slice(0, 200);
|
||||
old.name = sym.name;
|
||||
store.run('UPDATE symbols SET name = ? WHERE id = ?', name, sym.id);
|
||||
changes.name = name;
|
||||
}
|
||||
if (body.decorated !== undefined) {
|
||||
assertUnique(body.decorated, sym.id);
|
||||
old.decorated = sym.decorated;
|
||||
store.run('UPDATE symbols SET decorated = ? WHERE id = ?', body.decorated, sym.id);
|
||||
changes.decorated = body.decorated;
|
||||
}
|
||||
if (body.retire === true) {
|
||||
if (sym.kind !== 'global') throw new HttpError(400, 'only globals are retired this way');
|
||||
let annotated = '';
|
||||
|
|
@ -892,14 +938,18 @@ export function createApp(ctx) {
|
|||
annotated = repo.git(['grep', '-l', '-w', '-e', `GLOBAL: ${sym.id}`, 'HEAD', '--', '.']).trim();
|
||||
} catch {}
|
||||
if (annotated) throw new HttpError(409, `${sym.id} is still annotated on main (${annotated.split('\n').map((l) => l.replace(/^HEAD:/, '')).join(', ')}); remove those annotations with a maintenance batch first`);
|
||||
if (!v) throw new HttpError(409, 'no current version');
|
||||
old.retired = !store.get('SELECT 1 FROM symbol_addresses WHERE symbol_id = ? AND version_id = ?', sym.id, v.id);
|
||||
store.run('DELETE FROM symbol_addresses WHERE symbol_id = ? AND version_id = ?', sym.id, v.id);
|
||||
changes.retired = true;
|
||||
}
|
||||
if (!Object.keys(changes).length) throw new HttpError(400, 'nothing to change: give size, name or retire: true');
|
||||
store.event('symbol_corrected', user.id, { id: sym.id, ...changes, reason });
|
||||
});
|
||||
if (!Object.keys(changes).length) throw new HttpError(400, 'nothing to change: give size, name, decorated, swapDecoratedWith or retire: true');
|
||||
store.event('symbol_corrected', user.id, { id: sym.id, ...changes, old, new: changes, reason });
|
||||
store.setMeta('symbolCorrectionsRevision', Number(store.getMeta('symbolCorrectionsRevision') ?? 0) + 1);
|
||||
return { id: sym.id, ...changes };
|
||||
});
|
||||
return result;
|
||||
});
|
||||
|
||||
// --- the original function (assembly, pseudocode) for contributors without a local Ghidra.
|
||||
// Derived from the exe, so approved accounts only, like the exe itself.
|
||||
|
|
@ -1211,6 +1261,7 @@ export function createApp(ctx) {
|
|||
const handler = async (req, res) => {
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
try {
|
||||
if (ctx.runsWorker === false) ctx.verifier?.refreshPublishedProject();
|
||||
if ((req.method === 'GET' || req.method === 'HEAD') && url.pathname === '/discord') {
|
||||
res.writeHead(302, { location: 'https://discord.gg/5fDwn7tVNj', 'cache-control': 'no-store' });
|
||||
return res.end();
|
||||
|
|
@ -1224,7 +1275,7 @@ export function createApp(ctx) {
|
|||
res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8', 'cache-control': 'no-cache', 'x-content-type-options': 'nosniff' });
|
||||
return res.end(script);
|
||||
}
|
||||
if (req.method === 'GET' && url.pathname.startsWith('/download/')) return serveDownload(url.pathname.slice(10), req, res);
|
||||
if ((req.method === 'GET' || req.method === 'HEAD') && url.pathname.startsWith('/download/')) return serveDownload(url.pathname.slice(10), req, res);
|
||||
if (req.method === 'GET' && !url.pathname.startsWith('/api/')) return serveStatic(url.pathname, res);
|
||||
const m = router.match(req.method, url.pathname);
|
||||
if (!m) throw new HttpError(404, 'not found');
|
||||
|
|
@ -1284,6 +1335,8 @@ function withUpdateNote(req, result) {
|
|||
}
|
||||
|
||||
function serveDownload(name, req, res) {
|
||||
// Published names are overwritten on every release, including SHA256SUMS and gzip copies.
|
||||
res.setHeader('cache-control', 'no-store');
|
||||
const file = DOWNLOADS.has(name) && downloadsRoot ? path.join(downloadsRoot, name) : null;
|
||||
if (!file || !fs.existsSync(file)) {
|
||||
res.writeHead(404, { 'content-type': 'text/plain' });
|
||||
|
|
@ -1302,6 +1355,7 @@ function serveDownload(name, req, res) {
|
|||
vary: 'accept-encoding',
|
||||
...(zipped ? { 'content-encoding': 'gzip' } : {}),
|
||||
});
|
||||
if (req.method === 'HEAD') return res.end();
|
||||
fs.createReadStream(sent).pipe(res);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -115,7 +115,7 @@ export class CheckQueue {
|
|||
}
|
||||
|
||||
pump() {
|
||||
if (this.stopped) return;
|
||||
if (this.stopped || this.verifier.repositoryBlocked) return;
|
||||
const max = this.config.remoteCheckConcurrency ?? 2;
|
||||
while (this.running < max) {
|
||||
const job = this.store.tx(() => {
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ export class Repo {
|
|||
|
||||
git(args, opts = {}) {
|
||||
try {
|
||||
return execFileSync('git', args, { cwd: this.dir, encoding: opts.encoding === undefined ? 'utf8' : opts.encoding, maxBuffer: 64 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'], input: opts.input });
|
||||
return execFileSync('git', args, { cwd: this.dir, encoding: opts.encoding === undefined ? 'utf8' : opts.encoding, maxBuffer: 64 * 1024 * 1024, stdio: ['pipe', 'pipe', 'pipe'], input: opts.input, timeout: opts.timeout, env: opts.env });
|
||||
} catch (e) {
|
||||
// git's own explanation, not just "Command failed".
|
||||
const why = `${e.stderr ?? ''}${e.stdout ?? ''}`.trim();
|
||||
|
|
@ -28,10 +28,19 @@ export class Repo {
|
|||
|
||||
isAncestor(commit) {
|
||||
try {
|
||||
this.git(['merge-base', '--is-ancestor', commit, 'HEAD']);
|
||||
return true;
|
||||
return this.ancestorOf(commit, 'HEAD');
|
||||
} catch {
|
||||
return false;
|
||||
return false; // a submitted base may not exist locally
|
||||
}
|
||||
}
|
||||
|
||||
ancestorOf(commit, descendant) {
|
||||
try {
|
||||
this.git(['merge-base', '--is-ancestor', commit, descendant]);
|
||||
return true;
|
||||
} catch (e) {
|
||||
if (e.status === 1) return false;
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -82,6 +91,37 @@ export class Repo {
|
|||
return this.git(['status', '--porcelain']).trim() === '';
|
||||
}
|
||||
|
||||
/** Fetches the branch we push, without discarding local commits or working-tree edits. */
|
||||
sync(remote) {
|
||||
const local = this.head();
|
||||
let branch;
|
||||
try {
|
||||
branch = this.git(['symbolic-ref', '--quiet', 'HEAD']).trim();
|
||||
} catch (e) {
|
||||
throw new Error(`repository sync refused at local HEAD ${local}: no checked-out branch; check out the branch to synchronize`, { cause: e });
|
||||
}
|
||||
const target = `${remote} ${branch}`;
|
||||
try {
|
||||
// Explicit refspec: never trust a stale tracking ref or fetch a differently named upstream.
|
||||
this.git(['fetch', '--no-tags', '--', remote, branch], { timeout: 120_000, env: { ...process.env, GIT_TERMINAL_PROMPT: '0', GCM_INTERACTIVE: 'never' } });
|
||||
const upstream = this.git(['rev-parse', '--verify', 'FETCH_HEAD^{commit}']).trim();
|
||||
const result = { remote, branch, before: local, head: local, remoteHead: upstream };
|
||||
if (!this.isClean()) {
|
||||
throw new Error(`dirty server working tree at local HEAD ${local}, remote ${upstream}; preserve and commit or move the edits before retrying; no working-tree files were changed`);
|
||||
}
|
||||
if (local === upstream) return { ...result, state: 'current' };
|
||||
if (this.ancestorOf(upstream, local)) return { ...result, state: 'ahead' };
|
||||
if (!this.ancestorOf(local, upstream)) {
|
||||
throw new Error(`diverged: local HEAD ${local}, remote ${upstream}; a maintainer must reconcile these histories explicitly and retry; no reset, rebase or merge was performed`);
|
||||
}
|
||||
// --no-overwrite-ignore also preserves ignored local files that become tracked upstream.
|
||||
this.git(['-c', 'merge.autoStash=false', 'merge', '--ff-only', '--no-autostash', '--no-overwrite-ignore', upstream]);
|
||||
return { ...result, state: 'fast-forward', head: this.head() };
|
||||
} catch (e) {
|
||||
throw new Error(`repository sync from ${target} failed (local HEAD ${local}): ${e.message}`, { cause: e });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Three-way merges submitted files (written against `base`) onto HEAD.
|
||||
* @returns {{files: Record<string,string>, conflicts: string[]}}
|
||||
|
|
|
|||
|
|
@ -50,6 +50,8 @@ export class Maintenance {
|
|||
}
|
||||
|
||||
async verifyAndMerge(batch) {
|
||||
// The verifier pump synchronized the repository at the idle cohort boundary, before this
|
||||
// batch entered a lane. Never fetch here: another batch may already be compiling.
|
||||
const files = JSON.parse(batch.files_json);
|
||||
const back = (returned, details = {}) => ({ state: 'returned', result: { returned, ...details } });
|
||||
const sourceDir = this.project.sourceDirs[0];
|
||||
|
|
|
|||
|
|
@ -200,7 +200,7 @@ export class Merger {
|
|||
this.settleInactive();
|
||||
}
|
||||
const started = [];
|
||||
while (!this.stopped && this.running < MERGE_SLOTS) {
|
||||
while (!this.stopped && !this.verifier.repositoryBlocked && this.running < MERGE_SLOTS) {
|
||||
const pick = this.next(this.runningBig > 0);
|
||||
if (!pick) break;
|
||||
const big = (pick.batch.cost ?? 0) > BIG_BATCH;
|
||||
|
|
@ -470,6 +470,7 @@ export class Merger {
|
|||
this.log.error(`push to ${this.config.pushRemote} failed: ${e.message}`);
|
||||
}
|
||||
}
|
||||
this.store.setMeta('repository_project_head', commit);
|
||||
this.onMerged?.(commit);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ import { freeTextErrorsAcross, executionCharset } from '../shared/freetext.js';
|
|||
import { checkUnits } from '../shared/check.js';
|
||||
import { reconcile } from '../shared/reconcile.js';
|
||||
import { STATUS, sameOriginal } from '../shared/compare.js';
|
||||
import { listSources, isUnit, unitSettings } from '../shared/project.js';
|
||||
import { listSources, isUnit, unitSettings, loadProject, loadToolchainLock } from '../shared/project.js';
|
||||
import { changedNames, unitsMentioning } from './affected.js';
|
||||
import { undefinedDeclarations } from './undefined.js';
|
||||
import { sourceIncluders } from './includers.js';
|
||||
|
|
@ -56,6 +56,7 @@ export class Verifier {
|
|||
|
||||
start() {
|
||||
this.stopped = false;
|
||||
this.needsSync = !!this.config.pushRemote;
|
||||
this.store.run(`UPDATE batches SET state = 'queued' WHERE state = 'verifying'`);
|
||||
this.store.run(`UPDATE attempts SET state = 'queued' WHERE state = 'verifying'`);
|
||||
// Queue order needs every batch's size (batches from before sizes were recorded).
|
||||
|
|
@ -79,8 +80,77 @@ export class Verifier {
|
|||
this.stopped = true; // finishing verifications must not start new ones
|
||||
}
|
||||
|
||||
// A queued batch drains the existing cohort, including merge and remote-check lanes. Git
|
||||
// and project reload are synchronous, so no job can start between the idle check and FF.
|
||||
get syncPending() {
|
||||
return !!this.config.pushRemote && (this.needsSync || !!this.store.get(`SELECT 1 FROM batches WHERE state = 'queued' AND kind IN ('code', 'maintenance') LIMIT 1`));
|
||||
}
|
||||
|
||||
get repositoryBlocked() {
|
||||
return this.syncPending || !!this.syncError;
|
||||
}
|
||||
|
||||
/** Read one immutable commit, including a newly selected compile-command path. */
|
||||
refreshProject(head = this.repo.head(), { invalidateChecks = true } = {}) {
|
||||
if (this.projectHead === head) return false;
|
||||
const readFile = (file) => this.repo.showMany(head, [file]).get(file);
|
||||
const project = loadProject(this.repo.dir, readFile);
|
||||
const lock = loadToolchainLock(this.repo.dir, readFile);
|
||||
const digest = lock[project.toolchain]?.clSha256 ?? null;
|
||||
const changed = JSON.stringify(project) !== JSON.stringify(this.project) || JSON.stringify(lock) !== JSON.stringify(this.config.toolchainLock);
|
||||
if (changed) {
|
||||
// Every component holds this same object; remove obsolete optional settings as well.
|
||||
for (const key of Object.keys(this.project)) delete this.project[key];
|
||||
Object.assign(this.project, project);
|
||||
this.config.envDigest = digest;
|
||||
this.config.toolchainLock = lock;
|
||||
this.baselineAt?.clear();
|
||||
this.trust?.forget();
|
||||
// Compile results from old flags, roots or policy cannot be reused by mergeCheck.
|
||||
if (invalidateChecks) this.store.run('DELETE FROM batch_checks');
|
||||
}
|
||||
this.projectHead = head;
|
||||
return changed;
|
||||
}
|
||||
|
||||
refreshPublishedProject() {
|
||||
if (!this.config.pushRemote) return;
|
||||
const head = this.store.getMeta('repository_project_head');
|
||||
if (head) this.refreshProject(head, { invalidateChecks: false });
|
||||
}
|
||||
|
||||
syncRepository() {
|
||||
if (!this.config.pushRemote) {
|
||||
this.syncError = null;
|
||||
this.needsSync = false;
|
||||
return;
|
||||
}
|
||||
if (this.running || this.merger?.busy || this.checks?.running) throw new Error('repository sync requires idle verification, merge and remote-check lanes');
|
||||
const result = this.repo.sync(this.config.pushRemote);
|
||||
this.refreshProject();
|
||||
this.store.setMeta('repository_project_head', result.head);
|
||||
this.syncError = null;
|
||||
this.needsSync = false;
|
||||
this.store.run(`UPDATE batches SET result_json = json_remove(result_json, '$.repositorySync') WHERE state = 'queued' AND json_extract(result_json, '$.repositorySync') IS NOT NULL`);
|
||||
if (result.state === 'fast-forward') this.log.info(`repository fast-forwarded from ${result.before} to ${result.head} (${result.remote} ${result.branch})`);
|
||||
return result;
|
||||
}
|
||||
|
||||
pump() {
|
||||
if (this.stopped) return;
|
||||
if (this.repositoryBlocked) {
|
||||
if (this.running || this.merger?.busy || this.checks?.running) return;
|
||||
try {
|
||||
this.syncRepository();
|
||||
} catch (e) {
|
||||
if (this.syncError !== e.message) this.log.error(e.message);
|
||||
this.syncError = e.message;
|
||||
// Infrastructure failures leave submissions queued for a clean retry, with the actual
|
||||
// diagnostic visible in the batch result instead of a misleading bad-base rejection.
|
||||
this.store.run(`UPDATE batches SET result_json = json_set(COALESCE(result_json, '{}'), '$.repositorySync', json(?)) WHERE state = 'queued' AND kind IN ('code', 'maintenance')`, JSON.stringify({ error: e.message }));
|
||||
return;
|
||||
}
|
||||
}
|
||||
const max = this.config.verifyConcurrency ?? 3;
|
||||
while (this.running < max) {
|
||||
// Verification compiles only the submitted units, so every batch is small here; the units
|
||||
|
|
|
|||
|
|
@ -26,13 +26,16 @@ export const DEFAULT_PROJECT = {
|
|||
const UNIT_EXT = new Set(['.c', '.cpp', '.cc', '.cxx']);
|
||||
const SOURCE_EXT = new Set([...UNIT_EXT, '.h', '.hpp', '.inl']);
|
||||
|
||||
export function loadProject(repoDir) {
|
||||
const file = path.join(repoDir, 'data', 'decomp.json');
|
||||
const cfg = fs.existsSync(file) ? JSON.parse(fs.readFileSync(file, 'utf8')) : {};
|
||||
export function loadProject(repoDir, readFile = (file) => readProjectFile(repoDir, file)) {
|
||||
const cfg = JSON.parse(readFile('data/decomp.json') ?? '{}');
|
||||
const project = { ...DEFAULT_PROJECT, ...cfg, matchPolicy: { ...DEFAULT_PROJECT.matchPolicy, ...(cfg.matchPolicy ?? {}) } };
|
||||
// Optional exact per-unit command lines: { env, shapes: [[args]], units: { file: shapeIndex } }.
|
||||
// Args may use ${SRC} (repo root) and ${TC} (toolchain root); the runner fills them in.
|
||||
if (cfg.compileCommands) project.compile = JSON.parse(fs.readFileSync(path.join(repoDir, cfg.compileCommands), 'utf8'));
|
||||
if (cfg.compileCommands) {
|
||||
const commands = readFile(cfg.compileCommands);
|
||||
if (commands === null) throw new Error(`missing compile commands: ${cfg.compileCommands}`);
|
||||
project.compile = JSON.parse(commands);
|
||||
}
|
||||
return project;
|
||||
}
|
||||
|
||||
|
|
@ -71,9 +74,13 @@ export function addUnitsToCompileCommands(project, text, files) {
|
|||
return { text: JSON.stringify(doc, null, 1), added, units: doc.units };
|
||||
}
|
||||
|
||||
export function loadToolchainLock(repoDir) {
|
||||
const file = path.join(repoDir, 'data', 'toolchain.lock.json');
|
||||
return fs.existsSync(file) ? JSON.parse(fs.readFileSync(file, 'utf8')) : {};
|
||||
export function loadToolchainLock(repoDir, readFile = (file) => readProjectFile(repoDir, file)) {
|
||||
return JSON.parse(readFile('data/toolchain.lock.json') ?? '{}');
|
||||
}
|
||||
|
||||
function readProjectFile(repoDir, file) {
|
||||
const absolute = path.join(repoDir, file);
|
||||
return fs.existsSync(absolute) ? fs.readFileSync(absolute, 'utf8') : null;
|
||||
}
|
||||
|
||||
/** Compiler flags and toolchain for one translation unit. */
|
||||
|
|
|
|||
322
test/repo-sync.test.js
Normal file
322
test/repo-sync.test.js
Normal file
|
|
@ -0,0 +1,322 @@
|
|||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { Repo } from '../src/server/git.js';
|
||||
import { openServer, createApp } from '../src/server/app.js';
|
||||
import { now } from '../src/server/db.js';
|
||||
import { fixture as binaryFixture } from './helpers.js';
|
||||
|
||||
const quiet = { info() {}, error() {} };
|
||||
|
||||
function repositories(t) {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'td-repo-sync-'));
|
||||
const cleanup = [];
|
||||
t.after(async () => {
|
||||
for (const close of cleanup.reverse()) await close();
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
const remote = path.join(root, 'remote.git');
|
||||
const producer = path.join(root, 'producer');
|
||||
const dataDir = path.join(root, 'server');
|
||||
const local = path.join(dataDir, 'repo');
|
||||
const git = (dir, ...args) => execFileSync('git', args, { cwd: dir, encoding: 'utf8', stdio: 'pipe' }).trim();
|
||||
fs.mkdirSync(dataDir);
|
||||
git(root, 'init', '--bare', '-q', '-b', 'main', remote);
|
||||
git(root, 'clone', '-q', remote, producer);
|
||||
const commit = (dir, files, message = 'fixture change') => {
|
||||
for (const [file, content] of Object.entries(files)) {
|
||||
fs.mkdirSync(path.dirname(path.join(dir, file)), { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, file), typeof content === 'string' ? content : JSON.stringify(content));
|
||||
}
|
||||
git(dir, 'add', '-A');
|
||||
git(dir, '-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'commit', '-q', '-m', message);
|
||||
return git(dir, 'rev-parse', 'HEAD');
|
||||
};
|
||||
const base = commit(producer, {
|
||||
'data/decomp.json': { name: 'before', sourceDirs: ['src'], includeDirs: ['include'], headerGlobs: ['include/**'], obsoleteSetting: true },
|
||||
'src/base.cpp': 'int base_value;\n',
|
||||
'.gitattributes': '* text eol=lf\n',
|
||||
'.gitignore': 'ignored.txt\n',
|
||||
});
|
||||
git(producer, 'push', '-q', 'origin', 'HEAD');
|
||||
git(root, 'clone', '-q', remote, local);
|
||||
git(local, 'remote', 'rename', 'origin', 'publish');
|
||||
git(local, 'remote', 'add', 'origin', path.join(root, 'wrong-remote.git'));
|
||||
const advance = (files) => {
|
||||
const head = commit(producer, files);
|
||||
git(producer, 'push', '-q', 'origin', 'HEAD');
|
||||
return head;
|
||||
};
|
||||
return { root, remote, producer, dataDir, local, git, commit, base, advance, cleanup, repo: new Repo(local) };
|
||||
}
|
||||
|
||||
function serverFixture(t, repos) {
|
||||
fs.writeFileSync(path.join(repos.dataDir, 'server.json'), JSON.stringify({ pushRemote: 'publish', verifyConcurrency: 2, publicUrl: 'http://127.0.0.1', listen: { host: '127.0.0.1', port: 0 } }));
|
||||
const ctx = openServer(repos.dataDir, { log: quiet });
|
||||
const exe = path.join(repos.dataDir, 'target.exe');
|
||||
const pe = binaryFixture().pe;
|
||||
fs.writeFileSync(exe, pe);
|
||||
ctx.store.run(`INSERT INTO versions (id, sha256, file, size, is_current, created_at) VALUES ('test', 'test', ?, ?, 1, ?)`, exe, pe.length, now());
|
||||
const user = Number(ctx.store.run(`INSERT INTO users (name, role, status, created_at) VALUES ('maintainer', 'maintainer', 'active', ?)`, now()).lastInsertRowid);
|
||||
repos.cleanup.push(() => { ctx.verifier.stop(); ctx.store.db.close(); });
|
||||
const batch = (kind, base = repos.base, files = { 'forbidden.txt': 'invalid\n' }) => {
|
||||
const id = Number(ctx.store.run(`INSERT INTO batches (user_id, base_commit, files_json, state, kind, created_at) VALUES (?, ?, ?, 'queued', ?, ?)`, user, base, JSON.stringify(files), kind, now()).lastInsertRowid);
|
||||
if (kind === 'code') {
|
||||
ctx.store.run(`INSERT OR IGNORE INTO symbols (id, kind, unit, created_at) VALUES ('fn_1', 'function', 'src/base.cpp', ?)`, now());
|
||||
const task = Number(ctx.store.run(`INSERT INTO tasks (symbol_id, state, updated_at) VALUES ('fn_1', 'verifying', ?)`, now()).lastInsertRowid);
|
||||
ctx.store.run(`INSERT INTO attempts (task_id, user_id, base_commit, files_json, state, batch_id, created_at) VALUES (?, ?, ?, ?, 'queued', ?, ?)`, task, user, base, JSON.stringify(files), id, now());
|
||||
}
|
||||
return id;
|
||||
};
|
||||
return { ctx, batch, user };
|
||||
}
|
||||
|
||||
async function drained(verifier) {
|
||||
for (let i = 0; i < 100 && verifier.running; i++) await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
assert.equal(verifier.running, 0, 'verification finishes');
|
||||
}
|
||||
|
||||
test('sync fetches pushRemote and fast-forwards its current branch, preserving local-ahead work', (t) => {
|
||||
const r = repositories(t);
|
||||
const remoteHead = r.advance({ 'src/remote.cpp': 'int remote_value;\n' });
|
||||
assert.deepEqual(r.repo.sync('publish'), { remote: 'publish', branch: 'refs/heads/main', before: r.base, head: remoteHead, remoteHead, state: 'fast-forward' });
|
||||
assert.equal(fs.readFileSync(path.join(r.local, 'src/remote.cpp'), 'utf8'), 'int remote_value;\n');
|
||||
assert.equal(r.repo.sync('publish').state, 'current');
|
||||
const ahead = r.commit(r.local, { 'src/local.cpp': 'int local_value;\n' });
|
||||
assert.equal(r.repo.sync('publish').state, 'ahead');
|
||||
assert.equal(r.repo.head(), ahead);
|
||||
assert.equal(r.repo.show('HEAD', 'src/local.cpp'), 'int local_value;\n');
|
||||
});
|
||||
|
||||
test('divergence gives both commits and leaves both histories and worktree intact', (t) => {
|
||||
const r = repositories(t);
|
||||
const local = r.commit(r.local, { 'src/local.cpp': 'int local_value;\n' });
|
||||
const remote = r.advance({ 'src/remote.cpp': 'int remote_value;\n' });
|
||||
assert.throws(() => r.repo.sync('publish'), (e) => /diverged/.test(e.message) && e.message.includes(local) && e.message.includes(remote) && /maintainer.*reconcile/.test(e.message));
|
||||
assert.equal(r.repo.head(), local);
|
||||
assert.equal(r.repo.show('HEAD', 'src/local.cpp'), 'int local_value;\n');
|
||||
assert.equal(r.repo.show(remote, 'src/remote.cpp'), 'int remote_value;\n');
|
||||
assert.equal(r.repo.isClean(), true);
|
||||
});
|
||||
|
||||
test('dirty tracked, staged and untracked files are preserved, as are ignored files upstream would track', (t) => {
|
||||
const r = repositories(t);
|
||||
r.advance({ 'src/base.cpp': 'int upstream_value;\n', 'new.txt': 'remote\n' });
|
||||
fs.writeFileSync(path.join(r.local, 'src/base.cpp'), 'int operator_edit;\n');
|
||||
fs.writeFileSync(path.join(r.local, 'new.txt'), 'local\n');
|
||||
r.git(r.local, 'add', 'src/base.cpp');
|
||||
const index = r.git(r.local, 'show', ':src/base.cpp');
|
||||
assert.throws(() => r.repo.sync('publish'), /dirty server working tree/);
|
||||
assert.equal(r.repo.head(), r.base);
|
||||
assert.equal(r.git(r.local, 'show', ':src/base.cpp'), index);
|
||||
assert.equal(fs.readFileSync(path.join(r.local, 'new.txt'), 'utf8'), 'local\n');
|
||||
});
|
||||
|
||||
test('ignored untracked content is not overwritten by a fast-forward', (t) => {
|
||||
const r = repositories(t);
|
||||
fs.writeFileSync(path.join(r.local, 'ignored.txt'), 'operator data\n');
|
||||
fs.writeFileSync(path.join(r.producer, 'ignored.txt'), 'remote data\n');
|
||||
r.git(r.producer, 'add', '-f', 'ignored.txt');
|
||||
r.advance({ 'src/remote.cpp': 'int remote_value;\n' });
|
||||
assert.throws(() => r.repo.sync('publish'), /overwritten|untracked/i);
|
||||
assert.equal(r.repo.head(), r.base);
|
||||
assert.equal(fs.readFileSync(path.join(r.local, 'ignored.txt'), 'utf8'), 'operator data\n');
|
||||
});
|
||||
|
||||
test('failed fetch never reuses FETCH_HEAD or reports success', (t) => {
|
||||
const r = repositories(t);
|
||||
r.repo.sync('publish');
|
||||
r.git(r.local, 'remote', 'set-url', 'publish', path.join(r.root, 'unavailable.git'));
|
||||
assert.throws(() => r.repo.sync('publish'), /repository sync.*failed.*local HEAD/s);
|
||||
assert.equal(r.repo.head(), r.base);
|
||||
});
|
||||
|
||||
test('code and maintenance admission waits for every active lane and retries dirty sync safely', async (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx, batch } = serverFixture(t, r);
|
||||
const remote = r.advance({ 'src/remote.cpp': 'int remote_value;\n' });
|
||||
const code = batch('code', remote);
|
||||
const maintenance = batch('maintenance', remote);
|
||||
for (const lane of [ctx.verifier, ctx.merger, ctx.checks]) {
|
||||
lane.running = 1;
|
||||
ctx.verifier.pump();
|
||||
assert.equal(r.repo.head(), r.base);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', code).state, 'queued');
|
||||
lane.running = 0;
|
||||
}
|
||||
fs.writeFileSync(path.join(r.local, 'operator.txt'), 'preserve me\n');
|
||||
ctx.verifier.pump();
|
||||
for (const id of [code, maintenance]) {
|
||||
const row = ctx.store.get('SELECT state, result_json FROM batches WHERE id = ?', id);
|
||||
assert.equal(row.state, 'queued');
|
||||
assert.match(JSON.parse(row.result_json).repositorySync.error, /dirty server working tree/);
|
||||
}
|
||||
assert.equal(ctx.verifier.repositoryBlocked, true);
|
||||
assert.equal(fs.readFileSync(path.join(r.local, 'operator.txt'), 'utf8'), 'preserve me\n');
|
||||
fs.renameSync(path.join(r.local, 'operator.txt'), path.join(r.root, 'preserved.txt'));
|
||||
ctx.verifier.pump();
|
||||
await drained(ctx.verifier);
|
||||
assert.equal(r.repo.head(), remote);
|
||||
assert.equal(ctx.verifier.repositoryBlocked, false);
|
||||
for (const id of [code, maintenance]) {
|
||||
const row = ctx.store.get('SELECT state, result_json FROM batches WHERE id = ?', id);
|
||||
assert.equal(row.state, 'returned');
|
||||
assert.match(row.result_json, /static gate/);
|
||||
assert.equal(JSON.parse(row.result_json).repositorySync, undefined);
|
||||
}
|
||||
});
|
||||
|
||||
test('sync failure blocks new merge/check work and an operator reconciliation resumes queued work', async (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx, batch, user } = serverFixture(t, r);
|
||||
const queued = batch('maintenance');
|
||||
const approved = batch('code');
|
||||
ctx.store.run(`UPDATE batches SET state = 'open' WHERE id = ?`, approved);
|
||||
ctx.store.run(`UPDATE attempts SET state = 'approved' WHERE batch_id = ?`, approved);
|
||||
const checkId = Number(ctx.store.run(`INSERT INTO check_jobs (user_id, files_hash, files_json, state, created_at) VALUES (?, 'test', '{}', 'queued', ?)`, user, now()).lastInsertRowid);
|
||||
const local = r.commit(r.local, { 'src/local.cpp': 'int local_value;\n' });
|
||||
const remote = r.advance({ 'src/remote.cpp': 'int remote_value;\n' });
|
||||
ctx.verifier.pump();
|
||||
const error = JSON.parse(ctx.store.get('SELECT result_json FROM batches WHERE id = ?', queued).result_json).repositorySync.error;
|
||||
assert.ok(error.includes(local) && error.includes(remote));
|
||||
await ctx.merger.pump();
|
||||
ctx.checks.pump();
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', approved).state, 'open');
|
||||
assert.equal(ctx.store.get('SELECT state FROM check_jobs WHERE id = ?', checkId).state, 'queued');
|
||||
assert.equal(r.repo.head(), local);
|
||||
// Deliberate operator reconciliation, not an automatic reset/rebase by sync.
|
||||
r.git(r.producer, 'fetch', '-q', r.local, 'main');
|
||||
r.git(r.producer, '-c', 'user.name=Test', '-c', 'user.email=test@example.invalid', 'merge', '-q', '--no-edit', 'FETCH_HEAD');
|
||||
r.git(r.producer, 'push', '-q', 'origin', 'HEAD');
|
||||
ctx.verifier.pump();
|
||||
await drained(ctx.verifier);
|
||||
assert.equal(ctx.verifier.repositoryBlocked, false);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', queued).state, 'returned');
|
||||
ctx.checks.pump();
|
||||
for (let i = 0; i < 100 && ctx.checks.running; i++) await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
assert.equal(ctx.store.get('SELECT state FROM check_jobs WHERE id = ?', checkId).state, 'done');
|
||||
assert.equal(ctx.repo.show('HEAD', 'src/local.cpp'), 'int local_value;\n');
|
||||
assert.equal(ctx.repo.show('HEAD', 'src/remote.cpp'), 'int remote_value;\n');
|
||||
});
|
||||
|
||||
test('concurrent batches retain their snapshot until maintenance and code both finish', async (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx, batch } = serverFixture(t, r);
|
||||
ctx.config.verifyConcurrency = 2;
|
||||
const releases = [];
|
||||
const hold = () => new Promise((resolve) => releases.push(resolve));
|
||||
const code = batch('code');
|
||||
const maintenance = batch('maintenance');
|
||||
ctx.verifier.verifyBatch = async (b) => { await hold(); ctx.store.run(`UPDATE batches SET state = 'returned' WHERE id = ?`, b.id); };
|
||||
ctx.maintenance.run = async (id) => { await hold(); ctx.store.run(`UPDATE batches SET state = 'returned' WHERE id = ?`, id); };
|
||||
ctx.verifier.pump();
|
||||
assert.equal(ctx.verifier.running, 2);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', code).state, 'verifying');
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', maintenance).state, 'verifying');
|
||||
const remote = r.advance({ 'src/cohort.cpp': 'int cohort_value;\n' });
|
||||
const next = batch('maintenance');
|
||||
ctx.verifier.pump();
|
||||
releases.shift()();
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
assert.equal(ctx.verifier.running, 1);
|
||||
assert.equal(r.repo.head(), r.base);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', next).state, 'queued');
|
||||
releases.shift()();
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
assert.equal(r.repo.head(), remote);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', next).state, 'verifying');
|
||||
releases.shift()();
|
||||
await drained(ctx.verifier);
|
||||
});
|
||||
|
||||
test('FF reloads compile-command paths and invalidates old checks before maintenance uses new roots', async (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx, batch } = serverFixture(t, r);
|
||||
const id = batch('maintenance', r.base, { 'fresh/empty.h': '' });
|
||||
ctx.store.run(`INSERT INTO batch_checks (batch_id, head, check_json) VALUES (?, ?, '{}')`, id, r.base);
|
||||
const shared = ctx.project;
|
||||
const remote = r.advance({
|
||||
'data/decomp.json': { name: 'after', sourceDirs: ['newsrc'], includeDirs: ['fresh'], headerGlobs: ['fresh/**'], toolchain: 'newtool', compileCommands: 'data/new-commands.json' },
|
||||
'data/new-commands.json': { shapes: [['/c']], units: {} },
|
||||
'data/toolchain.lock.json': { newtool: { clSha256: 'new-digest' } },
|
||||
});
|
||||
ctx.verifier.pump();
|
||||
await drained(ctx.verifier);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', id).state, 'merged');
|
||||
assert.equal(ctx.repo.isAncestor(remote), true);
|
||||
assert.equal(ctx.repo.show('HEAD', 'fresh/empty.h'), '');
|
||||
assert.equal(ctx.project, shared);
|
||||
assert.equal(ctx.maintenance.project, shared);
|
||||
assert.equal(ctx.tasks.project, shared);
|
||||
assert.equal(ctx.project.name, 'after');
|
||||
assert.equal(ctx.project.obsoleteSetting, undefined);
|
||||
assert.deepEqual(ctx.project.compile, { shapes: [['/c']], units: {} });
|
||||
assert.equal(ctx.config.envDigest, 'new-digest');
|
||||
assert.equal(ctx.store.get('SELECT COUNT(*) AS n FROM batch_checks').n, 0);
|
||||
});
|
||||
|
||||
test('split web serves only published config and lock snapshots, even during an unpublished FF', async (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx } = serverFixture(t, r);
|
||||
ctx.verifier.syncRepository();
|
||||
const web = openServer(r.dataDir, { log: quiet });
|
||||
web.runsWorker = false;
|
||||
const server = createApp(web);
|
||||
await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
|
||||
r.cleanup.push(async () => { await new Promise((resolve) => server.close(resolve)); web.store.db.close(); });
|
||||
const info = async () => {
|
||||
const response = await fetch(`http://127.0.0.1:${server.address().port}/api/info`);
|
||||
assert.equal(response.status, 200);
|
||||
return response.json();
|
||||
};
|
||||
r.advance({ 'data/decomp.json': { name: 'published', toolchain: 'newtool' }, 'data/toolchain.lock.json': { newtool: { clSha256: 'published-digest' } } });
|
||||
r.repo.sync('publish'); // Repo moved, but the worker has not reloaded/published project data yet.
|
||||
let response = await info();
|
||||
assert.equal(response.project, 'before');
|
||||
assert.equal(response.toolchain, 'msvc2010');
|
||||
assert.deepEqual(response.toolchainLock, {});
|
||||
fs.writeFileSync(path.join(r.local, 'data/decomp.json'), '{partial worktree write');
|
||||
ctx.verifier.refreshProject();
|
||||
ctx.store.setMeta('repository_project_head', r.repo.head());
|
||||
response = await info();
|
||||
assert.equal(response.project, 'published');
|
||||
assert.equal(response.toolchain, 'newtool');
|
||||
assert.deepEqual(response.toolchainLock, { newtool: { clSha256: 'published-digest' } });
|
||||
assert.equal(web.config.envDigest, 'published-digest');
|
||||
});
|
||||
|
||||
test('startup refreshes an idle worker even with no queued batches', (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx } = serverFixture(t, r);
|
||||
const remote = r.advance({ 'data/decomp.json': { name: 'startup-refreshed' } });
|
||||
ctx.verifier.start();
|
||||
ctx.verifier.stop();
|
||||
assert.equal(r.repo.head(), remote);
|
||||
assert.equal(ctx.project.name, 'startup-refreshed');
|
||||
assert.equal(ctx.store.getMeta('repository_project_head'), remote);
|
||||
});
|
||||
|
||||
test('missing incoming compile commands block verification without publishing stale project data', async (t) => {
|
||||
const r = repositories(t);
|
||||
const { ctx, batch } = serverFixture(t, r);
|
||||
ctx.verifier.syncRepository();
|
||||
const id = batch('maintenance');
|
||||
const remote = r.advance({ 'data/decomp.json': { name: 'broken', compileCommands: 'data/commands.json' } });
|
||||
ctx.verifier.pump();
|
||||
assert.equal(r.repo.head(), remote);
|
||||
assert.equal(ctx.project.name, 'before');
|
||||
assert.equal(ctx.store.getMeta('repository_project_head'), r.base);
|
||||
const row = ctx.store.get('SELECT state, result_json FROM batches WHERE id = ?', id);
|
||||
assert.equal(row.state, 'queued');
|
||||
assert.match(JSON.parse(row.result_json).repositorySync.error, /missing compile commands: data\/commands\.json/);
|
||||
const repaired = r.advance({ 'data/commands.json': { shapes: [['/c']], units: {} } });
|
||||
ctx.verifier.pump();
|
||||
await drained(ctx.verifier);
|
||||
assert.equal(ctx.project.name, 'broken');
|
||||
assert.equal(ctx.store.getMeta('repository_project_head'), repaired);
|
||||
assert.equal(ctx.verifier.repositoryBlocked, false);
|
||||
assert.equal(ctx.store.get('SELECT state FROM batches WHERE id = ?', id).state, 'returned');
|
||||
});
|
||||
246
test/symbol-corrections.test.js
Normal file
246
test/symbol-corrections.test.js
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { Store, hashToken, now } from '../src/server/db.js';
|
||||
import { Accounts } from '../src/server/accounts.js';
|
||||
import { createApp, DEFAULT_CONFIG } from '../src/server/app.js';
|
||||
import { Workspace } from '../src/client/workspace.js';
|
||||
import { correctSymbol, refreshSymbols } from '../src/client/ops.js';
|
||||
|
||||
async function fixture(t) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'td-symbol-corrections-'));
|
||||
const store = new Store(path.join(dir, 'td.sqlite'));
|
||||
const accounts = new Accounts(store);
|
||||
for (const [id, current] of [['current', 1], ['previous', 0]]) {
|
||||
store.run('INSERT INTO versions (id, sha256, file, size, is_current, created_at) VALUES (?, ?, ?, 1, ?, ?)', id, 'fixture', 'unused.exe', current, now());
|
||||
}
|
||||
const symbols = ['?First@@YAXXZ', '?Second@@YAXXZ', null].map((decorated, i) => {
|
||||
const { symbol } = store.ensureSymbol({ kind: 'function', versionId: 'current', addr: 0x401000 + i * 0x100, size: 16, name: `Function${i}`, unit: 'src/functions.cpp' });
|
||||
store.run('UPDATE symbols SET decorated = ? WHERE id = ?', decorated, symbol.id);
|
||||
store.run('INSERT INTO symbol_addresses (symbol_id, version_id, addr, size) VALUES (?, ?, ?, ?)', symbol.id, 'previous', 0x501000 + i * 0x100, 12);
|
||||
store.run("INSERT INTO func_status (symbol_id, file, status, score, commit_hash, updated_at, confirmed, code_hash) VALUES (?, 'src/functions.cpp', 'equivalent', 0.9, 'original-commit', 123, 1, 'original-code')", symbol.id);
|
||||
store.run("INSERT INTO tasks (symbol_id, state, updated_at) VALUES (?, 'done', 123)", symbol.id);
|
||||
return symbol.id;
|
||||
});
|
||||
const user = (role) => {
|
||||
const id = Number(store.run("INSERT INTO users (name, role, status, created_at) VALUES (?, ?, 'active', ?)", role, role, now()).lastInsertRowid);
|
||||
const token = accounts.createApiToken(id, 'test').token;
|
||||
const session = `tds_${role}`;
|
||||
store.run('INSERT INTO sessions (user_id, token_hash, created_at, expires_at) VALUES (?, ?, ?, ?)', id, hashToken(session), now(), now() + 60_000);
|
||||
return { id, token, cookie: `td_session=${session}` };
|
||||
};
|
||||
const maintainer = user('maintainer');
|
||||
const contributor = user('contributor');
|
||||
const config = { ...DEFAULT_CONFIG, dataDir: dir };
|
||||
const server = createApp({ store, accounts, config, project: {}, runsWorker: false });
|
||||
await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
|
||||
t.after(async () => {
|
||||
await new Promise((resolve) => server.close(resolve));
|
||||
store.db.close();
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
const base = `http://127.0.0.1:${server.address().port}`;
|
||||
const post = async (id, body, auth = { token: maintainer.token }) => {
|
||||
const res = await fetch(`${base}/api/admin/symbols/${id}`, {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json', ...(auth.token && { authorization: `Bearer ${auth.token}` }), ...(auth.cookie && { cookie: auth.cookie }), ...(auth.csrf && { 'x-td-csrf': '1' }) },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
return { status: res.status, body: await res.json() };
|
||||
};
|
||||
const snapshot = async (version = 'current', etag) => {
|
||||
const res = await fetch(`${base}/api/symbols?version=${version}`, { headers: etag ? { 'if-none-match': etag } : {} });
|
||||
if (res.status === 304) return { status: 304 };
|
||||
const body = await res.json();
|
||||
return { status: res.status, etag: res.headers.get('etag'), symbols: new Map(body.symbols.map((row) => [row[0], Object.fromEntries(body.columns.map((c, i) => [c, row[i]]))])) };
|
||||
};
|
||||
const state = () => Object.fromEntries(['symbols', 'symbol_addresses', 'func_status', 'tasks'].map((table) => [table, store.all(`SELECT * FROM ${table} ORDER BY rowid`).map((row) => ({ ...row }))]));
|
||||
const events = () => store.all("SELECT user_id, type, data FROM events WHERE type = 'symbol_corrected' ORDER BY id").map((event) => ({ ...event, data: JSON.parse(event.data) }));
|
||||
return { store, symbols, maintainer, contributor, post, snapshot, state, events, base, dir };
|
||||
}
|
||||
|
||||
test('decorated corrections require maintainer authorization and session CSRF', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
const body = { decorated: '?Corrected@@YAXXZ', reason: 'Original export and disassembly identify this function.' };
|
||||
for (const [auth, status] of [[{}, 401], [{ token: 'td_invalid' }, 401], [{ token: f.contributor.token }, 403], [{ cookie: f.maintainer.cookie }, 403]]) {
|
||||
assert.equal((await f.post(f.symbols[0], body, auth)).status, status);
|
||||
}
|
||||
assert.deepEqual(f.state(), before);
|
||||
assert.deepEqual(f.events(), []);
|
||||
assert.equal((await f.post(f.symbols[0], body, { cookie: f.maintainer.cookie, csrf: true })).status, 200);
|
||||
});
|
||||
|
||||
test('decorated corrections reject malformed values without coercion, truncation or partial writes', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
for (const decorated of [null, 0, true, {}, [], '', ' ', ' ?Name@@YAXXZ', '?Name@@YAXXZ ', 'bad\nname', 'bad\n', 'bad\r', 'bad\u0000name', 'bad\u007fname', 'caf\u00e9', 'x'.repeat(2049)]) {
|
||||
assert.equal((await f.post(f.symbols[0], { name: 'Must not persist', size: 99, decorated, reason: 'Evidence' })).status, 400, JSON.stringify(decorated));
|
||||
}
|
||||
for (const body of [null, [], 'wrong', { decorated: '?New@@YAXXZ' }, { decorated: '?New@@YAXXZ', reason: {} }, { decorated: '?New@@YAXXZ', reason: ' ' }, { decorated: '?New@@YAXXZ', reason: 'x'.repeat(501) }, { decorated: '?New@@YAXXZ', reason: 'Evidence', addr: 123 }]) {
|
||||
assert.equal((await f.post(f.symbols[0], body)).status, 400, JSON.stringify(body));
|
||||
}
|
||||
assert.deepEqual(f.state(), before);
|
||||
assert.deepEqual(f.events(), []);
|
||||
});
|
||||
|
||||
test('duplicate decorated names roll back every requested correction', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
const result = await f.post(f.symbols[0], { decorated: '?Second@@YAXXZ', name: 'Must not persist', size: 99, reason: 'Evidence' });
|
||||
assert.equal(result.status, 409);
|
||||
assert.match(result.body.error, new RegExp(f.symbols[1]));
|
||||
assert.deepEqual(f.state(), before);
|
||||
assert.deepEqual(f.events(), []);
|
||||
});
|
||||
|
||||
test('audited decorated corrections preserve identity and confirmation and invalidate every version snapshot', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
const snapshots = await Promise.all(['current', 'previous'].map((v) => f.snapshot(v)));
|
||||
const reason = 'Original export and disassembly identify this function.';
|
||||
const result = await f.post(f.symbols[0], { decorated: '?Corrected@@YAXXZ', reason });
|
||||
assert.equal(result.status, 200, JSON.stringify(result.body));
|
||||
assert.deepEqual(result.body, { id: f.symbols[0], decorated: '?Corrected@@YAXXZ' });
|
||||
const expected = structuredClone(before);
|
||||
expected.symbols[0].decorated = '?Corrected@@YAXXZ';
|
||||
assert.deepEqual(f.state(), expected);
|
||||
assert.deepEqual(f.events(), [{ user_id: f.maintainer.id, type: 'symbol_corrected', data: { id: f.symbols[0], decorated: '?Corrected@@YAXXZ', old: { decorated: '?First@@YAXXZ' }, new: { decorated: '?Corrected@@YAXXZ' }, reason } }]);
|
||||
for (const [i, version] of ['current', 'previous'].entries()) {
|
||||
const snapshot = await f.snapshot(version, snapshots[i].etag);
|
||||
assert.equal(snapshot.status, 200);
|
||||
assert.notEqual(snapshot.etag, snapshots[i].etag);
|
||||
assert.equal(snapshot.symbols.get(f.symbols[0]).decorated, '?Corrected@@YAXXZ');
|
||||
assert.equal((await f.snapshot(version, snapshot.etag)).status, 304);
|
||||
}
|
||||
});
|
||||
|
||||
test('atomic decorated swaps audit both directions and preserve addresses and function state', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
const cached = await f.snapshot();
|
||||
const reason = 'Original call targets show the two recovered names were reversed.';
|
||||
const result = await f.post(f.symbols[0], { swapDecoratedWith: f.symbols[1], reason });
|
||||
assert.equal(result.status, 200, JSON.stringify(result.body));
|
||||
assert.deepEqual(result.body, { id: f.symbols[0], decorated: '?Second@@YAXXZ', swapped: { id: f.symbols[1], decorated: '?First@@YAXXZ' } });
|
||||
const expected = structuredClone(before);
|
||||
[expected.symbols[0].decorated, expected.symbols[1].decorated] = [expected.symbols[1].decorated, expected.symbols[0].decorated];
|
||||
assert.deepEqual(f.state(), expected);
|
||||
const events = f.events();
|
||||
assert.deepEqual(events.map((e) => [e.user_id, e.data.id, e.data.old.decorated, e.data.new.decorated, e.data.swapDecoratedWith, e.data.reason]), [
|
||||
[f.maintainer.id, f.symbols[0], '?First@@YAXXZ', '?Second@@YAXXZ', f.symbols[1], reason],
|
||||
[f.maintainer.id, f.symbols[1], '?Second@@YAXXZ', '?First@@YAXXZ', f.symbols[0], reason],
|
||||
]);
|
||||
const fresh = await f.snapshot('current', cached.etag);
|
||||
assert.equal(fresh.status, 200);
|
||||
assert.deepEqual(f.symbols.slice(0, 2).map((id) => fresh.symbols.get(id).decorated), ['?Second@@YAXXZ', '?First@@YAXXZ']);
|
||||
});
|
||||
|
||||
test('unsafe swaps reject missing, ambiguous or conflicting mappings transactionally', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
for (const [body, status] of [
|
||||
[{ swapDecoratedWith: f.symbols[0] }, 400],
|
||||
[{ swapDecoratedWith: f.symbols[2] }, 409],
|
||||
[{ swapDecoratedWith: 'fn_99999' }, 404],
|
||||
[{ swapDecoratedWith: null }, 400],
|
||||
[{ swapDecoratedWith: 123 }, 400],
|
||||
[{ swapDecoratedWith: f.symbols[1], decorated: '?Other@@YAXXZ' }, 400],
|
||||
[{ swapDecoratedWith: f.symbols[1], name: 'Other' }, 400],
|
||||
[{ swapDecoratedWith: f.symbols[1], size: 99 }, 400],
|
||||
[{ swapDecoratedWith: f.symbols[1], retire: true }, 400],
|
||||
]) {
|
||||
assert.equal((await f.post(f.symbols[0], { ...body, reason: 'Evidence' })).status, status, JSON.stringify(body));
|
||||
}
|
||||
assert.deepEqual(f.state(), before);
|
||||
f.store.run('UPDATE symbols SET decorated = ? WHERE id = ?', '?First@@YAXXZ', f.symbols[2]);
|
||||
const ambiguous = f.state();
|
||||
assert.equal((await f.post(f.symbols[0], { swapDecoratedWith: f.symbols[1], reason: 'Evidence' })).status, 409);
|
||||
assert.deepEqual(f.state(), ambiguous);
|
||||
assert.deepEqual(f.events(), []);
|
||||
});
|
||||
|
||||
test('decorated swaps reject function-vtable exchanges without audit or state changes', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const { symbol: vtable } = f.store.ensureSymbol({ kind: 'vtable', versionId: 'current', addr: 0x405000, size: 16, name: 'Widget vtable' });
|
||||
f.store.run('UPDATE symbols SET decorated = ? WHERE id = ?', '??_7Widget@@6B@', vtable.id);
|
||||
const before = f.state();
|
||||
const revision = f.store.getMeta('symbolCorrectionsRevision');
|
||||
const snapshot = await f.snapshot();
|
||||
const result = await f.post(f.symbols[0], { swapDecoratedWith: vtable.id, reason: 'Mistaken cross-kind correction' });
|
||||
assert.equal(result.status, 409);
|
||||
assert.deepEqual(f.state(), before);
|
||||
assert.deepEqual(f.events(), []);
|
||||
assert.equal(f.store.getMeta('symbolCorrectionsRevision'), revision);
|
||||
assert.equal((await f.snapshot('current', snapshot.etag)).status, 304);
|
||||
});
|
||||
|
||||
test('competing corrections cannot bind one decorated name to two IDs', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const results = await Promise.all(f.symbols.slice(0, 2).map((id) => f.post(id, { decorated: '?Contested@@YAXXZ', reason: 'Evidence' })));
|
||||
assert.deepEqual(results.map((r) => r.status).sort(), [200, 409]);
|
||||
const winner = results.find((r) => r.status === 200).body.id;
|
||||
assert.deepEqual(f.store.all('SELECT id FROM symbols WHERE decorated = ?', '?Contested@@YAXXZ').map((r) => r.id), [winner]);
|
||||
assert.deepEqual(f.events().map((e) => e.data.id), [winner]);
|
||||
});
|
||||
|
||||
test('an audit insert failure rolls back both halves of a decorated swap', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = f.state();
|
||||
f.store.db.exec(`CREATE TRIGGER fail_second_audit BEFORE INSERT ON events WHEN NEW.type = 'symbol_corrected' AND json_extract(NEW.data, '$.id') = '${f.symbols[1]}' BEGIN SELECT RAISE(ABORT, 'audit unavailable'); END`);
|
||||
assert.equal((await f.post(f.symbols[0], { swapDecoratedWith: f.symbols[1], reason: 'Evidence' })).status, 500);
|
||||
assert.deepEqual(f.state(), before);
|
||||
assert.deepEqual(f.events(), []);
|
||||
});
|
||||
|
||||
test('decorated correction cache invalidation reaches another web process database connection', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const otherStore = new Store(path.join(f.dir, 'td.sqlite'));
|
||||
const otherServer = createApp({ store: otherStore, accounts: new Accounts(otherStore), config: { ...DEFAULT_CONFIG, dataDir: f.dir }, project: {}, runsWorker: false });
|
||||
await new Promise((resolve) => otherServer.listen(0, '127.0.0.1', resolve));
|
||||
try {
|
||||
const url = `http://127.0.0.1:${otherServer.address().port}/api/symbols`;
|
||||
const cached = await fetch(url);
|
||||
const etag = cached.headers.get('etag');
|
||||
await cached.json();
|
||||
assert.equal((await f.post(f.symbols[0], { decorated: '?CrossProcess@@YAXXZ', reason: 'Evidence' })).status, 200);
|
||||
const changed = await fetch(url, { headers: { 'if-none-match': etag } });
|
||||
assert.equal(changed.status, 200);
|
||||
const snapshot = await changed.json();
|
||||
assert.equal(snapshot.symbols.find((row) => row[0] === f.symbols[0])[snapshot.columns.indexOf('decorated')], '?CrossProcess@@YAXXZ');
|
||||
} finally {
|
||||
await new Promise((resolve) => otherServer.close(resolve));
|
||||
otherStore.db.close();
|
||||
}
|
||||
});
|
||||
|
||||
test('decorated names retain C linkage and maximum-length spellings exactly', async (t) => {
|
||||
const f = await fixture(t);
|
||||
for (const decorated of ['_Function@4', '@Function@4', 'Function', '?'.repeat(2048)]) {
|
||||
const result = await f.post(f.symbols[2], { decorated, reason: 'Original symbol spelling' });
|
||||
assert.equal(result.status, 200, JSON.stringify(result.body));
|
||||
assert.equal(f.store.get('SELECT decorated FROM symbols WHERE id = ?', f.symbols[2]).decorated, decorated);
|
||||
}
|
||||
});
|
||||
|
||||
test('client corrections and swaps replace cached and persisted decorated-name lookups', async (t) => {
|
||||
const f = await fixture(t);
|
||||
const ws = new Workspace(path.join(f.dir, 'workspace'));
|
||||
ws.config = { server: f.base, token: f.maintainer.token };
|
||||
ws.save();
|
||||
await refreshSymbols(ws);
|
||||
assert.equal(ws.findSymbol('?First@@YAXXZ').id, f.symbols[0]);
|
||||
await correctSymbol(ws, f.symbols[0], { decorated: '?ClientCorrected@@YAXXZ', reason: 'Evidence' });
|
||||
assert.equal(ws.findSymbol('?First@@YAXXZ'), null);
|
||||
assert.equal(ws.nameIndex.get('?ClientCorrected@@YAXXZ'), f.symbols[0]);
|
||||
assert.equal(new Workspace(ws.dir).findSymbol('?ClientCorrected@@YAXXZ').id, f.symbols[0]);
|
||||
await correctSymbol(ws, f.symbols[0], { swapDecoratedWith: f.symbols[1], reason: 'Evidence' });
|
||||
assert.equal(ws.nameIndex.get('?ClientCorrected@@YAXXZ'), f.symbols[1]);
|
||||
assert.equal(ws.nameIndex.get('?Second@@YAXXZ'), f.symbols[0]);
|
||||
const persisted = fs.readFileSync(ws.symbolsFile, 'utf8');
|
||||
await assert.rejects(correctSymbol(ws, f.symbols[0], { decorated: '?ClientCorrected@@YAXXZ', reason: 'Evidence' }), (error) => error.status === 409);
|
||||
assert.equal(fs.readFileSync(ws.symbolsFile, 'utf8'), persisted);
|
||||
assert.equal(new Workspace(ws.dir).findSymbol('?ClientCorrected@@YAXXZ').id, f.symbols[1]);
|
||||
});
|
||||
122
test/update.test.js
Normal file
122
test/update.test.js
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import http from 'node:http';
|
||||
import crypto from 'node:crypto';
|
||||
import { execFile } from 'node:child_process';
|
||||
import { promisify } from 'node:util';
|
||||
import { gzipSync } from 'node:zlib';
|
||||
import { createApp, DEFAULT_CONFIG } from '../src/server/app.js';
|
||||
|
||||
const exec = promisify(execFile);
|
||||
const ASSET = process.platform === 'win32' ? 'td-windows-x64.exe' : 'td-linux-x64';
|
||||
const hash = (data) => crypto.createHash('sha256').update(data).digest('hex');
|
||||
const installed = Buffer.from('installed td build');
|
||||
const published = Buffer.from('newly published td build');
|
||||
|
||||
async function listen(t, server, dir) {
|
||||
await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve));
|
||||
t.after(async () => {
|
||||
await new Promise((resolve) => server.close(resolve));
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
return `http://127.0.0.1:${server.address().port}`;
|
||||
}
|
||||
|
||||
async function runUpdater(dir, server) {
|
||||
// Isolate the SEA gate and executable/home paths, not the updater's HTTP, hashing or writes.
|
||||
const script = `
|
||||
import { createRequire } from 'node:module';
|
||||
import path from 'node:path';
|
||||
const [dir, server, updater] = process.argv.slice(1);
|
||||
const require = createRequire(import.meta.url);
|
||||
require('node:sea').isSea = () => true;
|
||||
require('node:os').homedir = () => dir;
|
||||
process.execPath = path.join(dir, 'td');
|
||||
const { maybeUpdate } = await import(updater);
|
||||
const logs = [];
|
||||
const result = await maybeUpdate({ server, force: true, log: (message) => logs.push(message) });
|
||||
console.log(JSON.stringify({ result, logs }));
|
||||
`;
|
||||
const { stdout } = await exec(process.execPath, ['--input-type=module', '-e', script, dir, server, new URL('../src/client/update.js', import.meta.url).href], {
|
||||
env: { ...process.env, TD_NO_UPDATE: '0' },
|
||||
timeout: 15_000,
|
||||
});
|
||||
return JSON.parse(stdout);
|
||||
}
|
||||
|
||||
async function cachedDownloads(t, corrupt = false) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'td-update-'));
|
||||
const exe = path.join(dir, 'td');
|
||||
fs.writeFileSync(exe, installed);
|
||||
const requests = [];
|
||||
const server = http.createServer((req, res) => {
|
||||
requests.push(req.url);
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
if (url.pathname === '/download/SHA256SUMS') {
|
||||
// A cache serves the old manifest unless the client requests revalidation.
|
||||
const revalidate = /(?:no-cache|no-store|max-age=0)/.test(req.headers['cache-control'] ?? '');
|
||||
return res.end(`${hash(revalidate ? published : installed)} ${ASSET}\n`);
|
||||
}
|
||||
if (url.pathname === `/download/${ASSET}`) {
|
||||
// The mutable URL is stale; only a new content-keyed URL reaches the new build.
|
||||
const keyed = url.searchParams.get('sha256') === hash(published);
|
||||
return res.end(keyed && !corrupt ? published : installed);
|
||||
}
|
||||
res.writeHead(404);
|
||||
res.end();
|
||||
});
|
||||
return { dir, exe, requests, base: await listen(t, server, dir) };
|
||||
}
|
||||
|
||||
test('self-update revalidates the manifest and bypasses a stale unversioned binary', async (t) => {
|
||||
const f = await cachedDownloads(t);
|
||||
const { result } = await runUpdater(f.dir, f.base);
|
||||
assert.equal(typeof result, 'string', 'the downloaded update was installed');
|
||||
assert.deepEqual(fs.readFileSync(f.exe), published);
|
||||
assert.deepEqual(f.requests, ['/download/SHA256SUMS', `/download/${ASSET}?sha256=${hash(published)}`]);
|
||||
assert.equal(fs.existsSync(f.exe + '.new'), false);
|
||||
});
|
||||
|
||||
test('self-update rejects mismatched bytes even at the expected SHA256 URL', async (t) => {
|
||||
const f = await cachedDownloads(t, true);
|
||||
const { result } = await runUpdater(f.dir, f.base);
|
||||
assert.deepEqual(f.requests, ['/download/SHA256SUMS', `/download/${ASSET}?sha256=${hash(published)}`]);
|
||||
assert.equal(result, null);
|
||||
assert.deepEqual(fs.readFileSync(f.exe), installed, 'checksum failure must preserve the installed executable');
|
||||
assert.equal(fs.existsSync(f.exe + '.new'), false);
|
||||
assert.equal(fs.existsSync(f.exe + '.old'), false);
|
||||
});
|
||||
|
||||
test('mutable downloads are not stored by caches for GET or HEAD, including gzip and missing assets', async (t) => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'td-download-'));
|
||||
const downloads = path.join(dir, 'downloads');
|
||||
fs.mkdirSync(downloads);
|
||||
fs.writeFileSync(path.join(downloads, ASSET), published);
|
||||
const zipped = gzipSync(published);
|
||||
fs.writeFileSync(path.join(downloads, ASSET + '.gz'), zipped);
|
||||
const manifest = `${hash(published)} ${ASSET}\n`;
|
||||
fs.writeFileSync(path.join(downloads, 'SHA256SUMS'), manifest);
|
||||
const server = createApp({ config: { ...DEFAULT_CONFIG, dataDir: dir }, project: {}, runsWorker: false });
|
||||
const base = await listen(t, server, dir);
|
||||
for (const method of ['GET', 'HEAD']) {
|
||||
for (const [asset, encoding, body, length] of [
|
||||
[ASSET, 'gzip', published, zipped.length],
|
||||
[`${ASSET}?sha256=${hash(published)}`, 'identity', published, published.length],
|
||||
['SHA256SUMS', 'identity', Buffer.from(manifest), Buffer.byteLength(manifest)],
|
||||
]) {
|
||||
const res = await fetch(`${base}/download/${asset}`, { method, headers: { 'accept-encoding': encoding } });
|
||||
assert.equal(res.status, 200);
|
||||
assert.equal(res.headers.get('cache-control'), 'no-store');
|
||||
assert.equal(res.headers.get('content-encoding'), encoding === 'gzip' ? 'gzip' : null);
|
||||
assert.equal(Number(res.headers.get('content-length')), length);
|
||||
assert.deepEqual(Buffer.from(await res.arrayBuffer()), method === 'HEAD' ? Buffer.alloc(0) : body);
|
||||
}
|
||||
const missing = await fetch(`${base}/download/missing`, { method });
|
||||
assert.equal(missing.status, 404);
|
||||
assert.equal(missing.headers.get('cache-control'), 'no-store');
|
||||
await missing.arrayBuffer();
|
||||
}
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue