UAsset Reference MCP: The Bottleneck Was Never the Language

Indexing a Unity project means parsing YAML looking for {fileID, guid, type} records. It is the most obviously compute-heavy step in the pipeline, which made it the obvious candidate for a native rewrite. So a Rust extractor was built, it produced byte-identical output, and it was benchmarked properly.

It parsed about 2.9x faster and delivered results about 38x slower. Then measuring the real project moved the bottleneck somewhere neither implementation touched.

The Obvious Optimization Was Obvious for the Wrong Reason

The proof of concept lives in rust/uasset-ref-extractor, parses with Rayon, and is invoked as a subprocess over NDJSON by a benchmark script. It was never wired into product code.

Against the same 2,000 prefab records, with every mode producing 2,000 outputs, 1,998 edges, zero unresolved references, and exact output parity:

The parse itself went from 3.146 ms to 1.074 ms. Delivering that parse cost roughly 118 ms of process startup plus NDJSON serialization, which the in-process Node path never pays.

That is not a result about Rust. Rust parsed faster, exactly as expected. It is a result about a subprocess being the wrong delivery shape for work measured in single-digit milliseconds. An in-process binding would erase the overhead entirely — which turns the interesting question into a different one. If the parse were free, how much would indexing actually save?

The Fixture That Said 3%, and the Fixture That Said 76%

This is the part worth telling straight, because both attempts to answer that question were wrong.

The repository’s own benchmark fixture generated 2,000 Unity-shaped prefab assets of about 100 bytes each, carrying one reference apiece. Measured against it at concurrency 8, total indexing was roughly 110 ms and the extraction phase roughly 40 ms, most of it file reads — the parse itself was around 3 ms. Removing parsing entirely would have saved under 3% of runtime.

That number was used to shelve the Rust port. It was also measuring the wrong thing: 100-byte single-reference files measure per-file overhead, not parse throughput. The fixture answered a question about open/read/close, and the question was about parsing.

So a denser fixture was built for the write-path benchmark, generating about 139 references per asset. Against that corpus, SQLite writes looked like roughly 76% of the work — a completely different conclusion, pointing at a completely different phase.

The real project has 0.89 references per asset. The second fixture was around 150x too dense, wrong in the opposite direction from the first.

Both synthetic corpora encoded an assumption about the shape of a Unity project, and both assumptions were wrong. Neither was a rhetorical setup; they were two real mistakes made in sequence, and the only thing that settled the question was measuring a project that actually exists.

What the Real Project Showed

The profile was captured against a real Unity project of 32,579 indexed assets and 28,842 edges, on macOS with 8 logical CPUs. The project’s own .asset-memory/index.db was never written to; every run used a scratch database.

At concurrency 8, medians over four runs with about 4% total spread:

Reference parsing alone, measured across the project’s 1,827 YAML assets totalling 128.8 MB, takes 485 ms serial — about 266 MB/s. Against a ~5,000 ms total, that is roughly 2% once parallelized.

The Node parser is not fast in absolute terms. It is simply not the bottleneck, and a 2.9x faster parser applied to 2% of runtime is a rounding error with a cross-compilation matrix attached.

Most of the Dominant Phase Was Not Filesystem Work

Scan was 60%, so scan is where the time was. Decomposing it is where the actual finding is.

Pure directory traversal across the project measured around 160 ms. Reading every .meta file measured around 400 ms. That is roughly 560 ms of real filesystem work inside a phase that reports 1,600-3,000 ms.

Around 65-80% of the dominant phase was not filesystem I/O at all. It was promise-scheduling overhead — the cost of concurrency pools nested per directory, one pool spawned per level of recursion, each one coordinating work that was mostly waiting anyway.

The bottleneck was coordination structure, in a language whose scheduler you do not rewrite by switching languages.

What Actually Got Faster

v0.4.0 ships bounded concurrency for scanning and reference extraction, with SQLite writes still transactional and serialized. On the repository fixture at concurrency 8 that measured about 2.7x total speedup with identical output.

Correctness was the gate, not the speedup. A full index at concurrency 1 and at concurrency 8 produces byte-identical assets and edges tables, verified by hashing the fully sorted tables rather than by comparing row counts.

The concurrency cap itself turned out to be the largest remaining measured win, and the reason is the same finding restated:

DEFAULT_INDEX_CONCURRENCY = Math.min(8, Math.max(2, availableParallelism()))

That is a CPU-count heuristic applied to a workload that is roughly 93% filesystem I/O. Workers spend their time waiting on syscalls, not computing, so throughput keeps improving well past the core count. On the real project, with edge counts identical across every run:

The default cap did not change in this release. The gains above 16 carry 8-9% coefficient of variation, close to their own effect size, and every number here comes from one macOS machine. A spinning disk, a network share, or a virus scanner walking the same directories could reorder the phases entirely. So --concurrency <n> ships as a flag to tune per machine, and the default stays where it can be defended.

UV_THREADPOOL_SIZE was checked and is not the lever: raising it from 4 to 32 moved scan about 8% and total about 5%, near the noise floor.

The Write Path Was Measured Too, and Mostly Left Alone

The same benchmark ran nine variants of the write configuration against a dense corpus, interleaved across rounds with a discarded warm-up so drift hit every variant equally, and asserting row-count parity so a variant that lost rows could not read as a win.

WITHOUT ROWID on edges measured consistently slower at 0.92x — the key is four wide TEXT columns, so the b-tree rows are large. synchronous = NORMAL, foreign_keys = OFF, and a larger cache_size all landed within noise, and the first two buy nothing measurable in exchange for durability and integrity.

One finding holds independently of any timing: idx_edges_from is redundant. edges is keyed PRIMARY KEY (from_guid, to_guid, ref_kind, context), so from_guid is already the leftmost column of the primary key index, and the extra index costs write time and disk for no read benefit.

None of those write-path changes shipped in v0.4.0. They are measured candidates with their own validation still ahead, and the write phase is ~6% of runtime, so the ceiling on all of them together is small. Saying that plainly is cheaper than shipping a change whose benefit cannot be distinguished from noise.

The Decision Is Written Down So the Argument Happens Once

Keeping extraction in TypeScript is recorded as ADR 0020, with two conditions for revisiting, in order:

1. A phase breakdown from a real Unity project — not a synthetic fixture — showing that parsing is a materially larger share than measured here.
2. An in-process binding rather than a subprocess. The ~118 ms overhead is structural and cannot be tuned away.

If both ever hold, native extraction ships as prebuilt per-platform binaries selected at install time, never as a source build that requires a toolchain on a user’s machine.

The Rust crate was kept rather than deleted, on an unmerged branch. It is not built, not published, and not on any code path a user reaches — but the measurements in this post are only reproducible because it still exists, and a decision defended by numbers should stay re-runnable.

The lesson is not that Rust is slow, because it was not. It is that “the parser is the bottleneck” was a hypothesis that felt too obvious to test, and the two fixtures built to test it were each shaped by an assumption about Unity projects that the actual Unity project did not share.