Can we run a serious model on hardware we already own?
Quantisation compresses the model's numbers so it fits in less memory and runs faster. Fluent drafting survives it better than multi-step reasoning, and abstention is the behaviour most likely to soften, so test the exact build you intend to deploy.
What compression actually changes
A model is mostly numbers. Each weight is a value the model learned during training, normally stored at sixteen bits of precision. Quantisation stores those values at lower precision — eight bits, four bits, and in aggressive builds fewer still — so the file is smaller and moving it around memory, which is the real bottleneck when a model generates text, is faster. The mechanism is rounding. Instead of holding a precise value, the model holds the nearest value from a small set, and the small errors accumulate across the enormous number of multiplications that produce one word. Different methods round differently. Post-training quantisation compresses a finished model, which is quick and widely used. Quantisation-aware training builds the rounding into training so the model can adapt to it, which generally preserves more quality and costs more to produce. Beyond the weights, the other variables are the group size — how many weights share a scale — and whether the attention cache and activations are also compressed. The cache matters more than people expect, because a long context multiplies it. This is why 'four bit' tells you very little on its own. Two builds with the same nominal precision can differ materially depending on method, group size and tooling.
What you lose, and where
Quality loss from quantisation is not a uniform tax. It concentrates, and it concentrates in exactly the places legal work is hard. Long chains of dependent reasoning degrade first, because each step's error feeds the next. Arithmetic and date logic suffer, which matters when a chronology is the deliverable. Unusual tokens suffer most — party names, case names, statute references, unusual spellings, non-Latin scripts — because the compressed model has less room to distinguish rare patterns. Instruction following on less common formats gets shakier, so a model may hold a schema for eight rows and lose it on the ninth. And abstention, in our assessment, is among the first behaviours to soften: a refusal is a rarer pattern than an answer, so it has less support in the weights to begin with, and precision loss erodes thin support before thick support. Treat that as a hypothesis to test on your own tasks rather than a law of nature. Meanwhile, drafting and summarisation — the tasks most fee-earners reach for first — survive compression noticeably better, because they reward fluency and general pattern rather than exactness. A compressed model can be perfectly serviceable for extracting dates into a schema and unreliable for comparing three clauses and telling you which is weakest.
The hardware arithmetic nobody does first
Firms tend to buy on the parameter count in the model's name and discover the constraint afterwards. The constraint is memory. Weights, attention cache and working overhead all have to fit in the memory you have, with headroom for concurrent users. Compress the weights by a factor of four and the weight memory falls by roughly the same factor, which is the whole appeal — but the cache does not shrink unless you compress it separately, and it grows with context length and with the number of people using the system at once. Work it out as a sanity check rather than a specification: parameters times bits per weight, divided by eight, gives the weight memory in bytes, before cache and overhead are added. Then compare that with the tooling's published requirement, which is the number to trust, and with the machine you actually have. If the model does not fit, the outcome is not graceful: it either refuses to load or spills into slower memory and drops to a speed nobody will use. Throughput is the second half. Generation speed is bounded by memory bandwidth, and a single workstation serves one or two users comfortably, not a firm of forty. Before committing, test it with two people working at once rather than one.
Test the build you will run, not the model you read about
Your evaluation of a hosted, full-precision version does not transfer to a compressed local build. They are different systems, and the differences show up in the tasks that matter. Re-run the same task set on the exact build you intend to deploy: the extraction into schema, the chronology with dates checked, the quote-fidelity test, the unanswerable questions, and one multi-step analysis. The abstention set is the most important of these, because it is the behaviour least visible when it degrades — a model that has stopped refusing still produces fluent, plausible output, and nobody notices until a fabricated clause reaches a file note. Record what you tested precisely: repository, model revision, quantisation method, group size, file hash and the tooling version. Community builds of open models vary widely in quality and provenance, and two files with the same name from different uploaders are not the same artefact. Where the quantisation method is undisclosed, treat the build as untested regardless of who published it. And re-test when the base model or the tooling updates, because a compressed build is a combination of both.
Choosing what to shrink
Decide the tasks first, then the smallest model and the most aggressive compression that passes them. Doing it the other way round — buying a machine, then looking for something that fits — reliably lands firms on a model that is good at drafting and unreliable at the analysis they actually wanted. Prefer methods that are documented and reproducible rather than a file with an anonymous name. Keep the build register entry alongside the evaluation record, and reference it in the configuration register, so a supervisor can tell which artefact produced an output. And think about the honest alternatives before you shrink a model below what the task needs: one modest server shared across the firm, a private hosted tenancy where the provider runs the full-precision model in an isolated environment, or simply narrowing the task to the part the small model handles well and leaving the rest manual. Where a firm's genuinely difficult analysis task needs a large model, we do not think a heavily quantised build on a desktop is a substitute, and we would rather say that plainly than sell the compromise as a win. The right answer for that task may be that it stays with a lawyer.
What to watch
- Hardware sized from the parameter count in the model's name without allowing memory for the attention cache and concurrent users
- Bench behaviour on a hosted full-precision model assumed to hold for the four-bit build on your workstation
- Anonymous community quantised builds with no record of method, group size or revision
- Abstention and instruction-following quietly degrading while drafting still looks perfectly fluent
- Long-context work run on a compressed build whose attention cache was also compressed, with no re-test
- Buying a server for a task that a smaller model on existing machines, or a narrow manual process, would do better
Start with the task list, not the hardware. For each candidate local build, work out the memory needed for weights plus attention cache plus overhead, compare it with the tooling's published requirement, and test throughput with two concurrent users rather than one. Then run your evaluation set on the exact artefact — same repository, revision, quantisation method and group size — with the abstention questions first, because that is the behaviour compression erodes most quietly. Record the build details and file hash in the configuration register. If the analysis task needs full precision, put one shared server or a private hosted tenancy on the table before you accept a compressed model that fails the work.