Which settings should we standardise across the firm?
Temperature and top-p control how the model chooses each word, not how good it is. Standardise a small set of named configurations for extraction, summarisation and drafting — and pin the version, schema and retrieval settings too.
What the dials are doing underneath
At each step a model computes a probability for every word it could produce next. Temperature rescales that distribution before a choice is made: low temperature sharpens it toward the most likely options, high temperature flattens it and lets unlikely words through. Top-p, usually called nucleus sampling, throws away the tail — it keeps the smallest set of candidate words whose probabilities add up to the chosen threshold, and samples from those. Top-k does something cruder, keeping a fixed number of candidates. At temperature zero, with the tail discarded, the model takes the single most likely word at every step. That is deterministic at the level of the settings, and it is why people describe it as the same answer every time. It is not. The computation runs across many operations in parallel on hardware where the order of arithmetic is not fixed, so the same input can produce slightly different numbers on different machines, in different batches, or between a full-precision model and a quantised version of it. The practical consequence for a firm is uncomfortable but important: you cannot claim reproducibility from a temperature setting. You can claim consistency of configuration, and you should record it — but the evidence for an output is the output itself, stored, not the settings that produced it.
Why zero is not automatically right for legal work
Extraction wants zero. Given a fixed schema and the same document, you want the same dates and parties pulled out every time, and any drift is a defect you have to chase. Drafting and analysis are different. At zero, the model always takes the most likely continuation. On long outputs that shows up as flat, repetitive prose, a tendency to fall into a loop, and the same phrasing every time you ask. It also removes the model's ability to offer you a second angle: you cannot ask the same question twice and compare, because you will get the same text twice, and a hedge that happens to be wrong is now repeated identically in every draft. There is a useful practice hiding in that. For a genuinely uncertain question — does this clause conflict with that one — run the task two or three times at a low non-zero temperature and compare the answers. Where the model agrees with itself, the position is probably robust; where it diverges, either the documents are ambiguous or the retrieved context was too thin to support an answer. Divergence is a triage signal, and it is one of the few things sampling settings are good for. What you should not do is turn the temperature up on a reasoning model in the hope of a more creative answer, which mostly buys you a longer walk to the same place.
Settings by task, not by firm
A single number for the whole firm is the wrong goal. The right goal is a small set of named configurations, each tied to a task, written down once and used everywhere. Extraction and classification: temperature at or near zero, a strict output schema, no latitude in formatting. Summarisation of a bundle: also low, with an instruction to cite the source line for every date and party. Drafting from precedent: slightly higher, enough to avoid the looped-flat problem while staying well short of anything that reads as inventive. Analysis and second opinions: low, and on a reasoning model, lower still, because raising it lengthens the trace without improving the conclusion. Then name them. Extraction (strict), Summary (cited), Draft (standard), Analysis (low). Put them in the AI policy with the exact numbers, the output schema and the model version each is approved for. A configuration table is short, boring and one of the most useful single pages you will write, because it turns an argument about settings into a question with an answer.
Everything else you have to pin
Temperature is one dial among many, and standardising it while leaving the rest to individual taste accomplishes very little. The settings that change outcomes as much or more are: the system prompt, the output schema, the model version, the quantisation of that version, the retrieval configuration — how many passages, how they are split, whether results are reranked — the maximum context and truncation policy, the maximum output length, and any stop sequences. Two versions of the same model under the same temperature, one at full precision and one compressed to run on a workstation, are not the same system and will not behave identically. Neither are two retrieval settings that return four passages and twelve. If a fee-earner pastes a different prompt into the same tool, you have a new configuration whether or not anybody wrote it down. So the register is the control: a named configuration, the model version and build, the prompt, the retrieval settings, the date approved and the person who approved it. When an output goes into a matter file, the file note references the configuration by name. That is what makes a result reproducible in the only sense a supervisor needs — being able to say what produced it.
Standardising without freezing
Settings need an owner. One person — usually the COLP working with whoever runs IT — approves changes to the register, and every change is dated with a note saying why. Behaviour that motivated the change should be recorded, even briefly: the extraction configuration moved because dates were drifting in the schedule-only documents. Two disciplines keep the register honest. The first is that a settings change is a configuration change, and a configuration change is a reason to re-run your evaluation tasks, because you have changed the system you tested. The second is that the register is not a moat: fee-earners will find a task it does not cover, and the answer is to add a configuration with evidence, not to invent one on the fly and keep it quiet. Where you are uncertain, say so. We do not have a defensible view about the ideal temperature for a drafting task in your practice; it depends on your precedent and your readers, and it is the sort of thing two versions of the same test will settle for your firm. What we would defend is that the number should be chosen deliberately, written down, and the same for everyone doing the same task.
What to watch
- Temperature zero treated as a guarantee of identical output, when batching, hardware and quantisation all move the numbers
- Turning the temperature up to get a second opinion instead of running the same task twice and comparing the answers
- Standardising temperature while prompts, schemas and retrieval settings stay in individual heads
- A configuration table that names a model but not its version or build, so nothing can be reproduced
- Extraction run at anything above zero without a strict schema to catch the drift
- Settings visible in the tool but absent from the file note, so a supervisor cannot tell what produced the output
Write one page: four named configurations — extraction, summarisation, drafting, analysis — each with a temperature, top-p, output schema, model version and retrieval setting. Approve it once, date it, give it an owner, and put it in the AI policy. Configure the tools so the defaults match the register rather than the vendor's defaults, and make referencing the configuration by name part of how outputs are recorded in a matter file. Then treat any change to the register as a trigger to re-run your evaluation tasks. Do not expect the settings to be right forever; expect them to be written down and changed deliberately.