Private deployment sounds like "move the model behind the firewall." The time sink is almost never the model. Five mistakes, from our own log.
One, optimistic memory estimates. Model weights are not the whole story — you also pay for the KV cache, concurrency and context length. We now size hardware by target concurrency × average context, with a load test. Not by a theoretical number on a spec sheet.
Two, no request queue. Concurrency arrives, memory dies. Queueing and a degradation strategy have to exist before launch: overflow waits in line instead of taking the service down.
Three, context gets truncated silently. The long-document trap: the model does not error, but a middle section quietly disappeared, and the answer sounds reasonable while being wrong. Explicit chunking and citation tagging on the input side fix this.
Four, no model versioning. One fine-tune round, one base-model swap, and behaviour shifts — with nobody able to say what is running in production. Model artefacts need version numbers and change records, exactly like code.
Five, no evaluation baseline. Without a benchmark defined before launch, quality judgements run on vibes afterwards. Even 100 hand-labelled samples give later optimisation somewhere to aim.
None of these are hard. The hard part is deciding them before launch. Model capability sets the ceiling; engineering detail decides whether anything runs at all.
