LinkedIn Post Ideas for ML Engineers

Our model hit 96% accuracy in the notebook. In production, it was useless within a week. The notebook trained on a clean, static dataset. Production features came from a real-time pipeline with a subtly different preprocessing step — a normalization constant computed on training data that drifted as live traffic patterns shifted. Predictions degraded silently. No errors, no crashes, just slowly worse recommendations that nobody flagged until conversion dipped and someone finally checked. The fix wasn't a better model. It was matching training and serving preprocessing exactly, with a shared code path instead of two separate implementations that quietly diverged. Training-serving skew doesn't announce itself. It just erodes results until someone goes looking.

You probably don't need fine-tuning. You need better retrieval. I've said this in three project kickoffs this year alone. Fine-tuning is expensive, slow to iterate on, and locks your knowledge into model weights that go stale the moment your data changes. For most business use cases — answering questions about a company's docs, summarizing internal content — a well-tuned RAG pipeline gets you 90% of the way there in a fraction of the time. My actual decision rule: fine-tune only when the task needs a stylistic or behavioral shift the prompt can't achieve, not when it's really a knowledge-access problem in disguise. Most teams reach for fine-tuning because it sounds more serious. Retrieval is usually the more serious engineering choice.

How we detect data drift before the business notices bad predictions, not after. We track distribution shifts on every input feature weekly, comparing against a rolling baseline rather than a fixed one from training time — fixed baselines go stale and start crying wolf. When labels lag, which is most of the time in our case, we monitor proxy metrics: prediction confidence distributions, and a small human-reviewed sample each week that acts as an early warning before the real label ever arrives. Alert thresholds took real tuning — our first version paged someone for noise so often it got ignored within a month. The system that matters is the one nobody has learned to dismiss yet.

We cut inference costs 70% without losing accuracy. Exact stack, no hand-waving. Quantization to int8: roughly 35% of the savings, with accuracy loss under 0.3% on our eval set. Dynamic batching, grouping requests instead of processing one at a time: another 20%. Right-sizing our instance type after realizing we'd over-provisioned GPU memory for a model that didn't need it: 10%. Response caching for repeated queries, which turned out to be about 15% of our traffic: the remaining 5%, plus a latency win as a bonus. Total: 70% cost reduction, accuracy essentially unchanged on our held-out set. None of these required a smaller or worse model. They required actually profiling where the money was going.

Our model aced every offline metric. Precision, recall, F1 — all strong. Users rejected it anyway. The predictions were accurate but arrived after the moment they'd have been useful — our pipeline took 40 minutes to score a batch, and by the time results reached the team, the decision window had closed. We'd also shipped zero explanation alongside each prediction. Users didn't trust a number with no reasoning attached, no matter how accurate it turned out to be historically. We rebuilt for latency first, cutting scoring time to under 2 minutes, and added a simple feature-importance explanation to each output. Adoption changed completely, with the same underlying model. Evaluation suites measure the model. They don't measure whether anyone will actually use it.

Five MLOps investments I made too late, each one after the pain that should have prompted it earlier. Experiment tracking, added only after we couldn't reproduce a model from six weeks earlier that a stakeholder suddenly wanted explained. A feature store, built only after discovering three separate teams had built three slightly different versions of the same customer-tenure feature. Automated rollback, added only after a bad model sat in production for two days because nobody had an easy way to revert it. Data validation at the pipeline entry point, added only after a schema change upstream silently broke training for a week. On-call ownership for models, assigned only after an incident with no clear owner. Every one of these would have cost a fraction as much built proactively.

LLM evals are the new unit tests. Most teams shipping LLM features still have none, and I include a past version of my own team in that. We shipped a summarization feature with zero systematic evaluation beyond 'it looked good in the demo.' Three weeks later, a prompt change silently broke behavior on a whole class of longer documents, and we only found out from a support ticket. We now maintain a golden set of about 200 real examples with expected behavior ranges, run automatically on every prompt or model change, plus periodic human judgment sampling for the parts a rubric can't fully capture. If your LLM feature has no eval suite, you're not shipping a product. You're shipping a demo that occasionally works in production.

Debugging a 3am inference latency spike. Full forensic trail, no shortcuts skipped. The p99 latency graph showed a clean, sudden jump at 2:47am — not a gradual creep, which usually points to something discrete rather than load-related. First suspect: the deploy from that evening. Rolled it back. Latency stayed elevated. Not that. Second suspect: infrastructure, checked GPU utilization and memory — both normal. Not that either. Found it an hour in: a tokenizer edge case. A batch of inputs containing an unusual Unicode sequence was hitting a slow fallback path in the tokenizer library, adding milliseconds per token on exactly those requests, and that day's traffic happened to include more of them than usual. Fixed with input sanitization before tokenization. Three hours, one very specific root cause.

Seven questions I ask before training any model from scratch, because most of the time the answer changes the whole plan. 1. Would a simple heuristic or rule actually solve this well enough? 2. Have we audited the data for quality, not just quantity? 3. What's our actual serving latency and cost budget? 4. Who owns this model's maintenance a year from now, specifically? 5. What's our baseline — has anyone tried the boring approach first? 6. How will we detect when this model's performance degrades? 7. What's the cost of a wrong prediction, and does that change the bar we need to hit? Question 1 alone has killed more proposed ML projects on my team than every other question combined.

ML folks: what's the dumbest root cause you've ever found in production? Mine still makes me laugh. A recommendation model's quality dropped every Saturday and Sunday, reliably, for months before anyone connected the pattern. Root cause: a feature counting 'days since last purchase' was computed using business days in one part of the pipeline and calendar days in another, and the mismatch only showed up on weekends when the two counts diverged. Took two engineers and a very confused Slack thread to find. This field runs on absurd, specific causes hiding behind statistical-sounding symptoms — a timezone bug in a feature, an emoji breaking a tokenizer, a retrained model accidentally reading test data. Drop yours below.