Nothing has beaten our 14b classifier

About this talk

A 14-billion-parameter classifier has been running in our production pipeline for almost a year, processing millions of documents a month. It’s not out of attachment: we have tried to replace it with newer and better models, ones that on paper beat it on most benchmarks, and each time the replacement cost us more than it bought. In this talk, I’ll walk you through what happened each time we tried to upgrade, what we learned about the numbers we’d been trusting, and the bar a new model now has to clear before it gets near production. It’s a low bar. Almost nothing clears it.