The Bitter Lesson - Rich Sutton

I’m late to the party, I know.

I got a recommendation to read The Bitter Lesson by Rich Sutton (2019). This was a fairly quick read and for some, it might stay with you. In my opinion, it is too early to make any conclusions if the article will give any future value, more than being a reflection of when it was written.

So, reading Rich Sutton’s essay on The Bitter Lesson is a bit of a reality check for anyone who follows what is going on in AI and LLM. Even though he (Rich Sutton) wrote this back in 2019, it has become more relevant with the current AI expansion, or explosion might be more fitting. The idea is simple, but a little hard to swallow for me. He argues that whenever we try to build our own clever human knowledge into AI, we eventually lose to raw computation.

Sutton points out that for seventy years, researchers have tried to teach machines how to play chess or recognize speech using human rules and logic. It works in the short term, but eventually, Moore’s Law catches up. The moment we stop trying to hand-hold the machine and instead just give it massive amounts of data and processing power, it leaps ahead of us. We saw it with Deep Blue in chess and AlphaGo.

It is called a bitter lesson because it hurts our ego. We want to believe that our specific understanding of the world is what makes AI smart. But the truth is that scaling methods like search and learning is what actually move the needle. Instead of building on what we have discovered, we should build systems that can discover things on their own (and we are well on our way to doing so).

Looking at the way large language models work today, Sutton was clearly right. They do not have a set of grammar rules programmed into them by linguists. They have enough computation and data to find the patterns themselves. One thing I don’t think is talked about enough is how we are making “AI” more specialized and efficient, leapfrogging discussions of computational power.

The Bitter Lesson was published in 2019. When I’m writing this, it is 2026 and things are moving fast and in several directions at once and no one knows where we are going.