Pretraining data, not verifiability, is why LLMs are especially good at math (and coding).

A LessWrong post argues that pretraining data, not the ease of verifying math answers, explains why large language models are especially good at math and coding. It is a follow-up to a post arguing LLMs are still mostly powered by imitative learning rather than reinforcement learning.

Written by AI from LessWrong's published text. Read the original for full details.