Fri Sep 20 14:00:00 UTC 2024: ## LLMs Struggle with Basic Math: Researchers Propose “Scaffolding Learning” to Improve Performance
**San Jose, CA** – A new research paper published in PLOS ONE challenges the way large language models (LLMs) are trained to solve math problems. Researchers found that while LLMs like ChatGPT excel at complex problems, they often struggle with basic arithmetic, such as multiplication and division with multiple digits.
This disparity led the authors to propose a new training approach called “Scaffolding Learning,” which mimics how humans learn math. This approach involves first training the LLM on specific tasks, such as individual arithmetic operations, before moving on to more generic tasks like solving word problems.
**Why Scaffolding Learning?**
Traditionally, LLMs have been trained on a vast pool of problems with varied levels of difficulty. While this approach has yielded impressive results in complex scenarios, it seems to hinder the models’ ability to master basic skills. “Scaffolding Learning” addresses this by breaking down the learning process into manageable steps, starting with specific tasks and building towards more complex ones.
**The Experiment**
Researchers tested the efficacy of Scaffolding Learning by training the Pythia-6.9B language model on arithmetic operations before exposing it to word problems. Their findings were striking: after mastering arithmetic, the model achieved high accuracy in solving word problems with a surprisingly small number of training examples. This contrasted sharply with the performance of models trained directly on word problems, which showed significantly lower accuracy even after thousands of examples.
**Potential Applications**
The researchers argue that Scaffolding Learning has significant implications for improving LLM performance in various fields, particularly in science and engineering. They suggest that by training LLMs on fundamental skills related to specific domains, we can significantly reduce the amount of training data needed to tackle complex real-world problems.
**Looking Ahead**
While Scaffolding Learning shows promising results, researchers acknowledge limitations. The approach is best suited for tasks where numerous training examples can be generated for sub-problems. Additionally, the model still needs fine-tuning to apply learned skills to the original task.
**Overall, the research highlights the need for more nuanced training methods for LLMs, mirroring human learning processes. Scaffolding Learning represents a promising step in this direction, with potential to revolutionize LLM performance in various fields.**
First Piper 