{
  "id": "2205.11916",
  "version": "v1",
  "published": "2022-05-24T09:22:26.000Z",
  "updated": "2022-05-24T09:22:26.000Z",
  "title": "Large Language Models are Zero-Shot Reasoners",
  "authors": [
    "Takeshi Kojima",
    "Shixiang Shane Gu",
    "Machel Reid",
    "Yutaka Matsuo",
    "Yusuke Iwasawa"
  ],
  "categories": [
    "cs.CL",
    "cs.AI",
    "cs.LG"
  ],
  "abstract": "Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding ``Let's think step by step'' before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with an off-the-shelf 175B parameter model. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted through simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.",
  "revisions": [
    {
      "version": "v1",
      "updated": "2022-05-24T09:22:26.000Z"
    }
  ],
  "analyses": {
    "keywords": [
      "large language models",
      "zero-shot reasoners",
      "outperforms zero-shot llm performances",
      "zero-shot knowledge hidden inside llms",
      "reasoning tasks"
    ],
    "note": {
      "typesetting": "TeX",
      "pages": 0,
      "language": "en",
      "license": "arXiv",
      "status": "editable"
    }
  }
}