The video explores the development of Orca, a 13 billion parameter language model created by Microsoft that can mimic the reasoning process of larger models like GPT4, outperforming smaller models like Llama and Alpaca. Orca matches GPT4 in logical reasoning tests and performs well on language benchmarks like the GMAT and GRE. The authors utilized teacher models, including ChatGPT and GPT4, and fine-tuned Orca with prompts from Google’s Flan collection, reducing the gap in its capabilities and achieving better performance. The video speculates that Microsoft’s research on Orca may be a response to Google’s concerns regarding the limitations of training large models from scratch. The leaders of OpenAI express different opinions on the gap between open source and private models, with Elia Satsukova believing the gap is growing wider and Sam Altman suggesting that OpenAI’s ability to figure out what comes next sets it apart.
Watch on YouTube…