AI summary: Leaked details about GPT4 reveal a model size of 1.8 trillion parameters across 120 layers, utilizing a Mixture of Experts (MoE) for efficient scaling. The model was trained on 13 trillion tokens, with a context length of 8K to 32K. OpenAI used around 25,000 Nvidia A100 GPUs for training, costing approximately $25,000. The estimated cost to train GPT4 with today’s equivalent hardware would be around $63 million. Despite its impressive specs, GPT4’s computational resource utilization was reported at 32% to 36%, indicating room for efficiency improvements.