AI summary: Researchers have developed the Focused Transformer (FOT), a technique that addresses the challenge of incorporating new knowledge into language models. FOT extends the context length in models, improving tasks requiring long-context modeling. The method can be applied to existing models without changing their architecture, making it a cost-effective solution for enhancing models with memory. The resulting models, known as LONGLLAMAs, have shown significant improvements in tasks that benefit from extended context.
Read more at MarkTechPost…