HyperwriteAI is developing a self-operating computer framework that uses multimodal models to perform tasks. The system mimics human operators, viewing the screen and deciding on mouse and keyboard actions to achieve objectives. Despite current challenges with error rates in estimating mouse click locations, the team is focused on refining this accuracy, aiming for human-level performance in computer operation. The framework is designed for compatibility with various multimodal models and is currently integrated with GPT-4v.
Read more at GitHub…