Anthropic revealed that it tested its latest AI model, Claude 3.7 Sonnet, using the classic Game Boy game Pokémon Red. The company equipped the model with essential features like basic memory and screen pixel input, allowing it to interact with the game by pressing buttons and navigating the screen continuously.
One standout capability of Claude 3.7 Sonnet is its “extended thinking,” which enables it to tackle complex problems by utilizing more computing power and taking additional time. This feature proved beneficial during the gameplay of Pokémon Red.
Unlike its predecessor, Claude 3.0 Sonnet, which struggled to leave the starting area of Pallet Town, Claude 3.7 Sonnet managed to defeat three Pokémon gym leaders and earn their badges. Although Anthropic did not specify the exact computing resources used or the time taken to achieve these milestones, they noted that the model executed 35,000 actions to reach the final gym leader, Surge.
This achievement demonstrates the model’s enhanced capabilities in navigating the game. While using Pokémon Red as a benchmark may seem unconventional, it fits into a broader trend where various games are used to evaluate AI performance.
Recently, numerous apps and platforms have emerged to test AI models’ gaming skills across different titles, from Street Fighter to Pictionary. Anthropic’s innovative approach highlights the ongoing exploration of AI’s potential in interactive environments and its ability to learn and adapt through gameplay. As AI continues to evolve, it will be interesting to see how these gaming benchmarks contribute to the development of more advanced models in the future.
