Massive Multitask Language Understanding (MMLU) is a comprehensive test designed to measure the multitask accuracy of AI text models across a wide array of subjects. This test includes 57 diverse tasks spanning elementary mathematics, US history, computer science, law, and more, aiming to assess the depth and breadth of a model's academic and professional understanding.

To excel in this test, models must possess extensive world knowledge as well as problem-solving abilities. Despite recent advances in Natural Language Processing (NLP) models, most still struggle with MMLU, especially in areas requiring complex reasoning like morality and law. This test is siginficant because it moves beyond evaluating basic linguistic skills or common sense knowledge, focusing instead on a broader spectrum of real-world text understanding.

The MMLU test covers subjects in humanities, social sciences, hard sciences, and other areas, using multiple-choice questions collected from various sources. These subjects include, but are not limited to, law, philosophy, history, economics,sociology, physics, computer science, and mathematics. The test's format allows it to identify knowledge gaps and blind spots in AI models, revealing areas where models perform poorly.

In experiments, models like GPT-3, GPT-4 and Gemini were evaluated using the MMLU test. The performance varied significantly across different domains, with lopsided results indicating substantial gaps in knowledge and understanding.