Everyone’s talking about DeepSeek, and there’s palpable excitement about the surging capabilities of AIModels and GenAI technologies.
- Leaders are quoting Jevon’s paradox (Satya Nadela, “As AI gets more efficient and accessible, we will see its use skyrocket, turning it into a commodity we just can’t get enough of.”) and Napolean (Sam Altman, “A revolution can be neither made nor stopped. The only thing that can be done is for one of several of its children to give it a direction by dint of victories.”). Source:https://www.cnbc.com/2025/01/27/chinas-deepseek-ai-tops-chatgpt-app-store-what-you-should-know.html
- Then DeepSeek published this table for their V3 Capabilities (Source:https://www.deepseek.com/) But do you know what these Benchmark (metric) means?(see Column 1).
- If you don’t it’s really good to know about this. Quick primer is here:
Overall Categories:
English: Evaluates the model’s performance on English language tasks.
Code: Assesses the model’s ability to generate and understand code.
Math: Tests the model’s mathematical reasoning and problem-solving abilities.
Chinese: Evaluates the model’s performance on Chinese language tasks.
Specific Benchmarks (Metrics):
English:
MMLU (EM): Massive Multitask Language Understanding. A comprehensive benchmark covering various language understanding tasks. EM stands for Exact Match, meaning the model’s response must perfectly match the correct answer.
MMLU-Redux (EM): A variation of MMLU, possibly with a different set of questions or a focus on specific subtasks. Again, EM means Exact Match.
MMLU-Pro (EM): Another variation of MMLU, possibly focusing on more challenging or specialized language understanding tasks. EM still denotes Exact Match.
DROP (3-shot F1): DROP (Discrete Reasoning Over Paragraphs) measures the model’s ability to reason over paragraphs and answer questions. “3-shot” means the model is given 3 examples before being asked to answer the question. F1 is a metric that combines precision and recall.
IF-Eval (Prompt Strict): IF-Eval (Instruction Following Evaluation) measures how well the model follows instructions. “Prompt Strict” suggests a strict evaluation of how closely the model adheres to the given prompt.
GPQA-Diamond (Pass@1): GPQA (Google Professional Question Answering) Diamond is a benchmark using questions from various professional domains. “Pass@1” means the model is considered successful if it provides the correct answer in its first attempt.
SimpleQA (Correct): A benchmark consisting of simple question-answering tasks. “Correct” indicates the model’s answer must be completely accurate.
FRAMES (Acc.): FRAMES (Few-shot Reading Comprehension with Multiple Sentences) measures reading comprehension abilities. “Acc.” stands for accuracy.
LongBench v2 (Acc.): LongBench is a benchmark designed to evaluate the model’s ability to handle long-form text. “Acc.” represents accuracy.
Code:
HumanEval-Mul (Pass@1): HumanEval is a benchmark for code generation. “Mul” likely indicates multiple test cases per problem. “Pass@1” signifies the model is considered successful if it generates correct code on its first attempt.
LiveCodeBench (Pass@1-COT): LiveCodeBench is another code generation benchmark. “COT” likely stands for Chain-of-Thought prompting, a technique to improve reasoning. “Pass@1” has the same meaning as above.
LiveCodeBench (Pass@1): Same as above but without Chain-of-Thought prompting
Codeforces (Percentile): Codeforces is a competitive programming platform. This metric likely measures the model’s performance relative to human competitors on Codeforces, expressed as a percentile.
SWE Verified (Resolved): SWE likely refers to Software Engineering. This metric probably evaluates how well the model can solve software engineering problems that have been verified as having correct solutions. “Resolved” might indicate the problems have been checked by humans
Aider-Edit (Acc.): Aider is a tool for code editing. This benchmark likely measures the accuracy of the model’s code edits.
Aider-Polyglot (Acc.): Similar to Aider-Edit, but “Polyglot” suggests the model is evaluated on its ability to edit code in multiple programming languages.
Math:
AIME 2024 (Pass@1): AIME (American Invitational Mathematics Examination). “Pass@1” means the model is successful if it gets the correct answer on its first try.
MATH-500 (EM): A math benchmark with 500 problems. “EM” stands for Exact Match.
CNMO 2024 (Pass@1): CNMO (China National Mathematics Olympiad). “Pass@1” has the same meaning as before.
Chinese:
CLUEWSC (EM): CLUE (Chinese Language Understanding Evaluation) Word Sense Classification. “EM” stands for Exact Match.
C-Eval (EM): C-Eval is a comprehensive Chinese language evaluation benchmark. “EM” denotes Exact Match.
C-SimpleQA (Correct): Simple question answering in Chinese. “Correct” indicates the model’s answer must be entirely accurate.
This breakdown should give you a better understanding of the metrics used to evaluate the LLMs in the table. Remember that each benchmark focuses on different aspects of language understanding, coding, and reasoning. Comparing performance across these benchmarks provides a comprehensive view of each model’s capabilities.
