Open-source Large Language Models (LLMs) have rapidly advanced, offering developer communities powerful alternatives to proprietary systems. This article provides a deep dive into five major open LLMs – their architectures, training specifics, and how they stack up on intelligence benchmarks. We examine Meta’s latest LLaMA 3, the efficient Mistral model, UAE’s Falcon, community-driven models like OpenChat/OpenHermes, and new challengers like DeepSeek (with a note on Yi). We’ll also explain the key evaluation metrics (MMLU, ARC, HellaSwag, TruthfulQA, GSM8K, BBH) and leaderboards used to compare LLM intelligence.

Meta’s LLaMA 3: Scaling Up Open Models

Meta’s LLaMA 3 is the third-generation LLM from the LLaMA family, pushing the boundaries of open model scale. Released in April 2024, LLaMA 3 debuted with 8B and 70B-parameter models (Meta’s Upcoming Release of the Largest Llama 3 Model). These models were pre-trained on approximately 15 trillion tokens from “publicly available sources,” and the instruction-tuned versions incorporated over 10 million human-annotated examples (Llama (language model) – Wikipedia). The architecture follows the Transformer decoder design with improvements carried from LLaMA 2 (such as efficient RoPE positional embeddings and swiGLU activation). Notably, LLaMA 3’s 70B model showed such strong learning that it was “still learning even at the end of the 15T tokens” of training (Llama (language model) – Wikipedia) – an indication of under-training relative to its capacity.

LLaMA 3 demonstrated state-of-the-art performance among open models upon release. Meta reported the 70B model outperforming Google’s Gemini Pro 1.5 and Anthropic’s Claude 3 (Sonnet) on most benchmarks in April 2024 (Llama (language model) – Wikipedia). By July 2024, Meta introduced LLaMA 3.1, including an enormous 405B-parameter model – one of the largest openly available to date (Llama (language model) – Wikipedia). This 405B version extended the model’s context window dramatically to up to 128k tokens (for long inputs), compared to an 8k context in the initial LLaMA 3 (Llama (language model) – Wikipedia) (Llama (language model) – Wikipedia). Such a long context length enables LLaMA 3.1 to handle very large documents or conversations, far beyond the 4k tokens of LLaMA 2.

The LLaMA 3 series continued to evolve through 2024 with versions 3.2 and 3.3 focusing on specialization. LLaMA 3.2 (Sept 2024) introduced smaller models (1B, 3B, 11B) optimized for edge devices and even a 90B model, plus early multimodal vision support (Llama (language model) – Wikipedia). By LLaMA 3.3 (Dec 2024), Meta had refined multilingual capabilities and integration into their Meta AI assistant products (Llama (language model) – Wikipedia). All LLaMA 3 models use a SentencePiece BPE tokenizer (likely ~32k vocabulary) similar to previous LLaMA versions. They remain “source-available” under a community license permitting commercial use with some restrictions (Llama (language model) – Wikipedia). Meta also provided instruction-tuned variants (chat models) alongside base models, making LLaMA 3 a versatile foundation for fine-tuning.

In summary, LLaMA 3 delivers unprecedented scale in the open domain (up to 405B parameters) and strong performance across tasks. Its Transformer architecture is largely standard but exhibits emergent capabilities from scale – e.g. the 8B LLaMA 3 was “nearly as powerful as the largest LLaMA 2” (70B) in early tests (Llama (language model) – Wikipedia). Meta’s commitment to open release (the models are available for download) and the inclusion of instruction tuning set a high bar. LLaMA 3’s roadmap (multilingual, multimodal, coding proficiency) (Llama (language model) – Wikipedia) indicates that it’s designed to be a general-purpose powerhouse in the open AI ecosystem.

Mistral: Small Model, Big Impact

Mistral 7B proved that a well-engineered 7-billion-parameter model can punch above its weight. Released by the startup Mistral AI in Sept 2023, Mistral-7B v0.1 “outperformed LLaMA 2 13B” on many benchmarks despite having half the parameters (Top 10 Large Language Models on Hugging Face- Analytics Vidhya). The secret lies in technical innovations in its architecture for efficiency:

• Grouped-Query Attention (GQA) – Mistral uses grouped-query attention, where multiple attention heads share key/value projections. This reduces memory usage and speeds up inference by “allowing faster inference and lower cache size” (Mistral), with minimal loss in modeling power.
• Sliding Window Attention (SWA) – Instead of full 8k context attention (which is memory heavy), Mistral was trained with a 8k context window and a fixed cache size, but uses a sliding window mechanism that can theoretically extend attention to 128k tokens (Mistral). In practice, this means the model processes long inputs in segments (e.g. 4096-token windows) with overlap, enabling extremely long context handling at lower compute cost.
• Efficient Training – Mistral employs FlashAttention and other optimizations (RMSNorm, RoPE, etc.), focusing on making a smaller model reach the performance of larger ones (Mistral 7B Explained: Towards More Efficient Language Models | by Bradney Smith | TDS Archive | Medium) (Mistral 7B Explained: Towards More Efficient Language Models | by Bradney Smith | TDS Archive | Medium). Its tokenizer is a custom Byte Pair Encoding (BPE) with a byte-level fallback, which ensures robust handling of rare or out-of-vocabulary characters (Top 10 Large Language Models on Hugging Face- Analytics Vidhya) (similar in spirit to GPT-3’s tokenizer that can byte-decode any string).

The result is a 7B model that set a new standard for parameter efficiency. Mistral-7B achieves strong results on reasoning and knowledge tasks that previously required 13B+ models (Top 10 Large Language Models on Hugging Face- Analytics Vidhya). It’s a decoder-only Transformer like LLaMA, but the “careful architectural design” lets it “exceed the performance of much larger models using a fraction of the parameters” (Mistral 7B Explained: Towards More Efficient Language Models | by Bradney Smith | TDS Archive | Medium). Notably, Mistral-7B is fully open-source under the Apache 2.0 license, allowing free commercial use. This openness spurred a wave of community fine-tunes – for example, OpenOrca-Mistral-7B and OpenHermes-2.5 are built on Mistral’s base and topped the leaderboards for 7B models (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data).

Mistral AI has not stopped at 7B. By late 2024, they teased larger models (referred to as Mistral Large) under a research license, and specialized versions: coding-optimized Codestral, vision-enabled Pixtral, and a multilingual model Mistral Nemo. Their documentation indicates models with up to 131k token context and even an 8×7B expert ensemble dubbed “Mixtral” (OpenHermes-2.5: This Local LLM Is All You Need) (OpenHermes-2.5: This Local LLM Is All You Need). These developments hint at mixture-of-experts (MoE) approaches (e.g. 8×7B experts) to scale parameters without linear compute cost. In fact, the community has already experimented with merging Mistral checkpoints – one example is Dolphin 2.5 Mixtral 8×7B, an uncensored chatbot that uses 8 Mistral experts (OpenHermes-2.5: This Local LLM Is All You Need).

In summary, Mistral stands out for delivering Llama-2-13B level performance from a 7B model (Top 10 Large Language Models on Hugging Face- Analytics Vidhya), thanks to innovations like GQA and SWA. It supports an 8k context (128k with sliding windows) (Mistral), making it practical for longer inputs than many older models. For developers, Mistral-7B’s small size (fits on a single GPU) and Apache-2 license make it an attractive choice for fine-tuning and deployment. It set a template for efficient LLM design that others are following.

Falcon: High-Flying 40B to 180B Models from TII

The Falcon series, developed by the Technology Innovation Institute (TII) in UAE, has been a flagship for large open models. Falcon-40B (released mid-2023) and Falcon-7B quickly gained popularity for strong performance and an Apache 2 license. Falcon models are decoder-only Transformers trained on the RefinedWeb dataset – a massive curated web crawl focusing on high-quality content. In September 2023, TII took a leap further by releasing Falcon-180B, a 180-billion-parameter model that was (at that time) “the largest openly-available LLM” (Falcon180B: authors open source a new 180B version! : r/LocalLLaMA) (Falcon 180B: The Powerful Open Source AI Model … That Lacks …).

Falcon-180B’s specs are impressive: it was trained on 3.5 trillion tokens of RefinedWeb data plus additional curated corpora (tiiuae/falcon-180B · Hugging Face). The model architecture is optimized for inference with multi-query attention (MQA) (tiiuae/falcon-180B · Hugging Face) – a technique where all attention heads share a single set of key/value vectors (proposed by Shazeer et al., 2019). MQA (similar to GQA) greatly reduces memory usage for large models. This means Falcon can maintain speed and memory efficiency even at 180B scale, by cutting down redundant computations in multi-head attention. Falcon models use rotary positional embeddings and standard transformer layers, with training optimizations to handle such a long training corpus.

Upon release, Falcon-180B was state-of-the-art among open models. It “outperforms LLaMA-2, StableLM, RedPajama, MPT, etc.” on many benchmarks (tiiuae/falcon-180B · Hugging Face). Indeed, Falcon-180B topped the Hugging Face Open LLM Leaderboard for a time in late 2023. TII also provided an instruction-tuned variant, Falcon-180B-Chat, aligned for dialogue. However, running Falcon-180B is resource-intensive – it requires ~400GB of memory for inference in full precision (tiiuae/falcon-180B · Hugging Face) (though 4-bit quantization can shrink this to around 100GB). Most developers use the smaller Falcon-40B (which fits on ~2×24GB GPUs in 8-bit). Falcon-40B itself was trained on 1T tokens and demonstrated excellent knowledge and reasoning ability for its size, often beating LLaMA-2 70B on open benchmarks as of 2023.

Falcon’s contributions include not just the models but also the RefinedWeb dataset and an open-source training recipe. The open model community benefited from Falcon’s release under a permissive license, which TII explicitly allows for commercial use (tiiuae/falcon-180B · Hugging Face). This contrasts with LLaMA’s more restricted license. Falcon models support an input context of 2048 tokens (out of the box) and use a typical GPT-style tokenizer. While not explicitly multilingual, the training data’s breadth gives decent performance across English and other languages present on the web.

In summary, Falcon models represent the large end of open-source LLMs – reaching 180B parameters and competing with the best closed models of 2023. Their use of multi-query attention and massive training corpora produced models that are both powerful and (relatively) efficient in inference (tiiuae/falcon-180B · Hugging Face). For developers who need maximum horsepower and are willing to handle the deployment complexity, Falcon-180B is a top choice. Meanwhile, Falcon-40B remains a strong general-purpose model that is easier to fine-tune and deploy, benefiting from the same design principles.

OpenChat and OpenHermes: Fine-Tuning Open Models to New Heights

Not all breakthroughs come from new base models – some come from fine-tuning existing open models with clever techniques. OpenChat and OpenHermes are two community-driven projects that took LLaMA/Mistral bases and tuned them to rival proprietary chatbots. These models show how open-source LLMs can be adapted with alignment and instruction-following to achieve ChatGPT-like capabilities on your own hardware.

OpenChat is a series of fine-tuned models (versions 3.x) originating from the OpenAccess AI community. OpenChat 3 was based on LLaMA 2 (in 7B and 13B flavors) and has since incorporated LLaMA 3. The OpenChat team introduced a novel fine-tuning strategy called C-RLFT (Consistent Reinforcement Learning from Human Feedback, offline) (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data). In essence, they fine-tune on a mix of high-quality and imperfect conversational data without explicit preference labels, simulating an RLHF-like outcome without needing human comparisons. This approach allowed even a 7B model to “deliver exceptional performance on par with ChatGPT” (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data), according to the OpenChat authors. For example, OpenChat 3.5 (7B), released in late 2023, reportedly surpassed ChatGPT on various benchmark tests (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data) – including knowledge and reasoning evaluations – while still running on a single GPU. By early 2024, OpenChat 3.6 (using LLaMA 3 8B as the base) was released, and it outperformed Meta’s official LLaMA 3 8B Instruct model in the Open LLM Leaderboard evaluations (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data).

OpenChat models place heavy emphasis on multi-turn dialogue, coding, and instruction following. The fine-tuning datasets include open instruction corpora (like OASST, Orca, etc.) and code tasks, which led to notable improvements in coding benchmarks. In fact, an update to OpenChat 3.5 in Dec 2023 “improved coding by 15 points” on HumanEval (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data). The OpenChat 3.x models are freely available for commercial use and have been integrated into various chat interfaces. They typically maintain the base model’s context length (4k or 8k tokens) but add a conversational formatting and the ability to follow user instructions more reliably. This showcases how LoRA fine-tuning or full fine-tunes on open bases can yield highly capable assistants without new parameter training from scratch.

OpenHermes is another exemplar – a fine-tuned model focusing on conversational prowess. OpenHermes-2.5 (7B) was built on the Mistral-7B base by community contributors (notably, Teknium). It has been lauded as “one of the best performing Mistral-7B fine-tune models” (OpenHermes-2.5: This Local LLM Is All You Need). OpenHermes combined the strengths of Mistral’s efficient base with extensive chat fine-tuning, including additional training on code and dialogue. It adopts the ChatML prompt format (from OpenAI) for better multi-turn consistency (teknium/OpenHermes-2.5-Mistral-7B – Hugging Face), and was reported to improve benchmarks across the board. For instance, OpenHermes-2.5 reached an MMLU score of ~64 and GSM8K math score of ~74, significantly above the original Mistral-7B base (which had ~52 MMLU) (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data). This put OpenHermes on par with some 13B models on these evaluations.

The success of OpenHermes and OpenChat demonstrates the impact of fine-tuning methods on model “intelligence.” Techniques like reward modeling and reinforcement learning (as done implicitly by OpenChat’s C-RLFT) and careful curation of instruction data can make a smaller model much more useful in interactive settings. Many fine-tunes also utilize Direct Preference Optimization (DPO) or similar loss functions to better incorporate preference data without the complexity of full RLHF. The result: models that are safer, more factual, and better at following user intent.

From a developer’s perspective, these community models offer ready-to-use chatbots that rival closed-source ones. They often come in quantized formats (e.g. 4-bit QLoRA weights or GGML binaries) for efficiency, meaning you can run a ChatGPT-like model on a consumer GPU or even CPU. In summary, OpenChat and OpenHermes exemplify how open LLMs plus open research in fine-tuning can yield highly capable conversational agents. They bridge the gap between raw model and practical AI assistant.

DeepSeek (and Yi): Next-Generation Open LLMs with New Approaches

The open-source LLM landscape in 2024–2025 has also seen newcomers built entirely from scratch, aiming to leapfrog earlier models. Two notable projects in this vein are DeepSeek and Yi – both pushing the frontier with massive training corpora and novel architectures.

DeepSeek is an open LLM initiative that has made waves with its unusual design. The latest model, DeepSeek V3, uses a Mixture-of-Experts (MoE) Transformer architecture to achieve extremely high capacity. While the model has a total of 671 billion parameters, only a subset (~37B) are active for any given token (GitHub – deepseek-ai/DeepSeek-V3). This MoE approach (inspired by Switch Transformers) allows scaling the model’s knowledge without a proportional increase in computation. DeepSeek V3’s effective capability rivals the largest dense models: on benchmarks, it outperforms or matches LLaMA 3.1 405B and other frontier models. For example, DeepSeek V3 scored 88.5% on MMLU (English) – comparable to LLaMA 3.1’s 88.6 – and surpassed it on a tougher MMLU-Pro subset (GitHub – deepseek-ai/DeepSeek-V3). It also excels at reasoning-heavy tasks: e.g., on DROP (reading comprehension) it hit 91.6 F1, higher than any 400B+ dense model (GitHub – deepseek-ai/DeepSeek-V3). These results led the team to claim DeepSeek V3 is the best-performing open-source model on many benchmarks, “especially on math and code tasks.” (GitHub – deepseek-ai/DeepSeek-V3)

DeepSeek’s training emphasizes reasoning, coding, and multilingual ability. It was trained from scratch on a diverse, massive dataset (reports indicate on the order of 2 trillion tokens for the base 67B model (DeepSeek LLM: Let there be answers – GitHub)). The architecture features not only MoE layers but also support for very long context lengths (up to 128k tokens) (GitHub – deepseek-ai/DeepSeek-V3), making it adept at handling long documents or dialogues. Despite its complexity, DeepSeek is openly released – the 67B base model weights are on Hugging Face (deepseek-ai/deepseek-llm-67b-base – Hugging Face). However, due to its MoE nature, running DeepSeek can be non-trivial (it may require custom inference code to handle expert routing). For those who can leverage it, DeepSeek offers an open model that rivals the closed GPT-4 class in certain domains (GitHub – deepseek-ai/DeepSeek-V3).

Another notable project is Yi by 01.AI, a Chinese startup. Yi-34B is a 34B-parameter dense Transformer trained on an astonishing 3 trillion token multilingual corpus (GitHub – 01-ai/Yi: A series of large language models trained from scratch by developers @01-ai). Targeted as a bilingual model (Chinese and English), Yi-34B achieved extraordinary evaluation results in 2023. It “ranked first among all existing open-source models (such as Falcon-180B, Llama-70B, Claude)” on many benchmarks, including the HuggingFace Open LLM Leaderboard (pre-training tasks) and the Chinese CEval exam (GitHub – 01-ai/Yi: A series of large language models trained from scratch by developers @01-ai). On the AlpacaEval leaderboard for instruction-following, Yi-34B-Chat was second only to GPT-4, even outperforming other top models like Claude and Mistral-based fine-tunes (GitHub – 01-ai/Yi: A series of large language models trained from scratch by developers @01-ai). What’s remarkable is that Yi achieved this with a 34B model, thanks to extremely high-quality training data and techniques to maximize efficiency. The developers note that Yi adopts the same model architecture as LLaMA (Transformer decoder with similar configurations) but was built from scratch (no LLaMA weights) (GitHub – 01-ai/Yi: A series of large language models trained from scratch by developers @01-ai). This means they could open-source it without license issues. A smaller variant, Yi-13B, has been made fully open, while the 34B chat model’s weights are semi-open (available for research/commercial license). The Yi series highlights how data scale and training optimization can sometimes beat sheer parameter count – a 34B model topping a 180B model on certain tasks (GitHub – 01-ai/Yi: A series of large language models trained from scratch by developers @01-ai).

For developers, both DeepSeek and Yi herald a new era where open models are not just following Big Tech releases but proactively advancing the state of the art. These models incorporate multilingual training, enormous token counts, and novel architectures (MoE) to achieve superior general intelligence. Many of them also support the usual toolkit: exporting to smaller precisions (the Yi repo notes quantized models run on 3090 GPUs easily (GitHub – 01-ai/Yi: A series of large language models trained from scratch by developers @01-ai)) and fine-tuning hooks for customization. While they may be less famous than LLaMA or Falcon yet, their impact is being felt on leaderboards and will likely trickle down to mainstream use via derivatives.

Model Feature Comparison

The following table summarizes key features of the five open LLMs discussed:

Model	Architecture & Params	Context Length	Training Data (approx.)	Notable Features & License
LLaMA 3 (Meta)	Transformer (dense); 8B, 70B, 405B	8k (128k in 3.1 version) (Llama (language model) – Wikipedia) (Llama (language model) – Wikipedia)	~15T tokens web+docs; +10M human examples (Llama (language model) – Wikipedia)	Instruction-tuned variants; state-of-art performance (Llama (language model) – Wikipedia); Community License (commercial use allowed) (Llama (language model) – Wikipedia).
Mistral 7B	Transformer (dense); 7B	8k (128k theoretical via SWA) (Mistral)	~1.3T tokens web data (est.)	Grouped Query Attn & Sliding Window for efficiency (Mistral); outperforms LLaMA2-13B (Top 10 Large Language Models on Hugging Face- Analytics Vidhya); Apache 2.0 license.
Falcon 180B	Transformer (dense); 180B	2k (2048 tokens)	3.5T tokens RefinedWeb + curated (tiiuae/falcon-180B · Hugging Face)	Multi-Query Attn for fast inference (tiiuae/falcon-180B · Hugging Face); largest open model in 2023 (Falcon 180B: The Powerful Open Source AI Model … That Lacks …); strong multitask ability; Apache 2.0 license.
OpenChat 3.5 (7B)	LLaMA2-based decoder; 7B	4k	Fine-tune on multi-turn chats, code, instructions	C-RLFT alignment (offline RLHF) (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data); ChatGPT-level responses at 7B (GitHub – imoneoi/openchat: OpenChat: Advancing Open-source Language Models with Imperfect Data); improved coding ability; open commercial use.
DeepSeek V3	Transformer MoE (dense+experts); 67B active (671B total) (GitHub – deepseek-ai/DeepSeek-V3)	16k–128k (extended) (GitHub – deepseek-ai/DeepSeek-V3)	2T+ tokens (code, text, reasoning data)	Mixture-of-Experts architecture; SOTA on math/reasoning (GitHub – deepseek-ai/DeepSeek-V3) (GitHub – deepseek-ai/DeepSeek-V3); bilingual (English/Chinese) eval strength; open (research license).

Table: Comparison of key models’ architecture, size, data, and features. Param = total parameters.

How LLM Intelligence is Measured

When we say one model “outperforms” another, it’s usually based on standardized evaluation benchmarks. These benchmarks test various aspects of AI capability in an apples-to-apples way. Here we explain some of the key metrics and tests commonly used to compare LLMs:

• MMLU (Massive Multitask Language Understanding): A benchmark of 57 diverse subjects (history, math, science, law, etc.) with over 15,000 multiple-choice questions (What Are LLM Benchmarks? | IBM). It evaluates the breadth and depth of a model’s world knowledge and problem-solving. Models are tested in zero-shot or few-shot mode (no fine-tune on the tasks), and the score is simply the percentage of questions answered correctly (LLM Benchmarks Explained: Everything on MMLU, HellaSwag, BBH, and Beyond – Confident AI) (What Are LLM Benchmarks? | IBM). A high MMLU score indicates a model that learned a lot of factual and commonsense knowledge during pre-training.
• ARC (AI2 Reasoning Challenge): A set of grade-school science exam questions designed to probe reasoning. It has an Easy set and a Challenge set, totaling 7,000+ questions (What Are LLM Benchmarks? | IBM). Questions often require combining factual knowledge with logical reasoning – beyond simple retrieval. Models earn 1 point per correct answer (or partial credit if they list multiple choices with one correct) (What Are LLM Benchmarks? | IBM). ARC was one of the early benchmarks where models like GPT-3 struggled, but newer LLMs have made strong progress, especially on the easy set. It’s a good test of commonsense reasoning and basic science understanding.
• HellaSwag: A commonsense inference benchmark with an adversarial twist. Models are given a partial description of a situation and must choose the most plausible continuation from four options. The dataset was constructed with “harder endings” and adversarially generated wrong answers to trip up models (What Are LLM Benchmarks? | IBM). For example, a prompt might describe a person opening a door, and the model must pick the sensible next action. HellaSwag measures the model’s grasp of everyday physical and social commonsense. Performance is measured by accuracy (percent choosing the correct ending) in zero-shot and few-shot settings (What Are LLM Benchmarks? | IBM). It’s challenging: GPT-3 sized models were near random accuracy initially, but later LLMs improved with better world knowledge.
• TruthfulQA: A benchmark that tests whether the model tells the truth (and resists false or misleading prompts). It consists of over 800 questions across 38 categories, many of which are adversarial or tricky (containing myths, traps, or requiring careful factual recall) (What Are LLM Benchmarks? | IBM). TruthfulQA evaluates the percentage of responses that are rated as truthful (and informative) (LLM Benchmarks Explained: Everything on MMLU, HellaSwag, BBH, and Beyond – Confident AI). A special GPT-trained judge (GPT-Judge) or human evaluation checks if an answer is true or a hallucination (LLM Benchmarks Explained: Everything on MMLU, HellaSwag, BBH, and Beyond – Confident AI). This benchmark is crucial because LLMs often hallucinate – a high TruthfulQA score means the model more reliably produces correct, non-fabricated information (LLM Benchmarks Explained: Everything on MMLU, HellaSwag, BBH, and Beyond – Confident AI). Models fine-tuned on factual data or with retrieval help tend to do better here.
• GSM8K (Grade School Math 8K): A set of 8,500 math word problems (at about a U.S. grade school level) designed to assess mathematical reasoning (What Are LLM Benchmarks? | IBM). Each problem is given in natural language; the model must produce the correct answer (often a number or simple phrase). Importantly, GSM8K often requires multi-step reasoning – something LLMs struggle with unless they can perform step-by-step “chain-of-thought.” Many evaluations let the model output its reasoning (which isn’t directly checked) and then the final answer. The metric is accuracy: the fraction of problems solved correctly. This benchmark has become a gold standard for testing logic and arithmetic in LLMs. Top models in 2025 (like GPT-4 or DeepSeek) can exceed 80-90% on GSM8K, whereas earlier models were below 50%, highlighting how far reasoning has come (GitHub – deepseek-ai/DeepSeek-V3).
• BIG-Bench Hard (BBH): BIG-Bench is a large collection of challenging tasks; BBH is a curated subset of the 23 most difficult tasks from that collection (LLM Benchmarks Explained: Everything on MMLU, HellaSwag, BBH, and Beyond – Confident AI). These tasks cover things like logical deduction, nuanced understanding, or extreme few-shot learning. They were considered “beyond the capabilities” of models when released (LLM Benchmarks Explained: Everything on MMLU, HellaSwag, BBH, and Beyond – Confident AI). BBH serves as a torture test for advanced reasoning and understanding – essentially, can the model solve problems that stumped earlier LLMs? Each task has its own metric (accuracy, F1, etc.), but models are often ranked by how many of the 23 tasks they significantly surpass a baseline on. BBH is useful to distinguish the very best models: for instance, an advanced model might solve 15+ of the tasks, while a weaker one solves only a few. It’s a measure of extreme generalization ability.

In addition to these, many other benchmarks exist (HumanEval for coding, MT-Bench for multi-turn dialogue, Winogrande for pronoun resolution, etc.), but the above are among the most widely cited for “general intelligence” of LLMs.

Leaderboards and Community Evaluations

To keep track of the many benchmarks, researchers rely on LLM leaderboards. A leaderboard aggregates multiple test results into a ranking of models, often with an overall score. One prominent example is the Hugging Face Open LLM Leaderboard, which ranks open-source models on a suite of benchmarks including ARC, HellaSwag, MMLU, GSM8K, TruthfulQA, and others (What Are LLM Benchmarks? | IBM) (What Are LLM Benchmarks? | IBM). Models are evaluated under identical conditions (usually 0-shot or few-shot) and the results are updated as new models are added. For instance, as of early 2025, you might see DeepSeek V3, LLaMA 3.1, Falcon 180B, etc. vying for the top spots. Such leaderboards provide a quick way for developers to see which models are currently the “smartest” by these metrics.

Another popular evaluation is via LMSYS’s Chatbot Arena (by the Vicuna team). This is a crowd-sourced Elo rating system where real users (or a proxy like GPT-4) compare two models in a chat conversation and vote for the better response (What Are LLM Benchmarks? | IBM). The LMSYS Arena yields an Elo score indicating overall quality and conversational skill. Open models like Vicuna, OpenAssistant, and others were ranked here against closed models. By mid-2024, some fine-tuned open models (e.g. Vicuna-33B, etc.) had Elo scores not far from ChatGPT. The MT-Bench mentioned earlier is part of this, using GPT-4 to grade model responses on multi-turn tasks (What Are LLM Benchmarks? | IBM). Leaderboards like LMSYS Arena are valuable because they capture interactive performance and qualitative aspects (like helpfulness, coherence) that static benchmarks might miss.

When evaluating models, it’s important to consider which benchmarks matter for your use case. A coding assistant might prioritize HumanEval and MBPP scores. A knowledge bot might emphasize MMLU and TruthfulQA. The great thing in 2025 is that the open-source community has assembled a rich set of evaluation data and made many results public – so we have a clearer picture than ever of how these LLMs compare.

Conclusion

The open-source LLM ecosystem in 2025 is vibrant and quickly closing the gap with proprietary models. Meta’s LLaMA 3 has set new records in openness and scale, Mistral has shown the way to efficiency, and Falcon demonstrated that even 100B+ models can be open access. Meanwhile, community fine-tunes like OpenChat and OpenHermes prove that with clever training, smaller models can achieve remarkable chat performance. Emerging projects like DeepSeek (and Yi) indicate the next wave of innovation, with techniques like MoE and massive multilingual data to push intelligence further.

For developers, the choices can be overwhelming – but also empowering. Depending on your needs (model size, license, multilinguality, etc.), you can pick an open LLM and have confidence in its evaluated capabilities. And you can fine-tune or even contribute to these models. The benchmarks and leaderboards help in navigating this landscape, offering an objective guide to an otherwise subjective question: How “smart” is this AI?

One thing is clear: open-source LLMs are here to stay, and collaboration plus transparency are driving them forward. Whether you need a 7B model to deploy in an app or a 180B giant for research, the open models discussed above cover the spectrum – and they are only getting better. The race towards more capable, more accessible AI is on, and the open-source community is leading from the front.