2.4T Parameters: Qwen 3.8 Model Open-Sourced, Signaling Convergence in Domestic Ultra-Large Model Architectures

Half a month after the open-sourcing of Kimi K3, Alibaba’s Qwen officially released the model weights for Qwen3.8-2.4T on August 13.

Beyond being a large-parameter model with 2.4T parameters, this open-source release holds additional significance for Alibaba.

Although Qwen has previously open-sourced flagship models multiple times, it had long established two distinct tracks: one for open-weight models and another for larger, closed-source Max models. When introducing its first-generation products in 2024, Qwen clearly distinguished between the “large open-source model Qwen-72B” and the “large closed-source model.” Later, Qwen-Max-0428 was provided exclusively through Qwen Chat and the DashScope API, without releasing weights. This pattern continued with Qwen2.5-Max into 2025, where only the API was made available, while Qwen2.5-72B served as its open-weight counterpart.

This marks the first Max-scale model from Qwen to have its weights opened.

The current open-source release of Qwen3.8 does not include a corresponding ready-to-use model, but rather provides basic information on the architecture, post-training, inference infrastructure, and open-source rules. Nevertheless, we can glean significant insights from these details.

Architecture

Qwen3.8 consists of 92 layers (compared to K3’s 96 layers), with a total of 2.4T parameters and 95B activated parameters per token. It utilizes 512 experts (compared to K3’s 896), selecting 10 routing experts at a time, along with 1 shared expert. Compared to Kimi K3, each expert is larger, but the total activation volume is slightly smaller (Kimi K3 uses 16 experts with 103B activations).

Regarding the attention mechanism, Qwen3.8 inherits the hybrid architecture of Qwen3.5. The 92 layers are actually composed of 23 groups of identical structures, each containing 3 layers of Gated DeltaNet and 1 layer of Gated Attention. This results in 69 layers of linear attention and 23 layers of traditional attention, achieving a ratio close to 3:1.

This is very similar to Kimi K3. K3’s 93 layers include 69 layers of Kimi Delta Attention (KDA) and 24 layers of Gated MLA (Gated Multi-Head Latent Attention), also approximating a 3:1 ratio. Both even belong to similar technical lineages, with KDA viewed as a further granular refinement of the forgetting mechanism in Gated DeltaNet.

However, K3 takes an additional step in inter-layer information flow. It introduces AttnRes (Attention Residuals), allowing the model to recombine outputs from multiple historical layers. According to the config.json, Qwen3.8 follows the implementation of Qwen3.5. The publicly available code for Qwen 3.5 still uses a relatively traditional Pre-Norm residual structure, where each sub-layer’s output is directly added to the residual entering that sub-layer.

Qwen3.8 has a native context window of 262,144 tokens, expandable to 1,010,000 tokens, and underwent multi-step MTP (Multi-Token Prediction) training.

Infrastructure

Here, it is important to distinguish between two types of infrastructure. Training Infrastructure answers “how this model was trained,” covering aspects such as optimizers, parallel training, communication, memory management, checkpoints, and fault tolerance. Inference Infrastructure answers “how to deploy the model after training is complete, ensuring each token is generated quickly and cost-effectively.”

Qwen has not systematically disclosed its training infrastructure in the same manner as the Kimi K3 technical report, so it is difficult to determine exactly how the 2.4T scale training was accomplished from this release. However, information regarding the inference side is relatively abundant.

First is quantization. Qwen officially provides BF16 and FP8 checkpoints, while Inferact further offers NVFP4 and MXFP4 quantized versions. The full BF16/FP8 versions require at least two NVIDIA B300 or AMD MI355X nodes, whereas the FP4 version can be compressed for deployment on a single node.

Second is parallelism. Qwen3.8 employs a combination of TP, DP, and EP during the inference phase. The Attention part relies more on TP to split matrix calculations, while the MoE part depends on EP to distribute different experts across different GPUs. Meanwhile, the inference framework aims to minimize communication overhead caused by expert scheduling through fused communications, specialized MoE Kernels, and high-speed interconnect networks.

Kimi K3 faces the same challenge. When training Stable LatentMoE with 2.8T parameters and 896 experts, Moonshot did not simply expand traditional EP. Instead, it proposed MoonEP, allowing popular experts to be dynamically replicated to more GPUs based on current load.

Finally, there is MTP. Qwen3.8 was trained with Multi-Token Prediction capabilities, with example configurations predicting 3 candidate tokens at once. If a candidate is accepted by the main model, it effectively advances multiple tokens in one forward pass, thereby reducing the serial latency associated with autoregressive token-by-token generation. Notably, DeepSeek V4 also explicitly utilized MTP in its technical report.

Post-Training

Compared to the architecture, Qwen disclosed fewer details about specific post-training techniques, as these are not included in the config documents.

However, its post-training objectives are very clear.

In the model card, Qwen explicitly lists Agent Execution as a core upgrade direction, emphasizing stronger autonomous planning, the ability to handle environment feedback, and more reliable end-to-end task completion.

Yet, virtually all current post-training efforts emphasize these points.

Officially published results reflect this inclination. In Qwen’s own comparisons, moving from Qwen3.7-Max to 3.8-Max, Terminal Bench 2.1 scores rose from 74.5 to 86.6, PaperBench from 64.8 to 93.0, JobBench from 31.3 to 53.4, and Toolathlon Verified from 49.7 to 72.5.

In contrast, GPQA Diamond only changed from 92.4 to 92.6, and HLE rose from 41.4 to 43.6. At least based on the presentation of official evaluations, the most significant gains in this generation clearly occurred in Coding Agents, General Agents, and professional workflows, rather than in traditional single-turn knowledge and reasoning benchmarks.

Additionally, Qwen3.8 enables preserve_thinking by default, retaining the reasoning context from previous turns. This allows an Agent to continue utilizing the previously formed reasoning state after a tool call, rather than restarting with only the final answer and tool results.

Kimi K3 has adopted almost the same direction. It explicitly states that its post-training retains reasoning history. During multi-turn interactions and tool calls, the previous reasoning content and tool calls must be fully sent back to the model.

The fact that both companies have reached this point indicates that “retaining reasoning state” is gradually shifting from being a responsibility of the harness to becoming part of the training and interface paradigm of flagship Agent models themselves.

Open-Source Rules

The open-sourced Qwen3.8-2.4T-A95B is not entirely equivalent to the cloud-based Qwen3.8-Max. The open version is currently text-only, runs only in thinking mode, and has a native context window of 262K. The official Qwen3.8-Max is built upon this model, additionally offering visual input, non-thinking mode, a default 1M context window, and official built-in tools.

Releasing enhanced versions on the cloud while keeping the base model open has become a common pattern for current open-source models, allowing companies to retain certain commercial capabilities. For instance, the open-sourcing of Minimax H3 kept material understanding affecting final output quality, 2K regeneration, and sparse attention implementations in the cloud.

The license has also changed. During the Qwen3 era, many models used Apache 2.0, but Qwen3.8 has switched to a specialized Qwen3.8-Max License. The license still permits using, copying, modifying, distributing, fine-tuning, deploying, hosting, and even selling derivative products. However, if a commercial product exceeds 100 million monthly active users or generates monthly revenue exceeding $20 million, the model name must be prominently displayed in the interface. If an enterprise engages in Model-as-a-Service or AI Work Assistant businesses, and its group revenue exceeds $50 million for 12 consecutive months, separate permission from Qwen is required before commercial use.

This is very similar to Kimi K3. K3 also adopts its own Kimi K3 License instead of Apache 2.0; it similarly allows broad use and modification but sets attribution requirements for ultra-large-scale commercial products and additional authorization thresholds for MaaS businesses. The differences lie mainly in the specific thresholds and the scope of covered business activities.

In other words, for 3T-class flagship models, both companies have chosen open-weight + custom commercial licenses, rather than the more permissive Apache-style openness seen in the era of smaller models.

Viewing Qwen3.8 and Kimi K3 together reveals a quite obvious convergence emerging in flagship models by 2026.

Total parameters have entered the 2–3T range, with activated parameters controlled around 100B. Attention mechanisms no longer rely solely on traditional Transformers, instead using approximately three-quarters recursive/linear state layers combined with one-quarter global Attention, with context targets stabilizing at the million-token level. Post-training focus has shifted to Coding, Research, and long-horizon Agents. Ultimately, deployability increasingly depends on FP4/FP8, fused kernels, Expert Parallelism, and rack-level high-speed interconnects.

Kimi K3 is somewhat more radical in its model architecture, simultaneously introducing KDA, AttnRes, and Stable LatentMoE; Qwen3.8 appears more like a direct scaling of the hybrid attention architecture already validated in Qwen3.5 to the Max level, complementing the entire ecosystem around long-horizon Agents and production inference.

Regardless, their architectural choices represent minor optimization differences rather than major distinctions. The infrastructure aspects may differ more significantly, but balancing various parallelisms remains the most critical issue.

The foundational recipe has begun to converge, making it even more anticipated to see who can break this mold and introduce something new next.

Risk Warning and Disclaimer

The market involves risks, and investment should be approached with caution. This article does not constitute personal investment advice, nor does it consider the specific investment goals, financial status, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article align with their specific circumstances. Investment decisions made based on this content are the sole responsibility of the investor.