Baidu officially published the pre-trained weights for ERNIE-5.5-Flash-Base on November 5, 2026, delivering an open-source foundation model engineered specifically for high-efficiency downstream adaptation. Spanning 280 billion total parameters with 14 billion activated per token, the model was pre-trained entirely from scratch on 33.5 trillion tokens. The checkpoint represents the raw state of the network at the conclusion of pre-training, intentionally omitting mid-training or instruction post-tuning to provide a clean baseline for academic researchers and enterprise fine-tuning teams.
The backbone incorporates a hybrid attention mechanism termed Latent-Gated Sparse Attention that drastically reduces KV-cache memory consumption. Rather than computing uniform full-context attention at every layer, only one out of every seven layers retains token-level access using Multi-head Latent Attention (MLA) coupled with DeepSeek-style Sparse Attention (DSA). The intervening 37 layers utilize Heavily Compressed Attention (HCA), which compresses 128 sequential tokens into a single cached representation while attending to a localized window of uncompressed recent tokens. In addition, Low-Rank Gated Attention derives gating scalars directly from query latents to eliminate 75 percent of projection parameters.
Feed-forward layers rely on a Multi-Router Latent Mixture-of-Experts topology comprising 512 routed experts and one dedicated shared expert. Instead of depending on a single monolithic router whose routing decisions can entrench early sub-optimal selections, Baidu deploys multiple independently initialized sigmoid routers that jointly score candidate experts. Tokens are first mapped from the 4096 hidden dimension down to a 2048-dimensional latent space where experts operate, before a shared up-projection restores full width. This compression roughly halves inter-accelerator all-to-all communication traffic and expert floating-point operations.
Residual pathways are replaced with Manifold-Constrained Hyper-Connections (mHC), enabling each Transformer layer to read from and write to four parallel residual streams through constrained linear mixing matrices. At the output boundary, Baidu introduces MultiMax, which substitutes a standard single softmax temperature with a learnable piecewise scaling function executed inside a chunked Triton kernel. For future token forecasting, the architecture discards auxiliary multi-token prediction heads in favor of ECHO, an approach that reuses the models final layer to predict subsequent tokens, directly improving speculative decoding acceptance rates without adding detached parameters.
The training corpus was processed using a customized right-to-left (R2L) byte-pair tokenizer of 200,000 tokens designed to align numerical place values in groups of up to three digits. Pre-training was driven by the Muon optimizer operating in FP8 mixed precision across multi-thousand accelerator clusters. To overcome the high synchronization overhead of Muon, Baidu merged dense and MoE gradient transfers into unified collective operations and implemented memory-balanced sharding to prevent out-of-memory bottlenecks. Attention computation utilized FlashMask-4, a custom CuTe-DSL kernel for SM90 and SM100 GPUs supporting query-key head dimensions up to 576 and value dimensions of 512.
Standardized evaluation on identical harnesses against peer open-weight base checkpoints demonstrated broad capability gains. ERNIE 5.5 Flash Base recorded 93.3 on MMLU (5-shot exact match), outperforming DeepSeek-V4-Flash-Base (89.4) and Hy3-preview-Base (87.5). On MMLU-Pro, ERNIE registered 78.4 compared to 69.2 for DeepSeek V4 Flash, alongside a 60.2 score on GPQA. Long-context evaluations confirmed that the six MLA-DSA layers maintain effective global retrieval across 128K context benchmarks despite heavy compression across the majority of Transformer blocks.