From its inception, Azure Maia has been built around a single architectural philosophy: optimize the entire system for LLM inference so that every watt and every dollar delivers more intelligence. Now in its third generation , Maia 300 is the fullest expression of that philosophy. Maia 300 is built as a complete system, with software, silicon, physical rack infrastructure, networking, and datacenters co-designed to operate as a single platform to meet the demands of cloud-scale agentic inference.
Learn more about Maia 300 here That system-level approach is increasingly necessary as agentic workloads, including multi-step execution, long-context decoding, and tool use, change the computational profile and arithmetic intensity of inference. Meeting these demands requires sustaining locality, supplying compute with enough memory capacity and bandwidth, minimizing data movement, scaling communication efficiently, and fitting the most capability within a datacenter’s fixed power and cooling envelope.
At Microsoft, we see this transformation firsthand through the products and platforms our customers use every day, from Microsoft Copilot and GitHub Copilot to Microsoft Foundry. Operating these workloads at scale gives us a unique view into how AI is being adopted across industries and the infrastructure needed to support the next generation of AI applications and agents.
Maia 300 is designed to maximize useful tokens per watt and useful tokens per dollar, bringing more capacity, capability, and performance to these products and Microsoft’s broader AI portfolio for our customers. Energy-efficient inference through software-defined dataflow Maia 200 introduced a new class of accelerator architecture called Software Defined Locally Accessed Dataflow Architecture , or SDLA.
It begins with the observation that efficient data management, including storing, moving and converting data, is one of the central challenges in modern AI systems. The Maia programmable dataflow architecture separates control path programming from data path execution and allows software to explicitly coordinate data placement, movement, conversion and synchronization across specialized local memories, DMA engines, compute engines, and the network.
Maia 300 system-on-chip architecture Maia 300 evolves this architecture for long-context, reasoning-intensive, and decode-heavy inference. First, memory moves from the uniform memory access (UMA) organization of the SoC to a locality-aware non-uniform (NUMA) organization across chiplets, so software can place data near the compute that uses it. Second, new narrow-precision formats with hardware-accelerated data type conversion reduce the bytes moved for each token.
Third, the rack doubles to 96 accelerators and the scale-up domain grows to a superpod of 12,288 accelerators, enabling more accelerators to work effectively as a single system. Rather than scaling performance through increased power and silicon area alone, Maia 300 improves performance per watt through a set of architectural innovations that optimize data movement, locality, precision, and memory efficiency across the system: Multi-chiplet architecture boosts performance by placing compute on TSMC N2P, where the advanced process delivers the greatest performance-per-watt benefit, while separating memory-controller and I/O functions onto dies optimized for bandwidth, connectivity, and cost.
This increases the silicon area dedicated to compute performance and allows each function to use the technology best suited to its role, while SDLA manages locality and data movement across the package. This strategy increases the system’s compute FLOPs by 36%, adds 2. 5 times HBM bandwidth, and provides 14% more scale-up bandwidth.
Optimized low precision inference, introducing enhanced MXFP4 and sub-3-bit storage formats. Narrower representations reduce memory footprint and data movement, allowing the system to support larger models, longer contexts, and more concurrent sessions within the same power envelope. Optimized inference kernel through ISA enhancement: The vector engine adds optimizations for key inference kernels, including softmax, low precision data casting, and other operations .
Efficient on-die data movement , Maia 300 changes memory from a uniform organization to a non-uniform (NUMA) one across chiplets. Each compute region has nearby memory, and SDLA’s explicit control over DMA engines, local SRAM, HBM and the network-on-chip places data next to the compute that uses it. Data travels shorter distances, which reduces the energy spent moving it.
Memory hierarchy optimized for bandwidth per watt, combining software-managed on-die SRAM with eight HBM4 stacks providing 288 GB of capacity and 18 TB/s of bandwidth per accelerator. Compared with Maia 200 , this delivers 2. 5 times the memory bandwidth, 33% more capacity, and approximately 2 times the HBM bandwidth per watt, keeping more model and KV state close to compute.
Maia 300 brings 2. 5 times the memory bandwidth, 33% more capacity, and approximately 2 times the HBM bandwidth per watt compared with Maia 200 One scale-up fabric: From pod to superpod Maia 300 makes the network a native extension of its software-defined dataflow architecture. Maia’s AI network controller implements custom AI transport based on Ethernet.
A two-tier network builds a high-performance superpod with up to 12,288 accelerators. Maia presents one communication model to software—the same semantics to move data directly to any accelerator within a superpod. Half-switched in-rack topology Each accelerator divides its scale-up connectivity between direct links and switched paths.
Direct links connect two accelerators in a tray, while the switched fabric connects all 96 accelerators in the rack to form a pod. All in-rack connectivity uses electrical links to reduce power and cost. Optical second tier across racks Second-tier switches connect up to 128 racks together to form a superpod.
Originally published at azure.microsoft.com
