ZAYA1: Zyphra’s Groundbreaking MoE Model Trained Exclusively on AMD 

In a major milestone for AI infrastructure, Zyphra has successfully trained ZAYA1, a large-scale Mixture-of-Experts (MoE) foundation model, using only AMD’s full-stack platform — including Instinct MI300X GPUs, AMD Pensando Pollara networking, and the ROCm open software stack.

This achievement demonstrates that AMD’s hardware and software ecosystem is now not only capable of supporting small to medium models — but also frontier-scale AI training. In this article, we explore the technical innovations, performance implications, and strategic significance of ZAYA1, and why it matters for the future of AI infrastructure.

Why ZAYA1 Is a Big Deal

1. First Large-Scale MoE on an AMD Platform

ZAYA1 is the first large-scale MoE foundation model that has been fully trained on AMD hardware. Unlike many AI models that rely on NVIDIA GPUs, Zyphra opted for a “pure AMD” stack — validating AMD’s viability in high-stakes, large-model training.

2. High Efficiency Through MoE Architecture

With 8.3 billion total parameters but only 760 million active parameters at any time, ZAYA1 uses a sparse Mixture-of-Experts approach. This design boosts computational efficiency by activating only a small subset of “expert” sub-networks, reducing computational cost without sacrificing model capacity.

3. Hardware–Model Co-Design

Zyphra co-designed ZAYA1 with AMD to optimize performance on the specific hardware used. This includes custom kernels, parallelism strategies, and model sizing tailored to AMD’s MI300X GPUs and network fabric.

4. High Throughput & Accessibility

The training cluster built for ZAYA1 — in collaboration with IBM Cloud — delivered over 750 PFLOPs of compute performance. This shows that AMD-based infrastructure can scale to match or rival traditional frontier AI platforms.

5. Competitive Performance

Benchmarking reveals that ZAYA1-base (8.3 B total / 760 M active) performs on par with — and in some cases outperforms — other leading open models. Zyphra reports that ZAYA1 matches or exceeds models such as Qwen3-4B (Alibaba) and Gemma3-12B (Google) on reasoning, math, and coding benchmarks. It also surpasses Llama-3-8B (Meta) and OLMoE.

Technical Innovations & Architecture of ZAYA1

Co-Designed for AMD’s MI300X

• Memory Advantage: The AMD Instinct MI300X GPUs used in the training cluster each have 192 GB of high-bandwidth memory (HBM). This high capacity enabled Zyphra to reduce or avoid the need for complex expert-sharding or tensor-sharding strategies, simplifying distributed training.
  
• High-Speed Networking: Each node includes AMD Pensando Pollara 400 Gbps networking hardware, delivering rails-only topology communication. This ensures that gradient exchanges and collective operations across GPUs remain efficient even at scale.
  
• ROCm Software Stack: Training was done using AMD’s ROCm (Radeon Open Compute) stack, which is optimized for both compute and memory throughput.
  

Model Architecture & Efficiency Improvements

Zyphra introduced several architectural features in ZAYA1 to exploit the underlying hardware efficiently:

Advanced Routing

• ZAYA1 uses a router that decides which “expert” sub-networks to activate. Zyphra’s design includes an MLP-based router (rather than a simple linear gate), allowing for richer expert specialization.

Compressed Convolutional Attention (CCA)

• CCA reduces memory usage by compressing KV-caches (key-value caches) in attention layers. According to Zyphra, this optimization reduces memory use by ~32% while increasing long-context throughput by ~18%.

Lightweight Residual Scaling

• To balance training stability and efficiency, Zyphra applies residual scaling techniques that reduce overhead while maintaining the ability to optimize effectively.
  

Parallelism Strategy

• Given the large memory per GPU, Zyphra used data parallelism with ZeRO-1 (a parameter-sharding optimizer) across the nodes rather than more complicated parallel strategies.

Robust Training Infrastructure

• Fault Tolerance & Checkpointing: Zyphra’s technical report details fault-tolerant training pipelines and efficient checkpointing strategies suitable for long-running large-model training.
  
• Microbenchmarking: The team performed detailed network and compute benchmarking, including collective communication (all-reduce, reduce-scatter) on AMD’s Pollara interconnect.

Performance & Benchmark Comparisons

Zyphra’s performance evaluation shows that ZAYA1 is not just a proof of concept — it’s competitive at scale:

• On reasoning, mathematics, and coding benchmarks, ZAYA1-base performs similarly to or better than open models like Qwen3-4B and Gemma3-12B.
  
• Compared to Llama-3-8B and OLMoE, Zyphra reports that ZAYA1 outperforms in multiple tasks.
  
• Training efficiency benefits from 10× faster model save times, thanks to AMD’s optimized distributed I/O layers.
  

Strategic Implications: What This Means for AI & Hardware Ecosystems

1. Validating AMD as a Serious AI-Training Platform

ZAYA1’s success challenges the dominance of NVIDIA in large-scale AI training. The fact that a frontier MoE model can be trained entirely on AMD hardware shows that AMD’s compute, memory, and networking stack has matured to “production-grade” for big AI workloads.

2. Efficiency Through Co-Design

By jointly designing the model and infrastructure, Zyphra and AMD achieved strong efficiency gains — co-optimization is becoming essential for high-performance AI training. This approach may become more common, especially for organizations seeking to reduce cost and maximize performance.

3. Open-Platform Advantage

Training on a fully open stack (hardware + software) offers transparency and flexibility. Organizations that prefer avoiding vendor lock-in or rely on open ecosystems can now consider AMD as a viable alternative for large-scale AI.

4. Hyperscaler & Cloud Adoption

The cluster for ZAYA1 was built with IBM Cloud, showing that major cloud providers are ready to support cutting-edge AMD-based training. This can pave the way for broader AMD adoption in cloud AI infrastructure.

5. Scalable Future Models

ZAYA1 is only the beginning. With a demonstrated production-level AMD stack, more researchers and companies may explore MoE architectures, long-context models, and efficient routing in future models trained on AMD infrastructure.

Challenges & Considerations

While ZAYA1 is a strong proof point, there are important challenges and caveats to keep in mind:

• Hardware Cost & Availability: Building a 128-node cluster with high-bandwidth GPUs and custom networking is non-trivial. Not every organization can replicate this scale easily.
  
• Software Complexity: To fully exploit the hardware, Zyphra had to implement custom kernels and optimizations. Enterprises may need significant engineering expertise to replicate similar performance.
  
• Specialized Architecture Risks: MoE models can be more complex to manage, deploy, and fine-tune than dense models. The performance gains may come at the cost of increased system complexity.
  
• Generalization: While ZAYA1 performs well on benchmarks, real-world adoption will require robust fine-tuning, safety validation, and deployment testing.
  
• Ecosystem Lock-in: Though AMD’s stack is open, co-designing with specific hardware could lead to some platform dependence unless portability is carefully maintained.
  

Why This Matters for the Future of AI

ZAYA1 is not just a technical milestone — it is a significant signal in the evolution of AI infrastructure:

• It affirms that MoE architectures are not only efficient but also practical for training on non-NVIDIA hardware.
  
• It underscores the importance of hardware/software co-design in building next-gen AI systems.
  
• It democratizes access to large-model training by broadening the viable hardware stack.
  
• It encourages cloud providers to support diverse AI compute ecosystems, reducing reliance on a single semiconductor vendor.
  

Conclusion

ZAYA1 — Zyphra’s Mixture-of-Experts foundation model — represents a landmark achievement not just for Zyphra, but for the broader AI hardware ecosystem. By proving that AMD’s full stack (compute, networking, software) can support high-performance, large-scale AI training, Zyphra and AMD together have opened a new chapter in the diversity of AI infrastructure.

This milestone underscores the value of hardware–model co-design, architectural efficiency, and open platforms. As AI continues to evolve, breakthroughs like ZAYA1 will likely drive more competition, innovation, and infrastructure flexibility — ultimately expanding access to next-generation AI for a wider range of organizations.

For enterprises, researchers, and cloud providers, ZAYA1 is more than a technical curiosity: it’s a powerful proof point that large-scale, efficient AI training on AMD hardware is not only possible — it’s already happening.

FAQs

What is ZAYA1?

ZAYA1 is a large-scale Mixture-of-Experts (MoE) foundation model developed by Zyphra, trained entirely on AMD hardware (MI300X GPUs) and networking.

Why is training ZAYA1 on AMD hardware significant?

It demonstrates that AMD’s GPU (Instinct MI300X), networking (Pensando Pollara), and software (ROCm) stack is mature enough for frontier-scale AI training, offering a viable alternative to other platforms.

What is a Mixture-of-Experts (MoE) model?

An MoE model activates only a subset of its “experts” (sub-networks) for each input, enabling large parameter capacity with lower computation per inference or training step, thereby improving efficiency.

How large is ZAYA1?

ZAYA1 has 8.3 billion total parameters, but only 760 million parameters are active at a time during inference/training.

What performance benchmarks does ZAYA1 achieve?

According to Zyphra, ZAYA1-base matches or outperforms models such as Qwen3-4B and Gemma3-12B on reasoning, mathematics, and coding tasks, and outperforms Llama-3-8B and OLMoE.

What infrastructure was used to train ZAYA1?

The training cluster comprised 128 nodes × 8 AMD MI300X GPUs, connected via AMD Pollara 400 Gbps networking and supported by IBM Cloud’s high-performance fabric.

What architectural innovations did Zyphra introduce in ZAYA1?

Key innovations include Compressed Convolutional Attention (CCA) for memory efficiency, an MLP-based expert router, and lightweight residual scaling for training stability.