What is the NVIDIA H200 Chip?

The NVIDIA H200 in short: an AI accelerator on NVIDIA’s Hopper architecture, announced on November 13, 2023. Systems and cloud instances were scheduled to ship from the second quarter of 2024. It carries 141 GB of HBM3e memory with 4.8 TB/s of bandwidth. The PCIe version, the H200 NVL, sells for about €38,000 in Germany, and an eight-GPU HGX H200 server runs about $350,000 to $500,000.

This guide covers the H200’s release date, specs, price, power draw, and export status, and how it compares with the H100 and the China-market H20.

What’s New

• First H200 deliveries to China: According to the Financial Times, ByteDance and Tencent have each received about 10,000 chips. Beijing wants most of the hardware to stay outside the mainland, largely in Hong Kong (The Next Web).

• Prices have moved: The old rule of thumb of $25,000 to $40,000 per card is out of date. An 8-GPU server now costs $350,000 to $500,000, and cloud rental starts around $2 to $5 per GPU hour (Jarvislabs).

• Successors are on the market: Blackwell and Rubin follow Hopper. The H200 is no longer NVIDIA’s most powerful chip.

NVIDIA H200 at a Glance

What Is the NVIDIA H200?

The H200 succeeds the H100. It uses the same Hopper chip but pairs it with faster and much larger memory. That is the key point: large language models are often limited by memory bandwidth at runtime, not by raw compute. NVIDIA says the H200 boosts LLM inference speed by up to 2 times compared with the H100 (NVIDIA). HGX H200 boards are compatible with existing HGX H100 systems, so operators can upgrade on the same server infrastructure.

It is used for AI training and inference of large models, for high-performance computing (HPC), and in cloud data centers.

NVIDIA H200 Release Date

NVIDIA announced the H200 on November 13, 2023. Systems and cloud instances were due to begin arriving in Q2 2024. The initial cloud providers listed by NVIDIA are Amazon Web Services, Google Cloud, Microsoft Azure, Oracle Cloud Infrastructure, Core Weave, Lambda, and Vultr. Server vendors listed by NVIDIA include Dell, HPE, Lenovo, Supermicro, ASUS, and GIGABYTE. For how Germany fits into this hardware wave, read Nvidia and Germany.

Specs: H200 SXM vs H200 NVL

Most cloud providers offer the SXM version. If you want a single card for your own server, you buy the NVL. All figures come from NVIDIA’s product page, which marks them as preliminary.

H100 vs H200

The H200 pays off when models do not fit into the H100’s memory or when bandwidth is the bottleneck. For smaller models, the H100 is often the cheaper choice.

H20 vs H200: The China-Market Chip Compared

The H20 is a cut-down Hopper variant built for the Chinese market.

On memory and bandwidth, the two chips are closer than most people expect. The big gap is compute: reports put the H200 at roughly six times the performance of the H20.

How Much Does the NVIDIA H200 Cost?

The H200 is not sold like a graphics card. The price depends on the form factor.

Figures come from the Jarvislabs price guide. Prices change quickly, so check with the provider before you decide.

Buy or Rent an H200?

For companies, the key question is utilization, not price. A simplified example, excluding power, cooling, operations, and depreciation:

• Cloud at a specialist provider: 8 GPUs × $4.50 = about $36 per hour.

• Buying a server for $350,000: break-even after about 9,700 hours, roughly 13 months of nonstop use.

• Buying a server for $500,000: break-even after about 13,900 hours, roughly 19 months of nonstop use.

• At a hyperscaler (about $10.60 per GPU hour), break-even comes much sooner.

For the basics of buying compute as a service, see Data Center as a Service.

NVIDIA H200 Power Consumption

The H200 SXM draws up to 700 W, and the H200 NVL up to 600 W. A server with eight SXM GPUs reaches 5.6 kW for the GPUs alone, before CPUs, networking and cooling. The SXM version is designed for liquid-cooled HGX systems, while the NVL runs in air-cooled servers.

Example at €0.30 per kWh: one 700 W GPU running nonstop costs about €0.21 per hour, roughly €1,840 per year. That is the GPU only. Actual consumption depends on load and cooling.

H200 Export to China: Where Things Stand

The H200 sits at the center of the fight over AI chip exports. The sequence, as reported:

1. First: the US government signals it will allow sales to selected customers in China, with a 25% cut for the US. Licenses are reviewed case by case.

2. Then: Beijing holds the chips at customs at first and later clears purchases only under conditions.

3. Most recently: according to the Financial Times, ByteDance and Tencent have each received about 10,000 chips. US licenses allow far more per company (reports cite 75,000 to 100,000). Beijing wants most of the hardware to stay outside the mainland, largely in Hong Kong (The Next Web, Tom’s Hardware).

The H200 is at least two generations behind NVIDIA’s newest chips, which remain barred from China. For background and consequences, read AI chip bans and geopolitics. China’s response is covered in Huawei’s AI hardware, the Huawei CANN vs CUDA debate, and Ant Group’s domestic chips.

H200 Successor

Blackwell and then Rubin follow the Hopper generation. For Blackwell in enterprise use, see the NVIDIA Blackwell RTX Pro 6000 Server Edition. A look at the next generation is in our piece on Nvidia Rubin. The market keeps moving too, for example, with Nvidia’s $20 billion Groq deal and talent deal and the OpenAI and Nvidia partnership. 

While Nvidia did execute a massive $20 billion deal with AI chip startup Groq, it was not a full corporate acquisition. It was legally structured as an asset purchase and non-exclusive IP licensing agreement. Nvidia absorbed Groq's core Language Processing Unit (LPU) intellectual property and hired its core engineering team (including founder Jonathan Ross), but Groq technically continues to exist as an independent cloud business. 

Who Should Use the H200?

• Cloud users with large models: yes, if the models do not fit into the H100’s memory.

• Companies with constant load: check buying; see the example above.

• Teams with small models or tests: rent, or use a smaller GPU.

• Anyone who needs the latest hardware: keep an eye on Blackwell or Rubin.

Conclusion

The NVIDIA H200 is an H100 with much more and faster memory. That makes it strong for large language models, but not automatically the best choice: price, power draw, and utilization decide. Whether buying, renting, or a smaller setup fits your company is something our AI consulting for businesses can help you work out.

FAQ

When was the NVIDIA H200 released?

NVIDIA announced it on November 13, 2023. Systems and cloud instances were expected from the second quarter of 2024.

What is the NVIDIA H200?

The NVIDIA H200 is a Hopper-generation AI accelerator with 141 GB of HBM3e memory and 4.8 TB/s of bandwidth, built for AI training, inference, and HPC.

How much does the NVIDIA H200 cost?

The PCIe version H200 NVL costs from about €38,000 at retail. An eight-GPU server costs about $350,000 to $500,000. In the cloud, one GPU costs from about $2 to $5 per hour, and around $10 at hyperscalers.

Can I rent an NVIDIA H200?

Yes. Many cloud providers rent H200 GPUs by the hour. Specialist providers are usually cheaper than the big hyperscalers.

How much power does the H200 use?

Up to 700 W for the SXM version and up to 600 W for the NVL version.

What is the difference between H100 and H200?

Both use the Hopper chip. The H200 has more memory (141 vs 80 GB) and more bandwidth (4.8 vs 3.35 TB/s).

What is the difference between H20 and H200?

The H20 is the China-market variant with 96 GB HBM3 and 4.0 TB/s. The H200 has more memory, more bandwidth, and, according to reports, about six times the compute.

What is the difference between H200 SXM and H200 NVL?

The SXM is a module for HGX servers at up to 700 W. The NVL is a double-wide PCIe card at up to 600 W that runs air-cooled in standard servers.

Is the H200 NVIDIA’s most powerful chip?

No. The H200 belongs to the Hopper generation. Newer generations such as Blackwell and Rubin are more powerful.

Can China buy the H200?

Partly. The US allows sales to selected customers for a 25% cut. According to reports, China allows only small quantities in.

Sources

• NVIDIA: H200 Tensor Core GPU

• Jarvislabs: H200 Price Guide

• The Next Web: H200 shipments to China

• Tom’s Hardware: H200 deliveries to ByteDance and Tencent