The Evolution of AI Server Racks: Power Density, Liquid Cooling and 800V DC

AI computing infrastructure is undergoing a major architectural transformation. As large language models, high-performance computing, and AI inference workloads continue to expand, traditional server rack designs are facing increasing demands for power density, thermal management, power distribution, and network connectivity.

The next generation of AI server racks is evolving toward higher power density, full liquid cooling, high-voltage DC power distribution, and tightly integrated rack-scale computing architectures. At the same time, the separation of AI training and inference workloads, heterogeneous computing, and the integration of computing with energy infrastructure are becoming important industry trends.

Image source: itserverrack.com

1. Increasing Power Density: Two Different Development Paths

AI server rack power density is rising rapidly, but training and inference infrastructure are following different development paths.

For high-performance AI training, rack power requirements are moving from approximately 40 kW toward 120–140 kW and 190–230 kW. Future architectures, including NVIDIA Rubin Ultra and Kyber-related rack-scale systems, point toward potential rack designs reaching 600 kW or even megawatt-class power levels. GPU power consumption is also increasing, with individual accelerators moving beyond 1,000W and potentially reaching 2,000W in future generations.

Inference infrastructure, particularly large-scale Token factories, follows a more balanced approach. Rack configurations in the 40–60 kW range using direct-to-chip liquid cooling are expected to remain attractive for many deployments. Rather than pursuing maximum density, operators prioritize total cost of ownership (TCO), operational stability, power availability, and maintenance requirements.

This distinction is important. Although individual GPU power consumption continues to increase, improvements in computing efficiency can reduce the energy required to generate each Token. Lower energy consumption per Token is an important foundation for the commercial scalability of AI inference services.

2. Thermal Management: Liquid Cooling Becomes Essential

Air cooling is becoming increasingly difficult to apply to high-density AI computing infrastructure. Liquid cooling is therefore moving from a specialized option toward a core component of next-generation data center design.

Direct-to-chip liquid cooling is currently one of the most practical approaches for both existing facilities and new AI inference clusters. It transfers heat directly from processors through cold plates and liquid circulation systems, reducing dependence on high-speed internal server fans.

For rack densities of approximately 40–140 kW, direct liquid cooling offers a relatively mature engineering solution. Depending on the overall facility design and operating conditions, advanced cooling architectures can target a Power Usage Effectiveness (PUE) of approximately 1.10–1.15.

For future high-density training clusters and tightly integrated supernodes, immersion cooling and advanced two-phase cooling technologies may become additional options. Certain specialized designs could support rack-level thermal loads exceeding 200 kW, with potential expansion toward megawatt-class configurations. Under suitable conditions, highly optimized facilities may target PUE values in the 1.03–1.08 range.

Another important development is the expansion of liquid cooling beyond GPUs. High-speed optical modules, power distribution components, and advanced switching chips are also generating considerable heat. Integrating thermal management across the entire system can help eliminate localized hotspots and improve overall operating stability.

Air cooling will continue to serve lower-power inference systems, storage equipment, and conventional enterprise applications. However, it is becoming less suitable as the primary cooling solution for newly built high-density AI computing facilities.

3. Power Architecture: The Transition Toward 800V DC

Power distribution is becoming one of the most important engineering challenges in high-density AI data centers.

When rack power consumption exceeds 100 kW, conventional low-voltage AC distribution requires increasingly high currents. This creates challenges involving cable sizing, electrical losses, heat generation, busbar design, and installation complexity.

The industry is therefore exploring a transition from traditional low-voltage AC systems toward 240V high-voltage DC (HVDC) and ultimately 800V DC power architectures.

In an 800V DC architecture, higher-voltage power can be distributed closer to the computing equipment and converted to the required voltage levels near the GPU or other loads. This approach can reduce current, simplify certain distribution stages, and improve overall power delivery efficiency. System efficiency targets above 96% are possible in suitably designed architectures, although actual performance depends on conversion topology and operating conditions.

AI workloads also introduce rapid changes in power demand. GPU utilization can fluctuate significantly within short periods, creating transient electrical loads that conventional power systems must manage carefully.

Two levels of energy buffering are therefore becoming relevant:

• Rack-level buffering: Supercapacitors or other fast-response energy storage components can help mitigate millisecond-scale power fluctuations.
• Facility-level energy storage: Battery energy storage systems can provide support over longer periods, helping manage minute-scale load variations and broader power requirements.

This combination can improve power stability and support the continuous operation required by large-scale AI inference facilities.

4. Hardware Architecture: From Independent Servers to Rack-Scale Supernodes

Traditional AI infrastructure often consists of independent eight-GPU servers connected through external InfiniBand or Ethernet switches. While this architecture offers flexibility, data movement between servers can introduce communication overhead and limit scaling efficiency.

The next generation of computing infrastructure is moving toward tightly integrated rack-scale supernodes.

In these architectures, GPUs, high-bandwidth memory (HBM), high-speed interconnects, optical switching, power distribution, and liquid cooling systems are designed as part of a more integrated computing platform.

Technologies such as NVIDIA NVLink and other high-speed interconnect architectures help establish high-bandwidth communication between accelerators. Shorter data paths and more coordinated hardware integration can improve communication efficiency for large-scale AI workloads, including Mixture of Experts (MoE) models.

However, maximum integration is not necessarily the best solution for every workload.

AI infrastructure is increasingly being divided into separate training and inference resource pools. Training clusters prioritize maximum computing density and tightly integrated supernodes, while inference facilities emphasize standardized rack configurations, balanced power density, operational stability, and cost efficiency.

Heterogeneous computing is another emerging direction. Domestic NPUs, international GPUs, and dedicated AI inference ASICs may coexist within the same data center campus. Unified resource scheduling and energy management platforms can help allocate different workloads to the most appropriate hardware.

5. Connectivity: The Shift from Copper to Optical Interconnects

As AI clusters become larger and more interconnected, communication bandwidth and latency are increasingly important performance factors.

In the near term, 400G and 800G optical transceivers are becoming important components of high-performance data center networks. Optical connections are increasingly used for both rack-to-rack communication and high-speed network infrastructure.

For large AI clusters, lossless InfiniBand and RoCE networking technologies help support high-throughput communication and reduce packet loss during intensive data exchange.

Looking further ahead, optical switching matrices integrated into rack-scale systems could enable more flexible network topologies. Dynamic optical connectivity may help optimize communication paths, reduce unnecessary switching stages, and adapt to changing AI workload patterns.

This development is particularly relevant to large-scale distributed training and inference systems, where communication bottlenecks can significantly affect overall computing efficiency.

6. Green Computing: Integrating Data Centers with Renewable Energy and Storage

Energy availability and electricity costs are becoming strategic considerations in AI infrastructure development. As computing density increases, data center operators must consider not only server performance but also the availability, cost, and reliability of electricity.

One important direction is the integration of AI data centers with renewable energy, battery storage, and intelligent energy management systems.

Solar power, battery energy storage systems (BESS), and grid electricity can be coordinated through an Energy Management System (EMS). This allows operators to consider electricity prices, renewable generation forecasts, and computing demand when managing energy consumption.

For facilities located in areas with abundant renewable resources, solar-plus-storage systems can provide additional energy flexibility. Solar power and hybrid energy solutions for telecommunications infrastructure demonstrate how renewable generation, battery storage, and intelligent power management can support remote and energy-intensive infrastructure. Similar energy integration principles are increasingly relevant to distributed computing facilities.

Standardized rack designs and prefabricated infrastructure are also becoming important. Open standards such as the Open Compute Project (OCP) help promote hardware compatibility and modular deployment, potentially reducing data center construction time and simplifying future expansion.

At the operational level, integrated monitoring is essential. Rack power consumption, coolant flow rates, chip temperatures, queries per second (QPS), and Token throughput can be collected and analyzed through unified management platforms.

This visibility enables operators to manage computing loads, respond to thermal or electrical constraints, and optimize resource allocation according to real-time operating conditions.

7. Choosing the Right Rack Architecture for Training and Inference

There is no single rack configuration suitable for every AI computing workload. The choice depends on computing objectives, power availability, cooling infrastructure, capital expenditure, and operational requirements.

For AI training clusters: High-density rack-scale supernodes offer advantages in computing integration, interconnect bandwidth, and large-model training performance. Operators must be prepared for higher capital expenditure, more complex liquid cooling infrastructure, and advanced power distribution systems, potentially including 800V DC architectures.

For AI inference and Token factories: Balanced rack density is often a more practical approach. Configurations in the 40–60 kW range using direct-to-chip liquid cooling can offer a reasonable balance between performance, reliability, energy efficiency, and maintenance requirements.

Not every inference facility needs to adopt 100 kW or higher-density racks. Excessive rack density can increase the requirements for electrical infrastructure, battery storage, cooling distribution, and facility engineering.

For many inference-oriented projects, a carefully designed 40–60 kW rack architecture may provide a more manageable path toward long-term operation and commercial scalability.

The evolution of AI server racks is being driven by increasing power density, liquid cooling, high-voltage DC distribution, rack-scale computing integration, and optical interconnect technologies.

At the same time, the separation of training and inference workloads is creating different infrastructure requirements. Training clusters continue to pursue higher computing density and tightly integrated supernodes, while inference Token factories place greater emphasis on balanced rack configurations, energy efficiency, and operational stability.

The next generation of AI data centers will therefore depend on more than high-performance GPUs. Server architecture, rack infrastructure, cooling systems, power distribution, renewable energy, and battery storage must work together as an integrated system.

Ultimately, the competitive advantage of AI infrastructure will come from balancing computing performance, electricity consumption, thermal management, and total cost of ownership.