Marvell: AI Infrastructure Bottlenecks Shift from Compute to Connectivity
At the Optica Executive Forum, Dave Lazovsky, EVP and GM of the Data Center Networking Group at Marvell Technology, argued that AI infrastructure has reached an inflection point where system-level efficiency—not raw compute—is the defining constraint. As AI workloads evolve toward reasoning models, mixture-of-experts architectures, and agentic workflows, the pressure on memory systems and interconnects is scaling faster than compute capability itself.
Lazovsky emphasized that current AI clusters remain structurally inefficient, with many systems achieving only 25–30% utilization due to memory bottlenecks. In many deployments, the majority of time is spent waiting on data movement across the network. This has profound implications for both performance and energy: in some cases, more than half of total system power is consumed by moving data rather than performing computation.
The result is a shift in architectural priority. Networking—across scale-up, scale-out, and increasingly “scale-across” domains—is becoming the central design challenge. Lazovsky noted that inference infrastructure will dominate data center investment over the remainder of the decade, requiring systems that are not only high performance but economically efficient. This places new emphasis on latency, bandwidth, and cost per bit, particularly within tightly coupled scale-up domains.
To address this, Marvell is advancing a continuum of interconnect technologies spanning copper, co-packaged optics (CPO), and ultimately photonic fabric architectures. The company’s photonic fabric approach—originating from its Celestial AI acquisition—targets sub-200ns latency, multi-rack reach, and significant reductions in power consumption by eliminating DSPs and moving toward analog-driven optical interconnects. Lazovsky stressed that more integrated optical designs also improve reliability and serviceability, reinforcing observations shared earlier in the day by hyperscalers.
At the system level, Marvell is positioning itself as an end-to-end connectivity provider, combining custom silicon, optical engines, switching, and storage interconnects into tightly co-designed solutions. By working backward from hyperscaler workloads, the company aims to enable modular transitions from copper to optics without requiring disruptive redesigns—supporting a gradual but inevitable shift toward optical-first AI infrastructure.
- AI infrastructure bottlenecks are shifting from compute to memory access and network connectivity
- Many AI systems operate at only ~25–30% utilization due to data movement constraints
- Data movement can account for >50% of total system power in some workloads
- Inference workloads will drive the majority of AI infrastructure investment going forward
- Scale-up networks are becoming the primary performance constraint vs. traditional scale-out
- Optical interconnects (NPO/CPO) are critical to reducing power and improving latency
- Photonic fabric targets:
- <200ns XPU-to-XPU latency
- ~2.5 pJ/bit energy efficiency
- 50m reach (beyond rack scale)
- Bandwidth scaling to 100s of Tbps per package
- Cost targets for scale-up interconnects approach ~$0.05/Gbps (competitive with copper)
- Transition path: copper → NPO → CPO → photonic fabric
- Architecture trend: multi-rack scale-up pods interconnected via scale-out networks
- Enables memory disaggregation, in-network collectives, and larger XPU domains
“Compute remains a prerequisite—but it’s no longer the constraint. In many of these systems, over half the power is spent just moving data, not computing it.”
