Inference Moves Next Door: The Distributed Compute Thesis

The first wave of AI infrastructure was about training: a few giant campuses in remote, power rich places. The next wave is inference, the work of running models every time someone asks a question or an agent takes an action. Inference is already the majority of AI compute, it never switches off, and it pays rent. It also behaves differently from training. It can be split into small jobs, it cares about latency, and it can run on hardware that is not a top-end training GPU. That combination is pushing compute out of the mega campus and into metro areas, existing buildings, heat reuse sites, telecom edges and enterprise basements. We think this shift creates a new layer of venture scale companies, and that the winners will be the ones who solve power, placement and cost per token at the same time.

This piece covers why the shift is happening, where compute is going, which parts of the stack are investable, and the questions we ask founders building here.

Inference is now the main event

Inference crossed over this year. Deloitte forecast that inference would make up about two thirds of all AI compute in 2026, up from roughly one third in 2023 (Deloitte TMT Predictions 2026). McKinsey’s demand model has AI inference growing about 35% a year, from roughly 21 GW of global data center demand in 2025 to about 93 GW by 2030, overtaking training (McKinsey, Dec 2025).

The economics are different too. Training is a large, lumpy capital expense that is hard to tie to revenue. Inference is a recurring operating cost tied directly to usage, so every new user, copilot seat and agent workflow adds load. Agentic workloads make this sharper, because one agent task can chain dozens of model calls where a chatbot made one.

Three properties of inference matter for where it runs:

It is divisible. Requests can be served independently, so inference does not need one tightly synchronized cluster. McKinsey puts typical inference racks at 30 to 150 kW, versus 100 kW to 1 MW for frontier training.
It is latency sensitive. Real time apps, voice and agents want compute near users and data. Training can tolerate distance; inference often cannot.
It is cost per token driven. Once a model is good enough, buyers optimize price, power and availability, which opens the door to new hardware and new sites.

The mega campus is hitting walls

Power, not chips, now sets the pace of AI build out. JLL’s 2026 outlook puts the average US wait for a utility grid connection at about four years, and about seven years for a 100 MW connection in Northern Virginia (Construction Owners, summarizing JLL). A modern data center shell takes 18 to 24 months to build, so in many markets the building is finished years before the power arrives.

The queue numbers explain why. ERCOT’s large load queue stood near 410 GW in April 2026, roughly five times the Texas grid’s generating capacity, with data centers about three quarters of it. Large power transformers carry two to four year lead times. McKinsey reports time to market in constrained regions has stretched from 12 to 18 months in 2022 to 36 months or more.

For training, operators can work around this by going somewhere remote and building their own generation. For inference, remote is a worse answer: it adds latency and distance from customers’ data. That leaves a gap between where inference demand is and where large blocks of new power can be found. The gap is the opportunity.

Where inference is moving

Inference is not moving onto phones. Deloitte is clear that most inference will still run on data center and enterprise server hardware rather than cheap edge chips. What is changing is which data centers: smaller, more numerous, and closer to people and power. McKinsey expects a shift from large, power hungry facilities toward smaller, modular, distributed sites, with a significant share of inference moving to the edge.

We see five tiers forming:

Tier	Where	Why it works	Main constraint
Metro inference hubs	Tier 2 cities and the edges of existing cloud campuses	Low latency to large user bases, power 12 to 24 months faster than tier 1 hubs	Still needs utility power at scale
Buildings and stranded power	Office towers, industrial sites, existing substations with spare capacity	Uses power and space already connected to the grid	Heat, floor loading, landlord deals
Heat reuse sites	District heating plants, pools, greenhouses	Waste heat becomes a revenue line or a free cooling sink	Matching compute load to heat demand
Telecom edge	Central offices, cable headends, regional hubs	Real estate, fiber and power already in place	Utilization, carrier economics
Enterprise and sovereign	On prem racks, national clouds	Data residency, security, regulated industries	Procurement cycles

The heat reuse model is no longer a novelty. UK startup Deep Green has run small immersion cooled data centers that heat public swimming pools, cutting one pool’s gas use by about 62% (Data Centre Review). Telecom operators are moving too: at NVIDIA’s GTC 2026, several major carriers including AT&T and Comcast announced plans to run GPU inference across their distributed network sites (Nvidia). Analysts caution that cell tower GPUs are not yet economical, and that metro sites such as central offices are the realistic near term opportunity (Data Center Knowledge).

Startups are already building out these tiers. Flume, a New York fiber operator with access rights inside buildings across the city (Light Reading), is turning that access into in building GPU inference sites. Simply Silicon, based in Calgary, places inference compute at district heating plants and feeds the GPUs’ waste heat into the heating network. General Compute is building an inference cloud on chips other than NVIDIA’s, helped by a reported customer deal worth $100 million in annual recurring revenue (Runtime Wire). Underneath all of them, Lightmatter builds photonic interconnects that move data between chips with light (Lightmatter), which matters more as compute spreads across more, smaller sites.

Sovereignty is a further tailwind. Deloitte expects close to $100 billion of investment in sovereign AI compute in 2026, which favors local, smaller deployments over a handful of US mega campuses.

The silicon is diversifying

NVIDIA still dominates, but inference is where its grip loosens. Purpose built inference chips trade flexibility for lower cost per token, lower latency or lower power, and buyers running a fixed model at scale care about exactly those things.

The market has priced this in over the past year:

NVIDIA paid to own the idea. In December 2025 NVIDIA signed a non-exclusive licensing deal for Groq’s inference technology and hired its founder and senior team. CNBC reported the price at about $20 billion; NVIDIA described it as a license, not an acquisition (Constellation Research).
Independent challengers reached public markets. Cerebras went public in May 2026, and Etched is reported to have raised close to $1 billion for a transformer only inference chip (Hashrate Index).
Hyperscalers are building their own. AWS has said more than half the tokens on its Bedrock inference service run on its own Trainium chips (Data Center Knowledge). Google’s Ironwood TPU was its first designed specifically for inference.
Inference chips are a real market. Deloitte expects inference optimized chips to exceed $50 billion in 2026.

Two adjacent layers matter as much as the accelerator itself. Memory is the bottleneck for serving large models, which is why SRAM heavy and memory centric designs are drawing capital. Interconnect, including photonics, determines how cheaply many smaller chips can act as one, which matters more when compute is spread across many sites.

For distributed inference, this diversity is the enabler. A building or a heating plant cannot host a 100 MW training cluster, but it can host racks of efficient inference hardware matched to the power and cooling it has.

Where startups win

Distributed inference creates value at every layer, but not every layer suits venture capital.

Orchestration is the control point. Once compute lives in hundreds of sites with different chips, prices and power profiles, someone has to decide where each request runs. The software that routes workloads across that fleet sees all the demand, compounds data on price and performance, and is hard to rip out. That is where we expect the largest software outcomes.

Site developers win on access, not hardware. The scarce asset is energized, permitted space: a building with spare substation capacity, a heating plant that will pay for heat, a carrier office with fiber. Companies that sign repeatable contracts for these sites, and standardize the deployment, can scale far faster than a greenfield builder. The risk is that they become capital heavy real estate businesses, so we look closely at who owns the hardware and how it is financed.

Silicon and power are real, but capital hungry. Chip companies can produce very large outcomes, as Groq showed, but they need hundreds of millions before revenue. Power and cooling businesses often look more like project finance. We back these selectively, usually where the company sells technology rather than owns the assets.

What would make us wrong

Efficiency outruns demand. McKinsey notes that better hardware, smaller models and new precision formats could slow inference demand growth to 4 to 7% a year. If cost per token falls faster than usage rises, spare hyperscale capacity could absorb inference without new sites.
The hyperscalers solve power first. Behind the meter gas, nuclear deals and grid reform could shorten time to power and pull inference back into large campuses.
Utilization never fills small sites. Distributed capacity only pays if it runs hot. Telecom edge projects have stalled before for exactly this reason.
Hardware cycles strand assets. A site built around one chip generation can lose its cost edge within 18 months.
The incumbent absorbs the category. NVIDIA’s Groq deal shows the largest player will buy what threatens it. That is a good exit for early investors, but it caps independent outcomes.
Permitting and community push back reach small sites. Opposition has so far focused on large campuses; it could spread to compute in mixed-use buildings.

How we evaluate companies here

When founders pitch us a distributed inference company, these are the questions we work through:

What is your all in cost per million tokens, and how does it compare with a hyperscaler today? We want the full stack: hardware, power, cooling, site, network and staff.
How fast can you energize a new site? Months, not years, is the advantage. We ask for the record on the last three sites.
Who owns the hardware, and how is it financed? Equity funded GPUs are an expensive way to build a real estate business.
What is utilization, and who is the demand? Signed offtake or committed customers beat a pipeline of interest.
Is the site contract repeatable? One heating plant is a project; a standard agreement signed with fifty is a company.
How hardware agnostic is the stack? The winners will move to the next chip generation without rebuilding.
What do you earn besides compute? Heat sales, grid services and data residency premiums improve unit economics and are harder to copy.
What is the moat in three years? Site access, routing data and customer workflows compound. A single hardware advantage does not.

We are actively looking at companies across this stack. If you are building here, we would like to hear from you Pitch Us Here

Sources

This article is for informational purposes only and is not an offer to sell or a solicitation of an offer to buy any security.