PyTorch on TPU and Trainium: How Far Can You Really Escape Nvidia?

Running PyTorch on TPU and Trainium now takes a device switch with TorchTPU and TorchNeuron. We break down the numbers, limits, real costs and who should try first.

Main report

Running PyTorch on Google TPUs or AWS Trainium used to mean an intermediary library and a fair amount of rework. On October 19–20, 2026, just ahead of PyTorch Conference, both companies open-sourced a native PyTorch backend: TorchTPU (Google, co-developed with Meta's PyTorch team) and TorchNeuron (AWS, in public beta). The shared goal: PyTorch code written for Nvidia GPUs should run on their chips with very few changes.

• Switch the device, don't rewrite. For TorchNeuron, AWS says the only change is model.to('neuron'). For TorchTPU, Google promises "minimal code changes"; in Hugging Face Transformers v5 you pass device_map="tpu".
• Both execution modes. Both support eager mode (op by op, easy to debug) and torch.compile (whole-graph compilation for production speed).
• Distributed training out of the box. DDP, FSDP2 and DTensor work on both; TorchNeuron adds tensor and pipeline parallelism.
• Wired into the serving ecosystem. TorchTPU ships backends for vLLM and SGLang; TorchNeuron has vLLM Neuron plus support in Transformers v5, Diffusers and trl.
• Apache 2.0 open source for both projects on GitHub.

Analysis

Analysis: is one line of code enough to leave Nvidia GPUs?

Short answer: enough to try, not enough to migrate. The real change is in how the backends integrate; actual savings depend heavily on the models you run.

What is different from before?

The key difference is PrivateUse1, PyTorch's official extension point for new hardware. A tensor on a TPU or Trainium is a normal PyTorch tensor, not a wrapper. That is why libraries such as Hugging Face, vLLM and TorchTitan no longer need a separate branch for each chip.

Why does it run without a rewrite?

When you call torch.compile, PyTorch uses TorchDynamo to capture the compute graph. Instead of handing that graph to a GPU compiler, TorchTPU lowers it to StableHLO and lets XLA optimize it for TPUs. TorchNeuron follows a similar path through Torch-MLIR, StableHLO and the Neuron compiler. Developers keep writing familiar PyTorch; translation to the chip happens underneath.

TorchTPU's "Fused Eager" mode automatically groups many small ops into larger chunks before sending them to the chip. Google has reported gains of 50% to more than 100% over op-by-op execution. That is a vendor number with no independent measurement yet.

Who is affected, and how?

• ML engineers: can run the same training loop on GPU, TPU and Trainium to compare. Custom kernels still require learning Pallas (TPU) or NKI (Trainium).
• Engineering managers: gain leverage when negotiating GPU pricing. A cloud contract is no longer tied to CUDA if models run on several chips.
• Small and mid-sized businesses: most consume models through APIs, so the effect is indirect. Teams self-hosting open models (Qwen, Llama) on Google Cloud or AWS benefit directly.
• Marketers and product teams: nothing changes right away. Any benefit would arrive as cheaper API pricing from providers using these chips.

What does it actually cost?

Neither company published new pricing. The only cost figure is Deepgram's: about 60% more audio processed per dollar than its current GPUs. The table below is a hypothetical example applying that figure at two scales, with an assumed migration cost of 30,000 USD (roughly two engineers for 1–2 months).

The takeaway: on a small bill, switching chips rarely pays off. On a large bill with a stable model, the math starts to work. The 60% figure comes from one customer and one workload (audio); your model could do better, worse, or see no gain at all.

Risks and limitations

• Operator coverage is incomplete. In the August 2026 PyTorch test run, about 22,000 tests were skipped because TorchNeuron lacked a matching operator. The first release does not support dynamic shapes, sparse tensors or NestedTensor; FP8 is in private beta.
• Dynamic shapes remain a TPU weak spot. Google acknowledges that changing sequence lengths or batch sizes trigger recompilation, and first-iteration latency is high. A chatbot with constantly varying prompt lengths is exactly the kind of workload affected.
• Lock-in just moves. TPUs exist only on Google Cloud and Trainium only on AWS. Leaving CUDA can mean tying yourself more tightly to one cloud.
• CUDA-only code will not run. CUDA graphs, NCCL-specific calls and hand-written CUDA kernels still need rewriting.

What still needs verification

• The 52% MFU and 8,192-chip figures are Google's internal TorchTitan tests, with no independent comparison against similarly priced GPUs.
• The Zenteiq (about 4x) and Deepgram (+60% per dollar) figures are customer quotes selected by the vendors.
• The "85% of AI apps use PyTorch" figure cited by TechCrunch has no named source; TBN Grow could not verify it.
• How to check yourself: take a model in production, measure tokens/second and cost per 1 million tokens on your current GPUs, then run the same model on TPU v6e or Trainium2 with the same input lengths.

TBN Grow's take

This is an infrastructure-software shift with real value for teams that train or self-host open models on Google Cloud or AWS. Teams with GPU bills in the tens of thousands of dollars per month should test now on a stable model. Small teams, API users, or workloads with highly variable inputs should wait one or two releases, until operator coverage and dynamic-shape support mature.

What to watch

What to watch when running PyTorch on TPU and Trainium

What you can do now

• [ ] Find every direct cuda call in your code; replace it with the torch.accelerator API to make switching chips easier.
• [ ] Try TorchTPU: pip install torch_tpu==0.2.0 on a TPU v6e VM (the project README calls this the easiest start; add --extra-index-url https://download.pytorch.org/whl/cpu to avoid installing the CUDA build).
• [ ] Try TorchNeuron: pip install torch-neuronx on a trn2 instance and switch the device to neuron.
• [ ] Run your own project's test suite to find unsupported operators before estimating costs.
• [ ] Compare cost per 1 million tokens or per hour of audio, not just speed.

Timeline and what is not yet announced

• AWS plans to make TorchAO and FP8 generally available in the next release and close operator gaps over time; no dates yet.
• Google aims to add dynamic-shape support to torch.compile and a library of precompiled kernels; no timeline yet.
• Google has not labeled TorchTPU as beta or GA, nor published a full list of supported TPU generations.
• Nvidia has not responded officially.

Further reading: China's AI chip market shifts: Nvidia at 34%, Huawei at 28% and the payback problem facing AI infrastructure spending.

Verification sources: Google Cloud Blog, Google Developers Blog, TorchTPU on GitHub, PyTorch Blog – TorchNeuron, AWS What's New, TechCrunch.