DEV Community: Grace Gong

TPUs vs. GPUs: What They Are, How They Differ, and Which Workloads Belong on Each

Grace Gong — Fri, 01 May 2026 01:53:27 +0000

If you've worked with machine learning on Google Cloud, you've hit the choice: GPU instance or TPU? Most teams default to GPU because that's what they already know. But as inference costs climb and TPU tooling matures, it's worth understanding what each chip actually does and when one outperforms the other.

This post covers what GPUs and TPUs are, how they work, and which workloads run better on each. It ends with a look at Google's current TPU lineup, including the eighth-generation chips announced at Google Cloud Next 2026.

Why TPUs exist

Image source: Google Cloud

GPUs were originally built for rendering video games. They handle AI workloads well because the underlying math, large parallel floating-point operations is the same. Researchers figured this out around 2012, and GPUs became the default for training neural networks.

Google ran into a problem in 2013. Engineers at Google Brain calculated that if every Android user used voice search for just three minutes a day, Google would need to double its global data center capacity. Running inference on general-purpose GPUs at that scale was too expensive and power-hungry.

Their solution was to build a chip designed specifically for neural network math. The first TPU went into production in Google's data centers in 2015. Google made Cloud TPUs publicly available in 2018. The core idea, strip out everything a GPU carries from its graphics origins and focus entirely on matrix multiplication still drives every TPU generation today.

How a GPU works

Image source: Google Cloud. Some images of GPUs.

A GPU is a parallel processor with thousands of smaller cores. Where a CPU has 8 to 64 powerful general-purpose cores, a high-end GPU like the NVIDIA H100 has thousands of smaller ones that run the same instruction across many data points at once. This is called SIMD (Single Instruction, Multiple Data) parallelism.

GPUs support a wide range of precision formats: FP32, FP16, BF16, INT8, FP8. They run PyTorch, TensorFlow, JAX, CUDA libraries, simulations, rendering pipelines. That broad support is useful, but it means a GPU carries hardware for texture mapping, branch prediction, and other operations that sit completely idle during a matrix multiplication.

The NVIDIA H100 has 80GB of HBM2e memory on-package. Memory bandwidth matters a lot for AI workloads because moving data between memory and compute units is often what limits throughput, not the raw math.

How a TPU works

Image source: Google Cloud

A TPU is built for one job: tensor math. Specifically, the matrix multiplications at the core of neural network training and inference.

The key piece of hardware is the systolic array. In a standard processor, every operation reads inputs from memory, computes, and writes the result back. In a systolic array, data flows through a grid of multiply-and-accumulate units. You load the weights once, pass inputs through the grid, and results flow from unit to unit without going back to main memory. This removes the constant memory round-trips that slow conventional chips.

Google built BF16 support into TPUs from early generations; GPUs added it later. Recent chips support FP8 natively, which helps throughput for inference workloads.

The limitation: TPUs work poorly with dynamic control flow, variable-length sequences, and custom operations. They are best suited for static computation graphs, which is what most transformer models produce.

Side-by-side comparison

When to use a GPU

Image source: Google Cloud.

Recommended GPUs based on workload type.PyTorch-first teams. Most research code on GitHub, most open-source model checkpoints, and most fine-tuning guides assume a GPU. If your team works primarily in PyTorch, starting on GPU is faster.

Models with TensorFlow ops that are not available on Cloud TPU (see the list of available TensorFlow ops)

Models with dynamic inputs. Variable-length sequences, conditional branches, custom CUDA extensions - these work on GPUs and can be tricky to run on TPUs.

Medium-to-large models with larger effective batch sizes

Multi-cloud or on-prem deployments. TPUs only exist in Google Cloud. If your infrastructure is on AWS, Azure, or your own servers, you don't have a choice.

Mixed workloads. If the same team does ML training, scientific simulation, and rendering, GPUs handle all of it. TPUs don't.

Small teams moving fast. GPU tooling (profilers, debuggers, community tutorials) is more mature. Diagnosing a performance problem on a GPU is easier today than on a TPU.

When to use a TPU

Models relying on embeddings: Cloud TPUs feature SparseCores, which are dataflow processors specifically built to accelerate models that heavily use embeddings. This makes them ideal for applications like recommendation systems. - Google Cloud

Training massive deep learning models: If you're building and training large and complex deep learning models, especially large language models (LLMs), Cloud TPUs are designed to handle the immense number of matrix calculations involved efficiently.

Models dominated by matrix computations

Models that train for weeks or months

Models with ultra-large embeddings common in advanced ranking and recommendation workloads

Large-scale transformer training. TPU pods scale to tens of thousands of chips through Google's Inter-Chip Interconnect (ICI). Training something like Gemma on a TPU pod tends to be faster and cheaper per token than an equivalent GPU cluster.

High-volume production inference. TPU v6e (Trillium) and Ironwood were built specifically for inference workloads. Ironwood delivers more than 4x better performance per chip for inference compared to TPU v6e (Trillium).

Models with no custom PyTorch/JAX operations inside the main training loop

Google open-weight models. Gemma 4 (released April 2026) is built and optimized for TPU serving. Google publishes JAX reference implementations for every Gemma variant, and there are community guides for deploying Gemma 4 via vLLM on Cloud TPU.

Cloud TPUs are not suited to the following workloads:

Linear algebra programs that require frequent branching or contain many element-wise algebra operations
Workloads that require high-precision arithmetic
Neural network workloads that contain custom operations in the main training loop

Google's current TPU lineup

TPU v5e, available now

Good starting point. Used for smaller inference workloads and fine-tuning. Lower per-chip cost than newer generations.

TPU v6e (Trillium), available now

4.7x the peak compute of v5e, with 67% better energy efficiency. Scales to 256 chips per pod. Still widely used for inference, particularly for teams where cost per chip-hour matters more than raw throughput. vLLM supports TPU v6e for both offline batch inference and online API serving.

TPU v7 (Ironwood), generally available since late 2025

Announced at Google Cloud Next 2025. Specs per chip: 4,614 FP8 TFLOPS, 192GB of HBM3E memory, 7.37 TB/s memory bandwidth, 9.6 Tb/s inter-chip interconnect. Scales to 9,216 chips in a single superpod, delivering 42.5 FP8 ExaFLOPS per pod. That's more than 4x the performance per chip of
TPU v6e (Trillium) and 10x of TPU v5p.
Each Ironwood chip contains two TensorCores and four SparseCores in a dual-chiplet design. Anthropic's Claude models train and serve on TPUs, and Anthropic signed an agreement to access up to one million Ironwood TPUs through Google Cloud.
Ironwood is the first TPU generation where Google used AlphaChip - a reinforcement learning tool - to design the chip layouts.

TPU 8t and TPU 8i (eighth generation), coming later in 2026

Announced at Google Cloud Next 2026. For the first time, Google has split its TPU lineup into two chips with different architectures for training and inference.

TPU 8t is built for training. A single superpod holds 9,600 chips with 2 petabytes of shared HBM memory and 121 FP4 ExaFLOPS of compute, nearly tripling compute per pod versus Ironwood. ICI bandwidth is 19.2 Tb/s per chip, double Ironwood. The new Virgo Network fabric can link 134,000 chips across a data center and theoretically over 1 million chips across sites. TPUDirect RDMA and TPU Direct Storage bypass the host CPU entirely, doubling bandwidth for large data transfers. Google targets 97% goodput meaning 97% of compute cycles go toward actual learning rather than overhead.

TPU 8i is built for inference. It scales to 1,152 chips per pod and delivers 11.6 FP8 ExaFLOPS. Each chip carries 288GB of HBM, more than the 8t training chip and 384MB of on-chip SRAM, 3x what Ironwood had. Google reports 80% better performance-per-dollar versus Ironwood for inference, and 2x better performance-per-watt.

The 8i uses a new Boardfly interconnect that reduces the maximum number of network hops from 16 to 7. This matters for Mixture-of-Experts models, where data needs to move quickly between expert layers. The chip also replaces Ironwood's SparseCores with a Collectives Acceleration Engine (CAE), which cuts the latency of collective operations by 5x - important when many agents are running concurrently and small latency multiplies across thousands of calls.

The reason the inference chip has more memory than the training chip: large MoE inference is memory-bandwidth-bound. The chip serving tokens needs to stream weights and KV-cache faster than the chip training the model. Both 8t and 8i run on Google's Axion ARM host CPU and use liquid cooling.

More info from TPU Overview

The software side

TPUs run best with a few specific tools:
JAX is Google's ML framework. Its jit, vmap, pmap, and shard_map primitives map directly onto TPU hardware. If you're new to TPUs and want to get the most out of them, JAX is where to start.
MaxText is Google's open-source LLM reference implementation for TPUs, available at AI-Hypercomputer/maxtext on GitHub. It's a practical starting point for training large language models on TPU pods.
Pallas is Google's Python-based kernel language for writing low-level, hardware-aware kernels. Supported on both Ironwood and the eighth-generation chips.
vLLM now has first-class TPU support. You can run offline batch inference or an OpenAI-compatible API server on a Cloud TPU VM with standard configuration.
PyTorch on TPU is in preview as of the eighth-generation launch. If your team is on PyTorch, you can now bring existing models to TPU hardware without rewriting them in JAX.
Google's Gemma 4 (April 2026) is optimized for TPU serving. The google-deepmind/gemma GitHub repo has JAX reference implementations for every model variant.

Summary

GPUs are the practical default for most research and development work. The tooling is mature, the community is large, and most models you'll find online were built on GPUs.
TPUs are worth the switch when you're running workloads at sustained scale on Google Cloud, especially for inference. Ironwood is available today. The eighth-generation 8t and 8i chips, which separate training and inference into dedicated hardware, are coming later in 2026. If you want to try TPUs before committing, Google Colab's free TPU runtime lets you run a JAX or Keras model on one without any setup.

Resources

Google's eighth-generation TPUs: two chips for the agentic era
TPU 8t and TPU 8i technical deep dive - Google Cloud Blog
Ironwood: The first Google TPU for the age of inference
Training large models on Ironwood TPUs - Google Cloud Blog
Performance per dollar of GPUs and TPUs for AI inference - Google Cloud Blog
Building production AI on Google Cloud TPUs with JAX
MaxText: LLM reference implementation for TPUs - GitHub
Gemma open-weight LLM library - Google DeepMind GitHub
Serve and Inference Gemma 4 on TPU
Google Cloud unveils eighth-generation TPUs - TechRadar

TPUSprint

Google Cloud TPU Architecture Versions Explained: From v1 to the Eighth Generation

Grace Gong — Fri, 01 May 2026 01:44:23 +0000

A guide to Cloud TPU generations, what changed between them, and how to choose the right one for your workload

Image source: Google Cloud

If you've looked at Google Cloud TPU pricing or documentation recently, you've probably noticed there are a lot of versions to choose from. TPU v5e, v5p, v6e, Ironwood, now TPU 8t and 8i. Each one has different specs, different use cases, and different tradeoffs. This post walks through every major TPU generation, what changed at each step, and what those changes mean for the people running workloads on them.

The building blocks: what's inside a TPU chip
Before going generation by generation, it helps to know what the key components are, because the names come up repeatedly across every version.

Image source: Google Cloud

Matrix Multiply Unit (MXU). This is the core compute engine inside every TPU TensorCore. It performs the multiply-and-accumulate operations that power neural network math. On most TPU generations up through v5p, the MXU is a 128x128 systolic array - 16,384 multiply-accumulators working simultaneously. Starting with Trillium (v6e), the MXU expanded to 256x256, quadrupling the operations per cycle.

TensorCore. A TensorCore contains one or more MXUs, a vector processing unit (VPU), and a scalar unit. Depending on the generation, a single TPU chip may have one or two TensorCores.

High Bandwidth Memory (HBM). This is the on-chip memory that stores model weights and activations. HBM capacity and bandwidth are often the real bottleneck for large models, not compute. Each generation has brought more HBM and faster access speeds.

Inter-Chip Interconnect (ICI). The network that connects chips inside a pod. ICI bandwidth determines how fast chips can synchronize gradients during training. Higher bandwidth means less time waiting for communication and more time computing.

SparseCore. Introduced in TPU v4, SparseCores are specialized processors for embedding operations - the kind that power recommendation systems and large vocabulary models. v5p and Ironwood have four SparseCores per chip. v6e has two.

Topology. How chips are wired together in a pod. Earlier generations use a 2D torus (chips connect to four neighbors). Starting with v4, Google moved to a 3D torus for larger-scale pods, which reduces the maximum number of hops between any two chips and cuts communication latency.

Generation by generation

TPU v1 (2015, internal only)
The first TPU was built for one purpose: inference. It was not publicly available and could not train models. The chip contained a 256x256 systolic array of 8-bit multiply-accumulators, yielding 92 TOPS of INT8 compute. It consumed about 40 watts, which was remarkably efficient for its time.
Google kept it secret for over a year. When Sundar Pichai announced it at Google I/O 2016, he said it had already been running in Google's data centers for more than a year, powering services like Search, Maps, and Street View. The entire motivation for building it was to avoid doubling Google's data center capacity to handle growing neural network inference demand.
Not available on Google Cloud. Historical context only.

TPU v2 (2017)

TPU v2 was the first generation capable of training neural networks. This required a fundamental change: switching from 8-bit integer math to bfloat16, a 16-bit floating-point format that Google invented specifically for this purpose. BF16 retains the same 8-bit exponent as FP32, which gives it the wide dynamic range training needs, while cutting memory use in half compared to FP32.
Each v2 chip delivered approximately 45 TFLOPS of BF16 compute. A single board held four chips. A full v2 Pod contained 512 chips connected by a 2D torus ICI.
TPU v2 is no longer recommended for new workloads but represents an important milestone: it established bfloat16 as a training standard that the broader ML community eventually adopted.

TPU v3 (2018)

TPU v3 more than doubled the compute of v2, reaching approximately 420 TFLOPS per chip. To handle the increased power density, Google switched to liquid cooling, the first TPU generation to require it.
Each v3 chip contained two TensorCores, each with a 128x128 MXU. A full v3 Pod scaled to 1,024 chips in a 2D torus. The v3 architecture is described in detail in Google's paper "A Domain Specific Supercomputer for Training Deep Neural Networks."
TPU v3 is documented on Google Cloud and remains available in some configurations, though newer generations offer substantially better performance per dollar.

TPU v4 (2021)

TPU v4 was a major architectural shift. Performance more than doubled over v3, and Google moved from a 2D torus to a 3D torus interconnect. In a 3D torus, each chip connects to six neighbors instead of four. For a 4,096-chip pod, this reduces the maximum number of hops between any two chips from roughly 128 to about 48, which meaningfully cuts all-reduce latency during distributed training.
A single v4 Pod contained 4,096 chips. Google also introduced SparseCore in v4, four dedicated processors per chip optimized for embedding-heavy workloads like recommendation models.
TPU v4 has been compared favorably to the NVIDIA A100 in benchmarks: Google's 2023 paper reported TPU v4 running 5 to 87 percent faster on ML workloads depending on the model. TPU v4 also has an inference-optimized variant, v4i, that does not require liquid cooling.

TPU v5e (2023)

TPU v5e is Google's cost-optimized fifth-generation chip. The "e" stands for efficient. Where v5p prioritized maximum performance, v5e was designed to minimize cost per inference query and cost per training FLOP for medium-scale jobs.
Each v5e chip contains one TensorCore with four MXUs, a vector unit, and a scalar unit. The chip uses a 2D torus topology and scales to 256 chips per pod. Google returned to air cooling for v5e. HBM capacity is 16 GB per chip with 819 GB/s bandwidth.
TPU v5e delivers 2.5x better price-performance than v4 for inference workloads. It is currently available on Google Cloud and is a practical starting point for teams new to TPUs, especially for serving workloads where cost matters more than peak throughput.

TPU v5p (2023)

TPU v5p is the performance-focused fifth-generation chip, released alongside v5e. The "p" stands for performance. Each chip has two TensorCores, each with four MXUs, giving it roughly double the compute of v5e per chip. HBM capacity is 95 GB per chip with 2,765 GB/s bandwidth - nearly 6x v5e's memory capacity.
v5p uses a 3D torus topology and scales to 8,960 chips per pod with 4,800 Gbps of ICI bandwidth per chip. Google reports v5p trains large LLM models 2.8x faster than v4 and includes second-generation SparseCores, which deliver 1.9x better performance than v4's SparseCores for embedding-dense models.
v5p is suited for large training runs where you need the largest possible pod size and maximum per-chip compute. Available on Google Cloud in North America (US East region).

TPU v6e, Trillium (2024)

Trillium was announced at Google I/O 2024 and became generally available in late 2024. It is the sixth-generation chip and represents the biggest architectural leap between consecutive generations since v2.
The most significant change: the MXU expanded from 128x128 to 256x256. This quadruples the number of multiply-accumulate operations per cycle. Combined with a higher clock speed, Trillium delivers 4.7x the peak compute of v5e per chip. HBM capacity doubled to 32 GB per chip, and ICI bandwidth doubled to 3,200 Gbps per chip. Trillium is 67% more energy efficient than v5e.
Each v6e chip has one TensorCore with two MXUs (the larger 256x256 array), plus two SparseCores. The topology is a 2D torus scaling to 256 chips per pod, the same footprint as v5e.
On the technical documentation side, Google refers to Trillium as v6e in all APIs and logs. The v6e-8 VM type (all 8 chips attached to a single VM) is optimized specifically for inference, making it easy to serve large models on a single host.
Trillium is available now on Google Cloud in North America (US East), Europe (West), and Asia (Northeast).

TPU v7, Ironwood (2025)

Ironwood is the seventh-generation TPU, announced at Google Cloud Next 2025 and generally available since late 2025. It is purpose-built for inference and large-scale training at the scale needed for frontier models.
Key specs per chip: 4,614 FP8 TFLOPS, 192 GB of HBM3E memory, 7.37 TB/s memory bandwidth, 9.6 Tb/s ICI bandwidth. A full superpod contains 9,216 chips delivering 42.5 FP8 ExaFLOPS. That is 4x better performance per chip over Trillium and 10x over TPU v5p.
Ironwood introduced FP8 as a native precision format, which is critical for inference throughput. It uses a dual-chiplet design: each Ironwood chip contains two TensorCores and four SparseCores connected by a high-speed die-to-die interface. The 3D torus topology returns for large pod configurations, with 3D connectivity for pods of 4x4x4 or larger.
Ironwood is also the first TPU generation where Google used AlphaChip, a reinforcement learning tool, to optimize the physical chip layout.
Anthropic's Claude models train and serve on TPUs. As part of a multi-billion dollar agreement, Anthropic committed to access up to one million Ironwood TPUs through Google Cloud.
Available now in North America (Central) and Europe (West).

TPU 8t and TPU 8i (eighth generation, coming 2026)

Announced at Google Cloud Next 2026, the eighth generation is the first time Google has split the TPU lineup into two chips with distinct architectures for training and inference.
TPU 8t is built for large-scale pre-training and embedding-heavy workloads. A single superpod holds 9,600 chips with 2 petabytes of shared HBM and 121 FP4 ExaFLOPS of compute, nearly tripling per-pod compute versus Ironwood. ICI bandwidth is 19.2 Tb/s per chip, double Ironwood. The new Virgo Network fabric can link over 134,000 chips in a single data center, and theoretically over 1 million chips across sites. TPUDirect RDMA and TPU Direct Storage bypass the host CPU for data movement, effectively doubling bandwidth for large transfers. Google targets 97% goodput on 8t, meaning 97% of compute cycles go toward actual learning. It delivers 2.7x better performance-per-dollar over Ironwood for large-scale training.
TPU 8i is built for post-training and inference. It scales to 1,152 chips per pod and delivers 11.6 FP8 ExaFLOPS. Each chip carries 288 GB of HBM, more than the 8t training chip, and 384 MB of on-chip SRAM - 3x Ironwood's on-chip SRAM. The reason for more memory on the inference chip: large Mixture-of-Experts models at inference time are memory-bandwidth-bound, not compute-bound. The chip serving tokens needs to stream weights and KV-cache faster than the chip training the model. The 8i uses a Boardfly interconnect that reduces maximum network hops from 16 to 7, which reduces all-to-all latency for MoE routing. The Collectives Acceleration Engine (CAE) replaces Ironwood's SparseCores and cuts collective operation latency by 5x. Google reports 80% better performance-per-dollar over previous generations for low-latency inference on large MoE models.
Both chips run on Google's Axion ARM-based CPU host and use fourth-generation liquid cooling. Both are coming later in 2026.

Key architectural trends across generations

A few patterns stand out looking across all eight generations:

MXU size stayed constant for a long time, then doubled. From v2 through v5p, the MXU was 128x128. Trillium (v6e) expanded it to 256x256. This gave a step-change in throughput rather than incremental gains.

Topology alternates by use case. Cost-efficient chips (v5e, v6e) use 2D torus topologies, which are simpler and scale well to 256 chips. Performance chips (v4, v5p, Ironwood) use 3D torus, which reduces communication latency at larger pod sizes (4,096 to 9,216 chips).

Memory capacity has grown dramatically. From 16 GB per chip on v5e to 192 GB on Ironwood to 288 GB on TPU 8i. Memory capacity is increasingly what determines which models you can run and at what batch size.

Generation 8 split training and inference. Every prior generation was a single chip asked to handle both workloads. The 8t and 8i split acknowledges that the two jobs have fundamentally different hardware requirements.

Quick reference: which generation for which use case

Use case Recommended generation Getting started, cost-sensitive inference TPU v5e Medium training jobs, best perf/watt today TPU v6e (Trillium) Large training runs, >256 chips TPU v7 (Ironwood) Frontier model pre-training, 2026 TPU 8t Agentic AI inference, MoE serving, 2026 TPU 8i

Code sample: detecting your TPU version and topology in JAX

Once you have a Cloud TPU VM, this snippet uses JAX to confirm the TPU version, count available devices, and print the topology. It runs on any currently available generation (v5e, v6e, Ironwood).

import jax import jax.numpy as jnp

`# Print the number of TPU devices visible to JAX
print(f"Number of TPU devices: {jax.device_count()}")
print(f"Number of local devices: {jax.local_device_count()}")

Print device details including TPU version

devices = jax.devices()
for i, device in enumerate(devices):
print(f"Device {i}: {device}")

Run a simple matrix multiply to confirm the TPU is working

On Trillium (v6e) and Ironwood, this uses the 256x256 or larger MXU

key = jax.random.PRNGKey(0)
x = jax.random.normal(key, (1024, 1024))
y = jax.random.normal(key, (1024, 1024))
result = jnp.dot(x, y)
print(f"\nMatrix multiply output shape: {result.shape}")
print(f"Result device: {result.devices()}")

Check available memory per device

Useful for estimating what model sizes will fit

for device in jax.devices():
stats = device.memory_stats()
if stats:
total_bytes = stats.get('bytes_limit', 0)
used_bytes = stats.get('bytes_in_use', 0)
total_gb = total_bytes / (1024 ** 3)
used_gb = used_bytes / (1024 ** 3)
print(f"\nDevice: {device}")
print(f" Total HBM: {total_gb:.1f} GB")
print(f" In use: {used_gb:.2f} GB")`

To run this on a Cloud TPU VM:
# 1. Create a TPU VM (v6e example - swap accelerator-type for other versions) gcloud compute tpus tpu-vm create my-tpu \ --zone=us-east1-d \ --accelerator-type=v6e-8 \ --version=tpu-ubuntu2204-base

`# 2. SSH into the VM
gcloud compute tpus tpu-vm ssh my-tpu --zone=us-east1-d

3. Install JAX with TPU support

pip install "jax[tpu]" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html

4. Run the script

python tpu_check.py`

On a v6e-8 (8 chips, single host), you will see 8 devices listed. On a multi-host slice like v6e-32, each VM sees 8 devices and JAX handles coordination across hosts via the ICI.

Summary

Each TPU generation has made a specific set of tradeoffs: v5e optimizes cost, v5p maximizes compute per pod, Trillium doubled the MXU and energy efficiency, Ironwood added native FP8 and massive HBM, and the eighth generation splits the chip entirely for training and inference. Knowing which generation fits your workload is the difference between overpaying for compute you don't need and hitting memory or bandwidth limits on a chip that isn't designed for what you're running.
If you're just getting started, TPU v5e is the easiest and cheapest entry point. For production inference today, Trillium or Ironwood. For the next generation of frontier model training, keep an eye on TPU 8t.

Resources

TPU architecture - Google Cloud Documentation
TPU versions overview - Google Cloud Documentation
TPU v5e documentation
TPU v5p documentation
TPU v6e (Trillium) documentation
TPU v7 (Ironwood) documentation
Introducing Trillium, sixth-generation TPUs - Google Cloud Blog
Introducing Cloud TPU v5p and AI Hypercomputer - Google Cloud Blog
Ironwood: The first Google TPU for the age of inference
Google's eighth-generation TPUs: two chips for the agentic era
Run JAX on Cloud TPU VM - Google Cloud Documentation

TPUSprint

vLLM on Google Cloud TPU: A Model Size vs Chip Cheat Sheet (With Interactive Tool)

Grace Gong — Thu, 30 Apr 2026 09:38:40 +0000

Picking a Cloud TPU slice for vLLM inference involves three decisions that most tutorials skip over: how much HBM your model actually needs at runtime, how many chips to shard across, and whether the cost is justified for your workload. Get it wrong in either direction and you're either OOMing on startup or paying for memory you're not using.

This post walks through how to make that decision, with a reference table for popular models and a live interactive tool where you can select your model, toggle precision, and see exactly which TPU configurations fit and what they cost.

Try the interactive cheat sheet here: ggongg.github.io/vllm-tpu-notes

The site is a mini-project based on data pulled from 4/30/2026 and may change. Please refer to the official docs & site and double check.

What is vLLM and why run it on a TPU?

vLLM is an open-source LLM inference engine built for high-throughput, memory-efficient serving. It uses a technique called PagedAttention to manage the KV cache more efficiently than naive implementations, which translates to higher throughput and the ability to serve larger batches from the same hardware.

It has historically been GPU-first, but vLLM now has first-class support for Google Cloud TPUs, covering v5e, v6e (Trillium), and Ironwood. You can run both offline batch inference and an OpenAI-compatible API server on a Cloud TPU VM with the same command-line interface you'd use on a GPU.

Why TPU instead of GPU?

The short answer is cost at scale. For sustained, high-volume LLM inference, TPUs often deliver better performance per dollar than equivalent GPU setups, particularly with Google's newer generations. Trillium (v6e) delivers 4.7x more compute per chip than v5e while being 67% more energy efficient. Ironwood brings 192 GB of HBM per chip, which opens up model sizes and context lengths that simply aren't practical on a single GPU node. For teams already in the Google Cloud ecosystem, the operational simplicity of a managed TPU VM also reduces setup overhead.

What the interactive tool does

The cheat sheet at ggongg.github.io/vllm-tpu-notes lets you:
Select a model from Gemma 2B through Llama 405B
Toggle between BF16 and FP8 precision
Switch between on-demand, 1-year CUD, and 3-year CUD pricing
See which TPU slices fit the model, the recommended --tensor-parallel-size for each, and the hourly cost
Get a generated vllm serve command for the cheapest compatible chip

All pricing is pulled from the official Google Cloud TPU pricing page. Memory estimates use weights × 1.25 to account for KV cache and activation overhead.

How much memory does a model actually need?

The naive calculation is parameters × bytes per parameter. A 7B model in BF16 is 7B × 2 bytes = 14 GB. But that only covers the weight footprint. At inference time, you also need headroom for:
KV cache, which grows with batch size and context length
Activations during the forward pass
vLLM's internal buffers

A practical floor is weights × 1.25. A 7B BF16 model needs roughly 17.5 GB, which means a single v5e chip (16 GB HBM) will OOM, but a v5e-4 slice (64 GB across 4 chips) fits comfortably.
FP8 halves the weight footprint, so the same 7B model drops to about 8 GB in FP8, leaving far more room for KV cache on a given slice. Trillium (v6e) and Ironwood both support native FP8. TPU v5e supports INT8 quantization via vLLM's tpu_int8 flag instead.

TPU chip HBM and pricing reference

These are the currently available Cloud TPU slices for vLLM inference. Per-chip pricing is from cloud.google.com/tpu/pricing, US region, as of April 30, 2026. Slice prices are chips × per-chip rate.

Model size vs chip: the cheat sheet

The key vLLM flags for TPU

vllm serve MODEL_ID \
--tensor-parallel-size TP \
--max-model-len MAX_TOKENS \
--dtype bfloat16 \
--device tpu
--tensor-parallel-size sets how many chips the model is sharded across. It needs to be a power of 2 (1, 2, 4, or 8) and match your slice size or be a divisor of it.
--max-model-len caps the context window, which controls KV cache memory usage. If you're OOMing on a slice that should theoretically fit the model, reduce this first before scaling up to a larger slice.
--dtype bfloat16 is the right default for TPU. For native FP8 on v6e or Ironwood, use --dtype fp8 instead.
--device tpu tells vLLM to use the TPU backend.

Example commands

Gemma 7B on v5e-4
vllm serve google/gemma-7b-it \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--dtype bfloat16 \
--device tpu
16 GB weights fit on a single v5e chip. With 64 GB total on a v5e-4, there is plenty of room for KV cache at reasonable context lengths.
Llama 3.1 70B on v6e-8
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--dtype bfloat16 \
--device tpu
140 GB of weights need roughly 175 GB with overhead. The v6e-8 has 256 GB total across 8 chips, so TP4 puts 64 GB on each of 4 chips and leaves headroom for the KV cache.
Llama 3.1 8B on v6e with FP8
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--tensor-parallel-size 1 \
--max-model-len 8192 \
--dtype fp8 \
--device tpu
FP8 brings the 8B model down to about 8 GB, fitting easily on a single v6e chip (32 GB) and leaving 24 GB for KV cache. This is a good setup for high-concurrency serving where you want to maximize batch size.
Gemma 4 31B on v6e-8
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm \
-e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:gemma4 \
--model google/gemma-4-31B-it \
--tensor-parallel-size 8 \
--max-model-len 16384 \
--device tpu
Gemma 4 has a dedicated Docker image (vllm/vllm-tpu:gemma4) that includes multimodal support. For the 31B variant on a v6e-8, TP8 across all 8 chips is the recommended configuration.

Provisioning a TPU VM for vLLM

Create a v6e-8 VM

gcloud compute tpus tpu-vm create my-vllm-tpu \
--zone=us-east1-d \
--accelerator-type=v6e-8 \
--version=tpu-ubuntu2204-base

SSH in

gcloud compute tpus tpu-vm ssh my-vllm-tpu --zone=us-east1-d

Install vLLM TPU

pip install vllm-tpu
The v6e-8 configuration is the one Google designed specifically for single-host inference. For Ironwood, you need to use GKE rather than the TPU VM API. Step-by-step instructions for both are in the vLLM TPU quickstart.

Picking the right configuration

v5e-4 or v5e-8 makes sense if you're running models up to 27B and want the lowest cost per hour. At $1.20/chip/hr, it's the cheapest path to TPU inference. The 16 GB per chip limits what fits, but it covers the 7B to 13B range that handles most practical use cases.
v6e-4 or v6e-8 (Trillium) is the best general-purpose option today. The 32 GB per chip opens up models to 70B with enough chips, and the 4.7x compute improvement over v5e gives meaningfully better throughput for the same money. The v6e-8 in particular was designed by Google for single-host inference and is the easiest starting point for production workloads.
Ironwood is the right choice when you genuinely need the memory: 70B at long context, 405B models, or high-concurrency serving where maximizing KV cache capacity matters. At $12/chip/hr, it's worth confirming you actually need the 192 GB per chip before provisioning it.

Try it yourself

The interactive tool is live at ggongg.github.io/vllm-tpu-notes. Select your model, toggle precision and pricing tier, and get the recommended chip configuration and vLLM command in one view. The source is on GitHub at github.com/ggongg/vllm-tpu-notes.

Resources

Interactive cheat sheet
vLLM TPU quickstart
vLLM TPU installation
Cloud TPU pricing
TPU v6e documentation
TPU v7 (Ironwood) documentation
Gemma 4 vLLM recipe
vLLM TPU offline inference example

TPUSprint program.

Check cloud.google.com/tpu/pricing for most up to date info!

Ill be speaking at the 2026 Grace Hopper Celebration (GHC)!

Grace Gong — Thu, 30 Apr 2026 06:59:02 +0000

GHC 2026
I’m Speaking at Grace Hopper Celebration 2026!

My path into VC hasn’t been a straight line. As an engineer at Microsoft and a Venture Fellow at Alumni Ventures, I’ve had the privilege of sitting at the intersection of technology and investment learning pitches, understanding what investors are actually looking for through the fellowship training, and learning from venture capital coursework through Berkeley Law, NVCA, and VCForward. Alongside that, I’m completing coursework from Georgia Tech and the University of Illinois, where technical and business strategy have added a new lens to how I think about building and funding companies.

GHC felt like the natural place to bring these learnings together because this community deserves a seat at the table in these conversations.

What You’ll Learn at My Session

Whether you’re a founder looking to raise, an engineer curious about the business side, or someone who’s just VC-curious, this session is designed to be approachable and practical. We’ll cover:

How investors evaluate deals and what they’re really looking for beyond the pitch deck
The frameworks VCs use to make funding decisions
How you as a technologist or founder can navigate the VC ecosystem with confidence
No finance background required.

Why This Matters

Women remain significantly underrepresented both as founders receiving VC funding and as investors making funding decisions. One of the best ways to change that is education and access: knowing the language, the process, and the people. I hope this session is a small step toward that.

Join Me in Anaheim

GHC 2026 runs October 27–30 in Anaheim, CA. If you’re attending, I’d love to see you at my session!

🔗 Register: GHC.AnitaB.org

Let’s connect too, find me on Twitter/X and LinkedIn

I'll be speaking at Major League Hacking (MLH) Hackcon 2024!

Grace Gong — Tue, 13 Aug 2024 16:52:23 +0000

Hi everyone! This year I'll be speaking at Major League Hacking (MLH) Hackcon 2024 on "How to Build a Campus Community!"
I'm excited to share learnings and a high level rundown of the event in a future post.

More info on Hackcon: https://hackcon.mlh.io/

Join Hundreds of Hackathon Organizers.
Hackcon brings together 300+ hackathon organizers and campus tech leaders for two days of interactive sessions on community building, event planning, and the hacker experience. Come for the learning, stay for the camp activities!

Meet Community Leaders
Hackcon is a unique opportunity to meet like-minded individuals from across the country. It’s where friendships are formed, partnerships are started, and tech leaders meet all together in-person one last time before they go off into the real world.

Enjoy the Great Outdoors
Unplug from your devices and enjoy the beautiful scenery of upstate New York. Enjoy a range of fun outdoor activities like swimming, s’mores, and tie dye! By the end of the event, you’ll feel relaxed and recharged for the new school year.

Learn from the Best
There is no better place to learn how to organize a hackathon and run a community on campus. At Hackcon, you’ll have the chance to meet like-minded individuals and hear from the most experienced leaders in our community.

Join us at Camp Pontiac
Located in Copake, New York, Camp Pontiac is an active summer camp that becomes the venue for Hackcon. This scenic location creates the perfect atmosphere for community talks, discussion groups, and lasting friendships.

Get Tickets here: https://ti.to/mlh/mlh-hackcon-2024

Making Chatbot Applications with Dialogflow: My HackItTogether 2024 Project

Grace Gong — Sat, 20 Apr 2024 20:41:14 +0000

I participated in a hackathon in nyc earlier this term! As a highlight, I was able to work with a team of students from brooklyn to create a chatbot. We were given the challenge and objective to work on an Individualized Recommendation System that leverages AI
techniques to understand users' preferences and recommend similar content.

I conducted research on various Google Cloud Services to help support this project, and ultimately decided to work on a chatbot using Dialogflow to help achieve this.

Things that I considered during our project:

Architecture for the chat, db &/api
If we want to use dialog flow / azure openai as a templated UI for the chat or coding the interface ? (which would be time consuming GCP pros: we had credits)
Which API’s to use, and if we use multiple, how to consolidate/aggregate the data in one unified chat
Which UI to use for the website, and how to customize it? A premade template would be ideal
Which data sources we want to take and how to integrate this data
Are we enforcing guardrails (tell it to not give info) only picking from specific sections of info and prevent hallucinations

The other members in my group used other api's such as spotify api to help provide traditional recommendations to a search bar.
The earliest iteration of our UI looked like this:

The Dialogflow CX documentation was simple and easy to follow - I added our data sources through a combination of docs, website links, csv and files. After, I tested the chat application by asking questions to ensure the data was retrieved and provided adequate responses.

The Chatbot Application was simple to integrate into our React UI -> it appears on the bottom right corner of our site.

While the cloud credits have paused so the chat might not be loading at the moment, you can still see the UI for our project here: https://your-north-star.netlify.app/ {UI done in react}

Inspiration
The North Star has been a symbol of guidance and direction for centuries. In ancient times, navigators and travelers used the North Star as a way to determine their location and navigate the seas. We think that our web-based project with the help of AI will help the users navigate the quiz to find their way to their favorite book, movie or music.

What it does
The website will present a welcoming page to “Your North Star” ChatBot where it asks the user what they’re looking for. They’re presented with three buttons: music, book and movie. This leads to the next question which allows the user to specify the genre of their chosen media. The input given by the user is fed to an API. The APIs return the recommendations as a list, where each recommendation provides metadata, such as: title, cover, rating, overview, etc. We chose the constellation-theme because this is meant to be a journey to imagination. Your taste will bring you to the destination.

Challenges we ran into
The challenges we faced were with the implementation of a chatbot into the website.

Accomplishments that we're proud of
Being able to successfully implement the APIs from Google Cloud and Spotify Web.

What we learned
We learned how to customize APIs; how to navigate on different errors when using javascript or react. Most importantly we learned how to negotiate between team members and try to make our best based on the time provided.

What's next for Your North Star
By using AI we can explore the characters of the books/movies to elevate the user's experience to the next level. The user can communicate the characters and be friends with them.

Built With

chatbot-ui
css
dialogflow-es
figma
github
google-books-api
google-cloud
html
imdb-api
javascript
omdb-api
react
spotify
vertexai-api

GitHub Field Day USA 2024: Microsoft Reactor New York City

Grace Gong — Sun, 11 Feb 2024 06:54:59 +0000

Attention all Students in the United States and Canada!
GitHub Field Day USA 2024 will be taking place on March 15, 2024 at Microsoft Reactor New York City.
Please Apply here: https://githubfieldday.com/nyc2024/

An unconference for leaders of technical student communities
Friday, March 15th, 2024 at the Microsoft Reactor in New York City!

Hosted by GitHub Education

A day of discussions
Field Day brings together a group of student leaders for a day of conversation in the form of lightning talks, discussions, games, or whatever suits the topic best.
**
Schedule**
10:00 am Registration Opens
11:00 am Welcome to Field Day
11:20 am Keynote
11:50 pm Icebreakers
12:30 pm Talks/Discussions 1
1:05 pm Talks/Discussions 2
1:40 pm Talks/Discussions 3
2:10 pm Lunch
3:05 pm Talks/Discussions 4
3:40 pm Talks/Discussions 5
4:15 pm Closing Remarks
4:30 pm Goodbye + Networking Event

Topics
Inclusivity in your community
Internships
Building your community
Inter-community connections
Finding events near you
Open source
Apply to contribute your discussion topic!
FAQ
Who can attend Field Day?
Field Day is for student leaders of technical communities. Most attendees will be undergraduate students and young professionals.

What's a student leader?
A student leader is someone who's actively involved in improving their local technical community. They might be the president or chairperson of their school's tech club or be one of the organizers of the annual hackathon. They might run the student hardware group or be actively involved in introducing other students to programming.

What's an unconference?
A loosely structured conference emphasizing the informal exchange of information and ideas between participants (to quote the Google definition). Most content at Field Day will be in the form of lightning talks and discussions conducted by attendees.

I'm not in New York, but this sounds cool! Can I come?
If you're based in North America, you're welcome to apply!

So I registered to attend, what happens now?
We're just screening applications to make sure you're eligible to attend, you should hear from us within a few days. Once confirmed, all you need to do is show up at the venue that will be sent to you on the day of the event and take part!

How much does Field Day cost?
Field Day is free to attend!

Is financial aid for attending the event available?
We expect to have travel reimbursements available for students in areas surrounding New York City, we encourage you to apply. We will be notifying accepted students with the status of their travel reimbursemens within one week after acceptances are released.

What COVID restrictions are there?
We'll be following local guidance and will announce more specific details closer to the event.

Will there be food?
Yes! We will be serving a light breakfast and lunch! You will not go hungry.

Will there be swag?
Yes!

When can I find out if I'm receiving financial aid?
We hope to send out confirmations of travel reimbursements within 1 week of acceptance.

Is there a Code of Conduct for this event?
Yes! You can view it here.

Who is organizing Field Day?
Field Day is organized by GitHub Campus Experts, and technical student leaders.

About me
My name is Grace, and I’m a computer science student and GitHub Campus Expert for Western University based in Canada. As a GitHub Campus Expert and Developer Student Club Lead, we organize technical and career development events, creating opportunities for our communities to leverage GitHub and Student Developer Pack. We strive to build diverse and inclusive spaces to learn skills, share our experiences, and build projects together. Some of our flagship events that we host at Hackathons include Intro to Cloud, ML API's Workshops, and GitHub Workshops, where we help students get started with GitHub and share the GitHub Student Developer Pack.

Earlier this November, I had the opportunity to go to GitHub Universe in person in San Francisco, thanks to the GitHub Education team.
We met at the GitHub San Francisco headquarters, where we did a tour of the HQ, and attended Sessions at the center.

Throughout the Semesters, I collaborate with tech clubs across campus at Western University to organize workshops on GitHub, as well as other technical topics. We share resources, share events with our broader communities, and aim to inform and educate students so they can utilize these skills to work on impactful projects in and outside of school.

How I became a GHC (Grace Hopper Conference) Speaker, from a 2023 GHC Speaker

Grace Gong — Sat, 23 Dec 2023 02:36:52 +0000

I’m excited to have spoken this year at the Grace Hopper Conference as a computer science student! My talk was focused on empowering individuals to build a campus community. I shared my experience starting the women in computer science committee and how I leverage the resources from partner organizations, clubs, and my school to create compelling, engaging, and valuable events for the student population.
I also went through how I marketed the events and got hundreds of students to attend these events this year and join the community.

I will try to keep this article concise, short, and sweet in honor of GHC approaching - I know everyone is preparing for exams or the upcoming holiday season, and if anything, I hope this article can potentially be a catalyst for future conversations at the event and beyond!

My process was a bit unique, and I hope it can provide some insights for others interested in getting involved in the GHC community. (This is virtual, and I am sure there will be other requirements for the person)

I had just finished my sophomore year of CS and realized there was no significant in-person presence at my campus. On campus, I mean, when you walk up to the doors of the computer science building, there is a rush of people immediately, but there is no commonplace that you can call "community.” The classrooms upstairs are mostly unknown to most students, with the exception of labs, and the space itself feels foreign even after having spent years here.

There was a new room where people could study in groups, but it holds probably 20 people max and is usually filled with students on a time crunch doing group projects or empty throughout the day, and no significant space is created for those who are in computer science and want to meet others in the community.

Some clubs would hold virtual events and generally fall under the area of technology, but for computer science specifically, there was no common community. Noticing that there were many women in CS clubs on campuses across Canada, such as Waterloo and UBC, I wanted to bring this energy to my university specifically with a focus on collaboration with the school faculty for increased sponsorship, and applied for the AnitaB Pass It On Grant.

As a DSC (Developer Student Club) lead, I believe this strong partnership between our committees would enable us to deliver even more impact, and it for sure did - that year, we were able to organize numerous workshops with corporate partners like Salesforce on Technical, Behavioural DSA, and General Interviews, and give students a community to learn from each other. We organized an International Women’s Day event in partnership with the Ivey Business School and their Hackathon event in March, where we were able to support students in navigating the new normal as emerging technologies continue to evolve - students were able to learn from Ada, a TPM at Google in ML, as well as other Developers from the Google Developer Community to facilitate workshops on Flutter, a cross-platform mobile application development SDK which enabled students to develop their skills during the challenge.

I also had multiple discussions with the chair of the department of computer science, the chair of the faculty of science, where we discussed the experiences of marginalized students in science and computer science and how we could partner to improve the student experience.

I was the campus lead for the Percentage Project, a data-driven campaign to

This is among the many events we had during the school year with DSC, and I’m proud of our progress as the membership grew from 100 to over 450 members.

My GitHub Universe 2023 Experience with GitHub Campus Experts & GitHub Education

Grace Gong — Fri, 10 Nov 2023 08:16:17 +0000

Attending GitHub Universe 2023 was amazing!

Whats GitHub Universe?
GitHub Universe brings together the world's developers, enterprise leaders, and security professionals for two days of interactive sessions on AI, security, and the developer experience. Come for the learning, stay for the fun. We'll see you there!
How Did you get the opportunity to attend?
As a GitHub Campus Expert, this year I had the opportunity to attend GitHub Universe in person and the sessions and discussions I had there were impactful, inspiring, and a great testament to the incredible work and progress the Github Community has put together to improve the developer experience this past year.

How was the experience?
As my friend Harsh from OurTechCommunity (OTC) said - All the Campus Experts were so inspiring! It was so good to talk about each other's hardships openly, hear about their efforts and goals, see their drive and hard work, and learn from their experiences and knowledge.
We were also able to learn from the overall tech community from Universe, whether it be about Open Source, or what other companies are doing to adopt and adapt to the current evolving tech landscape. I am so grateful for GitHub, the GitHub Education Team, and everyone who worked to make this possible. Big Thank you to Luis, Juan, Dyson, Stormy Peters and everyone else who worked to make us, as students feel heard, and give us spaces to learn and bring our learnings to our overall communities.

What activities did you attend?

GitHub Open Source Pre-event Networking @ Minna Art Gallery
GitHub Platform Engineering Talk @ MoMa
Telus GitHub Actions Talk @ MoMa
Developer Experience @ Discussion Lounge
VSCode Productivity @ Discussion Lounge
Observability Solutions @ Discussion Lounge
Education Panel @ Hyatt
Ignacio's Panel on AI In Education
Keynotes, Chat with Stormy + Martin
Booths: Microsoft, Datadog, Postman, Arm, GitHub CoPilot, Security,
Demos: GitHub Security, CoPilot
GitHub Certification
GitHub Women in Tech Event

What did you learn from attending GitHub Universe?
I learned additional ways to support and engage my community and how to uplift marginalized/underrepresented communities. Specifically, through the panels on specific GitHub Actions and Features that I can now talk about in the workshops and at my internship companies who use GHE, specific actions I can take to expand and make my community more consistently engaged.
Through conversations with those in open source (leading communities or as an active contributor), I was able to learn about current open source projects, how to get involved, and how OSPO's work(An open source program office (OSPO) serves as the center of competency for an organization's open source operations and structure.) I was able to learn about the operational / logistics side of these groups and how to keep these communities alive and engaged. The key themes continued to tie back to Developer Experience and how understanding the Developer + being able to take steps to make it easy for developers to know how and when to contribute is often key to success in maintaining or making it accessible for the community.
The GitHub Women in Tech event enabled me to learn how to support and uplift other marginalized genders, create safe spaces for discussion, and continue to be involved and immersed in my community. This conference was life-changing because it enabled me to be connected with inspiring women at GitHub + Beyond and from various walks of life. I learned from women of diverse backgrounds and their experiences/strategies in the workforce. I also learned about the struggles they faced in the process and some things I need to look out for / consider in my career.

Who did you meet?
Story Peters, Martin Woodward, Michael Hanley, Joseph K, Ingrid Yan @github Security, VPs of Product, PM @ GitHub Sponsors, Anjuan from GitHub Sponsors, Thomas Dohmke, GitHub Education Team, Datadog Employees (PM, PMM, Sales), Open Source Contributors (CS50, Huan Li, Employees from Slack that work on Electron, etc) David Malan, Demetris Cheatham,(Chief of Staff, GitHub) and of course, the Campus Experts! Workshop Speaker-> Telus, Tech lead from Platform Engineering Talk (Leslie @ny Times)
-The Women in Tech workshop had various women from students my age (I wasn't able to get her name but she is a masters at UWashington doing a PhD to senior / VP positions) and a Bootcamp grad from 2016 who is now a tech lead at HubSpot, IBM VP, A Digital Assets and Workplace Activation Lead at Aristocrat, GitHub Business Ops Staff, Robinhood Developer Advocate

Best thing about the event?
The opportunity to learn from others in the field from diverse backgrounds (technical + nontechnical) and hear their experiences working with their communities and achieving positive impacts.
Learning from the developers and community leaders about their projects, their contributors, their ideas for the future, and what is happening with emerging technologies
The GitHub Women in Tech Event, where I had the opportunity to meet inspiring women and their career journeys
Developing my interpersonal and technical skills through valuable workshops and in-person demos. I was even able to have my code reviewed at the Microsoft booth because at my internship, I was using an SDK they had released as Open Source. I learned from the Dev Advocates on best code practices and made advancements in my code's bugs
The Dev Advocates + other employees also gave me valuable advice and motivation by sharing their experiences and career journeys. This conference and talking to others enabled me to see that these and other issues, whether systemic or

All of GitHub Universe keynotes, announcements, and most popular sessions
You can now watch 40+ on-demand sessions from GitHub Universe!
https://reg.githubuniverse.com/flow/github/universe23/sessioncatalog/page/sessioncatalog?search.deliveryformat=1692799009854004CQlx

The highlights

GitHub Copilot Enterprise: a new subscription tier that culminates the entirety of our Copilot offerings and personalizes them with the context of your organization's codebases, available in February 2024.
GitHub Copilot Chat, an AI companion that'll allow you to write and understand code using whatever language you speak, will be generally available in December 2023 at no additional cost. We're also integrating Copilot Chat directly into github.com and mobile, so developers can code from anywhere.
Now available in preview: Code scanning autofix and secret scanning for generic secrets provide new capabilities to help you prevent and detect more security vulnerabilities across your organization.

How to get started with MongoDB as a Student

Grace Gong — Thu, 14 Sep 2023 04:00:32 +0000

As student with a passion for the developer community, I have participated in varous hackathons, and gotten involved in my student community.

Throughout these experiences, I have had the opportunity to learn and develop projects using the MERN stack (MERN stands for MongoDB, Express, React, Node, after the four key technologies that make up the stack. MongoDB — document database. Express(.js) — Node.js web framework. React(.js) — a client-side JavaScript framework.) More information can be seen on their blog here.

As a software engineering intern, I have had the opportunity to also see how often MongoDB is used within projects, and how its capabilities have contributed to more efficient and reliable projects.

Finally, In my previous semester, I had the opportunity to take a course on unstructured data, (for any students at western, CS4417 is the course!) where I was able to learn more about unstructured data, noSQL, and work on an assignment involving MongoDB, Aggregation, and MapReduce. Im happy to make seperate blog posts about these projects as well for any students interested in exploring applications and ways to use MongoDB in their own projects.

Overall, MongoDB is a great tool to be able to leverage in your projects! Now you might be wondering, how do I get started?

MLH (Major League Hacking) is a large hackathon organizer and a community that I have been involved in since the summer of 2020.

As mentioned in their website post,

MongoDB Atlas – Best Database Platform for Your Next Hackathon
MongoDB Atlas is a multi-cloud database with an integrated set of related services that allow development teams to address the growing requirements for today’s wide variety of modern applications, all in a unified and consistent user experience.

MongoDB Atlas is more than a general-purpose database, it’s a full developer data platform – easily deploy and manage databases on-demand when and where you need them.

Below are some ways you could use MongoDB Atlas in your next hackathon project.

1) Simplify the way you integrate database functionality into your hackathon project by starting a free cluster or using your introductory $50 Atlas credits for students.

2) Take your hackathon project to the next level, and deploy a database to the cloud in minutes. Signing up is easy and hassle-free with no credit card required!

Key Benefits for Students
MongoDB is one of the most prevalent database providers in the world, and we’ve got a range of opportunities to help you get started building on their platform.

1) Receive $50 of free MongoDB Atlas credits through the GitHub Student Developer Pack, along with a MongoDB certification ($150 in value). The Forever Free Tier is available if you are no longer a student.

2) Enter the “Best Use of MongoDB Atlas” challenge at an upcoming MLH event for a chance to win a M5GO iOT Starter Kit. To participate in the challenge, check out our website and register for an upcoming event!

Interested in taking advantage of your MongoDB Atlas perks? Check out their website here.

MongoDB also has developed "MongoDB University" a series of courses and ressources to support the learning and application of those learnings! I have taken some of the modules and can confirm the videos and tutorials go in depth and with the interactive nature of their tutorials, you will be engaged with the content.

Finally, MongoDB has User Groups
which bring people together to learn from and connect over their shared interest in MongoDB technologies. These are communities of developers with various skill levels and an interest in learning new technologies and applications including MongoDB and beyond, and are a great opportunity to get involved! Whether you are a beginner or a skilled individiaul, this is a safe space for users and novices to come together, learn from each other, and share what theyre doing with mongodb and other tech.

I recently became involved as a co-organizer for the Toronto User Group and confirm the community is welcoming for people with diverse experiences and has enabled me to meet and learn from people around the company and community.

These are just starting points - there are many options for ways to get involved with MongoDB beyond this!

You could build an application or library you've built using MongoDB, contribute to open source related to MongoDB,
talk at a local MongoDB event, create a video or tutorial sharing what you've learned or created about MongoDB,
get involved and provide support through Stack Overflow, the MongoDB Community Forum or elsewhere, and more!

If you had any questions feel free to leave them in the comments!

Toronto MongoDB User Group Meeting Sept 5

Grace Gong — Fri, 08 Sep 2023 01:46:08 +0000

I had a great time co-organizing the Toronto MUG event this past week!

Event Summary
Toronto MUG Organizers: Chris Dellaway, Grace Gong
Agenda:
6:00PM Doors Open, Supper and Networking
6:30PM Prashant (MongoDB) Amplifying Retail Operations with Generative AI: The Untapped Potential
7:00PM Lachlan (Community) - Company journey with MongoDB
7:20PM Devang (Community) Using MongoDB Clustered Collection to Boost Query Performance
7:30PM Networking

The event was held at the Microsoft Toronto headquarters from 6-8pm on Tuesday September 5 2023 with the aim of fostering connections and sharing knowledge among members of the MongoDB Community.
The crowd was truly engaged, and the diversity of the attendees resulted in vibrant discussions and insightful questions.

Learned

Attendance Accuracy: Future events should either emphasize the importance of accurate RSVPing or anticipate a specific turnout percentage to prevent underutilisation of space.
Improving Check-in: An advanced and digital check-in system can greatly improve the event’s commencement process.
Balancing Time: A clear mechanism to ensure speakers stick to their allocated time can make the event more streamlined.
Loved

Speakers: They were highly informative, and their sessions were greatly appreciated by all attendees.
Community Connection: Attendees left the event feeling more connected with the MongoDB Community and inspired.
Feedback: The considerable number of feedback forms received demonstrated the attendees’ involvement and commitment to improving future events.
Diverse Discussions: The diverse background of attendees enriched the discussions and questions during the event.
Ideas/Action Items for the Next Event

Allocate budget for name tags and markers to facilitate easier recognition and networking. Also next time we can introduce volunteers and have the slide for them as well
Explore advanced check-in systems for a smoother start to the event.
Liaise with adjacent event organisers to coordinate and ensure both events run seamlessly without confusion for attendees.

Part 1: Intro to serverless

Grace Gong — Mon, 03 Jul 2023 17:59:03 +0000

As Im learning about cloud computing and reading up on various sources including aws, I thought I would share some of my learnings.

Why serverless?
Building Secure Reliable and Cost effective architecture is important for IT professionals. Many organizations are replacing monolithic methods with microservices methodology.

What are Microservices?

Architectural and organizational approach to software development.
Software is composed of small independent services.
Communicate over well-defined APIs.
Owned by small, self-contained teams.

Tightly versus loosely coupled

Monolithic application (tightly coupled)
In a monolithic application, if one piece were to fail, the entire application would crash, as this example illustrates. If there is a spike in demand, the entire architecture must be scaled. Adding features to a monolithic application becomes more complex as time goes on. Pieces of the code base must work with each other to sync properly.

Microservices application (loosely coupled)
In a microservices architecture, an application is built as independent components that run each application process as a service. These services communicate through a well-defined interface by using lightweight APIs. Services are built for business capabilities, and each service performs a single function. Because they are independently run, each service can be updated, deployed, and scaled to meet demand for specific functions of an application. If one component fails, the other components remain active, as illustrated.

Benefits of microservices

Use public APIs
Microservices communicate over APIs. In fact, doing so is the only way to realize the benefits of microservices. Don't be tempted to circumvent that rule. By communicating with APIs, you can modify a database and maintain it without affecting communications or having to repoint interfaces. The public API is a contract between your microservices and its clients.

Use the right tool for the job
Because every service is built for explicit functionality, you can choose the right tool for the job. The development team for Microservices B can add Amazon Aurora Service or change to Amazon OpenSearch without affecting Microservices A. You can also make changes to the entire language framework, which gives you the flexibility to code in the language that is best suited to an application. By choosing the programming framework that works for the team, the development team for Microservices B can avoid depending on other teams for their knowledge of specific frameworks.

Secure your services
Microservices must be architected to provide security at every step. The defense- in-depth approach is a common method of reviewing each step of the application to determine areas where additional security controls can be implemented. In a microservices environment, this arrangement includes the application level, API level, network layer, and AWS Identity and Access Management (IAM) configurations. The API is the entry point. Amazon API Gateway can perform authentication and authorization for services and act as security control for microservices by enforcing throttling and obfuscation of internal resources.

Be a good citizen
As this environment of microservices grows and expands, it becomes important to be a good citizen. As a service owner, you must understand how other services are calling you. The service owner will identify the client's use case and register them in the service registry. As a client of services, you must understand the service that you are calling.

Company transformation
Going from a monolith to microservices is more than a technological transformation. The shift requires transformation in the organization and communication mechanisms between teams. Automation is the key to rapidly deploying stable applications. Small development teams ("two-pizza teams" or no more than the number of people who can be fed with two pizzas) focus on producing a single microservice. They can send the service through an automated delivery pipeline and release a service into production with everything, from the build step forward, being automated. By organizing the software as a grouping of microservices, the development teams are decoupled and can efficiently release updates to their microservices on their own.

Automate everything
Some examples of some tools can help with automation.

Downsides of monoliths

All processes are tightly coupled and run as a single service.
Adding or improving a monolithic application's features becomes more complex as the code base grows.
Dependent and tightly coupled processes increase the impact of a single process failure.

Serverless applications dont require you to provision, scale or manage any servers.

You can build them for nearly any type of application or backend service, and everything required to run and scale your application with high availability is handled for you.

Why serverless?

developers dont need to worry about managing and operating servers of runtimes and can focus on other areas, thus can focus on core product

Event-driven architectures

Uses events to initiate actions and communication between decoupled services.
Change in state, a user request, or an update.
Information published for other services to consume it.
Events are the primary mechanism for sharing information across services.

• Microservices is a method of building your architecture.

• Serverless is a service that is used to build your architecture.

• Microservices is the method of loosely coupling sections of your architecture.

• Serverless services are the tools that you use to build out those sections of your architecture.

• It's common to use a blend of both serverless and nonserverless services.