What is CUDA and what happens when you launch a kernel?

CUDA is NVIDIA's platform for running general-purpose programs on NVIDIA GPUs. It includes a programming model, language extensions for C and C++, a compiler toolchain, runtime and driver libraries, and tools for profiling and debugging. When people say "the code runs in CUDA", they usually mean that part of the program is compiled for the GPU and started from ordinary code running on the CPU.

That last sentence hides most of the interesting detail. A typical CUDA program has a host portion running on the CPU and a device portion running on the GPU, and the two cooperate. Understanding a kernel launch means understanding what the host asks for, what the GPU actually receives, and when the host finds out what happened.

Two sides of one program

The host side decides what work exists. It allocates memory, prepares inputs, chooses how many threads to start, and asks the driver to launch a function called a kernel. The device side is that kernel: a function written once and executed by many GPU threads, each with its own indices.

A minimal mental model, written as pseudocode rather than tested code:

host:
  allocate device buffers in, out
  copy input array → in
  launch scale(in, out, n) with G blocks of B threads
  copy out → result array
  wait, then check for errors

kernel scale(in, out, n):            # runs once per thread
  i = blockIndex * blockSize + threadIndex
  if i < n:
    out[i] = 2 * in[i]

The bounds check matters. Threads are started in whole blocks, so the total thread count is usually rounded up beyond n. Without the check, the extra threads would read and write past the end of the arrays. Nothing in the launch prevents that; the kernel author must.

Threads, blocks and the grid

CUDA organises threads into blocks, and blocks into a grid. Threads in the same block can cooperate through fast on-chip shared memory and can synchronise with one another. Blocks are meant to be independent: the hardware may run them in any order, at the same time or one after another, depending on how many fit. CUDA programming model (opens in a new tab)

That independence is the deal you make. If your algorithm needs block 7 to finish before block 8 starts, a plain launch does not promise it. You either restructure the work, split it into multiple launches, or use features designed for that coordination.

On NVIDIA hardware, threads within a block execute in groups of 32 called warps. When threads in a warp take different branches, the paths are handled with reduced efficiency. When neighbouring threads read neighbouring addresses, memory requests can be combined. These are performance properties, not correctness rules—but they explain why two kernels with identical logic can behave very differently. CUDA best practices (opens in a new tab)

What a launch actually does

When the host launches a kernel, it does not normally wait for the kernel to finish. The launch is placed into a stream—an ordered queue of GPU work—and the host may continue before the GPU has completed it. Work in one stream runs in order; work in different streams may overlap. CUDA programming guide (opens in a new tab)

This asynchrony has three practical consequences.

Timing submission is not timing completion. A host timer around the launch line measures the cost of submitting the work. That can include overhead, one-time initialisation on first use, or blocking if a queue is full, so it is not a reliable measure of how long the kernel ran on the device. To time the kernel, wait for it explicitly or use GPU-side events.

Errors can arrive later. Some problems, such as an invalid launch configuration, can be reported at launch. Others, such as an illegal memory access inside the kernel, may only surface at a later synchronising call. The line that reports an error is not always the line that caused it.

Ordering is what makes results safe to read. A copy back to the host placed in the same stream after the kernel will wait for the kernel. Reading host memory that the GPU is still writing, without that ordering, is a bug even if it sometimes appears to work.

A thought experiment: one million elements

Suppose we scale one million 32-bit floats with blocks of 256 threads. We need 3,907 blocks, because 3,906 × 256 falls 64 threads short of a million. The final block contains 192 threads with no element to process, which is exactly why the bounds check exists.

Nothing in that arithmetic tells us how fast the kernel is. The work per element is one multiply, against reading and writing four bytes each. For sufficiently large arrays, a kernel like this is often limited by memory traffic rather than arithmetic, and the transfer to and from a discrete GPU may cost more than the kernel itself—but only measurement on a given system says which. That accounting is the subject of how a GPU works.

What to take away

CUDA is not "C++ that runs faster". It is a way to describe many independent pieces of work, place them in an ordered queue, and later collect the result. The questions worth asking about any kernel are: Is the work independent per block? Is every thread's memory access in bounds? Where does the host wait, and what does the program measure?

If Rust is your host language, the same model applies with different safety boundaries—see CUDA in Rust. For the CPU-side foundations, continue with how programs execute and memory and data movement.