CUDA C++ Tutorial: Getting Started

Overview

This CUDA C++ tutorial is the starting point for the series. It introduces CUDA, explains why GPUs are a strong fit for data-parallel work, and builds an intuition for how a CUDA application is split between the CPU (host) and GPU (device).

The chapter is intentionally practical: it lays down the minimum set of ideas needed to read CUDA code confidently—then ends with a first program that launches a simple kernel.

What is CUDA?

CUDA (Compute Unified Device Architecture) is NVIDIA’s platform and API for running general-purpose programs on the GPU.

At a high level, CUDA extends C/C++ with a few keywords and runtime APIs so the CPU can launch work on the GPU. The GPU then runs that work using many lightweight threads.

CUDA is most effective for problems that can be expressed as massive numbers of parallel, independent operations, aligning perfectly with the GPU’s execution model. Common examples include:

• Image/Video processing

• Scientific simulation

• Linear algebra and machine learning

GPU vs. CPU: Understanding the Difference

CUDA starts to make a lot more sense once you clearly understand how CPUs and GPUs differ.

• CPUs are built for low-latency tasks. They have a small number of very fast cores and are excellent at handling complex logic, branching, and “do this, then that” kinds of workflows.
• GPUs, on the other hand, are built for throughput. They contain many simpler cores designed to perform the same operation over and over again—on lots of data at the same time.

In practice, using CUDA often means rethinking how you describe a problem. Instead of one loop doing all the work step by step, the work is broken into many small pieces so thousands of GPU threads can run in parallel.

That’s why GPUs excel at data-parallel workloads—situations where the same operation is applied across large collections of data, such as pixels, vectors, matrices, or particles.

The Appeal of GPU Acceleration: Why It Matters?

GPU acceleration isn’t magic—and it doesn’t make every program faster. But when a workload is a good match, the performance gains can be dramatic. Here’s why GPUs are so compelling in the right situations:

• Big speedups (when the problem fits): By running thousands of operations in parallel, GPUs can reduce runtimes from hours to minutes for the right kinds of tasks.

• High throughput on large datasets: Many data-heavy workloads naturally map to GPU execution, making it easier to process massive amounts of data efficiently.

• Simple scaling: On many systems, increasing compute power can be as straightforward as adding more GPUs.

• Energy efficiency: For certain workloads, GPUs can deliver more performance per watt than CPUs.

• Wide applicability: CUDA is used across a broad range of fields, from scientific simulations and video processing to machine learning and AI.

The key phrase here is “when the problem fits.” If a task is mostly sequential, full of complex branching, or spends most of its time waiting on I/O, a GPU may offer little benefit. This series focuses on the kinds of problems that do map well to GPU execution.

What is Parallel Computing?

In CUDA C++, parallel computing means doing many things at the same time. Instead of solving a problem step by step in a single sequence, the work is broken into lots of smaller pieces that can run simultaneously.

In CUDA, those “small pieces” are usually threads. Each thread is responsible for a tiny slice of the overall task—for example, processing one pixel, handling one element of a vector, or working on a single row or column.

When a problem can be divided this way—so that each thread can work mostly on its own with very little coordination—the speedups can be substantial.

In CUDA, parallelism generally shows up in two main forms:

• Task parallelism: different tasks or functions run independently at the same time.

• Data parallelism: the same operation runs across many data elements in parallel (very common in GPU workloads).

What Is Heterogeneous Computing?

In CUDA, “heterogeneous” simply means the program runs on more than one kind of processor. Most CUDA applications use both the CPU and the GPU, each doing the part it’s best at.

Typically, the CPU handles orchestration and control flow (setting things up, moving data, launching work), and the GPU handles the heavy parallel work.

A simple way to picture it:

• the CPU decides what should happen and when

• the GPU does the repetitive, data-parallel math at scale

A heterogeneous CUDA application usually has two parts:

• Host code (CPU): runs on the CPU and manages high-level logic (setup, memory transfers, launching kernels).
• Device code (GPU): runs on the GPU and performs parallel work in kernels, where many threads process data at the same time.

What are the different types of Computer Architecture?

In computer architecture, one common way to understand how a computer processes instructions and data is Flynn’s Taxonomy. It classifies computer systems based on how many instruction streams and how many data streams they can handle at the same time.

Flynn’s taxonomy divides computer architectures into four main types:

• Single Instruction Single Data (SISD): Single Instruction, Single Data (SISD) is the simplest way a computer can work. The system handles one instruction at a time and applies it to one piece of data. Everything happens in a straight line—finish one step, then move to the next. Early computers and basic single-core processors worked this way. While this approach is easy to understand, it becomes slow when the computer has to deal with large amounts of data or repetitive tasks.

• Single Instruction Multiple Data (SIMD): Single Instruction, Multiple Data (SIMD) improves performance by doing the same operation on many pieces of data at the same time. Instead of processing each data item one by one, the computer applies a single instruction across multiple data elements in parallel. This is especially useful for tasks like image processing, video rendering, or scientific calculations, where the same computation must be repeated many times. Modern GPUs rely heavily on SIMD to achieve high speed.

• Multiple Instruction Single Data (MISD): Multiple Instruction, Single Data (MISD) is a much less common architecture. In this model, the same data is processed simultaneously by multiple instructions. The goal here is usually not speed, but reliability. By running different operations on the same data, the system can detect errors or inconsistencies. Because of this specialized purpose, MISD is rarely seen in everyday computers and is mostly used in critical systems such as aerospace or fault-tolerant applications.

• Multiple Instruction Multiple Data (MIMD): Multiple Instruction, Multiple Data (MIMD) represents how most modern computers work today. In this architecture, multiple processors execute different instructions on different data at the same time. Each processor operates independently, which allows true parallel processing. Multi-core CPUs, cloud servers, and distributed systems all use MIMD, making it the most flexible and powerful architecture for handling complex and large-scale workloads.

How To Set Up Your Development Environment?

Before you start working with CUDA or GPU programming, it’s essential to choose a development environment that matches both your hardware and your learning goals. In general, there are two practical ways to get started: setting up CUDA on your local machine or using a cloud-based platform such as Google Colab. Each option has its own strengths, and understanding these will help you decide what works best for you.

This CUDA C++ Tutorial recommends both approaches depending on your needs:

If you already have access to an NVIDIA GPU and plan to do serious development, profiling, or long-term experimentation, a local setup is usually the better choice. On the other hand, if your goal is to learn the basics quickly or experiment without dealing with installations and configuration, a cloud environment can save a lot of time.

Using a Local Machine with Visual Studio Code, Eclipse, or NVIDIA Nsight Eclipse

Setting up CUDA locally means installing the CUDA Toolkit along with compatible NVIDIA GPU drivers on your system. This approach gives you full control over your development environment and allows you to work directly with your hardware. While the initial setup may take some effort, it provides the most flexibility and is well suited for real-world development and performance optimization.

Once CUDA is installed, modern development environments such as Visual Studio Code, Eclipse, or NVIDIA Nsight Eclipse can significantly improve your workflow. These IDEs support CUDA-related extensions that make writing and managing code easier. Features like syntax highlighting, intelligent code completion, and integrated debugging help you focus on solving problems rather than fighting tooling issues.

A local setup is especially valuable when you need advanced debugging tools, detailed performance profiling, or a stable environment for large projects. It closely resembles how CUDA applications are developed and deployed in professional and research settings.

Using Google Colab with GPU Support

Google Colab offers a convenient cloud-based alternative for running CUDA programs without setting up a local GPU environment. With Colab, you can access GPU resources directly through a web browser, making it an excellent option for beginners or for quick experiments. You simply select a GPU runtime, and the environment is ready to use within seconds.

Colab notebooks allow you to write and run CUDA-enabled C++ code directly inside notebook cells, which is particularly useful for learning and testing small examples. This approach removes the need to install drivers or toolkits manually and lets you focus entirely on understanding concepts and writing code.

However, cloud environments like Colab do come with limitations. Session timeouts, restricted customization, and limited access to advanced profiling tools make them less suitable for long-term or production-level development. Despite these constraints, Colab remains a great starting point for learning CUDA and exploring GPU programming fundamentals.

Writing The First CUDA Program

The easiest way to understand CUDA is to run a very small program and see it in action. The example below launches a simple GPU kernel that prints a message from multiple threads. This is not a performance example; it is meant to build intuition about how CUDA works.

This program highlights two key ideas. CUDA kernel code runs on the GPU, not the CPU, and launching a kernel creates many parallel threads instead of a single execution path. These concepts form the foundation of CUDA programming.

To keep things simple, this tutorial focuses on the basic host–device split. In Google Colab, the CUDA code is written inside a cell marked with %%cu. The kernel is defined using the global keyword, which indicates that the function runs on the GPU but is launched from the CPU. Once this relationship is clear, CUDA’s execution model becomes much easier to follow.

Now that the core idea is clear, let’s move on to the code. This first example is deliberately simple and meant to show how a CUDA kernel is written and executed on the GPU. Focus on the structure rather than the details—understanding this flow will make the rest of CUDA much easier to grasp.

Here is the code:

%%cu
#include <iostream>
 
__global__ void helloWorld() {
    printf("Hello, World from thread %d!\n", threadIdx.x);
}

• Launch the kernel from main: In a separate code cell, implement main and launch the kernel.

%%cu
int main() {
    helloWorld<<<1, 10>>>();
    cudaDeviceSynchronize();
    return 0;
}

• The output should show ten lines—one per thread—printing the thread index:

Output:
Hello, World from thread 0!
Hello, World from thread 1!
Hello, World from thread 2!
Hello, World from thread 3!
Hello, World from thread 4!
Hello, World from thread 5!
Hello, World from thread 6!
Hello, World from thread 7!
Hello, World from thread 8!
Hello, World from thread 9!

Summary

This CUDA C++ tutorial introduced CUDA as a way to run general-purpose C++ code on NVIDIA GPUs using kernels and massive parallelism. It also clarified why GPUs excel at throughput-oriented workloads, how heterogeneous programs split responsibilities between CPU and GPU, and what “parallel computing” means in the CUDA context.

With the environment set up (local or Colab) and a first kernel running, the next step is to dig deeper into threads, blocks, and grids—then connect those ideas to memory and performance.