Engineering specifications for NVIDIA’s CUDA (Compute Unified Device Architecture) span both hardware microarchitecture requirements and programming-model constraints. CUDA maps software parallel execution to underlying physical Streaming Multiprocessors (SMs) via a strict hardware/software abstraction layer.
[1] (https://en.wikipedia.org/wiki/CUDA),
[2] (https://developer.download.nvidia.com/compute/cuda/docs/CUDA_Architecture_Overview.pdf),
[3] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e)
1. Hardware Execution & Core Infrastructure
The foundational processing block of a CUDA-capable GPU is the Streaming Multiprocessor (SM).
[1] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e)
• The Warp: The atomic unit of execution. The hardware schedules and executes threads in groups of 32 threads (a warp). All threads in a warp execute via the SIMT (Single Instruction, Multiple Threads) paradigm, running the same instruction in lock-step.
[1] (https://docs.nvidia.com/cuda/cuda-programming-guide/pdf/cuda-programming-guide.pdf),
[2] (https://www.youtube.com/watch?v=NJjnRiJVsoY),
[3] (https://www.youtube.com/watch?v=yMNlAy6X4pg)
• Core Types:
• CUDA Cores: Primary ALUs handling integer, FP32, and FP64 arithmetic.
• Tensor Cores: Matrix-multiply-accumulate (MMA) pipelines built for mixed-precision AI operations (supporting FP64, TF32, FP16, BF16, FP8, and FP4 depending on architecture generation).
• RT Cores: Hardware blocks dedicated to bounded volume hierarchy (BVH) ray-tracing math.
[1] (https://dev.to/maxvyaznikov/cuda-compute-capability-what-it-is-and-why-it-matters-for-ml-engineers-1mhg),
[2] (https://jarvislabs.ai/blog/cuda-cores),
[3] (https://www.naddod.com/ai-insights/a-comprehensive-guide-to-cuda-cores-in-nvidia-gpu)
2. Thread Hierarchy Limits
CUDA organizes tasks through an abstract software hierarchy—Threads, Blocks, and Grids—which the hardware runtime scheduler binds to execution resources.
[1] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e),
[2] (https://www.youtube.com/watch?v=q2mo_RS-zOk)
Metric / Parameter Hard Specification Limit
Max Threads per Thread Block 1,024 threads
Warp Size 32 threads (constant across all architectures)
Max Thread Block Dimensions 1024 x 1024 x 64 (X, Y, Z coordinates)
Max Grid Dimensions 2³¹-1 for X, 65,535 for Y and Z
Max Threads per SM 1,536 to 2,048 (Architecture dependent, e.g., Ampere/Hopper/Blackwell)
Max Blocks per SM 16 to 32 (Architecture dependent)
3. Memory Hierarchy & Capacity Specifications
Data movement is strictly segmented into physical and logical boundaries with massive variations in bandwidth and latency.
[1] (https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html),
[2] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e)
[Registers: Fast, Private] ➔ [Shared Memory / L1 Cache: Low Latency, Per Block] ➔ [Global Memory (VRAM): High Capacity, Shared by All Blocks]
• Registers: The fastest memory on the chip, private to each thread. Typically 64KB to 256KB total register file per SM. Max registers per thread is tightly limited to 255.
[1] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e)
• Shared Memory (SRAM): On-chip, software-managed cache allocated per thread block. Max capacity varies drastically by Compute Capability (CC):
• Turing (CC 7.5): up to 64 KB
• Ampere (CC 8.0): up to 164 KB
• Hopper (CC 9.0): up to 228 KB
• Blackwell (CC 10.7): up to 328 KB
[1] (https://developer.nvidia.com/cuda/gpus),
[2] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e),
[3] (https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html),
[4] (https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/compute-capabilities.html)
• Global Memory (VRAM): High-latency off-chip memory (GDDR6X / HBM3e) accessible by all threads.
[1] (https://www.facebook.com/xmodulo/videos/nvidia-cuda-architecture/4228309480648482/),
[2] (https://medium.com/@neuraldev/cuda-and-its-architecture-a-complete-beginner-friendly-guide-to-how-gpus-really-compute-504f1d047f7e)
• Memory Byte Ordering: Native Little-endian representation across all supported platforms.
[1] (https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/compute-capabilities.html)
4. Compilation & Intermediate Representation (ISA)
CUDA provides forward and backward compatibility using a dual-stage compilation pipeline.
[1] (https://www.youtube.com/watch?v=NJjnRiJVsoY)
• PTX (Parallel Thread Execution): A machine-independent, virtual Instruction Set Architecture (ISA). The nvcc compiler compiles device-level C++ code into PTX.
[1] (https://gist.github.com/agirault/d0d66c12c85dbc2662c95569ef039a1d),
[2] (https://developer.download.nvidia.com/compute/cuda/docs/CUDA_Architecture_Overview.pdf),
[3] (https://www.youtube.com/watch?v=NJjnRiJVsoY)
• SASS (Source Absolute Assembly): The native machine code specific to a physical GPU architecture. The NVIDIA graphics driver utilizes a Just-In-Time (JIT) compiler to target PTX into SASS at runtime.
[1] (https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/cuda-platform.html),
[2] (https://www.youtube.com/watch?v=NJjnRiJVsoY),
[3] (https://myzhar.tech/tutorials/cuda/cuda-compute-capability/)
• Binary Compatibility: Guaranteed within a major Compute Capability generation (e.g., binaries compiled for sm_80 will execute on sm_86), but strictly broken across major architectural shifts (e.g., sm_80 to sm_90).
[1] (https://docs.nvidia.com/cuda/cuda-programming-guide/pdf/cuda-programming-guide.pdf),
[2] (https://docs.nvidia.com/cuda/cuda-programming-guide/01-introduction/cuda-platform.html)
5. Compute Capability (CC) Engineering Milestones
NVIDIA tracks major architectural and structural feature sets by a version integer:
[1] (https://www.mathworks.com/help/rtw/ref/computecapability.html),
[2] (https://gpusmith.com/articles/en/cuda-compute-capability-table-gpu)
• CC 7.0 / 7.5 (Volta / Turing): First-generation Tensor Cores, introduction of independent thread scheduling (mitigating warp-divergence penalties), and hardware ray-tracing.
[1] (https://www.youtube.com/watch?v=U3xJJ8h25Fc),
[2] (https://developer.nvidia.com/blog/navigating-gpu-architecture-support-a-guide-for-nvidia-cuda-developers/),
[3] (https://dev.to/maxvyaznikov/cuda-compute-capability-what-it-is-and-why-it-matters-for-ml-engineers-1mhg)
• CC 8.0 / 8.6 / 8.9 (Ampere / Ada Lovelace): Sparsity acceleration in Tensor Cores, TF32 and FP8 data types, and Multi-Instance GPU (MIG) hardware partitioning.
[1] (https://vrlatech.com/vrla-tech-workstations/cuda-toolkit/),
[2] (https://jarvislabs.ai/blog/cuda-cores),
[3] (https://dev.to/maxvyaznikov/cuda-compute-capability-what-it-is-and-why-it-matters-for-ml-engineers-1mhg)
• CC 9.0 (Hopper): Asynchronous Distributed Shared Memory (TMA – Tensor Memory Accelerator), Thread Block Clusters, DPX instructions for dynamic programming, and the Transformer Engine.
[1] (https://docs.nvidia.com/cuda/cuda-programming-guide/pdf/cuda-programming-guide.pdf),
[2] (https://jarvislabs.ai/blog/cuda-cores),
[3] (https://dev.to/maxvyaznikov/cuda-compute-capability-what-it-is-and-why-it-matters-for-ml-engineers-1mhg)
• CC 10.0 / 12.0 (Blackwell generation): Dual-channel execution, native FP4 tensor execution, and FlashAttention-3 physical hardware acceleration paths.
[1] (https://developer.nvidia.com/cuda/gpus),
[2] (https://vrlatech.com/vrla-tech-workstations/cuda-toolkit/),
[3] (https://dev.to/maxvyaznikov/cuda-compute-capability-what-it-is-and-why-it-matters-for-ml-engineers-1mhg)
