NVIDIA
Interview Prep

CPUs have few, powerful cores optimized for sequential tasks with complex branch prediction and large caches. GPUs have thousands of simpler cores designed for massive parallelism — perfect for the matrix multiplications that dominate deep learning. A forward pass through a transformer is essentially a series of GEMM (general matrix multiply) operations that can be parallelized across thousands of cores simultaneously. CUDA's programming model exposes this parallelism through thread blocks and warps.

Walk through the profiling process (NVIDIA Nsight, torch.profiler, nvprof), what the bottleneck was (memory bandwidth vs. compute-bound, kernel launch overhead, memory allocation), and the optimization technique you applied (kernel fusion, memory coalescing, tensor core utilization via FP16, operator overlap with CUDA streams, reducing memory copies between host and device). Quantify the improvement.

Walk through a specific CUDA kernel you wrote or optimized: the computation it performed, the thread hierarchy you designed (grid, block, warp sizing), memory management (shared memory usage, register pressure, memory access patterns), synchronization primitives, and the performance outcome vs. the naive implementation. Discuss any challenges with numerical precision or race conditions.

Systematic approach: 1) Measure throughput (samples/sec, tokens/sec) and identify if you're compute-bound or I/O-bound, 2) Profile with Nsight or PyTorch Profiler to identify bottleneck operators, 3) Check GPU utilization (low utilization = CPU bottleneck or data loading bottleneck, high utilization = compute bottleneck), 4) Optimize bottlenecks: data loading (multiple workers, prefetching, pinned memory), kernel efficiency (mixed precision, operator fusion), memory (gradient checkpointing, smaller batch size to fit in memory).

Walk through the latency target, the model's baseline performance, the optimization techniques you applied (quantization — INT8 or FP16 — reducing precision loss, knowledge distillation — training a smaller model to match a larger one's outputs, TensorRT or ONNX Runtime for inference engine optimization, batching strategy, KV cache management for LLMs), and the latency/throughput/accuracy trade-offs you navigated. Quantify the improvement.

NVLink is NVIDIA's high-bandwidth interconnect between GPUs on the same node, providing much higher bandwidth than PCIe (for A100: 600 GB/s bidirectional vs. PCIe's ~32 GB/s). NVSwitch is the switch fabric that connects all GPUs in a node via NVLink, enabling any-to-any all-reduce operations at NVLink bandwidth. This matters for large model training because all-reduce operations (synchronizing gradients across GPUs) become the bottleneck at large scale — NVLink reduces this from minutes per step to seconds.

Walk through the computational problem, why parallelization was necessary (scale, latency requirements), how you partitioned the data or computation (by data, by model layers, by pipeline stage), what the synchronization requirements were (barrier sync, all-reduce, point-to-point communication), what the performance characteristics were (computation vs. communication ratio), and the result.

Show genuine technical depth and forward-looking thinking: discuss memory bandwidth (HBM capacity and bandwidth are often the limiting factor for large model training), interconnect (inter-node bandwidth is the bottleneck for clusters beyond 1000 GPUs), new compute paradigms (photonic computing, analog in-memory computing, specialized attention hardware), energy efficiency (training GPT-4 class models consumes enormous power), and the memory wall (compute growing faster than memory bandwidth). Share your own view on which of these is most tractable.