Parallel Computing Concepts

This webinar, marking the first of the year, introduces parallel computing concepts primarily for non-programmers using supercomputers. The COMPLeX (Comprehensive Learning for End-Users to Effectively Utilize Cyberinfrastructure) program, funded by the NSF, aims to train users to effectively leverage supercomputing resources. Topics covered will include Linux tools, Bash scripting, security, batch computing, data management, and interactive computing. The session focuses on understanding why parallel computing is necessary, its broad applications beyond traditional science, common myths, and the structure of parallel computers.

The session delves into the differences between processes and threads, which are both independent sequences of execution. A process is an instance of a program with its own memory, while threads are lightweight entities executing within a process, sharing memory. MPI (Message Passing Interface) is the de facto standard for parallelizing C, C++, and Fortran codes across distributed memory systems, allowing communication between processes. OpenMP is an API for shared memory programming, primarily implemented by compilers, enabling threading within a single node. While MPI is dominant for distributed memory, OpenMP is key for shared memory parallelism. Users do not need to be programmers but should be aware of these concepts, especially when building executables.

Hybrid applications combine both distributed and shared memory parallelism, typically using MPI and OpenMP, to benefit from both approaches. This strategy is crucial as the number of CPU cores per processor increases. A typical scenario involves running one process per node with multiple threads within that node. However, the optimal balance between processes and threads depends on the application. Users are advised to benchmark their applications with different combinations to achieve the best performance. Tasks in job schedulers like Slurm generally correspond to processes.

Amdahl's Law defines the theoretical limit on the speedup of a code based on the fraction that can be parallelized (P) and the number of processes (N). Even a small percentage of sequential code (S) can severely limit scalability. For example, 99% parallel content only yields a maximum speedup of 100, meaning a 128-core system would be underutilized. Beyond Amdahl's Law, other factors limit scalability, including problem size (not enough work for all processors), uneven load balancing (processors sitting idle), and communications overhead (data transfer between processes). These factors can significantly degrade performance in real-world applications.

Despite the limitations, high scalability can be achieved through various strategies, often implemented by software developers. These include growing the problem size with the number of cores, overlapping communication with computation, and dynamic load balancing. For end-users, the most practical approach to determine optimal resource usage is to conduct a 'scaling study.' This involves running a representative problem on varying numbers of processors and analyzing the runtime. Visualizing scaling data with logarithmic axes is crucial for accurately assessing performance, unlike linear plots that can obscure details at higher core counts.

When deciding on core counts for parallel applications, consider the sensitivity to completion time. For workloads like parameter sweeps, prioritize 100% efficiency, possibly running many smaller jobs. For critical applications, higher core counts might be necessary, even if it sacrifices some parallel efficiency. Parallel efficiency is vital, especially for shared resources like supercomputers or cloud platforms, as it prevents wasting valuable compute cycles or financial resources. The webinar concludes by reiterating the importance of understanding these fundamentals for effective use of parallel computing and points to further training resources.