Discussion

In this chapter we have given a historical review of data parallel languages, High Performance Fortran (HPF), message-passing systems for HPC, and high-level libraries for distributed arrays. Data parallelism means a task with large data structures can be split across nodes. Historically, data parallel programming approach has gradually evolved from SIMD (Single Instruction Multiple Data) via MIMD (Multiple Instruction Multiple Data) to SPMD (Single Program Multiple Data). There are two approaches to data parallelism: languages and communication libraries. HPF is an extension of Fortran 90 to support the data parallel programming model on distributed memory parallel computers. Despite of the limited success of HPF, many of the originally defined goals for HPF standardization of a distributed data model for SPMD programming are important and affect contemporary data parallel languages. Message-passing is a different approach from HPF. It explicitly lets processes communicate through messages on classes of parallel machines with distributed memory. The Message Passing Interface (MPI) established a standard API for message-passing library routines, inspired by PVM, Zipcode, Express, p4, PARMACS, etc. The major advantages making MPI successful are its portability and scalability. High-level libraries for distributed arrays, such as PARTI, The Global Array Toolkit, and NPAC PCRC Runtime Kernel - Adlib, are designed and implemented in order to avoid low-level, and complicated communication patterns for distributed arrays. HPF, message-passing, and high-level libraries for distributed arrays have influenced the HPspmd programming model we will introduce in the next chapter. Our model takes features from HPF, is programmable in message passing paradigm, and is closely involved with high-level libraries and distributed arrays.