Parallel Programming and Locations

We have seen so far how to create a distributed array. It is time to discuss how to use a distributed array for parallel programming in HPJava through a simple example. Figure 3.5 is a basic HPJava program for a parallel matrix addition.

Figure 3.5: A parallel matrix addition.

  Procs2 p = new Procs2(2, 3) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    double [[-,-]] a = new double [[x, y]] on p ; 
    double [[-,-]] b = new double [[x, y]] on p ;
    double [[-,-]] c = new double [[x, y]] on p ;

    ... initialize values in `a', `b'

    overall(i = x for :)
      overall(j = y for :)
        c [i, j] = a [i, j] + b [i, j] ;
  }

The overall construct is another control construct of HPJava. It represents a distributed parallel loop, sharing some characteristics of the forall construct of HPF. The index triplet of the overall construct represents a lower bound, an upper bound, and a step, all of which are integer expressions, as follows:

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} overall(i = x for l : u : s)\end{verbatim}\end{minipage}$$

The step in the triplet may be omitted as follow:

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} overall(i = x for l : u)\end{verbatim}\end{minipage}$$

where the default step is $1$. The default of the lower bound, l, is 0 and the default of the upper bound, u, is $N-1$, where $N$ is the extent of the range before the for keyword. Thus,

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} overall(i = x for :)\end{verbatim}\end{minipage}$$

means ``repeat for all elements, i, in the range x'' from figure 3.5. An HPJava range object can be thought of as a collection of abstract locations. From Figure 3.5, the two lines

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} Range x = n... ...im(0)) ; Range y = new BlockRange(N, p.dim(1)) ;\end{verbatim}\end{minipage}$$

mean that each of the range x and y has N locations. The symbol i scoped by the overall construct is called a distributed index. Its value is a location, rather an abstract element of a distributed range than an integer value. It is important that with a few exceptions we will mention later, the subscript of a multiarray must be a distributed index, and the location should be an element of the range associated with the array dimension. This restriction is a important feature, ensuring that referenced array elements are locally held. Figure 3.6 is an attempt to visualize the mapping of locations from x and y. We will write the locations of x as x[0], x[1], ..., x[N -1]. Each location is mapped to a particular group of processes. Location x[1] is mapped to the three processes with coordinates $(0,0)$, $(0,1)$ and $(0,2)$. Location y[4] is mapped to the two processes with coordinates $(0,1)$ and $(1,1)$.

Figure 3.6: Mapping of x and y locations to the grid p.

In addition to the overall construct, there is an at construct in HPJava that defines a distributed index when we want to update or access a single element of a multiarray rather than accessing a whole set of elements in parallel. When we want to update a[1, 4], we can write:

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} double [[-,... ... ... at(i = x[1]) at(j = y[4]) a [i, j] = 19 ;\end{verbatim}\end{minipage}$$

The semantics of the at construct is similar to that of the on construct. It limits control to processes in the collection that hold the given location. Referring again to Figure 3.6, the outer

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} at(i = x [1])\end{verbatim}\end{minipage}$$

construct limits execution of its body to processes with coordinates $(0,0)$, $(0,1)$ and $(0,2)$. The inner

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} at(j = y [4])\end{verbatim}\end{minipage}$$

then restricts execution down to just process $(0,1)$. This is the process that owns element a[1,4]. The restriction that subscripts must be distributed indices helps ensure that processes only manipulate array elements stored locally. If a process has to access elements non-locally held, some explicit library call should be used to fetch them. We often need to get the integer global index associated with a distributed index. But, using a distributed index as an integer value is not allowed in HPJava. To get the integer global index value for the distributed loop, we use the backquote symbol as a postfix operator on a distributed index, for example, i`, read ``i-primed''. The following nested overall loop is an example using an integer global index value:

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} overall(i = x for :) overall(j = y for :) a[i, j] = i\lq + j\lq ;\end{verbatim}\end{minipage}$$

Figure 3.7 is a complete example of Laplace equation by Jacobi relaxation that uses HPJava language features introduced in this section.

Figure 3.7: Solution for Laplace equation by Jacobi relaxation in HPJava.

  Procs2 p = new Procs2(P, P) ;
  on(p) {
    Range x = new BlockRange(N, p.dim(0)) ;
    Range y = new BlockRange(N, p.dim(1)) ;

    double [[-,-]] a = new double [[x, y]] on p ;

    // Initialize `a': set boundary values.    
    overall(i = x for :) 
      overall(j = y for :)
        if(i` == 0 || i` == N - 1 || j` == 0 || j` == N - 1)
          a [i, j] = i` * i` - j` * j`;
        else
          a [i, j] = 0.0;
    
    // Main loop.
    double [[-,-]] n = new double [[x, y]] on p ; 
    double [[-,-]] s = new double [[x, y]] on p ;
    double [[-,-]] e = new double [[x, y]] on p ; 
    double [[-,-]] w = new double [[x, y]] on p ;
    double [[-,-]] r = new double [[x, y]] on p ;
  
    do {
      Adlib.shift(n, a,  1, 0) ;
      Adlib.shift(s, a, -1, 0) ;
      Adlib.shift(e, a,  1, 1) ;
      Adlib.shift(w, a, -1, 1) ;

      overall(i = x for 1 : N - 2 ) 
        overall(j = y for 1 : N - 2) {
          double newA = (double)0.25 * (n [i, j] + s [i, j] + 
                                        e [i, j] + w [i, j]) ;
          r [i, j] = Math.abs(newA - a [i, j]) ;
          a [i, j] = newA ;
        }
    } while(Adlib.maxval(r) > EPS);

    // Output results
    ...
  }