Computation of new values for grid variables can now be done concurrently, with each grid process computing new values for the grid points in its local section. For a grid point in the interior of a local section, it is easy to see that the data required to compute its new values (old values at the point and neighboring points) is readily available locally, but for grid points on the boundary of a local section, the required data may reside in another process. This is handled by surrounding each local section with a ghost boundary, which is used as a buffer to contain a shadow copy of data from neighboring local sections. Figure 3 illustrates a local section plus ghost boundary. Note that the ghost boundary can have width greater than 1; if, for example, the computation of new values for a grid point requires values from points 2 grid cells away, then the ghost boundary should have width 2.
Figure 3: Local section with ghost boundary.
Values for array elements in the ghost boundaries are filled in via a boundary exchange operation, in which grid processes send data from the boundaries of their local sections to the ghost boundaries of their neighbors. Once this is done, values for all points in a local section (possibly excluding those on the boundary of the global array) can be computed exactly as one would compute them in a sequential program, with the ghost boundaries supplying neighboring values for points on the boundaries of the local sections.
As an example, consider performing the computation shown in figure 1 to the points in the local section shown in figure 3. (Both arrays mentioned by the formula - newval and oldval - have a local section of the form shown.) The code would look something like this:
parameter (N=6)
parameter (M=4)
real newval(0:N+1, 0:M+1)
real oldval(0:N+1, 0:M+1)
.... boundary exchange ....
do i=1,N
do j=1,M
newval(i,j) = f(oldval(i-1,j), oldval(i+1,j),
oldval(i,j-1), oldval(i,j+1))
enddo
enddo
Observe that no special handling is required for points on the
boundaries of the local section (i = 1 or N,
j = 1 or M); values for oldval at neighboring
points
come from the ghost boundaries, which have been filled in by
a boundary exchange operation.