MPICH-SCore version 1.0 (ch_score): Running an MPI application

The binary generated by an MPI compiler provided by MPICH-SCore runs on the SCore-D operating system only. Thus you should prepare for one of the Single user environment or the Multiple user environment of SCore-D operating system.

Running in the Single user environment

First you need to run the scout shell program. scout provides a remote shell environment. Then, you may run your MPI application using mpirun on the shell. For example, when you run the application using four nodes:

$ setenv SCBDSERV server.pccluster.org $ msgb -group pcc& $ scout -g pcc [comp0-3]: SCOUT(3.1.0): Ready. $ . . . . . . $ mpirun -np 4 ./mpi_program args ... . . . . . . $ exit SCOUT: session done $

MPICH-SCore supports clusters consists of SMP nodes. Runtime code of the MPICH-SCore spawns multiple MPI processes on a SMP node.

You may specify the number of MPI processes on a SMP node using -np option of mpirun. For example, when you need to run eight MPI processes on four dual processor machines, specifies the options as follows:

$ mpirun -np 4x2 ./my_mpi_program args...

When you do not specify the number of MPI processes on a SMP node, MPICH-SCore spawns MPI processes for each processor of the cluster. Thus the "-np 4" is equal to "-np 2x2" on a cluster consists of dual processor nodes.

You may use scrun instead of mpirun. For example, when you run the MPI application using 16 processors on eight dual processor nodes:

$ scrun -nodes=8x2 ./my_mpi_program args...

When you do not specify -nodes option for scrun, MPICH-SCore spawns MPI processes for each processor of all nodes reserved by scout.

Running the sample application in a single user environment

Here we show an example to run the sample application alltoall, that measures performance of MPI_Alltoall function. This application requires two command line arguments. The first is message length for the all-to-all communication. The second is number of iteration. The result consists of three fields, number of MPI processes, message length and elapsed time for one all-to-all communication. Unit of the time is micro seconds:

$ setenv SCBDSERV server.pccluster.org $ msgb -group pcc& $ scout -g pcc [comp0-3]: SCOUT(3.1.0): Ready. $ . . . . . . $ scrun -nodes=4x2 alltoall 3000 10000 SCORE: Connected (jid=1) <0:0> SCORE: 8 nodes (4x2) ready. 8 3000 1052.230600 $ . . . . . . $ exit SCOUT: session done $

Running in the Multiple user environment

You may run the application just issuing mpirun when using the multiple user environment. Remember you should specify the hostname of SCore-D server as follows:

$ mpirun -np 4x2 -score scored=comp3.pccluster.org ./mpi_program args...

You are able to specify the hostname using environment variable SCORE_OPTIONS instead of the mpirun option:

$ export SCORE_OPTIONS=scored=comp3.pccluster.org $ mpirun -np 4x2 ./mpi_program args...

You may also use scrun instead of mpirun:

$ export SCORE_OPTIONS=scored=comp3.pccluster.org $ scrun -nodes=4x2 ./mpi_program args...

The way to specify the number of MPI processes in an SMP node is same to that for the single user environment. See the previous section.

Improving the performance

Sliding border lines to change the protocols

MPICH-SCore version 1.0 (ch_score) transfers MPI messages using three protocols described below. The runtime code choose the one of them by the message size sending. You may change the border lines for choosing the protocol. The change makes possible to improve the performance of some applications.

Short protocol
transfers a MPI header and message body within a single packet of PM, the low level message passing system of SCore. The total size of the MPI header and the MPI message is limited to MTU of PM. The sender does not wait ready of the receiver. Thus a message unexpected by the receiver stops at a temporary buffer of the receiver.
Eager protocol
transfers a MPI header and message body using multiple packets of PM. The sender does not wait ready of the receiver as the short protocol. Thus a message unexpected by the receiver stops at a temporary buffer of the receiver.
Rendezvous protocol
transfers a MPI message synchronously. The sender waits ready of the receiver. Then the message body is carried. The remote memory access (RMA) facility of PM is used when the user enable it. Otherwise MPICH-SCore transfers the message body dividing it to multiple packets of PM.

The threshold to change the eager protocol from the short protocol is minimum length of MTU values of PM device.s

You can change the borderline between the eager protocol and the rendezvous protocol. The default is 16 kbytes. To change this value, use mpi_eager option when running the application. For example, to specify 300 kbytes as the borderline use mpi_eager option as follows:

$ mpirun -np 4x2 -score mpi_eager=300000 ./mpi_program args...

Alternatively, you can use scrun:

$ scrun -nodes=4x2,mpi_eager=300000 ./mpi_program args...

Using PM Remote Memory Access facility (Zero-copy/One-copy transfer)

Currently, the remote memory access (RMA) facility of PM is supported on PM/Myrinet and PM/Shmem, which is a PM that supports inter process communication within an SMP node. You are able to use PM RMA when using the rendezvous protocol only. To enable PM RMA, use mpi_zerocopy option:

$ mpirun -np 4x2 -score mpi_zerocopy=on mpi_program args...

Alternatively, you can use scrun as follows:

$ scrun -nodes=4x2,mpi_zerocopy=on ./mpi_program args...

Some RMA implementations, such as PM/Myrinet transmit data using DMA only. We call the message transfer using such a RMA the Zero-copy transfer since no memory copy by CPU is required when transmitting. MPICH-SCore realizes Zero-copy transfer when using RMA of PM/Myrinet. Zero-copy transfer improves maximum bandwidth of point-to-point message transfer because it reduces congestion of memory access. Zero-copy transfer is effective for some application. However it is not so effective for others since it involves overhead to synchronize the sender and the receiver.

The message transfer using PM/Shmem RMA is the One-copy transfer. Since PM/Shmem realizes copy between virtual memory spaces using the PM/Shmem device driver, the RMA is implemented as one copy. MPICH-SCore version 1.0 enables One-copy transfer only when Zero-copy transfer is enabled. This restriction has been removed on the version 2.0.

Restrictions

Message transfer using remote memory access facility is supported only on the communication over Myrinet and shared memory between multiple MPI processes in an SMP node.
Threshold to change the eager protocol and rendezvous protocol is unique to the all communication devices underlying. This restriction has been removed on the version 2.0.
The user cannot use PM/Myrinet RMA and PM/Shmem RMA independently. This restriction has been removed on the version 2.0.
Currently the user cannot use RMA (rendezvous protocol) when checkpointing.