Checkpointing

This page describes how to checkpoint and restart your parallel process, and the limitations of checkpointing.

How to checkpoint your parallel process

You can specify the checkpoint option on the scrun command in the following manner to periodically checkpoint your parallel process:

scrun -checkpoint=interval a.out

If you want to checkpoint with single user mode, you must execute with group option wuthout the SCOUT environment.

scrun -group=pcc,checkpoint=interval a.out

where a.out is the program name and interval is the checkpointing period specified in format of [1-9][0-9]*[sSmMhHdD]. The sSmHhHdD is a suffix specifying a time unit. s/S, m/M, h/H, and d/D are interpreted as in seconds, minutes, hours, and days, respectively. The default unit is in minutes.

After the parallel process is started, you will see the following messages printed to stderr periodically:

SCORE: Checkpointing ... done.

This message is printed after each successful checkpoint.

Another way to checkpoint your process is to send SIGQUIT to the FEP (Front-End Process) process. This is done by simply pressing "^\" in the console window where you executed the program. You will be informed of the successful checkpoint with the above message.

How to checkpoint your multithreaded process

You must load the CRT (Checkpoint/Restart for Threads) module in the following manner on each node to checkpoint your multithreaded process:

installdir/crt.target/etc/init.d/crt start

You can execute the scrun with checkpoint option.

How your parallel process is restarted from a checkpoint

If SCore-D terminates for some reason, your FEP process will simply be terminated with a following message printed to stderr:

FEP:ERROR SCore-D unexpectedly terminated.

By specifying the -checkpoint option, you can avoid this behavior; i.e., the FEP process will stay alive and the following message is shown:

FEP:WARNING SCore-D unexpectedly terminated. FEP: [07/Feb/2000 15:28:40] Waiting for SCore-D to be restarted ...

Do not kill this FEP process if you want to restart your parallel process from a checkpoint. After a system crash, if the system administrator runs SCore-D with the restart option, your parallel process will be automatically restarted from the latest checkpoint, as long as the FEP process is still alive. You will then see the following message:

FEP: [07/Feb/2000 15:30:43] SCore-D restarted.

and your parallel process will continue from the last checkpoint image. If a checkpoint image was not taken before a system crash, your parallel process will execute from the beginning again.

If the system administrator does not rerun SCore-D with the restart option, then your FEP process will be killed and the checkpoint for the parallel process is lost. You will see the following message:

FEP: [07/Feb/2000 16:31:17] restart canceled by SCore-D.

How your multithreaded process is restarted from a checkpoint

You must load the CRT module if you want to restart your multithreaded process from a checkpoint. If the system administrator runs SCore-D with the restart option, your multithreaded process will be automatically restarted from the latest checkpoint.

Limitations

Following limitations exist when using the checkpointing facility in SCore-D:

Only the latest checkpoint can be used to restart the parallel process. A new checkpoint overrides the previous one.
File contents cannot be resumed. You must preserve the contents of files so they remain unchanged from the checkpoint time until the parallel process is restarted.
Sockets, pipes, and shared memory cannot be resumed. The only exception to this is the shared memory area which is mmap(2)'ed with the MAP_SHARED option and the file descriptor is binded to a persistent file. Such a memory area can be resumed as long as the file contents are preserved at the checkpoint.
Only a part of kernel/hardware status is restored. For example, gettimeofday(2) will return the current system clock time, not adjusted by the interval time from the checkpoint.
Unable to resume from multiple disk crashes. Single disk crash can be restored when a soare host is specified.
You must compile the CRT module to the booting kernel.

Messages

The following messages may appear while checkpointing/restarting:

Warning messages:
- Initializing checkpointer failed: error-code
  This message is shown while initializing the application program. The program will be normally started, but a request for checkpointing will be ignored.
- Checkpointing failed: error-code
  This message is shown while checkpointing. The requested checkpoint is cancelled for this time. Subsequent checkpoints, however, will be normally executed.
- Restarting failed: error-code
  This message is shown while restarting. The program will normally be started without restarting.
- Dynamically linked program is uncheckpointable.
  This message is shown when you request checkpointing by sending SIGQUIT to the FEP process. The requested checkpoint is cancelled.
Error messages:
- Dynamically linked program is uncheckpointable
  This message is shown when you specify the -checkpoint=number option to your scrun command. The execution will be totally cancelled.
- Fatal error in restarting: error-code
  This message is shown while restarting. Due to some hazardous error, the program will not be started even normally. The execution will be totally cancelled.

PC Cluster Consortium