PBS/SCore(Open PBS/SCore) Administrator's Guide

Index

  1. Introduction
  2. Host Configuration
  3. Common file setting
    1. /var/scored/pbs/pbs_server_name
  4. File setup on PBS server host
    1. /var/scored/pbs/ directory
    2. /var/scored/pbs/server_priv/nodes
  5. File setup on compute hosts
    1. /var/scored/pbs/ directory
    2. /var/scored/pbs/mom_priv/config
  6. File setup on user hosts
  7. File setup on the host to submit SCore jobs
    1. /var/scored/pbs/ directory
    2. /var/scored/pbs/mom_priv/config
  8. Initiating PBS servers
    1. pbs_mom
    2. pbs_sched
    3. pbs_server
    4. Confimation
    5. Creation of default queue
    6. Start of scheduling
  9. The difference between PBS and PBS/SCore
    1. score property
    2. Host to invoke SCore jobs

1. Introduction

PBS/SCore must be configired independently from the SCore configuration. This document describes hot to configure PBS/SCore. For further deials, please refer to the Administrator Guide attached to PBS(/SCore). In this document, it is assumed that SCore is already installed and configured.

If SCore system has been installed by using EIT, all the configuration procedure described in this document has been automatically done by EIT.

2. Host Configuration

There are 4 host types in PBS/SCore. An administrator must decide which host is which type.

  1. PBS server host

    One of the hosts must be a PBS server host on which PBS server (pbs_server) and PBS scheduler (pbs_sched) are running.

    It is possible that msgbserv and/or scoreboard server programs of SCore can also run on the PBS server host.

  2. Compute host

    Parallel jobs run on compute hosts. On each compute host, pbs_mom server must be running to monitor host resource and invoke user parallel jobs. To run SCore jobs, compute hosts must be registered in scorehosts.db file, and scoreboard server is up and running on some host.

  3. User hosts

    To submit parallel jobs from user hosts, PBS commands, such as qsub and qdel, must be installed on the user hosts.

  4. Host to invoke SCore jobs

    An SCore administrator may specify a host from which SCore jobs are invoked via the scout command. This host should not be a compute host. Like a computer host, pbs_mom must be initiated on this host too. If this host is not allocated, one of compute hosts which are allocated to a submitted job is used.

In this document, it is assumed that above hosts are cofigured like the following.

  1. PBS server host
    server.pccluster.org
  2. Compute hosts
    comp0.pccluster.org
    comp1.pccluster.org
    comp2.pccluster.org
    comp3.pccluster.org
  3. User hosts
    server.pccluster.org
  4. Host to invoke SCore jobs
    server.pccluster.org

3. File setup on all hosts

Every (possible) host must be configured as in the following.

/opt/score/etc/pbs_server_name

Default PBS server host name must be specified in the /opt/score/etc/pbs_server_name file. 

	server.pccluster.org

4. File setup on PBS server host

/var/scored/pbs/ directory

Execute the following command to setup the /var/scored/pbs/. This operation is not needed if SCore system has been installed by using the binary rpms or EIT. 

	# cd /opt/score/install
	# ./pbs_mkdirs aux server
	# cp dedicated_time holidays resource_group sched_config \
	    /var/scored/pbs/sched_priv

/var/scored/pbs/server_priv/nodes

All compute hosts must be listed in the /var/scored/pbs/server_priv/nodes file. If scorehosts.db file is already created and scoreboard server is up and running, then the scbd2pbs command creates the nodes file via the scoreboard database. 

	% /opt/score/sbin/scbd2pbs pcc > /var/scored/pbs/server_priv/nodes

In this example, all hosts belong to pcc host group are listed, and then the nodes file is created. The follwoing is an example of the created nodes file. 

	comp0.pccluster.org	pcc score
	comp1.pccluster.org	pcc score
	comp2.pccluster.org	pcc score
	comp3.pccluster.org	pcc score

Each line consists of hostname and property names of the host. As a PBS property name, host group name, "pcc" in this example, is used. In scorehosts.db file, if a host record has an attribute named "pbs", then its associated value(a) is added to the line as property name(s).

For more detail on the nodes file, please refer to Section 3.3.2. Declaring Nodes of PBS Administrator Guide

5. File setup on compute hosts

/var/scored/pbs/ directory

Type the following command to setup the /var/scored/pbs/. This operation is not needed if SCore system has been installed by using the binary rpms or EIT. 

	# /opt/score/install/pbs_mkdirs aux mom

/var/scored/pbs/mom_priv/config

In the /var/scored/pbs/mom_priv/config file, PBS server hostname must be included as in the following. 

	$logevent 0x1ff
	$clienthost server.pccluster.org

For more detail on the config file, please refer to Section 9.2. pbs_mom of PBS Administrator Guide

6. File setup on user hosts

There is nothing to do with user hosts.

7. File setup on the host to submit SCore jobs

/var/scored/pbs/ directory

Type the following command to setup the /var/scored/pbs/. This operation is not needed if SCore system has been installed by using the binary rpms or EIT. 

	# /opt/score/install/pbs_mkdirs aux mom

/var/scored/pbs/mom_priv/config

In the /var/scored/pbs/mom_priv/config file, PBS server hostname and all compute hostnames must be listed. 

	$logevent 0x1ff
	$clienthost server.pccluster.org
	$clienthost comp0.pccluster.org
	$clienthost comp1.pccluster.org
	$clienthost comp2.pccluster.org
	$clienthost comp3.pccluster.org

For more detail on the config file, please refer to Section 9.2. pbs_mom of PBS Administrator Guide

8. Initiating PBS Servers

pbs_mom

Initiate the pbs_mom program on all compute host. If there is a host to submit SCore jobs, then the pbs_mom program must be initiate on the host too.

Red Hat or Turbo Linux: 
	% su
	# /etc/rc.d/init.d/pbs_mom start
SuSE Linux: 
	% su
        # /etc/init.d/pbs_mom start

pbs_sched

Initiate PBS scheduler on PBS server host.

Red Hat or Turbo Linux: 
	% su
	# /etc/rc.d/init.d/pbs_sched start
SuSE Linux: 
	% su
        # /etc/init.d/pbs_sched start

pbs_server

Initiate the pbs_server program on the server host. If there is a host to submit SCore jobs, add -m option followed by the name of the host. If this is the first time to initiate PBS server, add -t option to setup databse.

For the first time, 
	% su
	# /opt/score/sbin/pbs_server -t create -m server.pccluster.org
Otherwise,
Red Hat or Turbo Linux: 
	% su
	# /etc/rc.d/init.d/pbs_server start
SuSE Linux: 
	% su
        # /etc/init.d/pbs_server start -m

Confirmation

To confirm if the PBS server processes are running, invoke the following command. 

	% su
	# /opt/score/bin/qmgr -c 'p n @active'

If they are running properly, then the following output is obtained. 

	#
	# Create nodes and set their properties.
	#
	#
	# Create and define comp0.pccluster.org
	#
	# create node comp0.pccluster.org # unsuppored operation
	set node comp0.pccluster.org state = free
	set node comp0.pccluster.org properties = any
	set node comp0.pccluster.org properties += score
	set node comp0.pccluster.org properties += score-pcc
	set node comp0.pccluster.org ntype = cluster
	#
	# Create and define comp1.pccluster.org
	#
	# create node comp1.pccluster.org # unsuppored operation
	set node comp1.pccluster.org state = free
	set node comp1.pccluster.org properties = any
	set node comp1.pccluster.org properties += score
	set node comp1.pccluster.org properties += score-pcc
	set node comp1.pccluster.org ntype = cluster
	#
	# Create and define comp2.pccluster.org
	#
	# create node comp2.pccluster.org # unsuppored operation
	set node comp2.pccluster.org state = free
	set node comp2.pccluster.org properties = any
	set node comp2.pccluster.org properties += score
	set node comp2.pccluster.org properties += score-pcc
	set node comp2.pccluster.org ntype = cluster
	#
	# Create and define comp3.pccluster.org
	#
	# create node comp3.pccluster.org # unsuppored operation
	set node comp3.pccluster.org state = free
	set node comp3.pccluster.org properties = any
	set node comp3.pccluster.org properties += score
	set node comp3.pccluster.org properties += score-pcc
	set node comp3.pccluster.org ntype = cluster

Creation of default queue

Create a default queue. In the following example, the created queue is named as "default." 

	# qmgr -c "create queue default queue_type=execution"
	# qmgr -c "set queue default enabled=true"
	# qmgr -c "set queue default started=true"
	# qmgr -c "set server default_queue=default"

For more detail on the config file, please refer to Section 3.5.2. Queue Configuration of PBS Administrator Guide

Start of scheduling

Finally, activate PBS scheduling queue. 

	# qmgr -c "set server scheduling=true"

If PBS/SCore is installed from a binary package, then C based scheduler is implemented. For more detail on the C based scheduler, please refer to Section 9.9. C Based Scheduler of PBS Administrator Guide

9. The difference between PBS and PBS/SCore

score property

In PBS/SCore, the property name, score is treated as a special name to distinguish not-SCore jobs and SCore jobs.

Host to invoke SCore jobs

Any SCore jobs must be invoked via the scout and scrun programs. The scout program creates a parallel execution environment in a cluster, and the scrun program invokes user parallel progam in the parallel environment. The scout program can be invoked outside of a cluster.

In contrast, parallel program invokation scheme of PBS is different. PBS assumed user parallel program is firstly invoked on a compute host. And then the program itself spawns remote processes on the other compute hosts.

"-m" option is added to pbs_server to specify the host to invoke SCore jobs. As described above, pbs_mon daemon process must be running on this host. However, this host cannot be allocated to a designated compute host. By default, the host is allocated dynamically one of the compute hosts. 

   pbs_server -m <hostname>[:<portno>]
PCCC logo PC Cluster Consotium
CREDIT
This document is a part of the SCore cluster system software developed at PC Cluster Consortium, Japan. Copyright (C) 2002 PC Cluster Consortium.