About PM/Infiniband for Topspin InfinBand fabric

The current PM/Infiniband TS is a PM/Infiniband communication library for TopSpin InfiniBand Fabric.

The PM/Infiniband TS is implemented using TopSpin OS. The user library of the PM/InfiniBand is released, only. Its library requeires the TopSpin InfiniBand driver, TopSpin OS 2.0.0 build 436, and the PM/InfiniBand TS driver module. You must get its load moduls from InfiniBand vendor to use the PM/InfiniBand TS.

SCore installation on the server host and compute hosts

First of all, please install the SCore system for Ethernet using the EIT tool or by hand.

PM/InfiniBand TS installation on all compute hosts

All compute hosts must be set up as follows:
  1. Install the TopSpin OS
    Install the TopSpin OS rpm packages as follow: 
            # rpm -i topspin-ib-rh80-2.0.0-436.i686.rpm
        # rpm -i topspin-ib-mod-rhXX-2.4.XX-2.0.0-436.i686.rpm
    The rpm package, topspin-ib-mod-XXXX-YYYY-2.0.0-436.i686.rpm, depends on the kernel version and the distribution on compute hosts.

  2. Install the PM/InfiniBand TS driver
    Copy the PM/InfiniBand TS driver as follow: 
            # cp pm_ibats.o /lib/modules/`uname -r`/ib/pm_ibats.o
  3. Setup the IP address for IPoIB
    Create a configration files, /etc/sysconfig/network-scripts/ifcfg-ib0, as follow: 
            DEVICE=ib0
	ONBOOT=yes
        BOOTPROTO=static
	IPADDR=192.168.0.1
	NETMASK=255.255.255.0
	BROADCAST=192.168.0.255
	NETWORK=192.168.0.0
    The network parameter depends the each compute hosts.

  4. reboot the compute host.

SCore Set up on the server host

Setup the configration files for SCore.
  1. Setup the PM/InfiniBand TS
    Create the PM/InfiniBand TS confiration file as follow: 
    	# ( count=0; for hosts in `scorehosts -g pcc`; do \
	      echo -n "$count	$hosts	"; \
	      rsh $hosts /sbin/ifconfig ib0 | grep inet | sed 's/.*inet addr:\([0-9.]*\).*/\1/' ; \
	      count=`expr $count + 1`; \
	    done \
          ) > /opt/score/etc/pm-infiniband.conf
  2. Configure the scoreaboard database
    Create the /opt/score/etc/scoreboard.db file as follows: The following entries are an example for PM/Infiniband TS where the server host name is server.pccluster.org and the compute host names are comp0.pccluster.org -- comp3.pccluster.org. 
    	/* PM/InfiniBand TS */
	infiniband	type=ib-ts -config:file=/opt/score/etc/pm-infiniband.conf

	/* PM/Shmem */		
	shmem0		type=shmem -node=0
	shmem1		type=shmem -node=1

	#define PCC     MSGBSERV(server.pccluster.org:8766) \
	                cpugen=pentium-4 speed=3000 smp=2 \
	                network=infiniband,shmem0,shmem1 group=pcc

	comp0.pccluster.org	PCC
	comp1.pccluster.org	PCC
	comp2.pccluster.org	PCC
	comp3.pccluster.org	PCC
  3. reboot the server host.
PCCC logo PC Cluster Consotium
CREDIT
This document is a part of the SCore cluster system software developed at PC Cluster Consortium, Japan. Copyright (C) 2003-2004 PC Cluster Consortium.