About PM/Infiniband for Topspin InfinBand fabric

The current PM/Infiniband TS is a PM/Infiniband communication library for TopSpin InfiniBand Fabric.

The PM/Infiniband TS is implemented using TopSpin OS. The user library of the PM/InfiniBand is released, only. Its library requeires the TopSpin InfiniBand driver, TopSpin OS 2.0.0 build 436, and the PM/InfiniBand TS driver module. You must get its load moduls from InfiniBand vendor to use the PM/InfiniBand TS.

SCore installation on the server host and compute hosts

First of all, please install the SCore system for Ethernet using the EIT tool or by hand.

PM/InfiniBand TS installation on all compute hosts

All compute hosts must be set up as follows:
  1. Install the TopSpin OS
    Install the TopSpin OS rpm packages as follow:
            # rpm -i topspin-ib-rh80-2.0.0-436.i686.rpm
            # rpm -i topspin-ib-mod-rhXX-2.4.XX-2.0.0-436.i686.rpm
    The rpm package, topspin-ib-mod-XXXX-YYYY-2.0.0-436.i686.rpm, depends on the kernel version and the distribution on compute hosts.

  2. Install the PM/InfiniBand TS driver
    Copy the PM/InfiniBand TS driver as follow:
            # cp pm_ibats.o /lib/modules/`uname -r`/ib/pm_ibats.o
  3. Setup the IP address for IPoIB
    Create a configration files, /etc/sysconfig/network-scripts/ifcfg-ib0, as follow:
    The network parameter depends the each compute hosts.

  4. reboot the compute host.

SCore Set up on the server host

Setup the configration files for SCore.
  1. Setup the PM/InfiniBand TS
    Create the PM/InfiniBand TS confiration file as follow:
    	# ( count=0; for hosts in `scorehosts -g pcc`; do \
    	      echo -n "$count	$hosts	"; \
    	      rsh $hosts /sbin/ifconfig ib0 | grep inet | sed 's/.*inet addr:\([0-9.]*\).*/\1/' ; \
    	      count=`expr $count + 1`; \
    	    done \
              ) > /opt/score/etc/pm-infiniband.conf
  2. Configure the scoreaboard database
    Create the /opt/score/etc/scoreboard.db file as follows: The following entries are an example for PM/Infiniband TS where the server host name is server.pccluster.org and the compute host names are comp0.pccluster.org -- comp3.pccluster.org.
    	/* PM/InfiniBand TS */
    	infiniband	type=ib-ts -config:file=/opt/score/etc/pm-infiniband.conf
    	/* PM/Shmem */		
    	shmem0		type=shmem -node=0
    	shmem1		type=shmem -node=1
    	#define PCC     MSGBSERV(server.pccluster.org:8766) \
    	                cpugen=pentium-4 speed=3000 smp=2 \
    	                network=infiniband,shmem0,shmem1 group=pcc
    	comp0.pccluster.org	PCC
    	comp1.pccluster.org	PCC
    	comp2.pccluster.org	PCC
    	comp3.pccluster.org	PCC
  3. reboot the server host.

PCCC logo PC Cluster Consotium

$Id: pmibts.html,v 1.1 2004/08/05 04:50:28 a-hasega Exp $