[SCore-users] Slow communication

[Shinji Sumimoto]

Hi.

From: JJ <jj at guest.xenya.si>
Subject: Re: [SCore-users] Slow communication
Date: Mon, 12 Aug 2002 11:56:05 +0200 (CEST)
Message-ID: <1029146165.3d5786357ff13 at webmail.stimm-mz.si>

jj> Shinji,
jj> 
jj> thanks a lot for your prompt reply. I changed the  value of 
jj> maxnsend to 16 and value of backoff to 2400 and everything works
jj> much better now. However, the performance for small packets is 
jj> still far from the expected one.
jj> 
jj> For example, here is a part of the table for mpi_bw2:
jj> 
jj> size (Bytes) Linksys (83820) Tamarack (tc902x)
jj> 4		222059		39286
jj> 48		2.527e+06	472280
jj> 128		6.37e+06	1.28e+06
jj> ...
jj> 1024		2.80e+07	1.61e+07
jj> ...
jj> 8192		6.94e+07	4.22e+07
jj> ...
jj> 98304		9.22e+07	8.76e+07
jj> ...
jj> 524288		1.03e+08	1.11e+08
jj> 
jj> The performance of tc902x is better for huge messages, as you can see.
jj> I will read the suggested manual, but do you have any quick hint how
jj> to get even closer to desired performance?

How about mpi_rtt and mpi_bw results?

Many Gigabit Ethernet NIC supports interrupt coalescing function which
increase huge latencies. The round trip latency is very importrant
because the mpi_bw2 is ping-pong banad-width performance.

Some NICs have parameters to control the interrupt coalescing
function, ex intel e1000, bcm5700.

Shinji.

jj> The driver was added as a source to Longshine gigabit card and
jj> it was written by Craig Rich, craig_rich at sundanceti.com. There
jj> is nothing written about the license.
jj> 
jj> Jure
jj> 
jj> uoting Shinji Sumimoto <s-sumi at flab.fujitsu.co.jp>:
jj> 
jj> > Hi.
jj> > 
jj> > Have you tuned some paramters in pm-ethernet.conf?
jj> > 
jj> > Important parameters are maxnsend and backoff.
jj> > If you do not try, could you try it?
jj> > 
jj> > The parameters depend on the NIC hardware.
jj> > The descriptions of the parameters are:
jj> > 
jj> > http://www.pccluster.org/score/dist/score/html/en/man/man5/pm-ether-conf.html
jj> > 
jj> > May be following parameter is that you try first.
jj> > 
jj> > maxnsend 16
jj> > backoff 2400
jj> > 
jj> > PS: I have not seen the linux driver for Tamarack 9021.
jj> >     Where have you gotten the driver?
jj> > 
jj> > Shinji.
jj> > 
jj> > From: Jure <jj at guest.xenya.si>
jj> > Subject: [SCore-users] Slow communication
jj> > Date: Mon, 12 Aug 2002 10:53:41 +0200 (CEST)
jj> > Message-ID: <1029142421.3d57779548a5c at webmail.stimm-mz.si>
jj> > 
jj> > jj> Hello,
jj> > jj> 
jj> > jj> I am experimenting with score on two machines. For communication  I
jj> > first used 
jj> > jj> Linksys gigabit coper cards (nc83820) and everything worked well.
jj> > Then I 
jj> > jj> replaced Linksys cards with longshine fiber gigabit cards. 
jj> > jj> 
jj> > jj> I changed mac addresses in /opt/score/etc/pm-ethernet.conf
jj> > jj> and in /opt/score/etc/ndconf/0 and /opt/score/etc/ndconf/1.
jj> > jj> 
jj> > jj> First I tested the regular mpi communication over tcp and it works
jj> > jj> much better on longshine cards than on linksys.
jj> > jj> 
jj> > jj> However, when I want to run mpitest (rtt, bw, bw2) over pm, the
jj> > jj> communication is very, very slow.
jj> > jj> 
jj> > jj> Is there anything I can do? Did I perhaps forget something in score
jj> > 
jj> > jj> configuration?
jj> > jj> 
jj> > jj> The chipset on longshine card is Tamarack 9021.
jj> > jj> 
jj> > jj> I would very much appreciate any hint.
jj> > jj> 
jj> > jj> Jure
jj> > jj> 
jj> > jj> 
jj> > jj> 
jj> > jj> _______________________________________________
jj> > jj> SCore-users mailing list
jj> > jj> SCore-users at pccluster.org
jj> > jj> http://www.pccluster.org/mailman/listinfo/score-users
jj> > jj> 
jj> > jj> 
jj> > ------
jj> > Shinji Sumimoto, Fujitsu Labs
jj> > 
jj> > 
jj> _______________________________________________
jj> SCore-users mailing list
jj> SCore-users at pccluster.org
jj> http://www.pccluster.org/mailman/listinfo/score-users
jj> 
------
Shinji Sumimoto, Fujitsu Labs