[SCore-users-jp] Re: [SCore-users] Slow communication

[Shinji Sumimoto]

Hi.

From: JJ <jj at guest.xenya.si>
Subject: [SCore-users-jp] Re: [SCore-users] Slow communication
Date: Mon, 12 Aug 2002 12:35:01 +0200 (CEST)
Message-ID: <1029148501.3d578f55924fe at webmail.stimm-mz.si>

jj> I decreased the backoff to 100 and maxnsend to 16 and
jj> we are slowly getting better.

I think backoff 100 is too small to use on MPI application.

Maybe, you have to check some other parameters of device driver,
especially number of TX(RX) descriptors.

If I can see the source code of the driver, it is easy to check.
But it depends on the code license.

PS: Could you let us know your hardware environment?
    We are now collecting hardware environments on which SCore is able to run.

Shinji.

jj> Latencies are still 25 % worse for small packets on tc902
jj> than on 83820.
jj> 
jj> rtt:
jj> 
jj> size	83820		tc902x
jj> 4 	3.59e-05	5.21e-05 
jj> 
jj> 
jj> bw2
jj> 4	222059		155000 
jj> 
jj> but tc902x is getting better already at the 3072 bytes packets.
jj> 
jj> Do you believe I can still squize a bit more with some 
jj> tunning? 
jj> 
jj> I will check the driver for the coalescing function but I don not believe
jj> that there is something like that build in.
jj> 
jj> Jure
jj> 
jj> Quoting Shinji Sumimoto <s-sumi at flab.fujitsu.co.jp>:
jj> 
jj> > Hi.
jj> > 
jj> > From: JJ <jj at guest.xenya.si>
jj> > Subject: Re: [SCore-users] Slow communication
jj> > Date: Mon, 12 Aug 2002 11:56:05 +0200 (CEST)
jj> > Message-ID: <1029146165.3d5786357ff13 at webmail.stimm-mz.si>
jj> > 
jj> > jj> Shinji,
jj> > jj> 
jj> > jj> thanks a lot for your prompt reply. I changed the  value of 
jj> > jj> maxnsend to 16 and value of backoff to 2400 and everything works
jj> > jj> much better now. However, the performance for small packets is 
jj> > jj> still far from the expected one.
jj> > jj> 
jj> > jj> For example, here is a part of the table for mpi_bw2:
jj> > jj> 
jj> > jj> size (Bytes) Linksys (83820) Tamarack (tc902x)
jj> > jj> 4  222059  39286
jj> > jj> 48  2.527e+06 472280
jj> > jj> 128  6.37e+06 1.28e+06
jj> > jj> ...
jj> > jj> 1024  2.80e+07 1.61e+07
jj> > jj> ...
jj> > jj> 8192  6.94e+07 4.22e+07
jj> > jj> ...
jj> > jj> 98304  9.22e+07 8.76e+07
jj> > jj> ...
jj> > jj> 524288  1.03e+08 1.11e+08
jj> > jj> 
jj> > jj> The performance of tc902x is better for huge messages, as you can
jj> > see.
jj> > jj> I will read the suggested manual, but do you have any quick hint
jj> > how
jj> > jj> to get even closer to desired performance?
jj> > 
jj> > How about mpi_rtt and mpi_bw results?
jj> > 
jj> > Many Gigabit Ethernet NIC supports interrupt coalescing function which
jj> > increase huge latencies. The round trip latency is very importrant
jj> > because the mpi_bw2 is ping-pong banad-width performance.
jj> > 
jj> > Some NICs have parameters to control the interrupt coalescing
jj> > function, ex intel e1000, bcm5700.
jj> > 
jj> > Shinji.
jj> > 
jj> > jj> The driver was added as a source to Longshine gigabit card and
jj> > jj> it was written by Craig Rich, craig_rich at sundanceti.com. There
jj> > jj> is nothing written about the license.
jj> > jj> 
jj> > jj> Jure
jj> > jj> 
jj> > jj> uoting Shinji Sumimoto <s-sumi at flab.fujitsu.co.jp>:
jj> > jj> 
jj> > jj> > Hi.
jj> > jj> > 
jj> > jj> > Have you tuned some paramters in pm-ethernet.conf?
jj> > jj> > 
jj> > jj> > Important parameters are maxnsend and backoff.
jj> > jj> > If you do not try, could you try it?
jj> > jj> > 
jj> > jj> > The parameters depend on the NIC hardware.
jj> > jj> > The descriptions of the parameters are:
jj> > jj> > 
jj> > jj> >
jj> > http://www.pccluster.org/score/dist/score/html/en/man/man5/pm-ether-conf.html
jj> > jj> > 
jj> > jj> > May be following parameter is that you try first.
jj> > jj> > 
jj> > jj> > maxnsend 16
jj> > jj> > backoff 2400
jj> > jj> > 
jj> > jj> > PS: I have not seen the linux driver for Tamarack 9021.
jj> > jj> >     Where have you gotten the driver?
jj> > jj> > 
jj> > jj> > Shinji.
jj> > jj> > 
jj> > jj> > From: Jure <jj at guest.xenya.si>
jj> > jj> > Subject: [SCore-users] Slow communication
jj> > jj> > Date: Mon, 12 Aug 2002 10:53:41 +0200 (CEST)
jj> > jj> > Message-ID: <1029142421.3d57779548a5c at webmail.stimm-mz.si>
jj> > jj> > 
jj> > jj> > jj> Hello,
jj> > jj> > jj> 
jj> > jj> > jj> I am experimenting with score on two machines. For
jj> > communication  I
jj> > jj> > first used 
jj> > jj> > jj> Linksys gigabit coper cards (nc83820) and everything worked
jj> > well.
jj> > jj> > Then I 
jj> > jj> > jj> replaced Linksys cards with longshine fiber gigabit cards. 
jj> > jj> > jj> 
jj> > jj> > jj> I changed mac addresses in /opt/score/etc/pm-ethernet.conf
jj> > jj> > jj> and in /opt/score/etc/ndconf/0 and /opt/score/etc/ndconf/1.
jj> > jj> > jj> 
jj> > jj> > jj> First I tested the regular mpi communication over tcp and it
jj> > works
jj> > jj> > jj> much better on longshine cards than on linksys.
jj> > jj> > jj> 
jj> > jj> > jj> However, when I want to run mpitest (rtt, bw, bw2) over pm,
jj> > the
jj> > jj> > jj> communication is very, very slow.
jj> > jj> > jj> 
jj> > jj> > jj> Is there anything I can do? Did I perhaps forget something in
jj> > score
jj> > jj> > 
jj> > jj> > jj> configuration?
jj> > jj> > jj> 
jj> > jj> > jj> The chipset on longshine card is Tamarack 9021.
jj> > jj> > jj> 
jj> > jj> > jj> I would very much appreciate any hint.
jj> > jj> > jj> 
jj> > jj> > jj> Jure
jj> > jj> > jj> 
jj> > jj> > jj> 
jj> > jj> > jj> 
jj> > jj> > jj> _______________________________________________
jj> > jj> > jj> SCore-users mailing list
jj> > jj> > jj> SCore-users at pccluster.org
jj> > jj> > jj> http://www.pccluster.org/mailman/listinfo/score-users
jj> > jj> > jj> 
jj> > jj> > jj> 
jj> > jj> > ------
jj> > jj> > Shinji Sumimoto, Fujitsu Labs
jj> > jj> > 
jj> > jj> > 
jj> > jj> _______________________________________________
jj> > jj> SCore-users mailing list
jj> > jj> SCore-users at pccluster.org
jj> > jj> http://www.pccluster.org/mailman/listinfo/score-users
jj> > jj> 
jj> > ------
jj> > Shinji Sumimoto, Fujitsu Labs
jj> > 
jj> > 
jj> _______________________________________________
jj> SCore-users mailing list
jj> SCore-users at pccluster.org
jj> http://www.pccluster.org/mailman/listinfo/score-users
jj> _______________________________________________
jj> SCore-users-jp mailing list
jj> SCore-users-jp at pccluster.org
jj> http://www.pccluster.org/mailman/listinfo/score-users-jp
jj> 
jj> 
------
Shinji Sumimoto, Fujitsu Labs