[SCore-users] Slow communication

[JJ]

Hi,

I decreased the backoff to 100 and maxnsend to 16 and
we are slowly getting better.

Latencies are still 25 % worse for small packets on tc902
than on 83820.

rtt:

size	83820		tc902x
4 	3.59e-05	5.21e-05 


bw2
4	222059		155000 

but tc902x is getting better already at the 3072 bytes packets.

Do you believe I can still squize a bit more with some 
tunning? 

I will check the driver for the coalescing function but I don not believe
that there is something like that build in.

Jure

Quoting Shinji Sumimoto <s-sumi at flab.fujitsu.co.jp>:

> Hi.
> 
> From: JJ <jj at guest.xenya.si>
> Subject: Re: [SCore-users] Slow communication
> Date: Mon, 12 Aug 2002 11:56:05 +0200 (CEST)
> Message-ID: <1029146165.3d5786357ff13 at webmail.stimm-mz.si>
> 
> jj> Shinji,
> jj> 
> jj> thanks a lot for your prompt reply. I changed the  value of 
> jj> maxnsend to 16 and value of backoff to 2400 and everything works
> jj> much better now. However, the performance for small packets is 
> jj> still far from the expected one.
> jj> 
> jj> For example, here is a part of the table for mpi_bw2:
> jj> 
> jj> size (Bytes) Linksys (83820) Tamarack (tc902x)
> jj> 4  222059  39286
> jj> 48  2.527e+06 472280
> jj> 128  6.37e+06 1.28e+06
> jj> ...
> jj> 1024  2.80e+07 1.61e+07
> jj> ...
> jj> 8192  6.94e+07 4.22e+07
> jj> ...
> jj> 98304  9.22e+07 8.76e+07
> jj> ...
> jj> 524288  1.03e+08 1.11e+08
> jj> 
> jj> The performance of tc902x is better for huge messages, as you can
> see.
> jj> I will read the suggested manual, but do you have any quick hint
> how
> jj> to get even closer to desired performance?
> 
> How about mpi_rtt and mpi_bw results?
> 
> Many Gigabit Ethernet NIC supports interrupt coalescing function which
> increase huge latencies. The round trip latency is very importrant
> because the mpi_bw2 is ping-pong banad-width performance.
> 
> Some NICs have parameters to control the interrupt coalescing
> function, ex intel e1000, bcm5700.
> 
> Shinji.
> 
> jj> The driver was added as a source to Longshine gigabit card and
> jj> it was written by Craig Rich, craig_rich at sundanceti.com. There
> jj> is nothing written about the license.
> jj> 
> jj> Jure
> jj> 
> jj> uoting Shinji Sumimoto <s-sumi at flab.fujitsu.co.jp>:
> jj> 
> jj> > Hi.
> jj> > 
> jj> > Have you tuned some paramters in pm-ethernet.conf?
> jj> > 
> jj> > Important parameters are maxnsend and backoff.
> jj> > If you do not try, could you try it?
> jj> > 
> jj> > The parameters depend on the NIC hardware.
> jj> > The descriptions of the parameters are:
> jj> > 
> jj> >
> http://www.pccluster.org/score/dist/score/html/en/man/man5/pm-ether-conf.html
> jj> > 
> jj> > May be following parameter is that you try first.
> jj> > 
> jj> > maxnsend 16
> jj> > backoff 2400
> jj> > 
> jj> > PS: I have not seen the linux driver for Tamarack 9021.
> jj> >     Where have you gotten the driver?
> jj> > 
> jj> > Shinji.
> jj> > 
> jj> > From: Jure <jj at guest.xenya.si>
> jj> > Subject: [SCore-users] Slow communication
> jj> > Date: Mon, 12 Aug 2002 10:53:41 +0200 (CEST)
> jj> > Message-ID: <1029142421.3d57779548a5c at webmail.stimm-mz.si>
> jj> > 
> jj> > jj> Hello,
> jj> > jj> 
> jj> > jj> I am experimenting with score on two machines. For
> communication  I
> jj> > first used 
> jj> > jj> Linksys gigabit coper cards (nc83820) and everything worked
> well.
> jj> > Then I 
> jj> > jj> replaced Linksys cards with longshine fiber gigabit cards. 
> jj> > jj> 
> jj> > jj> I changed mac addresses in /opt/score/etc/pm-ethernet.conf
> jj> > jj> and in /opt/score/etc/ndconf/0 and /opt/score/etc/ndconf/1.
> jj> > jj> 
> jj> > jj> First I tested the regular mpi communication over tcp and it
> works
> jj> > jj> much better on longshine cards than on linksys.
> jj> > jj> 
> jj> > jj> However, when I want to run mpitest (rtt, bw, bw2) over pm,
> the
> jj> > jj> communication is very, very slow.
> jj> > jj> 
> jj> > jj> Is there anything I can do? Did I perhaps forget something in
> score
> jj> > 
> jj> > jj> configuration?
> jj> > jj> 
> jj> > jj> The chipset on longshine card is Tamarack 9021.
> jj> > jj> 
> jj> > jj> I would very much appreciate any hint.
> jj> > jj> 
> jj> > jj> Jure
> jj> > jj> 
> jj> > jj> 
> jj> > jj> 
> jj> > jj> _______________________________________________
> jj> > jj> SCore-users mailing list
> jj> > jj> SCore-users at pccluster.org
> jj> > jj> http://www.pccluster.org/mailman/listinfo/score-users
> jj> > jj> 
> jj> > jj> 
> jj> > ------
> jj> > Shinji Sumimoto, Fujitsu Labs
> jj> > 
> jj> > 
> jj> _______________________________________________
> jj> SCore-users mailing list
> jj> SCore-users at pccluster.org
> jj> http://www.pccluster.org/mailman/listinfo/score-users
> jj> 
> ------
> Shinji Sumimoto, Fujitsu Labs
> 
>