[SCore-users-jp] [SCore-users] Network trunking problems
Shinji Sumimoto
s-sumi ¡÷ flab.fujitsu.co.jp
2004ǯ 10·î 12Æü (²Ð) 21:22:57 JST
Dear Nick.
How about the following parameter?
=======================
maxnsend 16
backoff 800
=======================
Here are some results on Intel Dual Giga-E card and SuperMicro MB
using MTU 1.5K, but the results are not good. I will investigate the
problem.
Shinji.
*** Dual Giga-E ****
==============================================
SCore-D 5.8.1 connected.
#---------------------------------------------------
# PALLAS MPI Benchmark Suite V2.2, MPI-1 part
#---------------------------------------------------
# Date : Tue Oct 12 20:55:46 2004
# Machine : i686# System : Linux
# Release : 2.4.21score5.8
# Version : #2 SMP Mon Sep 6 13:47:39 JST 2004
#
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# PingPong
# Sendrecv
#---------------------------------------------------
# Benchmarking PingPong
# ( #processes = 2 )
# ( 2 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 15.35 0.00
1 1000 15.50 0.06
2 1000 15.49 0.12
4 1000 15.72 0.24
8 1000 15.89 0.48
16 1000 15.93 0.96
32 1000 16.35 1.87
64 1000 17.67 3.45
128 1000 19.67 6.20
256 1000 23.33 10.46
512 1000 33.99 14.36
1024 1000 45.98 21.24
2048 1000 65.63 29.76
4096 1000 80.81 48.34
8192 1000 103.38 75.57
16384 1000 177.73 87.92
32768 1000 278.15 112.35
65536 1000 450.74 138.66
131072 1000 830.70 150.48
262144 640 1578.08 158.42
524288 320 3037.10 164.63
1048576 160 6054.12 165.18
2097152 80 11752.26 170.18
4194304 40 27332.49 146.35
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# ( #processes = 2 )
# ( 2 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 15.96 15.97 15.97 0.00
1 1000 16.14 16.15 16.14 0.12
2 1000 16.23 16.23 16.23 0.23
4 1000 16.46 16.47 16.46 0.46
8 1000 16.64 16.64 16.64 0.92
16 1000 16.61 16.61 16.61 1.84
32 1000 16.99 17.00 16.99 3.59
64 1000 18.48 18.48 18.48 6.60
128 1000 20.40 20.42 20.41 11.96
256 1000 24.24 24.25 24.24 20.13
512 1000 31.91 31.94 31.93 30.58
1024 1000 46.10 46.11 46.11 42.36
2048 1000 112.50 112.52 112.51 34.71
4096 1000 476.83 478.01 477.42 16.34
8192 1000 495.49 495.52 495.50 31.53
16384 1000 2214.85 2216.06 2215.45 14.10
32768 1000 2423.52 2425.49 2424.51 25.77
65536 1000 2701.98 2703.39 2702.69 46.24
131072 1000 3396.19 3396.76 3396.48 73.60
262144 640 4823.91 4824.50 4824.20 103.64
524288 320 7815.02 7816.31 7815.66 127.94
1048576 160 13664.31 13667.36 13665.83 146.33
2097152 80 25498.25 25505.10 25501.68 156.83
4194304 40 54413.87 54424.60 54419.23 146.99
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# ( #processes = 4 )
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 16.98 16.99 16.99 0.00
1 1000 17.26 17.27 17.26 0.11
2 1000 17.30 17.30 17.30 0.22
4 1000 17.45 17.46 17.46 0.44
8 1000 17.50 17.52 17.51 0.87
16 1000 17.49 17.50 17.49 1.74
32 1000 18.00 18.02 18.01 3.39
64 1000 19.44 19.45 19.44 6.28
128 1000 21.30 21.32 21.31 11.45
256 1000 25.91 25.94 25.92 18.82
512 1000 32.59 32.62 32.60 29.94
1024 1000 47.36 47.40 47.38 41.20
2048 1000 68.89 68.94 68.92 56.66
4096 1000 82.46 82.48 82.47 94.72
8192 1000 111.64 111.71 111.68 139.87
16384 1000 238.85 238.91 238.89 130.80
32768 1000 447.44 447.49 447.46 139.67
65536 1000 800.36 800.49 800.42 156.16
131072 1000 1525.62 1526.29 1525.93 163.80
262144 640 2985.35 2987.49 2986.53 167.36
524288 320 5966.47 5975.08 5971.48 167.36
1048576 160 11971.57 11994.16 11984.03 166.75
2097152 80 24619.02 24748.15 24685.33 161.63
4194304 40 50143.33 50580.92 50363.74 158.16
#=====================================================
#
# Thanks for using PMB2.2
#
# The Pallas team kindly requests that you
# give us as much feedback for PMB as possible.
#
# It would be very helpful when you sent the
# output tables of your run(s) of PMB to
#
# #######################
# # #
# # pmb ¡÷ pallas.com #
# # #
# #######################
#
# You might also add
#
# - personal information (institution, motivation
# for using PMB)
# - basic information about the machine you used
# (number of CPUs, processor type e.t.c.)
#
#=====================================================
*** Single Giga-E ****
=====================================================================
SCore-D 5.8.1 connected.
#---------------------------------------------------
# PALLAS MPI Benchmark Suite V2.2, MPI-1 part
#---------------------------------------------------
# Date : Tue Oct 12 20:56:53 2004
# Machine : i686# System : Linux
# Release : 2.4.21score5.8
# Version : #2 SMP Mon Sep 6 13:47:39 JST 2004
#
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# PingPong
# Sendrecv
#---------------------------------------------------
# Benchmarking PingPong
# ( #processes = 2 )
# ( 2 additional processes waiting in MPI_Barrier)
#---------------------------------------------------
#bytes #repetitions t[usec] Mbytes/sec
0 1000 18.47 0.00
1 1000 18.47 0.05
2 1000 18.39 0.10
4 1000 18.68 0.20
8 1000 18.74 0.41
16 1000 18.90 0.81
32 1000 19.38 1.57
64 1000 20.89 2.92
128 1000 22.81 5.35
256 1000 26.85 9.09
512 1000 34.64 14.09
1024 1000 49.94 19.55
2048 1000 67.83 28.79
4096 1000 86.55 45.13
8192 1000 119.63 65.31
16384 1000 223.56 69.89
32768 1000 362.53 86.20
65536 1000 652.64 95.77
131072 1000 1215.95 102.80
262144 640 2317.14 107.89
524288 320 4610.97 108.44
1048576 160 8991.56 111.22
2097152 80 17884.11 111.83
4194304 40 35684.34 112.09
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# ( #processes = 2 )
# ( 2 additional processes waiting in MPI_Barrier)
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 19.32 19.33 19.33 0.00
1 1000 19.43 19.44 19.43 0.10
2 1000 19.36 19.36 19.36 0.20
4 1000 19.76 19.77 19.77 0.39
8 1000 19.80 19.81 19.81 0.77
16 1000 19.62 19.63 19.62 1.55
32 1000 20.18 20.19 20.18 3.02
64 1000 21.95 21.95 21.95 5.56
128 1000 23.74 23.74 23.74 10.28
256 1000 27.83 27.85 27.84 17.54
512 1000 36.06 36.08 36.07 27.07
1024 1000 51.19 51.23 51.21 38.13
2048 1000 74.77 74.82 74.79 52.21
4096 1000 95.45 95.50 95.48 81.80
8192 1000 127.14 127.18 127.16 122.86
16384 1000 228.08 228.10 228.09 137.00
32768 1000 366.22 366.22 366.22 170.66
65536 1000 647.03 647.06 647.05 193.18
131072 1000 1244.26 1244.28 1244.27 200.92
262144 640 2488.19 2488.23 2488.21 200.95
524288 320 4987.30 4987.57 4987.43 200.50
1048576 160 9946.71 9946.96 9946.83 201.07
2097152 80 20048.08 20048.67 20048.38 199.51
4194304 40 42708.02 42728.23 42718.12 187.23
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# ( #processes = 4 )
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 19.15 19.17 19.16 0.00
1 1000 19.53 19.55 19.54 0.10
2 1000 19.46 19.49 19.47 0.20
4 1000 19.52 19.55 19.54 0.39
8 1000 19.54 19.56 19.55 0.78
16 1000 19.66 19.68 19.67 1.55
32 1000 20.03 20.05 20.04 3.04
64 1000 21.53 21.56 21.54 5.66
128 1000 23.49 23.56 23.53 10.36
256 1000 27.59 27.62 27.60 17.68
512 1000 35.52 35.55 35.54 27.47
1024 1000 54.62 54.69 54.65 35.72
2048 1000 75.10 75.17 75.13 51.96
4096 1000 92.74 92.82 92.79 84.17
8192 1000 127.26 127.38 127.32 122.66
16384 1000 293.29 293.49 293.40 106.48
32768 1000 504.30 504.68 504.47 123.84
65536 1000 867.29 867.74 867.56 144.05
131072 1000 1521.44 1522.03 1521.82 164.25
262144 640 3161.80 3164.15 3163.17 158.02
524288 320 6549.42 6560.77 6555.03 152.42
1048576 160 13667.64 13712.74 13690.57 145.85
2097152 80 27897.05 28024.57 27959.47 142.73
4194304 40 57200.03 57689.10 57468.21 138.67
#=====================================================
#
# Thanks for using PMB2.2
#
# The Pallas team kindly requests that you
# give us as much feedback for PMB as possible.
#
# It would be very helpful when you sent the
# output tables of your run(s) of PMB to
#
# #######################
# # #
# # pmb ¡÷ pallas.com #
# # #
# #######################
#
# You might also add
#
# - personal information (institution, motivation
# for using PMB)
# - basic information about the machine you used
# (number of CPUs, processor type e.t.c.)
#
#=====================================================
From: Nick Birkett <nick ¡÷ streamline-computing.com>
Subject: [SCore-users-jp] [SCore-users] Network trunking problems
Date: 12 Oct 2004 09:59:51 +0100
Message-ID: <1097571591.15000.120.camel ¡÷ zeralda.streamline>
nick> We need some help on this.
nick>
nick> I am getting the same problems as I got when I tried this before.
nick>
nick> Each single network performs extremely well.
nick>
nick> But using PMB on 2x gigabit, the Sendrecv is very poor and uneven
nick> compared to single gigabit.
nick>
nick> Hardware set up: Smicro dual Xeon front end,
nick> 8x Smicro dual Xeon 3.06GHz cpus comp nodes,
nick> 4096 Mbytes
nick> 2x inboard e1000
nick> 1x offboard e1000 workstation card in PCI-X slot
nick>
nick> 3 x Nortel baystack Gbit switches (not linked)
nick> 3 independent networks eth0,eth1,eth2
nick>
nick> System network is eth1 - connects Front end and compute nodes.
nick> Message networks eth0,eth2
nick>
nick> eth1,eth2 or the onboard gbit
nick> eth0 is offboard gbit.
nick>
nick> Software: RedHat 9, Score 5.8.1 (compiled from source received end of
nick> Sept 2004). Kernel 2.4.21-2SCORE smp , Intel e1000 driver, v 5.2.39 .
nick>
nick> I have also tried it using Score 5.6.1 compiled for RedHat 9
nick> and with Score 5.4.1 on RedHat 7.3,
nick>
nick> I can run jobs on all 3 networks and dual network:
nick>
nick> scout -F hosts -e scrun -nodes=2x1,network=ethernet0 ./PMB-MPI1 Pingpong
nick> Sendrecv
nick>
nick> works fine - see ethernet0.out.
nick>
nick> scout -F hosts -e scrun -nodes=2x1,network=ethernet1 ./PMB-MPI1 Pingpong
nick> Sendrecv
nick>
nick> works fine
nick>
nick> scout -F hosts -e scrun -nodes=2x1,network=ethernet2 ./PMB-MPI1 Pingpong
nick> Sendrecv
nick>
nick> works fine - see ethernet2.out
nick>
nick> scout -F hosts -e scrun -nodes=2x1,network=ethernetx2 ./PMB-MPI1
nick> Pingpong Sendrecv
nick>
nick> problems !!
nick>
nick> Attached files:
nick>
nick> Score configuration files:
nick> score-etc-5.8.1.tgz
nick>
nick> Output for PMB for ethernet0,ethernet2 and trunked ethernetx2:
nick> ethernet0.out
nick> ethernet2.out
nick> ethernetx2.out
nick>
nick> Info files from cat /proc/pm :
nick>
nick> eth0.info
nick> eth2.info
nick>
nick> I have tried the following parameters: maxnsend 16,32
nick> backoff 1024,2048 4096
nick>
nick> I have also tried with Jumbo frames set on eth0, eth2 network
nick> (MTU=9000). The attached results are with Jumbo frames off.
nick>
nick> I have also tested with interrupt reaping off.
nick>
nick> This was much worse than Intreap 1 .
nick>
nick>
nick> Any ideas ?
nick>
nick> Best wishes,
nick>
nick> Nick
nick>
nick>
nick> --
nick>
nick>
nick> ----
nick>
nick>
nick> PM Ethernet 0330
nick> Version : "$Id: pm_ethernet_dev.c,v 1.5 2004/04/22 04:30:17 kameyama Exp $"
nick> Device : eth0
nick> Irq : 0
nick> MTU : 1468 Bytes
nick> Maxnodes : 512
nick> MaxContext : 16
nick> Intreap : on
nick> Checksum : on
nick> Dev error : 3
nick> Header err : 0
nick> Header cksum err: 0
nick> Data cksum err : 0
nick> Underrun err : 0
nick> ----
nick>
nick>
nick> PM Ethernet 0330
nick> Version : "$Id: pm_ethernet_dev.c,v 1.5 2004/04/22 04:30:17 kameyama Exp $"
nick> Device : eth2
nick> Irq : 0
nick> MTU : 1468 Bytes
nick> Maxnodes : 512
nick> MaxContext : 16
nick> Intreap : on
nick> Checksum : on
nick> Dev error : 1
nick> Header err : 0
nick> Header cksum err: 0
nick> Data cksum err : 0
nick> Underrun err : 0
nick> --
nick> Nick Birkett
nick> Streamline Computing Ltd
nick>
------
Shinji Sumimoto, Fujitsu Labs
_______________________________________________
SCore-users mailing list
SCore-users ¡÷ pccluster.org
http://www.pccluster.org/mailman/listinfo/score-users
SCore-users-jp ¥á¡¼¥ê¥ó¥°¥ê¥¹¥È¤Î°ÆÆâ