NetBSD network speed under VMware ESXi

[Joel Carnat]

Hello,

I've just tested running NetBSD and OpenBSD as a VMware ESXi Virtual
Machines.
I expected to see differences between NetBSD and OpenBSD because I read
OpenBSD had vic(4), and understood this was an optimized driver for
VMware.
I previously ran NetBSD on ESX and got poor network rate transfer (10Mb/s).

My actual configuration is somehow weird and probably far from inefficient
(ESXi boots from USB on a laptop and VM disks are accessed from NFS ran by
a Kurobox...) but I still got better transfer rate than I expected.

The NetBSD's network card goes:
pcn0 at pci0 dev 17 function 0: AMD PCnet-PCI Ethernet
pcn0: Am79c970A PCnet-PCI II rev 0, Ethernet address 00:0c:29:0a:24:b3
pcn0: VMware Tx segment count bug detected
pcn0: interrupting at ioapic0 pin 18
pcn0: 10base5, 10base5-FDX, 10baseT, 10baseT-FDX, auto, auto-FDX

And ifconfig says:
pcn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
address: 00:0c:29:0a:24:b3
media: Ethernet autoselect (autoselect)
supported Ethernet media:
media 10base5
media 10base5 mediaopt full-duplex
media 10baseT
media 10baseT mediaopt full-duplex
media autoselect
media autoselect mediaopt full-duplex
inet 10.15.5.28 netmask 0xffffff00 broadcast 10.15.5.255
inet6 fe80::20c:29ff:fe0a:24b3%pcn0 prefixlen 64 scopeid 0x1

Using FTP, the NetBSD VM was able to transfer the netbsd kernel file and
i386cd.iso at 5.39 MiB/s (according to ftp(1)). On the OpenBSD VM, using
the same transfer test, I got 4.65 MB/s.

As far as I understand those two measures, this looks like about half a
100 Mb/s LAN and about 4 times a 10Mb/s LAN. I am wrong expecting a 10
MiB/s transfer rate in a decent configuration ?

Does NetBSD support 100Mb/s as a VMware Virtual Machine ?

TIA,
Jo

[Joerg Sonnenberger]

Post by Joel Carnat
Does NetBSD support 100Mb/s as a VMware Virtual Machine ?

There is a switch for VMware Server at least to get Intel EtherExpress
emulation. You might need to change the .vmi file by hand though.

Post by Joel Carnat
TIA,
Jo

[Joel Carnat]

Post by Joerg Sonnenberger

Post by Joel Carnat
Does NetBSD support 100Mb/s as a VMware Virtual Machine ?

There is a switch for VMware Server at least to get Intel EtherExpress
emulation. You might need to change the .vmi file by hand though.

I added 'Ethernet0.virtualDev = "e1000"' to the .vmx file and the
interface is now known as wm0, Intel i82545EM 1000BASE-T. The FTP transfer
is about the same, but seems a bit less constant - probably an issue with
my Kurobox.

Post by Joerg Sonnenberger
ftp(1) (and any other FTP client) is not an appropriate network
benchmark.
Especially not under VMware because the disk access is slow (at least in
my experiences with my VMware server).
Please use a more appropriate tool like "ttcp" or "netio" which are both
available in "pkgsrc".

Yeah, this was just a quick&dirty look just after a bare install.
Now that those numbers gave me hope, I'll try using correct benchmark tools.


Thanks both.
Regards,
Jo

[Joel Carnat]

Post by Joel Carnat

Post by Matthias Scheler
Especially not under VMware because the disk access is slow (at least in
my experiences with my VMware server).
Please use a more appropriate tool like "ttcp" or "netio" which are both
available in "pkgsrc".

Yeah, this was just a quick&dirty look just after a bare install.
Now that those numbers gave me hope, I'll try using correct benchmark tools.

Here are ttcp numbers:
(sending: ttcp -v -t peer < /tmp/i386cd.iso)
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.15.5.28
ttcp-r: 224348160 bytes in 38.21 real seconds = 5733.84 KB/sec +++
ttcp-r: 40109 I/O calls, msec/call = 0.98, calls/sec = 1049.70
ttcp-r: 0.0user 1.0sys 0:38real 2% 0i+0d 0maxrss 0+2pf 33111+8241csw

(sending: ttcp -v -t peer < /dev/zero during ~40sec)
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 10.15.5.28
ttcp-r: 468470368 bytes in 40.48 real seconds = 11302.92 KB/sec +++
ttcp-r: 71735 I/O calls, msec/call = 0.58, calls/sec = 1772.31
ttcp-r: 0.0user 2.4sys 0:40real 6% 0i+0d 0maxrss 0+2pf 55359+29155csw

And netio numbers:
NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

TCP connection established.
Packet size 1k bytes: 11496 KByte/s Tx, 11079 KByte/s Rx.
Packet size 2k bytes: 11451 KByte/s Tx, 11368 KByte/s Rx.
Packet size 4k bytes: 11494 KByte/s Tx, 11456 KByte/s Rx.
Packet size 8k bytes: 11355 KByte/s Tx, 11390 KByte/s Rx.
Packet size 16k bytes: 11509 KByte/s Tx, 11452 KByte/s Rx.
Packet size 32k bytes: 11499 KByte/s Tx, 11452 KByte/s Rx.
Done.

And finally, trying to understand ttcp difference between sending
/tmp/i386cd.iso and /dev/zero :
# dd if=/dev/zero of=/tmp/TEST count=10m
908497+0 records in
908496+0 records out
465149952 bytes transferred in 62.012 secs (7500966 bytes/sec)

Keep in mind that this / is located on a remote NFS server (for easy
testings).

Should those network numbers assume that NetBSD is ready for VMware
production ?

TIA,
Jo

[Martti Kuparinen]

Virtual machine (on VMware ESXi)
--------------------------------
# uname -srm
NetBSD 4.99.73 amd64

# ifconfig wm0
wm0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

capabilities=2bf80<TSO4,IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx,TCP6CSUM_Tx,UDP6CSUM_Tx>
enabled=0
address: 00:0c:29:ef:64:67
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active


Real machine (on Dell PowerEdge 2900)
-------------------------------------
# uname -srm
NetBSD 4.0_STABLE amd64

# ifconfig bnx0
bnx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500

capabilities=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>

enabled=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
address: 00:18:8b:38:b7:3a
media: Ethernet autoselect (100baseTX full-duplex)
status: active


Scenario 1: TCP between two virtual machines
--------------------------------------------
ttcp-t: 232183808 bytes in 1.23 real seconds = 1444.00 Mbit/sec +++
ttcp-t: 232183808 bytes in 0.91 CPU seconds = 1950.54 Mbit/cpu sec
ttcp-t: 28343 I/O calls, msec/call = 0.04, calls/sec = 23104.17


Scenario 2: UDP between two virtual machines
--------------------------------------------
ttcp-t: 232183808 bytes in 1.31 real seconds = 1351.73 Mbit/sec +++
ttcp-t: 232183808 bytes in 1.03 CPU seconds = 1726.16 Mbit/cpu sec
ttcp-t: 28347 I/O calls, msec/call = 0.05, calls/sec = 21630.87


Scenario 3: TCP between virtual machine and real machine
--------------------------------------------------------
ttcp-r: 232183808 bytes in 19.89 real seconds = 89.08 Mbit/sec +++
ttcp-r: 232183808 bytes in 1.03 CPU seconds = 1722.91 Mbit/cpu sec
ttcp-r: 159331 I/O calls, msec/call = 0.13, calls/sec = 8012.17


Scenario 4: UDP between virtual machine and real machine
--------------------------------------------------------
ttcp-t: 232183808 bytes in 1.75 real seconds = 1012.72 Mbit/sec +++
ttcp-t: 232183808 bytes in 1.71 CPU seconds = 1037.14 Mbit/cpu sec
ttcp-t: 28347 I/O calls, msec/call = 0.06, calls/sec = 16206.01

[Martti Kuparinen]

Joel, your mail address is invalid or you have configuration problem...

<***@carnat.net>: host smtp2.altran.com[194.98.79.76] said: 550 #5.1.0 Address
rejected. (in reply to RCPT TO command)

[Joerg Sonnenberger]

Post by Joel Carnat
I added 'Ethernet0.virtualDev = "e1000"' to the .vmx file and the
interface is now known as wm0, Intel i82545EM 1000BASE-T. The FTP transfer
is about the same, but seems a bit less constant - probably an issue with
my Kurobox.

Also take a look at CPU usage. It should be much better with the
emulated Intel.

Joerg

[Matthias Scheler]

Post by Joel Carnat
Using FTP, the NetBSD VM was able to transfer the netbsd kernel file and
i386cd.iso at 5.39 MiB/s (according to ftp(1)).

ftp(1) (and any other FTP client) is not an appropriate network
benchmark.
Especially not under VMware because the disk access is slow (at least in
my experiences with my VMware server).

Please use a more appropriate tool like "ttcp" or "netio" which are both
available in "pkgsrc".

Kind regards

[David Laight]

Post by Matthias Scheler

Post by Joel Carnat
Using FTP, the NetBSD VM was able to transfer the netbsd kernel file and
i386cd.iso at 5.39 MiB/s (according to ftp(1)).

ftp(1) (and any other FTP client) is not an appropriate network
benchmark.
Especially not under VMware because the disk access is slow (at least in
my experiences with my VMware server).

FWIW I've done measurements using ftp to tranfser a sparse file
to /dev/null - which removes most of the disk traffic...

David