TS509, Link Aggregation, 3COM 2916 = good

[Dennis Wood]

We just got a 16 port switch in here specifically because it was the only layer 2 managed switch I could find for under $300 that had full support for 803.2ad...or automatic link aggregation. Trunking and link aggregation was supported on the previous switch, the DLINK 1216T, but it does not fully support this specification...only static trunking.

Enter the TS509 with dual ports and the magic claim that it supported load balancing. With a static trunk set up and 2 LAN cables used, the TS509 did not load balance on the trunk which was not a surprise. QNAP specifies the switch must support automatic link aggregation, 803.2ad. Well, there are some issues there under multiple loads (and I'm working with Qnap on this) but what a surprise in testing today. The test workstation uses an ASUS (P5W-DH) board with Marvel dual LAN PCIe connected gigabit and Vista SP1 with ICH7 based RAID 0 drives. The Marvel driver supports 3 modes, basic, static, dynamic which are used depending on the switch capabilities. Dynamic is what you want, and can only be used with switches fully supporting 803.2ad....like the 3COm 2916. This switch has an amazing feature set for under $300.

My previous tests with link aggregation on the DLINK 1216T switch (using the aforementioned workstation) showed that link aggregation worked, but only in "static" mode on the Marvel driver with a trunk configured. Now with the 3COM switch, I can use "dynamic" mode, and the TS509 does work in load balancing mode.

Performance from the same workstation to the TS509 (neither teamed) was previously 45MB/s write, and about 60MB/s read. With the same workstation now configured with two gigabit LAN connections, the TS509 connected in the same manner, and the LACL trunks set up, I got a surprise. Write speeds jumped to 50MB/s and read speads are much higher at 83 MB/s using our 5.3GB test set of files. This is with the TS509 using RAID 5.

In our large file copy (200GB) tests from the NAS to the desktop, Vista is now jugging along at sustained 97.5 MB/s read. Link aggregation is not supposed to increase speed of one "pipe", but clearly something else is going on here. Any LACL experts out there? It would seem that "dynamic" LACL is capable of doubling the pipe, and increasing a single workstation's (IP address) performance.

Note that with the latest firmware 2.0.2, performance with 2 workstations loading the unit simultaneously is completely unacceptable at less then 3MB/s ... so some more code updates will be required.

[thiggins]

So "load balance" on the TS509 Pro is the same as teaming? Or only with the dynamic mode on the 3Com switch?

[Dennis Wood]

For those who are not aware, the QNAP TS509 has two gigabit LAN ports that support dual LAN, failover, or fault tolerant operation. The TS509 network interface allows you to choose "Load Balancing" as an option, but the literature indicates you must use an 803.2ad compliant switch for this to work. So you may choose this, with both LAN ports connected, but with a normal switch...there's no load balancing. All of the traffic goes to one port.

With a switch like the DLink 1216T that supports manual (static) trunking, teaming, or static link aggregation (used to describe the same thing by various manufacturers)...no joy. The TS509 still only uses one port, even with a static trunk setup on the switch. Now if you have a motherboard with the Marvel Yukon family dual gigabit ports...that will work as a server and load balance just fine, as long as the driver is set to "static" mode. So the Marvel driver will support basic "teaming" even on a cheap switch but the links cannot dynamically configure themselves, and two identical teamed workstation will never be any faster with respect to LAN traffic between them. These types of motherboards would be a perfect base for a cheap but high performing NAS under multiple loads as each (assuming 2) workstation would end up with it's own pipe. The Nvidia 680i motherboard chipset supports teaming as well, but it seems to be very basic. It does not work if a trunk is configured on either of the switches, static or LACL.

With the 3COM switch you can set up either static trunks, or LACL trunks. An LACL trunk is evidently a fully compliant 803.2ad construct. From what I understand of the 803.2ad spec, this spec allows dynamically reconfiguring of the the ports used in a trunk. A trunk can contain up to 8 ports on the switch. With a four port PCIe NIC, you could therefore create a 4 port trunk that would likely reconfigure as required for higher bandwidth. I believe that what I'm seeing now is the absolute limit of the TS509 unit drive performance. 50MB/s write and 83MB/s read is by far the best RAID 5 performance I"ve seen over the LAN in our testing. To compare, the best we could do with an onboard ICH7 RAID 5 was 35MB/s local write!

Next step...our own NAS with dual LAN ports, and the RocketRaid 4 port controller?

[thiggins]

The Broadcom BRCM 5787 Ethernet chips in the TS509 PRO are PCIe. So there must be some other limitation that is preventing a single PCIe Ethernet port from reaching the > 100 MB/s that my IxChariot testing showed that it was capable of.

[Dennis Wood]

Yes, it may be at the NAS end, or the workstation end...hard to tell. Iozone, for the first time, is not accurately measuring large file read and write rates between a workstation and NAS with LACL used on both. What we're actually measuring is about 20% higher.

In fact, none of the tools work in this configuration. I suspect the 3COM switch is routing upstream and downstream traffic dynamically over the 2 LAN connections and causing some measurement confusion. It's the only way to explain the 25% improvement in read rates from the NAS.

[iwod]

Originally Posted by Dennis Wood

Yes, it may be at the NAS end, or the workstation end...hard to tell. Iozone, for the first time, is not accurately measuring large file read and write rates between a workstation and NAS with LACL used on both. What we're actually measuring is about 20% higher.

In fact, none of the tools work in this configuration. I suspect the 3COM switch is routing upstream and downstream traffic dynamically over the 2 LAN connections and causing some measurement confusion. It's the only way to explain the 25% improvement in read rates from the NAS.

So what you are saying is the results inaccurate?
Becasuse it sounds too good to be true

[Dennis Wood]

What I'm saying, after many, many tests is that if you hook up the TS509 in fault tolerance mode (2 ports connected to the switch), to a an LACL capable switch, with a trunk configured, here's what you will see in actual measured file transfer times. This assumes you're using Vista SP1 on a machine with at least 2 drives in RAID 0 (to keep up with the NAS). It also assumes that this workstation has PCIe connected gigabit and you're connected to the switch with one port.

1. Read speeds will exceed 85 MB/s on a mixed set of files. If you are copying a single 200GB file off the NAS, Vista will report read rates at about 97 MB/s. This I believe is as good you'll ever get over gigabit ethernet. If you try copying two different 200GB files off the NAS to two different Vista SP1 workstations (both with RAID 0), each will report 63MB/s for a total rate of about 120MB/s. This is outstanding.

2. Write speeds will be at 50MB/s.

If you load the NAS from two workstations (at least in our tests) the write speed will drop dramatically. I'm very confident this will be addressed with a code update. Confident enough to order a second TS509 to see how it performs with rsync (synchronizing two units for offsite backup over the WAN)

[Dennis Wood]

A few more tests with the TS509 using RAID5 and in load balancing mode using our 5.3 GB mixed file set.

Single drive XP SP3 workstation with PCI gigabit:

Read: 38 MB/s
Write: 36 MB/s

Single drive Vista SP1 laptop with PCIe gigabit:

Read: 30 MB/s
Write: 44MB/s

Using Storagecraft backup, we were able to back the laptop up to the NAS (56GB) at a rate of approx. 45 MB/s which is excellent.

Simultaneous two workstation loading (Vista SP1, RAID0)

Read: 61MB/s (total aggregate read rate at 122MB/s)
Write: 3 MB/s (appears to be a code issue with the NAS)

Single workstation, Nvidia 680i SLI chipset, Vista SP1, RAID 0 onboard)

Read 83MB/s
Write: 52MB/s

[SirBC]

Originally Posted by Dennis Wood

Confident enough to order a second TS509 to see how it performs with rsync (synchronizing two units for offsite backup over the WAN)

Dennis, did you get a chance to test the remote replication feature yet? I would be interested to hear your thoughts as I am considering purchasing two of these for the same purpose.

Cheers,
Dave

[Dennis Wood]

Dave, I've got 5 WD green drives 1TB sitting here and a TS509 on the way. I'll definitely post my thoughts on rsync performance when they're here. It will be interested to test the NAS with the WD green drives and compare to the unit using the 5 Seagate Enterprise 1TB drives which case $100 more each. With the money saved, one could purchase six of the green drives, and still have $400 in your pocket. We use several of the WD green drives in external drive enclosures as theoretically they should run cooler.

On another note, I've just tested some beta firmware and the dual workstation loading test shows much improved results. The TS509 is still in load balancing network mode to an LACL capable switch. Reads are still (to a RAID 0 Vista SP1 workstation) an impressive 97MB/s on large files, and writes at around 47MB/s.

On two workstations loading the TS509 simultaneously, write rates were dropping to 3MB/s and are now improved to approximately 10MB/s. in the beta code for an aggregate total of approx. 20MB/s. These are XP workstations, and I have not tested two Vista workstations yet. I believe the QNAP folks can potentially tune this to where I believe it should be, about 40 to 45MB/s aggregate.