Gigabit Ethernet Need-to-Know

[a.k.a.]

Hello everyone,

This is in ref to the article that appeared on August 19 on Gigabit Ethernet.

THANK YOU for getting an article up that tackles Gig-E speeds. I moved a NAS/DAS storage plan to the back burner because I wasn't finding much on Google about real-world Gig-E speeds compared to the alternative options.

This is my first venture into network solutions, and I'm keeping it simple/stupid, but I'm hitting a roadblock in my understanding of the mechanisms behind network transfer speeds. Specifically, how do buffers fit into this picture? Here's the issue I need help understanding:

One additional comparative benchmark that would make the article even more helpful is against the data transfer rates for a PCIe ExpressCard 1.0 (Type II) <-> eSATA adapter. Why? Well, this is what is available to anyone with a laptop who needs dedicated storage.

I have read that ExpressCard eSATA adapters get real-world sustained read speeds of anywhere between 30-75 MB/s (MB/s, not Mbs). Sustained speed is important -- it's the speed after the buffer is full, and when the hard drive speed becomes the bottleneck. I don't have figures for sustained write speeds, but I assume theyr'e about the same, since we're talking about data moving between one maxed-out HDD and another maxed-out HDD.

If I'm following this right, the Gig-E unidirectional speeds mentioned in the article are around 640Mbs for a PCI card and 900 Mbs for a PCIe card, or about 80MB/s to 112MB/s. Sounds a lot faster!

But then, when we're dealing with backing up multiple-gigabyte folders or entire HDDs, a HDD-to-HDD backup process over Gig-E still runs into the same speed bottlenecks as I'm reading about with the ExpressCard eSATA solution, doesn't it? Wouldn't the two data transfer rates be roughly equivalent in the end? Also, I've heard that the TCP/IP packet overhead is pretty large and eats up bandwidth regardless, and so one might even see slower transfer speeds than with an ExpressCard solution.

Or am I missing something pretty basic? Is there something about buffers that I didn't pick up from this article? If we're talking about backing up multiple-Gigabyte folders, then what kind of a NAS setup could one set up that gets around the full/empty-buffer bottleneck and shows higher transfer speeds over a sustained data load?

As you can tell, at this point I'm still feeling like quite the novice, but it would be HUGE to get a better grasp of these speed factors.

A little more info about my laptop: It's a ThinkPad T61p Core 2 Duo laptop (2.2GHz). It has an Intel 82566MM Gigabit Adapter, a 667MHz front-side bus rate, and 3 Mini PCI Express buses, some of which are already taken up, but I can't entirely decipher everything I'm looking at under Vista's Device Manager console. There's also the iSCSI controller. I'm running Vista x64, but I've got a TechNet subscription. That doesn't allow access to Home Server, but when SBS 2008 is finally released, that should be available to tinker with.

Thank you very much in advance for your time and your advice.

a.k.a.

[thiggins]

I can't speak to eSATA transfer speeds, sorry.

The testing in this article was done with IxChariot and a script that is very bandwidth efficient. The tests performed were not intended to simulate file transfer performance, but to explore what raw TCP/IP transfers could do.

File transfers involve plenty of protocol overhead that differ by OS and network file system used. OS buffering and caching has a huge effect on file transfer performance and is a very complicated subject.

I am exploring NAS performance limits in another series, so you might want to follow that one. So far I have been able to get writes in the 80-90 MB/s range, but reads are limited to 60-70 MB/s. Peeling that onion is taking some time.

[rogerbinns]

The file transfers are the killer for me. With 100Mbps your file transfers are limited by the speed of the network. With GigE you are limited by the speed of your hard disks.

If you use a DV camcorder, the files are 10GB per hour. My backups are 27GB.

A related place this performance matters is if you are using network boot and filesystem. For example I have PXE boot Ubuntu Linux setup (using an expanded LiveCD). It is almost 10x faster using GigE.

The next big thing is 10GigE (1 million ports shipped in 2007 according to Wikipedia) with 100GigE currently under development.

[thiggins]

But what good is 10GigE if the limit is the hard drive speed?

Tim,
Modern PCI-E Gigabit adapters actually net wirespeed on TX and RX (945-950 on Chariot) and 1860ish Mbs on Bi-directional. The game isn't about throughput now, it's about at what cost of CPU utilization you achieve these results at. Modern Adapters with support for Large Send Offload (TCP Segmentation offloading) net wirespeed at less than 10% CPU utilization, while RX (optimized with Receive Side Scaling) nets less than 20% CPU.

This can all be done using standard size ethernet frames. Jumbo frames only tend to eat buffers these days.

Jason Folk
CCNP

oh, and of course power required to in active and low power states.

Regards,
Jason Folk

[rogerbinns]

Originally Posted by thiggins

But what good is 10GigE if the limit is the hard drive speed?

The 10GigE and 100GigE comment was not in the context of hard drives, just pointing out that GigE is not currently the fastest conventional networking you can get :-)

The current fastest hard drives have a 3GBps interface so a single drive could theoretically saturate a GigE connection, although platter speeds aren't that high (my 750GB drives get around 60-70% of a GBps). However an array of drives as well as data being served out of cache to one or more clients easily could saturate GigE.

[Dennis Wood]

We tested an Asus G2S laptop using Vista SP1 in real measured tests using a combination of file 5GB (total) file tranfers, iozone tests, and media encode tests. This laptop has an Esata port as well as PCIe connected gigabit so we tested both.

From Vista SP1 to our test XP SP1 workstation (raid 0) our best transfer over gigabit was measured at 47.5MB/s

From the laptop's Esata port, the best transfer rate was 45.89MB/s. The external drive used was the less-than-speedy WD "green" 1 TB drive.

The transfer rate between this laptop and our Intel 4 drive NAS (SS4200) was just over 45MB/s (read). In other words, with Vista's support of SMB2, and what I'm guessing is the Intel's support of this on the NAS, you might find using a fast NAS about the same as using the Esata drive where read performance was concerned. Use RAID0 instead of RAID5 on the NAS and write speeds would likely be on par.

As an aside, we regularly use this laptop to transfer files from SxS Expresscards containing HD video from a Sony EX1 camera, as well as DVCPRO100 footage from an HPX500 (via a P2 PCMCIA to Expresscard adapter) Transferring directly from the SxS card to our XP editing workstation (RAID0) we typically hover in the 29MB/s area as reported by Vista over an 8GB transfer. Btw, aka your search for real world numbers is the same as what we're exploring and what brought me here to this site. We've got a comprehensive review coming which will certainly (with Tim's permission) link back here to a few of these excellent articles.

[a.k.a.]

Tim, Roger, and Dennis,

It's very helpful to hear all this. Thanks for jumping into the discussion.

Maybe I'm misunderstanding how processors handle I/O activity, but where's the NAS storage server/client app that does what the torrents did for P2P? How about a backup app that takes advantage of dual core processing to route the data simultaneously through both the Gig-E and the eSATA ports? You'd need a dual core NAS OS on the other end, presumably, but it would be worth every penny if you could break through the data transfer bottleneck. Maybe it could route data into RAM until the HDD caches were available to write.

Roger, PXE caught my attention just now, in an equally half-baked way. Could you explain what makes it faster? You're saying that file transfer speeds are much higher under PXE, not just boot speed? Is Pixie a full-fledged OS distro, that in this case is installed on a server and booting an Ubuntu Live CD onto a second machine? Or is Pixie more an app running under a different Linux distro, that's handling file transfer (in this case a Live CD) for the OS? Is PXE the kind of user-friendly, workhorse distro that you could use as a primary OS on a workstation/laptop?

Dennis, count me among the people who think your review will be doing everyone a favor. I am definitely looking forward to it. I had two comments on this very worthy project:

1. On the WD external drive, is that inside a simple external enclosure, or a multi-volume RAID rack? If it happens to be in a simple enclosure, is it a dedicated eSATA-only enclosure? Or is it some kind of eSATA + FW800 / IDE / USB enclosure? I've read that dual interface enclosures route data through a chip, whereas in enclosures without the FW800 / IDE / USB, the data routes directly to/from the drive, and can move faster.

2. By chance if you haven't finished the write-up yet, it would be incredibly helpful to see the file transfers checked out over two different brands of ExpressCard eSATA adapter as well. The vast majority of laptop users don't have dedicated eSATA ports. It's been a while since I've looked, but I remember finding only patchy speed comparisons for ExpressCard-eSATA adapters, but I'd wager there are significant differences. Obviously, you can't test them all, but you hear through forums which ones are getting all the buzz. (Mainly one hears mentions of Silicon Image (SIIG), Rosewill, Apiotek, Sonnet, Adaptec and CalDigit, roughly in that order.)

Another measure of eSATA ExpressCard differences would be whether you can boot a machine from it. Most, seemingly, cannot, and it doesn't seem to be strictly whether your BIOS can change ports for its boot priority. Everyone is asking this question.

Some 2-port eSATA ExpressCards are even touted as good 2-volume RAID controllers, but my assumption is that they aren't dedicated pathways, they're just post multipliers that are splitting bandwidth. Maybe some ExpressCard 54 RAID controllers are legit. (It would be even nicer to have a two-layered RAID card that taps the stacked top+bottom ExpressCard slots that a lot of computers have.)

Just because I'm thanking each of you for your insights, it doesn't mean this thread is closed!

Happy summer weekend to readers of this thread.

a.k.a.

[Dennis Wood]

One thing we don't have is an Expresscard Esata adapter :-( The G2S has an Esata port standard, as well as a 1920x1200 res screen, which is why we use it in our studio in combination with external 1TB drives. The drive enclosure we're using is a single drive "SMART" brand (about $45) which has both Esata and USB interfaces. In a test between the same drive/enclosure to an Intel based workstation (RAID0) running XP SP3, we saw 67MB/s reading from the Esata port and writing to the workstation's RAID0 array.

USB tests on the same drive have been testing about 29MB/s which is fairly consistent in both reads and writes...albeit with higher CPU overhead. In other words, this particular drive enclosure claims to have a SATA II interface and in testing is performing as such.