Jumbo Frames, iSCSI and Disabling Nagle

So the new project (which I get to spend about 15 minutes a week on) has been to remove the mechanical harddisk from my HTPC and have it run completely silently off the SSD.   The first stage was purchasing the ReadyNAS NV+.  2TB drives have dropped below $100 so a NAS device with 1Gb networking was the no brainer solution.

Given I have two ATI CableCARD tuners (which BTW are no longer being made or supported by ATI), I needed to make sure the network and NAS had enough bandwidth and performance to write two simultaneous HD streams, while reading a third.  Now in theory, with a 1Gb network, you should be able to get roughly 100MB thoughput, unfortunately the real world doesn’t work that way.

My initial testing using Lan Speed Test (more on that later) showed I was getting around 24MBps writes and 45MBps reads.  (Remember big Bs are Bytes and little bs are bits.  8 bits to a byte)  So looking at my Comcast HD recordings, it turns out Fox is broadcasting 1920×1080 MPEG2 at around 14Mbps.   This isn’t that bad.  Note that DirectTV uses MPEG4 at around 5Mbps.  Since mp4 delivers better quality at lower bit rates,  the DirectTV advertisements aren’t lying when they say the picture is better.   For comparison, a typical DVD is MPEG2 @ 9.8Mbs and a BlueRay is MPEG4 at 40Mbps.   So if you haven’t figured it out yet, HD can really mean anything you want.

But the answer is if you want to record 2 HD TV channels from Comcast, you need about 2MBps per stream, or really just 4MBs, but you definitely want to have some head room, and there’s also the need to play DVD images back from the drive

I was hoping to get a little better than 25% of maximum throughput so started looking for solutions.    The first thing I poked at was the Nagle’s algorithm attributes.    This basically tells the TCP stack to gather all the small requests in to one big one before sending.     Turns out gamers want to disable this feature to make sure every keystroke/click is sent the moment they enter it, and not let things batch up on the network card.    For media streaming, you want the opposite behavior, but you’re rarely sending small data packets anyways.    But just for kicks, I set the following registry settings

HKLM/SYSTEM/CurrentControlSet/services/Tcpip/Parameters/Interfaces/{nicid}/

GlobalMaxTcpWindowSize = 0x01400000 (DWORD)
TcpWindowSize = 0x01400000 (DWORD)
Tcp1323Opts = 3 (DWORD)
SackOpts = 1 (DWORD)
TcpAckFrequency = 4 (DWORD)
TcpDelAckTicks = 2 (DWORD)

And ran this command:

C:\ netsh interface tcp set global rss=disabled chimney=disabled autotuninglevel=disabled congestionprovider=none

Mostly from recommendations from this iSCSI site (more on that below):

Basically I’m saying the opposite.   Package up as many small bits as possible into larger ones to avoid the overhead.  Difficult to measure the differences here, but I’m recording them here for myself in case I run into problems.   Though, it turns out there is another TCP feature along the same lines that does help with streaming media and was much more noticeable.

I had already done all the cache optimizations possible on the ReadyNAS, and configured it for Raid 0 since I really don’t need redundancy for recorded TV shows and I’m using Amazon S3 for offsite backup as described here.   One of the features I found on the ReadyNAS was support for TCP Jumbo Frames.   So it turns out the standards the Internet still runs on today were defined over 30 years ago.   Given the reliability of Ethernet at the time, the designers decided that 1500 bytes was the largest amount of data to be communicated in each packet so if the receiver didn’t get the packet, the resend wouldn’t be so large.    In today’s home gigabit switched networks, collisions and data corruption are almost unheard of.  So rather than waste all the CPU & interrupt time splitting and joining small packets, you just build one big one.   This is also a bit more efficient because each packet also requires header and footer  information describing where it should be delivered to.   Unfortunately, because everyone has to follow a standard, the largest the Jumbo Frame packet goes to is 9K, but that’s still almost a 4x increase in the data delivered with the same header and footer used for the original frame size.

So I started looking at the configuration for the on-board network adapter on my Intel P35.  No Jumbo Frame option, but this I found this note:

Note: The Intel PRO/1000 PL Network Connection supports jumbo frames in Microsoft* Windows* operating systems only when Intel® PROSet for Windows Device Manager is installed.

Cool!  So Installed Intel ProWin and still couldn’t find the JF option.   Do a little more research and find this:

The following gigabit LAN components included with Intel® Desktop Boards do not support jumbo frames:

• Intel® 82566DM Gigabit Ethernet Controller
• Intel® 82566DC Gigabit Ethernet Controller
• …

Eeeekk! My motherboard chipset doesn’t support jumbo frames!    So it was off to Amazon to see how much a 1Gb PCIe Network card with JF support would set me back.    Since 1GB is not longer the bleeding edge (they now have 10Gb NIC over Cat6),  this Startect card was just under $25.    This also allowed me to do some real world performance testing between the two use Lan Speed Test:

HW	Intel 82566DC	StarTech ST1000SPEX
SMB Read MB/s	48.05	45.8
SMB Write MB/s	24.28	33.43

So I’m pleased with the 30% speed improvement on write.   I read somewhere that JFs aren’t used for reads, hence there wasn’t any significant difference there.    So I’m all set right?   Oops, wait a minute.   It turns out that Windows Media Center won’t record to a network drive.    This is part of the DRM associated with CableCARD, which I’ve ranted about before.   It turns out the new ATI BIOS relaxed OCUR standards addressed most my CableCARD concerns.  This left two possible solutions:

1. Record to the SSD and use DVRMSToolbox to move the recordings to the NAS (after commercial detection)
2. Use iSCSI rather than SMB (Microsoft File Sharing)

Once again, going back to 30 years, there were a couple competing standards for attaching disk drives to computers.    Once of these was called SCSI and was championed by Apple and Sun (as well as many others).    SCSI had a bit high level command structure and some interesting chaining features that are similar to today’s USB features.   PCs meanwhile were using IDE interfaces which have evolved in their own direction.    Fast forward 30 years, and have these network cables which are now as fast as those big thick SCSI cables, so why not send the SCSI protocol over that?   Now you have iSCSI.

So the cool part is, you use iSCSI, and Windows thinks the drive is a local SCSI drive, not a remote NAS drive.     Of course, since I bought the cheaper ReadyNAS NV+ rather than the latest and greatest ReadyNAS Ultra, iSCSI support was not yet built in.   Enter the OpenSource world to the rescue.    Since the ReadyNAS NV+ is basically a little Sparc machine running Linux, Stephan at http://whocares.de/ ported the Linux iSCSI Target daemon.    If you go this route, be sure to check out his support page which was a little tricky to find.    In short, the original directions pointed you to the wrong config file, as I explain here:

Downloaded and installed 1.4.20.2. After following the instructions verbatim, I realized my target was not being created and spent a lot of time searching the net for the cause of this message:
iscsi_trgt: iscsi_target_create(131) The length of the target name is zero

I finally came back here and read all the comments. The problem the whole time was ietd.conf needs to be /etc/ietd, not /etc like the instructions say. 🙁

Hopefully google will find this comment for the next guy who comes along.

I mention a couple other quirks on the support page, but the above is the only one that matters.   So back to performance testing via Lan Speed Test:

Protocol	iSCSI	SMB
Read MB/s	21.05	45.8
Write MB/s	191.28	33.43

Whoa, check that out!    More than 6x improvement in write performance!    In fact, it now writes almost twice as fast as the theoretical network maximum… umm… wait-a-second….   That’s probably not right…

A little more investigation showed that because Windows considers it a SCSI drive, there’s lots of local caching going on which was fooling Lan Speed Test.    Using Blackmagic’s Disk Speed Test (which also won’t work on a network drive), write speeds were around 14MB/s.    So the freeware implementation of iSCSI leaves a bit to be desired performance wise.    There may be some other things you could do via direct device access and later versions of iSCSI, but I decided to go back to the DVRMSToolbox solution.

So I’m actually pretty happy with the current solution where I record to a temporary directory on the SSD and then move the file over to the NAS.   This also allows the Dragon Global Showanalyzer to work on the files locally rather than scanning them over the network.