Moving A Petabyte of Data

(With apologies for the headfake of posting this entry then taking it down - my fingers were working faster than my brain, and I accidentally posted the entry without completing it. Or proofreading it.) I made a speech last week at which I asserted it was faster to send a petabyte of data from San Francisco to Hong Kong by sailboat, than by the internet. I got quite a few "how can that possibly be true?" kinds of questions, so here's the math. (Full disclosure, I am a mathematician by training, which guarantees me a lifetime of small "off by one" errors in all subsequent calculations - so if I get something wrong, be gentle). A petabyte is a thousand terabytes, which is a million gigabytes, or a billion megabytes. Or 8 billion megabits. With me so far? So if you had a half megabit per second internet connection, which is relatively high in the US (relatively low compared to residential bandwidth available in, say, Korea), it'd take you 16 billion seconds, or 266 million minutes, or 507 years to transmit the data. Can you sail to Hong Kong faster than that? At a full megabit, just divide the time in half. Even at a hundred megabits (about the highest, generally available, of any carrier I've seen), it's a few years. As Hal Stern once said to me, "Never understimate the bandwidth of a station wagon full of storage driving down the [New] Jersey Turnpike" - and now you understand why tape based storage has such a lasting appeal to so many enterprises recording, compiling, transporting or just plain archiving, very large quantities of data. From video surveillance to trading data. Standard tapes are 500GB each (currently), and fit nicely into cardboard boxes with overnight express labels. One other big benefit to tape as an archive format? When the data's at rest, it consumes no electricity - just imagine a petabyte of data spinning on even the most power efficient disk storage (for reference, a petabyte of active disk-based storage is the equivalent of more than 40 Thumpers, each drawing more than a kilowatt - and tipping the scales at something north of 150 lbs, slightly tougher to put on a sailboat, or in an overnight envelope). For data to be available, disks have to be kept spinning and cool (tape has no equivalent requirement). Now there is no one hammer for all nails, and tape isn't perfect for a lot of applications (near line storage, eg) - but it plays a prominent role in some remarkably cutting edge high performance computing applications, along with social networking and content aggregation sites (who think nothing of gathering terabytes of data every day) - tape archive isn't just for banks or telcos running mainframes (although we're good there, too). So yes, at least for now, it's faster to send a petabyte of data via a sailboat than the internet (at least defined by the bandwidth to which most of us have access). Which btw, is another reason we're refreshing our Solaris on DVD program - it's more efficient for many folks to get a 4 Gigabyte DVD in the mail (for FREE) than nurse our download centers, a megabit at a time. (And I apologize for how slow the DVD deliveries have been - we haven't exactly executed perfectly here, but hopefully it's getting better as I type.) And I don't want to even think about moving a zettabyte.

Date: March 12, 2007
Category: Main

More »

Others News