There has been a lot of sound and fury today (well, technically yesterday) surrounding Verizon and Google’s joint net neutrality policy announcement. I didn’t really know about it until Tim brought it to my attention. He wanted to know my thoughts about this, and blogs are practically designed for uninformed opinions, so why not?

In the interest of full disclosure, I’ve worked at Google in the past. Several friends of mine are either current or former Googlers. Money from Google’s generous intern salary paid for all the furniture in my living room. I’m totally biased, I freely admit. The rest of this post will also probably reveal my profound ignorance, but that’s why there are comments.

Frankly, I’m surprised at the amount of rage that I’m picking up about this announcement. This isn’t a net-neutrality-crushing alliance between Google and Verizon. It’s not binding. There’s nothing in the language that says that it’s not open for discussion and feedback. This is basically these two companies saying, “This is how we think we’re going to deal with the net neutrality thing right now.”

The notion of allowing carriers to “offer new types of services that are not part of the open Internet” is also not all that surprising to me. In my view, net neutrality says “if two computers on the Internet want to talk to one another, the carriers providing Internet access to either of the two shouldn’t be able to stop them”. Under “shouldn’t be able to stop them”, I mean that my computer shouldn’t be unable to talk to Google’s computers because I’m on a “Microsoft-friendly” ISP. What Google and Verizon seem to be proposing here is that companies should be able to create services that don’t operate over the public Internet as long as those services don’t circumvent net neutrality. It’s not really clear what they mean by “not part of the open Internet” here; are we saying that the computers making up this service aren’t world-routable? If so, then companies are already doing this internally. Large corporate intranets could be seen as an example of a service that operates separately from the open Internet. Do they mean creating a parallel network over which these services run? In that case, data center networks seem to fit that description pretty well and clearly companies are providing dedicated networks for data center networks, or just buying their own. Again, on the face of it I’m not too terribly distressed.

Most of the rage seems to be directed at the fact that they don’t think that wireless access to the Internet should be protected in the same way that wired access is. Judging by Google’s blog post on the subject (see how biased I am?) their justification seems to be that they don’t know what the mobile Internet really looks like yet. It seems to me that this is basically a political compromise; Google caves somewhat on the wireless issue so that Verizon will agree to come on board with the agreement. Notice that the transparency requirement opens the door for any company that tries to pull any pro-neutrality funny business to be publicly reamed by their customers (you need only look at the Comcast P2P filtering scandal to see how much good press anti-neutrality practices generate for the telcos that implement them). Google promises to not be non-neutral in access to its properties, and it and Verizon get to be great friends; everybody wins.

Are Google and Verizon both huge multinational corporations that are interested primarily in pulling in a lot of cash? Absolutely. Does the language of this announcement open up the possibility of abuse, corruption and rampant mismanagement? Of course, but so does every other agreement between more than one person. Does this announcement violate Google’s “Do no evil” mission statement? Hardly. To me, this just sounds like Verizon and Google getting all buddy-buddy and announcing the sort of long-term, under-defined recommendations that make PR people all warm and fuzzy because they don’t actually require you to promise to do anything. Everything I’m hearing from Google indicates they’re really self-conscious about this whole thing (“it’s not an aggreement”, “we are still committed to net neutrality”, “we’re not sure what the mobile market looks like, so we have to keep our options open”), which gives me some comfort; the second you get a “so what?” announcement out of a company Google’s size (see Comcast’s P2P shenanigans, Antennagate), it’s time to be concerned. Yes, I know I’m totally defending Google here. I told you, I’m biased.

It might sound like I don’t care about net neutrality, but nothing could be further from the truth. People are absolutely right when they say that the Internet is the most democratic medium ever conceived, and it’s really important that it stays that way. The Internet is big business, and these days it’s dominated by big businesses with a lot of influence and a lot of money. As citizens of a democracy, we have every right (and, some may argue, the responsibility) to be outraged when a large, powerful entity looks like it’s trying to gain control of a fundamentally democratic communications medium like the Internet. I just don’t think that this announcement represents anything close to such a power grab.

And that is my opinion on the subject. Release the trolls!

I’m going to assert that KVM switches are awful. There are a couple reasons why:

1. You have to switch back and forth between machines with a key combo. Often, this combo is not documented anywhere on the switch itself. Almost always, it’s non-modifiable. Always, it’s cumbersome.
2. KVM switches are stuck in 1997. Most KVM switches use PS/2 and VGA connectors. Getting this to work with computers that use USB and DVI requires a bunch of separate dongles that only nerds like me have in abundance. Good luck finding a native USB/DVI KVM switch for under $150.
3. More cables. Seriously, I’ve got enough cables cluttering up the back of my desk, I’d rather not introduce more cables if I can avoid it.

Thankfully, there are software solutions that allow you to bypass the need for a KVM switch altogether.

Synergy allows you to use a single keyboard and mouse among an arbitrary collection of computers. Synergy runs a server on the machine with the keyboard and mouse attached and clients on all the other machines. The user specifies how the displays of these multiple machines are arranged with respect to one another; for example, I could say that my laptop’s display is to the left of my workstation’s display. Synergy detects when the mouse hits the edge of a display and “switches” the keyboard and mouse from one computer to another based on the configured arrangement. So when my mouse hits the left edge of my workstation’s display, it is seamlessly transferred to the right edge of my laptop’s display. I’ve used Synergy this way for a while now, and I’ll never even think of buying a KVM just to share a keyboard and mouse again.

Of course, Synergy doesn’t take care of sharing a single display between multiple computers. Most monitors have multiple inputs, and if you’re willing to press the input change button, Synergy might still work, but it’s still cumbersome. Another alternative is to use VNC. VNC is a communications protocol, and is implemented by a ton of different clients for all platforms (Google “VNC” for a list of the big ones). It essentially allows you to view and control another computer remotely. Those of you running Mac OS X Snow Leopard will probably use Screen Sharing for this, but under the covers it’s still VNC.

Both these solutions are cross-platform, pretty easy to set up and have a wide range of graphical helper applications and front-ends. They share the common disadvantage that they send data unencrypted, which means that any enterprising hacker could view your keystrokes as they travel over the network. Thankfully, it’s easy to solve this problem. There are a number of applications that will create a secure tunnel between two computers (Google “SSH tunnel” for more information). Alternative protocols to VNC like NX provide all the functionality of VNC over an encrypted connection. There are a lot of options, and the best part is that almost all of them are free.

So in short, if you’re thinking of buying a KVM, save $100 in cables and dongles and use software instead.

Megabytes or Mibibytes?

As a bit of exposition: a common question when buying hard drives is “I just bought a 500 GB drive but it only has 465 GB of capacity when I hook it to my computer. I want my extra space back!” This discrepancy isn’t because hard drive companies want to screw you over (OK, they might be trying to screw you over, but not with this), it has to do with how computer science deals with bytes. In a network setting, measurement is typically done in powers of 10, so a gigabyte is 1000 megabytes. With storage, measurement is typically done in powers of 2, so a gigabyte is 1024 megabytes. So when you buy a 500 GB drive, your computer will tell you it’s a 465.7 GB drive because they gave you the powers-of-10 capacity and the OS displays it in powers-of-two. I know that Snow Leopard no longer does this to avoid confusion, but I’m not sure what Windows 7 does.

In an effort to resolve the confusion, the International Electrotechnical Commission (IEC) instituted a set of units of measurement that used powers of two, and kept powers of ten as the default. So according to IEC, a megabyte is 1000 kilobytes, and a mibibyte is 1000 kibibytes. The ‘-’-bi’ was supposed to invoke the word “binary” in the user’s mind. Personally I think they deliberately did this to make computer scientists sound ridiculous. Once you get up high enough you hit yobibytes, for Pete’s sake; that sounds like something you’d give Scooby Doo’s second-cousin Yobi Doo to make him cooperate.

Google Calculator deals explicitly in IEC units, but doesn’t follow the standard: if you ask it “how many bytes in a yottabyte” and “how many bytes in a yobibyte”, you get the same answer. This is intensely irritating if you care about powers of ten when you’re dealing with bytes.

Enter Wolfram Alpha

Wolfram Alpha is basically a thin veneer on top of Mathematica that allows you to access some of its simpler functionality for free. Mathematica is a mathematical toolbox that is so absurdly sophisticated and full-featured that I’m expecting it to gain sentience any minute. Just to give you an idea, the Mathematica documentation – complete with typesetting and figures – was written entirely in Mathematica. Scary.

I was hunting around for an alternative to Google Calculator’s unit conversion (and considering writing my own, such was my level of desperation) when I decided to ask Alpha. After all, you can ask Alpha “How many roads must a man walk down before you can call him a man?” so it must be able to do unit conversion, right? Turns out it can, and it honors the IEC specification! So if I ask it “how many bytes in a kilobyte”, not only will it give me powers of ten, it will ask me if that’s what I wanted and suggest kibibytes as an alternative!

It can also convert between decimal, binary (twos-compliment and unsigned), hex and octal. Hot damn.

I noticed a couple days ago that my Mac mini (which acts as a storage server but will probably be doing more media serving once I get a better TV) was not visible to the rest of my home network. It turns out that the mini’s network interface was having serious issues, because the mini couldn’t even get an IP address. Doing a little digging into the log, I found a lot of log segments that look like this:

kernel  AppleYukon2: error - FATAL: SkGeStopPort() does not terminate (Rx)
kernel  AppleYukon2: error - Event queued in Init Level 0
configd[14] network configuration changed.
Firewall[66]  krb5kdc is listening from :::88 proto=6
Firewall[66]  krb5kdc is listening from 0.0.0.0:88 proto=6
kernel  Ethernet [AppleYukon2]: Link up on en0, 100-Megabit, Full-duplex, Symmetric flow-control, Debug [796d,6c0c,0de1,0200,4de1,4000]
configd[14] network configuration changed.
Firewall[66]  krb5kdc is listening from :::88 proto=6
Firewall[66]  krb5kdc is listening from 0.0.0.0:88 proto=6
kernel  AppleYukon2: 00000000,00000000 sk98nif - deadmanCheck - nothing received, soft reset of chip
kernel  AppleYukon2: 00000000,00000000 sk98nif - deadmanCheck - nothing received, soft reset of chip
kernel  AppleYukon2: 00000000,00000000 sk98nif - deadmanCheck - nothing received, soft reset of chip
kernel  AppleYukon2: 00000000,00000000 sk98nif - deadmanCheck - nothing received, soft reset of chip
kernel  AppleYukon2: 00000000,00000000 sk98nif - deadmanCheck - nothing received, soft reset of chip
kernel  AppleYukon2: 00000000,00000000 sk98nif - deadmanCheck - still nothing received, hard reset of chip

There is only one group of people who know exactly what these messages mean, and they are the guys that wrote the AppleYukon2 driver.  Thankfully, a little Googling and some link clicking later, I had a plausible explanation. Apparently this sort of error occurs when one of the networking-related preferences that OS X stores is corrupted somehow. This apparently happened quite a bit when users upgraded to 10.5.7, and it also happened to me. The driver reads a corrupted preferences file, takes some ridiculous action and wedges the underlying hardware in some kind of bizarre state that only a reboot can correct.

Thankfully, OS X can regenerate its preferences files. To do this cleanly:

• Reboot holding down the Shift key. This will force the computer to boot in Safe Mode and (as a side-effect) rebuild its caches, which is a good thing if they’ve been populated with garbage from your corrupted preferences files, which they probably have been.
• Delete the following files in /Library/Preferences/SystemConfiguration: NetworkInterfaces.plist, com.apple.airport.preferences.plist, com.apple.network.identification.plist
• Reboot and re-apply your network settings (IP address, et al)

Special thanks to Daniel Palmer, the poor soul who sat on the phone with Apple to get this resolved. Here’s the thread containing the solution.

Update 3/3/10: This problem has reared its ugly head once again, and I’ve found out a little more information about it:

When you set the speed and duplex settings of your wired network adapter manually rather than keeping the setting at “Automatically” and you disable IPv6, the driver no longer triggers hard resets for whatever reason. In my case, it triggers soft resets at precise 6-minute intervals.

Also, as reader lafber pointed out, this problem only seems to occur when there is no traffic on the wired network. To solve this problem, I wrote a one-line AppleScript script. The application start a command that pings my router every 30 seconds. The command itself runs in a virtual terminal, so I don’t have to keep an application running in the foreground. The AppleScript looks like this:

do shell script “screen -m -d ping -i 30 192.168.1.1″

To get it to start at bootup, I saved that AppleScript script as an application and added the application to my default user’s Login Items in the Accounts system preference pane.

One of TextMate’s nicer features is code folding, which allows you to collapse a block of code (a function, a conditional block, etc.) down to a single line in the editor. This often makes a large piece of code much easier to navigate. TextMate determines where code can be folded by evaluating each line of the code and tagging particular lines with start and end markers.

Doing code folding for Python in this method (only considering a single line at a time) is impossible, since the only clear end of a function in Python is a reduction in indentation level on the next line. For example, here’s a function in C followed by a statement:

int foo(int value) {
   printf("%d", value + 5);
}

foo(24);

Here’s the same code written in Python:

def foo(value):
   print "%d" % (value + 5)

foo(24)

As you can see, if you look at the code one line at a time, it’s not clear where the definition of foo in Python ends, whereas the closing curly brace (}) is clearly the end of foo’s definition in C.

By default, TextMate punts: it defines the start of a function or class definition as the start of a foldable region and the first subsequent blank line as the end of that foldable region.  This doesn’t really work well, because function definitions without any blank lines aren’t really readable past a certain size.

There is, however, a way to hack your way around this issue. Another of TextMate’s killer features is that you can customize almost everything, including the definition for a given language.

Open the Language Editor through the “Bundles > Bundle Editor >Edit Languages …” menu item (as of TextMate 1.5.8, anyway).

Pull up the language editor for Python, find a line that looks like this:

foldingStopMarker = '^\s*$|^\s*\}|^\s*\]|^\s*\)|^\s*"""\s*$';

and replace it with this:

foldingStopMarker = '^\s*#\s*END_DEF ([a-zA-Z0-9_<]+)';

Now, if you want to fold a function definition in Python, stick a comment at the end of it:

def foo(value):
   print "%d" % (value + 5)
#END_DEF foo

Hope fully this helps out some of you who are using TextMate for Python programming.

I’ve always been the sort of person that saves receipts. Until very recently, I had an entire filing box full of receipts, warranty cards and software license keys. This system bothered me in much the same way that keeping paper copies of research papers did: they can’t be searched (did I file the receipt for my coffee table under C for coffee table, T for table, I for Ikea, F for furniture?), and they take up space in my apartment, where space is pretty limited. There’s the additional problem of a paper receipt’s relative fragility: there’s only one copy of it, and it can be burned, crumpled, crushed, soaked, can fade into unreadability, and so on. The solution was pretty straightforward; I scanned everything in the box, gave them descriptive file names and let Spotlight index them.  To provide some additional protection against data loss, I burned two copies of the newly-scanned PDFs onto two separate DVDs and mailed one to my parents. This got me thinking; sure, this way it’s much less likely that my important documents will be destroyed in the event of a disaster. Really though, how safe is my data? More importantly, how safe can my data really be?

Some studies point to the fact that the lifespan of recordable CDs and DVDs is quite short (in the neighborhood of three to five years), which means that any “permanent” storage on DVDs would need to be refreshed on about that interval to remain current. Now, given that most of what I burned on those DVDs don’t need to be kept for more than five years, that shouldn’t really be a problem. Also, these are kept in relatively controlled conditions – no direct sunlight, ambient temperature where they are doesn’t get much colder than about 55 or much hotter than about 100. But what about things like photos, home videos? Stuff that is irreplaceable and would need to be stored over a period of, say, decades?

Every data retention scheme you can name has its breaking point, some condition that could irretrievably kill your data. In my opinion, the important factors are the likelihood and inevitability of the failure and if the failure can be proactively avoided. Let’s take a look at a few ways to store data sorted from least paranoid to “tin-foil hat” paranoid.

Single disk:

• Data is lost when: the disk fails.
• How likely/inevitable is this?: All disks will fail, it’s just a matter of when.
• Proactive avoidance?: Unless you go to some outside storage source, none.

Scarily, this is what most people are relying on. Professional drive recovery companies might be able to get your data off the disk, but it will cost a pretty penny.

RAID array (multiple, redundant disks):

• Data is lost when:
  • All redundant disks fail before the array can be rebuilt. Especially heinous if your disks all croak of the same hardware problem, or your power supply kills them by catching on fire.
  • (Hardware RAID only) RAID controller fails, no replacement or equivalent controller can be found
• How likely/inevitable is this?: With enough diligence (and enough backup disks), this shouldn’t be too likely.
• Proactive avoidance?: More redundant disks, software RAID, quick monitoring and early detection of drive errors, always having a spare drive or two handy.

Burning to CD/DVD:

• Data is lost when: the disk becomes unreadable (crushed, melted, thoroughly scratched, hacked into un-spinnable chunks, or some other loss of structural integrity)
• How likely/inevitable is this?: Some might argue that it will happen eventually, but it’s a lot less likely than your disk failing, especially if you don’t expect to touch the data itself frequently.
• Proactive avoidance?: More copies, stored in more places. Re-burning periodically.

Backup to off-site computer/external disk that’s kept somewhere else:

• Data is lost when: both the storage system storing the original and the storage system storing the backup fail before an additional backup can be brought online
• How likely/inevitable is this?: Unless you’re not paying attention to your backup for long periods of time or there’s a nuclear war, you’re probably OK. The amount of time you have to react depends, of course, on the storage setup of the two machines.
• Proactive avoidance?: More than one backup, as widely distributed geographically as you can afford.

Cloud Storage (Amazon’s S3, others):

• Data is lost when: hopefully never. Companies providing cloud storage services spend a lot of money making sure that this doesn’t ever happen. Amazon’s SLA for S3 doesn’t even mention data loss, just the minimum amount of time they guarantee that the service will be available per year.
• How likely/inevitable is this?: Much like the nuclear war scenario, if S3 loses data it will probably be on the news, at least in the Silicon Valley.
• Proactive avoidance?: Multiple storage clouds, owned by multiple vendors. This is truly the peak of tinfoil-hat paranoia.

In my opinion, there are only two real downsides to the cloud storage approach. The first is that upload bandwidth, for most residential Internet customers in the United States, just plain sucks, and uploading a non-trivial amount of data is painfully slow. Fiber to the home (and a technology-friendly executive branch) might change this, but it won’t happen in the immediate future. Second, S3 costs money, both to upload data and to store it there. If you’re not willing (or able) to pay the bill, this isn’t for you.

This is by no means an exhaustive list, and there are a lot of hybrid approaches. I back up my computer’s hard drives to external drives periodically, and those drives spend most of their time in my desk drawer. My various media (photos, music, digital copies of DVDs) are stored on a software RAID-1 in the media center PC. As an additional layer of protection, all my photos are on Flickr, all my music is synchronized with my iPod fairly regularly and my DVDs are, well, DVDs.

This setup, while pretty decent, is by no means foolproof. One major concern that I haven’t talked about here is data corruption, and I’ve got pretty much zero defense against that. Once Apple releases Snow Leopard, hopefully I’ll be able to transfer all my data over to ZFS volumes. I could nerd out over ZFS for a whole other post, but the short of it is that ZFS makes far fewer assumptions about data’s integrity and comes pretty close to eliminating the data corruption problem.

This was going to be an extended rant on how people use databases where people shouldn’t use databases, but the more I wrote the more I realized that this had been analyzed quite a bit by many in the systems research community and blogosphere at large, many members of which are far more knowledgeable than I. So I’ll summarize my rant in a paragraph and then move onto more philosophical, “meta”-type comments.

Twitter’s architecture (as much as they’ve shown us) is a Ruby on Rails app backed by a MySQL database. This combination is the Golden Hammer of Web 2.0. A frighteningly large number of web application developers seem to follow the mantra, “If I need to store data, use SQL as a Big-Ass Table (no, not that Big-Ass Table). Who needs high-speed middleware? I’ll write everything in Ruby!” The problem is that schema design is as close to alchemy as CS gets and tuning databases is tedious and hard to do right. If you are writing something that must process tens of thousands of messages a day, do not think you can write it in an interpreted language and have it frequently converse with a database. If you think this will work, you are living in a magical dream world. I’m talking directly to you, Twitter, you poor sad whipping boy of the Web 2.0 universe. Please, for your own sake, rewrite Starling in C or C++ and use a more suitable back-end.

That concludes the synopsis of my multi-page rant of doom. Now, for the meta: if I were to write an essay for NPR’s This I Believe, the following would be that essay.

I believe in telling systems what I want, not how to get it, and having them give it to me as quickly as possible. I believe that programmers are lazy, and that middleware should give them the ability to do the right thing the easy way. I believe in intrinsic scalability and building on sound principles. I believe that the disk is evil and writing to it should be avoided until you have no other choice. I believe in most of what databases do and in the potential of what their descendant systems can and will do.

I believe in the awesome potential of automatic everything.

If you look closely, you’ll notice that the point cloud is really noisy around the edges. A simple high-pass filter later and that same frame looks like this:

That’s a little more manageable. I figured, why stop at point when you can have 3D surfaces? One of the more straightforward ways to make a 3D surface out of a bunch of points is to stick a bunch of triangles in between the points, creating what’s called a Delaunay triangulation. This is a really compute-intensive calculation and I don’t exactly have a supercomputer on hand, so I did a lot of fudging and approximation. Even with all that fudging, each of these frames took as much as 5 minutes to render. This process has been running for most of last week while I’ve been at work. That same frame above looks like this when Delaunay-triangulated:

Notice that it’s a little noisy, which is mainly due to some approximation on my part as well as some leftover noise in the point cloud. The video below shows what happens when you sequence all 2100 frames together. Enjoy!

If I want to download an update from Software Update today on my home computers, I have to do it three times - once for my Mac Mini (file server/backup server/media center), once for my laptop and once for my tower. The actual update binary is, in most cases, identical. If I wanted to only download the update once, I’d have to find where Software Update keeps the update’s installer file, copy it to the other machines and run it there. In some cases I have to download tens or hundreds of megabytes of file that could easily be transferred over my home network, saving both my time and the update provider’s money.

The thing that’s the most irritating about this is that it’s a completely solved problem. Blizzard, for example, distributes updates to World of Warcraft over Bittorrent. My roommate just started playing WoW again and had to install a patch (~2 GB) on two of his computers. He downloaded and installed the patch on the first computer, which took about an hour and a half. The download-and-install process for the second computer took all of about five minutes because the computer automatically recognized that a source for the update existed on its local network and downloaded the file peer-to-peer from the other machine.

Imagine if everyone interested in downloading the iPhone patch could download it not only from Apple but from each other. After the first few hundred downloads (which would have to pull directly from Apple) most of the remaining transfer would be peer-to-peer. If iTunes needs to authenticate the phone with Apple before installing, that’s fine; the load on the servers from authorization would be far lower and of a much shorter duration than the load from patch downloading. Security, of course, is an issue with Bittorrent-esque downloads, but there are relatively straightforward ways to deal with that.

I’m just saying it’s about time that someone did something about this, because it’s getting a little ridiculous.