Automatic Everything

This was going to be an extended rant on how people use databases where people shouldn’t use databases, but the more I wrote the more I realized that this had been analyzed quite a bit by many in the systems research community and blogosphere at large, many members of which are far more knowledgeable than I. So I’ll summarize my rant in a paragraph and then move onto more philosophical, “meta”-type comments.

Twitter’s architecture (as much as they’ve shown us) is a Ruby on Rails app backed by a MySQL database. This combination is the Golden Hammer of Web 2.0. A frighteningly large number of web application developers seem to follow the mantra, “If I need to store data, use SQL as a Big-Ass Table (no, not that Big-Ass Table). Who needs high-speed middleware? I’ll write everything in Ruby!” The problem is that schema design is as close to alchemy as CS gets and tuning databases is tedious and hard to do right. If you are writing something that must process tens of thousands of messages a day, do not think you can write it in an interpreted language and have it frequently converse with a database. If you think this will work, you are living in a magical dream world. I’m talking directly to you, Twitter, you poor sad whipping boy of the Web 2.0 universe. Please, for your own sake, rewrite Starling in C or C++ and use a more suitable back-end.

That concludes the synopsis of my multi-page rant of doom. Now, for the meta: if I were to write an essay for NPR’s This I Believe, the following would be that essay.

I believe in telling systems what I want, not how to get it, and having them give it to me as quickly as possible. I believe that programmers are lazy, and that middleware should give them the ability to do the right thing the easy way. I believe in intrinsic scalability and building on sound principles. I believe that the disk is evil and writing to it should be avoided until you have no other choice. I believe in most of what databases do and in the potential of what their descendant systems can and will do.

I believe in the awesome potential of automatic everything.

Messing with Thom Yorke’s head

Radiohead’s latest music video was shot without cameras. Instead, they used a combination of reflected light and lasers to generate clouds of points in 3D. Google was nice enough to provide the rest of the world with some of the 3D point cloud data collected for that music video. A big piece of that data is about 2100 frames of lead singer Thom Yorke’s head. A frame of the original data (when output via Processing) looks like this:

If you look closely, you’ll notice that the point cloud is really noisy around the edges. A simple high-pass filter later and that same frame looks like this:

That’s a little more manageable. I figured, why stop at point when you can have 3D surfaces? One of the more straightforward ways to make a 3D surface out of a bunch of points is to stick a bunch of triangles in between the points, creating what’s called a Delaunay triangulation. This is a really compute-intensive calculation and I don’t exactly have a supercomputer on hand, so I did a lot of fudging and approximation. Even with all that fudging, each of these frames took as much as 5 minutes to render. This process has been running for most of last week while I’ve been at work. That same frame above looks like this when Delaunay-triangulated:

Notice that it’s a little noisy, which is mainly due to some approximation on my part as well as some leftover noise in the point cloud. The video below shows what happens when you sequence all 2100 frames together. Enjoy!

This is the future – why are my updates still failing?

So anyone who has an iPhone or iPod Touch will be pretty aware that Apple’s update servers basically fell over in response to all the demand today due to the new iPhone firmware. Recently, Firefox’s update servers suffered exactly the same problem. Now I’m sure that these guys have a really expensive load balancer in front of their update server cluster, but why in the world are so many major companies still having all their users go to a single place for updates?

If I want to download an update from Software Update today on my home computers, I have to do it three times - once for my Mac Mini (file server/backup server/media center), once for my laptop and once for my tower. The actual update binary is, in most cases, identical. If I wanted to only download the update once, I’d have to find where Software Update keeps the update’s installer file, copy it to the other machines and run it there. In some cases I have to download tens or hundreds of megabytes of file that could easily be transferred over my home network, saving both my time and the update provider’s money.

The thing that’s the most irritating about this is that it’s a completely solved problem. Blizzard, for example, distributes updates to World of Warcraft over Bittorrent. My roommate just started playing WoW again and had to install a patch (~2 GB) on two of his computers. He downloaded and installed the patch on the first computer, which took about an hour and a half. The download-and-install process for the second computer took all of about five minutes because the computer automatically recognized that a source for the update existed on its local network and downloaded the file peer-to-peer from the other machine.

Imagine if everyone interested in downloading the iPhone patch could download it not only from Apple but from each other. After the first few hundred downloads (which would have to pull directly from Apple) most of the remaining transfer would be peer-to-peer. If iTunes needs to authenticate the phone with Apple before installing, that’s fine; the load on the servers from authorization would be far lower and of a much shorter duration than the load from patch downloading. Security, of course, is an issue with Bittorrent-esque downloads, but there are relatively straightforward ways to deal with that.

I’m just saying it’s about time that someone did something about this, because it’s getting a little ridiculous.

Working for The Man

Here’s something you never thought you’d be hearing from me: I’m working for Microsoft this summer.

Well, I’m working for Microsoft Research this summer, anyway.

Wait, put out those torches, put down the pitchforks and let me explain.

Some of you may know that I’ve been somewhat … critical … of Microsoft in the past. My anti-Microsoft sentiments have mellowed somewhat in recent years, however, The XBox 360 may have been to blame for that, and my hatred of Windows has ebbed since I realized that building operating systems is hard and keeping them working is even harder. Also, it’s no longer my job to fix PCs running Windows (thank goodness) so I haven’t seen the Windows installations of security-casual college students in a while.

So why MSR? Microsoft Research is where a lot of the interesting corporate systems research is happening right now, and the project I’ll be working on is really closely related to my current research. I’ll get to spend three months in the Seattle area during the three months that the area’s weather is really nice. Compared to what I would make at UCSD in three months it’s a huge chunk of cash. It’s a huge stack of win all the way around.

Cure for a scratchy speaker

I had a problem with my speakers for a while, and it was getting on my nerves so much today that I finally sat down and tried to fix it. Essentially, when you adjusted the volume on these speakers, sound would cut in and out on the left speaker unless you had it set on just the right volume or you jiggled the knob for a few minutes. Even after lots of jiggling and turning, the knob was picky; every once in a while the left speaker would just  sort of stop working (and you’d wonder if you’d suddenly gone deaf in one ear).I had figured from the beginning that the problem was the knob itself. The knob is just a potentiometer (an electronic doodad that variably limits current flow based on the position of the knob), and sometimes the contact between parts of a potentiometer can get gunked up, causing the entire works to go crazy – at least that’s what the Internet tells me. Love the technical explicitness of that last paragraph? Blame the fact that I hated my electronics course in college.

Anyway, I called up my dad and asked for his advice on the subject. I was ready to rip the speaker open and spray lubricant on the knob, which may or may not have been a good idea. He suggested just turning the knob through its entire range of motion about 50 times. It turns out that worked like a charm – the speakers are as responsive as the day I bought them. Rocket scientist dad for the win!

Leopard

Mac OS 10.5 is here!

First off, a warning. Don’t buy the Leopard Family Pack. I did, because I’m a sucker. I paid an extra $100 for a sticker on my box and an extra word on my receipt. Seriously, Apple, at least give me a CD key or some paper authorization of my extra licenses. Don’t just give me a sticker. I can fake a sticker.

Here are some of the things I really like about Leopard:

• NetInfo Manager is gone! I hated that program so much. It’s been replaced by about 3 different control panels, but things are where you’d expect them to be instead of in an obscure, atrociously ugly utility.
• Frostier frosted glass menus! Eat it, Vista! 
• Terminal finally has tabs. iChat also finally has tabs. It’s a tab fiesta.
• Although Screen Sharing uses VNC, it’s not glacially slow; their compression algorithm isn’t half bad.
• Based on my benchmarks Spotlight is exactly a gajillion times faster than it was in Tiger.
• Believe it or not, Leopard actually runs really well on my 4-and-a-half-year-old Powerbook G4. I am really surprised.
• Apparently the OpenGL drivers are much improved, although I don’t really run games anymore.
• NFS mounting (and auto-mounting for that matter) is no longer a gross hack. Hooray for nfs://
• It’s shiny. Wait, you probably already knew that … but oh man it’s shiny.

I had this problem with my Macbook Pro that basically involved my keyboard suddenly not working for minutes at a stretch, but a clean reinstall seems to have solved that problem (fingers crossed on that one). I haven’t tried Time Machine yet, but I’m looking forward to it.