MarkupServe – A DIY Evernote Alternative

I’ve gushed at length in the past about how much I love Evernote. I upgraded to Evernote Pro about half a year ago so that I could attach arbitrary file types to notes. After a while, though, a couple of things about my workflow with Evernote were starting to irk me.

First, I wasn’t even coming close to using Evernote Pro’s generous data cap. Second, I wasn’t sure that paying $5 a month so that I could attach a PDF to a note was worth it given how infrequently I was attaching PDFs to notes. Third, and maybe most critically, their WYSIWYG editor has always kind of bugged me; lists never really indent the right way, sometimes formatting leaks to the next paragraph in unpredictable ways, etc. It’s just really hard generally to get precision control over how your text looks in Evernote’s editor (at least on the Mac, I have no experience with the Windows editor). Its lack of Linux compatibility isn’t a problem for me anymore, but it might become a problem if I start working on Linux boxes again in the future.

I figured the “paying for space I don’t use” problem would be solved if I could find something that would sync with Dropbox. I have a couple GB of free space with them that is really underutilized and I would prefer using that space instead of paying for more underutilized space. For the “imprecise editing” problem, I decided that what I really wanted was the ability to write notes in a markup language like Markdown (which I also love and have gushed about here), do my writing in Emacs, and be able to view the notes in HTML easily.

I couldn’t find any pre-existing solution that I really liked, so I decided to roll my own.

Dropbox support wasn’t a problem; I just exported my notes from Evernote, used html2text to convert the exported notes into Markdown, and moved the folder containing the converted notes to my Dropbox folder, where it happily synchronized. Dropbox, you are awesome.

At first, I used Marked to render and view notes individually. Marked is a great app, but I wanted to be able to interact with rendered versions of my files more quickly. I had thought about writing up a read-only filesystem with FUSE that would do the rendering of Markdown to HTML transparently and just present a filesystem of HTML files, but that sounded like overkill even for me. I figured that writing up a simple web server with Bottle to display the rendered files would get me to a working solution much more quickly. After a couple hours of coding on Thanksgiving, I had MarkupServe working.

MarkupServe is pretty basic at this point. It’s given a directory tree containing markup files and presents that directory tree as a directory listing similar to the one httpd gives you if you don’t have an index page. It has simple keyword search (which just runs grep at the root of the directory tree and HTML-ifies the results) and renders notes to HTML on-the-fly when they’re clicked on. I made MarkupServe extensible enough that it should support more than just Markdown, in case others would find something like this useful but want to use other markup languages. It’s not the fastest or prettiest thing ever, but it works.

The last hurdle was making attaching files in Emacs comfortable. Evernote exports all attachments for a note note.html in a directory named note.resources, so I figured I’d stick to that convention. I made MarkupServe ignore directories that ended in .resources so that it wouldn’t clutter up the file listing. Then I wrote a couple little elisp functions that create a .resources directory for a note and “attach” a file by copying it into the appropriate .resources directory and inserting a Markdown-style link to the new copy. I’ve posted those functions in a GitHub gist if you’re interested in looking at those.

One big piece that this system is missing is a mobile app. Evernote’s iPhone app is terrific, and I’m going to miss it. At this point, my solution is to use Elements to edit notes and add photos taken with my phone’s camera to notes manually if the need arises. It’s sort of awkward, but I used the image attachment feature so infrequently in Evernote that I’m not really concerned about it. The system also lacks Evernote’s slick image OCR capability, but that was another feature I never really used (my handwriting’s pretty awful and the OCR could never really parse it well).

I’m sure I’ll tweak this setup considerably as I get more experience with it, but it was surprisingly quick to throw together and has proven really useful so far. Hopefully open-sourcing this stuff will help any other people who might have a similar itch to scratch.

Escape to New York

I got the opportunity to spend some time in New York a few weeks ago. Officially, I was there to attend Hadoop World, but I flew up a few days early to visit friends and see the city. I hadn’t been there since middle school (and I can’t really say I’ve been to New York if I was twelve years old and with a tour group). My friend Mangesh and his roommate graciously let me crash on their couch for a couple of days until my official (and reimbursable!) conference hotel room was available. They are wonderful people.

I had a couple of days to walk around and enjoy the city. Spent most of Sunday at the American Museum of Natural History. The museum was just as great as I remember it being back in 1997, and their newly-renovated planetarium is absolutely spectacular.

I spent most of Monday wandering around Manhattan. Since a lot of the touristy things like museums are closed on Mondays, I spent a good deal of time in Central Park and wandered around Times Square and Rockefeller Center. There’s this amazingly stark contrast between Central Park and the rest of Manhattan. The decision to put the park there rather than developing that land was a really inspired choice on the part of city planners; it was a great place to wander around and relax after spending so much time getting jostled on the streets. The weather was perfect while I was there, which was a lucky break considering that the weather is apparently pretty erratic there in November.

I spent Tuesday and Wednesday at Hadoop World, hobnobbing with fellow big data geeks and interested businesspeople, talking about the state of Hadoop and the big data landscape in general. Since I’m in this weird pre-thesis-writing-but-thinking-about-graduation state right now, I spent much of the time getting a feel for what the various companies in the big data space were up to and doing a bit of shameless evangelizing of our group’s work with Themis (our follow-on work that has grown out of TritonSort). My friend Yanpei (who is now wrapping up a Ph.D. at Cal) gave a great talk along with Todd Lipcon from Cloudera on measuring and improving Hadoop’s performance that you should check out if you’re interested in that kind of thing.

Tuesday night was spent at EMC/Greenplum’s bowling-for-charity event, in which I affirmed that a) I am not that great of a bowler and b) tech companies know how to throw a party. The attendees (with the help of sponsors) ended up raising over $20,000 for Artists for Elephants, which is also pretty cool. If the choice of charity sounds random, it’s worth noting that Hadoop’s mascot is a little yellow elephant. We are actively considering adopting an adorable mascot for the Themis project.

In general, my impression of New York was a lot more positive than I thought it would be. I’m not a huge fan of crowds, but for some reason Manhattan felt a lot less crowded than I imagined it would be, although there were still a ton of people everywhere. I can definitely see why people want to live in New York; there’s so much cultural diversity and so many things to do and see there. On the other hand, the ridiculously high cost of living (granted, my exposure to the city at this point has been Manhattan, which is probably not representative) and the constant crowds would kind of deter me from moving there.

I took a ton of photos during my trip that are up on Flickr.

tmux: the new hotness for terminal management

Some time ago I talked about using GNU Screen to effectively manage a bunch of terminal windows. Turns out screen has some serious competition: tmux.

One thing that’s pretty cool about tmux is the way it handles windows. tmux’s window management model is purely client-server; windows are clients, and clients are managed by a single server tmux process. This allows you to do things like move a window between sessions or attach the same window in multiple sessions. I haven’t run into a situation where this was useful yet, but it’s nice to know that it’s possible.

The thing that I have found useful is the fact that tmux is reasonably scriptable. In order to get the list of windows in a screen session, I had to create a dummy window, stuff the real window list into it, dump the contents of the dummy window to a file and parse that file. With tmux, I call

tmux list-windows

from any window in the session, or

tmux list-windows -t <session name>

outside the session, and parse the output. Not only that, but tmux displays none of the weird “I can only make calls that change a session once per second” problems that I’ve been seeing in practice with screen.

Like screen, tmux won’t auto-sort windows by name or make their numbering contiguous. My py-screnum tool for sorting and compacting Screen windows takes several seconds to re-arrange windows and is 95 lines of Python (according to sloccount). The analog for tmux, tmux-screnum, is only 57 lines, and re-arranges windows instantly. I won’t claim that either of these programs are minimal, but 50% fewer lines of code and faster is a winning combination.

There are times when I still have to use one or the other; many non-BSD machines come with screen installed by default but not tmux. I’m starting to prefer tmux whenever I have a choice, though.