Update (2011-04-03): Comcast’s user account pages now appear to require JavaScript, which makes it impossible to scrape the usage data using a simple script. As a result, this script no longer works.

Comcast has often advertised their high speed Internet service as providing “unlimited” data transfer, but when they say “unlimited”, what they really mean is “limited to 250GB a month”.

Just before the new year, Comcast finally rolled out a data usage meter to users in the Portland, Oregon area so we can actually tell when we’re in danger of exceeding that 250GB ceiling. I find this usage meter incredibly helpful in achieving my goal of using as much of my monthly 250GB data allotment as I possibly can. I feel it’s my duty to get my full money’s worth.

Unfortunately, the meter is buried several pages deep in Comcast’s account site, which is a slow and ugly beast that requires a login, several redirects, and a click or two. So I whipped up a little Ruby script to do the dirty work for me and just print out my current usage total.

Before using the script, you’ll need to install the Mechanize gem:

gem install mechanize

Here’s the script:

#!/usr/bin/env ruby

require 'rubygems'
require 'mechanize'

URL_LOGIN = 'https://login.comcast.net/login?continue=https://login.comcast.net/account'
URL_USERS = 'https://customer.comcast.com/Secure/Users.aspx'

abort "Usage: #{$0} <username> <password>" unless ARGV.length == 2

agent = Mechanize.new

agent.follow_meta_refresh = true
agent.redirect_ok = true
agent.user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'

login_page = agent.get(URL_LOGIN)

login_form = login_page.form_with(:name => 'signin')
login_form.user = ARGV[0]
login_form.passwd = ARGV[1]

redirect_page = agent.submit(login_form)
redirect_form = redirect_page.form_with(:name => 'redir')

abort 'Error: Login failed' unless redirect_form

account_page = agent.submit(redirect_form, redirect_form.buttons.first)

users_page = agent.get(URL_USERS)
usage_text = users_page.search("div[@class='usage-graph-legend']").first.content

puts usage_text.strip

Save it to an executable file (I called it capmon.rb), then run it like so, passing in your Comcast.net username and password (they’ll be sent securely over HTTPS):

./capmon.rb myusername mypass

The script will log into your Comcast account, go through all those painful redirects and clicks, and eventually spit out your usage stats, which will look something like this:

166GB of 250GB

Couldn’t be simpler! Naturally, this script won’t work for you unless you’re a Comcast customer in a region where the usage meter is currently available. Also, the script will break if Comcast changes their login flow or page structure, but I’ll try to keep this post updated if that happens.

This script is available as a GitHub gist as well. If you’d like to modify it and make it better, please fork the gist.

The Net::IMAP standard library distributed with Ruby 1.8.6, 1.8.7, and 1.9.1 contains a response parsing bug that can cause an endless hang (in 1.8.x) or raise an exception (in 1.9.1) when switching between mailboxes on a Dovecot 1.2.x server.

The bug has been fixed in Ruby’s SVN trunk and should eventually make it into the 1.9.2 release, but if you’re using Net::IMAP with a current or older Ruby release and need a fix for this, the following monkeypatch (which just replaces the old buggy method with the fixed one from SVN) should do the trick.

Fortunately, this fix is the only difference from the 1.8.6, 1.8.7, and 1.9.1 versions of this method, so the monkeypatch works for all three versions. Just add it to your own code at some point after requiring Net::IMAP.

if RUBY_VERSION <= '1.9.1'
  module Net # :nodoc:
    class IMAP # :nodoc:
      class ResponseParser # :nodoc:
        private

        # This monkeypatched method is the one included in Ruby SVN trunk as
        # of 2010-02-08.
        def resp_text_code
          @lex_state = EXPR_BEG
          match(T_LBRA)
          token = match(T_ATOM)
          name = token.value.upcase
          case name
          when /\A(?:ALERT|PARSE|READ-ONLY|READ-WRITE|TRYCREATE|NOMODSEQ)\z/n
            result = ResponseCode.new(name, nil)
          when /\A(?:PERMANENTFLAGS)\z/n
            match(T_SPACE)
            result = ResponseCode.new(name, flag_list)
          when /\A(?:UIDVALIDITY|UIDNEXT|UNSEEN)\z/n
            match(T_SPACE)
            result = ResponseCode.new(name, number)
          else
            token = lookahead
            if token.symbol == T_SPACE
              shift_token
              @lex_state = EXPR_CTEXT
              token = match(T_TEXT)
              @lex_state = EXPR_BEG
              result = ResponseCode.new(name, token.value)
            else
              result = ResponseCode.new(name, nil)
            end
          end
          match(T_RBRA)
          @lex_state = EXPR_RTEXT
          return result
        end
      end

    end
  end
end

If you’re a Larch user, the latest Larch development gem includes this fix.

I bought a Mac Mini last week to use as an HTPC running Plex (replacing a noisy old PC laptop running MediaPortal), and I couldn’t be happier with it. I’ve cobbled together quite a few HTPCs over the years, and they generally left much to be desired. The Mini blows them all out of the water for a pretty reasonable price.

What I bought

• Mac Mini (2.0GHz Intel Core 2 Duo, 2GB RAM, 120GB HD) – $649
• Mini DVI to HDMI adapter – $15.85

I went for the smallest hard drive in the Mini since I stream my video from a Drobo. If you don’t plan to use an external drive or a NAS, I’d recommend getting a bigger drive.

What I already had lying around

• Apple Remote – $19.00
• TOSLINK optical cable with mini adapter – $10.35
• HDMI cable – $5.95

Putting it all together

1. Plug the Mini into the wall and the HDMI and TOSLINK cables into the Mini (via the aforementioned adapters). Plug the other ends into your receiver or TV.
2. Boot up OS X. You’ll want a keyboard and mouse for this part, and possibly a monitor if OS X doesn’t automatically figure out a good resolution to use for your TV.
3. Configure the display settings to match your TV’s native resolution. Check your TV’s manual for this number or, if you can’t find it, go for a standard HDTV resolution like 1280×720 (720p) or 1920×1080 (1080i/p). If OS X doesn’t list your TV’s native resolution (my Vizio LCD’s is 1366×768), you may need to use SwitchResX to get it working, but it’s not hard.
4. Install and configure Plex.

You may have noticed that I haven’t mentioned anything about a TV tuner. That’s because there’s really not much point to having one these days. I download all my TV off the Internets. It’s so much more convenient.

If, like me, you want to stream HD video over an 802.11n network, I strongly recommend setting Plex’s Local Network Video Cache to 16MB. There’ll be a few seconds of buffering when you start a video, but it’ll save you from annoying pauses and dropouts if the signal quality fluctuates while you’re watching something.

If you want to stream HD video over anything less than 802.11n, you’re probably going to be disappointed.

Once you’ve got Plex installed, be sure to check out the Plex App Store, where you’ll find apps that seamlessly integrate Hulu, YouTube, and other awesome content into Plex. There’s even an app that will stream recorded content directly from a TiVo if you have one. Despite the name, all the apps in the App Store are free, just like Plex itself.

I’ve already watched a ridiculous number of hours of video on this thing, and I couldn’t be more pleased with it. If you’ve been looking for an awesome, relatively inexpensive, easy to set up HTPC, I highly recommend the Mac Mini and Plex.

My coworker Stoyan Stefanov wrote a helpful blog post a few weeks ago describing how to create a simple TextMate bundle that allows you to quickly run the current file through JSLint. I’ve extended Stoyan’s bundle command to prettify the JSLint output for display in an HTML window.

Here’s what the ouput looks like (click for full size):

To use this command, just follow the instructions in Stoyan’s blog post using the script below in place of his bundle command, then select “Show as HTML” from the Output dropdown below the command edit box.

#!/usr/bin/env ruby
require 'cgi'

lint = `java org.mozilla.javascript.tools.shell.Main ~/Library/JSLint/jslint.js "$TM_FILEPATH"`

lint.gsub!(/^(Lint at line )(\d+)(.+?:)(.+?)\n(?:(.+?)\n\n)?/m) do
  "<p><strong>#{CGI.escapeHTML($1)}<a href=\"txmt://open?url=file://TM_FILEPATH&line=#{CGI.escapeHTML($2)}\">#{CGI.escapeHTML($2)}</a>#{CGI.escapeHTML($3)}</strong>#{CGI.escapeHTML($4)}" <<
    ($5 ? "<pre>#{CGI.escapeHTML($5)}</pre>" : '')
end

lint.gsub!(/^(jslint:)(.+?)$/, '<p><strong>\1</strong>\2</p>')
lint.gsub!(/TM_FILEPATH/, ENV['TM_FILEPATH']) 

print <<HTML
<!doctype>
<html>
<head>
  <style type="text/css">
    p { margin-bottom: 0; }
    pre {
      background: #f5f5f5;
      border: 1px solid #cfcfcf;
      font-size: 12px;
      margin-top: 2px;
      padding: 2px 4px;
    }
  </style>
</head>
<body>
  #{lint}
</body>
</html>
HTML

Update (2009-05-07): Added line number linkage courtesy of Steve Spencer.

Now that Ruby 1.9.1 is out, a lot of Ruby developers are going to want to run it alongside a 1.8.x build to test their apps. Here’s how to compile and install Ruby 1.9.1 alongside an existing Ruby installation on Mac OS X Leopard without disturbing that installation or any gems.

Note: You’ll need to have Xcode installed, since it provides the development tools necessary for the compilation step. If you don’t already have Xcode, you can install it from your Mac OS X installation DVD or download it from Apple for free.

First, download and extract the release:

curl ftp://ftp.ruby-lang.org/pub/ruby/ruby-1.9.1-p0.tar.bz2 -o ruby-1.9.1-p0.tar.bz2
tar xjf ruby-1.9.1-p0.tar.bz2

Next, compile Ruby, specifying a suffix for the binaries. This will create binaries named ruby19, irb19, gem19, etc., which will coexist peacefully with the stable ruby, irb, and gem binaries:

cd ruby-1.9.1-p0/
autoconf
./configure --program-suffix=19 --enable-shared --with-readline-dir=/usr/local
make && sudo make install

That’s all there is to it.

Update: Updated to resolve potential readline-related errors.

Update 2: The stable release of 1.9.1 is now out, so I’ve updated the instructions accordingly.

Everyone and their dog seems to have written an example of how to use the Yahoo! Search BOSS API to build a simple search tool. I wanted to take that one step further and build something that would serve both as an example and as a usable service, and that could be extended and enhanced by other developers. Since my day job involves constant tradeoffs between making Yahoo! Search slower (by adding features) and making it faster (by optimizing those features), my primary goal here was to make something as fast as technically possible.

To do this, I wrote a very simple JavaScript module called Trogdor that uses dynamic script nodes to make cross-domain JSONP requests to the BOSS API as you type your query. Search results are returned and rendered almost instantly on each keystroke, and you can use the up and down arrow keys (or tab) and enter to quickly select the result you want—no mouse necessary.

Trogdor doesn’t require a JavaScript framework and works great in all modern browsers (and even some ancient, crappy browsers like IE6). The entire package (HTML, CSS and JS) weighs just a smidge under 2KB after minification and gzip, and it’s wonderfully fast.

Try it out for yourself at pieisgood.org/search and be sure to grab the heavily-commented source code on GitHub. If you’ve got ideas for features and improvements, fork the repo and go nuts (and be sure to let me know what you come up with). You’re also welcome to use Trogdor (modified or unmodified) in your own projects, although I do ask that you please use your own BOSS API key rather than the one included in the example.

Update, 11/26: Changed the name of the library from FastSearch to Trogdor, since dragons are awesome (and apparently there’s a Microsoft search product called FastSearch).

The YUI Get Utility makes it easy to dynamically load CSS and JavaScript on demand, but there’s one tiny gotcha: Firefox, Safari, and other Gecko/WebKit-based browsers don’t provide a reliable way to tell when a CSS file has finished loading. As a result, when you load CSS in one of these browsers using the Get.css() method, the onSuccess callback is fired instantly.

In many cases this isn’t a problem, but if you need to ensure that you don’t perform an action (such as appending HTML to the page) until after the CSS has loaded, this can cause a race condition. There’s no way for the Get Utility to automatically work around this problem, but with a little bit of extra effort on our part, we can work around it ourselves.

The trick is to include a rule at the end of the dynamically-loaded CSS file that will send a message to our script by setting a style property on a special element. Once we see that property change, we know the CSS has finished loading.

First, the contents of our CSS file:

/* ... imagine lots of other CSS here ... */

#css-done { display: none; }

And now the JavaScript that will load it:

(function () {
  var Y = YAHOO;

  Y.util.Get.css('example.css', {
    onSuccess: function () {
      var el, poll;

      function finished() {
        alert('CSS has finished loading.');
      }

      if (Y.env.ua.gecko || Y.env.ua.webkit) {
        el    = document.body.appendChild(document.createElement('div'));
        el.id = 'css-done';

        poll = function () {
          if (Y.util.Dom.getStyle(el, 'display') === 'none') {
            // Once the element's display property changes, we know the CSS has
            // finished loading.
            el.parentNode.removeChild(el);
            finished();
          } else {
            // The element's display property hasn't changed yet, so call this
            // function again in 50ms.
            setTimeout(poll, 50);
          }
        };

        poll();
      } else {
        finished();
      }
    }
  });
})();

Here’s how to get the latest version of MySQL (5.0.51b as of this writing) working with the MySQL/Ruby gem and the Apple-supported version of Ruby that comes with Mac OS X Leopard. These instructions are scattered around the Internets in various places, but I’m sick of searching for them every time I need them so I’m compiling them here in a single place for convenience.

First, download and install MySQL for Mac OS X 10.5 (x86). Don’t install the x86_64 build or Ruby will refuse to speak to it. If you’ve already installed the x86_64 build, backup your databases, install the x86 build on top of it, and restore your databases.

Once you’ve got the correct build of MySQL installed, pop open a terminal and run the following to install the MySQL/Ruby gem:

sudo env ARCHFLAGS="-arch i386" gem install mysql -- \
  --with-mysql-dir=/usr/local/mysql --with-mysql-lib=/usr/local/mysql/lib \
  --with-mysql-include=/usr/local/mysql/include

That should do the trick.

One of the awesome things about getting to work on Yahoo! Search Assist is that I got to toss a few extra little features in there just for myself. I haven’t seen these mentioned in the press or blog coverage, possibly because nobody’s noticed them yet.

Quick access to the search box

On search.yahoo.com, the cursor is automatically placed in the search box when the page is loaded, but on the search result page, we don’t automatically give focus to the search box because that would prevent you from scrolling using the keyboard.

So, what if you want to edit your query without moving your hand all the way from the keyboard to the mouse to click on the search box? Well, now it’s easy: after the page loads, just press tab. Voila, cursor in search box, ready to edit, no mouse necessary.

Quick access to the Search Assist tray

By default, Search Assist isn’t displayed unless you appear to be having difficulty deciding what to search for. It does this by analyzing your typing speed and noticing when you pause longer than usual. Sometimes, though, you want to see suggestions right away without waiting. You can use the mouse to click on the little arrow beneath the search box, but that requires more pesky hand movement. Luckily, there’s another option.

When the cursor is in the search box, you can simply press the down arrow on your keyboard to expand the tray. If you decide you don’t want the tray after all, hit the escape key to close it again. As has been mentioned elsewhere, you can also use the arrow keys to select suggestions and concepts.

Mouse wheel scrolling

If you’re a mouse person and aren’t fond of having to click the tiny arrows to scroll the lists of suggestions and concepts, just move your cursor over the list you want to scroll and use the mouse wheel. When the cursor isn’t over a suggestion list, the mouse wheel causes the entire page to scroll, but when the cursor is over a list, the mouse wheel only scrolls that list.

A little something extra

I can’t take credit for this one, but it’s one of my favorite new features nonetheless: Yahoo! Search goes up to 11. Google only goes to 10. You do the math.