Update (2011-04-03): Comcast’s user account pages now appear to require JavaScript, which makes it impossible to scrape the usage data using a simple script. As a result, this script no longer works.
Comcast has often advertised their high speed Internet service as providing “unlimited” data transfer, but when they say “unlimited”, what they really mean is “limited to 250GB a month”.
Just before the new year, Comcast finally rolled out a data usage meter to users in the Portland, Oregon area so we can actually tell when we’re in danger of exceeding that 250GB ceiling. I find this usage meter incredibly helpful in achieving my goal of using as much of my monthly 250GB data allotment as I possibly can. I feel it’s my duty to get my full money’s worth.
Unfortunately, the meter is buried several pages deep in Comcast’s account site, which is a slow and ugly beast that requires a login, several redirects, and a click or two. So I whipped up a little Ruby script to do the dirty work for me and just print out my current usage total.
Before using the script, you’ll need to install the Mechanize gem:
gem install mechanize
Here’s the script:
#!/usr/bin/env ruby
require 'rubygems'
require 'mechanize'
URL_LOGIN = 'https://login.comcast.net/login?continue=https://login.comcast.net/account'
URL_USERS = 'https://customer.comcast.com/Secure/Users.aspx'
abort "Usage: #{$0} <username> <password>" unless ARGV.length == 2
agent = Mechanize.new
agent.follow_meta_refresh = true
agent.redirect_ok = true
agent.user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'
login_page = agent.get(URL_LOGIN)
login_form = login_page.form_with(:name => 'signin')
login_form.user = ARGV[0]
login_form.passwd = ARGV[1]
redirect_page = agent.submit(login_form)
redirect_form = redirect_page.form_with(:name => 'redir')
abort 'Error: Login failed' unless redirect_form
account_page = agent.submit(redirect_form, redirect_form.buttons.first)
users_page = agent.get(URL_USERS)
usage_text = users_page.search("div[@class='usage-graph-legend']").first.content
puts usage_text.strip
Save it to an executable file (I called it capmon.rb), then run it like so, passing in your Comcast.net username and password (they’ll be sent securely over HTTPS):
./capmon.rb myusername mypass
The script will log into your Comcast account, go through all those painful redirects and clicks, and eventually spit out your usage stats, which will look something like this:
166GB of 250GB
Couldn’t be simpler! Naturally, this script won’t work for you unless you’re a Comcast customer in a region where the usage meter is currently available. Also, the script will break if Comcast changes their login flow or page structure, but I’ll try to keep this post updated if that happens.
This script is available as a GitHub gist as well. If you’d like to modify it and make it better, please fork the gist.
Comments
Availability
Apparently, Vancouver doesn’t fall into the “Portland Area” as defined by Comcast. Poo.
Nicely done.
This is awesome. Almost makes me want to switch to Comcast. And move to Portland.
Re: Nicely done.
Thanks!
I think Comcast has now rolled out the usage meter to all customers (or at least more customers), so presumably this script should work even outside of Portland now. Of course, I have no way of testing that.
Error
I keep getting this error c:/temp_comcast.rb:18:in `‘: undefined method `user=’ for nil:NilClass (NoMethodError) although sometimes it just works can anyone explain what is up?
Error
I fixed it.. it seems i was being presented a different page to log in sometimes.
oopsie
seems to have stopped working with some new changes to the page and login process
Re: oopsie
Looks like Comcast’s user account pages now require JavaScript. It appears to be impossible to access the usage meter without executing JS. Guess that’s the end of this script.
Re: oopsie
Darn it Comcast, this script was fantastic and now it’s ruined.
Damn!
Damn! Now my geektool layout is ruined. Ugh, I hate Comcast.
Usage meter dead
I had this script incorporated into a dashboard in my house and to send me email alerts, as well as kick off some automated tasks at certain thresholds. Noooooooooooooo. Now I have to find a new method to grab this data. Dumb ISP!
Node.js to the rescue?
@Ryan,
It might be possible to re-do the script with Zombie.js on node; since you could then run the JS code.
Re: Node.js to the rescue?
You read my mind as usual, Eric. I was just looking at zombie.js a little while ago (as evidenced by the Pinboard bookmark in the sidebar!). I’m in the midst of moving at the moment, but I plan to jump right on that as soon as I have free time again.
Re: Node.js to the rescue?
Partway there. I’ve got a Zombie.js script that should work, but it fails on the first connection with a “socket hang up” error. As far as I can tell, this may be a Node.js bug (even in 0.4.5).
The script in progress: https://gist.github.com/905116 . Anyone feel like helping debug it?
Re: Node.js to the rescue?
I’d love to help, but I’m useless when it comes to JavaScript.
https?
are you sure zombie support https? https support seems to be big on want lists
Re: https?
According to the Zombie.js Changelog, https has been supported since version 0.8.6.
Found a python script that works!
http://thehobbsfamily.net/arch ive2011/comcappy-comcast-inter net-usage-script/