wonko.com

I wasn’t at Yahoo! when Paul Graham was. He was there a long time ago. I can’t speak to whether his blog post accurately reflects what Yahoo! was like then. I can tell you that it doesn’t mesh at all with the experiences I’ve had at Yahoo! since I joined the company in early 2007.

The main point of Graham’s article is that Yahoo! didn’t have a hacker-centric culture. If there was a time when that was true, it must have been before I joined.

A company without a hacker-centric culture doesn’t encourage the kind of risk-taking and experimentation I saw when I was at Yahoo! Search. As an engineer, I had direct input into product features at every level, from ideation to design to implementation to launch. If I had a crazy idea, I was encouraged not just to tell people about it (up to and including executives), but to implement it and see if it tested well with users. I was able to add my own personal touch to parts of the product (sometimes big parts) without needing to ask permission or wade through excessive red tape.

This may not sound impressive to someone who’s used to the way things work at startups or small companies. But this was at one of the largest Internet companies in the world, on one of the most visited websites in the world. For Yahoo! to give me and other engineers the kind of freedom and power we had is not normal for a company or a product that operates at this scale.

Earlier this year, I transferred to the YUI team, where I get to work with some of the smartest frontend engineers on the planet on an open source JavaScript library that we develop not just for use by Yahoo!, but also by thousands of other developers around the world. The ideas and the work that I see coming from this team, and from the other teams we work with throughout Yahoo!, are amazing and often groundbreaking.

I have my gripes about Yahoo!, sure. It hasn’t been all kittens and rainbows. But the hacker-centric culture and the brilliant people Paul Graham seems to think don’t exist here are the reason I’ve been here for 3.5 years and counting, and they’re the reason I don’t plan on leaving anytime soon.

I originally wrote this as an answer to a question on Quora. I thought it was worth reposting here. My opinions, as always, are my own, and don’t necessarily reflect the views of my employer.

Like anyone with certain hot buzzwords on their résumé or LinkedIn profile, I’m often contacted by recruiters. Usually they’re perfectly friendly and polite: nice people doing an important and often thankless job. Sometimes they’re less polite, or use sketchy tactics.

Whenever I get annoyed by a recruiter, I try to remember how happy I would have been to have gotten a call from them back in the dark days of ought-one, when much of my time was spent sitting on a ratty couch eating Eggo waffles and watching Zoboomafoo with whichever of my roommates also happened to be unemployed at the time.

There are good recruiters and there are sketchy recruiters, but the worst kind of recruiter is an incompetent recruiter. Like Bjoern, who sent me the following email today.

From: bjoern@[redacted]
To: ryan@wonko.com
Subject: Silicon Explosion? // interesting startup opportunities

Email not displaying correctly? View it in your browser.

Already we’re off to a bad start. Bjoern is so certain of his own incompetence at such a simple task as formatting an email that, before even saying hello, he has offered me the opportunity to skip the email entirely and instead view a web page, presumably generated by some automated tool so foolproof that even he couldn’t screw it up. Confidence has not been instilled.

That’s one interpretation. Another interpretation is that the email itself was generated by an automated tool. Which makes it insulting. But for some reason I keep reading.

Hi Ryan,

You stumbled over your profile a couple days ago … I was impressed. Congratulations!

Just one word into the opening sentence, Bjoern has justified his earlier self-doubt.

I’m not sure exactly what Bjoern is accusing me of here—I swear I don’t remember stumbling over anything a couple days ago—but apparently an impressive profile was involved. So impressive that I deserve congratulating. Go me! I must be super awesome.

You probably noticed that recently there was an explosion of freshly funded startups out there looking for senior devs., tech VPs and CTOs.

We are working with a number of funded startups handson and help them to accelerate with funding, people, prototyping, media, going global, etc.

I don’t have the slightest idea why Bjoern thinks any of this is relevant to me, but hey, I’m impressed he managed to string together so many empty, meaningless words. And that “etc.” at the end totally seals the deal. It tells me that he’s doing so many incredible “handson” things for so many freshly funded startups that he can’t even be bothered to list them all. He must be almost as awesome as I am.

Hey, Bjoern? Just one thing. What is it with you and explosions?

Finding good people – as you probably know – is always the hardest part!

The hardest part? Of what? All those things he listed earlier? Maybe just the “people” part? Or did he mean the “etc.” part? I appreciate the implication that I’m smart enough to know what he’s talking about, but since I’m not sure he knows what he’s talking about, I feel uncomfortable jumping to conclusions.

As it turns out it is the best to ask good people for good people. Could you recommend anybody who is looking for a new opportunity? :)

Bjoern

Well shit. Bjoern had me all worked up about how awesome I was, and then he went and dashed my hopes. It wasn’t me he was interested in after all; it was my awesome friends!

Or maybe he’s actually interested in me and is just being really sneaky about it. Is that what the smiley means? Maybe that’s Bjoern’s way of saying, “Hey, I know your boss reads your personal email, so let’s pretend we’re talking about OTHER PEOPLE and not you. Got it? Wink wink!”

If my boss were reading my personal email, you’d think he’d be smart enough to at least delete the recruiter emails before I saw them.

But wait, there’s more good stuff after the signature.

You were recommended to me :)

Unsubscribe ryan@wonko.com from this list.

Our mailing address is:
[redacted]

Add us to your address book

Copyright © 2010 [redacted] All rights reserved.

Forward this email to a friend
Update your profile

I was recommended to him? But I thought he stumbled across my profile? Somebody stumbled across a profile, at any rate. Maybe the smiley after this sentence means that it, too, is code for something? But for what? Bjoern may have overestimated my intuitive abilities, because his encoded meaning is lost on me.

But then, at the very end, there’s a link to my profile. That must be the one someone stumbled over at the beginning of the email! Boy, I sure don’t remember creating a profile, so I’d better click that link and find out what’s going on!

Of course, clicking that link would route me through a tracking redirect and tell Bjoern that I read his message.

So, yeah. Let’s just not click that. Let’s not click that at all.

In my latest post on the YUI Blog, I describe the results of my recent research into mobile browser cache limits for Android, iOS (including iPhone 4) and webOS. Be sure to check it out if you develop websites for mobile browsers.

Steve Jobs describing Apple’s stance on privacy at the recent D8 Conference:

Privacy means people know what they’re signing up for, in plain English, and repeatedly. That’s what it means.

I’m an optimist. I believe people are smart, and some people want to share more data than other people do. Ask them. Ask them every time. Make them tell you to stop asking them if they get tired of you asking them. Let them know precisely what you’re going to do with their data. That’s what we think.

The privacy discussion begins at about 1:09:30 into the full video of the interview.

Failing at privacy is easy. Anyone can do it. I can tell you how in a single sentence:

To fail at privacy, make it an attainable state rather than the default state.

As luck would have it, the inverse is also true:

To succeed at privacy, make it the default state rather than an attainable state.

Surprisingly, succeeding at privacy is in many cases easier than failing at it, yet so many products still fail at it. Why? Because they choose to. Privacy and marketing are often incompatible.

If a website launches a new feature that’s disabled by default, most users won’t even notice it’s there and will never use it. So when Facebook, Google, Yahoo!, and other websites launch new features, they typically enable them by default and allow users to opt out.

The opt-out model achieves marketing objectives and dramatically increases user uptake of a new feature, but if the new feature involves changes to the handling of information that was previously private, achieving the marketing objectives may come at the cost of violating users’ expectations of privacy and betraying their trust.

With enough time and effort, you can almost always convince a user to try out a new feature. It’s much harder to convince a user to trust you again once they feel their trust has been betrayed.

Unfortunately, in an industry where success is typically measured by how many users have a feature enabled and not by how many users are actually using or deriving value from that feature, conservatism doesn’t pay. If you want the usage numbers, you have to make it hard for users not to use your stuff.

Someone with a more mathematical mind than mine could probably derive an algorithm to describe the typical level of privacy you can expect from any given company or product. A younger, newer company or product will typically err on the side of caution. They haven’t yet proven themselves, so user trust is extremely valuable to them; in addition, they typically have less market pressure to drive massive user uptake of new features, and they tend to communicate more directly with their users, which gives them a better idea of what their users want and need.

Larger, older companies and products have less to lose by betraying a user’s trust and have significantly more to lose by not meeting business objectives, particularly if the company is traded publicly. Products at larger companies sometimes suffer from design-by-committee, and the people at those companies may find it harder to keep in touch with their users’ wants and needs (not surprisingly, the wants and needs of shareholders are often given higher priority).

The value of a user’s trust is something that’s hard to measure, especially in an industry as young as the Internet industry. It’s something that isn’t apparent in the short term, but can slowly become very apparent over the long term. Betraying your users’ trust today might not have any noticeable effect on your company right away or even for a few years, but do it enough and it will eventually catch up with you.

This is a lesson that many Internet companies will soon begin learning the hard way. Some already are. Give yourself an advantage by learning from their mistakes: make privacy the default. Encourage your users to trust you, and reward them when they do by valuing their trust and treating it as your top priority. Think about the future, not the now.

Your users are far more valuable as advocates of your product than as numbers in a usage report.

A little over three years ago, I opened a purple box containing a job offer and some boring forms to sign. It yodeled at me when I opened it. That’s when I knew I’d made the right choice in deciding to join Yahoo!.

Working on Yahoo! Search has been an incredible experience. I can now say that I’ve written code that’s used by hundreds of millions of people around the world. I can also say that I’ve broken code that’s used by hundreds of millions of people around the world (fortunately that only happened once). During my time there I did a little bit of everything, from writing build tools and pushing pixels around for bucket tests to leading frontend development on the September 2009 Search redesign—the biggest in the history of Yahoo! Search—which the team managed to pull off in only a matter of months.

At its best, working on Search was exciting and fulfilling in a way no other job has been for me; at its worst, it was stressful and relentlessly demanding. But whether it was exciting or stressful, fulfilling or demanding, I always learned something new every day, and I got to work with some of the smartest people in the business. As far as I’m concerned, that’s what makes a thing worth doing.

It’s in this spirit that I’m launching myself on a new journey. Not an altogether different one—I’ll still be at Yahoo!—but certainly one that I expect will be both incredibly challenging and a lot of fun. As a member of the YUI team, I’ll get to work on an awesome project that I love, with awesome people from whom I will doubtless learn truly epic amounts of stuff on a daily basis.

A little over three years ago, if someone had told me I’d be this lucky, I wouldn’t have believed them.

Comcast has often advertised their high speed Internet service as providing “unlimited” data transfer, but when they say “unlimited”, what they really mean is “limited to 250GB a month”.

Just before the new year, Comcast finally rolled out a data usage meter to users in the Portland, Oregon area so we can actually tell when we’re in danger of exceeding that 250GB ceiling. I find this usage meter incredibly helpful in achieving my goal of using as much of my monthly 250GB data allotment as I possibly can. I feel it’s my duty to get my full money’s worth.

Unfortunately, the meter is buried several pages deep in Comcast’s account site, which is a slow and ugly beast that requires a login, several redirects, and a click or two. So I whipped up a little Ruby script to do the dirty work for me and just print out my current usage total.

Before using the script, you’ll need to install the Mechanize gem:

gem install mechanize

Here’s the script:

#!/usr/bin/env ruby

require 'rubygems'
require 'mechanize'

URL_LOGIN = 'https://login.comcast.net/login?continue=https://login.comcast.net/account'
URL_USERS = 'https://customer.comcast.com/Secure/Users.aspx'

abort "Usage: #{$0} <username> <password>" unless ARGV.length == 2

agent = Mechanize.new

agent.follow_meta_refresh = true
agent.redirect_ok = true
agent.user_agent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'

login_page = agent.get(URL_LOGIN)

login_form = login_page.form_with(:name => 'login-form')
login_form.user = ARGV[0]
login_form.passwd = ARGV[1]

redirect_page = agent.submit(login_form)
redirect_form = redirect_page.form_with(:name => 'redir')

abort 'Error: Login failed' unless redirect_form

account_page = agent.submit(redirect_form, redirect_form.buttons.first)

users_page = agent.get(URL_USERS)
usage_text = users_page.search("div[@class='usage-graph-legend']").first.content

puts usage_text.strip

Save it to an executable file (I called it capmon.rb), then run it like so, passing in your Comcast.net username and password (they’ll be sent securely over HTTPS):

./capmon.rb myusername mypass

The script will log into your Comcast account, go through all those painful redirects and clicks, and eventually spit out your usage stats, which will look something like this:

166GB of 250GB

Couldn’t be simpler! Naturally, this script won’t work for you unless you’re a Comcast customer in a region where the usage meter is currently available. Also, the script will break if Comcast changes their login flow or page structure, but I’ll try to keep this post updated if that happens.

This script is available as a GitHub gist as well. If you’d like to modify it and make it better, please fork the gist.

I’ve written an article on YUIBlog introducing my new Storage Lite YUI 3 Gallery module, which provides a lightweight, cross-browser client-side storage API without relying on any browser plugins. This is an updated version of the storage library I mentioned last year in my post about the development of Yahoo! Search Pad.

Head over to YUIBlog to read the article, and feel free to fork Storage Lite on GitHub and add awesomeness (or subtract crappiness).

The Net::IMAP standard library distributed with Ruby 1.8.6, 1.8.7, and 1.9.1 contains a response parsing bug that can cause an endless hang (in 1.8.x) or raise an exception (in 1.9.1) when switching between mailboxes on a Dovecot 1.2.x server.

The bug has been fixed in Ruby’s SVN trunk and should eventually make it into the 1.9.2 release, but if you’re using Net::IMAP with a current or older Ruby release and need a fix for this, the following monkeypatch (which just replaces the old buggy method with the fixed one from SVN) should do the trick.

Fortunately, this fix is the only difference from the 1.8.6, 1.8.7, and 1.9.1 versions of this method, so the monkeypatch works for all three versions. Just add it to your own code at some point after requiring Net::IMAP.

if RUBY_VERSION <= '1.9.1'
  module Net # :nodoc:
    class IMAP # :nodoc:
      class ResponseParser # :nodoc:
        private

        # This monkeypatched method is the one included in Ruby SVN trunk as
        # of 2010-02-08.
        def resp_text_code
          @lex_state = EXPR_BEG
          match(T_LBRA)
          token = match(T_ATOM)
          name = token.value.upcase
          case name
          when /\A(?:ALERT|PARSE|READ-ONLY|READ-WRITE|TRYCREATE|NOMODSEQ)\z/n
            result = ResponseCode.new(name, nil)
          when /\A(?:PERMANENTFLAGS)\z/n
            match(T_SPACE)
            result = ResponseCode.new(name, flag_list)
          when /\A(?:UIDVALIDITY|UIDNEXT|UNSEEN)\z/n
            match(T_SPACE)
            result = ResponseCode.new(name, number)
          else
            token = lookahead
            if token.symbol == T_SPACE
              shift_token
              @lex_state = EXPR_CTEXT
              token = match(T_TEXT)
              @lex_state = EXPR_BEG
              result = ResponseCode.new(name, token.value)
            else
              result = ResponseCode.new(name, nil)
            end
          end
          match(T_RBRA)
          @lex_state = EXPR_RTEXT
          return result
        end
      end

    end
  end
end

If you’re a Larch user, the latest Larch development gem includes this fix.

Version 1.2.0 of Sanitize, my whitelist-based HTML sanitizing library for Ruby, is now available. Consult the HISTORY file for a complete list of changes.

Introducing Transformers

This release adds a major new feature called transformers. Transformers allow you to filter and alter HTML nodes using your own custom logic, on top of (or instead of) Sanitize’s core filter. A transformer is any Ruby object that responds to call() (such as a lambda or proc) and returns either nil or a Hash containing certain optional response values.

To use one or more transformers, pass them to the :transformers config setting:

Sanitize.clean(html, :transformers => [transformer_one, transformer_two])

Input

Each registered transformer’s call() method will be called once for each element node in the HTML, and will receive as an argument an environment Hash that contains Sanitize config information and a reference to a Nokogiri::XML::Node object.

The transformer has full access to the Nokogiri::XML::Node that’s passed into it and to the rest of the document via the node’s document() method. Any changes will be reflected instantly in the document and passed on to subsequently-called transformers and to Sanitize itself. A transformer may even call Sanitize internally to perform custom sanitization if needed.

Transformers have a tremendous amount of power, including the power to completely bypass Sanitize’s built-in filtering.

Output

A transformer may return either nil or a Hash. A return value of nil indicates that the transformer does not wish to act on the current node in any way. A returned Hash may contain instructions that tell Sanitize to whitelist certain attributes or nodes, or to replace the current node with a new node (see the README for specifics).

Example: Transformer to whitelist YouTube video embeds

The following example demonstrates how to create a Sanitize transformer that will safely whitelist valid YouTube video embeds without having to blindly allow other kinds of embedded content, which would be the case if you tried to do this by just whitelisting all <object>, <embed>, and <param> elements:

lambda do |env|
  node      = env[:node]
  node_name = node.name.to_s.downcase
  parent    = node.parent

  # Since the transformer receives the deepest nodes first, we look for a
  # <param> element or an <embed> element whose parent is an <object>.
  return nil unless (node_name == 'param' || node_name == 'embed') &&
      parent.name.to_s.downcase == 'object'

  if node_name == 'param'
    # Quick XPath search to find the <param> node that contains the video URL.
    return nil unless movie_node = parent.search('param[@name="movie"]')[0]
    url = movie_node['value']
  else
    # Since this is an <embed>, the video URL is in the "src" attribute. No
    # extra work needed.
    url = node['src']
  end

  # Verify that the video URL is actually a valid YouTube video URL.
  return nil unless url =~ /^http:\/\/(?:www\.)?youtube\.com\/v\//

  # We're now certain that this is a YouTube embed, but we still need to run
  # it through a special Sanitize step to ensure that no unwanted elements or
  # attributes that don't belong in a YouTube embed can sneak in.
  Sanitize.clean_node!(parent, {
    :elements   => ['embed', 'object', 'param'],
    :attributes => {
      'embed'  => ['allowfullscreen', 'allowscriptaccess', 'height', 'src', 'type', 'width'],
      'object' => ['height', 'width'],
      'param'  => ['name', 'value']
    }
  })

  # Now that we're sure that this is a valid YouTube embed and that there are
  # no unwanted elements or attributes hidden inside it, we can tell Sanitize
  # to whitelist the current node (<param> or <embed>) and its parent
  # (<object>).
  {:whitelist_nodes => [node, parent]}
end

For more details on transformers, consult the README.

Installing

To install or upgrade Sanitize via RubyGems, run:

gem install sanitize