Designing a flexible framework is tricky. Most framework users are blinded by choice and would rather just connect the dots. And boy do they hate it if the dots don’t connect up to create exactly the picture they want.

Framework designers need to understand this and at least provide clearly defined guideposts for the users who want to connect the dots, while also making it possible for more adventurous users to blaze their own trails without too much effort.

Framework users should strive to see beyond their immediate goals and look at the ways the framework can make their work easier, not just the ways it can do their work for them.

I recently needed a quick and easy way to minify CSS and JS for the new YUI Library website (launching soon!). In the past I’ve written powerful and complicated tools for doing static asset management and minification, but this time I wanted something simple.

A good old-fashioned makefile turned out to be the perfect tool for the job. Here’s what I came up with. Feel free to use it in your own projects. This version requires the YUI Compressor, but that can easily be replaced with Closure Compiler, Uglify, or any other tool of your choice.

# Patterns matching CSS files that should be minified. Files with a -min.css
# suffix will be ignored.
CSS_FILES = $(filter-out %-min.css,$(wildcard \
	public/css/*.css \
	public/css/**/*.css \
))

# Patterns matching JS files that should be minified. Files with a -min.js
# suffix will be ignored.
JS_FILES = $(filter-out %-min.js,$(wildcard \
	public/js/*.js \
	public/js/**/*.js \
))

# Command to run to execute the YUI Compressor.
YUI_COMPRESSOR = java -jar yuicompressor-2.4.6.jar

# Flags to pass to the YUI Compressor for both CSS and JS.
YUI_COMPRESSOR_FLAGS = --charset utf-8 --verbose

CSS_MINIFIED = $(CSS_FILES:.css=-min.css)
JS_MINIFIED = $(JS_FILES:.js=-min.js)

# target: minify - Minifies CSS and JS.
minify: minify-css minify-js

# target: minify-css - Minifies CSS.
minify-css: $(CSS_FILES) $(CSS_MINIFIED)

# target: minify-js - Minifies JS.
minify-js: $(JS_FILES) $(JS_MINIFIED)

%-min.css: %.css
	@echo '==> Minifying $<'
	$(YUI_COMPRESSOR) $(YUI_COMPRESSOR_FLAGS) --type css $< >$@
	@echo

%-min.js: %.js
	@echo '==> Minifying $<'
	$(YUI_COMPRESSOR) $(YUI_COMPRESSOR_FLAGS) --type js $< >$@
	@echo

# target: clean - Removes minified CSS and JS files.
clean:
	rm -f $(CSS_MINIFIED) $(JS_MINIFIED)

# target: help - Displays help.
help:
	@egrep "^# target:" Makefile

To use this, save it as a makefile, customize it as necessary, and then run make minify to minify your .js and .css files. Minified files will be saved with a -min suffix alongside the originals. Only files that have changed since the last time you minified them will be processed.

This file is also available as a gist if you’d like to fork it and improve it. Enjoy!

Version 1.1.0 of Larch is now available.

Larch is a command line tool for copying email quickly and reliably from one IMAP server to another. It’s smart enough not to copy messages that already exist on the destination server, robust enough to pick up where it left off when interrupted, and also capable of more advanced operations like syncing message flags from the source server to the destination server. Larch is particularly well-suited for copying messages to, from, and between Gmail and Google Apps accounts.

Installing

Larch is a Ruby application and requires Ruby 1.8.6 or higher (1.9.2 is recommended). Once Ruby is installed, install or upgrade Larch via RubyGems:

gem install larch

What’s new

This release is the culmination of over a year of bug fixes and feature development. Larch is now faster and more reliable than ever. The most significant new features are:

This is only a partial list of changes. For the complete list, see the HISTORY file. For more details on how to use Larch’s new features, see the comprehensive README.

Try it out, report bugs

Please use Larch’s GitHub issue tracker to report bugs and request features. If you have questions or need help with Larch, the first thing you should do is read the very detailed documentation. If that fails, search the archives of the Larch mailing list on Google Groups. If you still can’t find an answer, send your question to the list.

Version 2.0.0 of Sanitize, my whitelist-based HTML filtering library for Ruby, is now available. This release includes several new features and some changes to existing features. I’ll cover the big stuff in this blog post; for the complete list of changes, see the HISTORY.md file.

Installing

To install or upgrade Sanitize via RubyGems, run:

gem install sanitize

Sanitize is fully compatible with Ruby 1.8.7, 1.9.1 and 1.9.2.

Transformers

The most significant change in this release is that Sanitize’s core filtering logic is now implemented entirely as a set of always-on transformers. This simplifies the core code and means that Sanitize itself is now built on the same powerful transformer architecture that you can use in your own apps to enhance or alter Sanitize’s functionality.

The environment object provided as input to transformers now contains a slightly different set of data, and transformer output has been simplified. Transformers are no longer required to return anything, and are expected to make any desired alterations directly to the current node and/or document.

Sanitize now has the ability to traverse the document and execute transformers using either depth-first traversal (the default behavior, same as before) or breadth-first traversal (new in 2.0.0). If necessary, you can even run one set of transformers using one traversal method and another using the other method. This allows for greater flexibility and less complexity when writing certain types of transformers.

The README has more details on these changes and new features.

Other notable changes

• Sanitize now outputs HTML4/HTML5 markup by default instead of XHTML (e.g., <img src="foo.jpg"> instead of <img src="foo.jpg" />, etc.). If you prefer the old behavior, you can set the :output config to :xhtml.
• Some new elements and attributes (including several HTML5 elements) have been added to the built-in basic and relaxed whitelists. See HISTORY.md for the complete list.
• Elements like <br>, <p>, and others are now replaced with whitespace when they’re removed in order to preserve the readability of the remaining text content. The list of elements that will be replaced with whitespace when removed is configurable using the :whitespace_elements setting.

Be aware that if you expect specific output from Sanitize in your unit tests, you may need to update your tests. The HTML output from this release may not precisely match the output from previous releases.

Try it out, report bugs

As always, you can try out Sanitize’s built-in filters using the test page at sanitize.pieisgood.org. Please use Sanitize’s GitHub issue tracker to report bugs and file feature requests.

I’ve written an article on YUIBlog introducing my new Storage Lite YUI 3 Gallery module, which provides a lightweight, cross-browser client-side storage API without relying on any browser plugins. This is an updated version of the storage library I mentioned last year in my post about the development of Yahoo! Search Pad.

Head over to YUIBlog to read the article, and feel free to fork Storage Lite on GitHub and add awesomeness (or subtract crappiness).

Version 1.2.0 of Sanitize, my whitelist-based HTML sanitizing library for Ruby, is now available. Consult the HISTORY file for a complete list of changes.

Introducing Transformers

This release adds a major new feature called transformers. Transformers allow you to filter and alter HTML nodes using your own custom logic, on top of (or instead of) Sanitize’s core filter. A transformer is any Ruby object that responds to call() (such as a lambda or proc) and returns either nil or a Hash containing certain optional response values.

To use one or more transformers, pass them to the :transformers config setting:

Sanitize.clean(html, :transformers => [transformer_one, transformer_two])

Input

Each registered transformer’s call() method will be called once for each element node in the HTML, and will receive as an argument an environment Hash that contains Sanitize config information and a reference to a Nokogiri::XML::Node object.

The transformer has full access to the Nokogiri::XML::Node that’s passed into it and to the rest of the document via the node’s document() method. Any changes will be reflected instantly in the document and passed on to subsequently-called transformers and to Sanitize itself. A transformer may even call Sanitize internally to perform custom sanitization if needed.

Transformers have a tremendous amount of power, including the power to completely bypass Sanitize’s built-in filtering.

Output

A transformer may return either nil or a Hash. A return value of nil indicates that the transformer does not wish to act on the current node in any way. A returned Hash may contain instructions that tell Sanitize to whitelist certain attributes or nodes, or to replace the current node with a new node (see the README for specifics).

Example: Transformer to whitelist YouTube video embeds

The following example demonstrates how to create a Sanitize transformer that will safely whitelist valid YouTube video embeds without having to blindly allow other kinds of embedded content, which would be the case if you tried to do this by just whitelisting all <object>, <embed>, and <param> elements:

lambda do |env|
  node      = env[:node]
  node_name = node.name.to_s.downcase
  parent    = node.parent

  # Since the transformer receives the deepest nodes first, we look for a
  # <param> element or an <embed> element whose parent is an <object>.
  return nil unless (node_name == 'param' || node_name == 'embed') &&
      parent.name.to_s.downcase == 'object'

  if node_name == 'param'
    # Quick XPath search to find the <param> node that contains the video URL.
    return nil unless movie_node = parent.search('param[@name="movie"]')[0]
    url = movie_node['value']
  else
    # Since this is an <embed>, the video URL is in the "src" attribute. No
    # extra work needed.
    url = node['src']
  end

  # Verify that the video URL is actually a valid YouTube video URL.
  return nil unless url =~ /^http:\/\/(?:www\.)?youtube\.com\/v\//

  # We're now certain that this is a YouTube embed, but we still need to run
  # it through a special Sanitize step to ensure that no unwanted elements or
  # attributes that don't belong in a YouTube embed can sneak in.
  Sanitize.clean_node!(parent, {
    :elements   => ['embed', 'object', 'param'],
    :attributes => {
      'embed'  => ['allowfullscreen', 'allowscriptaccess', 'height', 'src', 'type', 'width'],
      'object' => ['height', 'width'],
      'param'  => ['name', 'value']
    }
  })

  # Now that we're sure that this is a valid YouTube embed and that there are
  # no unwanted elements or attributes hidden inside it, we can tell Sanitize
  # to whitelist the current node (<param> or <embed>) and its parent
  # (<object>).
  {:whitelist_nodes => [node, parent]}
end

For more details on transformers, consult the README.

Installing

To install or upgrade Sanitize via RubyGems, run:

gem install sanitize

Last week, Google banned my PHP port of JSMin from Google Code due to a quibble over a line in the license stating that “The Software shall be used for Good, not Evil”, which they believe makes the license non-free. When I asked Google’s Chris DiBona whether all Google Code projects including JSMin would be subject to bans due to this clause in the license, he replied, “Sadly, yes”.

Today, Etherpad (which was recently acquired by Google) released their source code on Google Code. Unfortunately, their source tree contains at least two different JSMin ports (one in JavaScript and one in Python), thus making Etherpad non-free and violating Google Code’s terms of service. I’ve notified Google via an email to the Google Code mailing list.

I bring this up not because I have anything against Etherpad or Google Code, and not because I want to start a fight, but because it demonstrates the slipperiness of the slope Google launched themselves down when they banned jsmin-php last week. While I may disagree with their interpretation of the JSMin license as non-free, Google is certainly within their rights to refuse to host it. However, since JSMin is so widely used by so many open source projects, Google now has to choose between banning popular, high profile projects (including their own) or applying their rules selectively and thus promoting a double standard.

So what will it be, Google? Will you remove JSMin from Etherpad, ban Etherpad, or just be—dare I say it—evil and ignore your own rules when they’re inconvenient?

If you need a new host for the Etherpad project, the lovely folks over at GitHub don’t seem to have any problem hosting JSMin.

Update (2009-12-18): Chris clarifies: “As a side note, it’s not a matter of violating the terms of service, which don’t mention specific licenses, it is against our practices, though.” I’ve updated the title of this post accordingly. Chris has also asked the Etherpad maintainers to remove JSMin, which seems to indicate that Google is going to do the right thing and follow their own rules. Admirable!

Update 2 (2009-12-18): There are several other Google-sponsored projects that fall afoul of this ban as well:

• Google’s Page Speed Firefox plugin includes JSMin.
• The SoC project, which lists Chris DiBona as an owner, includes JSLint, which is distributed under the same license as JSMin.
• The GMaps Utility Library also includes JSLint.

Google’s Chris DiBona emailed me this morning to tell me that unless I removed a specific line from the license of my jsmin-php project (a PHP port of Douglas Crockford’s JSMin), Google Code would no longer host the project.

The license in question is the one attached to the original jsmin.c, and is a slightly modified version of the MIT License. Here it is with the offending line emphasized:

Copyright (c) 2002 Douglas Crockford (www.crockford.com)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

The Software shall be used for Good, not Evil.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

As Google (and some others) interpret it, this additional requirement constitutes a vague use restriction and thus makes the license non-free. Chris explained that if I were to remove that line from the license and “return to a proper open source license that we support”, then jsmin-php could stay on Google Code. Otherwise, he said, “we can’t host you”.

Of course, I can’t change the license, because it’s not my license. It’s Douglas’s license, and he wants people who use his software and derivative works of his software to use it for good and not evil. All derivative works and copies of jsmin.c either include this license or are in violation of it.

I added jsmin-php to Google Code in 2007. Since then, it’s been downloaded over 20,000 times. As of today, its new home is GitHub.

I don’t really mind moving the project—I’ve been intending to do it for a while anyway—and I certainly appreciate the fact that Chris was kind enough to send me a personal email about this before taking any action. But jsmin-php is unlikely to be the only project affected by Google’s discovery of JSMin’s license.

In my reply to Chris, I asked him:

There are quite a few other projects on Google Code that are ports of jsmin.c or include either ports or the original. Does this mean those projects will also be banned from Google Code unless jsmin.c's license changes?

Chris responded: “Sadly, yes.”

I don’t know if Google intends to proactively hunt down all projects using JSMin or whether they’ll only take action when someone rats you out, but if you currently have a project on Google Code that is derived from or includes jsmin.c, you might want to consider migrating to a new host with less restrictive policies.

I asked Douglas what he thought of this. He responded: “When did Google stop being against evil?”

Update (2009-12-09): Via @miraglia, here’s a hilarious excerpt from Doug’s talk, “The JSON Saga”, in which he gives some background on why he added this clause to the license and how often people ask him to remove it:

When I put the reference implementation onto the website, I needed to put a software license on it. I looked up all the licenses that are available, and there were a lot of them. I decided the one I liked the best was the MIT license, which was a notice that you would put on your source, and it would say: "you're allowed to use this for any purpose you want, just leave the notice in the source, and don't sue me." I love that license, it's really good.

But this was late in 2002, we'd just started the War On Terror, and we were going after the evil-doers with the President, and the Vice-President, and I felt like I need to do my part.

[laughter]

So I added one more line to my license, which was: "The Software shall be used for Good, not Evil." I thought I'd done my job. About once a year I'll get a letter from a crank who says: "I should have a right to use it for evil!"

[laughter]

"I'm not going to use it until you change your license!" Or they'll write to me and say: "How do I know if it's evil or not? I don't think it's evil, but someone else might think it's evil, so I'm not going to use it." Great, it's working. My license works, I'm stopping the evil doers!

Audience member: If you ask for a separate license, can you use it for evil?

Douglas: That's an interesting point. Also about once a year, I get a letter from a lawyer, every year a different lawyer, at a company--I don't want to embarrass the company by saying their name, so I'll just say their initials--IBM...

[laughter]

...saying that they want to use something I wrote. Because I put this on everything I write, now. They want to use something that I wrote in something that they wrote, and they were pretty sure they weren't going to use it for evil, but they couldn't say for sure about their customers. So could I give them a special license for that?

Of course. So I wrote back--this happened literally two weeks ago--"I give permission for IBM, its customers, partners, and minions, to use JSLint for evil."

[laughter and applause]

And the attorney wrote back and said: "Thanks very much, Douglas!"

You can see the full video of the talk at YUI Theater (the excerpt above is from 39:45).