The eclectic musings of a bitter software engineer.

XMLHttpRequest, savior of the Internets

Friday January 28, 2005 @ 12:33 AM (PST)

For the last few weeks, I've been plugging away nonstop at work on an internal website that will be used to manage a heavily data-driven external customer website. The primary users of this internal site are not at all technically savvy—at least, not outside their specific spheres of expertise—so my primary goal was to make the site highly intuitive, mainly so as to avoid being pestered with questions.

One thing I learned a long time ago is that most people do not find web-based forms intuitive at all. Sure, anyone can use Google, but as soon as you mix it up a bit with a few text input fields, a select box, maybe a checkbox or two, people start getting confused. Adding inline help text and tooltips doesn't solve anything, because people automatically tune them out. They'll always, always assume it's easier to ask someone for help than to look around for an answer themselves, even when it's right in front of them.

So I decided early on that this site, despite being incredibly input-centric and thus necessitating the use of forms, would use them completely transparently. I wanted the users to forget they were using a website. I didn't want them to feel restricted; I wanted them to be able to do whatever felt natural. If a user is looking at a table of customer information and decides they want to change a customer's name, they shouldn't have to click a link and open a form. They should be able to click on the name and start editing, and their changes should take effect transparently. No visible forms, no submit buttons, no page refreshes.

Since I had free reign to do whatever I wanted, and since I didn't have a looming deadline, I decided to take my time, do some research, and try some new things. The first thing I experimented with, of course, was the XMLHttpRequest object, which has been around for a while but has only recently gotten a lot of attention thanks to Gmail. I was not disappointed; XMLHttpRequest is an incredibly useful tool when used appropriately, and it's supported by Mozilla, Safari and Internet Explorer. Opera has lagged behind, but I hear it's in the latest beta (no doubt thanks to Gmail).

I found myself writing a lot of JavaScript and, what's more, I found myself enjoying it. JavaScript is an incredibly flexible language, and while browser differences are occasionally a pain to deal with, they've presented a lot less trouble than I thought they would. I've been developing for Firefox and Internet Explorer. Firefox, of course, has been an absolute dream to work with. Internet Explorer has been less pleasant, but still not as bad as it's made out to be. I'm far more frustrated by the incompleteness of IE's CSS support than I am by the quirks in its JavaScript implementation.

XMLHttpRequest is making things seamless on the back end, between the browser and the server, with data being transferred in the form of serialized JavaScript objects, and the DOM is making things transparent on the front end, between the user and the screen. Tables are populated with data (and sometimes created from nothingness) on the fly, rows are instantly sorted when the user clicks a column header, changes are validated and saved instantly, error messages are displayed inline and attached to the appropriate interface object, etc.

Currently, almost all the client-side work is being done in two JavaScript classes: DynamicForm (for creating transparent forms with seamless validation and persistence) and PagedTable (for creating a data-driven table with sorting, filtering and next/previous-style paging). The site is barely even halfway finished, but I'm already dreaming up ways to use these techniques in other projects (Poseidon, specifically).

A year ago, I would have scoffed at the idea of writing a web application that actually required a user's browser to support JavaScript. Now I'm starting to find it hard to believe that anyone would ever write a web application that didn't. For years web developers have suffered from trying to cope with misbehaving browsers, a lack of standards, and the necessity to support users of old software. Finally, finally, I think the tide has turned. Firefox is rapidly gaining popularity, Internet Explorer, for all its faults, is not impossible to deal with, and Opera and Safari are doing an excellent job filling in the gaps; we've got more standards than we know what to do with; and the popularity of Gmail has paved the way for other highly dynamic websites.

People are starting to expect more, and they're finally willing to upgrade their damn browsers to get it. Thank God.

Comments

i'd like to think that there might be maybe one or two other people in the world that keep javascript disabled by default.

damn pop-up ads.

i also have meta redirect disabled my default so everytime i log into wonko.com i have to manually go back to the page i wanted to post a comment to. i do it because i don't like web pages being able to control vital functions of my web browser, but i guess i'm just behind the times.

ideally, every browser would have a long list of JavaScript functions and events that you could disable or enable depending on what you wanted. specifically, the ability to disable those damn message boxes that think they can prevent you from copying text, to keep the screen from being resized, and to keep javascript from changing the page without a prompt.
Friday January 28, 2005 @ 01:41 AM (PST) Posted by rdude
There's no need to disable JavaScript in order to get rid of popups. Just use Firefox. I can't remember the last time I saw a popup ad. Even Internet Explorer has popup prevention these days, although I have no idea how effective it is.

Disabling meta redirects is asking for trouble. They're very widely used. I know they can sometimes be annoying, but is the inconvenience really worth it?

Mozilla and Firefox do allow you to disable specific JavaScript functions, events, and properties. It's called Configurable Security Policies, and it's a very robust system. You can even define rules on a site-by-site basis if you want to. There are also countless Firefox extensions available that keep JavaScript from being used to block text selection or mouse click actions.

Friday January 28, 2005 @ 11:40 AM (PST) Posted by Ryan Grove

Yea, the last time I saw a pop-up ad was on a Windows 95 machine with like IE 4 in my school's library. If you don't have a pop-up blocking solution by now then your for sure obsolete. Wow wonko, this little project of yours is pretty interesting, you are a genius.

Friday January 28, 2005 @ 02:00 PM (PST) Posted by summercom
CONFIGURABLE SECURITY POLICIES

FIREFOX

I HAVE SEEN THE LIGHT!

thanks for pointing that out to me!
Saturday January 29, 2005 @ 12:58 AM (PST) Posted by rdude
Does the Preview thingie work here at wonko.com?

Well off course it does, but then why won't it work for me at my site http://weasel.dk/content.php?title=Bug+hunt ?? Sorry for the this Off Topic posting... but I'm getting desperate :-/

Hey - that spell check is quite nice - i thought it was broken? - guess you fixed it :o)
Sunday January 30, 2005 @ 07:06 AM (PST) Posted by Weasel
The preview bug occurs when you use Poseidon with PHP 5. I came across this a long time ago and spent quite a while trying to come up with a workaround, but to no avail. Poseidon isn't doing anything that PHP 5 shouldn't be able to handle, but for some reason PHP just refuses to display that preview. No errors, no complaints, it just refuses to work, no matter what.

However, if you don't mind waiting another month or two, Poseidon 0.7, which is pretty much a complete refactoring of the code, is specifically designed to run on PHP 5 (and to take full advantage of its new capabilities). The new version isn't quite complete enough to be useful yet, but it's getting closer every day. As soon as I have something usable, I'll commit it to CVS and you can start playing with it.

Sunday January 30, 2005 @ 01:43 PM (PST) Posted by Ryan Grove
Ooh my!
A new version is within reach - i am thrilled, and you can be sure i will love to try it as soon as it is possible.

Ps.: For the record, weasel.dk is English language, and my new site familieklubbenfonnesbæk.dk is Danish language. It was at the new site i discovered the preview bug.
Sunday January 30, 2005 @ 09:46 PM (PST) Posted by Weasel
I've only lightly touched on using XMLHttpReq previously, and only for interacting with a database, however your article spurred me on to try and be a little more adventurous and use it on a image-update script I had to put together, basically I spent the day trying to work out how to transmit the binary image from a file input over the javajscript call to xmlhttprequest -- time constraints meant I had to abandon the idea, but its still on the back burner, just wondered if your newfound experience means you'd be able to suggest if i was completely wasting my time? ;)

Definiately the way forward!
Friday February 04, 2005 @ 03:38 AM (PST) Posted by test
Problem is, when you use JavaScript, 9 times out of 10 anyone using a screenreader is screwed over. Lots of blind people surf the web (I didn't believe it until I ran into a few of them), and Javascript, badly designed forms, PDFs, and graphics without alt tags are what screw them up the most.

Hence, I stay away from Javascript like the plague.
Friday February 04, 2005 @ 10:41 AM (PST) Posted by Innismir
Sadly, yes, trying to get JavaScript to upload files via XMLHttpRequest is a waste of time. I spent half a day on that myself, but there's just no good cross-browser way to do it. For security reasons, it's virtually impossible to manipulate the local filesystem using JavaScript.

However, if you don't mind being tied to a specific browser, there may be some hope for you. In IE, it can supposedly be done by using an instance of the ADODB.Stream ActiveX object to read the local file. It requires relaxed security settings, of course.

It's also possible to gain access to the local file system in Mozilla, but only by enabling the proper security privileges (I believe the "UniversalXPConnect" privilege is what you need for doing local file I/O, but I may be wrong). Like many security measures, it's a big inconvenience for the user.

In the end, I used a standard HTML file upload form and had it post to a hidden iframe. The server returns a serialized JavaScript object to the iframe indicating whether the upload succeeded or failed and containing a descriptive status message, which I then retrieve from the iframe and parse just as if it had come in over the XMLHttpRequest connection. This way the user experience is still seamless, and we don't have to do any local file I/O with JavaScript.

Friday February 04, 2005 @ 10:51 AM (PST) Posted by Ryan Grove
That's not necessarily true. It takes a little bit of extra work and attention (and possibly a tradeoff here and there) to design an accessible web application that makes heavy use of JavaScript, but it's not impossible.

Admittedly, accessibility hasn't been a primary goal for this particular project since it's intended for internal use by a very specific group of people, none of whom (currently) have any special needs, but I can only think of a few places where a screen reader might get really confused. Assuming, of course, that the screen reader watches for dynamic changes to page elements instead of just activating itself when a new page is loaded.

Still, you have a valid point. But accessibility is no harder to implement in a web-based app using JavaScript than it is in a client-side app. In fact, I'd go so far as to say that writing accessible web apps, even highly-dynamic ones, is a great deal easier than writing accessible client-side apps, if only because of the existence of widely-accepted standards and documented best-practices.

Friday February 04, 2005 @ 12:24 PM (PST) Posted by Ryan Grove
From the Anti-Javascript FAQ (http://www.panix.com/~aahz/javascript.html):

...text-only browsers such as Lynx that don't support JavaScript (lots of blind people use Lynx -- in fact, some people have threatened to sue sites for non-compliance with the Americans with Disabilities Act (ADA)).

The BBC also recently had an article about blind people using modified versions of lynx (which is how I knew this).
Friday February 04, 2005 @ 01:59 PM (PST) Posted by tabor
A large part of ensuring the accessibility of JavaScript-enabled web apps is ensuring that they still work when JavaScript is disabled. In fact, the WCAG even requires this. However, unless I'm mistaken, Section 508 (the only US law I'm aware of that actually defines legal requirements for web accessibility) doesn't require websites to work when JavaScript is disabled; it only requires that they use JavaScript in an accessible manner. In any case, Section 508 only applies to US government websites.

I've got nothing against blind people, and I've got nothing against people who prefer to use Lynx, but filing lawsuits because your browser of choice doesn't support JavaScript is stupid. The ADA doesn't even mention websites, and it would take a great deal of creative interpretation to claim that websites fall under its jurisdiction.

Friday February 04, 2005 @ 05:23 PM (PST) Posted by Ryan Grove

In no way did I agree that the ADA applied to you or your employer's websites (although at least for some things it probably does apply to your employer). I was just pointing out that it is hard to write accessible javascript if the majority of web browsers for blind people do not support javascript.

Sunday February 06, 2005 @ 02:59 PM (PST) Posted by tabor

Lynx isn't a web browser for blind people. Are there actually any web browsers written specifically for blind people? I'd imagine most blind people would just use a normal browser with a screen reader.

Sunday February 06, 2005 @ 03:58 PM (PST) Posted by Ryan Grove
Lynx isn't a web browser for blind people.

Thank You Captain Obvious.

Are there actually any web browsers written specifically for blind people?

Yes. BLYNX (based on Lynx, http://leb.net/blinux/blynx/). Like I said in my original comment, blind people often use "modified versions of lynx." Well, technically BLYNX is not "modified" because the source code is not modified, but it is highly configured for the blind (the lynx.cfg file for blynx is 1725 lines long and the .lynxrc file for blynx is 193 lines long).

Note: Blynx is actually more than just the customized version of lynx. It is a whole project devoted to helping blind people browse the web.
Monday February 07, 2005 @ 11:20 AM (PST) Posted by tabor
We've been using this nice little object in IE for quite a long time. We've built our user interface framework around this communication mechanism.

You might be interested in seeing it all at http://www.xicommunity.ca. This site hosts the XI-Factory tool that allows you to design/generate complete web application prototypes online! XI-Factory and the applications it generates all uses the XMLHttpRequest object.

Right now it works only in IE but we are looking into making all this stuff compatible with Mozilla and Safari.

btw, great blog!
Alex

http://endfunction.blogspot.com
Wednesday February 09, 2005 @ 01:25 PM (PST) Posted by alexbf
'your ideas are intriguing to me and I wish to subscribe to your newsletter' ;)

sorry for the delayed response, hectic week.

Cheers for the informed musings.. I was afraid of that.
I found your suggestion of using the serialised js object from the hidden iframe a lot cleaner from the user POV than the modal dialog I'd put together though, thanks for that suggestion.

I think I'll be having a play with UniversalXPConnect when I get a bit more time, but the iframe isn't too bad by any stretch.
Monday February 14, 2005 @ 02:07 AM (PST) Posted by test

Man are you gonna love backbase!
http://www.backbase.com/

Sunday September 18, 2005 @ 03:55 AM (PDT) Posted by Jack

If you want to download binary files/streams using XMLHttpRequest object in Javascript, have a look at
http://mgran.blogspot.com/2006/08/downloading-binary-streams-with.html
[]s,

Monday August 21, 2006 @ 06:00 PM (PDT) Posted by Marcus
Post a comment

Basic XHTML (including links) is allowed, just don't try anything fishy. Your comment will be auto-formatted unless you use your own <p> tags for formatting. You're also welcome to use Textile.

Don't type anything here unless you're an evil robot:


And especially don't type anything here:

Copyright © 2002-2008 Ryan Grove. All rights reserved.
Powered by Thoth.