YUI from the outside

Update, 2012-09-10: I’ve posted a followup to this post entitled Why I believe in YUI. Please read it.

When I quit my job at Yahoo! a few months ago, it didn’t feel like I was really quitting. I was leaving the company, sure, but I still planned to continue working on YUI, even though Yahoo! would no longer be paying me to do it. Getting paid to work on an open source project has many benefits, and one of them is that if you love your job, you can keep right on doing it even if you decide to leave the company.

I’m extra lucky that my new employer loves YUI, uses it extensively, and is more than happy to let me contribute the work I do back to YUI whenever it makes sense. Since I left Yahoo! I’ve sent ten pull requests to YUI. Some for minor bug fixes, enhancements, or performance improvements; others for brand new components like LazyModelList, Template.Micro, and node-scroll-info.

This has been one of the most frustrating experiences I’ve ever had contributing to an open source project.

Pull request graveyard

Even though I’m still in touch with the core team and talk to many of them on a daily basis, several of my pull requests sat in limbo alongside over 40 other open pull requests on the YUI 3 project (some dating back nearly a year). Compare this to jQuery, which — despite being more popular than YUI — currently has only 22 open pull requests, none older than two months.

Not all my pull requests were large. One was a minor change that fixed a critical performance bug in Y.Attribute#getAttrs(). Another was a minor change that improved the performance of Y.merge(), a core utility method used throughout the library. In both cases, the pull requests sat unreviewed as release windows passed them by, even though reviewing them would have taken minutes.

No external committers

Why send pull requests? Why not just commit these minor changes myself? Because there are no YUI committers who don’t work for Yahoo!. Despite having been a trusted senior member of the core team for over two years, I lost commit access to YUI the instant I stepped off the Yahoo! campus for the last time.

What many people don’t know about YUI is that the YUI GitHub repository is just a read-only mirror. YUI’s actual source-of-truth repo lives on a server inside the Yahoo! corporate firewall, and can only be accessed by Yahoo! employees. The reasons for this have to do primarily with typical Yahoo! bureaucracy and not with any desire on the part of the YUI team to exclude external developers, but nevertheless, that’s exactly the effect it has. YUI has many external contributors, but the only way to get commit access is to draw a Yahoo! salary.

The YUI team also has no actual set procedure or policy for dealing with pull requests. In theory, when budgeting time and making development estimates for each release cycle, team members are expected to budget time for handling pull requests, responding to community questions, etc. In practice, the team members who actually care about that stuff make time for it, and the ones who don’t care as much don’t make time for it. So some team members respond to pull requests, hang out on IRC, and drive YUI’s community engagement efforts, while others are completely invisible and unreachable to the YUI community.

Unfortunately, some of the team members who don’t care very much about engaging with the community outside Yahoo!‘s walls are also very senior members of the team who “own” many important parts of the library. Because they avoid spending time interacting with the community, they’re often out of touch with what YUI’s users need and care about, and tend to say “no” far more often than they say “yes”. They also tend to either ignore pull requests or excuse themselves from dealing with them by citing how busy they are.

As a result, some of the most important parts of YUIY.Node, Y.Event, Y.Base, Y.Attribute, and more — are also the most ignored, and the hardest for external developers to contribute to.

Long story short? Important changes and improvements to YUI are frequently blocked by people who rarely interact with the community and cannot be held to task by the community. These team members mean well, and they do what they do because they genuinely think it’s good for YUI. But it’s often not, and I’ve learned that fighting with them from the outside is far more difficult than fighting with them from the inside was.

The Template.Micro fiasco

A few weeks ago I opened a pull request for a new YUI component called Template.Micro. This is something I created for my own use, but I thought it would be a great addition to YUI. Some fantastic community discussion occurred on that pull request — lots of developers, including members of the core team, had excellent feedback, and we worked together to make Template.Micro awesome. Eric Ferraiuolo and I even chatted about Template.Micro in a public Google Hangouts on Air session and posted the recording on YUI Theater.

Once everyone who participated was happy with the pull request, Eric (who is a member of the core team) arranged to discuss Template.Micro at one of YUI’s weekly engineering meetings. These meetings are normally private, but the team occasionally invites external contributors to join in, and they invited me to talk about Template.Micro.

Every YUI team member in that room who had an opinion had already participated in the discussion on the pull request, except for — guess who? — the ones who rarely engage with the community. They hadn’t looked at it. Not even a glance. Naturally, they were the only ones who objected to merging it in. They objected strenuously and continuously for nearly two hours as I summarized and reiterated every discussion that had already taken place on the pull request or on IRC. Eventually, with the help of others on the team who had participated from day one, I managed to convince them it should be merged.

Then nothing happened, and nothing has continued to happen for over a week. Rumor has it that someone changed their mind (or just “doesn’t have time” to review the fiftyish lines of code, not counting comments) and has decided Template.Micro shouldn’t be merged after all. Code freeze for the next release approaches. I can’t actually get an answer on the pull request itself, so it sits unmerged, and I have no idea whether I or anyone else who wants to see this in the library will be able to use it in the next release.

I’m left feeling that I’ve wasted my time, and I no longer have any desire to contribute to YUI. It’s not worth it. This is just one of the battles I’ve had to fight recently, and fighting battles is not how I enjoy spending my free time, nor is it a good way for me to spend my employer’s time.

What needs to change

Problem #1: There’s no incentive for YUI core team members to review or merge pull requests.

Everyone on the core team is busy (usually incredibly busy) with their own tasks. No explicit time is ever set aside for managing external contributions, so the only time this happens is when someone takes the initiative to do it. There are people on the team who rarely take this initiative, and others who are often too busy to take this initiative. As a result, even small pull requests tend to sit around for a long time. This provides a significant disincentive for external developers to contribute minor fixes, and an incentive for them to simply bake monkeypatches into their own code, which YUI will never benefit from.

My recommendation: YUI should make it part of everyone’s job to triage and manage pull requests. They should strive for zero open pull requests at the end of every sprint — any pull request that shouldn’t go into the current sprint should either be denied or scheduled for a future sprint (and then should not be forgotten when that sprint rolls around).

Problem #2: There are no external contributors with commit access to YUI.

Obviously only highly trusted contributors should have commit access to YUI. The core team is highly trusted. Luke Smith (another recent former YUI team member who still contributes) and I were highly trusted when we were on the core team. As soon as we turned in our Yahoo! badges, we became just as untrusted as everyone else on the Internet. This is strange.

The weirdest thing about this? I’m now dogfooding YUI far more than I ever did when I was paid to work on it, so I’m naturally finding a lot of things I want to improve. I feel like I have a much better idea of YUI’s pain points and opportunities for improvement than I ever did before — but I also feel like there’s so much of a process barrier that the amount of effort required to get my changes back into YUI begins to approach the amount of effort required to simply fork YUI. That’s really scary to me, because it makes me wonder how many other YUI users and potential contributors have the same feeling.

My recommendation: Make GitHub YUI’s source of truth repo. Grant commit access to highly trusted external contributors, on the condition that non-trivial changes must be reviewed by the core team first. If an external contributor screws up, it’s not a big deal — revert the commit. If they screw up often, revoke their commit access.

Problem #3: There’s not much external visibility into YUI’s release planning and scheduling process.

I didn’t realize how frustrating this could be until I was on the other side. I want to contribute lots of stuff to YUI. Not just new stuff, but improvements and fixes. Naturally, I want to do this in a way that aligns with YUI’s own plans and schedules, but I also want to do it in such a way that I don’t have to babysit my contributions for months until they can be integrated into core.

Knowing YUI’s release plans and what the team is focusing on for an upcoming release would be a huge help in budgeting my time. I could look at the team’s plans and say, for example, “Hey, it looks like YUI is focusing on performance improvements for this release. I’ve got some of those on the back burner, so this would be a great time to contribute them.” Or “Well, I’ve cooked up this new widget, but it looks like the next YUI release will be a bug fix release, so I should roll this into a gallery module for now instead of letting it rot in my fork for three months.”

Efforts like YUI Open Hours and This Week in YUI help a little, but they don’t provide the nitty gritty insight into release planning and scheduling that are really needed, and Open Hours in particular requires a level of time commitment that people just can’t always make — it’s a lot harder to watch an hour-long screencast and try to extract meaningful info than it would be to read a blog post that simply says “here’s what we’ve got planned for the next release and here’s how you can help”. Even just knowing the dates for preview release windows would help contributors immensely in prioritizing our contributions.

My recommendation: Post the results of planning discussions and other non-confidential internal discussions on the blog (or on GitHub, with an announcement on the blog or Twitter) soon after they happen. A public document of some kind should be the end result of any non-confidential planning meeting or discussion. Consider inviting trusted external contributors to participate (via G+ Hangouts) in discussions that won’t involve confidential Yahoo! information.

Problem #4: Creating and maintaining gallery modules is far too much work.

The usual response to “it’s too hard to contribute to YUI core” or “YUI core doesn’t want my awesome new widget” is “well, just put your stuff in the gallery”. But it’s so painful to create, publish, and maintain a simple gallery module that I’d just rather not do it at all. It eats up far too much time that could be better spent writing code.

The main problem is how much manual labor there is, and how difficult it is to organize and maintain gallery modules as a result. Dav and others on the team are aware of my specific complaints in this regard, but until something is done, I think the team needs to stop using the gallery as a crutch to justify not solving problems 1 through 3 above.

My recommendation: Change the gallery to rely on the exact same build tools and procedures that YUI core uses. Remove the requirement for all CDN-hosted gallery modules to be maintained in a single yui3-gallery repo, so that gallery modules can use their own GitHub repos as their source of truth. Remove the manual form-filling required to create a gallery module on the website, and instead rely on the module’s own metadata and a simple tool that publishes gallery modules similar to how npm publishes npm modules.

If it were as easy to create a gallery module as it is to create an npm module, I’d have so many gallery modules people’s heads would explode. In a good way.

Problem #5: YUI can be a community project or a Yahoo! project. It can’t be both.

The primary mission of any good internal Yahoo! project is to further Yahoo!‘s corporate goals. The primary mission of any good open source project is to further the goals of its community. Since its inception, YUI has tried to do both. This isn’t working.

I can tell you from my time working at Yahoo! on both YUI and other projects that YUI is unlike any other project at Yahoo!. It sometimes doesn’t feel like it’s even part of the same company. And I can tell you from my time working on open source projects that YUI is unlike any other open source project, and in many ways doesn’t feel like an open source project. YUI tries to be a hybrid of both, and as such it tries to serve two masters — the corporation and the community.

This is why YUI is subject to Yahoo!‘s bureaucracy, it’s why some team members feel justified never engaging with the community, and it’s why contributing to YUI is a major pain in the ass. When I worked for Yahoo!, it was easy to assume that the external users (and non-users) who complained about this were just being too sensitive, too selfish, or too shortsighted. Now that I’m one of them, the shoe’s on the other foot and I’ve achieved enlightenment.

Despite the efforts and dedication of many members of the core team — seriously, YUI has an incredible group of core developers — YUI is not really an open source project. It’s a Yahoo! project whose source is public. The open source community has zero involvement in the management of the project. They have zero transparency into day-to-day happenings except for the mirrored source tree and whatever information leaks out via the blog, IRC, and Open Hours. There is no true sense of community ownership or responsibility for the project, and community members often feel frustrated at their inability to influence YUI’s direction.

My recommendation: Create a YUI foundation, separate from Yahoo!, and entrust the governance of the YUI project to it. Encourage Yahoo! and other companies that benefit from YUI to provide funding. This isn’t a new idea — it’s something that’s been discussed both inside and outside of Yahoo! for years, and is a structure embraced by many other open source projects — but it’s more and more apparent now that it’s crucial to YUI’s future.

I still love you, YUI

YUI has many flaws, whether we’re talking about the code itself or how the project is governed. But it’s also an amazing library with a vibrant, intelligent, devoted community. I plan to continue using it, and I hope to be able to contribute to it again at some point. But until some things improve, contributing to YUI just isn’t worth my time.