David Baron's weblog: August 2006

Friends & Colleagues

Friday 2006-08-18

More W3C Controversy (17:58 -0700)

There's been a good bit of controversy recently due to concerns in the Web browser community about the SVG Tiny 1.2 specification moving to Candidate Recommendation (the W3C's call for implementations stage). Ian Hickson (a long time Web standard developer and the man behind the WHATWG) objected to its duplicating features of many other Web standards. Boris Zbarsky (a leading Gecko/Mozilla developer) objected to the group failing to clarify basic concepts and to not following W3C process, said he no longer thinks the SVG working group is acting in good faith, and said he no longer plans to work with the SVG working group. Maciej Stachowiak (a leading WebKit/Safari developer) objected to the addition of text layout features to SVG that largely duplicate, but with significant differences, features that already exist in HTML+CSS, and said that he no longer thinks it makes sense for Web browser vendors to participate in the development of SVG at the W3C. Ian Hickson also objected to the SVG spec adding a new script processing model that is incompatible with the one used by HTML on the Web. Robert O'Callahan (another leading Gecko/Mozilla developer), objected to SVG's choosing link targeting behavior that was compatible with WebCGM rather than with HTML, since being compatible with HTML ought to be more important for a standard intended for the Web. Björn Höhrmann (a frequent and very knowledgeable commenter on many Web standards) detailed his formal objections and the alleged violations of W3C process through which they were ignored.

The technical comments in these messages are just the latest in a very long string of comments, and there are dozens or hundreds of technical problems of a similar magnitude. However, the frustration with the entire process has been expressed more explicitly than it was in the past.

Around the same time, there has been concern raised by prominent figures in the Web development community about lack of progress within the W3C. Jeffrey Zeldman criticized the W3C for its lack of concern for the needs of Web developers. Eric Meyer (a leading author about CSS) echoed these concerns, and pointed out that Molly Holzschlag's rebuttal to Zeldman's criticism actually accepted one of Zeldman's key points.

While these two areas of criticism may initially appear unrelated, I think they're actually very closely related. But explaining how they're related requires a bit of a detour into understanding how the W3C works.

The W3C today

The first thing to understand about the W3C is that it is a consortium. Over 400 companies pay the W3C to be members of the W3C, which allows them to participate in many W3C activities. The W3C then has over sixty technical employees who work on the things that the members are paying for.

The first thing that might surprise readers here is that there are over 400 member companies. Web developers might wonder if there are that many companies that make browsers or authoring tools? Or if there are a lot of medium size Web design companies in the membership? Neither is actually the case. And that should give a pretty clear explanation of what Molly Holzschlag called the W3C's “frightening disregard for the real needs of the workaday Web world.” If most of the member companies are paying the W3C to work on other things, then the W3C will probably end up working on other things.

So what do the W3C members want? For a start, have a look at the six domains that the W3C organizes its work into: Architecture, Interaction, Quality Assurance, Technology and Society, Ubiquitous Web, and Web Accessibility Initiative. Two of these six (Quality Assurance and Web Accessibility Initiative) are interested largely in refining the technologies produced in the other domains. Of the remaining four, the stuff that Web browsers do lives mostly in the Interaction and Architecture Domains, and it's mostly the Interaction Domain where there is interest in developing new standards relevant to Web browsers. So I want to focus on what W3C members want from the Interaction Domain.

"Follow the money" is often given as a good way to figure out motive. Why would companies want to be members of W3C? Because it helps their business. For example:

These motives lead groups within the W3C to spend significant amounts of time on things that don't help the Web. For example, a company that is using W3C technologies in a non-Web environment may push the issues that arise in their environment to the agendas of working group meetings. Essentially, they're paying the W3C to have experts on the technology (the working group) solve their problems. And those experts are often quite willing, since work to make one's invention used in more places can appeal to the vanity of the inventor.

This problem is minor compared to the time that's been spent lately in fights between working groups and the communities they're associated with. The biggest such disputes have generally been between people involved with browsers for personal computers (some of which also run on mobile devices) and browsers developed primarily for mobile devices. While I don't know all that much about the cell phone industry, my general impression is that they're interested in providing Web browsing (although perhaps at pricing plans tend to make it relatively rarely used and thus probably not all that profitable). However, cell phone providers also provide a lot of content that's part of their network—content from which they can make money more directly. At one time they were requiring software on cell phones that supported WML so they could produce their content in WML, and also allowing access to WML on the Internet in the hopes that a separate Mobile Web would develop. Due perhaps to a combination of the lack of momentum of this Mobile Web, the increasing capabilities of the hardware on cell phones, and political pressure from organizations like the W3C that wanted one Web rather than two (one of which they didn't get the standardization business for), the companies decided to move to using Web standards on their mobile phones, perhaps with the high-level goal of actually being compatible with the Web.

However, compatibility is really in the details. Web browser developers all know that being able to display Web pages requires being compatible with other browsers not just by supporting the same standards and doing what those standards say (or in some cases all doing the same thing that's different from what the standards say), but by being compatible on minor details that are not mentioned in the standards (especially the poorly written standards). Even if this level of compatibility were met, there would still be huge obstacles to making the same Web pages work on desktop and mobile devices, which have vastly different displays or other output methods, and vastly different keyboards, keypads, pointers, or other input methods.

In other words, a large part of the mobile industry is using software that implements some Web standards, but largely not compatibly enough with the content on the Web to be able to use it. However, they're still developing content in their own walled gardens. And since that content works, it's the stuff that drives their development. Making that content work interoperably across multiple implementations requires standardizing what standards are required by Mobile Web browsers and standardizing much of the detailed behavior that's needed for compatibility. The problem is that they don't have much incentive to choose behavior that's compatible with the content on the Web. Or something roughly like that; I don't actually understand the details, but I have seen the results.

So we end up in a world where mobile browsers implement some of the same standards as desktop browsers (HTML, CSS) although generally not as well, and where they implement some different standards (e.g., SVG), perhaps better. The level of interoperability between these two worlds is not high enough to make it easy to develop content that works well in both. So, naturally, left to their own, the two worlds would diverge, into two different sets of rules for handling all the ambiguities in the standards, sets of common “bugs” (where all implementations would disagree with the standard in the same way), and eventually (or already) different sets of supported standards. In other words, using technology for both mobile and desktop based on the same underlying specifications doesn't actually do anything useful if the implementations of those specifications don't interoperate. However, the W3C staff and other W3C policies tend to try to force them to converge even if neither side actually wants this convergence. Generally, W3C process forces reuse of standards even when those standards are inappropriate. So whichever side of the divergence (desktop CSS, or mobile SVG and CDF, or in some cases other communities within the W3C) is the first to write down the rules that they depend on for interoperability supposedly writes the official W3C way of handling the ambiguity. At least, this is the way it is according to the rules at the W3C, although neither side really likes it. This causes each side to attack and try to block the advancement of the specifications on the other side.

This picture should make clearer the causes of both sets of criticism described above. The Web browser community has to stop things in SVG that are incompatible with or inappropriate for the Web, because if we don't stop them we'll never be able to standardize the Web behavior at the W3C. Second, the Web browser community isn't making much progress on standards relevant to the Web because it's spending all of its time fighting the larger Mobile Web community.

Ideal standards for the Web

If we want to fix these problems, we should think about what it is that the Web really needs from standardization.

One of the reasons we want standards is to get interoperability: a situation where there are multiple clients and multiple servers that can all talk to each other, or multiple authors and multiple readers that can all read what they've written. However, interoperability can be achieved by copying rather than by standardization. But standardization has some advantages over copying:

To get interoperability on the Web, what we really need are well written specifications with clear conformance requirements and clear error handling requirements. And we need good test suites that thoroughly test the specifications. The Web has already suffered many times when poorly written or poorly tested specifications caused loss of interoperability.

The other important thing to consider is that we want to choose the right behavior to standardize. (This isn't really specific to why we want standards; we want the right behavior whether or not there's a standard.) This means that we want to consider the needs of all participants. For document formats, this means authors, users, and implementors of the tools that they use. (The W3C process currently relies on the implementors to represent the other two. And that doesn't always work out so well; sometimes authors will get more representation than users, and sometimes the other way around.) This includes getting input from experts on issues like internationalization and accessibility, which affect some users.

If the Web would benefit from using the same standards that are used elsewhere, then it may be worth considering the needs of the other potential users of the same standards. However, the Web is large enough and important enough that the advantages to taking a backseat to other users (larger user base, additional tools) are unlikely to be nearly as large as the disadvantages (technologies less well suited to the Web).

A way forward?

I used to think that the W3C should focus on things that are compatible (in terms of architecture, interoperability, and maintenance of existing invariants) with what is already on the Web and designed primarily to improve the Web. This belief led me to complain about the fragmentation of document format specifications and complain about the results of a W3C workshop. However, this belief is not shared by many W3C members, and is not the underlying focus behind much of the standardization business that W3C wants to attract in order to attract members.

I think such a focus is necessary if we assume that W3C standards are supposed to be implemented by Web browsers—an assumption that some W3C staff and some member companies vehemently insist is true. It is the combination of this assumption (held by some people) and the lack (from other people) of common belief in this narrow focus that has led to many of the recent controversies about W3C specifications, including those about SVG that I mention above. These controversies, in turn, reduce the resources spent on development of what really matters to Web developers.

Such a focus is also necessary to ensure that all W3C specifications can be interoperably implemented together. For example, if two communities (say, the Web and the Mobile Web) build on the same underlying technology (say, the subset of CSS used in SVG) but using different specifications (say, the rest of CSS, or SVG) then the content built using one of the latter specifications might depend on ambiguities in CSS being clarified one way or on having certain bugs, and the content built using the other might depend on different clarifications or bugs. This creates an environment where somebody who wants to implement both the rest of CSS and SVG on top of the subset of CSS in SVG can't interoperate with both sets of existing content. This causes the two communities (the Web and the Mobile Web) to diverge, and eventually not to demand the same standards be implemented. In other words, the only way this focused model works is if the W3C produces new specifications slowly enough that everybody involved can implement all of them. And that doesn't fit with the W3C's business model.

So I've now come to believe the opposite. In other words, given the breadth of activity within the W3C, we can no longer assume that all the W3C's specifications are part of a single plan. Groups within the W3C should be allowed to produce specifications whose features overlap with those of other W3C specifications. No members of the W3C should be obliged to implement any specifications, or criticized for failing to do so simply because the specification they do not implement is a W3C Recommendation. Instead, specifications should compete on their own merits among implementors, authors, and users.

Accepting this does require giving up something that some might consider significant: the ability to put pressure on Microsoft to implement the W3C standards that are already interoperably implemented by Mozilla, Opera, and Safari, such as many parts of CSS and the DOM. Or, rather, the ability to put such pressure on Microsoft on the basis that these things are W3C standards. Microsoft does recognize the legitimacy of the W3C because the W3C has been a leading Web standards organization since the time when Microsoft was the competitor trying to beat the market leader (Netscape). But I don't think it has a history of yielding to pressure to implement W3C standards simply on the basis of their being W3C standards, rather than their meeting the needs of authors and users. And I don't think accepting this idea of the W3C being broader than the Web removes any ability to complain about bugs in existing implementations, at least for specifications already shown by other implementors to be compatible with the Web. In the end, pressure to implement new specifications really needs to come from authors who are using the new features that Mozilla, Opera, and Safari are implementing, which means making these engines more interoperable, making them easier to write for, and getting them more market share. (Furthermore, large organizations may be less hesitant (for legal reasons, perhaps?) to implement new specifications produced by a standards organization that they already know and understand than specifications produced elsewhere.)

In any case, don't think the W3C can continue trying to be both a focused organization and a broad organization. I think it currently tries to be both at the same time, and gets the technical disadvantages of both approaches, the technical advantages of neither, and the financial advantages of the latter. I've come to accept that it's not going to be the focused organization that I'd like it to be. Given that, I think the W3C and the community around it needs to fully accept the consequences of being a broader organization.

It's time for the Web browser community to stop using up its resources attacking specifications that we're not interested in implementing. One of the reasons there's been so little advancement of the standards used in Web browsers is that we've been spending most of our standardization work fighting against the proposals of others—proposals that don't fit with the Web, or working to improve proposals by others that aren't the top priorities for authors and users of the Web. We should work on, and implement, the standards that we think are appropriate for Web browsers, and ignore the rest. We should spend our time improving what Web developers and users want, not waste our time improving what is less important or criticizing what isn't going to work in the first place. That requires considering what's important at a high level before delving in—something that isn't always easy, and is easily forgotten. But we should spend the effort so that we work on what matters.

Monday 2006-08-14

Geotagged photos (14:52 -0700)

Last night, due to accumulated frustration with existing flickr + maps mashups like geobloggers, which disappeared, flyr, which has a tiny fixed-size map view and uses the older Google Maps API which is no longer updated (why not?) with new map data, and mylocalguru which has a slightly larger fixed size map view and user interface that obscures part of the map, I actually did what I'd been thinking about doing for a few weeks: wrote my own, so that I can see my geotagged photos more easily.

There are two parts to doing this: dealing with the flickr API to get the photo data, and dealing with the Google Maps API to to build the viewing application. I've worked with the Google Maps API before, so I was able to get something simple working for that part in less than an hour.

But doing this required that I have photo data to use, which was the part I stumbled on before. What I really want is to be able to get the data without a server-side component, but that requires I wait for an access control standard. (The Google Maps API found a really neat way to work around this problem: the API requires no server-side component other than a server to serve images. But that solution only works for a small set of problems: maps just happen to be one of them. It doesn't work for flickr which really does need to query a huge database, or for lots of other things that could be done with APIs like this.)

So I decided to start solving only my own problem, and getting the data needed to display just my photos, and not anybody else's. Maybe I'll generalize later, but doing so would require some fancier server-side stuff, something that would take me more time than I have to spend on it.

I wrote a very simple python program (my first, believe it or not, with some help from Mark Pilgrim) that used FlickrClient 0.2 to turn the information I needed about my own photos into a JSON object. (The functional programmer in me wanted to write pretty much the whole program as one statement, as a bunch of nested list comprehensions, but I decided to just use variables, partly since I was dealing with a whitespace sensitive language and didn't want to have to deal with the entire thing being a single line or wonder what would happen if I broke the line.) I then just pasted the JSON into my script and wrote the Google Maps API side of the app.

So I now have a way of viewing my geotagged photos that fills the entire browser window with a map. There are a bunch of relatively simple tweaks to improve the user interface that I could make (some easy, some more complicated), that I'll probably be making at some point in the future. And maybe frustration with regenerating my JSON every time I geotag more photos will eventually lead me to improve the flickr API side of the app, although I suspect I'll be satisified with it. Maybe if I make the Google Maps half useful enough somebody else will be interested.

Thursday 2006-08-10

The terrorists are winning, and other musings on politics (23:22 -0700)

The terrorists are winning. The new security restrictions on flying, especially those in the UK, have now reached the level where I no longer intend to fly. I'd be willing to fly from San Francisco to Europe even if it weren't safer than my (short) daily commute to work, which it probably has been for the past few decades. But it's not worth traveling if I have to give up the ability to bring reading material onto flights (UK only, so far), and I don't trust airlines to provide enough water. But our government (and the UK government) would rather we live in a state of terror, so traveling at slightly higher but still reasonable risk is not an option.

While I'm on the topic of politics, I've been thinking about the war in Lebanon. I've seen some speculation about why the Israeli political and military leaders were foolish enough to respond so aggressively to provocation—a response that greatly reduces the prospects for peace in the region. But that idea just doesn't seem to add up—I have more confidence in the ability of the Israeli government to act rationally than I have in the United States government (although I suppose that isn't saying much), and the reasons I've seen for the response just don't ring true. A theory that makes more sense to me, and which I think an article in The Economist hinted at, is that either the United States or Israel is preparing to attack Iran to destroy its nuclear weapons program. The Economist article suggests that if Iran were attacked, its method of retaliation would likely be to use Hezbollah as a proxy to attack Israel. Thus it would make some sense for Israel to preemptively attack Hezbollah if the United States and/or Israel were preparing to attack Iran without wider international support. (Attacking Iran unilaterally or bilaterally is itself a very bad idea. That the United States can't get wider support for blocking Iran's nuclear program is yet another sign of the ineffectiveness of the Bush administration's foreign policy.)

And one more thing about politics. It's clear that today's Republican party doesn't believe in democracy. Bobby Kennedy Jr.'s article about the 2004 presidential election in Ohio alleges that their tactics changed the result of the 2004 presidential election (see also two rebuttals: 1, 2). I don't think it's clear whether whether or not their tactics actually changed the result of the 2004 election. But what's alarming is that they don't believe in letting the people choose, and they're getting away with it (in Florida, in Ohio, in Alabama, in Missouri, and elsewhere).

Kind of like the way they use homeland security as a political tool rather than as a way to keep people safe.

I'm waiting for the day when the Republicans try to stop the teaching of the American Revolution in schools because it might give the kids dangerous ideas.

Sunday 2006-08-06

Switched hosting providers (09:25 -0700)

Just so that the most recent entry in my weblog doesn't say that my email is bouncing, I'll say here that (about a month ago, when I posted the previous entry) my email was down for about 24-48 hours, depending on how long it took for the DNS change to my new hosting provider to propagate.