Software development. And stuff.

Just got back from the O’Reilly Open Source Convention in Portland. Fantastic conference – met lots of really interesting people (and the odd nutter).

It was a good conference for SiteMesh. It opened my eyes to two things:

# SiteMesh rocks. People who have tried SiteMesh, love it and don’t turn back. Their preferred choice for web framework changes, but SiteMesh remains constant.
# Our marketing sucks. Despite it being around for 5 years, most of the Java web app community have never felt the need to try it.

I was there to present a session on SiteMesh but a lot of other speakers beat me to it. It kept slipping into other sessions…

h4. Using AppFuse for Test driven Web Development, Matt Raible (details)

Matt gave an overview of the technology stack used in his AppFuse application. Despite having 5 versions of his app that use different frameworks (Struts, WebWork, Tapestry, Spring MVC and JavaServer Faces), all used SiteMesh. Good!

h4. Integrate: Building a Site from Open Source Gems, Erik Hatcher (details)

Erik walked us through the open source products he used to build his Lucene Book website and what customizations he made. The focus, of course, was Lucene and I learned a lot of great tricks about Lucene that hadn’t occurred to me before – such as using “sounds like” queries with soundex and indexing images by colors. I continue to love Lucene.

A great point that Erik mentioned was the need to become intimate with the projects you use. If you truely want to make the most of your frameworks, understand how they work, join the community and extend them.

Erik chose Tapestry to build the site but he also had Blojsom and some static content, so SiteMesh was useful to integrate these and he created some custom code to build SiteMesh decorators with Tapestry.

He pointed out that despite submitting this useful Tapestry integration to the SiteMesh project, nothing had made it into the SiteMesh release. Feeling embarressed, I committed his changes immediately, inadvertently breaking the build and providing great ammunition for Eric Pugh’s session on the importance of continuous integration.

h4. WebWork vs Spring MVC Smackdown, Matthew Porter and Matt Raible (details)

The basic plot was this… Matthew Porter was arguing why Spring MVC sucks and WebWork rocks. Matt Raible was arguing why Spring MVC rocks and WebWork sucks. The only thing they both agreed on was SiteMesh rocked. A fairly heated and passionate debate – great fun to watch. I would have opted for more violence though.

Matthew Porter got the final laugh when he pointed out that he compared the Spring MVC and WebWork versions of Matt Raible’s AppFuse framework and the Spring MVC version had about 25% (I think) more code, not including comments.

(more)

h4. The Evolution of Web Application Architectures, Craig McClanahan (details)

This was an interesting session where Craig compared the approaches taken by Struts, WebWork, Spring MVC, Tapestry and JavaServer Faces. He had done detailed research and, despite his heavy involvement with Struts and JSF, gave a very fair and objective view of the pros and cons of each.

This work could be useful for people evaluating which frameworks to choose and possibly could be overlayed with a guide based on values. The bottom line is there’s no single ‘ultimate’ web framework and depending on your needs and values you should choose the most suitable. I think it would be beneficial to all to have a guide indicating which values each of these frameworks are suited/not-suited for.

So, my question is this: Which values are more important to you when choosing a web framework and in which priority?

These are some example values that spring to mind: commercial support, testability (unit and functional), popularity, extensibility/customization, integration with other frameworks, rich widget support, REST friendlyness, simplicity vs magicness, AJAX friendlyness, learning curve, configuration, etc.

Anyhoo, SiteMesh was probably mentioned enough times to attract another load of people to the SiteMesh session.

I’m really glad these people mentioned SiteMesh and said such kind words about it – it resulted in a lot of interest and a full house for the SiteMesh session. I hope to get these presentations online shortly and write a bit more about how Subversion, Microsoft Word and SiteMesh can be combined to create a rich Content Management System.

h2. Improving SiteMesh’s marketing

The fact still remains that SiteMesh has terrible marketing. I’d love some ideas of how to spread the word more and encourage more people to try it but I honestly have no idea what to do. Any suggestions?

Before talking about how the new SiteMesh HTML processor works (to be released in SiteMesh 3), I thought I’d write a bit about how the current parser has evolved since it’s first attempt in 1999 – purely in the interest of nostalgia.

The original version used a bunch of regular expressions to extract the necessary chunks of text from the document. This was easy to get running, but very error prone as the matches had no context about where they were in a document. For example, a <title> element in a <head> block is very important to SiteMesh, however sometimes they appear elsewhere, such as in a comment, <script> or <xml> block.

This was dumped, in favour of a DOM based parser, which initially used JTidy to convert HTML to XHTML so it could be traversed as a standard DOM tree. Much nicer, but very slooow. Too slow, so I switched to OpenXML, an XML parser that was tolerant to nasty HTML, giving a slight boost to performance. I was much happier with OpenXML – even though it still added a fair amount of overhead and rewrote bits of HTML that I didn’t want it to.

Annoyingly, not long after that, the OpenXML project merged with the IBM XML4J parser project, rebranded itself as the mighty Apache Xerces and promptly dropped support for HTML parsing. So now I was dependant on a library that no longer existed.

By this time, SiteMesh had been open-sourced, and along came Victor Salaman, who was the third user to discover it (after Mike Cannon-Brookes and Joseph Ottinger). He saw the potential but hated the parser. About three hours later, he’d produced his own version that used low-level string manipulation. It wasn’t pretty, but it went like the clappers – twelve times faster than the OpenXML one, with the bonus feature of not rewriting great chunks of the document. This brought SiteMesh into the mainstream as it was now ready for use on high-traffic sites. 1.0 was released.

This parser really is the core of SiteMesh. It’s been our friend thanks to its speed and reliability. It’s been our enemy because of it’s awkwardness to understand and change. For a couple of years it remained barely untouched, except when we occasionally poked at it from afar with a long pointy stick for the odd change. Three years later, Chris Miller and Hani Suleiman took the plunge and gave its guts an overhaul – making it six times faster! Very brave.

Despite its awkwardness, it proudly lived on and is still the primary ingredient of SiteMesh today. It’s even been ported to VB.Net!

I’ve kept my eye on other HTML parsers, such as HotSAX, NekoHTML and TagSoup, always with the intention of implementing an easier to maintain parser, but I just couldn’t get the performance to be anything like what Victor, Chris and Hani achieved.

The problem is that most HTML parsers try to represent an HTML document as tree of nodes, like XML. This makes sense as that’s what HTML is meant to be, however, to do this, every single tag in a document must be analysed and balanced accordingly. This is hard, error-prone and adds a lot of overhead.

There’s another approach though. The new parser focusses on ease-of-use and ability to customize, without compromising on performance and robustness. I hope you’ll like it…

Update: Sorry, I forget to mention Hani in the original posting of this. how could I forget!