The Butler Dispute, Round 2

Nearly 4 month have passed by since the renaming of Hudson project to Jenkins, which marks the climax of the dispute between the old Hudson developers and the guys from Oracle and Sonatype.

Jenkins has made a great job since then, delivering 15 releases in a weekly schedule. Build number 1.400 was hit in March, wich is not a particular significant release but shows how well things go on. The way from Hudson to Jenkins is as easy as it could be, and it seems like many users are going it.

Indeed there are a lot of reasons why to choose Jenkins over Hudson, just to name a few:

• Support by the fabulous Hudson core development team – with Kohsuke Kawaguchi, the creator of Hudson, and other brave guys.
• Strong community activity – measured in figures like commit counts and mailing list traffic, see this post for some numbers.
• Most of the plugins moved over to Jenkins – 5 of the top 5 and 19 of the top 25 plugins continue primary development with Jenkins, see here for some statistics.
• High quality and regular releases – the weekly schedule led to 15 high quality releases, each of them providing a couple of bug fixes and new features (see changelog). Moreover, a few weeks ago, Jenkins governance board proposed to start another release line for most stable baselines with a 3-months schedule.

Even the Hudson board seems to have observed that Jenkins outperforms Hudson in many ways, at least they are thinking about "how to make it more attractive for plug-in developers to support both Hudson and Jenkins" (see this message on Hudson-Dev list). The author's perception is that "Hudson also appears to be slowing down development wise" and "another place where Hudson appears to be slowing down, is when you compare changelogs". Some of the ideas deal with copying approaches that are working fine for the Jenkins project.

Hence, it seems Jenkins is the winner of the battle and has in fact benefited from the fork.... until today.

Because today, Oracle submitted a proposal to move Hudson to the Eclipse Foundation. This is, well, somewhat astonishing since that means Oracle will lose both control and the Hudson trademarks – which was the main background of the original dispute with the community.

As part of the proposal, other big players have announced support for the project, including IBM, VMware, Tasktop and Intuit. That means, moving the Hudson project to Eclipse will for sure result in higher attention and more resources (developers).

Does this change anything? Will Jenkins be the unlucky loser, after all? I don't think so. The heavens didn't really smile on Hudson since the fork (kind of bad karma) and I don't see why the move to Eclipse should change that. It's all about people, not code.

Moreover, Jenkins has been invited by Sonatype to reunite with Hudson. But... why should they do that? Jenkins is a vibrant project today, so what is the benefit? Also, there have been some deep disappointments on personal level that are not forgotten yet.

It's going to be interesting!

Jenkins: Pimp It Up!

Some days ago, I started to review what plugins are available for Jenkins, my favorite CI server. I haven't done so for a long time, so I was somewhat surprised to see a full universe of plugins (380+) listed in the Wiki...

There is next to everything you can imagine. Among the plugins I would like to suggest for consideration are these:

• Dependency Graph View Plugin – Shows the dependency graph of the Jenkins projects using graphviz. This greatly helps to keep track of dependencies between all your Jobs.
  
  



• Disk Usage Plugin – This plugin calculates and records disk usage (space for builds and workspace) per project and per build, and can display trend graphs.
  
  



• Global Build Stats Plugin – can be used to gather and display global build result statistics, monitoring over time, and show nice graphics.
  
  



• And, of course, all the static analyzing plugins that scan result files of several static code analysis tools and visualize the results as trend graphs:
  • Checkstyle Plug-in
  • Cobertura Plugin
  • FindBugs Plugin
  • JavaNCSS Plugin
  • PMD Plugin
  • Task Scanner Plugin
  • Warnings Plugin
  
  
  

Have fun!

DocBook with Maven Issue

We are using DocBook for writing technical documentation for all our projects and in-house frameworks. We are actually quite happy with this approach, especially because we are able to automatically publish the docs in a number of formats, including HTML and PDF. To do so, we use the docbkx-maven-plugin in the project's nightly build.

So, all has been in best order... until I decided to upgrade this docbkx-maven-plugin from version 2.0.8 to current version 2.0.11 (due to some issues we had). After doing so, the document conversion issues an error which breaks the build:

[ERROR] Failed to execute goal com.agilejava.docbkx:docbkx-maven-plugin:2.0.11:generate-pdf (pdf) on project builddoc-ma
ven-plugin: Failed to transform to PDF: org.apache.fop.fo.ValidationException: null:30:723: Error(30/723): fo:table-body
 is missing child elements.
[ERROR] Required Content Model: marker* (table-row+|table-cell+)

Well, this is somewhat unexpected because I didn't change anything but the plugin version, and I don't see any reason it could not work as before. In particular, we are still using the same docbook version in our POM. Here is the relevant snippet:

<plugin>
  <groupId>com.agilejava.docbkx</groupId>
  <artifactId>docbkx-maven-plugin</artifactId>
  <version>2.0.11</version>
  <dependencies>
    <!-- the DocBook XML DTD and catalog files (see http://www.oasis-open.org/docbook) -->
    <dependency>
      <groupId>org.docbook</groupId>
      <artifactId>docbook-xml</artifactId>
      <version>4.4</version>
      <scope>runtime</scope>
    </dependency>
  </dependencies>

  <executions>
    <execution>
      <id>pdf</id>
      <goals>
        <goal>generate-pdf</goal>
      </goals>
      <phase>post-site</phase>
      <configuration>
        ...
      </configuration>
    </execution>
    ...
  </executions>

  <configuration>
    <htmlStylesheet>css/html.css</htmlStylesheet>
    <htmlCustomization>${basedir}/src/doc/xsl/html_chunk_customization.xsl</htmlCustomization>
    <foCustomization>${basedir}/src/doc/xsl/fopdf_customization.xsl</foCustomization>
    ...
  </configuration>
</plugin>

It's important to understand that we are using the advanced customizing capabilities of DocBook, i.e. we customized the stylesheets used for rendering HTML and PDF. The created custom stylesheets contain an import to urn:docbkx:stylesheet, and in the Maven POM the htmlCustomization and foCustomization properties point to those custom stylesheets. This is how it's supposed to be, and this is how it worked all along.

I found out that the error message is correct when building with plugin version greater than 2.0.8, since the for-each element indeed does not return any element which results in an empty fo:table-body. In fact, none of the xsl:value-of in our customized stylesheet returned any value any more....

So here is why: since docbkx-maven-plugin version 2.0.9, the plugin is using namespaced stylesheets. That is, we must use a namespace in our custom stylesheet to be able to select any docbook element! See this or this post for related comments.

Thus, all I have to do is to add the docbook namespace declaration at the top and add the docbook namespace prefix to all references to element names in my customization layer. See highlighted lines in this XSL snippet:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:fo="http://www.w3.org/1999/XSL/Format"
                xmlns:db="http://docbook.org/ns/docbook"
                exclude-result-prefixes="date"
                version="1.0">

  <xsl:template name="book.titlepage.separator">
    <fo:block>
      <fo:table table-layout="fixed" width="163mm">
        ...
        <fo:table-body text-align="left">
          <xsl:for-each select="/db:book/db:bookinfo/db:revhistory/db:revision">
            ...
          </xsl:for-each>
        </fo:table-body>
      </fo:table>
    </fo:block>
  </xsl:template>
  ...
</xsl:stylesheet>

Well, that did the trick – after spending a couple of hours of investigation... I think that issue should be clearly noted with the docbkx-maven-plugin Maven plugin, because in the end it is an incompatability between versions 2.0.8 and 2.0.9. Alas, I did not find this information on the plugin's Changes Report page. At least, nothing that pointed me (not being a DocBook expert) into this direction... :-(

The Way From Hudson To Jenkins

Some time has gone by since the Hudson/Jenkins fork... and there has been even more talk in the community. However, slowly the dust settles, everybody is getting back to business. And finally, we decided to switch from Hudson to Jenkins! This is about why and how.

Why move to Jenkins?

But wait: who has forked, anyways? Is it Jenkins that forked Hudson, or is it Hudson that did the fork of Jenkins? There is some evidence that the community just did a rename of the project (due to trademark conflicts), and after that Oracle forked Jenkins, using the Hudson name they claim holding the trademark on.

You may think this question is a purely theoretical one, but actually it's not. I'll have to legitimate the decision to move to Jenkins to my stakeholders, and using a fork would be a "smell". Project forks are usually not as good as the "original", are possibly done out of selfish reasons, are considered to harm the community etc. Hence, not moving to a fork but instead following the "real" project is a good reason for the move to Jenkins.

An even better one is "project vibrancy", that is the pace of development and level of support provided by the community. This is usually measured by indicators such as the number of commits, the mailing list traffic, the quantity, quality and regularity of releases etc. See this post for such an analysis on commit counts and mailing lists post counts. This is more than four weeks old now and covers not more than two weeks, but nevertheless the result is obvious: Jenkins moves much faster than Hudson does, and community is much more agile. This is confirmed by following the dev mailing lists of both: for Hudson, most of the relevant posts are by either Oracle or Sonatype engineers – seems the Hudson community has become pretty small... Moreover, as this post shows, most of the top plugins will continue primary development under Jenkins.

Last not least, I really respect Kohsuke Kawaguchi (the original creator of Hudson) and what he has done for us. I feel ashamed by how Oracle is dealing with him and the rest of the core team, that's why I have a strong tendency to follow the "good guys" with Jenkins.

As I blogged before, Maven integration is probably one of the most important features of any CI server (at least for me). I guess Sonatype is doing better with Maven integration – it's "The Maven Company", right? – and they are working with Oracle on Hudson. At least, they are putting huge efforts into rock-solid integration. However, after having seen a Sonatype Webinar about their plans with Hudson, I'm not that convinced any more. Current features looked a bit awkward and also does the GWT based UI they are using. So, from my point of view, this point is not yet decided.

Counting it all together, there are some really good reasons to move from Hudson to Jenkins, so we did.

How to upgrade

Now... how do you actually migrate from Hudson to Jenkins? Well, it couldn't be easier. There is a Wiki page about Upgrading from Hudson to Jenkins. To make it short, the involved steps are:

1. Backup your current installation – just for the good feeling.


2. Change Update Site: In your Hudson, go to Manage Hudson > Plugin Management > Advanced > Update Site and enter "http://updates.jenkins-ci.org/update-center.json" as URL for Jenkins update site.


3. Choose to upgrade automatically on Manage Hudson page, just as you did so many times to update Hudson. This will download the new JAR.


4. Restart Hudson, eh, Jenkins.... and there it is!

That's it. Took less than 5 min! Hudson indeed is a drop-in replacement, so you usually do not have to change anything (environment variables, system properties, start scripts, job configuration etc).

Well, there is only one thing: the name of the WAR file is still hudson.war! Is Oracle aware of this? ;-)

The Butler Dispute

I thought it would be time to resurrect my blog, after not having posted for a couple of months. That was mainly because I have been really busy with some interesting stuff I should post about in the future, like Xtext upgrade...

Oracle vs. the Community

But today, I just bumped the ongoing dispute between Hudson and Jenkins guys. We are using Hudson since around 2008, coming from Cruise Control. We really liked the web interface, being able to setup everything by just using your browser. Of course, the features also have been impressive since then. Setting up a build farm is just fun with Hudson.

You probably know that there has been a fork of Hudson which is named Jenkins (others say Hudson has been renamed to Jenkins and then forked into Hudson). This all started with the Hudson team being unhappy with the infrastructure provided by java.net (which is driven by Oracle since Sun acquisition), due to its poor reliability. The community talked about moving parts of the project to other servers, and first candiate was issue tracking. Suddenly, the project is locked due to the migration of java.net projects to new Kenai infrastructure, which was announced by Oracle but somehow missed by the project owners. Frustrated by the migration, the community decides to move code to GitHub and mailing list to Google Groups. See "Who's driving this thing?" for the facts.

This is the point where Oracle steps in, claiming to have a trademark on the name. If the project decides to move, it must be using another name: "Because it is open source, we can't stop anybody from forking it. We do however own the trademark to the name so you cannot use the name outside of the core community. We acquired that as part of Sun." (BTW, that might not be true after all). Later he stated that "the final decision of what to do w.r.t. infrastructure belongs to Oracle".

Guess what: this really concerned the community. There have been some talks between key community members and Oracle representives, in an attempt to agree on a "proposal for a stable structure and arrangement" which later would be proposed to the community. But, with no success. That finally led to the decision of the community to move to GitHub and at the same time rename the project to another butler's name: "Jenkins".

Of course, Oracle's view on the subject is a bit different...

Welcome Jenkins!

Well, so now you have the choice: use Hudson, or use Jenkins. You know, competition is usually a good thing, so let the race begin. The majority of the community seems to have made the switch to Jenkins (given the blogs and mailing list traffic). This is because Oracle's behavior is not quite understood and does not cast a positive light on their comprehension of Oracle's role in the Hudson community.

However, Oracle is putting enormous resources (people and hardware) into the Hudson project. And what's even more important, Sonatype is helping to drive Hudson to the next level. Sonatype? Right, that's the company behind Maven and all the great Maven tools like Nexus and m2eclipse.

Maven Support – the Killer Feature?

One of the most important features (for me, but also for possibly the majority of other users) is Maven 3 support. Sure, Hudson/Jenkins already support Maven 3 since version 1.392 (end of 2010, see changelog). But hey, Sonatype entered the scene, and they will surely do better.

Sonatype, too, have put some full-time engineers into the Hudson project, making sure that "Hudson users can look forward to a long, bright future". See this or this post. Sonatype in the end of the day wants to earn money with Hudson (and Maven), so I expect to see outstanding features related to Maven 3 support, Eclipse integration and workflow extensions for Hudson. See here for some of their ideas.

Well, this really makes a thrilling game. I honestly appreciate what Kohsuke Kawaguchi and others have built up with Hudson from the ground up, and would like to see them win on the "evil company that pushed them out of the project". And by the way, Sonatype seems to be in good companion when talking about being evil – they removed the oldest commiter of m2eclipse from the project a year ago.

So, is this again the good vs. evil story? I don't know. In the end, both projects will have their users. And they will learn and benefit from each other. So let's wait and see.... Time will tell. It's going to be an interesting year, though!