Call a spade a spade, and a Nightly a Snapshot

This is a blog post to sum up some of my thoughts I shared on the Nebula-dev mailing list with Wim Jongmam and later on Twitter with Zoltán Ujhelyi and Dave Carver about naming build types at Eclipse.org and scheduling them.
It is a topic open to debate. My goal with it is to find out practices and names that are relevant and useful for community (contributors and, mainly, consumers) and also to give food for thoughts about how to deal with builds.

Background

Historically, Eclipse has 4 to 5 classical qualifier for binary artifacts:

  • Release
  • Maintenance
  • Stable
  • Integration
  • Nightly

This wording is specific to Eclipse, only Eclipse people do understand the meaning of it. Even worse, some of these are not used accurately. Although this is more or less official wording, I am not sure it is used by most projects.

This guy has driven a revolution in the way we build and deliver software

Now  Eclipse.org provides continuous integration, to automate and manage builds executions, and most of them happen whenever a change happen on your VCS. Continuous integration has highly changed the way binary are produced and made available to consumers, it is now much easier to get builds, lots of project have a ping of less that 10 minutes between a commit and a build ready to be released.

That was the starting point of my thoughts, with the Nebula build job: Why calling nightly a job that does run on a build any time a commit happen?

Requalifying binaries to make consumption clearer

As a producer of builds, here are my opinion on these qualifiers:

  • Release: The heartbeat of the product for consumers
  • Maintenance: is a release, but with 3rd qualifier digit different than 0
  • Stable: Is nothing but an integration build that was put on a temporary update-site for the release train. And to be honest, I don’t use it, I directly point the latest good continuous integration builds to the release train builder
  • Integration: The heartbeat of the product for developers, built on each commit
  • Nightly: What is in it? What does nighly mean at the same time for a developer in Ottawa and a developper in Beijing? Who cares that it is build nightly and not a 2pm CEST? For most projects, the nightly built artifacts are not at all different from the ones that would have been built with the last commit. What is the added value of artifacts built during a nighly scheduled build over the latest artifact built from last commit? Why building something every night if there was no commit in it for 3 monthes?

So I am in favor of removing Maintenance, Stable and Nightly. That gives

  • Release
  • Integration

Wow, 2 kinds of builds, reminds me a tool… Oh yes! Maven! Maven has 2 types of binaries: Release and SNAPSHOTS. That’s great, we finally arrive to the same conclusion as Maven, except that what Maven calls “snapshot” is called “integration” in the Eclipse teminology.

But now, let’s consider the pure wording: why having 2 names for the same thing? How to call a binary that is work in progress? An “Integration” or a “snapshot”?

Let’s be pragmatic: This report tells us there are about 9000000 Java developers in the world. This onetells us 53% of this population use Maven. Let’s say that about 60% of Maven users do understand the meaning of SNAPSHOT. That means 9000000 * 53% * 60% = 2,862,000 people do know what SNAPSHOT means.  Wayne confirmed me that the size of the Eclipse community is 132,284 people, who might know the meaning of “integration” in an Eclipse update-site name. That’s the number of people who have an account on eclipse.org sites – wiki, forum, bugzilla, marketplace. Even if we assume that 100% of them do understand the differences between the different qualifiers, and I am pretty sure that less than the 600 committers actually do, that makes that Snapshot is 21 times a more popular and understood word than integration.

So, Eclipse projects and update sites would be easier to understand and consume for the whole Java community by accepting the Maven wording, and using it on the websites and update-sites name.
Following the Maven naming and the release/snapshots dogmas would make consumption easier, but also avoid duplication of built artifacts, and also make things go more “continuously”. Your project is on rails, it’s going ahead with Snapshots, and sometimes make stops at a Release. That’s also a step ahead towards the present (see how GitHub works) of software delivery: continuous improvement, continuous delivery.

Requalifying build job to make production clearer

So now let’s speak about build management, no more about delivery.

You need several jobs to keep both quality and fast enough feedback, then set up several jobs!

Dave Carver reminded me about some basics of Continuous Integration: keep short feedback loops, one builds for each thing. That’s true. If you have a “short” build for acceptance and a “long” build for QA, you need to have separated jobs. Developers need acceptance feedback, they also need QA feedback, but they for sure cannot wait for a long build to have short-term feedback. Otherwise, they’ll drink too much coffee while waiting, it is bad for their health.


But do not create several builds until you have real needs (metrics can also help you to see whether you have need). If you have a 40 minutes full-build, that fails rarely, with slow commit activity, and that nobody is synchronously waiting for this build to go ahead, then multiplying builds and separating reports could be expensive for low Return On Investment. That’s the case for GMF-Tooling: we have a 37 minutes build with compile + javadoc + tests + coverage + signing, and we are currently quite happy with it, no need to spend more effort on it now. Let’s see how we feel when there will be a Sonar instance at Eclipse.org and that we’d have enabled static analysis… Maybe it will be time to split job.

Avoid scheduling build jobs, it makes you less agile

Before scheduling a job that could happen on each commit, just think about how long will be the feedback loop between a commit and the feedback you’ll get: it is the time that will happen between the commit and the clock event starting your job. It can be sooooo long! Why not starting it on commit and get results as soon as possible? Also why schedule and run builds when nothing may change between to schedule triggers?
I can only see one use case where scheduling job executions is relevant: when you have lots of build stuff in the pipe of your build server, and you have limited resources. Then, in that specific but common case, you want to set up priorities: you don’t want a long-running QA job to slow down your dev team who is waiting for feedbacks from a faster acceptance build. For this reason, I would use scheduling, but I would do so because I have no better idea on how to ensure the team will get the acceptance feedback when necessary, it is a matter of priority. Maybe inversting in more hardware would be useful. Then you could stop scheduling, and get all your builds giving feedbacks as soon as they can. Nobody would wait for a schedule event to be able to go ahead the project.
As I said to Zoltán on Twitter: “The best is to have all build reports as soon as possible, as often as possible” (and of course only when necessary, don’t build something that has already been built!). Scheduling goes often against those objectives, but it is sometimes helps to avoid some bottlenecks in the build queue, and then save time projects.

Name it to know what it provides!

Remember Into the Wild or Doctor Zhivago... "By its right name"...

Ok, at Eclipse, there are “Nightly” jobs. I don’t like this word. Once again, “Nightly” is meaningless in a worldwide community. And the most important issue are “What does this build provide?”, “Why is it nightly, can’t I get one on each commit?”.
If this build is run on each commit, then don’t call it “Nightly”, because it feeds the consumer with false information. You can think about having both “acceptance” and “QA” jobs, then put that in their names rather than a schedule info, that’s a far more relevant information.

Conclusion

Continuous integration has changed the way we produce and deliver software, we must benefit of it and adopt the good practices that come with it. Continuous integration is the first step towards what seems to be the present of near future of software delivery: continuous improvement and continuous delivery. We must not miss that step if we want to stay efficient.

About these ads
  1. #1 by Zoltán Ujhelyi on December 7, 2011 - 1:25 pm

    Thanks for writing this blog post. And at least here I have more then 140 characters to respond. :)

    I mostly agree: maintenance and stable is somewhat redundant (I don’t use them very often). I have a single case for maintenance builds: they represent for the end-user that its binary compatible (as suggested in Eclipse versioning). Ok, I know, that this information is already represented in the version number, so it really is redundant.

    About scheduling: if possible, then yes, you need a full build every time. Even in case of multiple builds. That depends on the project and infrastructure.

    However, changing the build type names might help Java developers (esp. Maven users), but it would create some kind of confusion for existing Eclipse developers, who uses builds. Existing documentation, articles, blog posts use this terminology, including various resources available from eclipse.org. As these sources are here to stay, a very clear and unambigously specified translation shall be available, and they have to be pushed hardly.

    Alltogether, I don’t mind the renaming, but I think, that will be a hard shift.

  2. #2 by Pascal Rapicault on December 7, 2011 - 3:11 pm

    The problem you show is that there is a plethora of names for only two different types of build:
    – Integration, which is building from known tags
    – Nightly, which builds from whatever the trunk of our SCM contains (remember this got invented

    Stable, Integration, Milestone (you forgot that one) are just Integration builds with various level of testing and control applied to the input, which set consumers expectations. For example an Milestone build is a special Integration build that received more testing, the Final build is a milestone build with even more testing. Maintenance builds are different since they represent the axis of maintenance and on this axis there also is some Integration builds.

    I agree that with Git and Hudson the process could be rethought and it is already happening, however I’m not convinced that adopting the Maven convention is not necessarily the best thing either because it sets expectations that are not met.

    Snapshots in an OSGi universe end up having a real version (I know snapshot have a real version too) whereas in OSGi you see “Snapshot”, and you refer to them differently when you express a dependency (e.g 1.0.0-SNAPSHOT not a range)
    As for release, the process of creating a release build is not promotion of a well tested content but a rebuild (which IMO invalidates any sort of manual testing you could have done since the bytes you are delivering are not those that you tested). For example if someone had committed something between the snapshot you tested and the moment you cut your release, then you would get it… Of course there are ways around this (shameless product plug) with the awesome staging functionality of http://sonatype.com/nexus

    I’m all ear on seeing how we can improve this issue.

    • #3 by Mickael Istria on December 7, 2011 - 3:27 pm

      I received the same kind of feedback from Paul Webster via Twitter, and I think the issues you show here are true for the Eclipse project, but not for most of other projects at Eclipse.org.

      My post and suggestion indeed forget the use-case of Eclipse Platform project, and probably others, that have several levels of acceptance.

      > because it sets expectations that are not met.

      Which ones?

      > As for release, the process of creating a release build is not promotion of a well tested content but a rebuildwell tested content but a rebuild

      In projects I work for, release is a promotion, not a release. And it is quite convenient and efficient. Things are way easier when any build, even a snapshot (or a Nightly) should be ready to be promoted as a release.

      • #4 by Paul Webster on December 7, 2011 - 4:01 pm

        I guess it’s about legacy. The Eclipse project used those names and they mean specific things, and make sense. Even nightlies, which are that way because of the commit rate for a large # of components coupled with scarce build resources. As Eclipse was the first project, those terms have been bandied around in the new projects … with less and less meaning. EGit comes to mind where they only have a snapshot build that they publish during development (every night :-), and release builds. All the other build names probably don’t make sense in the context of a smaller/newer project.

  3. #5 by Ian Bull on December 7, 2011 - 11:26 pm

    You can call your non-integration builds whatever you want. I prefer the name nightly as it gives me some indication of how often it’s build (every night). But if you want to call them SNAPSHOT, or UNSTABLE, go for it.

    However, make sure the artifacts have ‘real’ version numbers. Bug reports that say: 1.0.0-SNAPSHOT has an NPE with the following stacktrace, are meaningless, unless you only ever have one 1.0.0-SNAPSHOT. 1.0.0-SNAPSHOT-20111201, that at least tells me that it was the version built on Dec 1st 2011, now we can start to make sense of this.

    Also, please make sure that your version numbers are monotonically increasing.

    • #6 by Mickael Istria on December 8, 2011 - 2:14 pm

      > You can call your non-integration builds

      Ok, but what is an “integration build” and a “non-integration” build? Only a very few people know the difference, using this words makes Eclipse build practices less transparent. That’s what I’d like to make clearer by finding better wording.

      > I prefer the name nightly as it gives me some indication of how often it’s build (every night).

      But is it useful to have this build scheduled nightly. Why not just running it when necessary, ie when something happened to the code? And then call it snapshot, or latest or unstable…?

      > However, make sure the artifacts have ‘real’ version numbers
      > Also, please make sure that your version numbers are monotonically increasing.

      Versions have a timestamp as a qualifier, for snapshots or releases. In this post, I do not speak about versions qualifiers, but about build and artifacts description. Nothing technical such as versions is in the topic, just agility and communication.

  4. #7 by Nick Boldt (@nickboldt) on December 11, 2011 - 8:16 pm

  5. #8 by David Williams on December 15, 2011 - 5:52 am

    One thing I haven’t seen mentioned in these discussions (sorry if I missed it) is the matter of prereqs. I know from my WTP experiences, our “milestones” very much cared and were meant to line up with other’s milestones (or, releases, if no changes were coming) whereas our “integration” builds are not so strict about lining up with other’s I-builds. Sometimes they might (if we knew of a sensitive change) but more often not, since … the idea was … for 4 or 5 weeks we’d focus only on “our stuff” and not be distracted by churn below us, then spend a week or so to make sure we also integrated exactly with our “prereq stack” (through both builds and testing efforts). And “releases” at Eclipse are special, since it means not only is code ready and lined up prereqs are ready, but also that the IP staff (and others) at Eclipse are ready. So, that’s why I think there are three levels. I’m sure there are ways to change terminology that “still fits” (without having these exact three names) … but … thought I’d make this small point about prereqs coordination and “legal clearance” being important factors.

  1. Mickael Istria: Call a spade a spade, and a Nightly a Snapshot
  2. Call a spade a spade, and a Nightly a Snapshot | Eclipse | Syngu
  3. SNAPSHOTS, Nightly, Milestones… oh my! « EclipseSource Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: