There are a lot of improvements in the pipe for Nick and myself to make build more and more agile, and to make it produce more and more interesting results. I feel very interested in going a step ahead of continuous integration, and open the road towards continuous improvement, automating QA to get reports about static analysis and code coverage on each build. I bet it will help developers to avoid mistakes and improve there code. (Advertising: You can learn more on this topic during next EclipseCon ).

Here are some of these topics that are worth this blog post:

QA with Sonar

About static analysis, my choice fully goes to Sonar, a webapp that provides a dashboard aggregating reports from several QA “services” (Findbugs, Checkstyle, Test reports, coverage…). Sonar provides nice integration with Maven and Jenkins & Hudson, and getting the whole working together is about a few minutes of configuration. The return on investment when setting up Sonar is very high and immediate. In my humble opinion Sonar is a must-have nowadays.

So, there are 2 bugs I encourage you to look at/vote for, in order to leverage Sonar, QA and continuous improvements in your favourite projects:

• @Eclipse.org : Set up a Sonar instance of Eclipse projects
• @JBoss.org: Set up Sonar for JBoss Tools (and more JBoss stuff?)

Jacoco, without Sonar

Jacoco is a very easy-to-set-up code coverage tool, with a very convenient Maven integration. With Jacoco, there is no need for instrumentation step to get coverage results, all is about giving a -java.agent=jacocoagent.jar property to your JVM. While using it, I did not detect any effect on performances. That’s pretty cool.

The current issue for Jacoco adoption is the reporting. Here are the known ways to analyse reports from a Jacoco coverage output file (aka jacoco.exec):

• Use the Jacoco Ant task to generate HTML reports
• Use the Jacoco maven:site integration to generate reports as part of maven:site
• Une EclEmma plugin to analyse Jacoco reports in your IDE
• Use Sonar

As a fan of Sonar, I think the Sonar-based approach is the best one. But both previous bugs show something clear: despite of my full enthusiasm for Sonar, community-wide infrastructures such as Eclipse.org or JBoss.org cannot set up a Sonar instance immediately. I guess this is because of issues with credentials, or hardware, or IT stuff. So we need to be able to consume Jacoco reports without Sonar.
The issues with all these approaches is that it introduces the need for a new build step or a new developer tool to analyze reports. The main dashboard for JBoss.org builds in Jenkins (and Hudson for Eclipse.org), then the ideal place for such reports would be  a Jenkins plugin for Jacoco. Unfortunately this does not existt (yet), but a Jenkins issue is open for this. Now that I have lost some optimism in getting Sonar available for Eclipse.org or JBoss.org community quickly (as an impatient guy, quickly means today), this Jacoco Jenkins is now my #1 wish in the whole Java software world. Although setting up Jacoco for execution is very easy, setting up integrated reporting is still almost a blocking point for adoption.

So please, vote for this bug!!! iJacoco Jenkins plugin

Some other stuff

Since I’m writing here a wish list, I’d also like to share with you thib Jenkins issue I opened: Add an option to select Build status when no Test is found. The idea is simple: Even if no-one should never do that, it happens that we need to run a Jenkins build with skipped tests. Ok, it’s bad, but it happens. In such case, the fileset for Test results in Jenkins JUnit plugin does not match any existing report, and then build turn to FAILED/Red. When you have cascading of jubs, this is pretty annoying, since you’d rather see this build UNSTABLE/Yellow -my favourite- or SUCCESS/Blue. So this issue simply requests the ability to change the result of plugin when no test is found.
If you find this useful, you can vote for it.

Working for PetalsLink was a quite interesting experience:

On the technical side, I enjoyed moving all the XML-based tooling of PetalsStudio to a more powerful EMF-based approach for Petals JBI editor – for those who don’t know JBI, it is a standard that allows to define SOA artifacts in your ESB. Moving to EMF allowed us to provide better tooling faster, because most of the complexity in manipulating JBI can be removed with very few efforts leveraging EMF ExtendedMetaData. That was the first time I faced this part of EMF, and I got pretty impressed of how well it works (working with EMF always gives this impression of “being well”). I also improved the ability to plug new JBI components into the Studio, which is a critical point when you have to deal with connectors for almost everything – Mail, SFTP, Talend, XSLT…. So that was an interesting challenge in term of conception and development.
Petals Studio was also the pretext to start using Git, GitHub and Sonar. I am pretty happy to have learnt these 3 tools that clearly improved the way I work.

Also, I had the great opportunity to work closely to several Eclipse projects:

• I could contribute the Tycho build of GMF Tooling, put it on Hudson, get source moved to Git/mirrored to GitHub, improve wiki… GMF Tooling is a project I’ve used for 3 years now and I often saw in it some critical organization points to improve to make it more dynamic in term of development. Working at PetalsLink gave me the opportunity to do what I think was necessary to keep the project healthy. With the help of Michael Golubev, I now think this was an real success.
• I could contribute to Nebula the TreeMapper widget, which will probably have some very interesting use-cases soon. As I became a committer, I also helped in improving Tycho build and CI, nad it seems like the project liked it if we look at the new p2 update-sites.
• I contributed some small improvements to Eclipse BPEL designer, tried (unsuccessfully) to make SWTBot use Tycho, and developed a useful extension for Draw2d.

The only thing I wish I would be able to do here is to push ahead the usage of Sonar at Eclipse, at least for GMF Tooling and Nebula.

But I probably learnt even more things from PetalsLink by discovering another company organisation that is very different from what I could experiment before (OpenWide and BonitaSoft): PetalsLink is focused on the Research about SOA and agility of Systems of Information. It is a wide topic! Petals products are quite good compared to other alternatives in the SOA landscape, but they don’t meet the success they deserve, it was a bit frustrating for a developer.

I enjoyed working for PetalsLink, all the expectations are fulfilled, so it is time for me to go ahead, to find a new experience, a new team, new challenges, new issues… I love discovering new things!

That’s why I’ll start working for JBoss|Red Hat tomorrow, as part of the team developing JBoss Tools and JBoss Developer Studio.

I’ll have the opportunity to work with a great team! My main occupation for the next monthes will be to assist Nick Boldt in making JBoss Tools CI and build infrastructure better and better. I’d also like to open the road towards efficient QA for JBoss Tools, including -among other- usage of Jacoco and Sonar. Then I’ll also work on developing nice stuff for some JBoss Tools modules, most probably on the SOA/BPM part.

That’s gonna be a lot of fun! I’m eager to be tomorrow and actually get started for this new team/employer/project/product/users.

Let’s keep in touch via this blog and twitter

Background

Historically, Eclipse has 4 to 5 classical qualifier for binary artifacts:

• Release
• Maintenance
• Stable
• Integration
• Nightly

This wording is specific to Eclipse, only Eclipse people do understand the meaning of it. Even worse, some of these are not used accurately. Although this is more or less official wording, I am not sure it is used by most projects.

This guy has driven a revolution in the way we build and deliver software

Now  Eclipse.org provides continuous integration, to automate and manage builds executions, and most of them happen whenever a change happen on your VCS. Continuous integration has highly changed the way binary are produced and made available to consumers, it is now much easier to get builds, lots of project have a ping of less that 10 minutes between a commit and a build ready to be released.

That was the starting point of my thoughts, with the Nebula build job: Why calling nightly a job that does run on a build any time a commit happen?

Requalifying binaries to make consumption clearer

As a producer of builds, here are my opinion on these qualifiers:

• Release: The heartbeat of the product for consumers
• Maintenance: is a release, but with 3rd qualifier digit different than 0
• Stable: Is nothing but an integration build that was put on a temporary update-site for the release train. And to be honest, I don’t use it, I directly point the latest good continuous integration builds to the release train builder
• Integration: The heartbeat of the product for developers, built on each commit
  
• Nightly: What is in it? What does nighly mean at the same time for a developer in Ottawa and a developper in Beijing? Who cares that it is build nightly and not a 2pm CEST? For most projects, the nightly built artifacts are not at all different from the ones that would have been built with the last commit. What is the added value of artifacts built during a nighly scheduled build over the latest artifact built from last commit? Why building something every night if there was no commit in it for 3 monthes?

So I am in favor of removing Maintenance, Stable and Nightly. That gives

• Release
• Integration

Wow, 2 kinds of builds, reminds me a tool… Oh yes! Maven! Maven has 2 types of binaries: Release and SNAPSHOTS. That’s great, we finally arrive to the same conclusion as Maven, except that what Maven calls “snapshot” is called “integration” in the Eclipse teminology.

But now, let’s consider the pure wording: why having 2 names for the same thing? How to call a binary that is work in progress? An “Integration” or a “snapshot”?

Let’s be pragmatic: This report tells us there are about 9000000 Java developers in the world. This onetells us 53% of this population use Maven. Let’s say that about 60% of Maven users do understand the meaning of SNAPSHOT. That means 9000000 * 53% * 60% = 2,862,000 people do know what SNAPSHOT means.  Wayne confirmed me that the size of the Eclipse community is 132,284 people, who might know the meaning of “integration” in an Eclipse update-site name. That’s the number of people who have an account on eclipse.org sites – wiki, forum, bugzilla, marketplace. Even if we assume that 100% of them do understand the differences between the different qualifiers, and I am pretty sure that less than the 600 committers actually do, that makes that Snapshot is 21 times a more popular and understood word than integration.

So, Eclipse projects and update sites would be easier to understand and consume for the whole Java community by accepting the Maven wording, and using it on the websites and update-sites name.
Following the Maven naming and the release/snapshots dogmas would make consumption easier, but also avoid duplication of built artifacts, and also make things go more “continuously”. Your project is on rails, it’s going ahead with Snapshots, and sometimes make stops at a Release. That’s also a step ahead towards the present (see how GitHub works) of software delivery: continuous improvement, continuous delivery.

Requalifying build job to make production clearer

So now let’s speak about build management, no more about delivery.

You need several jobs to keep both quality and fast enough feedback, then set up several jobs!

Dave Carver reminded me about some basics of Continuous Integration: keep short feedback loops, one builds for each thing. That’s true. If you have a “short” build for acceptance and a “long” build for QA, you need to have separated jobs. Developers need acceptance feedback, they also need QA feedback, but they for sure cannot wait for a long build to have short-term feedback. Otherwise, they’ll drink too much coffee while waiting, it is bad for their health.

But do not create several builds until you have real needs (metrics can also help you to see whether you have need). If you have a 40 minutes full-build, that fails rarely, with slow commit activity, and that nobody is synchronously waiting for this build to go ahead, then multiplying builds and separating reports could be expensive for low Return On Investment. That’s the case for GMF-Tooling: we have a 37 minutes build with compile + javadoc + tests + coverage + signing, and we are currently quite happy with it, no need to spend more effort on it now. Let’s see how we feel when there will be a Sonar instance at Eclipse.org and that we’d have enabled static analysis… Maybe it will be time to split job.

Avoid scheduling build jobs, it makes you less agile

Before scheduling a job that could happen on each commit, just think about how long will be the feedback loop between a commit and the feedback you’ll get: it is the time that will happen between the commit and the clock event starting your job. It can be sooooo long! Why not starting it on commit and get results as soon as possible? Also why schedule and run builds when nothing may change between to schedule triggers?
I can only see one use case where scheduling job executions is relevant: when you have lots of build stuff in the pipe of your build server, and you have limited resources. Then, in that specific but common case, you want to set up priorities: you don’t want a long-running QA job to slow down your dev team who is waiting for feedbacks from a faster acceptance build. For this reason, I would use scheduling, but I would do so because I have no better idea on how to ensure the team will get the acceptance feedback when necessary, it is a matter of priority. Maybe inversting in more hardware would be useful. Then you could stop scheduling, and get all your builds giving feedbacks as soon as they can. Nobody would wait for a schedule event to be able to go ahead the project.
As I said to Zoltán on Twitter: “The best is to have all build reports as soon as possible, as often as possible” (and of course only when necessary, don’t build something that has already been built!). Scheduling goes often against those objectives, but it is sometimes helps to avoid some bottlenecks in the build queue, and then save time projects.

Name it to know what it provides!

Remember Into the Wild or Doctor Zhivago... "By its right name"...

Ok, at Eclipse, there are “Nightly” jobs. I don’t like this word. Once again, “Nightly” is meaningless in a worldwide community. And the most important issue are “What does this build provide?”, “Why is it nightly, can’t I get one on each commit?”.
If this build is run on each commit, then don’t call it “Nightly”, because it feeds the consumer with false information. You can think about having both “acceptance” and “QA” jobs, then put that in their names rather than a schedule info, that’s a far more relevant information.

Conclusion

Continuous integration has changed the way we produce and deliver software, we must benefit of it and adopt the good practices that come with it. Continuous integration is the first step towards what seems to be the present of near future of software delivery: continuous improvement and continuous delivery. We must not miss that step if we want to stay efficient.

• http://eclipse.org/nebula/widgets/treemapper/treemapper.php
• http://www.eclipsecon.org/2012/sessions/%C2%A1new-nebula-treemapper-widget

Very often, one has to define links or mapping between objects. Here is the way I learnt to represent the concept of matching items when I was 4.

A matching game

Now I’m 26 and I use the concept of mapping everyday at work. It can represent reference, transformation, association… But that’s now more complicated, because I now use structured data!

With the BPEL “Assign” element in mind, I wanted to find a widget that allowed me to map structured data, represented as trees, using the simple “Draw a line” method that I have been using for years. Except that “Draw a line” would be represented by “Drag’n'drop” on a computer. After some research, I could not find an open-source widget for that. Then I started to develop a new one. And here is what it looks like:

The TreeMapper in action

Although it is primarly intended to be used in Eclipse BPEL Designer, I made it part of Nebula because I hope it is helpful for lots of other projects in the Eclipse community. I can remember some projects I work(ed) on that have nice use-cases for this widget in the past (for Eclipse JWT and Scarbo, Bonita Studio, Petals Studio…); and I now think about how to simplify the GMFMap editor with it.

If you are interested in it and if you plan to attend EclipseCon 2012, then vote for this submission to learn more: http://www.eclipsecon.org/2012/sessions/%C2%A1new-nebula-treemapper-widget

PS: If you like it, then thank the French National Research Agency for funding it

GMF Tooling 2.4.0 “for Indigo” released

A lot of people were waiting for it, it finally occurs: GMF Tooling (finally) has a release that works with the Eclipse Indigo release! It is mainly made of bug fixes and compatibility improvements. Here is the p2 repository: http://download.eclipse.org/modeling/gmp/gmf-tooling/updates/releases/

A new lead: Michael “Borlander” Golubev

After lots of mails, the GMF Team, helped by the Modeling PMC, was able to nominate a new lead to overview the GMF Tooling contributors team and development. He is Michael Golubev, often known as “borlander” on bugs and forums. who works for Montages as a full-time developer for GMF Tooling. He has also been the lead of UML2 Tools.

Simple build process thanks to Tycho and host build on hudson.eclipse.org

GMF Tooling now has a Tycho builder, that is far easier to maintain and run than the legacy one. So contributors can now run tests very easily with a “mvn clean install” to ensure their work did not break anything. That makes contributing much easier. Moreover, the build is hosted on hudson.eclipse.org so that it is easy and transparent to get an idea of how healthy is the code. Also, going to continuous integration on Eclipse servers allows to produce builds that are equivalent to the one that will be released (including signing and all the necessary Eclipse stuff). So there is no more additional difficulty building a release than building a snapshot.

Get GMF Tooling back into the Modeling discovery service

The Modeling Discovery wizard is a wizard that appears when downloading the Eclipse Modeling package to suggest you some projects to install and use. GMF Tooling just get back into it as I am writing this post!

Guarantee GMF Tooling will make it in Juno release train

We also made the efforts to ensure the future of GMF Tooling will be less chaotic than it was for Indigo. We already did most of the necessary stuff to get GMF Tooling in the Juno release train. So, no stress this year! More details here.

Improved documentation

See this effort in my previous post. This is an undefinitely work in progress, so feel free to contribute directly by making the wiki easier to navigate.

So… what’s next?

A lot of things + what the community will think about contributing. The project plan for GMF-Tooling 3.0 (yes, 3.0!) is not yet finished, but here are some key objectives:

• Easier and more intuitive Tooling – with a high-level graphical editor to define your own graphical editor (a “meta-”editor)
• Improve integration/collaboration with other Modeling projects (EEF, XText…)
• Move to Git
• Enroll more contributors in GMF Tooling development
• Simplify generator code
• Extensibility of generator and tooling to make it easy to add support for new things in GMF Tooling from 3rd-party bundles.

I think GMF Tooling just achieved a major step, and I bet this is the beginning of a new, leaner, era for Graphical Modeling!

SoftShake 2011 was now several weeks ago! This was a very nice conference, quite well organized, with different tracks that make easy for you to always find something interesting to learn.

Some sessions I liked

Before speaking of the sessions I like, I have to say that there were a lot of sessions in just 2 days, and I had been busy during the conference to talk with some people during the sessions. So I missed most of them, maybe some very good ones. So this is not of “best of all sessions”, but more a “best of the 9 sessions I saw”.

Kudos to Hamlet D’Arcy who presented how the compile-time annotations, such as the one proposed by the Lombok project, are processed by compilers to modify the internal AST model of your code, and add behavior to your code automatically. It was interesting, simple to understand, and easy to find how it applies in everyday’s developer life.

I also liked the presentation “Data Grids vs Databases” from Galder Zamarreño from the Infinispan team. I am not very easy will all this Data-driven stuff, but after this presentation, I now know when to use a Data Grid or a Database. However, I am still not sure how all these things compare to Object Databases, such as Objectivity.

And I also found the présentation from Sébastion Douche interesting. It told us the story of the product and team he manages, how his team leverage Git, how they industrialize builds, and everything they do to make their working life easier. That’s alwasys good to hear that kind of tips.

Our sessions

Here is the slides of my “Modeling with Eclipse” presentation, with materials available on GitHub:

I enjoyed a lot presenting it, although I had my laptop taking lot of time between 2 operations, so that demonstrations got longer than expected. I counted 26 attendees. I hope I gave them enough examples to understand how powerful is the Modeling ecosystem at Eclipse, and how productive are the tools available.

And here are the slides from my colleague Jean-Christophe Reigner (PetalsLink Chief Services Officer) that he used to explain what is an ESB, why and when it is useful, and to show some concrete use-cases of ESB:

I liked his presentation, it was quite clear, and I could learn more about ESB in a general IT architect point of view. I think people in the room felt the same thing.

As a speaker

As a speaker too, it was really very cool to be there since the organizing team made everything necessary to get speakers happy, providing help and chocolate during the conference, and inviting us to a very good restaurant where we could discover, for those who did not know that already, the Swiss gastronomy.

Swiss Gastronomy Part I: "La fondue moit-moit"

Swiss gastronomy Part II: "Les meringues double-creme"

So, I hope I’ll make it again for SoftShake 2012 !

When you write a UI test, your test does depend on UI components you use. This dependency is specific to your test bundle, and then must be explicitly defined in your MANIFEST.MF. For example, if your test will click on the “New > Java Project” menu, so it highly depends on org.eclipse,jdt,ui, which provides this contribution. Then do not forget to add this in your dependencies!

A bit easier than "Where is Waldo"

It can work in some cases when you are sure your test platform already contains the contribtutor of the UI elements you manipulate (here org.eclipse.jdt.ui) . Then the menu is already there – as a 3rd party contribution-, although you did not add the dependency to it in your test. But that’s more or less a lucky case, or a case that requires rigorous management of your test platform.

With Tycho, your test platform is, by default, made of your test bundle and all its dependencies (computed from MANIFEST.MF). Then if you don’t explicit your dependencies to UI contributions, your test will probably run in a target platform which does not include the UI elements you interact with, and will fail. When you have this depedency to the UI contributor (such as org.eclipse.jdt.ui) in your test MANIFEST.MF, you are sure you’ll have the menu available whenever you execute your test. Moreover, you are sure that all installation of your test bundle with p2 will contain the necessary stuff to get it working.

To sum it up: If your test depends on UI elements, then it depends on plugins that contribute these UI elements. So tell it in its MANIFEST.MF. That’s all!

The "SoftShake speaker badge"

I am very excited by this presentation since I think Modeling is something very productive and powerful, and I know how helpful are the tools provided by Eclipse. It is the first time I have the opportunity to democratize Modeling, Model-Driven Stuff and Eclipse Modeling Project in a conference! However, this requires a lot of work to prepare a good presentation, but that’s a work I love to do.

On a site note, my PetalsLink colleague Jean-Christophe Reigner was also accepted as a speaker. He will democratize the usage of ESB: what use-cases it resolves, what is its role in your SI and so on. That should be also quite interesting since ESB are very powerful and scalable middlewares and most people don’t get the actual value yet. So if you are interested in SOA, System of Informations, Middleware, ESB and so on, you should for sure come to see his talk, you’ll definitely learn interesting things.

GMF-Tooling and Indigo...

However, this was just an exception: GMF Tooling is already back on the Juno train! In order to be more reactive to the release train requirement, GMF Tooling moved its build to Tycho, making the build system quite easy to maintain, and release train rythm much easier to follow.

Thanks to Tycho, GMF-Tooling is already in the train

And you can see it here:alrea

GMF Tooling in Juno

Then you won’t have to search a lot in order to get GMF Tooling in Juno.

The first answer I have is in refactoring documentation to show more “Getting started” tutorials. Then people need to spend less time to find a way to get started.

What you see on the main GMF wiki page

Also, I thought the tutorial was too monolithic. There are a lot of things you can do with GMF, but not all will be useful for your use-case. Users need to clearly understand what use case of piece of documentation actually resolves, in order either to skip it, or to spend more time on it. This can be achieved by reorganizing the titling of the documentation. Then it becomes easier to find out what are the different steps when creating a diagram, which ones you can skip, which ones interest you.

Structured organization and titling

Finally, by digging into current GMF documentation, I discovered that GMF has lots and lots of resources to help people to leverage it. There are several tutorials, including one that includes Flash videos to show how to generate a diagram. This is a very-high value resource, but it was unfortunately not very easy to find. This resource deserved to be highlighted! A sad thing is that (AFAIK) there is currently no way to integrate these movies into the wiki. I hope Bug 352735 gets fixed soon so that people will really benefit from these tutorials without effort.

If you know anything you could improve to the refreshed documentation, please do it. Remember of your first experience with GMF or of the first time you met a classical use-case: How did you search answer? How could the answer be easier to find? How could the answer be easier to understand? What in the documentation makes it less efficicent? If you have ideas, feel free to edit the documentation accordingly.

I learnt a few lessons from this work:

• Very new users are really the people to target for tutorials.
• For tutorials, a video is more attractive and efficient than 5 pages of documentation.
• Do not try to be exhaustive in tutorials, prefer being modular or pluggable, so that the tutorial remains easy to follow, and that people can extend it by adding a sub-sub-title. It will make the tutorial easier to maintain, without making it easier to read (people are able to skip your “tutorial modules”)
• Make your tutorial incremental. Resolve use-case one after the other. Be very explicit about the use-case. People need to understand where and why you go before reading the “How”
• Sometimes, there is alreadly a lot of documentation available, there is no need for more. Instead, it can be useful to spend time on organizing documentation and making it more visible. A documentation that nobody finds or reads is very sad, it is waste.

These are my very first steps with documentation, and my first thoughts after working a little bit on GMF documentation. There are probably some things you’d like to tell me on this topic according to your experience. I’d be glad to learn from you, please tell me