Archive for category Uncategorized
Today is my last day working for PetalsLink.
Working for PetalsLink was a quite interesting experience:
On the technical side, I enjoyed moving all the XML-based tooling of PetalsStudio to a more powerful EMF-based approach for Petals JBI editor – for those who don’t know JBI, it is a standard that allows to define SOA artifacts in your ESB. Moving to EMF allowed us to provide better tooling faster, because most of the complexity in manipulating JBI can be removed with very few efforts leveraging EMF ExtendedMetaData. That was the first time I faced this part of EMF, and I got pretty impressed of how well it works (working with EMF always gives this impression of “being well”). I also improved the ability to plug new JBI components into the Studio, which is a critical point when you have to deal with connectors for almost everything – Mail, SFTP, Talend, XSLT…. So that was an interesting challenge in term of conception and development.
Petals Studio was also the pretext to start using Git, GitHub and Sonar. I am pretty happy to have learnt these 3 tools that clearly improved the way I work.
Also, I had the great opportunity to work closely to several Eclipse projects:
- I could contribute the Tycho build of GMF Tooling, put it on Hudson, get source moved to Git/mirrored to GitHub, improve wiki… GMF Tooling is a project I’ve used for 3 years now and I often saw in it some critical organization points to improve to make it more dynamic in term of development. Working at PetalsLink gave me the opportunity to do what I think was necessary to keep the project healthy. With the help of Michael Golubev, I now think this was an real success.
- I could contribute to Nebula the TreeMapper widget, which will probably have some very interesting use-cases soon. As I became a committer, I also helped in improving Tycho build and CI, nad it seems like the project liked it if we look at the new p2 update-sites.
- I contributed some small improvements to Eclipse BPEL designer, tried (unsuccessfully) to make SWTBot use Tycho, and developed a useful extension for Draw2d.
The only thing I wish I would be able to do here is to push ahead the usage of Sonar at Eclipse, at least for GMF Tooling and Nebula.
But I probably learnt even more things from PetalsLink by discovering another company organisation that is very different from what I could experiment before (OpenWide and BonitaSoft): PetalsLink is focused on the Research about SOA and agility of Systems of Information. It is a wide topic! Petals products are quite good compared to other alternatives in the SOA landscape, but they don’t meet the success they deserve, it was a bit frustrating for a developer.
I enjoyed working for PetalsLink, all the expectations are fulfilled, so it is time for me to go ahead, to find a new experience, a new team, new challenges, new issues… I love discovering new things!
I’ll have the opportunity to work with a great team! My main occupation for the next monthes will be to assist Nick Boldt in making JBoss Tools CI and build infrastructure better and better. I’d also like to open the road towards efficient QA for JBoss Tools, including -among other- usage of Jacoco and Sonar. Then I’ll also work on developing nice stuff for some JBoss Tools modules, most probably on the SOA/BPM part.
That’s gonna be a lot of fun! I’m eager to be tomorrow and actually get started for this new team/employer/project/product/users.
Let’s keep in touch via this blog and twitter
This is a blog post to sum up some of my thoughts I shared on the Nebula-dev mailing list with Wim Jongmam and later on Twitter with Zoltán Ujhelyi and Dave Carver about naming build types at Eclipse.org and scheduling them.
It is a topic open to debate. My goal with it is to find out practices and names that are relevant and useful for community (contributors and, mainly, consumers) and also to give food for thoughts about how to deal with builds.
Historically, Eclipse has 4 to 5 classical qualifier for binary artifacts:
This wording is specific to Eclipse, only Eclipse people do understand the meaning of it. Even worse, some of these are not used accurately. Although this is more or less official wording, I am not sure it is used by most projects.
Now Eclipse.org provides continuous integration, to automate and manage builds executions, and most of them happen whenever a change happen on your VCS. Continuous integration has highly changed the way binary are produced and made available to consumers, it is now much easier to get builds, lots of project have a ping of less that 10 minutes between a commit and a build ready to be released.
That was the starting point of my thoughts, with the Nebula build job: Why calling nightly a job that does run on a build any time a commit happen?
Requalifying binaries to make consumption clearer
As a producer of builds, here are my opinion on these qualifiers:
- Release: The heartbeat of the product for consumers
- Maintenance: is a release, but with 3rd qualifier digit different than 0
- Stable: Is nothing but an integration build that was put on a temporary update-site for the release train. And to be honest, I don’t use it, I directly point the latest good continuous integration builds to the release train builder
- Integration: The heartbeat of the product for developers, built on each commit
- Nightly: What is in it? What does nighly mean at the same time for a developer in Ottawa and a developper in Beijing? Who cares that it is build nightly and not a 2pm CEST? For most projects, the nightly built artifacts are not at all different from the ones that would have been built with the last commit. What is the added value of artifacts built during a nighly scheduled build over the latest artifact built from last commit? Why building something every night if there was no commit in it for 3 monthes?
So I am in favor of removing Maintenance, Stable and Nightly. That gives
Wow, 2 kinds of builds, reminds me a tool… Oh yes! Maven! Maven has 2 types of binaries: Release and SNAPSHOTS. That’s great, we finally arrive to the same conclusion as Maven, except that what Maven calls “snapshot” is called “integration” in the Eclipse teminology.
But now, let’s consider the pure wording: why having 2 names for the same thing? How to call a binary that is work in progress? An “Integration” or a “snapshot”?
Let’s be pragmatic: This report tells us there are about 9000000 Java developers in the world. This onetells us 53% of this population use Maven. Let’s say that about 60% of Maven users do understand the meaning of SNAPSHOT. That means 9000000 * 53% * 60% = 2,862,000 people do know what SNAPSHOT means. Wayne confirmed me that the size of the Eclipse community is 132,284 people, who might know the meaning of “integration” in an Eclipse update-site name. That’s the number of people who have an account on eclipse.org sites – wiki, forum, bugzilla, marketplace. Even if we assume that 100% of them do understand the differences between the different qualifiers, and I am pretty sure that less than the 600 committers actually do, that makes that Snapshot is 21 times a more popular and understood word than integration.
So, Eclipse projects and update sites would be easier to understand and consume for the whole Java community by accepting the Maven wording, and using it on the websites and update-sites name.
Following the Maven naming and the release/snapshots dogmas would make consumption easier, but also avoid duplication of built artifacts, and also make things go more “continuously”. Your project is on rails, it’s going ahead with Snapshots, and sometimes make stops at a Release. That’s also a step ahead towards the present (see how GitHub works) of software delivery: continuous improvement, continuous delivery.
Requalifying build job to make production clearer
So now let’s speak about build management, no more about delivery.
You need several jobs to keep both quality and fast enough feedback, then set up several jobs!
Dave Carver reminded me about some basics of Continuous Integration: keep short feedback loops, one builds for each thing. That’s true. If you have a “short” build for acceptance and a “long” build for QA, you need to have separated jobs. Developers need acceptance feedback, they also need QA feedback, but they for sure cannot wait for a long build to have short-term feedback. Otherwise, they’ll drink too much coffee while waiting, it is bad for their health.
But do not create several builds until you have real needs (metrics can also help you to see whether you have need). If you have a 40 minutes full-build, that fails rarely, with slow commit activity, and that nobody is synchronously waiting for this build to go ahead, then multiplying builds and separating reports could be expensive for low Return On Investment. That’s the case for GMF-Tooling: we have a 37 minutes build with compile + javadoc + tests + coverage + signing, and we are currently quite happy with it, no need to spend more effort on it now. Let’s see how we feel when there will be a Sonar instance at Eclipse.org and that we’d have enabled static analysis… Maybe it will be time to split job.
Avoid scheduling build jobs, it makes you less agile
Before scheduling a job that could happen on each commit, just think about how long will be the feedback loop between a commit and the feedback you’ll get: it is the time that will happen between the commit and the clock event starting your job. It can be sooooo long! Why not starting it on commit and get results as soon as possible? Also why schedule and run builds when nothing may change between to schedule triggers?
I can only see one use case where scheduling job executions is relevant: when you have lots of build stuff in the pipe of your build server, and you have limited resources. Then, in that specific but common case, you want to set up priorities: you don’t want a long-running QA job to slow down your dev team who is waiting for feedbacks from a faster acceptance build. For this reason, I would use scheduling, but I would do so because I have no better idea on how to ensure the team will get the acceptance feedback when necessary, it is a matter of priority. Maybe inversting in more hardware would be useful. Then you could stop scheduling, and get all your builds giving feedbacks as soon as they can. Nobody would wait for a schedule event to be able to go ahead the project.
As I said to Zoltán on Twitter: “The best is to have all build reports as soon as possible, as often as possible” (and of course only when necessary, don’t build something that has already been built!). Scheduling goes often against those objectives, but it is sometimes helps to avoid some bottlenecks in the build queue, and then save time projects.
Name it to know what it provides!
Ok, at Eclipse, there are “Nightly” jobs. I don’t like this word. Once again, “Nightly” is meaningless in a worldwide community. And the most important issue are “What does this build provide?”, “Why is it nightly, can’t I get one on each commit?”.
If this build is run on each commit, then don’t call it “Nightly”, because it feeds the consumer with false information. You can think about having both “acceptance” and “QA” jobs, then put that in their names rather than a schedule info, that’s a far more relevant information.
Continuous integration has changed the way we produce and deliver software, we must benefit of it and adopt the good practices that come with it. Continuous integration is the first step towards what seems to be the present of near future of software delivery: continuous improvement and continuous delivery. We must not miss that step if we want to stay efficient.
This blog is not yet ready-to-go. I’ll soon update it to post stuff related to Eclipse development, among other things. But for the moment, there is nothing to read here. See post on http://www.bonitasoft.org/blog/author/mickael-istria-eclipse/ .