on Dec 12th, 2011SNAPSHOTS, Nightly, Milestones… oh my!
There have been a discussion about build naming conventions lately. SNAPSHOTS, nightly, milestone, GA, maintenance, integration, stable, etc…
In both Nick’s and Mickael’s posts, I noticed a theme: all builds are the same. Of course when they are built, or the process by which the built artifacts are vetted may change, but the build process is the same for all build types: namely, take what’s in HEAD/Master/Trunk/Tip/Top/Main Branch, tag it and build it.
This isn’t the case for the Eclipse project. The Nightly builds are actually done differently from the integration, milestone, stable, and maintenance builds.
The build processes that Nick and Michael mentioned, assume that once something is in Master/Head, it’s ready to be released and the only other variable is release date. The Eclipse process doesn’t assume this. In fact, I can release a wad of changes the night before Juno ships and likely none of them will be in build. This is because Eclipse requires explicit tagging of what you want built.
Here is how it works:
- Development teams continually release stuff to master. Each night master is built to ensure that nobody has introduced compile errors (or broke unit tests). This is not bleeding edge! This is not intended for anyone. This is not similar to maven SNAPSHOTS.
- When teams are happy with their work, they tag the bundle.
- Integration builds happen weekly, and only what’s tagged will be built. Integration builds are similar to SNAPSHOTS, and can be consumed (if you want the most bleeding edge stuff).
- This has recently changed, so instead of tagging each week, we merge (from master into ‘integration’). This means, what’s in master won’t necessarily be in a build each week.
I’m pretty sure I can hear the GitClub screaming feature branch, feature branch, feature branch. Why don’t you just use feature branches?
Please keep in mind:
- Eclipse is over 10 years old, and predates Git. Would you really do feature branches with CVS? (maybe)
- The Eclipse build takes a while to complete:
- Let’s assume it takes 6 hours to build and run the tests (1/4 of a day).
- Let’s assume there are 10 teams (SWT, Debug, Workbench, PDE, JDT, UA, etc…)
- Let’s assume each team has 2 features on the go at all times: 20 feature branches
- Right now (for better or worse) a full Eclipse build, plus platform tests must run on the Eclipse build infrastructure
- So, if we had 20 feature branches that we had to build in sequence, it would take 5 days to build todays nightly
- Tomorrow’s nightly would start once that’s done!
Yes I realize some of those assumptions might be rather simplifying — maybe we could run the builds in parallel and off-load the costs to the hardware, but having a central build of 20 feature branches is still not feasible. Also, some teams would likely want to combined builds (now build the SWT + Workbench feature branch together).
I know some people will head down the path of build restructuring (completely refactor the Eclipse build so developers can run them locally, thus removing the necessity for a centralized builds). We’ve heard this before many times. Instead of suggesting this, sponsoring the effort (with money + people power) would be more helpful. Also remember that Eclipse has successfully shipped on-time for the pat 10 years. So, if you really want the team to change their process to the development flavour of the week, be prepared to show some evidence that process XYZ is better.
This was not meant to start a development war, but rather explain a bit about how the Eclipse build process works. For better or worse, it’s what we have.
BTW, what do others do? Does everyone use feature branches and assume what’s in Master is ready? What did you do before Git (did you use branches for development in CVS/SVN)?
Related posts:
- Merging branches in Eclipse git (EGit)
- Tips for Successful PDE Product Builds
- The Future of Eclipse PDE builds
- Tagging Eclipse for the next integration build
+1 to
“I know some people will head down the path of build restructuring (completely refactor the Eclipse build so developers can run them locally, thus removing the necessity for a centralized builds). We’ve heard this before many times. Instead of suggesting this, sponsoring the effort (with money + people power) would be more helpful.”
I would really love it it people took the time to help improve our build story + donate hardware to make our builds faster. If you look at the Eclipse Hudson instance today, the queues are usually quite busy, especially around release time. I find often find it ironic that people remark that the Eclipse (project) build is so slow etc. but the number of patches I’ve received from people other that our own committers is very small. In fact, the only one I can remember is Anthony from Compuware who submitted a patch for the test framework last year.
Some history lessons.
+ SNAPSHOT means not a release.
+ SNAPSHOT is used for vetting code changes (such as api changes, compile issues, unit testing, integration testing, etc..).
+ SNAPSHOTS can be used by anyone for any non-release purpose.
+ SNAPSHOTS has no guarantees of existing forever. (in fact SNAPSHOT artifacts are continuously cleaned up in active projects)
+ RELEASE is an artifact that has been vetted by the team building it.
+ RELEASE artifacts are tagged in the scm automatically as part of the maven-release-plugin
+ RELEASE artifacts are built / unit-tested from the scm tag as part of the release lifecycle.
+ RELEASE artifacts are never deleted (note: best practice is to use release staging techniques with a proper artifact repository, exposed via a temporary staging repository base url, to prevent bad releases)
The follow are rules placed on RELEASE artifacts if they want to exist on maven central.
+ RELEASE artifacts can have no reference to SNAPSHOT dependencies (direct, transitive, and plugins)
+ RELEASE artifacts are signed by the releasing individual/team (this is not the same as jar signing)
+ RELEASE artifacts have no reference to alternate repositories using the or configurable.
+ RELEASE artifacts must have a -source.jar classifier artifact (if of a java package type: jar, war, etc..)
+ RELEASE artifacts must have a -javadoc.jar classifier (if of a java package type with source)
ok, now that we got that out of the way….
Point #1 in “How it works” is the SNAPSHOT deploy process, nearly verbatim.
Point #2 in “How it works” is the RELEASE process / lifecycle.
Point #3 in “How it works” has no bearing on SNAPSHOT vs RELEASE, that is a CI related task and can constitute use by either SNAPSHOT or RELEASE artifacts and dependencies. And frankly, this needs to not be treated special and done weekly, but should be part of your core CI policy.
In a proper CI infrastructure, using maven and hudson (jenkins too), you have a much more efficient build of the infrastructure, with knowledge of the downstream/upstream deps (at hudson level) and changes / rebuilds in an artifact can cascade build all dependent artifacts as well (even if those artifacts had no code change themselves).
In short.
Use the tools you have correctly: git, maven, and hudson.
You’ll have less work, faster knowledge of impacts in code changes, less noise, less to explain, less redundant builds on CI.
“In fact, I can release a wad of changes the night before Juno ships and likely none of them will be in build. This is because Eclipse requires explicit tagging of what you want built.”
Uh…that is what Branches are for. The problem I’ve always had with the eclipse build project has been the map files. Yes, I know why they are there, to make sure the version numbers don’t change, and items aren’t re-downloaded. That is an eclipse particular requirement, and it DOES make things more complicated from a build stand point.
Your map is basically a Tag/Branch that points to particular items to build. Make life simple, put your changes into a 3.8-Dev branch, and have a 3.8-release branch. Merge changes into 3.8-release and build from that. Same as if you used Master as the vetted code, and a branch for development.
@David: See point 4 above, that’s exactly what we are doing now. Our dev branch is called ‘master’, and our release branch is called ‘integration’.
One of the origin stories for maven was that it was created to ease the effort involved with picking up and contributing to another open source project with apache. The progenitors realized that every T hours someone spent trying to understand and get the build system working and the testing infrastructure working in order to test a simple change was 60 minutes x T that someone could get frustrated and just stop trying to contribute. They realized that if you would create _one_ system that had a lifecycle that someone could learn _once_ and apply that knowledge forward to other use cases…they could cut that time down into a minimal amount. Proof is in the pudding as they say, there are a great many maven projects you can checkout from scm, tweak a single source file, run ‘mvn install’ and have something you can test, or write a unit test for it under a certain directory and it just gets run.
Maven haters will say this never works and I will not disagree that it has failed in some places, I have seen my fair share of terrible maven builds, but the origin of _those_ builds more often then not is someone trying to abuse maven, not maven abusing them.
Anyway, I mention all of this because eclipse has a bespoke build system. I posit the reason you don’t get many contributors to the build system of eclipse is _because_ it is bespoke and it a minor nightmare to work with. The orbit build system is no better, just try and build a local copy of that beast to test your changes, you can’t easily and have to rely on committing changes for a cruise control system to tell you it worked.
If you learn how to use maven, you know how to operate any other maven build. Tycho is a good example of this, there is a new set of plugins to understand but the lifecycle is still there and as rock solid a core as its always been. So there may be plugin differences, doing different things, but fundamentally its all about the lifecycle and how plugins attach to it. If I sank the time necessary into understanding how eclipse builds itself together, what tangible skill does that let a contributor add to their resume?
Skills: ‘I know how eclipse builds! \o\ \o/ /o/’
Not something I look for on a resume
Anyway, my long winded point is that eclipse projects ought to look at how they can further reduce the burden on contributors necessary for them _to_ contribute. A great many projects have moved to git which is a great step, many others are moving to tycho for their plugin development which is another great step.
Which is a much saner approach.
Jesse, there is a historical reason for why we don’t use Maven to build. Until the introduction of Tycho, it didn’t build OSGi bundles. When Eclipse bundles were introduced as artifacts, there wasn’t an existing build system that could build them, this is why we use our own.
We had meetings at the Eclipse Build workshop in the September 2006 with some Maven folks and they asked what they could do to build Eclipse with Maven. We gave them our requirements and said that building the RCP feature would be a good start for them to test using Maven to build OSGi bundles.
http://docs.codehaus.org/display/MAVEN/Barriers+to+Building+Eclipse+with+Maven
Fast forward many years and Tycho is introduced. Igor from Sonatype is working currently on building the SDK with Tycho as part of the Eclipse Long Term support initiative. It will be interesting to see the results of his work.
The reality is that the mandate of the Eclipse project was never to provide a build system. However, this is what we had to do build our artifacts.
Yes, it would be nice if we could build everything the same way. Realistically, this would take a lot of people effort to migrate for the SDK build to a new underlying build engine, when we have these priorities for the next milestone
1) Finish Git migration
2) Switch primary build to 4.2 instead of 3.8
3) Run build on foundation hardware
+ everything else
It’s difficult to get funding as committers to change the underlying build engine when the resulting binary bits will be the same. It’s much more easy to get funding for implementing cool new features and projects …see Orion
@jesse, I’m certainly not a Maven hater — actually, I try not to attach emotional feelings to technology in general — but I think the Eclipse build process deserves *some* credit. You’re right, “I know how to build Eclipse” may not be a selling point on a resume, but if someone told me they “helped architect a development methodology that managed to ship software, on-time, for 10 years straight”, I would be impressed.
A lot has changed in 10 years, but shipping on-time is still something we (as an industry) struggle with. Does the Eclipse build and release process contribute to the success? I don’t know. Will changing to a different system improve things? I don’t know. Do we have something that work? Yes. Could it be better? Yes. sure. Is Maven, Tycho, Jenkins, Hudson, etc, etc, etc… the answer? I don’t know.
Glad to see my article inspired people who do builds to compare different approaches.
The Eclipse Platfrom build is for sure a more complex one than any build I worked on. I was not aware of how it actually works, thanks Ian for sharing these informations.
When reading it, there are for sure ideas that come to minds in order to improve it, but as you notice, having ideas is not enough. Only real contribution and sponsoring matter.
What I think when comparing the Eclipse platform builds with other builds for higher-level projects is that there are different approaches:
* the Eclipse Platform build is what I’d call a “tag-based build”, which means that the team is in charge of setting up a tag on sources, that will represent the built artifacts.
* Other builds, that relies more modern technologies such as Tycho, Git, Hudson… are more often “promotion-based builds”. Which means that builds results are always ready to get released. The team is responsible of ensuring the quality of one of this build to make it a release. Releasing is just a matter of testing and promoting.
As you may have understood, I personnally feed much better with the second kind of builds. It is IMHO the better way to go nowadays. It reduces the overall number of builds and reduce the difficulty to create a release, and it is quite easy to maintain. The future will probably tell us whether this could also apply to the Eclipse Platform build.
I’d love to give a try to improve the Platform build one day, but like any developer and project, I’d need for sponsoring for doing it
Some good points:
- For any non-simple project, the build should always be against some “integration” branch. Development happens in feature branches (which aren’t built on the CI server by default). That isolates developers and allows to develop features that span several releases (among other things).
It also has some smells. For example, the build it taking much too long. Where is that time spent? My guesses from my shallow knowledge of the Eclipse build process:
- You build things like org.eclipse.core.resources every time even if they didn’t change. Solution: Dependency management and a more fine grained release process. The release train is nice from a consumer point of view (developer using the IDE, not working on it) but this is clearly one undesirable artifact.
- Tests take too long. The main reason here is that the Eclipse design actively prevents mockup tests. For example, if you want to test any code related to SWT, you need to start a graphical environment (an X server). I’ve rubbed my brain against StyledText for several weeks. With a patched SWT, I could run thousands of unit tests in the same time it takes to run the setup of a single test with the real thing. The same is true for anything that depends on the Workbench, Bundles and similar stuff that depends on old-style singletons.
@Aaron
> You build things like org.eclipse.core.resources every time even if they didn’t change. Solution: Dependency management
> and a more fine grained release process
But remember, Eclipse uses the JDT (which is obviously a moving target) to compile Eclipse. So, just because the source doesn’t change, doesn’t mean the binaries won’t change. We need to know if / when this happens.
Good point.
Does it only rebuild everything when the compiler source code changes?
JDT is a lot of code: The Java debugger, UI code, Java indexing, resource bundles for UI texts. Is the build system fine grained enough to handle this?
@Aaron,
No, I don’t think we have anything that sophisticated. Also, Eclipse uses it’s own build tools (PDE/Build), so there’s another moving target.
So right now we re-compile everything each time just to be sure.
You’re right, there are plenty of areas to improve things. But as Kim and I both mentioned, this costs money. It’s also very difficult to convince the ‘Penny Pushers’ that changing something that works is a good idea.