The engineering team at Jive Software is growing fast (btw, we’re hiring!). I thought it might be interesting to talk about some of the engineering process changes we’re making to cope with that growth, especially since they directly affect product releases. First, a bit of history. We’ve always had a fairly agile development process – lots of iterations, tools like unit tests and continuous integration, etc. But, we’ve consistently had a lot of pain around our current process:
Not enough time for QA* . It’s always scheduled, but gets squeezed due to lack of time. For example, the “official” QA time period for the Spark 2.5.0 release got crunched down to almost nothing.
Stress when we don’t need it* . The stress is caused by having to cram a ton of work into a short period of time to make internally set product release dates. It’s also caused by scheduling extra features into releases at the last minute and by having to do emergency patch releases due to bugs we missed. We like working really hard, but there should be a way to do that without excessive stress.
Unpredictable release dates* . The bigger a release is, the harder it is to make accurate work effort estimates. That means that dates slip and that it’s hard to communicate to the outside world exactly when a release will ship.
So, there are some clear problems that we want to fix. The trick for us was to come up with a process that fixes those problems but that’s still light-weight enough to avoid a huge administrative burden. We’re now experimenting with a strict release train model for Spark and Smack. Assuming it goes well, the same model will be applied across all our products, including Clearspace. A visual summary of the model is below (click the image for a larger version):
We put out a new release every three weeks (although each release will have gone through a nine week process total).
Three weeks before each development cycle is reserved for product management; three weeks after each development cycle is reserved for QA. But notice that all three processes are all happening at the same time.
There are all sorts of subtleties to the system like how to deal with new features that take more than three weeks of engineering time, but so far everything is going quite well.
The start of the three week cycle is always on a Monday, and we have a big meeting to determine exactly which features will go into the release. Product management has already done the work of prioritizing the features so we just need to choose exactly what we’re going to work on and come up with the plan about how to get it done over the next three weeks. On the second and third Mondays during each cycle we go through more detailed updates than in our SCRUM meetings to see what’s on track and what’s not.
At the end of the cycle, we:
Do the product release coming out of QA on Thursday.
On Friday, finish all development and create a branch (“spark_2_5_2_branch” for example).
Switch the builds in our continuous integration environment, TeamCity. The new branch becomes the “stable” release of the product and trunk becomes “development”.
Release a beta version out of the new branch. In other words, the world generally always sees a stable official product release and a beta release that will become official in three weeks.
There’s way more that I could talk about with this process (including some of the potential drawbacks), but I’ll save that for future blog entries. Things we’re jazzed about so far include solid release dates, having a better process in place to deal with new feature requests and bug reports, more stable quality in each release, and always having a place to check in new feature development (trunk). We’re still early on in the new system, but we’ll all know how exactly how well this works in the next two months. Of course, we’d love your feedback as time goes on about how well you think we’re doing.
The release train sounds like a great idea; it’ll be interesting to see how it’s received by the community. It seems like there is always a balancing act going on between getting new releases out the door that include new features/bug fixes while at the same time not having users feel like they constantly have to upgrade.
This sounds like a great thing. Do we know what features will be included in the first release train once 2.5.0 is finalized??? I’m hoping SSO is since it was promised to be in 2.5.0 but couldn’t make it. I’m really looking forward to SSO as a part of Spark.
Excellent post! Thank you for sharing! I’m curious why CI is done off the “stable” branch instead of the trunk. Is this because the code base is not always “pristine” (buildable)? If so, does that make it more difficult to integrate changes between multiple developers?
White Spy – I also answered your question in your own blog entry. But to re-paste here: we do continuous integration on both trunk and the branch at the same time ??? that???s what “development” and “stable” are.
Hello, we have isa server 2004 and we are installing spark program 2.5.2 we can chating word but for voice we can’t speaking what we do or what is voice port number
i want permission voice on isa server2004 but i dont know how pls help we thank you
I am suspicious of any software process that claims to make releases predictable. Making release DATES predictable is easy, but – now that you have had some time to try out this process – are you finding that these releases are also high-quality and sufficiently feature-rich to justify the overhead (regression testing, upgrade support, etc.) of each new release? Have you had to tweak the process yet?
Mark – I completely agree. Any process that promises to fix everything is obviously BS. But, we’re quite happy with what this set of limitations has given us so far. I’m planning to do a follow-up blog entry with more details on our progress in the not too distant future. One thing that’s emerged already is timing big demos and massive happy hours for the end of each train. We had a total blow-out happy hour with an ice theme on Friday. Maybe I’ll have to get some pictures up…
the big point is knowing how far along in a given project you are by a binary method, not by asking the developer ???how far along are you???. you???ll see in the video, he explains it way better than i could.
I actually wanted to comment on how things are going for me personally. I think the one BIG benefit of this is that it allows the engineer to stay very focused on the tasks at hand and to be less reactionary to feedback. I’ve talked with matt about this in depth and I think he would agree that when you take out some of the “free time” in development, the experimentation side of development just doesn’t happen as much, and the quality does go up. However, my one complaint is that the creativity part of the job goes down just a bit. No longer can I change the color of a tab to lighter blue without it being approved for the release heh.
This sounds like classic waterfall process, only with fewer stages in the pipeline and using Scrum to encapsulate the missing pieces. We use something similar for the company I work at, but I’m not happy with it. We have a weekly release cycle here, which compounds the problems. QA’s builds are usually quite out of synch with the dev-team’s builds. Finding bugs introduces “bubbles” into the pipeline which delays the release train.
I think it’d be much better to use an extreme programming-style development model. That is, each iteration, estimate what goes into the product and work to make it a reality. Things which do not make it in (including partially implemented features) are dropped in favor of maintaining the scheduled release date. Dropped features are deferred for the subsequent iteration’s planning session. This may take longer to “finish” the product, but the goods are always on time and of very high quality.
i tried Sparks. Looks fine. But 75MB for roster and one chat? ROFL! Hire some stuff for real programing and use brain! Do you use own product? Or your standard is 4GB RAM? I’m glad you program “just” IM
Why don’t you help to make better Gajim, PSI or other good programs, insted of tons of code ended with useless SW? Nice pictures and skins are the last, first you need program.
Try google to find some basics about memory management and Jabber clients
[Clearspace|http://www.igniterealtime.org/blog/wp-content/uploads/2007/04/train.png|train_sm all.png]