Improving the software development process

As many know, there has been a push in the community to move our VCS from subversion to git. I agree with them that it is time to make that move, and I hope the other contributors and especially the project leads will get on board. I had reservations about making this move myself, not so much specific to git itself, but for the fact that I didn’t think that it offered enough advantage over the cost of actually doing the move due to the limited resources available. If we were all paid to do this kind of work, these issues simply wouldn’t exist. It isn’t simply switching version control, but the interactions between that and the existing infrastructure (Jira and Bamboo) and the like and also implementing new software, build and release processes.

With regards to the software development process, this is where git, or more specifically, some of the available management tools for git really shine. They add more than enough value to the process that I think we should adopt them.

So, I am proposing that we get and set up Atlassian Stash (the enterprise version of BitBucket). It has excellent integration with the existing development tools we are using and will provide a much better software development and management process then we currently have (which admittedly isn’t that hard).

  • Branches can be linked to their Jira tickets.
  • Code review of a pull request will automatically update a Jira ticket to code review status.
  • Accepting a pull request will update the status to resolved.
  • Configure builds to automatically build any branch.
  • Force merge of pull request to require a successful build before allowing a merge.
  • Inline code reviews
  • Would allow private repos
  • Can be branded to Igniterealtime

From what I know, the feature set is very similar to Github, but I don’t know where Github stands with respect to Jira and Bamboo integration. I am sure the build server has no issues either way with simply building the code base, but I have no idea if it offers any actual integration the other way around.

4 Likes

I think this might be a better bet than moving to Github. Atlassian’s whole ecosystem is what we use in our company and we couldn’t be happier with it.

Since Ignite’s already using JIRA/Fisheye/Bamboo is there any advantage in moving to Github vs Atlassian Stash?

Project exposure is the big one that Stash does not bring to the table. Single Sign On (github uses oauth). Significantly larger community (for support and stuff). No license to worry about (even if we got an open source license, it’s still one more thing to keep track of). Otherwise they are pretty similar.

@rcollier - I do believe there is Github plugins for almost all Altassian products. So our existing Jira/Bamboo should integrate with Github no problems. Either route, Github or Stash, we will ditch Fisheye because it’s redundant of both Github or Stash.

This alone should shed some light on things: https://github.com/atlassian

Altassian themselves use Github for things. Stash is still a reletively new product for Altassian (I’ve been watching it for a few months, considering a purchase for our company for proprietary stuff). They are iterating rapidly on it, adding all kinds of features and stuff to compete with Github/Github Enterprise.

From what I can tell, Stash is, or is almost, identical to BitBucket.com (Altassians hosted option). Bitbucket.com is free for Open and Closed/Private repos and offers all the same Altassian integration (because it’s their own product). It does not have the same community as Github and is also younger to the market, but is nice itself.

Stash is pretty nice looking.

As long as it’s Git, I’m on board, although I’d prefer Github personally.

1 Like

I don’t get the project exposure comment. I would bet money that people find the projects they are interested by Google search (90%) followed up by Stackoverflow (9%). All percentages are compeletely my guess .

As for sign on, for write access you would ideally use the same sign on that anyone would already require for the forums. Anyone taking part in the project will likely have or need an account on Igniterealtime anyway. I would think that the number of people that are going to contribute without ever having asked a question first before writing code is pretty small. In any case though, I don’t think this is much of an issue in either direction.

According to the Jira documentation (I just looked), the same or at least similar interoperability is supported as with Stash. So the only thing I am wondering is if a the build server can block a

merge like you can set up with Stash?

So there may be next to no differences functionality wise, other than my last two points on private repos and branding. I will admit the the usage of private repos would be pretty small and can be satisfied by Bitbucket anyway (for up to 5 users). The branding is certainly a nice to have as it creates a more cohesive environment, but not a necessity.

More input from other contributors would be nice, and by this I mean for all projects. Smack will go one way or the other anyway, but it would be nice if all other projects did the same. Otherwise it will kind of fragment the community here. I think I will try to ping a couple of people to speak up.

I suspect everyone will simply have their preference based on what they are familiar with. It would be nice to hear from someone who has some experience with both.

PS:

Buy it for your enterprise ventures, especially if you use Jira already. As you mentioned, it is being updated constantly and is a fraction of the cost of Github Enterprise. We have been using it for awhile now and it is pretty sweet. Integrates well with Hudson/Jenkins as well (which we use at work instead of Bamboo).

1 Like

My thinking has not changed since the last time this was discussed

I share Walter’s view about time being very precious to all of us and being better spent on fixing things rather than administrative tasks, but if we could spend less time with more hands and better tools, then moving from SVN to Git is a good idea.

I also share Guus’s view not to move to github.com and keep everything here in one place at igniterealtime.org. I see the benefits of Git, but can’t see the greener grass at github.com

Could we spend effort on a having a Git repository here rather than moving to github.com and keeping both in sync.

As this proposal to use Stash answers my question, I am in full support of the proposal. Unfortunately, I have never used Stash and don’t have much to add to the choice of which Git

1 Like

I am fine with moving to Stash. My dev ressources were focusing on Spectrum and Openfire recently. Spark is de facto on hold. We integrated the new Smack internally for testing purposes. How much effort is needed to move to Stash?

I’m ok with anything which will work with Jira (and Bamboo or other CI). Does Atlassian provide open source license for Stash or will we ask Jive to buy a license for us?

I think it is free for open source projects like us

It is on their list of software on the OSS requisition form, so yes, we can get the license for free.

The moving of code will not be too painful. The bigger administrative hurdle is simply adapting the release methodology, but it is an expected one time hit that the gains in the development process more than make up for.

I am using GIT as a file management tool for everything. I like it very much, especially the branch concept

Also for SPARK development I am using a tool called git-svn (http://git.or.cz/course/svn.html) to allow me use GIT features also for SVN repositories like SPARK repo

Inside the company, for our projects we are using GitHub. I am fine with GitHub. I am sure I will like Atlassian Stash too because it uses GIT, if you feel suits better for SPARK.

As long as we are talking GIT, I am onboard.

Few plugins of possible interest if we move to Stash:

https://marketplace.atlassian.com/plugins/org.tmatesoft.subgit.stash-svn-importe r

https://marketplace.atlassian.com/plugins/com.atlassian.bamboo.plugins.bamboo-st ash-plugin (bundled with Bamboo >= 4.4)

I think if we get the open source license, we can also get licensing for several/most (all?) altassian plugins in the market place.

I’m stilll not a big fan of modifying anything, as I don’t see real added benefit. I have seen a lot of testimonies of GIT being preferred over SVN, listings of features and such, but I’m struggling to see how this would actually get us more development resources. I don’t think it’s the tools that interest people - it is the project itself.

Nonetheless, I won’t oppose a change, if only if my involvement has been very limited these past few months. If changes are desirable, I would be a strong proponent of keeping the code on servers that are in our (well, Jive’s) control. In that sense, I’m preferring Stash over Github.

I hear you Guus, and I agree that it will not get us more development resources. I believe (the same as you) that if someone is interested in contributing, they will do so because of the project itself, even if the source control is CVS. In this regard, I don’t see any benefit to moving to Git.

The value in this tool (and Github for that matter) is the much improved process for software development. One of the problems with the current process is that there really isn’t any. Anyone with commit priviliges can simply commit code, sight unseen, any time they wish. This is not conducive to writing/maintaining a robust code base, as it typically means finding and fixing things after they have been pushed into the codebase and possibly even a release. Forcing at least one other developer to review any contribution before it gets merged in will reduce errors on this front. It’s always harder to change things after they have been put in.

It just so happens that tools like Stash support a much better process, due in no small part to the natural strengths of branching/merging that are inherent in many DVCS solutions. In fact, Stash can be used with Git or Mercurial (it was originally a Mercurial only tool). Git seems to be the better choice in this case simply due to its popularity, even though many consider Mercurial to be easier to use.

In any case, I will go ahead and order a license for Stash on behalf of Igniterealtime.

Is it easier to review a patch in vcs or as an attachment to a ticket in JIRA? Well, this all makes sense if there is more than one person doing commits

It’s easier in Stash (or Github), but instead of a patch you would preferably review a pull request. This can be done online and review comments done inline with the code. Patches can still be attached to tickets like they are now, but they typically require you to download the code, apply the patch and then check the changes via a diff tool. The pull request doesn’t require these extra steps as you don’t need to actually apply the patch to check for differences. Plus the mechanism for commenting and requesting changes is built in to that request. The poster of the pull request can then make changes and resubmit. It is more cohesive then either PMing the patch writer or simply pointing out the issues in the forums.

Under this model, ideally every commit involves at least 2 people. 1+ people to write the code for submission and another to review and merge it. Nothing gets merged without someone reviewing it. The people with review and merge privileges are the guardians of your code quality, so while anyone can write code and submit pull requests, only a few would actually have the privilige of pushing it into the actual product.

I just discovered this blog post, would have expected something like this to be a discussion somewhere in the forums. But anyway I don’t care if it’s github or a self-hosted Stash instance. Both have minor advantages and disadvantages.

And yes github provides a higher project exposure. Which may attract more developers. If this really happens will the future show. But remeber that the community already created a Smack git repo back in 2009, http://repo.or.cz/w/Smack.git, where patches accumulated that didn’t get processed by the offical Smack developers. And I also credit the fact that aSmack has already 3 times as many active developers as Smack to the fact that aSmack uses git exclusively.

And why do you think that the old open source project sites like freashmeat (now freecode) or sourceforge are now almost meaningless or dead: It’s because of git. The used VCS makes a difference in FOSS development.

If we choose Stash, then we could ask the github folks to create a sync between the repos on your github organization site and your Stash repos.

A bit late to the party, but still a big fan!

I am supportive of making the switch to Git (via Stash or Github), assuming we can keep the exitsting Ignite community intact. Specifically, I have a few logistical questions:

  • How will existing SVN access credentials/privileges/roles be mapped/ported to the new infrastructure?
  • Are the current project administrators comfortable with the new responsibilities implicit in the switch to a Git-based development model?
  • Will existing/former committers (like myself) effectively lose the ability to commit changes to the corresponding repos?
  • Do all contributions going forward become fork/pull requests, rather than allowing commit (push) access to the project repos?

I suppose my main concern is that I have been frustrated in the past on various projects where forks and pull requests have sat idle, presumably due to Git-wise administration overhead.

  1. I’m of the opinion that all credentials should be reset (we likely have a lot of legacy/unused credentials). Key contributors can request new access and be granted based on Team Lead judgment.

  2. I don’t think there is anything really “new” to the git development process that would incur additional overhead. In fact, I think it will streamline things long term. There will absolutely be some administration overhead up front as the Team Leads and community adjust and learn the new process, but that’s expected and should be controlled and short term.

  3. See #1 above.

  4. I think to best take advantage of the destributed nature of git, all contributions should be done via fork/pull-request workflow. This truely enables anyone, anywhere to work on the project as often as they want for as long as they want before pushing anything back into the public (via a reviewed and accepted pull request). Again, Key contributors can be given direct push access, however I really think to keep things well organized and jiving well (pun intended) with git, things need to be fork/pull-request workflow. It forces code review in some sense to guard against bad commits, but also helps ensure each contribution does not get lost.

To be frank, you main concern occurs already. I myself have several patches for Spark that are months old posted in jira and/or on the forums and have yet to be reviewed and committed. It boils down to Team Leads either losing interest in the project, running out of spare time, etc. With a fork/pull-request workflow, it’s actually harder to miss pending/submitted patches since when you log into your account, they are presented right in front of you as a notification. So in your past case, I’d wager this was the product of a lazy project admin who chose to ignore your pull request on purpose for any reason – actually, I’ve done the same myself on several of my projects on github i’ve personally lost interest in, but people issue pull requests for. Maybe in my case on here, my submitted patches are terrible (likely the case ), but I woudln’t ever know unless someone provided direct feedback, or better yet, my pull request was reviewed, commented on, closed, etc.

Closing thoughts – Why are we so dead set on self-hosting the repos? We are a small community of volunteers, we do not have some big IT staff or people willing to spend time setting up a server, maintaining it, backing it up, fixing problems when they occur. Why not let someone else deal with all that? We also have zero need for crazy custom rules and setups, having these custom setups is an indication we are doing something wrong! We are not exceptional in any regard and should be able to live very happily within normal dev tool bounds.

With this logic, I suggest we rule out Stash, although I find it quite nice. Instead, we should only be considering Github or Bitbucket.