Automated Software Release
Part of an efficient software development pipeline is automatic deployment and release. This is best implemented relatively early on, because it makes later development easier and more stable.
Current development pipeline has the following steps:
- Create task (task now in Inbox)
- Accept task (task now in Backlog)
- Schedule task (task now in Next)
- Pick up task (task now In Progress)
- Create new branch from Develop branch
- Implement task
- Increment version number in
- Create pull request from new branch back to Develop branch
- Task is done when pull request is merged
- Manually build develop branch, and use Twine to upload to PyPi (assuming Python task)
The steps in bold are now done manually, and should be automated.
- Releases must have a meaningful version number
- Version number must adhere to the most common convention
- The deployment and release process must be using the same pipeline as the existing review process.
- Releases must only be done from stable branches
- Only one release may be done per version
- Credentials for uploading to deployment / release must be kept confidential (can't be in source code)
- Deployment must be done by currently used tools as much as possible (GitHub + Travis)
Git Flow is currently one of the more popular implementations of a distributed workflow.
It proscribes two long-lived branches, with other, shorter, branches forking from one, and merging back into one, or both.
develop branch is the core branch for development. A new
feature branch is branched from here for each task. When the task is done, the branch is merged back into develop.
In theory, this branch interaction model is sufficient for multiple programmers to cooperatively develop software. In practice, develop is not sufficiently stable for release to end users.
master is the branch intended for stable software releases. This branch should only receive code that is thoroughly tested, and does not contain partially finished features.
In order to verify that code on
develop is ready for primetime, new
release branches are created regularly. These branch from
develop either after a set interval (end of Scrum sprint), or when a feature is complete.
At this point developers can continue to add new features to
develop, while the
release branch is used to verify that its frozen set of features are bug-free, and meet requirements.
Bug fixes made on
release branches should also be merged into develop. No new functionality should be added to
When the software on
release passes all checks, it is merged into
Occasionally, despite best efforts,
master will have bugs. For the most critical of them,
hotfix branches are branched directly from
master to fix the issue. When ready, these branches are merged both into
Brewblox as a project has no special requirements that mandate any changes to this model. At time of writing, we do still need to implement meaningful and automatic regression tests that should be run against
We do need to formulate a standard approach for conditions not described explicitly by Git Flow:
- Contributors working on forked repositories
- Pull Requests
Git Flow: Forked
The default Git Flow model does not describe forked repositories making upstream pull requests. The forked model is attractive to projects with open-source contributors, as it allows contributions without granting write access to strangers.
Contributors fork the base repository on GitHub into their own. At that point they can create their own branches, or use their local version of
develop - the upstream repository doesn't care. When they are done, they make a pull request to the correct target branch on the base repository. Pull requests can be commented on by many, but only merged by somebody with write access to the base repository.
We'll refer to the base repository as
upstream, and to the contributor repository as
In a forked Git Flow repository, there will be no
hotfix branches in
upstream. This is all done in
Contributors branch new
downstream/feature branches from
upstream/develop, and when ready, make a pull request back into
release branches are created in
upstream. If any fixes need to be made, these are directly done in
downstream/release (or a forked branch), and merged back in through pull request.
The same goes for
downstream/hotfix is branched from
upstream/master, and merged back into
Normally, only 3 branches will exist in
develop, and (when active)
- Long running feature branches that involve multiple developers. They can use a feature branch on
upstreamto synchronize work. They could do pull requests to the
- Multiple release branches can exist in parallel, for example the latest stable release and a new release candidate for the next major version.
In general: small features do not require a feature branches on the upstream repo.
In order to avoid "dependency hell" in a project using many third-party libraries, Semantic Versioning was introduced.
Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make incompatible API changes,
- MINOR version when you add functionality in a backwards-compatible manner, and
- PATCH version when you make backwards-compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.
This can be used without any adjustments or special cases by the Brewblox project.
Git itself supports tags. These can provide extra information about the repository. Specifically, they can be used to mark where releases were done, so that the HEAD can easily be set at that point.
GitHub offers additional functionality for tags with GitHub Releases.
We don't particularly need the package distribution offered by GitHub Releases, as we release our Python code to PyPi.
More useful is the easy display/editing of tag descriptions. These can be used as release change logs.
Deployment credentials (PyPi requires a username and password) can be kept private by adding environment variables to the brewblox (
upstream) runner configuration.
downstream builds can't access the value of these variables. Any attempt to check in code exposing credentials would need to be approved in a pull request.
Every release to PyPi should be using a new (semantic) version number, and each version number should be released to PyPi.
Version numbers should be set in repository tags, in source code, and in
setup.py (determining python package version). Version numbers in all these places should match, and 'bumping' (adding +1 to MAJOR/MINOR/PATCH) should be a single action. This makes the process less error-prone.
Bumping can be automated by using bump2version, which will take care of both the tag, and the version in source code.
When to bump
bugfix branches are active simultaneously, bumping the version inside those branches would create race conditions in version numbering.
Therefore, version numbers should only be increased on the
master branch. Multiple
bugfix branch results can then also be grouped in a single release, or combined with commits from a
When the version is bumped, Travis should automatically deploy it. This can be accomplished by setting the
on: tags condition in the
Some features will possibly be relevant in the future, but are not specified or implemented at this time.
- Releasing Dev/Alpha/Beta/Gamma/rc versions to PyPi.
- Automatically setting dev version in Python package when building locally.
- Automating version bumping + release after
bugfixbranches are merged
- Requires automatic determination of whether the release is MAJOR/MINOR/PATCH