As I wrote in my previous post about automation I would continue
my post. So here it is.
The following parts are left from my process:
The process
- Create an issue for myself (if already existing move to step 2)
- Create a branch within the appropriate project based on the issue
- Assign the issue to myself
- Migrate the issue into the state of
IN PROGRESS
- Done.
- Done.
- Done.
- Done.
- Close the appropriate jira issue with a reference to the commit in comment.
If I work on updates of dependencies or parents in our builds, it is required to make a JIRA ticket for each of those updates (ok not really required but it is useful). It makes sense to test each of those updates separately in Jenkins. In the end it is an important information for the end-users, so they can see what has been changed.
Automation Step 4
The first step is to create a ticket with some information like a summary, a description and giving Affected versions
and Fix Version/s
useful values. This is only needed if we do not have a ticket yet for the work.
So far I was going into my browser opened the appropriate project and pressed the Create
button and typed in the appopriate values. I already
mastered the copy and paste parts from browser url to command line etc. After some time, I thought about this and came to the conclusioni, there must be a more convenient way.
I tried to use the REST API of JIRA with some JavaScript which I used earlier to create some release notes but it was too complex to handle that via JavaScript. After some research, I found a command line client which looks promising.
I started to play around with the command line client. There are some things to mention about it:
- Single binary artifact; simply download it and put on the path.
Name it as you like (I decided to name it
jira-cli
) - Templating mechanism to create custom commands
- Simple and powerful default commands already existing
- Easy configuration via yaml file.
The first thing is, you need to login into the JIRA instance which can be handled by the following:
$ jira-cli login --endpoint=https://issues.apache.org/jira
You will be prompted for your password. The user which is used to login is derived by the user you are working with. For me it was wrong. So I needed to add a parameter:
$ jira-cli login --endpoint=https://issues.apache.org/jira --user=khmarbaise
So now I can view a ticket on command line very easy:
$ jira-cli view MEJB-118
issue: MEJB-118
created: 339 days ago
status: Closed
summary: Remove "J2EE" from plugin description and site
project: MEJB
issuetype: Task
assignee: khmarbaise
reporter: afloom
fixVersions: 3.0.1
priority: Trivial
votes: 0
description: |
The term "J2EE" was repalced by "Java EE" by Sun/oracle logn time ago and we shouldn't use "J2EE". It's currently used at least in the plugin's description (pom.xml) and on the intro page of the plugin's site.
Replace "J2EE Enterprise Javabean (EJB)" with "Java Enterprise JavaBean (EJB)".
comments:
- | # khmarbaise, 338 days ago
Done in [r1805332|https://svn.apache.org/r1805332]
- | # hudson, 338 days ago
SUCCESS: Integrated in Jenkins build maven-plugins #9069 (See [https://builds.apache.org/job/maven-plugins/9069/])
[MEJB-118] Remove "J2EE" from plugin description and site (khmarbaise: [http://svn.apache.org/viewvc/?view=rev&rev=1805332])
* (edit) maven-ejb-plugin/pom.xml
* (edit) maven-ejb-plugin/src/site/apt/index.apt.vm
Just to take a short look on a particular issue it is much faster than going via the browser.
After more detailed reading the documentation I found that I can put the information about the end point and the user which should be used for jira into a configuration file (within your home directory) .jira.d/config.yml
which looks like this:
endpoint: https://issues.apache.org/jira/
user: khmarbaise
login: khmarbaise
With this the login command can be reduced to jira-cli login
. Easy? The jira-cli
will safe session information in .jira.d/
directory which is stored in cookies.js
file. For my convenience I have created a Git repository in .jira.d
and added cookies.js
to the .gitignores
file to prevent me to commit the cookies.js
file accidently.
Now I have the ability to create JIRA tickets via command line.
So far so good. But how do I know for which project I need to create a ticket?
But wait a second. The pom.xml
contains the information project.issueManagement.url
which is the URL for the JIRA tracker. So I needed to find a way to extract it from the pom.xml
.
Luckily the maven-help-plugin is here the way to go which looks like this (Just for brevity I have added \
; usually this is a single line):
ISSUE_URL=$(mvn org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate \
-Dexpression=project.issueManagement.url \
-q \
-DforceStdout)
This will extract the URL for JIRA out of the pom file. So you get something like this:
https://issues.apache.org/jira/browse/MDEPLOY
but unfortunately this is not the project you need for jira-cli
. What I need is the MDEPLOY
. This can easily achieved by:
PROJECT=$(basename $ISSUE_URL)
These are the information which are needed to create a JIRA ticket. Hm...really? No. What about the version for Fixed versions/s
and Affected versions
? This can be done by:
PLAIN_VERSION=$(mvn build-helper:parse-version \
-Dx=\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion} \
org.apache.maven.plugins:maven-help-plugin:3.1.0:evaluate \
-Dexpression=x \
-q \
-DforceStdout)
This will extract the version out of the pom file without the trailing -SNAPSHOT
. It assumes that the version number has three digits.
So now comes a favorite feature of jira-cli
. You can create your own commands very easy. You simply define things like this in your config.yml
file:
custom-commands:
- name: mine
help: display issues assigned to me
script: |-
{{jira}} list --template table --query "resolution = unresolved and assignee=currentuser() ORDER BY priority asc, created"
So now you can simply call jira-cli
like this:
$ jira-cli mine
and you will see the list of issues which are assigned to yourself. See more details in the documentation of jira-cli
. Based on the above, I have created a custom command which looks like this:
custom-commands:
- name: createdependencyupgrade
help: creates an Dependency Upgrade ticket.
options:
- name: affected
help: The affected version
- name: fix
help: The fix versions.
- name: project
default: $JIRA_PROJECT
- name: description
default: ""
- name: priority
default: "Minor"
args:
- name: summary
required: true
script: |-
{{jira}} create --project={{options.project}} --issuetype="Dependency upgrade" -o summary="{{args.summary}}" -o description="{{options.description}}" -o fixVersions="{{options.fix}}" -o versions="{{options.affected}}" -o priority="{{options.priority}}" --noedit
So as the result of this, I can create a JIRA ticket via this:
$ jira-cli createdependencyupgrade --project MWAR --fix="3.0.2" --affected="3.0.2" "Summary Text"
These have been the prerequisite to finally create a JIRA ticket in a very convenient way.
The result of previous things and integrating some other parts is now my
createdependencyupgradeissue.sh script which I can simply call like this:
$ createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
The output of that script looks at the moment like this:
$ createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
Checking if we are on a branch...done.
Check that we are on master...done.
Extracting issue url from pom.xml file...done.
Extracting version from pom.xml file...done.
Creating JIRA issue...done.
OK MWAR-419 https://issues.apache.org/jira/browse/MWAR-419
M pom.xml
Switched to a new branch 'MWAR-419'
As you might already have realized, I have added things to create a branch in Git based on the JIRA issue name and switched to that branch.
The fourth step of automation accomplished which means more improvement but not enough.
Automation Step 5
If I like to assign a ticket to myself so far I had to go to the ticket page and click on assign to me
link which prerequisites to know the issue number and type in the correct URL etc.
So based on my assumptions that the branch name is the JIRA issue number the only thing I need to do, is to extract the number from the branch. This can simply be achieved by:
BRANCH=$(git symbolic-ref --short HEAD)
So next the thing is to assign the ticket to myself. This
can be done with jira-cli
easily like this:
jira-cli take $BRANCH
I have to add some sanity checks in case of errors and the result is the takeissue.sh script which I can use within the branch like this:
$ takeissue.sh
The fifth step of automation is accomplished which means becoming comfortable but I think, we could do more.
Automation Step 6
This step is more or less identical to the previous one cause I would like to change the state of an issue into IN PROGRESS
. This can easily be achieved by using jira-cli
like this:
$ jira-cli start MDEP-234
Based on the ideas before, this can be baked into a script startissue.sh which results in such a simple call
$ startissue.sh
The sixth step of automation accomplished which means more improvement is possible.
Automation Step 7
Ok I can create an issue in JIRA including the creation of the branch in Git so what about to commit here? Usually I update files or add some new ones which is done by git add .
and afterwards I can commit the state.
So based on the issue the commit message should look like this:
[XXX-123] - Summary
Optional description
In the majority of the cases I need only the first line. In other words this is copy and paste from the issue I created before. Let us make that easier. I created, of course you know already a script commitonissue.sh which exactly does that.
It identifies the branch based on the name of the branch in Git and gets the information about the summary via jira-cli
from JIRA which results in such line:
SUMMARY=$(jira-cli view $BRANCH | grep "^summary: " | cut -d " " -f2-)
and finally I need to do the real commit which can be handled by this:
git commit -a -m"[$BRANCH] - $SUMMARY"
The seventh step of automation accomplished which means we are on the home straight. But not yet crossed the line.
Automation Step 8
This step is nothing really new, it is just improving something already existing. The gitmergeclean script of the first post. This is needed to be tweaked. It should also close the JIRA ticket after the branch has been merged to master with an appropriate comment on it.
I have added the following lines to the script to handle this.
#
# Get the latest commit HASH
#
COMMITHASH=$(git rev-parse HEAD)
#
# Get the GIT URL from pom file:
# TODO: Can we do some sanity checks? Yes: scm:git:.. if not FAIL!
echo -n "Get the git url from pom file..."
GITURL=$(mvn help:evaluate -Dexpression=project.scm.connection -q -DforceStdout | cut -d":" -f3-)
echo " URL: $GITURL"
GITPROJECT=$(basename $GITURL)
GITBASE=$(dirname $GITURL)
#
echo "Closing JIRA issue $BRANCH"
jira-cli close -m"Done in [$COMMITHASH|$GITBASE?p=$GITPROJECT;a=commitdiff;h=$COMMITHASH]" --resolution=Done $BRANCH
## Error handling?
echo "Closing finished."
The BRANCH
information is already extracted at the beginning of the script. The final lines of the script are to call jira-cli
to close the issue with an appropriate comment and set the resolution
to a useful value. In our case it is simply Done
. The GITURL
etc. is extacted to create a useful comment which contains a link to the real git repository with showing the diff of the commit which has been made. You can take a look at the MCLEAN-87 issue which shows a link to the git repository.
A full output of the new gitmergeandclean script looks like this:
~/ws-git-maven/plugins/maven-rar-plugin (MRAR-79)$ gitmergeandclean.sh
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
Updating 1ffdfba..055bbda
Fast-forward
pom.xml | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
Total 0 (delta 0), reused 0 (delta 0)
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-rar-plugin.git
remote: 1ffdfba..055bbda 055bbda0907661aa6515cc8e79ba66cee43aec12 -> master
remote: Syncing refs/heads/master...
To https://gitbox.apache.org/repos/asf/maven-rar-plugin.git
1ffdfba..055bbda master -> master
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-rar-plugin.git
remote: - [deleted] MRAR-79
remote: Syncing refs/heads/MRAR-79 (FORCED)...
To https://gitbox.apache.org/repos/asf/maven-rar-plugin.git
- [deleted] MRAR-79
Deleted branch MRAR-79 (was 055bbda).
Get the git url from pom file... URL: https://gitbox.apache.org/repos/asf/maven-rar-plugin.git
Closing JIRA issue MRAR-79
OK MRAR-79 https://issues.apache.org/jira/browse/MRAR-79
Closing finished.
Conclusion
So finally for a dependency upgrade the process looks like this:
- Change the appropriate information in
pom.xml
file createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
commitonissue.sh
gitpushwithlease.sh
takeissue.sh
-
startissue.sh
After that I have to wait for the result of Jenkins. If the build is successful I can merge back to master otherwise I need find the cause of the problem. In this case the result was successful so I can do it like this: gitmergeandclean.sh
The output looks in total like this:
~/ws-git-maven/plugins/maven-linkcheck-plugin (master *)$ createdependencyupgradeissue.sh "Upgrade maven-plugins parent to version 32"
Checking if we are on a branch...done.
Check that we are on master...done.
Extracting issue url from pom.xml file...done.
Extracting version from pom.xml file...done.
Creating JIRA issue...done.
OK MLINKCHECK-32 https://issues.apache.org/jira/browse/MLINKCHECK-32
M pom.xml
Switched to a new branch 'MLINKCHECK-32'
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32 *)$ commitonissue.sh
[MLINKCHECK-32 cd34b97] [MLINKCHECK-32] - Upgrade maven-plugins parent to version 32
1 file changed, 1 insertion(+), 1 deletion(-)
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ gitpushwithlease.sh
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.53 KiB | 1.53 MiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-linkcheck-plugin.git
remote: * [new branch] cd34b97ddb661d5881bf7fc01e3291a9b0f67041 -> MLINKCHECK-32
remote: Syncing refs/heads/MLINKCHECK-32...
To https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
* [new branch] MLINKCHECK-32 -> MLINKCHECK-32
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ takeissue.sh
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ startissue.sh
Here is the time where I have to wait for the results of Jenkins. So in this case the result was successful so I can continue like this:
~/ws-git-maven/plugins/maven-linkcheck-plugin (MLINKCHECK-32)$ gitmergeandclean.sh
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
Updating ff5dfab..cd34b97
Fast-forward
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Total 0 (delta 0), reused 0 (delta 0)
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-linkcheck-plugin.git
remote: ff5dfab..cd34b97 cd34b97ddb661d5881bf7fc01e3291a9b0f67041 -> master
remote: Syncing refs/heads/master...
To https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
ff5dfab..cd34b97 master -> master
remote: Sending notification emails to: ['"commits@maven.apache.org" <commits@maven.apache.org>']
remote: To git@github:apache/maven-linkcheck-plugin.git
remote: - [deleted] MLINKCHECK-32
remote: Syncing refs/heads/MLINKCHECK-32 (FORCED)...
To https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
- [deleted] MLINKCHECK-32
Deleted branch MLINKCHECK-32 (was cd34b97).
Get the git url from pom file... URL: https://gitbox.apache.org/repos/asf/maven-linkcheck-plugin.git
Closing JIRA issue MLINKCHECK-32
OK MLINKCHECK-32 https://issues.apache.org/jira/browse/MLINKCHECK-32
Closing finished.
So if you look in my previous post you might be astonished that the number process
steps has not been reduced dramatically? Those scripts are replacing a large number of manual steps (git command line typing, jira in browser, clicking typing etc.) which is now simply handled by scripts which means the steps are done always in the same way and I am sure I do not forget steps. Furthermore many of scripts can be used in other situations as well which makes my life easier for example in handling pull request etc.
So this was a much longer post. I hope you enjoyed it.
Top comments (2)
Have you thought about having Jenkins close the issue once a commit message including its id lands on the master branch? Especially in concert with a pull request workflow, that would let the build server handle just about everything
gitmergeandclean.sh
does (Jenkins would need permissions to automatically delete the remote branch and wouldn't be able to clean your local repository). But you might have some specific process requirements that necessitate kicking off the merge manually.Hi Dian,
first thanks for your comment.
And yes this could be done but unfortunately this can't be done on the Apache Software Foundation side, cause there we do not use pull request (or something which something like PR's). A PR must be "authorized" by a committer which means we have to create a branch for it and commit it later which means a name of a committer is given here (difference between committer/author)..cause the main source of truth is not GitHub but it is gitbox.apache.org/repos instead which does not directly support PR's in the way GitHub does. That's the reason why we need to go that way.
I'm currently experimenting with scripts to handle this also making this easier for me to support the contributions from other people in more simpler and faster way...