Automated tests in JIRA with Xray for JIRA and Bamboo


Posted by
Corentin BRESSON

April 12, 2016

This is a guest blog by Xpand Add-ons. A leading Atlassian Expert, Xpand Add-ons helps teams make the most of their Atlasssian tools with its flagship products Xray for JIRA and Xporter. In this blog, the team explains how automated tests in JIRA using Xray for JIRA can help continuously integrate the software development process, enabling teams to get high quality products to market quicker. 

We all know that the process of testing can, for a software tester, be an everyday challenge. In fact, it’s one of the most important processes of the software development lifecycle. Testing must assure that the product can be implemented with the same characteristics, that all the business and technical requirements in terms of design and development are met, and above all, that it works as expected.

Considering this, continuously integrating your testing process is super important in enabling a quicker time-to-market. This approach means that most integration bugs are identified and fixed on a regular basis, with the continuous cycle of Code->Build->Test->Deploy increasing the probability of achieving high quality, bug-free releases.

What’s more, this is all completely possible by automating your tests and test sets. Let’s see how Xray can assist with managing automated tests in JIRA.

With Xray, automated tests in JIRA are executed through an external tool controlling both the execution of tests and the comparison of actual outcomes to predicted outcomes. It can automate some repetitive tasks in a formalised testing process already in place, or add additional testing that it would be difficult to perform manually.

As the testing tool we will use Cucumber, since it allows you to execute plain-text functional descriptions as automated tests. To get more info about how automated tests are created in Xray for JIRA as well as more information about Cucumber, you can take a look at the Cucumber website and Xray for JIRA documentation.

It’s also worth highlighting that this is possible because Xray also makes its rest API available, and with that, we are able to export test definitions and import execution results.

1. Creating Xray for JIRA test issues

Let’s start by creating an Xray for JIRA test issue or a test set issue (in case we need to group tests in one JIRA issue). In this example we will use a test set issue, and add the tests (that in this case were already created), not forgetting that these must be automated.

A Cucumber test is composed of two parts. Firstly, you define your test specification in Gherkin, a very simple language that allows you to use plain text for the test steps. Secondly, you need to implement your test steps definition in code. This code will parse the appropriate sentences and trigger the desired actions.

Xray for JIRA only stores the test specification in plain text. The implementation code must be stored outside of JIRA, more commonly in a SCM (Source Code Management System) or repository.

1

2

2. Creating a Plan in Bamboo

Jumping to Bamboo, we will start by creating a build plan, where we will point out the several tasks we want it to go through.

Our plan will consist of five tasks:

  • Pull the Cucumber project from our SCM
  • Export the test definitions from Xray and JIRA. This will generate .feature files ready to be executed by Cucumber
  • Unzip the exported features into our Cucumber project
  • Execute the exported .feature files with Cucumber
  • Import the execution results outputted by Cucumber back into JIRA

2.1. Pull the Cucumber project from SCM

 

3

 

After we configure the repository where our code to be tested is located, the first task will be a command task type. Here we will demand that Bamboo exports all the .feature files from the automated tests we created in Xray. For that to be possible, we will use a cURL command that will act as a data transfer tool from our Xray for JIRA to our Bamboo.

 

4

2.2. Export features from Xray

Select cURL as the executable, or if you don’t have it in your list yet, add a new executable by configuring the path where it is located. If your agent is a Windows machine, you should install it.

Our example argument will be: 

-u admin:admin http://xraydemo.xpand-it.com/rest/raven/1.0/export/test?keys=CALC-173 -o features/features.zip

Note that CALC-173 is the Test Set ID we just created in Xray for JIRA.

10

2.3.Unzip the .feature files

If multiple feature files are generated by Xray, a zip file will be exported from JIRA. If this is the case, you must first unzip the .features files.

This one will be also a command task type – select unzip as the executable, or if you don’t have it, add a new executable by following the same procedure as above.

The argument should be something like:

 features/features.zip -d features

11

2.4. Execute tests in Cucumber

After all the feature files are unziped, it’s time to execute them in Cucumber. In the next task (also a command type one), and after Cucumber is installed in your agent machine, add it as new executable.

In terms of the argument, it should be something like:

-x -f json -o report.json

In this step, Cucumber will test the feature files and create a json file report.

2.5. Import Execution results

This json file report will be used in our last command type task. Here we will import the report back to Xray, before creating a test execution issue with the results of each test from the test set we first created.

This will be done by using our cURL friend once again as the executable, and we will use this argument as an example:

-H “Content-Type: application/json” -X POST -u admin:admin –data @report.json http://xraydemo.xpand-it.com/rest/raven/1.0/import/execution/cucumber

X

3. Run the created plan in Bamboo

After all the tasks are correctly configured, you can enable and create the plan. You are now ready to run the plan.

13

 

If everything goes as it is supposed to, it will be created as a test execution Issue in Xray, similar to what you can see in the following image.

 

14

Good luck in continuously integrating your tests!