Editor’s note: The study “Observations in Software Testing“, originally written in French, was presented at the JFTL 2017 conference, a French event gathering the country’s testing professionals. The study collected responses from more than 600 testing professionals and compared it to their results in 2013. This is what we found out.
During the last few years, best practices in testing have evolved significantly to include new trends around agile testing. By managing more frequent releases, continuous integration, and even the automation of certain aspects of testing, the industry has increased productivity. As you can imagine, these trends have significant impacts on the daily life of software testers.
In this context of rapid change however, the question of benchmarking still lingers in the background.
What are the best practices for testing teams? How should these teams organise themselves? What tools should they use? What themes should we recognise which have recently changed the game?
Organising software testing activities
Project organisation
Two major trends are evident. Firstly, testing is being clearly identified as a separate profession within software development. This is good news, as testing very much uses a different skill set and this separation of labour makes testing more objective and effective. Secondly, another important trend is the reintegration of testers into project teams. This move has improved the communication between development teams and testers – ultimately improving the product’s quality.
Which situation best represents the organisation for testing activities related to your projects?*
| Responses | 2017 | 2013 |
| Developers execute the testing | 2.94% | 6.72% |
| Testers are integrated into the project | 50.09% | 35.34% |
| Testing is partially managed by a specific team which is independent of the project | 17.80% | 23.79% |
| Testing is completely managed by a specific team which is independent of the project | 29.17% | 34.17% |
Agile or waterfall model?
The Agile method is relatively new in the development ecosystem. It should be highlighted that it is gaining significance in France compared to the waterfall model, as Agile methods are used in nearly as many projects as traditional approaches.
Project context*
| Responses | 2017 | 2013 |
| Projects using an Agile approach (Scrum, Kanban, etc.) | 49.72% | / |
| Projects using a waterfall method | 50.28% | / |
Testing teams: internal or external?
It is interesting to observe that outsourced testing activities consist of nearly half of the responses in the study. Is this due to the desire to increase profits or because in-house skills are lacking? The study doesn’t analyse this point.
In which context do you execute your job?*
| Responses | 2017 | 2013 |
| Internally – in the company of my employment | 53.06% | 56.67% |
| Externally – at the customer of my company | 46.94% | 43.33% |
Software testing and tools
Test management tools – what to use?
The results of the study are reassuring in the sense that companies are moving towards a more mature direction in terms of software testing. In effect, nearly all companies surveyed use a test management tool, and 75% percent of those companies use a requirement management tool. The majority use an editor solution, and this figure increased between 2013 and 2017.
However, the study does not clarify if the tools are sufficiently integrated to allow for actual traceability of requirements and synergetic collaboration between testers, developers and project managers.
Do you have an integrated testing solution?*
| Responses | 2017 | 2013 |
| Yes, with an editing solution | 61.76% | 56.82% |
| Yes, with an open source solution | 19.02% | 17.99% |
| Yes, with a set of editing solutions, open source or internally developed (according to the projects) | 11.25 % | 11.74% |
| No | 7.98% | 13.45% |
Do you have a solution for managing requirements?*
| Responses | 2017 | 2013 |
| Yes, with an editing solution | 48.37% | 42.80% |
| Yes, with an open source solution | 12.45% | 15.15% |
| Yes, with a set of editing solutions, open source or internally developed (according to the projects) | 12.45% | 10.42% |
| No | 26.73% | 31.63% |
Automated testing: a dream to reality
Automation is the hot subject for testing managers as the benefits are numerous: optimisation of test regression coverage, multiplication of data sets for the same test, decrease in the number of test campaigns, execution of tests in different environments, implementation of tests in continuous integration and notable financial gains.
However, it is fascinating that the number of companies implementing automated approaches has not significantly increased during the past 4 years.
What percentage would you estimate for how much of your testing is automated?*
| Responses | 2017 | 2013 |
| Less than 10% | 43.28% | 43.71% |
| Between10 and 25% | 28.36% | 28.82% |
| Between 25 and 50% | 15.76% | 17.21% |
| Between 50 and 75% | 8.40% | 7.74% |
| More than 75% | 4.20% | 2.51% |
Possible reasons for this stagnation are due to the challenges in driving change – resistance from decision makers, difficulties in identifying the right tools and lack of involvement by relevant parties.
How is success in software testing defined in 2017?
Mastering testing activities guarantees product quality and customer satisfaction. The key factors of success – along with areas for improvement – identified by the study’s participants outline the challenges within the world of testing for the years to come.
Factors for success
- Understanding of the product – Mastering the product not only allows testers to design tests adapted to the situation, but also to understand the origins of a bug and be able to interact with different teams on the project. The challenge is therefore to give the testers the means to understand the functional aspects of the product.
- Take the time to design tests early in the development – Involving testers in early phases of the project allows them to anticipate the needs and have a thorough coverage for the tests – which directly impacts the robustness of the final application.
- The relationships between other teams – Humans are at the heart of all successful projects. Knowing how to collaborate with different teams and communicate the right information at the right time is essential.
- Concentrate on tests which have a strong added-value – We could test everything, yet for economic and practical reasons it is important to identify which tests actually have real added-value.
Areas for improvement
- Automation – Automation is still in its naissance. However the objective remains to relieve testers of low-level testing activities and allow for more testing to be conducted at a lower cost.
- Adopting Agile – A change in how the development team works naturally leads to a mirrored change for the testing team. With the challenges arising from continuous integration and the management of frequent releases, teams must invent new ways of testing which are Agile and robust.
- Integrating test requirements from product design – Allowing for the integration of testing from the start of the project gives a voice to the test teams and assimilates their inputs into the initial steps of the projects. This reduces friction down the road and streamlines communication between project teams.
- Conduct testing activities sooner in the project life cycle – Testing earlier allows for potential bugs to see the light sooner, thus allowing them to be corrected more rapidly by the development teams.
- Decrease costs – The economic stakes are real. Testing certainly validates the product’s quality, but to what extent should testing be carried out? Should testing be outsourced or should these teams be internal? These questions have a direct correlation to how to optimise the project’s final budget.
Atlassian tools for test management: A relevant solution?
Historically, test management solutions which were available on the market came with baggage: they were expensive, often cumbersome to implement and – from the user’s point of view – were not very intuitive tools. If you are wondering what the alternatives are, it may be worth considering the number one tool used by Agile teams – JIRA Software.
Test teams already approve it for its bug tracking and requirements management capabilities. Coupled with acclaimed add-ons from the Atlassian Marketplace, Zephyr and Xray, JIRA Software can easily be transformed into a true test management tool. If you want to know which add-on is best for your situation, contact us here to discuss with one of our consultants.
 |
 |
 |
 |
Discover our JIRA test management solution
