The Current State of Open-Source Testing Tools

Tricentis and three survey project collaborators recently published findings that provide new insights on global trends in open-source testing.

The study focused on the open-source tool testing industry, not how or if companies and software developers test their code. This is an issue that confronts both open-source and commercial or proprietary software.

The results revealed that a lack of technical skills is the major roadblock to open-source tool adoption, with 30 percent of organizations surveyed saying they lacked the skills to adopt open-source tools, according to Kevin Dunne, senior vice president for strategic initiatives at Tricentis.

Cost is the single, largest driver of open-source tool adoption, with 39 percent of those surveyed stating it as the number one benefit. Other factors, such as community support (16 percent) and ease of customization (14 percent) also rank highly.

The published report is gleaned from 2,000 global survey respondents that work in functional testing, load/performance testing, or behavior-driven development.

The results shows open-source testing tools gaining support as viable options for even the largest and most security-focused enterprises. But some respondents still face opposition to open-source security and support within their organizations.

“With the current economic climate impacting tool budgets at many organizations, we believed it was critical to further examine the developments related to open source and free testing tools,” Dunne told LinuxInsider.

Survey Parameters

Open-source testing tools are most often used by those identified as testers or quality assurance workers. However, what was surprising was the low level of engagement from developers and engineers within this survey, noted Dunne.

Respondents averaged almost nine years of experience in the engineering field. The majority (52 percent) are in quality assurance roles. Software development engineers comprised 14 percent of the survey respondents. Development engineers (8 percent), and other roles (26 percent) comprised the rest of the survey participants.

Tricentis collaborated with TestProject, SpecFlow, and Flood to capture a snapshot of where the industry stands today and track how open-source testing trends evolve in the years to come.

Tricentis claims one of the largest open-source testing communities in the world.

The survey was conducted in April 2020 and was distributed online via SurveyMonkey. Sponsor companies promoted the survey via mailing lists, social media, and partnerships with testing media outlets, Dunne said.

Results Across the Board

Tricentis assembled a panel of experts across each testing category or domain to take a deep dive into the results. The panels focused on Behavior-driven development (BDD), functional testing and load/performance testing.

Here are the key findings in each area, according to Dunne:

• Functional Testing: 84 percent of organizations still employ dedicated testers to perform functional test automation, despite an alleged trend to have developers replace dedicated testers to build functional test automation.
• BDD: Only 21 percent of organizations consider themselves experts when it comes to BDD. Fifty-one percent of organizations are developing their test scenarios in a “Given-when-then” format, but there is still room to grow when it comes to automating these tests completely and organizing that code for reuse.
• Load/Performance Testing: Though performance is critical to customer onboarding and retention, only 56 percent of companies are performing load/performance testing on a regular basis. Time, training, technical skills and proper testing environments remain the largest impediments to more frequent load/performance testing.

These are the most significant high-level takeaways the survey reveals, Dunne maintained:

Offshore testing still seems to be popular and thriving, with 61 percent of respondents coming from Asia. Most are located in India and employed with large system integrators.

“With COVID-19 allowing everyone to work remotely, the previous notion that agile development would shift focus away from offshoring seems to not be the case,” he said.

Software Testing Not Universal

Open-source platforms vary wildly when it comes to testing. Many of them are not tested on a regular basis, or they are not tested completely, according to Thomas Hatch, CTO and co-founder at SaltStack.

“The Tricentis survey gives us a good insight into the trends impacting open-source testing and testing tools. I think it is a step in a positive direction, particularly with regard to awareness of the issues,” he told LinuxInsider.

The roadblocks highlighted in the results can be boiled down to an unwillingness to change and a general lack of expertise. The expertise issue highlights a broader problem with software engineering today, he offered. Our tools are more customizable than they are easy to use.

“This speaks to a fundamental issue in open source. It creates significant amounts of innovation, but not very much refinement,” explained Hatch.

The trends regarding industry practices for open-source testing swing like pendulums. He sees a greater shift toward the usage of open-source testing tools.

“But as it becomes clear that we need expertise and the overhead associated with using these open-source tools is more widely known, I think we will see a shift back toward the adoption of enterprise-grade tools,” Hatch countered.

Testing’s Key Role

The survey shows a trend toward automation. More companies reported using automated functional testing than manual testing (50 percent) approaches. One surprising result, according to the report, is the low number (31 percent) of respondents performing unit testing of their code.

Other results in this category include:

• Coded tests (67 percent)
• Exploratory Testing (32 percent)
• Load Testing (23 percent)
• Security Testing (14 percent)

Benefits and Importance Identified

The largest reason for using open-source testing tools instead of commercial or proprietary tools was cost (39 percent) according to respondents. The second-most popular usage reason was a three-way tie for community support (16 percent) and integration to tools/workstations (15 percent), and ease of customization (14 percent).

The least popular reasons for using open-source testing tools were freedom from vendor locking (7 percent) and scalability (6 percent).

Open source importance was another keen trend with 92 percent of responding organizations agreeing that having an open-source functional testing solution was important (34 percent) or very important (58 percent) to them. The remaining 8 percent of respondents tied at 4 percent each for open source being important only in regard to certain conditions or not important at all.

Results were similar when respondents rated the degree of importance for an open-source solution for performance and/or load testing. Again, 92 percent found it either very important (54 percent) or important (38 percent). Only 3 percent found it important in regard to certain conditions, and 5 percent said it was not important.

Responsibility Factor

Only 35 percent of the companies surveyed specifically task performance engineers with doing performance testing. Instead, quality assurance and others had responsibility for functional and performance testing.

A higher proportion of developers had responsibility for performance testing than for functional testing. Researchers found that very surprising. Here is that breakdown:

• Quality/Assurance Testing – 47 percent
• Site Reliability Engineering – 3 percent
• Performance Engineering – 22 percent
• Development – 12 percent
• Operations – 8 percent
• Nobody – 7 percent
• Other -1 percent

Roadblocks to Testing

Performance and /or Load testing shared scarce resources. The majority of impediments for was technical training (39 percent) and time (38 percent).

The roadblocks steepened through Test Environments (34 percent), Test Data (23 percent), Network or Security (20 percent), Tool Complexity (18 percent), and Tool Costs (17 percent).

Time proved to be a critical roadblock with agile development according to 50 percent of the respondents. Technical training and skills tallied at 40 percent.

Other impediments to testing in agile development include: Test environments (38 percent), Test Data (31 percent), Tool Costs (27 percent), Tool Complexity (25 percent), and Network or Security (16 percent).

Behavior Driven Development

Respondents indicated a heavy use of the Given When Then format (51 percent) to increase automation rates and reuse. The survey also revealed that the use of Given When Then to align teams on requirements and to streamline documentation is still lagging.

More Category Results

Testing location preferences matter. A majority of companies (65 percent) are still doing some form of testing outside their build pipelines. Report researchers expect to see the remaining 35 percent get smaller as the adoption of CI/CD will demand that testing be closely integrated within the pipeline to automate deployment decisions.

The full report can be found here.