Corey Bradley | Lead Performance Tester

BlueCross BlueShield of TN Selects NeoLoad to Streamline its Continuous Delivery Process & DevOps Initiative

BCBST has launched a DevOps initiative aimed at building, testing, and releasing software more rapidly/frequently. In order to ensure that this culture is supported by a robust Continuous Delivery Process, they recognized the need for a modern performance testing tool that would accelerate testing without any compromise to software quality. After completing an extensive proof of concept between NeoLoad and their current load testing solution, LoadRunner, BCBST selected NeoLoad.

BlueCross BlueShield of Tennessee Context

Part of the BlueCross BlueShield Association of nationwide health care plans, BCBST is Tennessee’s largest health benefit plan company. As a leader in health care financing, BCBST serves over 3.3 million people, provides benefits to nearly 12,000 companies, and pays more than $82.2 million in claims annually. IT applications range from in-house to third-party vendors but can be heavily customized for BCBST. They must be rapid and reliable in order to sustain the highest level of service Blue Cross Blue Shield customers and partners have come to expect. BCBST is establishing a culture of DevOps to release software more quickly. With sprints shortening from six weeks to four (a duration that continues to shrink), the team is able to produce performance testing reports within hours using NeoLoad compared to a three to four-day turnaround with LoadRunner. Performance tests can easily be completed without holding up the release.

The Challenges of Performance Testing in the BCBST Continuous Delivery Process

  • Before NeoLoad, performance testing was a key barrier to the BCBST Continuous Delivery process.
  • Performance tests were complex and executed over long periods of time. As a result, project timelines requiring allotment for proper performance testing would slip, complicating resource coordination and negatively impacting delivery timing.
  • Projects mandated by pre-determined delivery dates would compress, resulting in significant reduction of end-product testing windows. This would often lead to decreased team collaboration and coordination.
  • Under a compressed timeline with an inability to conduct “proper” testing protocol, BCBST was typically executing performance testing as pre-launch “validation,” or even dropping altogether in deference to handling post-launch as a “break-fix” exercise. The legacy tool used for performance testing (LoadRunner) was not appropriate for performance testing in the BCBST Continuous Delivery environment. The tool was complicated and inefficient which impacted the following:
    •  Finding and retaining resources to run performance tests was difficult.
    • Outsourcing to performance testing engineers was an extremely expensive and distracting proposition.

NeoLoad Enables Shift Left to Support Continuous Delivery

The objective of the BCBST Continuous Delivery Process goal is to deliver software, faster. As development cycles continue to shorten, so too are performance testing windows. In this context, it is required that BCBST start its performance testing during the first stages of the software development lifecycle, at the component level, even before a GUI is delivered. The cost associated with defect resolution rises exponentially the longer it takes to identify it.


  • This is why BCBST starts performance testing on components (E.g. SOAP calls) well before acceptance tests are executed.
  • Because it is easy to learn, NeoLoad enabled BCBST to have developers execute their own smoke
    tests – something not possible with their legacy tool.
  • Performance test design and reporting are each available within hours using NeoLoad.
    • According to Corey Bradley, Lead Performance Tester, these are just some of the benefits
      which BCBST has been able to seize since switching to NeoLoad. Here’s what he had to
      say about the switch:
      “… developers have been up and running within three days compared to multiple weeks using LoadRunner… This is a significant achievement since the same test would have taken three or four days with our legacy tool, which is incompatible with the pace of our releases… Now we’re conducting performance testing on a weekly basis…

In order to accelerate and automate an expanded performance testing schedule beyond current practice, BCBST plans to implement the NeoLoad-Selenium integration. This feature will enable BCBST to:

  • Re-use functional tests designed with Selenium for performance testing.
  • Automate the maintenance of performance test scripts once NeoLoad’s process automatically identifies and updates based on changes in the Selenium script.

NeoLoad Integrates into BCBST Continuous Delivery Pipeline

NeoLoad is an open solution that provides several APIs. “It was easy to integrate with our Continuous Delivery pipeline supported by IBM Urban Code,” says Bradley. Performance tests are automatically launched on a weekly basis, which enables BCBST to detect performance issues earlier in the sprint. Continuous Delivery is implemented using a Continuous Integration tool – IBM Urban Code.


In order to streamline performance testing communication across Dev, QA, and Ops teams, BCBST leverages the Neotys Team Server to distribute test results amongst the groups. This allows each team member the opportunity to make corrections with maximum responsiveness to support the internal DevOps culture.


The Results: Before and After NeoLoad

  • We completed the same six-week-long project within the LoadRunner environment in less
    than two weeks using NeoLoad.”
  • “We were able to quickly educate other department members on NeoLoad’s intuitive GUI as
    compared to LoadRunner’s scripts.”/li>
  • “We experienced 60-70% faster script design compared to LoadRunner.” In addition,
  • “Script 12 SOAP calls were completed in minutes, not hours, as experienced with LoadRunner.”
  • “The greatest value we’ve seen thus far using NeoLoad is with its easy-to-learn GUI for
    developers. The more it gets out that we’re using NeoLoad, the more our staff continues to ask
    for it. I am impressed by the quality and support being provided by Neotys.”

With NeoLoad’s easy-to-learn and use interface, BCBST has been able to spread its performance testing
function across five team members vs. one single user with LoadRunner (Corey himself). This not only has provided faster test cycles but increased testing coverage and greater software quality with a wider team managing the workload.

Health Insurance
USA – Tennessee

With NeoLoad, we completed the same six-week-long LoadRunner project in less than two weeks & experienced 60-70% faster script design.

Folge uns