It wasn’t long ago that the Dev and test teams would work late hours, focused and rushed to meet a deadline: rapid fixing, reprioritizing and deferring bugs to close out the bug list, move everything to the staging server, do one last run of the regression and pass it over to Ops/IT to move to production. What happened after? No one knew. For most Dev and test team members, Ops is a black box. More often than not, they are oblivious to what happens at Ops, the roles, responsibilities and timelines. One day—long after the drop-dead deadline for Dev and test teams—after delays, questions and changes, the product went live.
Does this scenario sound familiar? Well, DevOps is supposed to be a game changer.
DevOps means many things to many people. But for test teams, continuous testing (CT) is the primary task in DevOps. And, CT is not merely rerunning the same set of “full regression” tests over and over, as product gets delivered and deployed. This is actually a recipe for disaster and not the intent of continuous testing.
What is CT and How Do Traditional Test Teams Fit In?
Recently, test teams were challenged to ramp up their practices by moving to agile and scrum. DevOps will propel the teams to now ramp their practices up even more—more automation, more technical testing and more tools to leverage. If your test team already has mature agile practices, with continuous testing, they can mature further to eventually accelerate the product release.
Continuous testing grows from continuous integration (CI).
Starting with CI, test teams joined the process with automated smoke tests. A strict view on CI is about automating the build process. Yet for anybody doing CI today, that is just the start. The reality of CI using tools such as Jenkins, Bamboo or TeamCity is that after automating the build process, the unit tests are rerun, the automated smoke tests (at the user interface, API level or wherever) are run and many companies build on that to include automatic rerunning the full regression in the test environment, then automatically running some set of tests against some number of VMs. This growing in CI levels of “awesomeness” is becoming a common practice. You may ask, “Isn’t this continuous testing or approaching continuous testing?” CI is the foundation. It grows from there.
Continuous testing is a lean process of quality at every step. It includes quality user stories, quality environments, quality unit tests and quality performance tests. It is “testing at every step” and not “QA” at the end. It is also shifting testing tasks from Ops at the end, by testing earlier and testing all along the way. CT is adding more tasks as well.
Test Strategy
The success of CT is in its ability to get everyone involved and not just the test team.
Running the right tests at the right time on the right environments was never an easy task. In modern development, testing—unit tests, performance tests, security tests, functional tests and workflow tests—is everyone’s responsibility. It’s the foundation of continuous testing.
The test strategy for continuous testing is quite different than agile testing strategies. In agile, testing very often relies on a regression or a release sprint before deployment for bigger and longer tests, followed by Ops teams doing their own performance and security tests on a staging environment. In DevOps and the “shift left” approach, all these tests need to be run earlier and more often as part of the test team’s suites.
So far we have developer unit tests and automated smoke tests run with CI, an automated full regression that gets run often. These tests in addition to manual tests, which are the normal part of new user story development during sprints, continue to be a part of continuous testing.
Continuous testing gets bigger from here. Ops/IT builds environments and data for test teams through virtualization, VMs, containers, in the cloud, etc. Test teams will automate more and run different types of tests, much earlier, against the virtualized environments that are the same as in production. The idea is to take what used to be late development and final tests and shift them left, earlier in test cycles. Thus, eliminating a regression or release sprint and having the product ready for delivery continuously.
Longer workflow and integration tests now must be automated and run against the builds for continuous delivery, as well as load, performance and security tests run at various points during development. This presents many problems for testing and Ops to resolve.
This brings up the question, when do test teams do usability, soap opera, end-to-end path, workflow and task-based tests? These are tests focused on completing a full user task as opposed to testing an isolated individual function. Are those automated? Longer tests tend to break more and are more complicated to automate. Many teams automate their regression tests but exclude from the complex, difficult or long scenario tests. These require a higher level of automation skill. Too many automation tests today are small, isolated tests that validate individual functions. We all know that is not how users use a product. They use functions in various sequences, combinations and varieties of data. It is certainly not an easy task to automate and maintain all of those more complex tests.
Also, many test teams have not been responsible for running performance, load and security tests. Traditional test teams may, in the DevOps world, still not be the team writing them. They may manage or execute these new testing tasks earlier as part of their regular automated suites. A good strategy will define when and how often these tests will be run.
Remember, the goal of CT is immediate feedback. A common question that arises is that why can’t every test be put into one giant automated suite and have it run all the time. That is not required. We don’t need to run the same tests so often for immediate feedback. It is better to have lean and mean suites of tests that are run strategically. Run unit tests, feedback, go to the next step, run smoke tests, feedback, go to the next tests and so on.
Automation
I still see teams struggle with agile automation. They can’t automate enough in the same sprint, the automation is fragile, there is no methodology, which can lead to the suite staying too small, or conversely, getting bloated and slow when we need lean and mean. These problems need to be fixed before attempting CT.
Equally important is that some teams still don’t have adequate resources to manage and maintain an automation framework, code new tests, thoughtfully retire old tests and effectively work on new user stories in new sprints. Where there are many teams that will easily slide in continuous testing, there will be more teams who cannot keep up with the new demands and compressed delivery requirements.
Leveraging technology
Many teams still suffer from data and environment issues. Luckily, virtualization, containers and the cloud, can take care of these by leveraging popular technologies. But most test teams will need the help of Ops to get this accomplished.
SOA/Microservices
There is a swiftly growing trend of building applications out of services. The trend is not new but has been picking up steam. From APIs to service-oriented architecture to web services to microservices, testing is evolving into exercising the integration of products made up of services. The use of containers has sped up the use of services. Test teams must have the knowledge, skills and tools to adequately test and automate the testing of these services.
Outsource service testing if you don’t have the skill within your test team—don’t skip it. Waiting to test these through a user interface (UI) is always possible, but isolating a defect once it happens is much tougher through a UI than at the service level. Running a service test through the UI is not quick and easy. The most important reasons we automate so much in continuous testing is to get quick and easy feedback on the health of the system. Skipping testing API or any service at that service level is not OK anymore.
Summary
Testing is no longer the responsibility of a silo QA team. It is everyone’s job!
Scrum and big movement on XP practices such as TDD or simply doing a bunch more unit testing has greased the skids for the whole team to be more responsible. Everyone on the team has to test. If you can’t, then DevOps is not for you.
Test teams will need sophisticated automation engineers, and Ops/IT staff to support environments, data, VMs, containers and all of the virtualization needs of the test team.
Intelligent, lean and mean test automation is the test team’s primary goal. If your dev and test teams already have mature and efficient agile practices, low maintenance test automation and a good continuous integration process in place, then continuous testing and DevOps is the next step in achieving hyper-efficiency. Everyone in the enterprise benefits by bringing Ops/IT and its processes into the dev process much earlier, by shifting left, and taking advantage of the advances in virtualization and infrastructure automation.
About the Author/Michael Hackett
Michael is a co-founder of LogiGear Corporation, and has over two decades of experience in software engineering in banking, securities, healthcare and consumer electronics.
Michael is a Certified Scrum Master and has co-authored two books on software testing. Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems (Wiley, 2nd ed. 2003), available in English, Chinese and Japanese, and Global Software Test Automation (HappyAbout Publishing, 2006). He is a founding member of the Board of Advisors at the University of California Berkeley Extension and has taught for the Certificate in Software Quality Engineering and Management at the University of California Santa Cruz Extension. As a member of IEEE, his training courses have brought Silicon Valley testing expertise to over 16 countries.
Michael holds a Bachelor of Science in Engineering from Carnegie Mellon University.
