Does testing matter if you're doing DevOps?

Yes!

Throughout the IT infrastructure landscape folks are starting to make significant investments in deployment automation and source control management. Many are realizing the time to value benefits.

Knowledge Management 101

Without tests, there isn’t a clear understanding of a definition of “fit for use”. There are significant consequences when that definition doesn’t exist. Without the definition,

• development and operations teams can unintentionally gild the lily by not knowing when to stop working.
• new team members are unable to engage in effective troubleshooting since they don’t have the knowledge
• “pay your dues” fallacy

What do you value?

Testing tells us what we value and by implication what we don’t value. Too often the implicit test strategy is “toss it over the wall and see if anyone complains”. This has negative consequences in terms of performance, cost of delivery, and trust. If you perform no testing before handing off to your clients there is an implication of disinterest, or at best ambivalence, about your customer’s experience. Manual testing is a valueable step in the proper direction, but it is not a

Things to say about testing

• Testing shows presence of defects, does not assert absence of defects
• Comprehensive testing is not possible
• The earlier a test catches a defect the lower the cost of remediation
• Pesticide testing will make certain you don’t break existing features (known bugs), it won’t prevent you breaking new features (unknown bugs)
• The nature of required testing is a function of the service being delivered

Things to test for

• required features
• regulatory compliance
• security compliance (a variant on required features)

What can you do about it

• document your definition of “fit for use”
• find out what testing tools and frameworks are in use by your company’s application delivery and DevOps teams
• identify the cost and volume of failure demand for the elements of your “fit for use” definition
• identify one or two elements, based upon high costs and/or high rates of failure, to create automated tests