Click here to Skip to main content
15,610,223 members
Articles / All Topics
Posted 25 Nov 2018

Tagged as


1 bookmarked

Mistakes – Are Your Automated Tests Really Testing?

Rate me:
Please Sign up or sign in to vote.
4.00/5 (1 vote)
25 Nov 2018CPOL3 min read
Are your automated tests really testing?


There’s a great deal of satisfaction to be had from seeing a consistently green dashboard for your automated testing, the business thinks the testers are writing good, reliable tests and their software is free of issues. The trouble is, it’s not that difficult to get a test that passes all the time if you’re not actually validating anything of note. Obviously, the validation you need to be performing in your tests depends entirely on your requirements and acceptance criteria, so there is no definitive list of what you must and musn’t check. To stick with what we are familiar with in previous articles, let’s use the term “assert”.

While there is no right or wrong assert, there is most definitely a good and bad assert.

Looking at a specific example, let’s take an assert of an expected string after performing an action on a page, in this example, a successful order being placed. Our acceptance criteria states that a successful order being placed should show a string saying:

"Thank you for your order! You will receive confirmation by email shortly!"

We could do an assert at the end of our test that says:

("Thank you for your order! You will receive confirmation by email shortly!");

Now the above will work, and will pass as long as our confirmation page contains that string. But that’s the trouble, if it CONTAINS the string. If our string came back as:

"Yo shopper. Just thought we’d say "Thank you for your order! 
You will receive confirmation by email shortly!" 
But in the mean time, you could have gone elsewhere and got it a lot cheaper, we’re overpriced!"

But because we’re only checking our string contains the message, it will still pass. It doesn’t care what else is before or after it, just as long as at some point, that string is there.

Which means basically, you aren’t really testing much at all. A better assert would be:

("Thank you for your order! You will receive confirmation by email shortly!");

Now, the only way our assert can pass is if our string is exactly what we are passing in. No extras, no typos, nothing. So straight away, our assert is a much better test of our order placement page.

But it’s important to point out that even in this most simple of examples, our assert for our test can be even more thorough to ensure even better test coverage. Let’s say our order confirmation page is on its own URL, and as well as that, it also has a ‘Continue Shopping’ button that will only appear when on that page. We can improve the validation of our test even more by adding the following:

("Thank you for your order! You will receive confirmation by email shortly!");
Assert.AreEqual($"{homePageUrl}/orderconfirmation", orderConfirmationPage.Url)

Now we are ensuring that not only is our text correct, but that we’ve also correctly navigated to the correct page and the expected button exists on the page. So we can be completely confident that if our test passes, we are testing everything, and now if we have a failure, something must be wrong.

It does not just asserts that can prove to be an issue with poor test validation. The test cases themselves can be the problem. If a test case is particularly troublesome to automate, then it can be tempting to adapt it in a way that you can at least automate part of it or a modified version of it. The trouble with this is that by moving away from the original test case, not only are you moving away from what needs to be tested, but now you potentially have a test case that no-one ever really asked for nor wanted.

These are also the types of tests that are easy to get to pass, because they do so little, or test nothing of note within the application/system they are written for. This is why it’s important at the start of a project, or even at the start of each sprint or iteration, to determine exactly what your test strategy will be so you can identify these areas unsuitable for automation early on.

The main focus of this article was to emphasise how important it is when writing automated tests, to not only have validation, but to have the correct validation. Never write tests with the intention of them always passing, write them with the test case and acceptance criteria in mind. This way, you rely on the development on being correct and the quality being high for your test to pass, rather than a weak test.

The post Mistakes – Are Your Automated Tests Really Testing? appeared first on Learn Automation.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Written By
CEO Steed Solutions Ltd
United Kingdom United Kingdom
I’ve been in the software industry for over 10 years now. Graduating from university with a 1st in Computer Science, I accidentally found myself in a software testing career instead of the development career I intended. But that was no bad thing… I quickly learned that via test automation, I still got to do all the coding and technical things that I loved. And I’ve loved it since.

My first role was developing test scripts in VB in a QTP framework, and from there, quickly moved in to roles where I was developing frameworks in C# for companies that had no previous automation in place. Fast forward to where I am now and I am freelancing as an automation consultant, going in to companies to build frameworks or provide support and expertise to those who already have automation in place.

Comments and Discussions

SuggestionSome other test cases Pin
Nick__J25-Nov-18 23:41
Nick__J25-Nov-18 23:41 
GeneralNice start, but keep growing that list Pin
madden25-Nov-18 21:57
madden25-Nov-18 21:57 
While the article explains that creating good unit tests is hard, your example of additional tests misses the mark. The general consensus is that each unit test should only contain one (1) Assert, and that the name of the unit test clearly states what is being asserted.
In your example it would mean that yes, you will test the same construct multiple times, but each unit tests tests a specific fail condition.

Other than that, keep the list growing.

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.