I was watching the Data from an API and I adamantly believe this to be wrong. But like all absolutes I need to clarify.
The Situation
I would like to have you get through this point in the training, but to allow you to skip I'll try to describe the situation.
He has a website which uses a card model for content. There can be several cards on the page and they each have a name and category and rarity.
He writes a test which will verify the card details are correct on the page.
Cards are provided by another API call. In the referenced chapter he demonstrates that by using the same API used by the developer we can cover all of the cards rather than write them by hand.
Breaking Things Apart
I will avoid saying this doesn't add to test coverage, but I want to explain why it provides limited coverage.
How Much Coverage
By writing this test can we say that we know the website displays all cards after the tests run?
✅ If the cards were provided at random, the test is unlikely to succeed.
❌ If a card has new content fields we are not checking it is displayed
Random
Basically any difference between the order the API provides and how the UI displays the cards runs the risk of failure.
This leaves us with needing to recreate the same logic as the developers, assuming the API is consistent and the UI is doing the sorting. I'm not saying this isn't usable to help quality, double or triple implementation has been a common approach in highly critical systems. It is common to reduce the amount of excess logic in tests (who is correct, tester or dev).
When we have a passing test we would know that none of these other considerations are true and have covered what we know.
New Card Types
He mentioned that using the API would allow tests to automatically be added as the new cards are added. This assumes we don't add to what a card can have.
If we are tracking the addition of cards we can update our test to handle these new card types. If the cards are from a third party how do you know what they are adding.
Expectations
What is actually happening here is that we are defining our expectations of a working system. However we are not being explicit about these expectations.
Mock
I believe the card interface should be tested with mocking the card data, we test around the expectation of valid card content. Do cards handle unicode, can html be injected.
Confirm Expectations
If the card API is controlled by your development then you may have some insight into the changes occurring to cards. If you have card types mocked then these new types can be added and verified, even if the new types are available in the backed yet.
If you have little insight into the card API changes then an API confirmation could be created. The idea here is to pull the cards and verify they match the data payload expected to the current functionality. If you are really concerned about card changes you want to be alerted to not only changes to used content but unknown additions to content which might be important for the card.
Concerns
One of the primary concerns of the suggested API testing is that it creates the appearance of large coverage but can distract from finding edge cases or miss the lack of coverage due to complexity in the system.
By isolation of UI testing we can simplify the system under test. It creates another concern that the mocks aren't updated to meet the real world data. While I provided a suggestion for mitigation, reality doesn't mean you'll go that far.
Top comments (6)
With an API it depends on what we are trying to test and what risks are we trying to mitigate. APIs are great for mocking up complicated UI testing scenarios or triggering state in a specific way before testing something else. There could be benefits as well around API contract testing during development changes. If you provide the API there can be some sanity checking done but if you're implementing an external API what's the point of testing it? Trust must be extended at some point otherwise we make lots of work for ourselves. There is a good use case for API contract testing such as PACT since end-to-end tests for everything in a large enough environment are too time-consuming and expensive.
I wasn't writing about API testing. EDIT: let me rephrase that, I suggested API contract testing on third party API to mitigate risks if that is a high enough risk for you, but yes I do think trust is needed at some point.
When you say
Do you mean behind a third party API consumer?
Contract testing is definitely the approach I lean on for automation.
Yeah was just referring to the original article writer and wondering what they're trying to convey to the audience.
I is had to write testing articles, so much I have to say starts negative. But I also feel there is so much misinformation and wrong recommendations.
I think it is probably natural to come off negative. Most of the time it seems like people are looking at things the wrong way or not evaluating any true risks. To be fair to the course writers they have to come up with a farcical example for a broad testing audience.
Tell me about it. My article on behavior Object model utilized login as an example, I've never actually used testing login as shown but wanted something familiar from other tutorials.
The code structure part was good, and I do believe that was the main point.