Software Manager, Agile Aficionado, Tech Dabbler, and Public Speaker. A lifelong learner, the prospect of something new and coffee are what get me up in the morning.
Testing a Black Box is just that, Black. When everything is an external dependency that have no control over its really hard, especially if there are changes in how the data you receive is structured and you don't know about it until you start showing failures and have to start checking. Or you check the feeds first and log that issues that things are not in the right format.
About the best way of handling things that are external is mocking them, simulating all your external dependencies and running tests Unit and or Functional using those mocks is a good way to expand your coverage. One of the bigger types of testing I hear more and more about now is Contract Tests, which I am sadly not as completely versed in as I would like to be since its not been something I have had a need of, in a situation like yours it may be a good fit. At least with mocks you should be able to automate your tests by bringing them up in a framework and running any tests you need, and if you have many external feeds you should be able to cheaply create multiple scenarios to test against.
Right now I have two or three API points to check. They are queried multiple times, though.
I feel it is useless to check all of this as unit tests :/
When i was doing a huge update, I wrote multiple functional tests, that were working on real data (all of the data is read-only for me, anyway), and check what I have to check manually, just in the debugger :/
However I in my functional tests I provided tests for the API point and was able to tell which failures were due to my bugs, and which were caused by external data.
Sadly, the corporations coverage reporting tool doesn't acknowledge my tests as a 'proper test' ;(
Software Manager, Agile Aficionado, Tech Dabbler, and Public Speaker. A lifelong learner, the prospect of something new and coffee are what get me up in the morning.
My view is if you cover the functional part once, that's enough, while different data inputs may be useful you shouldn't need more than one test to cover each case. Though tests should always be idempotent.
From what your saying about your testing its complete for me, and pretty much what I would do. I occasionally add in destructive tests when I can, but only if they are necessary. They should also be able to be cleaned up after and not leave the system in a bad state. Tests should be invisible to each other so it should never matter in what order they are run.
Coverage Tools can vary, but in some of them I know you can only capture certain types of actions. That's a deeper subject though.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Testing a Black Box is just that, Black. When everything is an external dependency that have no control over its really hard, especially if there are changes in how the data you receive is structured and you don't know about it until you start showing failures and have to start checking. Or you check the feeds first and log that issues that things are not in the right format.
About the best way of handling things that are external is mocking them, simulating all your external dependencies and running tests Unit and or Functional using those mocks is a good way to expand your coverage. One of the bigger types of testing I hear more and more about now is Contract Tests, which I am sadly not as completely versed in as I would like to be since its not been something I have had a need of, in a situation like yours it may be a good fit. At least with mocks you should be able to automate your tests by bringing them up in a framework and running any tests you need, and if you have many external feeds you should be able to cheaply create multiple scenarios to test against.
Right now I have two or three API points to check. They are queried multiple times, though.
I feel it is useless to check all of this as unit tests :/
When i was doing a huge update, I wrote multiple functional tests, that were working on real data (all of the data is read-only for me, anyway), and check what I have to check manually, just in the debugger :/
However I in my functional tests I provided tests for the API point and was able to tell which failures were due to my bugs, and which were caused by external data.
Sadly, the corporations coverage reporting tool doesn't acknowledge my tests as a 'proper test' ;(
My view is if you cover the functional part once, that's enough, while different data inputs may be useful you shouldn't need more than one test to cover each case. Though tests should always be idempotent.
From what your saying about your testing its complete for me, and pretty much what I would do. I occasionally add in destructive tests when I can, but only if they are necessary. They should also be able to be cleaned up after and not leave the system in a bad state. Tests should be invisible to each other so it should never matter in what order they are run.
Coverage Tools can vary, but in some of them I know you can only capture certain types of actions. That's a deeper subject though.