You want the client to really use/play with the application even when it isn't finished, you want to be sure you really understood the needs, work with expert in the field that know their job and so on.
This is actually a both true and interesting thought. I think working together with a customer and showing him/her the results iteratively could (and maybe should) be viewed as part of testing software.
From the view of a customer, it's not easy to understand the difference between "That's a bug" and "That's not what we wanted" and maybe there even shouldn't be a difference. I think I might add a section to my post :)
Noticably, I find that integration test can often also serve as acceptance and non regression tests and are fast to write and do test a wide variety of features with quite low cost.
In fact, we don't write real acceptance tests at all and us a lot of integration tests. When you use spring boot integration tests, there are quite fast. Despite that, the cost of writing and executing them is still higher than writing unit tests - but luckily not much anymore. I thing it depends on what system you build and what language you use. I worked for quite some time in automotive industry developing large C++ systems. The integration tests took 20min to run.
From the view of a customer, it's not easy to understand the difference between "That's a bug" and "That's not what we wanted" and maybe there even shouldn't be a difference. I think I might add a section to my post :)
Depending of the organisation and contract there may be lot of difference. But to me what really count to consider the product as really successful is that it is really used in production and that it actually solve customers problem and make the happy.
It is said in the industry that about half of the project in computer science are failure. Often it can been see as budget issue: there no more money to pour on it.
Even if this purely cost, one can wonder why we started such project if it was so expensive and the client didn't want to afford it. Often it is cost was minimized and that to actually fix the actual software lot of money is needed without any proof that the thing is gona to be worth it. The software at is doesn't really solve enough of the customer problem to be worth the investment.
This is as a core of any project, we need to be clear on the objectives, if they are achievable and go toward that. If everything is a success but the software doesn't meet the original objectives, that's a failure. Not of one person but collectively.
I remember one guy saying that if by multiplying the estimated cost by 2 and by dividing the estimated revenue by 2 you don't see that you'll make lot of money the project likely isn't worth it. Because everybody is too optimistic.
Despite that, the cost of writing and executing them is still higher than writing unit tests - but luckily not much anymore. I thing it depends on what system you build and what language you use.
Yep, I think the cost depend extremely heavily on the technology used, both how smart one write his tests but also how easy the technology and also the system overall make it easy to test or not.
Often not enought is invested to improve the tooling, speeding its run, train people to use it efficiantly and so on.
What I can say ? The cost of unit test vary a lot. But there 2 costs, people working on it, maintaining it cost and the cost of machines running the tests.
To me even in the country with the lowest wages, developpers, software engineer are extremely costly. They don't like to do things repeatly and their motivation lower significantly if you force them. Also the more people are needed to achieve something, the more is spent in communication, training, knowledge sharing and so on and the less doing the real stuff. Cost of software is exponential.
So one should invest a lot in the capacity to automate everything, to get thing done on cluster of slave machines so nobody has to wait and there no reason to not do things in parallel if a single machine is too slow. In big projects people may waste most of their time in validation, investing there can have huge impact on individual and so overall company productivity.
Getting a cluster of machine doing the validation is cheap compared to people. Being able to have a test system like production shall be easy as anyway in cloud day everything is automated and you have the script to deploy the whole system automatically scale it up and down, again automatically.
While you don't want to maybe push something that hasn't been tested and waste hours of CPU/machines for no reason and to go back to it 20 times and have long iterations, you can layer your testing, runing the few tests that matter locally and fast, and get the additional confidence from the testing infrastructure.
Thanks for your great comments.
This is actually a both true and interesting thought. I think working together with a customer and showing him/her the results iteratively could (and maybe should) be viewed as part of testing software.
From the view of a customer, it's not easy to understand the difference between "That's a bug" and "That's not what we wanted" and maybe there even shouldn't be a difference. I think I might add a section to my post :)
In fact, we don't write real acceptance tests at all and us a lot of integration tests. When you use spring boot integration tests, there are quite fast. Despite that, the cost of writing and executing them is still higher than writing unit tests - but luckily not much anymore. I thing it depends on what system you build and what language you use. I worked for quite some time in automotive industry developing large C++ systems. The integration tests took 20min to run.
Depending of the organisation and contract there may be lot of difference. But to me what really count to consider the product as really successful is that it is really used in production and that it actually solve customers problem and make the happy.
It is said in the industry that about half of the project in computer science are failure. Often it can been see as budget issue: there no more money to pour on it.
Even if this purely cost, one can wonder why we started such project if it was so expensive and the client didn't want to afford it. Often it is cost was minimized and that to actually fix the actual software lot of money is needed without any proof that the thing is gona to be worth it. The software at is doesn't really solve enough of the customer problem to be worth the investment.
This is as a core of any project, we need to be clear on the objectives, if they are achievable and go toward that. If everything is a success but the software doesn't meet the original objectives, that's a failure. Not of one person but collectively.
I remember one guy saying that if by multiplying the estimated cost by 2 and by dividing the estimated revenue by 2 you don't see that you'll make lot of money the project likely isn't worth it. Because everybody is too optimistic.
Yep, I think the cost depend extremely heavily on the technology used, both how smart one write his tests but also how easy the technology and also the system overall make it easy to test or not.
Often not enought is invested to improve the tooling, speeding its run, train people to use it efficiantly and so on.
What I can say ? The cost of unit test vary a lot. But there 2 costs, people working on it, maintaining it cost and the cost of machines running the tests.
To me even in the country with the lowest wages, developpers, software engineer are extremely costly. They don't like to do things repeatly and their motivation lower significantly if you force them. Also the more people are needed to achieve something, the more is spent in communication, training, knowledge sharing and so on and the less doing the real stuff. Cost of software is exponential.
So one should invest a lot in the capacity to automate everything, to get thing done on cluster of slave machines so nobody has to wait and there no reason to not do things in parallel if a single machine is too slow. In big projects people may waste most of their time in validation, investing there can have huge impact on individual and so overall company productivity.
Getting a cluster of machine doing the validation is cheap compared to people. Being able to have a test system like production shall be easy as anyway in cloud day everything is automated and you have the script to deploy the whole system automatically scale it up and down, again automatically.
While you don't want to maybe push something that hasn't been tested and waste hours of CPU/machines for no reason and to go back to it 20 times and have long iterations, you can layer your testing, runing the few tests that matter locally and fast, and get the additional confidence from the testing infrastructure.
Thank you for taking the time to write such a great comment!