I run my own programming blog called Unicorn Utterances, I do development of all kinds, usually focusing on web development.
I collect retro video games 🎮, like reading 📖, love teaching 👨🏫
The idea of running a CLI app in Docker to ensure that commands are in a clean environment is excellent! It also seems to run pretty well with what you're looking to do!
That said, I have some thoughts I'd love to hear your input on.
As I see it, there are a few papercuts in your approach:
1) You mention in the article:
The testing logic is non-trivial, too.
This is often a bad thing IMO. It often leads to bugs in your testing environment and debugging your tests themselves. I've even seen some with tests for their test logic.
2) It seems like - because you're using JSON - that you lose the ability to make per-test assertions.
What do you do if you need to do more custom testing logic than simply running a command? Often what I've seen in similar implementations is that "we'll add a new field to our JSON file," and after enough time, what ends up happening is that your JSON fields end up as a pseudo approximation of JS itself.
3) It seems like your commands are only ever able to be run synchronously currently.
This means that you can't do things like: "Hey, what was the output of the middle of a command" like you would hope to do for E2E tests
I hope I haven't come off as too mean with these points - I only mention them because I've run into them myself. In fact, I ran into these same problems so often that I wrote my own Testing Library to solve for many of these:
This testing library has pretty good tests written for them, to avoid having to test the testing logic elsewhere.
And, given you're in the React ecosystem, you may already be familiar with this testing API as well!
I know this is forward, but I think it would be a good fit for your project as well (and be reasonably easy to integrate!). If you'd be up for it, I'd love to help with seeing if you agree as well!
Full stack developer & undergrad student of Mathematics; very passionate about both. Enthusiastic about learning, teaching, writing, open source, linguistics and more.
The idea of running a CLI app in Docker to ensure that commands are in a clean environment is excellent! It also seems to run pretty well with what you're looking to do!
That said, I have some thoughts I'd love to hear your input on.
As I see it, there are a few papercuts in your approach:
1) You mention in the article:
This is often a bad thing IMO. It often leads to bugs in your testing environment and debugging your tests themselves. I've even seen some with tests for their test logic.
2) It seems like - because you're using JSON - that you lose the ability to make per-test assertions.
github.com/NitzanHen/agrippa/blob/...
What do you do if you need to do more custom testing logic than simply running a command? Often what I've seen in similar implementations is that "we'll add a new field to our JSON file," and after enough time, what ends up happening is that your JSON fields end up as a pseudo approximation of JS itself.
3) It seems like your commands are only ever able to be run synchronously currently.
github.com/NitzanHen/agrippa/blob/...
This means that you can't do things like: "Hey, what was the output of the middle of a command" like you would hope to do for E2E tests
I hope I haven't come off as too mean with these points - I only mention them because I've run into them myself. In fact, I ran into these same problems so often that I wrote my own Testing Library to solve for many of these:
github.com/crutchcorn/cli-testing-...
This testing library has pretty good tests written for them, to avoid having to test the testing logic elsewhere.
And, given you're in the React ecosystem, you may already be familiar with this testing API as well!
I know this is forward, but I think it would be a good fit for your project as well (and be reasonably easy to integrate!). If you'd be up for it, I'd love to help with seeing if you agree as well!
Let me know :) Happy to chat over DMs as well:
twitter.com/crutchcorn
Hi! Sorry for the delayed response.
You didn't come off as mean - I appreciate your direct (yet respectful) manner of speech.
Your points are all valid, and I'll keep them in mind as Agrippa's tests evolve (the same goes for the library you mentioned).
Cheers!