When working on my side projects I find it very important to get a straight testing process; since the projects can be untouched for months I want to be sure everything is ok when I push new code. Usually I use Netlify to deploy my sites; it's free (open source plan), it's easy to integrate with GitHub and it gives me deploy previews for each pull request.
Since e2e testing is all about being as close as possible to "real" environments I want my tests to run against Netlifys deploy previews instead of serve or some other local server setup (actually I've found bugs which was only produced in Netlifys production).
Netlify is creating deploy previews by listening for new PR's. I thought it was possible to deploy previews from CLI instead, and then wait for the response to run my e2e tests. Something like:
yarn netlify deploy && yarn e2e
To solve my problem I added an outgoing webhook who calls an URL upon deploy succeeded. I pointed the webhook to a custom AWS Lambda function I called netlify-travis-integration. netlify-travis-integration reads the deploy preview URL from the payload and triggers a Travis job to run e2e testing against.
Since I couldn't find a way to run a specific Travis job I had to distinguish my non-Netlify required jobs from my Netlify required jobs by checking if the build was triggered by API or not:
jobs: include: - stage: lint if: type != api script: yarn lint - stage: test if: type != api script: yarn test:ci && codecov - stage: packtracker if: type != api script: yarn build - stage: lighthouse if: type = api script: yarn lh -- --perf=96 --bp=96 --no-comment $SITE_URL - stage: e2e if: type = api script: yarn e2e
This is a bit ugly, but I couldn't find a better way to solve it when using Travis. I think this is easier to solve with CircleCI, so I'll prefer to choose that for my next project.
To keep track of my e2e tests I added a webhook to my Travis config to let it call another AWS Lambda function. This function checks if the Travis job was triggered by an API and then creates a pull request status.
After one year and 100 pull requests I'm satisfied with the solution and it has caught a lot of errors I've missed. Especially it's comfortable when using Renovate who creates PR's for dependency updates, then it's easy to find out whether something breaks or not.
I created netlify-travis-integration just to solve a problem I had, but in case there's others who'd like to solve the same problem it's not super easy to do the setup; you need to create an AWS account, setup AWS permissions, configure Travis and Netlify and also create a GitHub API token. It should be considered as a proof of concept. Maybe I'll take the next step and make it easier to integrate with new projects.
How are you solving this problem? Are you just running your e2e tests against a local web server?