Watch out for fixtures in cypress.io

Przemek Suchodolski — Thu, 25 Jun 2020 22:26:27 +0000

Today I would like to tell you a story about a bug that cost me two days of searching and debugging sessions. It turned out a trivial thing, and with a better error message, it could have taken seconds instead of days. Let's go!

Hey Przemek! Could you help me?

A few days ago, I noticed that our VRT (Visual Regression Tests) suite started to fail for one case. I've asked my colleague, Monica, to check it. She accepted the challenge. After a long day of searching the root cause, she told me that she doesn't have any idea why the test is failing. On the local machine, it has been passing all the time, but on our GitlabCI, we got an error. Weird thing, isn't it? Monica was resigned and asked me for help. After two days of trying, committing, pushing, waiting, we've finally found it.

Fake server

We use a lot of tools in our tests. For unit testing, we use jest. In E2E, we use py.test with webDriver bindings. We also have UI tests that check our app on a higher level (interactions between components, pages, or views). Recently we introduced another test suite - VRT (Visual Regression Tests). The last two (UI and VRT) are based on cypress.io. It is an excellent tool for writing tests - from unit to full E2E.

Backend in our app is very complicated, and it is tough to setup a local environment. Because of that, for UI and VRT tests, we use a killer feature from cypress.io - network stubbing. Cypress can plug in between our app and network request giving us a possibility to decide about the response from API endpoint.

it("test with network stubbing", () => {
  // First, we need to start fake server
  cy.server()
  // Next, declare the route that we want to stub
  cy.route("/api/endpoint", { value: 1 })
})

More info about stub responses can be found in official Cypress documentation.

Fixtures

Fixtures are another feature from cypress.io that we use a lot, especially in our VRT suite. A fixture is a simple file that holds the data. We can reuse this file in many places. It helps us in organizing tests and managing the common responses from stubbed network requests. To load a fixture, we use a cy.fixture command. It expects a path to the file that we want to load. The path should be relative to a folder specified to hold fixtures (cypress/fixtures by default). Let's assume that we have the following file structure:

- fixtures
    - myFixture.json
    - someSubFolder
          - mySecondFixture.json

And now let's look at code which loads fixtures:

it("test with fixtures", () => {
  // We don't need to specify the file extension
  // Cypress will try to figure it out
  cy.fixture("myFixture").then(data => {
    // Here we can read the data
  })

  // We can save the fixture as an alias ...
  cy.fixture("someSubFolder/mySecondFixture").as("myAlias")

  // ...and then use the alias in stub of response
  cy.route("/api/endpoint", "@myAlias")
})

Authors of Cypress took care of reducing a boilerplate needed to use a fixture in stubbing network requests 🔥🔥🔥. The cy.route command can take a shortcut to fixture as a response argument:

cy.route("/api/path", "fixture:myFixture")
cy.route("/api/endpoint", "fx:someSubFolder/mySecondFixture")

In this way, we stubbed a network request with data kept in reusable fixture files. Great job!

Where is the hero of the story?

Ok, but where did our bug go?

I've created a simple app to visualize the issue. In the beginning, the app displays the Loading… message, then makes a request and replaces the text with a downloaded response.

Fetching the data in old, good XHR way 😎

<body>
  <div id="main">Loading...</div>
  <script>
    const mainEl = document.querySelector("#main")

    const req = new XMLHttpRequest()
    req.open("GET", "/api/endpoint", true)
    req.onreadystatechange = function() {
      if (req.readyState == 4) {
        const msg = req.status == 200 ? req.responseText : "Error"
        mainEl.innerHTML = msg
      }
    }
    req.send(null)
  </script>
</body>

I've also written a test:

describe("Simple fixture test", () => {
  it("displays response", function() {
    cy.server()
    cy.route("/api/endpoint", "fixture:examplefixture")

    cy.visit("/")

    cy.get("#main").should("have.text", "Hello")
  })
})

And created a fixture file fixtures/exampleFixture.json:

Hello

Have you noticed a bug yet?

In my case, the screenshot from the failed test was very helpful. Cypress takes them by default for failing tests, which is neat 🔥!

And now...Have you noticed a bug yet?

A message about the status from the stubbed request caught my attention. It was 400 instead of 200. That was a clue.

The typo and file systems

Our bug, which we've been trying to solve with Monica, was a simple typo. The name of the fixture file was in camelCase, and we tried to load it via shortcut without the same naming convention.

exampleFixture vs examplefixture

Ok, but why does it work on the local machine and doesn't on CI?

99% of our frontend team works on MacBooks. Our CI runs the tests in the docker container (Linux). You can think - "so what?". The default file system on Linux is case sensitive. On the other hand, the default file systems on Mac or Windows are not. What does it mean in practice?

On Linux you can create two files with the "same" name (different letter case):

myAwesomeFile.js
myawesomefile.js

Linux treats them as separate files. Try to do the same on Mac or Windows - you can't do it. It has also impact on the way how you load the files, for example in nodejs. On Mac, there is no difference in load file by "myFixture" or "mYFiXtURe" names - the file will be loaded. On Linux, we will get an error - file not found.

Let's check it

If we modify the code of our test in this way:

cy.route("/api/endpoint", "fixture:ExAmPlEFiXTuRe")

The test is always green on Mac. On Linux we get a 400 status for stubbed network request and an error message in console.

CypressError: The following error originated from your application code, not from Cypress.

When Cypress detects uncaught errors originating from your application it will automatically fail the current test.

This behavior is configurable, and you can choose to turn this off by listening to the `uncaught:exception` event.

https://on.cypress.io/uncaught-exception-from-application

Wait, wait, wait...WAT? The following error originated from your application code, not from Cypress. Are you sure Cypress? 🤔

Let's try to load the fixture without a shortcut:

// We made a mistake in fixture name
cy.fixture("examplEFixture").as("response")
cy.route("/api/endpoint", "@response")

// With storing fixture in an alias we can use it in our assertions
// We don't need to hardcode the "Hello" string
cy.get("@response").then(data => {
  cy.get("#main").should("have.text", data)
})

The error message for this code is quite different:

Error: A fixture file could not be found at any of the following paths:

> cypress/fixtures/examplEFixture
> cypress/fixtures/examplEFixture{{extension}}

Cypress looked for these file extensions at the provided path:
.json, .js, .coffee, .html, .txt, .csv, .png, .jpg, .jpeg, .gif, .tif, .tiff, .zip

Provide a path to an existing fixture file.

And this is the error message that I've been counting on 👏 . We know right the way where we should start looking 😎.

Summary

There are two takeaways from this story:

small typo could make you cry for two days of debugging session
you are as good as the error message from your test runner ;)

I think that Cypress could return the better message about missing fixtures than CypressError. That's why I've created an issue in cypress GitHub repository - here you can check the status.

Thank you for your attention. I am going to try to solve the issue that I've created 😉. Maybe I will be able to add something to the OpenSource community to make cypress.io even better 😁

DEV Community: Przemek Suchodolski