Engineering

Challenges with testing Phoenix LiveView with Cypress

| 6 min read

In our current project, we are using Phoenix static HTML templates for frontend with some JavaScript code where needed. We decided on adding E2E tests from day one and employed Cypress for the job, which it did pretty well. After some time however, we came to a conclusion, that number of dynamic UI elements is big enough to justify using LiveView, at least in some specific components. The first one was basically the LiveView 101 - show a modal after clicking a button. We did it something like this:

# component_live.html.heex
<button
  id="open_modal_button"
  phx-click="open_modal">
</button>


# component_live.ex
def handle_event("open_modal", _value, socket) do
  {:noreply, socket |> assign(:is_modal_opened, true)}
end

Button fires open_modal event that is then handled by handle_event/3 function, which toggles modal visibility (there is a if in HTML that either displays the modal or hides it). And it works just fine - when we click it in a browser, the modal appears as it should.

Let's write acceptance test then

After that we began writing an acceptance test for the whole feature. And there was an interesting problem - the following test code was failing:

And('the open modal button appears', () => {
  cy.get('#open_modal_button').should('be.visible')
});

And('I click the modal button', async () => {
  cy.get('#open_modal_button').click();
});

Then('the modal appears', () => {
  cy.get('#my-modal').should('be.visible');
});

Or more specifically the the modal appears was failing, due to modal not being visible. The button was found and it was clicked (we checked that many times), but the modal indeed didn't show up.

The investigation

Funny enough, when we clicked the button in a Cypress-run browser, the modal appeared normally. So the hint was clear - cypress clicks the button too early... but why? What's going on there under the hood?

Live view life cycles

To figure that out, we need to dive a bit deeper into how LiveView works and how it serves the UI. As you probably know, the mount callback is executed when the live view is ready. But as you may not recall, it is actually executed twice:

  • The first time, when LiveView serves only the static HTML to the client
  • The second time, when LiveView upgrades your HTTP request to socket connection

What that means is that there are actually two phases of loading live view element - first one returns the HTML but without having socket connected and then the second one, which sets up everything needed for it to actually work. This is normally not noticeable by the end users, since the delay between first and second phase is super short, but apparently it's not quick enough for cypress test.

Going back to our test, let's see what exactly happens there

And('the modal button appears', () => {
  cy.get('#open_modal_button').should('be.visible')
});

And('I click the modal button', async () => {
  cy.get('#open_modal_button').click();
});

First clause makes sure that the button is visible and then the second clicks it. But as we know, the button will be served, and therefore visible, before all the socket connection stuff is ready. Let's quickly test that theory with our favourite tool - the evil wait function

...

And('I click the modal button', async () => {
  cy.get('#open_modal_button').wait(100).click();
});

...

We wait one hundred milliseconds between finding the button and clicking it, and, well, the test passes!

Finding the best solution

So it seems we fixed it, right? Just put wait everywhere LiveView is included and call it a day. Well, while waiting one hundred milliseconds works well now, on my machine, will it be enough for my colleague on older laptop? And how about the CI pipeline, usually way slower than our machines? The huge issue with wait is that it almost always leads to flaky tests, which depend on how fast and how loaded the executing machine is at the moment. Of course, you may decide to wait for a ridiculous amount of time, like, let's say 10 seconds, which should somehow provide a guarantee of the test not being flaky, but just imagine a test suite filled with 10 second waits all over the place - you basically kill one of the good test properties, Fast feedback. It's always low for E2E tests, but with waits all over the place it will make your E2E suit basically worthless - sooner or later you will decide to run them only once a day, during night, then once a week, once a month...

So what alternatives do we have?

There are at least two alternatives to waiting inside the test, and they both have one big drawback - they require you to pollute your production code with some ifs that are only needed for tests.

First alternative - serving HTML only on the second mount

The first alternative to waiting in tests is pretty simple at the first sight - we may just tell live view to serve all the HTML on the second phase only, like that:

# component_live.html.heex

<%= if @should_render do %>
  # the actual html

<% else %>

  Maybe a loading icon

<% end %>

# component_live.ex

def mount(_, _, socket) do

  should_render = connected?(socket)

  ...

  assign(socket, :should_render, should_render)
end

This way Cypress will only see the share button after the connection is upgraded to the websocket - and boom, our test passes without waiting. The huge tradeoff however is that this affects UX pretty badly, as the page is blinking for less than a second, before it serves the HTML contents, and that looks pretty bad. It could probably be solved by some animation magic or other stuff, but this requires even more code pollution just for test to pass.

Second alternative - telling Cypress to wait until everything is loaded

The better solution is to explicitly tell Cypress that it should wait for a whole page to load and establish a websocket connection before it clicks the button. This way we don't pollute our production code that much (although still a little), but we push the responsibility to cypress test code. As hard as it sounds, it's actually pretty easy when you know what to do. You can use LiveView JavaScript hooks for that, like this:

<button phx-hook="OpenModal" ...></button>

And then in your JS code, you add this hook to live view:

let Hooks = {}
Hooks.OpenModal = {
  mounted() {
    if (window.Cypress) {
      window.modalButtonComponentReady = true
    }
  }
}

let liveSocket = new LiveSocket('/live', Socket, {
  ...
  hooks: Hooks
});

And last but not least, you need to tell Cypress to wait until modalButtonComponentReady is set up:

And('the modal button appears', () => {
  cy.get('#open_modal_button').should('be.visible')

  cy.window().should(({modalButtonComponentReady}) => {
    expect(modalButtonComponentReady).to.eq(true)
  })
});

This works pretty well, but has one major drawback - it requires you to set up these hooks for every single component you want to click in Cypress, which may pile up pretty quick. However, as it turns out, we can use another functionality shipped with LiveView to achieve the same goal, bot globally, without hooking every component - we can just emitted events.

LiveView emits phx:page-loading-stop event when the page in question is fully loaded, meaning it has the connection already upgraded. And we can take advantage of that with our approach:

window.addEventListener("phx:page-loading-stop", (info) => {
    if (window.Cypress) {
      window.componentsReady = true
    }
  })

And in Cypress, analogically:

And('the modal button appears', () => {
  cy.get('#open_modal_button').should('be.visible')

  cy.window().should(({componentsReady}) => {
    expect(componentsReady).to.eq(true)
  })
});