Root causing bad network requests in failed tests

Many frontend test failures happen due to network requests which timed out, returned an error code, or returned unexpected data the frontend didn’t handle as expected. These failures can happen either in CI or in synthetic monitoring, and require manual investigation whenever they occur. Using Replay, we can automate this investigation to directly identify the network problem as the root cause of the failure.

To demonstrate this, let’s use an e2e test in the Cypress Real-World App (itself a demo app) repository. We recorded one of the tests in this repo passing, modified a backend handler so that certain requests did not return, and then recorded the same test failing afterwards. The failing test times out due to a [data-test*=transaction-item] locator not matching.

From the failure logs we don’t know why this locator failed to match any elements. This could be a problem with the locator itself, but it could also be a problem with the expected element not being present. In this case it’s the latter, and similar to what we did when fixing test failures caused by bad locators, we can automatically compare the passing and failing recordings to not only identify the root cause but explain why it caused the test to eventually fail.

That means we need to do the following:

Identify the element which was present in the passing recording but not the failure.

Identify the React component which created that element.

Find the network requests that React component depends on: everything that was fetched in order for the component to be created.

Look in these network requests for one which was made in the failing recording but timed out or errored.

List all the events which failed to occur due to the problematic network request, leading up to the final missing React component.

All of this information can be computed automatically from the passing and failing recordings. Let’s look closer at steps 3 and 5, which is where most of the work is happening.

To understand the effect that the network requests have on a page, we have to understand the dependencies between the code making those network requests and all the other code which executes while the page loads. If function A calls function B, B’s execution depends on A, and if function A creates a promise P and calls P.then(C) then C depends on both A and whatever resolved P.

These dependencies form a graph, and by computing and traversing that graph we can find all the code and any associated network requests that are transitively involved in creating the element we’re focusing on. When we identify the problematic network request, the chain of events connecting that request to the final element creation in the dependency graph is everything that should have happened but didn’t because the request failed.

Then, we have what we need to succinctly explain not just what the problem is but why it caused the test to fail:

plain text
When the test passes, the element matching the locator is only rendered
after a network request on http://localhost:3001/transactions/public succeeds.

When the test fails, this network request was made but did not return before
the test timed out. This prevented React components in the following files
from rendering, and the element matching the locator was not created:

TransactionInfiniteList.tsx, TransactionItem.tsx

Here is a video walking through the steps involved in automatically generating this explanation:

Capturing and analyzing the dependencies between events that happen on a page gives us precise insight into what caused this test failure. More generally, the dependency graph is a powerful tool for debugging and optimizing an application, which we’ll be exploring more in future posts. In the meantime, if you’re interested in tools to automatically diagnose and fix your test failures, we’d love to talk. Reach us at hi@replay.io or fill out our contact form and we’ll be in touch.