Exploring a more complete front-end testing strategy

2019-01-03

This article assumes that we are continuously developing a relatively large front-end project with domain-driven design and an object-oriented programming model. The front-end business logic is split into domain modules and UI components. For this kind of project, we need a more layered testing strategy to protect both delivery speed and code quality.

General front-end testing

Whether we use a traditional testing model or a continuous delivery model, we typically define three types of tests:

generic-test

End-to-end testing involves ensuring that the integrated components of an application function as expected. The entire application is tested in a real-world scenario such as communicating with the database, network, hardware and other applications.

Integration testing is a key software development life cycle strategy. Generally, small software systems are integrated and tested in a single phase, whereas larger systems involve several integration phases to build a complete system, such as integrating modules into low-level subsystems for integration with larger subsystems. Integration testing encompasses all aspects of a software system’s performance, functionality and reliability.

Unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.

What is wrong with general front-end testing?

When evaluating a testing strategy, we can use the following criteria:

Pass acceptance criteria
Catch bugs as early as possible
Faster running speed, especially at the top level of testing
Find bugs efficiently
Test writing costs and maintenance costs
Continuous refactoring risks

I think a better testing strategy should satisfy these criteria as consistently as possible.

In the general testing strategy above, E2E tests can cover more acceptance criteria, but they usually run less frequently. Integration tests run more frequently than E2E tests, but they often start a large integrated application and become bloated. Unit tests usually make up the largest portion of the strategy. They provide good logic coverage, but large refactors often require many unit tests to change as well.

If our goal is to catch bugs as early as possible, the general strategy can be improved. E2E tests are often too slow or unstable to run on every pull request or every commit, so they tend to run less frequently. They are usually the most expensive automated tests to execute and maintain.

Sometimes integration tests start a large integrated system with many mocks. They can be run repeatedly, but as the number of integration cases grows, we can no longer guarantee fast completion for each pull request. In a resource-constrained CI environment, the suite may take half an hour or longer.

As a system becomes more complex, we need test reports that help identify not only which cases failed, but also where the failure is likely to be. The cause might be network instability, backend API exceptions, front-end domain module errors, UI component errors, or something else. A generic testing strategy provides limited help for bug localization. For example, if unit tests pass but integration and E2E tests fail, the report alone may not provide enough signal.

The cost of writing tests must be balanced with continuous delivery. When acceptance criteria are clear, test code should align with them. In theory, implementing all acceptance criteria through E2E tests verifies product behavior directly, but this creates an unhealthy balance: high instability, slow feedback, and high maintenance cost.

If unit tests are extensive enough, can they guarantee acceptance criteria? Not necessarily. Unit tests are closely coupled to implementation details. Each refactor may require related unit tests to change as well, so we still need higher-level tests to protect behavior during refactoring.

What’s the key to solving the problem?

For an ATDD-oriented continuous delivery model, acceptance criteria are the most important anchor. A good testing strategy should give us confidence during refactoring while balancing execution speed, bug localization, and test maintenance cost. Instead of pushing one test type to the extreme, we should use a layered approach similar to Liebig’s law, where the weakest layer determines the overall quality of the strategy.

Proposing a more complete testing strategy

new-test-strategy

E2E tests should cover the most important acceptance criteria. Ideally, they should also support smoke testing, UI testing, and multi-browser compatibility testing.

IT3 is an integration test of the overall system based on mock services. It can reuse E2E test code, but it does not start a real browser; all tests run in Node.js. Because backend APIs and the real browser DOM are mocked, IT3 is faster than E2E and can run more frequently. The key point is reuse: IT3 should reuse E2E-level behavior descriptions, which general integration tests often fail to do.

IT2 tests the minimum set of UI components and domain modules needed for a behavior. It also uses mock services, including server APIs, DOM, and BOM. Because it starts only the minimum required module set, it is faster than IT3. IT2 and IT3 should have clear boundaries: IT2 validates a minimal behavior slice, while IT3 validates the whole application composition. This makes failure reports easier to interpret.

IT1 is an integration test for the minimum set of domain modules. It only requires mocked backend APIs. Because it starts only the minimum set of domain modules, it is faster than IT2. One or more acceptance-criteria steps can be converted into IT1 or IT2 tests. Comparing IT1, IT2, and IT3 reports makes it easier to infer the location and cause of a bug.

For underlying modules with few or no dependencies, especially core functions and helper functions, unit tests are still valuable. They can cover more input examples than higher-level tests and serve as an important complement to IT1/IT2/IT3.

it-cover

Different test types cover different factors. With this strategy, E2E covers almost all factors. IT3 removes real server APIs and real browsers. IT2 removes non-essential modules and UI components from IT3. IT1 removes UI components from IT2. UT focuses only on small core logic units.

With this layered strategy, we can define a better execution plan. For example, run UT/IT1 on commits, run UT/IT1/IT2 or even IT3 on pull requests, and run E2E on a regular schedule or before release. This keeps acceptance criteria protected while balancing feedback speed and bug localization.

How to implement this layered strategy

Example business code:


@Module()
class Foo {
    a() {}
    _x() {
        //No dependent module core logic
    }
}

@Module({ dependencies:['Foo'] })
class Bar {
    b() {}
     _y() {
        //No dependent module core logic
    }
    get name(){}
}

@Module({ dependencies:['Foo', 'Bar'] })
class Foobar {
    c() {}
    get name(){}
}

const store = createStore(
    //...factory module
);

const FoobarContainer = (props) => (
    <div onClick={props.foobar.c}>
        {props.foobar.name}
    </div>
);

const BarContainer = (props) => (
    <div onClick={props.bar.b}>
        {props.bar.name}
    </div>
);

class App extends Component {
    render() {
        return (
        <div>
            {this.props.foobar ? (
                <FoobarContainer {...this.props} />
            ): null }
            {this.props.bar ? (
                <BarContainer {...this.props} />
            ): null }
        </div>
        );
    }
}

render(
    <App store={store} />,
    mountNode
);

Example acceptance criteria:

Feature: AC

  Scenario Outline:
    Given User sees 'b' node
    When User clicks 'b'
    Then User should see 'b' changed
    When User clicks 'f'
    Then User should see 'f' changed

  Scenario Outline:
    Given User sees 'c' node
    When User clicks 'c'
    Then User should see 'c' changed
    When User clicks 'e'
    Then User should see 'e' changed

Example tests:

// E2E & IT3
test(() => {
    const app = getApp();
    app.find(nodeSelectorB).click();
    expect(result).toBe(expectedValue1);
    app.find(nodeSelectorF).click();
    expect(result).toBe(expectedValue2);
});

test(() => {
    const app = getApp();
    app.find(nodeSelectorC).click();
    expect(result).toBe(expectedValue1);
    app.find(nodeSelectorE).click();
    expect(result).toBe(expectedValue2);
});

// IT2
test(() => {
    const barContainer = getMinimalSet(BarContainer);
    barContainer.find(nodeSelector).click();
    expect(result).toBe(expectedValue2);
});

test(() => {
    const foobarContainer = getMinimalSet(FoobarContainer);
    foobarContainer.find(nodeSelector).click();
    expect(result).toBe(expectedValue1);
});


// IT1
test(() => {
    const app = getMinimalSet(Bar);
    app.b();
    expect(result).toBe(expectedValue);
});

test(() => {
    const app = getMinimalSet(Foobar);
    app.c();
    expect(result).toBe(expectedValue);
});

// UT
test(() => {
    const result = Foo.prototype._x();
    expect(result).toBe(expectedValue);
});

test(() => {
    const result = Bar.prototype._y();
    expect(result).toBe(expectedValue);
});

Conclusion

There are many factors to consider when designing a testing strategy. Acceptance criteria should remain the main source of truth, but we also need to consider execution strategy, speed, test maintenance cost, bug localization, and refactoring confidence. Instead of relying on one test type, an E2E/IT3/IT2/IT1/UT layered strategy can protect both code quality and engineering quality while remaining agile enough for continuous delivery.