da Agency, Autor auf Cloud Computing Köln

Model-Based Testing with Testcontainers and Jqwik

When testing complex systems, the more edge cases you can identify, the better your software performs in the real world. But how do you efficiently generate hundreds or thousands of meaningful tests that reveal hidden bugs? Enter model-based testing (MBT), a technique that automates test case generation by modeling your software’s expected behavior.

In this demo, we’ll explore the model-based testing technique to perform regression testing on a simple REST API.

We’ll use the jqwik test engine on JUnit 5 to run property and model-based tests. Additionally, we’ll use Testcontainers to spin up Docker containers with different versions of our application.

Model-based testing

Model-based testing is a method for testing stateful software by comparing the tested component with a model that represents the expected behavior of the system. Instead of manually writing test cases, we’ll use a testing tool that:

Takes a list of possible actions supported by the application

Automatically generates test sequences from these actions, targeting potential edge cases

Executes these tests on the software and the model, comparing the results

In our case, the actions are simply the endpoints exposed by the application’s API. For the demo’s code examples, we’ll use a basic service with a CRUD REST API that allows us to:

Find an employee by their unique employee number

Update an employee’s name

Get a list of all the employees from a department

Figure 1: Finding an employee, updating their name, finding their department, and registering a new employee.

Once everything is configured and we finally run the test, we can expect to see a rapid sequence of hundreds of requests being sent to the two stateful services:

Figure 2: New requests being sent to the two stateful services.

Docker Compose

Let’s assume we need to switch the database from Postgres to MySQL and want to ensure the service’s behavior remains consistent. To test this, we can run both versions of the application, send identical requests to each, and compare the responses.

We can set up the environment using a Docker Compose that will run two versions of the app:

Model (mbt-demo:postgres): The current live version and our source of truth.

Tested version (mbt-demo:mysql): The new feature branch under test.

services:
## MODEL
app-model:
image: mbt-demo:postgres
# …
depends_on:
– postgres
postgres:
image: postgres:16-alpine
# …

## TESTED
app-tested:
image: mbt-demo:mysql
# …
depends_on:
– mysql
mysql:
image: mysql:8.0
# …

Testcontainers

At this point, we could start the application and databases manually for testing, but this would be tedious. Instead, let’s use Testcontainers’ ComposeContainer to automate this with our Docker Compose file during the testing phase.

In this example, we’ll use jqwik as our JUnit 5 test runner. First, let’s add the jqwik and Testcontainers and the jqwik-testcontainers dependencies to our pom.xml:

<dependency>
<groupId>net.jqwik</groupId>
<artifactId>jqwik</artifactId>
<version>1.9.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.jqwik</groupId>
<artifactId>jqwik-testcontainers</artifactId>
<version>0.5.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>1.20.1</version>
<scope>test</scope>
</dependency>

As a result, we can now instantiate a ComposeContainer and pass our test docker-compose file as argument:

@Testcontainers
class ModelBasedTest {

@Container
static ComposeContainer ENV = new ComposeContainer(new File("src/test/resources/docker-compose-test.yml"))
.withExposedService("app-tested", 8080, Wait.forHttp("/api/employees").forStatusCode(200))
.withExposedService("app-model", 8080, Wait.forHttp("/api/employees").forStatusCode(200));

// tests
}

Test HTTP client

Now, let’s create a small test utility that will help us execute the HTTP requests against our services:

class TestHttpClient {
ApiResponse<EmployeeDto> get(String employeeNo) { /* … */ }

ApiResponse<Void> put(String employeeNo, String newName) { /* … */ }

ApiResponse<List<EmployeeDto>> getByDepartment(String department) { /* … */ }

ApiResponse<EmployeeDto> post(String employeeNo, String name) { /* … */ }

record ApiResponse<T>(int statusCode, @Nullable T body) { }

record EmployeeDto(String employeeNo, String name) { }
}

Additionally, in the test class, we can declare another method that helps us create TestHttpClients for the two services started by the ComposeContainer:

static TestHttpClient testClient(String service) {
int port = ENV.getServicePort(service, 8080);
String url = "http://localhost:%s/api/employees".formatted(port);
return new TestHttpClient(service, url);
}

jqwik

Jqwik is a property-based testing framework for Java that integrates with JUnit 5, automatically generating test cases to validate properties of code across diverse inputs. By using generators to create varied and random test inputs, jqwik enhances test coverage and uncovers edge cases.

If you’re new to jqwik, you can explore their API in detail by reviewing the official user guide. While this tutorial won’t cover all the specifics of the API, it’s essential to know that jqwik allows us to define a set of actions we want to test.

To begin with, we’ll use jqwik’s @Property annotation — instead of the traditional @Test — to define a test:

@Property
void regressionTest() {
TestHttpClient model = testClient("app-model");
TestHttpClient tested = testClient("app-tested");
// …
}

Next, we’ll define the actions, which are the HTTP calls to our APIs and can also include assertions.

For instance, the GetOneEmployeeAction will try to fetch a specific employee from both services and compare the responses:

record ModelVsTested(TestHttpClient model, TestHttpClient tested) {}

record GetOneEmployeeAction(String empNo) implements Action<ModelVsTested> {
@Override
public ModelVsTested run(ModelVsTested apps) {
ApiResponse<EmployeeDto> actual = apps.tested.get(empNo);
ApiResponse<EmployeeDto> expected = apps.model.get(empNo);

assertThat(actual)
.satisfies(hasStatusCode(expected.statusCode()))
.satisfies(hasBody(expected.body()));
return apps;
}
}

Additionally, we’ll need to wrap these actions within Arbitrary objects. We can think of Arbitraries as objects implementing the factory design pattern that can generate a wide variety of instances of a type, based on a set of configured rules.

For instance, the Arbitrary returned by employeeNos() can generate employee numbers by choosing a random department from the configured list and concatenating a number between 0 and 200:

static Arbitrary<String> employeeNos() {
Arbitrary<String> departments = Arbitraries.of("Frontend", "Backend", "HR", "Creative", "DevOps");
Arbitrary<Long> ids = Arbitraries.longs().between(1, 200);
return Combinators.combine(departments, ids).as("%s-%s"::formatted);
}

Similarly, getOneEmployeeAction() returns an Aribtrary action based on a given Arbitrary employee number:

static Arbitrary<GetOneEmployeeAction> getOneEmployeeAction() {
return employeeNos().map(GetOneEmployeeAction::new);
}

After declaring all the other Actions and Arbitraries, we’ll create an ActionSequence:

@Provide
Arbitrary<ActionSequence<ModelVsTested>> mbtJqwikActions() {
return Arbitraries.sequences(
Arbitraries.oneOf(
MbtJqwikActions.getOneEmployeeAction(),
MbtJqwikActions.getEmployeesByDepartmentAction(),
MbtJqwikActions.createEmployeeAction(),
MbtJqwikActions.updateEmployeeNameAction()
));
}

static Arbitrary<Action<ModelVsTested>> getOneEmployeeAction() { /* … */ }
static Arbitrary<Action<ModelVsTested>> getEmployeesByDepartmentAction() { /* … */ }
// same for the other actions

Now, we can write our test and leverage jqwik to use the provided actions to test various sequences. Let’s create the ModelVsTested tuple and use it to execute the sequence of actions against it:

@Property
void regressionTest(@ForAll("mbtJqwikActions") ActionSequence<ModelVsTested> actions) {
ModelVsTested testVsModel = new ModelVsTested(
testClient("app-model"),
testClient("app-tested")
);
actions.run(testVsModel);
}

That’s it — we can finally run the test! The test will generate a sequence of thousands of requests trying to find inconsistencies between the model and the tested service:

INFO com.etr.demo.utils.TestHttpClient — [app-tested] PUT /api/employeesFrontend-129?name=v
INFO com.etr.demo.utils.TestHttpClient — [app-model] PUT /api/employeesFrontend-129?name=v
INFO com.etr.demo.utils.TestHttpClient — [app-tested] GET /api/employees/Frontend-129
INFO com.etr.demo.utils.TestHttpClient — [app-model] GET /api/employees/Frontend-129
INFO com.etr.demo.utils.TestHttpClient — [app-tested] POST /api/employees { name=sdxToS, empNo=Frontend-91 }
INFO com.etr.demo.utils.TestHttpClient — [app-model] POST /api/employees { name=sdxToS, empNo=Frontend-91 }
INFO com.etr.demo.utils.TestHttpClient — [app-tested] PUT /api/employeesFrontend-4?name=PZbmodNLNwX
INFO com.etr.demo.utils.TestHttpClient — [app-model] PUT /api/employeesFrontend-4?name=PZbmodNLNwX
INFO com.etr.demo.utils.TestHttpClient — [app-tested] GET /api/employees/Frontend-4
INFO com.etr.demo.utils.TestHttpClient — [app-model] GET /api/employees/Frontend-4
INFO com.etr.demo.utils.TestHttpClient — [app-tested] GET /api/employees?department=ٺ⯟桸
INFO com.etr.demo.utils.TestHttpClient — [app-model] GET /api/employees?department=ٺ⯟桸
…

Catching errors

If we run the test and check the logs, we’ll quickly spot a failure. It appears that when searching for employees by department with the argument ٺ⯟桸 the model produces an internal server error, while the test version returns 200 OK:

Original Sample
—————
actions:
ActionSequence[FAILED]: 8 actions run [
UpdateEmployeeAction[empNo=Creative-13, newName=uRhplM],
CreateEmployeeAction[empNo=Backend-184, name=aGAYQ],
UpdateEmployeeAction[empNo=Backend-3, newName=aWCxzg],
UpdateEmployeeAction[empNo=Frontend-93, newName=SrJTVwMvpy],
UpdateEmployeeAction[empNo=Frontend-129, newName=v],
CreateEmployeeAction[empNo=Frontend-91, name=sdxToS],
UpdateEmployeeAction[empNo=Frontend-4, newName=PZbmodNLNwX],
GetEmployeesByDepartmentAction[department=ٺ⯟桸]
]
final currentModel: ModelVsTested[model=com.etr.demo.utils.TestHttpClient@5dc0ff7d, tested=com.etr.demo.utils.TestHttpClient@64920dc2]
Multiple Failures (1 failure)
— failure 1 —
expected: 200
but was: 500

Upon investigation, we find that the issue arises from a native SQL query using Postgres-specific syntax to retrieve data. While this was a simple issue in our small application, model-based testing can help uncover unexpected behavior that may only surface after a specific sequence of repetitive steps pushes the system into a particular state.

Wrap up

In this post, we provided hands-on examples of how model-based testing works in practice. From defining models to generating test cases, we’ve seen a powerful approach to improving test coverage and reducing manual effort. Now that you’ve seen the potential of model-based testing to enhance software quality, it’s time to dive deeper and tailor it to your own projects.

Clone the repository to experiment further, customize the models, and integrate this methodology into your testing strategy. Start building more resilient software today!

Thank you to Emanuel Trandafir for contributing this post.

Learn more

Clone the model-based testing practice repo.

Subscribe to the Docker Newsletter.

Visit the Testcontainers website.

Get started with Testcontainers Cloud by creating a free account.

Have questions? The Docker community is here to help.

New to Docker? Get started.

Quelle: https://blog.docker.com/feed/