Experimental: Kotlin and mutation testing

Since about a year and a half, I do a lot of presentations on Mutation Testing. In those, my point is to show that Code Coverage‘s only benefit is that it’s easy to compute but that it’s meaningless – hence Mutation Testing.

Since some time, I’ve been interested in Kotlin, a language from JetBrains that runs on the JVM. It’s only natural that I wanted to check how Mutation Testing could be applied to the Kotlin language. As Kotlin is too young to have its own dedicated mutation testing tool, I used Pit, a Java dedicated tool. I didn’t expected much, here are my findings.

I started with a Kotlin-port of my demo project. It has two simple classes and an associated test for each. In the first one, assertion is missing and in the second, the boundary condition of the < operator is not tested. This is the perfect use-case for Mutation Testing. It’s a Maven project, so the command is quite straightforward:

mvn org.pitest:pitest-maven:mutationCoverage

Interestingly enough, this works perfectly well, in terms of mutation coverage execution, but also regarding referencing the lines that are the source of the problem.

I wanted to go further, to use a real project. The Kotlin folks were kind enough to redirect me to KTor, a Kotlin-based web framework. I tried the same command, but limited on a single module – ktor-features/ktor-server-sessions (I have no clue what it does, that is not relevant anyway). Aye, there’s the rub.

First, Pit cannot correctly parse the generated bytecode at some places:

PIT >> WARNING : Found more than one mutation similar on same line in a finally block. Can't correct for inlining.

Worse, there are a lot of timeout errors. Since there’s some threading involved, that’s not really a blocker.

PIT >> WARNING : Slave exited abnormally due to TIMED_OUT

For this project, reports are really long, but some errors are really similar to what you’d expect from Java code, for example:

removed call to kotlin/jvm/internal/Intrinsics::checkParameterIsNotNull → NO_COVERAGE

Here’s a sample of the report:

Again, Pit is able to bind the real lines to the problems found. Isn’t life good? If you stop at this point, it probably is. But running Pit on another project – say, ktor-features/ktor-locations fails miserably.

The class org.jetbrains.ktor.locations.Locations$WhenMappings does not contain a source debug information. All classes must be compiled with source and line number debug information

It seems using the when construct in Kotlin generates an inner class that doesn’t contain debug information, which Pit need to works its magic. Let’s exclude the offending class and its inner mappings class:

mvn org.pitest:pitest-maven:mutationCoverage -DexcludedClasses=org.jetbrains.ktor.locations.Locations*

It works again and produces expected results:

There are no mutation testing tools for Kotlin (yet), and considering Java ecosystem’s history, there might never be one. However, mutation testing is an invaluable tool to assert the real quality of your tests. Even if Pit is not a perfect match for Kotlin, it would be foolish to discard it.