I was very happy to see that Kotlintest, a port of the excellent scalatest in Kotlin, supports property based testing.
I was introduced to property based testing through the excellent "Functional programming in Scala" book.
The idea behind property based testing is simple - the behavior of a program is described as a property and the testing framework generates random data to validate the property. This is best illustrated with an example using the excellent scalacheck library:
scalacheck would generate a random list(of integer) of varying sizes and would validate that this property holds for the lists. A similar specification expressed through Kotlintest looks like this:
If the generators have to be a little more constrained, say if we wanted to test this behavior on lists of integer in the range 1 to 1000 then an explicit generator can be passed in the following way, again starting with scalacheck:
and an equivalent kotlintest code:
Given this let me now jump onto another example from the scalacheck site, this time to illustrate a failure:
the second property described above is wrong - if two strings are concatenated together they are ALWAYS larger than each of the parts, this is not true if one of the strings is blank. If I were to run this test using scalacheck it correctly catches this wrongly specified behavior:
An equivalent kotlintest is the following:
on running, it correctly catches the issue with concatenate and produces the following result:
However there is an issue here, scalacheck found a simpler failure case, it does this by a process called "Test Case minimization" where in case of a failure it tries to find the smallest test case that can fail, something that the Kotlintest can learn from.
There are other features where Kotlintest lags with respect to scalacheck, a big one being able to combine generators:
However all in all, I have found the DSL of Kotlintest and its support for property based testing to be a good start so far and look forward to how this library evolves over time.
If you want to play with these samples a little more, it is available in my github repo here - https://github.com/bijukunjummen/kotlintest-scalacheck-sample
I was introduced to property based testing through the excellent "Functional programming in Scala" book.
The idea behind property based testing is simple - the behavior of a program is described as a property and the testing framework generates random data to validate the property. This is best illustrated with an example using the excellent scalacheck library:
import org.scalacheck.Prop.forAll
import org.scalacheck.Properties
object ListSpecification extends Properties("List") {
property("reversing a list twice should return the list") = forAll { (a: List[Int]) =>
a.reverse.reverse == a
}
}
scalacheck would generate a random list(of integer) of varying sizes and would validate that this property holds for the lists. A similar specification expressed through Kotlintest looks like this:
import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec
class ListSpecification : StringSpec({
"reversing a list twice should return the list" {
forAll{ list: List<Int> ->
list.reversed().reversed().toList() == list
}
}
})
If the generators have to be a little more constrained, say if we wanted to test this behavior on lists of integer in the range 1 to 1000 then an explicit generator can be passed in the following way, again starting with scalacheck:
import org.scalacheck.Prop.forAll
import org.scalacheck.{Gen, Properties}
object ListSpecification extends Properties("List") {
val intList = Gen.listOf(Gen.choose(1, 1000))
property("reversing a list twice should return the list") = forAll(intList) { (a: List[Int]) =>
a.reverse.reverse == a
}
}
and an equivalent kotlintest code:
import io.kotlintest.properties.Gen
import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec
class BehaviorOfListSpecs : StringSpec({
"reversing a list twice should return the list" {
val intList = Gen.list(Gen.choose(1, 1000))
forAll(intList) { list ->
list.reversed().reversed().toList() == list
}
}
})
Given this let me now jump onto another example from the scalacheck site, this time to illustrate a failure:
import org.scalacheck.Prop.forAll
import org.scalacheck.Properties
object StringSpecification extends Properties("String") {
property("startsWith") = forAll { (a: String, b: String) =>
(a + b).startsWith(a)
}
property("concatenate") = forAll { (a: String, b: String) =>
(a + b).length > a.length && (a + b).length > b.length
}
property("substring") = forAll { (a: String, b: String, c: String) =>
(a + b + c).substring(a.length, a.length + b.length) == b
}
}
the second property described above is wrong - if two strings are concatenated together they are ALWAYS larger than each of the parts, this is not true if one of the strings is blank. If I were to run this test using scalacheck it correctly catches this wrongly specified behavior:
+ String.startsWith: OK, passed 100 tests.
! String.concatenate: Falsified after 0 passed tests.
> ARG_0: ""
> ARG_1: ""
+ String.substring: OK, passed 100 tests.
Found 1 failing properties.
An equivalent kotlintest is the following:
import io.kotlintest.properties.forAll
import io.kotlintest.specs.StringSpec
class StringSpecification : StringSpec({
"startsWith" {
forAll { a: String, b: String ->
(a + b).startsWith(a)
}
}
"concatenate" {
forAll { a: String, b: String ->
(a + b).length > a.length && (a + b).length > b.length
}
}
"substring" {
forAll { a: String, b: String, c: String ->
(a + b + c).substring(a.length, a.length + b.length) == b
}
}
})
on running, it correctly catches the issue with concatenate and produces the following result:
java.lang.AssertionError: Property failed for
Y{_DZ<vGnzLQHf9|3$i|UE,;!%8^SRF;JX%EH+<5d:p`Y7dxAd;I+J5LB/:O)
at io.kotlintest.properties.PropertyTestingKt.forAll(PropertyTesting.kt:27)
However there is an issue here, scalacheck found a simpler failure case, it does this by a process called "Test Case minimization" where in case of a failure it tries to find the smallest test case that can fail, something that the Kotlintest can learn from.
There are other features where Kotlintest lags with respect to scalacheck, a big one being able to combine generators:
case class Person(name: String, age: Int)
val genPerson = for {
name <- Gen.alphaStr
age <- Gen.choose(1, 50)
} yield Person(name, age)
genPerson.sample
However all in all, I have found the DSL of Kotlintest and its support for property based testing to be a good start so far and look forward to how this library evolves over time.
If you want to play with these samples a little more, it is available in my github repo here - https://github.com/bijukunjummen/kotlintest-scalacheck-sample