Property Testing using QuickCheck

Bibliography

"QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs" Koen Claessen and John Hughes (ICFP'2000).

Web site:

http://www.cse.chalmers.se/~rjmh/QuickCheck/


Introductory tutorial:

https://www.schoolofhaskell.com/user/pbv/an-introduction-to-quickcheck-testing

What is QuickCheck?

Using QuickCheck

Example

import Test.QuickCheck

prop_rev_rev xs = reverse (reverse xs) == xs
   where types = xs::[Int]


Ask GHCi to check this with randomly generated lists:

Main> quickCheck prop_rev_rev
OK, passed 100 tests.

Counter-examples

If the property fails, then QuickCheck prints out a counter-example.

import Test.QuickCheck

prop_rev_id xs = reverse xs == xs -- WRONG!
   where types = xs::[Int]


Main> quickCheck prop_rev_id
*** Failed! Falsifiable
    (after 6 tests and 6 shrinks):
[0,1]

Writing properties

For the previous example:
$$ \text{reverse}~(\text{reverse}~ xs) = xs, \quad \forall xs :\hspace{-0.1ex}: [\alpha] $$
We choose to test for xs :: [Int]

Conditional properties

insert :: Ord a => a -> [a] -> [a]
insert x [] = [x]
insert x (y:ys)
   | x<y       = x:y:ys
   | otherwise = y:insert x ys


Let us test that insertion preserves list ordering, i.e.


ascending xs ⟹ ascending (insert x xs), ∀x, xs


The operator ==> allows expressing such conditional properties.

Conditional properties (2)

prop_ins_ord x xs
  = ascending xs ==> ascending (insert x xs)
  where types = x::Int

-- auxiliary definition;
-- check values are in ascending order
ascending :: Ord a => [a] -> Bool
ascending ... 

Testing it out

> quickCheck prop_ins_ord
*Main Data.List> quickCheck prop_ins_ord 
*** Gave up! Passed only 72 tests.

What happened?

Showing test cases

We can use verboseCheck to show the actual test cases:

Main> verboseCheck prop_ins_ord
Passed:
1
[]
Skipped (precondition false):
-1
[1,-2]
....

This too verbose; it is better to aggregate statistical data on test cases.

Collecting data

prop_ins_ord x xs
  = collect (length xs) $
    ascending xs ==> ascending (insert x xs)
  where types = x::Int


Main> quickCheck prop_ins_ord
*** Gave up! Passed only 75 tests:
38% 0
29% 2
25% 1
 4% 4
 2% 3

Collecting data (2)

38% were the empty list

25% were single element lists

29% were two element lists

Only 6% were 3 or 4 elements!

Why?

Because most randomly-list generated lists of larger sizes are not ordered --- the pre-condition biases test cases to "small" lists.

Quantified properties

We can restrict test cases using explicit generators:

forAll <generator> $ \pattern -> <property>

For our example we can use the orderedList generator for lists of values in ascending order (pre-defined in the library):

prop_ins_ord2 x
   = forAll orderedList $
     \xs -> ascending (insert x xs)
   where types = x :: Int

Quantified properties (2)

A custom generator ensures better distributed test cases.

prop_ins_ord2 x =
   forAll orderedList $ \xs ->
   collect (length xs) $ ascending (insert x xs)
   where types = x :: Int

Main> quickCheck prop_ins_ord2
 6% 0
 5% 1
 4% 32
 4% 22
 4% 2
...

Writing Generators

The Gen monad

Basic generators

choose :: Random a => (a,a) -> Gen a
          -- random choice from interval
elements :: [a] -> Gen a
          -- random choice from list

Examples:

dice :: Gen Int
dice = choose (1,6)

vowel :: Gen Char
vowel = elements "aeiouAEIOU"

Combining generators

-- using do-notation
sum_dice :: Gen Int
sum_dice = do d1 <- dice
              d2 <- dice
              return (d1+d2)

-- using applicative operators
sum_dice' :: Gen Int
sum_dice' = (+) <$> dice <*> dice

Sampling generators

We can use sample to show random samples of a generator.

Main> sample sum_dice
3
7
8
4
8
7
5
6
2
3
6

Combinators

oneof     :: [Gen a] -> Gen a
          -- uniform random choice
frequency :: [(Int,Gen a)] -> Gen a
          -- random choice with frequency

Examples:

fruit = elements ["apple", "orange", "banana"]
sweet = elements ["cake", "ice-cream"]

-- fruit and sweets equally probable
desert = oneof [fruit, sweet] 
-- healthy option: 4/5 fruits, 1/5 sweets
healthy = frequency [(4,fruit),(1,sweet)]

Samples

Main> sample healthy
"orange"
"ice-cream"
"apple"
"banana"
"ice-cream"
"apple"
"apple"
"banana"
"orange"
"banana"
"orange"

Custom generators

We can write a custom generator using basic combinators and do-notation.

data Person = Person String Int 

genPerson :: Gen Person
genPerson =
   do name <- elements ["Alice", "Bob",
                        "David", "Carol"]
      age <- choose (10, 99)
      return (Person name age)

Samples

Main> sample genPerson 
Person "Alice" 77
Person "David" 87
Person "Alice" 67
Person "Bob" 51
Person "David" 91
Person "David" 36
Person "Alice" 47
Person "Alice" 84
Person "Carol" 36
Person "Bob" 94

Default generators

A default generator for a type is specified in the Arbitrary type class.

class Arbitrary a where
    arbitrary :: Gen a

The library defines instances for all basic and parametric types:

instance Arbitrary Int 
instance Arbitrary Bool
...
instance Arbitrary a => Arbitrary [a]
instance (Arbitrary a,
          Arbitrary b) => Arbitrary (a,b)
...

Defining a default generator

We can define instances of Arbitrary for new types:

instance Arbitrary Person where
    arbitrary = genPerson

genPerson :: Gen Person
genPerson = ... 

List generators

listOf :: Gen a -> Gen [a]
   -- lists of random length

listOf1 :: Gen a -> Gen [a]
   -- non-empty lists of random length

vectorOf :: Int -> Gen a -> Gen [a]
   -- list of the given length

Example

Main> sample (vectorOf 5 $ elements "aeiou")
"ueuuo"
"aioua"
"iouui"
"iaoue"
"aeaii"
"iiuio"
"uuaoe"
"ieeue"
"ouiei"
"uoeoo"
"ueaeo"

Sized generators

Sizes of test data

  sized :: (Int -> Gen a) -> Gen a

Example: binary trees

data Tree a = Branch a (Tree a) (Tree a)
            | Leaf

Generate balanced trees

instance Arbitrary a =>
         Arbitrary (Tree a) where
    arbitrary = sized genTree
    
genTree :: Arbitrary a => Int -> Gen (Tree a)
genTree size
  -- size ~ number of branch nodes
  | size>0 = do v <- arbitrary
                l <- genTree (size`div`2)
                r <- genTree (size`div`2)
                return (Branch v l r)
  | otherwise = return Leaf 

Alternative (unbalanced trees)

genTree' :: Arbitrary a => Int -> Gen (Tree a)
genTree' size
  -- `size' ~ upper bound of inner nodes
  | size>0 = frequency [(4, genBranch),
                        (1, return Leaf)]
  | otherwise = return Leaf
  where
    genBranch = do v <- arbitrary
                   l <- genTree' (size`div`2)
                   r <- genTree' (size`div`2)
                   return (Branch v l r)

Shrinking

Simplifying counter-examples

Shrinking by example

prop_wrong xs ys = reverse (xs ++ ys) ==
                   reverse xs ++ reverse ys
    where types = (xs::[Int], ys::[Int])

QuickCheck randomly generates one counter-example, e.g.: xs = [2,2] and ys = [1].

Shrinking tries to simplify the lists recursively:

This process is iterated until no "smaller" counter-example is found.

Shrinking by example

Shrinking
xs=[0], ys=[1] obtained after two shrinking steps; further shrinking does not yield counter-examples (local minimum).

The shrink function

The shrinking function is defined in the Arbitrary type class.

class Arbitrary a where
   arbitrary :: Gen a -- generator
   shrink :: a -> [a] -- shrink function
   shrink _ = []      -- default: no shrinking

shrink should give a (finite) list of "smaller" values to test for counter-examples.

The default implementation yields an empty list --- i.e. shrinking is disabled.

Example: binary trees

data Tree a = Branch a (Tree a) (Tree a)
            | Leaf 


Heuristic to shrink a branching node:

No further shrinking when we reach a leaf.

Shrinking binary trees

instance Arbitrary a =>
         Arbitrary (Tree a) where
   arbitrary = ...
   shrink = shrinkTree

shrinkTree :: Arbitrary a => Tree a -> [Tree a]
shrinkTree (Branch v l r) =
    [Leaf, l, r] ++
    [Branch v l' r | l'<-shrinkTree l] ++
    [Branch v l r' | r'<-shrinkTree r] ++
    [Branch v' l r | v'<-shrink v]
shrinkTree _ = []  -- no shrinking for leaves

Requirements

A larger example

Rotations

Tree rotations

rotateL, rotateR :: Tree a -> Tree a 
rotateL (Branch x t1 (Branch y t2 t3)) 
    = Branch y (Branch x t1 t2) t3
rotateL t = t   -- nothing to do

rotateR (Branch x (Branch y t1 t2) t3) 
    =  Branch x t1 (Branch y t2 t3) 
rotateR t = t   -- nothing to do

QuickCheck property

prop_rotateL_inv t
   = toList t == toList (rotateL t) 
   where types = t :: Tree Int

prop_rotateR_inv t
   = toList t == toList (rotateR t) 
   where types = t :: Tree Int

-- list tree values in order
toList :: Tree a -> [a]
toList Leaf = []
toList (Branch v l r)
    = toList l ++ [v] ++ toList r

Testing it

*Main> quickCheck prop_rotateL_inv 
+++ OK, passed 100 tests.
Main> quickCheck prop_rotateR_inv 
*** Failed! Falsifiable
    (after 3 tests and 3 shrinks):
Branch 1 (Branch 0 Leaf Leaf) Leaf


We found an error!

Tips

Using newtypes

Using newtype allows customizing the generators for specific uses.

data Expr = Var String | ...

vs.

data Expr = Var Name | ...
newtype Name = Name String

instance Arbitrary Name where
    arbitrary = do n<-elements ["a","b","c","d"]
                   return (Name n)

Newtypes in the library

newtype NonNegative a = NonNegative a
newtype NonZero a = NonZero a
newtype Positive a = Positive a
newtype NonEmptyList a = NonEmpty [a]
newtype OrderedList a = Ordered [a]

Allow restricting the range of generated values:

prop_add (Positive x) y = x+y > y
   where types = (x::Integer,y::Integer)

prop_len (NonEmpty xs) = length xs > 0
   where types = xs :: [Int]