Property Testing using QuickCheck

Pedro Vasconcelos, DCC/FCUP

April 2016

Bibliography

“QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs” Koen Claessen and John Hughes (ICFP’2000).

Web site:

http://www.cse.chalmers.se/~rjmh/QuickCheck/


Introductory tutorial:

https://www.schoolofhaskell.com/user/pbv/an-introduction-to-quickcheck-testing

What is QuickCheck?

Using QuickCheck

Example

import Test.QuickCheck

prop_rev_rev xs = reverse (reverse xs) == xs
   where types = xs::[Int]


Ask GHCi to check this with randomly generated lists:

Main> quickCheck prop_rev_rev
OK, passed 100 tests.

Counter-examples

If the property fails, then QuickCheck prints out a counter-example.

import Test.QuickCheck

prop_rev_id xs = reverse xs == xs -- WRONG!
   where types = xs::[Int]


Main> quickCheck prop_rev_id
*** Failed! Falsifiable
    (after 6 tests and 6 shrinks):
[0,1]

Writing properties

For the previous example:
$$ \text{reverse}~(\text{reverse}~ xs) = xs, \quad \forall xs :\hspace{-0.1ex}: [\alpha] $$
We choose to test for xs :: [Int]

Conditional properties

insert :: Ord a => a -> [a] -> [a]
insert x [] = [x]
insert x (y:ys)
   | x<y       = x:y:ys
   | otherwise = y:insert x ys


Let us test that insertion preserves the list ordering:
$$ \text{ascending}~xs \implies \text{ascending}~(\text{insert}~x~xs), \quad \forall x, xs $$
The operator ==> allows writing such conditional properties.

Conditional properties (2)

prop_ins_ord x xs
  = ascending xs ==> ascending (insert x xs)
  where types = x::Int

-- auxiliary definition;
-- check elements are in ascending order
ascending :: Ord a => [a] -> Bool
ascending (x:x':xs) = x<=x' && ascending (x':xs)
ascending _ = True  -- empty and singleton lists

Testing it out

> quickCheck prop_ins_ord
*Main Data.List> quickCheck prop_ins_ord 
*** Gave up! Passed only 72 tests.

What happened?

Showing test cases

We can use verboseCheck to show the actual test cases:

Main> verboseCheck prop_ins_ord
Passed:
1
[]
Skipped (precondition false):
-1
[1,-2]
....

This too verbose; it is better to aggregate statistical data on test cases.

Collecting data

prop_ins_ord x xs
  = collect (length xs) $
    ascending xs ==> ascending (insert x xs)
  where types = x::Int


Main> quickCheck prop_ins_ord
*** Gave up! Passed only 75 tests:
38% 0
29% 2
25% 1
 4% 4
 2% 3

Collecting data (2)

38% were the empty list

25% were single element lists

29% were two element lists

Only 6% were 3 or 4 elements!

Why?

Because most randomly-list generated lists of larger sizes are not ordered — the pre-condition biases test cases to “small” lists.

Quantified properties

We can restrict test cases using explicit generators:

forAll <generator> $ \pattern -> <property>

For our example we can use the orderedList generator for lists of values in ascending order (pre-defined in the library):

prop_ins_ord2 x
   = forAll orderedList $
     \xs -> ascending (insert x xs)
   where types = x :: Int

Quantified properties (2)

A custom generator ensures better distributed test cases.

prop_ins_ord2 x =
   forAll orderedList $ \xs ->
   collect (length xs) $ ascending (insert x xs)
   where types = x :: Int

Main> quickCheck prop_ins_ord2
 6% 0
 5% 1
 4% 32
 4% 22
 4% 2
...

Writing Generators

The Gen monad

Basic generators

choose :: Random a => (a,a) -> Gen a
          -- random choice from interval
elements :: [a] -> Gen a
          -- random choice from list

Examples:

sum_dice :: Gen Int
sum_dice = do d1 <- choose (1,6)
              d2 <- choose (1,6)
              return (d1+d2)

one_vowel :: Gen Char
one_vowel = elements "aeiouAEIOU"

Sampling generators

We can use sample to show random samples of a generator.

Main> sample sum_dice
3
7
8
4
8
7
5
6
2
3
6

Combinators

oneof     :: [Gen a] -> Gen a
          -- uniform random choice
frequency :: [(Int,Gen a)] -> Gen a
          -- random choice with frequency

Examples:

fruit = elements ["apple", "orange", "banana"]
sweet = elements ["cake", "ice-cream"]

-- fruit and sweets equally probable
desert = oneof [fruit, sweet] 
-- healthy option: 4/5 fruits, 1/5 sweets
healthy = frequency [(4,fruit),(1,sweet)]

Samples

Main> sample healthy
"orange"
"ice-cream"
"apple"
"banana"
"ice-cream"
"apple"
"apple"
"banana"
"orange"
"banana"
"orange"

Custom generators

We can write a custom generator using basic combinators and do-notation.

data Person = Person String Int -- name & age

genPerson :: Gen Person
genPerson =
   do name <- elements ["Alice", "Bob",
                        "David", "Carol"]
      age <- choose (10, 99)
      return (Person name age)

Samples

Main> sample genPerson 
Person "Alice" 77
Person "David" 87
Person "Alice" 67
Person "Bob" 51
Person "David" 91
Person "David" 36
Person "Alice" 47
Person "Alice" 84
Person "Carol" 36
Person "Bob" 94

Default generators

A default generator for a type is specified in the Arbitrary type class.

class Arbitrary a where
    arbitrary :: Gen a

The library defines instances for all basic and parametric types:

instance Arbitrary Int 
instance Arbitrary Bool
...
instance Arbitrary a => Arbitrary [a]
instance (Arbitrary a,
          Arbitrary b) => Arbitrary (a,b)
...

Defining a default generator

We can define an instance of Arbitrary for our new types.

data Person = Person String Int 

genPerson :: Gen Person
genPerson =
   do name <- elements ["Alice", "Bob",
                        "David", "Carol"]
      age <- choose (10, 99)
      return (Person name age)

instance Arbitrary Person where
    arbitrary = genPerson

List generators

listOf :: Gen a -> Gen [a]
   -- lists of random length

listOf1 :: Gen a -> Gen [a]
   -- non-empty lists of random length

vectorOf :: Int -> Gen a -> Gen [a]
   -- list of the given length

Example

Main> sample (vectorOf 5 $ elements "aeiou")
"ueuuo"
"aioua"
"iouui"
"iaoue"
"aeaii"
"iiuio"
"uuaoe"
"ieeue"
"ouiei"
"uoeoo"
"ueaeo"

Sized generators

Sizes of test data

   sized :: (Int -> Gen a) -> Gen a

Example: binary trees

data Tree a = Branch a (Tree a) (Tree a)
            | Leaf

Generate balanced trees

instance Arbitrary a => Arbitrary (Tree a) where
    arbitrary = sized genTree
    
-- `size' : approximate # inner nodes
genTree size
  | size>0 = do v <- arbitrary
                l <- genTree (size`div`2)
                r <- genTree (size`div`2)
                return (Branch v l r)
  | otherwise = return Leaf 

Alternative (unbalanced trees)

instance Arbitrary a => Arbitrary (Tree a) where
    arbitrary = sized genTree'

-- `size' : upper bound on # inner nodes
genTree' size
  | size>0 = frequency [(4, genBranch),
                        (1, return Leaf)]
  | otherwise = return Leaf
  where
    genBranch = do v <- arbitrary
                   l <- genTree' (size`div`2)
                   r <- genTree' (size`div`2)
                   return (Branch v l r)

Shrinking

Simplifying counter-examples

Shrinking by example

prop_wrong xs ys = reverse (xs ++ ys) ==
                   reverse xs ++ reverse ys
    where types = (xs::[Int], ys::[Int])

QuickCheck randomly generates one counter-example, e.g.: xs = [2,2] and ys = [1].

Shrinking tries to simplify the lists recursively:

This process is iterated until no “smaller” counter-example is found.

Shrinking by example

Shrinking
We reached a “local minimum” xs=[0], ys=[1] after two shrinking steps; this minimal counter-example is reported.

The shrink function

The shrinking function is defined in the Arbitrary type class.

class Arbitrary a where
   arbitrary :: Gen a -- generator
   shrink :: a -> [a] -- shrink function
   shrink _ = []      -- default: no shrinking

The shrink function should yield a (finite) list of “smaller” values to test for counter-examples.

The default implementation yields an empty list — i.e. no shrinking candidates.

Example: binary trees

data Tree a = Branch a (Tree a) (Tree a)
            | Leaf 


Heuristic to shrink a branching node:

No more shrinking when we reach a leaf.

Shrinking binary trees

instance Arbitrary a => Arbitrary (Tree a) where
   arbitrary = ...
   shrink = shrinkTree

shrinkTree :: Arbitrary a => Tree a -> [Tree a]
shrinkTree (Branch v l r) =
    [Leaf, l, r] ++
    [Branch v l' r | l'<-shrinkTree l] ++
    [Branch v l r' | r'<-shrinkTree r] ++
    [Branch v' l r | v'<-shrink v]
shrinkTree _ = []  -- no shrinking for leaves

Requirements

A larger example

Rotations

Tree rotations

rotateL, rotateR :: Tree a -> Tree a 
rotateL (Branch x t1 (Branch y t2 t3)) 
    = Branch y (Branch x t1 t2) t3
rotateL t = t   -- nothing to do

rotateR (Branch x (Branch y t1 t2) t3) 
    =  Branch x t1 (Branch y t2 t3) 
rotateR t = t   -- nothing to do

QuickCheck property

prop_rotateL_inv t
  = toList t == toList (rotateL t) 
  where types = t :: Tree Int

prop_rotateR_inv t
  = toList t == toList (rotateR t) 
  where types = t :: Tree Int

-- list tree values in order
toList :: Tree a -> [a]
toList Leaf = []
toList (Branch v l r)= toList l ++ [v] ++
                       toList r

Testing it

*Main> quickCheck prop_rotateL_inv 
+++ OK, passed 100 tests.
Main> quickCheck prop_rotateR_inv 
*** Failed! Falsifiable
    (after 3 tests and 3 shrinks):
Branch 1 (Branch 0 Leaf Leaf) Leaf


We found an error!

Tips

Using newtypes

Using newtype allows customizing the generators for specific uses.

data Expr = Var String | ...

vs.

data Expr = Var Name | ...
newtype Name = Name String

instance Arbitrary Name where
    arbitrary = do n<-elements ["a","b","c","d"]
                   return (Name n)

Newtypes in the library

newtype NonNegative a 
newtype NonZero a 
newtype Positive a
newtype NonEmptyList a
newtype OrderedList a

Allow restricting the range of generated values:

prop_add (Positive x) y = x+y > y
   where types = (x::Integer,y::Integer)

prop_len (NonEmptyList xs) = length xs > 0
   where types = xs :: [Int]