Generative Testing Part 5 – The hard way
In part 4 of this discussion of "generative testing", we began transforming and assembling these generators to do much more powerful tasks. This time we will compose better sheepish-like strings in an attempt to trick the sheep-bleat? function with our unit tests.
Building examples with more control
We know sheepish has only a and b characters, but we want to try some other characters and sequences, so that we test invalid output as well. For this discussion, let's communicate using a nearly ideal metaphor, the regex-style repetition specifiers: * (0 or more), + (1 or more).
Let us say we want to compose the test string into 1: (before), 2: the b, 3: intermediate characters, 4: a characters, 5: more intermediate characters, 6: more a characters. Here is how we might compose a generator that would be matched by the regex .*b*.*a*.*a*. Of course regex matching is logical (it either matches or it does not). When generating data probabalistically, there is an additional dimension of probability distribution that we cannot express in regex. Here are the generators we'll use:
- UsuallyEmptyString:
.*– Usually empty string (0 or more of any character, but usually empty) - StringOfB:
b*– String of b characters (0 to 3) - StringOfA:
a*– String of a characters (0 to 3), usually 0. - SheepishExamples: Test case built from concatenating the values generated by
UsuallyEmptyString + StringOfB + UsuallyEmptyString + StringOfA + UsuallyEmptyStringTake + StringOfA.
Testing with clojure.test
Let's start with Clojure again because we can see the generated strings in Clojure's concise literal data representation.
;; 1. UsuallyEmptyString: `.*` – Usually empty string (0 or more of any character, but usually empty)
(s/def ::usually-empty-string
(s/with-gen string?
#(gen/frequency [[9 (gen/return "")]
[1 (s/gen string?)]])))
; (generate-examples ::usually-empty-string 20)
; yields ("" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "Uzjbma783HTsxptQa8ZkY5fDkc68" "" "" "" "" "")
;; 2. StringOfB: `b*` – String of b characters (0 to 3)
(s/def ::string-of-a
(s/with-gen string?
#(gen/fmap
(fn [n] (string/join (repeat (max n 0) \a)))
(s/gen (s/int-in -3 4)))))
;; 3. StringOfA: `a*` – String of a characters (0 to 3), usually 0.
(s/def ::string-of-b
(s/with-gen string?
#(gen/fmap
(fn [n] (string/join (repeat (max n 0) \b)))
(s/gen (s/int-in 0 3)))))
; (generate-examples ::string-of-a 20)
; yields ("aa" "" "" "aa" "" "aa" "" "aa" "" "" "" "" "" "aa" "" "aa" "aaa" "aa" "aaa" "aa" "")
;; 4. SheepishExamples: Test case built from concatenating the values generated by
;; `UsuallyEmptyString + StringOfB + UsuallyEmptyString + StringOfA + UsuallyEmptyStringTake + StringOfA`.
(s/def ::sheepish-like-substrings
(s/with-gen
(s/coll-of any?)
#(gen/tuple (s/gen ::usually-empty-string)
(s/gen ::string-of-b)
(s/gen ::usually-empty-string)
(s/gen ::string-of-a)
(s/gen ::usually-empty-string)
(s/gen ::string-of-a))))
; (generate-examples ::sheepish-like-substrings 4)
; yields (["" "b" "" "" "" "aa"] ["" "bb" "" "aa" "" "aaa"] ["" "bb" "" "aa" "" ""] ["" "b" "" "" "" "aaa"])
(s/def ::sheepish-like-string
(s/with-gen
string?
#(gen/fmap string/join (s/gen ::sheepish-like-substrings))))
; (generate-examples ::sheepish-like-string 4)
; yields ("bbaaaa" "b1oYCjf3tRz5S65oT8Ux8bpZHkxp49Ghaa" "bbaa" "baaavLEjQBNj2I2Kr4Vc6z6OLb7aaaa" "baa" "70E7l6Q27vYt6Er50Ehbbaa")
Now, let's plug it into the test framework, and test against the regex oracle, as we did in in part 2.
(deftest Testing_sheep-bleat?_with_oracle
; Run the test for each of the generated examples
(doseq [text (generate-examples ::sheepish-like-string 4)]
(is (= (some? (re-find #"^baa+$" text)) ; true when a match is found, false when nil is returned.
(sheep-bleat? text))
(str "Does the regex oracle agree about `" text "`?"))))
; (clojure.test/run-tests)
; prints:
; Testing sheepish.f-better-sheepish-examples
;
; FAIL in (Testing_sheep-bleat?_with_oracle) (form-init3243900215441859027.clj:4)
; Does the regex oracle agree about `baa`?
; expected: (= (some? (re-find #"^baa+$" text)) (sheep-bleat? text))
; actual: (not (= true false))
;
; Ran 1 tests containing 4 assertions.
; 1 failures, 0 errors.
; {:test 1, :pass 3, :fail 1, :error 0, :type :summary}
Testing with nUnit TestFixtureSource attribute (the hard way)
This example uses nUnit's [TestFixtureSource(nameof(SheepishExamples))] attribute to use examples from a property called SheepishExamples which uses FsCheck generators to build example strings.
[TestFixtureSource(nameof(SheepishExamples))]
public class F_Parameterized_Test_With_Better_Generators
{
// nUnit creates instances of the class
public F_Parameterized_Test_With_Better_Generators(SheepishTestCase testCase)
{
this.testCase = testCase;
isSheepBleat = Sheepish.IsSheepBleat(testCase.Text);
}
[Test]
public void SheepishIsAtLeast3CharactersLong()
{
if (isSheepBleat)
Assert.GreaterOrEqual(testCase.Text.Length, 3,
"Sheep bleat is at least 3 characters");
}
static Gen<string> UsuallyEmptyString =>
Gen.Frequency(
Tuple.Create(9, Gen.Constant("")),
Tuple.Create(1, Arb.Default.NonEmptyString().Generator.Select(s => s.Item)));
static Gen<string> StringOfA =>
Gen.Choose(-3, 3).Select(n => new string('a', Math.Max(n, 0)));
static Gen<string> StringOfB =>
Gen.Choose(0, 3).Select(n => new string('b', Math.Max(n, 0)));
static IEnumerable<SheepishTestCase> SheepishExamples =>
Gen.zip(
Gen.zip3(UsuallyEmptyString,
StringOfB,
UsuallyEmptyString),
Gen.zip3(StringOfA,
UsuallyEmptyString,
StringOfA))
.Select(t => t.Item1.Item1 + t.Item1.Item2 + t.Item1.Item3
+ t.Item2.Item1 + t.Item2.Item2 + t.Item2.Item3) // That's confusing!
.Select(text => new SheepishTestCase(text, "random")) // To test case
.Sample(4, 100); // take 100 examples
Here is an alternative way of generating similar data that constructs instances of SheepishParts using reflection to infer which data types to generate. FsCheck then uses either the default generators and shrinkers (together known as Arb (as in "Arb-itrary data type generator"). These may be registered by Arb.Register<SheepishPartsGenerators>();. See FsCheck for more.
[TestFixtureSource(nameof(SheepishExamples2))]
public class F_Parameterized_Test_With_Better_Generators
{
static IEnumerable<SheepishTestCase> SheepishExamples2 =>
Arb.Default.Derive<SheepishParts>().Generator
.Select(parts => parts.ToString())
.Select(text => new SheepishTestCase(text, "random")) // To test case
.Sample(4, 100); // take 100 examples
/// <summary>
/// A class to represent a "better" sheepish generation.
/// </summary>
public class SheepishParts
{
// FsCheck can call this constructor with reflection.
public SheepishParts(UsuallyEmptyString S0, StringOfBs B1, UsuallyEmptyString S2, StringOfAs A3, UsuallyEmptyString S4, StringOfAs A5)
{
this.S0 = S0;
this.B1 = B1;
this.S2 = S2;
this.A3 = A3;
this.S4 = S4;
this.A5 = A5;
}
public UsuallyEmptyString S0 { get; }
public StringOfBs B1 { get; }
public UsuallyEmptyString S2 { get; }
public StringOfAs A3 { get; }
public UsuallyEmptyString S4 { get; }
public StringOfAs A5 { get; }
public override string ToString() => "" + S0 + B1 + S2 + A3 + S4 + A5;
public class UsuallyEmptyString : Wrapper<string> { }
public class StringOfAs : Wrapper<string> { }
public class StringOfBs : Wrapper<string> { }
}
public class SheepishPartsGenerators
{
public static Arbitrary<T> FromStringGen<T>(Gen<string> gen)
where T : Wrapper<string>, new() =>
gen.Select(s => new T{ Value = s }).ToArbitrary();
public static Arbitrary<SheepishParts.UsuallyEmptyString> UsuallyEmptyString() =>
FromStringGen<SheepishParts.UsuallyEmptyString>(F_Parameterized_Test_With_Better_Generators.UsuallyEmptyString);
public static Arbitrary<SheepishParts.StringOfAs> StringOfAs() =>
FromStringGen<SheepishParts.StringOfAs>(F_Parameterized_Test_With_Better_Generators.StringOfA);
public static Arbitrary<SheepishParts.StringOfBs> StringOfBs() =>
FromStringGen<SheepishParts.StringOfBs>(F_Parameterized_Test_With_Better_Generators.StringOfB);
}
public class Wrapper<T>
{
public Wrapper() => Value = default(T);
public Wrapper(T value) => Value = value;
public T Value;
public override string ToString() => Value?.ToString();
}
static F_Parameterized_Test_With_Better_Generators()
{
try
{
Arb.Register<SheepishPartsGenerators>();
}
catch(Exception e)
{
e.ToString();
}
}
If that code looks a bit more involved, like we've jumped from "simple enough" to "complicated," that's because we did just issue a seemingly magical incantation to generate data and use reflection to create particular classes. FsCheck offers a somewhat involved way of registering generators so you can do it once and for all (for the rest of the life of the process). While this can be convenient and less repetitive, it comes with its challenges, like making sure the Arbitraries are registered before the test data is generated. In this case, I used a static constructor to accomplish this: static F_Parameterized_Test_With_Better_Generators(). In the next post, we will see a way of selecting the Arbitrary generators to use for particular tests, rather than once per process. This will be a bit less magical and make the intent perfectly clear, as we are able to override the default generators.
The trouble with doing it half-way
So here are how the tests appear in Visual Studio's test runner.
It is easy to generate a bunch of random tests, but when you try to run a specific test again, that example may not have been generated again, then nUnit can't find the test.
Next time we'll let the generative testing framework decide how to generate the examples and witness the power of this fully-operational battle station test suite.
Source code
If you want to follow along, the source code is here: