You say that a value is not used, prove it

Introduction 

Some time, we are doing tests by considering that some values are not relevant for the test so we put default value. But some time we actually rely on those default values to have our tests working. It can lead to hidden knowledge and false passed tests

For instance, if I create a class OldWomen. For my test instances It's reasonable to put 70 as a default value. It's an age that anyone (at least any developer) will consider as old.

Let say I have a medicine dispenser to test. I can have a story like this

Given a old woman with a cancer
when medicine dispenser is turned on
then the old woman remains in life


But for any reasons we decide that Women older than 80 always forget to plug in the medicine dispenser.

...our test will continue to work 



but in fact if the old woman is older than 80, she will die. (Indeed it's a simplified world)

Your test is not relevant and can make old women died.

Improving your default values

1. We are all using continuous integration, a same test in a normal team will probably be run more than 10 times a day (Most of the time just to say "Hello I'm here and I'm working fucking well").
2. We cannot test all cases 
3. We haven't the brain enough big to foreseen all impacts of our changes (If we could, we basically won't do any test).

So let's make the machine do the job. Let's implement a default value generator !!!

By randomly generating default value you will probably failed one day (and probably the first day) If your code isn't safe for all values.

Obviously this approach has some limitation and some corner cases will not be threated.

Some ideas to implement those generators

• Create a generic class  RandomValueGenerator<T>. You will be able to handle many cases this way
• When you create String value take care of alpha numeric/ASCII characters
• Generate nullable value if it's  relevant (for example 50% of the values returned will be null ). You can create a decorator to do that 

public static <U> RandomValueGenerator<U> nullable(final RandomValueGenerator<U> randomValueGenerator) {

• Generate zero value if it's  relevant (for example 20% of the value returned will be 0)  you can also create a decorator to handle that. 
• Support enum and allow to have only a limited number of value RandomValueGenerator.fromPossibleValues(E... values) 
• Use it every where in your unit test or acceptance test (How much entity builder do you have in your code ?)
• It's difficult to find the good ranges for values (Is it usefull to generate amount that are equals to 100000000000000 $ ?). But don't assert too mu
• You can also combine some random generator :

With a method  RandomValueGenerator.combine(RandomValueGenerator... randomGenerator) 

 you can have
RandomValueGenerator.combine(
 RandomValueGenerator.fromPossibleValues(1.,0.,NaN),RandomValueGenerator.doubleValue())

Here it will provide you :

• half of the time a value that is known to be error prone
• half of the time a value that is actually random

Reproduce your tests

A good test should be reproductible. Fortunatly most of the random generator are initialize with a seed and given that seed all value are predictible. Then it's easy to have two mode :

1. By default, the seed is randomly chosen and we reinitialized the random generator at the beginning of all test to this seed
2. In order to reproduce a failing test, if a property is set with the seed, we use this seed.

Here is an exemple in Java

private final static Random random;

static {
if (System.getProperty("test.random.seed") == null) {
seed = new Random().nextLong();
} else {
seed = Long.getLong(System.getProperty("test.random.seed"));
}
random = new Random(seed);
}

public static void newTest() {
LOGGER.info("running test with random generator initialized with seed " + seed);
random.setSeed(seed);
}

You need to find a good way to call this newTest method on every tests. It's your job here !!! 

Non regression testing

At this point we tested that unexpected change in GIVEN values has no impact on THEN values but we also want to be sure that only expected values has changed on a test. Let's take again our old  example:

Given a old woman aged of 90 with a cancer
when medicine dispenser is turned on
then the old woman will die

But it's not only that the old woman will die that changed, there is also a good news, the electricity bill will not increase dramatically before people discover the old woman.

When you code do you always have in mind this kind of change ? It why non regression testing are so important. Although, I don't want to  change my test just to handle that case (It probably doesn't worth it) but I want to acknowledge that this change in behavior is ok.


There is no easy solution to solve this problem, but here some ideas :
- You need to compare state of the system at the end of the test with the expected state. It could be difficult or simple depending on your system. If your are in a db oriented project you may store the resulting database after the test.
- For the first run you cannot test anything (It's non regression testing... ) but you should store the results of this first test in the source base of the project.
- For non regression testing always use the same seeds. You can use lot of different seed but for each seed you should expect to have the same result.
- Reuse the seeds that have in the past broken your test.