International Software Testing Contest - experience

Why I write about the contest: Since I am one of the winners of the first ISTC competition held in 2017, it is in my best interest to promote this competition in order to make famous.

The assignment description: The contest consisted of writing tests for two Java projects: Naive Bayes classifier and Prim's algorithm for calculation of minimum spanning trees. A copy of the assignment: download.

Strategy: In 2017, we were evaluated only based on the branch coverage (line coverage was ignored contrary to what was written in the invitation) and the mutation score. Only in the case of a tie, the count of test cases would have been taken into the consideration. Since no tie happened in 2017, the recommended strategy for the next years (under the assumption that the rules do not change) is simple: maximize branch coverage even at the expense of the count of test cases.

Furthermore, to maximize the mutation score you have to use asserts in your tests. My strategy was: print the result of the method into a console:
    System.out.println(someMethod());
And use the printed output in the assert:
    assertEquals("textOutput".trim(), someMethod().trim());
I used trim methods because I did not want to deal with the errors caused by wrongly copy-pasting too many/too few whitespace characters. Is it a test strategy I would use outside of the contest? No way, because the toString() format can change anytime. It may not catch all deviations. And not all objects have to implement toString(). But at ISTC 2017 it worked reasonably well.

Idea: Use a generator of unit tests like EvoSuite. However, it is critical to make sure that the generated unit tests work even in the evaluation framework because a single non-compiling unit test will result in zero mutation score. If EvoSuite does not work with the evaluation framework, consider using Randoop, which is possibly less sophisticated than EvoSuite, but generates clean unit tests without any non-standard dependency.

Mutation testing: To measure mutation score you may use PITest. If you use IDEA, a nice plugin providing integration of PITest into IDEA is Zester. Following mutators were used to calculate the mutation score:

1. Return Values Mutator
2. Negate Conditionals Mutator
3. Void Method Call Mutator
4. Conditionals Boundary Mutator
5. Increments Mutator

Warning: The branch coverage and the mutation score that you obtain from your favourite tools may not 100% agree with the scores reported by MoocTest - the tool used to evaluate these two metrics in the competition. Hence, if you can, train yourself against the framework used at the contest.

Another trickery I run into is that JDK7 was required. But because I also had JDK6 and JDK8 installed on my computer, I run into unpleasant clashes during the competition that has cost me 7 minutes of debugging. If you can, have just a single JDK installed on your computer.

Finally, the source codes were in their default packages and that was interfering with PITest (I realized only after the competition that Zester does NOT work if either the source code or the test code is in the default package). Hence, if you design an intricate plan how to win the competition, prepare also a simple fallback plan.