Scala, choosing a test lib

2010/09/29 | Published in

I note here, (partial) results for my testing tool research/evaluation.

Features :

  • required
    • run for maven
    • run from Eclipse via shortcut, without : Alt+Shift+X [S|T|N|any]
  • I would like
    • support single before/after for set of test (run once, eg : start/stop backend connection)
    • ability to use ScalaCheck to generate a set of test
    • nice assertion syntax (support scala type)
    • support for skip/pending test
    • to have less dependencies (jar) ass possible

I create a sample project to try test/bdd lib : prj-scala-only

To run test under eclipse, I follow the rules :

  • to be able to run test the project have to compile (fully), required
  • name the file same as test class (and package sould match directory layout)

To run test under maven : mvn test :

  • end filename with Test (or Suite => require custom configuration)
  • If testng jar is in the classpath, then surefire (the maven plugin that run test) only run testng file, so in the final project I comment TestNG dependencies and test using it

JUnit 3/4 :

  • :-) Maven :
  • :-) Eclipse : via JUnit Runner Plugin
  • :-| before/after : only on static method, not scala firendly (need to declare an object)
  • :-( scalacheck :
  • :-) assert syntax : via mix of SpecsMatchers or org.scalatest.Assertions, or ShouldMatchersForJUnit,…
  • :-( skip support : not found how to
  • my sample code :

TestNG :

  • :-) Maven :
  • :-) Eclipse : via TestNG Runner Plugin
  • :-) before/after :
  • :-( scalacheck :
  • :-) assert syntax : idem JUnit
  • :-) skip support : via beforeXxx, or annotation, or test dependency chain
  • my sample code :

Specs :

  • :-) Maven : extends SpecificationWithJUnit, or mix with JUnit (see doc, or sample linked below)
  • :-| Eclipse : I can’t run Specs via JUnit Runner Plugin, or as scala application if I follow Specs’ documentation, But works if I use @RunWith
  • :-) before/after :
  • :-) scalacheck :
  • :-) assert syntax :
  • :-) skip support :
  • my sample code :
  • documentation of the site as grow with version, may be, a little clean up should be done.

ScalaTest :

  • :-) Maven : extends JUnitSuite, or use @RunWith, or extends TestNGSuite
  • :-) Eclipse : via JUnit Runner Plugin
  • :-) before/after : via trait BeforeAndAfterAll/FixtureSuite, but not as friendly (IMO) as TestNG (failure in before => skipped test and not an error that stop every test)
  • :-) scalacheck : detect pass/fail with JUnitSuite
  • :-) assert syntax : several syntaxes
  • :-| skip support : a pending are converted into failure for JUnitSuite and TestNGSuite, but are ignored in FunSuite
  • my sample code :
  • The documentation is in the scaladoc

I stop here, my evaluation. But If you’re interested to continue, there is other solution like Specsy, assert from Scalaz,…

Conclusion :

I started this comparison because I was not able to use Specs under Eclipse, and to eval alternative. But at the end of the eval I found how to run Specs under eclipse. And I don’t like a lot the too many syntax of ScalaTest (Error when before failed). So currently, I’ll continue to use Specs.

I hope my feedback could help you.