Introduction
PEA is a code coverage tool to be used with JUnit. Its purpose is to help determine the quality of tests and test coverage, not just the amount of coverage.
One of the main weaknesses of existing code coverage tools is that all code that executes during a test run is considered covered, and covered equally. If a method under test executes and calls another method for which no test was specifically written, the other method looks like it was tested, because it has coverage. The situation is exacerbated the more that one method calls another. Sometimes method calls can be chained many, many levels from one test, and a method that never had a test written for it has 100% coverage.
While PEA works like code coverage tools in that the classes under test are instrumented to call a tabulating routine on method entry, which marks the method as covered, PEA is different because it examines the stack trace at the time of the call to the tabulating routine to see how many stack frames exist between it and a JUnit test case. The lower the number of stack frames, the more direct and presumably more intentional the test. Higher numbers indicate indirect testing, and very high numbers indicate accidental testing. The lowest number of stack frames encountered during a batch test run for a method results in its "test directness" score.
PEA runs as a debugger which watches the JUnit tests as they run, monitoring method entry and exit and detecting how far away from a test case each method call is.
PEA produces results in XML, which can then be turned into an HTML report with a provided XSL transform, or processed by any other program as needed. An XSL Schema for the output XML will be provided as well so the data can be bound into beans using JAXB.
PEA gets its name from the fairy tale of the princess and the pea. The princess could not sleep all night because she felt a pea that had been placed under 20 mattresses. Defects are not peas, and we developers are not princesses. We should be testing more directly, and PEA will help us find out when we are not, and help us avoid the false sense of security that tools such as EMMA, Cobertura, and JCoverage can give when coverage numbers are high.
Note: At this time there is no binary distribution and the source can only
be accessed via CVS.