Testing is a hard problem because there is no way to guarantee that a certain product or piece of code is 100% bug free. Many organizations have testing or “quality assurance” departments who are responsible for doing the majority of product testing before software goes to the customer. Even with a dedicated testing department, developers still have a role to play in testing their code. This article describes the developer testing philosophies used on the project I work on. The project is a server application that mainly manipulates and creates documents.
After the developer testing is complete, the build should be in a good state to enter testing by the quality assurance group. The beauty of this is that it all can happen automatically overnight.
Nightly Builds
A full build of the project is automatically run each night after the developers go home. This is not a testing method by itself but it provides a process for further testing. Developers know that they cannot check in code into version control that will not build that evening. After a new feature has been committed, all members of the team can have access to this the next morning and ensure it works properly. If required, this build can be given to others outside the development team. This is a fully working version of the product that may even go to the customer.
A report of the build is emailed to the team indicating if the build passed or failed. Each build has a unique build number that is the same as the Subversion revision number. This way the build number is unique and the developers can get that exact build code from the source code repository if needed.
Nightly Unit Testing
After the nightly build is done all of the unit tests are run for the whole project. We try to restrict the unit tests to test true “units”. That is to say, test a single class or a very small number of classes. All of the unit tests are either Java JUnit tests or Groovy unit tests. A report of the unit tests is generated using the junitreport ant task. This indicates which tests passed and failed with information on any errors.
Nightly Integration Testing
After the unit tests are complete a set of integration tests are run. These integration tests test the entire product. The integration test suite installs the product, runs the product and then performs a series of tests to ensure that the basic end-to-end functionality of the product works. On this product a bunch of test input files are processed through this running server. The outputs of these are validated to ensure everything works properly. A lightweight test harness was created in Java and Groovy to do run the integration tests and perform this validation. This framework was created from scratch rather than basing it on JUnit as these tests are specific to the application domain. A report for these tests is generated.
Performance Testing
Performance is an important part of this application. The same test harness that runs the integration tests can be used to run the performance tests. Instead of validating if the software works, the speed in which the software performs its tasks is measured. This can be compared to previous versions of the software. This is not run with every nightly build because it takes more than 24 hours to run. It is instead run on occasion over the weekend.