Python Test Cases and Test Suites

Python Test Cases and Test Suites

Unit testing revolves around the test case, which is the smallest building block of testable code for any circumstances that you’re testing. When you’re using PyUnit, a test case is a simple object with at least one test method that runs code; and when it’s done, it then compares the results of the test against various assertions that you’ve made about the results.

NOTE: PyUnit is the name of the package as named by its authors, but the module you import is called the more generic-sounding name unittest.

Each test case is subclassed from the TestCase class, which is a good, memorable name for it. The simplest test cases you can write just override the runTest method of TestCase and enable you to define a basic test, but you can also define several different test methods within a single test case class, which can enable you to define things that are common to a number of tests, such as setup and cleanup procedures.

A series of test cases run together for a particular project is called a test suite. You can find some simple tools for organizing test suites, but they all share the concept of running a bunch of test cases together and recording what passed, what failed, and how, so you can know where you stand.

Because the simplest possible test suite consists of exactly one test case, and you’ve already had the simplest possible test case described to you, in the following Try It Out you write a quick testing example so you can see how all this fits together. In addition, just so you really don’t have anything to distract you, you test arithmetic, which has no external requirements on the system, the file system, or, really, anything.

Try It Out Testing Addition

1. Use your favorite editor to create a file named test1.py in a directory named ch12. Using your programming editor, edit your file to have the following code:

import unittest

 

class ArithTest (unittest.TestCase):

    def runTest (self):

        """ Test addition and succeed. """

        self.failUnless (1+1==2, ‘one plus one fails!’)

        self.failIf (1+1 != 2, ‘one plus one fails again!’)

        self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

 

def suite():

    suite = unittest.TestSuite()

    suite.addTest (ArithTest())

    return suite

 

 

if __name__ == ‘__main__’:

    runner = unittest.TextTestRunner()

    test_suite = suite()

    runner.run (test_suite)

2. Now run the code using python:

.

———————————————————————-

Ran 1 tests in 0.026s

 

How It Works

In step 1, after you’ve imported unittest (the module that contains the PyUnit framework), you define the class ArithTest, which is a subclass of the class from unittest, TestCase. ArithTest has only defined the runTest method, which performs the actual testing. Note how the runTest method has its docstring defined. It is at least as important to document your tests as it is to document your code. Lastly, a series of three assertions takes place in runTest.

TestCase classes beginning with fail, such as failUnless, failIf, and failUnlessEqual, come in additional varieties to simplify setting up the conditions for your tests. When you’re programming, you’ll likely find yourself resistant to writing tests (they can be very distracting; sometimes they are boring; and they are rarely something other people notice, which makes it harder to motivate yourself to write them). PyUnit tries to make things as easy as possible for you.

After the unit test is defined in ArithTest, you may like to define the suite itself in a callable function, as recommended by the PyUnit developer, Steve Purcell, in the modules documentation. This enables you to simply define what you’re doing (testing) and where (in the function you name). Therefore, after the definition of ArithTest, you have created the suite function, which simply instantiates a vanilla, unmodified test suite. It adds your single unit test to it and returns it. Keep in mind that the suite function only invokes the TestCase class in order to make an object that can be returned. The actual test is performed by the returned TestCase object.

As you learned in Chapter 6, only when this is being run as the main program will Python invoke the TextTestRunner class to create the runner object. The runner object has a method called run that expects to have an object of the unittests.TestSuite class. The suite function creates one such object, so test_suite is assigned a reference to the TestSuite object. When that’s finished, the runner.run method is called, which uses the suite in test_suite to test the unit tests defined in test_suite.

The actual output in this case is dull, but in that good way you’ll learn to appreciate because it means everything has succeeded. The single period tells you that it has successfully run one unit test. If, instead of the period, you see an F, it means that a test has failed. In either case, PyUnit finishes off a run with a report. Note that arithmetic is run very, very fast.

Now, see what failure looks like.

Try It Out Testing Faulty Addition

1. Use your favorite text editor to add a second set of tests to test1.py. These will be based on the first example. Add the following to your file:

class ArithTestFail (unittest.TestCase):

    def runTest (self):

        """ Test addition and fail. """

        self.failUnless (1+1==2, ‘one plus one fails!’)

        self.failIf (1+1 != 2, ‘one plus one fails again!’)

        self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

        self.failIfEqual (1+1, 2, ‘expected failure here’)

        self.failIfEqual (1+1, 2, ‘second failure’)

 

def suite_2():

    suite = unittest.TestSuite()

    suite.addTest (ArithTest())

    suite.addTest (ArithTestFail())

    return suite

You also need to change the if statement that sets off the tests, and you need to make sure that it appears at the end of your file so that it can see both classes:

if __name__ == ‘__main__’:

    runner = unittest.TextTestRunner()

    test_suite = suite_2()

    runner.run (test_suite)

2. Now run the newly modified file (after you’ve saved it). You’ll get a very different result with the second set of tests. In fact, it’ll be very different from the prior test:

.F

======================================================================

FAIL: Test addition and fail.

———————————————————————-

Traceback (most recent call last):

  File "C:Python31ch12test1.py", line 22, in runTest

    self.failIfEqual(1+1,2, ‘expected failure here’)

AssertionError: expected failure here

 

———————————————————————-

Ran 2 tests in 0.062s

 

FAILED (failures=1)

>>>

How It Works

Here, you’ve kept your successful test from the first example and added a second test that you know will fail. The result is that you now have a period from the first test, followed by an “F” for “Failed” from the second test, all in the first line of output from the test run.

After the tests are run, the results report is printed out so you can examine exactly what happened. The successful test still produces no output at all in the report, which makes sense: Imagine you have a hundred tests but only two fail—you would have to slog through a lot more output to find the failures than you do this way. It may seem like looking on the negative side of things, but you’ll get used to it.

Because there was a failed test, the stack trace from the failed test is displayed. In addition, a couple of different messages result from the runTest method. The first thing you should look at is the FAIL message. It actually uses the docstring from your runTest method and prints it at the top, so you can reference the test that failed. Therefore, the first lesson to take away from this is that you should document your tests in the docstring! Second, you’ll notice that the message you specified in the runTest for the specific test that failed is displayed along with the exception that PyUnit generated.

The report wraps up by listing the number of test cases actually run and a count of the failed test cases.

Test Fixtures

Well, this is all well and good, but real-world tests usually involve some work to set up your tests before they’re run (creating files, creating an appropriate directory structure, generally making sure everything is in shape, and other things that may need to be done to ensure that the right things are being tested). In addition, cleanup also often needs to be done at the end of your tests.

In PyUnit, the environment in which a test case runs is called the test fixture, and the base TestCase class defines two methods: setUp, which is called before a test is run, and tearDown, which is called after the test case has completed. These are present to deal with anything involved in creating or cleaning up the test fixture.

WARNING: You should know that if setUp fails, tearDown isn’t called. However, tearDown is called even if the test case itself fails.

Remember that when you set up tests, the initial state of each test shouldn’t rely on a prior test having succeeded or failed. Each test case should create a pristine test fixture for itself. If you don’t ensure this, you’re going to get inconsistent test results that will only make your life more difficult.

To save time when you run similar tests repeatedly on an identically configured test fixture, subclass the TestCase class to define the setup and cleanup methods. This will give you a single class that you can use as a starting point. Once you’ve done that, subclass your class to define each test case. You can alternatively define several test case methods within your unit case class, and then instantiate test case objects for each method. Both of these are demonstrated in the next example.

Try It Out Working with Test

1. Use your favorite text editor to add a new file test2.py. Make it look like the following example. Note that this example builds on the previous examples. 

import unittest

class ArithTestSuper (unittest.TestCase):

    def setUp (self):

        print("Setting up ArithTest cases")

    def tearDown (self):

        print("Cleaning up ArithTest cases")

class ArithTest (ArithTestSuper):

    def runTest (self):

        """ Test addition and succeed. """

        print("Running ArithTest")

        self.failUnless (1+1==2, ‘one plus one fails!’)

        self.failIf (1+1 != 2, ‘one plus one fails again!’)

        self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

 

class ArithTestFail (ArithTestSuper):

    def runTest (self):

        """ Test addition and fail. """

        print("Running ArithTestFail")

        self.failUnless (1+1==2, ‘one plus one fails!’)

        self.failIf (1+1 != 2, ‘one plus one fails again!’)

        self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

        self.failIfEqual (1+1, 2, ‘expected failure here’)

        self.failIfEqual (1+1, 2, ‘second failure’)

 

class ArithTest2 (unittest.TestCase):

    def setUp (self):

        print("Setting up ArithTest2 cases")

    def tearDown (self):

        print("Cleaning up ArithTest2 cases")

    def runArithTest (self):

        """ Test addition and succeed, in one class. """

        print("Running ArithTest in ArithTest2")

        self.failUnless (1+1==2, ‘one plus one fails!’)

        self.failIf (1+1 != 2, ‘one plus one fails again!’)

        self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

 

    def runArithTestFail (self):

        """ Test addition and fail, in one class. """

        print("Running ArithTestFail in ArithTest2")

        self.failUnless (1+1==2, ‘one plus one fails!’)

        self.failIf (1+1 != 2, ‘one plus one fails again!’)

        self.failUnlessEqual (1+1, 2, ‘more trouble with one plus one!’)

        self.failIfEqual (1+1, 2, ‘expected failure here’)

        self.failIfEqual (1+1, 2, ‘second failure’)

 

def suite():

    suite = unittest.TestSuite()

    # First style:

    suite.addTest (ArithTest())

    suite.addTest (ArithTestFail())

    # Second style:

    suite.addTest (ArithTest2("runArithTest"))

    suite.addTest (ArithTest2("runArithTestFail"))

 

    return suite

if __name__ == ‘__main__’:

    runner = unittest.TextTestRunner()

    test_suite = suite()

    runner.run (test_suite)

2. Run the code:

Setting up ArithTest cases

Running ArithTest

Cleaning up ArithTest cases

.Setting up ArithTest cases

Running ArithTestFail

FCleaning up ArithTest cases

Setting up ArithTest2 cases

Running ArithTest in ArithTest2

Cleaning up ArithTest2 cases

.Setting up ArithTest2 cases

Running ArithTestFail in ArithTest2

FCleaning up ArithTest2 cases

 

======================================================================

FAIL: Test addition and fail.

———————————————————————-

Traceback (most recent call last):

  File "C:/Python31/test2.py", line 25, in runTest

    self.failIfEqual (1+1, 2, ‘expected failure here’)

AssertionError: expected failure here

 

======================================================================

FAIL: Test addition and fail, in one class.

———————————————————————-

Traceback (most recent call last):

  File "C:/Python31/test2.py", line 48, in runArithTestFail

    self.failIfEqual (1+1, 2, ‘expected failure here’)

AssertionError: expected failure here

 

———————————————————————-

Ran 4 tests in 0.396s

 

FAILED (failures=2)

>>>

How It Works

Take a look at this code before moving along. The first thing to note about this is that you’re doing the same tests as before. One test is made to succeed and the other one is made to fail, but you’re doing two sets, each of which implements multiple unit test cases with a test fixture, but in two different styles.

Which style you use is completely up to you; it really depends on what you consider readable and maintainable.

The first set of classes in the code (ArithTestSuper, ArithTest, and ArithTestFail) are essentially the same tests as shown in the second set of examples in test1.py, but this time a class has been created called ArithTestSuper. ArithTestSuper implements a setUp and tearDown method. They don’t do much but they do demonstrate where you’d put in your own conditions. Each of the unit test classes are subclassed from your new ArithTestSuper class, so now they will perform the same setup of the test fixture. If you needed to make a change to the test fixture, you can now modify it in ArithTestSuper’s classes, and have it take effect in all of its subclasses.

The actual test cases, ArithTest and ArithTestFail, are the same as in the previous example, except that you’ve added print calls to them as well.

The final test case class, ArithTest2, does exactly the same thing as the prior three classes that you’ve already defined. The only difference is that it combines the test fixture methods with the test case methods, and it doesn’t override runTest. Instead ArithTest2 defines two test case methods: runArithTest and runArithTestFail. These are then invoked explicitly when you created test case instances during the test run, as you can see from the changed definition of suite.

Once this is actually run, you can see one change immediately: Because your setup, test, and cleanup functions all write to stdout, you can see the order in which everything is called. Note that the cleanup functions are indeed called even after a failed test. Finally, note that the tracebacks for the failed tests have been gathered up and displayed together at the end of the report. 

This article is excerpted from chapter 12 "Testing" of the book "Beginning Python: Using Python 2.6 and Python 3.1" by  James Payne (ISBN: 978-0-470-41463-7, Wrox, 2010, Copyright Wiley Publishing Inc.)

Tags:

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *