Pytest: The Best Python Testing Module

Written by Dan Sackett on September 6, 2014

In most people's minds, writing tests for applications can be a boring task.

With pytest I've found that I quite enjoy the process.

For those that have never written tests for their applications, the idea is simple. We want to check that the code we wrote returns the expected result. By doing so, we have a system in place that can keep us in check. Take for example the case where we change a block of code in one section and it breaks a feature in another section. With tests, we will know when this other section is failing and we can avoid having errors on a production server.

It's best practice to keep your application tested and honestly, it's not too hard to do.

With most tests there is typically a boilerplate that you have to build up. For instance, look at Python's unittest.

import unittest
from unnecessary_math import multiply

class TestUM(unittest.TestCase):
    def test_numbers_3_4(self):
        self.assertEqual( multiply(3,4), 12)

In this case, we see that we have to import the test module, inherit from the TestCase class, and then assertEqual() the expression. While this is not too much in some terms, it can easily get larger. With pytest, we can do it so much simpler.

from unnecessary_math import multiply

def test_numbers_3_4():
    assert( multiply(3,4) == 12 )

See the difference?

This is why pytest can call itself a "Mature full-featured Python Testing Tool". It's also why I say that setting up tests isn't too hard. It's smart in how you build and find tests. Some of the other good parts about pytest are:

Convinced? Let's install it.

Installation

It's best to first get a virtual environment setup and use pip to install pytest.

$ pip install -U pytest

Alternatively you can use easy-install:

$ easy-install -U pytest

That's all it takes!

Basic Test

Now you have access to py.test from the terminal. Let's create our first real test in a file called test_funcs.py

def add_one(x):
    """Add one to your number"""
    return x + 1

def test_add_one():
    """Test add_one works"""
    assert add_one(3) == 5

Take note that your file must be named test_*.* to be found. Now go to your terminal and type:

$ py.test
================================ test session starts ================================
platform linux2 -- Python 2.7.3 -- py-1.4.23 -- pytest-2.6.1
collected 1 items 

test_funcs.py F

===================================== FAILURES ======================================
___________________________________ test_add_one ____________________________________

    def test_add_one():
        """Test add_one works"""
>       assert add_one(3) == 5
E       assert 4 == 5
E        +  where 4 = add_one(3)

test_funcs.py:7: AssertionError
============================= 1 failed in 0.01 seconds ==============================

We didn't need to specify a path or anything. Pytest is smart and has a test discovery engine that just works. As well, you can see the result is a full traceback which gives the reason why this test fails, the line it failed on, the file it failed in, and the time it took. We can now confidently go back into this file, edit our test, and rerun it until we have passing results.

Super simple.

py.test arguments

Working with pytest gives us a bunch of handy options from the terminal and it allows you to run tests exactly how you want to.

To see all of the py.test commands, run:

$ py.test -h

py.test -x

Run this command when you want to stop the test suite after the first failure. This is useful if you are debugging tests one by one. It saves time when you still need to get things just right.

py.test --maxfail=2

Run this command when you want to stop the test suite after a set number of failures. For instance, you may not want to end the suite after one failure. In this case, we can set maxfail=3 and if three failures happen then the test suite closes.

py.test test_name.py

Run this if you only want to run a specific test. Again, this is good for getting your tests just right or working on passing a failing test.

py.test PATH/TO/FILE

Run this command to run all tests on a given path.

py.test -k STRING_EXPRESSION

Run this command when you want to find tests based on an expression. For instance we can run py.test -k "MyClass and not method". If a test_funcs.py has a class TestMyClass and methods test_something() and test_method_simple() then only the test_method_simple() test will be run due to the string expression.

py.test --showlocals or py.test -l

Run this command when you want to see the local variables in a test in the traceback.

py.test --tb=X

X can be a couple different scenarios:

py.test --pdb

Run this command when you want to debug your failed tests. For instance, if we assert that 4 == 5 then this will obviously fail. With --pdb we will enter the traceback at the assertion point and we then can test the local variables to see what went wrong and what we can expect.

py.test --durations=5

Run this command to see the 5 slowest tests after the suite has run. This is great for finding bottlenecks and cleaning them up.

py.test --pastebin-failed and py.test --pastebin=all

Run this command if you want to send the debug log to Pastebin. This will create a new paste and return the URL for you to share it.

Assertions

As noted, you can simply use the python assert statement in tests. Moreso, we can add to this. For instance:

assert 3 % 2 == 0, "Result Message"

Here we set a message to display if this test fails. This can be great for debugging and stating what you expected.

As you've seen in some examples, the traceback is quite verbose. For a great example to see how helpful pytest can be, let's try another:

def test_sets():
    """Test sets"""
    assert set('1308') == set('8035')

$ py.test
================================ test session starts ================================
platform linux2 -- Python 2.7.3 -- py-1.4.23 -- pytest-2.6.1
collected 1 items 

test_funcs.py F

===================================== FAILURES ======================================
_____________________________________ test_sets _____________________________________

    def test_sets():
        """Test sets"""
>       assert set('1308') == set('8035')
E       assert set(['0', '1', '3', '8']) == set(['0', '3', '5', '8'])
E         Extra items in the left set:
E         '1'
E         Extra items in the right set:
E         '5'

test_funcs.py:3: AssertionError
============================= 1 failed in 0.01 seconds ==============================

We get detailed feedback to the point that we know the extra items in each side. Pytest does a lot for us as you can see.

As well, we can check that a function raises an exception.

import pytest

def f():
    raise SystemExit(1)

def test_mytest():
    with pytest.raises(SystemExit):
        f()

In this example we check that we raise the exception and since it does, it passes just fine.

Conclusion

There's plenty more to cover and I will be in the coming posts. These posts include:

Until then, good luck writing basic tests!


python pytest

comments powered by Disqus