Pytest: The Best Python Testing Module
In most people's minds, writing tests for applications can be a boring task.
With pytest I've found that I quite enjoy the process.
For those that have never written tests for their applications, the idea is simple. We want to check that the code we wrote returns the expected result. By doing so, we have a system in place that can keep us in check. Take for example the case where we change a block of code in one section and it breaks a feature in another section. With tests, we will know when this other section is failing and we can avoid having errors on a production server.
It's best practice to keep your application tested and honestly, it's not too hard to do.
With most tests there is typically a boilerplate that you have to build up. For instance, look at Python's unittest.
import unittest from unnecessary_math import multiply class TestUM(unittest.TestCase): def test_numbers_3_4(self): self.assertEqual( multiply(3,4), 12)
In this case, we see that we have to import the test module, inherit from the TestCase class, and then assertEqual() the expression. While this is not too much in some terms, it can easily get larger. With pytest, we can do it so much simpler.
from unnecessary_math import multiply def test_numbers_3_4(): assert( multiply(3,4) == 12 )
See the difference?
- We don't need to import the testing module
- We don't need to inherit from a class
- We can use the standard assert statement rather than a special assertEqual() statement
This is why pytest can call itself a "Mature full-featured Python Testing Tool". It's also why I say that setting up tests isn't too hard. It's smart in how you build and find tests. Some of the other good parts about pytest are:
- Runs posix/windows, python 2.6-3.4, and PyPy
- Has tons of third party plugins
- Has no true boilerplate to setup
- Gives great traceback reporting with print debugging
- Can be used for unit tests and functional tests
- Smart test discovery
Convinced? Let's install it.
It's best to first get a virtual environment setup and use pip to install pytest.
$ pip install -U pytest
Alternatively you can use easy-install:
$ easy-install -U pytest
That's all it takes!
Now you have access to
py.test from the terminal. Let's create our first real test in a file called test_funcs.py
def add_one(x): """Add one to your number""" return x + 1 def test_add_one(): """Test add_one works""" assert add_one(3) == 5
Take note that your file must be named test_*.* to be found. Now go to your terminal and type:
$ py.test ================================ test session starts ================================ platform linux2 -- Python 2.7.3 -- py-1.4.23 -- pytest-2.6.1 collected 1 items test_funcs.py F ===================================== FAILURES ====================================== ___________________________________ test_add_one ____________________________________ def test_add_one(): """Test add_one works""" > assert add_one(3) == 5 E assert 4 == 5 E + where 4 = add_one(3) test_funcs.py:7: AssertionError ============================= 1 failed in 0.01 seconds ==============================
We didn't need to specify a path or anything. Pytest is smart and has a test discovery engine that just works. As well, you can see the result is a full traceback which gives the reason why this test fails, the line it failed on, the file it failed in, and the time it took. We can now confidently go back into this file, edit our test, and rerun it until we have passing results.
Working with pytest gives us a bunch of handy options from the terminal and it allows you to run tests exactly how you want to.
To see all of the
py.test commands, run:
$ py.test -h
Run this command when you want to stop the test suite after the first failure. This is useful if you are debugging tests one by one. It saves time when you still need to get things just right.
Run this command when you want to stop the test suite after a set number of failures. For instance, you may not want to end the suite after one failure. In this case, we can set
maxfail=3 and if three failures happen then the test suite closes.
Run this if you only want to run a specific test. Again, this is good for getting your tests just right or working on passing a failing test.
Run this command to run all tests on a given path.
py.test -k STRING_EXPRESSION
Run this command when you want to find tests based on an expression. For instance we can run
py.test -k "MyClass and not method". If a test_funcs.py has a class TestMyClass and methods test_something() and test_method_simple() then only the test_method_simple() test will be run due to the string expression.
py.test --showlocals or py.test -l
Run this command when you want to see the local variables in a test in the traceback.
X can be a couple different scenarios:
- X=long: This is the default format and shows the full tracebacks
- X=native: This is the Python standard library format
- X=short: A minified traceback. (My personal preference)
- X=line: Only show one line on a failure
Run this command when you want to debug your failed tests. For instance, if we assert that 4 == 5 then this will obviously fail. With --pdb we will enter the traceback at the assertion point and we then can test the local variables to see what went wrong and what we can expect.
Run this command to see the 5 slowest tests after the suite has run. This is great for finding bottlenecks and cleaning them up.
py.test --pastebin-failed and py.test --pastebin=all
Run this command if you want to send the debug log to Pastebin. This will create a new paste and return the URL for you to share it.
As noted, you can simply use the python
assert statement in tests. Moreso, we can add to this. For instance:
assert 3 % 2 == 0, "Result Message"
Here we set a message to display if this test fails. This can be great for debugging and stating what you expected.
As you've seen in some examples, the traceback is quite verbose. For a great example to see how helpful pytest can be, let's try another:
def test_sets(): """Test sets""" assert set('1308') == set('8035') $ py.test ================================ test session starts ================================ platform linux2 -- Python 2.7.3 -- py-1.4.23 -- pytest-2.6.1 collected 1 items test_funcs.py F ===================================== FAILURES ====================================== _____________________________________ test_sets _____________________________________ def test_sets(): """Test sets""" > assert set('1308') == set('8035') E assert set(['0', '1', '3', '8']) == set(['0', '3', '5', '8']) E Extra items in the left set: E '1' E Extra items in the right set: E '5' test_funcs.py:3: AssertionError ============================= 1 failed in 0.01 seconds ==============================
We get detailed feedback to the point that we know the extra items in each side. Pytest does a lot for us as you can see.
As well, we can check that a function raises an exception.
import pytest def f(): raise SystemExit(1) def test_mytest(): with pytest.raises(SystemExit): f()
In this example we check that we raise the exception and since it does, it passes just fine.
There's plenty more to cover and I will be in the coming posts. These posts include:
- Parameterizing Tests
- Monkeypatching and Mocking
- Doomed tests
- Config Files
Until then, good luck writing basic tests!
comments powered by Disqus