pytest-notebook by example

See also

This notebook was rendered with myst-nb: tutorial_intro.ipynb

Python API

The principal component of pytest-notebook is the NBRegressionFixture class, which is an attrs class, whose parameters can be instantiated or set via attributes.

try:
    # Python <= 3.8
    from importlib_resources import files
except ImportError:
    from importlib.resources import files
from pytest_notebook import example_nbs
from pytest_notebook.nb_regression import NBRegressionFixture

fixture = NBRegressionFixture(exec_timeout=50)
fixture.diff_color_words = False
fixture

NBRegressionFixture(exec_notebook=True, exec_cwd=None, exec_allow_errors=False, exec_timeout=50, coverage=False, cov_config=None, cov_source=None, cov_merge=None, post_processors=('coalesce_streams',), process_resources={}, diff_replace=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=False, force_regen=False)

The main method is check(), which executes a notebook and compares its initial and final contents.

with files(example_nbs).joinpath("example1.ipynb") as path:
    fixture.check(str(path))

---------------------------------------------------------------------------
NBRegressionError                         Traceback (most recent call last)
Cell In[3], line 2
      1 with files(example_nbs).joinpath("example1.ipynb") as path:
----> 2     fixture.check(str(path))

File ~/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.9/site-packages/pytest_notebook/nb_regression.py:367, in NBRegressionFixture.check(self, path, raise_errors)
    365     raise regen_exc
    366 elif filtered_diff:
--> 367     raise NBRegressionError(diff_string)
    369 return NBRegressionResult(
    370     nb_initial, nb_final, full_diff, filtered_diff, diff_string, resources
    371 )

NBRegressionError: 
--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18

To return the results, without raising an exception, use raise_errors=False. This returns a NBRegressionResult instance.

with files(example_nbs).joinpath("example1.ipynb") as path:
    result = fixture.check(str(path), raise_errors=False)

result

NBRegressionResult(diff_full_length=2,diff_filtered_length=2)

print(result.diff_string)

--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18

The diff of the notebooks is returned as a list of nbdime DiffEntry objects.

result.diff_filtered

[{'op': 'patch',
  'key': 'cells',
  'diff': [{'op': 'patch',
    'key': 1,
    'diff': [{'op': 'replace', 'key': 'execution_count', 'value': 1}]},
   {'op': 'patch',
    'key': 2,
    'diff': [{'op': 'patch',
      'key': 'outputs',
      'diff': [{'op': 'patch',
        'key': 0,
        'diff': [{'op': 'patch',
          'key': 'text',
          'diff': [{'op': 'patch',
            'key': 1,
            'diff': [{'op': 'addrange', 'key': 5, 'valuelist': '2'},
             {'op': 'removerange', 'key': 5, 'length': 1}]}]}]}]}]}]},
 {'op': 'patch',
  'key': 'metadata',
  'diff': [{'op': 'patch',
    'key': 'language_info',
    'diff': [{'op': 'patch',
      'key': 'version',
      'diff': [{'op': 'addrange', 'key': 0, 'valuelist': ['3.9.18']},
       {'op': 'removerange', 'key': 0, 'length': 1}]}]}]}]

The notebooks are returned as nbformat.NotebookNode instances.

result.nb_final

{'cells': [{'cell_type': 'markdown',
   'metadata': {},
   'source': '# Markdown Cell'},
  {'cell_type': 'code',
   'execution_count': 1,
   'metadata': {},
   'outputs': [],
   'source': 'from IPython import display'},
  {'cell_type': 'code',
   'execution_count': 2,
   'metadata': {},
   'outputs': [{'output_type': 'stream',
     'name': 'stdout',
     'text': 'hallo1\nhallo2\n'}],
   'source': "print('hallo1')\nprint('hallo2')"}],
 'metadata': {'kernelspec': {'display_name': 'Python 3',
   'language': 'python',
   'name': 'python3'},
  'language_info': {'name': 'python',
   'version': '3.9.18',
   'mimetype': 'text/x-python',
   'codemirror_mode': {'name': 'ipython', 'version': 3},
   'pygments_lexer': 'ipython3',
   'nbconvert_exporter': 'python',
   'file_extension': '.py'}},
 'nbformat': 4,
 'nbformat_minor': 2}

pytest fixture

See also

{py:mod}`pytest_notebook.ipy_magic`,
for the notebook magic used to run pytest in a Jupyter Notebook.

%load_ext pytest_notebook.ipy_magic

A NBRegressionFixture instance can accessed via the nb_regression() fixture. This instance will be instatiated with parameters dictated by arguments parsed from the pytest command-line and configuration file(s).

Note

pytest-notebook command-line and configuration file parameter names are the same as for NBRegressionFixture, but with the prefix nb_.

The command-line parameter takes precedence over the configuration file one.

%%pytest -v --color=yes --disable-warnings --nb-exec-timeout 50

---
[pytest]
nb_diff_color_words = True
---
try:
    # Python <= 3.8
    from importlib_resources import files
except ImportError:
    from importlib.resources import files
from pytest_notebook import example_nbs

def test_notebook(nb_regression):
    with files(example_nbs).joinpath("example1.ipynb") as path:
        nb_regression.check(str(path))

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmp1juo62ku
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 1 item

test_ipycell.py::test_notebook FAILED                                    [100%]

=================================== FAILURES ===================================
________________________________ test_notebook _________________________________

nb_regression = NBRegressionFixture(exec_notebook=True, exec_cwd='/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/env...place=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=True, force_regen=False)

    def test_notebook(nb_regression):
        with files(example_nbs).joinpath("example1.ipynb") as path:
>           nb_regression.check(str(path))
E           pytest_notebook.nb_regression.NBRegressionError: 
E           --- expected
E           +++ obtained
E           ## replaced /cells/1/execution_count:
E           -  2
E           +  1
E           
E           ## modified /cells/2/outputs/0/text:
E           @@ -1,2 +1,2 @@
E           hallo1
E           hallo3hallo2
E           
E           ## modified /metadata/language_info/version:
E           -  3.6.7
E           +  3.9.18
E           
E           

test_ipycell.py:11: NBRegressionError
=========================== short test summary info ============================
FAILED test_ipycell.py::test_notebook - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.77s ===============================

pytest file collection

check() can be run automatically on all notebooks using the pytest collection mechanism. To activate this feature, set --nb-test-files on the command-line, or nb_test_files = True in the configuration file.

notebook1_content = files(example_nbs).joinpath("example1_pass.ipynb").read_text()
notebook2_content = files(example_nbs).joinpath("example1.ipynb").read_text()

%%pytest -v --color=yes --disable-warnings --nb-test-files

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "test_notebook2.ipynb")
***

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpry0qo8w_
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 2 items

test_notebook1.ipynb::nbregression(test_notebook1) FAILED                [ 50%]
test_notebook2.ipynb::nbregression(test_notebook2) FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1)
FAILED test_notebook2.ipynb::nbregression(test_notebook2)
======================== 2 failed, 2 warnings in 2.67s =========================

%%pytest -v --color=yes --disable-warnings

---
[pytest]
nb_test_files = True
---

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "test_notebook2.ipynb")
***

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpbvx8tbgz
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 2 items

test_notebook1.ipynb::nbregression(test_notebook1) FAILED                [ 50%]
test_notebook2.ipynb::nbregression(test_notebook2) FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1)
FAILED test_notebook2.ipynb::nbregression(test_notebook2)
======================== 2 failed, 2 warnings in 2.68s =========================

To restrict the notebook files pytest collects, one or more filename pattern matches can also be set (see fnmatch).

%%pytest -v --color=yes --disable-warnings

---
[pytest]
nb_test_files = True
nb_file_fnmatch = test_*.ipynb tutorial_*.ipynb
---

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "other_notebook2.ipynb")
***

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpz_0imyev
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 1 item

test_notebook1.ipynb::nbregression(test_notebook1) FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1)
========================= 1 failed, 1 warning in 1.57s =========================

Live Logging of Cell Execution

If you wish to view the progress of the notebook execution, you can use the pytest live-logging functionality:

%%pytest -v --color=yes --disable-warnings --nb-test-files --log-cli-level=info

***
(notebook1_content, "test_notebook1.ipynb")
***

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmppe4ujt8a
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 1 item

test_notebook1.ipynb::nbregression(test_notebook1) 
-------------------------------- live log call ---------------------------------
INFO     pytest_notebook.execution:execution.py:97 Executing notebook with kernel: python3
FAILED                                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


------------------------------ Captured log call -------------------------------
INFO     pytest_notebook.execution:execution.py:97 Executing notebook with kernel: python3
=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1)
========================= 1 failed, 1 warning in 1.58s =========================

Regenerating Notebooks

Failing notebooks can be regenerated by setting --nb-force-regen. This will overwrite failing notebooks with the output from the notebook execution.

Note

Notebooks will not be regenerated if they raise any unexpected exceptions, during execution.

This approach to regeneration mimics pytest-regressions.

%%pytest -v --color=yes --disable-warnings --nb-force-regen

---
[pytest]
nb_test_files = True
nb_force_regen = True
---

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "test_notebook2.ipynb")
***

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
NB force regen: True
rootdir: /tmp/tmpvycup08e
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 2 items

test_notebook1.ipynb::nbregression(test_notebook1) FAILED                [ 50%]
test_notebook2.ipynb::nbregression(test_notebook2) FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmpvycup08e/test_notebook1.ipynb
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:

--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmpvycup08e/test_notebook2.ipynb
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:

--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1)
FAILED test_notebook2.ipynb::nbregression(test_notebook2)
======================== 2 failed, 2 warnings in 2.78s =========================

The regeneration can be observed, if we run two tests on the same notebook.

%%pytest -v --color=yes --disable-warnings

import os, tempfile
try:
    # Python <= 3.8
    from importlib_resources import files
except ImportError:
    from importlib.resources import files
import pytest
from pytest_notebook import example_nbs

@pytest.fixture(scope="module")
def notebook():
    tmphandle, tmppath = tempfile.mkstemp(suffix=".ipynb")
    with open(tmppath, "w") as handle:
        handle.write(
            files(example_nbs).joinpath("example1.ipynb").read_text()
        )
    yield tmppath
    os.remove(tmppath)

def test_notebook1(nb_regression, notebook):
    nb_regression.force_regen = True
    nb_regression.check(notebook)

def test_notebook2(nb_regression, notebook):
    nb_regression.check(notebook)

============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmp9gndrmqc
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 2 items

test_ipycell.py::test_notebook1 FAILED                                   [ 50%]
test_ipycell.py::test_notebook2 PASSED                                   [100%]

=================================== FAILURES ===================================
________________________________ test_notebook1 ________________________________

nb_regression = NBRegressionFixture(exec_notebook=True, exec_cwd='/tmp', exec_allow_errors=False, exec_timeout=120, coverage=False, co...place=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=False, force_regen=True)
notebook = '/tmp/tmp1j6f05y4.ipynb'

    def test_notebook1(nb_regression, notebook):
        nb_regression.force_regen = True
>       nb_regression.check(notebook)
E       pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
E       - /tmp/tmp1j6f05y4.ipynb

test_ipycell.py:23: NBRegressionError
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:

--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.9.18


=========================== short test summary info ============================
FAILED test_ipycell.py::test_notebook1 - pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-forc...
========================= 1 failed, 1 passed in 2.83s ==========================