Configuring pytest-notebook#

See also

This notebook was rendered with myst-nb: tutorial_config.ipynb

Testing can be configured via;

  1. The pytest configuration file.

  2. The pytest command-line interface.

  3. The notebook and cell level metadata.

To see the options available for (1) and (2), pytest -h can be used (look for options starting nb), and for (3) see Notebook/Cell Metadata Schema.

To access notebook metadata, either open the notebook as a text document, or use the Jupyter Notebook interface:

Accessing notebook metadata.

import nbformat
%load_ext pytest_notebook.ipy_magic

Ignoring Elements of the Notebook#

When comparing the initial and final notebook, differences are denoted by “paths” in the notebook; a list of dictionary keys and list indices, delimited by ‘/’.

notebook = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
        nbformat.v4.new_code_cell(
            "print(1)",
            execution_count=3,
            outputs=[
                nbformat.v4.new_output(output_type="stream", name="stdout", text="2\n")
            ],
        ),
    ]
)
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpr7ylcyv5
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook1.ipynb F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1

## replaced /cells/1/execution_count:
-  3
+  2

## modified /cells/1/outputs/0/text:
@@ -1 +1 @@
-2
+1

## added /metadata/language_info:
+  codemirror_mode:
+    name: ipython
+    version: 3
+  file_extension: .py
+  mimetype: text/x-python
+  name: python
+  nbconvert_exporter: python
+  pygments_lexer: ipython3
+  version: 3.9.18


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1)
========================= 1 failed, 1 warning in 1.55s =========================

We can set paths to ignore, in the configuration file, notebook metadata, or cell metadata.

Note

If the path is set in a cell metadata, it should be relative to that cell, i.e. /outputs not /cells/0/outputs

notebook.metadata["nbreg"] = {"diff_ignore": ["/cells/0/outputs/"]}
notebook.cells[1].metadata["nbreg"] = {"diff_ignore": ["/outputs/0/text"]}
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
    /cells/1/execution_count
---

***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpr_w3pjjm
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook1.ipynb .                                                   [100%]

========================= 1 passed, 1 warning in 1.54s =========================

Wildcard * can also be used, in place of indices, to apply to all indices in the list.

notebook2 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
        nbformat.v4.new_code_cell("print(1)", execution_count=2, outputs=[]),
    ]
)
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
    /cells/*/outputs/
---

***
(nbformat.writes(notebook2), "test_notebook2.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpw_61z3fl
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook2.ipynb .                                                   [100%]

========================= 1 passed, 1 warning in 1.54s =========================

Regex Pattern Replacement#

Rather than simply ignoring cells, we can apply regex replacements to sections of the notebook, before the diff comparison is made. This is particularly useful for changeable outputs, such as dates and times:

notebook3 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            ("from datetime import date\n" "print(date.today())"),
            execution_count=1,
            outputs=[
                nbformat.v4.new_output(
                    output_type="stream", name="stdout", text="DATE-STAMP\n"
                )
            ],
        )
    ]
)
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
---

***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpu1qhmg7s
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook3.ipynb F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook3) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /cells/0/outputs/0/text:
@@ -1 +1 @@
-DATE-STAMP
+2023-12-01


=========================== short test summary info ============================
FAILED test_notebook3.ipynb::nbregression(test_notebook3)
========================= 1 failed, 1 warning in 1.53s =========================
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
nb_diff_replace =
    /cells/*/outputs/*/text \d{2,4}-\d{1,2}-\d{1,2} "DATE-STAMP"
---

***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpuo9il2es
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook3.ipynb .                                                   [100%]

========================= 1 passed, 1 warning in 1.63s =========================

Ignoring Exceptions#

To mark expected exceptions from a notebook cell, tag the cell as raises-exception:

Tagging exceptions.

notebook4 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            'raise Exception("expected error")',
            execution_count=1,
            outputs=[
                nbformat.v4.new_output(
                    output_type="error",
                    ename="Exception",
                    evalue="expected error",
                    traceback=[],
                )
            ],
        )
    ]
)
notebook4.metadata["nbreg"] = {
    "diff_ignore": ["/metadata/language_info", "/cells/0/outputs/0/traceback"]
}
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpi4n8zp0i
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook4.ipynb F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook4) ____________________
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
raise Exception("expected error")
------------------

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 raise Exception("expected error")

Exception: expected error
Exception: expected error
=========================== short test summary info ============================
FAILED test_notebook4.ipynb::nbregression(test_notebook4)
========================= 1 failed, 1 warning in 1.90s =========================
notebook4.cells[0].metadata["tags"] = ["raises-exception"]
%%pytest --color=yes --disable-warnings --nb-test-files

***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/tmpidwf0gvg
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook4.ipynb .                                                   [100%]

========================= 1 passed, 1 warning in 1.90s =========================

Skipping Notebooks#

To add the pytest skip decorator to a notebook, you can add skip=True to the notebook metadata.

notebook5 = nbformat.v4.new_notebook()
notebook5.metadata["nbreg"] = {"skip": True, "skip_reason": "Not ready for testing."}
notebook5
{'nbformat': 4,
 'nbformat_minor': 5,
 'metadata': {'nbreg': {'skip': True,
   'skip_reason': 'Not ready for testing.'}},
 'cells': []}
%%pytest -v --color=yes -rs --nb-test-files

***
(nbformat.writes(notebook5), "test_notebook5.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmp_1ma7y9x
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collecting ... collected 1 item

test_notebook5.ipynb::nbregression(test_notebook5) SKIPPED (Not read...) [100%]

=============================== warnings summary ===============================
../../home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.9/site-packages/pytest_notebook/plugin.py:273
  /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.9/site-packages/pytest_notebook/plugin.py:273: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to JupyterNbCollector is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return JupyterNbCollector.from_parent(parent, fspath=path)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] test_notebook5.ipynb: Not ready for testing.
======================== 1 skipped, 1 warning in 0.05s =========================

Post-processors#

Post-processors are applied to the final notebook, before diff comparison. These can be added by external packages, using the nbreg.post_proc group entry point:

# setup.py
from setuptools import setup

setup(
    name="myproject",
    packages=["myproject"],
    entry_points={
        "nbreg.post_proc": [
            "blacken_code = post_processors:blacken_code"
    ]},
)

See also

pytest_notebook.post_processors for the internally provided plugins.

Format Source Code#

An example of this is the blacken_code post-processor, which applies the black formatter to all source code cells.

This is particularly useful for re-generating notebooks.

notebook6 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            (
                "for i in range(5  ) :\n"
                "  x=i\n"
                "  a ='123'# comment\n"
                "c = ['a fairly long line of text', "
                "'another fairly long line of text',\n"
                "'yet another fairly long line of text']"
            ),
            execution_count=1,
            outputs=[],
        )
    ]
)
%%pytest --color=yes --disable-warnings --nb-test-files --nb-force-regen

---
[pytest]
nb_exec_notebook = False
nb_diff_ignore =
    /metadata/language_info
nb_post_processors =
    coalesce_streams
    blacken_code
---

***
(nbformat.writes(notebook6), "test_notebook6.ipynb")
***
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.3, pluggy-1.3.0
NB post processors: coalesce_streams blacken_code
NB force regen: True
rootdir: /tmp/tmpmen3qy9s
configfile: pytest.ini
plugins: pytest_notebook-0.10.0, anyio-4.1.0, cov-4.1.0
collected 1 item

test_notebook6.ipynb F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook6) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmpmen3qy9s/test_notebook6.ipynb
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:

--- expected
+++ obtained
## modified /cells/0/source:
@@ -1,5 +1,8 @@
-for i in range(5  ) :
-  x=i
-  a ='123'# comment
-c = ['a fairly long line of text', 'another fairly long line of text',
-'yet another fairly long line of text']

+for i in range(5):
+    x = i
+    a = "123"  # comment
+c = [
+    "a fairly long line of text",
+    "another fairly long line of text",
+    "yet another fairly long line of text",
+]



=========================== short test summary info ============================
FAILED test_notebook6.ipynb::nbregression(test_notebook6)
========================= 1 failed, 1 warning in 0.18s =========================

Format HTML/SVG outputs#

The beautifulsoup() post-processor may also be useful, for assessing differences in HTML and SVG outputs.

Note

This requires beautifulsoup4 to be installed.