Configuring pytest-notebook

Configuring pytest-notebook#

See also

This notebook was rendered with myst-nb: tutorial_config.ipynb

Testing can be configured via;

  1. The pytest configuration file.

  2. The pytest command-line interface.

  3. The notebook and cell level metadata.

To see the options available for (1) and (2), pytest -h can be used (look for options starting nb), and for (3) see Notebook/Cell Metadata Schema.

To access notebook metadata, either open the notebook as a text document, or use the Jupyter Notebook interface:

Accessing notebook metadata.

import nbformat

%load_ext pytest_notebook.ipy_magic

Ignoring Elements of the Notebook#

When comparing the initial and final notebook, differences are denoted by “paths” in the notebook; a list of dictionary keys and list indices, delimited by ‘/’.

notebook = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
        nbformat.v4.new_code_cell(
            "print(1)",
            execution_count=3,
            outputs=[
                nbformat.v4.new_output(output_type="stream", name="stdout", text="2\n")
            ],
        ),
    ]
)

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmp75jao1fl
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook1.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1

## replaced /cells/1/execution_count:
-  3
+  2

## modified /cells/1/outputs/0/text:
@@ -1 +1 @@
-2
+1

## added /metadata/language_info:
+  codemirror_mode:
+    name: ipython
+    version: 3
+  file_extension: .py
+  mimetype: text/x-python
+  name: python
+  nbconvert_exporter: python
+  pygments_lexer: ipython3
+  version: 3.12.13


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.34s ===============================

We can set paths to ignore, in the configuration file, notebook metadata, or cell metadata.

Note

If the path is set in a cell metadata, it should be relative to that cell, i.e. /outputs not /cells/0/outputs

notebook.metadata["nbreg"] = {"diff_ignore": ["/cells/0/outputs/"]}
notebook.cells[1].metadata["nbreg"] = {"diff_ignore": ["/outputs/0/text"]}

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
    /cells/1/execution_count
---

***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmp3g2vetvg
configfile: pytest.ini
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook1.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.18s ===============================

Wildcard * can also be used, in place of indices, to apply to all indices in the list.

notebook2 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
        nbformat.v4.new_code_cell("print(1)", execution_count=2, outputs=[]),
    ]
)

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
    /cells/*/outputs/
---

***
(nbformat.writes(notebook2), "test_notebook2.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmph8q7b698
configfile: pytest.ini
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook2.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1

## inserted before /cells/1/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1


=========================== short test summary info ============================
FAILED test_notebook2.ipynb::nbregression(test_notebook2) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.05s ===============================

Regex Pattern Replacement#

Rather than simply ignoring cells, we can apply regex replacements to sections of the notebook, before the diff comparison is made. This is particularly useful for changeable outputs, such as dates and times:

notebook3 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            ("from datetime import date\n" "print(date.today())"),
            execution_count=1,
            outputs=[
                nbformat.v4.new_output(
                    output_type="stream", name="stdout", text="DATE-STAMP\n"
                )
            ],
        )
    ]
)

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
---

***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpey0szl3c
configfile: pytest.ini
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook3.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook3) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /cells/0/outputs/0/text:
@@ -1 +1 @@
-DATE-STAMP
+2026-07-25


=========================== short test summary info ============================
FAILED test_notebook3.ipynb::nbregression(test_notebook3) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.18s ===============================

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
nb_diff_replace =
    /cells/*/outputs/*/text \d{2,4}-\d{1,2}-\d{1,2} "DATE-STAMP"
---

***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpy45e7e85
configfile: pytest.ini
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook3.ipynb
.                                                   [100%]

============================== 1 passed in 1.05s ===============================

Ignoring Exceptions#

To mark expected exceptions from a notebook cell, tag the cell as raises-exception:

Tagging exceptions.

notebook4 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            'raise Exception("expected error")',
            execution_count=1,
            outputs=[
                nbformat.v4.new_output(
                    output_type="error",
                    ename="Exception",
                    evalue="expected error",
                    traceback=[],
                )
            ],
        )
    ]
)
notebook4.metadata["nbreg"] = {
    "diff_ignore": ["/metadata/language_info", "/cells/0/outputs/0/traceback"]
}

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpe8pqmzlp
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook4.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook4) ____________________
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
raise Exception("expected error")
------------------


---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 raise Exception("expected error")

Exception: expected error
=========================== short test summary info ============================
FAILED test_notebook4.ipynb::nbregression(test_notebook4) - nbclient.exceptions.CellExecutionError: An error occurred while executing t...
============================== 1 failed in 1.13s ===============================

notebook4.cells[0].metadata["tags"] = ["raises-exception"]

%%pytest --color=yes --disable-warnings --nb-test-files

***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpu058wi5v
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook4.ipynb
.                                                   [100%]

============================== 1 passed in 1.14s ===============================

Skipping Notebooks#

To add the pytest skip decorator to a notebook, you can add skip=True to the notebook metadata.

notebook5 = nbformat.v4.new_notebook()
notebook5.metadata["nbreg"] = {"skip": True, "skip_reason": "Not ready for testing."}
notebook5

{'nbformat': 4,
 'nbformat_minor': 5,
 'metadata': {'nbreg': {'skip': True,
   'skip_reason': 'Not ready for testing.'}},
 'cells': []}

%%pytest -v --color=yes -rs --nb-test-files

***
(nbformat.writes(notebook5), "test_notebook5.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpvsfg7ums
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collecting ... collected 1 item

test_notebook5.ipynb::nbregression(test_notebook5) SKIPPED (Not read...) [100%]


=========================== short test summary info ============================
SKIPPED [1] test_notebook5.ipynb: Not ready for testing.
============================== 1 skipped in 0.04s ==============================

A notebook can also skip itself at runtime, by raising pytest.skip within a cell, for example if a required service or API key is not available:

notebook_skip = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            "import pytest\npytest.skip('skipping from within the notebook')"
        )
    ]
)

notebook_skip

{'nbformat': 4,
 'nbformat_minor': 5,
 'metadata': {},
 'cells': [{'id': '4a4216f9',
   'cell_type': 'code',
   'metadata': {},
   'execution_count': None,
   'source': "import pytest\npytest.skip('skipping from within the notebook')",
   'outputs': []}]}

%%pytest -v --color=yes -rs --nb-test-files

***
(nbformat.writes(notebook_skip), "test_notebook_skip.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpbu01g1cn
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collecting ... collected 1 item

test_notebook_skip.ipynb::nbregression(test_notebook_skip)
SKIPPED (...) [100%]

=========================== short test summary info ============================
SKIPPED [1] ../../home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/plugin.py:423: skipping from within the notebook
============================== 1 skipped in 1.46s ==============================

Setting Environment Variables for the Kernel#

The nb_exec_env option sets environment variables for the kernel executing the notebook, in addition to the inherited environment. Each line is of the format KEY=VALUE. This can be used, for example, to add a local package to the PYTHONPATH, or to pass credentials:

[pytest]
nb_exec_env =
    PYTHONPATH=src
    MY_API_KEY=xxx

The equivalent option for NBRegressionFixture is exec_env, as a dict.

Using an nbdime Configuration File#

If you already configure nbdime ignores with an nbdime_config.json file, setting nb_diff_use_nbdime_config will also load diff-ignore paths from the Ignore mappings in its Diff, GitDiff and NbDiff sections (with later sections taking precedence per path, mirroring the nbdiff command). These are merged with the nb_diff_ignore setting, or the default diff_ignore:

[pytest]
nb_diff_use_nbdime_config = True

{
  "Diff": {
    "Ignore": {
      "/cells/*/execution_count": true,
      "/cells/*/metadata": ["collapsed", "autoscroll"],
      "/metadata": ["language_info"]
    }
  }
}

Only the first file found is used, looked up in the current working directory, then the pytest root directory (other locations searched by nbdime itself, such as the Jupyter configuration directories, are not used).

There is no equivalent NBRegressionFixture option; instead call pytest_notebook.diffing.load_nbdime_ignore_config() and pass the result to diff_ignore.

Normalizing Outputs Before Diffing#

Notebook outputs often contain insignificant differences — ANSI colour codes, timestamps, memory addresses, or whitespace-only changes in text representations — which cause spurious regression failures. Rather than hand-writing nb_diff_replace regexes for these, the nb_diff_normalize option applies named normalizers to both the stored and executed notebook, before they are diffed:

[pytest]
nb_diff_normalize =
    strip_ansi
    mask_timestamps

The built-in normalizers are:

  • strip_ansi: remove ANSI escape sequences from text outputs and tracebacks

  • mask_timestamps: replace dates (YYYY-MM-DD) and times (HH:MM:SS[.fff]) with placeholders

  • mask_memory_addresses: replace memory addresses (e.g. <object at 0x7f...>) with a placeholder

  • mask_uuids: replace UUIDs with a placeholder

  • collapse_whitespace: collapse runs of spaces/tabs and remove trailing whitespace in text outputs (useful for e.g. pandas DataFrame text representations, whose alignment can change between pandas versions)

The equivalent option for NBRegressionFixture is diff_normalize, as a tuple of names, and normalizers can also be set (for a whole notebook) in the nbreg notebook metadata:

{"nbreg": {"diff_normalize": ["strip_ansi", "mask_timestamps"]}}

When combined with nb_diff_replace, normalizers are applied first, and the regex replacements then run on the normalized notebooks; nb_diff_ignore filtering happens afterwards, on the computed diff.

Like post-processors, normalizers are registered via an entry-point group (nbreg.diff_normalize), so external packages can provide their own — each is a function taking and returning a notebook (see pytest_notebook.normalizers).

Post-processors#

Post-processors are applied to the final notebook, before diff comparison. These can be added by external packages, using the nbreg.post_proc group entry point:

# setup.py
from setuptools import setup

setup(
    name="myproject",
    packages=["myproject"],
    entry_points={
        "nbreg.post_proc": [
            "blacken_code = post_processors:blacken_code"
    ]},
)

See also

pytest_notebook.post_processors for the internally provided plugins.

Format Source Code#

An example of this is the blacken_code post-processor, which applies the black formatter to all source code cells.

This is particularly useful for re-generating notebooks.

notebook6 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            (
                "for i in range(5  ) :\n"
                "  x=i\n"
                "  a ='123'# comment\n"
                "c = ['a fairly long line of text', "
                "'another fairly long line of text',\n"
                "'yet another fairly long line of text']"
            ),
            execution_count=1,
            outputs=[],
        )
    ]
)

%%pytest --color=yes --disable-warnings --nb-test-files --nb-force-regen

---
[pytest]
nb_exec_notebook = False
nb_diff_ignore =
    /metadata/language_info
nb_post_processors =
    coalesce_streams
    blacken_code
---

***
(nbformat.writes(notebook6), "test_notebook6.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
NB post processors: coalesce_streams blacken_code
NB force regen: True
rootdir: /tmp/tmp6k113tfz
configfile: pytest.ini
plugins: pytest_notebook-0.11.0, cov-7.1.0, anyio-4.14.2
collected 1 item

test_notebook6.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook6) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmp6k113tfz/test_notebook6.ipynb
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:

--- expected
+++ obtained
## modified /cells/0/source:
@@ -1,5 +1,8 @@
-for i in range(5  ) :
-  x=i
-  a ='123'# comment
-c = ['a fairly long line of text', 'another fairly long line of text',
-'yet another fairly long line of text']

+for i in range(5):
+    x = i
+    a = "123"  # comment
+c = [
+    "a fairly long line of text",
+    "another fairly long line of text",
+    "yet another fairly long line of text",
+]



=========================== short test summary info ============================
FAILED test_notebook6.ipynb::nbregression(test_notebook6) - pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-forc...
============================== 1 failed in 0.14s ===============================

Format HTML/SVG outputs#

The beautifulsoup() post-processor may also be useful, for assessing differences in HTML and SVG outputs.

Note

This requires beautifulsoup4 to be installed.