Configuring pytest-notebook

Configuring pytest-notebook#

See also

This notebook was rendered with myst-nb: tutorial_config.ipynb

Testing can be configured via;

  1. The pytest configuration file.

  2. The pytest command-line interface.

  3. The notebook and cell level metadata.

To see the options available for (1) and (2), pytest -h can be used (look for options starting nb), and for (3) see Notebook/Cell Metadata Schema.

To access notebook metadata, either open the notebook as a text document, or use the Jupyter Notebook interface:

Accessing notebook metadata.

import nbformat

%load_ext pytest_notebook.ipy_magic

Ignoring Elements of the Notebook#

When comparing the initial and final notebook, differences are denoted by “paths” in the notebook; a list of dictionary keys and list indices, delimited by ‘/’.

notebook = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
        nbformat.v4.new_code_cell(
            "print(1)",
            execution_count=3,
            outputs=[
                nbformat.v4.new_output(output_type="stream", name="stdout", text="2\n")
            ],
        ),
    ]
)

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpxzi4gvfe
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook1.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1

## replaced /cells/1/execution_count:
-  3
+  2

## modified /cells/1/outputs/0/text:
@@ -1 +1 @@
-2
+1

## added /metadata/language_info:
+  codemirror_mode:
+    name: ipython
+    version: 3
+  file_extension: .py
+  mimetype: text/x-python
+  name: python
+  nbconvert_exporter: python
+  pygments_lexer: ipython3
+  version: 3.12.13


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.33s ===============================

We can set paths to ignore, in the configuration file, notebook metadata, or cell metadata.

Note

If the path is set in a cell metadata, it should be relative to that cell, i.e. /outputs not /cells/0/outputs

notebook.metadata["nbreg"] = {"diff_ignore": ["/cells/0/outputs/"]}
notebook.cells[1].metadata["nbreg"] = {"diff_ignore": ["/outputs/0/text"]}

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
    /cells/1/execution_count
---

***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmp1my2sc8h
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook1.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.33s ===============================

Wildcard * can also be used, in place of indices, to apply to all indices in the list.

notebook2 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
        nbformat.v4.new_code_cell("print(1)", execution_count=2, outputs=[]),
    ]
)

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
    /cells/*/outputs/
---

***
(nbformat.writes(notebook2), "test_notebook2.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpco2g0am2
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook2.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1

## inserted before /cells/1/outputs/0:
+  output:
+    output_type: stream
+    name: stdout
+    text:
+      1


=========================== short test summary info ============================
FAILED test_notebook2.ipynb::nbregression(test_notebook2) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.24s ===============================

Regex Pattern Replacement#

Rather than simply ignoring cells, we can apply regex replacements to sections of the notebook, before the diff comparison is made. This is particularly useful for changeable outputs, such as dates and times:

notebook3 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            ("from datetime import date\n" "print(date.today())"),
            execution_count=1,
            outputs=[
                nbformat.v4.new_output(
                    output_type="stream", name="stdout", text="DATE-STAMP\n"
                )
            ],
        )
    ]
)

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
---

***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpe5_fifas
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook3.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook3) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /cells/0/outputs/0/text:
@@ -1 +1 @@
-DATE-STAMP
+2026-07-12


=========================== short test summary info ============================
FAILED test_notebook3.ipynb::nbregression(test_notebook3) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.21s ===============================

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

---
[pytest]
nb_diff_ignore =
    /metadata/language_info
nb_diff_replace =
    /cells/*/outputs/*/text \d{2,4}-\d{1,2}-\d{1,2} "DATE-STAMP"
---

***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmpkv0ot_72
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook3.ipynb
.                                                   [100%]

============================== 1 passed in 1.35s ===============================

Ignoring Exceptions#

To mark expected exceptions from a notebook cell, tag the cell as raises-exception:

Tagging exceptions.

notebook4 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            'raise Exception("expected error")',
            execution_count=1,
            outputs=[
                nbformat.v4.new_output(
                    output_type="error",
                    ename="Exception",
                    evalue="expected error",
                    traceback=[],
                )
            ],
        )
    ]
)
notebook4.metadata["nbreg"] = {
    "diff_ignore": ["/metadata/language_info", "/cells/0/outputs/0/traceback"]
}

%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files

***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmp826uk8cs
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook4.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook4) ____________________
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
raise Exception("expected error")
------------------


---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[1], line 1
----> 1 raise Exception("expected error")

Exception: expected error
=========================== short test summary info ============================
FAILED test_notebook4.ipynb::nbregression(test_notebook4) - nbclient.exceptions.CellExecutionError: An error occurred while executing t...
============================== 1 failed in 1.34s ===============================

notebook4.cells[0].metadata["tags"] = ["raises-exception"]

%%pytest --color=yes --disable-warnings --nb-test-files

***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /tmp/tmp6rdgrbpi
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook4.ipynb
.                                                   [100%]

============================== 1 passed in 1.33s ===============================

Skipping Notebooks#

To add the pytest skip decorator to a notebook, you can add skip=True to the notebook metadata.

notebook5 = nbformat.v4.new_notebook()
notebook5.metadata["nbreg"] = {"skip": True, "skip_reason": "Not ready for testing."}
notebook5

{'nbformat': 4,
 'nbformat_minor': 5,
 'metadata': {'nbreg': {'skip': True,
   'skip_reason': 'Not ready for testing.'}},
 'cells': []}

%%pytest -v --color=yes -rs --nb-test-files

***
(nbformat.writes(notebook5), "test_notebook5.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpfckeg38u
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 1 item

test_notebook5.ipynb::nbregression(test_notebook5) SKIPPED (Not read...) [100%]

=========================== short test summary info ============================
SKIPPED [1] test_notebook5.ipynb: Not ready for testing.
============================== 1 skipped in 0.05s ==============================

A notebook can also skip itself at runtime, by raising pytest.skip within a cell, for example if a required service or API key is not available:

notebook_skip = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            "import pytest\npytest.skip('skipping from within the notebook')"
        )
    ]
)

notebook_skip

{'nbformat': 4,
 'nbformat_minor': 5,
 'metadata': {},
 'cells': [{'id': '4b209627',
   'cell_type': 'code',
   'metadata': {},
   'execution_count': None,
   'source': "import pytest\npytest.skip('skipping from within the notebook')",
   'outputs': []}]}

%%pytest -v --color=yes -rs --nb-test-files

***
(nbformat.writes(notebook_skip), "test_notebook_skip.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmp2q_y58cb
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 1 item

test_notebook_skip.ipynb::nbregression(test_notebook_skip)
SKIPPED (...) [100%]

=========================== short test summary info ============================
SKIPPED [1] ../../home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/plugin.py:413: skipping from within the notebook
============================== 1 skipped in 1.53s ==============================

Setting Environment Variables for the Kernel#

The nb_exec_env option sets environment variables for the kernel executing the notebook, in addition to the inherited environment. Each line is of the format KEY=VALUE. This can be used, for example, to add a local package to the PYTHONPATH, or to pass credentials:

[pytest]
nb_exec_env =
    PYTHONPATH=src
    MY_API_KEY=xxx

The equivalent option for NBRegressionFixture is exec_env, as a dict.

Using an nbdime Configuration File#

If you already configure nbdime ignores with an nbdime_config.json file, setting nb_diff_use_nbdime_config will also load diff-ignore paths from the Ignore mappings in its Diff, GitDiff and NbDiff sections (with later sections taking precedence per path, mirroring the nbdiff command). These are merged with the nb_diff_ignore setting, or the default diff_ignore:

[pytest]
nb_diff_use_nbdime_config = True

{
  "Diff": {
    "Ignore": {
      "/cells/*/execution_count": true,
      "/cells/*/metadata": ["collapsed", "autoscroll"],
      "/metadata": ["language_info"]
    }
  }
}

Only the first file found is used, looked up in the current working directory, then the pytest root directory (other locations searched by nbdime itself, such as the Jupyter configuration directories, are not used).

There is no equivalent NBRegressionFixture option; instead call pytest_notebook.diffing.load_nbdime_ignore_config() and pass the result to diff_ignore.

Post-processors#

Post-processors are applied to the final notebook, before diff comparison. These can be added by external packages, using the nbreg.post_proc group entry point:

# setup.py
from setuptools import setup

setup(
    name="myproject",
    packages=["myproject"],
    entry_points={
        "nbreg.post_proc": [
            "blacken_code = post_processors:blacken_code"
    ]},
)

See also

pytest_notebook.post_processors for the internally provided plugins.

Format Source Code#

An example of this is the blacken_code post-processor, which applies the black formatter to all source code cells.

This is particularly useful for re-generating notebooks.

notebook6 = nbformat.v4.new_notebook(
    cells=[
        nbformat.v4.new_code_cell(
            (
                "for i in range(5  ) :\n"
                "  x=i\n"
                "  a ='123'# comment\n"
                "c = ['a fairly long line of text', "
                "'another fairly long line of text',\n"
                "'yet another fairly long line of text']"
            ),
            execution_count=1,
            outputs=[],
        )
    ]
)

%%pytest --color=yes --disable-warnings --nb-test-files --nb-force-regen

---
[pytest]
nb_exec_notebook = False
nb_diff_ignore =
    /metadata/language_info
nb_post_processors =
    coalesce_streams
    blacken_code
---

***
(nbformat.writes(notebook6), "test_notebook6.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
NB post processors: coalesce_streams blacken_code
NB force regen: True
rootdir: /tmp/tmprmlcgh4m
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collected 1 item

test_notebook6.ipynb
F                                                   [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook6) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmprmlcgh4m/test_notebook6.ipynb
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:

--- expected
+++ obtained
## modified /cells/0/source:
@@ -1,5 +1,8 @@
-for i in range(5  ) :
-  x=i
-  a ='123'# comment
-c = ['a fairly long line of text', 'another fairly long line of text',
-'yet another fairly long line of text']

+for i in range(5):
+    x = i
+    a = "123"  # comment
+c = [
+    "a fairly long line of text",
+    "another fairly long line of text",
+    "yet another fairly long line of text",
+]



=========================== short test summary info ============================
FAILED test_notebook6.ipynb::nbregression(test_notebook6) - pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-forc...
============================== 1 failed in 0.17s ===============================

Format HTML/SVG outputs#

The beautifulsoup() post-processor may also be useful, for assessing differences in HTML and SVG outputs.

Note

This requires beautifulsoup4 to be installed.