pytest-notebook by example

See also

This notebook was rendered with myst-nb: tutorial_intro.ipynb

Python API

The principal component of pytest-notebook is the NBRegressionFixture class, which is an attrs class, whose parameters can be instantiated or set via attributes.

try:
    # Python <= 3.8
    from importlib_resources import files
except ImportError:
    from importlib.resources import files
from pytest_notebook import example_nbs
from pytest_notebook.nb_regression import NBRegressionFixture

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028

fixture = NBRegressionFixture(exec_timeout=50)
fixture.diff_color_words = False
fixture

NBRegressionFixture(exec_notebook=True, exec_cwd=None, exec_allow_errors=False, exec_timeout=50, exec_env=None, coverage=False, cov_config=None, cov_source=None, cov_merge=None, post_processors=('coalesce_streams',), process_resources={}, diff_replace=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=False, force_regen=False)

The main method is check(), which executes a notebook and compares its initial and final contents.

with files(example_nbs).joinpath("example1.ipynb") as path:
    fixture.check(str(path))

/tmp/ipykernel_967/888726454.py:1: DeprecationWarning: pathlib.Path.__enter__() is deprecated and scheduled for removal in Python 3.13; Path objects as a context manager is a no-op
  with files(example_nbs).joinpath("example1.ipynb") as path:
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.

---------------------------------------------------------------------------
NBRegressionError                         Traceback (most recent call last)
Cell In[3], line 2
      1 with files(example_nbs).joinpath("example1.ipynb") as path:
----> 2     fixture.check(str(path))

File ~/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/nb_regression.py:385, in NBRegressionFixture.check(self, path, raise_errors)
    383     raise regen_exc
    384 elif filtered_diff:
--> 385     raise NBRegressionError(diff_string)
    387 return NBRegressionResult(
    388     nb_initial, nb_final, full_diff, filtered_diff, diff_string, resources
    389 )

NBRegressionError: 
--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13

To return the results, without raising an exception, use raise_errors=False. This returns a NBRegressionResult instance.

with files(example_nbs).joinpath("example1.ipynb") as path:
    result = fixture.check(str(path), raise_errors=False)

result

/tmp/ipykernel_967/278104123.py:1: DeprecationWarning: pathlib.Path.__enter__() is deprecated and scheduled for removal in Python 3.13; Path objects as a context manager is a no-op
  with files(example_nbs).joinpath("example1.ipynb") as path:
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.

NBRegressionResult(diff_full_length=2,diff_filtered_length=2)

print(result.diff_string)

--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13

The diff of the notebooks is returned as a list of nbdime DiffEntry objects.

result.diff_filtered

[{'op': 'patch',
  'key': 'cells',
  'diff': [{'op': 'patch',
    'key': 1,
    'diff': [{'op': 'replace', 'key': 'execution_count', 'value': 1}]},
   {'op': 'patch',
    'key': 2,
    'diff': [{'op': 'patch',
      'key': 'outputs',
      'diff': [{'op': 'patch',
        'key': 0,
        'diff': [{'op': 'patch',
          'key': 'text',
          'diff': [{'op': 'patch',
            'key': 1,
            'diff': [{'op': 'addrange', 'key': 5, 'valuelist': '2'},
             {'op': 'removerange', 'key': 5, 'length': 1}]}]}]}]}]}]},
 {'op': 'patch',
  'key': 'metadata',
  'diff': [{'op': 'patch',
    'key': 'language_info',
    'diff': [{'op': 'patch',
      'key': 'version',
      'diff': [{'op': 'addrange', 'key': 0, 'valuelist': ['3.12.13']},
       {'op': 'removerange', 'key': 0, 'length': 1}]}]}]}]

The notebooks are returned as nbformat.NotebookNode instances.

result.nb_final

{'cells': [{'cell_type': 'markdown',
   'metadata': {},
   'source': '# Markdown Cell'},
  {'cell_type': 'code',
   'execution_count': 1,
   'metadata': {},
   'outputs': [],
   'source': 'from IPython import display'},
  {'cell_type': 'code',
   'execution_count': 2,
   'metadata': {},
   'outputs': [{'output_type': 'stream',
     'name': 'stdout',
     'text': 'hallo1\nhallo2\n'}],
   'source': "print('hallo1')\nprint('hallo2')"}],
 'metadata': {'kernelspec': {'display_name': 'Python 3',
   'language': 'python',
   'name': 'python3'},
  'language_info': {'name': 'python',
   'version': '3.12.13',
   'mimetype': 'text/x-python',
   'codemirror_mode': {'name': 'ipython', 'version': 3},
   'pygments_lexer': 'ipython3',
   'nbconvert_exporter': 'python',
   'file_extension': '.py'}},
 'nbformat': 4,
 'nbformat_minor': 2}

pytest fixture

See also

{py:mod}`pytest_notebook.ipy_magic`,
for the notebook magic used to run pytest in a Jupyter Notebook.

%load_ext pytest_notebook.ipy_magic

A NBRegressionFixture instance can accessed via the nb_regression() fixture. This instance will be instatiated with parameters dictated by arguments parsed from the pytest command-line and configuration file(s).

Note

pytest-notebook command-line and configuration file parameter names are the same as for NBRegressionFixture, but with the prefix nb_.

The command-line parameter takes precedence over the configuration file one.

%%pytest -v --color=yes --disable-warnings --nb-exec-timeout 50

---
[pytest]
nb_diff_color_words = True
---
try:
    # Python <= 3.8
    from importlib_resources import files
except ImportError:
    from importlib.resources import files
from pytest_notebook import example_nbs

def test_notebook(nb_regression):
    with files(example_nbs).joinpath("example1.ipynb") as path:
        nb_regression.check(str(path))

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpeso_ui3b
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 1 item

test_ipycell.py::test_notebook
FAILED                                    [100%]

=================================== FAILURES ===================================
________________________________ test_notebook _________________________________

nb_regression = NBRegressionFixture(exec_notebook=True, exec_cwd=None, exec_allow_errors=False, exec_timeout=50, exec_env=None, covera...place=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=True, force_regen=False)

    def test_notebook(nb_regression):
        with files(example_nbs).joinpath("example1.ipynb") as path:
>           nb_regression.check(str(path))
E           pytest_notebook.nb_regression.NBRegressionError: 
E           --- expected
E           +++ obtained
E           ## replaced /cells/1/execution_count:
E           -  2
E           +  1
E           
E           ## modified /cells/2/outputs/0/text:
E           @@ -1,2 +1,2 @@
E           hallo1
E           hallo3hallo2
E           
E           ## modified /metadata/language_info/version:
E           -  3.6.7
E           +  3.12.13
E           
E           

test_ipycell.py:11: NBRegressionError
----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
=========================== short test summary info ============================
FAILED test_ipycell.py::test_notebook - pytest_notebook.nb_regression.NBRegressionError: 
========================= 1 failed, 1 warning in 1.47s =========================

pytest file collection

check() can be run automatically on all notebooks using the pytest collection mechanism. To activate this feature, set --nb-test-files on the command-line, or nb_test_files = True in the configuration file.

notebook1_content = files(example_nbs).joinpath("example1_pass.ipynb").read_text()
notebook2_content = files(example_nbs).joinpath("example1.ipynb").read_text()

%%pytest -v --color=yes --disable-warnings --nb-test-files

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "test_notebook2.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpus4frjv5
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 2 items

test_notebook1.ipynb::nbregression(test_notebook1)
FAILED                [ 50%]
test_notebook2.ipynb::nbregression(test_notebook2)
FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
FAILED test_notebook2.ipynb::nbregression(test_notebook2) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 2 failed in 2.12s ===============================

%%pytest -v --color=yes --disable-warnings

---
[pytest]
nb_test_files = True
---

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "test_notebook2.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmphpxzhoyw
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 2 items

test_notebook1.ipynb::nbregression(test_notebook1)
FAILED                [ 50%]
test_notebook2.ipynb::nbregression(test_notebook2)
FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
FAILED test_notebook2.ipynb::nbregression(test_notebook2) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 2 failed in 5.20s ===============================

To restrict the notebook files pytest collects, one or more filename pattern matches can also be set (see fnmatch).

%%pytest -v --color=yes --disable-warnings

---
[pytest]
nb_test_files = True
nb_file_fnmatch = test_*.ipynb tutorial_*.ipynb
---

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "other_notebook2.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpu7eywtl9
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 1 item

test_notebook1.ipynb::nbregression(test_notebook1)
FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.27s ===============================

Live Logging of Cell Execution

If you wish to view the progress of the notebook execution, you can use the pytest live-logging functionality:

%%pytest -v --color=yes --disable-warnings --nb-test-files --log-cli-level=info

***
(notebook1_content, "test_notebook1.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmphc_mdhia
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 1 item

test_notebook1.ipynb::nbregression(test_notebook1)

-------------------------------- live log call ---------------------------------
INFO     pytest_notebook.execution:execution.py:102 Executing notebook with kernel: python3
FAILED                                                                   [100%]


=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: 
--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
------------------------------ Captured log call -------------------------------
INFO     pytest_notebook.execution:execution.py:102 Executing notebook with kernel: python3
=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: 
============================== 1 failed in 1.27s ===============================

Regenerating Notebooks

Failing notebooks can be regenerated by setting --nb-force-regen. This will overwrite failing notebooks with the output from the notebook execution.

Note

Notebooks will not be regenerated if they raise any unexpected exceptions, during execution.

This approach to regeneration mimics pytest-regressions.

%%pytest -v --color=yes --disable-warnings --nb-force-regen

---
[pytest]
nb_test_files = True
nb_force_regen = True
---

***
(notebook1_content, "test_notebook1.ipynb")
(notebook2_content, "test_notebook2.ipynb")
***

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
NB force regen: True
rootdir: /tmp/tmp2kx_zkwm
configfile: pytest.ini
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 2 items

test_notebook1.ipynb::nbregression(test_notebook1)
FAILED                [ 50%]
test_notebook2.ipynb::nbregression(test_notebook2)
FAILED                [100%]

=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmp2kx_zkwm/test_notebook1.ipynb
----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
Diff before regeneration:

--- expected
+++ obtained
## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


____________________ notebook: nbregression(test_notebook2) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /tmp/tmp2kx_zkwm/test_notebook2.ipynb
----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
Diff before regeneration:

--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


=========================== short test summary info ============================
FAILED test_notebook1.ipynb::nbregression(test_notebook1) - pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-forc...
FAILED test_notebook2.ipynb::nbregression(test_notebook2) - pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-forc...
============================== 2 failed in 2.16s ===============================

The regeneration can be observed, if we run two tests on the same notebook.

%%pytest -v --color=yes --disable-warnings

import os, tempfile
try:
    # Python <= 3.8
    from importlib_resources import files
except ImportError:
    from importlib.resources import files
import pytest
from pytest_notebook import example_nbs

@pytest.fixture(scope="module")
def notebook():
    tmphandle, tmppath = tempfile.mkstemp(suffix=".ipynb")
    with open(tmppath, "w") as handle:
        handle.write(
            files(example_nbs).joinpath("example1.ipynb").read_text()
        )
    yield tmppath
    os.remove(tmppath)

def test_notebook1(nb_regression, notebook):
    nb_regression.force_regen = True
    nb_regression.check(notebook)

def test_notebook2(nb_regression, notebook):
    nb_regression.check(notebook)

/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.execution.ExecuteResult.exec_error" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_cwd" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.exec_env" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_config" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_source" type could not be converted.
  warnings.warn(  # noqa: B028
/home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/lib/python3.12/site-packages/pytest_notebook/utils.py:48: UserWarning: Field "pytest_notebook.nb_regression.NBRegressionFixture.cov_merge" type could not be converted.
  warnings.warn(  # noqa: B028
============================= test session starts ==============================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /home/docs/checkouts/readthedocs.org/user_builds/pytest-notebook/envs/latest/bin/python
cachedir: .pytest_cache
rootdir: /tmp/tmpdjywntbk
plugins: anyio-4.14.1, pytest_notebook-0.11.0, cov-7.1.0
collecting ... collected 2 items

test_ipycell.py::test_notebook1
FAILED                                   [ 50%]
test_ipycell.py::test_notebook2
PASSED                                   [100%]

=================================== FAILURES ===================================
________________________________ test_notebook1 ________________________________

nb_regression = NBRegressionFixture(exec_notebook=True, exec_cwd=None, exec_allow_errors=False, exec_timeout=120, exec_env=None, cover...place=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=False, force_regen=True)
notebook = '/tmp/tmpps74y82a.ipynb'

    def test_notebook1(nb_regression, notebook):
        nb_regression.force_regen = True
>       nb_regression.check(notebook)
E       pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
E       - /tmp/tmpps74y82a.ipynb

test_ipycell.py:23: NBRegressionError
----------------------------- Captured stderr call -----------------------------
[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption.
Diff before regeneration:

--- expected
+++ obtained
## replaced /cells/1/execution_count:
-  2
+  1

## modified /cells/2/outputs/0/text:
@@ -1,2 +1,2 @@
 hallo1
-hallo3
+hallo2

## modified /metadata/language_info/version:
-  3.6.7
+  3.12.13


=========================== short test summary info ============================
FAILED test_ipycell.py::test_notebook1 - pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-forc...
========================= 1 failed, 1 passed in 2.33s ==========================