Configuring pytest-notebook¶
Testing can be configured via;
The pytest configuration file.
The pytest command-line interface.
The notebook and cell level metadata.
To see the options available for (1) and (2), pytest -h can be used
(look for options starting nb), and for (3) see
Notebook/Cell Metadata Schema.
To access notebook metadata, either open the notebook as a text document, or use the Jupyter Notebook interface:
Accessing notebook metadata.¶
In:
import nbformat
In:
%load_ext pytest_notebook.ipy_magic
Ignoring Elements of the Notebook¶
When comparing the initial and final notebook, differences are denoted by “paths” in the notebook; a list of dictionary keys and list indices, delimited by ‘/’.
In:
notebook = nbformat.v4.new_notebook(
cells=[
nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
nbformat.v4.new_code_cell(
"print(1)",
execution_count=3,
outputs=[
nbformat.v4.new_output(output_type="stream", name="stdout", text="2\n")
],
),
]
)
In:
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files
***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmp7unvedjq
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook1.ipynb F [100%]
=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError:
--- expected
+++ obtained
## inserted before /cells/0/outputs/0:
+ output:
+ output_type: stream
+ name: stdout
+ text:
+ 1
## replaced /cells/1/execution_count:
- 3
+ 2
## modified /cells/1/outputs/0/text:
@@ -1 +1 @@
-2
+1
## added /metadata/language_info:
+ codemirror_mode:
+ name: ipython
+ version: 3
+ file_extension: .py
+ mimetype: text/x-python
+ name: python
+ nbconvert_exporter: python
+ pygments_lexer: ipython3
+ version: 3.6.7
===================== 1 failed, 1 warnings in 2.34 seconds =====================
We can set paths to ignore, in the configuration file, notebook metadata, or cell metadata.
Note
If the path is set in a cell metadata, it should be relative to that cell, i.e. /outputs not /cells/0/outputs
In:
notebook.metadata["nbreg"] = {"diff_ignore": ["/cells/0/outputs/"]}
notebook.cells[1].metadata["nbreg"] = {"diff_ignore": ["/outputs/0/text"]}
In:
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files
---
[pytest]
nb_diff_ignore =
/metadata/language_info
/cells/1/execution_count
---
***
(nbformat.writes(notebook), "test_notebook1.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmp8y3upprv, inifile: pytest.ini
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook1.ipynb . [100%]
===================== 1 passed, 1 warnings in 2.28 seconds =====================
Wildcard * can also be used, in place of indices, to apply to all
indices in the list.
In:
notebook2 = nbformat.v4.new_notebook(
cells=[
nbformat.v4.new_code_cell("print(1)", execution_count=1, outputs=[]),
nbformat.v4.new_code_cell("print(1)", execution_count=2, outputs=[]),
]
)
In:
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files
---
[pytest]
nb_diff_ignore =
/metadata/language_info
/cells/*/outputs/
---
***
(nbformat.writes(notebook2), "test_notebook2.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpgh2kmcrh, inifile: pytest.ini
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook2.ipynb . [100%]
===================== 1 passed, 1 warnings in 2.34 seconds =====================
Regex Pattern Replacement¶
Rather than simply ignoring cells, we can apply regex replacements to sections of the notebook, before the diff comparison is made. This is particularly useful for changeable outputs, such as dates and times:
In:
notebook3 = nbformat.v4.new_notebook(
cells=[
nbformat.v4.new_code_cell(
("from datetime import date\n" "print(date.today())"),
execution_count=1,
outputs=[
nbformat.v4.new_output(
output_type="stream", name="stdout", text="DATE-STAMP\n"
)
],
)
]
)
In:
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files
---
[pytest]
nb_diff_ignore =
/metadata/language_info
---
***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpoxy8ral2, inifile: pytest.ini
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook3.ipynb F [100%]
=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook3) ____________________
pytest_notebook.nb_regression.NBRegressionError:
--- expected
+++ obtained
## modified /cells/0/outputs/0/text:
@@ -1 +1 @@
-DATE-STAMP
+2019-08-13
===================== 1 failed, 1 warnings in 2.29 seconds =====================
In:
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files
---
[pytest]
nb_diff_ignore =
/metadata/language_info
nb_diff_replace =
/cells/*/outputs/*/text \d{2,4}-\d{1,2}-\d{1,2} "DATE-STAMP"
---
***
(nbformat.writes(notebook3), "test_notebook3.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpi35a1mfi, inifile: pytest.ini
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook3.ipynb . [100%]
===================== 1 passed, 1 warnings in 2.32 seconds =====================
Ignoring Exceptions¶
To mark expected exceptions from a notebook cell, tag the cell as
raises-exception:
Tagging exceptions.¶
In:
notebook4 = nbformat.v4.new_notebook(
cells=[
nbformat.v4.new_code_cell(
'raise Exception("expected error")',
execution_count=1,
outputs=[
nbformat.v4.new_output(
output_type="error",
ename="Exception",
evalue="expected error",
traceback=[],
)
],
)
]
)
notebook4.metadata["nbreg"] = {
"diff_ignore": ["/metadata/language_info", "/cells/0/outputs/0/traceback"]
}
In:
%%pytest --color=yes --show-capture=no --disable-warnings --nb-test-files
***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpbdfklwmx
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook4.ipynb F [100%]
=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook4) ____________________
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:
------------------
raise Exception("expected error")
------------------
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-1-ba9385a0cebd> in <module>
----> 1 raise Exception("expected error")
Exception: expected error
Exception: expected error
===================== 1 failed, 1 warnings in 2.45 seconds =====================
In:
notebook4.cells[0].metadata["tags"] = ["raises-exception"]
In:
%%pytest --color=yes --disable-warnings --nb-test-files
***
(nbformat.writes(notebook4), "test_notebook4.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpfzle7_4r
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook4.ipynb . [100%]
===================== 1 passed, 1 warnings in 2.50 seconds =====================
Skipping Notebooks¶
To add the pytest skip
decorator
to a notebook, you can add skip=True to the notebook metadata.
In:
notebook5 = nbformat.v4.new_notebook()
notebook5.metadata["nbreg"] = {"skip": True, "skip_reason": "Not ready for testing."}
notebook5
Out:
{'nbformat': 4,
'nbformat_minor': 2,
'metadata': {'nbreg': {'skip': True,
'skip_reason': 'Not ready for testing.'}},
'cells': []}
In:
%%pytest -v --color=yes -rs --nb-test-files
***
(nbformat.writes(notebook5), "test_notebook5.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- //anaconda/envs/pytest_nb/bin/python
cachedir: .pytest_cache
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmp3xmsj73j
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collecting ... collected 1 item
test_notebook5.ipynb::nbregression(test_notebook5) SKIPPED [100%]
=========================== short test summary info ============================
SKIPPED [1] test_notebook5.ipynb: Not ready for testing.
========================== 1 skipped in 0.06 seconds ===========================
Post-processors¶
Post-processors are applied to the final notebook, before diff
comparison. These can be added by external packages, using the
nbreg.post_proc group entry
point:
# setup.py
from setuptools import setup
setup(
name="myproject",
packages=["myproject"],
entry_points={
"nbreg.post_proc": [
"blacken_code = post_processors:blacken_code"
]},
)
See also
pytest_notebook.post_processors
for the internally
provided plugins.
Format Source Code¶
An example of this is the blacken_code post-processor, which applies
the black formatter to all source
code cells.
This is particularly useful for re-generating notebooks.
In:
notebook6 = nbformat.v4.new_notebook(
cells=[
nbformat.v4.new_code_cell(
(
"for i in range(5 ) :\n"
" x=i\n"
" a ='123'# comment\n"
"c = ['a fairly long line of text', "
"'another fairly long line of text',\n"
"'yet another fairly long line of text']"
),
execution_count=1,
outputs=[],
)
]
)
In:
%%pytest --color=yes --disable-warnings --nb-test-files --nb-force-regen
---
[pytest]
nb_exec_notebook = False
nb_diff_ignore =
/metadata/language_info
nb_post_processors =
coalesce_streams
blacken_code
---
***
(nbformat.writes(notebook6), "test_notebook6.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
NB post processors: coalesce_streams blacken_code
NB force regen: True
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpdr8y1ufi, inifile: pytest.ini
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook6.ipynb F [100%]
=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook6) ____________________
pytest_notebook.nb_regression.NBRegressionError: Files differ and --nb-force-regen set, regenerating file at:
- /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpdr8y1ufi/test_notebook6.ipynb
----------------------------- Captured stderr call -----------------------------
Diff before regeneration:
--- expected
+++ obtained
## modified /cells/0/source:
@@ -1,5 +1,8 @@
-for i in range(5 ) :
- x=i
- a ='123'# comment
-c = ['a fairly long line of text', 'another fairly long line of text',
-'yet another fairly long line of text']
+for i in range(5):
+ x = i
+ a = "123" # comment
+c = [
+ "a fairly long line of text",
+ "another fairly long line of text",
+ "yet another fairly long line of text",
+]
=========================== 1 failed in 0.65 seconds ===========================
Format HTML/SVG outputs¶
The beautifulsoup()
post-processor may also be useful, for assessing differences in HTML and
SVG outputs.
Note
This requires beautifulsoup4 to be installed.