Execution and Coverage Reporting¶
The core component of the notebook execution API is the
ExecuteCoveragePreprocessor
class, which is a subclass of
nbconvert.preprocessors.ExecutePreprocessor, that can
additionally create code coverage
analytics.
This class is called by
execute_notebook(), which returns an
ExecuteResult object.
In:
import ruamel.yaml as yaml
from pytest_notebook.execution import execute_notebook
from pytest_notebook.notebook import create_notebook, create_cell, dump_notebook
In:
notebook = create_notebook()
notebook.cells = [
create_cell("""
from pytest_notebook import __version__
from pytest_notebook.notebook import create_notebook
print(__version__)
"""
)
]
In:
result = execute_notebook(notebook, with_coverage=True)
result.notebook
Out:
{'nbformat': 4,
'nbformat_minor': 2,
'metadata': {'language_info': {'name': 'python',
'version': '3.6.7',
'mimetype': 'text/x-python',
'codemirror_mode': {'name': 'ipython', 'version': 3},
'pygments_lexer': 'ipython3',
'nbconvert_exporter': 'python',
'file_extension': '.py'}},
'cells': [{'cell_type': 'code',
'metadata': {},
'execution_count': 1,
'source': '\nfrom pytest_notebook import __version__\nfrom pytest_notebook.notebook import create_notebook\nprint(__version__)\n',
'outputs': [{'output_type': 'stream',
'name': 'stdout',
'text': '0.6.0\n'}]}]}
In:
result.coverage_data()
Out:
<CoverageData lines={5} arcs=None tracers={0} runs=[0]>
In:
print(yaml.dump(result.coverage_dict))
lines:
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/__init__.py: [4, 5]
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/diffing.py: [1, 2, 3, 4, 5,
7, 8, 9, 10, 11, 12, 24, 25, 66, 67, 98, 101, 116, 117, 162, 163]
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/notebook.py: [1, 2, 3, 4, 5,
6, 8, 9, 10, 11, 12, 13, 15, 16, 17, 19, 21, 25, 26, 49, 50, 80, 81, 108, 109,
112, 142, 151, 173, 174, 176, 178, 179, 180, 181, 184, 191, 192, 195, 202, 203,
204, 205, 207, 208, 209, 210, 214, 245, 246, 252, 253, 260, 267, 275, 291, 292,
293, 295, 308]
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/resources/__init__.py: [1]
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/utils.py: [1, 2, 4, 5, 6, 9,
29, 52, 86, 99, 136, 138, 142, 144, 100, 102, 106, 108, 112, 32, 35, 40, 41, 44,
45, 49, 114, 115, 116, 120, 122, 123, 124, 125, 126, 128, 134, 149, 151, 87, 89,
97, 156]
The coverage can be limited to particular files or modules, by setting
cov_source.
In:
result = execute_notebook(
notebook, with_coverage=True, cov_source=['pytest_notebook.notebook'])
print(yaml.dump(result.coverage_dict))
lines:
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/notebook.py: [1, 2, 3, 4, 5,
6, 8, 9, 10, 11, 12, 13, 15, 16, 17, 19, 21, 25, 26, 49, 50, 80, 81, 108, 109,
112, 142, 151, 173, 174, 176, 178, 179, 180, 181, 184, 191, 192, 195, 202, 203,
204, 205, 207, 208, 209, 210, 214, 245, 246, 252, 253, 260, 267, 275, 291, 292,
293, 295, 308]
Integration with pytest-cov¶
If the pytest-cov plugin is
installed, the
NBRegressionFixture will be
initialised with the settings and coverage.Coverage object,
that pytest-cov has created.
If the --nb-coverage flag is set, then nb_regression will run
coverage introspection, and merge the data back into the main
Coverage object.
In:
%load_ext pytest_notebook.ipy_magic
In:
%%pytest --disable-warnings --color=yes --cov=pytest_notebook --nb-coverage --log-cli-level=info
import logging
import importlib_resources
from pytest_notebook import example_nbs
def test_notebook(nb_regression):
logging.getLogger(__name__).info(nb_regression)
with importlib_resources.path(example_nbs, "coverage.ipynb") as path:
nb_regression.check(str(path))
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpxcs1xsgm
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_ipycell.py::test_notebook
-------------------------------- live log call ---------------------------------
INFO test_ipycell:test_ipycell.py:7 NBRegressionFixture(exec_notebook=True, exec_cwd=None, exec_allow_errors=False, exec_timeout=120, coverage=True, cov_config='.coveragerc', cov_source=('pytest_notebook',), cov_merge=<coverage.control.Coverage object at 0x103ec3e48>, post_processors=('coalesce_streams',), process_resources={}, diff_replace=(), diff_ignore=('/cells/*/outputs/*/traceback',), diff_use_color=True, diff_color_words=False, force_regen=False)
INFO pytest_notebook.execution:execution.py:147 About to execute notebook with 1 cells
INFO pytest_notebook.execution:execution.py:153 Executing notebook with kernel: python
INFO pytest_notebook.execution:execution.py:155 Recording coverage for notebook
INFO pytest_notebook.execution:execution.py:173 Executing cell 0
INFO pytest_notebook.nb_regression:nb_regression.py:306 Merging coverage.
PASSED [100%]
---------- coverage: platform darwin, python 3.6.7-final-0 -----------
Name Stmts Miss Cover
-------------------------------------------------------------------------------------------------
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/__init__.py 1 0 100%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/diffing.py 83 40 52%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/example_nbs/__init__.py 0 0 100%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/execution.py 103 61 41%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/ipy_magic.py 80 80 0%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/nb_regression.py 184 112 39%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/notebook.py 140 74 47%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/plugin.py 117 81 31%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/post_processors.py 93 69 26%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/resources/__init__.py 0 0 100%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/utils.py 58 23 60%
-------------------------------------------------------------------------------------------------
TOTAL 859 540 37%
===================== 1 passed, 1 warnings in 3.73 seconds =====================
Coverage.py warning: Module pytest_notebook was previously imported, but not measured (module-not-measured)
This is also the case, when using the pytest file collection approach.
In:
%%pytest --disable-warnings --color=yes --cov=pytest_notebook --log-cli-level=info
---
[pytest]
nb_coverage = True
nb_test_files = True
---
***
(dump_notebook(notebook), "test_notebook1.ipynb")
***
============================= test session starts ==============================
platform darwin -- Python 3.6.7, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /private/var/folders/dm/b2qnkb_n3r72slmpxlfmcjvm00lbnd/T/tmpzc571yr2, inifile: pytest.ini
plugins: cov-2.7.1, datadir-1.3.0, regressions-1.0.5, notebook-0.5.1
collected 1 item
test_notebook1.ipynb::nbregression(test_notebook1)
-------------------------------- live log call ---------------------------------
INFO pytest_notebook.execution:execution.py:147 About to execute notebook with 1 cells
INFO pytest_notebook.execution:execution.py:153 Executing notebook with kernel: python
INFO pytest_notebook.execution:execution.py:155 Recording coverage for notebook
INFO pytest_notebook.execution:execution.py:173 Executing cell 0
INFO pytest_notebook.nb_regression:nb_regression.py:306 Merging coverage.
FAILED [100%]
=================================== FAILURES ===================================
____________________ notebook: nbregression(test_notebook1) ____________________
pytest_notebook.nb_regression.NBRegressionError:
--- expected
+++ obtained
## replaced (type changed from NoneType to int) /cells/0/execution_count:
- None
+ 1
## inserted before /cells/0/outputs/0:
+ output:
+ output_type: stream
+ name: stdout
+ text:
+ 0.6.0
## added /metadata/language_info:
+ codemirror_mode:
+ name: ipython
+ version: 3
+ file_extension: .py
+ mimetype: text/x-python
+ name: python
+ nbconvert_exporter: python
+ pygments_lexer: ipython3
+ version: 3.6.7
------------------------------ Captured log call -------------------------------
INFO pytest_notebook.execution:execution.py:147 About to execute notebook with 1 cells
INFO pytest_notebook.execution:execution.py:153 Executing notebook with kernel: python
INFO pytest_notebook.execution:execution.py:155 Recording coverage for notebook
INFO pytest_notebook.execution:execution.py:173 Executing cell 0
INFO pytest_notebook.nb_regression:nb_regression.py:306 Merging coverage.
---------- coverage: platform darwin, python 3.6.7-final-0 -----------
Name Stmts Miss Cover
-------------------------------------------------------------------------------------------------
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/__init__.py 1 0 100%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/diffing.py 83 11 87%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/example_nbs/__init__.py 0 0 100%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/execution.py 103 61 41%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/ipy_magic.py 80 80 0%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/nb_regression.py 184 112 39%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/notebook.py 140 74 47%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/plugin.py 117 70 40%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/post_processors.py 93 69 26%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/resources/__init__.py 0 0 100%
/Users/cjs14/GitHub/pytest-notebook/pytest_notebook/utils.py 58 23 60%
-------------------------------------------------------------------------------------------------
TOTAL 859 500 42%
===================== 1 failed, 1 warnings in 3.49 seconds =====================
Coverage.py warning: Module pytest_notebook was previously imported, but not measured (module-not-measured)