Execute and render ipynb's similar to spin R / Rmd scripts

[Avsecz]

It would be nice to be able to specify a similar header in the jupyter notebook as in R. wBuild would then parse this configuration and run the notebook. The notebook can be parametrized and executed using the following code

import subprocess
import os

def jupyter_nbconvert(input_ipynb):
    """ .ipynb -> .html
    """
    subprocess.call(["jupyter",
                     "nbconvert",
                     input_ipynb,
                     "--to", "html"])


def render_ipynb(template_ipynb, rendered_ipynb, params=dict()):
    """Render the ipython notebook
    Args:
      template_ipynb: template ipython notebook where one cell defines the following metadata:
        {"tags": ["parameters"]}
      render_ipynb: output ipython notebook path
      params: parameters used to execute the ipython notebook
    """
    import papermill as pm  # Render the ipython notebook

    os.makedirs(os.path.dirname(rendered_ipynb), exist_ok=True)
    pm.execute_notebook(
        template_ipynb,  # input template
        rendered_ipynb,
        kernel_name="python3",  # default kernel
        parameters=params
    )
    jupyter_nbconvert(rendered_ipynb)


rule compile_notebook:
    """Run the notebook and convert it to html
    """
    input:
        example='example_file_{some_param}.csv',
        ipynb = "src/<ipynb-path>.ipynb"
    output:
        ipynb = "output/<ipynb-path>.ipynb",
        html = "output/<ipynb-path>.html"
    run:
        render_ipynb(input.ipynb, output.ipynb,
                     params=dict(example=input.example,
                                 some_param=wildcards.some_param))
        jupyter_nbconvert(output.ipynb)

Required packages

• papermill: https://github.com/nteract/papermill

Related

• https://bitbucket.org/snakemake/snakemake/issues/529/support-running-from-ipython-notebooks
• https://github.com/jmeppley/snakemake_magic/blob/master/prototype/notebooks/snakemake%20magic.ipynb

[Hoeze]

+1 for that, I would love using wBuild for creating reports.
ATM I have to pass environment variables to my notebook to parameterize it.

[stefanches7]

• Read YAML cell based on position (tag?)
• Use its contents to parametrically run notebook using papermill
• Run nbconvert to convert run notebook to HTML report
• Add HTML report to index

[Hoeze]

Just for reference:
JupyText is a jupyterlab extension which allows to transparently convert .ipynb to .Rmd.
This way, Jupyter notebooks can be directly versioned via git and executed with wbuild.