有选择地从另一个Jupyter Notebook导入 [英] Selectively import from another Jupyter Notebook

问题描述

我将Jupyter笔记本布置为:data.ipynb，methods.ipynb和results.ipynb.如何从 data 和 methods 笔记本中有选择地导入单元格，以在 results 笔记本中使用?

我知道nbimporter和ipynb，但是它们都不提供变量的选择性导入.有一个选项可以导入定义-包括大写的变量-但这对我不起作用，因为我必须将笔记本中的大多数变量转换为大写.

我宁愿导入所有内容，除了需要花费很长时间进行评估的两个或三个单元格之外.理想情况下，我想将某些分配的执行推迟到我访问它们的那一刻(惰性评估)-但我知道这可能很难实现.

这是概述，以伪代码表示(每行代表一个单元格):

data.ipynb:

raw_data = load_data()
dataset = munge(raw_data)
describe(dataset)             # I want this line to be skipped at import

methods.ipynb:

import data
method = lambda x: x * x
# showcase how the method works on a subset of the dataset
method(data.dataset[:5])      # I want this line to be skipped at import

results.ipynb:

import data
import methods
result = methods.method(data.dataset)
describe(result)

动机是我真正的数据和方法笔记本:

• 要长得多且复杂得多，因此我想使用导入系统
• 只有几个单元格需要花费超过几秒钟的时间来评估

此外，方法笔记本不能用methods.py文件替换.实际上，我有一个文件，其中包含方法的实现细节.笔记本更像是用于指定默认参数，展示我的方法如何工作并解释示例结果的地方.

此问题实质上是以下问题的组合:

• 如何从另一个ipython-notebook导入?和
• 简单在全部运行期间选择要在ipython笔记本中运行哪些单元格的方法

我通读了两个答案，但没有一个满足我的要求.

在下面的回答中，我介绍了使用自定义单元魔术和猴子补丁的解决方案.但是，我宁愿选择一种解决方案，该解决方案可以指定要排除/包含的单元格/表达式不在原始笔记本中(例如data.ipynb)，而是在目标笔记本中(例如methods.ipynb).

例如，它可以使用正则表达式:

# all variables starting with 'result' would be ignored
nbimporter.options['exclude'] = '^result.*'

或(甚至更好)惰性评估:

# only `a` and `b` would be evaluated and imported
from data import a, b

所有想法将不胜感激！

解决方案

到目前为止，我一直在猴子修补nbimporter并选择要使用细胞魔术术排除的细胞:

from IPython.core import magic

@magic.register_cell_magic
def skip_on_import(args, cell):
    get_ipython().ex(cell)

用于去除细胞的猴子的代码:

import ast

class SkippingTransformer(ast.NodeTransformer):
    # usage:
    # import nbimporter 
    # nbimporter.CellDeleter = SkippingTransformer

    def visit(self, node):
        if (
            isinstance(node, ast.Expr)
            and isinstance(node.value, ast.Call)
            and isinstance(node.value.func, ast.Attribute)
            and node.value.func.attr == 'run_cell_magic'
            and node.value.args[0].s == 'skip_on_import'
        ):
            return
        return node

还有一个实际示例data.ipynb:

和methods.ipynb(最后一个异常是有意的-表示成功！):

编辑:我将上述代码作为 jupyter-helpers的一部分发布了前一段时间.使用此软件包，只需将导入程序导入到导入笔记本中即可.

from jupyter_helpers.selective_import import notebooks_importer 

可以使用以下命令在导入的笔记本中导入cell-magic:

from jupyter_helpers.selective_import import skip_on_import

以下是示例导入的笔记本: Data.ipynb 和导入笔记本的示例: Results.ipynb

I arranged my Jupyter notebooks into: data.ipynb, methods.ipynb and results.ipynb. How can I selectively import cells from data and methods notebooks for use in the results notebook?

I know of nbimporter and ipynb but neither of those offers selective import of variables. There is an option to import definitions - including variables that are uppercase - but this does not work for me as I would have to convert most of the variables in my notebooks to uppercase.

I would rather import everything except for two or three cells that take a long time to evaluate. Ideally, I would like to defer the execution of some assignments to the very moment I access them (lazy evaluation) - but I understand that it might be difficult to implement.

Here is the overview, in pseudocode (each line repesents a cell):

data.ipynb:

raw_data = load_data()
dataset = munge(raw_data)
describe(dataset)             # I want this line to be skipped at import

methods.ipynb:

import data
method = lambda x: x * x
# showcase how the method works on a subset of the dataset
method(data.dataset[:5])      # I want this line to be skipped at import

results.ipynb:

import data
import methods
result = methods.method(data.dataset)
describe(result)

The motivation is that my real data and methods notebooks:

• are way much longer and complicated, hence I want to use an import system
• there are only a couple of cells that take more than seconds to evaluate

also, the methods notebook cannot be replaced with methods.py file. In fact, I have such a file which contains the implementation details of my method. The notebook is more of a place to specify default parameters, showcase how my method works and explain example results.

This question is essentially a combination of:

• How can I import from another ipython-notebook?, and
• Simple way to choose which cells to run in ipython notebook during run all

I read through answers to both and none satisfied my requirements.

In my answer below I present my solution that uses custom cell magics and monkey-patching. However, I would prefer a solution which allows specifying which cells/expressions to exclude/include not in the notebook of origin (e.g. data.ipynb) but in the target one (e.g. in methods.ipynb).

For example, it could use regular expressions:

# all variables starting with 'result' would be ignored
nbimporter.options['exclude'] = '^result.*'

or (even better) lazy evaluation:

# only `a` and `b` would be evaluated and imported
from data import a, b

All ideas will be appreciated!

解决方案

So far I've been monkey-patching nbimporter and selecting cells to exclude using cell magic:

from IPython.core import magic

@magic.register_cell_magic
def skip_on_import(args, cell):
    get_ipython().ex(cell)

The code used to monkey-patch of cell remover:

import ast

class SkippingTransformer(ast.NodeTransformer):
    # usage:
    # import nbimporter 
    # nbimporter.CellDeleter = SkippingTransformer

    def visit(self, node):
        if (
            isinstance(node, ast.Expr)
            and isinstance(node.value, ast.Call)
            and isinstance(node.value.func, ast.Attribute)
            and node.value.func.attr == 'run_cell_magic'
            and node.value.args[0].s == 'skip_on_import'
        ):
            return
        return node

And an actual example, data.ipynb:

And methods.ipynb (the exception at the end is intended - it means success!):

Edit: I published the above code as a part of jupyter-helpers some time ago. Using this package one simply needs to import the importer in the importing notebook:

from jupyter_helpers.selective_import import notebooks_importer 

and the cell-magic can be imported in the imported notebook with:

from jupyter_helpers.selective_import import skip_on_import

Here is example imported notebook: Data.ipynb and example importing notebook: Results.ipynb

这篇关于有选择地从另一个Jupyter Notebook导入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！