从IPython Notebook运行MRJob [英] Run MRJob from IPython notebook

问题描述

我正在尝试从IPython笔记本运行mrjob示例

I'm trying to run mrjob example from IPython notebook

from mrjob.job import MRJob


class MRWordFrequencyCount(MRJob):

def mapper(self, _, line):
    yield "chars", len(line)
    yield "words", len(line.split())
    yield "lines", 1

def reducer(self, key, values):
    yield key, sum(values)  

然后使用代码运行

mr_job = MRWordFrequencyCount(args=["testfile.txt"])
with mr_job.make_runner() as runner:
    runner.run()
    for line in runner.stream_output():
        key, value = mr_job.parse_output_line(line)
        print key, value

并收到错误消息:

TypeError: <module '__main__' (built-in)> is a built-in class

是否可以从IPython Notebook运行mrjob?

Is there way to run mrjob from IPython notebook?

推荐答案

我还没有找到完美的方法"，但是您可以做的一件事就是使用 %% file 魔术，将单元格内容写入文件:

I haven't found the "perfect way" yet, but one thing you can do is create one notebook cell, using the %%file magic, writing the cell contents to a file:

%%file wordcount.py
from mrjob.job import MRJob

class MRWordFrequencyCount(MRJob):

    def mapper(self, _, line):
        yield "chars", len(line)
        yield "words", len(line.split())
        yield "lines", 1

    def reducer(self, key, values):
        yield key, sum(values)

然后让 mrjob 在以后的单元格中运行该文件:

And then have mrjob run that file in a later cell:

import wordcount
reload(wordcount)

mr_job = wordcount.MRWordFrequencyCount(args=['example.txt'])
with mr_job.make_runner() as runner:
    runner.run()
    for line in runner.stream_output():
        key, value = mr_job.parse_output_line(line)
        print key, value

请注意，我调用了文件 wordcount.py ，并且从 wordcount 模块(文件名和模块)导入了类 MRWordFrequencyCount 必须匹配.此外，Python还会缓存导入的模块，并且当您更改 wordcount.py 文件时，iPython不会重新加载模块，而是使用已缓存的旧模块.这就是为什么我在其中放置 reload()调用的原因.

Notice that I called my file wordcount.py and that I import the class MRWordFrequencyCount from the wordcount module -- the filename and module has to match. Also Python caches imported modules and when you change the wordcount.py-file iPython will not reload the module but rather used the old, cached one. That's why I put the reload() call in there.

参考: https://groups.google.com/d/msg/mrjob/CfdAgcEaC-I/8XfJPXCjTvQJ

更新(较短)
对于较短的第二个笔记本单元，您可以通过从笔记本内部调用外壳程序来运行mrjob

Update (shorter)
For a shorter second notebook cell you can run the mrjob by invoking the shell from within the notebook

! python mrjob.py shakespeare.txt

参考: http://jupyter.cs.brynmawr.edu/hub/dblank/公共/Jupyter%20Magics.ipynb

Reference: http://jupyter.cs.brynmawr.edu/hub/dblank/public/Jupyter%20Magics.ipynb

这篇关于从IPython Notebook运行MRJob的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！