慢慢的ipython --pylab和ipython笔记本启动 [英] Slow ipython --pylab and ipython notebook startup
问题描述
我已经从ipython 0.10切换到1.1.0。现在我正在经历令人烦恼的启动过程缓慢下降。
虽然 ipython 单独仍然没有时间， ipython --pylab 起步很慢，即 ~8秒（在英特尔（R）核心上） （TM）2 Duo CPU P9500 @ 2.53GHz系统）甚至更多新的 ipython笔记本我在ipython 0.10中没有，即〜 12秒。
当我做 strace -o tessi.txt -tt ipython --pylab 我至少可以识别出似乎对这些延迟的大块负责的部分。任何摆脱这些的帮助都会非常受欢迎。
对于 - pylab 选项，strace的输出包含一个吃掉 ~5秒的部分，只在ipython开始时不存在，即
10：23：24.331968 stat（/ etc / resolv.conf，{st_mode = S_IFREG | 0644，st_size = 868，...}）= 0
10：23：24.332028 socket （PF_FILE，SOCK_STREAM | SOCK_CLOEXEC | SOCK_NONBLOCK，0）= 9
10：23：24.332074 connect（9，{sa_family = AF_FILE，path =/ var / run / nscd / socket}，110）= 0
10：23：24.332152 sendto（9，\\\\\\\\\\\\\\\\\\\\ DOM.AIN。......，41，MSG_NOSIGNAL，NULL，0）= 41
!! - > 10：23：24.332227民意调查（[{fd = 9，events = POLLIN | POLLERR | POLLHUP}]，1,5000）= 1（[{fd = 9，revents = POLLIN | POLLHUP}]）
!! - > 10：23：29.336301读取（9，\\\\\\\\\\\\\\\\\\\\\ \\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 \0\0，32）= 32
10：23：29.336533 close（9）= 0
10：23：29.336839 close（7）= 0
我用 MYHOST.DOM.AIN <替换我的实际主机名/ p>
在 ipython notebook 的情况下，似乎基本上有两个这样的区域。第一个似乎与 ipython --pylab 相同。第二个带走另一个 ~4秒
10:39： 31.823298 stat（/ etc / resolv.conf，{st_mode = S_IFREG | 0644，st_size = 868，...}）= 0
10：39：31.823358 socket（PF_FILE，SOCK_STREAM | SOCK_CLOEXEC | SOCK_NONBLOCK，0 ）= 9
10：39：31.823405 connect（9，{sa_family = AF_FILE，path =/ var / run / nscd / socket}，110）= 0
10：39：31.823499 sendto（ 9，\\\\\\\\\\\\\\\\\\\ 0MYHOST.DOM.AIN。...，41 ，MSG_NOSIGNAL，NULL，0）= 41
!! - > 10：39：31.824166民意调查（[{fd = 9，events = POLLIN | POLLERR | POLLHUP}]，1,5000）= 1（[{fd = 9，revents = POLLIN | POLLHUP}]）
!! - > 10：39：36.827298读取（9，\\\\\\\\\\\\\\\\\\\\\ \\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 \ 0：0，32）= 32
10：39：36.827503 close（9）= 0
10：39：36.827828 close（7）= 0
10：39：38.591774 setsockopt（6，SOL_TCP，TCP_NODELAY，[0]，4）= 0
10：39：38.591922民意调查（[{fd = 3，events = POLLIN}，{fd = 4，events = POLLIN}，{fd = 6，events = POLLIN}，{fd = 7，events = POLLIN}]，4,0）= 0（超时）
10：39：38.592007民意调查（[{fd = 3， events = POLLIN}，{fd = 4，events = POLLIN}，{fd = 6，events = POLLIN}，{fd = 7，events = POLLIN}]，4,0）= 0（超时）
！ ！ - > 10：39：38.592049民意调查（[{fd = 3，events = POLLIN}，{fd = 4，events = POLLIN}，{fd = 6，events = POLLIN}，{fd = 7，events = POLLIN}]，4 ，3600000）=？ ERESTART_RESTARTBLOCK（待重启）
!! - > 10：39：42.355658 --- SIGINT（中断）@ 0（0）---
10：39：42.355773写（5，\ 0，1）= 1
10:39 ：42.355916 rt_sigreturn（0x2）= -1 EINTR（系统调用中断）
10：39：42.356113 rt_sigaction（SIGINT，{0x7f26dc0fbfc6，[]，SA_RESTORER，0x7f26dbe5a2d0}，{0x7f26dc0fbfc6，[]，SA_RESTORER，0x7f26dbe5a2d0}， 8）= 0
10：39：42.356672克隆（child_stack = 0x7f26cfb49ff0，旗帜= CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID，parent_tidptr = 0x7f26cfb4a9d0，TLS = 0x7f26cfb4a700，child_tidptr = 0x7f26cfb4a9d0 ）= 11084
任何有关如何摆脱这些延误的帮助都是最受欢迎的。
最后，当我在它的时候。当我看到strace的剩余输出时，我看到，在启动时，ipython试图打开很多python相关文件，我会天真地认为它应该永远不会这样做，并且我的系统python绝对没有安装？所有这些开放的呼叫都失败了。例如：
11：11：26.465594 open（/ opt / intel / composerxe-2011.3.174 / compiler / lib / intel64 /tls/libpython2.7.so.1.0，O_RDONLY）= -1 ENOENT（没有这样的文件或目录）
11：11：26.465646 stat（/ opt / intel / composerxe-2011.3.174 / compiler / lib / intel64 / tls，0x7fff0efb0100）= -1 ENOENT（没有这样的文件或目录）
或
11：11：26.468293 stat（/ usr / local / lib / vtk-5.4 / tls / x86_64，0x7fff0efb0100）= -1 ENOENT（没有这样的文件或目录）
11：11：26.468347 open（/ usr / local / lib / vtk-5.4 / tls / libpython2 .7.so.1.0，O_RDONLY）= -1 ENOENT（没有这样的文件或目录）
这些呼叫的总时间占剩余慢启动的另一个重要部分，但我不能指向特定的时隙。知道如何减少这些调用吗？
1）我建议不要以开头 - pylab 我们鼓励人们在启动后使用％matplotlib 。使用 - pylab 时开始的开销很可能是由于导入了我们无法访问的软件包。例如，ArchLinux上有一些错误，其中每个 ipython --pylab 由于内存泄漏而在IPython代码中占用数百MB。
对套接字的访问看起来仍然很奇怪，要确切了解它的来源可能会有所帮助。
2）带笔记本的第二个可能是由于ZMQ和localhost上通过套接字进行的一些通信的开始。我们知道在某些情况下它可能需要一些时间，但我们还需要找出代码的哪一部分很慢。 NOtebook还启动了一个网络服务器，因此预计会进行网络操作。
3）可能是由于我们使用了一些库，我们不会在ipython中明确打开这样的tings。特别是VTK。你有一个加载特定东西的个人资料吗？
你可以尝试在IPython跟踪器上打开一个问题，但我们可能会要求你深入挖掘知道什么是在ipython中调用python需要花费很多时间。核心开发人员之间已经提出了启动时间的问题。
一致认为这将是尽力而为但我们确实是一个太小的问题优先考虑。仍然与vbech这样的工具集成以跟踪回归将是很好的，如果有人有动力去慢速调用，我们将很乐意尽力改善。
I have switched from ipython 0.10 to 1.1.0. Now I am experiencing very annoying slow-downs of the startup process.
While ipython alone is still up in no time, ipython --pylab takes a very slow start, i.e. ~8 secs (on an Intel(R) Core(TM)2 Duo CPU P9500 @ 2.53GHz system) and even more so the new ipython notebook which I did not have in ipython 0.10, namely ~12 secs.
When I do strace -o tessi.txt -tt ipython --pylab I can identify at least sections which seem to be responsible for large chunks of these delays. Any help on getting rid of those would very welcome.
For the --pylab option, strace's output contains a section which eats away ~5 secs, and which is not present when starting just ipython, namely
10:23:24.331968 stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=868, ...}) = 0
10:23:24.332028 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 9
10:23:24.332074 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
10:23:24.332152 sendto(9, "\2\0\0\0\4\0\0\0\35\0\0\0MYHOST.DOM.AIN."..., 41, MSG_NOSIGNAL, NULL, 0) = 41
!! -> 10:23:24.332227 poll([{fd=9, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=9, revents=POLLIN|POLLHUP}])
!! -> 10:23:29.336301 read(9, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
10:23:29.336533 close(9) = 0
10:23:29.336839 close(7) = 0
where I've replace my actual host name with MYHOST.DOM.AIN
In case of ipython notebook there seem to be essentially two such regions. The first one seems to be the same as for ipython --pylab. The second one takes away another ~4 secs
10:39:31.823298 stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=868, ...}) = 0
10:39:31.823358 socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 9
10:39:31.823405 connect(9, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = 0
10:39:31.823499 sendto(9, "\2\0\0\0\4\0\0\0\35\0\0\0MYHOST.DOM.AIN."..., 41, MSG_NOSIGNAL, NULL, 0) = 41
!! -> 10:39:31.824166 poll([{fd=9, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=9, revents=POLLIN|POLLHUP}])
!! -> 10:39:36.827298 read(9, "\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
10:39:36.827503 close(9) = 0
10:39:36.827828 close(7) = 0
10:39:38.591774 setsockopt(6, SOL_TCP, TCP_NODELAY, [0], 4) = 0
10:39:38.591922 poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 4, 0) = 0 (Timeout)
10:39:38.592007 poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 4, 0) = 0 (Timeout)
!! -> 10:39:38.592049 poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 4, 3600000) = ? ERESTART_RESTARTBLOCK (To be restarted)
!! -> 10:39:42.355658 --- SIGINT (Interrupt) @ 0 (0) ---
10:39:42.355773 write(5, "\0", 1) = 1
10:39:42.355916 rt_sigreturn(0x2) = -1 EINTR (Interrupted system call)
10:39:42.356113 rt_sigaction(SIGINT, {0x7f26dc0fbfc6, [], SA_RESTORER, 0x7f26dbe5a2d0}, {0x7f26dc0fbfc6, [], SA_RESTORER, 0x7f26dbe5a2d0}, 8) = 0
10:39:42.356672 clone(child_stack=0x7f26cfb49ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f26cfb4a9d0, tls=0x7f26cfb4a700, child_tidptr=0x7f26cfb4a9d0) = 11084
Any help on how to get rid of these delays would be most welcome.
Finally, and while I'm at it. When I look at strace's remaing output I see, that on startup, ipython tries to open a lot of 'python related' files in places where I would have naively thought that it should never do so, and where my systems python definitely has not been installed? All these many open calls fail. E.g.:
11:11:26.465594 open("/opt/intel/composerxe-2011.3.174/compiler/lib/intel64/tls/libpython2.7.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
11:11:26.465646 stat("/opt/intel/composerxe-2011.3.174/compiler/lib/intel64/tls", 0x7fff0efb0100) = -1 ENOENT (No such file or directory)
or
11:11:26.468293 stat("/usr/local/lib/vtk-5.4/tls/x86_64", 0x7fff0efb0100) = -1 ENOENT (No such file or directory)
11:11:26.468347 open("/usr/local/lib/vtk-5.4/tls/libpython2.7.so.1.0", O_RDONLY) = -1 ENOENT (No such file or directory)
The time total for these calls make up for another significant portion of the remaining slow startup, but I cannot point to a specific time slot. Any idea how to reduce those calls?
1) I suggest not starting with --pylab we encourage people to use %matplotlib once started. the overhead of starting when using --pylabis most probably due to importing packages we don't have access to. There is for example bugs on ArchLinux where each ipython --pylab takes hundreds of MB because of memory leak somewhere not in IPython code.
The access to sockets still seem weird, getting to know exactly where this come from could help.
2) The second one with notebook is probably due to ZMQ and the start of some communication over sockets on localhost. We know that in some occasion it can take some times, but we need also to find out which part of the code is slow to do smth about that. NOtebook also start a webserver so network operation are expected.
3) Is probably due to some library we use, we don't open such tings explicitly in ipython. especially VTK. Do you have a profile that load specific things ?
You could try to open an issue on IPython tracker, but we will probably ask you to dig deeper to know what are the python call in ipython that take so much time. The question of startup time have already been asked between core dev.
The consensus was that this would be "best effort" but we are definitively a too small teem to have this as priority. Still a integration with tool like vbech to track regression would be nice and if someone is motivated to digg into slow call, we'll be glad to do our best to improve things.
这篇关于慢慢的ipython --pylab和ipython笔记本启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
