如何清除numba中的缓存(或强制重新编译) [英] How to clear cache (or force recompilation) in numba

问题描述

我有一个相当大的用numba编写的代码库，并且我注意到，当启用了在另一个文件中调用另一个numba编译函数的函数的缓存时，在更改被调用函数时不会拾取被调用函数中的更改.当我有两个文件时会发生这种情况:

I have a fairly large codebase written in numba, and I have noticed that when the cache is enabled for a function calling another numba compiled function in another file, changes in the called function are not picked up when the called function is changed. The situation occurs when I have two files:

testfile2:

testfile2:

import numba

@numba.njit(cache=True)
def function1(x):
    return x * 10

测试文件:

import numba
from tests import file1

@numba.njit(cache=True)
def function2(x, y):
    return y + file1.function1(x)

如果在jupyter笔记本中，则运行以下命令:

If in a jupyter notebook, I run the following:

# INSIDE JUPYTER NOTEBOOK
import sys
sys.path.insert(1, "path/to/files/")
from tests import testfile

testfile.function2(3, 4)
>>> 34   # good value

但是，如果我进行更改，则将testfile2更改为以下内容:

However, if I change then change testfile2 to the following:

import numba

@numba.njit(cache=True)
def function1(x):
    return x * 1

然后我重新启动jupyter笔记本内核并重新运行笔记本，我得到以下信息

Then I restart the jupyter notebook kernel and rerun the notebook, I get the following

import sys
sys.path.insert(1, "path/to/files/")
from tests import testfile

testfile.function2(3, 4)
>>> 34   # bad value, should be 7

将两个文件同时导入笔记本电脑对不良结果没有影响.另外，仅在 function1 上设置 cache = False 也不起作用.起作用的是在所有njit'ted函数上设置 cache = False ，然后重新启动内核，然后重新运行.

Importing both files into the notebook has no effect on the bad result. Also, setting cache=False only on function1 also has no effect. What does work is setting cache=False on all njit'ted functions, then restarting the kernel, then rerunning.

我相信LLVM可能会内联一些被调用的函数，然后再也不检查它们.

I believe that LLVM is probably inlining some of the called functions and then never checking them again.

我查看了源代码，发现有一种方法可以返回缓存对象 numba.caching.NullCache()，实例化一个缓存对象并运行以下命令:

I looked in the source and discovered there is a method that returns the cache object numba.caching.NullCache(), instantiated a cache object and ran the following:

cache = numba.caching.NullCache()
cache.flush()

不幸的是，这似乎没有效果.

Unfortunately that appears to have no effect.

是否存在numba环境设置，或者可以通过其他方式手动清除conda env中的所有缓存功能?还是我只是做错了什么?

Is there a numba environment setting, or another way I can manually clear all cached functions within a conda env? Or am I simply doing something wrong?

我在Mac OS X 10.12.3.上使用Anaconda Python 3.6运行numba 0.33.

I am running numba 0.33 with Anaconda Python 3.6 on Mac OS X 10.12.3.

推荐答案

在看到Josh的回答后，我通过在项目方法中创建了一个实用程序来杀死缓存来通过黑客解决方案解决"了这个问题.

I "solved" this with a hack solution after seeing Josh's answer, by creating a utility in the project method to kill off the cache.

可能有更好的方法，但这可行.如果有人使用这种方法不太笨拙，我将保留这个问题.

There is probably a better way, but this works. I'm leaving the question open in case someone has a less hacky way of doing this.

import os


def kill_files(folder):
    for the_file in os.listdir(folder):
        file_path = os.path.join(folder, the_file)
        try:
            if os.path.isfile(file_path):
                os.unlink(file_path)
        except Exception as e:
            print("failed on filepath: %s" % file_path)


def kill_numba_cache():

    root_folder = os.path.realpath(__file__ + "/../../")

    for root, dirnames, filenames in os.walk(root_folder):
        for dirname in dirnames:
            if dirname == "__pycache__":
                try:
                    kill_files(root + "/" + dirname)
                except Exception as e:
                    print("failed on %s", root)

这篇关于如何清除numba中的缓存(或强制重新编译)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！