pyinstaller 可执行文件的差异更新(修改嵌入式 PYZ-00.pyz) [英] differential update of pyinstaller executable (modify embedded PYZ-00.pyz)
问题描述
我打算创建一个巨大的可执行目录并将其安装在某些设备上.
I'm planning to create a huge executable directory and install it on some devices.
想象一下,后来我在我的一个 Python 模块中发现了一个错误.有没有什么办法只传输/复制修改后的字节码,然后用新的字节码替换原来的字节码.
Imagine, that lateron I discover a bug in one of my python modules. Is there any way to transfer/copy only the modified byte code and replace the original byte code with the new one.
我想这样做的原因是,在我的上下文中带宽非常昂贵,我想远程修补代码.
The reason I want to do this is, that in my context bandwidth is very expensive and I'd like to patch the code remotely.
示例:我有一个包含两个文件的项目:prog.py
:(以下三行)
Example: I have a project with two files:
prog.py
: (with following three lines)
import mod1
if __name__ == "__main__":
mod1.hello()
mod1.py
:(以下两行)
def hello():
print("hello old world")
现在我使用 PYTHONHASHSEED=2 pyinstaller prog.py
创建我的目录并将其复制到我的设备
Now I use PYTHONHASHSEED=2 pyinstaller prog.py
to create my directory which I copy to my device
现在我修改mod1.py
:
def hello():
print("hello new world")
然后我用 PYTHONHASHSEED=2 pyinstaller prog.py
重新编译完整目录(去皮和压缩)大小约为 10Mdist/prog/prog
文件大小约为 1M
and I recompile with PYTHONHASHSEED=2 pyinstaller prog.py
The full directory has (tared and gzipped) a size of about 10M
The file dist/prog/prog
has a size of about 1M
使用 pyi-archive_viewer
我可以从我的可执行文件 dist/prog/prog
中提取 PYZ-00.pyz
在 PYZ-00.pyz
中,我可以找到并提取仅使用 133 个字节的 mod1
.
with pyi-archive_viewer
I can extract PYZ-00.pyz
out of my executable dist/prog/prog
In PYZ-00.pyz
I can find and extract mod1
which uses only 133 bytes.
现在,如果我将该文件复制到我的设备上,我该如何更新旧的dist/prog/prog
这样,它就有了新的 PYZ-00.pyz:mod1
字节码.
Now if I copy that file to my device, how could I update the old
dist/prog/prog
such, that it has the new PYZ-00.pyz:mod1
byte code.
我可以用什么代码来分解,我可以用什么代码在替换了一个特定的文件(模块)后重新组装?
What code could I use to decompose, what code could I use to reassemble after having replaced one specific file (module)?
替代方法:将 pyc 文件移动到 zip 文件启动性能不是那么重要.我也可以使用替代解决方案,其中没有创建 PYZ 文件并将其添加到可执行文件中,但是 dist 目录包含一个包含所有 .pyc 文件的 zip 文件
Alternative: Move pyc files to a zip file Startup performance is not that crucial. I could also live with an alternative solution, where no PYZ file is created and added to the executable, but where the dist directory contains a zip file with all the .pyc files
另一种选择:将 .pyc 文件复制到应用程序目录中这将导致 __file__
具有与 PYZ 模式完全相同的值.性能方面可能没有那么好,并且会创建大量文件,但如果增量更新至关重要,那么也许可以选择一种处理方式.
Another alternative: copy .pyc files into application directory
This would result in __file__
having exactly the same value as in the PYZ mode. Performance wise probably not that nice and creating a lot of files, but if incremental updates are crucial perhaps one option to handle it.
推荐答案
此解决方案既不能修补".PYZ
文件,也不能放置所有 .pyc
> 将文件压缩为 zip 文件.
This solution is neither capable of 'patching' a .PYZ
file nor capable of putting all .pyc
files into a zip file.
但到目前为止,它是我找到的唯一可行的解决方案,适用于具有大量第三方依赖项的大型项目.
But so far it is the only viable solution I found so far, that works for huge projects with loads of third party dependencies.
这个想法是删除所有(或大部分文件从 .PYZ 文件)并将相应的 .pyc 文件复制到工作目录中.
The idea is to remove all (or most files from the .PYZ file) and copy the corresponding .pyc files into the working directory.
我会随着时间的推移加强和详细说明这个答案.我还在试验中:
I will enhance and elaborate this answer over time. I'm still experimenting:
我通过修改规范文件来实现这一点:
I achieve this by modyfing the spec file:
- 确定spec文件所在的目录
MYDIR
- 创建一个目录,
MYDIR/src
,其中所有来自a.pure
的文件都将被复制到 - 将所有文件从 a.pure 复制到
MYDIR/src
.(具有与模块名称相对应的子目录.例如,模块mypackage.mod.common
将存储在MYDIR/src/mypackage/mod/common.py
中) - 遍历文件并将它们编译为
.pyc
文件,然后删除.py
文件. - 创建一个
PYZ
文件,其中仅包含未复制的文件.(在我的测试用例中,PYZ
中不保留.pyc
文件) - 使用修改后的
PYZ
创建exe - 收集所有应该收集的文件以及来自
MYDIR/src
的所有文件(例如使用a.datas + Tree(src")
- determine the directory
MYDIR
where the spec file is located in - create a directory,
MYDIR/src
where all the files froma.pure
shall be copied to - copy all files from a.pure to to
MYDIR/src
. (with subdirectories corresponding to the module's name. Modulemypackage.mod.common
would for example be stored inMYDIR/src/mypackage/mod/common.py
) - iterate through files and compile them to a
.pyc
file and remove.py
file afterwards. - create a
PYZ
file which contains only the files that are not copied. (in my test case, keep no.pyc
file in thePYZ
) - create exe with the modified
PYZ
- collect all files that should be collected plus also all files from
MYDIR/src
(e.g. witha.datas + Tree("src")
规范文件更改:一开始
import os
MYDIR = os.path.realpath(SPECPATH)
sys.path.append(MYDIR)
import mypyinsthelpers # allows to reuse the code in multiple projects
然后在(未修改的)a = Analysis(...
部分之后我添加.
Then after the (unmodified) a = Analysis(...
section I add.
to_rmv_from_pyc = mypyinsthelpers.mk_copy_n_compile(a.pure, MYDIR)
# modified creation of pyz`
pyz = PYZ(a.pure - to_rmv_from_pyc, a.zipped_data,
cipher=block_cipher)
我会进一步详细说明mypyinsthelpers.mk_copy_n_compile
函数
I will detail the function mypyinsthelpers.mk_copy_n_compile
further down
更改收集阶段:
代替
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
...
我写道:
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas + Tree("src"),
...
这里是mypyinsthelpers.mk_copy_n_compile()
import compileall
import os
import shutil
from pathlib import Path
def mk_copy_n_compile(toc, src_tree):
"""
- copy source files to a destination directory
- compile them as pyc
- delete source
"""
dst_base_path = os.path.join(src_tree, "src")
to_rm = []
# copy files to destination tree
for entry in toc:
modname, src, typ = entry
assert typ == "PYMODULE"
assert src.endswith(".py") or src.endswith(".pyw")
# TODO: might add logic to skip some files (keep them in PYC)
to_rm.append(entry)
if src.endswith("__init__.py"):
modname += ".__init__"
m_split = modname.split(".")
m_split[-1] += ".py"
dst_dir = os.path.join(dst_base_path, *m_split[:-1])
dst_path = os.path.join(dst_dir, m_split[-1])
if not os.path.isdir(dst_dir):
os.makedirs(dst_dir)
print(entry[:2], dst_path)
shutil.copy(src, dst_path)
# now compile all files and rmv src
top_tree = src_tree
src_tree = os.path.join(src_tree, "src")
curdir = os.getcwd()
os.chdir(dst_base_path)
for path in Path(dst_base_path).glob("**/*.py"):
# TODO: might add code to keep some files as source
compileall.compile_file(
str(path.relative_to(dst_base_path)), quiet=1, legacy=True)
path.unlink()
os.chdir(curdir)
return to_rm
这篇关于pyinstaller 可执行文件的差异更新(修改嵌入式 PYZ-00.pyz)的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!