Python pip从pip冻结的输出中找出基本要求 [英] Python pip find out basic requirements from output of pip freeze
问题描述
我的朋友刚刚开始学习Python和Flask，并且缺少许多最佳实践"，例如requirements.txt文件.
My friend just started learning Python and Flask, and is missing a lot of "best practices", e.g., a requirements.txt file.
他最近要求我提供帮助，并且为了使项目更整洁，我想设置一个CI服务(Travis)，但是我需要首先编制此文件.
He has recently asked me for assistance, and to make the project clean, I want to setup a CI service (Travis), but I need to work out this file first.
由于他最初没有requirements.txt，所以我能获得的所有信息都是他的import语句以及他的pip freeze输出.
Since he did not initially have a requirements.txt, all information I can have is his import statements, as well as his output of pip freeze.
由于无法区分项目的直接需求和其中一个软件包的间接需求，因此我想从列表中找出所有顶级"软件包. 顶级程序包"是列表中其他程序包不需要的程序包.例如，requests需要urllib3，因此，当存在requests时，urllib3最好不要出现在最终结果中.
As there's no way to distinguish a direct requirement by the project and an indirect requirement by one of the packages, I want to find out all "top-level" packages from the list. A "top-level package" is a package that's not required by another package in the list. For example, urllib3 is required by requests, so when requests is present, urllib3 may better not appear in the final result.
有没有办法做到这一点?
Is there a way to achieve this?
如果有人想帮助我解决这个特定实例，这是pip freeze的输出:
If anyone wants to help me with this specific instance, here's the output of pip freeze:
apturl==0.5.2
arrow==0.12.1
asn1crypto==0.24.0
binaryornot==0.4.4
blinker==1.4
Bootstrap-Flask==1.0.9
Brlapi==0.6.6
certifi==2018.1.18
chardet==3.0.4
Click==7.0
colorama==0.3.7
command-not-found==0.3
configparser==3.5.0
cookiecutter==1.6.0
cryptography==2.1.4
cupshelpers==1.0
decorator==4.1.2
defer==1.0.6
distro-info==0.18
dominate==2.3.5
Flask==1.0.2
Flask-Bootstrap4==4.0.2
Flask-Login==0.4.1
Flask-Mail==0.9.1
Flask-Moment==0.6.0
Flask-SQLAlchemy==2.3.2
Flask-WTF==0.14.2
future==0.17.1
httpie==0.9.8
httplib2==0.9.2
idna==2.6
ipython==5.5.0
ipython-genutils==0.2.0
itsdangerous==1.1.0
Jinja2==2.10
jinja2-time==0.2.0
keyring==10.6.0
keyrings.alt==3.0
language-selector==0.1
launchpadlib==1.10.6
lazr.restfulclient==0.13.5
lazr.uri==1.0.3
louis==3.5.0
macaroonbakery==1.1.3
Mako==1.0.7
MarkupSafe==1.1.0
mysqlclient==1.3.14
netifaces==0.10.4
oauth==1.0.1
olefile==0.45.1
pexpect==4.2.1
pickleshare==0.7.4
Pillow==5.1.0
poyo==0.4.2
prompt-toolkit==1.0.15
protobuf==3.0.0
pycairo==1.16.2
pycrypto==2.6.1
pycups==1.9.73
Pygments==2.2.0
pygobject==3.26.1
pymacaroons==0.13.0
PyNaCl==1.1.2
pyRFC3339==1.0
python-apt==1.6.3
python-dateutil==2.7.5
python-debian==0.1.32
pytz==2018.3
pyxdg==0.25
PyYAML==3.12
reportlab==3.4.0
requests==2.18.4
requests-unixsocket==0.1.5
ruamel.yaml==0.15.34
SecretStorage==2.3.1
simplegeneric==0.8.1
simplejson==3.13.2
six==1.11.0
SQLAlchemy==1.2.14
system-service==0.3
systemd-python==234
traitlets==4.3.2
ubuntu-drivers-common==0.0.0
ufw==0.35
unattended-upgrades==0.1
urllib3==1.22
usb-creator==0.3.3
visitor==0.1.3
wadllib==1.3.2
wcwidth==0.1.7
Werkzeug==0.14.1
whichcraft==0.5.2
WTForms==2.2.1
xkit==0.0.0
zope.interface==4.3.2
这是import陈述，还有他告诉我的另外一个pymysql.
and here are the import statements, with an additional pymysql he told me.
import os
from flask import *
from flask_bootstrap import Bootstrap
from flask_moment import Moment
from flask_wtf import FlaskForm
from wtforms import *
from wtforms.validators import *
from flask_sqlalchemy import SQLAlchemy
from flask_mail import Mail, Message
from werkzeug.security import generate_password_hash,check_password_hash
from flask_login import login_required , login_user,login_fresh,login_url,LoginManager,UserMixin,logout_user
推荐答案
首先，我想建议使用 pip 的 API ，但建议使用 > pip 仅作为 cmdline 工具(
First, I wanted to suggest using pip's API, but it's recommended to use pip as a cmdline tool only ([PyPA]: Using pip from your program). Note that I successfully used it, I just don't expose the code (at least for now).
Here's a way that uses pkg_resources ([ReadTheDocs]: Package Discovery and Resource Access using pkg_resources).
code.py :
#!/usr/bin/env python3
import sys
import os
import pkg_resources
def get_pkgs(reqs_file="requirements_orig.txt"):
if reqs_file and os.path.isfile(reqs_file):
ret = dict()
with open(reqs_file) as f:
for item in f.readlines():
name, ver = item.strip("\n").split("==")[:2]
ret[name] = ver, ()
return ret
else:
return {
item.project_name: (item.version, tuple([dep.name for dep in item.requires()])) for item in pkg_resources.working_set
}
def print_pkg_data(text, pkg_info):
print("{:s}\nSize: {:d}\n\n{:s}".format(text, len(pkg_info), "\n".join(["{:s}=={:s}".format(*item) for item in pkg_info])))
def main():
pkgs = get_pkgs(reqs_file=None)
full_pkg_info = [(name, data[0]) for name, data in sorted(pkgs.items())]
print_pkg_data("----------FULL LIST----------", full_pkg_info)
deps = set()
for name in pkgs:
deps = deps.union(pkgs[name][1])
min_pkg_info = [(name, data[0]) for name, data in sorted(pkgs.items()) if name not in deps]
print_pkg_data("\n----------MINIMAL LIST----------", min_pkg_info)
if __name__ == "__main__":
print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
main()
输出:
(py_064_03.06.08_test0) e:\Work\Dev\StackOverflow\q054292236>"e:\Work\Dev\VEnvs\py_064_03.06.08_test0\Scripts\python.exe" code.py
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
----------FULL LIST----------
Size: 133
Babel==2.6.0
Click==7.0
Django==2.1.4
Flask==1.0.2
Jinja2==2.10
Keras==2.2.4
Keras-Applications==1.0.6
Keras-Preprocessing==1.0.5
Markdown==3.0.1
MarkupSafe==1.1.0
Pillow==5.3.0
PyQt5==5.9.2
PyQt5-sip==4.19.13
PyYAML==3.13
Pygments==2.3.1
QtAwesome==0.5.3
QtPy==1.5.2
Send2Trash==1.5.0
Sphinx==1.8.3
Werkzeug==0.14.1
absl-py==0.6.1
alabaster==0.7.12
asn1crypto==0.24.0
astor==0.7.1
astroid==2.1.0
backcall==0.1.0
bleach==3.0.2
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
cloudpickle==0.6.1
colorama==0.4.1
cryptography==2.4.2
cycler==0.10.0
decorator==4.3.0
defusedxml==0.5.0
djangorestframework==3.9.0
docutils==0.14
entrypoints==0.2.3
fatiando==0.5
funcsigs==1.0.2
future==0.17.1
gast==0.2.0
grpcio==1.17.1
h5py==2.9.0
html5lib==1.0.1
idna==2.8
imagesize==1.1.0
ipaddr==2.2.0
ipykernel==5.1.0
ipython==7.2.0
ipython-genutils==0.2.0
ipywidgets==7.4.2
isort==4.3.4
itsdangerous==1.1.0
jedi==0.13.2
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.4
jupyter-console==6.0.0
jupyter-core==4.4.0
keyboard==0.13.2
keyring==17.1.1
kiwisolver==1.0.1
lazy-object-proxy==1.3.1
llvmlite==0.26.0
lxml==4.2.5
matplotlib==3.0.2
mccabe==0.6.1
mistune==0.8.4
nbconvert==5.4.0
nbformat==4.4.0
notebook==5.7.4
numba==0.41.0
numpy==1.15.4
numpydoc==0.8.0
opencv-python==3.4.4.19
packaging==18.0
pandas==0.23.4
pandocfilters==1.4.2
parso==0.3.1
patsy==0.5.1
pickleshare==0.7.5
pip==18.1
prometheus-client==0.5.0
prompt-toolkit==2.0.7
protobuf==3.6.1
psutil==5.4.8
pyOpenSSL==18.0.0
pycodestyle==2.4.0
pycparser==2.19
pycryptodome==3.7.2
pyflakes==2.0.0
pygame==1.9.4
pylint==2.2.2
pynput==1.4
pyparsing==2.3.0
python-dateutil==2.7.5
pytz==2018.7
pywin32==224
pywin32-ctypes==0.2.0
pywinpty==0.5.5
pyzmq==17.1.2
qtconsole==4.4.3
requests==2.21.0
rope==0.11.0
scapy==2.4.0
scipy==1.2.0
setuptools==40.6.3
sip==4.19.8
six==1.12.0
snowballstemmer==1.2.1
sphinxcontrib-websupport==1.1.0
spyder==3.3.2
spyder-kernels==0.3.0
statsmodels==0.9.0
tensorboard==1.12.1
tensorflow-gpu==1.12.0
tensorflow-tensorboard==1.5.1
termcolor==1.1.0
terminado==0.8.1
testpath==0.4.2
thrift==0.11.0
tornado==5.1.1
traitlets==4.3.2
typed-ast==1.1.1
urllib3==1.24.1
wcwidth==0.1.7
webencodings==0.5.1
wheel==0.32.3
widgetsnbextension==3.4.2
wrapt==1.10.11
xlrd==1.2.0
----------MINIMAL LIST----------
Size: 37
Babel==2.6.0
Click==7.0
Django==2.1.4
Flask==1.0.2
Keras==2.2.4
Keras-Applications==1.0.6
Keras-Preprocessing==1.0.5
Markdown==3.0.1
Pillow==5.3.0
PyQt5==5.9.2
PyQt5-sip==4.19.13
PyYAML==3.13
QtAwesome==0.5.3
QtPy==1.5.2
Sphinx==1.8.3
djangorestframework==3.9.0
fatiando==0.5
funcsigs==1.0.2
ipaddr==2.2.0
keyboard==0.13.2
lxml==4.2.5
opencv-python==3.4.4.19
pandas==0.23.4
patsy==0.5.1
pip==18.1
pyOpenSSL==18.0.0
pycryptodome==3.7.2
pygame==1.9.4
pynput==1.4
pywin32==224
scapy==2.4.0
spyder==3.3.2
statsmodels==0.9.0
tensorflow-gpu==1.12.0
tensorflow-tensorboard==1.5.1
thrift==0.11.0
xlrd==1.2.0
注释:
- (表示显而易见):为了获取 pkg 信息，需要安装 pkg .这就是为什么在我的示例中我没有使用您的文件(我将其命名为 requirements_orig.txt )，而是将 pkg 安装在我的 VEnv 上的原因
- 如您所见，在我的案例中， pkg 编号从 133 降为 37 ，我想说它非常易于管理(当然，可以进行更多过滤)
- 我基于 pkg 名称是 主键 (唯一标识 pkg ).如果这是 false ，则代码需要进行一些更改
- (Stating the obvious): In order to get a pkg info, that pkg needs to be installed. That's why in my example I didn't used your file (I named it requirements_orig.txt), but the pkgs installed on my VEnv
- As you can see, in my case the pkg number dropped from 133 to 37, which I'd say it's pretty manageable (of course, more filtering can be done)
- I created the data structures based on the assumption that a pkg name is a primary key (uniquely identifies a pkg). If this is false, the code would require a bit of change
最后的记录:如果您还想考虑模块的 import 列表(如果可能的话，剥离更多的 pkg )，您还可以尝试 [Python 3]:modulefinder-查找脚本使用的模块(我在
Final note: If you also want to consider your module's import list (to strip out even more pkgs, if possible), you could also try [Python 3]: modulefinder - Find modules used by a script (I used it in [SO]: What files are required for Py_Initialize to run? (@CristiFati's answer), only from cmdline, but it should be trivial to use it from a script)
这篇关于Python pip从pip冻结的输出中找出基本要求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
