python3的编码问题并单击包 [英] Encoding issue with python3 and click package

问题描述

当 lib click 检测到运行时是 python3 但编码是ASCII 然后它突然结束 python 程序:

When the lib click detects that the runtime is python3 but the encoding is ASCII then it ends the python program abruptly:

RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Either switch to Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.

我在我的案例中找到了这个问题的原因，当我从我的 Mac 连接到我的 Linux 主机时，Terminal.app 将 SSH 会话区域设置设置为我的 Mac 区域设置 (es_ES.UTF-8) 但是我的 Linux 主机没有'没有安装这样的语言环境(仅 en_US.utf-8).

I found the cause of this issue in my case, when I connect to my Linux host from my Mac, the Terminal.app set the SSH session locale to my Mac locale (es_ES.UTF-8) However my Linux host hasn't installed such locale (only en_US.utf-8).

我应用了一个初步的解决方法来修复它(但它有很多问题，请参阅已接受的答案):

I applied an initial workaround to fix it (but It had many issues, see accepted answer):

import locale, codecs
# locale.getpreferredencoding() == 'ANSI_X3.4-1968'
if codecs.lookup(locale.getpreferredencoding()).name == 'ascii':
    os.environ['LANG'] = 'en_US.utf-8'

有关更好的补丁，请参阅我接受的答案.

我所有的 linux 主机都安装了en_US.utf-8"作为语言环境(Fedora 默认使用它).

All my linux hosts have installed 'en_US.utf-8' as locale (Fedora uses it as default).

我的问题是:是否有更好(更健壮)的方式在 python3 脚本中选择/强制语言环境?例如，设置系统中可用的语言环境之一.

My question is: Is there a better (more robust) way to choose/force the locale in a python3 script ? For instance, setting one of the available locales in the system.

也许有其他方法可以解决此问题，但我没有找到.

Maybe there is a different approach to fix this issue but I didn't find it.

推荐答案

如果你有 python 版本 >= 3.7，那么你应该不需要做任何事情.如果您有 python 3.6，请参阅原始解决方案.

If you have python version >= 3.7, then you should not need to do anything. If you have python 3.6 see the original solution.

我看到py3.7有一个PEP 538，它会在启动时改变python3编码管理的整个行为，我认为新的方法会解决原来的问题:https://www.python.org/dev/peps/pep-0538/

I've seen that there is a PEP 538 for py3.7, that will change the entire behavior of python3 encoding management during startup, I think that the new approach will fix the original problem: https://www.python.org/dev/peps/pep-0538/

恕我直言，针对 python 3.7 编码问题的更改应该在几年前就计划好了，但我猜迟到总比没有好.

IMHO the changes targeted to python 3.7 for encoding issues, should have been planed years ago, but better late than never, I guess.

有一个未解决的问题(增强)，http://bugs.python.org/issue15216，这将允许轻松更改已创建(未使用)流中的编码(sys.std*).但是是针对python 3.7所以，我们还得等一会儿.

There is an opened issue (enhancement), http://bugs.python.org/issue15216, that will allow to change the encoding in a created (not-used) stream easily (sys.std*). But is targeted to python 3.7 So, we'll have to wait for a while.

注意:运行 python 版本 >= 3.7 的任何人都不需要此解决方案，请参阅 PEP 538

嗯，我最初的解决方法有很多缺陷，我必须通过关于编码的 click 库检查，但是编码本身不是固定的，所以当输入参数或输出有异常时我会得到异常非 ASCII 字符.

Well, my initial workaround had many flaws, I got to pass the click library check about the encoding, but the encoding itself was not fixed, so I get exceptions when the input parameters or output had non-ascii characters.

我必须实现一个更复杂的方法，包括 3 个步骤:设置语言环境、在 std 输入/输出中正确编码并重新编码命令行参数，此外我添加了一个友好"的如果第一次尝试设置语言环境没有按预期工作，则退出:

I had to implement a more complex method, with 3 steps: set locale, correct encoding in std in/out and re-encode the command line parameters, besides I've added a "friendly" exit if the first try to set the locale doesn't work as expected:

def prevent_ascii_env():
    """
    To avoid issues reading unicode chars from stdin or writing to stdout, we need to ensure that the 
    python3 runtime is correctly configured, if not, we try to force to utf-8, 
    but It isn't possible then we exit with a more friendly message that the original one.
    """
    import locale, codecs, os, sys
    # locale.getpreferredencoding() == 'ANSI_X3.4-1968'
    if codecs.lookup(locale.getpreferredencoding()).name == 'ascii':
        os.environ['LANG'] = 'en_US.utf-8'
        if codecs.lookup(locale.getpreferredencoding()).name == 'ascii':
            print("The current locale is not correctly configured in your system")
            print("Please set the LANG env variable to the proper value before to call this script")
            sys.exit(-1)
        #Once we have the proper locale.getpreferredencoding() We can change current stdin/out streams
        _, encoding = locale.getdefaultlocale()
        import io
        sys.stderr = io.TextIOWrapper(sys.stderr.detach(), encoding=encoding, errors="replace", line_buffering=True)
        sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding=encoding, errors="replace", line_buffering=True)
        sys.stdin = io.TextIOWrapper(sys.stdin.detach(), encoding=encoding, errors="replace", line_buffering=True)
        # And finally we need to re-encode the input parameters
        for i, p in enumerate(sys.argv):
            sys.argv[i] = os.fsencode(p).decode() 

这个补丁几乎解决了所有问题，但是它有一个警告，方法 shutils.get_terminal_size() 引发了 ValueError 因为 sys.__stdout__ 已分离，click lib 使用该方法打印帮助，要修复它，我必须在 click lib

This patch solves almost all issues, however it has a caveat, the method shutils.get_terminal_size() raises a ValueError because the sys.__stdout__ has been detached, click lib uses that method to print the help, to fix it I had to apply a monkey-patch on click lib

def wrapper_get_terminal_size():
    """
    Replace the original function termui.get_terminal_size (click lib) by a new one 
    that uses a fallback if ValueError exception has been raised
    """
    from click import termui, formatting
    
    old_get_term_size = termui.get_terminal_size
    def _wrapped_get_terminal_size():
        try:
            return old_get_term_size()
        except ValueError:
            import os
            sz = os.get_terminal_size()
            return sz.columns, sz.lines
    termui.get_terminal_size = _wrapped_get_terminal_size
    formatting.get_terminal_size = _wrapped_get_terminal_size

通过这些更改，当环境配置了错误的语言环境但系统支持 en_US.utf-8(这是 Fedora 的默认语言环境)时，我的所有脚本现在都可以正常工作了.

With this changes all my scripts work fine now when the environment has a wrong locale configured but the system supports en_US.utf-8 (It's the Fedora default locale).

如果您发现此方法有任何问题或有更好的解决方案，请添加新答案.

If you find any issue on this approach or have a better solution, please add a new answer.

这篇关于python3的编码问题并单击包的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！