验证python数据类中的详细类型 [英] Validating detailed types in python dataclasses
问题描述
Python 3.7是在不久前发布的，我想测试一些新的dataclass + typing功能.使用本机类型和typing模块中的本机类型，使提示正确工作很容易:
Python 3.7 was released a while ago, and I wanted to test some of the fancy new dataclass+typing features. Getting hints to work right is easy enough, with both native types and those from the typing module:
>>> import dataclasses
>>> import typing as ty
>>>
... @dataclasses.dataclass
... class Structure:
... a_str: str
... a_str_list: ty.List[str]
...
>>> my_struct = Structure(a_str='test', a_str_list=['t', 'e', 's', 't'])
>>> my_struct.a_str_list[0]. # IDE suggests all the string methods :)
但是我想尝试的另一件事是在运行时强制将类型提示作为条件，即，dataclass不可能存在错误的类型.可以很好地实现 __post_init__ :
But one other thing that I wanted to try was forcing the type hints as conditions during runtime, i.e. it should not be possible for a dataclass with incorrect types to exist. It can be implemented nicely with __post_init__:
>>> @dataclasses.dataclass
... class Structure:
... a_str: str
... a_str_list: ty.List[str]
...
... def validate(self):
... ret = True
... for field_name, field_def in self.__dataclass_fields__.items():
... actual_type = type(getattr(self, field_name))
... if actual_type != field_def.type:
... print(f"\t{field_name}: '{actual_type}' instead of '{field_def.type}'")
... ret = False
... return ret
...
... def __post_init__(self):
... if not self.validate():
... raise ValueError('Wrong types')
这种validate函数适用于本机类型和自定义类，但不适用于typing模块指定的那些类型:
This kind of validate function works for native types and custom classes, but not those specified by the typing module:
>>> my_struct = Structure(a_str='test', a_str_list=['t', 'e', 's', 't'])
Traceback (most recent call last):
a_str_list: '<class 'list'>' instead of 'typing.List[str]'
ValueError: Wrong types
是否有更好的方法来验证具有typing类型的列表的未类型列表?最好不包括检查任何list，dict，tuple或set中所有元素的类型，这些元素都是dataclass'属性.
Is there a better approach to validate an untyped list with a typing-typed one? Preferably one that doesn't include checking the types of all elements in any list, dict, tuple, or set that is a dataclass' attribute.
推荐答案
您应该使用isinstance而不是检查类型是否相等.但是您不能使用参数化的泛型类型(typing.List[int])来执行此操作，必须使用泛型"版本(typing.List).因此，您将能够检查容器类型，而不是所包含的类型.参数化的泛型类型定义了一个__origin__属性，您可以将其用于此目的.
Instead of checking for type equality, you should use isinstance. But you cannot use a parametrized generic type (typing.List[int]) to do so, you must use the "generic" version (typing.List). So you will be able to check for the container type but not the contained types. Parametrized generic types define an __origin__ attribute that you can use for that.
与Python 3.6相反，在Python 3.7中，大多数类型提示都具有有用的__origin__属性.比较:
Contrary to Python 3.6, in Python 3.7 most type hints have a useful __origin__ attribute. Compare:
# Python 3.6
>>> import typing
>>> typing.List.__origin__
>>> typing.List[int].__origin__
typing.List
和
# Python 3.7
>>> import typing
>>> typing.List.__origin__
<class 'list'>
>>> typing.List[int].__origin__
<class 'list'>
Python 3.8通过 typing.get_origin() 自省功能:
Python 3.8 introduce even better support with the typing.get_origin() introspection function:
# Python 3.8
>>> import typing
>>> typing.get_origin(typing.List)
<class 'list'>
>>> typing.get_origin(typing.List[int])
<class 'list'>
值得注意的例外是typing.Any，typing.Union和typing.ClassVar…嗯，任何typing._SpecialForm都没有定义__origin__.幸运的是:
Notable exceptions being typing.Any, typing.Union and typing.ClassVar… Well, anything that is a typing._SpecialForm does not define __origin__. Fortunately:
>>> isinstance(typing.Union, typing._SpecialForm)
True
>>> isinstance(typing.Union[int, str], typing._SpecialForm)
False
>>> typing.get_origin(typing.Union[int, str])
typing.Union
但是参数化类型定义了一个__args__属性，该属性将其参数存储为元组. Python 3.8引入了 typing.get_args() 函数来检索它们:
But parametrized types define an __args__ attribute that store their parameters as a tuple; Python 3.8 introduce the typing.get_args() function to retrieve them:
# Python 3.7
>>> typing.Union[int, str].__args__
(<class 'int'>, <class 'str'>)
# Python 3.8
>>> typing.get_args(typing.Union[int, str])
(<class 'int'>, <class 'str'>)
所以我们可以稍微改进类型检查:
So we can improve type checking a bit:
for field_name, field_def in self.__dataclass_fields__.items():
if isinstance(field_def.type, typing._SpecialForm):
# No check for typing.Any, typing.Union, typing.ClassVar (without parameters)
continue
try:
actual_type = field_def.type.__origin__
except AttributeError:
# In case of non-typing types (such as <class 'int'>, for instance)
actual_type = field_def.type
# In Python 3.8 one would replace the try/except with
# actual_type = typing.get_origin(field_def.type) or field_def.type
if isinstance(actual_type, typing._SpecialForm):
# case of typing.Union[…] or typing.ClassVar[…]
actual_type = field_def.type.__args__
actual_value = getattr(self, field_name)
if not isinstance(actual_value, actual_type):
print(f"\t{field_name}: '{type(actual_value)}' instead of '{field_def.type}'")
ret = False
这不是完美的，因为它不能解释例如typing.ClassVar[typing.Union[int, str]]或typing.Optional[typing.List[int]]，但是它应该可以使事情开始.
This is not perfect as it won't account for typing.ClassVar[typing.Union[int, str]] or typing.Optional[typing.List[int]] for instance, but it should get things started.
下一步是应用此检查的方法.
Next is the way to apply this check.
我将使用装饰器路线，而不是使用__post_init__:它可以用于具有类型提示的任何东西，而不仅是dataclasses:
Instead of using __post_init__, I would go the decorator route: this could be used on anything with type hints, not only dataclasses:
import inspect
import typing
from contextlib import suppress
from functools import wraps
def enforce_types(callable):
spec = inspect.getfullargspec(callable)
def check_types(*args, **kwargs):
parameters = dict(zip(spec.args, args))
parameters.update(kwargs)
for name, value in parameters.items():
with suppress(KeyError): # Assume un-annotated parameters can be any type
type_hint = spec.annotations[name]
if isinstance(type_hint, typing._SpecialForm):
# No check for typing.Any, typing.Union, typing.ClassVar (without parameters)
continue
try:
actual_type = type_hint.__origin__
except AttributeError:
# In case of non-typing types (such as <class 'int'>, for instance)
actual_type = type_hint
# In Python 3.8 one would replace the try/except with
# actual_type = typing.get_origin(type_hint) or type_hint
if isinstance(actual_type, typing._SpecialForm):
# case of typing.Union[…] or typing.ClassVar[…]
actual_type = type_hint.__args__
if not isinstance(value, actual_type):
raise TypeError('Unexpected type for \'{}\' (expected {} but found {})'.format(name, type_hint, type(value)))
def decorate(func):
@wraps(func)
def wrapper(*args, **kwargs):
check_types(*args, **kwargs)
return func(*args, **kwargs)
return wrapper
if inspect.isclass(callable):
callable.__init__ = decorate(callable.__init__)
return callable
return decorate(callable)
用法是
@enforce_types
@dataclasses.dataclass
class Point:
x: float
y: float
@enforce_types
def foo(bar: typing.Union[int, str]):
pass
Appart通过验证上一节中建议的某些类型提示，此方法仍存在一些缺点:
Appart from validating some type hints as suggested in the previous section, this approach still have some drawbacks:
- 类型提示:您可能要使用
inspect.signature代替； -
不是正确类型的默认值未得到验证:
inspect.getfullargspec不考虑使用字符串(class Foo: def __init__(self: 'Foo'): pass)的- type hints using strings (
class Foo: def __init__(self: 'Foo'): pass) are not taken into account byinspect.getfullargspec: you may want to usetyping.get_type_hintsandinspect.signatureinstead; a default value which is not the appropriate type is not validated:
@enforce_type
def foo(bar: int = None):
pass
foo()
不引发任何TypeError.您可能希望结合使用 inspect.Signature.bind inspect.BoundArguments.apply_defaults (如果您要对此进行说明(从而迫使您定义def foo(bar: typing.Optional[int] = None));
does not raise any TypeError. You may want to use inspect.Signature.bind in conjuction with inspect.BoundArguments.apply_defaults if you want to account for that (and thus forcing you to define def foo(bar: typing.Optional[int] = None));
此答案广受欢迎后，受其启发的库发布了，需要消除上述缺点已成为现实.因此，我在typing模块上做了更多的工作，并将在此处提出一些发现和新方法.
After this answer got some popularity and a library heavily inspired by it got released, the need to lift the shortcomings mentioned above is becoming a reality. So I played a bit more with the typing module and will propose a few findings and a new approach here.
对于初学者来说，typing在寻找何时可选参数方面做得很好:
For starter, typing is doing a great job in finding when an argument is optional:
>>> def foo(a: int, b: str, c: typing.List[str] = None):
... pass
...
>>> typing.get_type_hints(foo)
{'a': <class 'int'>, 'b': <class 'str'>, 'c': typing.Union[typing.List[str], NoneType]}
这非常整洁，绝对是对inspect.getfullargspec的改进，因此最好使用它，因为它还可以正确地将字符串作为类型提示来处理.但是typing.get_type_hints会为其他类型的默认值提供援助:
This is pretty neat and definitely an improvement over inspect.getfullargspec, so better use that instead as it can also properly handle strings as type hints. But typing.get_type_hints will bail out for other kind of default values:
>>> def foo(a: int, b: str, c: typing.List[str] = 3):
... pass
...
>>> typing.get_type_hints(foo)
{'a': <class 'int'>, 'b': <class 'str'>, 'c': typing.List[str]}
因此，即使您觉得这种情况非常棘手，您仍可能需要进行更严格的检查.
So you may still need extra strict checking, even though such cases feels very fishy.
接下来是typing提示用作typing._SpecialForm的参数的情况，例如typing.Optional[typing.List[str]]或typing.Final[typing.Union[typing.Sequence, typing.Mapping]].由于这些typing._SpecialForm的__args__始终是一个元组，因此可以递归地找到该元组中包含的提示的__origin__.结合以上检查，我们将需要过滤剩下的任何typing._SpecialForm.
Next is the case of typing hints used as arguments for typing._SpecialForm, such as typing.Optional[typing.List[str]] or typing.Final[typing.Union[typing.Sequence, typing.Mapping]]. Since the __args__ of these typing._SpecialForms is always a tuple, it is possible to recursively find the __origin__ of the hints contained in that tuple. Combined with the above checks, we will then need to filter any typing._SpecialForm left.
拟议的改进:
import inspect
import typing
from functools import wraps
def _find_type_origin(type_hint):
if isinstance(type_hint, typing._SpecialForm):
# case of typing.Any, typing.ClassVar, typing.Final, typing.Literal,
# typing.NoReturn, typing.Optional, or typing.Union without parameters
yield typing.Any
return
actual_type = typing.get_origin(type_hint) or type_hint # requires Python 3.8
if isinstance(actual_type, typing._SpecialForm):
# case of typing.Union[…] or typing.ClassVar[…] or …
for origins in map(_find_type_origin, typing.get_args(type_hint)):
yield from origins
else:
yield actual_type
def _check_types(parameters, hints):
for name, value in parameters.items():
type_hint = hints.get(name, typing.Any)
actual_types = tuple(
origin
for origin in _find_type_origin(type_hint)
if origin is not typing.Any
)
if actual_types and not isinstance(value, actual_types):
raise TypeError(
f"Expected type '{type_hint}' for argument '{name}'"
f" but received type '{type(value)}' instead"
)
def enforce_types(callable):
def decorate(func):
hints = typing.get_type_hints(func)
signature = inspect.signature(func)
@wraps(func)
def wrapper(*args, **kwargs):
parameters = dict(zip(signature.parameters, args))
parameters.update(kwargs)
_check_types(parameters, hints)
return func(*args, **kwargs)
return wrapper
if inspect.isclass(callable):
callable.__init__ = decorate(callable.__init__)
return callable
return decorate(callable)
def enforce_strict_types(callable):
def decorate(func):
hints = typing.get_type_hints(func)
signature = inspect.signature(func)
@wraps(func)
def wrapper(*args, **kwargs):
bound = signature.bind(*args, **kwargs)
bound.apply_defaults()
parameters = dict(zip(signature.parameters, bound.args))
parameters.update(bound.kwargs)
_check_types(parameters, hints)
return func(*args, **kwargs)
return wrapper
if inspect.isclass(callable):
callable.__init__ = decorate(callable.__init__)
return callable
return decorate(callable)
感谢 @ Aran-Fey 帮助我改善了这个答案.
Thanks to @Aran-Fey that helped me improve this answer.
这篇关于验证python数据类中的详细类型的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
