你能在 Python 的语法中添加新的语句吗? [英] Can you add new statements to Python's syntax?
问题描述
你能在 Python 的语法中添加新的语句(比如 print、raise、with)吗?
说，允许..
mystatement "Something"或者，
new_if 真:打印示例"如果你应该的话，不是那么多，而是如果可能的话(没有修改python解释器代码)
您可能会发现这很有用 - Python 内部结构:向 Python 添加新语句，这里引用:
<小时>本文试图更好地理解 Python 的前端是如何工作的.仅仅阅读文档和源代码可能有点无聊，所以我在这里采取了动手的方法:我将向 Python 添加一个 until 语句.
本文的所有编码都是针对 Python Mercurial 中的尖端 Py3k 分支完成的仓库镜像.
until 语句
有些语言，比如 Ruby，有一个 until 语句，它是对 while 的补充(until num == 0 等价于while num != 0).在 Ruby 中，我可以这样写:
num = 3直到 num == 0 做放置数数量 -= 1结尾它会打印:
321所以，我想为 Python 添加一个类似的功能.也就是说，能够写:
num = 3直到 num == 0:打印(数量)数量 -= 1语言倡导离题
本文不打算建议在 Python 中添加 until 语句.虽然我认为这样的声明会使一些代码更清晰，并且本文展示了添加是多么容易，但我完全尊重 Python 的极简主义哲学.我在这里真正想做的就是深入了解 Python 的内部工作原理.
修改语法
Python 使用名为 pgen 的自定义解析器生成器.这是一个 LL(1) 解析器，可将 Python 源代码转换为解析树.解析器生成器的输入是文件 Grammar/Grammar[1].这是一个简单的文本文件，指定了 Python 的语法.
[1]:从这里开始，对 Python 源代码中文件的引用相对于源代码树的根目录给出，这是您运行 configure 和 make 以构建 Python 的目录.
必须对语法文件进行两次修改.首先是为 until 语句添加定义.我找到了while 语句的定义位置(while_stmt)，并在[2] 下方添加了until_stmt:
compound_stmt: if_stmt |while_stmt |直到_stmt |for_stmt |try_stmt |with_stmt |功能定义 |类定义 |装饰的if_stmt: 'if' test ':'suite ('elif' test ':'suite)* ['else' ':'suite]while_stmt: 'while' test ':' 套件 ['else' ':' 套件]直到_stmt:'直到'测试':'套件[2]:这展示了我在修改我不熟悉的源代码时使用的一种常用技术:通过相似性工作.这个原则不能解决你所有的问题，但它绝对可以简化这个过程.由于必须为 while 完成的所有操作也必须为 until 完成，因此它是一个很好的指导方针.
请注意，我已决定从我对 until 的定义中排除 else 子句，只是为了让它有所不同(坦率地说，我不喜欢 untilcode>else 循环的子句，并且认为它不适合 Python 的 Zen).
第二个更改是修改 compound_stmt 的规则以包含 until_stmt，如您在上面的代码段中所见.又是在 while_stmt 之后.
修改Grammar/Grammar后运行make时，注意运行pgen程序重新生成Include/graminit.h 和 Python/graminit.c，然后重新编译几个文件.
修改AST生成代码
Python 解析器创建解析树后，这棵树被转换为 AST，因为 AST 是 在编译过程的后续阶段使用起来要简单得多.
所以，我们将访问定义 Python AST 结构的 Parser/Python.asdl，并为我们的新 until 语句添加一个 AST 节点，再次正确while 下方:
<代码>|While(expr test, stmt* body, stmt* orelse)|直到(expr 测试，stmt* 正文)
如果您现在运行 make，请注意在编译一堆文件之前，会运行 Parser/asdl_c.py 以从 AST 定义文件生成 C 代码.这(如 Grammar/Grammar)是 Python 源代码使用迷你语言(换句话说，DSL)来简化编程的另一个示例.另请注意，由于 Parser/asdl_c.py 是 Python 脚本，因此这是一种 引导 - 要从头开始构建 Python，Python 必须可用.
虽然 Parser/asdl_c.py 生成了代码来管理我们新定义的 AST 节点(进入文件 Include/Python-ast.h 和 Python/Python-ast.c)，我们仍然需要手动编写将相关解析树节点转换为它的代码.这是在文件 Python/ast.c 中完成的.在那里，一个名为 ast_for_stmt 的函数将语句的解析树节点转换为 AST 节点.再一次，在我们的老朋友 while 的指导下，我们直接进入大的 switch 来处理复合语句，并为 until_stmt 添加一个子句:
case while_stmt:返回 ast_for_while_stmt(c, ch);案例直到_stmt:返回 ast_for_until_stmt(c, ch);现在我们应该实现ast_for_until_stmt.这是:
静态stmt_tyast_for_until_stmt(struct 编译 *c, const 节点 *n){/*直到_stmt:'直到'测试':'套件*/请求(n，直到_stmt)；如果(NCH(n)== 4){expr_ty 表达式；asdl_seq *suite_seq;表达式 = ast_for_expr(c, CHILD(n, 1));如果(！表达式)返回空；Suite_seq = ast_for_suite(c, CHILD(n, 3));如果(！suite_seq)返回空；返回直到(表达式，suite_seq，LINENO(n)，n->n_col_offset，c->c_arena)；}PyErr_Format(PyExc_SystemError,"'until' 语句的标记数错误:%d",NCH(n));返回空；}同样，这是在仔细查看等效的 ast_for_while_stmt 时编码的，不同之处在于对于 until 我决定不支持 else 子句.正如预期的那样，AST 是递归创建的，使用其他 AST 创建函数，如用于条件表达式的 ast_for_expr 和用于 until 语句主体的 ast_for_suite.最后，返回一个名为 Until 的新节点.
请注意，我们使用诸如NCH 和CHILD 之类的宏来访问解析树节点n.这些都值得理解——它们的代码在 Include/node.h 中.
题外话:AST 组合
我选择为 until 语句创建一种新类型的 AST，但实际上这不是必需的.我本可以节省一些工作并使用现有 AST 节点的组合来实现新功能，因为:
直到条件:# 做东西在功能上等同于:
虽然不是条件:# 做东西不是在 ast_for_until_stmt 中创建 Until 节点，我可以用 While 创建一个 Not 节点作为孩子的节点.由于 AST 编译器已经知道如何处理这些节点，因此可以跳过该过程的后续步骤.
将 AST 编译成字节码
下一步是将 AST 编译成 Python 字节码.编译有一个中间结果，它是一个 CFG(控制流图)，但由于相同的代码处理它，我现在将忽略这个细节并将其留给另一篇文章.
我们接下来要看的代码是Python/compile.c.在while的引导下，我们找到了函数compiler_visit_stmt，它负责将语句编译成字节码.我们为 Until 添加一个子句:
case While_kind:返回 compiler_while(c, s);案例直到_kind:返回 compiler_until(c, s);如果您想知道 Until_kind 是什么，它是一个常量(实际上是 _stmt_kind 枚举的值)，从 AST 定义文件自动生成到 Include/Python-ast.h.无论如何，我们调用 compiler_until ，当然，它仍然不存在.我待会儿再说.
如果你和我一样好奇，你会注意到 compiler_visit_stmt 很特别.没有多少 grep-ping 源树显示它被调用的位置.在这种情况下，只剩下一个选项 - C macro-fu.事实上，一个简短的调查将我们引导到 Python/compile.c 中定义的 VISIT 宏:
#define VISIT(C, TYPE, V) {\if (!compiler_visit_ ## TYPE((C), (V))) \返回0；\用于调用compiler_body中的compiler_visit_stmt.回到我们的业务，但是...
正如承诺的那样，这是compiler_until:
静态整数compiler_until(struct compiler *c, stmt_ty s){basicblock *loop, *end, *anchor = NULL;int 常量 = expr_constant(s->v.Until.test);如果(常数 == 1){返回 1；}loop = compiler_new_block(c);end = compiler_new_block(c);如果(常数 == -1){锚点 = compiler_new_block(c);如果(锚== NULL)返回0；}if (loop == NULL || end == NULL)返回0；ADDOP_JREL(c, SETUP_LOOP, end);compiler_use_next_block(c, 循环);if (!compiler_push_fblock(c, LOOP, loop))返回0；如果(常数 == -1){VISIT(c, expr, s->v.Until.test);ADDOP_JABS(c, POP_JUMP_IF_TRUE, 锚点);}VISIT_SEQ(c, stmt, s->v.Until.body);ADDOP_JABS(c, JUMP_ABSOLUTE, 循环);如果(常数 == -1){compiler_use_next_block(c, 锚点);添加 (c, POP_BLOCK);}compiler_pop_fblock(c, LOOP, loop);compiler_use_next_block(c, end);返回 1；}我必须承认:这段代码不是基于对 Python 字节码的深入理解而编写的.与本文的其余部分一样，它是模仿 kin compiler_while 函数完成的.但是，通过仔细阅读，请记住 Python VM 是基于堆栈的，并查看 dis 模块的文档，该模块具有 Python 字节码列表 带有描述，可以理解发生了什么.
就这样，我们完成了...不是吗?
在进行所有更改并运行 make 后，我们可以运行新编译的 Python 并尝试新的 until 语句:
瞧，它起作用了！让我们看看使用 dis 模块为 new 语句创建的字节码如下:
导入文件def myfoo(num):直到 num == 0:打印(数量)数量 -= 1dis.dis(myfoo)结果如下:
4 0 SETUP_LOOP 36(到 39)>>3 LOAD_FAST 0 (数量)6 LOAD_CONST 1 (0)9 COMPARE_OP 2 (==)12 POP_JUMP_IF_TRUE 385 15 LOAD_NAME 0(打印)18 LOAD_FAST 0 (数量)21 CALL_FUNCTION 124 POP_TOP6 25 LOAD_FAST 0 (数量)28 负载常量 2 (1)31 INPLACE_SUBTRACT32 STORE_FAST 0 (数量)35 JUMP_ABSOLUTE 3>>38 POP_BLOCK>>39 LOAD_CONST 0 (无)42 RETURN_VALUE最有趣的操作是数字 12:如果条件为真，我们跳转到循环之后.这是 until 的正确语义.如果没有执行跳转，循环体将继续运行，直到它跳回到操作 35 处的条件.
对我的更改感觉良好，然后我尝试运行该函数(执行 myfoo(3))而不是显示其字节码.结果并不令人鼓舞:
回溯(最近一次调用最后一次):文件zy.py"，第 9 行，在myfoo(3)文件zy.py"，第 5 行，在 myfoo 中打印(数量)SystemError:加载打印"时没有本地人哇...这不可能是好事.那么出了什么问题?
缺少符号表的情况
Python 编译器在编译 AST 时执行的步骤之一是为其编译的代码创建一个符号表.PyAST_Compile 中对 PySymtable_Build 的调用调用符号表模块 (Python/symtable.c)，它以类似于代码生成函数.每个作用域都有一个符号表有助于编译器找出一些关键信息，例如哪些变量是全局的，哪些是作用域的局部变量.
为了解决这个问题，我们必须修改Python/symtable.c中的symtable_visit_stmt函数，添加处理until语句的代码，在 while 语句的类似代码之后[3]:
case While_kind:访问(st, expr, s->v.While.test);VISIT_SEQ(st, stmt, s->v.While.body);如果 (s->v.While.orelse)VISIT_SEQ(st, stmt, s->v.While.orelse);休息;案例直到_kind:VISIT(st, expr, s->v.Until.test);VISIT_SEQ(st, stmt, s->v.Until.body);休息;[3]:顺便说一句，如果没有此代码，Python/symtable.c 会出现编译器警告.编译器注意到symtable_visit_stmt 的switch 语句中没有处理Until_kind 枚举值并抱怨.检查编译器警告总是很重要的！
现在我们真的完成了.在此更改后编译源代码使 myfoo(3) 的执行按预期工作.
结论
在本文中，我演示了如何向 Python 添加新语句.尽管需要对 Python 编译器的代码进行大量修改，但实施更改并不难，因为我使用了类似的现有语句作为指导.
Python 编译器是一个复杂的软件块，我并不声称自己是这方面的专家.但是，我对 Python 的内部结构非常感兴趣，尤其是它的前端.因此，我发现这个练习对于编译器原理和源代码的理论研究非常有用.它将作为以后深入了解编译器的文章的基础.
参考资料
我使用了一些优秀的参考资料来构建这篇文章.它们在这里，没有特定的顺序:
- PEP 339:CPython 编译器的设计 - 可能是最重要的和全面的官方 Python 编译器文档.它非常简短，痛苦地显示了 Python 内部良好文档的稀缺性.
- Python 编译器内部"——Thomas Lee 的一篇文章
- Python:设计与实现"——Guido van Rossum 的演讲
- Python (2.5) 虚拟机，导览 - Peter Tröger 的演讲
Can you add new statements (like print, raise, with) to Python's syntax?
Say, to allow..
mystatement "Something"
Or,
new_if True:
print "example"
Not so much if you should, but rather if it's possible (short of modifying the python interpreters code)
You may find this useful - Python internals: adding a new statement to Python, quoted here:
This article is an attempt to better understand how the front-end of Python works. Just reading documentation and source code may be a bit boring, so I'm taking a hands-on approach here: I'm going to add an until statement to Python.
All the coding for this article was done against the cutting-edge Py3k branch in the Python Mercurial repository mirror.
The until statement
Some languages, like Ruby, have an until statement, which is the complement to while (until num == 0 is equivalent to while num != 0). In Ruby, I can write:
num = 3
until num == 0 do
puts num
num -= 1
end
And it will print:
3
2
1
So, I want to add a similar capability to Python. That is, being able to write:
num = 3
until num == 0:
print(num)
num -= 1
A language-advocacy digression
This article doesn't attempt to suggest the addition of an until statement to Python. Although I think such a statement would make some code clearer, and this article displays how easy it is to add, I completely respect Python's philosophy of minimalism. All I'm trying to do here, really, is gain some insight into the inner workings of Python.
Modifying the grammar
Python uses a custom parser generator named pgen. This is a LL(1) parser that converts Python source code into a parse tree. The input to the parser generator is the file Grammar/Grammar[1]. This is a simple text file that specifies the grammar of Python.
[1]: From here on, references to files in the Python source are given relatively to the root of the source tree, which is the directory where you run configure and make to build Python.
Two modifications have to be made to the grammar file. The first is to add a definition for the until statement. I found where the while statement was defined (while_stmt), and added until_stmt below [2]:
compound_stmt: if_stmt | while_stmt | until_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
while_stmt: 'while' test ':' suite ['else' ':' suite]
until_stmt: 'until' test ':' suite
[2]: This demonstrates a common technique I use when modifying source code I’m not familiar with: work by similarity. This principle won’t solve all your problems, but it can definitely ease the process. Since everything that has to be done for while also has to be done for until, it serves as a pretty good guideline.
Note that I've decided to exclude the else clause from my definition of until, just to make it a little bit different (and because frankly I dislike the else clause of loops and don't think it fits well with the Zen of Python).
The second change is to modify the rule for compound_stmt to include until_stmt, as you can see in the snippet above. It's right after while_stmt, again.
When you run make after modifying Grammar/Grammar, notice that the pgen program is run to re-generate Include/graminit.h and Python/graminit.c, and then several files get re-compiled.
Modifying the AST generation code
After the Python parser has created a parse tree, this tree is converted into an AST, since ASTs are much simpler to work with in subsequent stages of the compilation process.
So, we're going to visit Parser/Python.asdl which defines the structure of Python's ASTs and add an AST node for our new until statement, again right below the while:
| While(expr test, stmt* body, stmt* orelse)
| Until(expr test, stmt* body)
If you now run make, notice that before compiling a bunch of files, Parser/asdl_c.py is run to generate C code from the AST definition file. This (like Grammar/Grammar) is another example of the Python source-code using a mini-language (in other words, a DSL) to simplify programming. Also note that since Parser/asdl_c.py is a Python script, this is a kind of bootstrapping - to build Python from scratch, Python already has to be available.
While Parser/asdl_c.py generated the code to manage our newly defined AST node (into the files Include/Python-ast.h and Python/Python-ast.c), we still have to write the code that converts a relevant parse-tree node into it by hand. This is done in the file Python/ast.c. There, a function named ast_for_stmt converts parse tree nodes for statements into AST nodes. Again, guided by our old friend while, we jump right into the big switch for handling compound statements and add a clause for until_stmt:
case while_stmt:
return ast_for_while_stmt(c, ch);
case until_stmt:
return ast_for_until_stmt(c, ch);
Now we should implement ast_for_until_stmt. Here it is:
static stmt_ty
ast_for_until_stmt(struct compiling *c, const node *n)
{
/* until_stmt: 'until' test ':' suite */
REQ(n, until_stmt);
if (NCH(n) == 4) {
expr_ty expression;
asdl_seq *suite_seq;
expression = ast_for_expr(c, CHILD(n, 1));
if (!expression)
return NULL;
suite_seq = ast_for_suite(c, CHILD(n, 3));
if (!suite_seq)
return NULL;
return Until(expression, suite_seq, LINENO(n), n->n_col_offset, c->c_arena);
}
PyErr_Format(PyExc_SystemError,
"wrong number of tokens for 'until' statement: %d",
NCH(n));
return NULL;
}
Again, this was coded while closely looking at the equivalent ast_for_while_stmt, with the difference that for until I've decided not to support the else clause. As expected, the AST is created recursively, using other AST creating functions like ast_for_expr for the condition expression and ast_for_suite for the body of the until statement. Finally, a new node named Until is returned.
Note that we access the parse-tree node n using some macros like NCH and CHILD. These are worth understanding - their code is in Include/node.h.
Digression: AST composition
I chose to create a new type of AST for the until statement, but actually this isn't necessary. I could've saved some work and implemented the new functionality using composition of existing AST nodes, since:
until condition:
# do stuff
Is functionally equivalent to:
while not condition:
# do stuff
Instead of creating the Until node in ast_for_until_stmt, I could have created a Not node with an While node as a child. Since the AST compiler already knows how to handle these nodes, the next steps of the process could be skipped.
Compiling ASTs into bytecode
The next step is compiling the AST into Python bytecode. The compilation has an intermediate result which is a CFG (Control Flow Graph), but since the same code handles it I will ignore this detail for now and leave it for another article.
The code we will look at next is Python/compile.c. Following the lead of while, we find the function compiler_visit_stmt, which is responsible for compiling statements into bytecode. We add a clause for Until:
case While_kind:
return compiler_while(c, s);
case Until_kind:
return compiler_until(c, s);
If you wonder what Until_kind is, it's a constant (actually a value of the _stmt_kind enumeration) automatically generated from the AST definition file into Include/Python-ast.h. Anyway, we call compiler_until which, of course, still doesn't exist. I'll get to it an a moment.
If you're curious like me, you'll notice that compiler_visit_stmt is peculiar. No amount of grep-ping the source tree reveals where it is called. When this is the case, only one option remains - C macro-fu. Indeed, a short investigation leads us to the VISIT macro defined in Python/compile.c:
#define VISIT(C, TYPE, V) {\
if (!compiler_visit_ ## TYPE((C), (V))) \
return 0; \
It's used to invoke compiler_visit_stmt in compiler_body. Back to our business, however...
As promised, here's compiler_until:
static int
compiler_until(struct compiler *c, stmt_ty s)
{
basicblock *loop, *end, *anchor = NULL;
int constant = expr_constant(s->v.Until.test);
if (constant == 1) {
return 1;
}
loop = compiler_new_block(c);
end = compiler_new_block(c);
if (constant == -1) {
anchor = compiler_new_block(c);
if (anchor == NULL)
return 0;
}
if (loop == NULL || end == NULL)
return 0;
ADDOP_JREL(c, SETUP_LOOP, end);
compiler_use_next_block(c, loop);
if (!compiler_push_fblock(c, LOOP, loop))
return 0;
if (constant == -1) {
VISIT(c, expr, s->v.Until.test);
ADDOP_JABS(c, POP_JUMP_IF_TRUE, anchor);
}
VISIT_SEQ(c, stmt, s->v.Until.body);
ADDOP_JABS(c, JUMP_ABSOLUTE, loop);
if (constant == -1) {
compiler_use_next_block(c, anchor);
ADDOP(c, POP_BLOCK);
}
compiler_pop_fblock(c, LOOP, loop);
compiler_use_next_block(c, end);
return 1;
}
I have a confession to make: this code wasn't written based on a deep understanding of Python bytecode. Like the rest of the article, it was done in imitation of the kin compiler_while function. By reading it carefully, however, keeping in mind that the Python VM is stack-based, and glancing into the documentation of the dis module, which has a list of Python bytecodes with descriptions, it's possible to understand what's going on.
That's it, we're done... Aren't we?
After making all the changes and running make, we can run the newly compiled Python and try our new until statement:
>>> until num == 0:
... print(num)
... num -= 1
...
3
2
1
Voila, it works! Let's see the bytecode created for the new statement by using the dis module as follows:
import dis
def myfoo(num):
until num == 0:
print(num)
num -= 1
dis.dis(myfoo)
Here's the result:
4 0 SETUP_LOOP 36 (to 39)
>> 3 LOAD_FAST 0 (num)
6 LOAD_CONST 1 (0)
9 COMPARE_OP 2 (==)
12 POP_JUMP_IF_TRUE 38
5 15 LOAD_NAME 0 (print)
18 LOAD_FAST 0 (num)
21 CALL_FUNCTION 1
24 POP_TOP
6 25 LOAD_FAST 0 (num)
28 LOAD_CONST 2 (1)
31 INPLACE_SUBTRACT
32 STORE_FAST 0 (num)
35 JUMP_ABSOLUTE 3
>> 38 POP_BLOCK
>> 39 LOAD_CONST 0 (None)
42 RETURN_VALUE
The most interesting operation is number 12: if the condition is true, we jump to after the loop. This is correct semantics for until. If the jump isn't executed, the loop body keeps running until it jumps back to the condition at operation 35.
Feeling good about my change, I then tried running the function (executing myfoo(3)) instead of showing its bytecode. The result was less than encouraging:
Traceback (most recent call last):
File "zy.py", line 9, in
myfoo(3)
File "zy.py", line 5, in myfoo
print(num)
SystemError: no locals when loading 'print'
Whoa... this can't be good. So what went wrong?
The case of the missing symbol table
One of the steps the Python compiler performs when compiling the AST is create a symbol table for the code it compiles. The call to PySymtable_Build in PyAST_Compile calls into the symbol table module (Python/symtable.c), which walks the AST in a manner similar to the code generation functions. Having a symbol table for each scope helps the compiler figure out some key information, such as which variables are global and which are local to a scope.
To fix the problem, we have to modify the symtable_visit_stmt function in Python/symtable.c, adding code for handling until statements, after the similar code for while statements [3]:
case While_kind:
VISIT(st, expr, s->v.While.test);
VISIT_SEQ(st, stmt, s->v.While.body);
if (s->v.While.orelse)
VISIT_SEQ(st, stmt, s->v.While.orelse);
break;
case Until_kind:
VISIT(st, expr, s->v.Until.test);
VISIT_SEQ(st, stmt, s->v.Until.body);
break;
[3]: By the way, without this code there’s a compiler warning for Python/symtable.c. The compiler notices that the Until_kind enumeration value isn’t handled in the switch statement of symtable_visit_stmt and complains. It’s always important to check for compiler warnings!
And now we really are done. Compiling the source after this change makes the execution of myfoo(3) work as expected.
Conclusion
In this article I've demonstrated how to add a new statement to Python. Albeit requiring quite a bit of tinkering in the code of the Python compiler, the change wasn't difficult to implement, because I used a similar and existing statement as a guideline.
The Python compiler is a sophisticated chunk of software, and I don't claim being an expert in it. However, I am really interested in the internals of Python, and particularly its front-end. Therefore, I found this exercise a very useful companion to theoretical study of the compiler's principles and source code. It will serve as a base for future articles that will get deeper into the compiler.
References
I used a few excellent references for the construction of this article. Here they are, in no particular order:
- PEP 339: Design of the CPython compiler - probably the most important and comprehensive piece of official documentation for the Python compiler. Being very short, it painfully displays the scarcity of good documentation of the internals of Python.
- "Python Compiler Internals" - an article by Thomas Lee
- "Python: Design and Implementation" - a presentation by Guido van Rossum
- Python (2.5) Virtual Machine, A guided tour - a presentation by Peter Tröger
这篇关于你能在 Python 的语法中添加新的语句吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
