PEP 701 – Syntactic formalization of f-strings

[pablogsal]

• Changes in the C tokenizer
• Categorize failing tests
• Fix failing tests or modify/remove them as needed
• Changes in Python tokenizer

Linked PRs

• gh-102856: Initial implementation of PEP 701 #102855
• gh-102856: Skip test_mismatched_parens in WASI builds #103633
• gh-102856: Clean some of the PEP 701 tokenizer implementation #103634
• gh-102856: Allow comments inside multi-line f-string expresions #104006
• gh-102856: Python tokenizer implementation for PEP 701 #104323
• gh-102856: Tokenize performance improvement #104731
• gh-102856: Add changes related to PEP 701 in 3.12 What's New docs #104824
• [3.12] gh-102856: Add changes related to PEP 701 in 3.12 What's New docs (GH-104824) #104847
• gh-102856: Add missing quote to fix doctest #104852
• [3.12] gh-102856: Add missing quote to fix doctest (GH-104852) #104854
• gh-102856: Update "Formatted string literals" docs section after PEP701 #104861
• [3.12] gh-102856: Update "Formatted string literals" docs section after PEP701 (GH-104861) #104865

[pablogsal]

CC: @lysnikolaou @isidentical @mgmacias95

[pablogsal]

See this for the latest report on errors from @isidentical

[pablogsal]

Draft PR for the C tokenizer up: #102855

[pablogsal]

Things for the cleanup of #102855:

• Cleaning up the grammar and the action helpers (the names are still ridiculous and there are multiple rules commented out).
• Remove the old parsing code and check that we didn't break anything 😅
• Clean/refactor the tokenizer struct (better names, factor stuff into its own structure as needed).
• Consider factoring out tok_get_fstring_mode because is a monster.

[pablogsal]

Ok with #102855 we have the following failing tests:

• test_ast
• test_cmd_line_script
• test_eof
• test_exceptions
• test_fstring
• test_tokenize
• test_type_comments
• test_unparse

Most of these are updating error messages, line numbers and other stuff but some may have actual bugs so we should check them. Please, mention which ones are you working on so we don't clash with one another.

[mgmacias95]

Working on test_tokenize

[Eclips4]

Hello, Pablo!
Can I get work on test_ast?
Recently I sent some PR's about this file (for example, #102797). So, I have some experience in that =)

[ramvikrams]

I can work with test_type_comments and test_unparse.

[pablogsal]

@Eclips4 @ramvikrams wonderful! Just make PRs against my fork!

Report here or ping any of us if you find something that could be a bug (don't just fix the tests blindly because there may be bugs lurking).

[pablogsal]

@lysnikolaou can you work on cleaning up the grammar + the actions?

@isidentical can you work on cleaning up some of the tokenizer layers? (This is quite a lot so we can probably work together here).

[Eclips4]

@pablogsal
About test_ast.py
Seems thats like there only a one test will be failed, and how I undestand, that's a bug:

cpython/Lib/test/test_ast.py

Lines 779 to 780 in 7f760c2

	with self.assertRaises(SyntaxError):
	ast.parse('f"{x=}"', feature_version=(3, 7))

I think, there's two solutions:

1. Remove this test, because support of python3.7 will be ended soon.
2. Now errors raised by tokenizer.c instead of string_parser.c, so as I understand, we should change python_gram, is it right? ( We need access to feature_version, which in tokenizer inaccessible )

[pablogsal]

2. Now errors raised by tokenizer.c instead of string_parser.c, so as I understand, we should change python_gram, is it right? ( We need access to feature_version, which in tokenizer inaccessible )

Probably we can do this but on the other hand I would prefer to not overcomplicate this so I think (1) is better

[lysnikolaou]

@lysnikolaou can you work on cleaning up the grammar + the actions?

Will do!

[Eclips4]

Also, I can take a look at test_cmd_line_script. Seems easy.