🔒 How is implicit line joining implemented?

[hoary spruce]

Python has allowed syntax like this for a while (since python 2!)

x = (1 + 2
     + 3 + 
     4 + 5)

How is this implemented? I've looked at tokenize, and it doesn't seem to handle removing newlines there. However, the grammar spec doesn't allow for newline. The python docs of course reference it, but I've been unable to find where or how it's actually implemented.

I can think of some ways to implement it myself:

• Write a flag in the parser that checks if you're in an open parenthesis and discard
• Check the entire lexed token list before parsing and remove all newlines in between parenthesis
• During lexing, assume that all parenthesis will be closed and remove newlines there

I'd love to see the actual implementation though, or hear some insight on which method is the best!

[analog wedge]

@hoary spruce

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[vague zephyr]

I think you want to look at the level field of the tokeniser state object and where it's used:
https://github.com/python/cpython/blob/b041a456f1a5bf73d8cc3f5efcdd6dd24281d149/Parser/lexer/state.h#L90

[analog wedge]

Parser/lexer/state.h line 90

int level;          /* () [] {} Parentheses nesting level */```

[tribal wyvern]

From what I've seen working on Black, python has two types of newlines: syntactically important newlines NEWLINE, and unimportant ones NL. Tokenizing that code, you can see all the internal newlines are NLs, and so are basically treated like whitespace.

PS D:\rust_projects\ruff> echo @"
x = (1 + 2
     + 3 +
     4 + 5)
"@ | py -m tokenize
1,0-1,1:            NAME           'x'
1,2-1,3:            OP             '='
1,4-1,5:            OP             '('
1,5-1,6:            NUMBER         '1'
1,7-1,8:            OP             '+'
1,9-1,10:           NUMBER         '2'
1,10-1,11:          NL             '\n'
2,5-2,6:            OP             '+'
2,7-2,8:            NUMBER         '3'
2,9-2,10:           OP             '+'
2,11-2,12:          NL             '\n'
3,5-3,6:            NUMBER         '4'
3,7-3,8:            OP             '+'
3,9-3,10:           NUMBER         '5'
3,10-3,11:          OP             ')'
3,11-3,12:          NEWLINE        '\n'
4,0-4,0:            ENDMARKER      ''

Whereas the ending newline is syntactically important, and so gets a NEWLINE token.

[vague zephyr]

@hoary spruce ^^

[hoary spruce]

@tribal wyvern, @vague zephyr thank you both for the insight! I have a much better understanding now—it's interesting that the lexer does so much. Thanks for the input!

!close

[analog wedge]

Python help channel closed with !close

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.