Nesting analysis using Prism

[tompng]

#1024
Migrate nesting analysis from Ripper to Prism

Original nesting calculation with Ripper

Tokenize with ripper
Parse the tokens mainly focusing on open/close tokens
Collect open tokens per line

New nesting calculation with Prism

Traverse syntax tree
Check open_loc and closing_loc for a node that makes nesting, create a token-like object for it
Collect open token-like objects per line

Gemfile

Exclude prism-1.8.0 because test fails. (ruby/prism#3851)

[st0012]

Can we nudge Prism maintainers to cut v1.8.1 for the fix, and add prism as a dependency and require 1.3.0+?

[st0012]

Can we make this or parse_by_line take (code, local_variables: []) and call Prism.parse_lex here?
I feel Prism should be hidden from the callers of this class. Then we only need to require it here instead of all the places that'd use a nesting parser.

[tompng]

In a followup pull request #1160, parse_lex_result will be also used in RubyLex#should_continue?.
To avoid parsing the same code two times in dynamic_prompt calculation, argument needs to be parse_lex_result.

@context.io.dynamic_prompt do |lines|
  parse_lex_result = Prism.parse_lex(code, scopes: [@context.local_variables])
  line_results = IRB::NestingParser.parse_by_line(parse_lex_result)
  ...
  line_results.map.with_index do |...|
    # This part requires lex result
    continue = @scanner.should_continue?(tokens_until_line, line, line_num_offset + 1)
    ...
  end
end

Maybe we can find a better way to separate after migrating to Prism, I think.

[st0012]

Oh you did this in #1091 already. I guess we just need to wait for the release then