Syntax Highlight using Prism

[tompng]

#1024
Migrate syntax highlight from Ripper to Prism

Syntax highlighting

Colorize from these three information:

• Basically colorize from Prism token types
• Error coloring from error location
• Exceptional case from Prism syntax tree

Mix tokens and pseudo tokens created from error location and syntax tree which is prior than normal tokens.

Exceptional case

Symbol name can by IDENTIFIER, operator-token or keyword-token. It may not be placed after SYMBOL_BEGIN

:symbol; :<=>; :if

<<A; :\
A
symbol

Symbol and String colorize
Sometimes it's hard to distinguish symbol and string because both consists of same token types: STRING_CONTENT, EMBEXPR_BEGIN, EMBEXPR_END and STRING_END.
Traversing syntax tree is the easiest way to colorize it correctly.

%w[a b c]
%i[a b c]
:"symbol#{embed}symbol"
"string#{embed}string"

IRB colors defined method name blue. Doing this is really easy from syntax tree but hard from token.

def method_name_colored_blue_in_irb; end

[st0012]

Can we require 1.1.0+ or something like that so we don't need this?

[tompng]

I changed it to 1.3.0 which is the one bundled in ruby-3.4.0.
This will also avoid this error

Could not find compatible versions

Because every version of rbs depends on prism >= 1.3.0
  and Gemfile depends on prism = 1.0.0,
  rbs cannot be used.
So, because Gemfile depends on rbs >= 0,
  version solving has failed.

and the workaround I added in Gemfile

if RUBY_VERSION >= "3.0.0" && !is_truffleruby && ENV['PRISM_VERSION'] != '1.0.0'

[st0012]

This is great. Thank you

[tompng]

Failed test (truffleruby) seems unrelated to this change. Failed in master too, maybe related to Ripper change.
Most of the failed test will be solved in #1160 (Only one (probably) unrelated failure)

[Earlopain]

Prism by default parses code as the latest version it currently supports. In irb you probably want to parse as the version that the user is currently running as.

You can do that by passing in version: RUBY_VERSION (or version: "current" on more recent prism for better error messages)

[tompng]

That's a good point, I completely forgotten about version option.

In Ruby < 3.3.0, Prism.parse('code', version: :current) fails with

Prism.parse '1+2', version: 'current'
(irb):2:in `parse': invalid version: Requested to parse as `version: 'current'`;  3.2.2 is below the minimum supported syntax. (Prism::CurrentVersionError)

Is there a good way to use 'current', fallback to the nearest supported version?
(It's better to parse with the current version but it might not be such a big problem... maybe.)

[Earlopain]

Ah, prism cannot be used to parse ruby syntax below 3.3. I also added "current" only in prism 1.6.0 so I guess you need to bump the requirement in the gemspec.

Apart from bumping the minimum supported ruby version to 3.3 which might be a bit extreme, the best you can do is version: RUBY_VERSION >= "3.3.0" ? "current" : "3.3" I think