PSA: Test your textmate grammars in TextMate

[evoactivity]

So with the latest release of linguist, we had included fixes for our glimmer-js and glimmer-ts grammars. We used NovaLightshow to test our grammar as it was capable of reproducing the issues we were seeing, and everything works perfectly there so we had hoped the same would be true for GitHub. It also works perfectly in VSCode and ShikiJS (and possibly other textmate derived syntax highlighting engines).

When we began testing the latest rollout of linguist everything seemed fine until we tested the simplest case our authoring format allows. Namely this

// if you're looking at this 3 months or more after this post hopefully you're
// seeing the correct highlighting. A screenshot is posted below for posterity 
const hello = 'Bonjour';

<template>
  <div>{{hello}} world</div>
</template>

I was stumped on how to test this as it worked everywhere else except GitHub. That is until I decided to install TextMate and try our grammar there, what I saw was identical to how GitHub was highlighting (or not in this case).

Now I had somewhere I could test I managed to fix the grammar

So if you grammar works everywhere but GitHub, and you're wondering how to most accurately test your grammar for GitHub support, I can definitely recommend using TextMate as it seems GitHub's highlighting engine matches TextMate's one-to-one, whereas other engines do not and might highlight things perfectly fine even if GitHub wont.

[RedCMD]

• VSCode
  • vscode-textmate v9.2.0
    • vscode-oniguruma v1.7.0
      • oniguruma v6.9.8
• TextMate 2.0
  • textmate/Onigmo v5.13.5
    • k-takata/Onigmo
      • oniguruma
• GitHub
  • Linguist
  • PrettyLights
    • PCRE v8.45
  • Pygments
• NovaLightshow
  • starry-night
    • vscode-textmate
      • vscode-oniguruma
        • oniguruma v6.9.8
• ShikiJS
  • vscode-oniguruma
    • oniguruma v6.9.8

seems GitHub's highlighting engine matches TextMate's one-to-one

it doesn't
Github uses PCRE instead of TextMate's Onigmo
it also doesn't match TextMate's crazy match/begin/end shenanigans
doesn't allow comments within captures
and prob many other things

---

I assume this is your PR? lifeart/vsc-ember-syntax#77
I found the reason for the problem
its due to a bug in TextMate 2.0, that Github then copied
VSCode did not copy the bug

in your grammar
#main is included in 2 places and applied twice in the same instance
both in the root level patterns and within the injections
patterns is checked first; for rules to apply
because source.js was higher than #main, source.js had priority

the injections are then checked
the L: in the injection should have put the 2nd include of #main higher than the other 2 includes
but because #main has already been included in this instance
the 2nd reference was ignored
and thus the priority did not change

VSCode does not discard the 2nd #main (injection) and thus it then has priority over source.js

[DecimalTurn]

If someone like me who doesn't have a mac wants to test a grammar with TextMate, there is a way using a GitHub Action running on macOS like this one and you can get something like this :