Support for Koka v3

[chtenb]

Koka 3 added some new syntactical features. See the samples/syntax folder for example cases.
I notice the highlighting doesn't quite work for at least these files:
https://github.com/koka-lang/koka/blob/dev/samples/syntax/contexts.kk
https://github.com/koka-lang/koka/blob/dev/samples/syntax/qualifiers.kk
https://github.com/koka-lang/koka/blob/dev/samples/syntax/implicits.kk

[HeikoRibberink]

I am currently working on modifying the grammar, and updating the highlighting rules to accommodate these changes.

[chtenb]

Thanks for the heads up :) I was almost gonna pick some of it up when I had the time

[mtoohey31]

A quick recommendation for anyone attempting to update things: I originally based the grammar off of this file in the upstream repository. There have been some problems with it becoming out of date with respect to the actual implementation at points, but it seems like it's been updated pretty recently, so maybe you can use that.

[HeikoRibberink]

Thanks for the advice. I am having success with just modifying the highlighting queries and making minimal changes to the grammar, but I still don't know, and hence cannot check the full extent of Koka's syntax.

I will look into updating the grammar to help me understand the aforementioned.

[HeikoRibberink]

I am currently updating the grammar using the grammar of Koka's parser.

However, I do not understand Koka's lexer (yet), nor the external scanner of tree-sitter-koka. Can someone else check if the latter needs to be updated?

[mtoohey31]

Great!

About the lexer, the main thing that's special about it is the layout rule. I took a quick look at the recent changes to the lexer.l file (with this url then searching for lexer.l) and there aren't very many. I think this might be all that's needed in terms of changes to our lexer:

diff --git a/src/scanner.c b/src/scanner.c
index ade02bd..f7579fe 100644
--- a/src/scanner.c
+++ b/src/scanner.c
@@ -245,10 +245,10 @@ AFTER_WHITESPACE:
   case '&':
   case '*':
   case '+':
-  case '@':
+  case '~':
+  case '#':
   case '\\':
   case '^':
-  case '?':
   case '.':
   case '=':
   case ')':

Not 100% sure though. It's probably best if you add these changes, and finish yours, then run the parser on all of Koka's examples/tests to see if we fail to parse anything. If you see any failures that look layout related, then there might still be other stuff I have to fix in the lexer, so please ping me if that happens. Back when I was developing this I believe I used something like the following for testing:

for file in $(find $KOKA_ROOT/{lib/std,samples,test,util} -name '*.kk'); do
  if tree-sitter parse $file | grep ERROR >/dev/null; then
    echo $file
  fi
done

[HeikoRibberink]

I am writing a script that converts the Yacc grammar to a TS grammar file, because I think that is more maintainable. Once I finish that, it will probably be finished very soon.

[mtoohey31]

Ok, it might be worth checking if any of the existing yacc -> tree-sitter converters work, cause it looks like there are some projects for that already, though they might be out of date.

[HeikoRibberink]

I really only found one: miks1965/yacc-to-tree-sitter, but it is unusable per this issue

[HeikoRibberink]

Building a yacc to treesitter converter will take some more time, but I have previously improved the highlighting groups and the grammar, giving a decent result on modern koka. I can upload this as a temporary fix, if you want.

[mtoohey31]

Sure, if it's not too much work to push things, feel free to create a pull request.