Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tokenizing empty fenced code block doesn't apply codeFenceLine to opening fence line #255

Open
DivineDominion opened this issue Jul 5, 2024 · 1 comment

Comments

@DivineDominion
Copy link
Contributor

Hey Fletcher, hope things are fine!

I found a weird token thing again today.

Given a document that has this string content:

```\n```

I.e. opening and closing code block triple backticks with 1 newline, only the closing backticks are tokenized as generic fenced code block lines. A printout of the tree with ranges and token types:

  {0, 7}.blockCodeFenced
    {0, 4}.lineFenceBacktick3
      {0, 3}.codeFence
      {3, 1}.textNl
    {4, 3}.codeFenceLine
      {4, 3}.codeFence

Once you add anything in the code block, even another line break, the lineFenceBacktick3 will be "downgraded" to the generic one:

```\n\n```

produces

  {0, 8}.blockCodeFenced
    {0, 4}.codeFenceLine
      {0, 3}.codeFence
      {3, 1}.textNl
    {4, 1}.textNl
    {5, 3}.codeFenceLine
      {5, 3}.codeFence

I'm using this information for syntax highlighting of course, and I wonder how you deal with these inconsistencies in behavior in your own apps. Given previous questions about this were not received with the same level of 'OCD' that I show, I guess you don't mind this in practice somehow :)

@fletcher
Copy link
Owner

fletcher commented Jul 6, 2024

```
```

Isn't a fenced code block, it's a code span within a <p> with a single newline as the contents. It's valid Markdown (which doesn't support fenced code blocks).

Once you split this into multiple paragraphs, e.g. with two newlines, it becomes a fenced code block in MultiMarkdown and others (but not Markdown).

This can be difficult to correctly interpret in the parser, and has to be handled after the fact. (One reason that I partially regret supporting fenced code blocks since they break the rules established by the rest of the "pure" Markdown syntax.)

You have to look at the HTML output to fully understand what is happening here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants