Idea sharing

[zjp-CN]

Hi, I noticed this crate is trying to implement markdown rendering with ratatui.

I'm delighted to share the code about rendering markdown I write in term-rustdoc.

The markdown rendering code is complete and viable to be split from term-rustdoc. I write it for displaying Rust docs and Help. Screenshots.

Relevant code is here.

[kdheepak]

I wrote some code too for rendering markdown with Ratatui that is still in an unfinished unpublished project.

Another idea: syntax highlighting in code blocks.

I implemented something like this for translating code blocks to ratatui with language specific syntax highlighting using syntect:

use syntect::{
  easy::HighlightLines, highlighting::ThemeSet, parsing::SyntaxSet, util::LinesWithEndings,
};

// ...

  if let Some(Tag::CodeBlock(kind)) = tags.last() {
    let language = match kind {
      CodeBlockKind::Fenced(language) => language,
      CodeBlockKind::Indented => "",
    };
    let ps = SyntaxSet::load_defaults_newlines();
    let ts = ThemeSet::load_defaults();
    let syntax = ps.find_syntax_by_token(language).unwrap_or_else(|| {
      ps.find_syntax_by_name(language).unwrap_or_else(|| ps.find_syntax_plain_text())
    });
    let mut h = HighlightLines::new(syntax, &ts.themes["base16-ocean.dark"]);
    for line in LinesWithEndings::from(text.as_ref()) {
      let line_spans: Vec<ratatui::text::Span> = h
        .highlight_line(line, &ps)
        .unwrap()
        .into_iter()
        .filter_map(|(style, s)| into_span((style, s.to_string())).ok())
        .collect();
      let line = ratatui::text::Line::from(line_spans);
      lines.push(line);
    }

into_span's signature was this:

pub fn into_span(
  (style, content): (syntect::highlighting::Style, String),
) -> Result<ratatui::text::Span<'static>> {
  // ...
}

I'll share the full version of into_span shortly.

[joshka]

The markdown rendering code is complete and viable to be split from term-rustdoc. I write it for displaying Rust docs and Help. zjp-CN/term-rustdoc#1.

Hey that looks pretty good.

Taking a very quick look at the code, it seems like there's some code that is fairly app specific, intermixed with code that's not. There's a lot of complexity in that folder - is markdown actually fairly complex, or is that just because of what your app needs?

I don't see a license in the repo - are you able to license this as MIT/Apache 2.0?

Another idea: syntax highlighting in code blocks.

Yep yep yep...

This and themes are two necessary things

[zjp-CN]

are you able to license this as MIT/Apache 2.0?

It has MIT license which is written in Cargo.toml.

There's a lot of complexity in that folder - is markdown actually fairly complex, or is that just because of what your app needs?

Yeah... Let me explain what I need and what I do:

syntect is great at highlighting syntaxes including the whole markdown document, but the biggest problem for me are text wrapping and how links should be rendered?

for example, here are two versions of rendering markdown in syntect and term-rustdoc on the same screen for the same doc

syntect:

custom:

• the width of a line in syntect depends on source code, which means once a line is too long for your screen, it's truncated or wrapped to another line, or too short, then a large of empty space on the right; but mine will always wrap texts to fixed width as much as possible to look nice in any width

• links in syntect keep the style in source code, like referenced/inline link styles as they are written, but mine will be unified to referenced link with the link id enumerated from 0 to reduce visual complexity via listing links under paragraphs containing them (without handling the defining spots, it would be annoying to see a [foo][bar] link on the top, but [bar]: xxx may be not in visual range or close to the using spot since it can be anywhere).

• the way to handle links displaying also affacts the width of a line. This is how neovim's markdown rendering looks:
  
  It basically like directly using syntect to do rendering, but additionally hides the link id and only shows the link text in square brackets if the cursor is not in the line, and show them all if yes.
  It's really not a good experience for me because all lines are either too long or too short, especially many link are inside a line.
  You would consider wrapping text before passing md to syntect would solve one of the problems, but actually, no. When you want to show inline links, URLs are usually long and will quickly occupy majority of a line. That makes wrapping texts useless.

• yeah, some extra features only targeting at the app are

  • Rust docs has own link types: [`crate::Item`] is a referenced link but can without defining itself
  • heading jump /intra-doc link jump: this means continous areas or line numbers where the heading/target lies should be recorded.

Actually, I do use syntect to highlight codeblocks too. Since code snippets are not wrappable like markdown texts (it would change the meaning if we arbitrarily eliminate whitespaces to wrap code), syntect can directly emit the colors on each words and we just need to simply wrap the long line to next line by moving these words beyond the width to it.
P.S. in my code, the Block abstracts block elements (paragraph/quoteblock/codeblock) and Word abstract styled text which can be a word optionally trailing with a whitespace in markdown texts or a word including whitespaces in code.

[joshka]

Neat - thanks for the explanation. I think I'd like to handle that with something I'm experimenting with. I have some ideas around having actual widgets for each element, each implementing an Element trait, and then composing the widgets using Vec<Box<dyn Element>>.

E.g. a over-simplified Heading element (that can only have a string child) might be something like:

pub trait Element: WidgetRef + Debug {}

#[derive(Debug)]
pub struct Heading<'a> {
    level: u8,
    text: &'a str,
}

impl Element for Heading<'_> {}

impl WidgetRef for Heading<'_> {
    fn render_ref(&self, area: Rect, buf: &mut Buffer) {
        Text::from(self.text).style(self.style()).render(area, buf);
    }
}

impl Heading<'_> {
    pub fn new(level: u8) -> Self {
        let text = "";
        Self { level, text }
    }

    pub fn add_text(&mut self, text: &str) {
        self.text = text;
    }

    fn style(&self) -> Style {
        match self.level {
            1 => Style::default().fg(Color::Red),
            2 => Style::default().fg(Color::Green),
            3 => Style::default().fg(Color::Blue),
            4 => Style::default().fg(Color::Yellow),
            5 => Style::default().fg(Color::Magenta),
            6 => Style::default().fg(Color::Cyan),
            _ => Style::default(),
        }
    }
}

And the markdown widget might look something like:

#[derive(Debug)]
pub struct MarkdownDoc {
    elements: Vec<Box<dyn Element>>,
}

I think the wrapping should be handled by the widgets - Text seems like it will break down very quickly there. E.g. we want to wrap list items so they align with their item, and code blocks that are within list items should be indented.

One of the things that leads me to this approach is that it allows the user to interact with elements (e.g. adding focusable regions for code blocks that can be copied, or having headings / links be navigable.) I think you have something similar in your app.

I'm in agreement with your approach to handling the markdown by throwing away some of the data (e.g. the link type or choice of style for emphasis probably doesn't matter) This is going to be more an opinionated viewer library than a markdown editor library.

[joshka]

Thinking about this more, it seems that this approach has two directions it can go:

• use a different library (cormak / markdown-rs) that pulls out an AST and build the widget tree from the nodes
• continue with pulldown-cmark and build up the visual tree based on that (this might be the right approach because styled text becomes just a simple matter of keeping the current style as a variable and applying that to the current span instead of having to worry about deeply nested trees of text / styles).

[zjp-CN]

Thinking about this more, it seems that this approach has two directions it can go:

Hah, I had a similar thought when deciding to write my own rendering code.

There is a third choice: termimad. It contains incomplete markdown parsing code and also expresses a line with styles and purposes. But it's independent of ratatui. I didn't dive into it too much. The main drawback is no highlighting in codeblocks. I guess it won't be hard to support it by syntect in termimad! A good thing to see is termimad also supports scrolling and wrapping.

My app term-rustdoc is similar with termimad in rendering markdown:

• a rendering line should contain the displaying texts, desired styles and original metadata: you just use them to render the doc once they're all cached
• scrolling: since there is no general way to scroll texts, I wrote my own Scrollable widget. termimad does too.
• wrapping: I don't know how termimad handles rendering links, but its doc says it handles wrapping

But term-rustdoc is featured as follows

• term-rustdoc cares about rendering data caches: pub fn parse_doc(doc: &str, width: f64) -> (Vec<StyledLine>, Blocks, Headings) is the starting point. It needs the doc and width to generate
  • StyledLines in at most the input width
  • Blocks are parsed markdown with a vec of Block and global link definitions/footnotes: StyledLines and Headings are the outputs of Blocks.
  • Headings are locations in StyledLines on headings, and tells us where a heading click should jump to in StyledLines
• Blocks are flatten view instead of syntax tree: it ultimately delegates to Words instead of Nodes. But the full chain is Blocks-Block-Line-Word-{unwrappable text, style, metadata}.
  • Block is a vec of Line. It also contains the link definition/footnote ids. A block only knows ids used inside itself, and the ids will be rendered below wrapped lines as real contents.
  • Line is a vec of Word. textwrap::wrap_algorithms::wrap_optimal_fit(&[Word], ...) is called in rendering code for each line. Wrapping happens in a line, not in a block.
    • For common cases, pulldown-cmark emits a Paragraph event, and the paragraph is broken up into one line with Words pieces pushed onto the line. Why just one line here? Because the line will ultimately wrapped into multiple rendering lines!
    • But for codeblocks, a line in source markdown will be a Line equivalently. (code) That makes one code Line wrapped to next line if it is too long, and two neighbor code lines won't join to each other!

[zjp-CN]

Word is a basic and handy concept.

• How a Word is generated is configurable:
  • you can define how an intra-code display by with backticks or not. (code) Push backticks if you want them, or don't push them if not.
  • you can define how a link display: [link text][id] is chosen in term-rustdoc, but it's not hard to make it [link text] which is originially what I expect it to be. At first, I wish [link text] is clickable to jump and highlight the target URLs and verse versa. Now with Word and LinkedRegions, it's really simple to add the bidirectional jump feature. (I just didn't have time to do it! Rust referenced links are not handled: the link points to any item in a crate doc or even outer crate :)
• Word and wrapping are orthogonal: you'll never worry about adding a Word will influence the wrapping. Word consists of the mere unwrappable text(s).

When ` is a Word, it means it's allowed to lie at the end of a line

                      | max width
xxxxxxxxxxxxxxxxxxxx `
code`

If you don't want this behavior, just make a Word `code` to include the backticks!

                      | max width
xxxxxxxxxxxxxxxxxxxx 
`code`

That's the secret to see links are able to be wrapped in the middle.

[joshka]

There's lots of good info here, that I'd prefer doesn't get hidden away when an issue is closed, but there's nothing immediately actionable, so I'm moving it to a discussion.