Improve support for code blocks/block quotes nested in lists

[0xdevalias]

I've noticed that code blocks / block quotes nested in lists don't seem to be supported too well currently, and wanted to create an issue to document some of the bugs/things I've noticed.

The examples I give only show single level lists, but ideally the fixes would work properly no matter how deeply nested the list item containing the block quote / code block is.

Issues

Diff highlighting

When nesting a triple backtick codeblock in a list, while highlighting seems to be correctly applied for JS/similar, diffs don't seem to be highlighted properly:

Example Code:

- ```js
  - console.log("This was removed");
  + console.log("This was added");
  ```
- ```diff
  - console.log("This was removed");
  + console.log("This was added");
  ```

Within Sublime Text:

Within Sublime Text without being nested in a list:

I would expect that the 'start column' for the diff highlighting within a code block should be the 'base column' of the code block, not the literal start of the line in the file.

List continuation with nested triple backtick code blocks quotes

Usually when I am in a list item, and press enter to go to the next line, it will add a new - for the next list item, which is useful.

But if I am trying to create a nested code block within a list item, it would be more useful for it to not create the -, but instead move me to the same indentation level as the triple backticks.

Eg. If I am typing the following (| represents my caret):

- ```|

Currently when I press enter, it does this:

- ```
- |

Whereas ideally it would do this:

- ```
  |

Additional newlines while in a triple back tick codeblock would maintain this level of indentation (which they currently seem to do once I manually remove the created list -):

- ```
  This is my first line
  This is my second line
  |

Until I end the triple back tick codeblock:

- ```
  This is my first line
  This is my second line
  ```|

At which point, pressing enter should ideally create a new list - again:

- ```
  This is my first line
  This is my second line
  ```
- |

But instead, currently, it just does this:

- ```
  This is my first line
  This is my second line
  ```
  |

Ideally this would work for all code blocks, not just triple backtick (eg. sometimes you need to use quadruple backticks if you want to embed triple backticks within the codeblock without terminating it)

• https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks

List continuation with nested block quotes

Usually when I am in a list item, and press enter to go to the next line, it will add a new - for the next list item, which is useful.

But if I am trying to create a nested block quote within a list item, it would be more useful for it to not create the -, but instead to continue the block quote.

It might make sense for this to be configurable as to whether it is:

• disabled
• applied to any blockquote within a list
• applied only to multiline blockquotes within a list

Eg. If I am typing the following (| represents my caret):

- > This is my first line|

Currently when I press enter, it does this:

- > This is my first line
- |

Whereas ideally (in the case of 'applied to any blockquote within a list') it would do this:

- > This is my first line
  > |

For 'applied only to multiline blockquotes within a list', it would first need to start like this:

- > This is my first line
  > This is my second line|

And only once enter is pressed from that 2nd/etc line, would it continue the blockquote:

- > This is my first line
  > This is my second line
  > |

Otherwise (for 'applied only to multiline blockquotes within a list') if it just had a single line like this:

- > This is my first line|

When enter is pressed, it would just continue the list like normal (which is what it currently does):

- > This is my first line
- |

Etc

I know there was at least one other issue I noticed related to block quotes and nested lists, but I can't figure out how to create a reproduction for it right now. I'll update this issue with it when I can.

[deathaxe]

I would expect that the 'start column' for the diff highlighting within a code block should be the 'base column' of the code block, not the literal start of the line in the file.

That's unfortunatelly not possible with ST's syntax engine. We can't count and use indentation for highlighting nor does it support stripping certain amount of leading whitespace from lines of embedded code when highlighting.

If an embedded syntax relies on indentation or its absense, like diff, it just doesn't work. There's nothing this package can do to fix it.

Theoretically it would require to ship a special syntax for diff which doesn't require +/- to start at the very first column, but that's out of scope for this package.

Syntaxes are embedded by scope, so it would theoretically be possible to create a more lazy one overriding the default diff syntax, which would then be picked up.

List continuation...

Input helpers have various short commings as they rely significantly on sublime-macros with lots of regexp black magic. There's much to improve.

The noted cases shouldn't be too hard to fix, I guess.

[0xdevalias]

That's unfortunatelly not possible with ST's syntax engine.

Ah true, that's unfortunate, but fair enough. The diff highlighting aspect was definitely a minor improvement compared to some of the other idea/etc included above.

---

Theoretically it would require to ship a special syntax for diff which doesn't require +/- to start at the very first column, but that's out of scope for this package.

Syntaxes are embedded by scope, so it would theoretically be possible to create a more lazy one overriding the default diff syntax, which would then be picked up.

Thanks for the insight, that makes sense. I don't think it's a big enough issue for me to need to go down that path; but if it is for others, good to know how it theoretically could be achieved.

---

Input helpers have various short commings as they rely significantly on sublime-macros with lots of regexp black magic. There's much to improve.

The noted cases shouldn't be too hard to fix, I guess.

Yeah, that makes sense. I figured there was probably a bunch of 'black magick' going on with them.

If they can be improved for these sorts of cases, that would be awesome!

[0xdevalias]

I figured out part of the other 'rendering issue' I noticed previously:

Given this text:

- > This is some blockquote text nested in a list

It renders fine:

If we indent it 4 spaces, it loses it's highlighting/etc (presumably because it's being treated as a non-fenced codeblock):

    - > This is some blockquote text nested in a list

If we add earlier list items, it renders fine:

- This is an earlier list item
  - And another
    - > This is some blockquote text nested in a list

In some cases (which I have yet to figure out how to reproduce properly), when I am working with nested blockquotes like this, it seems to get stuck in a weird 'half and half' case, where some of the blockquote is properly noticed/rendered, but part of it is in that 'raw' state.

I'll try and pay attention when it happens again, and figure out a minimal reproduction for it.

[0xdevalias]

Ok, I found an example. Not sure if this is the only case where it happens, but it seems if there is a single _ in the proceeding line, the blockquotes on the following lines aren't coloured properly:

- https://twitter.com/_devalias/status/1753322272293896385
  - > The code is still pretty hacky.. but pulled the PoC's together into a usable script for filtering the noise in the diff output!
    > 
    > Still not perfect.. but I am really liking these stats so far!
    > 
    > For an 8.4mb _app.js file (250,022 lines):
    > 
    > Original diff: 33,399
    > Filtered: 7,516

Presumably this is because the highlighting code is assuming that I intended for that to be italics/similar.

If I quote it in backticks, you can see that the highlighting works again:

- https://twitter.com/_devalias/status/1753322272293896385
  - > The code is still pretty hacky.. but pulled the PoC's together into a usable script for filtering the noise in the diff output!
    > 
    > Still not perfect.. but I am really liking these stats so far!
    > 
    > For an 8.4mb `_app.js` file (250,022 lines):
    > 
    > Original diff: 33,399
    > Filtered: 7,516

Though ideally, that wouldn't be required to have it work (eg. GitHub's markdown highlighting seems to account for it ok)

[deathaxe]

I am aware of those, but supporting them causes a lot of more complexity in syntax definitions. ST's syntax engine parses text in a single run, only, while Markdown requires multiple stages to get it done right. To simulate that in a single run, many special case contexts and patterns are required to handle all combinations of those nested statements. Combined with ST's line blindnes, which requires extra complex rules to handle emphasis (italics, bold) across multiple lines, that's probably not wordth it to support all those weird edge cases. Markdown syntax is already slow enough.

[0xdevalias]

nods that's fair enough. It's not the end of the world, the other things mentioned earlier are definitely more 'real world UX' issues for me.

Since it's not possible/easy to do it well for all the edge cases though, I wonder if there is some validity to having the option to disable the colour change of quote blocks (without having to go and create a custom style override rule):

• Allow blockquote visual style changes to be disabled so I can have my >'s back #741
• Add option to restore > blockquote style #659

[deathaxe]

... option to disable the colour change of quote blocks

That's not possible via options.

[0xdevalias]

Fair.