Support word wrapping for HTML encoder

[nathany]

When viewing code on a small screen or with a large font, it's nice to have word-wrapping rather than panning around. Ideally this word wrapping would adhere to the rules of the language in question to make it more readable.

I've done some research and a mockup.

While I'm happy to continue this little experiment on my own, I'm curious as to what would be involved in incorporating it into CodeRay and supporting multiple languages? :-D

[nathany]

@korny Have you had a chance to look at this mockup? Do you have any interest in supporting this feature in CodeRay?

I could attempt to do this as an external gem, but to do it well, I think the lexers would need to be revised. The encoder wants to know if there can be a soft-wrap (before or) after a given token, which is specific to the language.

Often we can wrap the line while keeping the code valid, eg. after a . in Ruby.

After a variable we can't line wrap, nor should the space(s) after it cause a break. Eg. say a = 1 we can break after the = operator, but not before.

Interested in your thoughts.

[korny]

Mmh, interesting. Seems to work differently in Firefox and Safari, but I'm not sure I completely understand what it's doing ;)

Two things:

• Using Ripper makes this, as you write, a Ruby-only solution. I think this makes it a separate project, maybe something that can work together with or use CodeRay, but a standalone Gem. I would very much like to do "perfect scanning", but I think a real lexer (that would throw an error on invalid input) and a highlighter (that should eat everything) have different goals.
• As for the intelligent wrapping, I think this would be awesome as an option, especially looking at mobile. If I'm understanding this right, you're using .token-group to mark spans that should not be wrapped, and some clever CSS to let the browser do the indentation? Do you have a description of the general concept behind this, something that could be extended to all languages?

[korny]

By the way, I'm warming up to Solarized, only the red operators are a weird choice…

[nathany]

I've yet to write out the details of how this works. It's still experimental, and failing entirely in IE9.

Use of Ripper and Solarized styles are of course orthogonal to this particular encoder.

[nathany]

I discovered a better way to do wrapping. Rather than my zwsp (zero width space) with a bunch of CSS, we can just use a <wbr> tag. It's working well for me in Chrome 24, Firefox 18 and Safari 6... but not at all in IE 9. I'm downloading IE 10 to give that a try. If supporting IE is important, I found a lengthy article that could help.

At the most basic level, if there is a word separator (eg. ./\()\"'-:,.;<>~!@#$%^&*|+=[]{}~?``) that isn't followed by a space, we inject a tag so that the browser can optionally wrap it. The nice thing with` is that there is no actual space in the HTML code, so copy & paste continue to work nicely.

The stuff I was doing with .token-groups is a nice idea, but goes beyond what editors do (trying to keep the code valid).

I was also experimenting with hanging indents, which are nice, but still have some problems. Right now I'm using CSS for indentation, without any spaces for copy & paste. That would necessitate a Gist-style <> raw button. There's also a matter of handling hard-wrapped code, particularly when it is indented more than the typical 2 spaces. Finally, CSS ems are based on the height of the font, so it may not be that reliable for setting the text-indent. With the font I'm using 1em seems to be worth 2 characters, but it could vary heavily with other monospace fonts.

Still interested? I have yet to do a more thorough writeup.

[korny]

Still interested?

Yes, very. It could be a wonderful "wow"-feature for any highlighter (and sure be copied by others) – if we get it to work on IE, too. So the idea would be to use   for spaces (so it doesn't wrap by default) and <wbr> where it actually can? Do you have a working example?

Also, how can you add that functionality to scanners without cluttering the already-complex logic? Would it be easier with a DSL like Rouge/Pygments?

[nathany]

The only example I have so far is the mockup at http://www.tint.io/. I added you as a collaborator to my tint-web repo, if you want to see the hacky code behind it.

I'd like to start simple, and add basic wrapping, then hanging indents, and finally maybe look at lexer-specific line continuations.