Pango Font Rendering

[patrick96]

This is my comment in #2186 (comment) converted to an issue

TL;DR: Polybar font rendering is very basic and doesn't do unicode too well. We want to support font rendering through pango.

Using pango would allow us to address the following issues:

• Garbage displayed after 6-byte emoji. #2186
• Persian (Farsi, RTL, Right-to-Left) text not showing properly. #2062
• Use fontconfig fallbacks? #2247
• Regional Indicator Symbols are displayed incorrectly #686
• Support OpenType features #2369
• Ligature support #2360

If you are familiar with pango, we would appreciate your input, please read the last section.
For more info, background, and concerns, please read on.

---

The issue hints at the underlying issues polybar has with font rendering: We do it ourselves using the cairo low-level text api.

This is an excerpt from the cairo text api:

The functions with glyphs in their name form cairo's low-level text API. The low-level API relies on the user to convert text to a set of glyph indexes and positions. This is a very hard problem and is best handled by external libraries, like the pangocairo that is part of the Pango text layout and rendering library. Pango is available from http://www.pango.org/.

As I understand it, this means any kind of algorithm to display all kinds of unicode control characters would have to be done by us, which we don't by the way.

This creates a number of issues:

• The one described here because the symbols and their variant selectors (or other control characters) are potentially rendered using different fonts and have no chance to influence each other. This also means polybar can't handle things like different emoji skin tones, and displaying emoji flags.
• We can't handle right-to-left text Persian (Farsi, RTL, Right-to-Left) text not showing properly. #2062
• These are not related to unicode rendering, but can probably be solved by using pango:
  • Supporting ligatures and other font features (Support OpenType features #2369, Ligature support #2360)
  • Fontconfig fallbacks (Use fontconfig fallbacks? #2247)

---

Now, to address some comments in this thread:

From @parmort

what do you think? Should Polybar handle the variation selection better? If so, how?

Yes! We should definitely have proper font handling. Filtering out control characters is at best a temporary workaround to not mess up people's bars.

My uneducated guess is that Polybar is displaying each character at a time, so character sequences aren't picked up by fonts. If this were the case, the variation selectors should be filtered out or something.

This is part of the problem, but I am not sure that passing the character together with its variant selector would make a difference. It's likely that the character and variant of the character are two different glyphs inside the font, but I don't know enough about unicode and fonts to be sure.

Polybar doesn't always render each character at a time, but due to the way we do font fallback, the variant selector may be displayed using a different font. That's because we treat each character as a printable character and try to find the first font in the list that can display it. So what happens in the original post is, as @parmort has correctly identified, polybar sees that the emoji font can display the sun emoji, but it doesn't find a glyph for the variant selector in the emoji font. Why would it? It's not a printable character. It then goes further down the list and the unifont font happens to have a glyph for it because it tries to provide printable characters for almost every unicode code point (even if they're control characters).

From @oldmansutton and @mainrs

EDIT2: Alternately, defaulting to automatically display the variation glyphs like github is doing would also be acceptable (to me)

This wouldn’t solve the problem but just shift it. Now people that want the normal variation can’t properly display the emoji. :)

I agree.

---

Any guidance on where or what has to be changed to make this work? - @mainrs

None of this has been particularly actionable information.
On a high-level, here is what I think needs to happen: Polybar needs to be able to do font rendering using pango.

How exactly this will/should happen, I am not sure. I have barely any experience with pango, cairo, or font rendering in general (the font renderer was here before I started contributing to polybar).

From everything I have read, it may not be possible to just replace the font rendering with pango without breaking existing functionality.
For one, automatic font fallback may not be possible in polybar, though explicit font selection using %{T} could still work.

If anyone has experience with pango, I would appreciate your insight here:

1. What options do we have to define fallback fonts for pango? The only thing I have found was on the side of fontconfig, but nothing on the application side.
2. I have some concerns around our formatting tags and translating that to pango. If we have a string like %{F#ff0}some %{B#0ff}text, each text fragment that doesn't contain any tags is dispatched to the renderer and the renderer applies the currently set colors. From what I can tell, pango can also do that. What concerns me is how control characters like the left-to-right override (and other stateful format controls) behave. If the first text contains an LTR override character, is the second text also rendered LTR if I call pango_layout_set_text with it?

The answers to this will inform how exactly our solution will look like (maybe we need to translate all formatting tags to pango markup?).

In any case, I think pango rendering should be a separate code path while still leaving the current functionality in tact. There should maybe be a setting in the bar section to switch to the new system. If so, it should be as "simple" (the dispatch is simple, the rendering itself probably not) as to distinguish between the two cases in renderer::render_text and call the right font rendering code.

---

References:

• Cairo + Pango example: https://www.cairographics.org/FAQ/#using_pango

[OJFord]

I only have experience as an application user, but

If anyone has experience with pango, I would appreciate your insight here:

1. What options do we have to define fallback fonts for pango? The only thing I have found was on the side of fontconfig, but nothing on the application side.

Isn't that by design? That fonts and fallbacks are a user (and system) setting, not an application setting?

My i3 config for example contains:

font pango:Monospace 10

and what Monospace is, and its fallbacks, is defined (once, for i3 and for everything else) in my fontconfig/fonts.conf. (A bit more on that in #2247 which you linked above.)

Probably I'm misunderstanding or missing something, but I don't see why Polybar would need or want control of its own fonts and fallbacks, beyond well-knowns like monospace or sans?

[patrick96]

Isn't that by design? That fonts and fallbacks are a user (and system) setting, not an application setting?

Probably. It would also make polybar's job a lot easier. The reason I am looking for fallback on the application side is because that's what polybar currently provides and, if possible, I'd like to keep this functionality even in the new system to make the transition as smooth as possible.

If there is indeed no (good) way to do this, the pango rendering will deprecate both the font list and the %{T} font tag.

Thanks for your input, @OJFord 😃

[mainrs]

Isn't that by design? That fonts and fallbacks are a user (and system) setting, not an application setting?

Probably. It would also make polybar's job a lot easier. The reason I am looking for fallback on the application side is because that's what polybar currently provides and, if possible, I'd like to keep this functionality even in the new system to make the transition as smooth as possible.

If there is indeed no (good) way to do this, the pango rendering will deprecate both the font list and the %{T} font tag.

Thanks for your input, @OJFord smiley

I do understand that mindset of backwards compatibility. However, this is a rather large change, and a substantial one as well. I honestly think it would be better to mark it as a breaking change and bump the major version one up.
If backwards compatibility is kept in polybar, it would only introduce more code to maintain in the long run. The old font logic would be dropped anyway eventually. And I think it would be better to do so directly.

[patrick96]

I would disagree. The old font rendering should be kept in polybar, deprecated, for at the very least one release before it is removed in a major version bump. Otherwise, we break everyone's config without giving them time migrate.

It's also a good way to gather feedback and find missing features compared to the old system and giving users a way back if some use-cases are not possible yet.

It's not like the current font rendering code is being maintained right now. There has been one non-trivial change in the last five years. Also, once it's deprecated, it is also not supported anymore, everyone that has issues should migrate to the new system.

[mainrs]

I guess is just a perspective way. I would create a release candidate that sticks longer around to gather feedback. You would introduce the new feature in a minor bump before setting it as default :)

I do a lot of work with NodeJS and JS where major bumps happen pretty much regularly. Maybe that mindset just rubbed off onto me :p. People who want new features can just bump their version. People that don't can stick to the old release; they don't disappear.

But I do get your argument. It might fit the project better that way.

[patrick96]

I think there are also some practical reasons for this. In node, it's trivial to downgrade a package, on most distros, you don't have that option without compiling from source. Also upgrading is a non-choice, you have to explicitly decide not to upgrade. So, once people upgrade, it's already too late. We do place quite a big burden on people if we break existing configs.

Also, anything that changes existing behavior leads to tons of help requests on reddit and issues opened here. Breaking changes require lots of good documentation and communication.