10000 GTK applications display missing glyph instead space after emoji characters · Issue #2 · 13rac1/emojione-color-font · GitHub
[go: up one dir, main page]
Skip to content
This repository has been archived by the owner on Nov 10, 2018. It is now read-only.

GTK applications display missing glyph instead space after emoji characters #2

Closed
13rac1 opened this issue Feb 25, 2016 · 20 comments
Closed

GTK applications display missing glyph instead space after emoji characters #2

13rac1 opened this issue Feb 25, 2016 · 20 comments
Assignees
@13rac1
Labels
bug external unmasked regression
Milestone
v1.0

Comments

@13rac1
Copy link
Owner
13rac1 commented Feb 25, 2016

I thought this was a fluke earlier, but it's definitively not. The header in Firefox displays the missing glyph character in some instances. I may need to adjust the null character or add a space. Could be related to #1, but I doubt it at this point.

incorrect-glyph
@edent
Copy link
edent commented Feb 26, 2016

Just to say I also see this issue.
graphemica font issue

@13rac1 13rac1 added the bug label Feb 26, 2016
@13rac1
Copy link
Owner Author
13rac1 commented Feb 27, 2016

@edent I have confirmed the cause and a workaround for this issue, but I'm trying to figure out an ideal solution. Can you help?

The problem is: this font does not, on purpose, have a space character. The OpenType spec recommends all fonts have a space character: https://www.microsoft.com/typography/otspec/recom.htm There isn't one is because we want to fallback to the next font for all other characters. The space character has different widths in different fonts and font styles/weights. The width of a space is drastically different between a monospace font and a sans-serif font for example. This shouldn't be a problem (AFAIK), but FreeType(or something else in the font render stack) isn't switching back to the originally selected font after loading character(s) from Emoji One Color.

The workaround is to create multiple font files changing only the width of the space character, but that's ridiculous. The actual solution is, probably, to determine what is causing that extra space to load.

The DejaVu font family, primary default fallback on many Linux distributions, has five different space widths alone:

$ grep "\"space\" w" *
DejaVuSans-BoldOblique.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSans-Bold.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSansCondensed-BoldOblique.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSansCondensed-Bold.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSansCondensed-Oblique.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSansCondensed.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSans-ExtraLight.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSansMono-BoldOblique.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSansMono-Bold.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSansMono-Oblique.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSansMono.ttx:    <mtx name="space" width="1233" lsb="0"/>
DejaVuSans-Oblique.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSans.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSerif-BoldItalic.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSerif-Bold.ttx:    <mtx name="space" width="713" lsb="0"/>
DejaVuSerifCondensed-BoldItalic.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSerifCondensed-Bold.ttx:    <mtx name="space" width="641" lsb="0"/>
DejaVuSerifCondensed-Italic.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSerifCondensed.ttx:    <mtx name="space" width="585" lsb="0"/>
DejaVuSerif-Italic.ttx:    <mtx name="space" width="651" lsb="0"/>
DejaVuSerif.ttx:    <mtx name="space" width="651" lsb="0"/>
$ cat *.ttx | grep "\"space\" w" | sort | uniq | wc
      5      20     221

Apple sets it to zero:

$ grep "\"space\" w" Apple\ Color\ Emoji.ttx 
    <mtx name="space" width="0" lsb="0"/>

I've generated a font with a full-width space character to test. The space is amusingly obvious in the header:
toomuchspace

And a problem when a sans-serif font is requested:
notgoingtoworkspace

@13rac1
8000 Copy link
Owner Author
13rac1 commented Feb 27, 2016

Haha! It happens in more than just the application titles, tool tips and taskbar in Ubuntu 14.04:
bad-space-not-just-firefox

Pluma is set to use "Monospace". The font render system shows space from the previous font until any other character appears.

@13rac1
Copy link
Owner Author
13rac1 commented Feb 27, 2016

I've narrowed down the cause of this text display problem (other than the missing space in the font) and decided on a best case solution for now without going further down this rabbit hole.

I tried a Kubuntu 15.10 LIveCD:
kubuntu-is-fine
No incorrect space characters in the taskbar. Hmm, is this GTK vs QT?

I tried a few QT vs GTK applications on my install, then ran a QT text field demo and a GTK text field demo:
qt vs gtk
There we go. I've already looked into this issue enough, so I'm done for now.

Best case solution given current Linux distribution defaults is to make two versions of the font. One matching the width of spaces in DejaVu Serif/Sans-Serif, and the other matching DejaVu Monospace.

I'm going to leave this issue open for a while before making these two additional fonts to see if anyone has a better solution. Either way, the ideal solution is far outside of my control in this font.

@13rac1 13rac1 mentioned this issue Mar 4, 2016
Closed
@13rac1 13rac1 changed the title Firefox title/tabs displays missing glyph for space after emoji characters GTK applications display missing glyph instead space after emoji characters Mar 4, 2016
@13rac1 13rac1 self-assigned this Mar 4, 2016
@13rac1
Copy link
Owner Author
13rac1 commented Mar 5, 2016

According to the Fontconfig docs it seems possible to exclude a <range> of unicode characters in a font from being used. Emoji One Color doesn't need to be the default font if DejaVu can be overridden for a few unicode ranges. I'm having trouble figuring out what configuration is needed, plus this GTK bug (right? It's not a problem in OSX, Windows, or QT) would probably still cause the same problem.

Potential temporary workarounds:

  1. Create two versions of the font for Linux, one using the 651 space width of DejaVu Serif & Sans-Serif and one using the 1233 space width of DejaVu Sans Mono. Modify the fontconfig fonts.config recommendation to specify the correct fallback. Doubles disk space use, but would work. Technically would have the incorrect spacing for some fonts.
  2. Create DejaVu Serif, Sans-Serif and Monospace fonts without the emoji we need to override.
  3. ???

@IBBoard Any other ideas?

@IBBoard
Copy link
Contributor
IBBoard commented Mar 5, 2016

I've had a look at the Fontconfig documents and it isn't clear. From the tags listed in the docs then having <range> with a value that is above the ASCII range seems like it should work to say "Emojione only supplies things outside this range", but there are no decent examples for using it that I've found and trying to add it in where I think it should go just breaks the standard overrides!

(Also, sorry for the dupe. I don't know how I missed this ticket!)

@13rac1
Copy link
Owner Author
13rac1 commented Mar 5, 2016

I've had a look at the Fontconfig documents and it isn't clear.

+:100: It works really well, but the config is so complex.

(Also, sorry for the dupe. I don't know how I missed this ticket!)

No problem. I'm always careful when closing issues as dupes, because I don't want to deter contributions/help. I appreciate other people looking into this issue.

Bug description
When GTK (pango?) renders a unformatted text string with characters from multiple fonts using the font fallback system, it uses the whitespace character from the last used font until it encounters any other visible non-whitespace character. If the last selected font doesn't have that whitespace character an error character is displayed.

I've confirmed this is a problem for nbsp 0xa0 too, so the font needs all whitespace characters.

Relevant bugs on gnome.org:

@13rac1
Copy link
Owner Author
13rac1 commented Mar 6, 2016

I've updated the spaces are half as long after a unicode symbol gnome.org bug with a screenshot and relevant details (aka nothing about this font, because it's not relevant to the specific issue.)

Workaround Plan: I'm going to make two versions of the font for Linux. One with monospace whitespace characters and one with the normal DejaVu Sans lengths. Slight change to the font.conf and the error characters will disappear.

@13rac1 13rac1 removed the help wanted label Mar 6, 2016
@13rac1 13rac1 mentioned this issue Mar 6, 2016
Closed
@crepererum
Copy link
8000

Might be related (otherwise I'll open an extra issue):

Environment

Application: Evolution (GNOME 3.16)
Global configs (GNOME):

  • Font Scaling Factor = 0.8
  • Window Scaling (HiDPI) = 2

Screenshots

Without the fontconfig file:
screenshot from 2016-03-07 12-00-07

With the fontconfig file (from project README):
screenshot from 2016-03-07 12-00-34

@13rac1
Copy link
Owner Author
13rac1 commented Mar 7, 2016

@crepererum Yes, although I haven't seen this specific issue it looks like a manifestation of the same issue. I'll be creating a set of fonts to workaround this issue with GTK(pango). Please hold off on a separate issue until then. Thank you!

There may be a fontconfig conf solution so Emoji One Color can only be the default font for the specific unicode ranges it provides, but I cannot figure out what arcane XML magic is required. So any help on that is appreciated. Haha! There isn't see #17

@IBBoard
Copy link
Contributor
IBBoard commented Mar 7, 2016

@eosrei That's what I was looking at. The obvious <range><int>128</int><int>100000</int></range> just seemed to break the override and I hadn't found a schema, just the vague docs without examples!

(Unless, of course, I'm misunderstanding the documents and it has no way of specifying Unicode ranges to exclude - but then <blank> sounds like it should already ignore our bad blanks, which I guess it does everywhere but GTK/Pango)

@13rac1
Copy link
Owner Author
13rac1 commented Mar 7, 2016

@IBBoard I'm fairly sure that, even if we can determine a correct fontconfig, it won't fix this issue due to the bug (design "feature"?) in Pango/GTK. 😞

Note to anyone who may have something to add to that bug report: Please don't mention this font, it'll complicate the resolution.

@13rac1 13rac1 mentioned this issue Mar 9, 2016
Closed
@13rac1
Copy link
Owner Author
13rac1 commented Mar 11, 2016

Status update

There are 25 kinds of whitespace in unicode:https://en.wikipedia.org/wiki/Whitespace_character#Unicode I am creating two "starter" fonts. One using the DejaVuSans whitespace widths and the other using the DejaVuSansMono widths. I'll add a new option to SCFBuild to accept an input file, then generate two versions of the font for Linux and make a zip file with both.

@13rac1 13rac1 added this to the v1.0 milestone Mar 13, 2016
This was referenced Mar 13, 2016
Closed
Closed
@13rac1
Copy link
Owner Author
13rac1 commented Mar 15, 2016

The OpenType Spec recommends a font have the space character, but doesn't require it. Reference: https://www.microsoft.com/typography/otspec/recom.htm IMO, all of these programs expecting a space are at fault. Haha! it's a bit amusing really, but I wish I were wrong. I am going to add all of the recommended characters to both fonts to avoid this.

13rac1 added a commit to 13rac1/scfbuild that referenced this issue Mar 18, 2016
@13rac1
create_font: Add minimum recommended characters
9c32d91 
Specifically add notdef, null, CR and space. Space is hardcoded to
a width of 1024. Todo: Space width should be a config yaml option.

The font rendering systems are supposed to fallback to the space width
in the next available fallback font, but it doesn't work reliably.

Workaround for: 13rac1/emojione-color-font#2
Caused by: https://bugzilla.gnome.org/show_bug.cgi?id=757785
Related: 13rac1/emojione-color-font#17
13rac1 added a commit that referenced this issue Mar 21, 2016
@13rac1
build: Add space width equal to DejaVu/Vera Sans
e7eed4a 
Requires Bitstream Vera as the default serif, sans-serif, and
monospace font family since it contains no emoji characters.
Any other emoji-less font can be used, but most systems include
Bitstream Vera so it is used here as the example default. Noto
works well except primarily for 0x2639 and 0x263a.

See fonts.conf example:
fontconfig/user-bitstream-vera-fonts.conf

Workaround fix for #2
Related to #17
@13rac1
Copy link
Owner Author
13rac1 commented Mar 21, 2016

This issue is "fixed" by adding a space character with a width equal to DejaVu and Bitsteam Vera Sans then making Bitstream Vera or Noto the default font family. Bitstream Vera contains no emoji characters, and Noto contains very few. See soon-to-be-recommended-default fonts.conf here: https://github.com/eosrei/emojione-color-font/blob/245e659fe129eeceb1381d726c9ea717abcf6a73/fontconfig/user-bitstream-vera-fonts.conf New v1.0-beta3 release coming up soon.

@13rac1 13rac1 closed this as completed Mar 21, 2016
@13rac1 13rac1 mentioned this issue Mar 21, 2016
Closed
@IBBoard
Copy link
Contributor
IBBoard commented Mar 21, 2016

So wait, let me get this right - this new config will default the text to Bitstream fonts rather than whatever we've set in (for example) Gnome Tweak?
Fixing the font issues is good, but I like my Liberation Sans! I'll have to try fiddling with the configs.

@13rac1
Copy link
Owner Author
13rac1 commented Mar 21, 2016

@IBBoard You can change it to whatever you want, but I need to create/produce/distribute a working "standard" out-of-the-box install method. I've looked at fontconfig for hours and hours and I'm so over it. :-/ Bitstream Vera is the only solution I've found other than forking DejaVu/Noto/Liberation since they all contain emoji characters. Glyph blacklist (my initial hope in #17) has been a wishlist item for 10+ years. Ah yes, Liberation contains: ☺0x263A. So, I cannot recommend it either.

whatever we've set in (for example) Gnome Tweak?

AFAIK Gnome Tweak does not change the fontconfig. I recommend you install my provided conf, then adjust that for your specific needs. Beta3 cannot be your default font because it includes a space character and will break monospace formatting. Not having a space character is what caused this issue to exist, so it's sadly not an option either.

I'm open to any other solutions!

@IBBoard
Copy link
Contributor
IBBoard commented Mar 21, 2016

From a bit of poking then I don't quite know how Gnome Tweak does it. Replacing "Bitstream Vera" with "Liberation" (then replacing "Sans Mono" with "Mono") seems to work. I can managed with the white smiley face being B&W - it's the tiny one that I try to avoid anyway, and most people probably use the nearby one that's full-size.

I know what you mean about fiddling with Font Config, though. I didn't do anywhere near as much as you've gone and it was a pain. It seems like you should be able to do things (like selectively pick fonts for certain character ranges) but none of it is documented and none of it works.

Thanks for all the work!

@13rac1
Copy link
Owner Author
13rac1 commented Mar 21, 2016

I can managed with the white smiley face being B&W - it's the tiny one that I try to avoid anyway, and most people probably use the nearby one that's full-size.

Haha, it's actually a full size emoji in the source and works correctly with Bitstream Vera. I may create/document how to subset the other fonts in the future, but it's not something I want to deal with right now.

(like selectively pick fonts for certain character ranges)

Nope! It's only a feature request! 😭

Thanks for all the work!

Sure! Thanks for testing/using it.

@13rac1
Copy link
Owner Author
13rac1 commented Mar 24, 2016

FYI The GTK/Pango issue has been acknowledged and the creator of Pango has described a good solution. Not sure the resolution will make it downstream to current distributions, but it'll be corrected in the future. 🎉 😄

If font before and after space are both the same and they support space, then we should use that. However, if the font used for before and after are different, then currently we choose the one used for before. However, we should use the one between before/after that has a higher priority. Fixing that will fix this case.

@13rac1 13rac1 mentioned this issue Mar 26, 2016
Closed
@NBonaparte NBonaparte mentioned this issue Feb 5, 2017
Open
@13rac1 13rac1 mentioned this issue May 16, 2017
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@13rac1 13rac1

Labels
bug external unmasked regression
Projects
None yet
Milestone
v1.0
Development

No branches or pull requests
4 participants
@13rac1 @IBBoard @edent @crepererum
0