[Translation] Make plural translations more standard #10152

goetas · 2014-01-28T13:51:40Z

Hi everybody.

I'm developing a multilingual site. We are using pootle (http://pootle.translatehouse.org/) to provide a web interface to translators.

Pootle and many other tools use a different convention than symfony to define translation strings.

Gettext PO file using plural forms can be something like:

msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

XLIFF file using plural forms can be something like:

<group restype="x-gettext-plurals" id="untranslated-string-singular">
    <trans-unit id="untranslated-string-singular[0]">
      <source>untranslated-string-singular</source>
      <target>translated-string-case-0</target>
    </trans-unit>
    <trans-unit id="untranslated-string-singular[N]">
      <source>untranslated-string-singular</source>
      <target state="translated">translated-string-case-n</target>
    </trans-unit>
</group>

While using symfony conventions:

$translatos->transChoice(
  "{0} There are no apples|{1} There is one apple|]1,Inf] There are %count% apples", 
$number);

we have always one rappresentation:
Gettext:

msgid "{0} There are no apples|{1} There is one apple|]1,Inf] There are %count% apples"
msgstr "{0} Translated...|{1} Translated...|]1,Inf] Translated..."

XLIFF

<trans-unit id="untranslated-string-singular[0]">
  <source>{0} There are no apples|{1} There is one apple|]1,Inf] There are %count% apples</source>
  <target>{0} Translated...|{1} Translated...|]1,Inf] Translated...</target>
</trans-unit>

Using symfony plurals, tools like http://virtaal.translatehouse.org/ or http://pootle.locamotion.org/ can't detect plural forms and can't present those sentences in a good way.

For example: http://pootle.locamotion.org/fr/pootle/translate/pootle.po#search=Please%20correct%20&sfields=source,target&unit=182009
Here pootle automatically splits the translation into many textarea based on the number of French plurals (Polish has three plural forms).
This approach avoids translators' mistakes, and does not relay on translators' ability to manipulate plural expressions (1]10] | ]10],inf[).

I'm going to make some changes to transChoice implementations.
I suggest to pass an array as first parameter (instead of a string). Example:

$translator->transChoice(array("There is one apple", "There are %count% apples"),
    10, array('%count%' => 10)
);

Also twig templates can take advantage from this change.

Since I'm thinking to use arrays, this change would be BC compatible.

Involved parts should be also:

MessageCatalogue: messages can be an array... (this can be difficult)
LoaderInterface: have to read correctly msgstr[N] or <group restype="x-gettext-plurals"> expressions.. and save data into MessageCatalogue
DumperInterface: have to read MessageCatalogue, and arrays should be saved correctly

May this change be accepted?

The text was updated successfully, but these errors were encountered:

stof · 2014-01-28T18:34:56Z

Changing the way messages work would be a BC break, so it cannot be accepted in 2.x. And I'm not sure it makes sense to accept it for 3.0 either.
A better solution would be to build a loader/dumper supporting the file formats used by Pootle instead of modifying the whole component so that its internal storage of messages matches the Pootle format. This is why the loading and dumping is its own layer in the component. There is no need to change everything else.

and using an array as source of the translation does not make much sense IMO. The place where you really need alternatives is after the translation, not before (for instance, when using a translation key rather than the English string as source, you won't have alternatives in the source).

hason · 2014-01-29T07:30:24Z

Instead, we should provide support for MessageFormatter #6009.

goetas · 2014-01-29T09:31:21Z

@hason MessageFormatter can solve some problems (eg: message representation) but p refer to have a non "php only" solution.

I want to separate developer responsibility from translators responsibility and ability.
Most translators can encounter some difficulties to understand strings as
{0,choice, 0#Brak nowych wiadomości |1#Masz jedną nową wiadomość |1<Masz {0,number} nowe wiadomości |4<Masz {0,number} nowych wiadomości}.

Thinking in i18n way: the developer have to write code in his own language (eg: english) and have to care of it's pluralization rules.

transChoice(array("one apple", "%count% apples"),
    10, array('%count%' => 10)
);

Later, this code should be converted into PO/XLIFF... (english pluralization rules also here)
Then, using some tools, created for translators (not developers), the sentences should be translated.
I the end, a developer can import the new PO/XLIFF files into the application

goetas · 2014-01-29T09:45:51Z

@stof I agree that it can be complicated and BC breaking...
If my application (source code too) is written using polish language, i need three alternatives. Later this application has to be translated for an english audience.

Implementing pluralization inside loaders/dumpers can be a good idea.
The solution can be to "explode" strings with "|" symbol during translation dumping.
While loading, implode these strings with "|".

Can it be easier?

stof · 2014-01-29T09:51:08Z

@goetas it can be easier and fully BC, so it is a better idea.

aimeos · 2014-09-14T11:23:34Z

It's impossible to integrate existing .po files when using the Symfony translator. The plural forms are

msgid untranslated-string-singular
msgid_plural untranslated-string-plural
msgstr[0] translated-string-case-0
...
msgstr[N] translated-string-case-n

while expected by the Symfony Translator is

msgid "{0} There are no apples|{1} There is one apple|]1,Inf] There are %count% apples"
msgstr "{0} Translated...|{1} Translated...|]1,Inf] Translated..."

I would urgently suggest to change the format the loaders are expecting to the standard one too to make it widely usable.

…formatter (aitboudad) This PR was merged into the 3.4 branch. Discussion ---------- [Translation] added support for adding custom message formatter | Q | A | | --- | --- | | Branch? | master | | Bug fix? | no | | New feature? | yes | | BC breaks? | no | | Deprecations? | yes | | Tests pass? | yes | | Fixed tickets | #6009, #10152, one item in #11742, #11948 | | License | MIT | | Doc PR | ~ | Commits ------- 42183b0 [Translation] Support adding custom message formatter

goetas · 2017-09-12T12:42:49Z

yay! great!

This PR was squashed before being merged into the 3.4 branch (closes #31266). Discussion ---------- [Translator] Load plurals from po files properly | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #10152 (comment) | License | MIT | Doc PR | Plurals were not handled correctly when loading po files. ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Before, the po entry above was treated as two entries, which doesn't make sense: ``` 'foo' => 'bar' 'foos' => 'bar|bars' ``` With this PR, it is treated as one entry: ``` 'foo|foos' => 'bar|bars' ``` Commits ------- 6b69a99 [Translator] Load plurals from po files properly

This PR was squashed before being merged into the 3.4 branch (closes #31266). Discussion ---------- [Translator] Load plurals from po files properly | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | symfony/symfony#10152 (comment) | License | MIT | Doc PR | Plurals were not handled correctly when loading po files. ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Before, the po entry above was treated as two entries, which doesn't make sense: ``` 'foo' => 'bar' 'foos' => 'bar|bars' ``` With this PR, it is treated as one entry: ``` 'foo|foos' => 'bar|bars' ``` Commits ------- 6b69a99230 [Translator] Load plurals from po files properly

This PR was merged into the 3.4 branch. Discussion ---------- [Translator] Load plurals from mo files properly | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #10152 (comment) | License | MIT | Doc PR | Plurals were not handled correctly when loading mo files. ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Before, the mo entry above was treated as two entries, which doesn't make sense: ``` 'foo' => 'bar' 'foos' => '{0} bar|{1} bars' ``` With this PR, it is treated as one entry: ``` 'foo|foos' => 'bar|bars' ``` This PR does the same as #31266, just for mo files instead of po files. Note, however, that the old behavior differed between po and mo files: `'foos' => 'bar|bars'` for po files and `'foos' => '{0} bar|{1} bars'` for mo files. Commits ------- 97d28b5 Load plurals from mo files properly

This PR was merged into the 3.4 branch. Discussion ---------- [Translator] Load plurals from mo files properly | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | symfony/symfony#10152 (comment) | License | MIT | Doc PR | Plurals were not handled correctly when loading mo files. ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Before, the mo entry above was treated as two entries, which doesn't make sense: ``` 'foo' => 'bar' 'foos' => '{0} bar|{1} bars' ``` With this PR, it is treated as one entry: ``` 'foo|foos' => 'bar|bars' ``` This PR does the same as #31266, just for mo files instead of po files. Note, however, that the old behavior differed between po and mo files: `'foos' => 'bar|bars'` for po files and `'foos' => '{0} bar|{1} bars'` for mo files. Commits ------- 97d28b5e4e Load plurals from mo files properly

…tadly) This PR was submitted for the master branch but it was squashed and merged into the 4.4 branch instead (closes #31269). Discussion ---------- [Translator] Dump native plural formats to po files | Q | A | ------------- | --- | Branch? | master | Bug fix? | no | New feature? | yes | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #29963, #10152 | License | MIT | Doc PR | Implementing support for dumping to the native po plural format. ``` 'foo|foos' => 'bar|bars' ``` Before, the entry above was dumped directly: ``` msgid "foo|foos" msgstr "bar|bars" ``` With this PR, it is dumped using the native po plural format: ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Strings using explicit rules or contain more than 2 pluralization forms are still dumped directly, as the po format does not support such cases: ``` '{0} no foos|one foo|%count% foos' => '{0} no bars|one bar|%count% bars' ``` ``` msgid "{0} no foos|one foo|%count% foos" msgstr "{0} no bars|one bar|%count% bars" ``` This PR complements #31266, fixing loading of native po plural formats. Commits ------- dc31739 [Translator] Dump native plural formats to po files

This PR was squashed before being merged into the 3.4 branch (closes #31266). Discussion ---------- [Translator] Load plurals from po files properly | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | symfony/symfony#10152 (comment) | License | MIT | Doc PR | Plurals were not handled correctly when loading po files. ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Before, the po entry above was treated as two entries, which doesn't make sense: ``` 'foo' => 'bar' 'foos' => 'bar|bars' ``` With this PR, it is treated as one entry: ``` 'foo|foos' => 'bar|bars' ``` Commits ------- 6b69a99230 [Translator] Load plurals from po files properly

This PR was merged into the 3.4 branch. Discussion ---------- [Translator] Load plurals from mo files properly | Q | A | ------------- | --- | Branch? | 3.4 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | symfony/symfony#10152 (comment) | License | MIT | Doc PR | Plurals were not handled correctly when loading mo files. ``` msgid "foo" msgid_plural "foos" msgstr[0] "bar" msgstr[1] "bars" ``` Before, the mo entry above was treated as two entries, which doesn't make sense: ``` 'foo' => 'bar' 'foos' => '{0} bar|{1} bars' ``` With this PR, it is treated as one entry: ``` 'foo|foos' => 'bar|bars' ``` This PR does the same as #31266, just for mo files instead of po files. Note, however, that the old behavior differed between po and mo files: `'foos' => 'bar|bars'` for po files and `'foos' => '{0} bar|{1} bars'` for mo files. Commits ------- 97d28b5e4e Load plurals from mo files properly

guilhermeblanco mentioned this issue Sep 27, 2014

[DX][Translation][3.0] Adding support for intl message formatter and decoupling default formatter #12060

Closed

aitboudad mentioned this issue Jun 22, 2015

[Translator] Adding support for intl message formatter and decoupling default formatter. #15068

Closed

javiereguiluz added the Enhancement label Feb 1, 2016

aitboudad mentioned this issue Mar 25, 2016

[Translation] added support for adding custom message formatter #18314

Merged

shaggyz mentioned this issue May 10, 2016

Bypass the native php extention shaggyz/laravel-gettext#52

Closed

Koc mentioned this issue Sep 21, 2016

[Translation] Added intl message formatter. #20007

Closed

xabbuh closed this as completed Sep 12, 2017

This was referenced Apr 26, 2019

[Translator] Load plurals from po files properly #31266

Merged

[Translator] Load plurals from mo files properly #31267

Merged

[Translator] Dump native plural formats to po files #31269

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Translation] Make plural translations more standard #10152

[Translation] Make plural translations more standard #10152

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[Translation] Make plural translations more standard #10152

[Translation] Make plural translations more standard #10152

Comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!