8000 [intl] Support all locales by using CLDR data directly · Issue #11737 · symfony/symfony · GitHub
[go: up one dir, main page]

Skip to content

[intl] Support all locales by using CLDR data directly #11737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bojanz opened this issue Aug 22, 2014 · 7 comments
Closed

[intl] Support all locales by using CLDR data directly #11737

bojanz opened this issue Aug 22, 2014 · 7 comments
Labels

Comments

@bojanz
Copy link
bojanz commented Aug 22, 2014

If intl used the CLDR data directly, it could support all locales.

See https://github.com/commerceguys/intl for the how and why.
It's not attempting to mimic the intl extension API like symfony/intl does,
but it could easily be made to do so.

@javiereguiluz
Copy link
Member

@bojanz thanks for opening this issue and thanks for helping us to solve "the infamous intl issue". At this moment, @fabpot is also trying to simplify the intl requirements for Symfony (see associated issue at #11447).

Regarding your specific proposal, if I understand you correctly, you are proposing to embed all the CLDR data inside Symfony or even using the intl project developed by CommerceGuys. If this is the case, I have two main concerns:

  • I've just downloaded the commerceguys/intl code and it contains 953 files/dirs and takes more than 5MB of disk space. My guess is that Symfony core developers would consider that too much.
  • All the CLDR data has been transformed into YAML files. Given that there are hundreds of YAML files and each one has hundreds of lines, this could seriously impact the performance of the application.

@bojanz
Copy link
Author
bojanz commented Aug 22, 2014

I am proposing the general approach.
Whether symfony uses commerceguys/intl, uses and renames it, or just takes pieces of the code is probably irrelevant at this point.

  1. The locale data has its weight, and that's unavoidable.
    The scripts are doing a lot of deduplication (if en-US is the same as en, export only en and rely on locale fallback, etc), stripping out defaults, but there's only so far that can go, and I believe we've hit a limit.

  2. As I wrote at the end of the README, the assumption is that applications will want to import the definitions into another storage (the database, redis, etc). The library imposes no restrictions and makes no assumptions on what that storage is going to be.

For example, an application certainly wants to keep currencies and languages in the database, but might not need the same for number_formats (since they don't think the users would want to modify them), so number_formats would just be cached.
Most applications will also want to import only a portion of locales (very few sites have 200 translations, and will want to import currency/country/language translations for the few they have, plus all number formats).

The format of the files themselves can easily be switched to something else (JSON?), though the cost of hitting the hard drive remains.

@mvrhov
Copy link
mvrhov commented Aug 25, 2014

IMO, the best format for files would be a php. Thus the ones running APC or newer versions with zend optimizer would benefit directly. For the ones not there would not be any change.

@fabpot fabpot added the Intl label Sep 5, 2014
@webmozart
Copy link
Contributor

Hi @bojanz, thank you for your input! :) In fact, Symfony already does include the CLDR data. However, this data is not stored in the main repository, but in the Icu component.

The intl project you mention looks nice! It seems to me that it is practically doing the same as the Intl component, but excluding the stubs usable in case of a missing intl extension. The accessors for the CLDR data are being added in #9206. That PR isn't finished yet, but once @fabpot is done with fixing our current issues (should be soon), I am going to complete it.

The API of that PR is slightly different than the one of the commerceguys intl project. It does not represent individual I18N objects (currencies, locales etc.) as objects - for performance reasons. For example, if you need a list of locale names, initializing an array of hundreds of Locale objects first adds noticeable overhead. So an identifier-based API is used instead, where the data for individual identifiers is accessible via a central service:

if (Currency::exists('EUR')) { ... }

$euro = Currency::getName('EUR');
$names = Currency::getNames();

If you want to help completing the PR and contribute to Symfony directly, that'd be great! :) If that sounds like fun to you, we can have a chat and talk about the details. Cheers!

@bojanz
Copy link
Author
bojanz commented Sep 8, 2014

@webmozart

symfony/icu includes the CLDR data only in the v1.2 branch which requires the intl extension and icu. People without the intl extension are forced to use v1.0, which only has the "en" data.
In addition, the NumberFormatter in symfony/intl only supports the "en" locale.

So we have no solution that works for everyone (people with and without the intl extension), with all locales. That's the reason why I wrote commerceguys/intl. I'd be glad to help symfony directly, if we can figure out how to resolve the main issue.

@webmozart
Copy link
Contributor

People without the intl extension are forced to use v1.0, which only has the "en" data.

@bojanz You are right. This was going to change with the mentioned PR, otherwise the CLDR data accessors clearly make no sense.

Implementing NumberFormatter et al. for all locales (in other words: a complete PHP implementation of the intl extension) is an entirely different topic. Of course this would be nice to have, but somebody needs to do the work. So far, nobody volunteered. :)

I'd be glad to help symfony directly, if we can figure out how to resolve the main issue.

Awesome! Let's chat tomorrow? You can find me on Google Hangouts, for example.

@bojanz
Copy link
Author
bojanz commented Sep 9, 2014

I've talked to webmozart, and it turns out that symfony is indeed moving in the same direction, by deprecating symfony/icu and introducing symfony/intl-data which includes the CLDR/ICU data.

I've opened issues against that work: #11885, #11886, #11887, #11888.
Looking forward to your comments.
Once that part is finished, merging my NumberFormatter into symfony/intl should be trivial.

@bojanz bojanz closed this as completed Sep 9, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants
0