8000 Add PKG_CONFIG_PATH environment variables to phpbrew build process by markwu · Pull Request #1085 · phpbrew/phpbrew · GitHub
[go: up one dir, main page]

Skip to content

Add PKG_CONFIG_PATH environment variables to phpbrew build process#1085

Merged
morozov merged 1 commit intophpbrew:masterfrom
markwu:fix-pkgconfig-for-PHP74
Dec 21, 2019
Merged

Add PKG_CONFIG_PATH environment variables to phpbrew build process#1085
morozov merged 1 commit intophpbrew:masterfrom
markwu:fix-pkgconfig-for-PHP74

Conversation

@markwu
Copy link
Contributor
@markwu markwu commented Dec 10, 2019

I try to dig the source code and add PKG_CONFIG_PATH to build process, it makes easier for user when build php 7.4.

It is just a proof of concept, for discussion only.

With the PR, you can build php 7.4, with the following command only, no more export PKG_CONFIG_PATH=....

$ phpbrew --debug install --jobs "$(nproc)" 7.1.33 +default +dbs +dba +apxs2 +bz2="$(brew --prefix bzip2)" +zlib="$(brew --prefix zlib)" +ftp

But, there is a problem when build extensions, take ftp and soap for example. You can install it with phpbrew install 7.4.0 +default +ftp, the build process will set the PKG_CONFIG_PATH for you.

But, with this PR, you still can't install extension by following command only.

phpbrew ext install ftp
phpbrew ext install soap

Because the current extension manager install them by phpize, configure, make. Without the capability like VariantBuilder add extra settings for you...

So, You still need to export PKG_CONFIG_PATH, LDFLAGS, CPPFLAGS manually. Then, install the extension.

@morozov Need your review and ideas.

Reference: #1040 (comment)

Copy link
Contributor
@morozov morozov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markwu I see that there's a pattern in the changes made for the curl and openssl variant. Could you describe the change in the logic so that we could see if it's worth generalizing into a reusable concept similar to the PrefixFinder?

@markwu
Copy link
Contributor Author
markwu commented Dec 11, 2019

The logic of openssl and curl just follow the pattern you modified in libxml2,

I try to convert the logic current to the following, and remain the same

  1. lookup $prefix through Util::findPrefix with finders.
            $prefix = Utils::findPrefix(array(
                new BrewPrefixFinder('libxml2'),
                new PkgConfigPrefixFinder('libxml'),
                new IncludePrefixFinder('libxml2/libxml/globals.h'),
                new LibPrefixFinder('libxml2.a'),
            ));
            $prefix = Utils::findPrefix(array(
                new BrewPrefixFinder('curl'),
                new PkgConfigPrefixFinder('curl'),
                new IncludePrefixFinder('curl/curl.h'),
            ));
  1. Then, I check the version is older or newer than 7.4
            if ($build->compareVersion('7.4') < 0) {
                $options[] = '--enable-libxml';

                if ($prefix !== null) {
                    $options[] = '--with-libxml-dir=' . $prefix;
                }
            } else {
                $options[] = '--with-libxml';

                if ($prefix !== null) {
                    $build->setEnvVar('PKG_CONFIG_PATH', $prefix.'/lib/pkgconfig');
                }
            }

            $options = array_merge($options, array(
                '--enable-simplexml',
                '--enable-xml',
                '--enable-xmlreader',
                '--enable-xmlwriter',
                '--with-xsl',
            ));

            return $options;
            if ($build->compareVersion('7.4') < 0) {
                if ($val) {
                    return "--with-curl=$val";
                }

                if ($prefix !== null) {
                    return '--with-curl=' . $prefix;
                }
            } else {
                if ($prefix !== null) {
                    $build->setEnvVar('PKG_CONFIG_PATH', $prefix.'/lib/pkgconfig');
                }

                return '--with-curl';
            }

So, the only thing I really change is set the PKG_CONFIG_PATH environment variables.

$build->setEnvVar('PKG_CONFIG_PATH', $prefix.'/lib/pkgconfig');

And, why should I do this, because for none standard system library just like homebrew on mac, even brewPrefixFinder can find it. The php configure can't not find pkgconfig unless you set it PATH and other variables in environment. But, it will pollute the system default. That's why most mac user won't link openssl or curl to replace system default.

Actually, I also want the flexibility, I also plan accept $val in 7.4, so the code will looks like

            if ($build->compareVersion('7.4') < 0) {
                if ($val) {
                    return "--with-curl=$val";
                }

                if ($prefix !== null) {
                    return '--with-curl=' . $prefix;
                }
            } else {
                if ($val) {
                    $build->setEnvVar('PKG_CONFIG_PATH', $val.'/lib/pkgconfig');
                    return '--with-curl';
                }
        
                if ($prefix !== null) {
                    $build->setEnvVar('PKG_CONFIG_PATH', $prefix.'/lib/pkgconfig');
                }

                return '--with-curl';
            }

Then, I can pass my personal build or none standard curl/libxml2/openssl to phpbrew, and build php-7.4.x withouts problems.

@markwu
Copy link
Contributor Author
markwu commented Dec 11, 2019

Here comes the brew info openssl:

$ brew info openssl
openssl@1.1: stable 1.1.1d (bottled) [keg-only]
Cryptography and SSL/TLS Toolkit
https://openssl.org/
/usr/local/Cellar/openssl@1.1/1.1.1d (7,983 files, 17.9MB)
  Poured from bottle on 2019-12-03 at 02:20:51
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/openssl@1.1.rb
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /usr/local/etc/openssl@1.1/certs

and run
  /usr/local/opt/openssl@1.1/bin/c_rehash

openssl@1.1 is keg-only, which means it was not symlinked into /usr/local,
because openssl/libressl is provided by macOS so don't link an incompatible version.

If you need to have openssl@1.1 first in your PATH run:
  echo 'export PATH="/usr/local/opt/openssl@1.1/bin:$PATH"' >> ~/.bash_profile

For compilers to find openssl@1.1 you may need to set:
  export LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"
  export CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"

For pkg-config to find openssl@1.1 you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig"

You can see it says:

openssl@1.1 is keg-only, which means it was not symlinked into /usr/local,
because openssl/libressl is provided by macOS so don't link an incompatible version.

@morozov
Copy link
Contributor
morozov commented Dec 16, 2019

And, why should I do this, because for none standard system library just like homebrew on mac, even brewPrefixFinder can find it.

This is confusing. Isn't the sole purpose of the BrewPrefixFinder to find dependencies managed by Homebrew on macOS? If it doesn't do its job properly, can we fix it and enable all extensions that use it instead fixing only a couple of extensions by adding more duplicated code?

@markwu
Copy link
Contributor Author
markwu commented Dec 16, 2019

Okay, let me explain this way.

macOS has it's own building system from Xcode and CLT (command line environment). It has its own building tools, libraries and packages. It official and system wide.

homebrew is another building system, it also has its own building tools, libraries and packages.

To build PHP outside homebrew with homebrew's packages. You need to link them to system directories. For example, /usr/local/bin, /usr/local/lib, /usr/local/include and etc.

But, to avoid problems, homebrew only link missed libraries to system, and won't link same libraries to system. You have to do it manually.

Take openssl for example, macOS has it's own openssl:

# it is system wide
$ /usr/bin/openssl version
LibreSSL 2.6.5

homebrew has openssl 1.1.1d

# it is only available inside homebrew building system, unless you link to system directories
$ /usr/local/opt/openssl/bin/openssl version
OpenSSL 1.1.1d  10 Sep 2019

Okay, if we build PHP inside homebrew(through homebrew formula), it will find pkg-config information from /usr/local/opt/openssl/bin/openssl/lib/pkgconfig automatically without any problems.

But, if we install PHP outside homebrew, like phpbrew. There is no way to find pkg-config information unless we set PKG_CONFIG_PATH=/usr/local/opt/openssl/bin/openssl/lib/pkgconfig or link openssl to system wide.

So, even BrewPrefixFinder can find openssl, but if the configure doesn't understand them, there is no way to compile. Because we build PHP outside homebrew.

Before PHP 7.3(with), we can pass the openssl prefix to php configure from command line with --with-openssl=$(brew --prefix openssl) or from BrewPrefixFinder --with-openssl=$prefix, therefore there is no problem to compile it.

After PHP 7.4(with), there is no way to pass openssl prefix to php configure, because there is only --with-openssl available, you can't pass anything into configure. We need to pass pkg-config, ldflags, cppflags through environment variables.

That's why after 7.4, we have to export PKG_CONFIG_PATH manually.

The purpose of this PR:

  1. Pass the $val from phpbrew command or $prefix from XXXPrefixFinder to --with-openssl=($val|$prefix) before php 7.3.

  2. Pass the $val from phpbrew command or $prefix from XXXPrefixFinder to PKG_CONFIG_PATH with --with-openssl after php 7.4.

Hope this explanation is clear for you.

Reference: https://docs.brew.sh/How-to-build-software-outside-Homebrew-with-Homebrew-keg-only-dependencies

@morozov
Copy link
Contributor
morozov commented Dec 17, 2019

Thank you for this detailed writeup. It really helps understand the changes.

After PHP 7.4(with), there is no way to pass openssl prefix to php configure, because there is only --with-openssl available, you can't pass anything into configure. We need to pass pkg-config, ldflags, cppflags through environment variables.

For future reference, it happened in php/php-src#3654.

Copy link
Contributor
@morozov morozov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@markwu thanks for bearing with me. I agree with the approach but the implementation and the newly introduced APIs still need some rework.

@morozov
Copy link
Contributor
morozov commented Dec 17, 2019

Because the current extension manager install them by phpize, configure, make. Without the capability like VariantBuilder add extra settings for you...

This is a shortcoming of the current design. In fact, all the prefixes should be determined on the per-extension basis regardless whether it's phpbrew install or phpbrew ext install while variants would be combinations of extensions or their synonyms. We could approach refactoring this logic in the future.

@morozov morozov added Extension Installer Improvement PHP 7.4 GD OpenSSL Issues related to compiling PHP with OpenSSL XML labels Dec 17, 2019
@markwu
Copy link
Contributor Author
markwu commented Dec 19, 2019

I just fixed the issues we discussed. There are 4 extensions need to set PKG_CONFIG_PATH when building php-7.4, they are:

  1. libxml
  2. openssl
  3. curl
  4. zlib

I just test them with all 5.6/7/2/7.3/7.4 building process.

Copy link
Contributor
@morozov morozov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're almost there. Excellent job so far!

@morozov
Copy link
Contributor
morozov commented Dec 20, 2019
$ phpbrew --debug install --jobs "$(nproc)" 7.1.33 +default +dbs +dba +apxs2 +bz2="$(brew --prefix bzip2)" +zlib="$(brew --prefix zlib)" +ftp

BTW, from this command, it follows that PHPBrew doesn't properly discover the Homebrew-managed prefixes for zlib and bz2. Do you want to file another PR and rework these variants to use the prefix finders and employ the one for Homebrew?

@morozov morozov added this to the Release 1.25.0 milestone Dec 20, 2019
@markwu
Copy link
Contributor Author
markwu commented Dec 21, 2019
$ phpbrew --debug install --jobs "$(nproc)" 7.1.33 +default +dbs +dba +apxs2 +bz2="$(brew --prefix bzip2)" +zlib="$(brew --prefix zlib)" +ftp

BTW, from this command, it follows that PHPBrew doesn't properly discover the Homebrew-managed prefixes for zlib and bz2. Do you want to file another PR and rework these variants to use the prefix finders and employ the one for Homebrew?

Yes, I plan to use another PR for this, because it seems different issue. I'll add BrewPrefixFinder for bz2 and iconv for macOS, zlib already fixed in previous.

If you don't mind, I can add them in this PR, because it already done in my local branch.

@morozov
Copy link
Contributor
morozov commented Dec 21, 2019

Please address the CI failure. You'll want to rebase your branch first because some of the code style issues are already addressed in the upstream branch. Then please squash all commits.

configure errors for libxml2, openssl, curl and zlib.
@markwu
Copy link
Contributor Author
markwu commented Dec 21, 2019

@morozov I just fixed the issues you mentioned, and squashed all commits into one. Thanks for your review.

I will fire another PR for add BrewPrefixFinder support for iconv and bz2.

@morozov morozov merged commit 8f75e62 into phpbrew:master Dec 21, 2019
@morozov
Copy link
Contributor
morozov commented Dec 21, 2019

@morozov morozov added the macOS label Dec 21, 2019
@markwu markwu deleted the fix-pkgconfig-for-PHP74 branch December 22, 2019 08:34
@morozov
Copy link
Contributor
morozov commented Dec 24, 2019

Thank you @markwu!

I just checked this approach on macOS and it seems not working. First I tried phpbrew install and then just ran a raw ./configure command:

$ brew info libxml2
libxml2: stable 2.9.10 (bottled), HEAD [keg-only]
GNOME XML library
http://xmlsoft.org/
/usr/local/Cellar/libxml2/2.9.10 (280 files, 10.5MB)
  Poured from bottle on 2019-12-23 at 16:22:01
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/libxml2.rb
==> Dependencies
Required: python ✔, readline ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
libxml2 is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have libxml2 first in your PATH run:
  echo 'set -g fish_user_paths "/usr/local/opt/libxml2/bin" $fish_user_paths' >> ~/.config/fish/config.fish

For compilers to find libxml2 you may need to set:
  set -gx LDFLAGS "-L/usr/local/opt/libxml2/lib"
  set -gx CPPFLAGS "-I/usr/local/opt/libxml2/include"

==> Analytics
install: 57,321 (30 days), 143,787 (90 days), 630,154 (365 days)
install-on-request: 15,167 (30 days), 37,983 (90 days), 133,004 (365 days)
build-error: 0 (30 days)

$ env PKG_CONFIG_PATH=/usr/local/opt/libxml2/lib/pkgconfig ./configure '--with-libxml'

...

Configuring extensions
checking io.h usability... no
checking io.h presence... no
checking for io.h... no
checking for strtoll... yes
checking for atoll... yes
checking whether to build with LIBXML support... yes
checking for libxml-2.0 >= 2.7.6... no
configure: error: in `/Users/morozov/.phpbrew/build/php-7.4.1':
configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

Alternatively, you may set the environment variables LIBXML_CFLAGS
and LIBXML_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

To get pkg-config, see <http://pkg-config.freedesktop.org/>.
See `config.log' for more details

I've never used PHPBrew on macOS so I may be doing something wrong. Any ideas?

@morozov
Copy link
Contributor
morozov commented Dec 24, 2019

The same is if I do:

env LDFLAGS="-L/usr/local/opt/libxml2/lib" CPPFLAGS="-I/usr/local/opt/libxml2/include" ./configure

@markwu
Copy link
Contributor Author
markwu commented Dec 24, 2019

configure: error: The pkg-config script could not be found or is too old. Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

I think there are some homebrew dependency need to installed first to run phpbrew, for example pkg-config. Can you check if you installed pkg-config through brew list | grep pkg-config, it should be:

$ brew list | grep pkg-config
pkg-config

I build a formula before to install phpbrew for myself, you can install the dependency list in this script https://github.com/markwu/homebrew-php/blob/master/Formula/phpbrew.rb

  depends_on "autoconf"
  depends_on "automake"
  depends_on "curl"
  depends_on "gd"
  depends_on "gettext"
  depends_on "gmp"
  depends_on "icu4c"
  depends_on "bison"
  depends_on "jpeg"
  depends_on "libevent"
  depends_on "libtool"
  depends_on "libxml2"
  depends_on "mcrypt"
  depends_on "mhash"
  depends_on "openssl"
  depends_on "pcre"
  depends_on "re2c"

I think it is not a complete list, I need some time to figure them out.

Maybe we can host our own homebrew formula to install all these dependencies with phpbrew easier.

Mark

@markwu
Copy link
Contributor Author
markwu commented Dec 24, 2019

you can try the following steps to install phpbrew by homebrew:

  1. $ brew tap markwu/php
  2. $ brew update
  3. $ brew install phpbrew # Currently, 1.23.1
  4. $ phpbrew self-update # Update to 1.24.1
  5. You can copy your own version phpbrew to here homebrew folder with cp phpbrew /usr/local/opt/phpbrew/libexec/ # Update to your branch phpbrew
  6. $ phpbrew --debug install --jobs "$(nproc)" 7.4.0 +default +dbs +dba +apxs2 +ftp +soap +gettext +exif +iconv +bz2 +zlib

@morozov morozov self-assigned this Dec 24, 2019
@morozov
Copy link
Contributor
morozov commented Dec 24, 2019

I think there are some homebrew dependency need to installed first to run phpbrew, for example pkg-config.

Exactly. It was missing on my machine. After I installed it, I was able to build PHP 7.4. Added it to the requirements page.

@morozov
Copy link
Contributor
morozov commented Dec 24, 2019

Maybe we can host our own homebrew formula to install all these dependencies with phpbrew easier.

I think what we could do instead is to document all variant dependencies for all supported platforms and implement integration of PHPBrew with dependency managers (including Homebrew). This way, it could just run or offer to run the needed commands w/o the need to install additional software. But this is a really long-going plan.

@morozov morozov mentioned this pull request Jan 2, 2020
@theofidry
Copy link
Contributor
theofidry commented Jan 2, 2020

Thanks a lot for the work! 🥇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

411D

Successfully merging this pull request may close these issues.

3 participants

Comments

0