[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tests: include current directory when running test Perl commands #14124

Closed
wants to merge 1 commit into from

Conversation

vszakats
Copy link
Member
@vszakats vszakats commented Jul 8, 2024

Necessary to find generated files in the out-of-tree build directory.
E.g. tests/configurehelp.pm, for tests 1119 and 1167.

Before this patch macOS autotools builds were failing these two tests
due to falling back to the default preprocessor (cpp) instead of
the actual one configured. Then cpp failing to compile Apple SDK
headers referenced by curl headers.

Cherry-picked from #14097
Closes #14124

Necessary to find generated files in the out-of-tree build directory.
E.g. `tests/configurehelp.pm`, for test 1119 and test 1167.

Before this patch macOS autotools builds were failing these two tests
due to falling back to the default preprocessor (`cpp`) instead of
the actual one configured. Then `cpp` failing to compile Apple SDK
headers referenced by curl headers.

Cherry-picked from curl#14097
Closes #xxxxx
vszakats added a commit to vszakats/curl that referenced this pull request Jul 8, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to curl#14124
Closes #xxxxx
@vszakats vszakats closed this in efc2c51 Jul 9, 2024
@vszakats vszakats deleted the runner-out-of-tree branch July 9, 2024 02:15
vszakats added a commit to vszakats/curl that referenced this pull request Jul 12, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to efc2c51 curl#14124
Closes curl#14129
vszakats added a commit to vszakats/curl that referenced this pull request Jul 12, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to curl#14124
Closes #xxxxx
vszakats added a commit to vszakats/curl that referenced this pull request Jul 13, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to efc2c51 curl#14124
Closes curl#14129
vszakats added a commit to vszakats/curl that referenced this pull request Jul 14, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to efc2c51 curl#14124
Closes curl#14129
vszakats added a commit to vszakats/curl that referenced this pull request Jul 14, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to efc2c51 curl#14124
Closes curl#14129
vszakats added a commit to vszakats/curl that referenced this pull request Jul 15, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to efc2c51 curl#14124
Closes curl#14129
vszakats added a commit to vszakats/curl that referenced this pull request Jul 16, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

(I haven't tested running the tests.)

Follow-up to efc2c51 curl#14124
Closes curl#14129
vszakats added a commit that referenced this pull request Jul 16, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

Tested OK: https://github.com/curl/curl/actions/runs/9915343826

Besides Apple, it also supports any gcc and clang builds, and MSVC.
For other platforms, it defaults to `cpp` (like autotools).

Follow-up to efc2c51 #14124
Cherry-picked from #14097
Closes #14129
vszakats added a commit that referenced this pull request Jul 18, 2024
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.

Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.

I developed and tested all fixes under this PR, then merged them as
separate patches.

This PR retained CI jobs updates, extensively reworking and extending
them: [1]

At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.

Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.

Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.

1. GCC header compatibility layer ("hack" as GCC calls it)

The toughest issue is GCC's built-in compatibility layer:
  https://github.com/gcc-mirror/gcc/tree/master/fixincludes

This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
  https://github.com/iains/gcc-12-branch
  https://github.com/iains/gcc-13-branch
  https://github.com/iains/gcc-14-branch

The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.

An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.

Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]

More info: #10356 (comment)

db135f8 #14119 macos: add workaround for gcc, non-c-ares, IPv6, compile error
Ref: curl/curl-for-win@e2db3c4
Ref: curl/curl-for-win@f5c58d7

2. Homebrew GCC's `availability` extension

A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.

Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.

Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: llvm/llvm-project#81767

Commit implementing the `availability` macro:
gcc-12: iains/gcc-12-branch@fd5530b
gcc-13: iains/gcc-13-branch@cb7e4ec
gcc-14: iains/gcc-14-branch@ff62a10

That applied to Homebrew GCC (12.4.0):
Homebrew/homebrew-core@b904223#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48

Ref: #13700
More info: #14091 (comment)

e91fcba #14155 macos: undo `availability` macro enabled by Homebrew gcc

3. Proprietary Apple SDK macros

Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.

baa3270 #14134 build: fix llvm 16 or older + Xcode 15 or newer, and gcc

4. Apple SDK header bug

Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.

ff784af #14159 build: fix llvm 17 and older + macOS SDK 14.4 and newer

5. `TargetConditionals.h` requires `sys/types.h`

Fixed in curl. It caused feature-detection failurs with autotools, and
could break builds in certain configurations.

e1f6192 #14130 configure: fix `SystemConfiguration` detection

6. Differences between autotools and CMake compiler options

Fixed it by syncing compiler warning options.

59cadac #14128 build: sync warning options between autotools, cmake & compilers

7. Differences between autotools and CMake dependency detection

Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.

f43adc2 #14137 cmake: detect `libidn2` also via `pkg-config`
Ref: #14136 cmake: detect `nghttp2` via `pkg-config`, enable by default

8. libidn2 detection bug with CMake

Fixed the root cause and also the trigger in the CI config.

764fbab #14175 cmake: fix builds with detected libidn2 lib but undetected header

9. Suppressed compiler warnings inside Apple-specific curl code

Fixed these warnings, which allowed to stop silencing them.

b05dc7e #14122 sectransp: fix `HAVE_BUILTIN_AVAILABLE` checks to not emit warnings
5fa534b #14162 sectransp: fix clang compiler warnings, stop silencing them

10. CMake mis-detecting a CA bundle path on macOS

d2ef625 #14182 cmake: sync CA bundle/path detection with autotools

11. Failure to build tests with LibreSSL or wolfSSL with CMake

Fixed by dropping unnecessary includes, makign test builds dependent
on dependency headers.

3765d75 #14172 cmake: fix building `unit1600` due to missing `ssl/openssl.h`

12. curl tests with CMake

curl's CMake was missing bits for running the C preprocessor accurately.
It made tests 1119 and 1167 fail. I implemented the missing bits.

efc2c51 #14124 tests: include current directory when running test Perl commands
c09db8b #14129 cmake: create `configurehelp.pm` like autotools does
67cc1e3 #14125 test1119: adapt for `.md` input

13. GCC missing `__builtin_available()` support

curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.

14. Monotonic timer support with GCC

Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.

15. Runtime/test failures with GCC

I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.

16. Runtime/test failure in llvm/clang

AppleIDN support received a fix with two more remaining.

fd02508 #14179 #14176 IDN: fix ß with AppleIDN

17. Other issues found and fixed while working on this:

2c15aa5        GHA/macos: delete misplaced `CFLAGS`, drop redundant CMake option
80fb7c0 #14126 configure: limit `SystemConfiguration` test to non-c-ares, IPv6 builds
cfd6f43 #14127 build: tidy up `__builtin_available` feature checks (Apple)
bae5553 #14174 runtests: show name and keywords for failed tests in summary
09cdf7e #14178 cmake: delete unused `HAVE_LIBSSH2`, `HAVE_LIBSOCKET` macros
d3595c7 #14186 configure: CA bundle/path detection fixes
58772b0 #14187 runtests: set `SOURCE_DATE_EPOCH` to fix failing around midnight
18f1cd7 #14183 tests: sync feature names with `curl -V`
4c22d97 #14181 build: use `#error` instead of invalid syntax

Pending merge:
vszakats added a commit that referenced this pull request Jul 19, 2024
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.

Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.

I developed and tested all fixes under this PR, then merged them as
separate patches.

This PR retained CI jobs updates, extensively reworking and extending
them: [1]

At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.

Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.

Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.

1. GCC header compatibility layer ("hack" as GCC calls it)

The toughest issue is GCC's built-in compatibility layer:
  https://github.com/gcc-mirror/gcc/tree/master/fixincludes

This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
  https://github.com/iains/gcc-12-branch
  https://github.com/iains/gcc-13-branch
  https://github.com/iains/gcc-14-branch

The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.

An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.

Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]

More info: #10356 (comment)

db135f8 #14119 macos: add workaround for gcc, non-c-ares, IPv6, compile error
Ref: curl/curl-for-win@e2db3c4
Ref: curl/curl-for-win@f5c58d7

2. Homebrew GCC's `availability` extension

A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.

Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.

Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: llvm/llvm-project#81767

Commit implementing the `availability` macro:
gcc-12: iains/gcc-12-branch@fd5530b
gcc-13: iains/gcc-13-branch@cb7e4ec
gcc-14: iains/gcc-14-branch@ff62a10

That applied to Homebrew GCC (12.4.0):
Homebrew/homebrew-core@b904223#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48

Ref: #13700
More info: #14091 (comment)

e91fcba #14155 macos: undo `availability` macro enabled by Homebrew gcc

3. Proprietary Apple SDK macros

Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.

baa3270 #14134 build: fix llvm 16 or older + Xcode 15 or newer, and gcc

4. Apple SDK header bug

Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.

ff784af #14159 build: fix llvm 17 and older + macOS SDK 14.4 and newer

5. `TargetConditionals.h` requires `sys/types.h`

Fixed in curl. It caused feature-detection failurs with autotools, and
could break builds in certain configurations.

e1f6192 #14130 configure: fix `SystemConfiguration` detection

6. Differences between autotools and CMake compiler options

Fixed it by syncing compiler warning options.

59cadac #14128 build: sync warning options between autotools, cmake & compilers

7. Differences between autotools and CMake dependency detection

Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.

f43adc2 #14137 cmake: detect `libidn2` also via `pkg-config`
Ref: #14136 cmake: detect `nghttp2` via `pkg-config`, enable by default

8. libidn2 detection bug with CMake

Fixed the root cause and also the trigger in the CI config.

764fbab #14175 cmake: fix builds with detected libidn2 lib but undetected header

9. Suppressed compiler warnings inside Apple-specific curl code

Fixed these warnings, which allowed to stop silencing them.

b05dc7e #14122 sectransp: fix `HAVE_BUILTIN_AVAILABLE` checks to not emit warnings
5fa534b #14162 sectransp: fix clang compiler warnings, stop silencing them

10. CMake mis-detecting a CA bundle path on macOS

d2ef625 #14182 cmake: sync CA bundle/path detection with autotools

11. Failure to build tests with LibreSSL or wolfSSL with CMake

Fixed by dropping unnecessary includes, makign test builds dependent
on dependency headers.

3765d75 #14172 cmake: fix building `unit1600` due to missing `ssl/openssl.h`

12. curl tests with CMake

curl's CMake was missing bits for running the C preprocessor accurately.
It made tests 1119 and 1167 fail. I implemented the missing bits.

efc2c51 #14124 tests: include current directory when running test Perl commands
c09db8b #14129 cmake: create `configurehelp.pm` like autotools does
67cc1e3 #14125 test1119: adapt for `.md` input

13. GCC missing `__builtin_available()` support

curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.

14. Monotonic timer support with GCC

Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.

15. Runtime/test failures with GCC

I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.

16. Runtime/test failure in llvm/clang

AppleIDN support received a fix with two more remaining.

fd02508 #14179 #14176 IDN: fix ß with AppleIDN

17. Other issues found and fixed while working on this:

2c15aa5        GHA/macos: delete misplaced `CFLAGS`, drop redundant CMake option
80fb7c0 #14126 configure: limit `SystemConfiguration` test to non-c-ares, IPv6 builds
cfd6f43 #14127 build: tidy up `__builtin_available` feature checks (Apple)
bae5553 #14174 runtests: show name and keywords for failed tests in summary
09cdf7e #14178 cmake: delete unused `HAVE_LIBSSH2`, `HAVE_LIBSOCKET` macros
d3595c7 #14186 configure: CA bundle/path detection fixes
58772b0 #14187 runtests: set `SOURCE_DATE_EPOCH` to fix failing around midnight
18f1cd7 #14183 tests: sync feature names with `curl -V`
4c22d97 #14181 build: use `#error` instead of invalid syntax

Pending merges:

- #14185 runtests: fold test details for GitHub CI runs
- #14197 cmake: grab-bag of tidy-ups
- #14196 configure: limit `__builtin_available` test to Darwin

Summary:

In general GCC doesn't seem to be a good fit with curl and macOS for
now. These "lucky" combinations (GitHub Actions runner) will build out
of the box now: macos-14 + Xcode 15.0.1 + gcc-11, gcc-12, gcc-14. The
rest builds with the ugly workaround in place, but all this still leaves
some runtime issues.

More info and links in the commit messages and source code.

[1]: This PR:
- add info about target OS version requirements per feature, with OS
  names and release years.
- stop using `-Wno-deprecated-declarations` to suppress warnings.
- use `LDFLAGS=-w` to suppress 'object file was built for newer macOS
  version than being linked' warnings.
  (there were tens of thousands of them in some jobs)
- allow overriding Xcode version in all jobs.
- improve job names.
- abbreviate CMake as CM, autotools as AM for more compact job names.
- shorten job names by using `!` instead of `no-` and `non-`.
- bump parellel tests to 10 (from 5).
- drop using `--enable-maintainer-mode` `./configure` option.
- add gcc-12 no-ssl, autotools job with tests, ignore failing test
  results. (It's not yet clear why gcc-12 builds have different runtime
  results than clang/llvm ones.)
- add comments with OS names and release years next to version numbers,
  e.g. 10.15  # Catalina (2019)
- fix broken gcc-12 SecureTransport build.
- show compiler, Xcode, SDK, gcc hack SDK versions, Homebrew
  preinstalled packages and C compiler predefined macros for each job.
  Useful for debugging all the strange problems these builds might have.
- merge brew bundle and install steps.
- move step names to the top.
- dump configure log for both cmake and autotools also for successful
  builds. Useful for debugging.
- dump curl_config.h in short (sorted #defines) and full form.
- add support for the mainline llvm compiler.
- set sysroot for gcc and llvm.
- add timeout for cmake jobs.
- add new job matrix: combinations
  It supports building all possible compiler, runner image, Xcode/SDK
  combinations, with cmake and autotools, target OS versions and with or
  without SecureTransport. It's quick. GHA limits the maximum number of
  matrix jobs at 256.
  I used this as a test-rig to fix the macOS build fallouts with gcc and
  llvm.
  I settled with 16 jobs, trying to maximize fallout coverage.
- implement hack to make Homebrew gcc work with all available SDKs.
- add handy mini-table about Xcode / SDK versions, OS names, years for
  each GHA images, with the defaults.
- add tests for cmake jobs.
- make cmake config hack to link GnuTLS less intrusive.
- stop ignoring test 1452, seems fine now.
- fix to enable libpsl in autotools builds.
- enable libpsl in cmake builds.
- add an llvm job with tests (both autotools and cmake).
- delete similar macOS jobs from Circle CI. GHA is now arm64 too.

[2]: Homebrew GCC vs GHA runner images vs curl builds:
```
macOS      Xcode   gcc         gcc SDK hacks      Xcode SDK   SDK major Build Compile
           (*def)  (Homebrew)  (CommandLineTools)             versions        error
--------  -------- ----------  ------------------ ----------  --------- ----- ---------------------
macos-12   13.1    GCC 11.4.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 11.4.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 11.4.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 11.4.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 11.4.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 11.4.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12  *14.2    GCC 11.4.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13   14.1    GCC 11.4.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 11.4.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 11.4.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 11.4.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.1    GCC 11.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.2    GCC 11.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14   14.3.1  GCC 11.4.0  MacOSX14           MacOSX13.3  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14  *15.0.1  GCC 11.4.0  MacOSX14           MacOSX14.0
macos-14   15.1    GCC 11.4.0  MacOSX14           MacOSX14.2
macos-14   15.2    GCC 11.4.0  MacOSX14           MacOSX14.2
macos-14   15.3    GCC 11.4.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 11.4.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 11.4.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc@11/11.4.0/lib/gcc/11/gcc/aarch64-apple-darwin23/11/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12   13.1    GCC 12.4.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 12.4.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 12.4.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 12.4.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 12.4.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 12.4.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12  *14.2    GCC 12.4.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13   14.1    GCC 12.4.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 12.4.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 12.4.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 12.4.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.1    GCC 12.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.2    GCC 12.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14   14.3.1  GCC 12.4.0  MacOSX14           MacOSX13.3  MISMATCH
macos-14  *15.0.1  GCC 12.4.0  MacOSX14           MacOSX14.0
macos-14   15.1    GCC 12.4.0  MacOSX14           MacOSX14.2
macos-14   15.2    GCC 12.4.0  MacOSX14           MacOSX14.2
macos-14   15.3    GCC 12.4.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 12.4.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 12.4.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc@12/12.4.0/lib/gcc/12/gcc/aarch64-apple-darwin23/12/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12   13.1    GCC 13.3.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 13.3.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 13.3.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 13.3.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 13.3.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 13.3.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-12  *14.2    GCC 13.3.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13   14.1    GCC 13.3.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 13.3.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 13.3.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 13.3.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13   15.1    GCC 13.3.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13   15.2    GCC 13.3.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   14.3.1  GCC 13.3.0  MacOSX14           MacOSX13.3  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14  *15.0.1  GCC 13.3.0  MacOSX14           MacOSX14.0            FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   15.1    GCC 13.3.0  MacOSX14           MacOSX14.2            FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   15.2    GCC 13.3.0  MacOSX14           MacOSX14.2            FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   15.3    GCC 13.3.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 13.3.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 13.3.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc@13/13.3.0/lib/gcc/13/gcc/aarch64-apple-darwin23/13/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12   13.1    GCC 14.1.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 14.1.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 14.1.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 14.1.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 14.1.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 14.1.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12  *14.2    GCC 14.1.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13   14.1    GCC 14.1.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 14.1.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 14.1.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 14.1.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13   15.1    GCC 14.1.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13   15.2    GCC 14.1.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-14   14.3.1  GCC 14.1.0  MacOSX14           MacOSX13.3  MISMATCH
macos-14  *15.0.1  GCC 14.1.0  MacOSX14           MacOSX14.0
macos-14   15.1    GCC 14.1.0  MacOSX14           MacOSX14.2
macos-14   15.2    GCC 14.1.0  MacOSX14           MacOSX14.2
macos-14   15.3    GCC 14.1.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 14.1.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 14.1.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc/14.1.0_1/lib/gcc/current/gcc/aarch64-apple-darwin23/14/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
```
Source: https://github.com/curl/curl/actions/runs/9883956647/job/27299564218

This commit fixes earlier commit
1e75edd, reverted in
41a7e0dcc9681afd91e066411bcee4f369c23366, where I cut the commit
message in half by accident. The patch itself is identical.

Closes #14097
meslubi2021 pushed a commit to meslubi2021/curl that referenced this pull request Jul 19, 2024
Required by tests 1119 and 1167 to run a C preprocessor.

Tested OK: https://github.com/curl/curl/actions/runs/9915343826

Besides Apple, it also supports any gcc and clang builds, and MSVC.
For other platforms, it defaults to `cpp` (like autotools).

Follow-up to efc2c51 curl#14124
Cherry-picked from curl#14097
Closes curl#14129
meslubi2021 pushed a commit to meslubi2021/curl that referenced this pull request Jul 19, 2024
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.

Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.

I developed and tested all fixes under this PR, then merged them as
separate patches.

This PR retained CI jobs updates, extensively reworking and extending
them: [1]

At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.

Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.

Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.

1. GCC header compatibility layer ("hack" as GCC calls it)

The toughest issue is GCC's built-in compatibility layer:
  https://github.com/gcc-mirror/gcc/tree/master/fixincludes

This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
  https://github.com/iains/gcc-12-branch
  https://github.com/iains/gcc-13-branch
  https://github.com/iains/gcc-14-branch

The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.

An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.

Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]

More info: curl#10356 (comment)

db135f8 curl#14119 macos: add workaround for gcc, non-c-ares, IPv6, compile error
Ref: curl/curl-for-win@e2db3c4
Ref: curl/curl-for-win@f5c58d7

2. Homebrew GCC's `availability` extension

A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.

Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.

Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: llvm/llvm-project#81767

Commit implementing the `availability` macro:
gcc-12: iains/gcc-12-branch@fd5530b
gcc-13: iains/gcc-13-branch@cb7e4ec
gcc-14: iains/gcc-14-branch@ff62a10

That applied to Homebrew GCC (12.4.0):
Homebrew/homebrew-core@b904223#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48

Ref: curl#13700
More info: curl#14091 (comment)

e91fcba curl#14155 macos: undo `availability` macro enabled by Homebrew gcc

3. Proprietary Apple SDK macros

Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.

baa3270 curl#14134 build: fix llvm 16 or older + Xcode 15 or newer, and gcc

4. Apple SDK header bug

Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.

ff784af curl#14159 build: fix llvm 17 and older + macOS SDK 14.4 and newer

5. `TargetConditionals.h` requires `sys/types.h`

Fixed in curl. It caused feature-detection failurs with autotools, and
could break builds in certain configurations.

e1f6192 curl#14130 configure: fix `SystemConfiguration` detection

6. Differences between autotools and CMake compiler options

Fixed it by syncing compiler warning options.

59cadac curl#14128 build: sync warning options between autotools, cmake & compilers

7. Differences between autotools and CMake dependency detection

Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.

f43adc2 curl#14137 cmake: detect `libidn2` also via `pkg-config`
Ref: curl#14136 cmake: detect `nghttp2` via `pkg-config`, enable by default

8. libidn2 detection bug with CMake

Fixed the root cause and also the trigger in the CI config.

764fbab curl#14175 cmake: fix builds with detected libidn2 lib but undetected header

9. Suppressed compiler warnings inside Apple-specific curl code

Fixed these warnings, which allowed to stop silencing them.

b05dc7e curl#14122 sectransp: fix `HAVE_BUILTIN_AVAILABLE` checks to not emit warnings
5fa534b curl#14162 sectransp: fix clang compiler warnings, stop silencing them

10. CMake mis-detecting a CA bundle path on macOS

d2ef625 curl#14182 cmake: sync CA bundle/path detection with autotools

11. Failure to build tests with LibreSSL or wolfSSL with CMake

Fixed by dropping unnecessary includes, makign test builds dependent
on dependency headers.

3765d75 curl#14172 cmake: fix building `unit1600` due to missing `ssl/openssl.h`

12. curl tests with CMake

curl's CMake was missing bits for running the C preprocessor accurately.
It made tests 1119 and 1167 fail. I implemented the missing bits.

efc2c51 curl#14124 tests: include current directory when running test Perl commands
c09db8b curl#14129 cmake: create `configurehelp.pm` like autotools does
67cc1e3 curl#14125 test1119: adapt for `.md` input

13. GCC missing `__builtin_available()` support

curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.

14. Monotonic timer support with GCC

Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.

15. Runtime/test failures with GCC

I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.

16. Runtime/test failure in llvm/clang

AppleIDN support received a fix with two more remaining.

fd02508 curl#14179 curl#14176 IDN: fix ß with AppleIDN

17. Other issues found and fixed while working on this:

2c15aa5        GHA/macos: delete misplaced `CFLAGS`, drop redundant CMake option
80fb7c0 curl#14126 configure: limit `SystemConfiguration` test to non-c-ares, IPv6 builds
cfd6f43 curl#14127 build: tidy up `__builtin_available` feature checks (Apple)
bae5553 curl#14174 runtests: show name and keywords for failed tests in summary
09cdf7e curl#14178 cmake: delete unused `HAVE_LIBSSH2`, `HAVE_LIBSOCKET` macros
d3595c7 curl#14186 configure: CA bundle/path detection fixes
58772b0 curl#14187 runtests: set `SOURCE_DATE_EPOCH` to fix failing around midnight
18f1cd7 curl#14183 tests: sync feature names with `curl -V`
4c22d97 curl#14181 build: use `#error` instead of invalid syntax

Pending merge:
meslubi2021 pushed a commit to meslubi2021/curl that referenced this pull request Jul 19, 2024
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.

Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.

I developed and tested all fixes under this PR, then merged them as
separate patches.

This PR retained CI jobs updates, extensively reworking and extending
them: [1]

At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.

Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.

Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.

1. GCC header compatibility layer ("hack" as GCC calls it)

The toughest issue is GCC's built-in compatibility layer:
  https://github.com/gcc-mirror/gcc/tree/master/fixincludes

This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
  https://github.com/iains/gcc-12-branch
  https://github.com/iains/gcc-13-branch
  https://github.com/iains/gcc-14-branch

The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.

An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.

Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]

More info: curl#10356 (comment)

db135f8 curl#14119 macos: add workaround for gcc, non-c-ares, IPv6, compile error
Ref: curl/curl-for-win@e2db3c4
Ref: curl/curl-for-win@f5c58d7

2. Homebrew GCC's `availability` extension

A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.

Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.

Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: llvm/llvm-project#81767

Commit implementing the `availability` macro:
gcc-12: iains/gcc-12-branch@fd5530b
gcc-13: iains/gcc-13-branch@cb7e4ec
gcc-14: iains/gcc-14-branch@ff62a10

That applied to Homebrew GCC (12.4.0):
Homebrew/homebrew-core@b904223#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48

Ref: curl#13700
More info: curl#14091 (comment)

e91fcba curl#14155 macos: undo `availability` macro enabled by Homebrew gcc

3. Proprietary Apple SDK macros

Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.

baa3270 curl#14134 build: fix llvm 16 or older + Xcode 15 or newer, and gcc

4. Apple SDK header bug

Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.

ff784af curl#14159 build: fix llvm 17 and older + macOS SDK 14.4 and newer

5. `TargetConditionals.h` requires `sys/types.h`

Fixed in curl. It caused feature-detection failurs with autotools, and
could break builds in certain configurations.

e1f6192 curl#14130 configure: fix `SystemConfiguration` detection

6. Differences between autotools and CMake compiler options

Fixed it by syncing compiler warning options.

59cadac curl#14128 build: sync warning options between autotools, cmake & compilers

7. Differences between autotools and CMake dependency detection

Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.

f43adc2 curl#14137 cmake: detect `libidn2` also via `pkg-config`
Ref: curl#14136 cmake: detect `nghttp2` via `pkg-config`, enable by default

8. libidn2 detection bug with CMake

Fixed the root cause and also the trigger in the CI config.

764fbab curl#14175 cmake: fix builds with detected libidn2 lib but undetected header

9. Suppressed compiler warnings inside Apple-specific curl code

Fixed these warnings, which allowed to stop silencing them.

b05dc7e curl#14122 sectransp: fix `HAVE_BUILTIN_AVAILABLE` checks to not emit warnings
5fa534b curl#14162 sectransp: fix clang compiler warnings, stop silencing them

10. CMake mis-detecting a CA bundle path on macOS

d2ef625 curl#14182 cmake: sync CA bundle/path detection with autotools

11. Failure to build tests with LibreSSL or wolfSSL with CMake

Fixed by dropping unnecessary includes, makign test builds dependent
on dependency headers.

3765d75 curl#14172 cmake: fix building `unit1600` due to missing `ssl/openssl.h`

12. curl tests with CMake

curl's CMake was missing bits for running the C preprocessor accurately.
It made tests 1119 and 1167 fail. I implemented the missing bits.

efc2c51 curl#14124 tests: include current directory when running test Perl commands
c09db8b curl#14129 cmake: create `configurehelp.pm` like autotools does
67cc1e3 curl#14125 test1119: adapt for `.md` input

13. GCC missing `__builtin_available()` support

curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.

14. Monotonic timer support with GCC

Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.

15. Runtime/test failures with GCC

I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.

16. Runtime/test failure in llvm/clang

AppleIDN support received a fix with two more remaining.

fd02508 curl#14179 curl#14176 IDN: fix ß with AppleIDN

17. Other issues found and fixed while working on this:

2c15aa5        GHA/macos: delete misplaced `CFLAGS`, drop redundant CMake option
80fb7c0 curl#14126 configure: limit `SystemConfiguration` test to non-c-ares, IPv6 builds
cfd6f43 curl#14127 build: tidy up `__builtin_available` feature checks (Apple)
bae5553 curl#14174 runtests: show name and keywords for failed tests in summary
09cdf7e curl#14178 cmake: delete unused `HAVE_LIBSSH2`, `HAVE_LIBSOCKET` macros
d3595c7 curl#14186 configure: CA bundle/path detection fixes
58772b0 curl#14187 runtests: set `SOURCE_DATE_EPOCH` to fix failing around midnight
18f1cd7 curl#14183 tests: sync feature names with `curl -V`
4c22d97 curl#14181 build: use `#error` instead of invalid syntax

Pending merges:

- curl#14185 runtests: fold test details for GitHub CI runs
- curl#14197 cmake: grab-bag of tidy-ups
- curl#14196 configure: limit `__builtin_available` test to Darwin

Summary:

In general GCC doesn't seem to be a good fit with curl and macOS for
now. These "lucky" combinations (GitHub Actions runner) will build out
of the box now: macos-14 + Xcode 15.0.1 + gcc-11, gcc-12, gcc-14. The
rest builds with the ugly workaround in place, but all this still leaves
some runtime issues.

More info and links in the commit messages and source code.

[1]: This PR:
- add info about target OS version requirements per feature, with OS
  names and release years.
- stop using `-Wno-deprecated-declarations` to suppress warnings.
- use `LDFLAGS=-w` to suppress 'object file was built for newer macOS
  version than being linked' warnings.
  (there were tens of thousands of them in some jobs)
- allow overriding Xcode version in all jobs.
- improve job names.
- abbreviate CMake as CM, autotools as AM for more compact job names.
- shorten job names by using `!` instead of `no-` and `non-`.
- bump parellel tests to 10 (from 5).
- drop using `--enable-maintainer-mode` `./configure` option.
- add gcc-12 no-ssl, autotools job with tests, ignore failing test
  results. (It's not yet clear why gcc-12 builds have different runtime
  results than clang/llvm ones.)
- add comments with OS names and release years next to version numbers,
  e.g. 10.15  # Catalina (2019)
- fix broken gcc-12 SecureTransport build.
- show compiler, Xcode, SDK, gcc hack SDK versions, Homebrew
  preinstalled packages and C compiler predefined macros for each job.
  Useful for debugging all the strange problems these builds might have.
- merge brew bundle and install steps.
- move step names to the top.
- dump configure log for both cmake and autotools also for successful
  builds. Useful for debugging.
- dump curl_config.h in short (sorted #defines) and full form.
- add support for the mainline llvm compiler.
- set sysroot for gcc and llvm.
- add timeout for cmake jobs.
- add new job matrix: combinations
  It supports building all possible compiler, runner image, Xcode/SDK
  combinations, with cmake and autotools, target OS versions and with or
  without SecureTransport. It's quick. GHA limits the maximum number of
  matrix jobs at 256.
  I used this as a test-rig to fix the macOS build fallouts with gcc and
  llvm.
  I settled with 16 jobs, trying to maximize fallout coverage.
- implement hack to make Homebrew gcc work with all available SDKs.
- add handy mini-table about Xcode / SDK versions, OS names, years for
  each GHA images, with the defaults.
- add tests for cmake jobs.
- make cmake config hack to link GnuTLS less intrusive.
- stop ignoring test 1452, seems fine now.
- fix to enable libpsl in autotools builds.
- enable libpsl in cmake builds.
- add an llvm job with tests (both autotools and cmake).
- delete similar macOS jobs from Circle CI. GHA is now arm64 too.

[2]: Homebrew GCC vs GHA runner images vs curl builds:
```
macOS      Xcode   gcc         gcc SDK hacks      Xcode SDK   SDK major Build Compile
           (*def)  (Homebrew)  (CommandLineTools)             versions        error
--------  -------- ----------  ------------------ ----------  --------- ----- ---------------------
macos-12   13.1    GCC 11.4.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 11.4.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 11.4.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 11.4.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 11.4.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 11.4.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12  *14.2    GCC 11.4.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13   14.1    GCC 11.4.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 11.4.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 11.4.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 11.4.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.1    GCC 11.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.2    GCC 11.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14   14.3.1  GCC 11.4.0  MacOSX14           MacOSX13.3  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14  *15.0.1  GCC 11.4.0  MacOSX14           MacOSX14.0
macos-14   15.1    GCC 11.4.0  MacOSX14           MacOSX14.2
macos-14   15.2    GCC 11.4.0  MacOSX14           MacOSX14.2
macos-14   15.3    GCC 11.4.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 11.4.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 11.4.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc@11/11.4.0/lib/gcc/11/gcc/aarch64-apple-darwin23/11/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12   13.1    GCC 12.4.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 12.4.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 12.4.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 12.4.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 12.4.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 12.4.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12  *14.2    GCC 12.4.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13   14.1    GCC 12.4.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 12.4.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 12.4.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 12.4.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.1    GCC 12.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13   15.2    GCC 12.4.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14   14.3.1  GCC 12.4.0  MacOSX14           MacOSX13.3  MISMATCH
macos-14  *15.0.1  GCC 12.4.0  MacOSX14           MacOSX14.0
macos-14   15.1    GCC 12.4.0  MacOSX14           MacOSX14.2
macos-14   15.2    GCC 12.4.0  MacOSX14           MacOSX14.2
macos-14   15.3    GCC 12.4.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 12.4.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 12.4.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc@12/12.4.0/lib/gcc/12/gcc/aarch64-apple-darwin23/12/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12   13.1    GCC 13.3.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 13.3.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 13.3.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 13.3.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 13.3.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 13.3.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-12  *14.2    GCC 13.3.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13   14.1    GCC 13.3.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 13.3.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 13.3.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 13.3.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13   15.1    GCC 13.3.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13   15.2    GCC 13.3.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   14.3.1  GCC 13.3.0  MacOSX14           MacOSX13.3  MISMATCH  FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14  *15.0.1  GCC 13.3.0  MacOSX14           MacOSX14.0            FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   15.1    GCC 13.3.0  MacOSX14           MacOSX14.2            FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   15.2    GCC 13.3.0  MacOSX14           MacOSX14.2            FAIL  /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14   15.3    GCC 13.3.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 13.3.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 13.3.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc@13/13.3.0/lib/gcc/13/gcc/aarch64-apple-darwin23/13/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12   13.1    GCC 14.1.0  MacOSX12           MacOSX12.0
macos-12   13.2.1  GCC 14.1.0  MacOSX12           MacOSX12.1
macos-12   13.3.1  GCC 14.1.0  MacOSX12           MacOSX12.3
macos-12   13.4.1  GCC 14.1.0  MacOSX12           MacOSX12.3
macos-12   14.0.1  GCC 14.1.0  MacOSX12           MacOSX12.3
macos-12   14.1    GCC 14.1.0  MacOSX12           MacOSX13.0  MISMATCH  FAIL  /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12  *14.2    GCC 14.1.0  MacOSX12           MacOSX13.1  MISMATCH  FAIL  /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13   14.1    GCC 14.1.0  MacOSX13           MacOSX13.0
macos-13   14.2    GCC 14.1.0  MacOSX13           MacOSX13.1
macos-13   14.3.1  GCC 14.1.0  MacOSX13           MacOSX13.3
macos-13  *15.0.1  GCC 14.1.0  MacOSX13           MacOSX14.0  MISMATCH  FAIL  /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13   15.1    GCC 14.1.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13   15.2    GCC 14.1.0  MacOSX13           MacOSX14.2  MISMATCH  FAIL  /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-14   14.3.1  GCC 14.1.0  MacOSX14           MacOSX13.3  MISMATCH
macos-14  *15.0.1  GCC 14.1.0  MacOSX14           MacOSX14.0
macos-14   15.1    GCC 14.1.0  MacOSX14           MacOSX14.2
macos-14   15.2    GCC 14.1.0  MacOSX14           MacOSX14.2
macos-14   15.3    GCC 14.1.0  MacOSX14           MacOSX14.4
macos-14   15.4    GCC 14.1.0  MacOSX14           MacOSX14.5
macos-14   16.0    GCC 14.1.0  MacOSX14           MacOSX15.0  MISMATCH  FAIL  /opt/homebrew/Cellar/gcc/14.1.0_1/lib/gcc/current/gcc/aarch64-apple-darwin23/14/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
```
Source: https://github.com/curl/curl/actions/runs/9883956647/job/27299564218

This commit fixes earlier commit
1e75edd, reverted in
41a7e0dcc9681afd91e066411bcee4f369c23366, where I cut the commit
message in half by accident. The patch itself is identical.

Closes curl#14097
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

2 participants