Explicitly declare autodetected C++ toolchain when compiling on Linux

Goal

Show how to explicitly declare the C++ toolchain that is automatically detected by Bazel and used to compile by default on Linux.

About the example

In this example we will see how we can explicitly declare a C++ toolchain in Bazel taking as a reference the toolchain automatically generated.

Currently in Bazel there are two ways to use toolchains, one is with --cpu, --crosstool_top and --compiler, and the other one is using platforms. The first way is complex, lacks flexibility and it is planned to be deprecated, the second one will replace the first one. Because of this, in this example we will show how to use toolchains building with platforms.

Bazel toolchains

In order to have an easy and fast start, Bazel provies an automatic detenction of toolchains. It detects which compilers are available in your system and configures automatically the C++ toolchains. In case of C++, a toolchain is composed mainly by compiler, linker, compiler flags, linker flags, system include directories, and library directories.

Obtain the default generted toolchain

The first think that we need to do, is to know which toolchain Bazel is using to compile C++ and where to find it. To do that we can build any target specifying the option --toolchain_resolution_debug.

bazel build --toolchain_resolution_debug //:hello_world

When doing that we will see in the logs which toolchain Bazel decided to use. Not all toolchains that appear in the log are used, we need to pay special attention to the part that says Selected toolchain in the logs starting with INFO: ToolchainResolution:.

INFO: ToolchainResolution:     Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: Rejected toolchain @local_config_cc//:cc-compiler-armeabi-v7a; mismatching values: arm, android
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: execution @local_config_platform//:host: Selected toolchain @local_config_cc//:cc-compiler-k8
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain @local_config_cc//:cc-compiler-k8
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, 
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, 

In this case the relevant line is the one that contains Selected toolchain @local_config_cc//:cc-compiler-k8 and it is telling us that the toolchain selected is @local_config_cc//:cc-compiler-k8. We need to keep in mind that the toolchain selected might depend on what is installed on the computer and what operating system you are using.

Now that we know what toolchain has been used, the next step is to take it as a reference to create our own toolchain. The good part of the autogenerated toolchain is that exist in the file system inside the bazel generated folders and we can take it almost as is.

Let's take it:

• Create a folder called toolchain that is where we will store our toolchain.
• Copy the content of bazel-linux_toolchain/external/local_config_cc except the WORKSPACE file, inside the toolchain folder that you just created. Make sure that are real copies and it does not contain any soft link.

Make sure that the autogenerated toolchain is not used

Before proceeding with the configuration of our new toolchain that we just copied, we want to make sure that Bazel does not use the autogenerated one. To do that we can define the environment variable BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1. In addition we also want to use the new way to work with toolchains, for that we can use --incompatible_enable_cc_toolchain_resolution.

Let's see what happens when we build our target with this options:

BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 bazel build //:hello_world --toolchain_resolution_debug --incompatible_enable_cc_toolchain_resolution
INFO: Invocation ID: bc16c31d-46f3-4a2f-a868-740aade11872
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform @local_config_platform//:host: No toolchains found.
ERROR: While resolving toolchains for target //:hello_world: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
ERROR: Analysis of target '//:hello_world' failed; build aborted: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
INFO: Elapsed time: 0.193s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 0 targets configured)

The relevant line here is ERROR: While resolving toolchains for target //:hello_world: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. As we wanted, Bazel is not able to compile anymore, because it cannot find the C++ toolchain. That is good because with this way we are sure that does not take the autogenerated toolchain.

Configuring our explicitly declared toolchain

If we look in the BUILD file that we just copied inside the toolchain folder, we will see that contains a cc_toolchain and a cc_toolchain_suite target. To be able to use the toolchain building with platforms, we need to add an additional target called toolchain that contains the following information:

• The toolchain to be used, needs to point to the cc_toolchain target, in this case cc-compiler-k8
• The type of the toolchain, in this case C++
• The constraints of the platform where the toolchain will be executed
• The constraints of the platform that the toolchain is targeting. Meaning the platform where the code will be executed. In case that we would do cross-compilation then exec_compatible_with and target_compatible_with would be different.

The target that we need to add is as follows:

toolchain(
    name = "my_linux_toolchain",
    toolchain = ":cc-compiler-k8",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
    exec_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    target_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
)

And now we are ready for the last step, that is to inform Bazel that the toolchain is available. To do that we need to register the toolchain in the WORKSPACE file (or in a .bzl file loaded in the WORKSPACE file).

register_toolchains(
    "//toolchain:my_linux_toolchain",
)

Declaring the platform to be used

As we saw before, we might have two different platforms, one where the toolchain gets executed, and another one that is the target platform. If we want to be correct we should define the target platform and indicate it when we compile. If we do not do that, Bazel will assume that the target platform is the same like the host.

To declare the platform is pretty simple, we just need to create a platform target containing the constrains of our platform. In this case it is Linux on a x64 cpu.

To keep it structure, we put this target in a BUILD file in a platform folder:

platform(
    name = "linux_x64",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    visibility = ["//visibility:public"],
)

Compiling with the explicitly declared toolchain

Now that the toolchain is registered, and the platform is declared, we just need to compile specifying the target platform and Bazel will know which toolchain to use.

BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 bazel build //:hello_world --toolchain_resolution_debug --platforms=//platform:linux_x64 --incompatible_enable_cc_toolchain_resolution
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 7df3db4c-ed4e-4296-aa53-366cebda0142
INFO: ToolchainResolution:   Type @bazel_tools//tools/cpp:toolchain_type: target platform //platform:linux_x64: execution @local_config_platform//:host: Selected toolchain //toolchain:cc-compiler-k8
INFO: ToolchainResolution: Target platform //platform:linux_x64: Selected execution platform @local_config_platform//:host, type @bazel_tools//tools/cpp:toolchain_type -> toolchain //toolchain:cc-compiler-k8
INFO: ToolchainResolution: Target platform @local_config_platform//:host: Selected execution platform @local_config_platform//:host, 
INFO: ToolchainResolution: Target platform //platform:linux_x64: Selected execution platform @local_config_platform//:host, 
INFO: Analyzed target //:hello_world (17 packages loaded, 47 targets configured).
INFO: Found 1 target...
Target //:hello_world up-to-date:
  bazel-bin/hello_world
INFO: Elapsed time: 5.250s, Critical Path: 0.21s
INFO: 6 processes: 2 remote cache hit, 4 internal.
INFO: Build completed successfully, 6 total actions

Because the part of the command line BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 and --incompatible_enable_cc_toolchain_resolution is something that we always want to do to make sure that we do not use anymore autmatically generated toolchains, we can extract it in the .bazelrc file.

And now we can build just doing:

bazel build //:hello_world --platforms=//platform:linux_x64

Cleaning up the toolchain and next steps

At this point everything is working fine but there is some cleanup that we can do. The automatic generated toolchain, contains indeed more than one toolchain but we do not use them all. Also contains a target called cc_toolchain_suite that is used in case of using --cpu and --crosstool_top, but that is not our case. Because of that, we should clean our toolchain folder and remove all unused targets like the arm targets (armeabi_cc_toolchain_config.bzl), cc_toolchain_suite, and any other target that is not referenced.

Because the automatic generated toolchain assumes the possibility that the compiler could be gcc or clang, you can also remove all clang branches from cc_toolchain_config.bzl.

Last but not least, I would also recommend that you apply buildifier to the files inside the toolchain folder. The automatically generated files are usually not properly formated.

Wrapping up

Now you have an explicitly declared toolchain for Linux. The next step is to go over the files and understand them. Then you can adapt the toolchain to your needs, adding features or changing compiler flags. Once you understand them you will see that you can keep cleaning up and remove unneeded features.

One of the next changes that is worth to mention, is the freezing of the compiler version. If you look at the default generated toolchain you will see that some of the tool paths are /usr/bin/cpp, /usr/bin/gcc, and /usr/bin/gcov. The problem of using this path, is that does not say anything about the version of the compiler to be used. If you have a single version install this will always be the same, but if you want to make sure that the version use is the one that you want, you should change the path for /usr/bin/cpp-9, /usr/bin/gcc-9, and /usr/bin/gcov-9.

Related links

• Building with platforms
• Example showing how to use platforms with custom C++ toolchains
• Bazel Tutorial: Configure C++ Toolchains

Related Bazel github issues

• #12712: --incompatible_enable_cc_toolchain_resolution still tries to access local_config_cc
• #7260: incompatible_enable_cc_toolchain_resolution: Turn on toolchain resolution for cc rules
• #12767: Provide a way to create a toolchain from an autodetected one