8000 chore: remove protoc grpc from generation config (#3390) · rmehta19/sdk-platform-java@7ee0089 · GitHub
[go: up one dir, main page]
Skip to content

Commit 7ee0089

Browse files
JoeWang1127JoeWang1127
authored
chore: remove protoc grpc from generation config (googleapis#3390)
In this PR: - Remove `protoc_version` and `grpc_version` from generation config.
1 parent ac82c67 commit 7ee0089

15 files changed

+174
-311
lines changed

‎.cloudbuild/library_generation/library_generation.Dockerfile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -88,24 +88,23 @@ WORKDIR /protoc
8888
RUN source /src/library_generation/utils/utilities.sh \
8989
&& download_protoc "${PROTOC_VERSION}" "${OS_ARCHITECTURE}"
9090
# we indicate protoc is available in the container via env vars
91-
ENV DOCKER_PROTOC_LOCATION=/protoc
91+
ENV DOCKER_PROTOC_LOCATION=/protoc/bin
9292
ENV DOCKER_PROTOC_VERSION="${PROTOC_VERSION}"
9393

9494
# install grpc
9595
WORKDIR /grpc
9696
RUN source /src/library_generation/utils/utilities.sh \
9797
&& download_grpc_plugin "${GRPC_VERSION}" "${OS_ARCHITECTURE}"
9898
# similar to protoc, we indicate grpc is available in the container via env vars
99-
ENV DOCKER_GRPC_LOCATION="/grpc/protoc-gen-grpc-java-${GRPC_VERSION}-${OS_ARCHITECTURE}.exe"
100-
ENV DOCKER_GRPC_VERSION="${GRPC_VERSION}"
101-
99+
ENV DOCKER_GRPC_LOCATION="/grpc/protoc-gen-grpc-java.exe"
102100

103101
# Here we transfer gapic-generator-java from the previous stage.
104102
# Note that the destination is a well-known location that will be assumed at runtime
105103< B41A div class="diff-text-inner"># We hard-code the location string to avoid making it configurable (via ARG) as
106104
# well as to avoid it making it overridable at runtime (via ENV).
107105
COPY --from=ggj-build "/sdk-platform-java/gapic-generator-java.jar" "${HOME}/.library_generation/gapic-generator-java.jar"
108106
RUN chmod 755 "${HOME}/.library_generation/gapic-generator-java.jar"
107+
ENV GAPIC_GENERATOR_LOCATION="${HOME}/.library_generation/gapic-generator-java.jar"
109108

110109
RUN python -m pip install --upgrade pip
111110

@@ -130,6 +129,7 @@ RUN apk del -r npm && apk cache clean
130129
ADD https://maven-central.storage-download.googleapis.com/maven2/com/google/googlejavaformat/google-java-format/${JAVA_FORMAT_VERSION}/google-java-format-${JAVA_FORMAT_VERSION}-all-deps.jar \
131130
"${HOME}"/.library_generation/google-java-format.jar
132131
RUN chmod 755 "${HOME}"/.library_generation/google-java-format.jar
132+
ENV JAVA_FORMATTER_LOCATION="${HOME}/.library_generation/google-java-format.jar"
133133

134134
# allow users to access the script folders
135135
RUN chmod -R o+rx /src

‎hermetic_build/DEVELOPMENT.md

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,21 @@ as per [POSIX env var definition](https://pubs.opengroup.org/onlinepubs/96999197
8282
mv /path/to/jar "${HOME}/.library_generation/gapic-generator-java.jar"
8383
```
8484

85+
#### Put the protobuf compiler in its well-known location
86+
1. Download protobuf compiler from [GitHub releases](https://github.com/protocolbuffers/protobuf/releases).
87+
2. Move the folder into its well-know location.
88+
89+
```shell
90+
unzip /path/to/zipfile -d "${HOME}/.library_generation/"
91+
```
92+
#### Put the GRPC plugin in its well-known location
93+
1. Download GRPC plugin from [Maven Central](https://central.sonatype.com/artifact/io.grpc/protoc-gen-grpc-java/versions).
94+
2. Move the folder into its well-know location.
95+
96+
```shell
97+
mv /path/to/protoc-gen-grpc-java.exe "${HOME}/.library_generation/protoc-gen-grpc-java.exe"
98+
```
99+
85100
#### Put the java formatter jar in its well-known location
86101

87102
1. Download google-java-format-{version}-all-deps.jar from [Maven Central](https://central.sonatype.com/artifact/com.google.googlejavaformat/google-java-format)
@@ -197,11 +212,6 @@ python hermetic_build/library_generation/cli/entry_point.py generate \
197212
--repository-path=/workspace \
198213
--api-definitions-path=/workspace/apis
199214
```
200-
Note that if you specify the generator version using environment variable,
201-
`-e GENERATOR_VERSION="${LOCAL_GENERATOR_VERSION}"` in the above example,
202-
you should not set `gapic_generator_version` and `protoc_version` in the
203-
generation configuration because values in the generation configuration will
204-
take precedence.
205215

206216
# Debug the library generation container
207217
If you are working on changing the way the containers are created, you may want

‎hermetic_build/README.md

Lines changed: 5 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,6 @@ Running the docker image built from `hermetic_build/library_generation`
77
directory, you can generate a repository containing GAPIC client libraries (a
88
monorepo, for example, google-cloud-java) from a configuration file.
99

10-
Instead of running the docker image, if you prefer running the underlying python
11-
scripts directly, please refer to the [development guide](DEVELOPMENT.md#run-the-script)
12-
for additional instructions.
13-
1410
## Environment
1511

1612
- OS: Linux
@@ -124,13 +120,11 @@ The repository level parameters define the version of API definition (proto)
124120
and tools.
125121
They are shared by library level parameters.
126122

127-
| Name | Required | Notes |
128-
|:------------------------|:--------:|:---------------------------------------------|
129-
| gapic_generator_version | No | set through env variable if not specified |
130-
| protoc_version | No | inferred from the generator if not specified |
131-
| grpc_version | No | inferred from the generator if not specified |
132-
| googleapis_commitish | Yes | |
133-
| libraries_bom_version | No | empty string if not specified |
123+
| Name | Required | Notes |
124+
|:------------------------|:--------:|:------------------------------------------|
125+
| gapic_generator_version | No | set through env variable if not specified |
126+
| googleapis_commitish | Yes | |
127+
| libraries_bom_version | No | empty string if not specified |
134128

135129
### Library level parameters
136130

@@ -178,7 +172,6 @@ The GAPIC level parameters define how to generate a GAPIC library.
178172

179173
```yaml
180174
gapic_generator_version: 2.34.0
181-
protoc_version: 25.2
182175
googleapis_commitish: 1a45bf7393b52407188c82e63101db7dc9c72026
183176
libraries_bom_version: 26.37.0
184177
libraries:

‎hermetic_build/common/model/generation_config.py

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,6 @@ def __init__(
3939
libraries: list[LibraryConfig],
4040
gapic_generator_version: Optional[str] = None,
4141
libraries_bom_version: Optional[str] = None,
42-
grpc_version: Optional[str] = None,
43-
protoc_version: Optional[str] = None,
4442
):
4543
self.googleapis_commitish = googleapis_commitish
4644
self.libraries_bom_version = (
@@ -50,8 +48,6 @@ def __init__(
5048
gapic_generator_version
5149
)
5250
self.libraries = libraries
53-
self.grpc_version = grpc_version
54-
self.protoc_version = protoc_version
5551
# explicit set to None so that we can compute the
5652
# value in getter.
5753
self.__contains_common_protos = None
@@ -172,8 +168,6 @@ def from_yaml(path_to_yaml: str) -> GenerationConfig:
172168
config, "googleapis_commitish", REPO_LEVEL_PARAMETER
173169
),
174170
gapic_generator_version=__optional(config, GAPIC_GENERATOR_VERSION, None),
175-
grpc_version=__optional(config, "grpc_version", None),
176-
protoc_version=__optional(config, "protoc_version", None),
177171
libraries_bom_version=__optional(config, LIBRARIES_BOM_VERSION, None),
178172
libraries=parsed_libraries,
179173
)

‎hermetic_build/common/tests/model/config_change_unit_tests.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -309,8 +309,6 @@ def __get_a_gen_config(
309309
return GenerationConfig(
310310
gapic_generator_version="",
311311
googleapis_commitish=googleapis_commitish,
312-
grpc_version="",
313-
protoc_version="",
314312
libraries=libraries,
315313
)
316314

‎hermetic_build/common/tests/model/generation_config_unit_tests.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,6 @@ def test_generation_config_set_generator_version_from_env(self):
7474
def test_from_yaml_succeeds(self):
7575
config = from_yaml(f"{test_config_dir}/generation_config.yaml")
7676
self.assertEqual("2.34.0", config.gapic_generator_version)
77-
self.assertEqual(25.2, config.protoc_version)
7877
self.assertEqual(
7978
"1a45bf7393b52407188c82e63101db7dc9c72026", config.googleapis_commitish
8079
)

‎hermetic_build/common/tests/resources/test-config/generation_config.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
gapic_generator_version: 2.34.0
2-
protoc_version: 25.2
32
googleapis_commitish: 1a45bf7393b52407188c82e63101db7dc9c72026
43
libraries_bom_version: 26.37.0
54
libraries:

‎hermetic_build/common/tests/utils/generation_config_comparator_unit_tests.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,15 +39,11 @@ def setUp(self) -> None:
3939
self.baseline_config = GenerationConfig(
4040
gapic_generator_version="",
4141
googleapis_commitish="",
42-
grpc_version="",
43-
protoc_version="",
4442
libraries=[self.baseline_library],
4543
)
4644
self.current_config = GenerationConfig(
4745
gapic_generator_version="",
4846
googleapis_commitish="",
49-
grpc_version="",
50-
protoc_version="",
5147
libraries=[self.current_library],
5248
)
5349

‎hermetic_build/library_generation/generate_composed_library.py

Lines changed: 1 addition & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,6 @@ def generate_composed_library(
5959
:return None
6060
"""
6161
output_folder = repo_config.output_folder
62-
base_arguments = __construct_tooling_arg(config=config)
6362
owlbot_cli_source_folder = util.sh_util("mktemp -d")
6463
os.makedirs(f"{library_path}", exist_ok=True)
6564
for gapic in library.get_sorted_gapic_configs():
@@ -81,7 +80,7 @@ def generate_composed_library(
8180
)
8281
temp_destination_path = f"java-{gapic.proto_path.replace('/','-')}"
8382
effective_arguments = __construct_effective_arg(
84-
base_arguments=base_arguments,
83+
base_arguments=[],
8584
gapic=gapic,
8685
gapic_inputs=gapic_inputs,
8786
temp_destination_path=temp_destination_path,
@@ -120,19 +119,6 @@ def generate_composed_library(
120119
)
121120

122121

123-
def __construct_tooling_arg(config: GenerationConfig) -> List[str]:
124-
"""
125-
Construct arguments of tooling versions used in generate_library.sh
126-
:param config: the generation config
127-
:return: arguments containing tooling versions
128-
"""
129-
arguments = []
130-
arguments += util.create_argument("grpc_version", config)
131-
arguments += util.create_argument("protoc_version", config)
132-
133-
return arguments
134-
135-
136122
def __construct_effective_arg(
137123
base_arguments: List[str],
138124
gapic: GapicConfig,

‎hermetic_build/library_generation/generate_library.sh

Lines changed: 3 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,6 @@ case $key in
1414
destination_path="$2"
1515
shift
1616
;;
17-
--protoc_version)
18-
protoc_version="$2"
19-
shift
20-
;;
21-
--grpc_version)
22-
grpc_version="$2"
23-
shift
24-
;;
2517
--proto_only)
2618
proto_only="$2"
2719
shift
@@ -71,14 +63,6 @@ script_dir=$(dirname "$(readlink -f "$0")")
7163
source "${script_dir}"/utils/utilities.sh
7264
output_folder="$(get_output_folder)"
7365

74-
if [ -z "${protoc_version}" ]; then
75-
protoc_version=$(get_protoc_version)
76-
fi
77-
78-
if [ -z "${grpc_version}" ]; then
79-
grpc_version=$(get_grpc_version)
80-
fi
81-
8266
if [ -z "${proto_only}" ]; then
8367
proto_only="false"
8468
fi
@@ -171,16 +155,14 @@ case "${proto_path}" in
171155
proto_files="${proto_files//${removed_proto}/}"
172156
;;
173157
esac
174-
# download gapic-generator-java, protobuf and grpc plugin.
175-
# the download_tools function will create the environment variables "protoc_path"
176-
# and "grpc_path", to be used in the protoc calls below.
177-
download_tools "${protoc_version}" "${grpc_version}" "${os_architecture}"
158+
159+
protoc_path=$(get_protoc_location)
178160
##################### Section 1 #####################
179161
# generate grpc-*/
180162
#####################################################
181163
if [[ ! "${transport}" == "rest" ]]; then
182164
# do not need to generate grpc-* if the transport is `rest`.
183-
"${protoc_path}"/protoc "--plugin=protoc-gen-rpc-plugin=${grpc_path}" \
165+
"${protoc_path}"/protoc "--plugin=protoc-gen-rpc-plugin=$(get_grpc_plugin_location)" \
184166
"--rpc-plugin_out=:${temp_destination_path}/java_grpc.jar" \
185167
${proto_files} # Do not quote because this variable should not be treated as one long string.
186168
# unzip java_grpc.jar to grpc-*/src/main/java

‎hermetic_build/library_generation/tests/generate_library_unit_tests.py

Lines changed: 73 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,6 @@ def setUp(self):
3939
# in its well-known location
4040
self.simulated_home = get_bash_call_output("mktemp -d")
4141
bash_call(f"mkdir {self.simulated_home}/.library_generation")
42-
bash_call(
43-
f"touch {self.simulated_home}/.library_generation/gapic-generator-java.jar"
44-
)
45-
4642
# We create a per-test directory where all output files will be created into.
4743
# Each folder will be deleted after its corresponding test finishes.
4844
test_dir = get_bash_call_output("mktemp -d")
@@ -76,34 +72,84 @@ def _run_command_and_get_sdout(self, command, **kwargs):
7672
command, stderr=subprocess.PIPE, **kwargs
7773
).stdout.decode()[:-1]
7874

79-
def test_get_grpc_version_with_no_env_var_fails(self):
80-
# the absence of DOCKER_GRPC_VERSION will make this function to fail
81-
result = self._run_command("get_grpc_version")
75+
def test_get_generator_location_with_env_returns_env(self):
76+
os.environ["GAPIC_GENERATOR_LOCATION"] = "/gapic-generator-java"
77+
result = self._run_command_and_get_sdout("get_gapic_generator_location")
78+
self.assertEqual("/gapic-generator-java", result)
79+
os.environ.pop("GAPIC_GENERATOR_LOCATION")
80+
81+
def test_get_generator_location_without_env_with_local_returns_local(self):
82+
bash_call(
83+
f"touch {self.simulated_home}/.library_generation/gapic-generator-java.jar"
84+
)
85+
result = self._run_command_and_get_sdout("get_gapic_generator_location")
86+
self.assertEqual(
87+
f"{self.simulated_home}/.library_generation/gapic-generator-java.jar",
88+
result,
89+
)
90+
91+
def test_get_generator_location_with_no_env_no_local_file_failed(self):
92+
result = self._run_command("get_gapic_generator_location")
93+
self.assertEqual(1, result.returncode)
94+
self.assertRegex(result.stdout.decode(), "Can't find GAPIC generator in")
95+
96+
def test_get_protoc_location_with_env_returns_env(self):
97+
os.environ["DOCKER_PROTOC_LOCATION"] = "/protoc"
98+
result = self._run_command_and_get_sdout("get_protoc_location")
99+
self.assertEqual("/protoc", result)
100+
os.environ.pop("DOCKER_PROTOC_LOCATION")
101+
102+
def test_get_protoc_location_without_env_with_local_returns_local(self):
103+
bash_call(f"mkdir -p {self.simulated_home}/.library_generation/bin")
104+
result = self._run_command_and_get_sdout("get_protoc_location")
105+
self.assertEqual(
106+
f"{self.simulated_home}/.library_generation/bin",
107+
result,
108+
)
109+
110+
def test_get_protoc_location_with_no_env_no_local_file_failed(self):
111+
result = self._run_command("get_protoc_location")
82112
self.assertEqual(1, result.returncode)
83-
self.assertRegex(result.stdout.decode(), "DOCKER_GRPC_VERSION is not set")
113+
self.assertRegex(result.stdout.decode(), "Can't find protoc in")
114+
115+
def test_get_grpc_plugin_location_with_env_returns_env(self):
116+
os.environ["DOCKER_GRPC_LOCATION"] = "/grpc"
117+
result = self._run_command_and_get_sdout("get_grpc_plugin_location")
118+
self.assertEqual("/grpc", result)
119+
os.environ.pop("DOCKER_GRPC_LOCATION")
84120

85-
def test_get_protoc_version_with_no_env_var_fails(self):
86-
# the absence of DOCKER_PROTOC_VERSION will make this function to fail
87-
result = self._run_command("get_protoc_version")
121+
def test_get_grpc_plugin_location_without_env_with_local_returns_local(self):
122+
bash_call(
123+
f"touch {self.simulated_home}/.library_generation/protoc-gen-grpc-java.exe"
124+
)
125+
result = self._run_command_and_get_sdout("get_grpc_plugin_location")
126+
self.assertEqual(
127+
f"{self.simulated_home}/.library_generation/protoc-gen-grpc-java.exe",
128+
result,
129+
)
130+
131+
def test_get_grpc_plugin_location_with_no_env_no_local_file_failed(self):
132+
result = self._run_command("get_grpc_plugin_location")
88133
self.assertEqual(1, result.returncode)
89-
self.assertRegex(result.stdout.decode(), "DOCKER_PROTOC_VERSION is not set")
134+
self.assertRegex(result.stdout.decode(), "Can't find grpc plugin in")
90135

91-
def test_download_tools_without_baked_generator_fails(self):
92-
# This test has the same structure as
93-
# download_tools_succeed_with_baked_protoc, but meant for
94-
# gapic-generator-java.
136+
def test_get_formatter_location_with_env_returns_env(self):
137+
os.environ["JAVA_FORMATTER_LOCATION"] = "/java-formatter"
138+
result = self._run_command_and_get_sdout("get_java_formatter_location")
139+
self.assertEqual("/java-formatter", result)
140+
os.environ.pop("JAVA_FORMATTER_LOCATION")
95141

96-
test_protoc_version = "1.64.0"
97-
test_grpc_version = "1.64.0"
98-
jar_location = (
99-
f"{self.simulated_home}/.library_generation/gapic-generator-java.jar"
142+
def test_get_formatter_location_without_env_with_local_returns_local(self):
143+
bash_call(
144+
f"touch {self.simulated_home}/.library_generation/google-java-format.jar"
100145
)
101-
# we expect the function to fail because the generator jar is not found in
102-
# its well-known location. To achieve this, we temporarily remove the fake
103-
# generator jar
104-
bash_call(f"rm {jar_location}")
105-
result = self._run_command(
106-
f"download_tools {test_protoc_version} {test_grpc_version} {self.TEST_ARCHITECTURE}"
146+
result = self._run_command_and_get_sdout("get_java_formatter_location")
147+
self.assertEqual(
148+
f"{self.simulated_home}/.library_generation/google-java-format.jar",
149+
result,
107150
)
151+
152+
def test_get_formatter_location_with_no_env_no_local_file_failed(self):
153+
result = self._run_command("get_java_formatter_location")
108154
self.assertEqual(1, result.returncode)
109-
self.assertRegex(result.stdout.decode(), "Please configure your environment")
155+
self.assertRegex(result.stdout.decode(), "Can't find Java formatter in")

0 commit comments

Comments
 (0)
0