10000 Revert "Revert "[ML] Upgrade to Pytorch 2.1.2 and zlib 1.2.13 (#2588)"" · edsavage/ml-cpp@dc68361 · GitHub
[go: up one dir, main page]
Skip to content

Commit dc68361

Browse files
edsavageedsavage
committed
Revert "Revert "[ML] Upgrade to Pytorch 2.1.2 and zlib 1.2.13 (elastic#2588)""
This reverts commit 435d3f9.
1 parent 435d3f9 commit dc68361

File tree

24 files changed

+57
-101
lines changed

24 files changed

+57
-101
lines changed

‎.buildkite/pipelines/build_linux.json.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
"cpu": "6",
3737
"ephemeralStorage": "20G",
3838
"memory": "64G",
39-
"image": "docker.elastic.co/ml-dev/ml-linux-build:27"
39+
"image": "docker.elastic.co/ml-dev/ml-linux-build:28"
4040
},
4141
"aarch64": {
4242
"provider": "aws",
@@ -100,7 +100,7 @@ def main(args):
100100
"cpu": "6",
101101
"ephemeralStorage": "20G",
102102
"memory": "64G",
103-
"image": "docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:11"
103+
"image": "docker.elastic.co/ml-dev/ml-linux-aarch64-cross-build:12"
104104
},
105105
"commands": [
106106
".buildkite/scripts/steps/build_and_test.sh"

‎.buildkite/pipelines/build_macos.json.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ def main(args):
8282
"cpu": "6",
8383
"ephemeralStorage": "20G",
8484
"memory": "64G",
85-
"image": "docker.elastic.co/ml-dev/ml-macosx-build:17"
85+
"image": "docker.elastic.co/ml-dev/ml-macosx-build:18"
8686
},
8787
"commands": [
8888
f'if [[ "{args.action}" == "debug" ]]; then export ML_DEBUG=1; fi',

‎3rd_party/licenses/pytorch-INFO.csv

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
name,version,revision,url,license,copyright,sourceURL
2-
PyTorch,1.11.0,bc2c6edaf163b1a1330e37a6e34caf8c553e4755,https://pytorch.org,BSD-3-Clause,,
2+
PyTorch,2.1.2,a8e7c98cb95ff97bb30a728c6b2a1ce6bff946eb,https://pytorch.org,BSD-3-Clause,,

‎3rd_party/licenses/zlib-INFO.csv

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
name,version,revision,url,license,copyright,sourceURL
2-
zlib,1.2.12,21767c654d31d2dccdde4330529775c6c5fd5389,http://zlib.net,Zlib,Copyright (C) 1995-2022 Jean-loup Gailly and Mark Adler,
2+
zlib,1.2.13,04f42ceca40f73e2978b50e93806c2a18c1281fc,http://zlib.net,Zlib,Copyright (C) 1995-2022 Jean-loup Gailly and Mark Adler,

‎build-setup/linux.md

Lines changed: 3 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -319,7 +319,7 @@ Then copy the shared libraries to the system directory:
319319
sudo cp /opt/intel/mkl/lib/intel64/libmkl*.so /usr/local/gcc103/lib
320320
```
321321

322-
### PyTorch 1.13.1
322+
### PyTorch 2.1.2
323323

324324
(This step requires a reasonable amount of memory. It failed on a machine with 8GB of RAM. It succeeded on a 16GB machine. You can specify the number of parallel jobs using environment variable MAX_JOBS. Lower number of jobs will reduce memory usage.)
325325

@@ -338,7 +338,7 @@ sudo /usr/local/gcc103/bin/python3.10 -m pip install install numpy pyyaml setupt
338338
Then obtain the PyTorch code:
339339

340340
```
341-
git clone --depth=1 --branch=v1.13.1 git@github.com:pytorch/pytorch.git
341+
git clone --depth=1 --branch=v2.1.2 git@github.com:pytorch/pytorch.git
342342
cd pytorch
343343
git submodule sync
344344
git submodule update --init --recursive
@@ -366,7 +366,7 @@ export USE_MKLDNN=ON
366366
export USE_QNNPACK=OFF
367367
export USE_PYTORCH_QNNPACK=OFF
368368
[ $(uname -m) = x86_64 ] && export USE_XNNPACK=OFF
369-
export PYTORCH_BUILD_VERSION=1.13.1
369+
export PYTORCH_BUILD_VERSION=2.1.2
370370
export PYTORCH_BUILD_NUMBER=1
371371
/usr/local/gcc103/bin/python3.10 setup.py install
372372
```
@@ -380,44 +380,6 @@ sudo cp torch/lib/libtorch_cpu.so /usr/local/gcc103/lib
380380
sudo cp torch/lib/libc10.so /usr/local/gcc103/lib
381381
```
382382

383-
### Intel Extension for PyTorch
384-
If you are building on x86_64, you can optionally build the Intel Extension for PyTorch (IPEX). This extension provides additional optimizations for Intel CPUs. It is not available for aarch64. IPEX library is required to run PyTorch models quantized using the IPEX backend.
385-
386-
Begin by cloning the IPEX repository:
387-
388-
```bash
389-
git clone https://github.com/intel/intel-extension-for-pytorch.git
390-
cd intel-extension-for-pytorch
391-
git fetch --all
392-
git checkout v1.13.100+cpu
393-
git submodule sync
394-
git submodule update --init --recursive
395-
git config --global --add safe.directory `pwd`
396-
```
397-
398-
IPEX expects that PyTorch build directory contains a file `build-version`, which contains the PyTorch version string. This file is not created by the PyTorch build process, so we need to create it manually:
399-
```bash
400-
echo "1.13.1+cpu\n" > ${PYTORCH_SRC_DIR}/torch/build-version
401-
```
402-
403-
This assumes that you have cloned PyTorch in the directory `${PYTORCH_SRC_DIR}` in the step above. Make sure that this path is correct.
404-
405-
Building IPEX requires a lot of memory. To reduce this requirement, we can patch the IPEX build system to lower the number of parallel processes. We use `sed` to replace the call to `multiprocessing.cpu_count()` with a constant `1` in the file `setup.py`:
406-
```bash
407-
sed -i 's/multiprocessing.cpu_count()/1/g' setup.py
408-
```
409-
410-
Finally, we can build IPEX:
411-
```bash
412-
export CC=/usr/local/gcc103/bin/gcc
413-
export TORCH_VERSION="v1.13.1"
414-
export TORCH_IPEX_VERSION="1.13.100+cpu"
415-
/usr/local/gcc103/bin/python3.10 -m pip install -r requirements.txt
416-
/usr/local/gcc103/bin/python3.10 setup.py clean
417-
/usr/local/gcc103/bin/python3.10 setup.py build_clib ${PYTORCH_SRC_DIR}/torch
418-
cp build/Release/packages/intel_extension_for_pytorch/lib/libintel-ext-pt-cpu.so /usr/local/gcc103/lib
419-
```
420-
421383
### valgrind
422384

423385
`valgrind` is not required to build the code. However, since we build `gcc` ourselves, if you want

‎build-setup/macos.md

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -130,7 +130,7 @@ Download the graphical installer for Python 3.10.9 from <https://www.python.org/
130130

131131
Install using all the default options. When the installer completes a Finder window pops up. Double click the `Install Certificates.command` file in this folder to install the SSL certificates Python needs.
132132

133-
### PyTorch 1.13.1
133+
### PyTorch 2.1.2
134134

135135
PyTorch requires that certain Python modules are installed. To install them:
136136

@@ -141,7 +141,7 @@ sudo /Library/Frameworks/Python.framework/Versions/3.10/bin/pip3.10 install inst
141141
Then obtain the PyTorch code:
142142

143143
```
144-
git clone --depth=1 --branch=v1.13.1 https://github.com/pytorch/pytorch.git
144+
git clone --depth=1 --branch=v2.1.2 https://github.com/pytorch/pytorch.git
145145
cd pytorch
146146
git submodule sync
147147
git submodule update --init --recursive
@@ -158,6 +158,18 @@ external processes.
158158
Edit `tools/setup_helpers/cmake.py` and add `"DNNL_TARGET_ARCH"` to the list
159159
of environment variables that get passed through to CMake (around line 215).
160160

161+
For compilation on Apple silicon Macs edit `third_party/ideep/include/ideep/utils.hpp`
162+
and add:
163+
164+
```c++
165+
inline void to_bytes(bytestring& bytes, const unsigned long arg) {
166+
auto as_cstring = reinterpret_cast<const char*>(&arg);
167+
bytes.append(as_cstring, sizeof(unsigned long));
168+
}
169+
```
170+
171+
at around line 189. This is necessary to resolve a template specialization issue.
172+
161173
Build as follows:
162174
163175
```
@@ -171,7 +183,7 @@ export USE_MKLDNN=ON
171183
export USE_QNNPACK=OFF
172184
export USE_PYTORCH_QNNPACK=OFF
173185
[ $(uname -m) = x86_64 ] && export USE_XNNPACK=OFF
174-
export PYTORCH_BUILD_VERSION=1.13.1
186+
export PYTORCH_BUILD_VERSION=2.1.2
175187
export PYTORCH_BUILD_NUMBER=1
176188
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 setup.py install
177189
```

‎build-setup/windows.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ On the "Workloads" page that is displayed after a short while, check "Desktop de
5555

5656
### Git for Windows
5757

58-
Download `Git-2.16.3-64-bit.exe` from <https://github.com/git-for-windows/git/releases/download/v2.16.3.windows.1>.
58+
Download `Git-2.42.0.2-64-bit.exe` from <https://github.com/git-for-windows/git/releases/download/v2.42.0.windows.2/Git-2.42.0.2-64-bit.exe>.
5959

6060
Install it using mainly the default options suggested by the installer, but on the feature selection dialog also check "On the desktop" in the "Additional icons" section.
6161

@@ -65,25 +65,25 @@ As well as providing a Git implementation, Git for Windows comes with Windows po
6565

6666
### CMake
6767

68-
CMake version 3.19.2 is the minimum required to build ml-cpp. Download the MSI installer for version 3.23.2 from <https://github.com/Kitware/CMake/releases/download/v3.23.2/cmake-3.23.2-win64-x64.msi> (or get a more recent version).
68+
CMake version 3.19.2 is the minimum required to build ml-cpp. Download the MSI installer for version 3.23.3 from <https://github.com/Kitware/CMake/releases/download/v3.23.3/cmake-3.23.3-windows-x86_64.msi> (or get a more recent version).
6969

7070
Install it mainly using the default options _except_ on the "Install Options" dialog check "Add CMake to the system PATH for all users".
7171

7272
### zlib
7373

7474
Whilst it is possible to download a pre-built version of `zlib1.dll`, for consistency we want one that links against the Visual Studio 2019 C runtime library. Therefore it is necessary to build zlib from source.
7575

76-
Download the source code from <http://zlib.net/> - the file is called `zlib1212.zip`. Unzip this file under `C:\tools`, so that you end up with a directory called `C:\tools\zlib-1.2.12`.
76+
Download the source code from <http://zlib.net/> - the file is called `zlib1213.zip`. Unzip this file under `C:\tools`, so that you end up with a directory called `C:\tools\zlib-1.2.13`.
7777

7878
To build, start a command prompt using Start Menu -&gt; Apps -&gt; Visual Studio 2019 -&gt; x64 Native Tools Command Prompt for VS 2019, then in it type:
7979

8080
```
81-
cd \tools\zlib-1.2.12
81+
cd \tools\zlib-1.2.13
8282
nmake -f win32/Makefile.msc LOC="-D_WIN32_WINNT=0x0601"
8383
nmake -f win32/Makefile.msc test
8484
```
8585

86-
All the build output will end up in the top level `C:\tools\zlib-1.2.12` directory. Once the build is complete, copy `zlib1.dll` and `minigzip.exe` to `C:\usr\local\bin`. Copy `zlib.lib` and `zdll.lib` to `C:\usr\local\lib`. And copy `zlib.h` and `zconf.h` to `C:\usr\local\include`.
86+
All the build output will end up in the top level `C:\tools\zlib-1.2.13` directory. Once the build is complete, copy `zlib1.dll` and `minigzip.exe` to `C:\usr\local\bin`. Copy `zlib.lib` and `zdll.lib` to `C:\usr\local\lib`. And copy `zlib.h` and `zconf.h` to `C:\usr\local\include`.
8787

8888
### libxml2
8989

@@ -147,8 +147,8 @@ Start a command prompt using Start Menu -&gt; Apps -&gt; Visual Studio 2019 -&gt
147147
```
148148
cd \tools\boost_1_83_0
149149
bootstrap.bat
150-
b2 -j6 --layout=versioned --disable-icu --toolset=msvc-14.2 cxxflags="-std:c++17" linkflags="-std:c++17" --build-type=complete -sZLIB_INCLUDE="C:\tools\zlib-1.2.12" -sZLIB_LIBPATH="C:\tools\zlib-1.2.12" -sZLIB_NAME=zdll --without-context --without-coroutine --without-graph_parallel --without-mpi --without-python architecture=x86 address-model=64 optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_WIN32_WINNT=0x0601
151-
b2 install --prefix=C:\usr\local --layout=versioned --disable-icu --toolset=msvc-14.2 cxxflags="-std:c++17" linkflags="-std:c++17" --build-type=complete -sZLIB_INCLUDE="C:\tools\zlib-1.2.12" -sZLIB_LIBPATH="C:\tools\zlib-1.2.12" -sZLIB_NAME=zdll --without-context --without-coroutine --without-graph_parallel --without-mpi --without-python architecture=x86 address-model=64 optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_WIN32_WINNT=0x0601
150+
b2 -j6 --layout=versioned --disable-icu --toolset=msvc-14.2 cxxflags="-std:c++17" linkflags="-std:c++17" --build-type=complete -sZLIB_INCLUDE="C:\tools\zlib-1.2.13" -sZLIB_LIBPATH="C:\tools\zlib-1.2.13" -sZLIB_NAME=zdll --without-context --without-coroutine --without-graph_parallel --without-mpi --without-python architecture=x86 address-model=64 optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_WIN32_WINNT=0x0601
151+
b2 install --prefix=C:\usr\local --layout=versioned --disable-icu --toolset=msvc-14.2 cxxflags="-std:c++17" linkflags="-std:c++17" --build-type=complete -sZLIB_INCLUDE="C:\tools\zlib-1.2.13" -sZLIB_LIBPATH="C:\tools\zlib-1.2.13" -sZLIB_NAME=zdll --without-context --without-coroutine --without-graph_parallel --without-mpi --without-python architecture=x86 address-model=64 optimization=speed inlining=full define=BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS define=BOOST_LOG_WITHOUT_DEBUG_OUTPUT define=BOOST_LOG_WITHOUT_EVENT_LOG define=BOOST_LOG_WITHOUT_SYSLOG define=BOOST_LOG_WITHOUT_IPC define=_WIN32_WINNT=0x0601
152152
```
153153

154154
The Boost headers and appropriate libraries should end up in `C:\usr\local\include` and `C:\usr\local\lib` respectively.
@@ -193,7 +193,7 @@ On the "Advanced Options" screen, check "Install for all users" and "Add Python
193193

194194
For the time being, do not take advantage of the option on the final installer screen to reconfigure the machine to allow paths longer than 260 characters. We still support Windows versions that do not have this option.
195195

196-
### PyTorch 1.13.1
196+
### PyTorch 2.1.2
197197

198198
(This step requires a lot of memory. It failed on a machine with 12GB of RAM. It just about fitted on a 20GB machine. 32GB RAM is recommended.)
199199

@@ -209,7 +209,7 @@ Next, in a Git bash shell run:
209209

210210
```
211211
cd /c/tools
212-
git clone --depth=1 --branch=v1.13.1 git@github.com:pytorch/pytorch.git
212+
git clone --depth=1 --branch=v2.1.2 https://github.com/pytorch/pytorch.git
213213
cd pytorch
214214
git submodule sync
215215
git submodule update --init --recursive
@@ -265,7 +265,7 @@ set USE_QNNPACK=OFF
265265
set USE_PYTORCH_QNNPACK=OFF
266266
set USE_XNNPACK=OFF
267267
set MSVC_Z7_OVERRIDE=OFF
268-
set PYTORCH_BUILD_VERSION=1.13.1
268+
set PYTORCH_BUILD_VERSION=2.1.2
269269
set PYTORCH_BUILD_NUMBER=1
270270
python setup.py install
271271
```

‎dev-tools/docker/build_linux_aarch64_cross_build_image.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
HOST=docker.elastic.co
2323
ACCOUNT=ml-dev
2424
REPOSITORY=ml-linux-aarch64-cross-build
25-
VERSION=11
25+
VERSION=12
2626

2727
set -e
2828

‎dev-tools/docker/build_linux_aarch64_native_build_image.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ sleep 5
3434
HOST=docker.elastic.co
3535
ACCOUNT=ml-dev
3636
REPOSITORY=ml-linux-aarch64-native-build
37-
VERSION=11
37+
VERSION=12
3838

3939
set -e
4040

‎dev-tools/docker/build_linux_build_image.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ sleep 5
3434
HOST=docker.elastic.co
3535
ACCOUNT=ml-dev
3636
REPOSITORY=ml-linux-build
37-
VERSION=27
37+
VERSION=28
3838

3939
set -e
4040

0 commit comments

Comments
 (0)
0