rebuild for torch-sys is triggered everytime

Hi, I have exported LD_LIBRARY_PATH  and LIBTORCH as required and  have a working tch-rs with cuda . However everytime I tried to run tests, torch-sys is rebuild ("Compiling torch-sys v0.10.0") which takes long time.

However,It works fine on my other pc without setting env vars for LIBTORCH with only cpu

maybe relevant to this PR  : https://github.com/LaurentMazare/tch-rs/pull/184 