Finally I have found set of flags, that allows me to properly build torch for my hardware. So, I want to share with the community:
cmake -D CMAKE_BUILD_TYPE=Release \
-D USE_AVX=OFF \
-D USE_NNPACK=OFF \
-D USE_MKLDNN=OFF \
-D C_HAS_AVX_2=OFF \
-D C_HAS_AVX2_2=OFF \
-D CXX_HAS_AVX_2=OFF \
-D CXX_HAS_AVX2_2=OFF \
-D CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS=OFF \
..