I have followed instructions for the Mac on https://github.com/pytorch/pytorch#from-source . The instructions for the main build all completed with no errors. I then try to build the docs locally and get an error implying that an Onnix related library is not installed. I could pip install pytorch but I want to use my locally built Pytorch -which should include torch.onnix.
make html
Traceback (most recent call last):
File "/Users/davidradley/pytorch/docs/source/scripts/onnx/build_onnx_supported_aten_op_csv_table.py", line 8, in <module>
from torch.onnx import _onnx_supported_ops
ImportError: cannot import name '_onnx_supported_ops' from 'torch.onnx' (/opt/homebrew/anaconda3/lib/python3.10/site-packages/torch/onnx/__init__.py)
make: *** [onnx] Error 1
You might need to just install pytorch from source instead, instructions are on the same page. Another way is to just test in CI if you don’t want to bother building pytorch from source since that can take a bit
Thanks for you reply @marksaroufim . I successfully ran python3 setup.py develop in the root folder of my PyTorch fork. I am new at this - I thought that would have built PyTorch from source - how can I check that the source build worked and will be picked up as a library for the Python script that runs in the docs make file?
thanks - good tip . I see torch 1.12.1 /opt/homebrew/anaconda3/lib/python3.10/site-packages I assume I would expect it to point into my local git repo.
I ran python setup.py develop and see the same in pip list
I run whereis python and get python: /opt/homebrew/anaconda3/bin/python whereis python3 gives me python3: /usr/bin/python3 /opt/homebrew/share/man/man1/python3.1
I have more going on in my environment that is probably relevant. I am using brew pyenv to set 3.10 version of Python. Also I am trying to use conda-forge. So previously ran:
pip list shows I have a torch package (not local, but there is one) so why does the doc make file python file not find the torch onnx package?
Why is the local source not being found?
I am wondering whether creating a virtual environment for the make might help isolate the Python environment. This would seem best practise or I am always at the mercy of the python on my machine.
I see:
cat /opt/homebrew/anaconda3/lib/python3.10/site-packages/torch.egg-link
/Users/davidradley/pytorch
.
Googling around this format says there should be no new lines in this file format. But this does not seem to matter as the egg-link files are not used for resolve the path for imports.
The paths for imports should be resolved in cat /opt/homebrew/anaconda3/lib/python3.10/site-packages/easy-install.pth
I installed miniconda3. I installed make through brew as I did not have one. I followed all the instructions and then issue
make html in the docs folder and get
Traceback (most recent call last):
File "/Users/davidradley/pytorch/docs/source/scripts/build_opsets.py", line 74, in <module>
main()
File "/Users/davidradley/pytorch/docs/source/scripts/build_opsets.py", line 57, in main
aten_ops_list = get_aten()
File "/Users/davidradley/pytorch/docs/source/scripts/build_opsets.py", line 19, in get_aten
parsed_yaml = parse_native_yaml(NATIVE_FUNCTION_YAML_PATH, TAGS_YAML_PATH)
File "/Users/davidradley/miniconda3/lib/python3.10/site-packages/torchgen/gen.py", line 235, in parse_native_yaml
_GLOBAL_PARSE_NATIVE_YAML_CACHE[path] = parse_native_yaml_struct(
File "/Users/davidradley/miniconda3/lib/python3.10/site-packages/torchgen/gen.py", line 167, in parse_native_yaml_struct
error_check_native_functions(rs)
File "/Users/davidradley/miniconda3/lib/python3.10/site-packages/torchgen/gen.py", line 277, in error_check_native_functions
assert len(base_func_map[out_of_place_base_name]) > 0, (
AssertionError: resize_as_ is marked with tag: inplace_view. The codegen expects there to be a corresponding out-of-place view op with the name 'resize_as_' and matching schema, but it didn't find one.
This is strange as as the assert in gen.py is not intended to be driven for str(f.func.name) of “resize_as_” due to the if condition and str(f.func.name) != "resize_as_".
if (
"inplace_view" in f.tags
and str(f.func.name) != "resize_"
and str(f.func.name) != "resize_as_"
):
base_name = f.func.name.name
overload_name = f.func.name.overload_name
assert base_name.inplace, (
f"{f.func.name} is marked with tag: inplace_view, but it doesn't follow the naming "
"convention for inplace ops - the codegen expects the base name to have a trailing underscore. "
)
out_of_place_base_name = BaseOperatorName(
base_name.base, False, base_name.dunder_method
)
assert len(base_func_map[out_of_place_base_name]) > 0, (
f"{f.func.name} is marked with tag: inplace_view. The codegen expects there to be a corresponding "
f"out-of-place view op with the name '{base_name}' and matching schema, but it didn't find one. "
)
The version of gen.py in miniconda3 does not have the extra and str(f.func.name) != "resize_as_" condition. But the one in my local file system is correct and has the extra if condition. I am incorrectly picking up the back level version of gen.py from miniconda3. This seems to be the problem - I now seem to be missing the egg-links for torch an torchgen.
I am continuing to update this forum entry in case it helps anyone in the future
It looks like I uninstalled brew anaconda and installed the miniconda3. But I am having issues running the
setup.py now as errors. It seems something in my environment is referencing /opt/homebrew/anaconda3/ which is now deleted.
There was a warning saying .conda/environments.txt was not writable, I chmodded it o be writeable and removed the anaconda3 environment.
I also ran python3 setup.py develop --cmake
this allowed the make command to run
I reran python3 setup.py develop
I see the torch egg-link is present now.
ls /Users/davidradley/miniconda3/lib/python3.10/site-packages/*.egg-link
/Users/davidradley/miniconda3/lib/python3.10/site-packages/pytorch-sphinx-theme.egg-link
/Users/davidradley/miniconda3/lib/python3.10/site-packages/torch.egg-link
there is no egg-link for torchgen - I am guessing this is why it is not picking up the correct python file.
there is also no egg-info folder for torchgen
also there is no entry for torchgen in easy-install.pth
. I see