Hi,
I’m trying to adapt the code from the PyTorch tutorial “Hyper-parameter tuning with Ray Tune”. I set the config variable like this:
config = {"batch_size" : tune.sample_from(lambda _: 2**np.random.randint(2, 9))}
but when I use it in the training function:
n_samps = n_feats * config["batch_size"]
I get a
TypeError: unsupported operand type(s) for *: 'int' and 'Function'
Also if I use
config = {"lr" : tune.loguniform(1e-4, 1e-1)}
then doing
2 * config["lr"]
results in a
TypeError: unsupported operand type(s) for *: 'int' and 'Float'
Indeed, config["lr"] is a ray.tune.sample.Float, and I don’t undrstand how to use it.
Any help would be appreciated.
Thanks!
Are you actually running the tuning algorithm from ray tune? Like tune.run_experiments() or tune.run()?
Because if you do that then these values will be sampled and replaced in your config dict with normal float / int values. Only when you try to use your configuration without going through tune will it contain these ray.tune.sample.Float types. If you want to do the latter anyway, just for debugging or whatnot, then call .sample() on the ray.tune.sample.Float and it’ll produce a usable value.
Thanks for the super-fast reply!
I am indeed using tune.run(). Adding below the beginning of Ray Tune’s output. BTW, this specific run was on a Google Colab notebook connected to my Windows PC as localhost, but I also tried it on a colab hosted connection and got the same error.
2020-12-09 09:59:22,510 INFO services.py:1090 -- View the Ray dashboard at [http://127.0.0.1:8265](http://127.0.0.1:8265/) 2020-12-09 09:59:29,556 WARNING worker.py:1091 -- The dashboard on node PC01235821 failed with the following error: Traceback (most recent call last): File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\dashboard\dashboard.py", line 956, in <module> dashboard = Dashboard( File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\dashboard\dashboard.py", line 512, in __init__ build_dir = setup_static_dir(self.app) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\dashboard\dashboard.py", line 405, in setup_static_dir raise OSError( FileNotFoundError: [Errno 2] Dashboard build directory not found. If installing from source, please follow the additional steps required to build the dashboard(cd python/ray/dashboard/client && npm ci && npm run build): 'F:\\conda_parent\\envs\\pytorch\\lib\\site-packages\\ray\\dashboard\\client/build' 2020-12-09 10:01:41,960 ERROR syncer.py:63 -- Log sync requires rsync to be installed. 2020-12-09 10:02:53,042 WARNING worker.py:1091 -- Warning: The actor ImplicitFunc has size 245397724 when pickled. It will be stored in Redis, which could cause memory issues. This may mean that its definition uses a large array or other object. 2020-12-09 10:03:03,080 WARNING util.py:137 -- The start_trial operation took 115.11174321174622 seconds to complete, which may be a performance bottleneck.
== Status ==
Memory usage on this node: 25.5/63.9 GiB
Using AsyncHyperBand: num_stopped=0 Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 1/20 CPUs, 1/1 GPUs, 0.0/24.51 GiB heap, 0.0/8.45 GiB objects
Result logdir: C:\Users\asafbe\ray_results\inner_2020-12-09_09-59-36
Number of trials: 1/10 (1 RUNNING)
|Trial name|status|loc|batch_size|conv_out1|dropout|kern_size|lin_out1|lr|n_conv_blocks|n_head|n_hid_ff|n_secs_per_seq|n_trans_layers|stride|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|inner_9257a_00000|RUNNING||32|8|0.431199|5|16|0.00230475|1|1|4|4|2|1|
(pid=17072) 2020-12-09 10:03:33,796 ERROR function_runner.py:254 -- Runner Thread raised error. (pid=17072) Traceback (most recent call last): (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 248, in run (pid=17072) self._entrypoint() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 315, in entrypoint (pid=17072) return self._trainable_func(self.config, self._status_reporter, (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 575, in _trainable_func (pid=17072) output = fn() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 643, in inner (pid=17072) fn(config, **fn_kwargs) (pid=17072) File "<ipython-input-88-08f6fd4506a8>", line 2, in main_train_eval (pid=17072) File "<ipython-input-77-855e742065de>", line 5, in call_batchify (pid=17072) TypeError: unsupported operand type(s) for *: 'int' and 'Function' (pid=17072) Exception in thread Thread-2: (pid=17072) Traceback (most recent call last): (pid=17072) File "F:\conda_parent\envs\pytorch\lib\threading.py", line 932, in _bootstrap_inner (pid=17072) self.run() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 267, in run (pid=17072) raise e (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 248, in run (pid=17072) self._entrypoint() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 315, in entrypoint (pid=17072) return self._trainable_func(self.config, self._status_reporter, (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 575, in _trainable_func (pid=17072) output = fn() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 643, in inner (pid=17072) fn(config, **fn_kwargs) (pid=17072) File "<ipython-input-88-08f6fd4506a8>", line 2, in main_train_eval (pid=17072) File "<ipython-input-77-855e742065de>", line 5, in call_batchify (pid=17072) TypeError: unsupported operand type(s) for *: 'int' and 'Function'