# Using a Google Cloud Storage bucket for dataset

**URL:** <https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253>\
**Category:** data\
**Created:** [March 12, 2022, 8:53pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253 "2022-03-12T20:53:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![cool\_cheetah1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cool_cheetah1/32/45577_2.png) [@cool\_cheetah1](https://discuss.pytorch.org/u/cool_cheetah1)\
**Post date:** [March 12, 2022, 8:53pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/1 "2022-03-12T20:53:00Z")

</div>

I’ve been stuck with this for a while now, and I’m certain this is a problem people have dealt with in the past.

So I’m trying to use torch dataloaders on a Google Cloud VM with a GPU attached. I want to run training on ImageNet. So my options are to either get a persistent disk and download ImageNet onto it, or otherwise download ImageNet onto a GCS bucket.

If I were to go the bucket route, does anyone know what the best way to then have this interface with a torch dataloader would be? I’ve looked into WebDataset as well, and it seems cool - but I’d likely have to pass in URLs (which isn’t necessarily a problem).

Long story short, I’m not sure if I should be mounting the bucket, or if things “just work” if I were to use WebDataset, and overall just what the best/most economical path is going forward.

Thanks very much!

---

<div class="post-metadata">

**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\
**Post date:** [March 14, 2022, 1:42pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/2 "2022-03-14T13:42:46Z")

</div>

You may checkout TorchData library we just released. We do have a DataPipe that takes URL from Google drive [data/online.py at 652986b14e893e08edaea2c519e21bc61706de5c · pytorch/data · GitHub](https://github.com/pytorch/data/blob/652986b14e893e08edaea2c519e21bc61706de5c/torchdata/datapipes/iter/load/online.py#L93)  
Let us know if this covers your request

---

<div class="post-metadata">

**Author:** ![Nilabhra](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nilabhra/32/8492_2.png) [@Nilabhra](https://discuss.pytorch.org/u/Nilabhra)\
**Post date:** [March 16, 2022, 4:17pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/3 "2022-03-16T16:17:28Z")

</div>

I don’t think GDrive is the same as GCS. Being able to load data from AWS s3 buckets and GCP gcs are features I am looking forward to.

---

<div class="post-metadata">

**Author:** ![nivek](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivek/32/39672_2.png) [@nivek](https://discuss.pytorch.org/u/nivek)\
**Post date:** [March 16, 2022, 6:59pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/4 "2022-03-16T18:59:28Z")

</div>

The `fsspec` and `iopath` [DataPipes here](https://pytorch.org/data/0.3.0/torchdata.datapipes.iter.html#io-datapipes) should allow you to load data from S3. Let us know that works for you.

I don’t think we have looked in to Google Cloud Storage but I’ll make a note of it.

---

<div class="post-metadata">

**Author:** ![Nilabhra](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nilabhra/32/8492_2.png) [@Nilabhra](https://discuss.pytorch.org/u/Nilabhra)\
**Post date:** [March 18, 2022, 10:50am UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/5 "2022-03-18T10:50:36Z")

</div>

Thanks! I’ll check if the functionalities allow for usage of GCS buckets.

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 2, 2022, 8:02am UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/6 "2022-06-02T08:02:08Z")

</div>

Hi,  
I’ve also started to check this option out.  
what I have found is the following:

1. there is an issue with setting credentials to `fsspec` in the [FSSpecFileListerIterDataPipe](https://github.com/pytorch/data/blob/cd38927904836f6f67ce33bfaee094fff4078402/torchdata/datapipes/iter/load/fsspec.py#L69). a possible solution is to add a `token` argument to `fsspec.core.url`)  
`fs, path = fsspec.core.url_to_fs(self.root, token=path/to/json)`
2. the output from [fs.protocol](https://github.com/pytorch/data/blob/aab67f314d9c8329ea688e5841abc16cbe2c4f42/torchdata/datapipes/iter/load/fsspec.py#L710) is a tuple `('gcs', 'gs')` .  
I’ve currently patched the code to take the relevant prefix for our GCS path.  
I hope to update once I get the pipeline running

---

<div class="post-metadata">

**Author:** ![nivek](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivek/32/39672_2.png) [@nivek](https://discuss.pytorch.org/u/nivek)\
**Post date:** [June 2, 2022, 2:48pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/7 "2022-06-02T14:48:50Z")

</div>

Hi @sephib, thanks for spotting the issue, we appreciate it!

Please open an issue (and a PR if you have a proposed fix) [on GitHub](https://github.com/pytorch/data).

---

<div class="post-metadata">

**Author:** ![nivek](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivek/32/39672_2.png) [@nivek](https://discuss.pytorch.org/u/nivek)\
**Post date:** [June 3, 2022, 9:05pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/8 "2022-06-03T21:05:47Z")

</div>

We currently have a PR: [Improve fsspec DataPipe to accept extra keyword arguments by ejguan · Pull Request #495 · pytorch/data · GitHub](https://github.com/pytorch/data/pull/495)

Let us know if it doesn’t resolve your need to pass a `token` argument.

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 6, 2022, 2:25pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/9 "2022-06-06T14:25:26Z")

</div>

👌🏽  
BTW - currently investigating 2 strange issues with the hotfix of the `token`

1. I tried to implement [the datapipline/dataloader tutorial](https://pytorch.org/data/beta/tutorial.html) - but when trying to iterate on `DataLoader` the system doesn’t do anything, doesn’t time out but keeps of running…

```python
...
for idx, batch in enumerate(dl):
    print(idx)

```

1. Another issue is that when running twice the following example - on the first time the system works, but on the second time (without restarting the kernel) the `fs.protocole` in the `open_file_by_fsspec` class returns a `file_uri` without `gs://` thus failing 😕

```python
datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
ds = Mapper(file_dp, PIL_open)
for i in ds:     
    print(f'{i=}')
    show_image(i)

```

---

<div class="post-metadata">

**Author:** ![nivek](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nivek/32/39672_2.png) [@nivek](https://discuss.pytorch.org/u/nivek)\
**Post date:** [June 6, 2022, 2:42pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/10 "2022-06-06T14:42:59Z")

</div>

For 1, can you let us know what version of the libraries (`torch` and `torchdata`) you have installed? And what is the exact code snippet that you are running?

1. By the hotfix, are you running the code from [this PR](https://github.com/pytorch/data/pull/495/files) or the fix that you described above?

cc: @ejguan

---

<div class="post-metadata">

**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\
**Post date:** [June 6, 2022, 4:05pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/11 "2022-06-06T16:05:52Z")

</div>

> [@sephib](#):
>
> 1. Another issue is that when running twice the following example - on the first time the system works, but on the second time (without restarting the kernel) the `fs.protocole` in the `open_file_by_fsspec` class returns a `file_uri` without `gs://` thus failing 😕
> 
> ```auto
> datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
> file_dp = datapipe.open_file_by_fsspec(mode='rb')
> ds = Mapper(file_dp, PIL_open)
> for i in ds:     
> print(f'{i=}')
> show_image(i)
> 
> ```

What do you mean about running twice? You mean iterating `ds` twice or running the script twice?

Does your `image_bucket` start with `gs://`?

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 6, 2022, 4:51pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/12 "2022-06-06T16:51:32Z")

</div>

1. the version are

```auto
Name: torch
Version: 1.11.0

Name: torchdata
Version: 0.3.0

```

The above code is the exact code that I’m running  
here is a simpler version

```python
datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)

```

after this if you run

```python
datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
file_dp = datapipe.open_file_by_fsspec(mode='rb')
list(file_dp)

```

again - you get a `FileNotFoundError` .  
Once I restart the kernel it works.  
If I run the script as below there is no problem - I’m guessing it is something regarding the how jupyter kernel uses fsspec 😕

> python my\_script.py

there is no problem. something is

1. the hotfix - is my local one - where I added the`token=` to `fsspec.core.url_to_fs`

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 6, 2022, 4:52pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/13 "2022-06-06T16:52:22Z")

</div>

yes - my URI `image_bucket` starts with `gs://`

---

<div class="post-metadata">

**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\
**Post date:** [June 6, 2022, 5:11pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/14 "2022-06-06T17:11:02Z")

</div>

> [@sephib](#):
>
> here is a simpler version
> 
> ```auto
> datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
> file_dp = datapipe.open_file_by_fsspec(mode='rb')
> list(file_dp)
> 
> ```
> 
> after this if you run
> 
> ```auto
> datapipe = FSSpecFileLister(root=image_bucket, masks=['*.png'])
> file_dp = datapipe.open_file_by_fsspec(mode='rb')
> list(file_dp)
> 
> ```
> 
> again - you get a `FileNotFoundError` .

This is interesting. I am not sure if this is something with the caching within `fsspec`. Could you please open an issue on Github?

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 6, 2022, 5:12pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/15 "2022-06-06T17:12:07Z")

</div>

Rergarding the [the datapipline/dataloader tutorial](https://pytorch.org/data/beta/tutorial.html) - this is the code that i’m trying to run wich doesn’t return any answer.

```python
def build_datapipes(root_dir=image_bucket):
    datapipe = FSSpecFileLister(root=root_dir, masks=['*.png'])
    file_dp = datapipe.open_file_by_fsspec(mode='rb')  
    datapipe = file_dp.map(PIL_open)
    return datapipe

datapipe = build_datapipes()
dl = DataLoader(dataset=datapipe, batch_size=1, num_workers=1)
list(dl)

```

When killing the service I can see in the stack that it is waiting on the `python3.9/multiprocessing/connection.py", line 936,`

---

<div class="post-metadata">

**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\
**Post date:** [June 6, 2022, 6:49pm UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/16 "2022-06-06T18:49:53Z")

</div>

Are you running this in ikernel? I believe notebook has some problems with multiprocessing (num\_workers\>0)

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 7, 2022, 4:45am UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/17 "2022-06-07T04:45:15Z")

</div>

No - I’m running via a regular python environment

> python my\_script.py

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 7, 2022, 5:34am UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/18 "2022-06-07T05:34:36Z")

</div>

Running with

> dl = DataLoader(dataset=datapipe)

Solved the issue  
Will need to understand why the `num_workers` is causing issues on my setup

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [June 7, 2022, 5:50am UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/19 "2022-06-07T05:50:38Z")

</div>

Opened an [issue on GitHub](https://github.com/pytorch/data/issues/497)

---

<div class="post-metadata">

**Author:** ![sephib](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sephib/32/50189_2.png) [@sephib](https://discuss.pytorch.org/u/sephib)\
**Post date:** [July 21, 2022, 5:30am UTC](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253/20 "2022-07-21T05:30:44Z")

</div>

Hi  
here is a [write up](https://sephib.github.io/when-pytorch-datapipes-meets-gcs.html#when-pytorch-datapipes-meets-gcs) of my solution

[Next page](https://discuss.pytorch.org/t/using-a-google-cloud-storage-bucket-for-dataset/146253.md?page=2)