# Reaching \`transformer\` attribute while model is wrapped in DataParallel

**URL:** <https://discuss.pytorch.org/t/reaching-transformer-attribute-while-model-is-wrapped-in-dataparallel/64556>\
**Category:** nlp\
**Created:** [December 19, 2019, 4:28pm UTC](https://discuss.pytorch.org/t/reaching-transformer-attribute-while-model-is-wrapped-in-dataparallel/64556 "2019-12-19T16:28:27Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ernst\_Stavro\_Blofeld](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ernst_stavro_blofeld/32/3234_2.png) [@Ernst\_Stavro\_Blofeld](https://discuss.pytorch.org/u/Ernst_Stavro_Blofeld)\
**Post date:** [December 19, 2019, 4:28pm UTC](https://discuss.pytorch.org/t/reaching-transformer-attribute-while-model-is-wrapped-in-dataparallel/64556/1 "2019-12-19T16:28:27Z")

</div>

Hi folks.

I successfully managed to use Huggingface transformers with Pytorch using **a single GPU**.

Now, I’m trying to use multiple gpus with `DataParallel`. While wrapped in `DataParallel`, my model begins as follows:

```auto
DataParallel(
  (module): DataParallel(
    (module): CustomTransformerModel(
      (transformer): RobertaForSequenceClassification(
        (roberta): RobertaModel(
          (embeddings): RobertaEmbeddings(
            (word_embeddings): Embedding(50265, 768, padding_idx=1)
            (position_embeddings): Embedding(514, 768, padding_idx=1)
            (token_type_embeddings): Embedding(1, 768)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
[...]
etc..

```

Whareas while NOT wrapped in DataParallel, the model look like this:

```auto
CustomTransformerModel(
  (transformer): RobertaForSequenceClassification(
    (roberta): RobertaModel(
      (embeddings): RobertaEmbeddings(
        (word_embeddings): Embedding(50265, 768, padding_idx=1)
        (position_embeddings): Embedding(514, 768, padding_idx=1)
        (token_type_embeddings): Embedding(1, 768)
        (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
[...]
etc

```

As I try to reach the `transformer` as it is wrapped in `DataParallel`, I incur in the (predictable) error: `AttributeError: 'DataParallel' object has no attribute 'transformer'`.

I ran a quick search and found this thread: [How to reach model attributes wrapped by nn.DataParallel?](https://discuss.pytorch.org/t/how-to-reach-model-attributes-wrapped-by-nn-dataparallel/1373)

but still, `model.module.transformer` outputs `AttributeError: 'DataParallel' object has no attribute 'transformer'`.

The thread is old though (2017).

How can I reach `transformer` attribute with Pytorch 1.3.1? Thanks!!

---

<div class="post-metadata">

**Author:** ![cgnarendiran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cgnarendiran/32/27519_2.png) [@cgnarendiran](https://discuss.pytorch.org/u/cgnarendiran)\
**Post date:** [August 5, 2020, 8:53pm UTC](https://discuss.pytorch.org/t/reaching-transformer-attribute-while-model-is-wrapped-in-dataparallel/64556/2 "2020-08-05T20:53:54Z")

</div>

Looks like you have applied the Dataparallel class twice to the model