# Intermediate tensors in NeMo

**URL:** <https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714>\
**Category:** audio\
**Created:** [June 24, 2020, 12:22pm UTC](https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714 "2020-06-24T12:22:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jhh3](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c37758/32.png) [@jhh3](https://discuss.pytorch.org/u/jhh3)\
**Post date:** [June 24, 2020, 12:22pm UTC](https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714/1 "2020-06-24T12:22:23Z")

</div>

Hi,  
I’m looking at QuartzNet in NeMo and trying to probe some of the internal tensors. I see that I can use `evaluated_tensors = neural_factory.infer(tensors=[a,b,c])` to run inference and return the evaulations of a,b,c, but I can’t figure out how to get a list of the intermediate activation tensors, or to get a pointer to one. I’m looking for a method of either the neural\_factory or the individual models (like jasper\_encoder) that would return a list of tensors that I can pass to infer(). Any ideas?  
Thanks  
_Edit:_ I know infer() and the neural\_factory object are NeMo, so maybe out of scope for this board, but the underlying model is based on a PyTorch module, so I’m hoping a general PyTorch method for getting internal activations will be useful here. Here’s the inheritance tree for that encoder model.

```auto
[nemo.collections.asr.jasper.JasperEncoder,
 nemo.backends.pytorch.nm.TrainableNM,
 nemo.core.neural_modules.NeuralModule,
 abc.ABC,
 torch.nn.modules.module.Module,
 object]
```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 25, 2020, 5:03am UTC](https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714/2 "2020-06-25T05:03:50Z")

</div>

For a general PyTorch model you could use forward hooks to get the intermediate activations as described [here](https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6). Since NeMo seems to be using a PyTorch model internally, you would have to access its layers to register the hooks.

---

<div class="post-metadata">

**Author:** ![jhh3](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c37758/32.png) [@jhh3](https://discuss.pytorch.org/u/jhh3)\
**Post date:** [June 25, 2020, 8:00pm UTC](https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714/3 "2020-06-25T20:00:46Z")

</div>

Thanks ptrblck! That worked. It takes a little poking around to figure out the class structure, but not too much. For example, using your example, I was able to use your example and this line:  
`encoder.encoder[1].mconv[0].conv.register_forward_hook(get_activation('B1(a).mconv.1D'))`  
to capture the output of the 1D convolution in the first block of a QuartzNet encoder model.  
Again, thanks for the help.

---

<div class="post-metadata">

**Author:** ![jhh3](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/c37758/32.png) [@jhh3](https://discuss.pytorch.org/u/jhh3)\
**Post date:** [July 1, 2020, 11:10am UTC](https://discuss.pytorch.org/t/intermediate-tensors-in-nemo/86714/4 "2020-07-01T11:10:30Z")

</div>

After messing with this for a while, I wanted to add one caveat that I encountered. Some layers objects, like ReLU, are re-used throughout the network. I guess it’s any layer without parameters, but I’m not sure. The result is that if you put a hook on a ReLU layer, like `encoder.encoder[1].mout[0].register_forward_hook(get_activation('B1.mout.relu'))` in QuartzNet, _it gets called for every ReLU in the whole model_, not just the one you wanted. So the final result in the dictionary is actually the ReLU output for the final model output, not the ReLU output associated with the layer where you registered the hook.