# Hessian vector product optimization

**URL:** <https://discuss.pytorch.org/t/hessian-vector-product-optimization/40717>\
**Category:** Uncategorized\
**Created:** [March 23, 2019, 9:28pm UTC](https://discuss.pytorch.org/t/hessian-vector-product-optimization/40717 "2019-03-23T21:28:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fei\_Wang1](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/c5a1d2/32.png) [@Fei\_Wang1](https://discuss.pytorch.org/u/Fei_Wang1)\
**Post date:** [March 23, 2019, 9:28pm UTC](https://discuss.pytorch.org/t/hessian-vector-product-optimization/40717/1 "2019-03-23T21:28:13Z")

</div>

This is a piece of code that compute Hessian vector product (gradient of gradient with regard to a given vector).

input = torch.tensor([1.0, 2.0, 0.5, 0.2], requires\_grad=True)  
output = input.tanh().sum()  
grads = torch.autograd.grad(output, input, create\_graph=True, retain\_graph = True)  
flatten = torch.cat([g.reshape(-1) for g in grads if g is not None])  
for i in range(100):  
┆ v = torch.randn(4)  
┆ hvps = torch.autograd.grad([flatten @ v], input, allow\_unused=True, **retain\_graph = True** )  
┆ print("{} {} {}".format(output.data, grads[0].data, hvps[0].data))

PyTorch says the bold part ( **retain\_graph=True** in hvps computation) is necessary. Otherwise this error msg shows up.

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain\_graph=True when calling backward the first time.

However, I am wondering if that “ **retain\_graph=True** ” is REALLY necessary? I might be wrong, but it seems to me that computing Hessian vector product for v1 doesn’t depend on that for v2. Will this be unnecessary memory overhead? Could this code snippet be written differently to avoid keeping those graphs that are not needed?

---

<div class="post-metadata">

**Author:** ![Kushaj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushaj/32/58838_2.png) [@Kushaj](https://discuss.pytorch.org/u/Kushaj)\
**Post date:** [March 23, 2019, 9:50pm UTC](https://discuss.pytorch.org/t/hessian-vector-product-optimization/40717/2 "2019-03-23T21:50:11Z")

</div>

In the backward pass, to reduce the memory footprint all the intermediate results are cleared by default. To backprop again, you have to again build the graph, with `retain_graph=True` the intermediates are not deleted.

---

<div class="post-metadata">

**Author:** ![Fei\_Wang1](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/c5a1d2/32.png) [@Fei\_Wang1](https://discuss.pytorch.org/u/Fei_Wang1)\
**Post date:** [March 23, 2019, 10:00pm UTC](https://discuss.pytorch.org/t/hessian-vector-product-optimization/40717/3 "2019-03-23T22:00:58Z")

</div>

Hi Kushaj,

Thanks for the prompt reply.

However, in my case, I think the memory footprint can be further optimized.

Basically, I have a function **f** , that I compute gradient **g**. Then I compute **gradient of g** with regard to different vectors in the **for loop**.

I believe that each iteration in the **for loop** depends on the computation of **g** , but doesn’t depend on each other. However, pyTorch is asking me to **retain\_graph** for **all** computations, which I think is not necessary.

Do you see the special situation I am having here?

Fei

---

<div class="post-metadata">

**Author:** ![Kushaj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kushaj/32/58838_2.png) [@Kushaj](https://discuss.pytorch.org/u/Kushaj)\
**Post date:** [March 23, 2019, 10:04pm UTC](https://discuss.pytorch.org/t/hessian-vector-product-optimization/40717/4 "2019-03-23T22:04:18Z")

</div>

Use this as a reference as they go into more detail [link](https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795)