Hi Jerron!

The short story is that it appears that `MaskedTensor`

s don’t (currently)

broadcast against other `MaskedTensor`

s.

The somewhat longer story is that (for legitimate semantic reasons) pytorch

doesn’t like to perform, say, element-wise addition on `MaskedTensor`

s whose

masks differ. This probably bleeds over into the broadcasting case because

you would have to sort through what you want the semantics of broadcasting

the masks to be.

One straightforward approach to your use case: Use `MaskedTensor`

s to

compute the `.mean()`

and `.std()`

and then convert them to regular `Tensors`

to perform the normalization with broadcasting.

Consider:

```
>>> import torch
>>> print (torch.__version__)
2.0.1
>>>
>>> bad_value = 3.0
>>> y = torch.arange (15.).reshape (5, 3)
>>>
>>> mask = y != bad_value
>>> my = torch.masked.masked_tensor (y, mask)
<path_to_pytorch_install>\torch\masked\maskedtensor\core.py:156: UserWarning: The PyTorch API of MaskedTensors is in prototype stage and will change in the near future. Please open a Github issue for features requests and see our documentation on the torch.masked module for further information about the project.
warnings.warn(("The PyTorch API of MaskedTensors is in prototype stage "
>>> y_mean, y_std = torch.mean (my, dim = 0), torch.std (my, dim = 0)
>>>
>>> my
MaskedTensor(
[
[ 0.0000, 1.0000, 2.0000],
[ --, 4.0000, 5.0000],
[ 6.0000, 7.0000, 8.0000],
[ 9.0000, 10.0000, 11.0000],
[ 12.0000, 13.0000, 14.0000]
]
)
>>> y_mean, y_std
(MaskedTensor(
[ 6.7500, 7.0000, 8.0000]
), MaskedTensor(
[ 5.1235, 4.7434, 4.7434]
))
>>>
>>> my.shape, y.shape, (y - y_mean).shape, (y / y_std).shape
(torch.Size([5, 3]), torch.Size([5, 3]), torch.Size([5, 3]), torch.Size([5, 3]))
>>>
>>> y - y_mean # MaskedTensor y_mean broadcasts against regular Tensor y
MaskedTensor(
[
[ -6.7500, -6.0000, -6.0000],
[ -3.7500, -3.0000, -3.0000],
[ -0.7500, 0.0000, 0.0000],
[ 2.2500, 3.0000, 3.0000],
[ 5.2500, 6.0000, 6.0000]
]
)
>>>
>>> # my - y_mean # fails with ValueError -- y_mean doesn't broadcast against MaskedTensor my
>>>
>>> # one way to do this -- convert to regular Tensors for broadcasting ...
>>>
>>> nan = float ('nan') # use nans to track masked values
>>>
>>> y_norm = (my.to_tensor (nan) - y_mean.to_tensor (nan)) / y_std.to_tensor (nan) # regular Tensors will broadcast
>>> my_norm = torch.masked.masked_tensor (y_norm, ~y_norm.isnan()) # convert back to MaskedTensor, if desired
>>>
>>> y_norm
tensor([[-1.3175, -1.2649, -1.2649],
[ nan, -0.6325, -0.6325],
[-0.1464, 0.0000, 0.0000],
[ 0.4392, 0.6325, 0.6325],
[ 1.0247, 1.2649, 1.2649]])
>>> my_norm
MaskedTensor(
[
[ -1.3175, -1.2649, -1.2649],
[ --, -0.6325, -0.6325],
[ -0.1464, 0.0000, 0.0000],
[ 0.4392, 0.6325, 0.6325],
[ 1.0247, 1.2649, 1.2649]
]
)
>>>
>>> # check normalization
>>> y_norm.mean (dim = 0)
tensor([nan, 0., 0.])
>>> y_norm.std (dim = 0)
tensor([nan, 1., 1.])
>>>
>>> my_norm.mean (dim = 0)
MaskedTensor(
[ -0.0000, 0.0000, 0.0000]
)
>>> my_norm.std (dim = 0)
MaskedTensor(
[ 1.0000, 1.0000, 1.0000]
)
```

Best.

K. Frank