I want to calculate number of distinct values in a tensor as loss for image generation.
The target gray image should contain only several gray values in range of (0, 255). Large number of distinct values should get high loss.

torch.unique() method identifies uniques elements in a tensor. You can use (tensor).numel() method to find number of elements in a tensor.
Hence you can combine both and calculate total loss as:

If by “loss” you mean something that can be used to train a model (i.e., something you can get gradients from), then the answer is NO, since such loss would not be differentiable.

You can use the L-1 loss, since it’s the closest convex approximation of L-0: loss = torch.norm(output-target, p=1). If you feel confident and ok with discarding convexity, you can even try a fractional norm, e.g. L-1/2: loss = torch.norm(output-target, p=0.5). This latter choice may probably lead to unstable gradients, though.