Average Pooling with Outlier Ignored?

I’d like to use Average Pooling to downsample a depth map, but only including the points that are close to each other.

For example, if a pooling neighbourhood consists of

[4 4.1, 3.9 3.8] then we take the mean of the whole thing

but if it consists of

[4 4.1 3.9 0.1], then we should ignore 0.1 and take the mean of the rest

Does anything like that exist?