Lets say I have objective functions A, B and C. A is the original (GT supervised) objective function I want to minimize, while B and C are unsupervised alternatives (resulting in the same end-goal).
Now I want to find out if B or C are better in approximating A.
Therefore, I trained my model with A and calculated B and C along the way.
Is it correct to say that the objective function with the higher correlation coefficient to A will also give the best results?
You have to be careful with the following matters:
Association does not mean necessarily a causal relation between both variables. For example, there might be a third variable you have not considered and this third variable might be the explanation for the behavior of the other two.
Even if there is a causal relationship between the variables, the correlation coefficient does not tell you which variable is the cause and which is the effect.
If the coefficient is close to 0, it does not necessarily mean that there is no relation between the two variables. It means there isn’t a LINEAR relationship, but there might be another type of functional relationship (for example, quadratic or exponential).
Just to be clear: Are you suggesting that A influences Z and Z influences B and C? In that case, does it really matter when I just want to quantify which objective function leads to a better result?
As I am training with A and only observe B and C, the former must be the cause - or am I mistaken?
I trained for the first epoch using A and the correlation coefficients are: c(A,B) ~ 0.3 and c(A,C) ~ 0.5. Therefore, I believe they have a reasonably linear relationship.
Can I do something else or do I need to do ablation studies (i.e. training the whole network with B and C and compare results w.r.t. GT) ?