Generally, some numerical differences with different hardware are expected as different implementations will incur some differences due to operation reordering. Could you try inference on the same data (e.g., the same image) and check how much the resulting values differ between CPU and GPU?