How to program this learning schedule

lr = x^0.5 * min(step_num^0.5, step_num*warmup_steps^-1.5)
where warmup_steps = 5000

I’m unable to formulate this.

Where are you stuck at the moment, i.e. what is not working?
Note that Python uses ** to calculate powers, while ^ is an XOR.

@ptrblck

Let me reframe my question
How to formulate this learning rate

lrate = d_model ^ 0.5 * min( step_num ^ 0.5, step_num * warmup_steps ^ -1.5)

Replace thr ^ operations with ** and if all inputs are defined it should work fine.
I don’t know what d_model is but it might be a constant or any other value you could pass to the scheduler.
You should be able to define this schedule in e.g. lr_scheduler.LambdaLR.