How to use OneCycleLR

CasellaJr · August 24, 2022, 10:56am

I want to train on CIFAR-10, suppose for 200 epochs.
This is my optimizer:
optimizer = optim.Adam([x for x in model.parameters() if x.requires_grad], lr=0.001)
I want to use OneCycleLR as scheduler. Now, according to the documentation, these are the parameters of OneCycleLR:

torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, total_steps=None, epochs=None, steps_per_epoch=None, pct_start=0.3, anneal_strategy='cos', cycle_momentum=True, base_momentum=0.85, max_momentum=0.95, div_factor=25.0, final_div_factor=10000.0, three_phase=False, last_epoch=- 1, verbose=False)

I have seen that the most used are max_lr, epochs and steps_per_epoch. The documentation says this:

**max_lr** (float or list) – Upper learning rate boundaries in the cycle for each parameter group.
**epochs** (int) – The number of epochs to train for. This is used along with steps_per_epoch in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None
**steps_per_epoch** (int) – The number of steps per epoch to train for. This is used along with epochs in order to infer the total number of steps in the cycle if a value for total_steps is not provided. Default: None

About steps_per_epoch, I have seen in many github repo that it is used steps_per_epoch=len(data_loader), so if I have a batch size of 128, then this parameter it is equal to 128.
However I do not understand what are the other 2 parameters. If I want to train for 200 epochs, then epochs=200? Or this is a parameter that runs the scheduler only for epoch and then it restarts? For example, If I write epochs=10 inside the scheduler, but I train in total for 200, it is like 20 complete steps of the scheduler?
Then max_lr I have seen people using a value greater than the lr of the optimizer and other people using a smaller value. I think that max_lr must be greater than the lr (otherwise why it is called max ?)
However, if I print the learning rate epoch by epoch, it assumes strange values. For example, in this setting:

optimizer = optim.Adam([x for x in model.parameters() if x.requires_grad], lr=0.001)

scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr = 0.01, epochs=200, steps_per_epoch=128)

And this is the learning rate:

Epoch 1: TrL=1.7557, TrA=0.3846, VL=1.4136, VA=0.4917, TeL=1.4266, TeA=0.4852, LR=0.0004,
Epoch 2: TrL=1.3414, TrA=0.5123, VL=1.2347, VA=0.5615, TeL=1.2231, TeA=0.5614, LR=0.0004,
...
Epoch 118: TrL=0.0972, TrA=0.9655, VL=0.8445, VA=0.8161, TeL=0.8764, TeA=0.8081, LR=0.0005,
Epoch 119: TrL=0.0939, TrA=0.9677, VL=0.8443, VA=0.8166, TeL=0.9094, TeA=0.8128, LR=0.0005,

So it is incresing

ayansengupta17 · April 10, 2023, 9:50am

Did you figure it out?

CasellaJr · April 11, 2023, 10:55am

Maybe yes… I am not sure that I am using it in the correct way, however I do this:
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr = 0.01, epochs=10, steps_per_epoch=312)
Steps per epochs are dataset size / batch size, and in my case was 40k/128 that is 312. Then, according to the number of epochs of training, I change the epochs parameter of the scheduler. In particular, I do epochs = epochs_of_training/100. So, if I do 1000 training epochs, then I use 10 in scheduler. I do not know if the procedure is correct, however, learning rate is as I expect:

Summary

Epoch 1: TrL=0.4240, TrA=0.8672, VL=0.1108, VA=0.9652, TeL=0.0973, TeA=0.9696, LR=0.00040,et=67.21,tt=67.22,Current memory:210799.00,Peak memory:4102834.00,
Epoch 2: TrL=0.1529, TrA=0.9503, VL=0.0769, VA=0.9770, TeL=0.0616, TeA=0.9785, LR=0.00040,et=59.62,tt=126.84,Current memory:48315.00,Peak memory:3949948.00,
Epoch 3: TrL=0.1119, TrA=0.9639, VL=0.0650, VA=0.9808, TeL=0.0541, TeA=0.9825, LR=0.00040,et=62.42,tt=189.26,Current memory:83904.00,Peak memory:3992829.00,
Epoch 4: TrL=0.0931, TrA=0.9703, VL=0.0589, VA=0.9827, TeL=0.0497, TeA=0.9845, LR=0.00040,et=57.31,tt=246.57,Current memory:80447.00,Peak memory:3951891.00,
Epoch 5: TrL=0.0823, TrA=0.9744, VL=0.0638, VA=0.9814, TeL=0.0496, TeA=0.9838, LR=0.00041,et=60.23,tt=306.80,Current memory:38221.00,Peak memory:3946200.00,
Epoch 6: TrL=0.0735, TrA=0.9766, VL=0.0632, VA=0.9814, TeL=0.0561, TeA=0.9828, LR=0.00041,et=62.14,tt=368.95,Current memory:78349.00,Peak memory:3982316.00,
Epoch 7: TrL=0.0693, TrA=0.9780, VL=0.0588, VA=0.9816, TeL=0.0490, TeA=0.9840, LR=0.00041,et=56.89,tt=425.84,Current memory:40132.00,Peak memory:3948825.00,
Epoch 8: TrL=0.0657, TrA=0.9792, VL=0.0597, VA=0.9816, TeL=0.0492, TeA=0.9842, LR=0.00042,et=59.82,tt=485.66,Current memory:76542.00,Peak memory:3979579.00,
Epoch 9: TrL=0.0618, TrA=0.9806, VL=0.0593, VA=0.9836, TeL=0.0528, TeA=0.9821, LR=0.00042,et=60.25,tt=545.90,Current memory:74672.00,Peak memory:3983118.00,
Epoch 10: TrL=0.0556, TrA=0.9826, VL=0.0415, VA=0.9874, TeL=0.0362, TeA=0.9882, LR=0.00042,et=56.72,tt=602.63,Current memory:33622.00,Peak memory:3948425.00,
Epoch 11: TrL=0.0558, TrA=0.9823, VL=0.0595, VA=0.9816, TeL=0.0558, TeA=0.9827, LR=0.00043,et=58.99,tt=661.62,Current memory:83180.00,Peak memory:3984442.00,
Epoch 12: TrL=0.0556, TrA=0.9822, VL=0.0587, VA=0.9827, TeL=0.0464, TeA=0.9850, LR=0.00044,et=60.13,tt=721.74,Current memory:80848.00,Peak memory:3981613.00,
Epoch 13: TrL=0.0508, TrA=0.9842, VL=0.0696, VA=0.9799, TeL=0.0576, TeA=0.9830, LR=0.00044,et=58.41,tt=780.15,Current memory:31800.00,Peak memory:3948128.00,
Epoch 14: TrL=0.0533, TrA=0.9836, VL=0.0651, VA=0.9806, TeL=0.0465, TeA=0.9862, LR=0.00045,et=59.93,tt=840.08,Current memory:69915.00,Peak memory:3984620.00,
Epoch 15: TrL=0.0479, TrA=0.9851, VL=0.0470, VA=0.9850, TeL=0.0440, TeA=0.9854, LR=0.00045,et=58.37,tt=898.46,Current memory:78684.00,Peak memory:3946668.00,
Epoch 16: TrL=0.0494, TrA=0.9841, VL=0.0470, VA=0.9858, TeL=0.0345, TeA=0.9881, LR=0.00046,et=57.97,tt=956.42,Current memory:41165.00,Peak memory:3948696.00,
Epoch 17: TrL=0.0465, TrA=0.9849, VL=0.0682, VA=0.9815, TeL=0.0546, TeA=0.9848, LR=0.00047,et=57.45,tt=1013.88,Current memory:72783.00,Peak memory:3986455.00,
Epoch 18: TrL=0.0473, TrA=0.9855, VL=0.0481, VA=0.9864, TeL=0.0388, TeA=0.9871, LR=0.00048,et=56.45,tt=1070.33,Current memory:79092.00,Peak memory:3947839.00,
Epoch 19: TrL=0.0452, TrA=0.9854, VL=0.0441, VA=0.9870, TeL=0.0382, TeA=0.9880, LR=0.00049,et=57.51,tt=1127.84,Current memory:42246.00,Peak memory:3946173.00,
Epoch 20: TrL=0.0427, TrA=0.9861, VL=0.0451, VA=0.9865, TeL=0.0383, TeA=0.9886, LR=0.00050,et=57.55,tt=1185.38,Current memory:79252.00,Peak memory:3979537.00,
Epoch 21: TrL=0.0427, TrA=0.9871, VL=0.0457, VA=0.9870, TeL=0.0405, TeA=0.9879, LR=0.00051,et=57.90,tt=1243.28,Current memory:47368.00,Peak memory:3953251.00,
Epoch 22: TrL=0.0396, TrA=0.9876, VL=0.0508, VA=0.9856, TeL=0.0437, TeA=0.9881, LR=0.00052,et=57.74,tt=1301.03,Current memory:79830.00,Peak memory:3981201.00,
Epoch 23: TrL=0.0433, TrA=0.9857, VL=0.0509, VA=0.9850, TeL=0.0550, TeA=0.9828, LR=0.00053,et=59.54,tt=1360.57,Current memory:81248.00,Peak memory:3983598.00,
Epoch 24: TrL=0.0415, TrA=0.9867, VL=0.0473, VA=0.9868, TeL=0.0416, TeA=0.9873, LR=0.00054,et=56.99,tt=1417.56,Current memory:42684.00,Peak memory:3947342.00,
Epoch 25: TrL=0.0363, TrA=0.9885, VL=0.0357, VA=0.9897, TeL=0.0323, TeA=0.9892, LR=0.00055,et=57.70,tt=1475.26,Current memory:80246.00,Peak memory:3981230.00,
Epoch 26: TrL=0.0382, TrA=0.9874, VL=0.0407, VA=0.9884, TeL=0.0372, TeA=0.9880, LR=0.00056,et=57.86,tt=1533.12,Current memory:76544.00,Peak memory:3983662.00,
Epoch 27: TrL=0.0383, TrA=0.9876, VL=0.0490, VA=0.9857, TeL=0.0474, TeA=0.9855, LR=0.00058,et=57.88,tt=1591.01,Current memory:37351.00,Peak memory:3949360.00,
Epoch 28: TrL=0.0357, TrA=0.9886, VL=0.0452, VA=0.9875, TeL=0.0388, TeA=0.9876, LR=0.00059,et=59.37,tt=1650.37,Current memory:67526.00,Peak memory:3977071.00,
Epoch 29: TrL=0.0379, TrA=0.9884, VL=0.0447, VA=0.9865, TeL=0.0345, TeA=0.9895, LR=0.00060,et=56.98,tt=1707.35,Current memory:66913.00,Peak memory:3950425.00,
Epoch 30: TrL=0.0384, TrA=0.9879, VL=0.0417, VA=0.9883, TeL=0.0396, TeA=0.9867, LR=0.00062,et=58.14,tt=1765.50,Current memory:48262.00,Peak memory:3948222.00,
Epoch 31: TrL=0.0356, TrA=0.9888, VL=0.0415, VA=0.9889, TeL=0.0443, TeA=0.9858, LR=0.00063,et=59.87,tt=1825.36,Current memory:71651.00,Peak memory:3967141.00,
Epoch 32: TrL=0.0332, TrA=0.9898, VL=0.0378, VA=0.9889, TeL=0.0385, TeA=0.9886, LR=0.00065,et=59.91,tt=1885.27,Current memory:74074.00,Peak memory:3947147.00,
Epoch 33: TrL=0.0339, TrA=0.9892, VL=0.0578, VA=0.9826, TeL=0.0579, TeA=0.9815, LR=0.00066,et=58.63,tt=1943.90,Current memory:48831.00,Peak memory:3949584.00,
Epoch 34: TrL=0.0347, TrA=0.9891, VL=0.0357, VA=0.9901, TeL=0.0330, TeA=0.9891, LR=0.00068,et=58.26,tt=2002.17,Current memory:78539.00,Peak memory:3979483.00,
Epoch 35: TrL=0.0345, TrA=0.9887, VL=0.0529, VA=0.9854, TeL=0.0489, TeA=0.9862, LR=0.00070,et=60.29,tt=2062.46,Current memory:42346.00,Peak memory:3944023.00,
Epoch 36: TrL=0.0322, TrA=0.9896, VL=0.0402, VA=0.9881, TeL=0.0390, TeA=0.9874, LR=0.00071,et=59.50,tt=2121.96,Current memory:78760.00,Peak memory:3978536.00,
Epoch 37: TrL=0.0323, TrA=0.9898, VL=0.0425, VA=0.9888, TeL=0.0442, TeA=0.9880, LR=0.00073,et=58.31,tt=2180.27,Current memory:75898.00,Peak memory:3978317.00,
Epoch 38: TrL=0.0332, TrA=0.9894, VL=0.0395, VA=0.9882, TeL=0.0382, TeA=0.9891, LR=0.00075,et=57.30,tt=2237.57,Current memory:48337.00,Peak memory:3936139.00,
Epoch 39: TrL=0.0312, TrA=0.9899, VL=0.0401, VA=0.9879, TeL=0.0406, TeA=0.9881, LR=0.00077,et=59.92,tt=2297.49,Current memory:73261.00,Peak memory:3967967.00,
Epoch 40: TrL=0.0314, TrA=0.9900, VL=0.0347, VA=0.9897, TeL=0.0341, TeA=0.9894, LR=0.00078,et=59.15,tt=2356.65,Current memory:77448.00,Peak memory:3979555.00,
Epoch 41: TrL=0.0310, TrA=0.9898, VL=0.0396, VA=0.9881, TeL=0.0408, TeA=0.9881, LR=0.00080,et=58.38,tt=2415.03,Current memory:41330.00,Peak memory:3947706.00,
Epoch 42: TrL=0.0322, TrA=0.9896, VL=0.0406, VA=0.9878, TeL=0.0427, TeA=0.9869, LR=0.00082,et=57.53,tt=2472.55,Current memory:79673.00,Peak memory:3979802.00,
Epoch 43: TrL=0.0334, TrA=0.9901, VL=0.0480, VA=0.9855, TeL=0.0453, TeA=0.9868, LR=0.00084,et=60.34,tt=2532.89,Current memory:80391.00,Peak memory:3948386.00,
Epoch 44: TrL=0.0335, TrA=0.9894, VL=0.0415, VA=0.9884, TeL=0.0354, TeA=0.9887, LR=0.00086,et=58.81,tt=2591.70,Current memory:45782.00,Peak memory:3946322.00,
Epoch 45: TrL=0.0279, TrA=0.9911, VL=0.0385, VA=0.9889, TeL=0.0367, TeA=0.9887, LR=0.00089,et=59.92,tt=2651.62,Current memory:66838.00,Peak memory:3978414.00,
Epoch 46: TrL=0.0293, TrA=0.9908, VL=0.0398, VA=0.9884, TeL=0.0344, TeA=0.9901, LR=0.00091,et=57.11,tt=2708.74,Current memory:72266.00,Peak memory:3947244.00,
Epoch 47: TrL=0.0293, TrA=0.9912, VL=0.0361, VA=0.9892, TeL=0.0247, TeA=0.9922, LR=0.00093,et=60.09,tt=2768.83,Current memory:43274.00,Peak memory:3948759.00,
Epoch 48: TrL=0.0278, TrA=0.9912, VL=0.0509, VA=0.9866, TeL=0.0454, TeA=0.9854, LR=0.00095,et=57.84,tt=2826.67,Current memory:77812.00,Peak memory:3979816.00,
Epoch 49: TrL=0.0301, TrA=0.9904, VL=0.0385, VA=0.9885, TeL=0.0359, TeA=0.9892, LR=0.00097,et=57.89,tt=2884.56,Current memory:46499.00,Peak memory:3947767.00,
Epoch 50: TrL=0.0287, TrA=0.9909, VL=0.0409, VA=0.9883, TeL=0.0369, TeA=0.9883, LR=0.00100,et=58.07,tt=2942.62,Current memory:74036.00,Peak memory:3982030.00,

...

Epoch 950: TrL=0.0004, TrA=0.9999, VL=0.0929, VA=0.9919, TeL=0.1076, TeA=0.9902, LR=0.00037,et=65.77,tt=59650.92,Current memory:76609.00,Peak memory:3978763.00,
Epoch 951: TrL=0.0000, TrA=1.0000, VL=0.0932, VA=0.9917, TeL=0.1065, TeA=0.9905, LR=0.00036,et=66.05,tt=59716.97,Current memory:41913.00,Peak memory:3948618.00,
Epoch 952: TrL=0.0000, TrA=1.0000, VL=0.0965, VA=0.9924, TeL=0.1072, TeA=0.9903, LR=0.00035,et=64.37,tt=59781.34,Current memory:72166.00,Peak memory:3980393.00,
Epoch 953: TrL=0.0001, TrA=1.0000, VL=0.1010, VA=0.9920, TeL=0.1095, TeA=0.9901, LR=0.00035,et=69.57,tt=59850.91,Current memory:71183.00,Peak memory:3934187.00,
Epoch 954: TrL=0.0001, TrA=1.0000, VL=0.0965, VA=0.9914, TeL=0.1125, TeA=0.9898, LR=0.00034,et=70.48,tt=59921.39,Current memory:45779.00,Peak memory:3952097.00,
Epoch 955: TrL=0.0001, TrA=1.0000, VL=0.1032, VA=0.9912, TeL=0.1126, TeA=0.9896, LR=0.00033,et=64.03,tt=59985.42,Current memory:79340.00,Peak memory:3980659.00,
Epoch 956: TrL=0.0001, TrA=1.0000, VL=0.0942, VA=0.9917, TeL=0.1105, TeA=0.9896, LR=0.00032,et=65.45,tt=60050.87,Current memory:79338.00,Peak memory:3952674.00,
Epoch 957: TrL=0.0001, TrA=1.0000, VL=0.0944, VA=0.9921, TeL=0.1067, TeA=0.9903, LR=0.00031,et=64.66,tt=60115.52,Current memory:46077.00,Peak memory:3950648.00,
Epoch 958: TrL=0.0001, TrA=0.9999, VL=0.0943, VA=0.9917, TeL=0.1053, TeA=0.9902, LR=0.00031,et=67.41,tt=60182.94,Current memory:70667.00,Peak memory:3978887.00,
Epoch 959: TrL=0.0000, TrA=1.0000, VL=0.0991, VA=0.9924, TeL=0.1066, TeA=0.9900, LR=0.00030,et=63.21,tt=60246.15,Current memory:41089.00,Peak memory:3950803.00,
Epoch 960: TrL=0.0000, TrA=1.0000, VL=0.0971, VA=0.9920, TeL=0.1068, TeA=0.9901, LR=0.00029,et=65.53,tt=60311.68,Current memory:70512.00,Peak memory:3968131.00,
Epoch 961: TrL=0.0001, TrA=1.0000, VL=0.0973, VA=0.9914, TeL=0.1100, TeA=0.9897, LR=0.00029,et=64.73,tt=60376.41,Current memory:78866.00,Peak memory:3981242.00,
Epoch 962: TrL=0.0000, TrA=1.0000, VL=0.1007, VA=0.9914, TeL=0.1094, TeA=0.9899, LR=0.00028,et=64.57,tt=60440.98,Current memory:46980.00,Peak memory:3946711.00,
Epoch 963: TrL=0.0001, TrA=1.0000, VL=0.0964, VA=0.9921, TeL=0.1092, TeA=0.9897, LR=0.00027,et=66.97,tt=60507.95,Current memory:76000.00,Peak memory:3979301.00,
Epoch 964: TrL=0.0000, TrA=1.0000, VL=0.1014, VA=0.9917, TeL=0.1092, TeA=0.9897, LR=0.00026,et=76.15,tt=60584.10,Current memory:63680.00,Peak memory:3977664.00,
Epoch 965: TrL=0.0001, TrA=1.0000, VL=0.1005, VA=0.9919, TeL=0.1123, TeA=0.9896, LR=0.00026,et=66.81,tt=60650.91,Current memory:45920.00,Peak memory:3948534.00,
Epoch 966: TrL=0.0001, TrA=1.0000, VL=0.1029, VA=0.9921, TeL=0.1110, TeA=0.9896, LR=0.00025,et=70.71,tt=60721.62,Current memory:82551.00,Peak memory:3985541.00,
Epoch 967: TrL=0.0001, TrA=1.0000, VL=0.0968, VA=0.9915, TeL=0.1102, TeA=0.9900, LR=0.00024,et=65.80,tt=60787.43,Current memory:67624.00,Peak memory:3953681.00,
Epoch 968: TrL=0.0000, TrA=1.0000, VL=0.0995, VA=0.9921, TeL=0.1111, TeA=0.9897, LR=0.00024,et=65.73,tt=60853.16,Current memory:28327.00,Peak memory:3942472.00,
Epoch 969: TrL=0.0000, TrA=1.0000, VL=0.0954, VA=0.9920, TeL=0.1107, TeA=0.9899, LR=0.00023,et=67.15,tt=60920.31,Current memory:68015.00,Peak memory:3977520.00,
Epoch 970: TrL=0.0000, TrA=1.0000, VL=0.0994, VA=0.9920, TeL=0.1114, TeA=0.9899, LR=0.00022,et=63.95,tt=60984.27,Current memory:61759.00,Peak memory:3934954.00,
Epoch 971: TrL=0.0000, TrA=1.0000, VL=0.1000, VA=0.9919, TeL=0.1140, TeA=0.9894, LR=0.00022,et=67.44,tt=61051.71,Current memory:30082.00,Peak memory:3934965.00,
Epoch 972: TrL=0.0000, TrA=1.0000, VL=0.1008, VA=0.9915, TeL=0.1142, TeA=0.9893, LR=0.00021,et=68.01,tt=61119.72,Current memory:72779.00,Peak memory:3980157.00,
Epoch 973: TrL=0.0000, TrA=1.0000, VL=0.1036, VA=0.9914, TeL=0.1143, TeA=0.9895, LR=0.00021,et=64.15,tt=61183.87,Current memory:99846.00,Peak memory:3949640.00,
Epoch 974: TrL=0.0000, TrA=1.0000, VL=0.1006, VA=0.9922, TeL=0.1147, TeA=0.9897, LR=0.00020,et=69.94,tt=61253.81,Current memory:80302.00,Peak memory:3983363.00,
Epoch 975: TrL=0.0000, TrA=1.0000, VL=0.1020, VA=0.9919, TeL=0.1140, TeA=0.9895, LR=0.00019,et=65.88,tt=61319.70,Current memory:76899.00,Peak memory:3981327.00,
Epoch 976: TrL=0.0001, TrA=0.9999, VL=0.0966, VA=0.9916, TeL=0.1121, TeA=0.9899, LR=0.00019,et=65.27,tt=61384.97,Current memory:46257.00,Peak memory:3951595.00,
Epoch 977: TrL=0.0000, TrA=1.0000, VL=0.1000, VA=0.9922, TeL=0.1121, TeA=0.9899, LR=0.00018,et=69.12,tt=61454.09,Current memory:80615.00,Peak memory:3980992.00,
Epoch 978: TrL=0.0000, TrA=1.0000, VL=0.0981, VA=0.9925, TeL=0.1117, TeA=0.9900, LR=0.00018,et=72.73,tt=61526.82,Current memory:62643.00,Peak memory:3978405.00,
Epoch 979: TrL=0.0000, TrA=1.0000, VL=0.1054, VA=0.9921, TeL=0.1133, TeA=0.9899, LR=0.00017,et=69.10,tt=61595.92,Current memory:49827.00,Peak memory:3951977.00,
Epoch 980: TrL=0.0001, TrA=1.0000, VL=0.1026, VA=0.9923, TeL=0.1139, TeA=0.9898, LR=0.00016,et=64.23,tt=61660.15,Current memory:79752.00,Peak memory:3977999.00,
Epoch 981: TrL=0.0001, TrA=1.0000, VL=0.1014, VA=0.9923, TeL=0.1137, TeA=0.9898, LR=0.00016,et=65.48,tt=61725.64,Current memory:72575.00,Peak memory:3950870.00,
Epoch 982: TrL=0.0000, TrA=1.0000, VL=0.0980, VA=0.9920, TeL=0.1140, TeA=0.9898, LR=0.00015,et=64.89,tt=61790.53,Current memory:35974.00,Peak memory:3947350.00,
Epoch 983: TrL=0.0002, TrA=0.9999, VL=0.0968, VA=0.9919, TeL=0.1139, TeA=0.9899, LR=0.00015,et=66.24,tt=61856.77,Current memory:75258.00,Peak memory:3982870.00,
Epoch 984: TrL=0.0002, TrA=0.9999, VL=0.0981, VA=0.9920, TeL=0.1138, TeA=0.9898, LR=0.00014,et=65.23,tt=61922.00,Current memory:72081.00,Peak memory:3948031.00,
Epoch 985: TrL=0.0000, TrA=1.0000, VL=0.1000, VA=0.9921, TeL=0.1142, TeA=0.9898, LR=0.00014,et=66.00,tt=61988.00,Current memory:44009.00,Peak memory:3936938.00,
Epoch 986: TrL=0.0000, TrA=1.0000, VL=0.1002, VA=0.9924, TeL=0.1141, TeA=0.9899, LR=0.00013,et=67.08,tt=62055.09,Current memory:71353.00,Peak memory:3979114.00,
Epoch 987: TrL=0.0000, TrA=1.0000, VL=0.1040, VA=0.9916, TeL=0.1147, TeA=0.9899, LR=0.00013,et=65.63,tt=62120.72,Current memory:43950.00,Peak memory:3945127.00,
Epoch 988: TrL=0.0001, TrA=1.0000, VL=0.1000, VA=0.9917, TeL=0.1167, TeA=0.9898, LR=0.00012,et=68.00,tt=62188.72,Current memory:81638.00,Peak memory:3982465.00,
Epoch 989: TrL=0.0002, TrA=0.9999, VL=0.1021, VA=0.9915, TeL=0.1162, TeA=0.9900, LR=0.00012,et=65.74,tt=62254.46,Current memory:74750.00,Peak memory:3984044.00,
Epoch 990: TrL=0.0001, TrA=1.0000, VL=0.1010, VA=0.9918, TeL=0.1163, TeA=0.9899, LR=0.00011,et=63.74,tt=62318.20,Current memory:49538.00,Peak memory:3948849.00,
Epoch 991: TrL=0.0001, TrA=1.0000, VL=0.1031, VA=0.9917, TeL=0.1157, TeA=0.9900, LR=0.00011,et=65.37,tt=62383.57,Current memory:82265.00,Peak memory:3982122.00,
Epoch 992: TrL=0.0001, TrA=1.0000, VL=0.1048, VA=0.9916, TeL=0.1157, TeA=0.9899, LR=0.00011,et=64.70,tt=62448.26,Current memory:78110.00,Peak memory:3978930.00,
Epoch 993: TrL=0.0000, TrA=1.0000, VL=0.1036, VA=0.9922, TeL=0.1162, TeA=0.9900, LR=0.00010,et=63.22,tt=62511.48,Current memory:40405.00,Peak memory:3948143.00,
Epoch 994: TrL=0.0000, TrA=1.0000, VL=0.1018, VA=0.9922, TeL=0.1162, TeA=0.9899, LR=0.00010,et=64.18,tt=62575.66,Current memory:80440.00,Peak memory:3978485.00,
Epoch 995: TrL=0.0002, TrA=0.9999, VL=0.1041, VA=0.9921, TeL=0.1160, TeA=0.9898, LR=0.00009,et=63.67,tt=62639.33,Current memory:67993.00,Peak memory:3946542.00,
Epoch 996: TrL=0.0000, TrA=1.0000, VL=0.1032, VA=0.9923, TeL=0.1158, TeA=0.9899, LR=0.00009,et=63.87,tt=62703.20,Current memory:42809.00,Peak memory:3936186.00,
Epoch 997: TrL=0.0001, TrA=1.0000, VL=0.1009, VA=0.9916, TeL=0.1147, TeA=0.9900, LR=0.00008,et=64.48,tt=62767.68,Current memory:63228.00,Peak memory:3979363.00,
Epoch 998: TrL=0.0001, TrA=1.0000, VL=0.0978, VA=0.9920, TeL=0.1142, TeA=0.9901, LR=0.00008,et=67.61,tt=62835.29,Current memory:78797.00,Peak memory:3947377.00,
Epoch 999: TrL=0.0000, TrA=1.0000, VL=0.1120, VA=0.9916, TeL=0.1149, TeA=0.9901, LR=0.00008,et=65.29,tt=62900.57,Current memory:39926.00,Peak memory:3947388.00,
Epoch 1000: TrL=0.0000, TrA=1.0000, VL=0.1009, VA=0.9921, TeL=0.1152, TeA=0.9901, LR=0.00007,et=65.10,tt=62965.68,Current memory:78245.00,Peak memory:3981355.00,

See the LR column.