The Learning Rate Tuner with Relative Adaptation (LRT-RA) is a machine learning technique designed to dynamically adjust the learning rate of an algorithm based on shifts in the properties of the data it processes.
The Pathway to Achieving Sustainable Computing
Keywords:
adaptive learning rate, gradient descent, optimizations, deep learning, learning rate scheduler, sustainable computingAbstract
Identifying the most suitable learning rates (LRs) in deep learning (DL) has remained a persistent challenge for the research community. Earlier techniques such as learning rate scheduling (LRS) and adaptive learning rate (ALR) have been explored to tackle this problem. Optimizers like RMSProp and Adam have added to the training complexity by bringing in extra hyperparameters, which in turn raises the cost of cross-validation experiments. These methods tend to focus on local gradient behavior, which may fall short when multiple local optima exist near the global optimum. To address these shortcomings, a new approach called the Learning Rate Tuner with Relative Adaptation (LRT-RA) is introduced. It works by dynamically updating learning rates (LRs) during training based on the global loss function curve, removing the need for costly initial LR estimation through cross-validation. This strategy lowers training costs and reduces environmental impact while boosting training efficiency. It exhibits good performance in preventing the early convergence and provides valuable information about the behaviour of the optimizer, and the relationship between the distribution of the data set and the selection of the optimal learning rate. The proposed approach gets accuracy 84.96% on the CIFAR-10 data set. Throughout training and testing, power consumption was kept as low as 0.07 kWh, CO2 emissions at 0.05, and both SO2 and NOx emissions at 0.00003 pounds.
References
1. Lavin, A.; Gray, S. Fast Algorithms for Convolutional Neural Networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 4013–4021. [CrossRef]
2. Dey, S.; Pal, R.; Biswas, S. Deep Learning Algorithms for Efficient Analysis of ECG Signals to Detect Heart Disorders. In Biosignal Processing; Asadpour, V., Karaku¸s, S., Eds.; IntechOpen: Rijeka, Croatia, 2022; Chapter 6. [CrossRef]
3. Dey, S.; Biswas, S.; Nandi, S.; Nath, S.; Das, I. Deep Greedy Network: A Tool for Medical Diagnosis on Exiguous Dataset of COVID-19. In Proceedings of the 2020 IEEE 1st International Conference for Convergence in Engineering (ICCE), Kolkata, India, 5–6 September 2020; pp. 340–344. [CrossRef]
4. Wang, Z.; Luo, Q.; Chen, H.; Zhao, J.; Yao, L.; Zhang, J.; Chu, F. A high-accuracy intelligent fault diagnosis method for aero-engine bearings with limited samples. Comput. Ind. 2024, 159–160, 104099. [CrossRef]
5. Wang, Z.; Liang, P.; Bai, R.; Liu, Y.; Zhao, J.; Yao, L.; Zhang, J.; Chu, F. Few-shot fault diagnosis for machinery using multi-scale perception multi-level feature fusion image quadrant entropy. Adv. Eng. Inform. 2025, 63, 102972. [CrossRef]
6. Yang, L.; Shami, A. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing 2020, 415, 295–316. [CrossRef]
7. Andonie, R. Hyperparameter optimization in learning systems. J. Membr. Comput. 2019, 1, 279–291. [CrossRef]
8. Bergstra, J.; Bengio, Y. Random Search for Hyper-Parameter Optimization. J. Mach. Learn. Res. 2012, 13, 281–305.
9. Bergstra, J.; Bardenet, R.; Bengio, Y.; Kégl, B. Algorithms for Hyper-Parameter Optimization. In Proceedings of the Advances in Neural Information Processing Systems, Granada, Spain, 12–14 December 2011; Shawe-Taylor, J., Zemel, R., Bartlett, P., Pereira, F., Weinberger, K., Eds.; Curran Associates, Inc.: Red Hook, NY, USA, 2011; Volume 24.
10. Miikkulainen, R.; Liang, J.; Meyerson, E.; Rawal, A.; Fink, D.; Francon, O.; Raju, B.; Shahrzad, H.; Navruzyan, A.; Duffy, N.; et al. Chapter 15—Evolving Deep Neural Networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing; Kozma, R., Alippi, C., Choe, Y., Morabito, F.C., Eds.; Academic Press: Cambridge, MA, USA, 2019; pp. 293–312. [CrossRef]
11. Nath, S.; Bandyopadhyay, R.; Biswas, S.; Sing, J.K.; Sarkar, S.K. A New Global Routing Optimization Algorithm based on Pigeon Inspired Optimization. In Proceedings of the 2020 IEEE Calcutta Conference (CALCON), Kolkata, India, 28–29 February 2020; pp. 184–188. [CrossRef]
12. Budennyy, S.; Lazarev, V.; Zakharenko, N.; Korovin, A.; Plosskaya, O.; Dimitrov, D.; Arkhipkin, V.; Oseledets, I.; Barsola, I.; Egorov, I.; et al. Eco2AI: Carbon emissions tracking of machine learning models as the first step towards sustainable AI. arXiv 2022, arXiv:2208.00406. [CrossRef]
