ANALYSIS OF GRADIENT DECREASE ALGORITHMS
Keywords:
Gradient descent, Batch gradient, Stochastic gradient descent, Mini-Batch gradient descent.Abstract
This paper presents a comprehensive comparison of three gradient descent algorithms commonly used in machine learning and deep learning: Batch gradient descent, stochastic gradient descent, and mini-batch gradient descent. Differences between these algorithms in terms of gradient descent computational efficiency, stability, and learning dynamics are explained. Provides a clear and concise overview of each algorithm, their advantages and disadvantages, making it easy to understand their suitability for a specific problem and data set. Relevant information is provided to illustrate the difference between these algorithms.
References
H. Zaynidinov, O. Mallayev, Parallel algorithm for calculating the learning processes of an artificial neural network. AIP Conference Proceedings 2647, 050006 (2022). doi: https://doi.org/10.1063/5.0104178
Yusupov I, Nurmurodov J, Ibragimov S, Gofurjonov M, Qobilov S. “Calculation of Spectral Coefficients of Signals on the Basis of Haar by the Method of Machine Learning”, 14th International Conference, IHCI 2022, Tashkent, Uzbekistan, October 20–22, 2022, pp 547–558. https://link.springer.com/conference/ihci
Bottou, L. (2010). Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, Springer.
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747.
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT press.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction. Springer.
Shukla, P. (2019). The Gradient Descent Algorithm and Its Variants. arXiv preprint arXiv:1908.10448. doi: 10.1093/ptep/ptaa104
https://www.baeldung.com/cs/gradient-stochastic-and-mini-batch
