IEEE

Learning Error Refinement in Stochastic Gradient Descent-Based Latent Factor Analysis via Diversified PID Controllers

Abstract: In Big Data-based applications, high-dimensional and incomplete (HDI) data are frequently used to represent the complicated interactions among numerous nodes. A stochastic gradient descent ...
IEEE

AccSPS Learning Rate: Accelerated Convergence through Decision-Adjusted Levels for Stochastic Polyak Stepsize

Abstract: Recently, the application of stochastic gradient descent (SGD) with Polyak stepsizes has gained attention and exhibited promising performance for machine learning problems. However, when the ...
Scientific Research Publishing

Bottou, L. (2010) Large-Scale Machine Learning with Stochastic Gradient Descent. Proceedings of COMPSTAT 2010, Paris, 22-27 August 2010, 177-186.

ABSTRACT: This paper investigates the application of machine learning techniques to optimize complex spray-drying operations in manufacturing environments. Using a mixed-methods approach that combines ...
GitHub

"Gradient accumulation on training samples of variable size" example broken when using DeepSpeed

I've been stuck for a while trying to get gradient accumulation and multi-GPU training with DeepSpeed to work. I noticed that even when I keep my effective batch sizes the same, if I do one run with 2 ...