linear regression

Adam is one more optimization algorithm used in neural networks. It is based on adaptive estimates of lower-order moments. It has more hyper-parameters than classic Gradient Descent to tune externally

Good default settings for the tested machine learning problems are:

• α = 0.001, // learning rate. We have already seen this one in classic Gradient Descent.
• β₁ = 0.9,
• β₂ = 0.999
• eps = 10−8.

In this article we are going to use Scala mini-library for Deep Learning that we developed earlier in order to study basic linear regression task. We will learn model weights using perceptron model, which will be our single unit network layer that emits target value. This model will predict a target value yHat based on two trained parameters: weight and bias. Both are scalar numbers. Weights optimization is going to be based on implemented Gradient descent algorithm:

• Gradient Descent Optimization
• Training Loop

Model equation:

y = bias + weight * x