How do I apply stochastic gradient descent?

To apply stochastic gradient descent (SGD), you need to follow these steps, which are outlined in the context of optimizing an objective function, typically a loss function in machine learning.

Initialization

• Initialize the model parameters ( w ) randomly. This could include weights and biases for a linear regression or more complex models like neural networks235.

Setting Parameters

• Determine the number of iterations (epochs) and the learning rate ( \eta ) (often denoted as ( \alpha ) or ( lr )). The learning rate controls how quickly the parameters are updated during each iteration235.

Stochastic Gradient Descent Loop

• Repeat the following steps until convergence or the maximum number of iterations is reached:

  • Shuffle the Training Data: Randomly shuffle the training samples to introduce randomness and prevent cycles in the updates124.

  • Iterate Over Training Samples:

  • For each training sample (or a small batch of samples, known as a mini-batch), perform the following steps:

    • Compute the Gradient: Calculate the gradient of the loss function with respect to the model parameters using the current training sample or mini-batch. This is typically denoted as ( \nabla Q_i(w) ) for the ( i )-th sample123.

    • Update the Parameters: Update the model parameters by taking a step in the direction of the negative gradient, scaled by the learning rate. The update rule is: [ w := w - \eta \, \nabla Q_i(w) ] For mini-batches, the gradient is averaged over the samples in the batch124.

  • Optional: Gradient Clipping: To prevent exploding gradients, you can clip the gradients to a certain range. For example:

    gradient_weights = tf.clip_by_value(gradient_weights, -1, 1)
    gradient_bias = tf.clip_by_value(gradient_bias, -1, 1)

    This step is not mandatory but can help in stabilizing the training process2.

Convergence Check

• Periodically check for convergence by evaluating the loss function or the norm of the gradients. If the loss has stabilized or the gradient norm is below a certain tolerance, you can stop the iterations25.

Example Pseudocode

Here is a simplified pseudocode to illustrate the process:

# Initialize parameters
w = initialize_parameters()
learning_rate = set_learning_rate()
max_iterations = set_max_iterations()

# Repeat until convergence or max iterations
for epoch in range(max_iterations):
    # Shuffle training data
    shuffled_indices = shuffle(range(n_samples))
    X_shuffled = X[shuffled_indices]
    y_shuffled = y[shuffled_indices]

    # Iterate over mini-batches
    for i in range(0, n_samples, batch_size):
        X_batch = X_shuffled[i:i+batch_size]
        y_batch = y_shuffled[i:i+batch_size]

        # Compute gradient
        gradient_weights, gradient_bias = compute_gradient(X_batch, y_batch, w)

        # Update parameters
        w -= learning_rate * gradient_weights
        bias -= learning_rate * gradient_bias

    # Optional: Check convergence
    if epoch % 100 == 0:
        y_pred = predict(X, w)
        loss = mean_squared_error(y, y_pred)
        print(f"Epoch {epoch}: Loss {loss}")
        if np.linalg.norm(gradient_weights) < tolerance:
            print("Convergence reached.")
            break

Key Points

• Mini-Batches: Using mini-batches (a small subset of the training data) can offer a balance between the computational efficiency of SGD and the stability of batch gradient descent124.
• Learning Rate: The choice of learning rate is crucial. A large learning rate can lead to fast convergence but may also cause numerical instability. A common strategy is to start with a relatively large learning rate and decrease it over time134.
• Implicit Updates: For added stability, implicit stochastic gradient descent (ISGD) can be used, where the stochastic gradient is evaluated at the next iterate rather than the current one1.