How Machine Learning and Neural Networks Work

Interview answer (say this first). Machine learning fits a function to data by adjusting parameters to minimise a loss. A neural network is a flexible function built from layers of weighted sums and nonlinear activations, and it is trained by backpropagation, which computes how each parameter should change. Deep learning is simply a neural network with many layers.

Why this exists

Traditional programming asks you to write the rules:

def is_spam(subject):
    if "viagra" in subject.lower():
        return True
    if "free money" in subject.lower():
        return True
    return False

This fails immediately. Spammers write “v1agra”, “freee money”, or something you have never seen. To keep up, you would need an endless list of hand-written rules, each one a guess about the future.

The insight behind machine learning is different: instead of writing the rules, you show the computer examples and let it find the pattern.

examples of spam + examples of not-spam  →  a program that classifies new messages

Nobody writes what “spam” means. The model learns a mapping from the examples, and it can generalise to messages it has never seen.

This matters for the rest of the book because a large language model is exactly this idea at enormous scale. An LLM is a function that maps a sequence of text to a prediction of the next token, and every capability it appears to have — reasoning, coding, translating — comes from fitting that function to a very large amount of text.

Start from zero

Every word below is used for the rest of the phase, so pin them down now.

The most important contrast is training vs inference. Training changes the parameters and is slow and expensive. Inference uses the fixed parameters and is what you pay for on every request. Confusing the two causes a lot of bad decisions — for example, assuming you can “fine-tune per user request”.

The core idea

Picture an old radio with many dials. You cannot see the circuit inside, but you can hear the output. Your job is to turn the dials until the sound matches what you want.

• The dials are the parameters.
• The sound is the model’s output.
• The difference from the desired sound is the loss.
• Turning a dial to reduce the difference is gradient descent.

Training is this, done automatically, millions or billions of times. Crucially, the gradient tells you which way each dial should turn, and by roughly how much, so you are not guessing randomly.

Now the second half of the idea: a neural network is built by stacking simple operations.

output = activation( input @ weights + bias )

A weighted sum plus a bias is just a linear function: it can only draw straight lines. Stacking several of them without any nonlinearity would still be one big linear function, no more powerful than a single one. The activation function is what breaks this. It bends the line, and enough bends can approximate almost any smooth function.

flowchart LR
    A["input<br/>features"] --> B["linear<br/>input·W1+b1"]
    B --> C["activation<br/>(nonlinearity)"]
    C --> D["linear<br/>·W2+b2"]
    D --> E["output<br/>prediction"]
    E --> F["loss vs label"]
    F -->|"backpropagate gradients"| B

That diagram — forward to a prediction, measure loss, send gradients backward, update — is the entire training loop. Everything else in this phase is a detail of one box in it.

How it works

1. Initialise parameters randomly. Starting at zero often prevents learning because every unit computes the same thing. Random, small values break the symmetry. (Zero init is harmless for a single linear unit like Example 1 — there is only one unit, so there is no symmetry to break. It is multiple hidden units that all learn the same thing from a zero start.)
2. Forward pass. Feed a batch of inputs through the network to get predictions.
3. Compute the loss. Compare predictions with labels using a formula such as mean squared error or cross-entropy (cross-entropy measures how much probability the model assigned to the correct class; lower is better).
4. Backward pass (backpropagation). Using the chain rule from calculus, work backward from the loss to find the gradient of the loss with respect to every parameter.
5. Update the parameters. Move each parameter a small step against its gradient: param = param - learning_rate * gradient.
6. Repeat over batches and epochs. Each cycle reduces the loss a little, and the parameters drift toward values that fit the data.
7. Stop and evaluate. Check performance on data the model never trained on. This is the only honest measure.
8. Inference. Freeze the parameters and run forward passes only. No gradients, no updates, much cheaper.

Note:

Why the gradient works. Backpropagation is not a special trick; it is the chain rule applied systematically. The loss depends on the output, the output depends on the last layer, and so on backward. Multiplying the local derivatives along the chain gives the gradient for every parameter in one efficient pass, which is far cheaper than nudging each parameter to measure its effect.

The syntax you will use

You rarely write this by hand in production, but writing it once makes every framework obvious.

A model is a function with parameters. Here is linear regression: y = w*x + b.

import numpy as np

def predict(x, w, b):
    return w * x + b

def mse_loss(y_pred, y_true):
    return np.mean((y_pred - y_true) ** 2)

The training loop: forward, loss, gradient, update.

# the data: 200 points from y = 3x - 2, plus noise (same setup as Example 1)
rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 200)
y = 3 * x - 2 + rng.normal(0, 0.1, 200)

w, b = 0.0, 0.0
learning_rate = 0.1

for epoch in range(200):
    y_pred = predict(x, w, b)
    loss = mse_loss(y_pred, y)
    # gradients of the loss with respect to w and b
    error = y_pred - y
    dw = np.mean(2 * error * x)
    db = np.mean(2 * error)
    w -= learning_rate * dw
    b -= learning_rate * db

A hidden layer with an activation. This is the smallest possible “deep” network: input → hidden → output.

def forward(x, W1, b1, W2, b2):
    hidden = np.tanh(x @ W1 + b1)   # tanh is the nonlinearity
    return hidden @ W2 + b2

Activations you will see. The choice matters, but the concept does not change.

Why X @ W is everywhere. In a layer, X is a batch of inputs and W is the weight matrix. X @ W computes the weighted sum for every input and every unit at once. A matrix multiplication is just many weighted sums batched together — which is exactly why GPUs, built for matrix maths, dominate AI.

Examples: simple to real

Example 1 — learning a straight line. Given noisy data from y = 3x - 2, gradient descent recovers the parameters. The exact setup: seed 0, 200 inputs drawn uniformly from [-1, 1], Gaussian noise with standard deviation 0.1, learning rate 0.1, and 200 epochs.

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(-1, 1, 200)                 # 200 inputs in [-1, 1]
y = 3 * x - 2 + rng.normal(0, 0.1, 200)     # the true rule, plus noise

w, b = 0.0, 0.0
learning_rate = 0.1

for epoch in range(200):
    y_pred = w * x + b
    loss = np.mean((y_pred - y) ** 2)
    error = y_pred - y
    w -= learning_rate * np.mean(2 * error * x)
    b -= learning_rate * np.mean(2 * error)

# after 200 epochs, measured:
# learned w = 2.987, b = -2.008   (true values 3.0 and -2.0)
# loss fell from 6.389 to 0.0104

Nothing was told the rule. The loop discovered w ≈ 3 and b ≈ -2 from the data alone.

Example 2 — checking the gradient. Backpropagation is easy to get subtly wrong, so the standard practice is to compare it against a numerical estimate: nudge a parameter slightly and see how the loss changes.

# numeric gradient: (loss(p + eps) - loss(p - eps)) / (2 * eps)
# analytic gradient from backpropagation
# measured max difference: 8.6e-11 — they agree

If the two disagree, the backpropagation implementation has a bug. This check is used in real frameworks.

Example 3 — why nonlinearity matters. XOR is the classic test: its output is 1 when the inputs differ, and no straight line can separate those cases.

A linear model on XOR:      predictions [0, 0, 0, 0]   → cannot fit
A 2-input, 8-hidden, 1-output tanh network:
                            predictions [0, 1, 1, 0]   → fits

Both models use the same training loop. The only difference is the nonlinear hidden layer, which lets the network bend the decision boundary.

Example 4 — the danger of memorising. A model with enough parameters can fit the training data perfectly and still fail on new data.

training loss:   0.0001   (looks excellent)
validation loss: 1.85     (much worse)

That gap is overfitting. It is why you always hold back data the model never sees, and why “our model gets 99% on the training set” is not a result.

Example 5 — from this to an LLM. Replace the input with a sequence of tokens and the label with the next token:

input:  "The capital of France is"  →  token ids [791, 6864, 315, 9822, 374]
label:  "Paris"                      →  the token id for "Paris" (for example, 6342)

The model never sees the string “Paris”: the label is an integer token id, and the output is a probability over the whole vocabulary. Train that across a large text corpus and you get a language model. It is the same forward → loss → backward → update loop from Example 1, scaled up by many orders of magnitude.

In production

• Data quality beats model size. A clean, well-labelled dataset usually helps more than a bigger network. Most real ML failures are data failures, not maths failures.
• Training and inference are separate budgets. Training is a batch job on accelerators; inference is a latency-sensitive service. Optimise them differently and never run training inside a request.
• Always hold out validation and test data. Training loss only tells you the model memorised. Use a validation set to choose settings and a test set you touch once.
• Loss is not the product metric. A lower loss does not automatically mean a better experience. For LLMs, loss is a proxy; usefulness, correctness, latency, and cost are the real measures.
• Seeds and versions matter for reproducibility. Record the random seed, data version, and code version. Otherwise you cannot reproduce a result, which makes debugging nearly impossible.
• Overfitting and underfitting are diagnosed from the gap. Training and validation loss both high means underfitting; training low but validation high means overfitting. More data, regularisation (any penalty or constraint that discourages memorising the training set, such as weight decay or dropout), or a smaller model are the usual fixes.
• Watch for distribution shift. A model is only valid on data resembling its training data. Real traffic drifts, and performance quietly decays; monitor inputs and outputs over time.
• The learning rate is the most important hyperparameter. Too high and the loss explodes; too low and training never converges. Schedules that reduce it over time are standard.
• GPUs exist because of matrix multiplication. Almost all of a network’s compute is X @ W. Batching examples together keeps the hardware busy, which is why batching is central to both training and inference.
• An LLM is this loop at scale. Pretraining minimises next-token loss; fine-tuning continues the same loop on narrower data. Understanding this removes the mystery from phrases like “the model learned”.

Interview questions

1. What is machine learning, and how is it different from normal programming?

Answer. Normal programming has you write the rules explicitly. Machine learning instead fits a function to examples: you define a model with adjustable parameters, measure error with a loss, and adjust the parameters to reduce that error. The rules are discovered from data rather than written by hand.

Follow-up: “When would you still write rules?” When the logic is exact, stable, and cheap to express — for example, input validation or a business rule. Use ML where the pattern is fuzzy, high-dimensional, or changes over time.

Trap. Saying machine learning “understands” the data or “learns the concept”. It fits a mathematical function; we interpret the result.

2. What is a loss function, and why minimise it?

Answer. A loss is a single number that measures how wrong the model is across examples. Training is only possible because a smaller loss gives a concrete target; the gradient of that loss tells each parameter which way to move. Mean squared error and cross-entropy are the two most common.

Follow-up: “Can a lower loss be bad?” Yes. If the loss does not match the real goal, or if the model overfits, a lower training loss can mean a worse product. Loss is a proxy, not the objective itself.

Trap. Believing the loss is the product metric. For an LLM, token-level loss is a training signal; correctness and usefulness are what you measure in production.

3. What is backpropagation?

Answer. It is an efficient way to compute the gradient of the loss with respect to every parameter, using the chain rule from calculus. The forward pass caches intermediate values; the backward pass multiplies local derivatives from the output back to each parameter, so all gradients are obtained in roughly the cost of one extra forward pass.

Follow-up: “Why not just nudge each parameter?” That would require two full forward passes per parameter — billions of times too slow. Backpropagation computes them all at once.

Trap. Calling backpropagation a learning algorithm. It only computes gradients; the actual update step is gradient descent.

4. What are the gradient and the learning rate?

Answer. The gradient is the direction and rate at which the loss changes as a parameter changes. The learning rate is the size of the step taken against the gradient. Too high and training diverges; too low and it is impractically slow.

Follow-up: “What other optimisers exist?” SGD with momentum, RMSProp, and Adam. They adapt the effective step size per parameter and are the default in practice, but they are refinements of the same gradient idea.

Trap. Thinking a bigger learning rate always learns faster. Past a threshold it overshoots the minimum and the loss rises or becomes NaN.

5. Why do neural networks need nonlinear activation functions?

Answer. Because stacking linear layers without a nonlinearity still produces one linear function. A composition of linear maps is itself linear, so added depth would add no expressive power. The activation introduces curvature, and with enough units and layers a network can approximate very complex functions.

Follow-up: “Then why is a linear model ever useful?” When the relationship really is close to linear, a linear model is simpler, faster, and less prone to overfitting. Nonlinearity is power, and power can overfit.

Trap. Saying the activation is what makes the network “learn”. It adds expressiveness; learning comes from the training loop.

6. What is overfitting, and how do you detect and reduce it?

Answer. Overfitting is fitting the noise in the training data instead of the underlying pattern. You detect it by comparing training and validation loss: a large gap where training is much lower is the signature. Reduce it with more data, regularisation (weight decay, dropout), early stopping, or a smaller model.

Follow-up: “What is underfitting?” Both training and validation loss stay high: the model is too simple or undertrained. The fix is a larger model, longer training, or better features.

Trap. Reporting training accuracy as the model’s quality. Only held-out data gives an honest estimate.

7. What is the difference between training and inference?

Answer. Training adjusts parameters: it needs labels, gradients, and backpropagation, and it is expensive. Inference uses the frozen parameters for a forward pass only, with no gradients or updates, and it is what serves user requests. In an LLM, inference is the token-generation loop you pay for on every call.

Follow-up: “Why not fine-tune per request?” Training is far too slow and expensive to run inside a request, and it would change the shared model for everyone. Per-user adaptation belongs in the prompt, the retrieval context, or a small adapter trained offline.

Trap. Assuming fine-tuning is a runtime operation. It is an offline batch process; the runtime only does inference.

8. Why is it called “deep” learning, and why did it start working?

Answer. “Deep” means many stacked layers, which let the network build simple features into complex ones. Deep networks existed for decades but did not work well until three things came together: far more data, much faster hardware (GPUs), and better training techniques such as improved activations and regularisation. The mathematics did not change; the conditions did.

Follow-up: “Does that mean ideas are unimportant?” No. Architecture choices such as attention determine what a network can learn efficiently. But scale was necessary for deep networks to outperform hand-engineered features.

Trap. Attributing the rise of deep learning to a single breakthrough. It was the combination of data, compute, and training methods that made depth practical.

Remember this

• Machine learning fits a function from data by minimising a loss; it does not follow written rules.
• The loop is forward → loss → backward (gradients) → update, repeated over batches.
• Nonlinear activations are what let stacked layers express more than one linear function.
• Training changes parameters; inference does not. Different cost, different strategy.
• Held-out data is the only honest measure. Training loss alone means nothing.