The Neuron

A neuron runs two steps. First the weighted sum: each input is multiplied by its weight, all of them are added together, and a small fixed bias is added on top. A weight sets how much its input matters, and its sign decides direction — positive weights push toward firing (green lines), negative weights push against it (orange lines). Second the squish: that raw sum is passed through an activation function that squeezes it into a clean 0-to-1 range. When the result reaches 0.5, the neuron fires. Stack thousands of these and you have a neural network.

This lesson uses the sigmoid function, which squishes the sum into a 0-to-1 range — the cleanest way to show the idea. But real networks rarely use sigmoid everywhere. The choice depends on where the neuron sits. Hidden layers, the neurons in the middle, mostly use ReLU: output 0 for any negative input, and pass positive values through unchanged. ReLU replaced sigmoid there because sigmoid's curve goes flat at the extremes, and flat means near-zero gradient — so deep networks trained on sigmoid barely learn (the vanishing gradient problem). ReLU keeps gradients flowing, which is a big reason deep learning took off. Transformer and LLM feed-forward layers use smooth ReLU cousins like GELU or SwiGLU. Output neurons are chosen to match the answer: sigmoid for a yes/no probability, softmax for picking one class out of many (this is what makes the probability bars in the Language Models lesson), or no activation at all when predicting a raw number like a price. So there is no single current neuron — hidden layers favour the ReLU family, and outputs are picked to fit the task.