Information Theory
Information as surprise: entropy, bits, and the direct line from there to the loss functions used every day.
3 articles, newest first.
- Mutual information for feature selection, and where it misleads It catches relationships correlation misses, including ones you do not want. What it measures, why estimating it is hard, and why a high score can mean a leak.
- Cross-entropy, log loss and perplexity are the same idea Three names from three fields for one quantity: how surprised your model was. Where each shows up, how to read the units, and what a good value looks like.
- Entropy, or how to measure surprise Information as surprise, why bits fall out of a logarithm, and the straight line from there to the cross-entropy loss you train with and the splits a decision tree picks.