Calculate information gain from entropy before and after a split, or from parent and child class counts in a binary decision tree.
- All Statistics Calculators
- Gini Index Calculator
- Shannon-Hartley Theorem Calculator
- Channel Capacity Calculator
- Shannon-Wiener Index Calculator
- Simpson’s Diversity Index Calculator
Information Gain Formula
The following formula is used to calculate the information gain:
Variables:
- IG is the information gain
- E_{before} is the entropy before the split
- E_{after} is the child entropy weighted by each child’s share of the parent observations
To calculate information gain in bits, subtract the weighted child entropy from the parent entropy, using base-2 logarithms. In From Counts mode, the two children must preserve both parent class totals.
What is Information Gain?
Information gain is a metric used to quantify the reduction in entropy or impurity in a dataset due to the application of a feature or rule. It is a key concept in machine learning and is particularly important in the construction of decision trees, where it is used to select the features that result in the largest gain (or equivalently, the largest reduction in uncertainty about the target variable).
How to Calculate Information Gain?
The following steps outline how to calculate the Information Gain:
- First, determine the entropy before the split (E_{before}).
- Next, determine the weighted child entropy after the split (E_{after}).
- Use the formula IG = E_{before} – E_{after} to calculate the Information Gain (IG).
Example Problem:
Use the following variables as an example problem to test your knowledge.
Entropy before the split (E_{before}) = 1.0 bits
Weighted entropy after the split (E_{after}) = 0.5 bits; information gain = 1.0 − 0.5 = 0.5 bits.
