Information gain (decision tree)

<h2 id="general-definition">General definition</h2>
In general terms, the <a href="/facts/Expectation_value/szGNS7Dx">expected</a> information gain is the reduction in <a href="/facts/Information_entropy/NLg4NLvt">information entropy</a> Η from a prior state to a state that takes some information as given:

I
        G
        (
        T
        ,
        a
        )
        =
        
          H
        
        
          (
          T
          )
        
        −
        
          H
        
        
          (
          T
          
            |
          
          a
          )
        
        ,
      
    
    {\displaystyle IG(T,a)=\mathrm {H} {(T)}-\mathrm {H} {(T|a)},}

where 
 
 
 
 
 H
 
 
 (
 T
 
 |
 
 a
 )
 
 
 
 {\displaystyle \mathrm {H} {(T|a)}}
 
 is the <a href="/facts/Conditional_entropy/gHzPezGB">conditional entropy</a> of 
 
 
 
 T
 
 
 {\displaystyle T}
 
 given the value of <a href="/facts/Feature_(machine_learning)/nzAhYxfu">attribute</a> 
 
 
 
 a
 
 
 {\displaystyle a}
 
.
This is intuitively plausible when interpreting entropy Η as a measure of uncertainty of a random variable 
 
 
 
 T
 
 
 {\displaystyle T}
 
: by learning (or assuming) 
 
 
 
 a
 
 
 {\displaystyle a}
 
 about 
 
 
 
 T
 
 
 {\displaystyle T}
 
, our uncertainty about 
 
 
 
 T
 
 
 {\displaystyle T}
 
 is reduced (i.e. 
 
 
 
 I
 G
 (
 T
 ,
 a
 )
 
 
 {\displaystyle IG(T,a)}
 
 is positive), unless of course 
 
 
 
 T
 
 
 {\displaystyle T}
 
 is independent of 
 
 
 
 a
 
 
 {\displaystyle a}
 
, in which case 
 
 
 
 
 H
 
 (
 T
 
 |
 
 a
 )
 =
 
 H
 
 (
 T
 )
 
 
 {\displaystyle \mathrm {H} (T|a)=\mathrm {H} (T)}
 
, meaning 
 
 
 
 I
 G
 (
 T
 ,
 a
 )
 =
 0
 
 
 {\displaystyle IG(T,a)=0}
 
.

<h2 id="formal-definition">Formal definition</h2>
Let T denote a <a href="/facts/Training_set/V3mMQb2G">set of training examples</a>, each of the form 
 
 
 
 (
 
 
 x
 
 
 ,
 y
 )
 =
 (
 
 x
 
 1
 
 
 ,
 
 x
 
 2
 
 
 ,
 
 x
 
 3
 
 
 ,
 .
 .
 .
 ,
 
 x
 
 k
 
 
 ,
 y
 )
 
 
 {\displaystyle ({\textbf {x}},y)=(x_{1},x_{2},x_{3},...,x_{k},y)}
 
 where 
 
 
 
 
 x
 
 a
 
 
 ∈
 
 v
 a
 l
 s
 
 (
 a
 )
 
 
 {\displaystyle x_{a}\in \mathrm {vals} (a)}
 
 is the value of the 
 
 
 
 
 a
 
 th
 
 
 
 
 {\displaystyle a^{\text{th}}}
 
 attribute or <a href="/facts/Feature_(machine_learning)/nzAhYxfu">feature</a> of <a href="/facts/Feature_vector/nzAhYxfu">example</a> 
 
 
 
 
 
 x
 
 
 
 
 {\displaystyle {\textbf {x}}}
 
 and y is the corresponding class label. The information gain for an attribute a is defined in terms of <a href="/facts/Shannon_entropy/NLg4NLvt">Shannon entropy</a> 
 
 
 
 
 H
 
 (
 −
 )
 
 
 {\displaystyle \mathrm {H} (-)}
 
 as follows. For a value v taken by attribute a, let 
 
 
 
 
 S
 
 a
 
 
 
 (
 v
 )
 
 =
 {
 
 
 x
 
 
 ∈
 T
 
 |
 
 
 x
 
 a
 
 
 =
 v
 }
 
 
 {\displaystyle S_{a}{(v)}=\{{\textbf {x}}\in T|x_{a}=v\}}
 
be defined as the <a href="/facts/Set_(mathematics)/BfucfHMq">set</a> of training inputs of T for which attribute a is equal to v. Then the information gain of T for attribute a is the difference between the a priori Shannon entropy 
 
 
 
 
 H
 
 (
 T
 )
 
 
 {\displaystyle \mathrm {H} (T)}
 
 of the training set and the <a href="/facts/Conditional_entropy/gHzPezGB">conditional entropy</a> 
 
 
 
 
 H
 
 
 (
 T
 
 |
 
 a
 )
 
 
 
 {\displaystyle \mathrm {H} {(T|a)}}
 
.

 
 
 
 
 H
 
 (
 T
 
 |
 
 a
 )
 =
 
 ∑
 
 v
 ∈
 
 v
 a
 l
 s
 
 (
 a
 )
 
 
 
 
 
 
 
 |
 
 
 S
 
 a
 
 
 
 (
 v
 )
 
 
 |
 
 
 
 
 |
 
 T
 
 |
 
 
 
 
 ⋅
 
 H
 
 
 (
 
 
 S
 
 a
 
 
 
 
 (
 v
 )
 
 
 
 )
 
 
 .
 
 
 {\displaystyle \mathrm {H} (T|a)=\sum _{v\in \mathrm {vals} (a)}{{\frac {|S_{a}{(v)}|}{|T|}}\cdot \mathrm {H} \left(S_{a}{\left(v\right)}\right)}.}

I
        G
        (
        T
        ,
        a
        )
        =
        
          H
        
        (
        T
        )
        −
        
          H
        
        (
        T
        
          |
        
        a
        )
      
    
    {\displaystyle IG(T,a)=\mathrm {H} (T)-\mathrm {H} (T|a)}

The <a href="/facts/Mutual_information/HIUvsjvV">mutual information</a> is equal to the total entropy for an attribute if for each of the attribute values a unique <a href="/facts/Statistical_classification/jXXHRkXR">classification</a> can be made for the result attribute. In this case, the relative entropies subtracted from the total entropy are 0. In particular, the values 
 
 
 
 v
 ∈
 v
 a
 l
 s
 (
 a
 )
 
 
 {\displaystyle v\in vals(a)}
 
 defines a <a href="/facts/Partition_of_a_set/VeP9g8CA">partition</a> of the training set data T into <a href="/facts/Disjoint_sets/nLr8XRVd">mutually exclusive</a> and all-inclusive <a href="/facts/Subset/SJGERmbA">subsets</a>, inducing a <a href="/facts/Categorical_distribution/xpc6RWM2">categorical probability distribution</a> 
 
 
 
 
 P
 
 a
 
 
 
 (
 v
 )
 
 
 
 {\textstyle P_{a}{(v)}}
 
 on the values 
 
 
 
 v
 ∈
 v
 a
 l
 s
 (
 a
 )
 
 
 {\textstyle v\in vals(a)}
 
 of attribute a. The distribution is given 
 
 
 
 
 P
 
 a
 
 
 
 (
 v
 )
 
 :=
 
 
 
 
 |
 
 
 S
 
 a
 
 
 
 (
 v
 )
 
 
 |
 
 
 
 
 |
 
 T
 
 |
 
 
 
 
 
 
 {\textstyle P_{a}{(v)}:={\frac {|S_{a}{(v)}|}{|T|}}}
 
. In this representation, the information gain of T given a can be defined as the difference between the unconditional Shannon entropy of T and the expected entropy of T conditioned on a, where the <a href="/facts/Expectation_value/szGNS7Dx">expectation value</a> is taken with respect to the induced distribution on the values of a.
 
 
 
 
 
 
 
 I
 G
 (
 T
 ,
 a
 )
 
 
 
 =
 
 H
 
 (
 T
 )
 −
 
 ∑
 
 v
 ∈
 
 v
 a
 l
 s
 
 (
 a
 )
 
 
 
 
 P
 
 a
 
 
 
 (
 v
 )
 
 
 H
 
 
 (
 
 
 S
 
 a
 
 
 
 (
 v
 )
 
 
 )
 
 
 
 
 
 
 
 
 =
 
 H
 
 (
 T
 )
 −
 
 
 E
 
 
 
 P
 
 a
 
 
 
 
 
 
 [
 
 
 H
 
 
 (
 
 S
 
 a
 
 
 
 (
 v
 )
 
 )
 
 
 ]
 
 
 
 
 
 
 
 
 =
 
 H
 
 (
 T
 )
 −
 
 H
 
 
 (
 T
 
 |
 
 a
 )
 
 .
 
 
 
 
 
 
 {\displaystyle {\begin{alignedat}{2}IG(T,a)&=\mathrm {H} (T)-\sum _{v\in \mathrm {vals} (a)}{P_{a}{(v)}\mathrm {H} \left(S_{a}{(v)}\right)}\\&=\mathrm {H} (T)-\mathbb {E} _{P_{a}}{\left[\mathrm {H} {(S_{a}{(v)})}\right]}\\&=\mathrm {H} (T)-\mathrm {H} {(T|a)}.\end{alignedat}}}

<h2 id="another-take-on-information-gain-with-example">Another Take on Information Gain, with Example</h2>
For a better understanding of information gain, let us break it down. As we know, information gain is the reduction in information entropy, what is entropy? Basically, entropy is the measure of impurity or uncertainty in a group of observations. In engineering applications, information is <a href="/facts/Analogy/LYcGaTdm">analogous</a> to signal, and entropy is analogous to noise. It determines how a decision tree chooses to split data.<a class="footnote-ref" id="fnref:1" href="#fn:1">1</a> The leftmost figure below is very impure and has high entropy corresponding to higher disorder and lower information value. As we go to the right, the entropy decreases, and the information value increases.

Now, it is clear that information gain is the measure of how much information a feature provides about a class. Let's visualize information gain in a decision tree as shown in the right:
The node t is the parent node, and the sub-nodes tL and tR are child nodes. In this case, the parent node t has a collection of cancer and non-cancer samples denoted as C and NC respectively. We can use information gain to determine how good the splitting of nodes is in a decision tree. In terms of entropy, information gain is defined as:

<table><tbody><tr><td>Gain = (Entropy of the parent node) – (average entropy of the child nodes)<a class="footnote-ref" id="fnref:2" href="#fn:2">2</a></td> <td><table><tbody><tr><td rowspan="2"></td> <td></td> <td rowspan="2"></td></tr> <tr><td></td></tr></tbody></table></td> <td>i</td></tr></tbody></table>
To understand this idea, let's start by an example in which we create a simple dataset and want to see if gene <a href="/facts/Mutation/Ee4NnWbY">mutations</a> could be related to patients with cancer. Given four different gene mutations, as well as seven samples, the <a href="/facts/Training%2c_validation%2c_and_test_sets/V3mMQb2G">training set</a> for a decision can be created as follows:

<table><tbody><tr><th>Samples</th><th>Mutation 1</th><th>Mutation 2</th><th>Mutation 3</th><th>Mutation 4</th></tr><tr><td>C1</td><td>1</td><td>1</td><td>1</td><td>0</td></tr><tr><td>C2</td><td>1</td><td>1</td><td>0</td><td>1</td></tr><tr><td>C3</td><td>1</td><td>0</td><td>1</td><td>1</td></tr><tr><td>C4</td><td>0</td><td>1</td><td>1</td><td>0</td></tr><tr><td>NC1</td><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>NC2</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>NC3</td><td>1</td><td>1</td><td>0</td><td>0</td></tr></tbody></table>
In this dataset, a 1 means the sample has the mutation (True), while a 0 means the sample does not (False). A sample with C denotes that it has been confirmed to be cancerous, while NC means it is non-cancerous. Using this data, a decision tree can be created with information gain used to determine the candidate splits for each node.

For the next step, the entropy at parent node t of the above simple decision tree is computed as:<blockquote>H(t) = −[pC,t log2(pC,t) + pNC,t log2(pNC,t)]<a class="footnote-ref" id="fnref:3" href="#fn:3">3</a>
where,
probability of selecting a class ‘C’ sample at node t, pC,t = n(t, C) / n(t),
probability of selecting a class ‘NC’ sample at node t, pNC,t = n(t, NC) / n(t),

n(t), n(t, C), and n(t, NC) are the number of total samples, ‘C’ samples and ‘NC’ samples at node t respectively.</blockquote>Using this with the example training set, the process for finding information gain beginning with 
 
 
 
 
 H
 
 
 (
 t
 )
 
 
 
 {\displaystyle \mathrm {H} {(t)}}
 
 for Mutation 1 is as follows:
pC, t = 4/7
pNC, t = 3/7

H
 
 
 (
 t
 )
 
 
 
 {\displaystyle \mathrm {H} {(t)}}
 
 = −(4/7 × log2(4/7) + 3/7 × log2(3/7)) = 0.985
Note: 
 
 
 
 
 H
 
 
 (
 t
 )
 
 
 
 {\displaystyle \mathrm {H} {(t)}}
 
 will be the same for all mutations at the root.
The relatively high value of entropy 
 
 
 
 
 H
 
 
 (
 t
 )
 
 =
 0.985
 
 
 {\displaystyle \mathrm {H} {(t)}=0.985}
 
 (1 is the optimal value) suggests that the root node is highly impure and the constituents of the input at the root node would look like the leftmost figure in the above Entropy Diagram. However, such a set of data is good for learning the attributes of the mutations used to split the node. At a certain node, when the homogeneity of the constituents of the input occurs (as shown in the rightmost figure in the above Entropy Diagram), the dataset would no longer be good for learning.

Moving on, the entropy at left and right child nodes of the above decision tree is computed using the formulae:<blockquote>H(tL) = −[pC,L log2(pC,L) + pNC,L log2(pNC,L)]<a class="footnote-ref" id="fnref:4" href="#fn:4">4</a></blockquote><blockquote>H(tR) = −[pC,R log2(pC,R) + pNC,R log2(pNC,R)]<a class="footnote-ref" id="fnref:5" href="#fn:5">5</a></blockquote>where,
probability of selecting a class ‘C’ sample at the left child node, pC,L = n(tL, C) / n(tL),
probability of selecting a class ‘NC’ sample at the left child node, pNC,L = n(tL, NC) / n(tL),
probability of selecting a class ‘C’ sample at the right child node, pC,R = n(tR, C) / n(tR),
probability of selecting a class ‘NC’ sample at the right child node, pNC,R = n(tR, NC) / n(tR),
n(tL), n(tL, C), and n(tL, NC) are the total number of samples, ‘C’ samples and ‘NC’ samples at the left child node respectively,
n(tR), n(tR, C), and n(tR, NC) are the total number of samples, ‘C’ samples and ‘NC’ samples at the right child node respectively.
Using these formulae, the H(tL) and H(tR) for Mutation 1 is shown below:

H(tL) = −(3/4 × log2(3/4) + 1/4 × log2(1/4)) = 0.811
H(tR) = −(1/3 × log2(1/3) + 2/3 × log2(2/3)) = 0.918
Following this, average entropy of the child nodes due to the split at node t of the above decision tree is computed as:<blockquote>H(s,t) = PLH(tL) + PRH(tR)
where,
probability of samples at the left child, PL = n(tL) / n(t),

probability of samples at the right child, PR = n(tR) / n(t),</blockquote>Finally, H(s,t) along with PL and PR for Mutation 1 is as follows:
PL = 4/7
PR = 3/7
H(s, t) = (4/7 × 0.811) + (3/7 × 0.918) = 0.857
Thus, by definition from equation (i):<blockquote>(Information gain) = H(t) - H(s,t)</blockquote>
After all the steps, gain(s), where s is a candidate split for the example is:

gain(s) = 0.985 – 0.857 = 0.128

Using this same set of formulae with the other three mutations leads to a table of the candidate splits, ranked by their information gain:

<table><tbody><tr><th>Mutation</th><th>Gain(s)</th></tr><tr><td>3</td><td>0.522</td></tr><tr><td>4</td><td>0.292</td></tr><tr><td>1</td><td>0.128</td></tr><tr><td>2</td><td>0.006</td></tr></tbody></table>
The mutation that provides the most useful information would be Mutation 3, so that will be used to split the root node of the decision tree. The root can be split and all the samples can be passed though and appended to the child nodes. A tree describing the split is shown on the left.
The samples that are on the left node of the tree would be classified as cancerous by the tree, while those on the right would be non-cancerous. This tree is relatively accurate at classifying the samples that were used to build it (which is a case of <a href="/facts/Overfitting/5xnFLcMg">overfitting</a>), but it would still classify sample C2 incorrectly. To remedy this, the tree can be split again at the child nodes to possibly achieve something even more accurate.
To split the right node, information gain must again be calculated for all the possible candidate splits that were not used for previous nodes. So, the only options this time are Mutations 1, 2, and 4.
Note: 
 
 
 
 
 H
 
 
 (
 t
 )
 
 
 
 {\displaystyle \mathrm {H} {(t)}}
 
 is different this time around since there are only four samples at the right child.

PC, t = 1/4
PNC, t = 3/4

H
        
        
          (
          t
          )
        
      
    
    {\displaystyle \mathrm {H} {(t)}}
  
 = -(1/4 × log2(1/4) + 3/4 × log2(3/4)) = 0.811

From this new 
 
 
 
 
 H
 
 
 (
 t
 )
 
 
 
 {\displaystyle \mathrm {H} {(t)}}
 
, the candidate splits can be calculated using the same formulae as the root node:

<table><tbody><tr><td>Mutation</td><td>Gain(s)</td></tr><tr><td>4</td><td>0.811</td></tr><tr><td>1</td><td>0.311</td></tr><tr><td>2</td><td>0.123</td></tr></tbody></table>
Thus, the right child will be split with Mutation 4. All the samples that have the mutation will be passed to the left child and the ones that lack it will be passed to the right child.
To split the left node, the process would be the same, except there would only be 3 samples to check. Sometimes a node may not need to be split at all if it is a <a href="/facts/Hereditary_set/edFwmXa8">pure set</a>, where all samples at the node are just cancerous or non-cancerous. Splitting the node may lead to the tree being more inaccurate and in this case it will not be split.

The tree would now achieve 100% accuracy if the samples that were used to build it are tested. This isn't a good idea, however, since the tree would overfit the data. The best course of action is to try testing the tree on other samples, of which are not part of the original set. Two outside samples are below:
<table><tbody><tr><th>Samples</th><th>Mutation 1</th><th>Mutation 2</th><th>Mutation 3</th><th>Mutation 4</th></tr><tr><td>NC10</td><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>C15</td><td>1</td><td>1</td><td>0</td><td>0</td></tr></tbody></table>
By following the tree, NC10 was classified correctly, but C15 was classified as NC. For other samples, this tree would not be 100% accurate anymore. It could be possible to improve this though, with options such as increasing the depth of the tree or increasing the size of the training set.

<h2 id="advantages">Advantages</h2>
Information gain is the basic criterion to decide whether a feature should be used to split a node or not. The feature with the <a href="/facts/Optimal/oRn8Iv5I">optimal</a> split i.e., the highest value of information gain at a node of a decision tree is used as the feature for splitting the node.
The concept of information gain function falls under the <a href="/facts/C4.5_algorithm/GGsxBTtz">C4.5 algorithm</a> for generating the decision trees and selecting the optimal split for a decision tree node.<a class="footnote-ref" id="fnref:6" href="#fn:6">6</a> Some of its advantages include:

<ul><li>It can work with both <a href="/facts/Continuous_or_discrete_variable/TWl5feEw">continuous and discrete</a> variables.</li>
<li>Due to the factor –[p ∗ log(p)] in the entropy definition as given above, leaf nodes with a small number of instances are assigned less weight and it favors dividing rest of the data into bigger but homogeneous groups. And thus, as we dig deeper into the depth of the tree, the dataset becomes more homogeneous. This approach is usually more stable and chooses the most impactful features on the nodes.<a class="footnote-ref" id="fnref:7" href="#fn:7">7</a></li></ul>
<h2 id="drawbacks-and-solutions">Drawbacks and Solutions</h2>
Although information gain is usually a good measure for deciding the <a href="/facts/Relevance/AYaqXDTr">relevance</a> of an attribute, it is not perfect. A notable problem occurs when information gain is applied to attributes that can take on a large number of distinct values. For example, suppose that one is building a decision tree for some data describing the customers of a business. Information gain is often used to decide which of the attributes are the most relevant, so they can be tested near the root of the tree. One of the input attributes might be the customer's membership number, if they are a member of the business's membership program. This attribute has a high mutual information, because it uniquely identifies each customer, but we do not want to include it in the decision tree. Deciding how to treat a customer based on their membership number is unlikely to generalize to customers we haven't seen before (<a href="/facts/Overfitting/5xnFLcMg">overfitting</a>). This issue can also occur if the samples that are being tested have multiple attributes with many distinct values. In this case, it can cause the information gain of each of these attributes to be much higher than those without as many distinct values.
To counter this problem, <a href="/facts/Ross_Quinlan/skDLL2u5">Ross Quinlan</a> proposed to instead choose the attribute with highest <a href="/facts/Information_gain_ratio/kjFE8sfr">information gain ratio</a> from among the attributes whose information gain is average or higher.<a class="footnote-ref" id="fnref:8" href="#fn:8">8</a> This biases the decision tree against considering attributes with a large number of distinct values, while not giving an unfair advantage to attributes with very low information value, as the information value is higher or equal to the information gain.<a class="footnote-ref" id="fnref:9" href="#fn:9">9</a>

<h2 id="see-also">See also</h2>
<ul><li><a href="/facts/Information_gain/nh7SjlPE">Information gain</a> more broadly</li>
<li><a href="/facts/Decision_tree_learning/LbPMAwdI">Decision tree learning</a></li>
<li><a href="/facts/Information_content/EkUdL669">Information content</a>, the starting point of <a href="/facts/Information_theory/qNLH3U5P">information theory</a> and the basis of <a href="/facts/Entropy_(information_theory)/NLg4NLvt">Shannon entropy</a></li>
<li><a href="/facts/Information_gain_ratio/kjFE8sfr">Information gain ratio</a></li>
<li><a href="/facts/ID3_algorithm/cey9VCT7">ID3 algorithm</a>
<ul><li><a href="/facts/C4.5_algorithm/GGsxBTtz">C4.5 algorithm</a></li></ul></li>
<li><a href="/facts/Surprisal_analysis/YnRLGrhe">Surprisal analysis</a></li></ul>

<h2 id="further-reading">Further reading</h2>
<ul><li>Nowozin, Sebastion (2012-06-18). "Improved Information Gain Estimates for Decision Tree Induction". <a href="/facts/ArXiv_(identifier)/H6EtgnBe">arXiv</a>:<a href="https://arxiv.org/abs/1206.4620v1">1206.4620v1</a> [<a href="https://arxiv.org/archive/cs.LG">cs.LG</a>].</li>
<li>Shouman, Mai (2011). <a href="https://crpit.scem.westernsydney.edu.au/confpapers/CRPITV121Shouman.pdf">"Using decision tree for diagnosing heart disease patients"</a> (PDF). Proceedings of the Ninth Australasian Data Mining Conference. 121: 23–30.</li>
<li><a href="/facts/Tom_M._Mitchell/kvDb5xJL">Mitchell, Tom M.</a> (1997). <a href="https://www.cs.cmu.edu/~tom/mlbook.html">Machine Learning</a>. The Mc-Graw-Hill Companies, Inc. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0070428072.</li></ul>

<h2 id="references">References</h2>

<ol>
<li id="fn:1">Larose, Daniel T. (2014). Discovering Knowledge in Data: An Introduction to Data Mining. Hoboken, New Jersey: Wiley. pp. 174–179. ISBN 9780470908747. <a href="9780470908747" target="_blank">9780470908747</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></li>
<li id="fn:2">Gopalan, Bhuvaneswari (2020-12-10). "What is Entropy and Information Gain? How are they used to construct decision trees?". Numpy Ninja. Retrieved 2021-11-28. <a href="https://www.numpyninja.com/post/what-is-entropy-and-information-gain-how-are-they-used-to-construct-decision-trees" target="_blank">https://www.numpyninja.com/post/what-is-entropy-and-information-gain-how-are-they-used-to-construct-decision-trees</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></li>
<li id="fn:3">Jothikumar, R. (2018). "Predicting Life time of Heart Attack Patient using Improved C4.5 Classification Algorithm". Research Journal of Pharmacy and Technology. 11 (5): 1951–1956. doi:10.5958/0974-360X.2018.00362.1. <a href="https://rjptonline.org/HTML_Papers/Research%20Journal%20of%20Pharmacy%20and%20Technology__PID__2018-11-5-47.html" target="_blank">https://rjptonline.org/HTML_Papers/Research%20Journal%20of%20Pharmacy%20and%20Technology__PID__2018-11-5-47.html</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></li>
<li id="fn:4">Larose, Daniel T. (2014). Discovering Knowledge in Data: An Introduction to Data Mining. Hoboken, New Jersey: Wiley. pp. 174–179. ISBN 9780470908747. <a href="9780470908747" target="_blank">9780470908747</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></li>
<li id="fn:5">Larose, Daniel T. (2014). Discovering Knowledge in Data: An Introduction to Data Mining. Hoboken, New Jersey: Wiley. pp. 174–179. ISBN 9780470908747. <a href="9780470908747" target="_blank">9780470908747</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></li>
<li id="fn:6">Larose, Daniel T. (2014). Discovering Knowledge in Data: An Introduction to Data Mining. Hoboken, New Jersey: Wiley. pp. 174–179. ISBN 9780470908747. <a href="9780470908747" target="_blank">9780470908747</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></li>
<li id="fn:7">"machine learning - Why we use information gain over accuracy as splitting criterion in decision tree?". Data Science Stack Exchange. Retrieved 2021-12-09. <a href="https://datascience.stackexchange.com/questions/14433/why-we-use-information-gain-over-accuracy-as-splitting-criterion-in-decision-tre" target="_blank">https://datascience.stackexchange.com/questions/14433/why-we-use-information-gain-over-accuracy-as-splitting-criterion-in-decision-tre</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></li>
<li id="fn:8">Quinlan, J. Ross (1986). "Induction of Decision Trees". Machine Learning. 1 (1): 81–106. doi:10.1007/BF00116251. <a href="https://doi.org/10.1007%2FBF00116251" target="_blank">https://doi.org/10.1007%2FBF00116251</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></li>
<li id="fn:9">Milman, Oren (August 6, 2018). "What is the range of information gain ratio?". Stack Exchange. Retrieved 2018-10-09. <a href="https://stats.stackexchange.com/q/360901" target="_blank">https://stats.stackexchange.com/q/360901</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></li>
</ol>

Information gain (decision tree) open-in-new

Information gain (decision tree)