Model collapse

Model collapse is a phenomenon where <a href="/facts/Machine_learning/e0w0XJTu">machine learning</a> models gradually degrade due to errors coming from uncurated training on the outputs of another model, such as prior versions of itself. Such outputs are known as <a href="/facts/Synthetic_data/MVSNrFPI">synthetic data</a>. It is a possible mechanism for <a href="/facts/Mode_collapse/3LxUSdmw">mode collapse</a>.
Shumailov et al. coined the term and described two specific stages to the degradation: early model collapse and late model collapse. In early model collapse, the model begins losing information about the tails of the <a href="/facts/Probability_distribution/EpsKKVRu">distribution</a> – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data. In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its <a href="/facts/Variance/ULBJKXD1">variance</a>.

Model collapse open-in-new

Model collapse