Menu
Home
People
Places
Arts
History
Plants & Animals
Science
Life & Culture
Technology
Reference.org
Lossless compression
open-in-new
<h2 id="techniques">Techniques</h2> <p>Most lossless compression programs do two things in sequence: the first step generates a <i>statistical model</i> for the input data, and the second step uses this model to map input data to bit sequences in such a way that "probable" (i.e. frequently encountered) data will produce shorter output than "improbable" data. </p><p>The primary encoding algorithms used to produce bit sequences are <a href="/facts/Huffman_coding/dBgMBz48">Huffman coding</a> (also used by the <a href="/facts/Deflate/XnBt1GKV">deflate algorithm</a>) and <a href="/facts/Arithmetic_coding/WU0H4oNu">arithmetic coding</a>. Arithmetic coding achieves compression rates close to the best possible for a particular statistical model, which is given by the <a href="/facts/Information_entropy/NLg4NLvt">information entropy</a>, whereas Huffman compression is simpler and faster but produces poor results for models that deal with symbol probabilities close to 1. </p><p>There are two primary ways of constructing statistical models: in a <i>static</i> model, the data is analyzed and a model is constructed, then this model is stored with the compressed data. This approach is simple and modular, but has the disadvantage that the model itself can be expensive to store, and also that it forces using a single model for all data being compressed, and so performs poorly on files that contain heterogeneous data. <i>Adaptive</i> models dynamically update the model as the data is compressed. Both the encoder and decoder begin with a trivial model, yielding poor compression of initial data, but as they learn more about the data, performance improves. Most popular types of compression used in practice now use adaptive coders. </p><p>Lossless compression methods may be categorized according to the type of data they are designed to compress. While, in principle, any general-purpose lossless compression algorithm (<i>general-purpose</i> meaning that they can accept any bitstring) can be used on any type of data, many are unable to achieve significant compression on data that are not of the form for which they were designed to compress. Many of the lossless compression techniques used for text also work reasonably well for <a href="/facts/Indexed_color/ljGwHMGE">indexed images</a>. </p> <h3>Multimedia</h3> <p>These techniques take advantage of the specific characteristics of images such as the common phenomenon of contiguous 2-D areas of similar tones. Every pixel but the first is replaced by the difference to its left neighbor. This leads to small values having a much higher probability than large values. This is often also applied to sound files, and can compress files that contain mostly low frequencies and low volumes. For images, this step can be repeated by taking the difference to the top pixel, and then in videos, the difference to the pixel in the next frame can be taken. </p><p>A hierarchical version of this technique takes neighboring pairs of data points, stores their difference and sum, and on a higher level with lower resolution continues with the sums. This is called <a href="/facts/Discrete_wavelet_transform/19DVnk4n">discrete wavelet transform</a>. <a href="/facts/JPEG2000/GAzXdHI2">JPEG2000</a> additionally uses data points from other pairs and multiplication factors to mix them into the difference. These factors must be integers, so that the result is an integer under all circumstances. So the values are increased, increasing file size, but the distribution of values could be more peaked. </p><p>The adaptive encoding uses the probabilities from the previous sample in sound encoding, from the left and upper pixel in image encoding, and additionally from the previous frame in video encoding. In the wavelet transformation, the probabilities are also passed through the hierarchy.<a class="footnote-ref" id="fnref:3" href="#fn:3"><sup>3</sup></a> </p> <h3>Historical legal issues</h3> <p>Many of these methods are implemented in open-source and proprietary tools, particularly LZW and its variants. Some algorithms are patented in the <a href="/facts/United_States/P0sCdLe5">United States</a> and other countries and their legal usage requires licensing by the patent holder. Because of patents on certain kinds of <a href="/facts/Lempel%25E2%2580%2593Ziv%25E2%2580%2593Welch/hJfgYm9j">LZW</a> compression, and in particular licensing practices by patent holder Unisys that many developers considered abusive, some open source proponents encouraged people to avoid using the <a href="/facts/Graphics_Interchange_Format/HfIBWrIC">Graphics Interchange Format</a> (GIF) for compressing still image files in favor of <a href="/facts/Portable_Network_Graphics/mwgK6xw5">Portable Network Graphics</a> (PNG), which combines the <a href="/facts/LZ77_and_LZ78/HGH09JM9">LZ77</a>-based <a href="/facts/Deflate/XnBt1GKV">deflate algorithm</a> with a selection of domain-specific prediction filters. However, the patents on LZW expired on June 20, 2003.<a class="footnote-ref" id="fnref:4" href="#fn:4"><sup>4</sup></a> </p><p>Many of the lossless compression techniques used for text also work reasonably well for <a href="/facts/Indexed_image/ljGwHMGE">indexed images</a>, but there are other techniques that do not work for typical text that are useful for some images (particularly simple bitmaps), and other techniques that take advantage of the specific characteristics of images (such as the common phenomenon of contiguous 2-D areas of similar tones, and the fact that color images usually have a preponderance of a limited range of colors out of those representable in the color space). </p><p>As mentioned previously, lossless sound compression is a somewhat specialized area. Lossless sound compression algorithms can take advantage of the repeating patterns shown by the wave-like nature of the data—essentially using <a href="/facts/Autoregressive/LpboAQ0W">autoregressive</a> models to predict the "next" value and encoding the (possibly small) difference between the expected value and the actual data. If the difference between the predicted and the actual data (called the <i>error</i>) tends to be small, then certain difference values (like 0, +1, −1 etc. on sample values) become very frequent, which can be exploited by encoding them in few output bits. </p><p>It is sometimes beneficial to compress only the differences between two versions of a file (or, in <a href="/facts/Video_compression/gMFuY4FT">video compression</a>, of successive images within a sequence). This is called <a href="/facts/Delta_encoding/2NQlyAUc">delta encoding</a> (from the Greek letter <a href="/facts/Delta_(letter)/otPsSYj6">Δ</a>, which in mathematics, denotes a difference), but the term is typically only used if both versions are meaningful outside compression and decompression. For example, while the process of compressing the error in the above-mentioned lossless audio compression scheme could be described as delta encoding from the approximated sound wave to the original sound wave, the approximated version of the sound wave is not meaningful in any other context. </p> <h2 id="methods">Methods</h2> <p class="note">See also: Category:Lossless compression algorithms and <a href="/facts/List_of_lossless_compression_algorithms/g3uenNkc">List of lossless compression algorithms</a></p> <p>No lossless compression algorithm can efficiently compress all possible data (see § Limitations for more on this). For this reason, many different algorithms exist that are designed either with a specific type of input data in mind or with specific assumptions about what kinds of redundancy the uncompressed data are likely to contain. </p><p>Some of the most common lossless compression algorithms are listed below. </p> <h3>General purpose</h3> <ul><li><a href="/facts/Asymmetric_numeral_systems/1tBzSkxj">ANS</a> – Entropy encoding, used by <a href="/facts/LZFSE/an7HA61Q">LZFSE</a> and <a href="/facts/Zstandard/eA9hBQzN">Zstandard</a></li> <li><a href="/facts/Arithmetic_coding/WU0H4oNu">Arithmetic coding</a> – Entropy encoding</li> <li><a href="/facts/Burrows%25E2%2580%2593Wheeler_transform/yxIc9E94">Burrows–Wheeler transform</a> reversible transform for making textual data more compressible, used by <a href="/facts/Bzip2/waffQ2m0">bzip2</a></li> <li><a href="/facts/Huffman_coding/dBgMBz48">Huffman coding</a> – Entropy encoding, pairs well with other algorithms</li> <li><a href="/facts/LZ77_and_LZ78/HGH09JM9">Lempel-Ziv compression</a> (LZ77 and LZ78) – Dictionary-based algorithm that forms the basis for many other algorithms <ul><li><a href="/facts/Deflate/XnBt1GKV">Deflate</a> – Combines LZ77 compression with Huffman coding, used by <a href="/facts/ZIP_(file_format)/qDrrYIB9">ZIP</a>, <a href="/facts/Gzip/H23uqvyx">gzip</a>, and <a href="/facts/Portable_Network_Graphics/mwgK6xw5">PNG</a> images</li> <li><a href="/facts/Lempel%25E2%2580%2593Ziv%25E2%2580%2593Markov_chain_algorithm/XMBm3MBK">Lempel–Ziv–Markov chain algorithm</a> (LZMA) – Very high compression ratio, used by <a href="/facts/7zip/0Y8ItlHs">7zip</a> and <a href="/facts/XZ_Utils/HkIBGKps">xz</a></li> <li><a href="/facts/Lempel%25E2%2580%2593Ziv%25E2%2580%2593Storer%25E2%2580%2593Szymanski/KHe7a3eJ">Lempel–Ziv–Storer–Szymanski</a> (LZSS) – Used by <a href="/facts/WinRAR/74MRE7H6">WinRAR</a> in tandem with Huffman coding</li> <li><a href="/facts/Lempel%25E2%2580%2593Ziv%25E2%2580%2593Welch/hJfgYm9j">Lempel–Ziv–Welch</a> (LZW) – Used by <a href="/facts/GIF/HfIBWrIC">GIF</a> images and Unix's <a href="/facts/Compress/RLSgGNUX">compress</a> utility</li></ul></li> <li><a href="/facts/Prediction_by_partial_matching/cc621LIE">Prediction by partial matching</a> (PPM) – Optimized for compressing <a href="/facts/Plain_text/TSlSSAg2">plain text</a></li> <li><a href="/facts/Run-length_encoding/3k6x9afM">Run-length encoding</a> (RLE) – Simple scheme that provides good compression of data containing many runs of the same value</li></ul> <h3>Audio</h3> <ul><li><a href="/facts/Adaptive_Transform_Acoustic_Coding/kD9psz23">Adaptive Transform Acoustic Coding</a> (ATRAC)</li> <li><a href="/facts/Apple_Lossless/a0D0hzbK">Apple Lossless</a> (ALAC – Apple Lossless Audio Codec)</li> <li><a href="/facts/Audio_Lossless_Coding/ptpbl4hK">Audio Lossless Coding</a> (also known as MPEG-4 ALS)</li> <li><a href="/facts/Super_Audio_CD/f4a7Gu0V">Direct Stream Transfer</a> (DST)</li> <li><a href="/facts/Dolby_TrueHD/ySxj3zIY">Dolby TrueHD</a></li> <li><a href="/facts/DTS-HD_Master_Audio/zprgCEGx">DTS-HD Master Audio</a></li> <li><a href="/facts/Free_Lossless_Audio_Codec/TbqpHlWy">Free Lossless Audio Codec</a> (FLAC)</li> <li><a href="/facts/Meridian_Lossless_Packing/uUXCuWSE">Meridian Lossless Packing</a> (MLP)</li> <li><a href="/facts/Monkey%2527s_Audio/UIJz1IQG">Monkey's Audio</a> (Monkey's Audio APE)</li> <li><a href="/facts/MPEG-4_SLS/0hcHB4NH">MPEG-4 SLS</a> (also known as HD-AAC)</li> <li><a href="/facts/OptimFROG/NHF2DoKS">OptimFROG</a></li> <li><a href="/facts/Original_Sound_Quality/WUThq9tU">Original Sound Quality</a> (OSQ)</li> <li><a href="/facts/RealPlayer/IjS4t5h8">RealPlayer</a> (RealAudio Lossless)</li> <li><a href="/facts/Shorten_(file_format)/XvWCEsxJ">Shorten</a> (SHN)</li> <li>TTA (True Audio Lossless)</li> <li><a href="/facts/WavPack/MN8nT9Gj">WavPack</a> (WavPack lossless)</li> <li><a href="/facts/Windows_Media_Audio_9_Lossless/s5zePro7">WMA Lossless</a> (Windows Media Lossless)</li></ul> <h3>Raster graphics</h3> <ul><li>Lossless only encoding <ul><li><a href="/facts/BMP_file_format/I4LtCFLk">BMP</a></li> <li><a href="/facts/Portable_Network_Graphics/mwgK6xw5">PNG</a> – Portable Network Graphics</li> <li><a href="/facts/GIF/HfIBWrIC">GIF</a> – Graphics Interchange Format</li></ul></li> <li>Lossy and Lossless encoding options <ul><li><a href="/facts/AV1/d07UxHgf">AVIF</a> – AV1 Image File Format</li> <li><a href="/facts/FLIF/3C5cNcUY">FLIF</a> – Free Lossless Image Format</li> <li><a href="/facts/High_Efficiency_Image_File_Format/knA4bTT6">HEIF</a> – High Efficiency Image File Format, using <a href="/facts/High_Efficiency_Video_Coding/6gp7DC1a">HEVC</a></li> <li><a href="/facts/ILBM/S6tgTMPE">ILBM</a> – (RLE compression of <a href="/facts/Amiga/60B3A9of">Amiga</a> <a href="/facts/Interchange_File_Format/tuwp8bH7">IFF</a> images)</li> <li><a href="/facts/JBIG2/j0tIDhfg">JBIG2</a> – compression of B&W images</li> <li><a href="/facts/JPEG_2000/GAzXdHI2">JPEG 2000</a> – (via Le Gall–Tabatabai 5/3<a class="footnote-ref" id="fnref:5" href="#fn:5"><sup>5</sup></a><a class="footnote-ref" id="fnref:6" href="#fn:6"><sup>6</sup></a><a class="footnote-ref" id="fnref:7" href="#fn:7"><sup>7</sup></a> reversible integer <a href="/facts/Wavelet_transform/4vk3W5be">wavelet transform</a>)</li> <li><a href="/facts/Lossless_JPEG/6E9zWoj1">JPEG-LS</a></li> <li><a href="/facts/JPEG_XL/gU0o8Kcr">JPEG XL</a></li> <li><a href="/facts/JPEG_XR/2S78PWt3">JPEG XR</a> – formerly <i>WMPhoto</i> and <i>HD Photo</i></li> <li><a href="/facts/Discrete_cosine_transform/gPpDhPwP">LDCT</a> – <a href="/facts/Discrete_Cosine_Transform/gPpDhPwP">Discrete Cosine Transform</a><a class="footnote-ref" id="fnref:8" href="#fn:8"><sup>8</sup></a><a class="footnote-ref" id="fnref:9" href="#fn:9"><sup>9</sup></a></li> <li><a href="/facts/PCX/3c8YkPQS">PCX</a> – PiCture eXchange</li> <li><a href="/facts/QOI_(image_format)/fvQV7bWm">QOI</a> – Quite OK Image Format</li> <li><a href="/facts/Truevision_TGA/rQc1hHS4">TGA</a> – Truevision TGA</li> <li><a href="/facts/Tag_Image_File_Format/TjBJcLeT">TIFF</a> – Tag Image File Format</li> <li><a href="/facts/WebP/W5Gv7mhf">WebP</a></li></ul></li></ul> <h3>3D Graphics</h3> <ul><li><a href="/facts/OpenCTM/R6AUXsY1">OpenCTM</a> – Lossless compression of 3D triangle meshes</li></ul> <h3>Video</h3> <p>See <a href="/facts/List_of_codecs/e5nTCc5X">list of lossless video codecs</a> </p> <h3>Cryptography</h3> <p><a href="/facts/Cryptosystem/GyDVJ4D0">Cryptosystems</a> often compress data (the "plaintext") <i>before</i> encryption for added security. When properly implemented, compression greatly increases the <a href="/facts/Unicity_distance/wAFhU6LT">unicity distance</a> by removing patterns that might facilitate <a href="/facts/Cryptanalysis/7CPEQlJI">cryptanalysis</a>.<a class="footnote-ref" id="fnref:10" href="#fn:10"><sup>10</sup></a> However, many ordinary lossless compression algorithms produce headers, wrappers, tables, or other predictable output that might instead make cryptanalysis easier. Thus, cryptosystems must utilize compression algorithms whose output does not contain these predictable patterns. </p> <h3>Genetics and genomics</h3> <p><a href="/facts/Compression_of_genomic_sequencing_data/MoEIlAyt">Genetics compression algorithms</a> (not to be confused with <a href="/facts/Genetic_algorithm/WP2AFWuW">genetic algorithms</a>) are the latest generation of lossless algorithms that compress data (typically sequences of nucleotides) using both conventional compression algorithms and specific algorithms adapted to genetic data. In 2012, a team of scientists from Johns Hopkins University published the first genetic compression algorithm that does not rely on external genetic databases for compression. HAPZIPPER was tailored for <a href="/facts/International_HapMap_Project/L2ANDPC4">HapMap</a> data and achieves over 20-fold compression (95% reduction in file size), providing 2- to 4-fold better compression much faster than leading general-purpose compression utilities.<a class="footnote-ref" id="fnref:11" href="#fn:11"><sup>11</sup></a> </p><p>Genomic sequence compression algorithms, also known as DNA sequence compressors, explore the fact that DNA sequences have characteristic properties, such as inverted repeats. The most successful compressors are XM and GeCo.<a class="footnote-ref" id="fnref:12" href="#fn:12"><sup>12</sup></a> For <a href="/facts/Eukaryotes/SkwyPLMA">eukaryotes</a> XM is slightly better in compression ratio, though for sequences larger than 100 MB its computational requirements are impractical. </p> <h3>Executables</h3> <p class="note">Main article: <a href="/facts/Executable_compression/rb9HnGG5">Executable compression</a></p> <p>Self-extracting executables contain a compressed application and a decompressor. When executed, the decompressor transparently decompresses and runs the original application. This is especially often used in <a href="/facts/Demo_(computer_programming)/yKunscS7">demo</a> coding, where competitions are held for demos with strict size limits, as small as 1 <a href="/facts/Kilobyte/jIe6t4yv">kilobyte</a>. This type of compression is not strictly limited to binary executables, but can also be applied to scripts, such as <a href="/facts/JavaScript/XhhIIEsl">JavaScript</a>. </p> <h2 id="benchmarks">Benchmarks</h2> <p>Lossless compression algorithms and their implementations are routinely tested in head-to-head <a href="/facts/Benchmark_(computing)/XSS42cRe">benchmarks</a>. There are a number of better-known compression benchmarks. Some benchmarks cover only the <a href="/facts/Data_compression_ratio/eDEWGzTv">data compression ratio</a>, so winners in these benchmarks may be unsuitable for everyday use due to the slow speed of the top performers. Another drawback of some benchmarks is that their data files are known, so some program writers may optimize their programs for best performance on a particular data set. The winners on these benchmarks often come from the class of <a href="/facts/Context-mixing/xzrwxAsI">context-mixing</a> compression software. </p><p><a href="/facts/Matt_Mahoney_(computer_scientist)/puvtoFtK">Matt Mahoney</a>, in his February 2010 edition of the free booklet <i>Data Compression Explained</i>, additionally lists the following:<a class="footnote-ref" id="fnref:13" href="#fn:13"><sup>13</sup></a> </p> <ul><li>The <a href="/facts/Calgary_Corpus/zEGTndLU">Calgary Corpus</a> dating back to 1987 is no longer widely used due to its small size. Matt Mahoney maintained the Calgary Compression Challenge, created and maintained from May 21, 1996, through May 21, 2016, by Leonid A. Broukhis.</li> <li>The Large Text Compression Benchmark<a class="footnote-ref" id="fnref:14" href="#fn:14"><sup>14</sup></a> and the similar <a href="/facts/Hutter_Prize/6M1AvbXt">Hutter Prize</a> both use a trimmed <a href="/facts/Wikipedia/KDRNKWnc">Wikipedia</a> <a href="/facts/XML/clNgUfWk">XML</a> <a href="/facts/UTF-8/xlaVHSyf">UTF-8</a> data set.</li> <li>The Generic Compression Benchmark,<a class="footnote-ref" id="fnref:15" href="#fn:15"><sup>15</sup></a> maintained by Matt Mahoney, tests compression of data generated by random <a href="/facts/Turing_machine/ojCtglYu">Turing machines</a>.</li> <li>Sami Runsas (the author of NanoZip) maintained Compression Ratings, a benchmark similar to Maximum Compression multiple file test, but with minimum speed requirements. It offered the calculator that allowed the user to weight the importance of speed and compression ratio. The top programs were fairly different due to the speed requirement. In January 2010, the top program was NanoZip followed by <a href="/facts/FreeArc/z2LdmWlo">FreeArc</a>, CCM, flashzip, and <a href="/facts/7-Zip/0Y8ItlHs">7-Zip</a>.</li> <li>The Monster of Compression benchmark by Nania Francesco Antonio tested compression on 1Gb of public data with a 40-minute time limit. In December 2009, the top ranked archiver was NanoZip 0.07a and the top ranked single file compressor was ccmx 1.30c.</li></ul> <p>The Compression Ratings website published a chart summary of the "frontier" in compression ratio and time.<a class="footnote-ref" id="fnref:16" href="#fn:16"><sup>16</sup></a> </p><p>The Compression Analysis Tool<a class="footnote-ref" id="fnref:17" href="#fn:17"><sup>17</sup></a> is a Windows application that enables end users to benchmark the performance characteristics of streaming implementations of LZF4, Deflate, ZLIB, GZIP, BZIP2 and LZMA using their own data. It produces measurements and charts with which users can compare the compression speed, decompression speed and compression ratio of the different compression methods and to examine how the compression level, buffer size and flushing operations affect the results. </p> <h2 id="limitations">Limitations</h2> <p>Lossless data compression algorithms cannot guarantee compression for all input data sets. In other words, for any lossless data compression algorithm, there will be an input data set that does not get smaller when processed by the algorithm, and for any lossless data compression algorithm that makes at least one file smaller, there will be at least one file that it makes larger. This is easily proven with elementary mathematics using a counting argument called the <a href="/facts/Pigeonhole_principle/hDks9p4n">pigeonhole principle</a>, as follows:<a class="footnote-ref" id="fnref:18" href="#fn:18"><sup>18</sup></a><a class="footnote-ref" id="fnref:19" href="#fn:19"><sup>19</sup></a> </p> <ul><li>Assume that each file is represented as a string of bits of some arbitrary length.</li> <li>Suppose that there is a compression algorithm that transforms every file into an output file that is no longer than the original file, and that at least one file will be compressed into an output file that is shorter than the original file.</li> <li>Let <i>M</i> be the least number such that there is a file <i>F</i> with length <i>M</i> bits that compresses to something shorter. Let <i>N</i> be the length (in bits) of the compressed version of <i>F</i>.</li> <li>Because <i>N</i><<i>M</i>, every file of length <i>N</i> keeps its size during compression. There are 2<i>N</i> such files possible. Together with <i>F</i>, this makes 2<i>N</i>+1 files that all compress into one of the 2<i>N</i> files of length <i>N</i>.</li> <li>But 2<i>N</i> is smaller than 2<i>N</i>+1, so by the pigeonhole principle there must be some file of length <i>N</i> that is simultaneously the output of the compression function on two different inputs. That file cannot be decompressed reliably (which of the two originals should that yield?), which contradicts the assumption that the algorithm was lossless.</li> <li>We must therefore conclude that our original hypothesis (that the compression function makes no file longer) is necessarily untrue.</li></ul> <p>Most practical compression algorithms provide an "escape" facility that can turn off the normal coding for files that would become longer by being encoded. In theory, only a single additional bit is required to tell the decoder that the normal coding has been turned off for the entire input; however, most encoding algorithms use at least one full byte (and typically more than one) for this purpose. For example, <a href="/facts/Deflate/XnBt1GKV">deflate</a> compressed files never need to grow by more than 5 bytes per 65,535 bytes of input. </p><p>In fact, if we consider files of length N, if all files were equally probable, then for any lossless compression that reduces the size of some file, the expected length of a compressed file (averaged over all possible files of length N) must necessarily be <i>greater</i> than N. So if we know nothing about the properties of the data we are compressing, we might as well not compress it at all. A lossless compression algorithm is useful only when we are more likely to compress certain types of files than others; then the algorithm could be designed to compress those types of data better. </p><p>Thus, the main lesson from the argument is not that one risks big losses, but merely that one cannot always win. To choose an algorithm always means implicitly to select a <i>subset</i> of all files that will become usefully shorter. This is the theoretical reason why we need to have different compression algorithms for different kinds of files: there cannot be any algorithm that is good for all kinds of data. </p><p>The "trick" that allows lossless compression algorithms, used on the type of data they were designed for, to consistently compress such files to a shorter form is that the files the algorithms are designed to act on all have some form of easily modeled <a href="/facts/Redundancy_(information_theory)/HnbfBLE0">redundancy</a> that the algorithm is designed to remove, and thus belong to the subset of files that that algorithm can make shorter, whereas other files would not get compressed or even get bigger. Algorithms are generally quite specifically tuned to a particular type of file: for example, lossless audio compression programs do not work well on text files, and vice versa. </p><p>In particular, files of <a href="/facts/Random/QJQBoEX5">random</a> data cannot be consistently compressed by any conceivable lossless data compression algorithm; indeed, this result is used to <i>define</i> the concept of randomness in <a href="/facts/Kolmogorov_complexity/L9ezq8pY">Kolmogorov complexity</a>.<a class="footnote-ref" id="fnref:20" href="#fn:20"><sup>20</sup></a> </p><p>It is provably impossible to create an algorithm that can losslessly compress any data. While there have been many claims through the years of companies achieving "perfect compression" where an arbitrary number <i>N</i> of random bits can always be compressed to <i>N</i> − 1 bits, these kinds of claims can be safely discarded without even looking at any further details regarding the purported compression scheme. Such an algorithm contradicts fundamental laws of mathematics because, if it existed, it could be applied repeatedly to losslessly reduce any file to length 1.<a class="footnote-ref" id="fnref:21" href="#fn:21"><sup>21</sup></a> </p><p>On the other hand, it has also been proven<a class="footnote-ref" id="fnref:22" href="#fn:22"><sup>22</sup></a> that there is no algorithm to determine whether a file is incompressible in the sense of Kolmogorov complexity. Hence it is possible that any particular file, even if it appears random, may be significantly compressed, even including the size of the decompressor. An example is the digits of the mathematical constant <i><a href="/facts/Pi/vC0UBj1q">pi</a></i>, which appear random but can be generated by a very small program. However, even though it cannot be determined whether a particular file is incompressible, a <a href="/facts/Kolmogorov_complexity/L9ezq8pY">simple theorem about incompressible strings</a> shows that over 99% of files of any given length cannot be compressed by more than one byte (including the size of the decompressor). </p> <h3>Mathematical background</h3> <p>Abstractly, a <a href="/facts/Compression_algorithm/gMFuY4FT">compression algorithm</a> can be viewed as a <a href="/facts/Function_(mathematics)/CdReEKCh">function</a> on sequences (normally of octets). Compression is successful if the resulting sequence is shorter than the original sequence (and the instructions for the decompression map). For a compression algorithm to be <a href="/facts/Lossless/LS702KsN">lossless</a>, the compression map must form an <a href="/facts/Injective_function/5ES6tqf5">injection</a> from "plain" to "compressed" bit sequences. The pigeonhole principle prohibits a bijection between the collection of sequences of length <i>N</i> and any subset of the collection of sequences of length <i>N</i>−1. Therefore, it is not possible to produce a lossless algorithm that reduces the size of every possible input sequence.<a class="footnote-ref" id="fnref:23" href="#fn:23"><sup>23</sup></a> </p> <h3>Points of application in real compression theory</h3> <p>Real compression algorithm designers accept that streams of high information entropy cannot be compressed, and accordingly, include facilities for detecting and handling this condition. An obvious way of detection is applying a raw compression algorithm and testing if its output is smaller than its input. Sometimes, detection is made by <a href="/facts/Heuristics/BcBEkv9c">heuristics</a>; for example, a compression application may consider files whose names end in ".zip", ".arj" or ".lha" uncompressible without any more sophisticated detection. A common way of handling this situation is quoting input, or uncompressible parts of the input in the output, minimizing the compression overhead. For example, the <a href="/facts/ZIP_(file_format)/qDrrYIB9">zip</a> data format specifies the 'compression method' of 'Stored' for input files that have been copied into the archive verbatim.<a class="footnote-ref" id="fnref:24" href="#fn:24"><sup>24</sup></a> </p> <h3>The Million Random Digit Challenge</h3> <p>Mark Nelson, in response to claims of "magic" compression algorithms appearing in comp.compression, has constructed a 415,241 byte binary file of highly entropic content, and issued a public challenge of $100 to anyone to write a program that, together with its input, would be smaller than his provided binary data yet be able to reconstitute it without error.<a class="footnote-ref" id="fnref:25" href="#fn:25"><sup>25</sup></a> A similar challenge, with $5,000 as reward, was issued by Mike Goldman.<a class="footnote-ref" id="fnref:26" href="#fn:26"><sup>26</sup></a> </p> <h2 id="see-also">See also</h2> <ul><li><a href="/facts/Comparison_of_file_archivers/tK0fEnF7">Comparison of file archivers</a></li> <li><a href="/facts/Data_compression/gMFuY4FT">Data compression</a></li> <li><a href="/facts/David_A._Huffman/4fbz13XB">David A. Huffman</a></li> <li><a href="/facts/Entropy_(information_theory)/NLg4NLvt">Entropy (information theory)</a></li> <li><a href="/facts/Grammar-based_code/z308LVJV">Grammar-based code</a></li> <li><a href="/facts/Hutter_Prize/6M1AvbXt">Hutter Prize</a></li> <li><a href="/facts/Information_theory/qNLH3U5P">Information theory</a></li> <li><a href="/facts/Kolmogorov_complexity/L9ezq8pY">Kolmogorov complexity</a></li> <li><a href="/facts/List_of_codecs/e5nTCc5X">List of codecs</a></li> <li><a href="/facts/Lossless_Transform_Audio_Compression/TAfmp2xf">Lossless Transform Audio Compression</a> (LTAC)</li> <li><a href="/facts/Lossy_compression/jHW7DDv3">Lossy compression</a></li> <li><a href="/facts/Normal_number/nGb5cM0y">Normal number</a></li> <li><a href="/facts/Universal_code_(data_compression)/9zn1cM2M">Universal code (data compression)</a></li></ul> <h2 id="further-reading">Further reading</h2> <ul><li>Sayood, Khalid (October 27, 2017). <i>Introduction to Data Compression</i>. The Morgan Kaufmann Series in Multimedia Information and Systems (5 ed.). <a href="/facts/Morgan_Kaufmann/HfefrWju">Morgan Kaufmann</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-12809474-7. (790 pages)</li> <li>Sayood, Khalid, ed. (December 18, 2002). <i>Lossless Compression Handbook (Communications, Networking and Multimedia)</i> (1 ed.). <a href="/facts/Academic_Press/qxoKjzFy">Academic Press</a>. <a href="/facts/ISBN_(identifier)/15AdSPa9">ISBN</a> 978-0-12390754-7. (488 pages)</li></ul> <h2 id="external-links">External links</h2> <ul><li><a href="https://github.com/ning/compress/wiki/LZFFormat">"LZF compression format"</a>. <i>github</i>. Retrieved October 17, 2017.</li> <li>Phamdo, Nam. <a href="https://web.archive.org/web/20160508054615/http://www.data-compression.com/theory.html">"Theory of Data Compression"</a>. <i>Data Compression</i>. Archived from <a href="http://www.data-compression.com/theory.html">the original</a> on May 8, 2016. Retrieved October 17, 2017.</li> <li><a href="http://wiki.hydrogenaud.io/index.php?title=Lossless_comparison">"Lossless comparison"</a>. <i>Hydrogenaudio Knowledgebase</i>. January 5, 2015. Retrieved October 17, 2017.</li> <li><a href="http://www.bobulous.org.uk/misc/audioFormats.html">"Lossless and lossy audio formats for music"</a>. <i>Bobulous Central</i>. November 6, 2003. Retrieved October 17, 2017.</li> <li><a href="https://www.c10n.info/archives/439">"Image Compression Benchmark"</a>. <a href="https://web.archive.org/web/20130210011959/https://www.c10n.info/archives/439">Archived</a> from the original on February 10, 2013. overview of <ul><li><a href="http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7,096,360.PN.&OS=PN/7,096,360&RS=PN/7,096,360">US patent #7,096,360</a> <a href="https://web.archive.org/web/20170202002158/http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7,096,360.PN.&OS=PN/7,096,360&RS=PN/7,096,360">Archived</a> February 2, 2017, at the <a href="/facts/Wayback_Machine/nmQ3a6JC">Wayback Machine</a>, "[a]n "Frequency-Time Based Data Compression Method" supporting the compression, encryption, decompression, and decryption and persistence of many binary digits through frequencies where each frequency represents many bits."</li></ul></li></ul> <h2 id="references">References</h2> <ol> <li id="fn:1"><p>"Unit 4 Lab 4: Data Representation and Compression, Page 6". bjc.edc.org. Retrieved April 9, 2022. <a href="https://bjc.edc.org/bjc-r/cur/programming/4-internet/4-representation-compression/6-compression.html?topic=nyc_bjc/4-internet.topic&course=bjc4nyc.html&novideo&noassignment#:~:text=Lossless%20compression%20works%20by%20removing,an%20example%20of%20lossless%20compression." target="_blank">https://bjc.edc.org/bjc-r/cur/programming/4-internet/4-representation-compression/6-compression.html?topic=nyc_bjc/4-internet.topic&course=bjc4nyc.html&novideo&noassignment#:~:text=Lossless%20compression%20works%20by%20removing,an%20example%20of%20lossless%20compression.</a> <a href="#fnref:1" class="footnote-back-ref">↩</a></p></li> <li id="fn:2"><p>Price, Andy (March 3, 2022). "Lossless Streaming – the future of high res audio". Audio Media International. <a href="https://audiomediainternational.com/lossless-streaming-back-to-the-future/" target="_blank">https://audiomediainternational.com/lossless-streaming-back-to-the-future/</a> <a href="#fnref:2" class="footnote-back-ref">↩</a></p></li> <li id="fn:3"><p>Unser, M.; Blu, T. (2003). "Mathematical properties of the JPEG2000 wavelet filters" (PDF). IEEE Transactions on Image Processing. 12 (9): 1080–1090. Bibcode:2003ITIP...12.1080U. doi:10.1109/TIP.2003.812329. PMID 18237979. <a href="https://infoscience.epfl.ch/record/63104/files/unser0305.pdf" target="_blank">https://infoscience.epfl.ch/record/63104/files/unser0305.pdf</a> <a href="#fnref:3" class="footnote-back-ref">↩</a></p></li> <li id="fn:4"><p>"LZW Patent Information". About Unisys. Unisys. Archived from the original on June 2, 2009. <a href="https://web.archive.org/web/20090602212118/http://www.unisys.com/about__unisys/lzw" target="_blank">https://web.archive.org/web/20090602212118/http://www.unisys.com/about__unisys/lzw</a> <a href="#fnref:4" class="footnote-back-ref">↩</a></p></li> <li id="fn:5"><p>Sullivan, Gary (December 8–12, 2003). "General characteristics and design considerations for temporal subband video coding". ITU-T. Video Coding Experts Group. Retrieved September 13, 2019. <a href="https://www.itu.int/wftp3/av-arch/video-site/0312_Wai/VCEG-U06.doc" target="_blank">https://www.itu.int/wftp3/av-arch/video-site/0312_Wai/VCEG-U06.doc</a> <a href="#fnref:5" class="footnote-back-ref">↩</a></p></li> <li id="fn:6"><p>Unser, M.; Blu, T. (2003). "Mathematical properties of the JPEG2000 wavelet filters" (PDF). IEEE Transactions on Image Processing. 12 (9): 1080–1090. Bibcode:2003ITIP...12.1080U. doi:10.1109/TIP.2003.812329. PMID 18237979. <a href="https://infoscience.epfl.ch/record/63104/files/unser0305.pdf" target="_blank">https://infoscience.epfl.ch/record/63104/files/unser0305.pdf</a> <a href="#fnref:6" class="footnote-back-ref">↩</a></p></li> <li id="fn:7"><p>Bovik, Alan C. (2009). The Essential Guide to Video Processing. Academic Press. p. 355. ISBN 9780080922508. <a href="9780080922508" target="_blank">9780080922508</a> <a href="#fnref:7" class="footnote-back-ref">↩</a></p></li> <li id="fn:8"><p>Ahmed, Nasir; Mandyam, Giridhar D.; Magotra, Neeraj (April 17, 1995). Rodriguez, Arturo A.; Safranek, Robert J.; Delp, Edward J. (eds.). "DCT-based scheme for lossless image compression". Digital Video Compression: Algorithms and Technologies 1995. 2419. International Society for Optics and Photonics: 474–478. Bibcode:1995SPIE.2419..474M. doi:10.1117/12.206386. <a href="/wiki/N._Ahmed" target="_blank">/wiki/N._Ahmed</a> <a href="#fnref:8" class="footnote-back-ref">↩</a></p></li> <li id="fn:9"><p>Komatsu, K.; Sezaki, K. (1998). "Reversible discrete cosine transform". Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181). Vol. 3. pp. 1769–1772. doi:10.1109/ICASSP.1998.681802. ISBN 0-7803-4428-6. <a href="0-7803-4428-6" target="_blank">0-7803-4428-6</a> <a href="#fnref:9" class="footnote-back-ref">↩</a></p></li> <li id="fn:10"><p>Alfred J. Menezes; Paul C. van Oorschot; Scott A. Vanstone (October 16, 1996). Handbook of Applied Cryptography. CRC Press. ISBN 978-1-4398-2191-6. <a href="978-1-4398-2191-6" target="_blank">978-1-4398-2191-6</a> <a href="#fnref:10" class="footnote-back-ref">↩</a></p></li> <li id="fn:11"><p>Chanda, P.; Elhaik, E.; Bader, J.S. (2012). "HapZipper: sharing HapMap populations just got easier". Nucleic Acids Res. 40 (20): 1–7. doi:10.1093/nar/gks709. PMC 3488212. PMID 22844100. <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3488212" target="_blank">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3488212</a> <a href="#fnref:11" class="footnote-back-ref">↩</a></p></li> <li id="fn:12"><p>Pratas, D.; Pinho, A. J.; Ferreira, P. J. S. G. (2016). "Efficient compression of genomic sequences". Data Compression Conference (PDF). Snowbird, Utah.{{cite book}}: CS1 maint: location missing publisher (link) <a href="http://sweet.ua.pt/pratas/papers/Pratas-2016b.pdf" target="_blank">http://sweet.ua.pt/pratas/papers/Pratas-2016b.pdf</a> <a href="#fnref:12" class="footnote-back-ref">↩</a></p></li> <li id="fn:13"><p>Matt Mahoney (2010). "Data Compression Explained" (PDF). pp. 3–5. <a href="http://nishi.dreamhosters.com/u/dce2010-02-26.pdf" target="_blank">http://nishi.dreamhosters.com/u/dce2010-02-26.pdf</a> <a href="#fnref:13" class="footnote-back-ref">↩</a></p></li> <li id="fn:14"><p>"Large Text Compression Benchmark". mattmahoney.net. <a href="http://mattmahoney.net/dc/text.html" target="_blank">http://mattmahoney.net/dc/text.html</a> <a href="#fnref:14" class="footnote-back-ref">↩</a></p></li> <li id="fn:15"><p>"Generic Compression Benchmark". mattmahoney.net. <a href="http://mattmahoney.net/dc/uiq/" target="_blank">http://mattmahoney.net/dc/uiq/</a> <a href="#fnref:15" class="footnote-back-ref">↩</a></p></li> <li id="fn:16"><p>"Summary". September 1, 2016. Archived from the original on September 1, 2016. <a href="https://web.archive.org/web/20160901094802/http://compressionratings.com/rating_sum.html" target="_blank">https://web.archive.org/web/20160901094802/http://compressionratings.com/rating_sum.html</a> <a href="#fnref:16" class="footnote-back-ref">↩</a></p></li> <li id="fn:17"><p>"Compression Analysis Tool". Free Tools. Noemax Technologies. <a href="https://www.noemax.com/free-tools/compression-analysis-tool.asp" target="_blank">https://www.noemax.com/free-tools/compression-analysis-tool.asp</a> <a href="#fnref:17" class="footnote-back-ref">↩</a></p></li> <li id="fn:18"><p>Sayood 2002, p. 41. - Sayood, Khalid, ed. (December 18, 2002). Lossless Compression Handbook (Communications, Networking and Multimedia) (1 ed.). Academic Press. ISBN 978-0-12390754-7. <a href="#fnref:18" class="footnote-back-ref">↩</a></p></li> <li id="fn:19"><p>Bell, Tim (2015). "Surprising Computer Science". Informatics in Schools. Curricula, Competences, and Competitions. Lecture Notes in Computer Science. Vol. 9378. pp. 1–11. doi:10.1007/978-3-319-25396-1_1. ISBN 978-3-319-25395-4. See in particular pp. 8–9. <a href="978-3-319-25395-4" target="_blank">978-3-319-25395-4</a> <a href="#fnref:19" class="footnote-back-ref">↩</a></p></li> <li id="fn:20"><p>Sayood 2002, p. 38. - Sayood, Khalid, ed. (December 18, 2002). Lossless Compression Handbook (Communications, Networking and Multimedia) (1 ed.). Academic Press. ISBN 978-0-12390754-7. <a href="#fnref:20" class="footnote-back-ref">↩</a></p></li> <li id="fn:21"><p>Bell, Tim (2015). "Surprising Computer Science". Informatics in Schools. Curricula, Competences, and Competitions. Lecture Notes in Computer Science. Vol. 9378. pp. 1–11. doi:10.1007/978-3-319-25396-1_1. ISBN 978-3-319-25395-4. See in particular pp. 8–9. <a href="978-3-319-25395-4" target="_blank">978-3-319-25395-4</a> <a href="#fnref:21" class="footnote-back-ref">↩</a></p></li> <li id="fn:22"><p>Li, Ming; Vitányi, Paul (1993). An Introduction to Kolmogorov Complexity and its Applications. New York: Springer. p. 102. ISBN 0-387-94053-7. Theorem 2.6 The function C ( x ) {\displaystyle C(x)} is not partial recursive. <a href="0-387-94053-7" target="_blank">0-387-94053-7</a> <a href="#fnref:22" class="footnote-back-ref">↩</a></p></li> <li id="fn:23"><p>Joshi, Mark (2015). "The Pigeonhole Principle". Proof Patterns. pp. 19–23. doi:10.1007/978-3-319-16250-8_3. ISBN 978-3-319-16249-2. <a href="978-3-319-16249-2" target="_blank">978-3-319-16249-2</a> <a href="#fnref:23" class="footnote-back-ref">↩</a></p></li> <li id="fn:24"><p>".ZIP File Format Specification". PKWARE, Inc. chapter V, section J. <a href="http://www.pkware.com/documents/casestudies/APPNOTE.TXT" target="_blank">http://www.pkware.com/documents/casestudies/APPNOTE.TXT</a> <a href="#fnref:24" class="footnote-back-ref">↩</a></p></li> <li id="fn:25"><p>Nelson, Mark (June 20, 2006). "The Million Random Digit Challenge Revisited". Mark Nelson. <a href="https://marknelson.us/posts/2006/06/20/million-digit-challenge.html" target="_blank">https://marknelson.us/posts/2006/06/20/million-digit-challenge.html</a> <a href="#fnref:25" class="footnote-back-ref">↩</a></p></li> <li id="fn:26"><p>Craig, Patrick. "The $5000 Compression Challenge". Retrieved June 8, 2009. <a href="http://www.patrickcraig.co.uk/other/compression.htm" target="_blank">http://www.patrickcraig.co.uk/other/compression.htm</a> <a href="#fnref:26" class="footnote-back-ref">↩</a></p></li> </ol>