drummyfish/less_retarded_wiki
5
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Update

Browse source
This commit is contained in:
Miloslav Ciz 2025-03-15 23:42:07 +01:00
parent 11eb6b01b2
commit 6d2a78f17c
37 changed files with 2131 additions and 1952 deletions
Download patch file Download diff file Expand all files Collapse all files

4
entropy.md
View file

@ -1,6 +1,6 @@
# Entropy
Entropy is a quite cryptic, often misunderstood [scientific](science.md) term that may have different definitions depending on specific field and context, which can intuitively be interpreted as an amount of disorder, uncertainty or [randomness](randomness.md). There are two main kinds of entropy: [information](information.md) entropy (information theory) and thermodynamic entropy (physics).
Entropy is a quite cryptic, [often misunderstood](often_misunderstood.md) [scientific](science.md) term that may have different definitions depending on specific field and context, which can intuitively be interpreted as an amount of disorder, uncertainty or [randomness](randomness.md). There are two main kinds of entropy: [information](information.md) entropy (information theory) and thermodynamic entropy (physics).
## Information Entropy
@ -20,7 +20,7 @@ Now an **entropy of a random variable** *X*, which can take values *x1*, *x2*, *
Entropy is greater if unpredictability ("randomness") is greater -- it is at its maximum if all possible values of the random variable are equally likely. For example entropy of a coin toss is 1 bit, given both outcomes are equally likely (if one outcome was more likely than the other, entropy would go down).
More predictable events have lower entropy -- for example English text has quite low entropy because it is pretty easy to predict missing letters from other letters (there is a lot of redundancy in human language). Thanks to this we can compress the text, e.g. using [Huffman code](huffman_code.md) -- compression reduces size, i.e. removes redundancy/correlation/predictability, and so increases entropy.
More predictable events have lower entropy -- for example [English](english.md) text has quite low entropy because it is pretty easy to predict missing letters from other letters (there is a lot of redundancy in human language). Thanks to this we can compress the text, e.g. using [Huffman code](huffman_code.md) -- compression reduces size, i.e. removes redundancy/correlation/predictability, and so increases entropy.
**Example**: consider a weather forecast for a specific area, day and hour -- our weather model predicts rain with 55% probability, cloudy with 30% probability and sunny with 15% probability. Once the specific day and hour comes, we will receive a message about the ACTUAL weather that there was in the area. What entropy does such message have? According to the formula above: *H = 0.55 * log2(1/0.55) + 0.3 * log2(1/0.3) + 0.15 * log2(1/0.25) ~= 1.3 bits*. That is the entropy and amount of information such message gives us.