UTF-8: how letters become bytes
A computer saves a text file as numbers. UTF-8 is a shared set of rules for turning text into bytes, and back again. This is called a text encoding. Think of two friends using the same codebook: they can read each other’s messages because they agree on the rules.
Why the 8?
UTF means Unicode Transformation Format. The 8 refers to eight-bit units: bytes. A single Unicode code point takes one to four bytes in UTF-8. It does not mean every letter uses eight bytes. See the Unicode encoding explanation.
| Text | Code points | UTF-8 bytes |
|---|---|---|
A |
1 | 1 |
é |
1 | 2 |
🐢 |
1 | 4 |
Here é is the single-code-point spelling. An accent can also be stored separately; code points explains that surprise.
Try it in Pliro
# language: en
say "Café 🐢"
say length("Aé🐢")
The output is Café 🐢, then 3. length counts code points, so the second line does not print the seven bytes used by that text.
What is a BOM?
UTF-8 with BOM, sometimes written UTF-8-BOM, means a UTF-8 text file with an extra, invisible label at its very beginning. BOM is short for byte order mark. Think of a sticker on an envelope that tells the reader how the message inside was saved.
Your text editor can add this label when saving. You usually do not see it among the letters of your program. It is not the word BOM typed into the code, and it is not the # language: en line.
| Saving option | What is in the file? | Suitable for Pliro source? |
|---|---|---|
| UTF-8 without BOM | Your text saved as UTF-8 | Yes |
| UTF-8 with BOM / UTF-8-BOM | An invisible label followed by your UTF-8 text | No, Pliro currently rejects the label |
The label takes three extra bytes. UTF-8 does not need it. Other programs may accept it, so a file can look normal in your editor and still be rejected by Pliro. The Unicode FAQ explains the technical background.
How do I save without it?
If a Pliro source file made in another editor has this problem:
- Keep a copy of the original file.
- Find the editor’s encoding setting: the option that controls how it saves text. Choose UTF-8 without BOM. Some editors call this just UTF-8 and list UTF-8 with BOM separately; check the option’s description if it is unclear.
- Save the file and check it in Pliro again. Keep the source-language line at the top.
Changing the filename does not change this setting. You normally do not fix it by pressing Backspace on the first visible letter, because the label is hidden. If your editor does not show a clear encoding option, ask a teacher or another helper to find it with you.
You only need these steps when the problem occurs. You do not need to learn byte patterns to start programming.
If accents turn into strange symbols, check the editor’s encoding. Changing the file extension does not change its encoding. Keep a copy before converting an existing file.
Try replacing the turtle with two turtles. Predict what length prints before running again.