Sometimes you open two files and not a single character differs, yet diff says every line has changed, or that only the first or last line differs. The cause is usually characters you cannot see on screen. This guide covers the kinds of "invisible differences" and how to deal with them.
Line-break characters: CRLF, LF, CR
Operating systems follow different conventions for ending a line.
| Name | Bytes | Where it is mainly used |
|---|---|---|
| LF | 0A (\n) | Linux, macOS, most development tools |
| CRLF | 0D 0A (\r\n) | Many Windows programs such as Notepad; internet protocols such as email and HTTP |
| CR | 0D (\r) | Very old Mac OS (before version 9) |
If one file is saved with CRLF and the other with LF, a line-by-line comparison picks up an invisible \r at the end of every line as a difference, so it looks as if the whole file has changed. GNU diff can ignore this with --strip-trailing-cr, and git diff with --ignore-cr-at-eol.
Newline at end of file
By POSIX convention, lines in a text file end with a newline character. So many tools put a newline after the last line too, but some editors do not. This difference is shown in a unified diff like this:
@@ -1,2 +1,2 @@ host=example.com-port=8080\ No newline at end of file+port=8080\ No newline at end of file means the line right above has no newline at the end. In this example, the content is the same, and only the new file gained a final newline. Many code style guides require a newline at the end of files, and you can standardize it with your editor's "insert final newline" setting.
BOM: when only the first line differs
Sometimes the three bytes EF BB BF are placed at the very start of a UTF-8 file. This is called a byte order mark (BOM). UTF-8 does not need a byte order, but some programs add it as a marker meaning "this file is UTF-8". A BOM is invisible on screen, so if diff says only the first line differs, suspect a BOM. This site removes a leading BOM when loading a file, and when a whitespace-ignore option is on, it also treats a BOM (U+FEFF) inside pasted text like whitespace.
Tabs, spaces, and special spaces
- Tabs and spaces: one tab and four spaces may look the same on screen but are different characters.
- No-break space (NBSP, U+00A0): often slips into text copied from web pages or word processors.
- Ideographic (full-width) space (U+3000): can be entered by Chinese, Japanese, and Korean input methods.
- Zero-width space (U+200B): has no width, so it is completely invisible, but it counts as a character.
Most of these are filtered out when you turn on a whitespace-ignore option (though characters that are not classified as whitespace, such as U+200B, may remain even with the option on). If you switch the in-line diff to Character, the highlighting shows you where an invisible character is.
Unicode normalization: same character, different bytes
The Korean syllable "가" can be written either as one precomposed character (U+AC00) or as the initial consonant ㄱ (U+1100) followed by the vowel ㅏ (U+1161). Accented Latin letters work the same way: "é" can be the single character U+00E9 or "e" followed by a combining acute accent (U+0301). Unicode calls the precomposed form NFC and the decomposed form NFD (UAX #15). The two texts look identical on screen but compare as different characters. A typical example is file names created on macOS that appear with their Korean letters split apart, like "ㄱㅏ", on other operating systems. The fix is to convert one side to NFC before comparing so that both match.
Standardizing line endings in git
In a repository shared across operating systems, you can standardize line endings with git settings.
core.autocrlf: set totrue, it converts LF to CRLF on checkout and CRLF to LF on commit (mainly for Windows).inputconverts CRLF to LF only on commit..gitattributes: defines the rules per repository, so the result is the same even if each person's settings differ.
* text=auto
*.sh text eol=lf
*.bat text eol=crlftext=auto normalizes line endings to LF inside the repository for files git detects as text, and eol=lf / eol=crlf set the line endings used when files are checked out into the working folder.
Try it in this tool
Normalize line endings in the Text Diff Checker is on by default, so CRLF, CR, and LF are all treated as the same line break and a missing final newline is ignored. Turn this option off and a CR at the end of a line is shown as ␍, and a last line with no newline is marked with ∅, so you can see the differences that were invisible. Whitespace options are covered in Whitespace options, and the meaning of the symbols in the result in How to read a unified diff.