Crlf: Difference between revisions

From miki
Jump to navigation Jump to search
(Created page with 'Everything about ascii file line termination issue (LF, CR/LF, CR...) == Detection == * Hex dump <source lang=bash> hd <file> # 00000000 48 65 6c 6c 6f 0d 0a 48 65 6c 6c 6f 0d…')
 
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
Everything about ascii file line termination issue (LF, CR/LF, CR...)
Everything about ascii file line termination issue (LF, CR/LF, CR...)

== Reference ==
* [http://vim.wikia.com/wiki/File_format File format in Vim]

== Miscellaneous ==
* Normal Linux line ending is LF
* Normal DOS line ending is CR-LF
* In Bash, insert special character with {{kbctrl|V}}. For instance {{kbctrl|V}} {{kbctrl|M}} will insert a CR character
** Alternatively, use something like <code>r="$(printf "\r")"</code>,
** Or use <code>echo $'\r'</code>


== Detection ==
== Detection ==
Line 11: Line 21:


* Use ''<tt>file</tt>'':
* Use ''<tt>file</tt>'':
<source lang="bash">
<source lang=bash>
file crlf.txt cr_and_lf.txt lf.txt
file crlf.txt cr_and_lf.txt lf.txt
# crlf.txt: ASCII text, with CRLF line terminators
# crlf.txt: ASCII text, with CRLF line terminators
# cr_and_lf.txt: ASCII text, with CRLF, LF line terminators
# cr_and_lf.txt: ASCII text, with CRLF, LF line terminators
lf# .txt: ASCII text
# lf.txt: ASCII text
find . -print0 | xargs -0 file | grep CR # Search all files
</source>

* Using GREP:
<source lang=bash>
grep -IUr --color $'\r' # Add -l to have list of files
find . -name "*.[ch]" -print0 | xargs -0 grep -IUl $'\r' # Prefilter files to look at
</source>

== Conversion ==
Reference: [http://www.cyberciti.biz/faq/howto-unix-linux-convert-dos-newlines-cr-lf-unix-text-format/]

* Using package '''<tt>dos2unix</tt>''':
<source lang="bash">
dos2unix <file>
unix2dos <file>
mac2unix <file>
unix2mac <file>
</source>

For instance, to fix all CR/CRLF issues, and remove also any trailing blanks (i.e. sanitizing all sources):
<source lang="bash">
# Convert CRLF to LF
find . -name '*.[chCH]' -print0 | xargs -0 dos2unix
# ... twice in case we had "CRCRLF" in source files (from a bad LF->CRLF conversion)
find . -name '*.[chCH]' -print0 | xargs -0 dos2unix
# Convert CR to LF
find . -name '*.[chCH]' -print0 | xargs -0 mac2unix
# Remove trailing blanks
find . -name '*.[chCH]' -print0 | xargs -0 -n 1 sed -i -r 's/ +$//'
</source>


* Using command '''<tt>tr</tt>''':
<source lang="bash">
tr -d '\r' < file # Delete RETURN (i.e. CR, ^M, ascii 13)
tr -d '\n' < file # Delete NEWLINE (i.e. LF, ^J, ascii 10)
</source>

* Using '''Vim''':
<source lang="bash">
vim file
</source>
<source lang="vim">
:set ff=unix
:wq
</source>

* Using '''Perl''':
<source lang="bash">
perl -pi -e 's/\r\n/\n/g' input.file # CRLF to LF
</source>

* Using '''Sed''':
<source lang="bash">
sed 's/$/^M/" input.txt > output.txt # LF to CRLF
sed 's/^M$//' input.txt > output.txt # CRLF to LF
</source>
</source>

Latest revision as of 09:59, 14 October 2011

Everything about ascii file line termination issue (LF, CR/LF, CR...)

Reference

Miscellaneous

  • Normal Linux line ending is LF
  • Normal DOS line ending is CR-LF
  • In Bash, insert special character with ^V. For instance ^V ^M will insert a CR character
    • Alternatively, use something like r="$(printf "\r")",
    • Or use echo $'\r'

Detection

  • Hex dump
hd <file>
# 00000000  48 65 6c 6c 6f 0d 0a 48  65 6c 6c 6f 0d 0a 48 65  |Hello..Hello..He|
# 00000010  6c 6c 6f 0d 0a 48 65 6c  6c 6f 0d 0a              |llo..Hello..|
# 0000001c
  • Use file:
file crlf.txt cr_and_lf.txt lf.txt
# crlf.txt:       ASCII text, with CRLF line terminators
# cr_and_lf.txt:  ASCII text, with CRLF, LF line terminators
# lf.txt:         ASCII text
find . -print0 | xargs -0 file | grep CR                  # Search all files
  • Using GREP:
grep -IUr --color $'\r'                                   # Add -l to have list of files
find . -name "*.[ch]" -print0 | xargs -0 grep -IUl $'\r'  # Prefilter files to look at

Conversion

Reference: [1]

  • Using package dos2unix:
dos2unix <file>
unix2dos <file>
mac2unix <file>
unix2mac <file>

For instance, to fix all CR/CRLF issues, and remove also any trailing blanks (i.e. sanitizing all sources):

# Convert CRLF to LF
find . -name '*.[chCH]' -print0 | xargs -0 dos2unix
# ... twice in case we had "CRCRLF" in source files (from a bad LF->CRLF conversion)
find . -name '*.[chCH]' -print0 | xargs -0 dos2unix
# Convert CR to LF
find . -name '*.[chCH]' -print0 | xargs -0 mac2unix
# Remove trailing blanks
find . -name '*.[chCH]' -print0 | xargs -0 -n 1 sed -i -r 's/ +$//'


  • Using command tr:
tr -d '\r' < file            # Delete RETURN (i.e. CR, ^M, ascii 13)
tr -d '\n' < file            # Delete NEWLINE (i.e. LF, ^J, ascii 10)
  • Using Vim:
vim file
:set ff=unix
:wq
  • Using Perl:
perl -pi -e 's/\r\n/\n/g' input.file              # CRLF to LF
  • Using Sed:
sed 's/$/^M/" input.txt > output.txt              # LF to CRLF
sed 's/^M$//' input.txt > output.txt              # CRLF to LF