how to delete the repeat lines in emacs

In linux, select region, and type

M-| uniq <RETURN>

The result without duplicates are in new buffer.

Put this code to your .emacs:

(defun uniq-lines (beg end)
  "Unique lines in region.
Called from a program, there are two arguments:
BEG and END (region to sort)."
  (interactive "r")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (while (not (eobp))
        (kill-line 1)
        (yank)
        (let ((next-line (point)))
          (while
              (re-search-forward
               (format "^%s" (regexp-quote (car kill-ring))) nil t)
            (replace-match "" nil nil))
          (goto-char next-line))))))

Usage:

M-x uniq-lines

If you have Emacs 24.4 or newer, the cleanest way to do it would be the new delete-duplicate-lines function. Note that

• this works on a region, not a buffer, so select the desired text first
• it maintains the relative order of the originals, killing the duplicates

For example, if your input is

test
dup
dup
one
two
one
three
one
test
five

M-x delete-duplicate-lines would make it

test
dup
one
two
three
five

You've the option of searching from backwards by prefixing it with the universal argument (C-u). The result would then be

dup
two
three
one
test
five

Credit goes to emacsredux.com.

Other roundabout options, not giving quite the same result, available via Eshell:

1. sort -u; doesn't maintain the relative order of the originals
2. uniq; worse it needs its input to be sorted