Remove duplicates values within a field

A sed one:

sed '
  s/[^[:blank:]]\{1,\}/,&,/g;:1
  s/\(\(,[^,[:blank:]]*\)\(,[^,[:blank:]]*\)*\)\2,/\1,/;t1
  s/,\([^[:blank:]]*\),/\1/g'

(it processes all the fields that contain , characters and preserves spacing)


With perl:

perl -MList::MoreUtils=uniq -pe 's{\S*,\S*}{join ",", uniq split ",", $&}ge'

(it processes all the fields that contain , characters and preserves spacing)


Another perl solution:

perl -anle '                                                                    
    print "$F[0] ", join ",", grep {!$seen{$_}++} split ",",$F[1];              
    %seen=();                                                                   
' file
A 1,2,3,45,8
B 5,6,2,3,7

Tags:

Awk