Is there a way to 'uniq' by column?

sort -u -t, -k1,1 file

• -u for unique
• -t, so comma is the delimiter
• -k1,1 for the key field 1

Test result:

[email protected],2009-11-27 00:58:29.793000000,xx3.net,255.255.255.0 
[email protected],2009-11-27 01:05:47.893000000,xx2.net,127.0.0.1 

awk -F"," '!_[$1]++' file

• -F sets the field separator.
• $1 is the first field.
• _[val] looks up val in the hash _(a regular variable).
• ++ increment, and return old value.
• ! returns logical not.
• there is an implicit print at the end.

To consider multiple column.

Sort and give unique list based on column 1 and column 3:

sort -u -t : -k 1,1 -k 3,3 test.txt

• -t : colon is separator
• -k 1,1 -k 3,3 based on column 1 and column 3