Remove duplicate line using regex

Remove duplicate line using regex

Copy from Stackoverflow

I have a list of html links, most of them are duplicate like this example below -->

I don't need same link twice, so I need to remove duplicate and keep only one link. How can I do this using regex? Or SED / AWK (I am not sure which technology would be best.) ? I am using ubuntu OS and for text editing sublime text 3 .


Answer 1

Very trivial using awk:

which basically means:

So if the line is not in the array it will add to it and print it. All subsequent lines if they exist in the array will be skipped.

Answer 2

Remove Duplicate Lines with Sublime's Permute Unique Lines

For long lists with duplicates, you can automatically delete them by using Permute Unique Lines from Sublime. To do that, go to "Edit - Permute Lines - Unique".

copy from?

任 江風


:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: