http://www.5169.info/software/teach-you-how-to-remove-duplicate-lines-from-text-files.html
http://d3s.mff.cuni.cz/~holub/sw/shash/#a1
http://ir.hit.edu.cn/phpwebsite/index.php?module=pagemaster&PAGE_user_op=view_page&PAGE_id=153