u/Head_Capital_7772

🔥 Hot ▲ 73 r/dataengineering

how to remove duplicates from a very large txt file (+200GB)

Hi everyone,

I want to know what is the best tool or app to remove duplicates from a huge data file (+200GB) in the fastest way and without hanging the laptop (not using much memory)

reddit.com
u/Head_Capital_7772 — 1 day ago