~hackernews | Bookmarks (2)

You are a researcher, potentially in the life or social sciences, and your organization offers you access to a fast machine with many CPUs and lots of memory, potentially as part of a CPU cluster. You are familiar with Linux and use Python/Pandas and/or R to manipulate and analyze your data. You sometimes use SQLite...

The basic principle of the shuffle algorithm is to bucket the input data, where each bucket’s data can be processed independently and output results. This reduces the size of the hash table when aggregating data.Taking the group operator as an example, there are two implementation methods: sort group and hash group....