Faster pandas, even on your laptop

By Prakhar Mishra

[Source]

Couple of weeks back, I came across this amazing library that scales up the existing pandas code by changing just one line of code and making it at least 2x faster compared to the existing. Seeing such big claims gave me a reason to test it out and see the results myself. This is the project i came across check it out!


I will be importing a 2 different datasets of different sizes to compare the performances for both the methods. 

Dataset 1


#!/usr/bin/env python
import time
import pandas as pd duration = []
for i in range(3): start = time.time() data_df = pd.read_csv( 'data.txt', sep = '\s\|\|\s' ) stop = time.time()-start duration.append(stop) del data_df final_time_pd = sum(duration) / float(len(duration))
print ('Average time for 3 runs is {} sec'.format(final_time_pd))
>>> Average time for 3 runs is 12.120 sec import time
import modin.pandas as pd duration = []
for i in range(3): start = time.time() data_df = pd.read_csv( 'data.txt', sep = '\s\|\|\s' ) stop = time.time()-start duration.append(stop) del data_df final_time_pd = sum(duration) / float(len(duration))
print ('Average time for 3 runs is {} sec'.format(final_time_pd))
>>> Average time for 3 runs is 6.515 sec
Clearly, Modin wins this case. Let's try with another dataset.

Dataset 2


I used the same code and re-ran the experiment.
>>> Average time taken for pandas is approx 111.723 seconds
>>> Average time taken for modin pandas is approx 71.770 seconds

P.S. Unfortuately, modin does not support read_table method as of now, that’s why I had to use read_csv.


Results are really impressive! This is surely going to help me in handling good amount of data now in pandas while making use of pandas magics with speed. Modin uses Ray to provide an effortless way to speed up your pandas notebooks, scripts, and libraries also at the same time gives seamless integration and compatibility with existing pandas code. It uses all 4 physical cores, whereas in pandas, you are only able to use 1 core at a time when you are doing computation of any kind.

No doubt! that this is some realy good contribution for Data Science / ML Enthusiasts. Kudos! Do give it a try at-least once for your use case.

Feel free to share comment your thoughts on the same. — Thanks