Parallelize Pandas with Pandarallel
www.blog.dailydoseofds.com
Discover more from Daily Dose of Data Science
High-quality insights on Data Science and Python, along with best practices — shared daily.
Get a free 550+ page data science PDF guide and 450+ practice questions notebook.
Over 50,000 subscribers
Continue reading
Parallelize Pandas with Pandarallel
Pandas' operations do not support parallelization. As a result, it adheres to a single-core computation, even when other cores are available. This makes it inefficient and challenging, especially on large datasets.
"Pandarallel" allows you to parallelize its operations to multiple CPU cores - by changing just one line of code. Supported methods include apply(), applymap(), groupby(), map() and rolling().
Read more here: https://github.com/nalepae/pandarallel.