Daily Dose of Data Science

Share this post

The Most Common Misconception About Inplace Operations in Pandas

www.blog.dailydoseofds.com

The Most Common Misconception About Inplace Operations in Pandas

...and here's what happens in reality.

Avi Chawla
Apr 27, 2023
17
2
Share
Share this post

The Most Common Misconception About Inplace Operations in Pandas

www.blog.dailydoseofds.com

Pandas users often modify a DataFrame inplace expecting better performance. Yet, it may not always be efficient. Here's why.

The image compares the run-time of inplace and non-in-place operations. In most cases, inplace operations are slow.

Why?

Contrary to common belief, most inplace operations DO NOT prevent the creation of a new copy. It is just that inplace assigns the copy back to the same address.

But during this assignment, Pandas performs some extra checks (SettingWithCopy) to ensure that the DataFrame is being modified correctly. This, at times, can be an expensive operation.

Yet, in general, there is no guarantee that an inplace operation is faster.

What’s more, inplace operations do not allow chaining multiple operations, such as this:

No method chaining with inplace operations.

Thanks for reading Daily Dose of Data Science! Subscribe for free to learn something new and insightful about Python and Data Science every day. Also, get a Free Data Science PDF (250+ pages) with 200+ tips.

👉 Read what others are saying about this post on LinkedIn and Twitter.

👉 If you liked this post, don’t forget to leave a like ❤️. It helps more people discover this newsletter on Substack and tells me that you appreciate reading these daily insights. The button is located towards the bottom of this email.

👉 If you love reading this newsletter, feel free to share it with friends!

Share Daily Dose of Data Science


Find the code for my tips here: GitHub.

I like to explore, experiment and write about data science concepts and tools. You can read my articles on Medium. Also, you can connect with me on LinkedIn and Twitter.

17
2
Share
Share this post

The Most Common Misconception About Inplace Operations in Pandas

www.blog.dailydoseofds.com
Previous
Next
2 Comments
Jack Nelson
Apr 27Liked by Avi Chawla

Big fan of method chaining. To achieve the same level of readability as the inplace=True, usually nest the chain in parentheses.

i.e.

(

df

.reset_index()

.fillna(0)

.drop_duplicates()

)

Expand full comment
Reply
1 reply by Avi Chawla
1 more comment…
Top
New
Community

No posts

Ready for more?

© 2023 Avi Chawla
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing