Did you ever report these issues to polars developers?

I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)

Yeah, was in the Discord for a while

Most of the time it works, but there are still corner cases throughout the python ML ecosystem where polars fails.

This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)

Rewriting pandas to polars is relatively trivial for most tasks these days. Especially if you wrote your pandas code correctly.

I still prefer (and use) Pandas for EDA. I think matplotlib integration is a better choice for most viz.

Also, I'm probably in the top 3-5 worldwide for number of folks I've trained with pandas. I offer Polars training and there is little demand for it.

Use pandas if you want advanced analytics, visualization, or ml.

Use SQL if you need to move data around.

How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???

And in the end save processed data in aws s3 in another format like iceberg or whatever

In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)

Those are generally rules of thumbs for data pipelines. (Plus duckdb can read Excel these days I think).

If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse).

I agree that the API has warts, though typically it is more concise than polars.

The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas.

[1]https://bjarkehautop.github.io/Website/blog/data-wrangling-t...

Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base.

This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python.

Lots of folks complain about it before using it. After they use it it is a non issue.

If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda).

Ive seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising.

Would love to see examples.

I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...

The text right above the code says why you can't...

edit:

Let me clarify. From the blog-post:

> since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it:

Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision.

I misspoke, you need to use .groupby/.transform to add a new filtering column:

    (sales
      .assign(country_median=lambda df_: (
          df_.groupby("country")["amount"].transform("median")
      ))
      .query("amount <= country_median * 10")
      .assign(net=pd.col('amount') - pd.col('discount'))
      .groupby("country", as_index=False)
      .agg(total=("net", "sum"))
    )

Yes, so basically equivalent to the code I showed in the blog.

I'm confused, you can use filter after a groupby in pandas...

It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.

Most folks just need to learn how to use pandas well and that will open enough doors. Then they can move to polars or duck if needed.

I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.)

Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).

If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.

Out of curiousity, why wouldn't someone just migrate directly to Polars?

I spent a few tokens last year trying to create a Python lib for native (undocumented) Resolve files. I was making progress, but it was slow (I ended up scripting Resolve in different ways). I imagine that current models could reverse-engineer the native file format relatively quickly.

Highly recommend the studio version. The ability to edit by transcript is worth the price alone.

I thought DaVinci Resolve could automatically sync camera footage for a few years now. I did it about 4 years ago.

Yes, correct, there’s a feature for that. Unfortunately one of the audio channels from the camera footage on the motorcycle is never good enough for this feature to work for me.