Rendered at 23:03:42 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
benrutter 15 hours ago [-]
> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
altern8 12 hours ago [-]
Aren't major versions supposed to indicate breaking changes..?
That's how I thought semantic versioning worked
theptip 8 hours ago [-]
Concretely, TFA lists a bunch of input validation that is being made more strict in the default configuration.
suddenlybananas 12 hours ago [-]
>The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience
I don't know how to read this sentence other than "there are breaking changes we want to make"
magicmicah85 11 hours ago [-]
The migration guide does say there are breaking changes, but the interpretation I have is "this won't have new features but allows us to develop new features".
> Polars adheres to the semantic versioning specification:
And it does have breaking changes in 2.0. The original asker presumably missed that.
E:
On the other hand, that whole page on versioning seems inconsistent.
nicce 14 hours ago [-]
> Version bumps should really be about removing deprecated cruft rather than shiny new features.
Can there be deprecated cruft without new features? :-D
kvdveer 12 hours ago [-]
Ideally: no.
All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.
In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
illiac786 3 hours ago [-]
Good question. I guess sometimes stuff becomes unnecessary due to external factors and not due to new features.
SkiFire13 12 hours ago [-]
Yes, the features don't need to be added immediately.
dist-epoch 14 hours ago [-]
That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.
esafak 1 hours ago [-]
A library is a collection of features, and any of them could have breaking changes. That's why semver is insufficient. It would be good to have a standardized way to indicate breaking changes in components, like changesets.
duskdozer 12 hours ago [-]
It sounds like they should be on a version much higher than 2.x then.
sanderjd 9 hours ago [-]
Deprecating without breaking is fine in a minor version under semver.
tialaramex 5 hours ago [-]
There's also always a "strictness" curve. Rust has specific rules where you could write something which doesn't break those rules but does break people who've done something very specific
At some point you reach the Hyrum's Law / Spacebar Heating edge where sure, there's no reasonable technical definition which concludes your change "broke" people but somebody's solution used to work and now it doesn't and they're angry about that.
To give an example that's only slightly less ridiculous than spacebar heating, maybe somebody depended on being able to parse a file from your library, you re-designed the library, there are no API changes but they're parsing your source code and now their approach doesn't work.
Generally, libraries with fewer and lighter users tend to be more lax, maybe they forgot to make a structure opaque, that's a break but none of their users notice, or they forgot to make an error type non-exhaustive, but everybody using their type had a default clause so no harm done when they add a new value.
Hyrum means that as you get bigger and more important, some day, the spacebar heating problem happens and too bad. A language (like Rust) can help you forestall that day somewhat, and reduce the blast radius when it does, but it is inevitable, Hyrum's law wasn't something you can outright prevent from happening, it's a factor for engineers to consider when designing systems, like misuse resistance.
sanderjd 2 hours ago [-]
Yeah. Agreed.
0x457 5 hours ago [-]
Did you miss "remove or change features." ?
sanderjd 2 hours ago [-]
No, I was just making a narrower point about one aspect of the comment.
For me, the superpower of polars is production stability.
Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.
Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.
I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!
Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
sigseg1v 9 hours ago [-]
Do people actually use languages where it doesn't do any compile time checks on the API in 2026? Why would developers put up with the lack of that. I'm not in this ecosystem but what you are describing sounds like the bare minimum to me that should be table stakes.
perrygeo 8 hours ago [-]
I agree. But the data science/engineering space is enamored with Python. Makes good sense when doing interactive work. Makes no sense in production. But by that point, most developers consider it a sunk cost and just keep their Python apps limping along rather than rewriting.
That's why polars is a great option. Start prototyping in python, then a relatively easy port to a Rust app when the need hits.
maleldil 8 hours ago [-]
You can still have extensive "compile-time" checking with linters and static type checkers. This makes Polars even better in typed Python, as it type-checks much better than the dynamic mess that is Pandas.
perrygeo 8 hours ago [-]
Polars is much better than pandas in this regard. But using it via Python still carries the risk of runtime errors.
Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
paddy_m 13 minutes ago [-]
When I deal with datascience, I rarely have type problems in my programs that python typecheckers can't catch. I do have type and dirty data problems of the dataframes/csvs/parquet I'm reading.
trombonechamp 12 hours ago [-]
Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g.
https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.
jorelfermin 3 minutes ago [-]
Unfortunately (for your case) the ordering of group_by, join, and unique all run in parallel hash aggregation across the threads so the output order comes about by how it gets partitioned across cores. Which is why you can get different order of rows depending on the machine even when you have the same set of data and polars version (this has happened to me). To fix you can set maintain_order=True or probably better an explicit sort whereever you save or compare / diff the output.
nemothekid 9 hours ago [-]
If the correctness of my program depends on the ordering of data (for operations that would otherwise be commutative), that seems like that should be something explicit rather than implicit.
I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
hopfenspergerj 10 hours ago [-]
It's standard sql behavior, users always specify the ordering they want as part of the query.
Xunjin 8 hours ago [-]
Often people new to databases and SQL thinks ordering is implicit, I have teach dozens of juniors that believe this is the default. I always wondered why that is the thought process...
fwip 8 hours ago [-]
Because out of the first 100 queries they ran, 99 came back in order and they didn't notice that the 100th didn't.
dash2 7 hours ago [-]
I once persuaded the dplyr maintainers not to do an update that might re-order rows after a filter(). I think the human tendency to think of database rows as existing in a fixed, given order, which will only be changed explicitly, is deep.
Permik 10 hours ago [-]
This is a tricky field, the problem is not actually the non-determinism of the processing algorithms, but implicit ordering of the data.[1] The implicit ordering of the data is a footgun that -- as seen in the paper -- has already claimed victims.
Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
winniewinnie 9 hours ago [-]
By "implicit ordering", do you mean "implicitly assumed that the data is ordered a certain way"? Since if that assumption of data being sorted a certain way is broken on some systems and not others, the result might be both non-deterministic (which could be a bug if the result is not allowed to be non-deterministic, but may or may not be a bug regarding the algorithm's assumptions) as well as a bug if the algorithm's assumptions requires it to be sorted a certain way.
> Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
Permik 3 hours ago [-]
Yeah, by implicit ordering, I mean an underspecified pipeline which has an undeclared dependency on some of the inputs being sorted. Unfortunately this is why the code in the cited paper worked on some systems and not others as the os.listdir() call didn't guarantee sorted results and thus the assumed invariants were broken.
Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.
Lvl999Noob 7 hours ago [-]
> How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.
sanderjd 9 hours ago [-]
Is "non-deterministic" the right description for this? I read it as describing an implementation where ordering is not preserved, but deterministically. Is that a misreading?
dhampi 8 hours ago [-]
For example, polars internally hashes rows for some operations in ways that affect the ultimate ordering.
They do not guarantee stability of hashing algorithm or seed across versions and platforms.
(No complaints here! I agree ordering shouldn’t matter unless you make it explicit.)
0cf8612b2e1e 9 hours ago [-]
This behavior has repeatedly frustrated me. I am writing some new transformation, want to see the results, and my first few sentinel rows are nowhere to be seen because they have been shuffled.
I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
geysersam 5 hours ago [-]
Why not slap an order_by on the end of the query and you're all good? If that's expensive maybe you can add a limit clause to your source.
0cf8612b2e1e 4 hours ago [-]
Sure, I can change what I get, I just disagree that my ordered table is not guaranteed to stay ordered.
Neither R nor Pandas will shuffle results by default.
winniewinnie 11 hours ago [-]
Perhaps to get better results on benchmarks.
Bootvis 10 hours ago [-]
Or you know, just better performance for people that know how to use their tool of choice.
winniewinnie 10 hours ago [-]
Wouldn't such people just pass in
maintain_order=False
?
Bootvis 9 hours ago [-]
That’s an API design question. What is the more common use case.
You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
bobson_dugnutt5 14 hours ago [-]
I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.
anotherpaul 14 hours ago [-]
I gave up pandas in favor of polars after someone at work did the same and I am very happy with it. Pandas API is just so much worse and much slower.
winniewinnie 10 hours ago [-]
Have Polars' inconsistent versioning policy caused you any problems?
duskdozer 12 hours ago [-]
I guess I am a casual pandas user only. Reading a guide on migrating/differences, it's hard to see why polars would be obviously better.
bobson_dugnutt5 11 hours ago [-]
Here are a couple reasons:
- much faster, multithreaded by default. Read in a big csv with it and see how it feels.
- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.
- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.
- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.
Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much
0cf8612b2e1e 7 hours ago [-]
End of the day, they both get the job done.
Pandas is more ergonomic in that some ideas can be more tersely represented. The downside is that this results in more dynamism which can change if the underlying data gets updated. Polars is more strict in that it will not silently flip a data type on you. However this strictness does come at the cost of being a bit slower to type and some data idioms not having a good Polars equivalent.
People like to note the speed improvements, but that is the least interesting thing about the library. Rarely have I ever had a problem where I was bottlenecked by Pandas throughout.
Polars is very much a Pandas 2.0 with a bunch of lessons learned. I do not think it is earth shattering changes, but it is worth migrating when you can.
latexr 11 hours ago [-]
Taken out of context, your post looks like a conservationist who got fed up with pandas being a flagship species and made it their lifelong mission to replace them with polar bears.
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
bobson_dugnutt5 4 hours ago [-]
Thank you, your the only person to correctly interpret my comment :)
blitzar 11 hours ago [-]
You should learn Boa constrictor instead of Python.
mgaunard 14 hours ago [-]
Both have terrible syntax that make SQL look like the most readable thing ever.
condwanaland 14 hours ago [-]
Could not agree less. Ive always found SQL an unreadable mess but tools like polars and dplyr are such elegant ways to manipulate data.
Pandas is a mess though.
world2vec 13 hours ago [-]
There's no way SQL is more unreadable than polars. IMO it's the other way around.
benrutter 12 hours ago [-]
> There's no way SQL is more unreadable than polars. IMO it's the other way around.
I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
sanderjd 9 hours ago [-]
I would say that it is easier to decompose polars (and all dataframe api) queries and to build them up from pieces than it is to do the same with sql. Any time I find myself writing more than five or so lines of sql, or especially building a sql string in parts with logic, I wish I had a dataframe api instead. But the reverse is also somewhat true, that simpler and explicit expressions are nicer with sql.
aquafox 14 hours ago [-]
Coming from an R/dplyr background, I agree. Compare
df.select(
pl.col("x"),
(pl.col("w")/pl.col("z")).alias("y")
)
with
df |>
select(x, y = w/z)
orlp 13 hours ago [-]
from polars import col as C
df.select(C.x, y = C.w / C.z)
__mharrison__ 9 hours ago [-]
This is the way. Favor keyword arguments to alias.
dkga 12 hours ago [-]
Still, it’s a very good approximation but still an approximation to the more ergonomic and expressive tidyverse syntax
sanderjd 9 hours ago [-]
One person's "ergonomic and expressive" is another person's "wait what in the world is actually going on here".
jcattle 13 hours ago [-]
R really is/was the superior traditional data science language. Python ecosystem is slowly catching up though.
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
vovavili 12 hours ago [-]
Altair and Positron should be just as good for your Polars @ Python needs. With software like Marimo notebooks and VegaFusion, Polars/Python experience starts beating R by quite a substantial margin.
bobson_dugnutt5 14 hours ago [-]
Fair point, but you can do something like
`df.select("x", y=pl.col.w/pl.col.z)`
sanderjd 9 hours ago [-]
To me, I immediately wonder whether w, x, y, and z here are variables or column names. It would indeed be nice if python could more tersely represent the distinction between a name and a literal string (or worse, as in your R example, a variable reference), but alas. But I think trading some verbosity for explicitness about this distinction is a pretty good trade, and very in keeping with python style.
countrymile 11 hours ago [-]
Polars is a world away from pandas, but I feel that dplyr still offers the most simple and understandable introduction to data analysis for the beginner. The above is a good example of this.
geysersam 12 hours ago [-]
I agree sql is more elegant. The problems arise when you have to add logic on top of sql. Often I end up constructing queries via string manipulation and that is not very ergonomic. Polars api is more verbose and complex than sql but at least it's not meta-programming.
The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.
vovavili 10 hours ago [-]
You should be using dbt instead of string manipulation for serious query building.
sanderjd 9 hours ago [-]
It's never quite been clear to me what the advantage of dbt over a python program using sqlalchemy / duckdb / polars to transform data is. Can you enlighten me?
vovavili 7 hours ago [-]
At the minimum, it's just Jinja2 templates in your SQL queries - meaning, you can do pure SQL transformations with conditional logic in your templates. In addition to being able to run tests, specify custom macros, having version control and having some constrained way to organize your tables, you're turning SQL into a proper programming language with just one library.
throwaway7783 4 hours ago [-]
My issue with DBT is it is a mix of SQL, yaml, jinja2 flow controls (and metrics is whole another thing). SQL with jinja2 if/else can get really unmaintainable quickly. It's perhaps better than homegrown sql based transformers.
polars is code and can be version controlled too. Dataframes in my opinion are more elegant, and with the right backends and some lineage enhancements, could serve a much wider set of use cases than what DBT does
vovavili 4 hours ago [-]
Different tools for different tasks.
sanderjd 2 hours ago [-]
But is that ... good?
geysersam 7 hours ago [-]
I've been looking at dbt for exactly this reason but don't quite get the advantages if you're not interacting with a data warehouse of some sort.
vovavili 6 hours ago [-]
Embedded databases like DuckDB are just as suited for dbt work.
bobson_dugnutt5 14 hours ago [-]
What is it about polars syntax you don't like? The fact that is very verbose? At first I wasn't a fan, but over time I've grown to really like it. That never happened to me with pandas, always felt the syntax was messy
mihaelm 12 hours ago [-]
The verbosity takes a bit to get used too, but it sure beats the anything-goes feeling - messy as you put it - of pandas.
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
fzumstein 14 hours ago [-]
I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.
refactor_master 14 hours ago [-]
The cool thing about polars is that you can conditionally collect expressions over many layers of business logic, and then compute the result at the end. Doing this in SQL ends up in a hodgepodge of strings and trimmed ends to please the syntax. You can also pretty effortlessly write quite complex conditionals directly in polars, and bridge it easily to the surrounding python.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
fzumstein 13 hours ago [-]
This sounds interesting! Do you have a specific example by any chance or blog post/doc references?
refactor_master 12 hours ago [-]
It’s just the lazy/expression part of the API, which is really the bread and butter of polars, rather than just being “replacement syntax” for pandas. This allows you to tap into abstraction that SQL can’t keep up with:
WITH raw_data AS (
SELECT * FROM (
VALUES
('S01', 5000.0, 0.45, 120, 5),
('S02', 2400.0, 0.30, 45, 12),
('S03', 15000.0, 0.60, 300, 45),
('S04', 900.0, 0.15, 20, 2),
('S05', 3200.0, 0.50, 85, 8)
) AS t(store_id, revenue, margin, tx_count, returns)),
base_data AS (
SELECT
store_id,
revenue,
margin,
CAST(tx_count AS DOUBLE) AS tx_count,
CAST(returns AS DOUBLE) AS returns
FROM raw_data
)
SELECT
store_id,
revenue,
margin,
CAST(tx_count AS BIGINT) AS tx_count,
CAST(returns AS BIGINT) AS returns,
-- KPI Layer
returns / tx_count AS return_rate,
revenue / tx_count AS avg_order_value,
-- Threshold Layer (matching original alias names)
revenue > 1000.0 AS is_revenueabove1000,
revenue > 5000.0 AS is_revenueabove5000,
revenue > 10000.0 AS is_revenueabove10000,
tx_count > 50 AS is_tx_countabove50,
tx_count > 100 AS is_tx_countabove100,
tx_count > 200 AS is_tx_countabove200,
-- Interaction Layer (preserving exact numeric formula & aliases)
revenue / (margin + 1e-5) AS ratio_revenuepermargin,
revenue / (tx_count + 1e-5) AS ratio_revenuepertx_count,
revenue / (returns + 1e-5) AS ratio_revenueperreturns,
margin / (tx_count + 1e-5) AS ratio_marginpertx_count,
margin / (returns + 1e-5) AS ratio_marginperreturns,
tx_count / (returns + 1e-5) AS ratio_tx_countperreturns,
-- Segmentation Layer
CASE WHEN margin > 0.4 THEN 'High' ELSE 'Low' END AS margin_profile
FROM base_data;
throwaway7783 4 hours ago [-]
Well, I prefer the polars version. And now if I want to reuse the CTEs elsewhere, I have to reach out to another tool like DBT or hand roll something to do string manipulation.
sanderjd 9 hours ago [-]
... does it? I don't think it does, even in this form.
And now write it such that all the conditions and transformations are injected into the string (somehow) rather than written in explicitly. Much worse.
bobson_dugnutt5 7 hours ago [-]
Really? You've written out all the ratios and thresholds manually. If a user wanted to change the set of thresholds the polars way is far superior. In what way do you consider this better?
lmeyerov 10 hours ago [-]
Moving towards streaming and generally out-of-core is great
We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
Kydlaw 12 hours ago [-]
Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL.
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
f311a 12 hours ago [-]
I've been using clickhouse-local for quite some time, instead of DuckDB.
There is also chDB.
After using pandas for 10 years, I favor SQL now, for some reason.
Kydlaw 8 hours ago [-]
I haven't tried chDB yet, but I heard about it. Thank you for reminding me of that option.
I use SQL in data pipelines and processing that is going to require interoperability.
But for data exploration, I usually prefer Polars (imo it is easier to work with text, semi-structured data, etc.)
mihaelm 12 hours ago [-]
Maybe because it's like a swiss army knife for data work, regardless of whether you need it for OLTP or OLAP workloads. Having different SQL dialects is a bit annoying, but the base is the same more or less, so switching doesn't come at too big of a cost.
geysersam 5 hours ago [-]
Interesting, why do you typically prefer clickhouse local to duckdb?
f311a 4 hours ago [-]
Because we use regular CH in production. I'm pretty familiar with it and I can always run large queries on a production server with 1TB of RAM.
Regular CH also support external data sources, so I can read 500GB of JSON from S3 and group by it on production server very fast and in memory.
efromvt 12 hours ago [-]
As a huge duckdb fan, I'd love to see chDB to get proper windows support - that would make it real competition (having WASM coverage is already a big step) which would be good for the space as a whole.
dtj1123 4 hours ago [-]
> as soon as their ready.
So strange that it's now relatively normal to see a typo and think 'oh cool, a human wrote this, I can take this seriously' rather than 'oh dear, they can't spell, I can't take this seriously'.
bbstats 8 hours ago [-]
Bothering me like crazy that "Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for categorical → int." doesn't give the requisite code example!
You can find it in the migration guide. Let me know if you miss anything, if you'd like you can make an issue and I'll make sure to get to it before the 2.0 release!
arn3n 13 hours ago [-]
The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?
orlp 13 hours ago [-]
Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
sanderjd 9 hours ago [-]
When you say "in-cache batches", you mean that this cache is on disk? Is that only the case when data is quite large?
(Or a more general question: What is the best resource for me to read about how the streaming engine and cache work?)
orlp 8 hours ago [-]
Well... once my recent work on out-of-core lands the batch could be on disk when we run out of memory budget ;)
But no, that's not what I meant. I meant that the batch is meant to be of a size that fits in your CPU cache. This can be a huge throughput improvement as each bit of data stays in cache as it moves from data source to sink.
Compare this to column-at-a-time execution: by the time you start the next operation on this column the start of the column will be out of cache again, meaning you operate at RAM speed (or worse, disk speed) rather than cache speed.
Aaaaah, the cpu cache aspect is what I was missing! This makes tons of sense! I'll check out your talk as well.
arn3n 12 hours ago [-]
Cool, thanks for the explanation!
rfgplk 14 hours ago [-]
[flagged]
thibaut_barrere 14 hours ago [-]
I like when large projects do that. This gives leeway for sister projects (eg wrappers) to anticipate, room for apps that use it intensively to test things out (release candidate etc), something which has really helped me in the past.
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.
rfgplk 11 hours ago [-]
I was referring to the "land" verb choice, aka a clear Claudism. In fact looking at it more carefully, the whole post seems to be heavily AI written with minimal human intervention.
Philpax 10 hours ago [-]
Actually, I would say the exact opposite. This post is full of strange and grammatically incorrect phrases, weird paragraph pacing, and unintuitive clauses: that is to say, this reads as very strongly human-written to me, and it is refreshing.
marliechiller 14 hours ago [-]
What is your understanding of a Pre-Release then?
dbdr 14 hours ago [-]
Why?
irpap 14 hours ago [-]
I assume because “land” is a word Claude would choose.
mgaunard 14 hours ago [-]
Claude's wording (and knowledge) is based on what competent senior engineers would say.
blitzar 11 hours ago [-]
I really hope I don't have to work with these "competent senior engineers" anytime soon — my codebase couldn't handle it.
On the plus side I would spend all day hearing I am "absolutely right" from a superior being.
tecleandor 14 hours ago [-]
None of the competent senior engineers I know say "not this, but that" every two sentences.
pepperoni_pizza 11 hours ago [-]
But what about load-bearing seams that have provenance?
tancop 13 hours ago [-]
It's the only good word here. "Drop" can also mean the opposite and anything else sounds too formal. Don't get me started on "release will release".
gjm11 11 hours ago [-]
Well, they could have written something like "The 2.0 release itself will be in the next few weeks" or "It will be a few weeks before the actual 2.0 release".
(But I agree that "land" is fine here, and the rest of TFA doesn't strike me as obviously AI-written. And I'm not a fan of the "look, they did one thing that AIs often do! Must be AI and therefore bad!" thing in any case.)
rfgplk 11 hours ago [-]
"Polars 2.0 will be released (publicly) in the following weeks" sounds far more natural.
rfgplk 11 hours ago [-]
Clear Claudism. It wants to "land" everything, everywhere.
NSPG911 12 hours ago [-]
release will be [released] in the following weeks
luciana1u 11 hours ago [-]
every major version of polars is a reminder that the API you finally memorized was always just a suggestion
bluebarbet 13 hours ago [-]
What does this project have to do with Serbia? Are the developers in Belgrade?
mkl 12 hours ago [-]
It's just a play on the name, and it's pretty common. claude.ai has nothing to do with Anguilla, John Romero's rome.ro has nothing to do with Romania, twitch.tv has nothing to do with Tuvalu, etc.
bluebarbet 11 hours ago [-]
Indeed, and Bit.ly has nothing to do with Libya, nor Lemmy.ml with Mali (both failed states). I posit that domain hacking is an ugly, shortsighted, unserious habit that we should drop.
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
That's how I thought semantic versioning worked
I don't know how to read this sentence other than "there are breaking changes we want to make"
https://docs.pola.rs/development/versioning/
> Polars adheres to the semantic versioning specification:
And it does have breaking changes in 2.0. The original asker presumably missed that.
E:
On the other hand, that whole page on versioning seems inconsistent.
Can there be deprecated cruft without new features? :-D
All new shiny new features shouldn't have waited for the (N+1).0 version, they should already have been part of the (N).(M) version.
In practice, the removing the deprecated cruft will remove blockers for some new features, but that should be rare.
At some point you reach the Hyrum's Law / Spacebar Heating edge where sure, there's no reasonable technical definition which concludes your change "broke" people but somebody's solution used to work and now it doesn't and they're angry about that.
To give an example that's only slightly less ridiculous than spacebar heating, maybe somebody depended on being able to parse a file from your library, you re-designed the library, there are no API changes but they're parsing your source code and now their approach doesn't work.
Generally, libraries with fewer and lighter users tend to be more lax, maybe they forgot to make a structure opaque, that's a break but none of their users notice, or they forgot to make an error type non-exhaustive, but everybody using their type had a default clause so no harm done when they add a new value.
Hyrum means that as you get bigger and more important, some day, the spacebar heating problem happens and too bad. A language (like Rust) can help you forestall that day somewhat, and reduce the blast radius when it does, but it is inevitable, Hyrum's law wasn't something you can outright prevent from happening, it's a factor for engineers to consider when designing systems, like misuse resistance.
Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.
Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.
I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!
Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
That's why polars is a great option. Start prototyping in python, then a relatively easy port to a Rust app when the need hits.
Python's linters leave many holes unfilled - it catches some stuff but still leaves everything else to the runtime. Literally every Python application that I've worked on in the last 3 years a) is 100% type checked yet b) still hits massive numbers of novel runtime errors on the regular. Python's anemic type system is in no way a substitute for an advanced compiler.
I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
> Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
How would the library "make future data modeling ... more explicit" if this is a change to a default, which is implicit?
Requiring code to be explicit about its invariants by default is just plain good design and makes things more robust. And when being more exact about invariants, you can reap the algorithmic benefits.
It would become more explicit because where the order matters, you will now see a `maintain_order=True` where previously, you couldn't say whether it actually needed the fixed order or not.
They do not guarantee stability of hashing algorithm or seed across versions and platforms.
(No complaints here! I agree ordering shouldn’t matter unless you make it explicit.)
I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
Neither R nor Pandas will shuffle results by default.
You seem to suggest they did it for benchmarking reasons only. They could use the option there themselves without changing the default so that is unlikely to be the motivation.
- much faster, multithreaded by default. Read in a big csv with it and see how it feels.
- no index/MultiIndex. Pandas special treatment of index always felt like more trouble than it was worth, so no need to reset_index() everywhere.
- expressions are very portable. At first using pl.col everywhere feels like a bit much, but you can define them anywhere and then apply them to a dataframe whenever you want.
- once internalized, the syntax makes much more sense and is far more consistent compared to pandas.
Of course all depends on what your use cases are. If performance is important then I'd strongly recommend trying it out. If you just use it to have a look at the odd dataframe, maybe not worth your time as much
Pandas is more ergonomic in that some ideas can be more tersely represented. The downside is that this results in more dynamism which can change if the underlying data gets updated. Polars is more strict in that it will not silently flip a data type on you. However this strictness does come at the cost of being a bit slower to type and some data idioms not having a good Polars equivalent.
People like to note the speed improvements, but that is the least interesting thing about the library. Rarely have I ever had a problem where I was bottlenecked by Pandas throughout.
Polars is very much a Pandas 2.0 with a bunch of lessons learned. I do not think it is earth shattering changes, but it is worth migrating when you can.
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
Pandas is a mess though.
I think on basic queries, SQL is really nice, but when stuff gets more complex, with a bunch of CTEs, let alone functions requiring loops, it becomes pretty obtuse.
df.select(
)with
df |> select(x, y = w/z)
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
`df.select("x", y=pl.col.w/pl.col.z)`
The duckdb python api is okay, but it is a bit limited, no ctes, no as of join, and it can be slow at bind/interpretation time when you do stuff like unioning multiple relations in a loop (I think that becomes O(N^2), but I might be wrong). Most issues can be worked around, but Polars is designed from the ground up to be used from python.
polars is code and can be version controlled too. Dataframes in my opinion are more elegant, and with the right backends and some lineage enhancements, could serve a much wider set of use cases than what DBT does
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
And now write it such that all the conditions and transformations are injected into the string (somehow) rather than written in explicitly. Much worse.
We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
After using pandas for 10 years, I favor SQL now, for some reason.
I use SQL in data pipelines and processing that is going to require interoperability.
But for data exploration, I usually prefer Polars (imo it is easier to work with text, semi-structured data, etc.)
Regular CH also support external data sources, so I can read 500GB of JSON from S3 and group by it on production server very fast and in memory.
So strange that it's now relatively normal to see a typo and think 'oh cool, a human wrote this, I can take this seriously' rather than 'oh dear, they can't spell, I can't take this seriously'.
You can find it in the migration guide. Let me know if you miss anything, if you'd like you can make an issue and I'll make sure to get to it before the 2.0 release!
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
(Or a more general question: What is the best resource for me to read about how the streaming engine and cache work?)
But no, that's not what I meant. I meant that the batch is meant to be of a size that fits in your CPU cache. This can be a huge throughput improvement as each bit of data stays in cache as it moves from data source to sink.
Compare this to column-at-a-time execution: by the time you start the next operation on this column the start of the column will be out of cache again, meaning you operate at RAM speed (or worse, disk speed) rather than cache speed.
I gave a (fairly surface-level) talk on the streaming engine a bit over a year ago: https://pola.rs/posts/talk-polars-meetup-1-streaming-engine/.
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.
On the plus side I would spend all day hearing I am "absolutely right" from a superior being.
(But I agree that "land" is fine here, and the rest of TFA doesn't strike me as obviously AI-written. And I'm not a fan of the "look, they did one thing that AIs often do! Must be AI and therefore bad!" thing in any case.)