Professor John L. Renne

As an urban researcher or data scientist, you’ve probably relied heavily on Pandas for data manipulation. It’s been the go-to “Excel on steroids” program.

I’ve spent the past three years leaning Pandas but recently Polars has been catching my attention.

Polars, written in Rust, is a DataFrame library that’s gaining traction for being incredibly fast and efficient, especially with large datasets.

I recently listened to a couple fascinating episodes of the SuperDataScience Podcast. Most recently, in Episode 827, Ritchie Vink, the creator of Polars, shared the journey behind Polars, why it’s faster than Pandas, and where it’s heading. It’s definitely worth a listen if you’re curious about new developments in data science.

What Makes Polars So Special?

Polars can be 5-20x faster than Pandas in most cases—and sometimes even 100x faster. That’s because Polars processes data in columns, not rows, and is built entirely in Rust, a system-level language known for its speed and safety. The parallel processing capabilities in Polars mean it uses all the cores on your machine, unlike Pandas, which often only uses one. This alone is a game-changer if you’re working with big datasets.

From the SuperDataScience episode, I learned that Polars also uses Arrow, a memory model that makes sharing data between processes super efficient without copying or serializing it. This could be a big deal if you’re working with large urban datasets, like I often do. Whether it’s land use data, real estate data, transportation data, or demographic info, the ability to manage memory more effectively means faster, more efficient analysis.

Eager vs. Lazy Execution

One of the best features of Polars is its Lazy API. In Pandas, when you run a command, it executes immediately—this is called Eager execution. Polars offers that too, but what’s unique is the Lazy API, which waits to gather all the steps of your query before executing. The result? It optimizes the entire process, reducing unnecessary computations and saving time.

Think of it like this example from the show: if you’re making multiple trips to the kitchen, wouldn’t it be smarter to grab everything in one go instead of going back and forth? That’s what Lazy execution does—it gathers all the information before it does the work.

This has real benefits if you’re working with large datasets, something many urban researchers often face. Whether you’re analyzing transportation data or assessing real estate markets, the ability to delay execution and optimize the query could save a ton of processing time.

Scaling for Big Data

Polars isn’t just fast—it scales. In the SuperDataScience Podcast, they discussed how Polars plans to expand to handle even larger datasets through Polars Cloud, which will offer serverless, distributed data processing. Imagine running complex queries on datasets that don’t even fit in memory. That’s where Polars is heading, and for anyone dealing with big urban data, that’s important.

For researchers working with multi-gigabyte datasets, Polars’ ability to scale while staying efficient might be the key to deeper insights. Whether it’s sensor data or real-time traffic data, Polars helps you focus on the research, not the tool’s limitations.

Is Polars the Right Tool for You?

If you’re already using Pandas, you might be wondering if switching to Polars is worth the hassle. That’s a fair question. For many smaller datasets or quick, exploratory analysis, Pandas still works great. But if you’re working with larger datasets and need something faster, Polars might be worth the change. It handles bigger datasets, optimizes automatically, and it’s built for performance.

Have You Tried Polars Yet?

I’m curious to hear from others in the data science and urban research fields: Have you tried Polars? If so, how’s it been working out for you? Are you seeing the speed gains it promises, or are there any challenges you’ve run into? I’d love to hear your thoughts and experiences, especially if you’re comparing it directly to Pandas.

And if you haven’t yet, give Episode 827 of the SuperDataScience Podcast a listen—it goes into a lot more detail about what makes Polars so fast and what’s coming down the pipeline. You can also dive into the previous episode (Episode 815) with Marco Gorelli, another Polars developer, for even more insights.

Looking forward to hearing what you think—let’s keep this conversation going!

References:

• Super Data Science Podcast (Episode 827): Polars: Past, Present, and Future with Ritchie Vink – https://www.superdatascience.com/podcast/827

• Super Data Science Podcast (Episode 815): Polars: Faster DataFrame Ops with Marco Gorelli – https://www.superdatascience.com/podcast/815