pandas 3.0.0rc2 took 3402.5ms to write a CSV

With pandas 3.0.0rc2, a 2,000,000-row DataFrame had a median to_csv time of 3402.5ms and a median read_csv time of 882.7ms.

The heavy part was not the aggregation itself but CSV input and output. groupby finished in 238.1ms and sort_values in 470.0ms, while CSV writing was 3.9 times the read time. The exit was heavier than the processing. If you optimize the wrong place, polishing the aggregation will not change the experience.

The lighter-than-expected part was the processing on a DataFrame this large. value_counts took 6.1ms, and filtering then summing took 26.1ms. On a Surface Pro 11th Edition with a Snapdragon X Elite X1E80100, pandas itself was not the bottleneck in this part.

The measurement date was July 22, 2026. Python was the ARM64 build; pandas was 3.0.0rc2, NumPy 2.4.1, and matplotlib 3.10.8. pandas 3.0.0rc2 is a release-candidate build, so it needs care before treating these as stable-release representative values. This is a record of what this environment looked like on 2026-07-22.

Measuring the DataFrame and files in one flow

I created a 2,000,000-row, three-column DataFrame and ran the same operations several times to look at medians. The columns were a simple category-like column, a value to aggregate numerically, and a column used for sorting. I measured groupby, sort_values, value_counts, a filtered sum, CSV writing, CSV reading, and matplotlib PNG saving. It was a rough side-by-side of operations that often appear together in work.

The core of the script looked like this.

cases = {
    "groupby_mean": lambda: df.groupby("key")["value"].mean(),
    "sort_values": lambda: df.sort_values("score"),
    "value_counts": lambda: df["key"].value_counts(),
    "filter_sum": lambda: df.loc[df["score"] > 0.5, "value"].sum(),
}

I started with a simple measurement and planned to think about why CSV alone stood out afterward. Taking side-by-side numbers from the same DataFrame first helps prevent choosing a culprit based only on intuition.

For CSV, I wrote to a file and read the same file back. Turning it into an in-memory string would have departed from the intermediate-file workflow I actually use, so this run went through a file. The goal was to see the wait that happens locally, not to make a clean benchmark.

df.to_csv(csv_path, index=False)
pd.read_csv(csv_path)

For PNG, matplotlib drew a line chart and saved it. I did not retain the number of points in the JSON, so this article looks only at PNG output time and size. Adding that later would mix the source of the numbers.

Only CSV reached seconds

Here are all repetitions. To avoid making everything look tidy through medians alone, I am leaving them intact. csv_bytes and png_bytes are shown as the sizes of the same measurement files even for cases that did not use one of those operations. rows is the number of DataFrame rows, not the number of matplotlib points.

The CSV was 50342651 bytes, about 50.3MB in decimal units. Writing it in 3402.5ms works out to about 14.8MB/s. Reading took 882.7ms, or about 57.0MB/s.

That difference makes it hard to blame sequential disk-write speed itself. It looks more like the CPU work of converting numbers and strings to CSV text and emitting separators and line breaks is dominant than the SSD being slow. From the disk-write perspective too, 14.8MB/s felt unusually low.

String conversion is heavier than aggregation

The 6.1ms for value_counts is very light. It examines 2,000,000 rows yet barely waits. The filtered sum took 26.1ms, so even adding a small amount of preprocessing would not approach the 3402.5ms for CSV writing.

groupby_mean took 394.9ms on the first run and 238.1ms and 223.3ms afterward. Caching or internal warm-up probably had an effect, but it was still 8.6 to 15.2 times faster than CSV writing. sort_values varied from 470.0ms to 563.1ms to 433.3ms. I expected sorting to be among the heavier operations, but in this range it looked light beside to_csv.

I had thought of CSV as a slow but safe intermediate format. With these numbers, carrying intermediate data through CSV repeatedly means paying a tax measured in seconds each time. You can polish small operations as much as you like and still end up waiting at the final write.

read_csv is not free either. At 882.7ms it may not matter a few times, but dozens of reads in a batch will add up. Writing stood out most here: 3402.5ms is long, and combined with being 3.9 times the read time, it makes me reluctant to add CSV output casually.

parquet was planned but failed

I intended to measure parquet too, but I want to keep the failure. I expected it to be smaller and faster than CSV, but neither pyarrow nor fastparquet was installed in this environment, so the run stopped with an ImportError.

The error said that pandas could not find an available engine for parquet. It showed that importing pyarrow failed and that fastparquet was also required. As a result, the table cannot include to_parquet time or parquet file size.

This is a candidate for a rerun. Installing pyarrow would likely make it measurable, but this time I wanted to see how far the plain environment could go before adding dependencies. Therefore parquet is recorded as a failure, not as either “fast” or “slow.”

There is a practical lesson here: even if you want to escape CSV, another format becomes an option only together with its dependency packages. It is not necessarily a format that pandas alone always provides. Scripts and reproduction instructions need to name those dependencies, or the result may simply fail in someone else’s environment.

matplotlib saving also accumulates

Matplotlib PNG output had a median of 120.1ms. That is not CSV-writing territory, but a batch that emits many graphs cannot ignore it. The file was only 34669 bytes, so the image itself was not large. Drawing and saving still took time in the 100ms range.

One image is not a concern. A batch generating many graphs is different. PNG saving is not free, even if it is nowhere near 3402.5ms for CSV. Looking only at file size would be misleading: it is tempting to call 34669 bytes light, but the measured time was 120.1ms.

Stop passing intermediate data around as CSV

The operational conclusion is simple: consider stopping the cycle of repeatedly saving and rereading intermediate data as CSV.

It is fine to keep CSV as a final deliverable for people. It opens in Excel and is easy to use for checking differences. But for a temporary file used only to connect Python operations, there is less reason to choose CSV. In this measurement, reducing the number of CSV passes would help more than removing groupby or sort_values.

Next I would install pyarrow and measure parquet again. I want to check how much smaller it is than the 50.3MB CSV and how far its write time falls from 3402.5ms. I would also check in another run whether the same tendency appears in a stable pandas release rather than pandas 3.0.0rc2.

I did not get the impression that pandas was slow because this was Windows ARM64. The DataFrame operations were reasonably fast; CSV string conversion was where it got stuck. Now that I have seen that, I will question the file format before the formula when writing the next batch.