Python Performance

Optimising without measuring is guessing. This lesson measures first — timeit, cProfile, line_profiler — then tries the usual fixes on real timings. Every timing below is from one run of CPython 3.13 on a 4-core Linux machine (NumPy 2.5, line_profiler 5.0); yours will differ, and so will a second run here. Compare the ratios, not the absolute numbers.

timeit: timing small snippets

timeit runs a statement many times and returns the total; repeat does several such trials.

import timeit

setup = "data = list(range(10_000))"
loop = """
total = 0
for x in data:
    total += x
"""

t_sum = min(timeit.repeat("sum(data)", setup=setup, number=1_000, repeat=5))
t_loop = min(timeit.repeat(loop, setup=setup, number=1_000, repeat=5))

print(f"sum():    {t_sum * 1000 / 1000:.3f} ms per call")
print(f"for loop: {t_loop * 1000 / 1000:.3f} ms per call")
print(f"loop is {t_loop / t_sum:.1f}x slower")

$ python timeit_sum.py
sum():    0.077 ms per call
for loop: 0.270 ms per call
loop is 3.5x slower

Both add the same 10,000 numbers. sum() runs its loop in C; the for loop executes several bytecode instructions per element. Taking the minimum of five trials discards runs that were slowed by something else on the machine. From a shell, python -m timeit -s "data = list(range(10_000))" "sum(data)" does the same.

cProfile: where does the time go?

A profiler counts calls and time per function, so you optimise the right one.

import cProfile
import pstats

def parse(line: str) -> list[int]:
    return [int(x) for x in line.split(",")]

def is_valid(row: list[int]) -> bool:
    return sum(row) % 7 != 0

def process(lines: list[str]) -> int:
    return sum(1 for line in lines if is_valid(parse(line)))

lines = [",".join(str(i * j) for j in range(20)) for i in range(20_000)]

profiler = cProfile.Profile()
profiler.enable()
process(lines)
profiler.disable()
pstats.Stats(profiler).sort_stats("tottime").print_stats(4)

$ python profile_demo.py
         97146 function calls (77146 primitive calls) in 0.102 seconds

   Ordered by: internal time
   List reduced from 7 to 4 due to restriction <4>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    20000    0.065    0.000    0.074    0.000 /home/user/py-demo/performance/profile_demo.py:4(parse)
    17143    0.014    0.000    0.100    0.000 /home/user/py-demo/performance/profile_demo.py:11(<genexpr>)
    20000    0.009    0.000    0.009    0.000 {method 'split' of 'str' objects}
  20001/1    0.009    0.000    0.102    0.102 {built-in method builtins.sum}

parse dominates tottime — the time spent in its own code, the list comprehension of int() calls — so that’s the function worth optimising, not is_valid. cumtime includes callees, which is why the generator expression’s cumtime covers nearly the whole run. (Its 17,143 calls are one per resumption — once for each of the 17,142 valid rows it yields, plus the final stop.) Profiling adds overhead, so treat the absolute times as approximate.

line_profiler: line by line

Once you know the function, line_profiler (pip install line_profiler) shows which line inside it is slow.

from line_profiler import LineProfiler

def process_items(items):
    results = []
    for item in items:
        cleaned = item.strip().lower()
        words = cleaned.split()
        results.append(" ".join(sorted(words)))
    return results

profiler = LineProfiler()
wrapped = profiler(process_items)
wrapped(["  Hello World ", "foo BAR baz"] * 20_000)
profiler.print_stats(output_unit=1e-3, stripzeros=True)

$ python lines.py
Timer unit: 0.001 s

Total time: 0.0649172 s
File: /home/user/py-demo/performance/lines.py
Function: process_items at line 3

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     3                                           def process_items(items):
     4         1          0.0      0.0      0.0      results = []
     5     40001         11.6      0.0     17.9      for item in items:
     6     40000         15.2      0.0     23.4          cleaned = item.strip().lower()
     7     40000         15.9      0.0     24.6          words = cleaned.split()
     8     40000         22.2      0.0     34.1          results.append(" ".join(sorted(words)))
     9         1          0.0      0.0      0.0      return results

The sort-and-join line is the single biggest cost, about a third of the function’s time — and the bare for line costs more than you might expect, since each iteration is interpreted too. Line-level timing has more overhead than cProfile, so use it only on the function the profiler already pointed at.

Caching with lru_cache

If a pure function is called repeatedly with the same arguments, cache it.

import timeit
from functools import lru_cache

def fib(n: int) -> int:
    return n if n < 2 else fib(n - 1) + fib(n - 2)

@lru_cache(maxsize=None)
def fib_cached(n: int) -> int:
    return n if n < 2 else fib_cached(n - 1) + fib_cached(n - 2)

t_plain = timeit.timeit("fib(27)", globals=globals(), number=1)
t_cached = timeit.timeit("fib_cached.cache_clear(); fib_cached(27)", globals=globals(), number=1)
print(f"plain:  {t_plain * 1000:.1f} ms")
print(f"cached: {t_cached * 1000:.3f} ms")
print(fib_cached.cache_info())

$ python cache_fib.py
plain:  33.5 ms
cached: 0.029 ms
CacheInfo(hits=25, misses=28, maxsize=None, currsize=28)

The plain version makes over 600,000 recursive calls; the cached one computes each of the 28 values once. No micro-optimisation comes close to changing the algorithm’s complexity like this. Only cache pure functions — a cached function that reads a database returns stale data.

Membership: list vs set

in on a list scans; on a set it hashes.

import timeit

data = list(range(100_000))
data_set = set(data)
target = 99_999

t_list = min(timeit.repeat(lambda: target in data, number=1_000, repeat=3))
t_set = min(timeit.repeat(lambda: target in data_set, number=1_000, repeat=3))
print(f"list: {t_list * 1000:.1f} ms for 1,000 lookups")
print(f"set:  {t_set * 1000:.3f} ms for 1,000 lookups")

$ python membership.py
list: 699.4 ms for 1,000 lookups
set:  0.049 ms for 1,000 lookups

The target is the last element, the list’s worst case: each lookup compares 100,000 items. Converting to a set once costs one pass and makes every later lookup constant-time.

Building strings: += vs join

The textbook says repeated += on strings is quadratic. Measure it:

import timeit

words = ["hello"] * 100_000

def concat_plus(words):
    result = ""
    for w in words:
        result += w
    return result

def concat_join(words):
    return "".join(words)

t_plus = min(timeit.repeat(lambda: concat_plus(words), number=10, repeat=3))
t_join = min(timeit.repeat(lambda: concat_join(words), number=10, repeat=3))
print(f"+=:   {t_plus * 100:.2f} ms per call")
print(f"join: {t_join * 100:.2f} ms per call")

$ python strings.py
+=:   3.97 ms per call
join: 0.48 ms per call

join is still clearly faster — it computes the final size once and copies each piece once — but += isn’t catastrophic here, because CPython resizes a string in place when nothing else references it. That optimisation is an implementation detail: it doesn’t apply on PyPy or when another name holds the string, so "".join(...) remains the reliable choice.

NumPy vectorisation

For numeric work on large arrays, NumPy moves the loop into compiled code.

import timeit
import numpy as np

data = list(range(1_000_000))
arr = np.array(data, dtype=np.int64)

def python_sum_squares(values):
    return sum(x * x for x in values)

def numpy_sum_squares(a):
    return int((a * a).sum())

assert python_sum_squares(data) == numpy_sum_squares(arr)
t_py = min(timeit.repeat(lambda: python_sum_squares(data), number=5, repeat=3)) / 5
t_np = min(timeit.repeat(lambda: numpy_sum_squares(arr), number=5, repeat=3)) / 5
print(f"Python: {t_py * 1000:.1f} ms")
print(f"NumPy:  {t_np * 1000:.1f} ms  ({t_py / t_np:.0f}x faster)")

slow = sum(x for x in arr[:100_000])
fast = arr[:100_000].sum()
print(slow == fast, type(slow).__name__)

$ python numpy_vs_python.py
Python: 42.5 ms
NumPy:  0.9 ms  (45x faster)
True int64

One expression on the whole array replaced a million interpreted multiplications. The last line shows the anti-pattern: looping over a NumPy array in Python produces a NumPy scalar per element and is slower than looping over a list. Stay in array operations. (int64 also wraps silently on overflow, unlike Python ints — fine here, a trap with huge values.)

__slots__ for object-heavy code

__slots__ removes the per-instance dict, which helps when you create very many objects.

import timeit

class DictPoint:
    def __init__(self, x, y, z):
        self.x, self.y, self.z = x, y, z

class SlotPoint:
    __slots__ = ("x", "y", "z")
    def __init__(self, x, y, z):
        self.x, self.y, self.z = x, y, z

for cls in (DictPoint, SlotPoint):
    create = min(timeit.repeat(lambda: cls(1, 2, 3), number=500_000, repeat=3))
    p = cls(1, 2, 3)
    read = min(timeit.repeat(lambda: p.x + p.y + p.z, number=500_000, repeat=3))
    print(f"{cls.__name__}: create {create * 1000:.0f} ms, read {read * 1000:.0f} ms")

$ python slots_speed.py
DictPoint: create 59 ms, read 30 ms
SlotPoint: create 80 ms, read 30 ms

There’s no speed win here at all: in every run on this machine the slotted class was noticeably slower to create, and attribute reads were about the same. Recent CPython already stores ordinary instance attributes compactly and has specialised fast paths for them. The real benefit of __slots__ is memory (see memory management); don’t add it for speed alone, and measure if you do.

Checklist

1. Measure the whole operation, then profile to find the hot function.
2. Fix the algorithm first: caching, sets for lookups, avoiding repeated work.
3. Push loops into C: built-ins like sum, "".join, comprehensions, NumPy.
4. Consider multiprocessing for CPU-bound work and asyncio for I/O-bound work (concurrency).
5. Measure again to confirm the change helped.

Practice

1. Time [x * 2 for x in range(100_000)] against the equivalent for loop with .append.

import timeit

comp = "[x * 2 for x in range(100_000)]"
loop = """
out = []
for x in range(100_000):
    out.append(x * 2)
"""
t_comp = min(timeit.repeat(comp, number=50, repeat=3))
t_loop = min(timeit.repeat(loop, number=50, repeat=3))
print(f"comprehension: {t_comp / 50 * 1000:.2f} ms")
print(f"append loop:   {t_loop / 50 * 1000:.2f} ms")

$ python comp_vs_loop.py
comprehension: 4.55 ms
append loop:   5.12 ms

2. Profile process from the cProfile example again after replacing parse with list(map(int, line.split(","))). Did parse get faster?

import cProfile
import pstats

def parse(line: str) -> list[int]:
    return list(map(int, line.split(",")))

def is_valid(row: list[int]) -> bool:
    return sum(row) % 7 != 0

def process(lines: list[str]) -> int:
    return sum(1 for line in lines if is_valid(parse(line)))

lines = [",".join(str(i * j) for j in range(20)) for i in range(20_000)]
profiler = cProfile.Profile()
profiler.enable()
process(lines)
profiler.disable()
pstats.Stats(profiler).sort_stats("tottime").print_stats(1)

$ python profile_map.py
         97146 function calls (77146 primitive calls) in 0.068 seconds

   Ordered by: internal time
   List reduced from 7 to 1 due to restriction <1>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    20000    0.032    0.000    0.041    0.000 /home/user/py-demo/performance/profile_map.py:4(parse)

3. Use np.where to replace negative values with 0 in a 1,000,000-element array, and compare with a list comprehension doing the same thing.

import timeit
import numpy as np

rng = np.random.default_rng(0)
arr = rng.integers(-100, 100, size=1_000_000)
values = arr.tolist()

t_np = min(timeit.repeat(lambda: np.where(arr > 0, arr, 0), number=5, repeat=3)) / 5
t_py = min(timeit.repeat(lambda: [x if x > 0 else 0 for x in values], number=5, repeat=3)) / 5
print(f"np.where: {t_np * 1000:.1f} ms, comprehension: {t_py * 1000:.1f} ms")
print(np.where(arr > 0, arr, 0)[:6], values[:6])

$ python where.py
np.where: 6.4 ms, comprehension: 34.0 ms
[70 27  2  0  0  0] [70, 27, 2, -47, -39, -92]