Profiling and Performance

Interview answer (say this first). Profiling means measuring where a program actually spends time or memory before changing anything. Use timeit for tiny isolated comparisons, cProfile plus pstats for function-level hotspots, line_profiler for the hottest lines, py-spy for production or a running process, and tracemalloc for memory. Fix the biggest cost first, then measure again.

Why this exists

Programs are slow for two reasons: a bad algorithm (the shape of the work) or constant-factor overhead (the cost of each step). You cannot tell which one you have by reading the code, because humans are bad at estimating cost. The part that feels heavy is often not the part that is slow.

Here is a realistic guess: “the model call is what takes all the time, so optimizing my Python is pointless.” Sometimes true. But a service that parses and post-processes large LLM responses can spend most of its time in cheap-looking string and list code, not in the network. Without measurement you are optimising a rumour.

The other failure mode is optimising too early. Rewriting clear code into clever code for a “speedup” that is invisible next to a database call makes the code worse and helps nobody. Donald Knuth’s often-quoted line is that premature optimization is the root of all evil; the full idea is that you should optimise the critical 3%, and you can only find it by measuring.

This page is about the measuring. Profiling answers three questions:

1. Where does the time or memory go?
2. How much would fixing it buy?
3. Did the fix actually work?

Start from zero

Two facts to hold on to:

• Wall time vs CPU time. If a function is fast on the CPU but the program is slow, the problem is waiting (network, disk, locks), and rewrite micro-optimizations will not help.
• Sampling vs deterministic. py-spy can profile a live production process because it samples rarely. cProfile records every call, so it is precise but slows the program down and cannot attach to a running process.

The core idea

A doctor does not prescribe medicine after reading your diary. They measure first: temperature, blood test, then treatment, then a check that it worked.

Profiling is the same loop:

flowchart LR
    A["1. Measure<br/>benchmark or profile"] --> B["2. Locate<br/>the hotspot"]
    B --> C["3. Fix<br/>the biggest cost"]
    C --> D["4. Re-measure<br/>did it help?"]
    D -->|"yes, and still slow"| B
    D -->|"meets target"| E["Stop"]

The loop matters more than any single tool. Profilers tell you where to look; they do not tell you what to do. And a fix that is not re-measured is a guess.

The second mental model is a budget. Performance work is about the total time of one user-visible action, for example “answer one API request in under 300 ms.” Every candidate optimization buys a slice of that budget, and the slices are wildly unequal:

flowchart TB
    subgraph Big["Usually big wins"]
        B1["Better algorithm or data structure"]
        B2["Remove repeated work / cache"]
        B3["Batch I/O instead of one call per item"]
    end
    subgraph Small["Usually small wins"]
        S1["Local variable aliasing"]
        S2["Avoid one attribute lookup"]
        S3["Replace += with join"]
    end
    Big -->|"measure first"| Budget["Your time budget"]
    Small -->|"only if proven"| Budget

How it works

1. Define the goal. “P95 latency under 200 ms” or “peak memory under 512 MB”. A number makes the work testable and tells you when to stop.
2. Build a representative workload. Use realistic input sizes and shapes. A benchmark on ten items says nothing about one hundred thousand.
3. Warm up. The first run pays for imports, caches, and JIT-like specialisation. Discard it or run enough iterations that it does not dominate.
4. Measure a baseline. Record the number before any change, so you can prove the fix helped.
5. Time small pieces with timeit. It runs the snippet many times and reports the best average, which removes most noise.
6. Profile the whole program with cProfile. It counts every call and records total time.
7. Sort the profile with pstats. Sort by tottime (time inside the function itself) to find work, or cumulative (time including callees) to find the path.
8. Drill into the hottest function with line_profiler. It shows which line inside the function costs the most.
9. Profile a live or production process with py-spy. It samples the stack with very low overhead and does not need code changes.
10. Measure memory with tracemalloc. It attributes allocations to the exact source line.
11. Fix the biggest item, then go back to step 4. Repeat until the goal is met. Stop when it is.

Tip:

The rule that saves the most time. Never optimise without a baseline number and a target number. Without them you cannot tell improvement from noise, or know when to stop.

The syntax you will use

timeit for micro-benchmarks. Use it for small, isolated choices, never for I/O.

import timeit

best = min(timeit.repeat("999 in data_set", setup="data_set = set(range(1000))",
                         number=10000, repeat=5))
print(best)

timeit from the command line. Quick and convenient for one-liners.

python -m timeit -s "data = set(range(1000))" "999 in data"
# 20000000 loops, best of 5: 11.6 nsec per loop

cProfile and pstats. Profile the whole program, then sort by internal time.

import cProfile, pstats

with cProfile.Profile() as pr:
    main()

stats = pstats.Stats(pr)
stats.sort_stats("tottime").print_stats(10)
stats.dump_stats("profile.out")      # open later with snakeviz or pstats

cProfile as a module. One command, no code changes.

python -m cProfile -s cumulative myscript.py

line_profiler. Decorate the one function you suspect, then run through kernprof.

from line_profiler import profile

@profile
def slow_sum(n):
    total = 0
    for i in range(n):
        total += i * i
    return total

kernprof -l -v myscript.py

py-spy for live processes. It samples a running Python process without changing the code.

py-spy record -o profile.svg -- python myscript.py    # flame graph
py-spy top --pid 12345                                # live top-like view
py-spy dump --pid 12345                               # current stack of every thread

On macOS, py-spy needs root, so prefix with sudo. On Linux, launching a new process under py-spy works without root, but attaching to an existing PID usually needs root or a relaxed ptrace_scope; in a container you may need --cap-add SYS_PTRACE.

tracemalloc for memory. Start it, run the code, then read current and peak usage.

import tracemalloc

tracemalloc.start()
data = [i for i in range(1_000_000)]
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(f"current={current:,} peak={peak:,}")

Snapshots find the leaking line. Compare two snapshots to see what grew.

import tracemalloc

tracemalloc.start()
first = tracemalloc.take_snapshot()
work()
second = tracemalloc.take_snapshot()
for stat in second.compare_to(first, "lineno")[:5]:
    print(stat)

Caching with functools. lru_cache keeps the last N results; cache keeps them all. Both require hashable arguments.

from functools import lru_cache, cache

@cache
def token_count(text: str) -> int:
    return len(text.split())

token_count("hello world")
token_count("hello world")        # free: served from the cache
print(token_count.cache_info())   # CacheInfo(hits=1, misses=1, ...)

sys.getsizeof for the size of one object. It reports the object itself, not the objects it references.

import sys

sys.getsizeof(0)              # 28
sys.getsizeof(2**64)          # 36: bigger ints use more memory

Examples: simple to real

Example 1 — measure a data-structure choice with timeit.

Membership testing is the classic case. Lists scan; sets and dicts hash.

import timeit

setup = "data_list = list(range(1000))\ndata_set = set(range(1000))"
t_list = timeit.timeit("999 in data_list", setup=setup, number=10000)
t_set = timeit.timeit("999 in data_set", setup=setup, number=10000)
print(f"list={t_list:.4f}s set={t_set:.4f}s ratio={t_list / t_set:.0f}x")
# one machine: list=0.0530s set=0.0001s ratio=470x

The absolute numbers depend on the machine, but the shape does not: a list scan is O(n), a set lookup averages O(1). At n = 1000 that is hundreds of times faster. Changing the data structure is an algorithmic win; no amount of micro-tuning closes that gap.

Example 2 — find the hotspot with cProfile.

import cProfile, pstats

def find_duplicates(items):
    seen, dupes = [], []
    for x in items:               # accidental O(n²): `in` scans `seen`
        (dupes if x in seen else seen).append(x)
    return dupes

def summarize(rows):
    return [row["name"].upper() for row in rows]

def main():
    find_duplicates(list(range(3000)) * 2)
    summarize([{"name": f"user{i}"} for i in range(20000)])

with cProfile.Profile() as pr:
    main()
pstats.Stats(pr).sort_stats("tottime").print_stats(3)

         26005 function calls in 0.054 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.049    0.049    0.049    0.049 profile_demo.py:3(find_duplicates)
        1    0.003    0.003    0.054    0.054 profile_demo.py:12(main)
        1    0.001    0.001    0.002    0.002 profile_demo.py:9(summarize)

find_duplicates owns 0.049 of the 0.054 seconds. The “obvious” suspects — building 20,000 dicts and calling .upper() 20,000 times — are almost free. The profile corrected the guess.

Example 3 — read tottime and cumtime correctly.

A function with high cumtime but low tottime is a conductor, not the orchestra. Do not optimise it; optimise what it calls. Sort by tottime first to find work, then by cumulative to see the path.

Example 4 — go line by line with line_profiler.

When one function is hot, a function-level profile is too coarse. kernprof -l -v gives this:

Function: slow_sum at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     5         1          0.0      0.0      0.0      total = 0
     6    200001      19847.0      0.1     43.4      for i in range(n):
     7    200000      25877.0      0.1     56.6          total += i * i

Now you know the cost is the loop itself, not setup. % Time points straight at the line to change.

Example 5 — find a memory problem with tracemalloc.

A list comprehension materialises everything; a generator produces one item at a time.

import tracemalloc

tracemalloc.start()
lst = [i for i in range(1_000_000)]
_, list_peak = tracemalloc.get_traced_memory()
del lst

tracemalloc.reset_peak()
gen = (i for i in range(1_000_000))
_, gen_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(f"list peak={list_peak:,} bytes")
print(f"gen  peak={gen_peak:,} bytes")
# one machine: list peak=40,437,248 bytes, gen peak=3,632 bytes

The list allocated about 40 MB; the generator about 4 KB. This matters for agentic AI code that streams long token sequences or loads many embeddings: prefer generators and chunking when you do not need the whole collection at once.

Example 6 — caching turns exponential into linear.

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n: int) -> int:
    return n if n < 2 else fib(n - 1) + fib(n - 2)

assert fib(30) == 832040
print(fib.cache_info())     # CacheInfo(hits=28, misses=31, ...)

lru_cache makes repeated subproblems free, turning O(2ⁿ) into O(n). The same idea caches expensive LLM calls, embeddings, or parsed documents — but only when inputs are hashable and results are safe to reuse.

In production

• Measure in the target environment. A laptop profile does not predict a container with a CPU quota and a shared database. Profile where the code runs.
• Use wall time for user-facing goals, CPU time for CPU work. If wall time is high but CPU time is low, you are waiting on I/O, and micro-optimizing Python will do nothing.
• Warm up before benchmarking. The first call pays import and cache costs. timeit.repeat plus min discards most noise and outliers.
• Profile overhead changes the picture. cProfile can slow a program down several times over; its per-call overhead is charged to every function, so frequently-called functions absorb that cost and look relatively more expensive than they are. Confirm with py-spy in production.
• tottime before cumtime. Sorting by cumulative time points at wrappers and main; sorting by internal time points at where the work is.
• An O(n²) in a hot path dwarfs every micro-optimization. Fix the algorithm first. A set instead of a list can be hundreds of times faster; aliasing a local variable is a few percent.
• Cache only pure, repeated work. lru_cache on a function that depends on mutable global state or the current time returns stale answers. It also holds references, so an unbounded cache can become a memory leak.
• Never cache with unhashable arguments. A dict, list, or Pydantic model raises TypeError as a cache key unless you convert it to a hashable form.
• Beware micro-optimizations that are not real — but join still wins string building. On modern CPython, repeated str += is not the quadratic disaster folklore claims, because the interpreter can often grow the buffer in place. It is still measurably slower than "".join() for non-trivial strings, though, because join allocates once. Keep += only for tiny, bounded builds where a measurement shows no meaningful difference; readability wins there.
• Memory leaks hide in long-lived objects. Module-level caches, growing lists, exception tracebacks, and circular references all survive for the process lifetime. tracemalloc snapshots show what grew between two points.
• sys.getsizeof is shallow. It does not count referenced objects, so it understates containers. Use tracemalloc or pympler for real totals.
• Parallelism has a ceiling. Amdahl’s law: if 20% of the work is serial, the best possible speedup is 5x no matter how many cores you add. Profile before adding threads or processes.

Interview questions

1. How do you approach a slow Python function?

Answer. Measure first. Build a representative benchmark, record a baseline, then profile with cProfile and pstats to find the hotspot. Fix the largest cost — usually an algorithm or data-structure problem — then re-measure to confirm. Stop when the goal is met; do not optimise by intuition.

Follow-up: “And if it is still slow?” Repeat the loop. If wall time stays high while CPU time is low, the bottleneck is I/O, not Python, so change the I/O pattern (batching, caching, concurrency) rather than the code constants.

Trap. Starting with a rewrite because “this loop looks slow.” Without a profile you are guessing, and the guess is often wrong.

2. What is the difference between timeit and cProfile?

Answer. timeit answers “how long does this tiny snippet take?” by running it many times in isolation. cProfile answers “where does my whole program spend time?” by recording every function call. Use timeit to compare two implementations of one operation; use cProfile to find which part of a real workload is hot.

Follow-up: “Why does timeit report the best run?” The minimum is the least contaminated by other processes and garbage collection, so it is the most stable estimate of the code’s own cost.

Trap. Using timeit on code that does I/O or has side effects. It repeats the code many times and assumes each run is independent.

3. What do tottime and cumtime mean?

Answer. tottime is time spent inside a function excluding calls it makes; cumtime includes those calls. High tottime means the function itself is doing expensive work. High cumtime with low tottime means it is a caller of expensive work. Sort by tottime to find work and by cumtime to find the path.

Follow-up: “Which do you optimise?” The function with high tottime, or the deep function at the end of a high-cumtime chain. Optimising the wrappers themselves rarely helps.

Trap. Chasing the top cumtime entry, which is usually main or a framework entry point.

4. How do cProfile and py-spy differ?

Answer. cProfile is a deterministic profiler: it records every call, so it is exact but slows the program and cannot attach to an already-running process. py-spy is a sampling profiler: it periodically snapshots stacks from outside the process, so it has low overhead and can profile production, at the cost of statistical rather than exact counts.

Follow-up: “When would you use each?” cProfile while developing a function you can rerun. py-spy for a live service, a hung process, or when profiling overhead would change behaviour.

Trap. Quoting exact call counts from a sampling profiler. It samples; counts are estimates.

5. How do you find a memory leak in Python?

Answer. Take a tracemalloc snapshot, run the suspect workload, take another, and compare with compare_to(first, "lineno"). The top lines are your allocations. Common causes are module-level caches that never evict, lists that only grow, and references held by tracebacks or closures.

Follow-up: “How is that different from sys.getsizeof?” getsizeof measures one object shallowly and says nothing about growth. tracemalloc attributes allocations to source lines over time, which is what a leak is.

Trap. Assuming garbage collection will fix it. A live reference is not garbage, and CPython’s collector only handles reference cycles.

6. Algorithmic versus micro-optimization, and what is premature optimization?

Answer. Algorithmic changes matter far more because they change how cost grows with input size. Turning an O(n²) list membership check into an O(1) set lookup was about 500x in a real benchmark, while local-variable aliasing was within noise. Micro-optimizations are a finishing step. Premature optimization is spending effort and sacrificing clarity before measuring, and it is wrong when the code is not the bottleneck. Measure, change the biggest cost, measure again.

Follow-up: “Give an example of each.” Algorithmic: build a set once instead of calling list.index() in a loop, or batch database calls instead of one per row. Micro: alias an attribute to a local, or avoid one function call.

Trap. Two extremes: refusing all optimization, and rewriting on a hunch. Both ignore the same evidence.

7. When is caching the wrong tool?

Answer. When the function is not pure, when arguments are unhashable, when the cache is unbounded and grows forever, or when the hit rate is low. A cache that misses almost every time adds memory and lookup cost for nothing. It also introduces staleness: callers can get an old answer for a new input.

Follow-up: “How do you know the hit rate?” functools exposes cache_info() with hits, misses, and currsize. Track it in production; a low-hit cache is dead weight.

Trap. Caching a function that reads the current time, a database, or mutable global state, then being surprised by stale results.

8. How do you benchmark fairly?

Answer. Use a representative workload, warm up, run multiple times, report the best or median rather than a single sample, keep the machine and conditions stable, and compare against a baseline recorded the same way. Isolate the change: one variable at a time. Be honest about what the benchmark does not cover.

Follow-up: “Why is the minimum a fair statistic?” It is the run least disturbed by noise, so it best estimates the code’s own cost. Report median or percentiles too when you care about tail latency.

Trap. Benchmarking with trivial inputs, then applying the conclusion to production-sized data, where the complexity term dominates.

Remember this

• Measure, locate, fix, re-measure. No baseline and target means no real optimization.
• timeit for micro choices; cProfile + pstats for hotspots; line_profiler for lines; py-spy for live processes; tracemalloc for memory.
• Read tottime to find work and cumtime to find the path; do not optimize wrappers.
• Algorithm first: a better data structure can be hundreds of times faster; micro-tuning is a few percent.
• Cache only pure, repeated, hashable work; an unbounded cache is a memory leak waiting to happen.