Performance Optimization in C++

Performance Optimization in C++

C++ gives you the tools to write code that runs close to the hardware limit — but only if you understand where time actually goes. The biggest surprise for most developers is that algorithm complexity is only part of the story: a poorly cache-friendly O(n) loop can be slower than a cache-friendly O(n log n) sort on real hardware. Most performance wins come from memory access patterns and compiler-friendly code, not micro-optimizations. Profile first, optimize what the profiler shows you, and measure the impact.

Cache Hierarchy and Memory Locality

Modern CPUs are fast; main memory is slow. L1 cache access costs ~4 cycles; a cache miss to RAM costs ~200 cycles — 50x slower. Writing cache-friendly code is the single highest-leverage optimization most developers can make. The key insight: sequential memory access is fast (the prefetcher loads ahead), while random access is slow (every access may miss the cache).

Array of Structs vs Struct of Arrays:

#include <vector>
#include <numeric>
#include <chrono>
#include <iostream>

constexpr int N = 1'000'000;

// Array of Structs (AoS) — bad for field-wise operations
// Summing only x loads all fields including vx, vy, vz, mass — wasted bandwidth
struct ParticleAoS {
    float x, y, z;    // position
    float vx, vy, vz; // velocity — loaded but unused when summing x
    float mass;
};

// Struct of Arrays (SoA) — better cache utilization for field-wise ops
// Summing x only loads x values — every cache line byte is used
struct ParticlesSoA {
    std::vector<float> x, y, z;
    std::vector<float> vx, vy, vz;
    std::vector<float> mass;
};

float sumX_AoS(const std::vector<ParticleAoS>& particles) {
    float sum = 0;
    for (const auto& p : particles) sum += p.x;  // loads 28 bytes, uses 4
    return sum;
}

float sumX_SoA(const ParticlesSoA& particles) {
    float sum = 0;
    for (float x : particles.x) sum += x;  // all cache line bytes used
    return sum;
}

SoA is consistently 2-4x faster for field-wise operations because every cache line loaded is fully utilized.

False Sharing

False sharing is a concurrency performance bug that can make multi-threaded code slower than single-threaded code. Two threads modify different variables that happen to share a cache line (64 bytes on x86). Each write by one thread invalidates the other thread’s cached copy, causing constant cross-core cache traffic even though the threads are touching different data.

#include <thread>
#include <atomic>
#include <iostream>

// BAD: a and b are likely adjacent in memory — same cache line
struct BadCounters {
    long a = 0;  // thread 0 writes here
    long b = 0;  // thread 1 writes here — same cache line, constant invalidation
};

// GOOD: pad to force each counter onto its own cache line
struct alignas(64) GoodCounters {
    long a = 0;
    char pad[64 - sizeof(long)];  // padding forces b to a different cache line
    long b = 0;
};

void benchmark() {
    BadCounters bad;
    GoodCounters good;

    auto increment = [](long& val, int iters) {
        for (int i = 0; i < iters; ++i) ++val;
    };

    // bad: threads thrash the same cache line — slower than sequential
    std::thread t1(increment, std::ref(bad.a), 100'000'000);
    std::thread t2(increment, std::ref(bad.b), 100'000'000);
    t1.join(); t2.join();

    // good: each thread owns its cache line — scales with cores
    std::thread t3(increment, std::ref(good.a), 100'000'000);
    std::thread t4(increment, std::ref(good.b), 100'000'000);
    t3.join(); t4.join();
}

Branch Prediction

The CPU speculatively executes branches before it knows whether the condition is true. A misprediction flushes the pipeline and costs ~15 cycles. For hot loops with unpredictable branch patterns, eliminating the branch entirely is faster than any prediction.

#include <vector>
#include <algorithm>

// Branch-heavy: unpredictable data causes frequent mispredictions
int sumPositive_branchy(const std::vector<int>& v) {
    int sum = 0;
    for (int x : v)
        if (x > 0) sum += x;  // branch mispredicts ~50% on random data
    return sum;
}

// Branchless: compiler emits a conditional move instruction instead of a branch
int sumPositive_branchless(const std::vector<int>& v) {
    int sum = 0;
    for (int x : v)
        sum += x * (x > 0);  // no branch — predictable, vectorizable
    return sum;
}

// C++20: hint the compiler and CPU about the likely path
void process(int value) {
    if (value > 0) [[likely]] {
        fastProcess(value);  // this path is taken most of the time
    } else [[unlikely]] {
        slowProcess(value);  // this path is rare
    }
}

Compiler Optimization Flags

The compiler can do a tremendous amount of work for you if you tell it to. The difference between -O0 and -O3 is often 5-10x on real code. Link-Time Optimization and Profile-Guided Optimization push this further by optimizing across the whole program.

# Development — fast compile, full debug info
g++ -O0 -g main.cpp

# Balanced — good optimization, still debuggable
g++ -O2 -g main.cpp

# Maximum safe optimization
g++ -O3 main.cpp

# Aggressive (allows unsafe floating-point reordering — test carefully)
g++ -Ofast main.cpp

# Target the current CPU's instruction set — non-portable but fastest
g++ -O3 -march=native main.cpp

# Full pipeline: LTO + PGO instrumentation
g++ -O3 -flto -fprofile-generate main.cpp -o app_instrumented
./app_instrumented                         # run with representative input
g++ -O3 -flto -fprofile-use main.cpp -o app_optimized

Link-Time Optimization (LTO) allows the compiler to optimize across translation unit boundaries — inlining functions from other .cpp files. It typically gives 5-15% speedup on real programs at the cost of longer link times.

Profile-Guided Optimization (PGO) runs your program, collects branch frequency data, then recompiles using that data to make hot branches predicted correctly and inline aggressively where it matters.

Profiling Tools

Never guess where the bottleneck is. Profiling gives you data; guessing gives you busy work. Most performance bottlenecks are in a small fraction of the code — find it first, then optimize.

# gprof: compile-time instrumentation — simple but overhead affects results
g++ -O2 -pg -o app main.cpp
./app
gprof app gmon.out | less

# perf: hardware performance counters (Linux) — low overhead, very accurate
perf stat ./app                     # overall stats: cache misses, branch mispredicts
perf record -g ./app                # call-graph sampling
perf report                         # interactive browser

# Valgrind/Callgrind: instruction-level profiling — high overhead but very detailed
valgrind --tool=callgrind ./app
callgrind_annotate callgrind.out.*
kcachegrind callgrind.out.*         # GUI viewer

# Quick timing in code — when you need to benchmark a specific section
#include <chrono>
auto t0 = std::chrono::high_resolution_clock::now();
// ... code to measure ...
auto t1 = std::chrono::high_resolution_clock::now();
auto us = std::chrono::duration_cast<std::chrono::microseconds>(t1 - t0).count();
std::cout << us << " us\n";

SIMD Intrinsics

SIMD (Single Instruction, Multiple Data) processes multiple values in one CPU instruction. AVX2 handles 8 floats simultaneously. This is the last resort after cache layout, algorithmic improvements, and compiler flags — but when the profiler shows an arithmetic-heavy inner loop, SIMD can deliver 4-8x speedup.

#include <immintrin.h>  // AVX2
#include <vector>

// Scalar: 1 addition per instruction
void addScalar(const float* a, const float* b, float* out, int n) {
    for (int i = 0; i < n; ++i)
        out[i] = a[i] + b[i];
}

// AVX2: 8 additions per instruction — 8x throughput
void addAVX2(const float* a, const float* b, float* out, int n) {
    int i = 0;
    for (; i <= n - 8; i += 8) {
        __m256 va = _mm256_loadu_ps(a + i);   // load 8 floats
        __m256 vb = _mm256_loadu_ps(b + i);
        __m256 vc = _mm256_add_ps(va, vb);    // add 8 pairs simultaneously
        _mm256_storeu_ps(out + i, vc);         // store 8 results
    }
    for (; i < n; ++i)                         // handle remainder
        out[i] = a[i] + b[i];
}

Before writing intrinsics manually, check if -O3 -march=native already auto-vectorizes the loop — it often does. Use intrinsics only when the compiler fails to vectorize and profiling confirms it matters.

Parallel Algorithms (C++17)

C++17 parallel execution policies let you parallelize standard algorithms with a single word change. The standard library handles thread management, work distribution, and synchronization — you just express the intent.

#include <algorithm>
#include <execution>
#include <vector>
#include <numeric>

int main() {
    std::vector<int> v(10'000'000);
    std::iota(v.begin(), v.end(), 0);

    // Sequential baseline
    std::sort(std::execution::seq, v.begin(), v.end());

    // Parallel — uses all available CPU cores automatically
    std::sort(std::execution::par, v.begin(), v.end());

    // Parallel + vectorized — parallel across cores AND SIMD within each core
    std::sort(std::execution::par_unseq, v.begin(), v.end());

    long long sum = std::reduce(std::execution::par, v.begin(), v.end(), 0LL);
}

Avoiding Unnecessary Copies

Unnecessary copies are the most common performance problem in C++ code that looks correct. A function that takes std::string by value when it only reads it makes a copy on every call. A vector that grows without reserve reallocates multiple times. These are easy wins once you know what to look for.

#include <string>
#include <vector>

// BAD: copies the string on every call — allocation + memcpy
void process(std::string s);

// GOOD: const ref for read-only — no copy, no allocation
void processRef(const std::string& s);
void processMoved(std::string s);   // caller std::moves in when giving up ownership

// BAD: vector reallocates repeatedly as it grows (log n reallocations)
std::vector<int> buildData() {
    std::vector<int> v;
    for (int i = 0; i < 100'000; ++i)
        v.push_back(i);  // reallocates at size 1, 2, 4, 8, 16, ...
    return v;
}

// GOOD: reserve upfront — single allocation, no copies
std::vector<int> buildDataFast() {
    std::vector<int> v;
    v.reserve(100'000);  // one allocation for the final size
    for (int i = 0; i < 100'000; ++i)
        v.push_back(i);
    return v;  // NRVO eliminates the return copy
}

reserve() is one of the easiest wins in C++. If you know the approximate final size, call it before filling a vector. For strings built by concatenation, reserve() on the string or use std::ostringstream with a single final conversion.