Skip to main content
Data Engineering · lesson 18 of 23

Beginner

8

Intermediate

4

Advanced

11

Also part of Data Engineering

9
Open the Data Engineering page →
Data Engineering advanced Lesson 18 of 23

Also part of: Security Fundamentals Cryptography

Data Security and PII: Hashing, Masking, and Tokenization

Run the same email and SSN through three PII techniques and see what each gives up: hashing for joins, masking for display, tokenization for reversal.

By SkillByExample · Published

data-engineering security pii data-privacy

Run the same email and SSN through three different techniques, and see exactly what each one gives up in exchange for its guarantee.

Two customers, two sensitive fields each

import hashlib, secrets

customers = [
    {"customer_id": 1, "email": "ada@example.com", "ssn": "123-45-6789", "card": "4111111111111111"},
    {"customer_id": 2, "email": "grace@example.com", "ssn": "987-65-4321", "card": "5500000000000004"},
]

Hashing: deterministic, one-way, safe to join on

def hash_pii(value, salt="platform-salt-2026"):
    return hashlib.sha256((salt + value).encode()).hexdigest()[:16]

for c in customers:
    h = hash_pii(c["email"])
    print(f"  {c['email']:<20} -> {h}")
h_again = hash_pii(customers[0]["email"])
print(f"  same input hashed again: {h_again} (matches: {h_again == hash_pii(customers[0]['email'])})")
$ python security_pii.py
== hashing: deterministic, one-way, safe to join on ==
  ada@example.com      -> 5478b2ea4f25a8c3
  grace@example.com    -> 5cfdb214d49823d9
  same input hashed again: 5478b2ea4f25a8c3 (matches: True)

Same email hashed twice gives the same fingerprint — that determinism is exactly what makes hashed PII usable for joins and deduplication across tables without either table ever storing the real email. There is no unhash() function; the original value is gone once hashed.

Masking: format-preserving, irreversible, safe to display

def mask_email(email):
    local, _, domain = email.partition("@")
    return f"{local[0]}{'*' * (len(local) - 1)}@{domain}"

def mask_card(card):
    return f"{'*' * (len(card) - 4)}{card[-4:]}"

for c in customers:
    print(f"  email {c['email']:<20} -> {mask_email(c['email'])}")
    print(f"  card  {c['card']:<20} -> {mask_card(c['card'])}")
== masking: format-preserving, irreversible, safe to display ==
  email ada@example.com      -> a**@example.com
  card  4111111111111111     -> ************1111
  email grace@example.com    -> g****@example.com
  card  5500000000000004     -> ************0004

Masking keeps enough shape to be useful in a support UI — “is this the card ending in 1111?” — without exposing the full value, and without even needing a lookup: the masked form is derived directly from the input every time, nothing extra to store or protect.

Tokenization: reversible, but only through the vault

VAULT = {}

def tokenize(value):
    token = f"tok_{secrets.token_hex(6)}"
    VAULT[token] = value
    return token

def detokenize(token, requester_role):
    if requester_role != "compliance_officer":
        raise PermissionError(f"role '{requester_role}' cannot detokenize")
    return VAULT[token]

token = tokenize(customers[0]["ssn"])
print(f"  ssn {customers[0]['ssn']} -> token {token}")
try:
    detokenize(token, "support_agent")
except PermissionError as e:
    print(f"    DENIED: {e}")
print(f"  compliance_officer detokenizes: {detokenize(token, 'compliance_officer')}")
== tokenization: reversible, only for an authorized role ==
  ssn 123-45-6789 -> token tok_6a358ce01552
  support_agent tries to detokenize:
    DENIED: role 'support_agent' cannot detokenize
  compliance_officer detokenizes: 123-45-6789

secrets.token_hex generates a fresh random token every run, so the exact token string above will differ if this runs again — that unpredictability is the point, it carries zero information about the SSN it stands in for. The token can travel freely through logs, downstream tables, even third-party systems, because reversing it requires the VAULT, not just the token — and the vault enforces the same role check the data-governance lesson built, not the token itself.

Practice

1. Hash the same email with a different salt and confirm the fingerprint changes. 
print(hash_pii("ada@example.com", salt="a-different-salt"))
64145e9afdf12823

A different salt produces a completely unrelated hash for the same input — anyone comparing hashes across two systems using different salts will see no matches even for identical PII, which is why the salt itself must be shared (and protected) consistently, not regenerated.

2. Mask a card number shorter than 4 digits and see what breaks. 
print(mask_card("123"))
123

'*' * (len(card) - 4) evaluates to '*' * -1, which Python silently treats as zero repetitions rather than raising an error — the function quietly returns the number unmasked instead of failing loudly. A real implementation should validate card length first.

3. Tokenize the same SSN twice. Are the two tokens the same? 
t1 = tokenize(customers[0]["ssn"])
t2 = tokenize(customers[0]["ssn"])
print(t1 == t2)
False

Unlike hashing, tokenization is not deterministic by design — two tokens for the same value are unlinkable from each other without the vault, which is a deliberate privacy property (an attacker who sees the same token reused everywhere can infer it’s the same underlying value; fresh tokens per issuance avoid that).

4. Why can `hash_pii`'s output safely be used as a join key across two tables, but `tokenize`'s output cannot?

Hashing is deterministic — the same email always produces the same hash in both tables, so an equi-join on the hashed column works exactly like joining on the real email. Tokenization generates a fresh, unrelated token on every call (confirmed in task 3), so the same SSN tokenized independently in two tables produces two different tokens that would never join.

Next: observability — instrumenting a pipeline’s own operational health, not the data flowing through it.

Frequently Asked Questions

What is the difference between hashing, masking, and tokenization for PII?
Hashing produces a deterministic, one-way fingerprint — the same input always hashes the same way, useful for joining or deduplicating on a PII field without ever storing the real value, but never recoverable. Masking shows a partial, format-preserving version for display — irreversible by design. Tokenization swaps the value for a random reference and stores the real value in a separate, access-controlled vault — the only one of the three that's reversible, and only for whoever is authorized to query the vault.
Why hash with a salt instead of just hashing the raw value?
An unsalted hash of a low-entropy value (an email, an SSN) can be reversed by an attacker who precomputes hashes for every plausible value and looks up a match — a rainbow table. A salt (a secret string mixed in before hashing) makes precomputed tables useless unless the attacker also has the salt, which should be stored separately from the hashed data.
Is tokenization the same thing as encryption?
No. Encryption transforms the value itself and can be decrypted by anyone holding the key — the ciphertext still encodes the original data. Tokenization replaces the value with an unrelated random reference; the token itself carries zero information about the original, which only exists in the vault. Losing an encryption key can potentially be brute-forced with enough compute; a well-generated token cannot be reversed at all without the vault.
Where does encryption at rest fit if this lesson doesn't build it?
Encryption at rest (encrypting the whole storage volume or specific columns with a key management service) is usually infrastructure-layer, handled by the warehouse or cloud storage service itself rather than application code — see the cloud vendor tracks for the actual KMS/encryption configuration. This lesson focuses on the pipeline-level techniques a data engineer writes by hand: hashing, masking, and tokenization.