Unity Catalog: Namespaces, Grants, and Lineage

Unity Catalog is the governance layer over everything in the previous two lessons: one place that knows what tables and files exist, who may read them, and where each column came from. Its most useful property is that permissions are inherited, so adding a table does not mean adding a grant.

The namespace

metastore                    one per region, shared by workspaces
└── catalog        bookshop_prod
    └── schema     marts
        ├── table  customer_orders
        ├── view   revenue_summary
        ├── volume exports
        └── function mask_email

show catalogs;

catalog
---------------
bookshop_dev
bookshop_prod
system
samples

create catalog if not exists bookshop_prod
    managed location 's3://bookshop-lake/prod';

create schema if not exists bookshop_prod.marts;
use catalog bookshop_prod;

OK

Environment separation is a catalog boundary, not a workspace boundary. One workspace, two catalogs, different grants — and a promotion from dev to prod is a CREATE TABLE ... DEEP CLONE rather than a data copy between systems.

Managed and external tables

-- managed: Unity Catalog owns the storage
create table bookshop_prod.marts.customer_orders as
select c.customer_id, c.full_name, count(o.order_id) as order_count,
       coalesce(sum(o.amount), 0) as lifetime_value
from bookshop.raw.customers c
left join bookshop.raw.orders o using (customer_id)
group by 1, 2;

num_affected_rows  num_inserted_rows
-----------------  -----------------
                4                  4

-- external: you own the location
create table bookshop_prod.raw.legacy_orders
using delta
location 's3://partner-bucket/exports/orders';

OK

describe extended bookshop_prod.marts.customer_orders;

col_name       data_type
-------------  --------------------------------------------------
Type           MANAGED
Location       s3://bookshop-lake/prod/marts/customer_orders
Provider       delta
Owner          analytics-eng

DROP TABLE on a managed table removes the data (after a retention window); on an external table it removes only the registration. Use managed by default — Unity Catalog can then optimise, cluster and vacuum the files for you, which it will not do for storage it does not own.

Volumes, for files

create volume if not exists bookshop.raw.landing;

OK

dbutils.fs.ls("/Volumes/bookshop/raw/landing/")

[FileInfo(path='/Volumes/bookshop/raw/landing/orders_2026_01.csv', name='orders_2026_01.csv', size=124500),
 FileInfo(path='/Volumes/bookshop/raw/landing/orders_2025_12.csv', name='orders_2025_12.csv', size=118204)]

import pandas as pd
pd.read_csv("/Volumes/bookshop/raw/landing/orders_2026_01.csv").head(3)

   order_id  customer_id  ordered_at     status  amount
0      1001            1  2026-01-04  completed   25.50
1      1002            2  2026-01-05  completed   12.00
2      1003            1  2026-01-07   returned   40.00

A volume path works with ordinary local-file APIs — pandas, open(), shell commands — while still being access-controlled. That combination is why volumes replaced DBFS mounts, where a mount granted everyone in the workspace the same access.

Grants

create group if not exists analysts;

grant use catalog on catalog bookshop_prod to `analysts`;
grant use schema  on schema  bookshop_prod.marts to `analysts`;
grant select      on schema  bookshop_prod.marts to `analysts`;

OK

USE CATALOG and USE SCHEMA are traversal rights — without them a SELECT grant is invisible, and the error says the table does not exist. That is the most common Unity Catalog support question and it is always this.

Granting SELECT on the schema covers every table in it, including ones created tomorrow:

create table bookshop_prod.marts.new_mart as select 1 as x;

-- as a member of analysts
select * from bookshop_prod.marts.new_mart;

x
-
1

No new grant needed. Compare that with per-table grants, where the pipeline that adds a table must also remember to grant on it — and eventually does not.

show grants on schema bookshop_prod.marts;

principal   action_type  object_type  object_key
----------  -----------  -----------  ---------------------
analysts    USE SCHEMA   SCHEMA       bookshop_prod.marts
analysts    SELECT       SCHEMA       bookshop_prod.marts
data-eng    ALL PRIVILEGES SCHEMA     bookshop_prod.marts

show grants `analysts` on catalog bookshop_prod;

principal  action_type  object_type  object_key
---------  -----------  -----------  -------------
analysts   USE CATALOG  CATALOG      bookshop_prod

Ownership is separate from privileges: the owner can always grant, revoke and drop. Set owners to groups rather than individuals, or objects become unmanageable when someone leaves:

alter schema bookshop_prod.marts owner to `data-eng`;

OK

Masking columns and filtering rows

Unity Catalog applies these as functions attached to a table, so they follow the column everywhere it is referenced:

create or replace function bookshop_prod.marts.mask_email(email string)
returns string
return case when is_account_group_member('support') then email
            else regexp_replace(email, '.+@', '*****@') end;

alter table bookshop_prod.marts.customers
    alter column email set mask bookshop_prod.marts.mask_email;

OK

select customer_id, email from bookshop_prod.marts.customers limit 2;

customer_id  email
-----------  -----------------
          1  *****@example.com
          2  *****@example.com

create or replace function bookshop_prod.marts.country_filter(country string)
returns boolean
return is_account_group_member('analytics-admin')
    or country = current_user_country();

alter table bookshop_prod.marts.customers
    set row filter bookshop_prod.marts.country_filter on (country_code);

OK

A view built on a masked table stays masked, and so does a downstream job — the rule lives with the column, not with the query.

Lineage, for free

Every read and write through Unity Catalog is recorded. No instrumentation, no manual documentation:

select source_table_full_name, target_table_full_name, entity_type, event_time
from system.access.table_lineage
where target_table_full_name = 'bookshop_prod.marts.customer_orders'
order by event_time desc
limit 3;

source_table_full_name          target_table_full_name               entity_type  event_time
------------------------------  -----------------------------------  -----------  -------------------
bookshop.raw.orders             bookshop_prod.marts.customer_orders  NOTEBOOK     2026-09-09 10:14:02
bookshop.raw.customers          bookshop_prod.marts.customer_orders  NOTEBOOK     2026-09-09 10:14:02

Column-level too:

select source_column_name, target_column_name
from system.access.column_lineage
where target_table_full_name = 'bookshop_prod.marts.customer_orders'
  and target_column_name = 'lifetime_value';

source_column_name  target_column_name
------------------  ------------------
amount              lifetime_value

“Where does lifetime_value come from” is answerable in one query, across notebooks, jobs, dashboards and SQL. On a platform where the same table is touched by several teams and three languages, that is worth more than any documentation effort.

Auditing

select event_time, user_identity.email, action_name,
       request_params.full_name_arg as object
from system.access.audit
where service_name = 'unityCatalog'
  and action_name in ('getTable', 'deleteTable', 'createTable')
  and event_date >= current_date() - interval 1 day
order by event_time desc
limit 5;

event_time           email                  action_name  object
-------------------  ---------------------  -----------  -----------------------------------
2026-09-09 10:41:02  [email protected]      getTable     bookshop_prod.marts.customer_orders
2026-09-09 10:38:55  [email protected]    createTable  bookshop_prod.marts.new_mart
2026-09-09 09:22:41  [email protected]        getTable     bookshop.raw.customers

Practice

[TABLE_OR_VIEW_NOT_FOUND] The table or view `bookshop_prod`.`marts`.`customer_orders`
cannot be found.

x
-
1

Type      MANAGED
Location  s3://bookshop-lake/prod/marts/customer_orders

Type      EXTERNAL
Location  s3://partner-bucket/exports/orders

select source_table_full_name, source_column_name
from system.access.column_lineage
where target_table_full_name = 'bookshop_prod.marts.customer_orders'
  and target_column_name = 'lifetime_value';

source_table_full_name  source_column_name
----------------------  ------------------
bookshop.raw.orders     amount

Next: ingestion — Auto Loader, COPY INTO, and handling a schema that changes underneath you.