Repository navigation
Procedure for re-wrapping table encryption keys #18240
Description
Activity
in the future, also statistics files and other file types.
thanks @ggershinsky, I started 1st PR #18440 to define the public interface in action, once we get consensus for community we can move towards implementations.
Questions
Assuming the master key version is already rotated in the KMS before the action runs: since rotation doesn't rewrite the existing KEK ciphertext in encryption-keys, and
reuses any KEK under the 730-day lifespan, the manifest-list key would be re-encrypted under the pre-rotation KEK and the old version would stay pinned.Should the action mint a fresh KEK (and should keyEncryptionKeyID() then prefer the newest unexpired KEK rather than the first), or did you intend the KEK itself to be re-wrapped in place?
Plan
- API: the interface above, on its own.
- Core + Spark 4.2: the action, including the selection change.
- Spark 4.2: the procedure and docs.
- Backports to older Spark versions after that, if there's appetite.
I'll also follow up on statistics files and the other file types you mentioned once this lands.
@ggershinsky why not just just re-create the KEK, unwrap the manifest list with the old KEK and rewrap using the new KEK and then just store the encrypted version of KEK in the table metadata ...
The way i understand the proposal is we are just doing the metadata re-encryption, what is the rationale behind that !
Should the action mint a fresh KEK (and should keyEncryptionKeyID() then prefer the newest unexpired KEK rather than the first), or did you intend the KEK itself to be re-wrapped in place?
I think it'd be more consistent to have all Iceberg keys replaced (except for data file keys)
why not just just re-create the KEK, unwrap the manifest list with the old KEK and rewrap using the new KEK and then just store the encrypted version of KEK in the table metadata ...
The way i understand the proposal is we are just doing the metadata re-encryption, what is the rationale behind that !The owners of tables that were encrypted a few years ago, are sometimes required to replace the old encryption keys.
Like most things in security, this is a "defense in depth" approach, not a silver bullet solution. We add a layer of protection that replaces old keys (just in case they are compromised/leaked/overused) - but only where this is feasible, meaning all keys except for data file keys (since it's too expensive to re-encrypt the big parquet files). It's not perfect, but tables where this re-wrapping is done, are safer than tables were it was not done.
Feature Request / Improvement
All encryption keys in a table, except for the master key, are wrapped (encrypted) by a parent key.
Some security protocols require periodic re-wrapping with a fresh version of the parent keys.
The period is typically long, one year or above.
We need a procedure that generates new manifest file keys, and re-encrypts all active manifests with the new keys (so the data file keys are re-wrapped) - basically creating new manifest files.
Then it generates new manifest list file keys, and re-encrypts all active manifest lists with the new keys - basically creating new manifest list files.
Then it generates a fresh Key Encryption Key and uses it to encrypt the new manifest list file keys.
The new Key Encryption Key is wrapped with a rotated master table key.
For safety / fault-tolerance, the procedure can run only on the latest snapshot, so the older snapshots are available for recovery. However, it should be documented with clear instructions on when and how the old snapshots are retired, so no re-wrapped keys exist anymore.
Query engine
None
Willingness to contribute