Key Rotation Runbook

This runbook covers the complete procedure for rotating the Datafly Signal encryption key. Follow these steps in order.

Prerequisites

• Admin access to the Management API (OrgAdmin role)
• Ability to set environment variables on all Signal services
• Access to perform rolling restarts (Helm upgrade, Coolify redeploy, etc.)

Generate a New Key

Generate a cryptographically secure 32-byte AES-256 key:

# Using OpenSSL
openssl rand -hex 32
 
# Or using Signal's built-in generator (if you have the binary)
signal-cli generate-key

Save the output — this is your new ENCRYPTION_KEY.

Rotation Steps

ENCRYPTION_KEY=<new-64-char-hex-key>
ENCRYPTION_KEY_PREVIOUS=<old-64-char-hex-key>

helm upgrade datafly ./charts/datafly \
  --set encryption.key=$NEW_KEY \
  --set encryption.previousKey=$OLD_KEY

curl https://collect.example.com/health/crypto
curl https://api.example.com/health/crypto

{
  "status": "healthy",
  "dual_key_mode": true,
  "checks": {
    "key_valid": true,
    "round_trip": true,
    "previous_key_valid": true
  }
}

curl -X POST https://api.example.com/v1/admin/crypto/re-encrypt \
  -H "Authorization: Bearer <admin-token>"

curl https://api.example.com/v1/admin/crypto/re-encrypt \
  -H "Authorization: Bearer <admin-token>"

{
  "status": "completed",
  "total_records": 150,
  "processed_records": 150,
  "failed_records": 0,
  "started_at": "2026-04-06T14:30:00Z",
  "completed_at": "2026-04-06T14:30:12Z"
}

ENCRYPTION_KEY=<new-key>
# ENCRYPTION_KEY_PREVIOUS removed

curl https://api.example.com/health/crypto

Cloud KMS Key Rotation

If using a cloud KMS provider (GCP, AWS, Azure), key rotation follows the same dual-key procedure, but the key wrapping happens in the cloud:

GCP Cloud KMS

# Create a new key version
gcloud kms keys versions create \
  --key=signal-dek \
  --keyring=signal \
  --location=global
 
# Set the new version as primary
gcloud kms keys update signal-dek \
  --keyring=signal \
  --location=global \
  --primary-version=<new-version-number>

GCP KMS automatically handles decryption with any key version, so ENCRYPTION_KEY_PREVIOUS is not needed. However, you should still run the re-encryption job to ensure all data is encrypted under the latest key version.

AWS KMS

AWS KMS supports automatic annual key rotation:

aws kms enable-key-rotation --key-id <key-arn>

With automatic rotation enabled, AWS creates new key material annually. Decryption of old data is handled transparently. Run the re-encryption job periodically to migrate data to the latest key material.

Troubleshooting

Service fails to start after key change

The most common cause is a typo in the new key. Verify:

• The key is exactly 64 hexadecimal characters
• The key is the same across all services
• ENCRYPTION_KEY_PREVIOUS is set to the old key (not the new one)

Re-encryption reports failures

Check the management-api logs for the specific error. Common causes:

• Corrupted ciphertext: A database record was modified outside of Signal
• Missing key: The ENCRYPTION_KEY_PREVIOUS doesn’t match the key that encrypted the record
• Database timeout: Increase DB_MAX_CONNS if re-encrypting a large number of records

Crypto health check fails

Verify the key is valid:

# This should be exactly 64 characters, all hex (0-9, a-f)
echo -n "$ENCRYPTION_KEY" | wc -c

If using a KMS provider, check that the service has network access to the KMS endpoint and valid credentials.

Compliance Reference