🔑Key Generation

🔑 Managing Keys

The KeyGenerator is a crucial component for creating and managing the keys required by the system. It allows for the secure generation of key sets, with advanced options for metadata encryption and key sealing to protect sensitive keys at rest. Keys can be generated and managed through both a Command-Line Interface (CLI) and a Python SDK.

In addition, key sealing supports the use of a Key Encryption Key (KEK), making it easier to integrate with external Key Management Systems (KMS) for enhanced security. For users who prefer not to manage raw keys themselves, enVector also supports AWS-backed key storage using services such as Amazon S3 and AWS Secrets Manager, so long-term key management can be delegated to AWS instead of the application.

Generated Key Files

When you generate a key set, the following files are created in the specified directory (<key_path>/<key_id>/).

SecKey.json: The Secret Key. This is a private key used to decrypt encrypted vectors and derive the public keys. It must be kept confidential and secure at all times.
EncKey.json: The Public Encryption Key. This key is used to encrypt vectors before they are sent to the system. It can be shared publicly.
EvalKey.json: The Public Evaluation Key. This key enables the server to perform computations (like similarity searches) directly on encrypted data. It is sent to the server during the register_key process and is safe to share. By default, it supports vector dimensions up to 4096.
MetadataKey.json: The AES Metadata Key. This key is used to encrypt and decrypt metadata associated with your data in the index. It is only generated if metadata_encryption is enabled.

ls <key_path>/<key_id>
EncKey.json    EvalKey.json   SecKey.json  MetadataKey.json

Key Generation Configuration

These options control how the keys and their metadata are generated and secured.

metadata_encryption (boolean, default: true) When enabled, a MetadataKey.json is generated. All metadata stored in the index is then encrypted using this AES key before being written, providing an end-to-end security.
seal_mode (string, default: none) If set to aes, this enables key sealing. Sealing encrypts the SecKey.json and MetadataKey.json files using a Key Encryption Key (KEK). The encrypted keys are saved as SecKey.json and MetadataKey.json, and the original unsealed key files are no longer used by the system. This provides strong protection for your most sensitive keys at rest.
seal_path or seal_key_stdin (string) This specifies the Key Encryption Key (KEK) used for sealing when seal_mode is aes.
- seal_path: Provides the file path to the KEK.
- seal_key_stdin: Reads the KEK from standard input instead of a file. This is a more secure method for automated environments (e.g., CI/CD pipelines) as it avoids writing the KEK to disk.

Generating Keys

You can generate keys using either the CLI or the Python SDK.

Command-Line Interface (CLI)

The pyenvector-keygen utility is used for key generation and management from the command line.

Example: Generate and Seal Keys

This command generates a key set identified by ev-key inside the keys directory and seals the secret keys using a KEK from the aes.kek file.

pyenvector-keygen --key_path keys \
           --key_id ev-key \
           --metadata_encryption true \
           --seal_mode aes \
           --seal_key_path aes.kek

Example: Supply KEK via Standard Input

For enhanced security, you can pipe the KEK directly to the command, avoiding the need for a file on disk.

echo "your-32-byte-kek" | pyenvector-keygen \
    --key_path keys \
    --key_id ev-key \
    --metadata_encryption true \
    --seal_mode aes \
    --seal_key_stdin

Python SDK

The enVector SDK provides two convenient ways to generate keys programmatically.

Note: The Envector CLI supports AES-KEK–based sealing, but the Python SDK does not support AES-KEK–based sealing yet; in Python, only unsealed raw keys are currently supported.

1. Using the KeyGenerator Class For direct control over key generation, you can use the KeyGenerator class.

from pyenvector.crypto.key_manager import KeyGenerator

# Configure key generation options
keygen = KeyGenerator(
    key_path="keys/ev-key",
    metadata_encryption=True,
    seal_mode="aes",
    seal_kek_path="aes.kek"
)

# Generate the key files
keygen.generate_keys()

2. Using the ev.init() Helper Function The ev.init() function provides a streamlined way to initialize your client environment. It can automatically generate and register keys if they don't already exist, making it ideal for getting started quickly.

import pyenvector as ev

# Initialize the client, generating keys if the path doesn't exist
ev.init(
    host="your-hostname",
    port=50050,
    key_path="./keys",
    key_id="my-key",
    metadata_encryption=True,
    seal_mode="aes",
    seal_kek_path="aes.kek"
)

Key Generation with AWS Key Storage

With AWS key storage, key material is generated in memory and uploaded to AWS using the KeyManager or CLI with key_store="aws". The SDK takes care of mapping keys to the configured AWS resources:

EncKey.json and EvalKey.json are stored as objects in Amazon S3.
SecKey.json and MetadataKey.json are stored as encrypted secrets in AWS Secrets Manager.

Before You Begin

To use AWS key storage, make sure:

The target AWS region and S3 bucket exist and are provided via region name and bucket name.
AWS credentials are configured for an IAM principal that can create and read Secrets Manager secrets.

AWS Storage (CLI & Python)

For production environments, you may want to store keys in AWS instead of on the local filesystem.

CLI: Generate Keys and Store Them in AWS

You can configure the CLI to generate keys and upload them directly to AWS:

pyenvector-keygen \
    --key_store aws \
    --key_id aws-example \
    --region_name ap-northeast-2 \
    --bucket_name my-envector-key-bucket \
    --secret_prefix envector/keys/

This command generates a key set in memory and uploads it to AWS, storing public keys as S3 objects and secret keys as encrypted Secrets Manager entries using the bucket and prefix configuration above.

Python: Upload an Existing Key to AWS

If you already generated a key set (for example using KeyGenerator.generate_keys_stream() in a previous step), you can upload the resulting key_dict to AWS with KeyManager(key_store="aws"):

import pyenvector as ev
from pyenvector.crypto import KeyGenerator, KeyManager

keygen = KeyGenerator(key_id="aws-example", dim_list=512, preset="ip", eval_mode="rmp")
key_dict = keygen.generate_keys_stream()

manager = KeyManager(
    key_id="aws-example",
    key_store="aws",
    region_name="ap-northeast-2",
    bucket_name="my-envector-key-bucket",
    secret_prefix="envector/keys/",
)

manager.save(key_dict)
print("Keys for 'aws-example' uploaded to AWS")

PreviousAdvanced User Guide NextANN (IVF_FLAT)

Last updated 1 day ago