> For the complete documentation index, see [llms.txt](https://docs.envector.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.envector.io/1.4.x/operations-and-management/troubleshooting.md). # Troubleshooting & Diagnostics This document describes common issues encountered when using enVector and how to resolve them. ## Quick Reference Find issues by symptom first; for detailed log examples and root-cause analysis, refer to the body of each linked section. ### Client Python SDK | Symptom | Quick Fix | Details | | ------------------------------------------ | ---------------------------------------------------------------------------- | ----------------------------- | | `pip install` fails / `ImportError` | Verify Python version is `3.9~3.13` and reinstall using a supported version | [Installation](#sdk-install) | | Install/run fails on Intel macOS | Intel macOS is not supported; run inside a Linux container in Docker Desktop | [Installation](#sdk-install) | | Package conflict / import errors | Reinstall in a dedicated `virtualenv` instead of the system Python | [Installation](#sdk-install) | | Some features don't work on Colab | Reinstall with the `pyenvector[colab]` extras | [Installation](#sdk-install) | | Server connection fails | Verify server address, port, and that the server is running | [Connection](#sdk-connection) | | Server not running / address·port mismatch | Inspect `docker compose ps` and the `ENVECTOR_ENDPOINT_HOST_PORT` value | [Connection](#sdk-connection) | | API call fails after long idle period | Check `ev.is_connected()` and reconnect | [Connection](#sdk-connection) | | gRPC Health Check fails | Check status with `docker compose logs` and restart the server | [Connection](#sdk-connection) | | Key generation fails | Verify `keys/` permissions and free disk space | [Key](#sdk-key) | | Key registration/load fails | Check for duplicate key IDs and review the endpoint logs | [Key](#sdk-key) | | File missing on `Cipher` initialization | Verify the key files exist under `keys//` | [Key](#sdk-key) | | Score decryption unavailable | Recovery is impossible if `SecKey.json` is lost; use the backup | [Key](#sdk-key) | | Index name collision | Use a different index name or delete the existing one | [Index](#sdk-index) | | Dimension (dim) parameter mismatch | Confirm the embedding model output dimension matches the index setting | [Index](#sdk-index) | | Index load fails | Check memory and object storage status | [Index](#sdk-index) | | `InternalError` on shard load | Verify the `preset` matches the index's `eval_mode` | [Index](#sdk-index) | | Dimension mismatch | Use `vecs.shape` to align with the index dim | [Insert](#sdk-insert) | | Abnormal similarity scores | For IP search, L2-normalize vectors before insertion | [Insert](#sdk-insert) | | Error during encrypted insertion | Verify the `Cipher` key path matches the key registered with the index | [Insert](#sdk-insert) | | Bulk insertion timeout | Split the request into smaller batches | [Insert](#sdk-insert) | | Empty results returned | Verify data was inserted and check the query vector dimension | [Search](#sdk-search) | | Decryption fails | Provide the `SecKey.json` from the same key pair used for search/encryption | [Search](#sdk-search) | | Top-k metadata missing | Check `output_fields` and that metadata was provided at insertion time | [Search](#sdk-search) | | Slow search response | Inspect resource status; consult the performance tuning / scale-out guides | [Search](#sdk-search) | ### Infrastructure Operations | Symptom | Quick Fix | Details | | -------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------- | | Service in `Restarting` / `Exited` state | Identify the cause with `docker compose logs ` | [Docker Compose](#docker-compose-troubleshooting) | | Termination after license error | Verify `token.jwt` presence, mount path, and expiration | [Docker Compose](#docker-compose-troubleshooting) | | License token input error (code `-1`) | Verify the token file isn't empty and the path is correct | [Docker Compose](#docker-compose-troubleshooting) | | License signature verification failure (code `-4`) | Reissue the token or check expiration | [Docker Compose](#docker-compose-troubleshooting) | | License token format error (code `-5`) | Check the token file for corruption and replace with the original | [Docker Compose](#docker-compose-troubleshooting) | | Port conflict | Identify the port in use and change `*_HOST_PORT` in `.env` | [Docker Compose](#docker-compose-troubleshooting) | | Container OOM Killed | Use `docker stats` and `dmesg` to check for memory shortage | [Docker Compose](#docker-compose-troubleshooting) | | I/O error / insertion·index failure | Use `df -h` to check free space and clean up unused images/data | [Docker Compose](#docker-compose-troubleshooting) | | Image pull fails | Check network, DNS, and firewall status | [Docker Compose](#docker-compose-troubleshooting) | | GPU mode execution error | Verify `nvidia-smi` and NVIDIA Container Toolkit installation | [Docker Compose](#docker-compose-troubleshooting) | | Pod `CrashLoopBackOff` | Identify the cause with `kubectl describe pod` and `kubectl logs --previous` | [Kubernetes](#kubernetes-troubleshooting) | | Pod `OOMKilled` / `Pending` | Adjust `resources.requests/limits` in Helm values | [Kubernetes](#kubernetes-troubleshooting) | | PVC binding fails | Verify the `StorageClass` and that available PVs exist | [Kubernetes](#kubernetes-troubleshooting) | | Service unreachable from outside | Inspect `Service`, `Ingress`, and `endpoints` configurations | [Kubernetes](#kubernetes-troubleshooting) | ### Advanced Operations | Symptom | Quick Fix | Details | | ----------------------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------- | | Index data loss | If the object storage volume remains, reload the index | [Recovery](#advanced-recovery) | | Homomorphic encryption secret key lost | Not stored on the server, so use an external backup | [Recovery](#advanced-recovery) | | No performance improvement after adding workers | Identify the actual bottleneck (memory/I/O/network) with `docker stats` | [Scale-out](#advanced-scale-out) (coming soon) | | No scale-out effect on K8s | Check not only the Pod count but also the per-Pod resource allocation | [Scale-out](#advanced-scale-out) (coming soon) | *** ## Table of Contents 1. [Client Python SDK](#1-client-python-sdk) * [Installation](#sdk-install) * [Connection](#sdk-connection) * [Key](#sdk-key) * [Index](#sdk-index) * [Insert](#sdk-insert) * [Search](#sdk-search) * [Error Codes / Messages](#sdk-error-code-messages) 2. [Docker Compose](#docker-compose-troubleshooting) 3. [Kubernetes](#kubernetes-troubleshooting) 4. [Advanced Usage](#advanced-usage) * [Recovery](#advanced-recovery) * [Scale-out](#advanced-scale-out) *** ## 1. Client Python SDK ### Installation *** **pip install fails / ImportError** ``` ERROR: Could not find a version that satisfies the requirement pyenvector ``` ```python ImportError: No module named 'pyenvector' ``` * **Cause**: Using a Python version outside the supported range (3.9\~3.13) * **Resolution**: Verify with `python --version`, then install via `pyenv install 3.11` or similar *** **Install/run fails on Intel macOS** ``` ERROR: Could not find a version that satisfies the requirement pyenvector ``` * **Cause**: Intel macOS is not supported * **Resolution**: Install inside a Linux container on Docker Desktop *** **Package conflict / unexpected import error** ```python ImportError: cannot import name 'xxx' from 'grpc' ``` * **Cause**: Direct installation in the system Python causes dependency conflicts * **Resolution**: Create a dedicated virtual environment with `virtualenv` and install there *** **Some features don't work on Colab** ```python ModuleNotFoundError: No module named 'pyenvector.colab' ``` * **Cause**: `pyenvector[colab]` was not installed * **Resolution**: Install with `!pip install "pyenvector[colab]"` *** ### Connection *** **Server connection fails** ``` pyenvector.errors.EnvectorTransportError: Failed to connect to localhost:50050 | Action: Check server connectivity ``` * **Cause**: enVector server is not running, or address/port information is incorrect * **Resolution**: Verify connection information such as server address and port *** **Server not running / address·port mismatch** ``` pyenvector.errors.EnvectorTransportError: search failed: UNAVAILABLE | Action: Check server connectivity ``` * **Cause**: Server not running, or address/port mismatch * **Resolution**: Check status with `docker compose ps` and verify the port matches `ENVECTOR_ENDPOINT_HOST_PORT` *** **API call fails after long idle period** ``` pyenvector.errors.EnvectorTransportError: failed: UNAVAILABLE | Action: Check server connectivity ``` * **Cause**: gRPC channel idle timeout * **Resolution**: Verify with `ev.is_connected()` and reconnect *** **gRPC Health Check fails** ``` pyenvector.errors.EnvectorTransportError: gRPC health status for service 'envector' is 'NOT_SERVING' | Action: Ensure enVector server is healthy ``` * **Cause**: Server is running but not in a healthy state * **Resolution**: Check server state with `docker compose logs` and restart the server *** ### Key *** **Key generation fails** ``` pyenvector.errors.KeyManagementError: GenerateKey failed (return_code=...): ... ``` * **Cause**: Insufficient permissions on the key storage path or insufficient disk space * **Resolution**: Verify with `ls -ld keys/` and `df -h .` *** **Key registration/load fails** ``` pyenvector.errors.KeyManagementError: Failed to unwrap EncKey for key_id='my_key': ... ``` ``` pyenvector.errors.KeyManagementError: Failed to unwrap EvalKey for key_id='my_key': ... ``` * **Cause**: Duplicate key ID (on registration) or non-existent key (on load) * **Resolution**: Check server errors with `docker compose logs envector-endpoint` *** **File missing on Cipher initialization** ```python FileNotFoundError: [Errno 2] No such file or directory: 'keys/my_key/EncKey.json' ``` * **Cause**: `EncKey.json` / `SecKey.json` path mismatch * **Resolution**: Verify file existence with `ls -la keys//` *** **Score decryption unavailable** * **Cause**: Homomorphic encryption secret key (`SecKey.json`) is lost * **Resolution**: **Recovery is impossible** — key files must be backed up in a secure location *** ### Index *** **Index name collision** ``` pyenvector.errors.InvalidInputError: Index 'my_index' already exists ``` * **Cause**: An index with the same name already exists * **Resolution**: Use a different name or delete the existing index *** **Dimension (dim) parameter mismatch** ``` pyenvector.errors.EnvectorValidationError: Invalid index_encryption: ... Expected 'plain' or 'cipher'. ``` * **Cause**: Vector dimension mismatch or invalid parameter format * **Resolution**: Verify the embedding model's output dimension and align it with the index *** **Index load fails** ``` pyenvector.errors.NotReadyError: NotReady ``` ``` pyenvector.errors.ResourceLimitError: InsufficientDiskSpace ``` * **Cause**: Insufficient server memory or object storage error * **Resolution**: Check memory with `free -h`; inspect storage with `docker compose logs minio` *** **Internal error on shard load (preset / eval-mode mismatch)** ``` pyenvector.errors.InternalError: failed to load raw shard 2326df816d7d424da635: failed to load raw shard 2326df816d7d424da635: failed to load raw shard on node: failed to load raw shard on node '172.22.0.6:25123': rpc error: code = Internal desc = failed to ForwardLoadRawShard: An internal computation error occurred. Contact support with your request ID. | Request ID: 01b64af3931b3beae048 ``` * **Cause**: The `preset` is not compatible with the index's `eval_mode`. Since v1.4.0 the eval-mode/preset coupling is enforced, so a mismatched preset surfaces as an internal computation failure during shard load. * **Resolution**: Re-create the index with a `preset` that matches the configured `eval_mode` (e.g., the `mm32` eval mode requires its matching mm-family preset). *** ### Insert *** **Dimension mismatch** ``` pyenvector.errors.InvalidInputError: bad vector dim ``` * **Cause**: Inserted vector's dim ≠ index's dim * **Resolution**: Check shape with `print(vecs.shape)` *** **Abnormal similarity scores** * **Cause**: Vectors not normalized for inner product (IP) search * **Resolution**: L2-normalize before insertion: ```python vecs = vecs / np.linalg.norm(vecs, axis=1, keepdims=True) ``` *** **Error during encrypted insertion** ``` pyenvector.errors.KeyManagementError: InvalidKeyURL ``` ``` pyenvector.errors.KeyManagementError: FailedToUnpackKey ``` * **Cause**: Encryption key does not match the key registered with the index * **Resolution**: Verify that the `Cipher`'s `enc_key_path` matches the index's key *** **Bulk insertion timeout** ``` pyenvector.errors.EnvectorTransportError: insert failed: DEADLINE_EXCEEDED | Action: Retry with longer timeout ``` ``` pyenvector.errors.EnvectorTimeoutError: Timed out waiting for insert to become searchable (index='my_index', request_id='...', ...) ``` * **Cause**: Single request payload is too large * **Resolution**: Split into batches (e.g., 1000 items) *** ### Search *** **Empty results returned** * **Cause**: No data in the index or query dimension mismatch * **Resolution**: Verify that data has been inserted and check the query vector dimension *** **Decryption fails** ``` pyenvector.errors.KeyManagementError: FailedToUnpackKey ``` * **Cause**: Decryption key differs from the key pair used during encryption * **Resolution**: Specify the `SecKey.json` path from the same key pair *** **Top-k metadata missing** * **Cause**: `output_fields` error or metadata not provided at insertion * **Resolution**: Verify whether metadata was provided at insertion and check field names *** **Slow search response** ``` pyenvector.errors.EnvectorTransportError: search failed: DEADLINE_EXCEEDED | Action: Retry with longer timeout ``` * **Cause**: Index has grown large; server resources are insufficient * **Resolution**: See [Performance Tuning](https://github.com/CryptoLabInc/envector-docs/tree/v1.4.6/gitbook/en/operations/performance-tuning.md) and [Scale-out](https://github.com/CryptoLabInc/envector-docs/tree/v1.4.6/gitbook/en/operations/scale-out.md) (coming soon) *** ### Error Codes / Messages #### SDK Errors Exceptions raised by the SDK on the client side. They occur during input validation before server communication, or are mapped from the `ReturnCode` in server responses. | Exception Class | Main Cause | Resolution | | ------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------------------- | | `EnvectorValidationError` | Client input validation failure (dimension mismatch, invalid parameter format, etc.) | Check the parameter values shown in the error message | | `InvalidInputError` | Server returned invalid input (`InvalidInput`, `NotFoundError`) | Verify index name, dimension, and data format | | `KeyManagementError` | Key-related errors (`InvalidKeyURL`, `FailedToUnpackKey`) | Verify key path and key file integrity | | `NotReadyError` | Resource not ready (`NotReady`, `NoSuchIndex`) | Verify the index was created/loaded | | `ResourceLimitError` | Resource limit exceeded (`ResourceLimitError`, `InsufficientDiskSpace`) | Expand server memory/disk; clean up unused indexes | | `EnvectorTimeoutError` | Server-side processing timeout (`Timeout`) | Reduce batch size; increase the timeout parameter | | `DependencyError` | Server dependency service failure (`DependencyError`) | Inspect server infrastructure (storage, etc.) status | | `InternalError` | Internal server error (`Fail`, `UnknownError`) | Check server logs and contact an administrator | | `EnvectorTransportError` | gRPC channel-level communication failure (no server response) | Verify network connectivity and server running state | #### gRPC Errors Status codes returned by the gRPC layer when the SDK communicates with the server. They are converted to the SDK exceptions above internally but can be inspected directly in logs or during debugging. | gRPC Code | Mapped SDK Exception | Main Cause | Resolution | | ------------------- | ---------------------- | ----------------------------------------- | ----------------------------------- | | `UNAVAILABLE` | `DependencyError` | Server not running, network disconnection | Check server state and network | | `DEADLINE_EXCEEDED` | `EnvectorTimeoutError` | Large data processing, server overload | Reduce batch size; increase timeout | | `INVALID_ARGUMENT` | `InvalidInputError` | Dimension mismatch, invalid data format | Validate input parameters | | `NOT_FOUND` | `InvalidInputError` | Non-existent index or key ID | Verify the resource exists | *** ## Docker Compose *** **Service in Restarting / Exited state** ``` $ docker compose ps NAME STATUS envector-endpoint restarting (1) ``` * **Cause**: Configuration error, license issue, or dependent service not running * **Resolution**: Identify the cause with `docker compose logs ` *** **Termination after license error** ``` License verification failed: token file not found ``` * **Cause**: `token.jwt` not mounted, expired, or in an invalid format * **Resolution**: Verify with `ls ./token.jwt` and check the volume mount path *** **License token input error (error code: -1)** ``` JWT license verification failed: license_verify_token failed: -1 ``` * **Cause**: Token file is empty (`0 bytes`), or the `ENVECTOR_LICENSE_TOKEN` environment variable does not point to a valid path * **Resolution**: 1. Verify the file size is not 0 with `ls -l ./token.jwt` 2. Inspect environment variables and the mount path with `docker compose config | grep LICENSE` 3. If the file is empty, replace it with a valid license token file *** **License signature verification failure (error code: -4)** ``` JWT license verification failed: license_verify_token failed: -4 ``` * **Cause**: The token's signature does not match the server's verification key. Commonly caused by an expired license, using a token from a different environment, or a key change after a server version upgrade * **Resolution**: 1. Check whether the license has expired — confirm the validity period with the token issuer 2. Verify that the token matches the current server version 3. If expired or version-mismatched, reissue the token *** **License token format error (error code: -5)** ``` JWT license verification failed: license_verify_token failed: -5 ``` * **Cause**: The token file's JWT structure (`header.payload.signature`) is invalid. Occurs when content was truncated during file copy/transfer, when newlines or whitespace were inserted, or when an entirely different file was mounted * **Resolution**: 1. Verify the contents look like a JWT with `cat ./token.jwt | head -c 100` (must start with `eyJ...`) 2. Check for unnecessary newlines or whitespace: `wc -l ./token.jwt` (should be 1 line) 3. If corrupted, replace it with the original token file *** **Port conflict** ``` Error response from daemon: driver failed programming external connectivity: bind: address already in use ``` * **Cause**: HOST port conflict * **Resolution**: Identify the conflicting port with `ss -tlnp | grep {port}`, then change `*_HOST_PORT` in `.env` *** **Container OOM Killed** ``` $ docker inspect --format='{{.State.OOMKilled}}' true ``` * **Cause**: Out of memory * **Resolution**: Verify with `dmesg | grep "out of memory"` and `docker stats` *** **I/O error / insertion·index failure** ``` OSError: [Errno 28] No space left on device ``` * **Cause**: Insufficient disk space * **Resolution**: `df -h`, `docker system prune -a` *** **Image pull fails** ``` Error response from daemon: pull access denied ... or repository does not exist ``` * **Cause**: Internet/DNS issue or firewall blocking * **Resolution**: `ping 8.8.8.8`, `nslookup docker.io` *** **GPU mode execution error** ``` docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] ``` * **Cause**: NVIDIA Container Toolkit not installed / driver mismatch * **Resolution**: Verify GPU recognition with `nvidia-smi` and install the Toolkit *** ## Kubernetes *** **Pod CrashLoopBackOff** ``` $ kubectl get pods NAME READY STATUS RESTARTS envector-endpoint-xxx 0/1 CrashLoopBackOff 5 ``` * **Cause**: Configuration error, insufficient resources, or license issue * **Resolution**: Identify the cause with `kubectl describe pod` and `kubectl logs --previous` *** **Pod OOMKilled / Pending** ``` $ kubectl describe pod envector-endpoint-xxx ... State: Terminated Reason: OOMKilled ``` * **Cause**: Insufficient `resources.requests/limits` * **Resolution**: Adjust memory and CPU in Helm values *** **PVC binding fails** ``` $ kubectl get pvc NAME STATUS VOLUME CAPACITY STORAGECLASS data-pvc Pending ``` * **Cause**: StorageClass missing or insufficient available PVs * **Resolution**: Verify with `kubectl get pvc` and `kubectl get storageclass` *** **Service unreachable from outside** ``` $ curl http://:50050 curl: (7) Failed to connect to ... port 50050: Connection refused ``` * **Cause**: Misconfigured Service type, Ingress, or port settings * **Resolution**: Inspect with `kubectl get svc` and `kubectl get endpoints` *** ## Advanced Usage ### Recovery * **Index data loss**: If the object storage volume is preserved, reload the index. Note that `--down-volumes` deletes volumes — use it with caution in production. * **Homomorphic encryption secret key lost**: The homomorphic encryption secret key is not stored on the server, so **recovery is impossible**. Back it up to external storage immediately after key generation. ### Scale-out * **No performance improvement after adding workers**: First identify the bottleneck (memory, I/O, network) with `docker stats`. Memory usage scales proportionally with worker count. * **K8s scale-out**: If increasing the Pod count doesn't help, check the resource allocation per Pod. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.envector.io/1.4.x/operations-and-management/troubleshooting.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.