> For the complete documentation index, see [llms.txt](https://docs.envector.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.envector.io/1.4.x/tutorials/advanced-use-case/multi-modal-search.md).

# Encrypted Multi-Modal Search with enVector

In this tutorial, we introduce how enVector improves the security of Multi-Modal Search.

Recent advances in AI have enabled LLMs and multimodal models to understand user intent and process text, images, audio, and video in an integrated manner. At the core of this capability are embedding vectors, which numerically represent data and allow semantic similarity comparison across modalities. Although embeddings provide the expressiveness, they pose security risks because original data can potentially be reconstructed from them.

To address this issue, **enVector** performs **encrypted vector search using fully homomorphic encryption (FHE)**. enVector ensures strong data privacy and security against various threats while enabling safe and powerful multimodal search in untrusted environments.

## Understanding Multi-Modal Search with enVector

Multi-modal models align embeddings from different modalities, such as text and images, into a shared semantic space. Each modality is encoded separately, and multi-modal models ensures that they share embedding space, enabling cross-modal understanding and retrieval. Because all modalities lie in the same space, simple similarity measures like cosine similarity support efficient and scalable multi-modal search.

For example, we demonstrate **Text-to-Image Search using enVector**. Text-to-Image Search retrieves images whose embeddings are most similar to a user’s textual query, and is widely used in applications such as e-commerce and photo search. With enVector, the query embedding is encrypted and the server performs similarity search directly on encrypted vectors in the index without accessing the query or results in plaintext. For efficient search, we use an approximate nearest neighbor (ANN) method that scales to large datasets.

```python
import os
import json
import numpy as np
import pyenvector as ev
```

## Load Dataset

```python
data_dir = os.environ.get("IMAGE_PATH", "./images")

emb_path = os.path.join(data_dir, "embeddings.npy")
metadata_path = os.path.join(data_dir, "labels.json")
centroid_path: str = os.path.join(data_dir, "centroids.npy")
```

```python
# get embedding data
embeddings = np.load(emb_path)
embeddings /= np.linalg.norm(embeddings, axis=1, keepdims=True)
embeddings = embeddings.astype(np.float32)
print(f"Dataset Shape: {embeddings.shape}")

# get metadata to retrieve
with open(metadata_path, "r") as f:
    metadata = json.load(f)
print(f"Metadata Shape: {len(metadata)}")

assert len(embeddings) == len(metadata), "Embeddings and metadata length mismatch"
```

```
Dataset Shape: (1390353, 512)
Metadata Shape: 1390353
```

```python
# get trained centroids
n_list = 128
centroids = np.load(centroid_path).astype(np.float32)
print(f"Centroids Shape: {centroids.shape} Norm: {np.linalg.norm(centroids, axis=1).mean()}")
```

```
Centroids Shape: (128, 512) Norm: 1.0
```

## enVector Initialization

```python
# es2 initialization
ev.init(
    address="localhost:50050",
    key_path="./keys",
    key_id="tutorial_key",
)
```

```
<pyenvector.client.client.EnvectorClient at 0x7febeff299d0>
```

### Index Creation

```python
# configure index parameters
index_params = {
    "index_type": "IVF_FLAT",
    "nlist": 128,
    "default_nprobe": 1,
    "centroids": centroids.tolist()
}

# create index
index = ev.create_index(
    "txt2img_index", 
    dim=512,
    index_params=index_params,
)
```

```python
# insert data into index
ids = index.insert(embeddings, metadata)
```

```
Insert IVF_FLAT: 100%|██████████| 128/128 [08:46<00:00,  4.11s/it]
```

## Encrypted Similarity Search

### Get Query

```python
query = "My daughter rides a bicycle"
```

```python
from typing import Union, List
from fastembed import TextEmbedding

class MultiModalModel:
    def __init__(self):
        self.model = TextEmbedding("Qdrant/clip-ViT-B-32-text")

    def get_text_embedding(self, text: Union[str, List[str]]) -> np.ndarray:
        if isinstance(text, str):
            text = [text]
        text_embedding = list(self.model.embed(text))
        text_embedding = np.squeeze(text_embedding, axis=0)
        return text_embedding
```

```python
multimodal_model = MultiModalModel()
query_embedding = multimodal_model.get_text_embedding(query)
```

### Search

```python
results = index.search(query_embedding, top_k=3, output_fields=["metadata"])[0]
```

```python
from PIL import Image

top1_res = results[0]
img_path = top1_res["metadata"]["img_path"]

img = Image.open(img_path).convert("RGB")
img
```

![png](/files/Own8UbzAbq4QaFeH132Y)

enVector allows for powerful but secure cross-modal retrieval applications including multi-modal search. Multi-modal models align vectors from different modalities—such as text, images, and audio—into a shared embedding space, enabling semantic search across modalities through simple distance metrics like cosine similarity. As a result, enVector has become a core component of modern retrieval systems, allowing for flexible, scalable, and modality-agnostic search experiences!


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.envector.io/1.4.x/tutorials/advanced-use-case/multi-modal-search.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.