使用 Mixpeek 多模态嵌入进行视频搜索

Implement video search with the Mixpeek Multimodal Embed API and Supabase Vector.

Mixpeek Embed API 让你可以为各种类型的内容生成嵌入，包括视频和文本。你可以将这些嵌入用于：

🌐 The Mixpeek Embed API allows you to generate embeddings for various types of content, including videos and text. You can use these embeddings for:

文本到视频 / 视频到文本 / 视频到视频 / 文本到文本搜索
在你自己的视频和文本数据上进行微调

本指南演示如何使用 Mixpeek Embed 进行视频处理和嵌入，以及使用 Supabase Vector 来存储和查询嵌入，从而实现视频搜索。

🌐 This guide demonstrates how to implement video search using Mixpeek Embed for video processing and embedding, and Supabase Vector for storing and querying embeddings.

用 Poetry 创建一个新的 Python 项目 #

🌐 Create a new Python project with Poetry

Poetry 为 Python 提供了打包和依赖管理。如果你还没有安装，可以通过 pip 安装 poetry：

1
pip install poetry

然后初始化一个新项目：

🌐 Then initialize a new project:

1
poetry new video-search

设置 Supabase 项目 #

🌐 Setup Supabase project

如果你还没做过，安装 Supabase CLI，然后在你新创建的 poetry 项目的根目录中初始化 Supabase：

🌐 If you haven't already, install the Supabase CLI, then initialize Supabase in the root of your newly created poetry project:

1
supabase init

接下来，启动你本地的 Supabase 环境：

🌐 Next, start your local Supabase stack:

1
supabase start

这将会在本地启动 Supabase 堆栈，并打印出一堆环境详情，包括你本地的 DB URL。记下来以便以后使用。

🌐 This will start up the Supabase stack locally and print out a bunch of environment details, including your local DB URL. Make a note of that for later use.

安装依赖 #

🌐 Install the dependencies

把以下依赖添加到你的项目里：

🌐 Add the following dependencies to your project:

supabase：Supabase Python 客户端
mixpeek：Mixpeek Python 客户端用于生成嵌入

1
poetry add supabase mixpeek

导入必要的依赖 #

🌐 Import the necessary dependencies

在你的主 Python 脚本顶部，导入依赖并存储你的环境变量：

🌐 At the top of your main Python script, import the dependencies and store your environment variables:

1
from supabase import create_client, Client
2
from mixpeek import Mixpeek
3
import os
4
5
SUPABASE_URL = os.getenv("SUPABASE_URL")
6
SUPABASE_KEY = os.getenv("SUPABASE_API_KEY")
7
MIXPEEK_API_KEY = os.getenv("MIXPEEK_API_KEY")

为你的视频创建嵌入 #

🌐 Create embeddings for your videos

接下来，创建一个 seed 方法，它将创建一个新的 Supabase 表，为你的视频片段生成嵌入，并将嵌入插入到你的数据库中：

🌐 Next, create a seed method, which will create a new Supabase table, generate embeddings for your video chunks, and insert the embeddings into your database:

1
def seed():
2
    # Initialize Supabase and Mixpeek clients
3
    supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)
4
    mixpeek = Mixpeek(MIXPEEK_API_KEY)
5
6
    # Create a table for storing video chunk embeddings
7
    supabase.table("video_chunks").create({
8
        "id": "text",
9
        "start_time": "float8",
10
        "end_time": "float8",
11
        "embedding": "extensions.vector(768)",
12
        "metadata": "jsonb"
13
    })
14
15
    # Process and embed video
16
    video_url = "https://example.com/your_video.mp4"
17
    processed_chunks = mixpeek.tools.video.process(
18
        video_source=video_url,
19
        chunk_interval=1,  # 1 second intervals
20
        resolution=[720, 1280]
21
    )
22
23
    for chunk in processed_chunks:
24
        print(f"Processing video chunk: {chunk['start_time']}")
25
26
        # Generate embedding using Mixpeek
27
        embed_response = mixpeek.embed.video(
28
            model_id="vuse-generic-v1",
29
            input=chunk['base64_chunk'],
30
            input_type="base64"
31
        )
32
33
        # Insert into Supabase
34
        supabase.table("video_chunks").insert({
35
            "id": f"chunk_{chunk['start_time']}",
36
            "start_time": chunk["start_time"],
37
            "end_time": chunk["end_time"],
38
            "embedding": embed_response['embedding'],
39
            "metadata": {"video_url": video_url}
40
        }).execute()
41
42
    print("Video processed and embeddings inserted")
43
44
    # Create index for fast search performance
45
    supabase.query("CREATE INDEX ON video_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100)").execute()
46
    print("Created index")

把这个方法作为脚本添加到你的 pyproject.toml 文件里：

🌐 Add this method as a script in your pyproject.toml file:

1
[tool.poetry.scripts]
2
seed = "video_search.main:seed"
3
search = "video_search.main:search"

在使用 poetry shell 激活虚拟环境后，你现在可以通过 poetry run seed 运行你的 seed 脚本。你可以通过访问本地 Supabase 仪表板 localhost:54323 来查看生成的 embeddings。

🌐 After activating the virtual environment with poetry shell, you can now run your seed script via poetry run seed. You can inspect the generated embeddings in your local database by visiting the local Supabase dashboard at localhost:54323.

通过文本查询进行视频搜索 #

🌐 Perform a video search from a text query

使用 Supabase Vector，你可以查询你的嵌入。你可以用一个视频片段作为搜索输入，或者，你也可以从字符串输入生成一个嵌入，并用它作为查询输入：

🌐 With Supabase Vector, you can query your embeddings. You can use either a video clip as search input or alternatively, you can generate an embedding from a string input and use that as the query input:

1
def search():
2
    # Initialize Supabase and Mixpeek clients
3
    supabase: Client = create_client(SUPABASE_URL, SUPABASE_KEY)
4
    mixpeek = Mixpeek(MIXPEEK_API_KEY)
5
6
    # Generate embedding for text query
7
    query_string = "a car chase scene"
8
    text_emb = mixpeek.embed.video(
9
        model_id="vuse-generic-v1",
10
        input=query_string,
11
        input_type="text"
12
    )
13
14
    # Query the collection
15
    results = supabase.rpc(
16
        'match_video_chunks',
17
        {
18
            'query_embedding': text_emb['embedding'],
19
            'match_threshold': 0.8,
20
            'match_count': 5
21
        }
22
    ).execute()
23
24
    # Display the results
25
    if results.data:
26
        for result in results.data:
27
            print(f"Matched chunk from {result['start_time']} to {result['end_time']} seconds")
28
            print(f"Video URL: {result['metadata']['video_url']}")
29
            print(f"Similarity: {result['similarity']}")
30
            print("---")
31
    else:
32
        print("No matching video chunks found")

这个查询会从你的数据库中返回最相似的前5个视频片段。

🌐 This query will return the top 5 most similar video chunks from your database.

你现在可以通过运行 poetry run search 来测试它，你将会看到与查询“追车场景”最相关的视频片段。

🌐 You can now test it out by running poetry run search, and you will be presented with the most relevant video chunks to the query "a car chase scene".

结论 #

🌐 Conclusion

通过几个 Python 脚本，你就可以使用 Mixpeek Embed 和 Supabase Vector 实现视频搜索以及反向视频搜索。这种方法可以提供语义搜索功能，能够集成到各种应用中，让你可以通过文本和视频查询搜索视频内容。

🌐 With a couple of Python scripts, you are able to implement video search as well as reverse video search using Mixpeek Embed and Supabase Vector. This approach allows for semantic search capabilities that can be integrated into various applications, enabling you to search through video content using both text and video queries.