Skip to content
AI & Vectors

使用 OpenAI CLIP 进行图片搜索

Implement image search with the OpenAI CLIP Model and Supabase Vector.

OpenAI CLIP 模型 是在各种（图片，文本）对上训练的。你可以使用 CLIP 模型来：

🌐 The OpenAI CLIP Model was trained on a variety of (image, text)-pairs. You can use the CLIP model for:

  • 文本到图片 / 图片到文本 / 图片到图片 / 文本到文本 搜索
  • 你可以用常规的 SentenceTransformers 训练代码，在你自己的图片和文本数据上进行微调。

SentenceTransformers 提供了可以将图片和文本嵌入到同一向量空间的模型。你可以用它来找到相似的图片，也可以用来实现图片搜索。

你可以在 GitHub 上找到作为 Python Poetry 项目的完整应用代码。

🌐 You can find the full application code as a Python Poetry project on GitHub.

用 Poetry 创建一个新的 Python 项目 #

🌐 Create a new Python project with Poetry

Poetry 为 Python 提供了打包和依赖管理。如果你还没有安装，可以通过 pip 安装 poetry：

1
pip install poetry

然后初始化一个新项目：

🌐 Then initialize a new project:

1
poetry new image-search

设置 Supabase 项目 #

🌐 Setup Supabase project

如果你还没做过，安装 Supabase CLI，然后在你新创建的 poetry 项目的根目录中初始化 Supabase：

🌐 If you haven't already, install the Supabase CLI, then initialize Supabase in the root of your newly created poetry project:

1
supabase init

接下来，启动你本地的 Supabase 环境：

🌐 Next, start your local Supabase stack:

1
supabase start

这将会在本地启动 Supabase 堆栈，并打印出一堆环境详情，包括你本地的 DB URL。记下来以后用。

🌐 This will start up the Supabase stack locally and print out a bunch of environment details, including your local DB URL. Make a note of that for later user.

安装依赖 #

🌐 Install the dependencies

我们需要在项目中添加以下依赖：

🌐 We will need to add the following dependencies to our project:

  • vecs：Supabase 向量 Python 客户端。
  • sentence-transformers：一个用于句子、文本和图片嵌入的框架（与 OpenAI CLIP 模型一起使用）
  • matplotlib：用于显示我们的图片结果
1
poetry add vecs sentence-transformers matplotlib

导入必要的依赖 #

🌐 Import the necessary dependencies

在你的主 Python 脚本顶部，导入依赖，并将上面提到的 DB URL 存储在一个变量中：

🌐 At the top of your main python script, import the dependencies and store your DB URL from above in a variable:

1
from PIL import Image
2
from sentence_transformers import SentenceTransformer
3
import vecs
4
from matplotlib import pyplot as plt
5
from matplotlib import image as mpimg
6
7
DB_CONNECTION = "postgresql://postgres:postgres@localhost:54322/postgres"

为你的图片创建嵌入 #

🌐 Create embeddings for your images

在你的项目根目录下，创建一个名为 images 的新文件夹并添加一些图片。你可以使用 GitHub 上示例项目中的图片，或者在 Unsplash 上找到免费的图片。

🌐 In the root of your project, create a new folder called images and add some images. You can use the images from the example project on GitHub or you can find license free images on Unsplash.

接下来，创建一个 seed 方法，它将创建一个新的 Supabase 向量集合，为你的图片生成嵌入，并将这些嵌入插入或更新到你的数据库中：

🌐 Next, create a seed method, which will create a new Supabase Vector Collection, generate embeddings for your images, and upsert the embeddings into your database:

1
def seed():
2
    # create vector store client
3
    vx = vecs.create_client(DB_CONNECTION)
4
5
    # create a collection of vectors with 3 dimensions
6
    images = vx.get_or_create_collection(name="image_vectors", dimension=512)
7
8
    # Load CLIP model
9
    model = SentenceTransformer('clip-ViT-B-32')
10
11
    # Encode an image:
12
    img_emb1 = model.encode(Image.open('./images/one.jpg'))
13
    img_emb2 = model.encode(Image.open('./images/two.jpg'))
14
    img_emb3 = model.encode(Image.open('./images/three.jpg'))
15
    img_emb4 = model.encode(Image.open('./images/four.jpg'))
16
17
    # add records to the *images* collection
18
    images.upsert(
19
        records=[
20
            (
21
                "one.jpg",        # the vector's identifier
22
                img_emb1,          # the vector. list or np.array
23
                {"type": "jpg"}   # associated  metadata
24
            ), (
25
                "two.jpg",
26
                img_emb2,
27
                {"type": "jpg"}
28
            ), (
29
                "three.jpg",
30
                img_emb3,
31
                {"type": "jpg"}
32
            ), (
33
                "four.jpg",
34
                img_emb4,
35
                {"type": "jpg"}
36
            )
37
        ]
38
    )
39
    print("Inserted images")
40
41
    # index the collection for fast search performance
42
    images.create_index()
43
    print("Created index")

把这个方法作为脚本添加到你的 pyproject.toml 文件里：

🌐 Add this method as a script in your pyproject.toml file:

1
[tool.poetry.scripts]
2
seed = "image_search.main:seed"
3
search = "image_search.main:search"

在使用 poetry shell 激活虚拟环境后，你现在可以通过 poetry run seed 运行你的种子脚本。你可以通过访问本地 Supabase 仪表板 localhost:54323，选择 vecs 模式和 image_vectors 数据库，来查看生成的嵌入数据。

🌐 After activating the virtual environment with poetry shell you can now run your seed script via poetry run seed. You can inspect the generated embeddings in your local database by visiting the local Supabase dashboard at localhost:54323, selecting the vecs schema, and the image_vectors database.

根据文字查询进行图片搜索 #

🌐 Perform an image search from a text query

使用 Supabase Vector 我们可以查询我们的嵌入。我们可以使用图片作为搜索输入，或者也可以从字符串输入生成嵌入，并将其用作查询输入：

🌐 With Supabase Vector we can query our embeddings. We can use either an image as search input or alternative we can generate an embedding from a string input and use that as the query input:

1
def search():
2
    # create vector store client
3
    vx = vecs.create_client(DB_CONNECTION)
4
    images = vx.get_or_create_collection(name="image_vectors", dimension=512)
5
6
    # Load CLIP model
7
    model = SentenceTransformer('clip-ViT-B-32')
8
    # Encode text query
9
    query_string = "a bike in front of a red brick wall"
10
    text_emb = model.encode(query_string)
11
12
    # query the collection filtering metadata for "type" = "jpg"
13
    results = images.query(
14
        data=text_emb,                      # required
15
        limit=1,                            # number of records to return
16
        filters={"type": {"$eq": "jpg"}},   # metadata filters
17
    )
18
    result = results[0]
19
    print(result)
20
    plt.title(result)
21
    image = mpimg.imread('./images/' + result)
22
    plt.imshow(image)
23
    plt.show()

通过将查询限制为一个结果，我们可以向用户显示最相关的图片。最后，我们使用 matplotlib 向用户展示图片结果。

🌐 By limiting the query to one result, we can show the most relevant image to the user. Finally we use matplotlib to show the image result to the user.

去试试看吧，运行 poetry run search，你就会看到一张“红砖墙前的自行车”的图片。

🌐 Go ahead and test it out by running poetry run search and you will be presented with an image of a "bike in front of a red brick wall".

结论 #

🌐 Conclusion

只需几行 Python 代码，你就可以使用 OpenAI 的 CLIP 模型和 Supabase 向量实现图片搜索以及反向图片搜索。

🌐 With a couple of lines of Python you are able to implement image search as well as reverse image search using OpenAI's CLIP model and Supabase Vector.