混合搜索

混合搜索结合了全文搜索（按关键词搜索）和语义搜索（按意义搜索），以找到既与用户查询直接相关又在上下文上相关的结果。

🌐 Hybrid search combines full text search (searching by keyword) with semantic search (searching by meaning) to identify results that are both directly and contextually relevant to the user's query.

混合搜索的使用场景 #

🌐 Use cases for hybrid search

有时候，单一的搜索方法并不能完全抓住用户真正想找的内容。比如说，如果用户在一个烹饪应用上搜索“意大利番茄酱秘诀”，关键词搜索会提取文本中明确提到“意大利”、“秘诀”和“番茄酱”的秘诀。然而，它可能会漏掉那些本质上是意大利菜、使用番茄酱但没有明确标注这些词，或者使用了像“意面酱”或者“马里纳拉酱”等变体的菜肴。另一方面，语义搜索可能会理解烹饪的上下文，找到符合意图的秘诀，比如传统的“马里纳拉意大利面”，即使它们和确切的关键词不完全匹配。但它也可能会推荐一些上下文相关但并非用户真正想找的秘诀，比如“墨西哥莎莎酱”，因为它理解的上下文是广义的基于番茄的酱料。

🌐 Sometimes a single search method doesn't quite capture what a user is really looking for. For example, if a user searches for "Italian recipes with tomato sauce" on a cooking app, a keyword search would pull up recipes that specifically mention "Italian," "recipes," and "tomato sauce" in the text. However, it might miss out on dishes that are quintessentially Italian and use tomato sauce but don't explicitly label themselves with these words, or use variations like "pasta sauce" or "marinara." On the other hand, a semantic search might understand the culinary context and find recipes that match the intent, such as a traditional "Spaghetti Marinara," even if they don't match the exact keyword phrase. However, it could also suggest recipes that are contextually related but not what the user is looking for, like a "Mexican salsa" recipe, because it understands the context to be broadly about tomato-based sauces.

混合搜索结合了这两种方法的优点。它可以确保明确提到关键词的秘诀被优先显示，从而捕捉到符合关键词条件的直接匹配。同时，它还会包含那些通过语义理解识别出的相关秘诀，比如不同的意大利菜，它们通常使用番茄酱，但可能没有明确标记用户的搜索词。这样既能找到与用户查询直接相关的结果，也能找到上下文上相关的结果，同时尽量减少遗漏和不相关的建议。

🌐 Hybrid search combines the strengths of both these methods. It would ensure that recipes explicitly mentioning the keywords are prioritized, thus capturing direct hits that satisfy the keyword criteria. At the same time, it would include recipes identified through semantic understanding as being related in meaning or context, like different Italian dishes that traditionally use tomato sauce but might not have been tagged explicitly with the user's search terms. It identifies results that are both directly and contextually relevant to the user's query while ideally minimizing misses and irrelevant suggestions.

什么时候考虑混合搜索 #

🌐 When to consider hybrid search

是否使用混合搜索取决于你的用户在应用中寻找什么内容。对于需要开发者查找精确代码行或错误信息的代码库来说，关键词搜索可能是最理想的，因为它匹配特定的术语。而在一个用户寻找与自己感受相关建议或经历的心理健康论坛中，语义搜索可能更好，因为它根据查询的含义找到结果，而不仅仅是具体的词语。对于一个购物应用，用户可能会搜索特定的产品名称，同时也愿意看到相关的推荐，混合搜索就能结合两者的优势——既找到精确匹配，又能根据购物情境发现类似产品。

🌐 The decision to use hybrid search depends on what your users are looking for in your app. For a code repository where developers need to find exact lines of code or error messages, keyword search is likely ideal because it matches specific terms. In a mental health forum where users search for advice or experiences related to their feelings, semantic search may be better because it finds results based on the meaning of a query, not only specific words. For a shopping app where customers might search for specific product names yet also be open to related suggestions, hybrid search combines the best of both worlds - finding exact matches while also uncovering similar products based on the shopping context.

如何结合搜索方法 #

🌐 How to combine search methods

混合搜索将关键词搜索和语义搜索结合在一起，但这个过程是如何运作的呢？

🌐 Hybrid search merges keyword search and semantic search, but how does this process work?

首先，每种搜索方法都是单独执行的。关键词搜索是通过查找内容中存在的特定单词或短语来进行的，它会产生自己的一系列结果。同样，语义搜索则是通过理解搜索查询背后的上下文或含义，而不是具体使用的词语来进行的，它也会产生自己独特的结果。

🌐 First, each search method is executed separately. Keyword search, which involves searching by specific words or phrases present in the content, will yield its own set of results. Similarly, semantic search, which involves understanding the context or meaning behind the search query rather than the specific words used, will generate its own unique results.

现在有了这些单独的结果列表，下一步就是把它们合并成一个统一的列表。这一步叫做“融合”。融合会把两种搜索方法的结果根据一定的排名或评分系统合并在一起。这个系统可能会根据一些因素来优先排序，比如结果与搜索查询的相关性、它们在各自列表中的排名，或者其他标准。最终得到的就是一个整合了关键词搜索和语义搜索优势的列表。

🌐 Now with these separate result lists available, the next step is to combine them into a single, unified list. This is achieved through a process known as “fusion”. Fusion takes the results from both search methods and merges them together based on a certain ranking or scoring system. This system may prioritize certain results based on factors like their relevance to the search query, their ranking in the individual lists, or other criteria. The result is a final list that integrates the strengths of both keyword and semantic search methods.

互惠排名融合 (RRF) #

🌐 Reciprocal Ranked Fusion (RRF)

最常见的融合方法之一是互惠排名融合（RRF）。RRF的核心思想是在构建最终合并列表时，对每个单独结果列表中排名靠前的项目给予更高的权重。

🌐 One of the most common fusion methods is Reciprocal Ranked Fusion (RRF). The key idea behind RRF is to give more weight to the top-ranked items in each individual result list when building the final combined list.

在 RRF 中，我们会遍历每条记录并给它们分配一个分数（注意每条记录可能存在于一个或两个列表中）。分数的计算方法是将该记录在每个列表中的排名取倒数，然后将两个列表的结果相加。例如，如果一个 ID 为 123 的记录在关键词搜索中排名第三，在语义搜索中排名第九，那么它的分数就是 $\dfrac{1}{3} + \dfrac{1}{9} = 0.444$。如果记录只出现在一个列表而不在另一个列表中，那么在另一个列表中的分数为 0。然后根据这个分数对记录进行排序，形成最终列表。分数最高的项目排名靠前，分数最低的排名靠后。

🌐 In RRF, we iterate over each record and assign a score (noting that each record could exist in one or both lists). The score is calculated as 1 divided by that record's rank in each list, summed together between both lists. For example, if a record with an ID of 123 was ranked third in the keyword search and ninth in semantic search, it would receive a score of $\dfrac{1}{3} + \dfrac{1}{9} = 0.444$. If the record was found in only one list and not the other, it would receive a score of 0 for the other list. The records are then sorted by this score to create the final list. The items with the highest scores are ranked first, and lowest scores ranked last.

这种方法确保在多个列表中排名靠前的项目在最终列表中也会得到高排名。它还确保那些只在少数列表中排名靠前但在其他列表中排名较低的项目不会在最终列表中得到高排名。在计算分数时将排名放在分母中有助于惩罚排名较低的记录。

🌐 This method ensures that items that are ranked high in multiple lists are given a high rank in the final list. It also ensures that items that are ranked high in only a few lists but low in others are not given a high rank in the final list. Placing the rank in the denominator when calculating score helps penalize the low ranking records.

平滑常数 k#

🌐 Smoothing constant k

为了防止排在第一的项目得分过高（因为我们是用排名来除的），通常会在分母中加入一个 k 常数来平滑得分：

🌐 To prevent extremely high scores for items that are ranked first (since we're dividing by the rank), a k constant is often added to the denominator to smooth the score:

$\dfrac{1}{k+rank}$

这个常数可以是任何正数，但通常很小。常数为1意味着排名第一的记录分数为 $\dfrac{1}{1+1} = 0.5$ 而不是 $1$。这种调整可以帮助在创建最终合并列表时平衡单个列表中排名很高的项目的影响。

🌐 This constant can be any positive number, but is typically small. A constant of 1 would mean that a record ranked first would have a score of $\dfrac{1}{1+1} = 0.5$ instead of $1$. This adjustment can help balance the influence of items that are ranked very high in individual lists when creating the final combined list.

Postgres 中的混合搜索 #

🌐 Hybrid search in Postgres

在 Postgres 中使用 tsvector（关键词搜索）和 pgvector（语义搜索）实现混合搜索。

🌐 Implement hybrid search in Postgres using tsvector (keyword search) and pgvector (semantic search).

首先，你可以创建一个 documents 表来存储可以搜索的文档。这只是一个示例。根据你的应用结构进行调整。

🌐 First, you can create a documents table to store the documents that you can search over. This is an example. Adjust this to match the structure of your application.

1
create table documents (
2
  id bigint primary key generated always as identity,
3
  content text,
4
  fts tsvector generated always as (to_tsvector('english', content)) stored,
5
  embedding extensions.vector(512)
6
);

这张表有4列：

🌐 The table contains 4 columns:

• id 是记录的自动生成唯一 ID。我们稍后在执行 RRF 时会用它来匹配记录。
• content 包含我们将要搜索的实际文本。
• fts 是一个自动生成的 tsvector 列，它是根据 content 中的文本生成的。我们会用它来进行全文搜索（按关键词搜索）。
• embedding 是一个 向量列，用于存储我们嵌入模型生成的向量。我们会用它进行 语义搜索（按意思搜索）。这个例子里我们选择了 512 维，但你可以根据你选用的模型生成的嵌入向量大小调整这个数值。

接下来我们将在 fts 和 embedding 列上创建索引，这样它们各自的查询在大规模数据下依旧会很快:

🌐 Next we'll create indexes on the fts and embedding columns so that their individual queries will remain fast at scale:

1
-- Create an index for the full-text search
2
create index on documents using gin(fts);
3
4
-- Create an index for the semantic vector search
5
create index on documents using hnsw (embedding vector_ip_ops);

对于全文搜索，我们使用 广义倒排（GIN）索引，它专为处理像存储在 tsvector 中的复合值而设计。

🌐 For full text search we use a generalized inverted (GIN) index which is designed for handling composite values like those stored in a tsvector.

对于语义向量搜索，我们使用的是 HNSW 索引，这是一种高性能的近似最近邻（ANN）搜索算法。注意，我们在使用该索引时采用的是 vector_ip_ops（内积）操作符，因为我们计划在后续查询中使用内积（<#>）操作符。如果你打算使用其他操作符，比如余弦距离（<=>），请务必相应地更新索引。更多信息，请参见 距离操作符。

🌐 For semantic vector search we use an HNSW index, which is a high performing approximate nearest neighbor (ANN) search algorithm. Note that we are using the vector_ip_ops (inner product) operator with this index because we plan on using the inner product (<#>) operator later in our query. If you plan to use a different operator like cosine distance (<=>), be sure to update the index accordingly. For more information, see distance operators.

最后我们将创建我们的 hybrid_search 函数：

🌐 Finally we'll create our hybrid_search function:

1
create or replace function hybrid_search(
2
  query_text text,
3
  query_embedding extensions.vector(512),
4
  match_count int,
5
  full_text_weight float = 1,
6
  semantic_weight float = 1,
7
  rrf_k int = 50
8
)
9
returns setof documents
10
language sql
11
as $$
12
with full_text as (
13
  select
14
    id,
15
    -- Note: ts_rank_cd is not indexable but will only rank matches of the where clause
16
    -- which shouldn't be too big
17
    row_number() over(order by ts_rank_cd(fts, websearch_to_tsquery(query_text)) desc) as rank_ix
18
  from
19
    documents
20
  where
21
    fts @@ websearch_to_tsquery(query_text)
22
  order by rank_ix
23
  limit least(match_count, 30) * 2
24
),
25
semantic as (
26
  select
27
    id,
28
    row_number() over (order by embedding <#> query_embedding) as rank_ix
29
  from
30
    documents
31
  order by rank_ix
32
  limit least(match_count, 30) * 2
33
)
34
select
35
  documents.*
36
from
37
  full_text
38
  full outer join semantic
39
    on full_text.id = semantic.id
40
  join documents
41
    on coalesce(full_text.id, semantic.id) = documents.id
42
order by
43
  coalesce(1.0 / (rrf_k + full_text.rank_ix), 0.0) * full_text_weight +
44
  coalesce(1.0 / (rrf_k + semantic.rank_ix), 0.0) * semantic_weight
45
  desc
46
limit
47
  least(match_count, 30)
48
$$;

在哪里：

🌐 Where:

• 参数： 这个函数接受很多参数，但主要（必需）的参数是 query_text、query_embedding 和 match_count。

  • query_text 是用户的查询文本（稍后会详细说明）
  • query_embedding 是由嵌入模型生成的用户查询的向量表示。我们在这个例子中选择了 512 维，但可以根据你喜欢的模型生成的嵌入向量的大小进行调整。这必须与 documents 表上的 embedding 向量大小匹配（并使用相同的模型）。
  • match_count 是在 limit 子句中返回的记录数。

  其他参数是可选的，但可以更好地控制融合过程。

  • full_text_weight 和 semantic_weight 决定每种搜索方法在最终得分中占多少权重。默认情况下，它们都是 1，这意味着它们对最终排名的贡献是相等的。如果 full_text_weight 是 2 而 semantic_weight 是 1，那么全文搜索的权重就是语义搜索的两倍。
  • rrf_k 是添加到倒数排名中的 k 平滑常数。默认值是50。

• 返回类型： 该函数从我们的 documents 表返回一组记录。

• CTE: 我们创建了两个公共表表达式（CTE），一个用于全文搜索，一个用于语义搜索。在将它们合并之前，这两个查询会各自独立执行。

• RRF: 最终查询使用 互惠排名融合 (RRF) 将两个 CTE 的结果合并。

运行混合搜索 #

🌐 Running hybrid search

在 SQL 中使用这个函数，我们可以运行：

🌐 To use this function in SQL, we can run:

1
select
2
  *
3
from
4
  hybrid_search(
5
    'Italian recipes with tomato sauce', -- user query
6
    '[...]'::extensions.vector(512), -- embedding generated from user query
7
    10
8
  );

实际上，你很可能会通过 Supabase 客户端 或自定义后端层来调用它。下面是一个快速示例，展示了如何使用 JavaScript 从 Edge Function 调用它：

🌐 In practice, you will likely be calling this from the Supabase client or through a custom backend layer. Here is a quick example of how you might call this from an Edge Function using JavaScript:

1
import { createClient } from 'npm:@supabase/supabase-js@2'
2
import OpenAI from 'npm:openai'
3
4
const supabaseUrl = Deno.env.get('SUPABASE_URL')!
5
const supabaseSecretKey = Deno.env.get('SUPABASE_SECRET_KEY')!
6
const openaiApiKey = Deno.env.get('OPENAI_API_KEY')!
7
8
Deno.serve(async (req) => {
9
  // Grab the user's query from the JSON payload
10
  const { query } = await req.json()
11
12
  // Instantiate OpenAI client
13
  const openai = new OpenAI({ apiKey: openaiApiKey })
14
15
  // Generate a one-time embedding for the user's query
16
  const embeddingResponse = await openai.embeddings.create({
17
    model: 'text-embedding-3-large',
18
    input: query,
19
    dimensions: 512,
20
  })
21
22
  const [{ embedding }] = embeddingResponse.data
23
24
  // Instantiate the Supabase client
25
  // (replace service role key with user's JWT if using Supabase auth and RLS)
26
  const supabase = createClient(supabaseUrl, supabaseServiceRoleKey)
27
28
  // Call hybrid_search Postgres function via RPC
29
  const { data: documents } = await supabase.rpc('hybrid_search', {
30
    query_text: query,
31
    query_embedding: embedding,
32
    match_count: 10,
33
  })
34
35
  return new Response(JSON.stringify(documents), {
36
    headers: { 'Content-Type': 'application/json' },
37
  })
38
})

这使用 OpenAI 的 text-embedding-3-large 模型来生成嵌入（缩短到 512 维以加快检索速度）。可以根据需要替换为你喜欢的嵌入模型（和维度大小）。

🌐 This uses OpenAI's text-embedding-3-large model to generate embeddings (shortened to 512 dimensions for faster retrieval). Swap in your preferred embedding model (and dimension size) accordingly.

要测试这个，向函数的端点发送一个 POST 请求，同时传入包含用户查询的 JSON 数据。下面是一个使用 cURL 的 POST 请求示例：

🌐 To test this, make a POST request to the function's endpoint while passing in a JSON payload containing the user's query. Here is an example POST request using cURL:

1
curl -i --location --request POST \
2
  'http://127.0.0.1:54321/functions/v1/hybrid-search' \
3
  --header 'Authorization: Bearer <anonymous key>' \
4
  --header 'Content-Type: application/json' \
5
  --data '{"query":"Italian recipes with tomato sauce"}'

想了解如何创建、测试和部署边缘函数的更多信息，请参见 入门指南。

🌐 For more information on how to create, test, and deploy edge functions, see Getting started.

另请参阅 #

🌐 See also

• 嵌入概念
• 向量列
• 向量索引
• 语义搜索
• 全文（关键词）搜索