Azure AI Search relevance tuning for RAG - hybrid queries, semantic ranker, scoring profiles and measuring the result

#Start by measuring

Before changing anything, build a small retrieval test set: 50-100 real questions, each with the document (or chunk) IDs that should be found. Then measure recall@k (is a correct chunk in the top k?) for each configuration you try. Without this, tuning is guesswork.

def recall_at_k(results, expected_ids, k=5):
    return int(any(r["id"] in expected_ids for r in results[:k]))

#1. Hybrid search (keyword + vector)

Pure vector search misses exact terms such as error codes, product names and IDs. Pure keyword search misses paraphrases. Hybrid runs both and merges them with Reciprocal Rank Fusion (RRF).

POST https://<service>.search.windows.net/indexes/kb/docs/search?api-version=2024-07-01
Content-Type: application/json
api-key: <query-key>

{
  "search": "AKS node NotReady after upgrade",
  "vectorQueries": [
    { "kind": "text", "text": "AKS node NotReady after upgrade", "fields": "contentVector", "k": 50 }
  ],
  "select": "id,title,chunk,source,updated",
  "top": 10
}

• kind: text needs an integrated vectorizer defined on the index. Otherwise send kind: vector with your own embedding.
• Set the vector k higher than top (for example 50) so RRF has enough candidates to merge.

#2. Semantic ranker

The semantic ranker re-scores the top results (up to 50) with a cross-encoder model. It's usually the single biggest relevance gain for natural-language questions.

Define a semantic configuration on the index:

"semantic": {
  "configurations": [{
    "name": "default",
    "prioritizedFields": {
      "titleField": { "fieldName": "title" },
      "prioritizedContentFields": [{ "fieldName": "chunk" }],
      "prioritizedKeywordsFields": [{ "fieldName": "tags" }]
    }
  }]
}

Query with it:

{
  "search": "AKS node NotReady after upgrade",
  "vectorQueries": [{ "kind": "text", "text": "AKS node NotReady after upgrade", "fields": "contentVector", "k": 50 }],
  "queryType": "semantic",
  "semanticConfiguration": "default",
  "captions": "extractive",
  "top": 10
}

Use @search.rerankerScore to set a cut-off. Low-scoring chunks often do more harm than good in the prompt.

Tip

The semantic ranker is billed separately and has its own limits. Check current pricing and availability for your tier and region before enabling it for all traffic.

#3. Filters

Narrow the search space before ranking. Mark filter fields filterable in the index.

"filter": "product eq 'aks' and doc_type ne 'archived' and updated ge 2025-01-01T00:00:00Z"

For security trimming, filter on a group field (groups/any(g: search.in(g, 'grp1,grp2'))) using the user's group IDs, or use the built-in document-level access control features where available.

For vector queries, vectorFilterMode: "preFilter" (the default in recent API versions) applies the filter before the vector search, so you still get k results that pass it.

#4. Scoring profiles (boost fresh or authoritative content)

"scoringProfiles": [{
  "name": "fresh-and-official",
  "text": { "weights": { "title": 3, "chunk": 1 } },
  "functions": [
    { "type": "freshness", "fieldName": "updated", "boost": 2,
      "interpolation": "quadratic", "freshness": { "boostingDuration": "P365D" } },
    { "type": "tag", "fieldName": "tags", "boost": 2, "tag": { "tagsParameter": "boosttags" } }
  ]
}]

Query with "scoringProfile": "fresh-and-official", "scoringParameters": ["boosttags-runbook,official"].

Scoring profiles affect the keyword side. When the semantic ranker is used, it re-ranks the final top results, so check whether the boost survives. Recent API versions have options for applying scoring profiles after semantic ranking; check the documentation for your version.

#5. Chunk and field design

Ranking can't fix bad chunks:

• Put a title or heading path in its own field and in the chunk text ("AKS > Troubleshooting > Node NotReady").
• Keep chunks focused (a few hundred tokens), with some overlap.
• Store source, updated, doc_type and product as filterable fields.
• Add synonyms (synonym maps) for internal abbreviations.

#6. Compare configurations

from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery
from azure.identity import DefaultAzureCredential

client = SearchClient("https://<service>.search.windows.net", "kb", DefaultAzureCredential())

def run(q, semantic=False):
    kw = dict(search_text=q, top=10, select=["id"],
              vector_queries=[VectorizableTextQuery(text=q, k_nearest_neighbors=50, fields="contentVector")])
    if semantic:
        kw.update(query_type="semantic", semantic_configuration_name="default")
    return list(client.search(**kw))

score = sum(recall_at_k(run(q, semantic=True), ids) for q, ids in testset) / len(testset)
print("recall@5:", score)

Track recall@5, recall@10 and latency for: keyword only, vector only, hybrid, and hybrid + semantic. Keep the best one and re-run the test after every index change.

#Common problems

#Checklist

• Retrieval test set with expected IDs
• Hybrid search with vector k well above top
• Semantic configuration and reranker score cut-off
• Filters for scope and security trimming
• Freshness/authority boosting tested
• Recall measured after every change