Filtering with metadata

You can limit your vector search based on metadata. Pinecone lets you attach metadata key-value pairs to vectors in an index, and specify filter expressions when you query the index.

Searches with metadata filters retrieve exactly the number of nearest-neighbor results that match the filters. For most cases, the search latency will be even lower than unfiltered searches.

Searches without metadata filters do not consider metadata. To combine keywords with semantic search, see sparse-dense embeddings.

For more background information on metadata filtering, see: The Missing WHERE Clause in Vector Search.

Supported metadata types

You can associate a metadata payload with each vector in an index, as key-value pairs in a JSON object where keys are strings and values are one of:

• String
• Number (integer or floating point, gets converted to a 64 bit floating point)
• Booleans (true, false)
• List of String

Null metadata values are not supported. Instead of setting a key to hold a
null value, we recommend you remove that key from the metadata payload.

For example, the following would be valid metadata payloads:

JSON

{
    "genre": "action",
    "year": 2020,
    "length_hrs": 1.5
}

{
    "color": "blue",
    "fit": "straight",
    "price": 29.99,
    "is_jeans": true
}

Supported metadata size

Pinecone supports 40kb of metadata per vector.

Metadata query language

Pinecone’s filtering query language is based on MongoDB’s query and projection operators. We
currently support a subset of those selectors.

The metadata filters can be combined with AND and OR:

• $eq - Equal to (number, string, boolean)
• $ne - Not equal to (number, string, boolean)
• $gt - Greater than (number)
• $gte - Greater than or equal to (number)
• $lt - Less than (number)
• $lte - Less than or equal to (number)
• $in - In array (string or number)
• $nin - Not in array (string or number)
• $exists - Has the specified metadata field (boolean)

Using arrays of strings as metadata values or as metadata filters

A vector with metadata payload…

JSON

{ "genre": ["comedy", "documentary"] }

…means the "genre" takes on both values.

For example, queries with the following filters will match the vector:

JSON

{"genre":"comedy"}

{"genre": {"$in":["documentary","action"]}}

{"$and": [{"genre": "comedy"}, {"genre":"documentary"}]}

Queries with the following filter will not match the vector:

JSON

{ "$and": [{ "genre": "comedy" }, { "genre": "drama" }] }

And queries with the following filters will not match the vector because they are invalid. They will result in a query compilation error:

# INVALID QUERY:
{"genre": ["comedy", "documentary"]}

# INVALID QUERY:
{"genre": {"$eq": ["comedy", "documentary"]}}

Inserting metadata into an index

Metadata can be included in upsert requests as you insert your vectors.

For example, here’s how to insert vectors with metadata representing movies into an index:

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")
index = pc.Index("pinecone-index")

index.upsert(
  vectors=[
    {
      "id": "A", 
      "values": [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1], 
      "metadata": {"genre": "comedy", "year": 2020}
    },
    {
      "id": "B", 
      "values": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
      "metadata": {"genre": "documentary", "year": 2019}
    },
    {
      "id": "C", 
      "values": [0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3],
      "metadata": {"genre": "comedy", "year": 2019}
    },
    {
      "id": "D", 
      "values": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4],
      "metadata": {"genre": "drama"}
    }
  ]
)

Querying an index with metadata filters

Metadata filter expressions can be included with queries to limit the search to only vectors matching the filter expression.

For example, we can search the previous movies index for documentaries from the year 2019. This also uses the include_metadata flag so that vector metadata is included in the response.

For optimal performance, when querying pod-based indexes with top_k over 1000, avoid returning vector data (include_values=True) or metadata (include_metadata=True)

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")
index = pc.Index("pinecone-index")

index.query(
    vector=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1],
    filter={
        "genre": {"$eq": "documentary"},
        "year": 2019
    },
    top_k=1,
    include_metadata=True
)

# Returns:
# {'matches': [{'id': 'B',
#               'metadata': {'genre': 'documentary', 'year': 2019.0},
#               'score': 0.0800000429,
#               'values': []}],
#  'namespace': ''}

More example filter expressions

A comedy, documentary, or drama:

JSON

{
  "genre": { "$in": ["comedy", "documentary", "drama"] }
}

A drama from 2020 or later:

JSON

{
  "genre": { "$eq": "drama" },
  "year": { "$gte": 2020 }
}``
`

A drama from 2020 or later (equivalent to the previous example):



```JSON JSON
{
  "$and": [{ "genre": { "$eq": "drama" } }, { "year": { "$gte": 2020 } }]
}

A drama or a movie from 2020 or later

JSON

{
  "$or": [{ "genre": { "$eq": "drama" } }, { "year": { "$gte": 2020 } }]
}

Deleting vectors by metadata filter

Serverless and starter indexes do not support deleting by metadata. For serverless indexes, you can delete records by ID prefix instead.

To use metadata values to select vectors to be deleted, pass a metadata filter expression to the delete operation. This deletes all vectors matching the metadata filter expression.

Example

This example deletes all vectors with genre “documentary” and year 2019 from an index.

Python

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")
index = pc.Index("pinecone-index")

index.delete(
    filter={
        "genre": {"$eq": "documentary"},
        "year": 2019
    }
)

Managing high-cardinality in pod-based indexes

Cardinality is an important factor to consider when managing an index, as high cardinality can lead to performance issues, pod fullness, and a reduction in the number of possible vectors that can fit per pod.

For pod-based indexes, Pinecone indexes all metadata by default. When metadata contains many unique values, pod-based indexes will consume significantly more memory, which can lead to performance issues, pod fullness, and a reduction in the number of possible vectors that fit per pod.

To avoid indexing high-cardinality metadata that is not needed for filtering, use selective metadata indexing, which lets you specify which fields need to be indexed and which do not, helping to reduce the overall cardinality of the metadata index while still ensuring that the necessary fields are able to be filtered.

To set up selective metadata indexing, use the Pinecone API. With the API, you can create a metadata index and specify which fields need to be indexed. This will allow you to reduce the cardinality of the index while still ensuring that all of the necessary fields are indexed.

Serverless indexes are not affected by high-cardinality metadata.

Considerations for serverless indexes

For each serverless index, Pinecone clusters records that are likely to be queried together. When you query a serverless index with a metadata filter, Pinecone first uses internal metadata statistics to exclude clusters that do not have records matching the filter and then chooses the most relevant remaining clusters.

Note the following considerations:

• When you use a highly selective metadata filter (i.e., a filter that rejects the vast majority of records in the index), the chosen clusters may not contain enough matching records to satisfy the designated top_k.

• When filtering by numeric metadata that cannot be ordered in a meaningful way (e.g., IDs as opposed to dates or prices), the chosen clusters may not be accurate. This is because the metadata statistics for each cluster reflect the min and max metadata values in the cluster, and min and max are not helpful when there is no meaningful order.

  In such cases, it is best to store the metadata as strings instead of numbers. For string metadata, the metadata statistics for each cluster reflect a sampling of the values, which is much more effective when there is no meaningful order.

For more details about query execution, see Serverless architecture.