How to Use Query Manipulation
How to Use the Query Manipulation APIs and Query Profiles

How to Use Query Manipulation

Haven OnDemand Query Manipulation allows you to modify queries and query results in your applications.

There are three types of query manipulation that you can use in Haven OnDemand:

  • Promotions. Return additional promoted content with the query results. The additional content might be a specific document or set of documents, or dynamic results from a separate promotion query. For example, you might want to return a featured article, or results from a text index that contains details of special offers. You can return a document separately, or add it to the normal query results at a specified position.
  • Synonyms. Modify a query with synonymous terms. For example, you might want to expand a query for SDK to also return results for Software Development Kit.
  • Blacklists. Modify a query to remove terms that you define. For example, you might want to prevent users from searching for offensive words.

The following sections provide information about how to set up query manipulation, and to create query profiles to use to manipulate queries. For more information about the different types of rules available, see Query Manipulation.

Key Concepts

This section describes the things you need to know about before you set up query manipulation. Also see Text Indexing - Key Concepts for general information about Haven OnDemand text indexing.

Query Manipulation Text Index

To use query manipulation you must create a text index with the query manipulation flavor. Haven OnDemand text indexes store content for easy search and retrieval. For more information about text indexes, see Text Indexing - Key Concepts.

The query manipulation text index does not store ordinary index content, but it stores query manipulation rules.

Query Manipulation Rules

Query manipulation rules define when to apply a particular rule, and what to do when that rule is activated.

The rules are index documents in the query manipulation text index (see Text Indexing - Key Concepts). You create a JSON object, similar to an ordinary content index document, with a defined set of document fields. The fields describe the queries that you want the rule to match, and the manipulation to perform when it matches. For example, it might contain a list of blacklist terms, or the details of a query to use to find promotion documents.

Query Profiles

To use query manipulation, you must also create query profiles. Each query profile defines the types of manipulation to run (promotions, synonyms, blacklists, or a combination of the three), and refers to a query manipulation text index that contains the rules. You can also optionally restrict the rules that the profile uses by defining rule categories.

You can create multiple query profiles for different uses, and apply them to different types of queries. For example, you might create a technical query profile and a commercial query profile, which activate different sets of rules.

Rule Categories

Rule categories allow you to tailor your query profiles. You define categories in your query profiles, and in your rules. The rule activates only if the rule category matches one of the categories you set in the query profile.

For the example where you have two query profiles, technical, and commercial, you might have two sets of rules:

  • rules aimed at technical users to promote advanced content.
  • rules aimed at commercial users to promote sales and marketing materials.

You can use profile information for your application users to activate the appropriate query profile. In both cases you might have a rule that activates when you query for SDK, but you can use different categories to promote different documents in each case.

Create a Query Manipulation Text Index

To use query manipulation, you must create a text index with the query manipulation flavor to store the query profiles and rules. For more information about this flavor, see Query Manipulation Flavor.

You can create a query manipulation text index by using the Create Text Index API, or by using the Create Text Index wizard on the account page.

The following example Create Text Index API call creates an index called QMIndex (the text index name is not case sensitive):

http://api.havenondemand.com/1/api/sync/createtextindex/v1?index=QMIndex&flavor=querymanipulation

Create a Query Profile

You create a query profile by using the Create Query Profile API.

This API has two required parameters:

  • query_profile. The name of the query profile.
  • query_manipulation_index. The query manipulation flavor text index that you have created to store your rules.

You can optionally also set description to a brief description for the query profile, to help you identify it.

The rest of the parameters for this API are optional, but you must set the appropriate parameters to enable the rules that you want to run. For example:

http://api.havenondemand.com/1/api/sync/createqueryprofile/v1?query_profile=technical&query_manipulation_index=QMIndex&synonyms_enabled=true&synonym_categories=technical

This call creates a query profile named technical, which enables synonym rules and uses the synonym category technical. If you send a query with this query profile, Haven OnDemand applies any synonym rules that you store in the QMIndex that have the category technical (see Match Query Manipulation Rules).

When you successfully create the query profile, the API returns a response of the following form:

{
   "message": "query profile created",
   "query_profile": "technical"
}

See Query Manipulation and Create Query Profile for more information about the parameters to set in your query profile to enable particular types of query manipulations.

Create Query Manipulation Rules

The query manipulation rules are JSON objects, which you add to your query manipulation text index by using the Add to Text Index API.

The rule JSON must have the same format as any index document that you add by using Add to Text Index. For example, the rules must be in a document array, and the reference and content fields can occur only once in each rule.

In addition to the usual restrictions, there are additional requirements for rule objects, which allow Haven OnDemand to find and apply the rules. For example, each rule must have a ruletype field.

There are other required fields, depending on the type of rule. For example, for a synonym rule, you must set synonym_add and synonym_remove to define the synonym terms to add and remove from the query text.

{
   "document": [
      {
         "reference": "synonym_1",
         "ruletype": "SYNONYM",
         "content": "SDK API",
         "title": "synonym_1_title",
         "booleanrestriction": "SDK OR API",
         "synonym_remove": ["SDK"],
         "synonym_add": ["Software Development Kit", "SDK"],
         "category": ["technical"]
      }
   ]
}

For a complete list of fields and examples for all types of query manipulation rules, see Query Manipulation.

Match a Query Manipulation Rule

When you run a query with a query profile, Haven OnDemand sends the query text to the query manipulation text index. Therefore, for a query to activate a query manipulation rule, the rule document must match the query.

The first stage of this search is the same as for any Haven OnDemand query. The query text must match the content field in the rule. Haven OnDemand uses the same linguistic processing as for a normal query (such as stemming and stop word removal). For more information, see Use HavenOnDemand Search Functionality and Text Tokenization and Processing in Text Indexes.

After this first matching step, Haven OnDemand checks against additional restrictions:

  • The booleanrestriction field in the rule document must match the query text. This field contains a Boolean or proximity expression, which you can optionally add to your rules to restrict matches.
  • The category field in the rule document must contain at least one category that matches a category in the query profile, for the appropriate rule type. For example, if the rule document is a synonym rule, and the category field has the value technical, the synonym_categories configuration for the query profile must contain the value technical for the rule to match. If the query profile does not have any categories defined for the rule type, Haven OnDemand matches all categories.
  • s

Note: The content, booleanrestriction, and category fields are available for all rule types. The fields are optional (you do not have to set all of them). However, you must set at least one of them for Haven OnDemand to match the rule. If you do not set content, you must add the always_match field. See Always_Match Fields.

Example

The following JSON fragment is part of a SYNONYM type rule:

   "ruletype": "SYNONYM",
   "content": "server",
   "booleanrestriction": "server AND error"
   "category": ["technical"]

Then take the following query (using the technical query profile created earlier):

http://api.havenondemand.com/1/api/sync/querytextindex/v1?text=server&query_profile=technical

The term server in the query matches the content field in the rule, so the rule matches in this initial step.

The technical query profile has the technical synonym category, so the rule also satisfies this criteria.

However, the second matching step checks the query text against the booleanrestriction field, which requires that the text has both of the terms server and error. This query contains only one of the terms, so it does not match the rule.

Populate the Content Field

To ensure that a rule matches, you must ensure that the content field contains a value that matches every query that might match your booleanrestriction.

The simplest way to do this is to add every term that appears in the booleanrestriction to the content field. For example:

"content": "server error",
"booleanrestriction": "server OR error"

In some cases, it is not possible to create a content field that matches every query that you want, particularly if your Boolean expression contains a Wildcard value. For example, if your booleanrestriction contains the phrase comput* OR serv*, you would need to add every possible expansion for comput* and serv* to the content field.

In this case, you can use the always_match field instead of the content field.

Always_Match Fields

As the name suggests, if you add an always_match field to a rule with a non-zero value, all queries match the rule at the first matching step (bypassing the content match). The API then assesses whether these rules match according to the other criteria, including categories and Boolean restrictions.

For example:

   "ruletype": "SYNONYM",
   "always_match": "true",
   "booleanrestriction": "comput* OR serv*"
   "category": ["technical"]

Now when you send a query for anything, the rule matches at the text matching stage, and the API checks the booleanrestriction for this rule.

Note: For performance reasons, Haven OnDemand recommends that you use always_match fields only when you have to, especially if you have a large number of rules in your query manipulation text index.

Add Query Manipulation Rules to the Index

After you create your query manipulation rules, you add them to your query manipulation text index by using the Add to Text Index API, in the same way that you add index documents to an ordinary text index.

If you create your rules manually, as described in the previous sections, you can use the JSON input type. You must also set the index parameter to the name of your query manipulation flavor text index.

http://api.havenondemand.com/1/api/async/addtotextindex/v1?index=QMIndex&json={"documents": [{"reference": "synonym_1","ruletype": "SYNONYM","content": "SDK API","title": "synonym_1_title","booleanrestriction": ["SDK OR API"],"synonym_remove": ["SDK"],"synonym_add": ["Software Development Kit", "SDK"],"category": ["technical"]}]}

This example adds a synonym rule document to the QMIndex index. A successful add operation returns a response of the following type:

{
   "index": "qmindex",
   "references": [
      {
         "reference": "synonym_1",
         "id": 3
      }
   ]
}

For more information about Add to Text Index, see Introduction to Haven OnDemand Unstructured Text Indexing and the API documentation.

Modify and Remove Rules

You can modify rules by adding the updated rule to the text index with the Add to Text Index API.

The updated version of the rule must have the same value in the reference field, so that Haven OnDemand can match the existing rule. The default value of the duplicate_mode parameter for Add to Text Index automatically detects when a modified document has the same reference, and replaces the older version with the new one.

For example, to update the rule added in the previous section:

http://api.havenondemand.com/1/api/async/addtotextindex/v1?index=QMIndex&json={"documents": [{"reference": "synonym_1","ruletype": "SYNONYM","content": "SDK software development kit","title": "synonym_1_title","booleanrestriction": ["SDK OR (software AND development AND kit)"],"synonym_remove": ["SDK"],"synonym_add": ["Software Development Kit", "SDK"],"category": ["technical"]}]}

This API call changes the promotion reference for the rule added in the previous section.

If you want to remove a rule completely, you can use the Delete from Text Index API. For example:

http://api.havenondemand.com/1/api/async/deletefromtextindex/v1?index=QMIndex&index_reference=synonym_1

This removes the synonym_1 rule from the text index. A successful delete operation returns a response of the following type:

{
   "index": "qmindex",
   "documents_deleted": 1
}

For more information, see Advanced Haven OnDemand Unstructured Text Indexing and the API documentation.

Apply Rules to Queries

After you have created your query profiles and indexed your rules, you can apply them to your queries by adding the query_profile parameter. Set query_profile to the name of the query profile that you want to apply to a particular query.

This parameter is available for the following APIs:

  • Query Text Index
  • Find Similar
  • Get Parametric Values

When you add the query_profile parameter, Haven OnDemand sends the query to the query manipulation text index to search for rules that match the query, and which are enabled by the query profile.

For example:

http://api.havenondemand.com/1/api/sync/querytextindex/v1?text=server error&query_profile=technical&indexes=wiki_eng

The technical query profile created in this how to only enables synonym rules. If the terms in this query match a synonym rule, Haven OnDemand updates the query text with the synonyms, and forwards the query to the wiki_eng text index. It returns any query results that match the modified text.

Promotions

The examples in the previous sections all use simple synonym rules. When you use a query profile that has synonyms enabled, Haven OnDemand always applies any matching synonym rules and modifies the queries as appropriate. Similarly, matching blacklist rules are always applied (when enabled).

The behavior for promotion type rules is different, and depends on the API that you use.

  • Query Text Index has an additional promotion parameter, which determines whether it returns promotion documents. When you set promotion to false (the default), Query Text Index returns any matching CARDINAL_PLACEMENT documents as part of the query results. When you set promotion to true, Query Text Index only returns any DYNAMIC_PROMOTIONS, STATIC_REFERENCE_PROMOTIONS, and STATIC_CONTENT_PROMOTIONS that match the query (after synonym and blacklist modifications). In this case, Query Text Index returns only the promotion documents, and you must retrieve the normal results in a separate query.
  • Find Similar returns any matching CARDINAL_PLACEMENT documents as part of the query results. It does not return any other kind of promotions.
  • Get Parametric Values retrieves the promotion documents for any matching promotions (CARDINAL_PLACEMENT, DYNAMIC_PROMOTIONS, and STATIC_REFERENCE_PROMOTIONS). If the promotion documents contain the requested parametric fields, Get Parametric Values also returns the values that exist in these fields in the promotion documents, along with the values from the ordinary results.

For more information, see Query Manipulation.

Administer Query Profiles

You can return a list of your query profiles by using the List Resources API, with the type parameter set to query_profile. For example:

http://api.havenondemand.com/1/api/sync/listresources/v1?type=query_profile

This API returns a response of the following form:

{
   "public_resources": [],
   "private_resources": [
      {
         "resource": "technical",
         "type": "query_profile",
         "flavor": null,
         "description": null,
         "date_created": "Mon Mar 21 2016 15:35:42 GMT+0000 (UTC)"
      }
   ]
}

You can retrieve the query profile configuration by using the Retrieve Query Profile API.

https://api.havenondemand.com/1/api/sync/retrievequeryprofile/v1?query_profile=technical

This API returns a response of the following form:

{
   "query_profile": "technical",
   "description": null,
   "query_manipulation_index": "QMIndex",
   "promotions_enabled": false,
   "promotion_categories": [],
   "promotions_identified": true,
   "synonyms_enabled": true,
   "synonym_categories": [
      "technical"
   ],
   "blacklists_enabled": false,
   "blacklist_categories": []
}

You can also modify your query profiles at any time by using the Update Query Profile API.

Note: You must provide the whole query profile in the Update Query Profile API (that is, you cannot perform a partial update). If you omit a parameter, the query profile uses the default value. If you omit a rule category parameter, the query profile uses the existing category list, but if you want to add a category, you must also provide the existing categories.

https://api.havenondemand.com/1/api/sync/updatequeryprofile/v1?query_profile=Technical&query_manipulation_index=QMIndex&synonyms_enabled=true&synonym_categories=Technical&synonym_categories=Tech&blacklists_enabled=true&blacklist_categories=Technical&blacklist_categories=Tech

This API returns a response of the following form:

{
   "message": "query profile updated",
   "query_profile": "technical"
}