CX Works

A single portal for curated, field-tested and SAP-verified expertise for your SAP C/4HANA suite. Whether it's a new implementation, adding new features, or getting additional value from an existing deployment, get it here, at CX Works.

Search and Navigation in SAP Commerce Cloud - Troubleshooting Solr

Problems

This section first outlines problems and their potential solutions. Given there are many scenarios with the same solutions, we've covered the steps to execute the solutions in the "How To Identify" section.

Search Inconsistent Behavior

(warning) Problem: Running the same query in the front-end provides different results.

When more than one Solr node is used for querying, a "Round-Robin" algorithm is used for selecting which Solr node responds. 
Different results (including empty results) for the same query indicate that there are differences between Solr cores.

Suggestions:

  • Check for replication issues (see below).
  • Check for connection issues.

Search Produces a 404 Response

(warning) Problem: When performing a search, a 404 error message is displayed instead of the search results. 

There may be many different reasons for getting a 404 page. The best approach is to check the logs for more detailed information. 

Suggestions:

  • Check the SAP Commerce Cloud logs for more detailed information.
  • Check for missing categories.

Empty Search Results for all Types of Search 

(warning) Problem: Search for text and navigation over categories leads to an empty search result page.

Suggestions:

  • Check for connection issues. 
  • Check for replication issues.
  • Check if Solr's core and shards were defined.
  • Check if a category was removed.

Solr Data is Not Up to Date

(warning) Problem: Data in search results doesn't reflect the database state. Attributes are not up to date or new products are not displayed. 

Suggestions:

  • Check indexing.
  • Check for replication issues.

Facet is Not Visible

(warning) Problem: There is a facet defined but it is not visible in the search results.

Suggestions:

  • Check IndexedProperty settings. Check if this facet and visible attributes are set to true.

     

  • Check if Property Value Provider is CommerceClassificationPropertyValueProvider.
  • Run the query in the Solr admin console's query analysis. When the facet value that is not selected covers 100% results (all documents have this value for this facet for a given query), this facet value is then removed from the results. This occurs in the ConfigurableFacetSearchResultFacetsPopulator.
  • Check in the Commerce Search Backoffice for custom boost rules.

Query Doesn't Provide Expected Results

(warning) Problem: Solr provides results but different than what is expected.

Suggestions:

  • Check the query in the logs and run it through the Solr admin console's query analysis.

Indexing Issues

(warning) Problem: Indexing doesn't work properly.

Suggestions:

  • Verify the indexing Cronjobs.

Unexpected Order of Products in the Search Result

(warning) Problem: Solr provides results but elements that were expected to be the best matches are not at top of the list.

Suggestions:

  • Check the Commerce Search Backoffice for custom boost rules.
  • Check the documents score.

Solr Backoffice: Missing Fields in the Search Result  

(warning) Problem: Solr's Backoffice provides results but some fields are not returned even though they are declared as indexed fields.

Suggestions:

From the official documentation: "Since Backoffice needs only PK to display search results, it's crucial from the performance point of view not to include the indexed fields in the response. The SolrSearchConfig has to have restrictFieldsInResponse setting set to true".

When you go to the Solr admin console (for example, http://localhost:8983/solr/) and run a query, you won't see some fields, even if they are declared as indexed fields.

In the schema.xml file, there is a comment regarding this behavior:

Backoffice requires the following field in response:
pk, catalogVersion, code, autosuggest and spellcheck.
Therefore these fields have stored set to true and other fields are set to false

The fields mentioned above are declared as stored="true" while the others one are declared as "false".


Schema.xml
<field name="id" type="string" indexed="true" stored="false" multiValued="false" />
<field name="pk" type="long" indexed="true" stored="true" multiValued="false" />
<field name="_typeCode" type="string" indexed="true" stored="false" multiValued="false" />
<field name="catalogId" type="string" indexed="true" stored="false" multiValued="false" />
<field name="catalogVersion" type="string" indexed="true" stored="true" multiValued="false" />
<field name="text" type="textgen" indexed="true" stored="false" />
<field name="_version_" type="long" indexed="true" stored="true" multiValued="false" />
<field name="code_string" type="string" indexed="true" stored="true" multiValued="false" />


However, even if they are not being stored in Solr, it does not mean it cannot be searchable by its indexed field. To make sure the index is being handled by Solr, go to your Solr's admin console and choose "Schema". You can browse through the declared indexed fields and check its content.

To turn a type or specific field to be "Stored" in the Backoffice, open the schema.xml and change stored="false" to "true".

The below example turns all "int" fields to be stored:


<dynamicField name="*_int" type="int" indexed="true" stored="true" />


Restart the Solr server and run the query again.

How to Identify

Connection Issues

To check connection issues from SAP Commerce to Solr search for "No live Solr Servers available to handle this request" 

See example below:


ERROR [hybrisHTTP26] [SolrSearchRequestResponsePopulator] Exception while executing SOLR search de.hybris.platform.solrfacetsearch.search.FacetSearchException: No live SolrServers available to handle this request at de.hybris.platform.solrfacetsearch.search.impl.LegacyFacetSearchStrategy.search(LegacyFacetSearchStrategy.java:205) ~[solrfacetsearchserver.jar:?]   ...


Possible reasons:

  • Wrong Solr address (how to check the address defined in SAP Commerce)

  • The Solr node is down.

  • The SAP Commerce instance can't access Solr's node due to network restrictions.

Missing Core in Solr

In case SAP Commerce Cloud can connect to Solr but there is no proper core defined, look for the following message in the logs:


ERROR [hybrisHTTP5] [SolrSearchRequestResponsePopulator] Exception while executing SOLR search de.hybris.platform.solrfacetsearch.search.FacetSearchException: Error from server at http://localhost:8983/solr: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/master_electronics_Product_flip/select. Reason: <pre> Not Found</pre></p> </body> </html>   ...


Since the query gets to a slave node, it indicates that there are no connection issues. Replication should not be a problem, because to replicate, a slave has to have a core created first which occurs during full indexing.

Suggestions:

  • Check indexing. 
  • Run full indexing. Find the relevant full indexing Cronjob and execute it. 

Missing Category

One of the reasons for getting a 404 page may be that a category that is present in Solr as a facet for a product was removed from the system (or the code of this category has changed). This will cause issues because the system will try to find a localized name value for the category. If it can't be found, a 404 error message will be displayed for the search results.

This issue results in the following error in the logs:


ERROR [hybrisHTTP37] [solrIndexThreadLogger] Category with code '578' not found! (Active session catalogversions: electronicsProductCatalog.Online)

Replication Issues

It is important to properly configure the Solr master-slave replication.


The replication setup and current replication state can be inspected in the Solr admin page.

  1. Go to the Solr admin console (for example, https://localhost:8983/solr/).

  2. Select the relevant core from the drop-down on the left navigation (how to find proper core). For example, master_electronics_Product_flip

  3. Select Replication from the left navigation.

Solr replication screen (master node):



Checks for master node:

  1. replication enable should be set to true (green icon). This may be automatically set to false during the full indexing when using the DIRECT mode (see Indexing Process).

  2. Default settings for replicateAfter should be used: commit, startup 


Solr replication screen (slave node)



Check for slaves nodes:

  1. Green text should be visible after the text "Iterations" and provide information when last replication was performed. If data in the master core didn't change, replication will not be performed and data will not be updated.

  2. The master and slave Version and Gen (generation) should be the same. Depending on the network and the amount of data, replication may take longer. Having different values is not an issue.

  3. On the slave screen, the Next Run box should be displayed and shows when the next replication will occur.


Available documentation:

How To

Check Which Facet Search Configuration is Used

Log in as admin in the Backoffice.

  1. In left navigation, open Website > WCMS.

  2. Search for the relevant site and open it.

  3. In first tab, search for attribute Solr Search Configuration and double click to open.

There are other strategies how Facet Search Config can be mapped with context. If other than the default (reference on Website level) is used, go directly to the Facet Search Config from left navigation instead of points 1-3:

  • In the left navigation, open System > Facet Search > Facet Search Config.

Get Addresses for Solr Nodes from the Backoffice Configuration

In the Backoffice, find Facet Search Config. Then: 

  1. Search for the attribute Solr server configuration and double click to open.

  2. Check if the Endpoint URLs are correct. Double click on the elements to check these URLs. Verify these are assigned to master or slave nodes. 


Check the Currently used Solr Core

In the Backoffice, find Facet Search Config. Find the property named "Index name prefix". It is used when the Solr core is created. For example, if Index name prefix is set to "electronics" and products are indexed, the Solr core name will be:
master_electronics_Product.

When using the TWO_PHASE indexing approach (see Indexing Process ), not one but two cores will be created for each Facet Search Config. One is used for queries while the full indexing is running on the other. To distinguish these indexes, a suffix is added to the name. 

Core names:
SAP Commerce Cloud version 6.2 and later: 

  • master_electronics_Product_flip
  • master_electronics_Product_flop

SAP Commerce Cloud earlier than 6.2:

  • master_electronics_Product
  • master_electronics_Product_1

To verify which of these is currently used for queries check:

  • SAP Commerce Cloud version 6.2 and later: Search for SolrIndex that is connected with Facet Search Config. There should be two instances with qualifiers set as "flip" and "flop". The index that has the flag Active set to true indicates that the core is currently used for queries.
  • Before 6.2: Search for SolrIndexedCoresRecord attribute currentIndexDataSubDirectory. In older versions of SAP Commerce, this information wasn't stored in the database (there was no item defined). Due to this, a restart of Solr or SAP Commerce Cloud had a 50% chance that the wrong index was used.

Verify the Indexing Cronjobs

In the Backoffice, find  Facet Search Config. Go to CRON JOBS tab. Check the following cronjobs:

  • full-[ Facet Search Config. name ]-cronJob. For example, full-electronicsIndex-cronJob

  • update-[ Facet Search Config. name ]-cronJob. For example, update-electronicsIndex-cronJob

Check if:

  1. The last result has status SUCCESS. 
  2. The cronjob is scheduled correctly in the TIME SCHEDULE tab.
  3. Check the RUN AS tab and verify the User attribute is set to a user that has access to products.

Go back to Facet Search Config. Go to INDEXED TYPES and open the object that can be found there. Get the queries defined in the Indexer queries attribute. Use the HAC console to verify if these queries return products that should be indexed. Please note that in the HAC, you can provide the user that was defined in the relevant cronjob, so that the query is executed with the same restrictions.

Make sure that the dynamic parameters in the query are replaced before executing in the HAC. For example, ?lastIndexTime needs to be changed to something that the database will understand.



If no customization was done to the indexing process and there are no issues with indexing, the number of results from this query should be the same as Num Doc presented in the Solr console for core overview.

Extract Query from SAP Commerce Cloud

In cases when the results that are provided in the front-end are different than expected, it is a good idea to extract the query that is sent from SAP Commerce Cloud to Solr and manually run it on Solr where you can add troubleshooting modifications or checks. This article describes how to get the query from SAP Commerce Cloud.

Use the SAP Commerce Cloud Query in Solr

The Solr admin page permits performing queries directly as described in Understanding Solr Queries. A few tricks that may be useful when dealing with queries in Solr:

  • Change the parameter wt=javabin to wt=xml to get results in a more friendly format.
  • Change the rows parameter to 0 if you are interested in facets only.
  • Change the facet value to false if you are not interested in facets.
  • Change the facet.mincount to 0 to display facet values with 0 matching products for current search conditions. 
  • Change the facet.limit value to get more facets from Solr (by default it set to 50). 
  • Use the fl parameter to define which fields for documents should be returned. For example when dealing with sorting issues for searching by text, it may be good idea to use:
    fl=pk,id,ean,code_string,name_text_en,manufacturerName_text,categoryName_text_en_mv,score


    Field Name Description

    pk,
    id

    These fields will help to identify products in results.
    ean,
    code_string,
    name_text_en,
    manufacturerName_text,
    categoryName_text_en_mv
    List of fields that are used for text searching.
    Please note language specific suffix ( _en in this example).
    For checking other languages, change suffix to isocode defined for this Langage in SAP Commerce Cloud
    The configuration that is responsible for defining fields that are used for text search are described in Understanding Solr Queries.  
    score Returns the calculated relevance score for each document.


Check the Document Score

The Solr admin console can display a detailed explanation for the calculated document score. This may be useful when relevance order of documents returned from Solr is different than expected. To get additional score details, add the additional parameter queryDebug=true when using the query tool in the Solr admin console. More details can be found in Understanding Solr Queries.  

Check ID Details in Solr Core Document

It is possible to add a special request handler in Solr that will allow access to detailed information about the core and documents. Please note that this is an advanced option that usually doesn't need to be used in projects. More details can be found in https://wiki.apache.org/solr/LukeRequestHandler

Indexing Time is Too Long

To speed up indexing time:

  • Verify the query that is used to extract products (especially the one that is used for update).
  • Check if all languages are used.
  • Check if all IndexedProperties are used.
  • Consider removing the staged version from index.
  • Consider using partial update instead of update or full indexing approach as discussed in our article about indexing.

Other Resources

Solr is well recognized and is a widely used solution. For issues that are not specifically SAP Commerce Cloud related, refer to Apache and other community resources on the internet, for example,https://wiki.apache.org/solr/SolrRelevancyFAQ.

Conclusion

Solr is a powerful tool to help your customers find the products they are looking for. If you're looking for more details around how you can better leverage the Solr capabilities in SAP Commerce Cloud, feel free to look at the rest of the articles in our search and navigation series.