How to define Dimension & Property in Endeca for forgeless approach? - Oracle commerce 11.x

There are two ways to define dimension and properties for Forge-less approach:

1) Define at Product-sku-output-config(PSOC) level
Ex:
      DIMENSION:  <property name="pattern" type="string" output-name="Pattern" multiselect-type="multi-or"  is-dimension="true"  />

      PROPERTY:   <property name="description" type="string" output-name="P_Description" text-searchable="true"/>

2) Define at Index config Level
File location: /opt/app/endeca/apps/APPNAME/config/index_config/index-config.json


a) Add dimension and property as below mentioned in example and you can define searchable, filter, roll-up, dimension, multi select etc...
Ex:
{
  "indexConfig" : {
    "attributes" : {
      "jcr:primaryType" : "endeca:unstructured",
      "record.id" : {
        "propertyDataType" : "ALPHA",
        "jcr:primaryType" : "endeca:property",
        "isRecordFilterable" : true
                },
                "sku.margin" : {
          "propertyDataType" : "DOUBLE",
          "jcr:primaryType" : "endeca:property"
        },
                "product.repositoryId" : {
          "propertyDataType" : "ALPHA",
          "mergeAction" : "UPDATE",
          "isRollupKey" : true,
          "jcr:primaryType" : "endeca:property",
          "isRecordFilterable" : true
        },
                "product.price_range" : {
                  "rangeComparisonType" : "FLOAT",
                  "multiSelectType" : "OR",
                  "sourcePropertyNames" : ["sku.sortPrice"],
                  "jcr:primaryType" : "endeca:dimension"
                }
        },
    "precedenceRules" : {
    },
    "searchIndexConfig" : {
      "isWildcardEnabledInDimensionSearch" : true,
      "spellingDictMaxWordLength" : 16,
      "spellingDictMinNumWordOccurrences" : 4,
      "spellingDictMinWordLength" : 3
    }
  }
}

b) Once you update index-config.json make sure to set configurations at Endeca repository level.

    Here is the command:
   /opt/app/endeca/apps/APPNAME/control :>

                          ./index_config_cmd.sh set-config -f ../config/index_config/index-config.json -o all

c) Run Endeca indexing to see your dimension(s) and propertie(s) added

Endeca InvokeAssembler:Assembler Exception:---java.lang.NullPointerException and also Jboss log file fills in GB with StackOverflowError message

Problem Statement:

Endeca InvokeAssembler:Assembler Exception:---java.lang.NullPointerException is thrown and also Jboss Log is filling in GB in few minutes to Hour with below exception:

2015-07-09 03:46:06,987 ERROR [nucleusNamespace.atg.endeca.InvokeAssembler] (ajp-0.0.0.0-10109-20) InvokeAssembler.service():Assembler Excepti on:---java.lang.NullPointerException 2015-07-09 2015-07-09 03:46:07,664 FATAL [org.apache.jasper.runtime.JspFactoryImpl] (ajp-0.0.0.0-10109-24) Exception initializing page context java.lang.StackOverflowError

       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
       at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
       at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
       at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
       at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
       at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)
       at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216)
       at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544)

Solution:
If there is no default page defined you will get above exceptions.

Assembler checks for trigger condition(s), priority, segmentation, time-boxing etc..  Make sure default page does not have expired time box or segmentation.

For example : Home page will be created for each day with time boxed. Merchandiser will create home page versions for 1 week with time boxing. If there is no valid page with expired time-box then assembler never finds default Home page and throws errors in JBOSS log.

How to check time boxing on page:

Go to Endeca XM page and check for below highlighted spot.






How to Remove time box? 

Uncheck the time and make one of the page default.


FYI
XM Promote page script run is required to reflect any changes done on site.

How to schedule Endeca index using ATG ProductCatalogSimpleIndexingAdmin component? - Endeca Indexing Job

Component path: /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin 
# enableScheduledIndexing: defaults to false. Must be true for the above schedules to have any effect 
enableScheduledIndexing=true 
# baselineSchedule - Run a baseline every 2 Hrs 
baselineSchedule=calendar * * * * 0,2,4,6,8,10,12,14,16,18,20,22 00

OOTB - Endeca Record Filter Builder Components

The ATG-Endeca integration includes several record filter builder components. Here are few which are most commonly used:

 1) A filter that restricts the set of records returned to only those associated with specified sites. 

It uses product.siteId attribute.

 /atg/endeca/assembler/cartridge/manager/filter/SiteFilterBuilder 

 2) A filter that restricts the set of records returned to only those in the current language. 

It uses product.language attribute.

 /atg/endeca/assembler/cartridge/manager/filter/LanguageFilterBuilder

3) A filter that restricts the set of records returned to only those associated with the appropriate catalogs.

It uses product.catalogId attribute.

 /atg/endeca/assembler/cartridge/manager/filter/CatalogFilterBuilder

 To enable a specific record filter builder component, you add it to the recordFilterBuilders property of the  

/atg/endeca/assembler/cartridge/manager/NavigationStateBuilder component. 

recordFilterBuilders+=\
/atg/endeca/assembler/cartridge/manager/filter/CatalogFilterBuilder,

/atg/endeca/assembler/cartridge/manager/filter/SiteFilterBuilder 

Endeca Guided Search and Endeca deployment template ports - Endeca Ports

Service Ports:



Ports used during Endeca application creation using deployment template:

FYI:

Default ports can be replaced with any other port numbers but Port numbers can be no larger than 32767.

How to Export and Import Endeca Workbench content? - Oracle commerce 11.x version

EXPORT Commands:

1) Export all the content in public format
 runcommand[.bat/.sh] IFCR exportApplication export_folder_name

2) Export media, preview and editor configurations

runcommand.[.bat/.sh] IFCR legacyExportContent"/" export_folder_name

3)Export CAS configurations

index_config_cmd[.bat/.sh] get-config [-o OwnerName] -f FileName

4) Export all Users and tool permissions
/opt/app/endeca/ToolsAndFrameworks/11.1.0/admin/bin$

export_users.[.bat/.sh] --config config-prop-file --output output-file --admin-user admin-user --admin-password admin-password

 IMPORT Commands:

1) Import all Users and tool permissions
/opt/app/endeca/ToolsAndFrameworks/11.1.0/admin/bin$

import_users.[.bat/.sh] --input input-file --config config-file --default_user_pswd user-password -- single-app permissions --admin-user admin-user --admin-password admin-password

where:

input-file is the location of data file 

config-file is the location of the configuration property

file

2)Import all the content in public format

runcommand.[.bat/.sh] IFCR importApplication export_folder_name

3)Import CAS configurations

index_config_cmd[.bat/.sh] set-config [-o OwnerName] -f FileName

4) Import media, preview and editor configurations

 runcommand.[.bat/.sh] IFCR legacyUpdateContent "/" export_folder_name

How to test Endeca response content returned on the browser? - Endeca

There are two formats to render Endeca response content and that is XML or JSON.

Here is the URL structure:

For XML response content use parameter format=xml and for JSON content use format=json

http://HOST:PORT/browse/sweaters/women/_/N-27bh?format=xml

http://HOST:PORT/browse/sweaters/women/_/N-27bh?format=json

Endeca CAS | Multiple Record store merge - Oracle Commerce 11.1

Multiple record store could be used to merge data. This merging would be work as switch join.  Find out the steps below to use multiple record store :-

 1. Open <Endeca App Path>\APPNAME\config\cas\last-mile-crawl.xml file and add all record stores. Find out sample below. Multiple record store is highlighted below. We can add any number of record stores to merge.

                                <sourceConfig>

                                                <moduleId>

                                                                <id>com.endeca.cas.source.RecordStoreMerger</id>

                                                </moduleId>

                                                <moduleProperties>

                                                                <moduleProperty>

                                                                                <key>dataRecordStores</key>

                                                                                <value>APPNAME-data</value>

                                                                                <value>APPNAME-web-crawl</value>

                                                                </moduleProperty>

                                                                <moduleProperty>

                                                                                <key>dimensionValueRecordStores</key>

                                                                                <value>APPNAME-dimvals</value>

                                                                </moduleProperty>

                                                </moduleProperties>

                                                <excludeFilters />

                                                <includeFilters />

                                </sourceConfig>

  2. Run following command to update configuration in CAS

a.       <Installpath>\CAS\11.1.0\bin >cas-cmd.bat updateCrawls -f <Endeca App Path>\APPNAME\config\cas\last-mile-crawl.xml

  3.  Run Baseline.

Note :- Endeca CAS uses record.id as unique identifier. We can define our own. Suppose two record appears in record store A and B with same record id . CAS would discard one of the record.

Endeca Select feature aka set Selection - Controlling Endeca Record Values

The Select feature allows you to select specific keys (Endeca properties and/or dimensions) from the data so that only a subset of values will be transferred for Endeca records in a query result set.

This functionality prevents the transferring of unneeded properties and dimension values when they will not be used by the front-end Web application.

It therefore makes the application more efficient because the unneeded data does not take up network bandwidth and memory on the application server.

There are two possible scenarios:
1)  If you are making independent Endeca queries then you can use below API:

// Create a query

ENEQuery usq = new UrlENEQuery(request.getQueryString(),"UTF-8");

// Create an empty selection list

FieldList fieldList = new FieldList();

// Add an Endeca property to the list

fieldList .addField("P_DisplayName");

// Add an Endeca dimension to the list

fieldList .addField("P_RepositroyId");

// Add the selection list to the query

usq.setSelection(fieldList );

// Make the MDEX Engine query

ENEQueryResults qr = nec.query(usq);

2)If you are using ATG-Endeca integration then use below component and add only required fields.

 /atg/endeca/assembler/cartridge/handler/config/ResultsListConfig.properties

 fieldNames=\
  P_DisplayName,\
  P_RepositroyId

Endeca Derived properties to get Min and Max price for Sale/Clearance/List/MSRP prices - Endeca Features

A derived property is a property that is calculated by applying a function to properties or Dimension values from each member record of an aggregated record. Derived properties are created by Forge, based on the configuration settings in the Derived_props.xml file. After a derived property is created, the resultant derived property is assigned to the aggregated record.

Problem Statement:
Consider one product with 5 SKU's and each SKU has sale,List,clearance and MSRP price tagged and requirement is to traverse through all the prices tagged on SKU's and get the MIN and MAX for sale, Clearance, List  and MSRP.

Solution:
1) Change Derived_props.xml file in pipeline
/opt/app/endeca/apps/CRS/config/pipeline/Derived_props.xml
<?xml version="1.0" encoding="UTF-8" standalone="no" ?> <!DOCTYPE DERIVED_PROPS SYSTEM "derived_props.dtd"> <DERIVED_PROPS>
  <DERIVED_PROP DERIVE_FROM="sku.listPrice" FCN="MAX" NAME="P_SKU_ListPrice_Max"/>
  <DERIVED_PROP DERIVE_FROM="sku.listPrice" FCN="MIN" NAME="P_SKU_ListPrice_Min"/>
  <DERIVED_PROP DERIVE_FROM="sku.salePrice" FCN="MAX" NAME="P_SKU_SalePrice_Max"/>
  <DERIVED_PROP DERIVE_FROM="sku.salePrice" FCN="MIN" NAME="P_SKU_SalePrice_Min"/>
  <DERIVED_PROP DERIVE_FROM="sku.MSRP" FCN="MAX" NAME="P_SKU_MSRP_Max"/>
  <DERIVED_PROP DERIVE_FROM="sku.MSRP" FCN="MIN" NAME="P_SKU_MSRP_Min"/>
  <DERIVED_PROP DERIVE_FROM="sku.clearencePrice" FCN="MAX" NAME="P_SKU_ClearencePrice_Max"/>
  <DERIVED_PROP DERIVE_FROM="sku.clearencePrice" FCN="MIN" NAME="P_SKU_ClearencePrice_Min"/>
</DERIVED_PROPS>
2) Run baseline index
3) Check reference application for Derived properties
Endeca reference Application:

8056678  (5 Records)
DERIVED PROPERTIES:
P_SKU_ListPrice_Max:	39.990000
P_SKU_ListPrice_Min:	38.990000
P_SKU_SalePrice_Max:	39.990000
P_SKU_SalePrice_Min:	38.990000
P_SKU_MSRP_Max:	55.000000
P_SKU_MSRP_Min:	54.000000
P_SKU_ClearencePrice_Max:
P_SKU_ClearencePrice_Min:
REPRESENTATIVE REC PROPERTIES:
allAncestors.repositoryId:	cat130021
allAncestors.repositoryId:	cat130052
allAncestors.repositoryId:	cat130155
allAncestors.repositoryId:	rootCategory

FYI - Derived properties can be only created on Aggregated records.
Valid functions are MIN, MAX, AVG, or SUM

Endeca concurrent write exception - atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException???

Problem:You will see below exception in Jboss log
ERROR [ProductCatalogSimpleIndexingAdmin]
atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40
at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:225)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForKey(AbstractRecordStoreAggregateSession.java:300)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForContext(AbstractRecordStoreAggregateSession.java:249)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForCurrentContext(AbstractRecordStoreAggregateSession.java:238)
at atg.endeca.index.AbstractRecordStoreAggregateSession.submitRecord(AbstractRecordStoreAggregateSession.java:334)
at atg.endeca.index.LocaledExporter.exportRecords(LocaledExporter.java:295)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:258)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:208)
at atg.endeca.index.LocaledExporter.performBaselineUpdate(LocaledExporter.java:345)
at atg.endeca.index.admin.IndexingTask.doTask(IndexingTask.java:401)
at atg.endeca.index.admin.IndexingTask.performTask(IndexingTask.java:359)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasks(IndexingPhase.java:265)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasksSerially(IndexingPhase.java:242)
at atg.endeca.index.admin.IndexingJob.performJobSerially(IndexingJob.java:264)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:657)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:636)
at atg.endeca.index.admin.SimpleIndexingAdmin$2.run(SimpleIndexingAdmin.java:1008)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:141) at $Proxy154.startTransaction(Unknown Source) at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:217)

Solution:
If you stop Endeca indexing from ATG(/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/) before it completes then it will acquire lock and then subsequent baseline updates will fail with above exception.

Solution#1: Restarting Endeca CAS which will remove the lock acquired - Temporary fix

Solution #2:  Make sure you restart CAS after every baseline update to avoid this issue plus some other issues related to CAS

Add /control/initialize_RS_services.sh script at the end of  /control/baseline_update.sh script - It will delete all the records stores and create new one each time and also helps to improve some level of performance on CAS side

Solution #3: Set resetActive flag to true on following components to rollback any previously active transactions
/dyn/admin/nucleus/atg/endeca/index/DataDocumentSubmitter/?propertyName=resetActive

/dyn/admin/nucleus/atg/endeca/index/DimensionDocumentSubmitter/?propertyName=resetActive
/dyn/admin/nucleus/atg/endeca/index/SchemaDocumentSubmitter /?propertyName=resetActive

How to check endeca process(MDEX, Platformservices, MDEX, CAS etc..) running on the box? - Endeca Process

Run below command:
ps -ef | grep endeca
You should see all below process running else need to restart manually.
 1) MDEX(Dgraphs)  Ex: /opt/app/endeca/MDEX/6.5.1/bin/dgraph
2) Platform Services Ex: /opt/app/endeca/PlatformServices/11.1.0/j2sdk/bin/java
3) Tools and Framework Ex: /opt/app/endeca/ToolsAndFrameworks/11.1.0/server/j2sdk/bin/java

How to check CAS is running?
ps -ef | grep cas  
You should see process with /bin/bash ./cas-service.sh  if not then restart CAS.