Endeca CAS : How to create new CAS Crawl and verify it?

New application created and even after provision Endeca Index is failing to find the Crawl. Try below steps:

1) Make sure CAS password is correct in below file
     /opt/app/endeca/apps/CRS/config/cas/last-mile-crawl.xml

2) Create new CAS crawl with updated details
/opt/app/endeca/CAS/11.1.0/bin> cas-cmd.sh createCrawls -f /opt/app/endeca/apps/CRS/config/cas/last-mile-crawl.xml

3)  Verify the crawler by running the below script
/opt/app/endeca/CAS/11.1.0/bin> cas-cmd.sh listCrawls

How to find Endeca component versions Ex: Dgraphs, Deployment template, Forge, CAS and Tools & Framework? - Endeca component versions

MDEX: 

Use below command
C:\Endeca\MDEX\6.5.1\bin>dgraph.exe --version
Endeca dgraph version 6.5.1.829811
Supports opti-XML version : 2.00
Supports binary IR version: 640
Interface specification :
  READS:dgidx=6.3.0a WRITES:dgraph=IR640
ICU version 51.2
OLT version OLT 1.4 140319
OpenSSL 1.0.0k 5 Feb 2013 

Deployment Template:

Use below command
C:\Endeca\apps\Discover\control>runcommand.bat --version
Deployment Template: 11.1.0

Platform Services: 

Use below command
C:\Endeca\PlatformServices\11.1.0\bin>forge.exe --version
INFO    01/19/16 03:14:25.991 UTC (1453173265991)       FORGE   {config}: forge 4.1-Internal-foo@bar-baz-beetle ("i86pc-MINGW32_NT-6.1")
INFO    01/19/16 03:14:25.996 UTC (1453173265995)       FORGE   {config}: Copyright 2001-2012 Oracle Endeca Technologies, Inc.
INFO    01/19/16 03:14:26.000 UTC (1453173265999)       FORGE   {config}: Command Line: forge.exe --version
forge 4.1-Internal-foo@bar-baz-beetle ("i86pc-MINGW32_NT-6.1")
Copyright 2001-2012 Oracle Endeca Technologies, Inc.
INFO    01/19/16 03:14:26.041 UTC (1453173266040)       FORGE   {config}: Initialized cURL, version: libcurl/7.16.4 OpenSSL/0.9.8 zlib/1.2.3
INFO    01/19/16 03:14:26.045 UTC (1453173266044)       FORGE   {config}: Forge completed successfully.

CAS:

Use below jar to find the version: 
C:\Endeca\CAS\11.1.0\lib\cas-cmd>cas-cmd-11.1.0.jar

Tools and Framework: 

Version is displayed on the login screen

How to verify Endeca FORGE data? - Endeca FORGE

Forge Process creates binary data in /APP_NAME/data/forge_ouput directory
For forge-less/CAS-based system data is in /APP_NAME/data/cas_ouput.

Step 1: Copy binary file Ex: Discover-sgmt0.records.binary to the below location
    Ex: C:\Endeca\PlatformServices\11.1.0\bin>

Step 2: Run below command
C:\Endeca\PlatformServices\11.1.0\bin>binary2xml.exe C:\Endeca\apps\Discover\data\forge_output\Discover-sgmt0.records.binary  C:\Endeca\out.xml  

Step 3: XML file created in the path mentioned
Ex: C:\Endeca\out.xml  

Endeca Forge recordstore adapter Fails with Exception - com.endeca.edf.adapter.AdapterException: java.lang.RuntimeException: java.io.IOException

Problem Statement:
Endeca Forge is failing with the following exception. 

Forge Log:
ERROR FORGE {baseline}: (AdapterRunner): java.lang.RuntimeException: java.io.IOException: http://localhost:8500/CRS_en_data/?wsdl returned response code 404 com.endeca.edf.adapter.AdapterException: java.lang.RuntimeException: java.io.IOException

CAS Log:

ERROR [] [main] com.endeca.itl.bootstrap.CasBootStrapServlet: Failed to initialize CAS: java.sql.SQLException: Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@6c679e04[file =/opt/app/endeca/CAS/workspace/state/cas/db/db.lck, exists=true, locked=false, valid=false, ]

Solution:

Above Exception means forge adapter is not able to communicate with CAS record store instance.  There is lock acquired on hsqldb.

Best way to release lock is kill CAS process and restart it.

Endeca Content Acquisition System (CAS) - CMS Connectors

CMS Connectors Provide a means to access and crawl data sources in a wide variety of CMS types. Each CMS data source has a corresponding connector. Endeca Content acquisition System supports nine licensed CMS connectors. 

1) Documentum eRoom
2) Documentum Content Server
3) Lotus Notes (Domino)
4) FileNet P8
5) FileNet Document and Image services
6) JSR -170 compliant repositories
7) Microsoft SharePoint (2003 & 2007)
8) OpenText LiveLink
9) Interwoven TeamSite

How to Migrate Dimension Value IDs between environments - (Forgeless) - Endeca commerce 11.1

Move Dimension Value IDs from UAT to production:

1. Run the export command on the UAT environment:
/opt/app/endeca/CAS/11.1.0/bin>cas-cmd.sh exportDimensionValueIdMappings -m <APP_NAME>-dimension-value-id-manager -f UAT_DIMVALS.txt 

2. Move the file UAT_DIMVALS.txt, to the PRODUCTION environment

3. Run the import command on the PRODUCTION environment: 
/opt/app/endeca/CAS/11.1.0/bin>cas-cmd.sh importDimensionValueIdMappings –m <APP_NAME>-dimension-value-id-manager –f  UAT_DIMVALS.txt

     
      4. Move dvalid_mappings_archive from UAT to PRODUCTION 
       location:  <APP_NAME>/data/dvalid_mappings_archive

How to list all record store instances created in CAS? - Endeca CAS

1) Go to CAS bin directory
Example: /opt/app/endeca/CAS/3.1.x/bin

2) Run below command
/opt/app/endeca/CAS/3.1.1/bin]
$ ./component-manager-cmd.sh list-components
 
NAME                         TYPE                  STATUS
CRS_en_dimvals        RecordStore        RUNNING
CRS_en_data              RecordStore        RUNNING
CRS_en_prules           RecordStore        RUNNING
CRS_en_schema         RecordStore        RUNNING

How to create/delete record store instances manually? [ERROR Message - CRS-en-dimvals/?wsdl returned response code 404] - Endeca CAS

Useful when you get below error message due to absence of record store instances in CAS:
 [ERROR Message - CRS-en-dimvals/?wsdl returned response code 404]
 [ERROR Message - CRS-en-data/?wsdl returned response code 404]
 [ERROR Message - CRS-en-schema/?wsdl returned response code 404]
 [ERROR Message - CRS-en-prules/?wsdl returned response code 404]  

1) Go to CAS bin directory
Example: /opt/app/endeca/CAS/3.1.x/bin

2) Use below command pattern to delete if any existing CAS record stores
Example: Delete 4 record stores created for ATG-Endeca integration 

./component-manager-cmd.sh delete-component -n CRS_en_schema
./component-manager-cmd.sh delete-component -n CRS_en_dimvals
./component-manager-cmd.sh delete-component -n CRS_en_prules
./component-manager-cmd.sh delete-component -n CRS_en_data

3) Use below command pattern to create CAS record stores

Example:Create 4 record stores required for ATG-Endeca integration

./component-manager-cmd.sh create-component -t RecordStore -n CRS_en_schema
./component-manager-cmd.sh create-component -t RecordStore -n CRS_en_data
./component-manager-cmd.sh create-component -t RecordStore -n CRS_en_dimvals
./component-manager-cmd.sh create-component -t RecordStore -n CRS_en_prules

4) Use below command pattern to Read CAS record stores

recordstore-cmd read-baseline -a CRS_en_data -n 50000 -f CRS_en_data.xml
recordstore-cmd read-baseline -a CRS_en_schema -n 50000 -f CRS_en_schema.xml
recordstore-cmd read-baseline -a CRS_en_dimvals -n 500 -f CRS_en_dimvals.xml
recordstore-cmd read-baseline -a CRS_en_prules -n 50000 -f CRS_en_prules.xml

How to Backup/Restore Endeca CAS state? - Endeca CAS

Backup CAS State:
1) Stop the Endeca CAS Service.
2) Copy state directory to outside CAS installation folder
     /opt/app/endeca/CAS/workspace/state
3) Restart CAS  

Restore CAS State:
 1) Stop the Endeca CAS Service.
2) Copy the backed up state directory to below location
      /opt/app/endeca/CAS/workspace/state
3) Restart CAS

How to Backup/Restore Endeca Dimension value Id's mapping? - Endeca CAS

You can back up dimension value Id mappings to a CSV file using the CAS Server Command-line Utility. 

To back up dimension value Id mappings:

/opt/app/endeca/CAS/11.1.0/bin>./cas-cmd.sh exportDimensionValueIdMappings -m APPNAME-dimension-value-id-manager -f dimvalIdMgr.csv 

To restore dimension value Id mappings:
/opt/app/endeca/CAS/11.1.0/bin>./cas-cmd.sh importDimensionValueIdMappings -m APPNAME-dimension-value-id-manager -f dimvalIdMgr.csv

You can find dimension value Id manager from below location:

/opt/app/endeca/apps/APPNAME/control:>vi initialize_RS_services.sh
DVAL_ID_MGR_NAME=APPNAME-dimension-value-id-manager

Endeca CAS | Multiple Record store merge - Oracle Commerce 11.1

Multiple record store could be used to merge data. This merging would be work as switch join.  Find out the steps below to use multiple record store :-

 1. Open <Endeca App Path>\APPNAME\config\cas\last-mile-crawl.xml file and add all record stores. Find out sample below. Multiple record store is highlighted below. We can add any number of record stores to merge.

                                <sourceConfig>

                                                <moduleId>

                                                                <id>com.endeca.cas.source.RecordStoreMerger</id>

                                                </moduleId>

                                                <moduleProperties>

                                                                <moduleProperty>

                                                                                <key>dataRecordStores</key>

                                                                                <value>APPNAME-data</value>

                                                                                <value>APPNAME-web-crawl</value>

                                                                </moduleProperty>

                                                                <moduleProperty>

                                                                                <key>dimensionValueRecordStores</key>

                                                                                <value>APPNAME-dimvals</value>

                                                                </moduleProperty>

                                                </moduleProperties>

                                                <excludeFilters />

                                                <includeFilters />

                                </sourceConfig>

  2. Run following command to update configuration in CAS

a.       <Installpath>\CAS\11.1.0\bin >cas-cmd.bat updateCrawls -f <Endeca App Path>\APPNAME\config\cas\last-mile-crawl.xml

  3.  Run Baseline.

Note :- Endeca CAS uses record.id as unique identifier. We can define our own. Suppose two record appears in record store A and B with same record id . CAS would discard one of the record.

Endeca concurrent write exception - atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException???

Problem:You will see below exception in Jboss log
ERROR [ProductCatalogSimpleIndexingAdmin]
atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40
at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:225)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForKey(AbstractRecordStoreAggregateSession.java:300)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForContext(AbstractRecordStoreAggregateSession.java:249)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForCurrentContext(AbstractRecordStoreAggregateSession.java:238)
at atg.endeca.index.AbstractRecordStoreAggregateSession.submitRecord(AbstractRecordStoreAggregateSession.java:334)
at atg.endeca.index.LocaledExporter.exportRecords(LocaledExporter.java:295)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:258)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:208)
at atg.endeca.index.LocaledExporter.performBaselineUpdate(LocaledExporter.java:345)
at atg.endeca.index.admin.IndexingTask.doTask(IndexingTask.java:401)
at atg.endeca.index.admin.IndexingTask.performTask(IndexingTask.java:359)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasks(IndexingPhase.java:265)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasksSerially(IndexingPhase.java:242)
at atg.endeca.index.admin.IndexingJob.performJobSerially(IndexingJob.java:264)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:657)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:636)
at atg.endeca.index.admin.SimpleIndexingAdmin$2.run(SimpleIndexingAdmin.java:1008)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:141) at $Proxy154.startTransaction(Unknown Source) at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:217)

Solution:
If you stop Endeca indexing from ATG(/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/) before it completes then it will acquire lock and then subsequent baseline updates will fail with above exception.

Solution#1: Restarting Endeca CAS which will remove the lock acquired - Temporary fix

Solution #2:  Make sure you restart CAS after every baseline update to avoid this issue plus some other issues related to CAS

Add /control/initialize_RS_services.sh script at the end of  /control/baseline_update.sh script - It will delete all the records stores and create new one each time and also helps to improve some level of performance on CAS side

Solution #3: Set resetActive flag to true on following components to rollback any previously active transactions
/dyn/admin/nucleus/atg/endeca/index/DataDocumentSubmitter/?propertyName=resetActive

/dyn/admin/nucleus/atg/endeca/index/DimensionDocumentSubmitter/?propertyName=resetActive
/dyn/admin/nucleus/atg/endeca/index/SchemaDocumentSubmitter /?propertyName=resetActive

How to check endeca process(MDEX, Platformservices, MDEX, CAS etc..) running on the box? - Endeca Process

Run below command:
ps -ef | grep endeca
You should see all below process running else need to restart manually.
 1) MDEX(Dgraphs)  Ex: /opt/app/endeca/MDEX/6.5.1/bin/dgraph
2) Platform Services Ex: /opt/app/endeca/PlatformServices/11.1.0/j2sdk/bin/java
3) Tools and Framework Ex: /opt/app/endeca/ToolsAndFrameworks/11.1.0/server/j2sdk/bin/java

How to check CAS is running?
ps -ef | grep cas  
You should see process with /bin/bash ./cas-service.sh  if not then restart CAS.

How to read data from Endeca CAS record store instance, Run ATG CMS and Endeca baseline update through Dynamo admin? - ATG Endeca Integration Issues

If you are not sure whether the ATG records are pushed at Endeca CAS level or not, then read the record store from the CAS by running below script.

/opt/app/endeca/CAS/version/bin>./recordstore-cmd.sh read-baseline -a <appname_en_data> -f <new-file-path>.xml

Ex. - /opt/app/endeca/CAS/version/bin>./recordstore-cmd.sh read-baseline -a APPNAME_en_data -f ../../APPNAME_en_data.xml

It will help to cross verify if any record is not showing in Endeca but merchandized in BCC and deployed. Sometime CMS(Catalog Maintenance Service) fails and does not pass latest records to Endeca. Try manually running CMS and Endeca Indexing.

Running ATG CMS(Catalog Maintenance Service):

http://localhost:port/dyn/admin/atg/commerce/admin/en/maintenance/startService.jhtml?process=BasicMaintProcess

Running Endeca Baseline update:
http://localhost:port/dyn/admin/nucleus//atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/