Endeca concurrent write exception - atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException???

Problem:You will see below exception in Jboss log
ERROR [ProductCatalogSimpleIndexingAdmin]
atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40
at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:225)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForKey(AbstractRecordStoreAggregateSession.java:300)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForContext(AbstractRecordStoreAggregateSession.java:249)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForCurrentContext(AbstractRecordStoreAggregateSession.java:238)
at atg.endeca.index.AbstractRecordStoreAggregateSession.submitRecord(AbstractRecordStoreAggregateSession.java:334)
at atg.endeca.index.LocaledExporter.exportRecords(LocaledExporter.java:295)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:258)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:208)
at atg.endeca.index.LocaledExporter.performBaselineUpdate(LocaledExporter.java:345)
at atg.endeca.index.admin.IndexingTask.doTask(IndexingTask.java:401)
at atg.endeca.index.admin.IndexingTask.performTask(IndexingTask.java:359)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasks(IndexingPhase.java:265)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasksSerially(IndexingPhase.java:242)
at atg.endeca.index.admin.IndexingJob.performJobSerially(IndexingJob.java:264)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:657)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:636)
at atg.endeca.index.admin.SimpleIndexingAdmin$2.run(SimpleIndexingAdmin.java:1008)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:141) at $Proxy154.startTransaction(Unknown Source) at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:217)

Solution:
If you stop Endeca indexing from ATG(/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/) before it completes then it will acquire lock and then subsequent baseline updates will fail with above exception.

Solution#1: Restarting Endeca CAS which will remove the lock acquired - Temporary fix

Solution #2:  Make sure you restart CAS after every baseline update to avoid this issue plus some other issues related to CAS

Add /control/initialize_RS_services.sh script at the end of  /control/baseline_update.sh script - It will delete all the records stores and create new one each time and also helps to improve some level of performance on CAS side

Solution #3: Set resetActive flag to true on following components to rollback any previously active transactions
/dyn/admin/nucleus/atg/endeca/index/DataDocumentSubmitter/?propertyName=resetActive

/dyn/admin/nucleus/atg/endeca/index/DimensionDocumentSubmitter/?propertyName=resetActive
/dyn/admin/nucleus/atg/endeca/index/SchemaDocumentSubmitter /?propertyName=resetActive

How to check endeca process(MDEX, Platformservices, MDEX, CAS etc..) running on the box? - Endeca Process

Run below command:
ps -ef | grep endeca
You should see all below process running else need to restart manually.
 1) MDEX(Dgraphs)  Ex: /opt/app/endeca/MDEX/6.5.1/bin/dgraph
2) Platform Services Ex: /opt/app/endeca/PlatformServices/11.1.0/j2sdk/bin/java
3) Tools and Framework Ex: /opt/app/endeca/ToolsAndFrameworks/11.1.0/server/j2sdk/bin/java

How to check CAS is running?
ps -ef | grep cas  
You should see process with /bin/bash ./cas-service.sh  if not then restart CAS.

How to read data from Endeca CAS record store instance, Run ATG CMS and Endeca baseline update through Dynamo admin? - ATG Endeca Integration Issues

If you are not sure whether the ATG records are pushed at Endeca CAS level or not, then read the record store from the CAS by running below script.

/opt/app/endeca/CAS/version/bin>./recordstore-cmd.sh read-baseline -a <appname_en_data> -f <new-file-path>.xml

Ex. - /opt/app/endeca/CAS/version/bin>./recordstore-cmd.sh read-baseline -a APPNAME_en_data -f ../../APPNAME_en_data.xml

It will help to cross verify if any record is not showing in Endeca but merchandized in BCC and deployed. Sometime CMS(Catalog Maintenance Service) fails and does not pass latest records to Endeca. Try manually running CMS and Endeca Indexing.

Running ATG CMS(Catalog Maintenance Service):

http://localhost:port/dyn/admin/atg/commerce/admin/en/maintenance/startService.jhtml?process=BasicMaintProcess

Running Endeca Baseline update:
http://localhost:port/dyn/admin/nucleus//atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/