Endeca concurrent write exception - atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException???

Problem:You will see below exception in Jboss log
ERROR [ProductCatalogSimpleIndexingAdmin]
atg.repository.search.indexing.IndexingException: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40
at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:225)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForKey(AbstractRecordStoreAggregateSession.java:300)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForContext(AbstractRecordStoreAggregateSession.java:249)
at atg.endeca.index.AbstractRecordStoreAggregateSession.getSessionForCurrentContext(AbstractRecordStoreAggregateSession.java:238)
at atg.endeca.index.AbstractRecordStoreAggregateSession.submitRecord(AbstractRecordStoreAggregateSession.java:334)
at atg.endeca.index.LocaledExporter.exportRecords(LocaledExporter.java:295)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:258)
at atg.endeca.index.LocaledExporter.export(LocaledExporter.java:208)
at atg.endeca.index.LocaledExporter.performBaselineUpdate(LocaledExporter.java:345)
at atg.endeca.index.admin.IndexingTask.doTask(IndexingTask.java:401)
at atg.endeca.index.admin.IndexingTask.performTask(IndexingTask.java:359)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasks(IndexingPhase.java:265)
at atg.endeca.index.admin.IndexingPhase.performPhaseTasksSerially(IndexingPhase.java:242)
at atg.endeca.index.admin.IndexingJob.performJobSerially(IndexingJob.java:264)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:657)
at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:636)
at atg.endeca.index.admin.SimpleIndexingAdmin$2.run(SimpleIndexingAdmin.java:1008)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.endeca.itl.recordstore.ConcurrentWriteException: Write in progress with generation 40 at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:141) at $Proxy154.startTransaction(Unknown Source) at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.beginSession(RecordStoreDocumentSubmitterSessionImpl.java:217)

Solution:
If you stop Endeca indexing from ATG(/atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/) before it completes then it will acquire lock and then subsequent baseline updates will fail with above exception.

Solution#1: Restarting Endeca CAS which will remove the lock acquired - Temporary fix

Solution #2:  Make sure you restart CAS after every baseline update to avoid this issue plus some other issues related to CAS

Add /control/initialize_RS_services.sh script at the end of  /control/baseline_update.sh script - It will delete all the records stores and create new one each time and also helps to improve some level of performance on CAS side

Solution #3: Set resetActive flag to true on following components to rollback any previously active transactions
/dyn/admin/nucleus/atg/endeca/index/DataDocumentSubmitter/?propertyName=resetActive

/dyn/admin/nucleus/atg/endeca/index/DimensionDocumentSubmitter/?propertyName=resetActive
/dyn/admin/nucleus/atg/endeca/index/SchemaDocumentSubmitter /?propertyName=resetActive