Caught an exception while checking provisioning EacCommunicationException - Endeca Provisioning

Problem Statement: 

    While doing Endeca provisioning got below exception

     $ /opt/app/endeca/apps/CRS/control/initialize_services.sh --force 

    Removing existing crawl configuration for crawl CRS-last-mile-crawl (ignore errors if crawl doesn't exist)
com.endeca.itl.cas.cmd.CasCmdException: CRS-last-mile-crawl was not found
Removing Record Store CRS-data (ignore errors if Record Store doesn't exist)
Successfully deleted component: CRS-data
Removing Record Store CRS-dimvals (ignore errors if Record Store doesn't exist)
Successfully deleted component: CRS-dimvals
Removing Dimension Value Id Manager CRS-dimension-value-id-manager (ignore errors if Dimension Value Id Manager doesn't exist)
Removing existing application provisioning...
[08.14.15 12:17:17] INFO: Removing application. Any active components will be forced to stop.
[08.14.15 12:17:17] INFO: Application 'CRS' is not defined.
Creating Dimension Value Id Manager CRS-dimension-value-id-manager
Creating Record Store CRS-data
Successfully created component: CRS-data
Successfully set recordstore configuration.
Creating Record Store CRS-dimvals
Successfully created component: CRS-dimvals
Setting EAC provisioning and performing initial setup...
[08.14.15 12:17:33] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
[08.14.15 12:17:33] INFO: Setting definition for application 'CRS'.
[08.14.15 12:17:33] SEVERE: Caught an exception while checking provisioning.
Caused by com.endeca.soleng.eac.toolkit.exception.EacCommunicationException
com.endeca.soleng.eac.toolkit.application.Application setDefinition - Caught exception while defining application 'CRS'.
Caused by com.endeca.eac.client.EACFault
sun.reflect.NativeConstructorAccessorImpl newInstance0 - null 

  Solution: 
         1)      Stop the EAC (Endeca HTTP Service)
         2)      Rename the directory  /opt/app/endeca/PlatformServices/workspace/state to state-backup
         3)      Create an empty state directory
         4)      Start the Endeca HTTP Service again
         5)      Now try to provision application using initialize_services.sh --force command

How to Remove the Endeca application? - Endeca

While removing an Endeca application, do not simply remove the Endeca application directories or delete EAC state directories (ENDECA_CONF/state/eac*) from in the file system.
This leaves application artifacts, which can cause naming conflicts when creating a new project with the same name. Clients running Oracle ATG Commerce, having random Endeca application artifacts can interfere with Commerce Reference Store installation and with indexing functionality.
To completely remove an Endeca application follow below steps:

Step 1:  Remove APP using runcommand
              <Endeca-app-dir>\control\runcommand.sh|bat --remove-app
              CRS\control\runcommand.sh|bat --remove-app

Step 2: Drop the corresponding Record stores from CAS
            Check names of application record stores registered in CAS.
             $CAS_ROOT\bin>component-manager-cmd.bat/.sh list-components
Remove the record stores corresponding to your Endeca application.  The record store names will be prefixed with the application name.
For example:
$CAS_ROOT/bin>component-manager-cmd.bat/.sh delete-component -n myAppName_en_data 
$CAS_ROOT/bin>component-manager-cmd.bat/.sh delete-component -n myAppName_en_schema
$CAS_ROOT/bin>component-manager-cmd.bat/.sh delete-component -n myAppName_en_prules
$CAS_ROOT/bin>component-manager-cmd.bat/.sh delete-component -n myAppName_en_dimvals

Step 3: Finally, remove the Endeca application directory from the file system

How to print all Endeca components status? - Endeca EAC

Displaying the status of Endeca application components.

1) Go to Endeca APP control directory
Example: /opt/app/endeca/apps/CRS/control

2)  Use runcommand to print status of all Endeca components 
$ ./runcommand.sh --print-status
Status of components defined in AppConfig.xml.
  AuthoringDgraph: Running
  DgraphA1: Failed
  LogServer: Running
  WeeklyReportGenerator: NotRunning
  DailyReportGenerator: NotRunning
  DaySoFarReportGenerator: NotRunning
  WeeklyHtmlReportGenerator: NotRunning
  DailyHtmlReportGenerator: NotRunning
  DaySoFarHtmlReportGenerator: NotRunning
  ConfigurationGeneratorForge: NotRunning
  Forge: NotRunning
  PartialForge: NotRunning
  Dgidx: NotRunning

Oracle recommendations - Multi-site Operations setup

Here is the standard recommendations from Oracle on multi-site Operations setup:

How to check the Endeca locks acquired and release them? - Endeca Scripts

1) Go to below directory
         /opt/app/endeca/PlatformServices/11.1.0/bin

2) To check all the flags acquired run below command
         ./eaccmd.sh list-flags --app CRS
         
Flag Examples: baseline_data_ready
                            update_lock

3) Depending on the output you can remove any flags acquired. Here is the command to remove flags one by one:
           ./eaccmd.sh remove-flag --app CRS--flag baseline_data_ready

4) you can remove all flags at once by using below command:
            ./eaccmd.sh remove-all-flags --app CRS

FYI - CRS is Endeca application name

Creating the Endeca EAC Application via Scripts - Deployment Template

How to create the Endeca EAC application?

1. On the machine where Oracle Commerce Guided Search with Experience Manager is installed, create the directory where the EAC application will reside
   Example: /opt/app/endeca/apps 
   
   
2. Copy the <ATG11dir>/CommerceReferenceStore/Store/Storefront/deploy directory to a temporary location on the machine where Oracle Commerce Guided Search with Experience Manager is installed.
   Example: /opt/app/ATGtemp
   
   
3. Execute the deploy script. 
   Example:
   /opt/app/endeca/ToolsAndFrameworks/11.1.0/deployment_template/bin >./deploy.sh --app /opt/app/ATGtemp/deploy/deploy.xml
   
   
4. Press Enter to continue with the install
5. Enter the following information for the script prompts:
   • Enter N to install a base deployment.
     
     
   • Enter CRS for the name of the application.
     
     
   • Enter the full path to the Apps directory, 
     Example: /opt/app/endeca/apps 
   • Enter the EAC port (default is 8888)
     
     
   • Workbench port number [8006]
     
     
   • Dgraph1 (Live Dgraph) port (default is 15000)
     
     
   • Authoring Dgraph port (default is 15002)
     
     
   • LogServer port (default is 15010)
     
     
   • Enter the CAS install directory using UNIX-style forward slashes, even if on Windows. 
     Example:/opt/app/endeca/CAS/11.1.0
     
     
   • Enter the CAS version number (default is 11.1.0).
     
     
   • Enter the CAS hostname (default is localhost).
     
     
   • Enter the CAS port (default is 8500).
     
     
   • Enter the language code (default is en).
     
     
   • Enter the fully-qualified Workbench hostname (localhost)
     
     
   • Production application server host name. This is the host where the ATGProduction server will be running. [localhost]
     
     
   • Production application server HTTP port number. 
     IMPORTANT: If you are following the instructions in this guide, these values are 7003 for WebLogic, 8080 for JBoss and Tomcat, and 9081 for WebSphere.
     
     
   • Preview host name [localhost]
     
     
   • Preview port number.
     
     
     If you are following the instructions in this guide, these values are 7003 for WebLogic, 8080 for JBoss and Tomcat, and 9081 for WebSphere.
     
     
   • Context root of the preview application [crs]
     
     
   • Host name of the user segment server. This is the server that will respond to Experience Manager requests for Business Control Center user segment data. Oracle recommends using the Content Adminstration server (also called the ATGPublishing server in this guide). [localhost]
     
     
   • HTTP port of the user segment server.  
     IMPORTANT: If you are following the instructions in this guide, these values are 7003 for WebLogic, 8080 for JBoss and Tomcat, and 9081 for WebSphere. 
     
     
   • Path for the Oracle Wallet jps-config.xml file
     Example:  /opt/app/endeca/ToolsAndFrameworks/11.1.0/server/workspace/credential_store/jps-config.xml
   • Application configuration archive path. This is the location where content promoted using the file-based method is stored,
     Example, /opt/app/endeca/apps/CRS/data/workbench/application_export_archive 
   You see a message that says “Application successfully deployed” when the deployment is finished. 
   
   
6. Change directories to the EAC applications directory, 
   Example: /opt/app/endeca/apps 
   The CRS application directory is added as child of the /apps directory.

How to provision the EAC application?

1. In a UNIX shell, change directories to your EAC application’s /control directory,
   Example: /opt/app/endeca/apps/CRS/control
   

2. Execute the initialize_services.sh script.
   ./initialize_services.sh

How to update Endeca configurations using runcommand? - Endeca Tips

If you update AppConfig.xml or any file in /opt/app/endeca/apps/APP_NAME/config/script folder you need to update your configuration for your changes to be applied.

/opt/app/endeca/apps/APP_NAME/control$>./runcommand.sh --update-definition
[07.12.15 19:14:43] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
[07.12.15 19:14:44] INFO: Updating provisioning for component 'Dgidx'.
[07.12.15 19:14:44] INFO: Updating definition for component 'Dgidx'.
[07.12.15 19:14:45] INFO: Definition updated.