Endeca Forge recordstore adapter Fails with Exception - com.endeca.edf.adapter.AdapterException: java.lang.RuntimeException: java.io.IOException

Problem Statement:
Endeca Forge is failing with the following exception. 

Forge Log:
ERROR FORGE {baseline}: (AdapterRunner): java.lang.RuntimeException: java.io.IOException: http://localhost:8500/CRS_en_data/?wsdl returned response code 404 com.endeca.edf.adapter.AdapterException: java.lang.RuntimeException: java.io.IOException
CAS Log:
ERROR [] [main] com.endeca.itl.bootstrap.CasBootStrapServlet: Failed to initialize CAS: java.sql.SQLException: Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@6c679e04[file =/opt/app/endeca/CAS/workspace/state/cas/db/db.lck, exists=true, locked=false, valid=false, ]

Solution:
Above Exception means forge adapter is not able to communicate with CAS record store instance.  There is lock acquired on hsqldb.
Best way to release lock is kill CAS process and restart it.

Promote content from Workbench 3.1.x to a Dgraph breaks the site - Endeca Site Down

Problem Statement:
When you run promote_content.bat|sh script site goes down immediately

Check IFCR Log :
Path: /opt/app/endeca/ToolsAndFrameworks\3.1.x/server/workspace/logs/ifcr.log
 
18.08.2015 19:49:38.641 *ERROR* [172.25.74.131 [1439945378629] GET /ifcr/sites/CRS/thesaurus.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Uncaught SlingException org.apache.sling.api.SlingException:
  ...
Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "en"

    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at java.net.URLDecoder.decode(URLDecoder.java:83)
    at org.apache.jsp.apps.endeca.thesaurus.html_jsp._jspService(html_jsp.java:118)
    at org.apache.sling.scripting.jsp.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.sling.scripting.jsp.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:419)
    ... 61 more


Solution:
Using a non-xml 1.0 safe character in a thesaurus entry causes the thesaurus content publish step within the promote_content process to fail. The rest of the content may succeed to be published, or the publish may fail entirely.

Obtain the following patch for Tools And Frameworks 3.1.x from support.oracle.com:

16863449: XM3.1.1: MDEX 6.4 INVALID XML CHARACTERS USED IN THESAURUS CAUSE PUBLISH FAILURE

Caught an exception while checking provisioning EacCommunicationException - Endeca Provisioning

Problem Statement: 
    While doing Endeca provisioning got below exception
     $ /opt/app/endeca/apps/CRS/control/initialize_services.sh --force 

    Removing existing crawl configuration for crawl CRS-last-mile-crawl (ignore errors if crawl doesn't exist)
com.endeca.itl.cas.cmd.CasCmdException: CRS-last-mile-crawl was not found
Removing Record Store CRS-data (ignore errors if Record Store doesn't exist)
Successfully deleted component: CRS-data
Removing Record Store CRS-dimvals (ignore errors if Record Store doesn't exist)
Successfully deleted component: CRS-dimvals
Removing Dimension Value Id Manager CRS-dimension-value-id-manager (ignore errors if Dimension Value Id Manager doesn't exist)
Removing existing application provisioning...
[08.14.15 12:17:17] INFO: Removing application. Any active components will be forced to stop.
[08.14.15 12:17:17] INFO: Application 'CRS' is not defined.
Creating Dimension Value Id Manager CRS-dimension-value-id-manager
Creating Record Store CRS-data
Successfully created component: CRS-data
Successfully set recordstore configuration.
Creating Record Store CRS-dimvals
Successfully created component: CRS-dimvals
Setting EAC provisioning and performing initial setup...
[08.14.15 12:17:33] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
[08.14.15 12:17:33] INFO: Setting definition for application 'CRS'.
[08.14.15 12:17:33] SEVERE: Caught an exception while checking provisioning.
Caused by com.endeca.soleng.eac.toolkit.exception.EacCommunicationException
com.endeca.soleng.eac.toolkit.application.Application setDefinition - Caught exception while defining application 'CRS'.
Caused by com.endeca.eac.client.EACFault
sun.reflect.NativeConstructorAccessorImpl newInstance0 - null
 

  Solution: 
         1)      Stop the EAC (Endeca HTTP Service)
         2)      Rename the directory  /opt/app/endeca/PlatformServices/workspace/state to state-backup
         3)      Create an empty state directory
         4)      Start the Endeca HTTP Service again
         5)      Now try to provision application using initialize_services.sh --force command

Endeca Relevance Ranking - Best practice

Best Practice: For applications dealing with Retail catalog data, the preferred order of modules is:
1. NTerms
2. MaxField
3. Glom
4. Exact
5. Static


Explanation: 
NTerms, the first module, ensures that in a multi-word search, the more words that match in the record, the higher the record is scored.
MaxField puts cross-field matches (see Allowing Cross-Field Matches) as high in priority as possible, to the point where they could tie with non-cross-field matches.
Glom, decomposes cross-field matches, effectively breaking any ties resulting from MaxField. Together, MaxField and Glom provide appropriate cross-field match ordering, depending upon what matched.
Exact module means that an exact match in a highly-ranked member of the search interface is placed higher than a partial or cross-field match.
Optionally, the Static module can be used to sort remaining ties by criteria such as Price or InstockWeb.


Example: Below Screenshot shows on how to configure Rel.Ranking Modules using Endeca Pipeline



Best Practice: For applications dealing with Document Repository Data, the preferred order of modules is:
1. NTerms
2. MaxField
3. Glom
4. Phrase

5. Static

Explanation:
NTerms, the first module, ensures that in a multi-word search, the more words that match in the record, the higher the record is scored.
MaxField puts cross-field matches (see Allowing Cross-Field Matches) as high in priority as possible, to the point where they could tie with non-cross-field matches.
Glom, decomposes cross-field matches, effectively breaking any ties resulting from MaxField. Together, MaxField and Glom provide appropriate cross-field match ordering, depending upon what matched.

Phrase module ensures that results containing the user's query as an exact phrase are given a higher priority than matches containing the user's search terms sprinkled throughout the text.
Optionally, the Static module can be used to sort remaining ties by criteria such as Price or SalesRank.