IFCR PromotionSite Error retrieving content bytes and Site loses all Endeca content - Endeca 3.1.x

Problem: IFCR Throws Below ERRORS and Site loses all Endeca content
++++++++++++++++++++
26.11.2015 01:15:35.779 *ERROR* [New I/O server worker #1-24] com.endeca.ifcr.publish.impl.content.server.PromotionSite Error retrieving content bytes. javax.jcr.RepositoryException: /51/49/9fec56a9409dbc581ca02b80efec/%7bhttp%3a%2f%2fwww.jcp.org%2fjcr%2f1.0%7ddata.0.bin: the specified resource does not exist
at org.apache.jackrabbit.core.value.BLOBInResource.getStream(BLOBInResource.java:111)
at org.apache.jackrabbit.core.value.BLOBInResource.copy(BLOBInResource.java:100)
...
...
Caused by: org.apache.jackrabbit.core.fs.FileSystemException: /opt/app/endeca/ToolsAndFrameworks/3.1.1/server/workspace/state/sling/_ifcr/jackrabbit/workspaces/default/blobs/51/49/9fec56a9409dbc581ca02b80efec/%7bhttp%3a%2f%2fwww.jcp.org%2fjcr%2f1.0%7ddata.0.bin does not denote an existing file
26.11.2015 01:15:46.646 Channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:639)
at org.jboss.net*ERROR* [New I/O server worker #1-26] com.endeca.ifcr.publish.impl.content.server.ContentServer Exception caught in channel handling. java.nio.channels.ClosedChannelException
++++++++++++++++++++
Solution:
Workaround Solution:
Try to execute multiple times promote content script
/opt/app/endeca/apps/APP_NAME/control> promote_content.sh| promote_content.bat

It could be due to:  
1) Increased number of application servers 
2) Increased number of Workbench clients
3) Larger promote_content snapshots
4) Endeca Resource Resolver issue

You can ask for a patch to fix it.

How to verify Endeca FORGE data? - Endeca FORGE

Forge Process creates binary data in /APP_NAME/data/forge_ouput directory
For forge-less/CAS-based system data is in /APP_NAME/data/cas_ouput.

Step 1: Copy binary file Ex: Discover-sgmt0.records.binary to the below location
    Ex: C:\Endeca\PlatformServices\11.1.0\bin>

Step 2: Run below command
C:\Endeca\PlatformServices\11.1.0\bin>binary2xml.exe C:\Endeca\apps\Discover\data\forge_output\Discover-sgmt0.records.binary  C:\Endeca\out.xml  

Step 3: XML file created in the path mentioned
Ex: C:\Endeca\out.xml  

How to gather Server Statistics During the Endeca Baseline? - Endeca Baseline

Run below command before you kick of baseline update.
 top -b -d 60 -n 20 > ServerStats.out
Above command gathers 'top' statistics every 60 seconds for the next 20 mins. 

How to check the Endeca Dgraph version?

/opt/app/endeca/MDEX/6.4.0/bin]

$ ./dgraph --version

Endeca dgraph version 6.4.0.743933

Supports opti-XML version : 2.00

Supports binary IR version: 640

Interface specification :

  READS:dgidx=6.3.0a WRITES:dgraph=IR640

ICU version 4.2.1

OLT version OLT 1.3 121107

Endeca Workbench EAC admin console disappeared from Workbench for ADMIN user - Endeca Workbench

Problem Statement:  EAC admin console disappeared from Workbench for ADMIN user 

Solution:
 1. Replace the following jar files in the production instance with the ones from your working environment in $ENDECA_TOOLS_CONF/state/sling/_ifcr/startup/25:
workbench-container-3.1.1.jar
ExperienceManagerContent-3.1.1-xmgr.jar
workbench-legacy-tools-definitions-3.1.1.jar

2. Reload the jar files -  To reload the jars, use the following Workbench URL:
http://host:port/ifcr/system/console/bundles 


The bundles with the names of the jars needed to be located and updated by clicking the "Install/Update" button, selecting the "Start Bundle" check box, Ensure that the "Start Level" value is the same as the value displayed when you expand the specific jar (ie: "25") and browsing for the specific jar file.This needs to be repeated for each specific jar.

3. Stop and restart the jar files 

In the same console windows, in the "Actions" column for each bundle that was updated, click the "Stop" button for each row. The "Status" for that bundle will change to "Resolved". Click the "Start" button for the bundle and the "Status" will say "Active".
At this point the missing components should become visible in Workbench.

4. Stop and restart Tools & Frameworks services

ATG-Endeca Site Crashed due to Endeca content repository Corruption - v3.1.1

Problem Statement:
Site got crashed and not able to import backed-up Experience Manager rules.

Error when you Import rules:
Nov 5, 2015 3:25:32 PM com.endeca.soleng.eac.toolkit.utility.IFCRUtility importNode
INFO: Uploading contents to: http://localhost:8006/ifcr/sites/APP_NAME/configuration/tools/xmgr
Nov 5, 2015 3:25:33 PM com.endeca.soleng.eac.toolkit.utility.IFCRUtility importNode
INFO: Finished uploading contents.
Caused by: com.endeca.soleng.eac.toolkit.utility.IFCRUtility$HttpStatusException: Internal Server Error (500): javax.jcr.ItemNotFoundException: 731f7677-dc43-47d3-968e-3d1
6c49970c2/{}contentCollectionReferences
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.execute(IFCRUtility.java:724)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.httpPost(IFCRUtility.java:671)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.httpPost(IFCRUtility.java:653)
        at com.endeca.soleng.eac.toolkit.utility.IFCRUtility.deleteNode(IFCRUtility.java:261)
        at com.endeca.soleng.eac.toolkit.component.IFCRComponent.importXml(IFCRComponent.java:195)
        at com.endeca.soleng.eac.toolkit.component.IFCRComponent.importXmlWithPrompt(IFCRComponent.java:144)
        ... 7 more


Solution:
1) Make sure you have a good copy of content repository export. To do this, execute export_site on an working environment.
 Ex: /opt/app/endeca/APP_NAME/control>./export_site.sh  XMRules.xml
2) Stop workbench.
3) Go to $ENDECA_TOOLS_CONF/state/
4) Rename sling and jackrabbit directory in here to anything other than sling and jackrabbit, or back them up to a different directory.
5) Restart workbench, this will cause the sling IFCR repository to be cleaned up.
   Note: If you have multiple apps, then all of them will be cleaned up. In this situation, run 6) and 7) below for each app.
6) Then run initialize_services./sh|bat --force from the app
         Ex: /opt/app/endeca/APP_NAME/control>./initialize_services./sh|bat --force 
7) Then import_site <export file>
       Ex: /opt/app/endeca/APP_NAME/control>./import_site.sh XMRules.xml
8) Execute a baseline_update from dyn/admin followed by a promote_content

Endeca Dgidx and Dgraph Flags - Endeca Flags

Dgidx Flags: 

http://docs.oracle.com/cd/E29584_01/webhelp/IAPAdmin/toc.htm#Dgidx%20flags 

Dgraph Flags:

http://docs.oracle.com/cd/E29584_01/webhelp/IAPAdmin/toc.htm#Dgraph%20flags

Oracle Commerce 11.2 Released - Release Notes

Known Issues in this release:
 

1) (18816983) BCC user segment not appearing in Experience Manager

2)(18780298) Cartridge example causes error

3)(18895941) New user segment not appearing
4)(21363913) Renaming and moving pages are not updated in Experience Manager projects

5)(21502844) Renaming and saving a site page produces "Page not found" error
6)(21649751) Authoring assemblers are not being updated correctly after importing Workbench content

7)(21657293) Error message/out of sync message is displayed after renaming a rule

8)(21671840) Renaming a folder displays misleading error message

9)(21801912) Preview does not refresh after adding content

10)(21822985) Increase and decrease icons are missing

11)(21824187) Intermittently cannot add new rule using Internet Explorer

12)(21864376) Cannot export an application when a folder name is changed from upper to lower case

More info in below thread:
http://docs.oracle.com/cd/E67226_01/Common.112/pdf/RelNotes.pdf

ATG-Endeca Oracle Commerce site performance degrades and toll on application tier memory - Best Practices

Problem Statement:

ATG-Endeca Commerce site performance degrades and toll on application tier memory

Solution:

If aggregation is defined in the Endeca with below configuration(allbins=2) and there are so many sub records are rolled up per product. 
/atg/endeca/assembler/cartridge/handler/config/ResultsListConfig
subRecordsPerAggregateRecord=TWO 
You can fix it as below:
1) Configure records Per Aggregate Record set to 1
 atg/endeca/assembler/cartridge/handler/config/ResultsListConfig
   # For aggregate records, sets the number of sub records that should be included in the results
 subRecordsPerAggregateRecord=ONE 

Endeca Forge recordstore adapter Fails with Exception - com.endeca.edf.adapter.AdapterException: java.lang.RuntimeException: java.io.IOException

Problem Statement:
Endeca Forge is failing with the following exception. 

Forge Log:
ERROR FORGE {baseline}: (AdapterRunner): java.lang.RuntimeException: java.io.IOException: http://localhost:8500/CRS_en_data/?wsdl returned response code 404 com.endeca.edf.adapter.AdapterException: java.lang.RuntimeException: java.io.IOException

CAS Log:

ERROR [] [main] com.endeca.itl.bootstrap.CasBootStrapServlet: Failed to initialize CAS: java.sql.SQLException: Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@6c679e04[file =/opt/app/endeca/CAS/workspace/state/cas/db/db.lck, exists=true, locked=false, valid=false, ]

Solution:

Above Exception means forge adapter is not able to communicate with CAS record store instance.  There is lock acquired on hsqldb.

Best way to release lock is kill CAS process and restart it.

Promote content from Workbench 3.1.x to a Dgraph breaks the site - Endeca Site Down

Problem Statement:
When you run promote_content.bat|sh script site goes down immediately

Check IFCR Log :
Path: /opt/app/endeca/ToolsAndFrameworks\3.1.x/server/workspace/logs/ifcr.log
 
18.08.2015 19:49:38.641 *ERROR* [172.25.74.131 [1439945378629] GET /ifcr/sites/CRS/thesaurus.html HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Uncaught SlingException org.apache.sling.api.SlingException:
  ...
Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "en"
    at java.net.URLDecoder.decode(URLDecoder.java:173)
    at java.net.URLDecoder.decode(URLDecoder.java:83)
    at org.apache.jsp.apps.endeca.thesaurus.html_jsp._jspService(html_jsp.java:118)
    at org.apache.sling.scripting.jsp.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.sling.scripting.jsp.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:419)
    ... 61 more

Solution:
Using a non-xml 1.0 safe character in a thesaurus entry causes the thesaurus content publish step within the promote_content process to fail. The rest of the content may succeed to be published, or the publish may fail entirely.

Obtain the following patch for Tools And Frameworks 3.1.x from support.oracle.com:

16863449: XM3.1.1: MDEX 6.4 INVALID XML CHARACTERS USED IN THESAURUS CAUSE PUBLISH FAILURE

Caught an exception while checking provisioning EacCommunicationException - Endeca Provisioning

Problem Statement: 

    While doing Endeca provisioning got below exception

     $ /opt/app/endeca/apps/CRS/control/initialize_services.sh --force 

    Removing existing crawl configuration for crawl CRS-last-mile-crawl (ignore errors if crawl doesn't exist)
com.endeca.itl.cas.cmd.CasCmdException: CRS-last-mile-crawl was not found
Removing Record Store CRS-data (ignore errors if Record Store doesn't exist)
Successfully deleted component: CRS-data
Removing Record Store CRS-dimvals (ignore errors if Record Store doesn't exist)
Successfully deleted component: CRS-dimvals
Removing Dimension Value Id Manager CRS-dimension-value-id-manager (ignore errors if Dimension Value Id Manager doesn't exist)
Removing existing application provisioning...
[08.14.15 12:17:17] INFO: Removing application. Any active components will be forced to stop.
[08.14.15 12:17:17] INFO: Application 'CRS' is not defined.
Creating Dimension Value Id Manager CRS-dimension-value-id-manager
Creating Record Store CRS-data
Successfully created component: CRS-data
Successfully set recordstore configuration.
Creating Record Store CRS-dimvals
Successfully created component: CRS-dimvals
Setting EAC provisioning and performing initial setup...
[08.14.15 12:17:33] INFO: Checking definition from AppConfig.xml against existing EAC provisioning.
[08.14.15 12:17:33] INFO: Setting definition for application 'CRS'.
[08.14.15 12:17:33] SEVERE: Caught an exception while checking provisioning.
Caused by com.endeca.soleng.eac.toolkit.exception.EacCommunicationException
com.endeca.soleng.eac.toolkit.application.Application setDefinition - Caught exception while defining application 'CRS'.
Caused by com.endeca.eac.client.EACFault
sun.reflect.NativeConstructorAccessorImpl newInstance0 - null 

  Solution: 
         1)      Stop the EAC (Endeca HTTP Service)
         2)      Rename the directory  /opt/app/endeca/PlatformServices/workspace/state to state-backup
         3)      Create an empty state directory
         4)      Start the Endeca HTTP Service again
         5)      Now try to provision application using initialize_services.sh --force command

Endeca Relevance Ranking - Best practice

Best Practice: For applications dealing with Retail catalog data, the preferred order of modules is:
1. NTerms
2. MaxField
3. Glom
4. Exact
5. Static

Explanation: 
NTerms, the first module, ensures that in a multi-word search, the more words that match in the record, the higher the record is scored.
MaxField puts cross-field matches (see Allowing Cross-Field Matches) as high in priority as possible, to the point where they could tie with non-cross-field matches.
Glom, decomposes cross-field matches, effectively breaking any ties resulting from MaxField. Together, MaxField and Glom provide appropriate cross-field match ordering, depending upon what matched.
Exact module means that an exact match in a highly-ranked member of the search interface is placed higher than a partial or cross-field match.
Optionally, the Static module can be used to sort remaining ties by criteria such as Price or InstockWeb.

Example: Below Screenshot shows on how to configure Rel.Ranking Modules using Endeca Pipeline



Best Practice: For applications dealing with Document Repository Data, the preferred order of modules is:
1. NTerms
2. MaxField
3. Glom
4. Phrase
5. Static

Explanation:
NTerms, the first module, ensures that in a multi-word search, the more words that match in the record, the higher the record is scored.
MaxField puts cross-field matches (see Allowing Cross-Field Matches) as high in priority as possible, to the point where they could tie with non-cross-field matches.
Glom, decomposes cross-field matches, effectively breaking any ties resulting from MaxField. Together, MaxField and Glom provide appropriate cross-field match ordering, depending upon what matched.
Phrase module ensures that results containing the user's query as an exact phrase are given a higher priority than matches containing the user's search terms sprinkled throughout the text.
Optionally, the Static module can be used to sort remaining ties by criteria such as Price or SalesRank.