Inserting Servlets in the DAF Pipeline

Inserting Servlets in the Pipeline
All pipeline servlet classes directly or indirectly implement interface atg.servlet.pipeline.PipelineableServlet.

The PipelineableServlet interface has two sub-interfaces that provide more flexibility for inserting new servlets into the pipeline:

  1. atg.servlet.pipeline.InsertableServlet
  2. atg.servlet.pipeline.DispatcherPipelineableServlet
InsertableServlet
The InsertableServlet interface lets a servlet insert itself into the pipeline when the service starts, without requiring changes to servlets already in the pipeline, through the insertAfterServlet property, which points back to the preceding servlet. The inserted servlet reads the preceding servlet’s nextServlet property and points to it as the next servlet to execute after itself.

Here are the steps to add an InsertableServlet to the servlet pipeline:
1) Write your servlet by extending atg.servlet.pipeline.InsertableServletImpl

package com.foo.endeca.assembler;  
import java.io.IOException;
import javax.servlet.ServletException;
import atg.core.util.StringUtils;
import atg.eai.framework.toolkit.IdLookup;
import atg.servlet.DynamoHttpServletRequest;
import atg.servlet.DynamoHttpServletResponse;
import atg.servlet.pipeline.InsertableServletImpl;     

public class SearchkeywordRedirectPipeline extends InsertableServletImpl {
               public void service(DynamoHttpServletRequest request, DynamoHttpServletResponse response) throws IOException, ServletException {
             String searchTerm = null;
            searchTerm = request.getParameter("Ntt");
            //Business logic goes here
            logDebug("URL for searchTerm : " + searchTerm);
            passRequest(request, response);
       }
      
2) Define the servlet as a globally scoped Nucleus component
      $class=com.foo.endeca.assembler.SearchkeywordRedirectPipeline
      $scope=global
      insertAfterServlet=/atg/dynamo/servlet/dafpipeline/CookieBufferServlet

Set the insertAfterServlet property of your servlet to point to the path of the pipeline servlet you want your servlet to follow. For example, you can insert a servlet after CookieBufferServlet as follows:

insertAfterServlet=/atg/dynamo/servlet/dafpipeline/CookieBufferServlet

When the inserted servlet finishes processing, it calls the method passRequest() (defined in InsertableServletImpl), which automatically passes the request and response objects to the next servlet in the pipeline.

Servlet Pipeline - All pipeline servlets in order of execution


You can use the Dynamo Server Admin Component Browser to view request handling pipeline servlets and their sequence within the pipeline:

Component path: /atg/dynamo/servlet/dafpipeline/DynamoHandler/


/atg/dynamo/servlet/dafpipeline/DynamoHandler
/atg/multisite/SiteContextPipelineServlet
/atg/dynamo/servlet/dafpipeline/SecurityServlet
/atg/dynamo/servlet/dafpipeline/TransactionServlet
/atg/dynamo/servlet/dafpipeline/ThreadUserBinderServlet
/atg/dynamo/servlet/dafpipeline/PathAuthenticationServlet
/atg/dynamo/servlet/dafpipeline/URLArgumentServlet
/atg/dynamo/servlet/dafpipeline/DynamoServlet
/atg/dynamo/servlet/dafpipeline/ProfileRequestServlet
/orion/locale/servlet/UpdateLocaleServlet
/orion/location/servlet/UpdateLocationServlet
/atg/dynamo/servlet/dafpipeline/ProfilePropertyServlet
/atg/dynamo/servlet/dafpipeline/SessionEventTrigger
/atg/dynamo/servlet/dafpipeline/SiteSessionEventTrigger
/atg/search/servlet/pipeline/SearchClickThroughServlet
/atg/dynamo/servlet/dafpipeline/PageViewServletTrigger
/atg/dynamo/servlet/dafpipeline/SessionSaverServlet
/atg/dynamo/servlet/dafpipeline/AccessControlServlet
/atg/dynamo/servlet/dafpipeline/PromotionServlet     
/atg/dynamo/servlet/dafpipeline/CommerceCommandServlet   
/atg/dynamo/servlet/pagecompile/DAFDropletEventServlet
/atg/dynamo/servlet/dafpipeline/MimeTyperServlet
/atg/dynamo/servlet/dafpipeline/ExpiredPasswordServlet
/atg/dynamo/servlet/dafpipeline/CookieBufferServlet
/foo/endeca/assembler/SearchkeywordRedirectPipeline                     - New Servlet
/atg/endeca/assembler/AssemblerPipelineServlet
/atg/dynamo/servlet/dafpipeline/MimeTypeDispatcher
    dynamo-internal/html : /atg/dynamo/servlet/dafpipeline/FileFinderServlet
/atg/dynamo/servlet/dafpipeline/TailPipelineServlet

DAF : Request-Handling Pipeline Servlets

You can use the Dynamo Server Admin Component Browser to view request handling pipeline servlets and their sequence within the pipeline:

Component path: /atg/dynamo/servlet/dafpipeline/DynamoHandler/

By default, PageFilter is mapped to handle JSP requests. When the application server invokes PageFilter, it checks the request and response for a reference to a Dynamo request and response pair. The pair does not exist, so PageFilter starts the DAF servlet pipeline by calling DynamoHandler, the first servlet in the pipeline.

/WEB-INF/web.xml



Servlet Pipeline(v11.1) - All pipeline servlets in order of execution.

           HTTP Request
                    |
                    |
                   V
/atg/dynamo/servlet/dafpipeline/DynamoHandler
/atg/multisite/SiteContextPipelineServlet
/atg/dynamo/servlet/dafpipeline/SecurityServlet
/atg/dynamo/servlet/dafpipeline/TransactionServlet
/atg/dynamo/servlet/dafpipeline/ThreadUserBinderServlet
/atg/dynamo/servlet/dafpipeline/PathAuthenticationServlet
/atg/dynamo/servlet/dafpipeline/URLArgumentServlet
/atg/dynamo/servlet/dafpipeline/DynamoServlet
/atg/dynamo/servlet/dafpipeline/ProfileRequestServlet
/orion/locale/servlet/UpdateLocaleServlet
/orion/location/servlet/UpdateLocationServlet
/atg/dynamo/servlet/dafpipeline/ProfilePropertyServlet
/atg/dynamo/servlet/dafpipeline/SessionEventTrigger
/atg/dynamo/servlet/dafpipeline/SiteSessionEventTrigger
/atg/search/servlet/pipeline/SearchClickThroughServlet
/atg/dynamo/servlet/dafpipeline/PageViewServletTrigger
/atg/dynamo/servlet/dafpipeline/SessionSaverServlet
/atg/dynamo/servlet/dafpipeline/AccessControlServlet
/atg/dynamo/servlet/dafpipeline/PromotionServlet                       — CORE COMMERCE
/atg/dynamo/servlet/dafpipeline/CommerceCommandServlet     — CORE COMMERCE
/atg/dynamo/servlet/pagecompile/DAFDropletEventServlet
/atg/dynamo/servlet/dafpipeline/MimeTyperServlet
/atg/dynamo/servlet/dafpipeline/ExpiredPasswordServlet
/atg/dynamo/servlet/dafpipeline/CookieBufferServlet
/atg/endeca/assembler/AssemblerPipelineServlet
/atg/dynamo/servlet/dafpipeline/MimeTypeDispatcher
    dynamo-internal/html : /atg/dynamo/servlet/dafpipeline/FileFinderServlet
/atg/dynamo/servlet/dafpipeline/TailPipelineServlet

How to load ATG component on Server startup?

When you start up an application, Nucleus reads the configuration path. The out-of-the-box DAS module has (/DAS/config/config.jar/Nucleus.properties) Nucleus.properties component.

Component path: /nucleus/
$class=atg.nucleus.Nucleus
# the default initially resolved service
initialServiceName=/Initial

The initialServiceName property instructs Nucleus to configure and start up its initial service using Initial.properties

Component path: /nucleus/Initial
Configuration: /nucleus/Initial/?propertyName=serviceConfiguration
$class=atg.nucleus.InitialService
initialServices=\
        atg/dynamo/service/PerformanceMonitor,\
/atg/Initial,\
VMSystem,\
/atg/dynamo/StartServers,\
        /atg/dynamo/server/ServerMonitor


In nucleus, to start any custom component at start up(when ATG server starts), Layer /nucleus/Initial.properties as below:

initialServices+=\
                /atg/search/CustomEndecaDimIdCacheService

ATG - How to enable priceLists to store prices?

Oracle Commerce users do not have a price list functionality available by default. By default prices are stored directly in the product catalog and OOTB configured using below component.

Component Path: /atg/commerce/pricing/ItemPricingEngine
Configuration:   /atg/commerce/pricing/ItemPricingEngine/?propertyName=serviceConfiguration

preCalculators=\
        calculators/ItemListPriceCalculator,\
        calculators/ItemSalePriceCalculator,\
        calculators/ConfigurableItemPriceCalculator



How to enable priceLists? 
To add price list functionality, configure the ItemPricingEngine to use the appropriate precalculator. 
Here is how you can override ItemPricingEngine component:
Component Path: /atg/commerce/pricing/ItemPricingEngine

preCalculators=\
    calculators/ItemPriceListCalculator,\
    calculators/ItemPriceListSaleCalculator,\
    calculators/ConfigurableItemPriceListCalculator,\
    calculators/ConfigurableItemPriceListSaleCalculator


This means the prices are retrieved from the PriceLists repository 
Price Lists created through BCC 
Price can be defined at Product, SKU or Product/SKU pairing

ATG User Profile Basics

OOTB ATG Manages user profiles with a SQL repository
SQL Repository Component : /atg/userprofiling/ProfileAdapterRepository
Definition File: /atg/userprofiling/userProfile.xml
Current user represented using Profile component (/atg/userprofiling/Profile) and exposes all the properties from the userProfile.xml

There are two types of users:
1) External Users

            Web Site registered users and are stored in ProfileAdapterRepository
2) Internal Users

        Company users who need access to BCC, CSC etc.. and are stored in InternalProfileRepository

Profile Repository definition file layering: (/atg/userprofiling/ProfileAdapterRepository/?propertyName=definitionFiles)

/DPS/config/profile.jar/atg/userprofiling/userProfile.xml
/DSS/config/config.jar/atg/userprofiling/userProfile.xml
/DCS/config/config.jar/atg/userprofiling/userProfile.xml
/APPLICATION_NAME/config/config.jar/atg/userprofiling/userProfile.xml

ProfileFormHanlder:
Component path: /atg/userprofiling/ProfileFormHandler/
This component helps to create, update, login, logout, change password etc…

Various methods available are:
  • handleCreate()
  • handleUpdate()
  • handleCancel()
  • handleDelete()
  • handleLogin()
  • handleLogout()
  • handleChangePassword()
How to create a new user?
Here is a code snippet:

 
 

How to check if user is logged in or Anonymous?
Using /atg/userprofiling/Profile.transient property. It returns false, if the Profile is saved. Otherwise true will be returned.

 

ATG Shopping Cart Basics

User can use Product Page or Cart page to perform below actions:
  1. Add Item(s) to the cart
  2. Remove Item from the cart
  3. Change quantity of item(s)
  4. Delete item(s) from the cart
How to add item(s) to the cart?
  • OOTB ATG provides request scope CartModifierFormHandler component with pre-build methods for add, update, delete etc..
  • Component path : /atg/commerce/order/purchase/CartModifierFormHandler


What happens when user adds item(s) to the cart?
  • When SKUs are added to the order, they become CommerceItem objects
  • CommerceItem Object represent each line item in the order
  • Order may contain many commerce items object
How to check the order object and commerce Items?
  •    ShoppingCart is session scoped component  (/atg/commerce/ShoppingCart)
  •    ShoppingCart uses OrderHolder Class to hold multiple orders
  •    At any given point only one Order will be active
  •    ShoppingCart holds current, saved and last orders
  •    Current Order object can be accessed by using either ShoppingCart.current or CartModifierFormHandler.order

ATG Component layering and how it works?

The config files (.properties) are layered in ATG. If the same .property files is present in modules which are dependent, then it will merge the property values.

Scenario #1:
Module A has X.properties with values name and age and depends on Module B which has X.properties with age and address.


When Module A component is involved it takes the values of age (overridden in A) and name from X.properties of A and address from X.properties of B.

Scenario #2:
Multi value property
#1    X.properties in the config of component contains

              vehicles=car, bike
        X.properties in the localconfig contains
              vehicles=bus
Then when this is used in the jsp then only bus is selected.

#2 To add all items, then, we need to use the below notation in the overridden file:
               vehicles+=bus

#3 If we need car and bus and bike to be removed then, we need to use the below notation in the overridden file:

              vehicles+=bus
              vehicles-=bike

ATG Order Reprice and How it works?

When Order reprice is typically called?
The Order is typically repriced after the following actions: 
  • Item is added to the cart 
  • Item is removed from the cart 
  • Quantity is changed on items in the cart 
  • User specifies a shipping address or shipping method and pushes "Checkout" button
How to do repricing on Order?
Use /atg/commerce/order/purchase/RepriceOrderDroplet to invoke repricing on order:



How it works behind the scenes?
Each time the code reprices the order, the ATG Promotions associated with the user's Profile are considered.
  • Fetch all of the Promotions/Discounts that the user is eligible for:
    • (Global + Individual Promotions) minus Promotions they don't qualify for
  • Calculate the price of each individual item on the order
    • Apply Item Level Discounts
  • Calculate the order total price(discounted item prices + shipping price)
    • Apply Order Level discounts
  • Calculate the shipping price
    • Apply Shipping Discounts
  • Generate a total price
Few scenarios when Order reprice is not working:
If you don't see your order getting repriced as you expected (the discount isn't being related), these are some things to check on:
  • Was the repricing code called yet?
  • Was the Promotion successfully added to the Profile?
  • Was the Promotion represented correctly in the repository?  
 

ATG - Promotions 101

What is Promotion?
A Promotion is an item created in the BCC that contains information about a possible discount that some (or all) users on your site may be able to get. Promotions specify a discount amount, and a conditional rule.

There are two types of Promotions:
 1) Global 

  • Automatically granted to all users
  •  If order qualifies, discount is given
 2) Individual 
  • Not automatically granted to all users
  • Something (typically a Scenario) has to grant the Promotion to the user before they can get the discount
What happens at User "Profile" component level?
  • When a user comes to your site, ATG will create a "Profile" component for this user. 
  • This ATG Profile component sticks around for the entire user session. It goes away when the user terminates their session. 
  • This Profile contains a property called "activePromotions" – this is a list of 0 or more Promotions that this user has been given. 
  • Each time your order is repriced, the code looks in Profile.activePromotions plus all Global Promotions to see which promotions the user is eligible for and reprices the order accordingly.

 How a user gets promotion(s)?
  •  If you create a Global Promotion, all users will automatically be given the Promotion (remember: they may not qualify – but it is always in their list of "possible promotions")
  •  If you create an Individual Promotion, the associated Promotion must first be "given" to the user. If the Individual Promotion is not in the Profile.activePromotions property, the user will not get the discount even if they qualify.
  • If a Scenario adds a Promotion to the user's Profile.activePromotions property, but the user doesn't actually qualify for it, a discount will not be given when the order is repriced
There are a number of ways Individual Promotions can be granted to the user. 
Commonly used one's are:
  1. Scenarios   -  Create a Scenario such as: If order.total is >= 50$, give user promotion: 10Off50, or if user logs in, and Profile.age > 62 and today is Wednesday, then give user a 20SeniorWednesday promotion 
  2. Coupons - User enters a coupon code associated with Promotion 10Off50. The Promotion is added to Profile.activePromotions

ATG Repository queries using dyn/admin

Product Catalog
Component: /atg/commerce/catalog/ProductCatalog/
 

Profile Adapter Repository
Component: /atg/userprofiling/ProfileAdapterRepository/
 


Order Repository
Component:  /atg/commerce/order/OrderRepository/
 


ATG Component scope

Component Scope:
1) global
2) session
3) request

global can access only global
session can access session and global
request can access request, session and global

Global:

$scope=global
- Default scope 
- Its instance is shared by users across all the sessions that exist for the application (Not similar to application scope of J2EE as global scope components are accessible across application but across ATG server instances)
- Any user which accesses the application can change the values of a global scoped component

Session:
$scope=session
- A separate instance is provided for each session or user
- Session by default valid for 30 mins


How to track and manage user sessions?
Use component:  /atg/dynamo/servlet/sessiontracking


 

Ex: OOTB Session based component /atg/userprofiling/Profile

Request:

$scope=request
 - Each request gets own copy of component
- A new instance will be generated for the component
- Formhandler is always request scoped

Single Sign-on : ATG and Endeca

Single sign-on (SSO) is a session and user authentication service that permits a user to use one set of login credentials (e.g., name and password) to access multiple applications. 

In Oracle 11.x, SSO is provided as an alternative to third-party SSO solutions. It integrates authentication for Endeca Workbench and the ATG Business Control Center, allowing a user to switch between tools without encountering additional login screens.

There will be dedicated ATG server instance where SSO and DPS.InternalUsersmodules included.

SSO module includes a web application that manages the single-sign on process. The Commerce SSO makes use of ticket granting tickets and service tickets algorithm.

In ATG pipeline, /atg/sso/servlet/SSODispatcherServlet component is added to dispatch requests to other servlets that provide the five SSO server functions

Here are five Functions:
1) login  - Manages the process of authenticating the user and issuing a service ticket


2) validate - Manages the process of validating requests based on the status of service tickets

3) keepAlive - Ensures that an SSO session remains active as long as there is activity in either the Business Control Center or the Workbench


4) control  - Handles configuration of the client logout URL (Only accessed by Endeca plug-in)

5) logout - Manages the process of deleting any tickets associated with the session and then redirecting to the login page


How to Integrate Workbench with Commerce SSO?
1) Go to webstudio.properties  

/opt/app/endeca/ToolsAndFrameworks//server/workspace/conf

2) Set com.endeca.webstudio.useSSO to true  

 com.endeca.webstudio.useSSO=true
 

3) Uncomment following properties and update SSO box host and port details

#com.endeca.webstudio.sso.loginURL=http://<host:port>/login
#com.endeca.webstudio.sso.controlURL=http://
<host:port>/control
#com.endeca.webstudio.sso.logoutURL=http://
<host:port>/logout
#com.endeca.webstudio.sso.validationURL=http://
<host:port>/validate
#com.endeca.webstudio.sso.keepAliveURL=http://
<host:port>/keepAlive
#com.endeca.webstudio.sso.keepAliveFrequency=1800

The Commerce SSO server uses ATG internal profile repository and does not automatically publish changes to the Endeca Workbench. Make sure each ATG user profile must have a corresponding workbench user with the same name.

How record sorting and relevance ranking works?

There are three ways of controlling the order in which records are returned:
1) Explicit sort key using Ns parameter
2) Default sort using Dgidx flag
3) Relevancy Ranking

The priority of record sorting/relevance ranking is as follows:

1) If none of the above  sorting methods is specified then Endeca uses order of internal ID generated by Dgidx during indexing

2) If Sort key is used(Ns parameter) then sort key overrides all other sorting and relevance ranking settings

3) If a default sort key is specified then records are returned in default sort order(Ties are broken using the internal order by Dgidx)

4) When searching against a search interface relevance ranking strategy is used(ties are broken using the default sort key and if no sort key then ties are broken using the internal order by Dgidx)

5) MDEX Engine query with relevance ranking parameter (overrides any relevance ranking strategies configured in the search interface)

Endeca Pipeline : Record Cache

In Endeca, all data sources feeding a join must be record cached. 

There are two exceptions: 
1) Switch join -  It does not do record comparisons and do not require record caches for their data sources
2) Left join - All of the right sources are record caches and hence the left source does not require a record cache

How to start Endeca HTTP service and Endeca Tools Service?

Windows:
Go to Start > Control Panel > Administrative Tools > Services

Select the Endeca HTTP service and the Endeca Tools Service and click Start.  

On UNIX: 
To start the Endeca HTTP service, run  
           $ENDECA_ROOT/tools/server/bin/startup.sh


To start the Endeca Tools Service, run  
           $ENDECA_TOOLS_ROOT/server/bin/startup.sh
 

Endeca Tips for troubleshooting long processing time - Core Endeca

         Is the long processing time for the Engine caused by limitations of hardware resources? 
o Identify whether long query time is caused by CPU, memory, or disk I/O utilization.

          Is a high number of records being returned by the MDEX Engine?
o Identify how many records are being returned per query by looking for large nbins values in queries as reported by Cheetah(Performance Monitoring Tool).  If this number is high, this can be expensive to compute and affects performance.

          Are all dimension refinements (dimension values) exposed for navigation? 
o Identify whether all dimension refinements are exposed by looking forallgroups=1 in the Dgraph request log (request URL parameter) or in Cheetah reports 

          Are your longest queries similar?
o Check the longest queries for similarities, such as whether they all use the same search interface with relevance ranking, wildcard search, or record filters.

          Is record search being used?
o Identify whether a record search is being used by any queries by looking for “attrs=search_interface_name” in a query. This indicates that a record search is being used which means that possibly expensive relevance ranking modules can be contributing to high computation time

          Which relevance ranking strategies are being used?
o Check the CRS.relrank_strategies.xml file for the presence of Exact, Phrase and Proximity ranking modules and test the same query with these modules removed.

          Is sorting enabled for properties or dimensions?
o Identify whether sorting with sort keys is enabled, for which properties and dimensions it is being used and whether it is needed. 
o The first time a sort key is issued to a Dgraph after startup the key must be computed which can slow down performance. To isolate this problem, test the query in the staging environment by removing the sort key. 
o If you confirm sort keys are the issue, consider using sort keys in a representative batch of queries used to warm up the Dgraph after startup. 
o The sorts will become cached and these queries will be faster. 

Versions of Endeca Commerce releases and Comparison chart

Oracle Endeca Commerce architectural comparison over multiple versions

How to Handle Special Characters in Search? - Endeca Best Practices

Endeca uses only alphanumeric characters to be searchable by default as part of Endeca search query.

Problem Statement:
If any search term contains any special characters then Endeca replaces special characters by "space" and makes endeca query. This may not give intended results.

Ex: If a search term is β Antibody, Endeca internally replaces search term special chars with space and search term would be Antibody".

Solution:
For Forge-based Application:
1. Open  /opt/app/endeca/apps/CRS/config/pipeline/CRS.search_chars.xml

2.  Add all characters that you want to make searchable.
3.  Run baseline
update 


For Forge-less  Application:
1. Open  /opt/app/endeca/apps/CRS/config/mdex/CRS.search_chars.xml
2.  Add all characters that you want to make searchable.
3.  Run baseline
update


Pipeline Changes Looks like as below: