Thursday, June 22, 2017

Elasticsearch: Extending ES using AOP

Elasticsearch provides the ability for extending the basic functionality using scripts or plug-ins. But at the same time ES has restrictions on changing or extending existing functionality like search actions etc. For example, say, I need to change the result set for every query or change the input search criteria etc. In such a case, we can use AOP to extend ES functionality.
AOP is a vast topic in itself so I will not cover it here. Basically, it allows to control what happens before a method execution, after a method execution, change the original input parameters, change the return values etc.
For this post we will try to monitor the ES search parameters using AOP. We will use AspectJ programming using maven project as follows:
  • Create the maven project
  • Find out the ES methods to be monitored
  • Define the Pointcuts & Advices
  • Compile the jar
  • Use Load Time Weaving
  • Start ES and monitor the queries

Source code
The source code for this example is located here.

Create the maven project
Create a simple  maven jar project with following POM xml.
  • Add dependencies for aspectjrt line 13-17 and aspectjweaver line 18-22
  • Add aspectj-maven-plugin line 31-48

ES methods to monitor
We will monitor the searches by ES. The actual search execution is done by the SearchService. We will monitor the executeSearch() and executeFetch() methods on this class.

Define Pointcuts & Advices
The SearchServiceAspect class defines the Pointcuts & Advices using annotations. Following is the definition to monitor the SearchService.executeSearch() method:
  • Line 5 defines the Around advice.
  • The advice is for ES method SearchService.executeQueryPhase() which takes two parameters: ShardSearchTransportRequest & SearchTask
  • It also mentions that these two should be passed to our method which will handle the execution.
  • Line 6 is the method in our class which will be executed when SearchService.executeQueryPhase() is to be executed.
  • In this method we can decide what to do i.e. do some processing or continue execution using joinPoint.proceeed() etc.
  • For this example, we are just logging the search query information.

Next we define the aop.xml which acts as input to the AOP compiler.
  • Line 4 declares that our aspect pointcuts & advices are in class my.elasticsearch.aspects.SearchServiceAspect
  • Line 6 tells the weaver to be more verbose to help troubleshooting the issues if any.
  • Line 7 tells the weaver which class in the target application is to be woven.
  • The aop.xml needs to be present at src/main/resources/META-INF in the project.

Deploying the jars
  • Compile the project and generate the jar using command: mvn install
  • Copy the generated es5x-method-interceptor-0.0.1-SNAPSHOT.jar into the elasticsearch/lib folder as per the ES installation.

Configure ES startup
Now, we need to start ES using the aspect weaver as java agent. 
  • Copy the aspectjweaver-1.8.9.jar into elasticsearch/bin folder.
  • Open the bin/elasticsearch.bat or bin/ file (I have verified with Windows batch file)
  • Add following line before the java command execution to launch the ES
             SET ES_JAVA_OPTS=%ES_JAVA_OPTS% -javaagent:aspectjweaver-1.8.9.jar
  • With above change we are basically passing two additional command line parameters while start the ES:
    • javaagent: Using the aspectj weaver as java agent.
    • ES 5x uses java security manager and hence our aspect weaver won't work. Using the above parameter we are asking the java security manager to grant the required permissions. The content of enable_aspectj_classes.policy is as follows:
                    grant {
          • We are granting all permissions to the application. It is recommended that you find out granular permissions by enabling the logging at access level and update the above file as required.
          • The file is placed in elasticsearch/bin folder.
      The logs from weaver won't be included in the ES logs. So you can redirect the ES command line output (if not running as service/daemon) to a file as follows:
      elasticsearch.bat > c:\es5x.log 2>&1
      You should see following logs:
      Line 18 & 19 indicates that out pointcuts & advices have been woven.

      Populating test data
      • We will create two indices as aop_test_01 & aop_test_02 having file type with following data:
              {"name": "file_01", "size": 5678}
      • Populate as many entries as you want.
      • Make sure both indices have some data

      Querying the data
      Lets issue a simple search snapping both the indices as:
      You should see that our advice is getting executed from the following logs:
      • Since we are querying multiple indices resulting in multiple shards query we see both executeQueryPhase() & executeFetchPhase() advice getting called.
      • In executeQueryPhase() we are just logging the indices. We can log additional info like the query to be executed etc.
      • In the executeFetchPhase() we are also logging the docIds which the search returns.
      • You can read more on how search gets executed here.

      With this simple POC we have hooked our custom code into ES which will allow use to customize ES behavior as needed. Currently, we have just logged the input requests but can do more like change the input/output etc.


      1. Replies
        1. A IEEE project is an interrelated arrangement of exercises, having a positive beginning and end point and bringing about an interesting result in Engineering Colleges for a particular asset assignment working under a triple limitation - time, cost and execution. Final Year Project Domains for CSE In Engineering Colleges, final year IEEE Project Management requires the utilization of abilities and information to arrange, plan, plan, direct, control, screen, and assess a final year project for cse. The utilization of Project Management to accomplish authoritative objectives has expanded quickly and many engineering colleges have reacted with final year IEEE projects Project Centers in Chennai for CSE to help students in learning these remarkable abilities.

          Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining the authors explore the idea of using Java in Big Data platforms.
          Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai


      2. You write this post very carefully I think, which is easily understandable to me. Not only this, but another post is also good. As a newbie, this info is really helpful for me. Thanks to you.
        Tally ERP 9 Training
        tally classes
        Tally Training institute in Chennai
        Tally course in Chennai

      3. I feel satisfied to read your blog, you have been delivering a useful & unique information to our vision.keep blogging.
        Digital Marketing Course In Kolkata
        Web Design Course In Kolkata

      4. Good post. I learn something new and challenging on sites I stumbleupon on a daily basis. It's always interesting to read content from other writers and practice a little something from their web sites
        angular js training in chennai

        angular js online training in chennai

        angular js training in bangalore

        angular js training in hyderabad

        angular js training in coimbatore

        angular js training

        angular js online training