Elasticsearch provides the ability for extending the basic functionality using scripts or plug-ins. But at the same time ES has restrictions on changing or extending existing functionality like search actions etc. For example, say, I need to change the result set for every query or change the input search criteria etc. In such a case, we can use AOP to extend ES functionality.
AOP is a vast topic in itself so I will not cover it here. Basically, it allows to control what happens before a method execution, after a method execution, change the original input parameters, change the return values etc.
For this post we will try to monitor the ES search parameters using AOP. We will use AspectJ programming using maven project as follows:
Source code
The source code for this example is located here.
Create the maven project
Create a simple maven jar project with following POM xml.
ES methods to monitor
We will monitor the searches by ES. The actual search execution is done by the SearchService. We will monitor the executeSearch() and executeFetch() methods on this class.
Define Pointcuts & Advices
The SearchServiceAspect class defines the Pointcuts & Advices using annotations. Following is the definition to monitor the SearchService.executeSearch() method:
aop.xml
Populating test data
Summary
With this simple POC we have hooked our custom code into ES which will allow use to customize ES behavior as needed. Currently, we have just logged the input requests but can do more like change the input/output etc.
AOP is a vast topic in itself so I will not cover it here. Basically, it allows to control what happens before a method execution, after a method execution, change the original input parameters, change the return values etc.
For this post we will try to monitor the ES search parameters using AOP. We will use AspectJ programming using maven project as follows:
- Create the maven project
- Find out the ES methods to be monitored
- Define the Pointcuts & Advices
- Compile the jar
- Use Load Time Weaving
- Start ES and monitor the queries
Source code
The source code for this example is located here.
Create the maven project
Create a simple maven jar project with following POM xml.
- Add dependencies for aspectjrt line 13-17 and aspectjweaver line 18-22
- Add aspectj-maven-plugin line 31-48
ES methods to monitor
We will monitor the searches by ES. The actual search execution is done by the SearchService. We will monitor the executeSearch() and executeFetch() methods on this class.
Define Pointcuts & Advices
The SearchServiceAspect class defines the Pointcuts & Advices using annotations. Following is the definition to monitor the SearchService.executeSearch() method:
- Line 5 defines the Around advice.
- The advice is for ES method SearchService.executeQueryPhase() which takes two parameters: ShardSearchTransportRequest & SearchTask
- It also mentions that these two should be passed to our method which will handle the execution.
- Line 6 is the method in our class which will be executed when SearchService.executeQueryPhase() is to be executed.
- In this method we can decide what to do i.e. do some processing or continue execution using joinPoint.proceeed() etc.
- For this example, we are just logging the search query information.
aop.xml
Next we define the aop.xml which acts as input to the AOP compiler.
- Line 4 declares that our aspect pointcuts & advices are in class my.elasticsearch.aspects.SearchServiceAspect
- Line 6 tells the weaver to be more verbose to help troubleshooting the issues if any.
- Line 7 tells the weaver which class in the target application is to be woven.
- The aop.xml needs to be present at src/main/resources/META-INF in the project.
Deploying the jars
- Compile the project and generate the jar using command: mvn install
- Copy the generated es5x-method-interceptor-0.0.1-SNAPSHOT.jar into the elasticsearch/lib folder as per the ES installation.
Configure ES startup
Now, we need to start ES using the aspect weaver as java agent.
- Copy the aspectjweaver-1.8.9.jar into elasticsearch/bin folder.
- Open the bin/elasticsearch.bat or bin/elasticsearch.sh file (I have verified with Windows batch file)
- Add following line before the java command execution to launch the ES
- With above change we are basically passing two additional command line parameters while start the ES:
- javaagent: Using the aspectj weaver as java agent.
- java.security.policy: ES 5x uses java security manager and hence our aspect weaver won't work. Using the above parameter we are asking the java security manager to grant the required permissions. The content of enable_aspectj_classes.policy is as follows:
- We are granting all permissions to the application. It is recommended that you find out granular permissions by enabling the logging at access level and update the above file as required.
- The file is placed in elasticsearch/bin folder.
The logs from weaver won't be included in the ES logs. So you can redirect the ES command line output (if not running as service/daemon) to a file as follows:
elasticsearch.bat > c:\es5x.log 2>&1
You should see following logs:
Line 18 & 19 indicates that out pointcuts & advices have been woven.
Populating test data
- We will create two indices as aop_test_01 & aop_test_02 having file type with following data:
- Populate as many entries as you want.
- Make sure both indices have some data
Querying the data
Lets issue a simple search snapping both the indices as:
http://localhost:9200/aop_test_*/_search
You should see that our advice is getting executed from the following logs:
- Since we are querying multiple indices resulting in multiple shards query we see both executeQueryPhase() & executeFetchPhase() advice getting called.
- In executeQueryPhase() we are just logging the indices. We can log additional info like the query to be executed etc.
- In the executeFetchPhase() we are also logging the docIds which the search returns.
- You can read more on how search gets executed here.
Summary
With this simple POC we have hooked our custom code into ES which will allow use to customize ES behavior as needed. Currently, we have just logged the input requests but can do more like change the input/output etc.
First of all thanks for the useful info. It's very useful for my research.
ReplyDeleteSAS Training in Chennai
sas training fees in chennai
SAS Training in Adyar
SAS Training in Velachery
SAS Training in Tambaram
clinical sas training in chennai
Placement Training in Chennai
soft skills training institutes in chennai
A IEEE project is an interrelated arrangement of exercises, having a positive beginning and end point and bringing about an interesting result in Engineering Colleges for a particular asset assignment working under a triple limitation - time, cost and execution. Final Year Project Domains for CSE In Engineering Colleges, final year IEEE Project Management requires the utilization of abilities and information to arrange, plan, plan, direct, control, screen, and assess a final year project for cse. The utilization of Project Management to accomplish authoritative objectives has expanded quickly and many engineering colleges have reacted with final year IEEE projects Project Centers in Chennai for CSE to help students in learning these remarkable abilities.
DeleteSpring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining the authors explore the idea of using Java in Big Data platforms.
Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai
I really admired with your post, do share more updates.
ReplyDeleteTally Training in Chennai
Tally course
Tally institute in Chennai
AngularJS course in Chennai
ccna Training in Chennai
PHP course in Chennai
Salesforce course in Chennai
Web Designing Training in Chennai
Tally Course in Velachery
ReplyDeleteYou write this post very carefully I think, which is easily understandable to me. Not only this, but another post is also good. As a newbie, this info is really helpful for me. Thanks to you.
Tally ERP 9 Training
tally classes
Tally Training institute in Chennai
Tally course in Chennai
I feel satisfied to read your blog, you have been delivering a useful & unique information to our vision.keep blogging.
ReplyDeleteDigital Marketing Course In Kolkata
Web Design Course In Kolkata
TITIAN MOUNTAIN BUNDLE RACING POTTER
ReplyDeleteThese bikes come in 2 variants. titanium hair straightener The standard standard can titanium rings be resized 2-pole 인터넷 바카라 design is also titanium armor well crafted. Each variation is designed with where can i buy titanium trim
Internet Download manager Serial keys are below.IDM Crack is the Best Software till now for downloading videos, files, Songs, Movies With very Less time. IDM Full Version Download
ReplyDeleteWhen you will grow and see the photographs of your second birthday, then you will realize how happy we are right now. My Beautiful Princess Quotes
ReplyDeleteThese bonuses allow you to play at no cost with out risking your individual cash. With these bonuses, you'll be able to|you presumably can} play at no 온라인 카지노 cost with out risking your individual cash. It is crucial to learn the casino’s phrases and conditions rigorously for a specific bonus to understand the wagering requirement. We are all about entertainment and having fun in a secure and responsible gaming surroundings.
ReplyDelete