Head first elastic search on java with spring boot and data features
In this article I’ll try to give you an easy introduction on how to use Elastic Search in a Java project. As Spring Boot is the easiest and fastest way to begin our project I choose to use it. Futhermore, we will heavily use Repository goodies of beloved Spring Data.
Let’s begin by installing Elastic Search on our machine and run our elastic server for the first time.
I go to elastic-folder\bin and run elasticsearch.bat (yeah I’m using Windows) but no luck. I get this:
“Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.”
What a great start!
In my bin folder there’s a “elasticsearch.in.bat” file. I set ES_MAX_MEM=1g to ES_MAX_MEM=512mb and voila it is fixed.
I start a new server without problem after that.
Now it is time to define the document we will index in elastic search. Assume we have movie information to index. Our model is quite straightforward. Movie has a name, rating and a genre in it. I chose “elastic_sample” as index name which sounds good as a database name and “movie” as type which is good for a table name if we think in relational database terms. Nothing fancy in the model as you can see.
@Document(indexName = "elastic_sample", type = "movie") public class Movie { @Id private String id; private String name; @Field(type = FieldType.Nested) private List < Genre > genre; private Double rating; public Double getRating() { return rating; } public void setRating(Double rating) { this.rating = rating; } public void setId(String id) { this.id = id; } public List < Genre > getGenre() { return genre; } public void setGenre(List < Genre > genre) { this.genre = genre; } public String getId() { return id; } public String getName() { return name; } public void setName(String name) { this.name = name; } @Override public String toString() { return "Movie{" + "id=" + id + ", name='" + name + '\'' + ", genre=" + genre + ", rating=" + rating + '}'; } }
For those who wonder what Genre is here is it. Just a POJO.
public class Genre { private String name; public Genre() { } public Genre(String name) { this.name = name; } public String getName() { return name; } @Override public String toString() { return "Genre{" + "name='" + name + '\'' + '}'; } public void setName(String name) { this.name = name; } }
Not it is time to create DAO layer so we can save and load our document to/from our elastic search server. Our Repository extends the classic ElasticserchRepository (no idea why it is search and not Search). As you probably know Spring Data can query one or more fields with these predefined methods where we use our field names. findByName will search in the name field, findByRating will search in the rating field so on so forth. Furthermore thanks to Spring Data we don’t need to write implementation for it, we just put method names in the interface and that’s finished.
public interface MovieRepository extends ElasticsearchRepository < Movie, Long > { public List < Movie > findByName(String name); public List < Movie> findByRatingBetween(Double beginning, Double end); }
Our DAO layer will be called by a Service layer:
@Service public class MovieService { @Autowired private MovieRepository repository; public List < Movie > getByName(String name) { return repository.findByName(name); } public List < Movie > getByRatingInterval(Double beginning, Double end) { return repository.findByRatingBetween(beginning, end); } public void addMovie(Movie movie) { repository.save(movie); } }
Here is the main Class we will use to run our application. EnableAutoConfiguration will auto-configure everything it recognizes under our classpath. ComponentScan will scan for Spring annotations under the main Class’ directory.
@Configuration @EnableAutoConfiguration @ComponentScan public class BootElastic implements CommandLineRunner { @Autowired private MovieService movieService; private static final Logger logger = LoggerFactory.getLogger(BootElastic.class); // add star wars and // princess bride as a movie // to elastic search private void addSomeMovies() { Movie starWars = getFirstMovie(); movieService.addMovie(starWars); Movie princessBride = getSecondMovie(); movieService.addMovie(princessBride); } private Movie getSecondMovie() { Movie secondMovie = new Movie(); secondMovie.setId("2"); secondMovie.setRating(8.4d); secondMovie.setName("The Princess Bride"); List < Genre > princessPrideGenre = new ArrayList < Genre >(); princessPrideGenre.add(new Genre("ACTION")); princessPrideGenre.add(new Genre("ROMANCE")); secondMovie.setGenre(princessPrideGenre); return secondMovie; } private Movie getFirstMovie() { Movie firstMovie = new Movie(); firstMovie.setId("1"); firstMovie.setRating(9.6d); firstMovie.setName("Star Wars"); List < Genre > starWarsGenre = new ArrayList < Genre >(); starWarsGenre.add(new Genre("ACTION")); starWarsGenre.add(new Genre("SCI_FI")); firstMovie.setGenre(starWarsGenre); return firstMovie; } public void run(String... args) throws Exception { addSomeMovies(); // We indexed star wars and pricess bride to our movie // listing in elastic search //Lets query if we have a movie with Star Wars as name List < Movie > starWarsNameQuery = movieService.getByName("Star Wars"); logger.info("Content of star wars name query is {}", starWarsNameQuery); //Lets query if we have a movie with The Princess Bride as name List < Movie > brideQuery = movieService.getByName("The Princess Bride"); logger.info("Content of princess bride name query is {}", brideQuery); //Lets query if we have a movie with rating between 6 and 9 List < Movie > byRatingInterval = movieService.getByRatingInterval(6d, 9d); logger.info("Content of Rating Interval query is {}", byRatingInterval); } public static void main(String[] args) throws Exception { SpringApplication.run(BootElastic.class, args); } }
If we run it the result is:
015-02-28 18:26:12.368 INFO 3616 --- [ main] main.BootElastic: Content of star wars name query is [Movie{id=1, name='Star Wars', genre=[Genre{name='ACTION'}, Genre{name='SCI_FI'}], rating=9.6}] 2015-02-28 18:26:12.373 INFO 3616 --- [ main] main.BootElastic: Content of princess bride name query is [Movie{id=2, name='The Princess Bride', genre=[Genre{name='ACTION'}, Genre{name='ROMANCE'}], rating=8.4}] 2015-02-28 18:26:12.384 INFO 3616 --- [ main] main.BootElastic: Content of Rating Interval query is [Movie{id=2, name='The Princess Bride', genre=[Genre{name='ACTION'}, Genre{name='ROMANCE'}], rating=8.4}]
As you can see the interval query only retrieved Princess Bride. We did not do any configuration right? It is unusual. I have to share the huge configuration file with you:
spring.data.elasticsearch.cluster-nodes=localhost:9300 # if spring data repository support is enabled spring.data.elasticsearch.repositories.enabled=true
Normally you would use port 9200 when you query your elastic server. But when we programmatically reach it we are using 9300. If you have more than one node you would separate them with a comma and use 9301, 9302 etc as port numbers. Our pom file is no surprise either. Just elastic starter pom and we are set to go.
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>caught.co.nr</groupid> <artifactid>boot-elastic-sample</artifactid> <version>1.0-SNAPSHOT</version> <packaging>war</packaging> <!-- Inherit defaults from Spring Boot --> <parent> <groupid>org.springframework.boot</groupid> <artifactid>spring-boot-starter-parent</artifactid> <version>1.2.2.RELEASE</version> </parent> <dependencies> <dependency> <groupid>org.springframework.boot</groupid> <artifactid>spring-boot-starter-data-elasticsearch</artifactid> </dependency> </dependencies> <!-- Needed for fat jar --> <build> <plugins> <plugin> <groupid>org.springframework.boot</groupid> <artifactid>spring-boot-maven-plugin</artifactid> </plugin> </plugins> </build> </project>
As you can see thanks to Spring Boot and Data it is quite easy to work with elastic search. Lets check what we indexed from the server api as well. I’ll use Sense -a chrome plug-in for elastic commands-.
Here’s the result json:
{ "took": 2, "timed_out": false, "_shards": { "total": 1, "successful": 1, "failed": 0 }, "hits": { "total": 2, "max_score": 1, "hits": [ { "_index": "elastic_sample", "_type": "movie", "_id": "1", "_score": 1, "_source": { "id": 1, "name": "Star Wars", "genre": [ { "name": "ACTION" }, { "name": "SCI_FI" } ] } }, { "_index": "elastic_sample", "_type": "movie", "_id": "2", "_score": 1, "_source": { "id": 2, "name": "The Princess Bride", "genre": [ { "name": "ACTION" }, { "name": "ROMANCE" } ] } } ] } }
- You can check out the whole project in the github.
Reference: | Head first elastic search on java with spring boot and data features from our JCG partner Sezin Karli at the caught Somewhere In Time = true; blog. |
great help. thanks!
Thanks a lot!, this was very helpful
Where is save() method in DAO layer in this project?
addMovie(Movie movie) in MovieService is where you save the data
i am getting
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/127.0.0.1:9200]][cluster/state] request_id [0] timed out after [5001ms]
error
Make sure you have this in your properties
spring.data.elasticsearch.cluster-nodes=localhost:9300
You must use 9300 as port not 9200.
Furthermore, try to fetch data directly from your server with Sense plugin or some other elastic client. I doubt you’ll get something from there as well. Root cause can be a firewall issue.
the movie is indexed but is not saved on dataBase is that normal?
No it is not Achraf. Do you get some kind of exception?
No exception, the difference is that i’m using a transport client just like that
cluster-nodes=”localhost:9300″ cluster-name=”elasticsearch” />
Nice artical, Thank you so much !!!!
I see that @Document,@Id and @Filed annotations are not being recognized by the IDE. What dependency do i need to add for this. Can you please help me with this.
The pom xml I provided should be enough. I suggest that you build the project again and reload it in your ide.
If you use intellij idea, do maven projects>reimport all maven projects.
Hi, thank you for this tutorial, actually, i’m learning spring data elasticsearch and i have a very urgent use case on ES Fuzzy queries, especially on how to set the fuzziness level to be a maximum level to match a very large number of possible documents. Do you have any idea about how to set the fuzziness level to a max level?
Thank you very much.
I am using elastic search 2.3.1 and running the above code. It’s not working.
I haven’t even been able to run the above code, neither after building the project myself nor downloading your version straight from your github. One of the problems I’m facing is that my IDE doesn’t recognize any of the Elasticsearch-related imports and cannot seem to resolve any of these terms. Additionally, when I built the project myself I tried to run the command: mvn package But it returned over a dozen errors related to the POM being malformed. Can you please clear up either of these issues? I’m new to Maven projects and Spring Data, but thought this project would… Read more »
Update:
I was able to run this project after importing it into IntelliJ with the correct Maven settings. However, it’s failing later on during run-time primarily because the node at port 9300. Please help.
I downloaded the project from your github but it cannot seem to find the configured nodes to be available. In fact, it’s not even finding the values that reside in the application.properties file. Is there something I’m missing in the path where this file has to be enabled, or something similar?
Can you add code for search by Genre?
can you please post the code
Same for getting all data…i.e getAll()
With elasticsearch-5.5.1, the application fails with the following errors:
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:1.2.2.RELEASE:run (default-cli) on project boot-elastic-sample: An exception occured while running. null: InvocationTargetException: Failed to execute CommandLineRunner: None of the configured nodes were available: [[#transport#-1][mh-Lenovo][inet[localhost/127.0.0.1:9300]]]: [][inet[localhost/127.0.0.1:9300]][index]: [][inet[localhost/127.0.0.1:9300]] Node not connected -> [Help 1]
In the elasticsearch console I got this:
[2017-08-15T15:24:15,235][WARN ][o.e.t.n.Netty4Transport ] [7CahJCt] exception caught on transport layer [[id: 0xcf7cc05d, L:/127.0.0.1:9300 – R:/127.0.0.1:48994]], closing connection
java.lang.IllegalStateException: Received message from unsupported version: [1.3.2] minimal compatible version is: [5.0.0]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1379) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-5.5.1.jar:5.5.1]
Looks like spring-boot-starter-data-elasticsearch is not compatible with latest version of elasticsearch.
Hi,
I use spring-boot-starter-data-elasticsearch dependency in my Java spring project to query ES 2.4.0
My json’s look like this,
{
“event”:{
“eventName”:”Man U vs Arsenal”,
“eventDateTime”:”Mon 24 Oct 12:00″
}
}
To avoid having to declare an empty “event” POJO wrapper, I use @JsonRootName(value=”event”) to mark that there’s a root element in my POJO and I have also specified the below property in my application.properties,
spring.jackson.deserialization.UNWRAP_ROOT_VALUE=true
The search is happening fine but the root element declaration does not take effect and results in objects with just nulls.
Is there a way to achieve this at all?
Thanks,
Arun