Apache Nutch is an open source framework written in Java. Its purpose is to help us crawl a set of websites (or the entire Internet), fetch the content, and prepare it for indexing by, say, Solr. A pretty useful framework if you ask me, however it is designed to be used onlymostly from the command line. You download the archive, unzip it, and run the binary file. It crawls and you get the data. However, I’ve got a project where this crawling had to be embedded into my own Java app. I realized that there is a complete absence of any documentation for that. Hence this blog post. It explains how you can use Nutch from Java, not from the command line.
I’ll be talking about Nutch 1.15. There is a later version 2+, but I didn’t manage to make it work. If you know how, leave your comment below.
I’d recommend you read this tutorial first, to understand how Nutch works from the command line. Well, it helped me anyway.
Now, let’s see how we can use Nutch without the command line. First, you need these dependencies in your pom.xml (Nutch uses Apache Hadoop, that’s why we need the second dependency):
<project>
<dependencies>
<dependency>
<groupId>org.apache.nutch</groupId>
<artifactId>nutch</artifactId>
<version>1.15</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.2</version>
</dependency>
[...]
</dependencies>
[...]
</project>Next, this is your Java code, which does all the work:
001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 096 097 098 099 100 101 102 103 104 105 106 | import java.io.IOException;import java.nio.file.Files;import java.nio.file.Paths;import java.util.Collection;import java.util.Collections;import java.util.Comparator;import java.util.List;import java.util.stream.Collectors;import org.apache.commons.io.FileUtils;import org.apache.hadoop.conf.Configuration;import org.apache.nutch.crawl.CrawlDb;import org.apache.nutch.crawl.Generator;import org.apache.nutch.crawl.Injector;import org.apache.nutch.fetcher.Fetcher;import org.apache.nutch.parse.ParseSegment;import org.apache.nutch.tools.FileDumper;public class Main { public static void main(String... args) throws Exception { // Create a default configuration object, which will read // the content of nutch-default.xml file from the classpath, // parse it and take its entire content as the default // configuration. Funny, but this interface is from Hadoop: Configuration conf = NutchConfiguration.create(); // Now it's possible to reset some configuration parameters // by using this set() method. This one is mandatory, if you // don't set it the crawling won't work. The value is used // as User-Agent HTTP header. conf.set("http.agent.name", "me, myself, and I"); // This one is also mandatory and we will discuss its // value below. You need to point Nutch to the directory // with compiled plugins and this collection is NOT in // its default JAR package, unfortunately. conf.set("plugin.folders", System.getProperty("nutch.plugins.dir")); // First, we need to have a directory where everything will // happen. I assume you are familiar with Maven, so let's use // its default temporary directory "target": Path home = new Path("target"); // Next, we have to create a file with a list of URLs Nutch will // start crawling from: final Path targets = new Path(home, "urls"); Files.createDirectory(Paths.get(targets.toString())); Files.write( Paths.get(targets.toString(), "list-of-urls.txt"), String.join("\n", urls).getBytes() ); // Next, we have to point Nutch to the directory with the // text file and let it "inject" our URLs into its database: new Injector(conf).inject( new Path(home, "crawldb"), // the directory with its database new Path(home, "urls"), // the directory with text files with URLs true, true // no idea what this is ); // Now, it's time to do a few cycles of fetching, parsing, and // updating. This is how Nutch works, in increments. Each increment // will bring new web pages to the database. The more increments // you run, the deeper Nutch will go into the Internet. Five here // is a very small number. If you really want to crawl deeper, // you will need hundreds of increments. I guess, anyway. I haven't tried it. for (int idx = 0; idx < 5; ++idx) { this.cycle(home, conf); } // Now it's time to dump what is fetched to a new directory, // which will contain HTML pages and all other files when // finished. Files.createDirectory(Paths.get(new Path(home, "dump").toString())); new FileDumper().dump( new File(new Path(home, "dump").toString()), // where to dump new File(new Path(home, "segments").toString()), null, true, false, true ); } private void cycle(Path home, Configuration conf) { // This is the directory with "segments". Each fetching cycle // will produce its own collection of files. Each collection // is called a segment. final Path segments = new Path(home, "segments"); // First, we generate a list of target URLs to fetch from: new Generator(conf).generate( new Path(home, "crawldb"), new Path(home, "segments"), 1, 1000L, System.currentTimeMillis() ); // Then, we get the path of the current segment: final Path sgmt = Batch.segment(segments); // Then, we fetch, parse and update: new Fetcher(conf).fetch(sgmt, 10); new ParseSegment(conf).parse(sgmt); new CrawlDb(conf).update( new Path(home, "crawldb"), Files.list(Paths.get(segments.toString())) .map(p -> new Path(p.toString())) .toArray(Path[]::new), true, true ); } private static Path segment(final Path dir) throws IOException { // Get the path of the most recent segment in the list, // sorted by the date/time of their creation. final List<Path> list = Files.list(Paths.get(dir.toString())) .map(p -> new Path(p.toString())) .sorted(Comparator.comparing(Path::toString)) .collect(Collectors.toList()); return list.get(list.size() - 1); }} |
Pay attention that Path here is not the Path from JDK. It’s the Path from Hadoop. Don’t ask me why.
This seems to be a pretty straight-forward algorithm, however there is one tricky part. Nutch, in order to work, needs a number of plugins, which are standalone JAR packages, which it doesn’t include in its default JAR. They exist in its binary distribution and they are pretty heavy (over 250MB in Nutch 1.15). Nutch expects you to download the entire distribution, unpack, and run the binary nutch they provide, which will work with the provided plugins.
What can we do, now that we are in Java, not in the command line? Here is what I suggest:
<project>
<build>
<plugins>
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<version>1.4.1</version>
<executions>
<execution>
<id>download-nutch</id>
<phase>generate-resources</phase>
<goals>
<goal>wget</goal>
</goals>
<configuration>
<url>http://artfiles.org/apache.org/nutch/1.15/apache-nutch-1.15-bin.zip</url>
<unpack>true</unpack>
<outputDirectory>${project.build.directory}</outputDirectory>
<overwrite>false</overwrite>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
[...]
</project>This Maven plugin will download the entire binary distribution of Nutch and will unpack it to target/apache-nutch-1.15. The plugins will be in target/apache-nutch-1.15/plugins. The only thing we still need to do is to set the system property for the unit test:
<project>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<nutch.plugins.dir>${project.build.directory}/apache-nutch-1.15/plugins</nutch.plugins.dir>
</systemPropertyVariables>
</configuration>
</plugin>
</plugins>
</pluginManagement>
[...]
</build>
[...]
</project>Actually, one more thing we have to do: copy the contents of the directory conf from their binary distribution to our src/main/resources directory. There are many files, including the most important nutch-default.xml. They all have to be available on classpath, otherwise Nutch will complain in so many places and won’t be able to load the Configuration.
You can see how it all works together in this GitHub repository I created to illustrate the example: yegor256/nutch-in-java.
If you have any questions or suggestions, feel free to submit a pull request or comment here.
Published on Java Code Geeks with permission by Yegor Bugayenko, partner at our JCG program. See the original article here: How to Use Nutch From Java, Not From the Command Line Opinions expressed by Java Code Geeks contributors are their own. |
