Being a strong advocate of Spring Batch, I’ve always talked about the notion of Spring Batch providing developers with a framework that allows them to focus on solving business needs. By doing so, it allows developers to not spend an inordinate amount of time solving all of the technical aspects to support the solution.
To illustrate what I mean by this, we’re going to take one of the previous Spring Batch examples I’ve written and enhance that a little bit for an additional business requirement that was needed.
The New Problem
In Part Three of my Spring Batch series, we introduced a tutorial for handling the output of large Excel files.
Later on it was determined that an additional business unit needed the same data, however they needed the data output in the format of a pipe-delimited text file with only three of the fields.
There are a couple of different ways to do this, but for this example, I’ll show you how to quickly implement your own ItemStreamReader that delegates the writing to your individual writers.
The first thing we need to do is to create the shell of our ItemStreamReader. I’m calling it the MultiFormatItemWriter. Here is what the shell looks like:
package com.keyhole.example;
import java.io.IOException;
import java.util.List;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.annotation.AfterStep;
import org.springframework.batch.core.annotation.BeforeStep;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemStreamException;
import org.springframework.batch.item.ItemStreamWriter;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;
import com.keyhole.example.poi.StockData;
@Component("multFormatItemWriter")
@Scope("step")
public class MultiFormatItemWriter implements ItemStreamWriter<StockData> {
@Override
public void write(List<? extends StockData> items) throws Exception {
}
@BeforeStep
public void beforeStep(StepExecution stepExecution) {
}
@AfterStep
public void afterStep(StepExecution stepExecution) throws IOException {
}
@Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
}
@Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
}
@Override
public void close() throws ItemStreamException {
}
}Next, we’ll need to make some adjustments to our existing StockDataExcelWriter from the previous example so that it will work as a delegate in our new MultiFormatItemWriter. I also found that there were some issues with the previous example related to the data stream coming from Nasdaq. The format of one of the fields had changed and the example was no longer working, so that had to be fixed before we could continue on.
- Bug fix: Changed the field type of marketCap on StockData from a BigDecimal to a String. The values were now coming across in the data feed as “$14.5M” and similar.
- Bug fix: Since the data format had changed and these blog articles use mostly static examples, I have created an input file of stock data named
companylist.csvin thedata.stockpackage under src/test/resources. - Bug fix: Modifed the stock data reader to use this data file instead of the live Nasdaq feed.
- Removed the
@Scope(“step”) annotation fromStockDataExcelWriter. This is required since theMultiFormatItemWriterwill be scoped at the step level. - Removed the
@BeforeStepand@AfterStepannotations fromStockDataExcelWriteras these methods will be called directly from the MultiFormatItemWriter. - Commented out the for loop inside that write method that was writing each record 300 times to the excel file. This was used for the large excel file demonstration, so that would need to be reverted for that example to work again.
Now that we’ve addressed the StockDataExcelWriter, we need to address the additional format output that the business needs. The second output should be in a pipe delimited text file and only contain the symbol, name and last sale fields.
For this delegate writer, we’re going to use the FlatFileItemWriter, which is one of the many output components provided with Spring Batch. To use this is a very simple configuration change and it looks like this:
<bean name="pipeDelimitedExtractFile" class="org.springframework.batch.item.file.FlatFileItemWriter"> <property name="resource" value="file:/data/example/excel/extract-example.txt" /> <property name="lineAggregator"> <bean class="org.springframework.batch.item.file.transform.DelimitedLineAggregator"> <property name="delimiter" value="|" /> <property name="fieldExtractor"> <bean class="org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor"> <property name="names" value="symbol,name,lastSale" /> </bean> </property> </bean> </property> </bean>
Thanks to Spring Batch having its foundation rooted in the Spring framework, it’s simple to configure the provided FlatFileItemWriter and wire the bean into the application code. In this case we’re creating the FlatFileItemWriter with the provided DelimitedLineAggregator, specifying the pipe character as the delimiter and setting the fieldExtractor to use the BeanWrapperFieldExtractor.
The BeanWrapperFieldExtractor takes the list of StockData records that is sent to the ItemStreamWriter and extracts the fields specified by the comma delimited list of field names that are found in the StockData bean. Finally, specifying the resource for output which in this case is the file extract-example.txt and is written to the /data/example/excel directory.
Now all we need to do is wire the two delegate writers into our MultiFormatItemWriter. Make sure the delegate writers are called in the appropriate methods and we’re done! Here is what the final code listing will look like for the MultiFormatITemWriter:
package com.keyhole.example;
import java.io.IOException;
import java.util.List;
import org.springframework.batch.core.StepExecution;
import org.springframework.batch.core.annotation.AfterStep;
import org.springframework.batch.core.annotation.BeforeStep;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemStreamException;
import org.springframework.batch.item.ItemStreamWriter;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;
import com.keyhole.example.poi.StockData;
import com.keyhole.example.poi.StockDataExcelWriter;
@Component("multFormatItemWriter")
@Scope("step")
public class MultiFormatItemWriter implements ItemStreamWriter<StockData> {
@Autowired
private StockDataExcelWriter stockDataExcelWriter;
@Autowired
@Qualifier("pipeDelimitedExtractFile")
private FlatFileItemWriter<StockData> extractWriter;
@Override
public void write(List<? extends StockData> items) throws Exception {
stockDataExcelWriter.write(items);
extractWriter.write(items);
}
@BeforeStep
public void beforeStep(StepExecution stepExecution) {
stockDataExcelWriter.beforeStep(stepExecution);
}
@AfterStep
public void afterStep(StepExecution stepExecution) throws IOException {
stockDataExcelWriter.afterStep(stepExecution);
}
@Override
public void open(ExecutionContext executionContext) throws ItemStreamException {
extractWriter.open(executionContext);
}
@Override
public void update(ExecutionContext executionContext) throws ItemStreamException {
extractWriter.update(executionContext);
}
@Override
public void close() throws ItemStreamException {
extractWriter.close();
}
}As you can see, there’s really not much work to do here and that’s what I wanted to point out. I hadn’t really shown how simple some business solutions can be by using some of the built-in readers and writers.
Final Thoughts
Now I did mention that there were a couple of ways to tackle this. The second would utilize the CompositeItemWriter that comes with Spring Batch. It does almost the exact same thing as I have done here, only it takes a list of ItemWriters and loops through them in each method that is implemented.
In that case, I would have converted my StockDataExcelWriter to implement the ItemStreamReader interface and the MultiFormatOutputWriter would be replaced with the CompositeItemWriter, which would be configured in the job configuration xml. Even less code.
So my point with this article today is to express how easy most common tasks and business solutions can be solved with several of the already implemented components that are provided with Spring Batch.
This and the other examples can be found over at GitHub at the following location: https://github.com/jonny-hackett/batch-example.
| Reference: | Spring Batch: Multiple Format Output Writer from our JCG partner Jonny Hackett at the Keyhole Software blog. |
