Throughout the years I’ve seen many teams using maven in many different ways. Maven can be used for many ci/cd tasks instead of using extra pipeline code or it can be used to prepare the development environment before running some tests.
Generally it is convenient tool, widely used among java teams and will continue so since there is a huge ecosystem around it.
The CloudStorage Maven plugin helps you with using various cloud buckets as a private maven repository. Recently CloudStorageMaven for s3 got a huge upgrade, and you can use it in order to download or upload files from s3, by using it as a plugin.
The plugin assumes that your environment is configured properly to access the s3 resources needed.
This can be achieved individually through aws configure
aws configure
Other ways are through environmental variables or by using the appropriate iam role.
Supposing you want to download some certain files from a path in s3.
<build>
<plugins>
<plugin>
<groupId>com.gkatzioura.maven.cloud</groupId>
<artifactId>s3-storage-wagon</artifactId>
<version>1.6</version>
<executions>
<execution>
<id>download-one</id>
<phase>package</phase>
<goals>
<goal>s3-download</goal>
</goals>
<configuration>
<bucket>your-bucket</bucket>
<downloadPath>/local/download/path</downloadPath>
<keys>1.txt,2.txt,directory/3.txt</keys>
</configuration>
</execution>
<executions>
<plugin>
<plugins>
</build>The files 1.txt,2.txt,directory/3.txt once the execution is finished shall reside in the local directory specified
(/local/download/path).
Be aware that the file discovery on s3 is done with prefix, thus if you have file 1.txt and 1.txt.jpg both files shall be downloaded.
You can also download only one file to one file that you specified locally, as long as it is one to one.
<execution>
<id>download-prefix</id>
<phase>package</phase>
<goals>
<goal>s3-download</goal>
</goals>
<configuration>
<bucket>your-bucket</bucket>
<downloadPath>/path/to/local/your-file.txt</downloadPath>
<keys>a-key-to-download.txt</keys>
</configuration>
</execution>Apparently files with a prefix that contain directories (they are fakes ones on s3) will downloaded to the directory specified in the form of directories and sub directories
<execution>
<id>download-prefix</id>
<phase>package</phase>
<goals>
<goal>s3-download</goal>
</goals>
<configuration>
<bucket>your-bucket</bucket>
<downloadPath>/path/to/local/</downloadPath>
<keys>s3-prefix</keys>
</configuration>
</execution>The next part is about uploading files to s3.
Uploading one file
<execution>
<id>upload-one</id>
<phase>package</phase>
<goals>
<goal>s3-upload</goal>
</goals>
<configuration>
<bucket>your-bucket</bucket>
<path>/path/to/local/your-file.txt</path>
<key>key-to-download.txt</key>
</configuration>
</execution>Upload a directory
<execution>
<id>upload-one</id>
<phase>package</phase>
<goals>
<goal>s3-upload</goal>
</goals>
<configuration>
<bucket>your-bucket</bucket>
<path>/path/to/local/directory</path>
<key>prefix</key>
</configuration>
</execution>Upload to the root of bucket.
<execution>
<id>upload-multiples-files-no-key</id>
<phase>package</phase>
<goals>
<goal>s3-upload</goal>
</goals>
<configuration>
<bucket>your-bucket</bucket>
<path>/path/to/local/directory</path>
</configuration>
</execution>That’s it! Since it is an open source project you can contribute or issue pull requests at github.
Published on Java Code Geeks with permission by Emmanouil Gkatziouras, partner at our JCG program. See the original article here: Upload and Download files to S3 using maven. Opinions expressed by Java Code Geeks contributors are their own. |
Please suggest with some example for resumable download and upload in case of connection break too. In case big size of file e.g. 50gb file.