S3Proxy is an open-source and highly customizable proxy server that enables developers to emulate Amazon S3 storage locally. It provides an S3-compatible API that allows us to test and develop applications requiring object storage without depending on an actual AWS S3 environment. With S3Proxy, we can integrate backend storage systems like local filesystems or databases and access them using the same S3 API calls your application uses in production. This article explores how to use the S3Proxy library to run S3Proxy in a local environment.
1. How S3Proxy Works
S3Proxy acts as a middleware between our application and various storage backends by emulating the Amazon S3 API. When our application sends S3-compatible requests (such as uploading or retrieving objects), S3Proxy intercepts these requests and translates them into operations supported by the configured storage backend. This allows us to interact with local or cloud-based storage systems using familiar S3 commands and tools.
Here is an explanation of how S3Proxy works:
- API Emulation: S3Proxy listens for requests using the S3 API (e.g.,
PUT,GET,DELETE) over HTTP or HTTPS. It interprets these requests and maps them to the underlying storage backend. - Backend Configuration: Storage backends like local filesystems, Google Cloud Storage, OpenStack Swift, Microsoft Azure Blob Storage, or Ceph RADOS Gateway are configured in the
s3proxy.conffile. S3Proxy translates S3 operations into backend-specific actions. - Request Handling: Incoming S3 requests, such as uploading files (
PUTobject) or fetching files (GETobject), are processed by S3Proxy. For instance, an upload request might write the file to the local filesystem if it is configured as the backend. - Security and Access Control: S3Proxy can be configured with credentials to authenticate requests, ensuring only authorized access. For example, it can use static credentials or mimic AWS authentication to validate requests.
- Local Development and Testing: By running S3Proxy locally, developers can emulate S3 interactions without needing access to AWS. Applications interact with S3Proxy as if it were a live S3 service, enabling seamless testing and debugging.
1.1 Example Workflow
Here is an example of how S3Proxy processes a file upload request when the local filesystem is used as the backend:
- Application: The application sends a
PUTrequest to S3Proxy with the object data. - S3Proxy: It receives the request and verifies the credentials if authentication is enabled.
- Translation: S3Proxy translates the
PUTrequest into a file write operation in the specified directory of the local filesystem.
2. Using S3Proxy in Java Applications
To use S3Proxy in your Java project, include the following Maven dependency:
<dependency>
<groupId>org.gaul</groupId>
<artifactId>s3proxy</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.28.23</version>
</dependency>
S3Proxy (org.gaul:s3proxy) offers a locally hosted endpoint that emulates the Amazon S3 API, while the AWS SDK for S3 (software.amazon.awssdk:s3) provides the tools to interact with this endpoint as though it were a real S3 service, making this combination ideal for local development and testing without needing an AWS account.
2.1 Sample Java Code for Using S3Proxy
The following code example demonstrates how to configure and use S3Proxy along with AWS SDK’s S3Client to interact with it.
public class S3ProxyExample {
public static void main(String[] args) throws Exception {
// Configure the backend BlobStore
Properties properties = new Properties();
properties.setProperty("jclouds.filesystem.basedir", "/tmp/s3proxy");
BlobStoreContext context = ContextBuilder
.newBuilder("filesystem")
.credentials("identity", "credential")
.overrides(properties)
.build(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
// Configure and start S3Proxy
S3Proxy s3Proxy = S3Proxy.builder()
.blobStore(blobStore)
.endpoint(URI.create("http://127.0.0.1:8080"))
.build();
s3Proxy.start();
// Wait for S3Proxy to start
while (!s3Proxy.getState().equals(AbstractLifeCycle.STARTED)) {
Thread.sleep(100);
}
System.out.println("S3Proxy started at: " + s3Proxy.getState());
// Configure S3Client
S3Client s3Client = S3Client.builder()
.endpointOverride(URI.create("http://127.0.0.1:8080"))
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create("access", "secret")))
.build();
// Perform S3 Operations
String bucketName = "test-bucket";
s3Client.createBucket(CreateBucketRequest.builder().bucket(bucketName).build());
System.out.println("Bucket created: " + bucketName);
String key = "example.txt";
String content = "Hello, S3Proxy!";
s3Client.putObject(PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build(),
RequestBody.fromBytes(content.getBytes()));
System.out.println("File uploaded: " + key);
// Verify the upload
List<S3Object> summaries = s3Client.listObjects(request
-> request.bucket(bucketName)
).contents();
summaries.forEach(summary -> System.out.println("File in bucket: " + summary.key()));
String downloadedContent = s3Client.getObject(
GetObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build(),
ResponseTransformer.toBytes()).asUtf8String();
System.out.println("Downloaded content: " + downloadedContent);
// Shutdown the proxy
s3Proxy.stop();
System.out.println("S3Proxy stopped.");
}
}
The following section provides a detailed explanation and overview of the example code above:
Configure the Backend BlobStore
Properties properties = new Properties();
properties.setProperty("jclouds.filesystem.basedir", "/tmp/s3proxy");
BlobStoreContext context = ContextBuilder
.newBuilder("filesystem")
.credentials("identity", "credential")
.overrides(properties)
.build(BlobStoreContext.class);
BlobStore blobStore = context.getBlobStore();
The purpose of this step is to configure a BlobStore to serve as the storage backend for S3Proxy. The key configuration includes setting jclouds.filesystem.basedir to specify the local directory (/tmp/s3proxy) for storing objects. Using ContextBuilder, a BlobStoreContext is created for the “filesystem” provider with dummy credentials (identity and credential). As a result, a BlobStore instance is created to interact with the backend storage.
Start S3Proxy
S3Proxy s3Proxy = S3Proxy.builder()
.blobStore(blobStore)
.endpoint(URI.create("http://127.0.0.1:8080"))
.build();
s3Proxy.start();
while (!s3Proxy.getState().equals(AbstractLifeCycle.STARTED)) {
Thread.sleep(100);
}
System.out.println("S3Proxy started at: " + s3Proxy.getState());
The purpose of this step is to configure and launch the S3Proxy server. Using S3Proxy.builder(), the server is set up to utilize the previously initialized BlobStore and bind it to the specified endpoint (http://127.0.0.1:8080). The s3Proxy.start() method is then called to start the server.
Configure the S3Client
S3Client s3Client = S3Client.builder()
.endpointOverride(URI.create("http://127.0.0.1:8080"))
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create("access", "secret")))
.build();
In the above code fragment, an S3Client instance is created to work with the S3Proxy endpoint. It is configured with the local S3Proxy address (endpointOverride), simple static credentials (“access” and “secret”), and an AWS region (though not strictly needed for S3Proxy). This setup provides a fully configured S3Client for performing S3 operations.
Perform S3 Operations
String bucketName = "test-bucket";
s3Client.createBucket(CreateBucketRequest.builder().bucket(bucketName).build());
String key = "example.txt";
String content = "Hello, S3Proxy!";
s3Client.putObject(PutObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build(),
RequestBody.fromBytes(content.getBytes()));
List<S3Object> summaries = s3Client.listObjects(request
-> request.bucket(bucketName)
).contents();
String downloadedContent = s3Client.getObject(
GetObjectRequest.builder()
.bucket(bucketName)
.key(key)
.build(),
ResponseTransformer.toBytes()).asUtf8String();
System.out.println("Downloaded content: " + downloadedContent);
In the above code fragment, we demonstrate how to interact with the S3Proxy server using the S3Client. First, a bucket named test-bucket is created to serve as a storage container. Next, a file named example.txt is uploaded to the bucket with the content "Hello, S3Proxy!". The bucket’s contents are then listed, displaying all stored files. Finally, the uploaded file is downloaded, and its content is retrieved to verify that it matches the original data, ensuring the upload and retrieval processes were successful.
These operations highlight how S3Proxy can replicate typical S3 functionality for development and testing purposes.
3. Conclusion
In this article, we explored how to set up and use S3Proxy, a Java library that mimics Amazon S3 functionality locally for development and testing. We walked through configuring a BlobStore backend, starting the S3Proxy server, and performing basic S3 operations such as creating a bucket, uploading files, listing bucket contents, and downloading files using S3Client. This setup enables us to test and build S3-compatible applications without incurring AWS costs or requiring an active internet connection.
Beyond Amazon S3 compatibility, S3Proxy also supports other storage backends, including Google Cloud Storage, OpenStack Swift, Microsoft Azure Blob Storage, and Ceph RADOS Gateway, making it a versatile solution for integrating and testing with various storage systems.
4. Download the Source Code
This article explored the Java S3Proxy library.
You can download the full source code of this example here: Java s3proxy library
