英文:
SSLException from AwsSDK S3AsyncClient GetObjectRequest
问题
I am using software.amazon.awssdk.services.s3.model.GetObjectRequest to retrieve an object in a S3AsyncClient getObject method call from an enterprise S3 instance that emulates AWS S3 functionality. This follows the pattern of Baeldung aws-s3-reactive demo found at https://www.baeldung.com/java-aws-s3-reactive
The above-mentioned demo was converted from maven to gradle and integrated into an existing project (demo is part of a much bigger tutorial project).
Executing the S3AsyncClient getObject call resulted in an exception of "javax.net.ssl.SSLException: SSLEngine closed already". Debugging the call using the logger produced the following output. There might be missing headers or something else that I'm missing. The more traditional AmazonS3.getObject call works and produces a few select headers not included in this S3AsyncClient.getObject.
I'm sharing the output, relevant configuration classes, and services below for both the S3AsyncClient getObject call (unsuccessful) and AmazonS3 getObject call (successful) below, with the build.gradle file at the bottom. Any input as to what I may be doing wrong with the S3AyncClient.getObject output is appreciated.
[The rest of the content you provided is code and configuration details, which I'm not translating based on your request.]
英文:
I am using a software.amazon.awssdk.services.s3.model.GetObjectRequest to retrieve an object in a S3AsyncClient getObject method call from an enterprise S3 instance that emulates AWS S3 functionality. This follows the pattern of Baeldung aws-s3-reactive demo found at
https://www.baeldung.com/java-aws-s3-reactive
The above-mentioned demo was converted from maven to gradle and integrated into an existing project (demo is part of a much bigger tutorial project).
Executing the S3AsyncClient getObject call resulted in exception of "javax.net.ssl.SSLException: SSLEngine closed already". Debugging the call using the logger produced the below output. I have to wonder if there are missing headers, or if I am missing something else. The more tranditional AmazonS3.getObject call works and produces a few select headers not included in this S3AsyncClient.getObject.
I share the output (edited to remove personal information), relevant configuration classes, and services below for both the S3AsyncClient getObject call (unsuccessful) and AmazonS3 getObject call (successful) below, with the build.gradle file at the bottom. Any input as to what I may be doing wrong with the
S3AyncClient.getObject (unsuccessful) output ("DEBUG***" denotes lines that begin with:
"DEBUG io.netty.handler.logging.LoggingHandler:147 - [id: {id}, L:{ip}:{port} - R:{namespace}.{endpoint}/{ip:port}]"):
GET /mybucket/resource.pdf HTTP/1.1
Host: {namespace}.{endpoint:port}
amz-sdk-invocation-id: {invocation-id}
amz-sdk-retry: 0/0/
Authorization: AWS4-HMAC-SHA256 Credential={namespace}-user01/20200831/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;x-amz-content-sha256;x-amz-date, Signature={signature}
User-Agent: aws-sdk-java/2.10.86 Windows_10/10.0 Java_HotSpot_TM__64-Bit_Server_VM/25.191-b12 Java/1.8.0_191 vendor/Oracle_Corporation io/async http/UNKNOWN
x-amz-content-sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200831T203041Z
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** FLUSH
[aws-java-sdk-NettyEventLoop-0-2] DEBUG*** READ COMPLETE
[aws-java-sdk-NettyEventLoop-0-2] DEBUG*** INACTIVE
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** FLUSH
[aws-java-sdk-NettyEventLoop-0-2] DEBUG*** UNREGISTERED
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** WRITE, DefaultHttpContent(data: EmptyByteBufBE, decoderResult: success), 0B
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** FLUSH
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** USER_EVENT: SslHandshakeCompletionEvent(javax.net.ssl.SSLException: Received fatal alert: handshake_failure)
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** WRITE, EmptyLastHttpContent, 0B
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** FLUSH
[aws-java-sdk-NettyEventLoop-0-4] DEBUG software.amazon.awssdk.request:84 - Retryable error detected. Will retry in 18ms. Request attempt number 2
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** READ COMPLETE
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** INACTIVE
[aws-java-sdk-NettyEventLoop-0-4] DEBUG*** UNREGISTERED
[sdk-ScheduledExecutor-2-0] DEBUG software.amazon.awssdk.request:84 - Retrying Request: DefaultSdkHttpFullRequest(httpMethod=GET, protocol=https, host={namespace}.{endpoint}, port={port}, encodedPath=/mybucket/resource.pdf, headers=[amz-sdk-invocation-id, User-Agent], queryParameters=[])
>> DEBUG io.netty.handler.logging.LoggingHandler:147 - [id: {id}] REGISTERED
>> DEBUG io.netty.handler.logging.LoggingHandler:147 - [id: {id}] CONNECT: {namespace}.{endpoint}/{ip:port}
>> DEBUG*** ACTIVE
>> DEBUG*** WRITE: software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor$StreamedRequest(DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /mybucket/resource.pdf HTTP/1.1
Host: {namespace}.{endpoint:port}
amz-sdk-invocation-id: {invocation-id}
amz-sdk-retry: {retry}
Authorization: AWS4-HMAC-SHA256 Credential={namespace}-user01/20200831/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;x-amz-content-sha256;x-amz-date, Signature={signature}
User-Agent: aws-sdk-java/2.10.86 Windows_10/10.0 Java_HotSpot_TM__64-Bit_Server_VM/25.191-b12 Java/1.8.0_191 vendor/Oracle_Corporation io/async http/UNKNOWN
x-amz-content-sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200831T203044Z)
***{{{{ below lines prior to stack trace begin with:
"DEBUG io.netty.handler.logging.LoggingHandler:147 - [id: {id}, L:{ip}:{port} - R:{namespace}.{endpoint}/{ip:port}]" }}}}
>> DEBUG*** FLUSH
>> DEBUG*** FLUSH
>> DEBUG*** WRITE, DefaultHttpContent(data: EmptyByteBufBE, decoderResult: success), 0B
>> DEBUG*** FLUSH
>> DEBUG*** USER_EVENT: SslHandshakeCompletionEvent(javax.net.ssl.SSLException: Received fatal alert: handshake_failure)
>> DEBUG*** WRITE, EmptyLastHttpContent, 0B
>> DEBUG*** FLUSH
>> DEBUG*** READ COMPLETE
>> DEBUG*** INACTIVE
>> DEBUG*** UNREGISTERED
org.apache.catalina.core.ApplicationDispatcher invoke
SEVERE: Servlet.service() for servlet [dispatcherServlet] threw exception
javax.net.ssl.SSLException: SSLEngine closed already
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:837)
at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:800)
at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:781)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:749)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:741)
at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:727)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:533)
at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:358)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:749)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:741)
at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:727)
at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:749)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:789)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:757)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.completeBody(HttpStreamsHandler.java:332)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.access$400(HttpStreamsHandler.java:43)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3$1.run(HttpStreamsHandler.java:310)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.executeInEventLoop(HttpStreamsHandler.java:376)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler.access$300(HttpStreamsHandler.java:43)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HttpStreamsHandler$3.complete(HttpStreamsHandler.java:307)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:274)
at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerSubscriber$3.operationComplete(HandlerSubscriber.java:271)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:57)
at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:31)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:57)
at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:31)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:57)
at io.netty.channel.DelegatingChannelPromiseNotifier.operationComplete(DelegatingChannelPromiseNotifier.java:31)
at io.netty.channel.AbstractCoalescingBufferQueue.releaseAndCompleteAll(AbstractCoalescingBufferQueue.java:340)
at io.netty.channel.AbstractCoalescingBufferQueue.releaseAndFailAll(AbstractCoalescingBufferQueue.java:207)
at io.netty.handler.ssl.SslHandler.releaseAndFailAll(SslHandler.java:1822)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1801)
at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1263)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1233)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Aug 31, 2020 4:30:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: SSLEngine closed already] with root cause
javax.net.ssl.SSLException: SSLEngine closed already
{rest of stack trace is exactly the same as above}
The GET method, S3ClientConfiguration class, and S3ClientConfigurationProperties class are shared below.
DownloadResource (unsuccessful). Attempted GET call goes through downloadFile
@RestController
@RequestMapping("/inbox")
@Slf4j
public class DownloadResource {
private final S3AsyncClient s3client;
private final S3ClientConfigurationProperties s3config;
public DownloadResource(S3AsyncClient s3client, S3ClientConfigurationProperties s3config) {
this.s3client = s3client;
this.s3config = s3config;
}
@GetMapping(path="/{filekey}")
public Mono<ResponseEntity<Flux<ByteBuffer>>> downloadFile(@PathVariable("filekey") String filekey) {
GetObjectRequest request = GetObjectRequest.builder()
.bucket(s3config.getBucket())
.key(filekey)
.build();
return Mono.fromFuture(s3client.getObject(request,new FluxResponseProvider()))
.map( (response) -> {
checkResult(response.sdkResponse);
String filename = getMetadataItem(response.sdkResponse,"filename",filekey);
log.info("[I65] filename={}, length={}",filename, response.sdkResponse.contentLength() );
return ResponseEntity.ok()
.header(HttpHeaders.CONTENT_TYPE, response.sdkResponse.contentType())
.header(HttpHeaders.CONTENT_LENGTH, Long.toString(response.sdkResponse.contentLength()))
.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=\"" + filename + "\"")
.body(response.flux);
});
}
/**
* Lookup a metadata key in a case-insensitive way.
* @param sdkResponse
* @param key
* @param defaultValue
* @return
*/
private String getMetadataItem(GetObjectResponse sdkResponse, String key, String defaultValue) {
for( Entry<String, String> entry : sdkResponse.metadata().entrySet()) {
if ( entry.getKey().equalsIgnoreCase(key)) {
return entry.getValue();
}
}
return defaultValue;
}
// Helper used to check return codes from an API call
private static void checkResult(GetObjectResponse response) {
SdkHttpResponse sdkResponse = response.sdkHttpResponse();
if ( sdkResponse != null && sdkResponse.isSuccessful()) {
return;
}
throw new DownloadFailedException(response);
}
static class FluxResponseProvider implements AsyncResponseTransformer<GetObjectResponse,FluxResponse> {
private FluxResponse response;
@Override
public CompletableFuture<FluxResponse> prepare() {
response = new FluxResponse();
return response.cf;
}
@Override
public void onResponse(GetObjectResponse sdkResponse) {
this.response.sdkResponse = sdkResponse;
}
@Override
public void onStream(SdkPublisher<ByteBuffer> publisher) {
response.flux = Flux.from(publisher);
response.cf.complete(response);
}
@Override
public void exceptionOccurred(Throwable error) {
response.cf.completeExceptionally(error);
}
}
/**
* Holds the API response and stream
* @author Philippe
*/
static class FluxResponse {
final CompletableFuture<FluxResponse> cf = new CompletableFuture<>();
GetObjectResponse sdkResponse;
Flux<ByteBuffer> flux;
}
}
Configuration Properties Class
@ConfigurationProperties(prefix = "aws.s3")
@Data
public class S3ClientConfigurationProperties {
private Region region = Region.US_EAST_1;
private URI endpoint;
private String accessKey;
private String secretKey;
// Bucket name we'll be using as our backend storage
private String bucket;
// AWS S3 requires that file parts must have at least 5MB, except
// for the last part. This may change for other S3-compatible services, so let't
// define a configuration property for that
private int multipartMinPartSize = 5*1024*1024;
}
S3ClientConfiguration (unsuccessful)
@Configuration
@EnableConfigurationProperties(S3ClientConfigurationProperties.class)
public class S3ClientConfiguration {
@Bean
public S3AsyncClient s3client(S3ClientConfigurationProperties s3props, AwsCredentialsProvider credentialsProvider) {
SdkAsyncHttpClient httpClient = NettyNioAsyncHttpClient.builder()
.writeTimeout(Duration.ZERO)
.maxConcurrency(64)
.build();
S3Configuration serviceConfiguration = S3Configuration.builder()
.checksumValidationEnabled(false)
.chunkedEncodingEnabled(true)
.build();
S3AsyncClientBuilder b = S3AsyncClient.builder()
.httpClient(httpClient)
.region(s3props.getRegion())
.credentialsProvider(credentialsProvider)
.serviceConfiguration(serviceConfiguration);
if (s3props.getEndpoint() != null) {
b = b.endpointOverride(s3props.getEndpoint());
}
return b.build();
}
@Bean
public AwsCredentialsProvider awsCredentialsProvider(S3ClientConfigurationProperties s3props) {
if (StringUtils.isBlank(s3props.getAccessKey())) {
// Return default provider
return DefaultCredentialsProvider.create();
}
else {
// Return custom credentials provider
return () -> {
AwsCredentials creds = AwsBasicCredentials.create(s3props.getAccessKey(), s3props.getSecretKey());
return creds;
};
}
}
}
For comparison, here is output for (successful) AmazonS3.getObject call:
>> "HEAD /mybucket/ HTTP/1.1"
>> "Host: {namespace}.{endpoint:port}"
>> "Authorization: AWS {namespace}-user01:{sig}="
>> "User-Agent: aws-sdk-java/1.11.415 Windows_10/10.0 Java_HotSpot(TM)_64-Bit_Server_VM/25.191-b12 java/1.8.0_191"
>> "amz-sdk-invocation-id: {invocation-id}"
>> "amz-sdk-retry: 0/0/500"
>> "Date: Mon, 31 Aug 2020 20:27:43 GMT"
>> "Content-Type: application/octet-stream"
>> "Connection: Keep-Alive"
<< "HTTP/1.1 200 OK"
<< "Date: Mon, 31 Aug 2020 20:27:43 GMT"
<< "Server: ViPR/1.0"
<< "x-amz-request-id: {request-id}"
<< "x-amz-id-2: {id}"
<< "x-emc-retention-period: 0"
<< "Content-Length: 0"
<< ""
>> "GET /mybucket/Calendar HTTP/1.1"
>> "Host: {namespace}.{endpoint:port}"
>> "Authorization: AWS {namespace}-user01:{sig}"
>> "User-Agent: aws-sdk-java/1.11.415 Windows_10/10.0 Java_HotSpot(TM)_64-Bit_Server_VM/25.191-b12 java/1.8.0_191"
>> "amz-sdk-invocation-id: {invocation-id}"
>> "amz-sdk-retry: 0/0/500"
>> "Date: Mon, 31 Aug 2020 20:27:44 GMT"
>> "Content-Type: application/octet-stream"
>> "Content-Length: 0"
>> "Connection: Keep-Alive"
>> ""
<< "HTTP/1.1 200 OK"
<< "Date: Mon, 31 Aug 2020 20:27:44 GMT"
<< "Cache-Control: no-cache"
<< "Server: ViPR/1.0"
<< "x-amz-request-id: {request-id}"
<< "x-amz-id-2: {amz-id-2}"
<< "ETag: "d9b5afd94610a7578384af7b36186b94""
<< "Last-Modified: Thu, 27 Aug 2020 15:38:29 GMT"
<< "x-emc-mtime: 1598542709166"
<< "x-amz-server-side-encryption: AES256"
<< "Content-Type: application/pdf"
<< "Content-Length: 44255"
AmazonS3.getObject (successful) method call is as seen below:
public S3Object get(String key) {
return s3Storage.getClient().getObject(new GetObjectRequest(s3Storage.getBucket(), key));
}
S3Configuration (successful):
@Configuration
@SuppressWarnings("deprecation")
public class S3Configuration {
@Bean
public S3Storage s3Storage(S3ServiceInfo s3ServiceInfo) {
final ClientConfiguration httpsClientConfig = new ClientConfiguration().withProtocol(Protocol.HTTPS).withSignerOverride("S3SignerType");
httpsClientConfig.getApacheHttpClientConfig().setSslSocketFactory(SSLSocketFactory.getSystemSocketFactory());
AmazonS3 client = AmazonS3ClientBuilder.standard()
.withPathStyleAccessEnabled(true)
.withForceGlobalBucketAccessEnabled(true)
.withClientConfiguration(httpsClientConfig)
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(s3ServiceInfo.getAccessKey(), s3ServiceInfo.getSecretKey())))
.withEndpointConfiguration(new EndpointConfiguration(endpointUrlWithNamespace(s3ServiceInfo.getEndpoint(), s3ServiceInfo.getAccessKey()), null))
.build();
return new S3Storage(client, s3ServiceInfo.getBucket(), s3ServiceInfo.getEndpoint());
}
protected String endpointUrlWithNamespace(String endpoint, String accessKey) {
//extract namespace from access-key (verify format)
Matcher matcher = Pattern.compile("((.+?-){3}ns\\d\\d)-").matcher(accessKey);
if (!matcher.find()) return endpoint;
String namespace = matcher.group(1);
return endpoint.contains("://s3-object") ? endpoint.replace("://", "://" + namespace + ".") : endpoint;
}
@Configuration
@Profile("cloud")
@Import(CloudScanConfiguration.class)
public class S3CloudConfiguration extends AbstractCloudConfig {
@Bean
public S3ServiceInfo s3ServiceInfo() {
return (S3ServiceInfo)cloud().getServiceInfos(S3ServiceInfo.class).get(0);
}
}
@Configuration
@Profile("!cloud")
public class S3LocalConfiguration {
@Bean
public S3ServiceInfo s3ServiceInfo(
@Value("${aws.s3.accessKey}") String accessKey,
@Value("${aws.s3.secretKey}") String secretKey,
@Value("${aws.s3.bucket}") String bucket,
@Value("${aws.s3.endpoint}") String endpoint) {
return new S3ServiceInfo(null, accessKey, secretKey, endpoint, bucket);
}
}
@Data
@AllArgsConstructor
public static class S3Storage {
private AmazonS3 client;
private String bucket;
private String endpoint;
}
}
build.gradle:
buildscript {
ext {
springBootVersion = '2.2.1.RELEASE'
awsSdkVersion = '2.10.86'
springVersion = '5.2.6.RELEASE'
springCloudVersion = 'Greenwich.RELEASE'
springCloudServicesVersion = '2.1.1.RELEASE'
lombokVersion = '1.18.10'
junitJupiterVersion ='5.2.0'
orgSlf4jVersion = '1.7.30'
logbackVersion = '1.2.3'
junitVersion ='4.12'
junitPlatformVersion ='1.2.0'
mavenSurefirePluginVersion = '2.15'
hamcrestVersion = '2.2'
hamcrestAllVersion = '1.3'
mockitoVersion = '3.3.0'
}
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
}
}
plugins {
id 'java'
id 'maven'
id 'io.spring.dependency-management' version '1.0.7.RELEASE'
id 'gradle-boost' version '2.0.0'
id 'org.springframework.boot' version "${springBootVersion}"
}
configurations.all {
exclude group: 'ch.qos.logback'
exclude group: 'org.logback'
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
exclude group: 'org.springframework.boot', module: 'logback-classic'
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
mavenBom "org.springframework.boot:spring-boot-dependencies:${springCloudServicesVersion}"
mavenBom "software.amazon.awssdk:bom:${awsSdkVersion}"
}
}
dependencies {
implementation "org.springframework.boot:spring-boot-starter-web:${springBootVersion}"
implementation "org.springframework.boot:spring-boot-starter-actuator:${springBootVersion}"
implementation 'io.springfox:springfox-swagger2:2.9.2'
implementation 'io.springfox:springfox-swagger-ui:2.9.2'
//other
compileOnly 'org.projectlombok:lombok:1.18.10'
annotationProcessor 'org.projectlombok:lombok:1.18.10'
//aws
compile 'com.amazonaws:aws-lambda-java-core:1.1.0'
compile 'com.amazonaws:aws-lambda-java-events:2.2.7'
compile 'com.amazonaws:aws-java-sdk-dynamodb:1.11.666'
/*********** TEST ***********/
testImplementation "org.springframework.boot:spring-boot-starter-test:${springBootVersion}"
testImplementation "org.springframework.security:spring-security-test:${springVersion}"
testAnnotationProcessor 'org.projectlombok:lombok'
/************************************ log 4J2 ************************************/
compileOnly group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.11.1'
compileOnly group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.11.1'
compileOnly group: 'org.apache.logging.log4j', name: 'log4j-slf4j-impl', version: '2.11.1'
compileOnly group: 'org.apache.logging.log4j', name: 'log4j-web', version: '2.11.1'
compileOnly group: 'org.apache.logging.log4j', name: 'log4j-jcl', version: '2.11.1'
compileOnly group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: '2.11.1'
/********************************************************* apache *********************************************************/
compileOnly group: 'org.apache.commons', name: 'commons-lang3', version: '3.9'
compileOnly group: 'org.apache.commons', name: 'commons-math3', version: '3.6.1'
compile group: 'org.apache.commons', name: 'commons-csv', version: '1.6'
compile group: 'commons-io', name: 'commons-io', version: '2.6'
compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.4'
compile "software.amazon.awssdk:bom:2.10.27"
compile "org.springframework.boot:spring-boot-starter-webflux:${springBootVersion}"
compile "software.amazon.awssdk:s3:${awsSdkVersion}"
compile "software.amazon.awssdk:netty-nio-client:${awsSdkVersion}"
testCompile "io.projectreactor:reactor-test:3.3.9.RELEASE"
compile "org.springframework.boot:spring-boot-dependencies:${springCloudServicesVersion}"
compile "org.springframework.boot:spring-boot-devtools:${springBootVersion}"
compile "org.springframework.boot:spring-boot-configuration-processor:${springBootVersion}"
}
答案1
得分: 1
在以高速率复制对象时,我遇到了类似的问题,以下是我最终采用的配置:
```java
S3AsyncClient.builder()
.httpClientBuilder(NettyNioAsyncHttpClient.builder()
.connectionMaxIdleTime(Duration.ofSeconds(180))
.connectionAcquisitionTimeout(Duration.ofSeconds(60))
.connectionTimeout(Duration.ofSeconds(120)).maxConcurrency(120)
.maxPendingConnectionAcquires(50000)
)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.retryPolicy(RetryPolicy.builder()
.backoffStrategy(BackoffStrategy.defaultStrategy())
.throttlingBackoffStrategy(BackoffStrategy.defaultThrottlingStrategy())
.numRetries(5)
.retryCondition(RetryCondition.defaultRetryCondition())
.build()
)
.apiCallTimeout(Duration.ofSeconds(120))
.build())
.build();
API 调用尝试超时是在 HTTP 请求完成前等待的时间,如果超时会中断请求。
<details>
<summary>英文:</summary>
I had a similar issue when copying objects at high rate, this is the configuration I ended up with
S3AsyncClient.builder()
.httpClientBuilder( NettyNioAsyncHttpClient.builder()
.connectionMaxIdleTime(Duration.ofSeconds(180))
.connectionAcquisitionTimeout(Duration.ofSeconds(60))
.connectionTimeout(Duration.ofSeconds(120)) .maxConcurrency(120)
.maxPendingConnectionAcquires(50000)
)
.overrideConfiguration( ClientOverrideConfiguration.builder()
.retryPolicy( RetryPolicy.builder()
.backoffStrategy(BackoffStrategy.defaultStrategy())
.throttlingBackoffStrategy(BackoffStrategy.defaultThrottlingStrategy())
.numRetries(5)
.retryCondition(RetryCondition.defaultRetryCondition())
.build()
)
.apiCallTimeout(Duration.ofSeconds(120)) .build()) .build();
The API call attempt timeout is the amount of time to wait for the HTTP request to complete before timing out.
</details>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论