2020年8月12日 18:54:23go评论73阅读模式

英文:

AWS S3 v2 GetObject Unable to unmarshall response (java.net.SocketTimeoutException)

问题

你正在尝试从S3获取一个大对象，并将输出直接流式传输到HttpServletResponse输出流。然而，由于SocketTimeoutException，它总是失败。

以下是代码部分：

public void downloadMergedResults(final String jobId, final OutputStream os) {
    get("jobs/" + jobId + "/results", os);
}

private boolean get(final String key, final OutputStream outputStream) {
    log.trace("GET {}", key);
    try {
        client.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build(), ResponseTransformer.toOutputStream(outputStream));
        return true;
    } catch (S3Exception e) {
        if (e.statusCode() == 404) {
            log.info("Item with key: {} does not exist.", key);
        }
        else {
            log.error("Could not get item {} from S3. Error: {} {}", key, e.getMessage(), e.awsErrorDetails());
        }
    }
    return false;
}

这是如何在Spring Boot应用程序中使用它的方式：

public Callable<ResponseEntity<StreamingResponseBody>> allData(HttpServletResponse res, @PathVariable("jobId") String jobId) {
    // ...
    return () -> {
        StreamingResponseBody streamingResponseBody = out -> {
            try {
                s3client.downloadMergedResults(jobId, out);
            } catch(Exception e) {
                log.error("{} Error downloading results: {}", jobId, e.getMessage(), e);
            }
        };
        return new ResponseEntity<>(streamingResponseBody, HttpStatus.OK);
    };
}

根据你提供的信息，你已经尝试了多种不同的S3Client初始化方式和超时配置，但仍然遇到了SocketTimeoutException的问题。你的Nginx和Spring Boot的配置似乎也没有问题。

要进一步诊断和解决这个问题，你可以考虑以下步骤：

增加日志信息: 在异常处理部分，记录更多有关异常的信息，例如堆栈跟踪、请求的详细信息等，以便更好地了解问题的根本原因。
监控S3请求: 使用Amazon CloudWatch或其他监控工具来监视S3请求的性能和错误情况，以查看是否有异常行为或频繁的超时。
调整超时设置: 你可以尝试进一步增加超时设置，以确保S3请求有足够的时间完成。这可能包括增加apiCallTimeout、connectionTimeout和socketTimeout的值。
网络问题: 确保服务器与S3之间的网络连接稳定。检查是否存在防火墙或网络设备可能干扰连接的情况。
S3 Bucket配置: 检查S3 Bucket的配置，确保没有设置特殊的限制或策略，可能会导致请求失败或超时。
考虑使用分段下载: 如果你处理的对象非常大，考虑使用分段下载，这样可以将对象分成多个部分下载，以减小每个请求的大小。
升级SDK版本: 有时SDK的特定版本可能存在问题，考虑升级到最新版本或一个已知稳定的版本。

最后，根据错误信息和日志来调查问题，可以更好地定位问题的原因。如果所有尝试都未成功，你可能需要联系Amazon Web Services支持以获取更进一步的帮助。

英文:

I'm trying to get a large object from S3 and steam the output straight to an HttpServletResponse output stream.

However it always fails on this object due to a SocketTimeoutException.

Here's the code that uses the S3Client to get the object:

    public void downloadMergedResults(final String jobId, final OutputStream os) {
        get(&quot;jobs/&quot; + jobId + &quot;/results&quot;, os);
    }

    private boolean get(final String key, final OutputStream outputStream) {
        log.trace(&quot;GET {}&quot;, key);
        try {
            client.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build(), ResponseTransformer.toOutputStream(outputStream));
            return true;
        } catch (S3Exception e) {
            if (e.statusCode() == 404) {
                log.info(&quot;Item with key: {} does not exist.&quot;, key);
            }
            else {
                log.error(&quot;Could not get item {} from S3. Error: {} {}&quot;, key, e.getMessage(), e.awsErrorDetails());
            }
        }
        return false;
    }

Here's how I'm using that within my springboot application:

    public Callable&lt;ResponseEntity&lt;StreamingResponseBody&gt;&gt; allData(HttpServletResponse res, @PathVariable(&quot;jobId&quot;) String jobId) {

        Progress progress = s3client.downloadCachedProgress(jobId);
        if (progress == null) {
            log.info(&quot;{} Could not find progress file&quot;, jobId);
            return () -&gt; new ResponseEntity&lt;&gt;(HttpStatus.NOT_FOUND);
        }

        res.setContentType(MediaType.APPLICATION_JSON_VALUE);
        res.setContentLengthLong(OPEN_ARRAY.length + progress.getDataSizeBytes() + (SEPARATOR.length * (progress.getMediaCount() - 1)) + CLOSE_ARRAY.length);

        return () -&gt; {
            StreamingResponseBody streamingResponseBody = out -&gt; {
                try {
                    s3client.downloadMergedResults(jobId, out);
                } catch(Exception e) {
                    log.error(&quot;{} Error downloading results: {}&quot;, jobId, e.getMessage(), e);
                }
            };
            return new ResponseEntity&lt;&gt;(streamingResponseBody, HttpStatus.OK);
        };
    }

This is what curl says:

curl: (18) transfer closed with 524076376 bytes remaining to read

This is the stack trace from my app:

software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (java.net.SocketTimeoutException). Response Code: 200, Response Text: OK
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:108) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:72) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.13.44.jar!/:na]
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:3539) ~[s3-2.13.44.jar!/:na]
at org.me.blobstorage.S3ClientWrapper.get(S3ClientWrapper.java:362) ~[classes!/:na]
at org.me.blobstorage.S3ClientWrapper.downloadMergedResults(S3ClientWrapper.java:253) ~[classes!/:na]
at org.me.RequestHandler.lambda$null$4(RequestHandler.java:241) ~[classes!/:na]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:328) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_265]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_265]
Caused by: software.amazon.awssdk.core.exception.NonRetryableException: java.net.SocketTimeoutException
at software.amazon.awssdk.core.exception.NonRetryableException$BuilderImpl.build(NonRetryableException.java:92) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:187) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.handle(BaseSyncClientHandler.java:168) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.13.44.jar!/:na]
... 40 common frames omitted
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:113) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:99) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.core.sync.ResponseTransformer.lambda$toOutputStream$2(ResponseTransformer.java:159) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:180) ~[sdk-core-2.13.44.jar!/:na]
... 42 common frames omitted
Caused by: java.net.SocketTimeoutException: null
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1257) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:814) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:584) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:522) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
... 51 common frames omitted

Here are the different ways I've tried initialising my S3Client:

        client = S3Client.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();

        client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.credentialsProvider(credentialsProvider)
.build();

        client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClient(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)).build())
.credentialsProvider(credentialsProvider)
.build();

        client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClientBuilder(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)))
.credentialsProvider(credentialsProvider)
.build();

My spring boot applications.properties setting for request timeouts:

spring.mvc.async.request-timeout=-1

and I have @EnableAsync on my bootstrap class.

The requests are proxied through nginx, with the settings on the location as:

upstream myupstream {
server [SERVER-REDACTED]:[PORT-REDACTED];
keepalive 15;
}
server {
...
location ~ ^/(v1/bigsearch|actuator)/ {
proxy_pass http://myupstream;
proxy_http_version 1.1;
proxy_set_header Connection &quot;Keep-Alive&quot;;
proxy_set_header Proxy-Connection &quot;Keep-Alive&quot;;
client_max_body_size 2G;
}
...
}

And the relevant nginx.conf entries itself:

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 1d;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
proxy_read_timeout 1d;
proxy_send_timeout 1d;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable &quot;msie6&quot;;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

Question

What haven't I done to make this work?

答案1

得分: 0

以下是翻译好的部分：

一如既往，我为一个问题苦苦挣扎了好几天，然后将它发布到 Stack Overflow，最终找到了解决方法。

尽管错误表现为 SocketTimeoutException，但问题实际上出在 nginx 代理缓冲上。

过了一段时间，我突然意识到下载总是在 1G 处停止，直到我专门搜索了这个问题，我才在这里找到了实际的问题和解决方案：https://github.com/owncloud/client/issues/5706

将以下设置添加到 nginx.conf 文件中解决了问题：

proxy_request_buffering off;
proxy_buffering off;

英文:

As always, I struggle with a problem for days, post it on SO and then figure it out.

Even though the error manifests as a SocketTimeoutException, the problem actually lies with nginx proxy buffering.

After a while it struck me as important the download always topped out at 1G, and it wasn't until I searched for that specifically that I discovered the actual problem and solution here: https://github.com/owncloud/client/issues/5706

Adding these settings into the nginx.conf file resolved it:

proxy_request_buffering off;
proxy_buffering off;

通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。

AWS S3 v2 GetObject 无法解组响应 (java.net.SocketTimeoutException)

问题

Question

答案1

在对ImmutableMap.copyof()进行迭代时出现ConcurrentModificationException。

寻找正确的算法来找出所有可能的二进制组合。

有没有Java的函数式方法来避免级联的NonNull检查？

CriteriaBuilder.createQuery和EntityManager.createQuery之间的区别是什么？

What's the correct way to type hint an empty list as a literal in python?

如何在Highcharts Gantt中更改本地化的星期名称

如何在同一个流中使用多个过滤器和映射函数？

如何使用Map/Set来将代码优化到O(n)？

.NET MAUI Android在GitHub Actions上构建失败，错误代码为1。

如何在Playwright视觉比较中屏蔽多个定位器？

在C++中，可以使用可变模板参数来检索类型的内部类型。

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: stale element not found

Creating and opening a URL to log in to Website via Basic Auth with Robot Framework/Selenium (Python)

AG Grid 在上下文菜单中以大文本形式打开

发表评论