Java HTTP请求中的内存泄漏。

huangapple
go评论54阅读模式
英文:

Java Memory Leak in HTTP requests

问题

我有一个不断运行的Java应用程序。该应用程序向云服务器发出HTTP请求。问题在于，每次请求时内存消耗都会增加，直到达到机器完全冻结的点。我隔离了代码的一部分，我确信问题出在这段代码块上，它正在发出这些HTTP请求。通过prometheus / Grafana分析JVM数字，我看到非堆内存（代码缓存和元空间）的使用不断增加，如这里所示。

在上面的图像中，每当线条下降时，都是在内存消耗达到98%时，Monit会终止应用程序。

导致这种内存消耗的方法如下（它大约执行300次，直到在初始化过程中消耗了略多于1.5GB的可用内存）。

public AbstractRestReponse send(RestRequest request){
        BufferedReader in = null;
        OutputStream fout = null;
        URLConnection conn = null;
        InputStreamReader inputStreamReader = null;
        String result = "";
        try {
            MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
            for (String key : request.getParams().keySet()) {
                String value = (String) request.getParams().get(key);
//                System.out.println(key + " = " + value);
                mb.addTextBody(key, value);
            }
            
            if (request.getFile() != null) {
                mb.addBinaryBody("file", request.getFile());
            }
            org.apache.http.HttpEntity e = mb.build();
    
            conn = new URL(request.getUrl()).openConnection();
            conn.setDoOutput(true);
            conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
            conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
            fout = conn.getOutputStream();
            e.writeTo(fout);// write multi part data...
            
            inputStreamReader = new InputStreamReader(conn.getInputStream());
            in = new BufferedReader(inputStreamReader);
            String line;
            while ((line = in.readLine()) != null) {
                result += line;
            }
            String text = result.toString(); 
            return objectMapper.readValue(text, FacialApiResult.class);
            
        }catch (Exception e) {
            e.printStackTrace();
            return null;
        }finally {
            try {
                inputStreamReader.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            try {
                conn.getInputStream().close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            try {
                fout.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            try {
                in.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            
        }
        
    }
英文:

I have a Java application that is runs constantly. This application makes HTTP requests to a cloud server. The problem is that at each request the memory consumption increases until it reaches the point that the machine complete freezes. I isolated parts of the code and I'm sure the problem is with this code block making this http requests. Analyzing the JVM numbers, via prometheus / Grafana, I see that the use of non-heap memory (codecache and metaspace) are constantly increasing, as shown here

In the image above, whenever there is a drop in the line, it is when 98% of memory consumption reached, and Monit kills the app.

The method that is causing this memory consumption, is below (it is executed approx. 300 times until it exhausts a little more than 1.5GB of available memory in the initialization).

public AbstractRestReponse send(RestRequest request){
BufferedReader in = null;
OutputStream fout = null;
URLConnection conn = null;
InputStreamReader inputStreamReader = null;
String result = "";
try {
MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
for (String key : request.getParams().keySet()) {
String value = (String) request.getParams().get(key);
//                System.out.println(key + " = " + value);
mb.addTextBody(key, value);
}
if (request.getFile() != null) {
mb.addBinaryBody("file", request.getFile());
}
org.apache.http.HttpEntity e = mb.build();
conn = new URL(request.getUrl()).openConnection();
conn.setDoOutput(true);
conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
fout = conn.getOutputStream();
e.writeTo(fout);// write multi part data...
inputStreamReader = new InputStreamReader(conn.getInputStream());
in = new BufferedReader(inputStreamReader);
String line;
while ((line = in.readLine()) != null) {
result += line;
}
String text = result.toString(); 
return objectMapper.readValue(text, FacialApiResult.class);
}catch (Exception e) {
e.printStackTrace();
return null;
}finally {
try {
inputStreamReader.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
conn.getInputStream().close();
} catch (IOException e) {
e.printStackTrace();
}
try {
fout.close();
} catch (IOException e) {
e.printStackTrace();
}
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}

答案1

得分: 1

((HttpURLConnection)conn).disconnect() 这个部分需要注意。另外，字符串拼接会消耗时间和内存。并且在去掉换行符时有一个小bug。

如果由于异常未能到达finally块，可能会引发NullPointerException。但你应该已经检查过了。

public AbstractRestReponse send(RestRequest request) {
    URLConnection conn = null;
    try {
        MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
        for (String key : request.getParams().keySet()) {
            String value = (String) request.getParams().get(key);
            mb.addTextBody(key, value);
        }

        if (request.getFile() != null) {
            mb.addBinaryBody("file", request.getFile());
        }
        org.apache.http.HttpEntity e = mb.build();

        conn = new URL(request.getUrl()).openConnection();
        conn.setDoOutput(true);
        conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
        conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
        try (OutputStream fout = conn.getOutputStream()){
            e.writeTo(fout);// write multi part data...
        }
        
        StringBuilder result = new StringBuilder(2048);
        try (BufferedReader in = new InputStreamReader(conn.getInputStream(),
                StandardCharsets.UTF_8)) { // Charset
            String line;
            while ((line = in.readLine()) != null) {
                result.append(line).append('\n'); // 换行
            }
        }
        String text = result.toString();
        return objectMapper.readValue(text, FacialApiResult.class);

    } catch (Exception e) {
        e.printStackTrace();
        return null;
    } finally {
        if (conn != null) {
            try {
                if (conn instanceof HttpURLConnection) {
                    ((HttpURLConnection) conn).disconnect();
                }
            } catch (IOException e) {
                e.printStackTrace(); //更好的日志记录
            }
        }
    }
    return null;
}
  • 我明确定义了字符集（UTF-8 可能不对） - 暂时使用服务器的默认值。
  • 使用了StringBuilder，并添加了缺失的换行符，这可能导致错误的解析。
  • 使用了自动关闭的try-with-resources，而且稍微早些关闭。希望这不会破坏任何东西。
  • 当连接是HttpURLConnection时断开连接。注意可能在单元测试中扮演角色的instanceof。
英文:

((HttpURLConnection)conn).disconnect() comes to mind. Also String concatenation is time and memory exhaustive. And there was a minor bug in dropping newlines.

NullPointerExceptions may arise in the finally block when an open was not reached due to an exception. But you should have checked that.

public AbstractRestReponse send(RestRequest request) {
URLConnection conn = null;
try {
MultipartEntityBuilder mb = MultipartEntityBuilder.create();// org.apache.http.entity.mime
for (String key : request.getParams().keySet()) {
String value = (String) request.getParams().get(key);
mb.addTextBody(key, value);
}
if (request.getFile() != null) {
mb.addBinaryBody("file", request.getFile());
}
org.apache.http.HttpEntity e = mb.build();
conn = new URL(request.getUrl()).openConnection();
conn.setDoOutput(true);
conn.addRequestProperty(e.getContentType().getName(), e.getContentType().getValue());// header "Content-Type"...
conn.addRequestProperty("Content-Length", String.valueOf(e.getContentLength()));
try (fout = conn.getOutputStream()){
e.writeTo(fout);// write multi part data...
}
StringBuilder resullt = new StringBuilder(2048);
try (BufferedReader in = new InputStreamReader(conn.getInputStream(),
StandardCharsets.UTF_8)) { // Charset
String line;
while ((line = in.readLine()) != null) {
result.append(line).append('\n'); // Newline
}
}
String text = result.toString();
return objectMapper.readValue(text, FacialApiResult.class);
} catch (Exception e) {
e.printStackTrace();
return null;
} finally {
if (conn != null) {
try {
if (conn instanceof HttpURLConnection) {
((HttpURLConnection) conn).disconnect();
}
} catch (IOException e) {
e.printStackTrace(); //Better logger
}
}
}
return null;
}
  • I explicitly defined the charset (UTF-8 might be wrong) - momentarily it is the server's default.
  • Used a StringBuilder, and added the missing newline, which might have lead to wrong parsing.
  • Try-with-resources for auto-closing, and a bit earlier. Hopefully this does not break anything.
  • Disconnecting the connection when it is an HttpURLConnection. Mind the instanceof which might play a role in unit tests mocking.

答案2

得分: 0

你似乎已经在finally块中处理了所有可能的关闭部分。无论如何，如果你的应用程序在Java 7+上运行，最好使用try-with资源来安全关闭所有Closeable对象。如果这样做不解决问题，可能会进一步隔离问题。

英文:

You seems to have handled all possible closing part in the finally block. Anyway it's better to use try-with resources to safely close all Closeable objects, if your application is running on Java 7+. That may isolate the problem further if it doesn't fix.

huangapple
  • 本文由 发表于 2020年7月30日 01:13:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/63158977.html
  • java
  • memory-leaks
  • urlconnection
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定