英文:
Downloading very large results from query via Java Webapp
问题
我一直在思考一个问题,并且想知道是否有人有任何建议。
我有一个网络应用程序,我需要添加一个用于报告的部分。它将根据用户 GUI 输入构建一个基于 T-SQL 查询,然后执行该查询,并允许用户通过其浏览器将文件作为 CSV 下载到他们的计算机上。单独看每个部分都不是特别复杂的。我以前做过这些事情。然而,有一个问题我以前没有解决过。查询结果可能会有数GB的数据量。与正常情况下有何不同(在代码方面),以防止浏览器超载或变得非常缓慢?显然不能一次性将所有数据都保存在内存中。如何防止发生这种情况?
英文:
I have been mulling over a bit of an issue, and I was wondering if anyone had any advice.
I have a web application that I need to add a section for reporting to. It will have a t-sql query constructed based on user gui inputs, execute, and allow the user to download the file to their computer through their browser as a csv. On it's own none of this is overly complicated. I have done these things before. There is one issue I have not grappled with before however. The results may end up as multiple GBs worth of data. What would I do differently (in terms of code) from normal to prevent it from overloading the browser or being tremendously slow? It clearly cannot all be held in memory at once? How do I prevent that from happening?
答案1
得分: 1
你可以直接将CSV数据写入传入请求的OutputStream中:
package example;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;
public class MyServlet extends HttpServlet {
private final CSVCreator csvCreator;
private MyServlet(CSVCreator csvCreator) {
this.csvCreator = csvCreator;
}
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
resp.setHeader("Content-Type", "text/comma-separated-values; charset=UTF-8");
try (Writer writer = new OutputStreamWriter(resp.getOutputStream(), StandardCharsets.UTF_8)) {
this.csvCreator.createCSV("some params you need to create the correct csv", writer);
}
}
}
你需要更改CSV生成部分,以便能够写入Writer... 例如:
package example;
import java.io.IOException;
public class CSVCreator {
public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
appendable.append("Column1,Column2,Column3\n");
appendable.append("Row1,Row1,Row1\n");
appendable.append("Row2,Row2,Row2\n");
}
}
你也可以使用apache-commons-csv来将CSV写入给定的Appendable(参见:https://javadoc.io/doc/org.apache.commons/commons-csv/latest/org/apache/commons/csv/CSVPrinter.html)
package example;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import java.io.IOException;
public class CSVCreator {
public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
printer.printRecord("Column1", "Column2", "Column3");
printer.printRecord("Row1", "Row1", "Row1");
printer.printRecord("Row2", "Row2", "Row2");
}
}
}
编辑:
你可以按照以下方式从ResultSet创建CSV文件:
package example;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import java.io.IOException;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
public class ResultSetToCSV {
public void createCSV(ResultSet rs, Appendable appendable) throws IOException, SQLException {
final ResultSetMetaData metaData = rs.getMetaData();
final int columnCount = metaData.getColumnCount();
try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
printHeaders(metaData, columnCount, printer);
printValues(rs, columnCount, printer);
}
}
private void printHeaders(ResultSetMetaData metaData, int columnCount, CSVPrinter printer) throws IOException, SQLException {
for (int i = 1; i <= columnCount; i++) {
printer.print(metaData.getColumnLabel(i));
}
printer.println();
}
private void printValues(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
while (rs.next()) {
printValue(rs, columnCount, printer);
}
}
private void printValue(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
for (int i = 1; i <= columnCount; i++) {
printer.print(rs.getString(i));
}
printer.println();
}
}
英文:
You can write the csv-data directly to the OutputStream of the incoming request:
package example;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;
public class MyServlet extends HttpServlet {
private final CSVCreator csvCreator;
private MyServlet(CSVCreator csvCreator) {
this.csvCreator = csvCreator;
}
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
// be nice to your consumer, tell them what they get
resp.setHeader("Content-Type", "text/comma-separated-values; charset=UTF-8");
try (Writer writer = new OutputStreamWriter(resp.getOutputStream(), StandardCharsets.UTF_8)) {
this.csvCreator.createCSV("some params you need to create the correct csv", writer);
}
}
}
You will have to change your csv-generation to be able to write to an Writer ... for example:
package example;
import java.io.IOException;
public class CSVCreator {
public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
appendable.append("Column1,Column2,Column3\n");
appendable.append("Row1,Row1,Row1\n");
appendable.append("Row2,Row2,Row2\n");
}
}
You could also, for example, use apache-commons-csv to write the csv to the given Appendable (see: https://javadoc.io/doc/org.apache.commons/commons-csv/latest/org/apache/commons/csv/CSVPrinter.html )
package example;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import java.io.IOException;
public class CSVCreator {
public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
printer.printRecord("Column1", "Column2", "Column3");
printer.printRecord("Row1", "Row1", "Row1");
printer.printRecord("Row2", "Row2", "Row2");
}
}
}
EDIT:
You can create a CSV-File from a ResultSet like this:
package example;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import java.io.IOException;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
public class ResultSetToCSV {
public void createCSV(ResultSet rs, Appendable appendable) throws IOException, SQLException {
final ResultSetMetaData metaData = rs.getMetaData();
final int columnCount = metaData.getColumnCount();
try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
printHeaders(metaData, columnCount, printer);
printValues(rs, columnCount, printer);
}
}
private void printHeaders(ResultSetMetaData metaData, int columnCount, CSVPrinter printer) throws IOException, SQLException {
for (int i = 1; i <= columnCount; i++) {
printer.print(metaData.getColumnLabel(i));
}
printer.println();
}
private void printValues(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
while (rs.next()) {
printValue(rs, columnCount, printer);
}
}
private void printValue(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
for (int i = 1; i <= columnCount; i++) {
printer.print(rs.getString(i));
}
printer.println();
}
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论