从Java Webapp下载非常大的查询结果

huangapple go评论115阅读模式
英文:

Downloading very large results from query via Java Webapp

问题

我一直在思考一个问题,并且想知道是否有人有任何建议。
我有一个网络应用程序,我需要添加一个用于报告的部分。它将根据用户 GUI 输入构建一个基于 T-SQL 查询,然后执行该查询,并允许用户通过其浏览器将文件作为 CSV 下载到他们的计算机上。单独看每个部分都不是特别复杂的。我以前做过这些事情。然而,有一个问题我以前没有解决过。查询结果可能会有数GB的数据量。与正常情况下有何不同(在代码方面),以防止浏览器超载或变得非常缓慢?显然不能一次性将所有数据都保存在内存中。如何防止发生这种情况?

英文:

I have been mulling over a bit of an issue, and I was wondering if anyone had any advice.
I have a web application that I need to add a section for reporting to. It will have a t-sql query constructed based on user gui inputs, execute, and allow the user to download the file to their computer through their browser as a csv. On it's own none of this is overly complicated. I have done these things before. There is one issue I have not grappled with before however. The results may end up as multiple GBs worth of data. What would I do differently (in terms of code) from normal to prevent it from overloading the browser or being tremendously slow? It clearly cannot all be held in memory at once? How do I prevent that from happening?

答案1

得分: 1

你可以直接将CSV数据写入传入请求的OutputStream中:

package example;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;

public class MyServlet extends HttpServlet {

	private final CSVCreator csvCreator;

	private MyServlet(CSVCreator csvCreator) {
		this.csvCreator = csvCreator;
	}

	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		resp.setHeader("Content-Type", "text/comma-separated-values; charset=UTF-8");

		try (Writer writer = new OutputStreamWriter(resp.getOutputStream(), StandardCharsets.UTF_8)) {
			this.csvCreator.createCSV("some params you need to create the correct csv", writer);
		}
	}
}

你需要更改CSV生成部分,以便能够写入Writer... 例如:

package example;

import java.io.IOException;

public class CSVCreator {

	public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
		appendable.append("Column1,Column2,Column3\n");
		appendable.append("Row1,Row1,Row1\n");
		appendable.append("Row2,Row2,Row2\n");
	}
}

你也可以使用apache-commons-csv来将CSV写入给定的Appendable(参见:https://javadoc.io/doc/org.apache.commons/commons-csv/latest/org/apache/commons/csv/CSVPrinter.html)

package example;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;

import java.io.IOException;

public class CSVCreator {

	public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
		try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
			printer.printRecord("Column1", "Column2", "Column3");
			printer.printRecord("Row1", "Row1", "Row1");
			printer.printRecord("Row2", "Row2", "Row2");
		}
	}
}

编辑:

你可以按照以下方式从ResultSet创建CSV文件:

package example;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;

import java.io.IOException;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;

public class ResultSetToCSV {

	public void createCSV(ResultSet rs, Appendable appendable) throws IOException, SQLException {
		final ResultSetMetaData metaData = rs.getMetaData();
		final int columnCount = metaData.getColumnCount();

		try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
			printHeaders(metaData, columnCount, printer);
			printValues(rs, columnCount, printer);
		}
	}

	private void printHeaders(ResultSetMetaData metaData, int columnCount, CSVPrinter printer) throws IOException, SQLException {
		for (int i = 1; i <= columnCount; i++) {
			printer.print(metaData.getColumnLabel(i));
		}
		
		printer.println();
	}

	private void printValues(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
		while (rs.next()) {
			printValue(rs, columnCount, printer);
		}
	}
	
	private void printValue(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
		for (int i = 1; i <= columnCount; i++) {
			printer.print(rs.getString(i));
		}
		
		printer.println();
	}
}
英文:

You can write the csv-data directly to the OutputStream of the incoming request:

package example;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;

public class MyServlet extends HttpServlet {

	private final CSVCreator csvCreator;

	private MyServlet(CSVCreator csvCreator) {
		this.csvCreator = csvCreator;
	}

	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		// be nice to your consumer, tell them what they get
		resp.setHeader(&quot;Content-Type&quot;, &quot;text/comma-separated-values; charset=UTF-8&quot;);

		try (Writer writer = new OutputStreamWriter(resp.getOutputStream(), StandardCharsets.UTF_8)) {
			this.csvCreator.createCSV(&quot;some params you need to create the correct csv&quot;, writer);
		}
	}
}

You will have to change your csv-generation to be able to write to an Writer ... for example:

package example;

import java.io.IOException;

public class CSVCreator {

	public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
		appendable.append(&quot;Column1,Column2,Column3\n&quot;);
		appendable.append(&quot;Row1,Row1,Row1\n&quot;);
		appendable.append(&quot;Row2,Row2,Row2\n&quot;);
	}
}

You could also, for example, use apache-commons-csv to write the csv to the given Appendable (see: https://javadoc.io/doc/org.apache.commons/commons-csv/latest/org/apache/commons/csv/CSVPrinter.html )

package example;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;

import java.io.IOException;

public class CSVCreator {

	public void createCSV(String someParamsYouNeedToCreateTheCSV, Appendable appendable) throws IOException {
		try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
			printer.printRecord(&quot;Column1&quot;, &quot;Column2&quot;, &quot;Column3&quot;);
			printer.printRecord(&quot;Row1&quot;, &quot;Row1&quot;, &quot;Row1&quot;);
			printer.printRecord(&quot;Row2&quot;, &quot;Row2&quot;, &quot;Row2&quot;);
		}
	}
}

EDIT:

You can create a CSV-File from a ResultSet like this:

package example;

import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;

import java.io.IOException;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;

public class ResultSetToCSV {

	public void createCSV(ResultSet rs, Appendable appendable) throws IOException, SQLException {
		final ResultSetMetaData metaData = rs.getMetaData();
		final int columnCount = metaData.getColumnCount();

		try (CSVPrinter printer = new CSVPrinter(appendable, CSVFormat.DEFAULT)) {
			printHeaders(metaData, columnCount, printer);
			printValues(rs, columnCount, printer);
		}
	}

	private void printHeaders(ResultSetMetaData metaData, int columnCount, CSVPrinter printer) throws IOException, SQLException {
		for (int i = 1; i &lt;= columnCount; i++) {
			printer.print(metaData.getColumnLabel(i));
		}
		
		printer.println();
	}

	private void printValues(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
		while (rs.next()) {
			printValue(rs, columnCount, printer);
		}
	}
	
	private void printValue(ResultSet rs, int columnCount, CSVPrinter printer) throws IOException, SQLException {
		for (int i = 1; i &lt;= columnCount; i++) {
			printer.print(rs.getString(i));
		}
		
		printer.println();
	}
}

huangapple
  • 本文由 发表于 2020年3月4日 05:34:52
  • 转载请务必保留本文链接:https://go.coder-hub.com/60515873.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定