尝试读取超过 700k 的数据时发生了“GC Overhead Limit Exceeded”错误。

huangapple go评论59阅读模式
英文:

Trying to read 700k+ of data and the Error "GC Overhead Limit Exceeded" occurred

问题

public class Demo2 {
    public static void main(String[] args) {	

        String url = "jdbc:mysql://localhost:3306/database";
        String username = "root";
        String password = "password";

        try {
            //Connect to the database
            Connection connection = DriverManager.getConnection(url, username, password);

            //Test on one table only
            String tableName = "Tickets";

            System.out.println("Connecting to TABLE " + tableName + "...");
            readCSVFile(tableName, connection);

            System.out.println();
            System.out.println("THE END");

            connection.close(); //close connection to the database
        }
        catch (SQLException e) {
            System.out.println("ERROR at main(): SQLException!!");
            e.printStackTrace();
        }
    }

    static int countNewRow = 0;
    static int countUpdatedRow = 0;

    //Method to read the CSV File
    static void readCSVFile(String tableName, Connection conn) {

        //Read CSV File
        try {
            String path = tableName + ".csv";

            BufferedReader br = new BufferedReader(new FileReader(path));
            br.readLine(); //skip the first line
            String inData;

            //Read The Remaining Line
            while ((inData = br.readLine()) != null) {
                String[] rowData = inData.split(",");
                ArrayList<String> rowDataList = new ArrayList<String>();

                for (int i = 0; i < rowData.length; i++)
                    rowDataList.add(rowData[i]);

                //To combine String that starts and ends with "
                for (int i = 0; i < rowDataList.size(); i++) {
                    if (rowDataList.get(i).charAt(0) == '"') {
                        String string1 = rowDataList.get(i).substring(1, rowDataList.get(i).length());
                        String string2 = rowDataList.get(i + 1).substring(0, rowDataList.get(i + 1).length() - 1);
                        String combined = string1 + "," + string2;

                        rowDataList.set(i, combined);
                        rowDataList.remove(i + 1);

                        break;
                    }
                }

                //Remove the RM
                for (int i = 0; i < rowDataList.size(); i++) {
                    if (rowDataList.get(i).startsWith("RM")) {
                        String string = rowDataList.get(i).substring(2);
                        rowDataList.set(i, string);
                    }
                }

                //This is just to keep track of the data that has been read
                System.out.println("[" + rowDataList.get(0) + "]");

                //Transfer the data to the database
                insertToDatabase(conn, tableName, rowDataList);
            }

            System.out.println("New Row Added : " + countNewRow);
            System.out.println("Updated Row   : " + countUpdatedRow);
            System.out.println("== Process Completed ==");
            br.close();
        } catch (FileNotFoundException e) {
            System.out.println("ERROR at readCSVFile(): FileNotFoundException!!");
            e.printStackTrace();
        } catch (IOException e) {
            System.out.println("ERROR at readCSVFile(): IOException!!");
            e.printStackTrace();
        } catch (SQLException e) {
            System.out.println("ERROR at readCSVFile(): SQLException!!");
            e.printStackTrace();
        } catch (ParseException e) {
            System.out.println("ERROR at readCSVFile(): ParseException!!");
            e.printStackTrace();
        }
    }

    static void insertToDatabase(Connection connection, String tableName, ArrayList<String> rowDataList)
            throws SQLException, ParseException {

        String tableIdName = tableName;
        if (tableIdName.charAt(tableIdName.length() - 1) == 's')
            tableIdName = tableIdName.substring(0, tableIdName.length() - 1);

        //To read row
        String rowID = rowDataList.get(0);
        String selectSQL = "SELECT * FROM " + tableName + " " + "WHERE " + tableIdName + "_ID = " + rowID;

        Statement statement = connection.createStatement();
        ResultSet result = statement.executeQuery(selectSQL);

        boolean value = result.next();

        //INSERT @ UPDATE row
        if (value == true) { //Update Row if the data is already existed
            updateStatementt(tableName, connection, rowDataList);
            countUpdatedRow++;
        } else { //Insert New Row
            insertStatementt(tableName, connection, rowDataList);
            countNewRow++;
        }
    }

    //Method to insert data to the database
    static void insertStatementt(String tableType, Connection conn, ArrayList<String> rowDataList)
            throws SQLException, ParseException {

        //Generate Question Mark
        String generateQuestionMark = null;

        if (rowDataList.size() == 1)
            generateQuestionMark = "?";
        else
            generateQuestionMark = "?, ";

        for (int i = 1; i < rowDataList.size(); i++) {
            if (i != rowDataList.size() - 1)
                generateQuestionMark += "?, ";
            else
                generateQuestionMark += "?";
        }

        //Insert sql
        String sql = "INSERT INTO " + tableType + " VALUES (" + generateQuestionMark + ")";

        PreparedStatement insertStatement = conn.prepareStatement(sql);

        //Insert data

        //There are other 'if' and 'else if' statements here for other tables

        else if (tableType.equals("Tickets")) {
            int ticketID = Integer.parseInt(rowDataList.get(0));
            int movieId = Integer.parseInt(rowDataList.get(1));
            int theaterId = Integer.parseInt(rowDataList.get(2));
            String[] date = rowDataList.get(3).split("/");
            String dateString = date[2] + "-" + date[1] + "-" + date[0];
            Date showDate = Date.valueOf(dateString);
            int showTimeId = Integer.parseInt(rowDataList.get(4));
            int cptId = Integer.parseInt(rowDataList.get(5));
            int pcId = Integer.parseInt(rowDataList.get(6));
            float amountPaid = Float.parseFloat(rowDataList.get(7));
            int year = Integer.parseInt(rowDataList.get(8));
            String month = rowDataList.get(9);

            insertStatement.setInt(1, ticketID);
            insertStatement.setInt(2, movieId);
            insertStatement.setInt(3, theaterId);
            insertStatement.setDate(4, showDate);
            insertStatement.setInt(5, showTimeId);
            insertStatement.setInt(6, cptId);
            insertStatement.setInt(7, pcId);
            insertStatement.setFloat(8, amountPaid);
            insertStatement.setInt(9, year);
            insertStatement.setString(10, month);
        }

        insertStatement.executeUpdate();
        insertStatement.close();
    }

    //Method to update the data from the database
    static void updateStatementt(String tableType, Connection conn, ArrayList<String> rowDataList)
            throws SQLException {
        Statement statement = conn.createStatement();
        String sql = "UPDATE " + tableType;

        //There are other 'if' and 'else if' statements here for other tables

        else if (tableType.equals("Tickets")) {
            String[] date = rowDataList.get(3).split("/");
            String dateString = date[2] + "-" + date[1] + "-" + date[0];

            sql += " SET movie_id = " + rowDataList.get(1) + ","
               

<details>
<summary>英文:</summary>

Alright so I need help in reviewing my codes because I&#39;m kinda still new in programming (currently in my second year of Diploma in Computer Science). I got this error as in the title ```GC Overhead Limit Exceeded``` when I tried running my code below.

A brief explanation of this code, I&#39;m trying to read data from a CSV File and then transfer it to a database. FYI, there are actually 10 tables/CSV files that I need to read, but on this I&#39;ll show this one table ```Tickets``` because the error only occurred when I tried to read that table/file. The other tables have hundreds of rows/data only while the table ```Tickets``` have 735,504 of rows/data. Furthermore, I&#39;ve succeeded in reading 450,028 of data after 6 hours of running the code before the error occurred.

What can I do to fix this error? What can be modified to improve my code? I really appreciate it if you guys can help me :)

public class Demo2 {
public static void main(String[] args) {

	String url = &quot;jdbc:mysql://localhost:3306/database&quot;;
String username = &quot;root&quot;;
String password = &quot;password&quot;;
try {
//Connect to the database
Connection connection = DriverManager.getConnection(url, username, password);
//Test on one table only
String tableName = &quot;Tickets&quot;;
System.out.println(&quot;Connecting to TABLE &quot; +tableName +&quot;...&quot;);
readCSVFile(tableName, connection);
System.out.println();
System.out.println(&quot;THE END&quot;);
connection.close();//close connection to the database
}
catch (SQLException e) {
System.out.println(&quot;ERROR at main(): SQLException!!&quot;);
e.printStackTrace();
}
}
static int countNewRow = 0;
static int countUpdatedRow = 0;
//Method to read the CSV File
static void readCSVFile(String tableName, Connection conn) {
//Read CSV File
try {
String path = tableName +&quot;.csv&quot;;
BufferedReader br = new BufferedReader(new FileReader(path));
br.readLine();//skip the first line
String inData;
//Read The Remaining Line
while((inData=br.readLine()) != null)
{
String[] rowData = inData.split(&quot;,&quot;);
ArrayList &lt;String&gt; rowDataList = new ArrayList&lt;String&gt;();
for (int i=0; i&lt;rowData.length; i++)
rowDataList.add(rowData[i]);
//To combine String that starts and ends with &quot;
for(int i=0; i&lt;rowDataList.size(); i++) {
if (rowDataList.get(i).charAt(0) == &#39;&quot;&#39;) {
String string1 = rowDataList.get(i).substring(1, rowDataList.get(i).length());
String string2 = rowDataList.get(i+1).substring(0, rowDataList.get(i+1).length()-1);
String combined = string1 +&quot;,&quot; +string2;
rowDataList.set(i, combined); 
rowDataList.remove(i+1);
break;
}
}
//Remove the RM
for(int i=0; i&lt;rowDataList.size(); i++) {
if (rowDataList.get(i).startsWith(&quot;RM&quot;)) {
String string = rowDataList.get(i).substring(2);
rowDataList.set(i, string);
}
}
//This is just to keep track of the data that has been read
System.out.println(&quot;[&quot; +rowDataList.get(0) +&quot;]&quot;);
//Transfer the data to the database
insertToDatabase(conn, tableName, rowDataList);
}
System.out.println(&quot;New Row Added : &quot; +countNewRow);
System.out.println(&quot;Updated Row   : &quot; +countUpdatedRow);
System.out.println(&quot;== Process Completed ==&quot;);
br.close();
}
catch (FileNotFoundException e) {
System.out.println(&quot;ERROR at readCSVFile(): FileNotFoundException!!&quot;);
e.printStackTrace();
}
catch (IOException e) {
System.out.println(&quot;ERROR at readCSVFile(): IOException!!&quot;);
e.printStackTrace();
}
catch (SQLException e) {
System.out.println(&quot;ERROR at readCSVFile(): SQLException!!&quot;);
e.printStackTrace();
}
catch (ParseException e) {
System.out.println(&quot;ERROR at readCSVFile(): ParseException!!&quot;);
e.printStackTrace();
}
}
static void insertToDatabase(Connection connection, String tableName, ArrayList &lt;String&gt; rowDataList) throws SQLException, ParseException {
String tableIdName = tableName;
if (tableIdName.charAt(tableIdName.length()-1) == &#39;s&#39;)
tableIdName = tableIdName.substring(0, tableIdName.length()-1);
//To read row
String rowID = rowDataList.get(0);
String selectSQL = &quot;SELECT * FROM &quot; +tableName +&quot; &quot;
+&quot;WHERE &quot; +tableIdName +&quot;_ID = &quot; +rowID;
Statement statement = connection.createStatement();
ResultSet result = statement.executeQuery(selectSQL);
boolean value = result.next();
//INSERT @ UPDATE row
if (value == true) { //Update Row if the data is already existed
updateStatementt(tableName, connection, rowDataList);
countUpdatedRow++;
}
else { //Insert New Row
insertStatementt(tableName, connection, rowDataList);
countNewRow++;
}
}
//Method to insert data to the database
static void insertStatementt(String tableType, Connection conn, ArrayList &lt;String&gt; rowDataList) throws SQLException, ParseException {
//Generate Question Mark
String generateQuestionMark = null;
if(rowDataList.size() == 1)
generateQuestionMark = &quot;?&quot;;
else
generateQuestionMark = &quot;?, &quot;;
for(int i=1; i&lt;rowDataList.size(); i++) {
if(i!=rowDataList.size()-1)
generateQuestionMark += &quot;?, &quot;;
else
generateQuestionMark += &quot;?&quot;;
}
//Insert sql
String sql = &quot;INSERT INTO &quot; +tableType +&quot; VALUES (&quot; +generateQuestionMark +&quot;)&quot;;
PreparedStatement insertStatement = conn.prepareStatement(sql);
//Insert data
//There are other &#39;if&#39; and &#39;else if&#39; statements here for other tables
else if (tableType.equals(&quot;Tickets&quot;)) {
int ticketID = Integer.parseInt(rowDataList.get(0));
int movieId = Integer.parseInt(rowDataList.get(1));
int theaterId = Integer.parseInt(rowDataList.get(2));
String[] date = rowDataList.get(3).split(&quot;/&quot;);
String dateString = date[2] +&quot;-&quot; +date[1] +&quot;-&quot; +date[0];
Date showDate = Date.valueOf(dateString);
int showTimeId = Integer.parseInt(rowDataList.get(4));
int cptId = Integer.parseInt(rowDataList.get(5));
int pcId = Integer.parseInt(rowDataList.get(6));
float amountPaid = Float.parseFloat(rowDataList.get(7));
int year = Integer.parseInt(rowDataList.get(8));
String month = rowDataList.get(9);
insertStatement.setInt(1, ticketID);
insertStatement.setInt(2, movieId);
insertStatement.setInt(3, theaterId);
insertStatement.setDate(4, showDate);
insertStatement.setInt(5, showTimeId);
insertStatement.setInt(6, cptId);
insertStatement.setInt(7, pcId);
insertStatement.setFloat(8, amountPaid);
insertStatement.setInt(9, year);
insertStatement.setString(10, month);
}
insertStatement.executeUpdate();		
insertStatement.close();
}
//Method to update the data from the database
static void updateStatementt(String tableType, Connection conn, ArrayList &lt;String&gt; rowDataList) throws SQLException {
Statement  statement = conn.createStatement();
String sql = &quot;UPDATE &quot; +tableType;
//There are other &#39;if&#39; and &#39;else if&#39; statements here for other tables
else if (tableType.equals(&quot;Tickets&quot;)) {
String[] date = rowDataList.get(3).split(&quot;/&quot;);
String dateString = date[2] +&quot;-&quot; +date[1] +&quot;-&quot; +date[0];
sql += &quot; SET movie_id = &quot; +rowDataList.get(1) +&quot;,&quot;
+  &quot; theater_id = &quot; +rowDataList.get(2) +&quot;,&quot;
+  &quot; showdate = \&quot;&quot; +dateString +&quot;\&quot;,&quot;
+  &quot; showtime_id = &quot; +rowDataList.get(4) +&quot;,&quot;
+  &quot; costperticket_id = &quot; +rowDataList.get(5) +&quot;,&quot;
+  &quot; personcategory_id = &quot; +rowDataList.get(6) +&quot;,&quot;
+  &quot; amount_paid = &quot; +rowDataList.get(7) +&quot;,&quot;
+  &quot; year = &quot; +rowDataList.get(8) +&quot;,&quot;
+  &quot; month = \&quot;&quot; +rowDataList.get(9) +&quot;\&quot;&quot;
+  &quot; WHERE ticket_id = &quot; +rowDataList.get(0);
}
statement.executeUpdate(sql);
}

}


</details>
# 答案1
**得分**: 1
对于简短的情况,只需读取一行并按照您的意愿处理。您的内存不足以容纳所有的 70 万行。
<details>
<summary>英文:</summary>
For short, read a single line and do whatever you want to do with it. You don&#39;t have enough memory for all 700k lines.
</details>
# 答案2
**得分**: 0
如果您真的想将所有这些数据读入Java堆中,可以通过例如使用`-Xmx`命令行开关来增加堆大小。由于文本数据在JVM中的编码方式,您可能需要比总数据大小更多的堆空间。
此外,您的代码中可能有一些地方可以减轻JVM内存管理系统的负担。例如,使用“+”连接字符串可能会生成大量临时数据,这将增加垃圾收集器的负担。使用`StringBuilder`组装字符串可能是一个简单、对资源要求较少的替代方法。
<details>
<summary>英文:</summary>
If you really want to read all this data into the Java heap, increase the heap size using, for example, the `-Xmx` command-line switch. Because of the way textual data is encoded in the JVM, you&#39;ll probably need much more heap that the total data size would suggest.
In addition, there might be some places in your code where you can take the strain off the JVM&#39;s memory management system. For example, concatenating strings using &quot;+&quot; can generate a lot of temporary data, which will increase the load on the garbage collector. Assembling strings using a `StringBuilder` might be a simple, less resource-hungry, alternative.
</details>
# 答案3
**得分**: 0
你应该为更新语句添加```statement.close()```。
<details>
<summary>英文:</summary>
You should add ```statement.close()``` for the update Statement.
</details>

huangapple
  • 本文由 发表于 2020年9月10日 19:27:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/63828727.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定