英文:
Jooq Java Code to handle more than 200k records from a table
问题
我的翻译如下：
我的数据库中的表有超过200,000条记录，我不想一次性从resultSet中获取所有记录进行处理。我正在尝试编写一个服务中的函数，调用另一个持久化类的函数，只获取1000条记录进行处理。
一旦处理完这1000条记录，它应该获取接下来的1000条记录。我在我的Java代码中使用了JOOQ数据库库。我只是分享了我脑海中的一个示例代码框架。
class RecordService {
RecordPersistence recordPersistence = new RecordPersistence();
public void processRecords() {
List<Record> records = recordPersistence.fetchRecords();
// 处理记录的代码
}
}
class RecordPersistence {
public List<Record> fetchRecords(){
Cursor<Record> records = dsl.select...fetchLazy();
while (records.hasNext()) {
records.fetchNext(1000);
}
return records;
}
}
如何从fetchRecords()函数中仅返回少量记录？我应该将其编写为异步函数吗？这是处理结果的正确方式吗？还是有更好的处理方式来处理我的情况？
英文:
The table in my database has more than 200k records and I don't want to fetch all of them at once in the resultSet to process. I am trying to write a function in a service that calls another persistence class function to bring only 1000 records to process.
Once it processes the 1000 records, it should fetch the following thousand records. I am using the JOOQ database library in my Java code. I am just sharing a sample skeleton code that I had in my mind.
class RecordService {
RecordPersistence recordPersistence = new RecordPersistence();
public void processRecords() {
List<Record> records = recordPersistence.fetchRecords();
// Code to process them
}
}
class RecordPersistence {
public List<Record> fetchRecords(){
Cursor<Record> records = dsl.select...fetchLazy();
while (records.hasNext()) {
records.fetchNext(1000);
}
return records
}
}
How to return only a few records from the fetchRecords() function? Should I write this is an async function? Is this the correct way to process result? or is there a better way to handle my case?
答案1
得分: 0
如评论中所提到的，问题在于你将处理逻辑放在了获取逻辑之外。只需将其移动到内部，例如：
class RecordService {
RecordPersistence recordPersistence = new RecordPersistence();
public void processRecords() {
recordPersistence.fetchRecords(
(List<Record> records) -> {
// 处理记录的代码
}
);
}
}
class RecordPersistence {
public List<Record> fetchRecords(Consumer<? super List<Record>> consumer) {
try (Cursor<Record> cursor = dsl.select...fetchLazy()) {
while (cursor.hasNext()) {
consumer.accept(cursor.fetchNext(1000));
}
}
}
}
对于这种类型的流式处理，你不需要进行任何范式转换（例如异步、响应式等）。只需从外部迭代切换到内部迭代即可。
英文:
As mentioned in the comments, the problem is that you're keeping the processing logic outside of the fetch logic. Just move it inside, e.g. as follows:
class RecordService {
RecordPersistence recordPersistence = new RecordPersistence();
public void processRecords() {
recordPersistence.fetchRecords(
(List<Record> records) -> {
// Code to process them
}
);
}
}
class RecordPersistence {
public List<Record> fetchRecords(Consumer<? super List<Record>> consumer) {
try (Cursor<Record> cursor = dsl.select...fetchLazy()) {
while (cursor.hasNext()) {
consumer.accept(cursor.fetchNext(1000));
}
}
}
}
You don't need any paradigm shift (e.g. async, reactive, etc.) for this type of streaming to work. Just switch from external to internal iteration
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论