英文:
Jooq Java Code to handle more than 200k records from a table
问题
我的翻译如下：
我的数据库中的表有超过200,000条记录，我不想一次性从resultSet中获取所有记录进行处理。我正在尝试编写一个服务中的函数，调用另一个持久化类的函数，只获取1000条记录进行处理。
一旦处理完这1000条记录，它应该获取接下来的1000条记录。我在我的Java代码中使用了JOOQ数据库库。我只是分享了我脑海中的一个示例代码框架。
class RecordService {RecordPersistence recordPersistence = new RecordPersistence();public void processRecords() {List<Record> records = recordPersistence.fetchRecords();// 处理记录的代码}}class RecordPersistence {public List<Record> fetchRecords(){Cursor<Record> records = dsl.select...fetchLazy();while (records.hasNext()) {records.fetchNext(1000);}return records;}}
如何从fetchRecords()函数中仅返回少量记录？我应该将其编写为异步函数吗？这是处理结果的正确方式吗？还是有更好的处理方式来处理我的情况？
英文:
The table in my database has more than 200k records and I don't want to fetch all of them at once in the resultSet to process. I am trying to write a function in a service that calls another persistence class function to bring only 1000 records to process.
Once it processes the 1000 records, it should fetch the following thousand records. I am using the JOOQ database library in my Java code. I am just sharing a sample skeleton code that I had in my mind.
class RecordService {RecordPersistence recordPersistence = new RecordPersistence();public void processRecords() {List<Record> records = recordPersistence.fetchRecords();// Code to process them}}class RecordPersistence {public List<Record> fetchRecords(){Cursor<Record> records = dsl.select...fetchLazy();while (records.hasNext()) {records.fetchNext(1000);}return records}}
How to return only a few records from the fetchRecords() function? Should I write this is an async function? Is this the correct way to process result? or is there a better way to handle my case?
答案1
得分: 0
如评论中所提到的，问题在于你将处理逻辑放在了获取逻辑之外。只需将其移动到内部，例如：
class RecordService {RecordPersistence recordPersistence = new RecordPersistence();public void processRecords() {recordPersistence.fetchRecords((List<Record> records) -> {// 处理记录的代码});}}class RecordPersistence {public List<Record> fetchRecords(Consumer<? super List<Record>> consumer) {try (Cursor<Record> cursor = dsl.select...fetchLazy()) {while (cursor.hasNext()) {consumer.accept(cursor.fetchNext(1000));}}}}
对于这种类型的流式处理，你不需要进行任何范式转换（例如异步、响应式等）。只需从外部迭代切换到内部迭代即可。
英文:
As mentioned in the comments, the problem is that you're keeping the processing logic outside of the fetch logic. Just move it inside, e.g. as follows:
class RecordService {RecordPersistence recordPersistence = new RecordPersistence();public void processRecords() {recordPersistence.fetchRecords((List<Record> records) -> {// Code to process them});}}class RecordPersistence {public List<Record> fetchRecords(Consumer<? super List<Record>> consumer) {try (Cursor<Record> cursor = dsl.select...fetchLazy()) {while (cursor.hasNext()) {consumer.accept(cursor.fetchNext(1000));}}}}
You don't need any paradigm shift (e.g. async, reactive, etc.) for this type of streaming to work. Just switch from external to internal iteration
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论