英文:
nodejs mongodb and android studio slow fetch speed
问题
我有一个系统,通过Android Studio,从我的服务器提取数据并将其存储到移动SQL数据库中。它能够工作,但速度非常慢,大约需要30分钟。我的数据库中有大约86000条记录,我想从服务器中提取所有这些记录。有什么最佳方法可以做到这一点?
目前,我从服务器获取计数,然后查询服务器数据库,直到找到每个ID,然后将结果发送回我的移动应用程序。
在我的拉取请求中,我使用了一个递归函数来处理每个ID。
如何加快速度或以不同的方式加快速度?
英文:
I have a system which, pulls data from my server and stores it to a mobile SQL database via android studio. It works but it is painful slow like 30mins. I have around 86000 records in my database and want to pull all of them out of the server. What is the best way to do this?
Presently I get the count from the server and then query the server database until i find each ID and then send that result back to my mobile app.
app.post("/get_data", function(req, res)
{
var Id_request = req.body.Id_request;//The requested ID
var query = {Val_String : Id_request};//Query value
//console.log(query);
//Data.find({}, function(err, result) {//Finds all data
Data.findOne(query, function(err, result) {//Finds all data
if (err) {
//res.status(400).send(err);
console.log("Sending error");
res.status(200).send();
} else {
return res.json(result);
}
});
});
I use a recersive function in my pull request for each ID
private void call_method()
{
HashMap<String, String> map = new HashMap<>();
map.put("Id_request", Integer.toString(data_pointer));//The ID value
Call<Fetch_result> call = retrofitInterface.executeGet_data(map);//Run the post
call.enqueue(new Callback<Fetch_result>() {
//call.enqueue(new Callback<Fetch_result>() {
@Override
public void onResponse(Call<Fetch_result> call, Response<Fetch_result> response) {
if (response.code() == 200)//Successful login
{
D1= response.body().getT1_String();
D2= response.body().getT2_String();
data_pointer = data_pointer + 1;
boolean result = BLE_DB.addData_Downloaded(D1,D2);//Add data
if(data_pointer<= Total_entries) {//Call method again
call_method();//Recursive here
}else if (data_pointer > Total_entries){
Utils.toast(getApplicationContext(),"All data received");
}
} else if (response.code() == 404) {
Utils.toast(getApplicationContext(), "Get data fail");
}
}
@Override
public void onFailure(Call<Fetch_result> call, Throwable t) {
Utils.toast(getApplicationContext(), "Get data error");
}
});
}
How Can I speed this up or do it differently to speed it up?
答案1
得分: 1
- 尽量一次获取尽可能多的数据(限制查询次数)。由于我不知道你的MongoDB集合是什么样的,所以很难告诉你如何做。
- 尽量减少请求次数。如果可以一次返回所有获取的数据,这将节省一些时间。
- 当在86000个文档上执行时,JSON可能会非常慢。
- 考虑将数据缓存以供将来的用户使用。
现在我怀疑限制你的是你正在对数据库进行86000次查询...如果你能够获取整个MongoDB集合,可能会快一些(参考注释)。
注释:
https://docs.mongodb.com/manual/reference/method/db.collection.find/#db-collection-find(省略查询参数将导致获取整个集合)
英文:
- Try to fetch as much data as possible at once ( limit the amount of queries you do ). It is hard to tell you how since I don't know your monogDB collection.
- Try to do this with as little requests as possible. If you can return all the fetched data at once, this will save you some time.
- JSON may be very slow when doing it on 86000 documents
- Consider caching the data for future users
Right now i suspect what is limiting you is the fact that you are doing 86000 queries to the db... If you can get the entire mongoDB collection it may be a bit faster( look at notes )
Notes:
https://docs.mongodb.com/manual/reference/method/db.collection.find/#db-collection-find ( omit the query parameter will result in fetching the entire collection )
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论