Node.js、MongoDB 和 Android Studio 的数据获取速度较慢。

huangapple go评论76阅读模式
英文:

nodejs mongodb and android studio slow fetch speed

问题

我有一个系统,通过Android Studio,从我的服务器提取数据并将其存储到移动SQL数据库中。它能够工作,但速度非常慢,大约需要30分钟。我的数据库中有大约86000条记录,我想从服务器中提取所有这些记录。有什么最佳方法可以做到这一点?

目前,我从服务器获取计数,然后查询服务器数据库,直到找到每个ID,然后将结果发送回我的移动应用程序。

在我的拉取请求中,我使用了一个递归函数来处理每个ID。

如何加快速度或以不同的方式加快速度?

英文:

I have a system which, pulls data from my server and stores it to a mobile SQL database via android studio. It works but it is painful slow like 30mins. I have around 86000 records in my database and want to pull all of them out of the server. What is the best way to do this?

Presently I get the count from the server and then query the server database until i find each ID and then send that result back to my mobile app.

app.post("/get_data", function(req, res)
{
	var Id_request = req.body.Id_request;//The requested ID

	var query = {Val_String : Id_request};//Query value
	//console.log(query);
	//Data.find({}, function(err, result) {//Finds all data
	Data.findOne(query, function(err, result) {//Finds all data
    if (err) {
      //res.status(400).send(err);
      console.log("Sending error");
      res.status(200).send();
    } else {
      return res.json(result);
    }
  });
});

I use a recersive function in my pull request for each ID

private void call_method()
    {
        HashMap<String, String> map = new HashMap<>();
        map.put("Id_request", Integer.toString(data_pointer));//The ID value
        Call<Fetch_result> call = retrofitInterface.executeGet_data(map);//Run the post
        call.enqueue(new Callback<Fetch_result>() {
            //call.enqueue(new Callback<Fetch_result>() {
            @Override
            public void onResponse(Call<Fetch_result> call, Response<Fetch_result> response) {
                if (response.code() == 200)//Successful login
                {
                    D1= response.body().getT1_String();
                    D2= response.body().getT2_String();
                  
                    data_pointer = data_pointer + 1;
                    boolean result = BLE_DB.addData_Downloaded(D1,D2);//Add data
    
                    if(data_pointer<= Total_entries) {//Call method again
                        call_method();//Recursive here
                    }else if (data_pointer > Total_entries){
                        Utils.toast(getApplicationContext(),"All data received");
                    }

                } else if (response.code() == 404) {
                    Utils.toast(getApplicationContext(), "Get data fail");
                }
            }

            @Override
            public void onFailure(Call<Fetch_result> call, Throwable t) {
                Utils.toast(getApplicationContext(), "Get data error");
            }
        });
    }

How Can I speed this up or do it differently to speed it up?

答案1

得分: 1

  1. 尽量一次获取尽可能多的数据(限制查询次数)。由于我不知道你的MongoDB集合是什么样的,所以很难告诉你如何做。
  2. 尽量减少请求次数。如果可以一次返回所有获取的数据,这将节省一些时间。
  3. 当在86000个文档上执行时,JSON可能会非常慢。
  4. 考虑将数据缓存以供将来的用户使用。

现在我怀疑限制你的是你正在对数据库进行86000次查询...如果你能够获取整个MongoDB集合,可能会快一些(参考注释)。

注释:
https://docs.mongodb.com/manual/reference/method/db.collection.find/#db-collection-find(省略查询参数将导致获取整个集合)

英文:
  1. Try to fetch as much data as possible at once ( limit the amount of queries you do ). It is hard to tell you how since I don't know your monogDB collection.
  2. Try to do this with as little requests as possible. If you can return all the fetched data at once, this will save you some time.
  3. JSON may be very slow when doing it on 86000 documents
  4. Consider caching the data for future users

Right now i suspect what is limiting you is the fact that you are doing 86000 queries to the db... If you can get the entire mongoDB collection it may be a bit faster( look at notes )

Notes:
https://docs.mongodb.com/manual/reference/method/db.collection.find/#db-collection-find ( omit the query parameter will result in fetching the entire collection )

huangapple
  • 本文由 发表于 2020年8月5日 19:20:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/63264096.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定