App Engine (Flask) memory limit: how should I cache "large" (3 MB) database calls? How can I monitor memory usage on a local server or during testing?

For some time now, I've been running into this error on my current project, golfcourse.wiki (App Engine F1, Python 3.9, Flask 2.0.2):

Exceeded hard memory limit of 384 MiB with 395 MiB after servicing 39 requests total. Consider setting a larger instance class in app.yaml.

This has been an issue more and more as my database has been growing. I thought it might be an issue of database calls (which I clearly understand is wrong now), so I added the builtin memcache to the system. This has been a huge pain, as adding a wsgi wrapper has make all my tests break, and I'm going to need to rewrite the __init__.py code using the app factory pattern and then go through the entire code rewriting app calls as current_app calls (haven't gotten around to that yet).

Unfortunately, the process of this has actually increased the number of times the app crashes:

Exceeded hard memory limit of 384 MiB with 424 MiB after servicing 539 requests total. Consider setting a larger instance class in app.yaml.

While doing this, I realized some of my larger database calls (3 MB) were over the memcache limit, and so I wrote the common serialization workaround to deal with it and my memory exploded:

Exceeded hard memory limit of 384 MiB with 904 MiB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.

I figured I'd bite the bullet and upgrade to F2... I couldn't believe it when it caused the memory usage to go up by 20%:

Exceeded hard memory limit of 768 MiB with 1105 MiB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.

I'm deeply confused. I cannot find anything related to this in the App Engine docs (I've really looked). I'm admittedly an amateur at this stuff, even as I've been working with python for over a decade.

Can someone explain to me:

1. What is actually happening here? I understand that without sharing my entire code base, that nobody can point out the inefficiencies, but I'm assuming now that I'm running up against the RSS limit? Because when I run the process locally, I'm not coming anywhere close to that! As far as I can tell the program, while running locally, is maybe reaching 150MiB.
2. How can I monitor this with testing?
3. Should I even worry about the number of database calls I'm making? I'm operating on a shoestring (obviously, being on F1), but I'm paying for MongoDB, and I don't think I'm anywhere near the quotas yet. It looks like I can cache most of the queries just fine, but the larger ones happen on the home page.

I've been trying to figure this out for months, any help would be really appreciated.

I understand that error to mean that your instance ran out of RAM, so adding a cache wouldn't necessarily help. A cache will save you the round-trip to the db, not how much you're loading into memory before return a result to your website.

> This has been an issue more and more as my database has been growing.

I visited your website. It looks like the problem is you are fetching your entire DB's list of golf courses to return the home page. You need to either:

1. only fetch the golf courses you need for that particular page load (i.e. only the golf courses within 100 miles of the user's location). Then when you user moves on the map, you query your backend for the golf courses at the new location
2. produce like a published list of all the golf courses, store it in a GCS bucket and have your website read that file from that bucket.

There are other solutions to this problem too, but those are the simplest. Pulling your entire DB table into RAM everytime the home page is just not a good idea.