OAuth2 & OpenID Connect - mobile app & backend server access & refresh token usage

So I'm trying to use Google Sign In and Sign in with Apple in my native mobile app which also communicates with my own backend server. I think I have the grasp of the OAuth2 flow and the concept of ID Tokens behind OpenID Connect. The only doubt I have is if I can/should use the access/refresh token pair to authorize access to my own endpoints? On apple's documentation this isn't clearly stated and on Google's site they mention you CAN use it to access Google APIs, but nowhere have I found you could use it for your own backend server. If not, how should this be approached (generating my own access/refresh token pair after validating the the authorization server's ID Token?)

Note that I only need to authenticate my users via these platforms, I don't want access to other Google APIs (for now).

TL;DR: Can I use Apple's/Google's access/refresh token pair to authorize access to my own backend's endpoints or should I generate my own/do something else?

You can use the tokens provided to you, but often you soon or later will want to customize what they contain and then introducing your own local provider can be a solution. So that your internal services trusts the tokens from your own service an your auth server trusts the tokens from Apple/Google.

The second problem is that your internal systems might need to trust multiple token issuers. In my experience is to internally only trust one token service and not multiple, even if this is not a hard requirement.