Could not login using access token from powershell

I am using the refresh token to get an access token and then trying to connect to ExchangeOnline and AzureAD like

$clientId = "7b2cd291-87e9-49fc-888e-xxxxxxxxxxxx"
$clientSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$tenantId = "baf62cb4-4cc6-4af9-a3c1-xxxxxxxxxxxx"
$refreshToken = "xxxxxxxx..."
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Content-Type", "application/x-www-form-urlencoded")

$body = "client_id=" + $clientId + "&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&refresh_token=" + $refreshToken + "&grant_type=refresh_token&client_secret=" + $clientSecret

$url = "https://login.microsoftonline.com/" + $tenantId + "/oauth2/v2.0/token"
$response = Invoke-RestMethod $url -Method 'POST' -Headers $headers -Body $body

Connect-ExchangeOnline -UserPrincipalName divyesh@myorg.com -AccessToken $response.access_token
Connect-AzAccount -AccessToken $response.access_token -AccountId 'divyesh@myorg.com'

I am getting access token successfully but could not able to connect to ExchangeOnline and AzureAD. It is giving me errors like the below for ExchangeOnline.

UnAuthorized
At C:\Program Files\WindowsPowerShell\Modules\ExchangeOnlineManagement.1.0\netFramework\ExchangeOnlineManagement.psm1:733 char:21
+                     throw $_.Exception;
+                     ~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : OperationStopped: (:) [], UnauthorizedAccessException
+ FullyQualifiedErrorId : UnAuthorized

and for AzureAD

WARNING: Unable to acquire a token for tenant 'organizations' with error 'Authentication failed.'
Connect-AzAccount : Authentication failed.
At line:16 char:1
+ Connect-AzAccount -AccessToken $response.access_token -AccountId 'div ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : CloseError: (:) [Connect-AzAccount], CloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand

I am not sure what is wrong here?

Depending on the type of access token, you need to use AccessToken parameter with the Organization parameter.

You've configured application permissions for the app in Azure Active Directory and have an access token for a daemon/background service, so you cannot combine AccessToken parameter with UserPrincipalName

Connect-ExchangeOnline -UserPrincipalName xxx -AccessToken yyy

Steps:

Generate access token with the correct scope

When generating access token you are using scope https://graph.microsoft.com/.default but the correct one should be https://outlook.office365.com/.default

To generate access token use this script

$clientId = "7b2cd291-87e9-49fc-888e-xxxxxxxxxxxx"
$clientSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$tenantId = "baf62cb4-4cc6-4af9-a3c1-xxxxxxxxxxxx"
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Content-Type", "application/x-www-form-urlencoded")
$body = "client_id=" + $clientId + "&scope=https%3A%2F%2Foutlook.office365.com%2F.default&grant_type=client_credentials&client_secret=" + $clientSecret

$url = "https://login.microsoftonline.com/" + $tenantId + "/oauth2/v2.0/token"
$response = Invoke-RestMethod $url -Method 'POST' -Headers $headers -Body $body

Connect

Use Organization and AccessToken parameters for connecting

Connect-ExchangeOnline -Organization '<your_domain_name>.onmicrosoft.com' -AccessToken $response.access_token

The steps above suppose that you've assigned the correct Azure role to your application and modified the manifest.