Restrict images access in my website limit the user to access specific image by htaccess and php

I am working on a chat website which has the feature of sending images to users which would be save into their user folder like this user/newuser/a.jpg (means whenever a user register I create a folder inside a main user folder name like this user/newuser).

But here the problem arises this image is meant to be seen by only sender and receiver not by everyone but everyone with the url can easily seen the image like this http://localhost/chat/user/abc.jpg.
So, I come up with that I would restrict access to user folder with htaccess and redirect all requests to user/index.php with this code.

RewriteRule ^user(/.*)?$ user/index.php [QSA]                                                                                 

and from this user/index.php I will redirect user to user/username/userimage.jpg with this code.

session_start();
$imageName = str_replace('/chat/','',$_SERVER['REQUEST_URI']);

if(in_array($imageName,$_SESSION['imageArray'])){

/**$_SESSION['imageArray'] contains all images name which can be accessed by the logged in user like this 
    *['user/newuser/abc.jpg','user/newuser/1.jpg',]
*/
    header('location:'.substr($imageName,strrpos($imageName,'/')+1)); //just output image.jpg

}
else{
echo 'you are not allowed to access this resource';
}

But it isn't working the way it should be if the user can access image it is ending up redirecting again and again and if not then it simply saying you are not allowed to access this resource.

Now, if you understand my question hopefully tell me how to achieve this and if it can't be achieved with htacess then what should I do and what is the best and efficient way to handle scenarios like this.I don't want to use readfile functions to read file and anything like that because it will consump a lot of server resource.Any help is highly appreciated. Thanks in advance.

I know that htaccess is redirecting when image can be accessed because of my htaccess condition that any request to user folder should be redirect to user/index.php. But I don't want to redirect any request which is made inside user/index.php to access any of user folder resource. I am not good with .htaccess code so any code you provide it would be great if you would explain it too.

Here is some rough example code that approximates what it might look like to store and retrieve files stored outside the docroot of your website.

/home/someuser/
  - htdocs/
    - get_file.php
    - serve_file.php
  - var/user_files/

get_file.php

$filename = tempnam('../var/user_file/', 'usrfile_');
move_uploaded_file($_FILES['userfile']['tmp_name'], $filename);
$realname = $_FILES['userfile']['name'];

YourApp::save_user_file_info($filename, $realname);

serve_file.php

if( ! YourApp::check_auth() ) {
  exit();
}
$some_identifier = $_GET['id'];
$file = YourApp::get_user_filename_info($some_identifier);

session_write_close();
header('Content-type: ' . $file->mimeType);
header('Content-Transfer-Encoding: Binary');
header('Content-length: ' . $file->size);
header('Content-disposition: attachment; filename="$file->realname"');
readfile($file->location);
exit();

Finally, I come with a working solution. Check it out and if there are any bugs kindly inform me in comments.
Let's Start, this line in .htaccess

RewriteRule ^user(/.*)?$ user/index.php [NC]

is redirecting all requests from user folder to user/index.php and in this file this is the basic example of what I did there.

if(in_array($url,$_SESSION['accessible-images')){                
header('Content-Type: image');
echo readfile(str_replace('user/','',$imageName)); }               
     else{ 
   header('location:404'); }

with this code only user can access the images which they are allowed to access. And anyone who can't access images are sent to 404 page. Here is my code where I am retrieving images.

echo '<img src=\'user/username/userfile.jpg\' >';

Thanks to Mr Sammitch Answer which help with the header function. If this answer or question would help anyone please upvote my anwer and question too.Thanks