Using ROW_NUMBER() Over to find closest date match to 0 or earlier

I have a need to compare all the recorded dates in one column to a deadline date in the other column. I can do a ROW_NUMBER() OVER to find the closest date, however, I cannot go past the deadline date. I know I can add in the WHERE clause "WHERE recorded_date <= deadline_date", but I need to show all the data for auditing purposes.

Here is the CREATE TABLE:

CREATE TABLE Vitals (
Player	nvarchar(50),
Vital	nvarchar(20),
Val	int,
Entry_Date	datetime,
Game_Date datetime
);

Here is the INSERT Values:

INSERT INTO Vitals (Player, Vital, Val, Entry_Date, Game_Date)

VALUES

('George', 'Weight', 227, '20230525 12:02:10', '6/1/23 20:00:00'),
('George', 'Weight', 226, '20230421 16:10:22', '6/1/23 20:00:00'),
('George', 'Weight', 227, '20230508 14:27:01', '6/1/23 20:00:00'),
('George', 'Weight', 228, '20230527 08:45:55', '6/1/23 20:00:00'),
('George', 'Weight', 230, '20230601 11:42:23', '6/1/23 20:00:00'),
('George', 'Weight', 229, '20230605 17:07:27', '6/1/23 20:00:00'),
('George', 'Weight', 223, '20230612 07:30:02', '6/1/23 20:00:00'),
('George', 'Weight', 228, '20230601 18:10:05', '6/1/23 20:00:00'),
('George', 'Weight', 229, '20230524 10:15:19', '6/1/23 20:00:00'),
('George', 'Weight', 228, '20230506 11:20:13', '6/1/23 20:00:00'),
('George', 'Weight', 230, '20230602 16:18:33', '6/1/23 20:00:00')

Here is the SQL Code I am currently using:

SELECT
 Player
,Vital
,Val
,Entry_Date
,Game_Date
,DATEDIFF(DAY, Game_Date, Entry_Date) 'Days Diff'
,ROW_NUMBER() OVER (PARTITION BY Player ORDER BY ABS(DATEDIFF(DAY, Game_Date, Entry_Date)) ASC) RN

FROM Vitals;

Here is what I am I trying to achieve:

As you can see, the row numbering is first ordering the Entry Dates BEFORE Game Date first then ordering the dates after the Game Date (missed the deadline, but have to show by how much to auditors). I have tried different ways to use Case statements in the Order By, but it doesn't work. Any suggestions would be appreciated. Thank you.

p.s., Note the difference between row number 1 and 2. Number 1 is very close to game time. I need to order closest to game time as possible.

with myData as 
(
    SELECT  Player
       ,Vital
       ,Val
       ,Entry_Date
       ,Game_Date
       ,DATEDIFF(DAY, Game_Date, Entry_Date) 'Diff'
    FROM Vitals
)
select Player
    ,Vital
    ,Val
    ,Entry_Date
    ,Game_Date
    ,Diff as 'Days Diff'
    ,Row_Number() over (Partition by Player
                        Order by case when Diff <= 0 then -1 else 1 end, abs(Diff)
                       ) as RN
from myData;

Here is DBfiddle demo