How to push items inside an array at the rate at which the screen is being getting captured

I am using getUserDisplay api to record the current tab of the user, Now I want to capture the users mouse move positions at the rate it is capturing the screen in (so that each frame could have one object of x and y coordinates)

const constraints = {
  audio: false,
  video: true,
  videoConstraints:{
    mandatory:
     {minFrameRate:60,maxFrameRate:90,maxWidth:1920,maxHeight:1080}} 
};

navigator.mediaDevices.getUserDisplay(constraints)
  .then(stream => {

    const mediaStream = stream; // 30 frames per second
    const mediaRecorder = new MediaRecorder(mediaStream, {
    mimeType: 'video/webm',
    videoBitsPerSecond: 3000000
    });
   

in short > I recorded the screen and send the recorded video to the backend, in the backend decoded the frames, now for each frame there should the mouse x and y coordinates just like doing in real, then I would stich the frames together and form a video
wants to do some editing with the recording

I don't wanna add the cursor on the video in the frontend js, but rather save the mouse coordiates and recorded video seperate and send both to backend

I try using requestAnimationFrame, but its not equal to the number of frames in the video, I tested and number of frames in the recorded video were like 570 and the array only contains 194 items.

 function tests() {
      testf.push('test');
      window.requestAnimationFrame(tests)
    }
    

Thank you so much for reading, any advice would be greatly appreciated

Unfortunately, it is technically impossible to capture the mouse position on each recorded frame because API doesn't provide an onFrame event or anything similar.

Additionally, because frame rates can technically be different on different devices, I'm not sure it is essential to stick to the recorded frames.

What I would recommend instead is to stick to the timing. What we need to know is that the mouse was at a specific position at the time when the certain frame was recorded. In other words, the mouse and picture on the frame should be in sync.

This can be achieved by starting a timer at the moment when the video recording starts and then using requestAnimationFrame() to schedule a function that will record { timestamp_msec, mouseX, mouseY, mouseButtons } where timestamp_msec is a time in msec since the moment when video recording started.

We will get a video and an array of the above objects on the receiving side. In the video, you can always obtain the current playing position. Having that position in msec, you can always find the mouse position object, which has a timestamp that is less or equal to the current video position.

Another thing is that requestAnimationFrame() gives you a way to precisely match the moment you record the mouse position with the screen update. Even if you record a video with a higher frame rate than requestAnimationFrame() can provide, some frames will be just duplicated and have little value.

I've built a few remote desktop like systems, and the approach described above works like a charm.

Here is a quick demo with requestAnimationFrame:

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-js -->

let recordingStart = Date.now(); // set this when you start recording
let lastKnownMousePosition = {};

window.addEventListener('mousemove', (event) => {
  lastKnownMousePosition = {
    mouseX: event.clientX,
    mouseY: event.clientY,
    mouseButtons: event.buttons,
  };
});

const frameHandler = () => {
  const mousePosition = {
    timestamp: Date.now() - recordingStart,
    ...lastKnownMousePosition,
  };
  // send mousePosition to server here
  console.log(mousePosition);
  requestAnimationFrame(frameHandler);
};
requestAnimationFrame(frameHandler);

<!-- end snippet -->

Please let me know if this helps.