Iterating over points in a rotated rectangle using coordinate transformation

I am trying to iterate over the points inside a rotated rectangle, which is basically an image matrix containing certain pixel values and modify them according to certain conditions. But the loop counter does not fill up rectangle properly and there are 'empty spaces' in the rectangle that are not being tracked.

My approach:

It is easy to iterate using a nested for loop over the individual points inside a non-rotated rectangle. My approach is to generate the points in a non-rotated rectangle and using a coordinate transformation using the rotation matrix to map those points into the rotated angle. The angle of rotation can be found easily.

The code I used is as follows-

% Define the coordinates of the vertices of the rectangle
x1 = 100;  % X-coordinate of vertex 1
y1 = 200;  % Y-coordinate of vertex 1
x2 = 400;  % X-coordinate of vertex 2
y2 = 300;  % Y-coordinate of vertex 2
x3 = 300;  % X-coordinate of vertex 3
y3 = 500;  % Y-coordinate of vertex 3
x4 = x1 + (x3 - x2);  % X-coordinate of vertex 4
y4 = y1 + (y3 - y2);  % Y-coordinate of vertex 4

% Calculate the rotation angle
angle = -atan2(y2 - y1, x2 - x1);  


% Calculate the center of the rectangle
centerX = (x1 + x2 + x3 + x4) / 4;
centerY = (y1 + y2 + y3 + y4) / 4;

% Define the dimensions of the rectangle
width = (x2 - x1);
height = (y4 - y1);

xv = [x1 x2 x3 x4 x1];
yv = [y1 y2 y3 y4 y1];
plot(xv,yv,'r','LineWidth',2)

% Iterate over points within the rectangle boundaries
for i = round(centerY - height/2):10: round(centerY + height/2)
    for j = round(centerX - width/2):10: round(centerX + width/2)
        

        % Apply inverse rotation transformation
        x = (cos(angle) * (j - centerX)) + (sin(angle) * (i - centerY)) + centerX;
        y = (-sin(angle) * (j - centerX)) + (cos(angle) * (i - centerY)) + centerY;
        
        x11=round(x);
        y11=round(y);

        hold on
        plot(x11,y11,'b*')
    end
end

The output and the problem:

I have shown how I have calculated the angle of rotation using the inverse tangent function.

Clearly, the points I plotted are not really filling up the rectangle properly. Note: I have set the loop counter at 5 points at each iteration for clarity but as shown by the yellow arrows, there are empty places that are not being filled. The purple arrows also show some points that just slip outside the intended rectangle. What should I do to correct this fault? Apparently it looks like there is some error in the rotation angle but I am not sure.

Thanks in advance!

Your coordinates x1y1.. don't describe rectangle but parallelogram (you can check that angle between two neighbor sides in not Pi/2).

So affine matrix of this transformation is not simple rotation but includes shear, and your Apply inverse rotation transformation is not correct.

You have a choice - or calculate rotated rectangle vertices exactly, either use given parallelogram with corresponding affine matrix.

You can get correct matrix of affine transform as described here or here with ready formulas

(for rectangle source expressions become a bit simpler due to some equal coordinates, but in Matlab you can just inverse and multuply matrices with inbuilt methods)