How to get corners of a traced image?

I am trying to use edge detecction to find 4 corners of a scanned paper, so i can crop it out. Here is what code I have, but it does not detect the points correctly. I am trying to do this without OpenCV.

        detectEdges(image);
HashMap<Integer, Integer> corners = new HashMap<>();
HashMap<Integer, Integer> reverseCorners = new HashMap<>();
int white = new Color(255, 255, 255).getRGB();
int red = new Color(255, 0, 0).getRGB();
int Blue = new Color(0, 0, 255).getRGB();
for (int i = 2; i < image.getWidth() - 2; i++) {
for (int j = 2; j < image.getHeight() - 2; j++) {
int color = image.getRGB(i, j);
int right = image.getRGB(i + 1, j);
int left = image.getRGB(i - 1, j);
int top = image.getRGB(i, j + 1);
int bottom = image.getRGB(i, j - 1);
int rightCheck = image.getRGB(i + 2, j);
int leftCheck = image.getRGB(i - 2, j);
int topCheck = image.getRGB(i, j + 2);
int bottomCheck = image.getRGB(i, j - 2);
if (color == white) {
//TR Corner
boolean isCornerLR = (((right != white && rightCheck != white) && (left == white && leftCheck == white)) || ((left != white && leftCheck != white) && (right == white && rightCheck == white)));
boolean isCornerTD = (((top != white && topCheck != white) && (bottom == white && bottomCheck == white)) || ((bottom != white && bottomCheck != white) && (top == white && topCheck == white)));
if (isCornerLR && isCornerTD) {
corners.put(i, j);
image.setRGB(i, j, red);
}
}
}
}
HashMap<Integer, Integer> extremeCorners = new HashMap<>();
/*Get opposite corners of shape. so, Bottom left to Top Right, or Top Left to Bottom Right
for case one, (LowKey,LowValue), (HighKey, HighKeyValue). Case two - (LowKey,HighValue),(HighKey,LowValue)
*/
List<Integer> valueOfValue = new ArrayList<>(corners.values());
for (int a : corners.keySet()) {
reverseCorners.put(corners.get(a), a);
}
Collections.sort(valueOfValue);
int yCorner1 = valueOfValue.get(valueOfValue.size() - 1);
int yCorner2 = valueOfValue.get(0);
int xCorner1 = reverseCorners.get(yCorner1);
int xCorner2 = reverseCorners.get(yCorner2);
System.out.println(corners);
extremeCorners.put(xCorner1, yCorner1);
image.setRGB(xCorner1, yCorner1, Blue);
image.setRGB(xCorner2, yCorner2, Blue);
extremeCorners.put(xCorner2, yCorner2);
File f = new File("EdgeImages/" + name);
ImageIO.write(image, "png", f);
return extremeCorners;
}

So what this code does, is that it looks for potential corners and marks them with red and adds them to a hashmap, which is pretty accurate in getting potential corners. from there, I tried to make another hashmap that stored 2 corners. This is where the code doesn't work, I am sure it is my logic which is the error, but I am not sure how to proceed from here.

Attached below is the image

Low accuracy solution:

Consider the blob formed by the outline and scan its pixels (the contour pixels can do). For every pixel, compute the values of X+Y and X-Y, and keep a trace of the pixels that yield the minimum and maximum of these quantities. These are the four corners. It might be that the outline is split in several pieces, and it suffices to use the pixels of all pieces.

High accuracy solution:

Using the above method, find the corners, roughly and draw the two diagonals. Then classify the pixels as belonging to one of the quadrants defined by these diagonals. It is advisable to discard the pixels close to the corners, as their classification is unsure.

Finally, fit a straight line on the four subsets of points, using your favorite line fitting method.