PyGame colliders don't scale with window

I should point out that I'm a beginner with PyGame. I have made a program that displays some simple graphics on the screen using PyGame. It blits every graphic on a dummy surface and the dummy surface gets scaled and blit to a 'real' surface that gets displayed on the screen in the end. This allows the program to have a resizable window without messing the graphics and UI.

I have also made my own 'Button' class that allows me to draw clickable buttons on the screen. Here it is:

import pygame

pygame.font.init()
dfont = pygame.font.Font('font/mfdfont.ttf', 64)

#button class   button(x, y, image, scale, rot, text_in, color, xoff, yoff)
class Button():
	def __init__(self, x, y, image, scale = 1, rot = 0, text_in = '', color = 'WHITE', xoff = 0, yoff = 0):
		self.xoff = xoff
		self.yof = yoff
		self.x = x
		self.y = y
		self.scale = scale
		width = image.get_width()
		height = image.get_height()
		self.image = pygame.transform.rotozoom(image, rot, scale)
		self.text_in = text_in
		self.text = dfont.render(self.text_in, True, color)
		self.text_rect = self.text.get_rect(center=(self.x +width/(2/scale) + xoff, self.y + height/(2/scale) + yoff))
		self.rect = self.image.get_rect()
		self.rect.topleft = (x, y)
		self.clicked = False

	def draw(self, surface):
		action = False
		#get mouse position
		pos = pygame.mouse.get_pos()

		#check mouseover and clicked conditions
		if self.rect.collidepoint(pos):
			if pygame.mouse.get_pressed()[0] == 1 and self.clicked == False:
				self.clicked = True
				action = True

		if pygame.mouse.get_pressed()[0] == 0:
			self.clicked = False

		#draw button on screen
		surface.blit(self.image, (self.rect.x, self.rect.y))
		surface.blit(self.text, self.text_rect)

		return action

When I need to draw one of these buttons on the screen I firstly define it like this:

uparrow = button.Button(128, 1128, arrow_img, 0.5, 0, "SLEW", WHITE, 0, 128)

Then I call it's draw function like this:

if uparrow.draw(screen):
		print('UP')

It works reasonably well when drawing it to a surface that doesn't get scaled. This is the problem. When I scale the dummy surface that it gets drawn to, the button's image and text scale just fine but it's collider does not. So when I click on it nothing happens, but if I click on the location of the screen the button would have been on the unscaled dummy surface it works.

Just for context, the dummy surface is 2048x1024 and the 'real' surface is much smaller, starting at 1024x512 and going up and down however the user resizes the window. The game maintains a 2:1 aspect ratio though, so any excess pixels in the game window are black. You can see this in the screenshot below:

Above is a screenshot of the game window. You can see the 'NORM' button at the top of the game screen, and the red box that roughly represents the same 'NORM' button's actual collider. It's basically where it would be on the dummy surface.

(I have previously posted a question on somewhat the same problem as this one, but at that time I didn't know the colliders actually worked and I thought my clicks just didn't register on the buttons, which is not the case).

I'd like to know what part of my button class causes this and how it should be refactored to fix this issue. Alternatively, if you think it's caused by my double surface rendering technique or anything else really, please do point me in the right direction.

In your setup you draw the buttons on an surface, scale the surface and blit that surface on the display. So you do something like the following:

dummy_surface = pygame.Surface((dummy_width, dummy_height)

while True:
    # [...]

    scaled_surface = pygame.transform.scale(dummy_surface, (scaled_width, scaled_height))
    screen.blit(scaled_surface, (offset_x, offset_y))

For click detection to work on the original buttons, you must scale the position of the mouse pointer by the reciprocal scale and shift the mouse position by the inverse offset:

def draw(self, surface):
    action = False
    
    # get mouse position
    pos = pygame.mouse.get_pos()

    scale_x = scaled_width / dummy_surface.get_width()
    scale_y = scaled_height / dummy_surface.get_height()
    mx = int((pos[0] - offset_x) / scale_x)
    my = int((pos[1] - offset_y) / scale_y)

    pos = (mx, my)

    # [...]