Swing: Component wont dissapear until I hover over the component

I am developing a small game in java swing as a small school project. I am done with all the logic and GUI.

The game(Snakes and Stairs) has 36 squares(JButtons) and each of those have Jpanels inside that can be used to put the players piece at(JButtons). In other words, I have 36 buttons which all have Jpanels inside them, and all the JPanels can reside buttons. On each square I have put an action listener that checks whose turn it is, if the player can move here, and moves the players button to that square only if those conditions(and more ofcourse) are true.

Now comes the buggy part. When a players piece moves, it appears on the new square and the old one. The piece only dissapears from the old square if I hover over it.

Some code that might help understand:

//this happens in another function. I only show this, because i think this is the only part relevant from the function
spots[i][j].addActionListener(new ActionListener() {
                    @Override
                    public void actionPerformed(ActionEvent e) {
                        //EventQueue.invokeLater(()->setGameSpotsAction(f,p,spotNr));
                        setGameSpotsAction(f,p,spotNr);
                    }
                });


//action to do when a spot/square is pressed
    public void setGameSpotsAction(JFrame f, JPanel p, int nr) {//nr is the spot where the piece should go
        if(nr == X*Y && playerPosition[playerTurn] + latestRoll == nr){//if dice is rolled
            f.remove(p);
            winnerwinnerchickendinner.setText(namesArr[playerTurn]+" WON!!!!!!");
            JPanel panel = new JPanel(new GridBagLayout());
            GridBagConstraints gbc = new GridBagConstraints();
            gbc.gridx=0;gbc.gridy=0;panel.add(winnerwinnerchickendinner,gbc);
            f.getContentPane().add(panel);
        } else if (latestRoll >= 2 && nr <= X*Y && playerPosition[playerTurn] + latestRoll == nr) {//
                int sot = snakeOrStairs[playerPosition[playerTurn] + latestRoll];//sot stands for Snake Or sTair
                //if just regular square/spot
                if(playerPosition[1] != playerPosition[2]){//if player moves and the previous spot is empty, make panel invisible.
                    spotPanels[playerPosition[playerTurn]].setVisible(false);
                }
                if (sot == 0) {
                    playerPosition[playerTurn] += latestRoll;//button has new position
                    movePlayerButton(nr);
                    //EventQueue.invokeLater(()->{movePlayerButton(nr);});
                } else if (sot > 0) {//if positive number, we can go up!!
                    playerPosition[playerTurn] += latestRoll + sot;//button has new position
                    movePlayerButton(nr + sot);
                    //EventQueue.invokeLater(()->{movePlayerButton(nr);});
                } else {//god damn it we going down
                    playerPosition[playerTurn] += latestRoll - sot;//button has new position
                    movePlayerButton(nr - sot);
                    //EventQueue.invokeLater(()->{movePlayerButton(nr);});
                }
            changePlayerTurn(diceLabelText[1], diceLabelText[2]);
            roll.setEnabled(true);//next player can now roll
        }

    }
public void movePlayerButton(int spotNr){
        GridBagConstraints gbc = new GridBagConstraints();
        gbc.gridx=0;gbc.gridy=playerTurn-1;
        spotPanels[spotNr].add(playerButtons[playerTurn],gbc);//move players button to the new spot
        spotPanels[spotNr].setVisible(true);//set the panel to visible
    }

What I have tried:

• I have tried to call "frame.pack()" after each time a piece moves. It seemed to work the first time it is called, but after that the frame begins to act wierd.(I at least tried something...)
• I have tried EventQueue.InvokeLater and EventQueue.invokeAndWait. This most likely didn't work because I don't really know how to use it properly. https://stackoverflow.com/questions/22534356/java-awt-eventqueue-invokelater-explained

When changing components held within a container, one that properly uses a layout manager, you always should call revalidate() on the container or one of its parent containers, since this will tell the layout manager and the managers of any nested containers to re-layout their components. You also often will want to call repaint() on the container to request a repainting of it and its children, mainly to clear any potentially left-over dirty pixels. This latter is especially true when removing components from the container.