Why does my accuracy score drop after hyperparameter tuning in XGBoost (multiclass model)?

I am trying to tune the multiclass model I've built, but every time I change hyperparameters my accuracy score drops significantly. I'm using RandomizedSearchCV and best_params_ to determine which parameters I need to change. In this specific case best_params_ recommends a learning rate of .29, while dropping the accuracy score from 0.6238 to 0.6192. The code I use to tune the parameters is below:

xgb = XGBClassifier(booster='gbtree', objective='multi:softmax', random_state=42, eval_metric="auc", 
                    num_class=num_of_classes, tree_method='gpu_hist', importance_type='gain')

xgb.fit(X_train,y_train)

params={
    "colsample_bytree":[1],
    "gamma":[0],
    "learning_rate":[0.3,0.29], 
    "max_delta_step":[0], 
    "max_depth":[6],
    "min_child_weight":[1],
    "n_jobs":[12],
    "subsample":[1]
    }

clf=RandomizedSearchCV(xgb,param_distributions=params,n_iter=1000,scoring='accuracy',cv=10,verbose=3)
clf.fit(X,Y)

And this is the code for measuring accuracy:

val = clf.predict(X_test)

lb = preprocessing.LabelBinarizer()
lb.fit(y_test)

y_test_lb = lb.transform(y_test)
val_lb = lb.transform(val)

accuracy_score(y_test_lb, val_lb)

The fluctuation of test accuracy is not necessarily a problem. Since the best hyperparameters are recommended based on the best CV score, it does not always correspond to the best test set accuracy. In your example, the difference is very minor.

However, there appear to be multiple issues with the code you provided:

• It seems like your test set is getting leaked into the training procedure.
• You are using 1000 iterations, but there are only 2 possible combinations for hyperparameters. Is this just an example?

The score difference is because your using cv, so you should expect a different score as the different splits of your data will produce different scores. The following will help more generally:

1. Change clf.fit(X,Y) to clf.fit(X_train,y_train) to stope leaking data.
2. Carefully monitor your training and testing scores, and use the differences to check what overfitting, underfitting, leakage etc... is happening.
3. Think about introducing a third validation set. Split your data into training, validation and holdout (60/20/20 is a sensible default). Train your models on training, tune your parameters by evaluating the model on your validation set. Then, once you are happy with your models performance, do a final test on the holdout set to get a good representation of how your model will perform in production.
4. Depending on the size of data, and variation within, cv=10 could be too high making it less practical.
5. Consider not using accuracy for multiclass classification.