Top Box = Crazzzzy wobble on the steering?

After installing and riding my scooter with a topbox for such a long time, I decided to remove it since it is a hot windy season. What I notice after removing it, my steering no longer wobbles at low and mid speed compared to having a topbox on it.

When I rode my scoot with the topbox, lifting my hands ever so slightly makes the steering wobble like being possessed.

My Ride: Honda Vario/Click 160 Topbox: Givi R32

However, riding with the topbox on wet road makes my scooter very grippy on the rear wheels. Probably due to additional rear weight and drag. I can go really fast confidently with it on such road condition.

Any riders face similar experience with their topbox? or is this an issue with the Givi R32?