How much difference in damage is there between identical gladiator and whimsy sets??

Is it worth the farm?