Time travel is more complicated than you think