I feel dumb for misinterpreting Boots of Spanish Leather

I have listened to Dylan casually for many years and I am now doing a deep dive to fully listen to his discography. Boots of Spanish Leather was one I had known for a while and I always thought the melody and words were very pretty. As I listened to the last three stanzas today, I got very confused because they do not alternate between the traveling person and the person left at home, but are just spoken by the person at home. For some reason I always interpreted it as the man being the person who traveled away who then realizes his lover is also traveling away and has lost interest. I had to look up the lyrics to figure out that it was the lady who traveled to Spain this whole time.

I think I just failed to listen to the lyrics with enough intent but I feel dumb for misinterpreting it for so many years. The story seems so plainly obvious to me now.

Does anybody else sometimes have trouble deciphering even the simplest of Dylan’s lyrics without having to look them up?