Hibiscus, Aphids and Lady Bugs.

I recently had to travel due to a sick relative and when I got back found one of my hibiscus infected with aphids and lady bugs on it. I regularly treat my plants with Neem Oil and Soap, which keep pests away.
I read that you should not spray your plants with anything when you see beneficial insects . Not so sure if you would harm the lady bugs if you spray it with neem oil . She's so infected that I don't know if at this point I should just ignored the lady bugs and go ahead and spray it. Suggestions!