Spells and artefacts sometimes not casting

First I want to point out the obvious about different cast types, and I use instant cast (cast immediately on click). However I've noticed something VERY annoying which is that sometimes my spells just won't cast. I will be trading with another hero in lane, press to use a skill and... nothing. Even if I press the skill multiple times (several times per second) nothing will happen, or it will take around a second to cast.

At first I thought you couldn't cast spells while (or a short time after) attacking, but that doesn't seem to be the case since just standing in base and attacking and pressing my spells casts them immediately.

This has happened while in "passive" conditions, just while farming jungle or lane, so it's not like I'm silenced or stunned.

It doesn't happen a lot, maybe 1/10 times so I don't think I have connection issues with the server (although I have unusually high ping compared to other games). Given that I assume I must be doing something wrong. Can someone enlighten me?