These are the people following you, not the ones you are following. So you can't remove them. It's not Facebook were people can deny a friend request.

I myself have 97 / 277, so quite a lot. But with you it's over 70%.

You can mute them I guess, but not that it would help. Because they are inactive anyway. So you won't be hearing from them in the first place.

The best thing to do is just ignore it.