It would surprise me, too, but absent beneficiary rewards that are logged by the chain, there is no way for them to take them.
That requires you authorize them to use your active key through steemconnect, or directly by giving it to them.