The label the enemy has been trying to place on your life will not stick. It currently has no adhesive ability because you know who you are in Christ and what He has called you do in service of His Kingdom. Because of that understanding, you have chosen to not take offense at the harshness of the label someone is trying to attach to your life. Neither have you tried to justify your calling. This has kept you safe and label-free.
Choose to walk forward in faith away from the label-makers. Keep your heart free of offense and defense no matter who is bringing an accusation against you. Let God alone be your defender and your protector. You have been called to wear only one label and that label reads, "A Child of God". Anything not under that banner heading is a lie.