YOU CANNOT RESPEC THROUGH WITHERS. YOU MUST USE RESPEC SPELL. 

You are tagged as an Oathbreaker and Paladin. Shadoweaver is also tagged Sorcerer and Necromancer.

Because of the way the game handles the Oathbreaker tag, you cannot respec normally through Withers. If you want to respec, you will need the Respec Spell mod, which will let you respec without issue.

If you still want the Oathbreaker Knight to show up at your camp, respec to Paladin, use Reclaim or break your oath spells to break your oath, long rest until he shows up and you can freely respect back to Blackguard. He'll still hang out at your camp and acknowledge you as an Oathbreaker (which means you can still hear the special Durge convo with him). If he disappears again it's a known bug with him unfortunately. Sometimes he just decides Halsin is stinky and there's nothing I can do about that, it's a vanilla bug.