I will be providing Youtube tutorials on how to add and remove followers mods. I will be using SSEdit/xEdit. This is an easy tool to use and should be using anyways.

First some rule of thumbs:
1. Follower mods must be above my mod. This is because we will be setting them as master files. That can't work if follower mod is below my mod.
2. When finish adding NPCs in xEdit, always right click on plugin and "check for errors"

ADDING FOLLOWERS

Now onto the guide using Chaconne mod as an example:

Link: https://www.youtube.com/watch?v=o652DMmwN1c
1. Place Chaconne above BYOB mods and run xEdit. Note: skip to 0:40 after starting xEdit if you want
2. Set Chaconne as master on "Build you Bandits.esp
3. Now it is just copying and pasting from this point. I'm using CTRL+C and CTRL+V
a. Copy Chaconne ID from left pane or right pane
b. Paste into desired bandit slot. Must paste into the TPLT-Template record

Optional: Adding follower mod into multiple bandits at once. Great for bulk editing. 
Link: https://www.youtube.com/watch?v=bEYJutT1mJo
1. Follow previous steps 1-3
2. Select all bandit records. Use shift or CTRL keys...
3. Compare all the records you just clicked
4. Go to the NPC you just copied a follower into (should be in bold), right click on your edit and the "copy to selected records"

--------------------------------------------------------------

REMOVING FOLLOWERS

https://www.youtube.com/watch?v=czmaURVzokY

1. Rule of thumb. If you remove an entire follower mod from my BYOB mod, you must clean BYOB plugin so that the follower mod is no longer a required file.
2. The "Reference by" tab as seen at 0:04 is one of the best pieces of information when working on any mod. It show what records are referenced. In this case, we can see where Ailana is being referenced. She is replacing 2 BYOB bandits.
3. From here can find all the instances of the npc you want to remove. Find them. Delete them.
4. Don't leave BYOB template blank. There must be a NPC follower mod here or you will get black faces on bandits