Version 2.0 is here - it is a MAJOR update.

TWO VERSIONS:  One requires Far Harbor and Nuka World.  One requires only the vanilla game.  Choose only one!

Simplified the OPTIONAL Community Clothing and Armor Pack ("CCAP") to require only Eli's Armor Compendium and Azar's Ponytail Hairstyles downloads - and nothing else.

I highly recommend the OPTIONAL CCAP because the work they did is great (see new images).

NOTE: Even though you may see some skimpy clothing in the screenshots, there is no skimpy clothing in the mod.

Better Settlers uses no scripts, and the base option uses only vanilla assets for maximum compatibility.

Compatible with Sim Settlements and all other major mods I know of.

Listed on Ultimate Fallout 4 Mod Compilation
As far as I know, the base mod is bug-free.  However, you should still read Known Game Engine Issues below before installing. 
The DLC mod has 2 known issues. 
Some Far Harbor Settlers have the wrong voice.  A fix is Here:  https://www.nexusmods.com/fallout4/mods/50024
Please let me know if you find any bugs - but you should not, because you should not.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Better Settlers is a lore friendly mod that adds more than 230 new settlers to the vanilla settler selections (for a total of 270+). You can choose to leave them with vanilla equipment and stats, or:  It allows you the OPTION to make lore-friendly changes to their equipment, stats, or mortality.  Other options include: mod-created equipment and hair (highly recommended), all female settlers, or raiders as settlers.  All of these options are randomly generated when the Settler spawns, so no two settlers should be exactly alike.

If it matters, I made it with FO4Edit.  Changelog is at bottom.  

Vanilla Version:

• This mod adds 230 new vanilla settler options with cosmetic variety in their faces.
• You can stop there if you want and you are done.
• Or, you can choose other lore friendly options listed below.
  

Optional Upgrades:

• If you choose OPTIONAL stat or equipment changes, the options are below:
• There is a choice between grimy and clean faces. 
• Settlers will no longer lollygag and will run to the bell when summoned.
• See the screenshots - they are self explanatory.
• There are approximately 230 face/race/hair/sex combinations now (instead of approx 28 in vanilla).
• There are approximately 30 civilian outfits and 25 hats over vanilla (approximately triple the vanilla amounts).
• All clothing and equipment is randomly generated on spawn.  No two settlers should be exactly alike.
• This only effects GENERIC SETTLERS.  Named Settlers (like the Longs), Provisioners and Children are unchanged.
• If you choose stat or equipment changes, they are only made to GENERIC SETTLERS.
• Generic Settlers have perks that help them with radiation and healing if they survive battle (important for the Mortal Pack).
• All have a chance to roll 0-2 stimpacks, which they absolutely will use in battle.
• All get better, but lore friendly, starter weapons.
• All get lore friendly equipment on spawn: clothes (55%), partial leather(15%), full leather (15%), full metal set (10%), or combat armor (5%).
• Metal and combat equipment sets and their better weapons do not show up until higher levels.
• There is an OPTION to give them only light or heavy equipment packages, if you wish.
• There is an OPTION to make settlers mortal (right now NPCs cannot kill them and defending settlements is pointless).
• There is an OPTION to make mortal settlers even tougher - for those of you that want to play crazy settlement raid games.
• There is an OPTION to make MOST new settlers female or male
• There is an OPTIONAL Community Clothing and Armor Pack (CCAP) that adds selections from the following mods.  Eli's Armor Compendium and Azar's Ponytail Hairstyles.  (Both of these files must be installed ABOVE all Better Settlers files in your load order).
  THOSE MODS ARE REQUIRED DOWNLOADS FOR THIS TO WORK.  DONT INSTALL THIS OPTION UNLESS YOU KNOW WHAT YOU ARE DOING.  I HIGHLY RECOMMEND IT! 
  
  
  
  Load Order (IMPORTANT):
  Better settles should go near the very bottom of your load order!
  
  YOU MUST LOAD UNOFFICIAL FALLOUT PATCH BEFORE BETTER SETTLERS OR YOU MIGHT GET CRASHES!!!
  
  <<<Eli's Armor Compendium and/or Azar's Ponytail Hairstyles (optional)>>>
  <<<Any other Settler Mods>>>
  BetterSettlers.esp
  <<<Any Better Settler Patches that you might use from other authors>>>>
  <<<Any Better Settler Optional Packs like Lollygagging, Clean face, or Light or Heavy Armor Types (optional)>>>
  <<<Better Settlers Mostly Female or Mostly Male (optional)>>>
  <<<CCAP MUST go here or it wont work (optional)>>>
  <<<Any Better Settler Mortal or Mortal Soldier Pack must go LAST or they wont work (optional)>>>

If you want to do a test to see variety and randomness, you can do this with console:
~tai  (turns off ai)
tcai (turns off combat ai)
tgm  (toggles god mod)
spawn 00020593 40  (40 is the number of settlers it will spawn).

let them spawn and move away a little bit and do it again.

I normally spawn about 120 settlers at a time and go through and look at them to check for bugs.  But you can use it to see the variety and make sure the ponytails and gear are showing up, etc. You can safely shoot the settlers and kill them to check their inventory.

Dont forget to tai tcai and tgm again or no one will move and you will be unkillable. :)

New in 2.0

• There are two versions.  One requires Far Harbor and Nuka World.  The other requires only vanilla game.  CHOOSE ONLY ONE.
• Now total 240+ settlers in Mainland levelled lists.
• Now 40+ settlers in Far Harbor levelled lists.
• Added Gender Ratio choice to Fomod (Mostly Female and Mostly Male)
• Raised settler levels in Mortal Options to higher levels, to compensate for harder mobs in DLC.
• Added Community Clothing and Accesory Pack choice to FOMOD.  Now, YOU MUST DOWNLOAD Eli's Armor Compendium (version 1.4 or higher) and Ponytail Hairstyles by Azar (version 2.5a or higher) from the Nexus and install them if you select this option.  NO OTHER FILES ARE REQUIRED.
• Removed all other CCAP Options for compatibility purposes. If you are already using the CCAP, its PROBABLY ok to install this version, as long as you dont unintall the other clothing mods.  Probably. But if you want to be completely safe, DO NOT INSTALL THIS VERSION UNTIL YOU START A NEW GAME.  OR YOU CAN BE DARING AND RISK IT.  BUT IT MIGHT CAUSE LOTS OF NAKED PEOPLE OR CRASHES OR OTHER TOMFOOLERY.
•  The following files are NO LONGER REQUIRED FOR THE CCAP (I Always recommend AE and Armor Keywords, even though they are not required).
  

• nsk13's DX's Shorts, DX's Adventurer Outfits;
• Elinaora's Apocalypse Attire, Apocalypse Accessories, Rugged Outfits and Armor Collection;
• Bordraw's Hoodless BOS Scribe Hat. 
• Ascendia's Patrolman Sunglasses (use the standalone version).
• Armorsmith Extended merged patches for the above.
  

Known Game Engine Issues - do not install without reading this.

• This is not a bug - but a game engine issue:  When you load or upgrade the mod, it will give some settlers equipment.  If it is a new game, there should be no problem.  In an existing game, or when you upgrade, sometimes this will create a conflict in their inventory and make them strip.  Occasionally a settler dressed like Hlormar Wine-Sot from Morrowind will appear. Or they will have "ghost" (0) items in their inventory.  These are both rare.  But if you trade with them you can equip clothes like normal.  Its just a cosmetic issue.  This will only occur if you unstall, or upgrade after I was forced to remove an item from the leveled lists for some reason.  This is not a bug, but normal game behavior with any mod that adds items.
• This is not a bug - but a game engine issue:  If you UNINSTALL it mid game, it will remove some items from settlers (not any you gave them - only ones the mod spawned for them).  This may result in the (0) item issue.  Again - this is just cosmetic and wont hurt your game.  This is not a bug, but normal game behaviour.  If you encounter it in any other manner, please report it.
• Read bug reports for other items that have yet to be confirmed.
  

Thank you to ButteryButters and merlimpt the mod spotlights:

This is still a beta - but waiting on final testing to become 1.0.  Still bug-testing, but I think we have fixed most of the fixable ones. Please let me know if you find any others.

DO NOT USE THIS WITH  Mortal Settlers MOD - THEY ARE INCOMPATIBLE - USE THE "MORTAL" PATCH HERE INSTEAD IF YOU WANT.

THIS ONLY EFFECTS GENERIC SETTLERS.  Companions, Children, and Named settlers (like the Longs and Finches) are unchanged, and will be equipped and behave as in vanilla.

There are a few files.

• Main File
• Clean/Dirty Face Pack.
• Mortal Pack (and subpatches) OPTIONAL (like in my mod Mod: Mortal Settlers).  My preferred way to play.  Will make the game harder.  Makes your settlers so they can die in NPC raids (right now they cannot die by raiders unless YOU accidentally kill them).  Consider before you choose this.  I made the settlers hardier, but they can still die pretty easily (2 grenades on survival).  Missile Turret spam is discouraged.  :) 
• Heavy Armor Only Pack OPTIONAL (only Settlers with Class III and Class IV equipment will spawn).
• Light Armor Only Pack OPTIONAL (only Class I and Class II equipment will spawn).  This is probably the most "lore friendly" version.
• Female Settlers Only Pack OPTIONAL.  Should do just what it says.  New settlers should be MOSTLY female or male, depending on your choice.
• Raider Settlers Pack OPTIONAL.  It replaces settler faces and gear with raider faces and gear.  EXPERIMENTAL.
• Community Clothing and Armor Pack OPTIONAL.  See Readme for that file details and required files.  DO NOT INSTALL UNLESS YOU KNOW WHAT YOU ARE DOING.
  

Are there compatibility issues?

This will conflict with any mod that changes settler stats, or their levelled lists.  

• 
  

If you have bugs, fixes, suggestions, etc, please post comments and I will try to address.

Installation
Make sure you have made all the necessary configurations to enable mods in Fallout 4.
If you haven't, check out this guide to learn how: FALLOUT 4 - Enabling Modding for PC by Gopher.

I seriously recommend installing through the Nexus Mod Manager. The installation wizard guides you through the steps, and it minimizes the potential for future errors. 

With Nexus Mod Manager (NMM):
1. Download the mod by clicking "Download with manager".
2. Activate the mod in Nexus Mod Manager

If you are using The Mortal Patches Or Heavy or Light Armor Patch, Or Raider, Female or Community Clothing Patches they must go after "Better Settlers" in your load order or it will not work.


Uninstallation
With Nexus Mod Manager (NMM):
1. Deactivate the mod by double-clicking it.
2. Mark the deactived mod and uninstall or delete it.
3.  Uninstalling will take away equipment given to them, and might result in them having some (0) items in their inventory.  This is a game function and not a bug.  Other than being annoying, it is cosmetic only and will not harm your game.

CREDITS:
drdanzel - I copied his installation instruction section!