Based on Sephumbra's Harder Taunting (modified with permission), with some fixes. Now compatible with Silver Tongue and the like.

This mod reacts to known Taunt Failure dialogue lines.

Whenever you fail a taunt, it will reduce the opponents Fight rating and restore their Flee rating - the base game has no penalty for failing Taunt (or Intimidate), other than the Disposition change. This could lead to characters with 5 Speechcraft and a lot of gold to just exploit the system by bribing and taunting until the opponent decides to attack.

Now Speechcraft is required for this to be reliable at all. If your Speechcraft, Personality, Luck, and Reputation ratings are lower than those of the other character,
the taunt fail impact could be pretty severe, and actually getting them to attack would be very difficult. The higher your persuasion attributes are compared to the character's, the less heavy the impact will be, down to the iPerMinChange GMST.

For anyone interested, the exact calculation for the value that gets added/subtracted from the flee/fight value of the character is exactly the reverse of the base formula for successful taunts.
You can find the pseudocode for those formulas here: https://wiki.openmw.org/index.php?title=Research:Disposition_and_Persuasion#Taunt

Anything seems wrong?

Go in MWSE's Mod Config (either on the Title Screen or on the Escape menu), go in Taunt Failure Penalty's config, enable Debug Mode, and try reproducing your issue. Then, send me your MWSE.log