This book is pretty short but a good mental model for when to quit.
TLDR: Everything worth doing in life is controlled by "The Dip". The Dip you experience after initial success of any project is there to weed out people don't want the result bad enough to do the work. Make it your best friend.