Depends on your market. If houses are moving and other inspectors are busy I would stay in it, but only if you feel qualified enough to not keep worrying about getting sued. Also, get E&O insurance-- look into E&O insurance that you pay per job until things pick up for you.

If you are as slow as you say you are, I hope this is just a weekend job for now.