This is a re-post from years ago. I still feel the same.
#1: Making Payroll
My first problem was simply cash flow. I was solely responsible for having enough money in the bank every week, and that could be nerve-wracking, especially when customers weren't paying their bills on time. It's not fun to think that a family won't be able to buy groceries if you can't collect your invoices. Still, that part didn't cause me to give up on employees. It was hard, but so long as my employees were working, we were making money, so there was always something coming in at some point. Somehow, I was able to pull it off.#2: Being Hated
Over time, some of my employees became jerks. This seemed to grow from envy and from stupid ideas about labor versus management. These guys decided that I was getting rich off of them, and demanded I pay them more: more than they deserved and more than the company could afford. And the really nasty part was this: It was the guys I had done the most for who hated me most. And when I sat down with them and explained why I couldn't pay them more, they started stealing from me. I fired the thieves, of course, but these experiences really soured me on employees. I had not only given these guys a job, but I had legitimately felt good about helping to feed their families. In return, they hated me, called me names, and stole from me. By itself, that was almost enough to make me swear off employing people, but not quite.#3: The IRS
What really drove me over the edge was dealing with the government, and the IRS in particular. They were abominable. I had to file forms with every payroll, and if anything on them was wrong, they penalized me... heavily. And if I paid them a single day late, they penalized me... heavily. And if they said I did something wrong – even if I didn't – there was no way to change their verdict. Reason and evidence simply didn't matter. I eventually talked to a tax lawyer who explained the situation to me. He said:Forget about fighting, Paul. There is no 'innocent until proven guilty' in tax court. You're automatically guilty, and you have to try to prove yourself innocent… which is very hard and very expensive. Just pay them. I know you hate that, but you have no other choice. Fighting them would ruin you.It wasn't just the money that got me about this – it was that they were heartless tyrants. Having the facts on my side didn't matter. Intelligent arguments didn't matter. Either I paid what they demanded or they would hurt me worse. It wasn’t terribly different than the local gang of street thugs demanding protection money. So, that's why I can't hire you: Having employees locked me into a single role in life, that of a despised slave. When I finally realized that, I walked away. I was lucky that I had the ability to move into specialties and to thrive in difficult niches; a lot of other guys probably couldn't have.