Joint Custody: The Joke Is On Me

My ex and I have joint custody.  This is crazy because....
 
1.  He was under the influence of drugs and alcohol for most of the last two years of our marriage.
 
2.  He assaulted me and the kids as well when they tried to defend me.
 
3.  His parental rights were taken away most of last summer by Child Protective Services and NOW he has unsupervised visitation...
 
4.  He is irresponsible and I think possibly insane.
 
How did this happen you ask??? 
 
1.  Money -- he drained our accounts and I had no access to cash when he left.  I was stuck in a town 1 hour away from home, with kids to pick up and daycare to pay and he had drained every cent we had.  So, I had no money for an attorney and had to borrow from my parents.
 
2.  Credit cards -- he had charged up ALL of our credit cards before he left - so I couldn't pay for an attorney that way either.
 
3.  What money I did have had to go to cleaning the flood he caused on the second floor of our home, trying to patch the air conditioner which he damaged during the summer (we are in a hot climate)
 
4.  He shaved himself for his hair drug test - think Brittney Spears.  He had warning of about a month before the screen - so he just abstained and shaved up for the test.
 
5.  The courts don't want to order additional tests of fingernails because it "wastes" their time.
 
So, here I sit on ANOTHER of his weekends where he just doesn't show.  The kids call him to find out why he isn't there.  I had my three year old son in tears.  He say's "I am still at work" - which is bullshit because I can hear that he is out on the town in the background.
 
I can't make plans on his weekends because I don't know whether he will pick them up or not.  I can't schedule a babysitter and then cancel when he does show up last minute.
 
Last night we had the MOTHER OF ALL arguments.  I asked him to sign over custody of the kids.  I told him that if he continued to miss his court assigned visitation that I would drag him back to court and go for full custody.  He told me to go ahead that no judge would take away his rights.
 
The interesting fact is that most states treat the non custodial parents visitation as a right - not a duty.  So, in essence he can choose whether he "wants" to see them and I'm stuck.  BUT... if I tell him he can't see them then I'm going against a court order.
 
Ladies - is there anyone else out there going through this?  I have to say that this is the first time in my life that I can truly say that I HATE someone...