How should i Get my child back from her grandmother?

ima 22 yr old male and my girlfriend is 25. we have one daughter who is not even 2 yrs old yet. My gf's mother often keeps our daughter for a few days at a time. Just last week she came to get our daughter for what i thought to be another casual visit. Now she is refusing to bring her back unless my GF leaves me and gets her own place. She is also black-mailing us saying that if we come to get her she will call the coops and DHS and tell them i deal weed. which i DO NOT. I DO however smoke it occionally, but never around my daughter. (usually after she's asleep) Can someone knowledgeable of the laws help me out on this one? My daughter is all i have and our family is not complete without her here. If they search my house they will find nothing except maybe some papers. I cannot,however, p a drug test but can they seriously take your child over weed?