Yeah, I thought in the book he escapes and is basically hunted by bolton for fun and after that he was tortured.
That's right, it's just the way it's portrayed in the show is slightly different and initially confusing. According to the Wiki, the timeline of events for Theon is:
Theon has been tortured repeatedly by Ramsay Bolton. Ramsay removes the skin on several of Theon's toes and fingers, leaving him in agony for days before removing the joints but only after Theon begs him to remove them. He broke and removed several of Theon's teeth because Ramsay hated his smiles. It is also implied, but not confirmed, that Ramsay removes Theon's genitals as well. Theon is forced to take on the guise of Reek, Ramsay's former serving man. Theon is forbidden to bathe and is caked from head to toe in filth and excrement.
Due to the torture he endured at Ramsay's hands, Theon's appearance has dramatically changed. He looks like he has aged forty years and several of his teeth have been knocked out, many more are loose meaning Theon cant even eat without suffering agony. Theon has lost three stone in weight, his skin has turned pasty, and his hair is white and brittle. He can no longer use a bow due to the loss of some fingers and hobbles like an old man due to the loss of several toes.
Before he endures the worst of Ramsay's tortures, Theon once escapes with the help of Kyra (his former bedwarmer at Winterfell). It turns out, though, that they are allowed to escape by Ramsay. Ramsay lets them get a day's head start before hunting them with his hounds. Theon is taken back to the Dreadfort dungeon and Kyra meets a cruel and grisly death at the hands of Ramsay and his hounds. Theon is mentally and physically broken by Ramsay and lives in great fear of him.
After Ramsay's father announces his return to the North, Ramsay has Theon washed and properly clothed, and sends him to deliver peace terms to the Ironborn occupying Moat Cailin, promising them food and safe passage if they surrender. When they do surrender, Ramsay has them all flayed alive and displays their skinless bodies along the road to the Moat. Theon is present when Ramsay is introduced to his bride "Arya Stark" who Theon instantly recognizes as Jeyne Poole.
Because Theon was a ward of Winterfell for ten years and is the closest thing to kin "Arya" has, he gives the bride away at the wedding. Jeyne pleads with Theon several times before the wedding to help rescue her. Theon refuses. Ramsay has Theon strip his bride for him and makes Theon watch as he humiliates and degrades her. Theon silently hopes that when Stannis comes he will kill Ramsay Bolton.
When several Bolton men are murdered at Winterfell, Theon is thought a suspect. The idea is quickly dismissed by Roose Bolton, who claims that Theon is too broken and weak to have carried out the murders. The murders cause tension in the castle between the Freys, Boltons, Manderlys and other Northern houses.
The musician at Winterfell "Abel" and six wilding spearwives disguised as maids are actually responsible for some of the deaths and use the confusion in the castle to free "Arya Stark," enlisting the help of a very reluctant Theon. In the end, Theon and Jeyne manage to escape and are caught by Mors Umber who sends them to rendezvous with Stannis's army a few days ride away.
Asha is a captive with Stannis's army and barely recognizes Theon due to his torture at the hands of Ramsay.
