Very valid point. At the end of the feudal system when property crime still hadn't quite become the primary focus of crime, most punishments were inflicted upon the body (we have a lot more fines today), and that didn't exactly do much. I think fear of being "pressed" or flayed or hung doesn't reduce crime. no one commits a crime intending to be caught, and being punished doesn't make you much less likely to commit another crime.
Rehabilitation is key, and the way we do it now is ****-poor. People come out having been identified as criminals, and adopt a criminal identity (ref Lemert and Becker re labelling theory), making them more likely to re-offend having been ostracised from society.
There doesn't have to be an "all or nothing" approach, just one better tailored to the individual at hand. Different offenders should be treated differently.