You best bet would actually be to get a low profile sub, or a small active bass unit with built in amp. Both are quite small.
A shelf mounted sub will take lot of building and will never be as good as a boxed sub. The bass won't be as sharp or accurate unless the shelf is really well made and supports the sub firmly and dosen't move, but there's still the lack of sharp bass from what I've heard in the past.
Pioneer make a low profile sub, the model name is the TS-SW3041D. MY friend had the original model of this and it sounded very good and the bass as a lot more accurate than my same sized infinity sub which had a full depth to it, running off the same amp and headunit in my car.
All you would need to do is make or aquire a sub enclosure designed for the sub so it's the exace specs to what the sub requires. Not a difficult task with the right tools and can take up less footprint than a normal box, could even be dropped into a small false floor to protect the amp too.
A sub will only be as good as its enclosure. I've heard the most expensive subs in a standard box and I have heard cheap subs in a taylor made enclosure. The chep subs won hands down, but were still constraned to a few specs, but were still better sounding.
You could build a low profile sub into a low profile box that sits under a shelf, now that might work. Experiment is what I say.
