My thoughts...
There could very well be a market for such a PC. I'd say it's worth performing some form of market analysis to try and ascertain whether the demand is sufficient, instead of relying on us computer geeks - we may or may not be interested, but I don't think we're representative of the general public (in terms of technological awareness).
Not too long ago there was a company that announced it was going to offer a free PC for those prepared to periodically sit through pop-up adverts. If I remember correctly, the spec of the PCs offered was hardly remarkable, but a surprising number of people registered interest. I believe the project never got off the ground, but it does show that people aren't necessarily bothered about having a high-end PC. It's not much of a step-up from a low-end PC displaying pop-ups to a low-end PC that costs less than a month's council tax for most of the country.
The marketing doesn't have to be expensive... create a web site, have a publicity stunt to generate awareness and let the punters do your advertising for you via word-of-mouth.
However, I think it would take a large company to be able to achieve this - not a sole trader. You'd need to rely on economies of scale to make this worthwhile, for everything from buying the components, to getting discounts with shipping companies. Doing it on a small scale, e.g. a couple per week (assuming you could generate the demand yourself locally) wouldn't yield sufficient returns. I can, nonetheless, imagine Google and someone like Mark Shuttleworth (founder of Ubuntu and Thawte) doing something exactly like this, in an attempt to push non-Microsoft technologies. To them, it probably wouldn't even matter if the PCs themselves were loss leaders.
In fact, the more I think about this, it would be a great project to do, just in order to raise the awareness of Linux and open source software. I'm almost tempted to set up a dummy company, knock up a whizzy web site to get people hooked, post an article to /. or The Register, do a few interviews with the BBC, Wired magazine, the tabloids, etc... Then just say that the company is "reconsidering its objectives" and pull the can on the entire thing - but at least the average person on the street would then be aware of the existence of Linux/FOSS.