I use the 23/1.4 more than the the 35/1.4 but that's more of a field of view decision. Both are startlingly good lenses.
I forced myself to use the 35/1.4 last night for an evening/nighttime staff party and have kind of fallen back in love with it. Autofocus (X-E2 with 4.00 firmware) doesn't feel as quick or accurate as the 23, I have no data to back that up.
35mm is available widely secondhand, cheaper in retail terms, considerably smaller and somewhat lighter. As it's a longer focal length it obviously makes it easier to throw backgrounds out of focus.
The 23mm is arguably a more general-purpose focal length. The lens itself is technically stunning. Rarely any need to stop down, especially now I have the electronic shutter option.
It's hard to give advice as lens choice is such a personal, subjective thing, and Fuji don't really make any dud lenses. The best way to decide is really what focal length is going to be of more use to you.
As an alternative setup, maybe consider Samyang 12/f2, Fuji 18/f2 and Fuji 35/1.4. I personally think the 18mm is a really underrated lens, gives great output, bright enough, compact, light and cheap.