So far, there have been 3 returning faces from Breaking Bad, but since BCS is set before BrBa, I think the most you'll be missing out on is the "oh, it's him/her!" feeling when you see familiar characters. Of course something might change and they might do some kind of flash forward or even time skip in the future, but based on what has been aired already, I say go for it.
I found the concept of BrBa far far more interesting and appealing, and so far I still do, since we kind of know where BCS is heading in terms of where Saul will end up in the near future, but these past 2 episodes have made me warm to the series a lot more. Some stellar acting too.