is the skin colour mismatch between them and the potential baby really a big issue?
Yes.
My wife works at the front end of children's services, and I am continually amazed at the depths of depravity and abuse that people will subject their children to.
These kids are then taken from that situation into the care system where they will end up in the care of many different adults. It's not uncommon for kids to have been looked after by multiple guardians as they are passed around the social care system. Every move is an attachment broken, trust is eroded and more damage done to the child's ego.
As a prospective adoptive parent every single aspect of your life and history is scrutinised in minute detail. Your mental, physical and financial health, your house and surrounding area. Your religion, political views, choice of friends and your childhood. Your history of past relationships. All of these things to ensure that the right match can be made.
The purpose of finding their "forever family" is to provide a solid, stable, trusting environment for the first time in the child's life. By the time the child reaches the adoptive home they are truly, utterly broken. They have no concept of identity, ego, trust, compassion, safety, love, cleanliness, warmth, satiety. Every adult they have know has either abused them or left them. They don't know how long they will be staying in this house, who these new people are, or if they are going to start doing bad things to them. At the very base level, if their skin colour is different, then that means they are different.
You have a melting pot for disaster and every step must be taken to mitigate that to aim for a smooth placement. There are so many different variables that will affect that. Race is one thing that you absolutely can control.
So it might seem politically correct and right-on to say things like "it's 2017, colour doesn't matter", but in terms of adoption, it really does.