I think... I can't check now but maybe it's to do with types of familiars to the user? Each familiar has an icon somewhere, I think it's the upper left of the familiar status, saying what type it is? (There's a specific word but cannot remember it!) So the Mite has an armoured looking icon?
When switching around familiars to who uses what, there are 3 tiny icons above the 3 slots for familiars, these are the familiar icons that are best suited for that user. They animate when you hover a "matched" type to that user, so the soldier icon thingy animates when you put the Mite on Oliver. I'm pretty sure it gets some bonuses and I guess XP is also part of this? I read this somewhere in the game but cannot for the life of me remember where but I wouldn't be surprised if it was this...
edit: Could be wrong though, apparently they have different growth rates?