I don't miss the really small stage honestly. We've got twins and that was a hard slog with few breaks and you entire existence revolves around the kids, their naps, their ability to do activities and want to do activities and the chances one of them was ill or grumpy for whatever reason was high.
2 onwards has been great and its getting better all the time. They are now proper little people who can talk with you, have quirky little personalities and are much broader in their interests and capabilities.
I imagine that you forget about them somewhat eventually but as your kids get older and have those, what I assume are harder years (teenager) you yearn for when they were cute and innocent and anything was possible. I don't know, its hard to say. I certainly overthink these things a little and when I see negative traits or my own reactions to their behaviour reflected in them I feel bad. Its so hard trying to raise kids that aren't little ***** but are also free and able to be happy and not worried about life too much. One of mine has little issue with this and the other takes things to heart a little more and I feel really bad.
I've just turned 39 and my partner will be 38 in August and we really need to decide if we want to go again. Its really hard to know what the right decision is because children are wonderful but 2 is manageable, 3 feels considerably harder in various ways and we have only just got to the stage where a lot of the baby/toddler strain is over. Going back to those restrictions seem mad but we would quite like a third.