I eventually went back to that mini-quarry to take more close-up photos and take some samples. There was very heavy rain that day, so the rocks are all soaking wet.
I took two samples back home to test :
I put some white vinegar on the rock on the left :
Nothing happened, there were no bubbles or any kind of fizzing or chemical reaction.
I then got a knife, the rock on the left was very difficult to scratch, hardly making a mark. And was blunting my knife. The rock on the right was easier to scratch.
Also, although you can't really see from these close-up photos, the rock on the left contains what looks like small quartz deposits, and sparkles.
I'm not convinced it's limestone, as I was expecting to see white bubbles with the vinegar. Is there anything else it could be?