Pottery, bone, metal, stone; rarely wood, leather, tissue.Our ancestors left a huge amount of litter, archaeologists literally make a living out of digging this stuff up. Ancient rubbish dumps have been excavated all over the place.
Nothing even close to the destructive power of plastic. Not even in the same ballpark.