I've got a dedicated server running qmail with around 3000 users who in total have around 1.8million emails stored, around 500gb of data. I'm trying to find the fastest way to back these up nightly to Amazon S3 and I'm losing the will to live now!
I've tried gzipping them all up in to a single tarball which I could then upload to s3 with aws tools but over 24 hours later it was still running so a daily backup wouldn't be possible. I've tried s3 tools to do one full backup and incremental but it appears it takes the same amount of time to just stat the files for upload. I've tried mounting S3 with fuse but it's even slower and flakey at best as it keeps losing the S3 bucket credentials somehow. I've now resorting to just tarring everything in to one uncompressed file and then uploading it with aws, 6 hours approx to tar every file and then however long it's going to take to upload.
Is there any easier/faster solution?
I've tried gzipping them all up in to a single tarball which I could then upload to s3 with aws tools but over 24 hours later it was still running so a daily backup wouldn't be possible. I've tried s3 tools to do one full backup and incremental but it appears it takes the same amount of time to just stat the files for upload. I've tried mounting S3 with fuse but it's even slower and flakey at best as it keeps losing the S3 bucket credentials somehow. I've now resorting to just tarring everything in to one uncompressed file and then uploading it with aws, 6 hours approx to tar every file and then however long it's going to take to upload.
Is there any easier/faster solution?