70,000 email database, how to not get blacklisted?

[oliver2601]

Hello,

My client has an email database of roughly 70,000 users for its e-commerce site. It sends newsletters out 5 times a week. They want their emails to be sent faster, but we want to avoid blacklists.

The client currently runs PHPLIST on a dedicated server. The client would like the emails to be sent faster, as it currently takes roughly 9 and a half hours to complete. Ultimately they would like them to be sent in around 4 hours.

Currently they are using batch processing and domain throttling which alters how fast the emails are sent out. The numbers are as follows:

###BATCH PROCESSING### (The processing of general emails)
Batch Size: 10,000
Batch Duration: 3600 seconds (1 hour)
10,000 per hour

###DOMAIN THROTTLING###(The process of emails to specific domains (e.g. hotmail,gmail)
Batch Size: 999
Batch Duration: 900 seconds (15 minutes)
4000 per hour

Now, 30,000 of these addresses are microsoft emails (hotmail,live,outlook) etc.

I believe that doubling the batch sizes (or halving the batch duration) would also half the amount of time it takes to completely send. As long as the server could handle it. But the main concern is avoiding blacklists, especially HOTMAIL since that's what 42% of their customers are using!

Is this a reasonable time to complete sending?
Roughly how fast can emails be sent before getting blacklisted?
Are their any ways to create a workaround? (Multiple servers? idk)
Suggestions/Advice?

Thank you for your time,
Oliver

 

[Mynight]

I assume they have an unsubscribe link in them?(might be urban myth but it's supposed to legitimise emails somewhat).

Another option would be to ask users to add your email address to their allowed list. I would assume Microsoft take into account if emails are marked as not spam/spam rather than just checking content.

I assume you have implemented dkim? (Not a guarantee but helpful)

Lastly the way the majority of spammers get around blocks is by using fresh IPs. I would assume if your company split the mail between multiple ips it wouldn't be as obvious a bulk mailing. Separate servers on same ip would achieve nothing.

It's not really the sort of thing they make public otherwise the spam filters would be useless.

I don't know what the cost is but there are a few bulk mailing companies that may be able to process it faster and for cheaper than your current bandwith/power usage. Amazon SES is supposed to be great.

 

[ocmanus]

As above. There are a few companies such as Sendgrid that allow up to 50000 per month for free and then the cost of sending is pretty cheap. A lot of hassle saved using a established one regarding delivery rate plus you get access to analysis tools for marketing activities.

 

[oldbag]

Use SPF records and DKIM keys, then build the reputation up from your IPs slowly (ramp up emails slowly).

 

[Mynight]

As above. There are a few companies such as Sendgrid that allow up to 50000 per month for free and then the cost of sending is pretty cheap. A lot of hassle saved using a established one regarding delivery rate plus you get access to analysis tools for marketing activities.

No reason why you couldn't split the emails between the various providers up to their max free limit

 

[DanF]

Sounds like the IP is already warmed up with the amount of emails you are currently sending. If it's a reputable ecommerce site then don't go and do anything crazy like splitting out IP and servers etc.

How are the email server resources (cpu etc) and outbound bandwidth coping with the current load? Often when the IP is warm you can just keep on creeping up the delivery speeds over a period of time but eventually you'll hit server or connectivity issues.

 

[oliver2601]

I assume they have an unsubscribe link in them?(might be urban myth but it's supposed to legitimise emails somewhat).

Another option would be to ask users to add your email address to their allowed list. I would assume Microsoft take into account if emails are marked as not spam/spam rather than just checking content.

I assume you have implemented dkim? (Not a guarantee but helpful)

Lastly the way the majority of spammers get around blocks is by using fresh IPs. I would assume if your company split the mail between multiple ips it wouldn't be as obvious a bulk mailing. Separate servers on same ip would achieve nothing.

It's not really the sort of thing they make public otherwise the spam filters would be useless.

I don't know what the cost is but there are a few bulk mailing companies that may be able to process it faster and for cheaper than your current bandwith/power usage. Amazon SES is supposed to be great.

Yes, we have an unsubscribe link and yes this will legitimize emails somewhat.

That'd be fine, but it think it would seem unproffessional for a reputable e-commerce site, and the problem is avoiding getting blacklisted not to resolve the effect.

I've checked and don't believe we have dkim, this is something I will have to look into.

Understood, but somebody's got to know roughly the limit?

As above. There are a few companies such as Sendgrid that allow up to 50000 per month for free and then the cost of sending is pretty cheap. A lot of hassle saved using a established one regarding delivery rate plus you get access to analysis tools for marketing activities.

I'm aware of companies like this, although we want to send 70,000 per day. We are an established eCommerce website, so we are not looking for something just because it's free. But I will have to look into this and similar companies.

Use SPF records and DKIM keys, then build the reputation up from your IPs slowly (ramp up emails slowly).

I've just used some lookup tools and have found that we don't have SPF records or DKIM keys, this is something we will have to look into. We are currently in the process of ramping up slowly and will have to see the results.

No reason why you couldn't split the emails between the various providers up to their max free limit

We want a system that is reliable and that we can keep

Sounds like the IP is already warmed up with the amount of emails you are currently sending. If it's a reputable ecommerce site then don't go and do anything crazy like splitting out IP and servers etc.

How are the email server resources (cpu etc) and outbound bandwidth coping with the current load? Often when the IP is warm you can just keep on creeping up the delivery speeds over a period of time but eventually you'll hit server or connectivity issues.

Yes, this is what I want to avoid

I will have to look into this and get some numbers, although at the moment I believe we're being limited by these batch processing methods.

 

[oldbag]

If you don't use spf and dkim there's a high chance of mail being delivered to junk or rejected and/or your ips black listed, especially from people like yahoo, google and hotmail.

 

[Mynight]

I'm not sure about being unprofessional to ask them to set your mail as a safe sender/add to address book.

One of the competitors to ocuk do it and they're rather large. Asus and adobe both use a similar message.

 

[abbotsmike]

I'm not sure about being unprofessional to ask them to set your mail as a safe sender/add to address book.

One of the competitors to ocuk do it and they're rather large. Asus and adobe both use a similar message.

Unfortunately they need to. Spam measures are getting more draconian, spammers are getting smarter to avoid them and the measures then step up another level!!

 

[oliver2601]

If you don't use spf and dkim there's a high chance of mail being delivered to junk or rejected and/or your ips black listed, especially from people like yahoo, google and hotmail.

This will be something to implement then. I've ran a blacklist lookup and we aren't listed which is good, but as the major providers aren't listed it's hard to see the overall damage.

I've found this on the mailchimp site:
"Technically it could take us 45 minutes to send 1 million emails. Beyond this ISPs require messages be throttled to groups of 2,000 or less and sent at rates from 5 to 30 minutes apart."

So this problem might still occur even with mailchip, anybody got any data on which ISPs throttle the most?

I'm not sure about being unprofessional to ask them to set your mail as a safe sender/add to address book.

One of the competitors to ocuk do it and they're rather large. Asus and adobe both use a similar message.

Unfortunately they need to. Spam measures are getting more draconian, spammers are getting smarter to avoid them and the measures then step up another level!!

Personally I haven't been sent an email asking me to add to safe senders list, but it's good to know some of the big names do it. I'll have a look into this, but might this set of alarm bells for blacklists?
(I guess a graphic would be undetectable though)

 

[daz]

So this problem might still occur even with mailchip, anybody got any data on which ISPs throttle the most?

In my experience, Gmail and Hotmail throttle the most.

The issue we find is that it's almost an arms race to see who can have the most strict spam filtering settings and that false positives aren't deemed the fault of Hotmail/Gmail, it's always deemed the fault of the sender!

 

[abbotsmike]

So this problem might still occur even with mailchip, anybody got any data on which ISPs throttle the most?

Personally I haven't been sent an email asking me to add to safe senders list, but it's good to know some of the big names do it. I'll have a look into this, but might this set of alarm bells for blacklists?
(I guess a graphic would be undetectable though)

In my experience, Gmail and Hotmail throttle the most.

The issue we find is that it's almost an arms race to see who can have the most strict spam filtering settings and that false positives aren't deemed the fault of Hotmail/Gmail, it's always deemed the fault of the sender!

I've noticed even less emails getting through to my gmail spam box, so they are being blocked before they even get there. I also know of people sending bulk mail out that are getting bounced back by gmail quite a lot.

Just had a quick look through my inbox, Camskill, Direct line, NS&I and Endsleigh all ask to be added to a 'whitelist' to avoid emails being spam boxed.

 

[BattFink]

I would get your mail server set up correctly.

• Setup DKIM & SPF for the domain

• Get your reverse DNS records established

Register your server as a bulk messaging sender with the larger ISP's e.g:

AOL - http://postmaster.aol.com/
Hotmail - http://mail.live.com/mail/postmaster.aspx
Google -https://support.google.com/mail/answer/81126?hl=en&ref_topic=3404235#thirdparty
Yahoo - http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html

This will help ensure that mail is delivered & not outright rejected as spam

 

[oliver2601]

I've noticed even less emails getting through to my gmail spam box, so they are being blocked before they even get there. I also know of people sending bulk mail out that are getting bounced back by gmail quite a lot.

Just had a quick look through my inbox, Camskill, Direct line, NS&I and Endsleigh all ask to be added to a 'whitelist' to avoid emails being spam boxed.

Ok, brilliant. It'll definitely be something I will be including in the template.

In my experience, Gmail and Hotmail throttle the most.

The issue we find is that it's almost an arms race to see who can have the most strict spam filtering settings and that false positives aren't deemed the fault of Hotmail/Gmail, it's always deemed the fault of the sender!

Yes, defiantly have some polishing to do

I would get your mail server set up correctly.

• Setup DKIM & SPF for the domain

• Get your reverse DNS records established

Register your server as a bulk messaging sender with the larger ISP's e.g:

AOL - http://postmaster.aol.com/
Hotmail - http://mail.live.com/mail/postmaster.aspx
Google -https://support.google.com/mail/answer/81126?hl=en&ref_topic=3404235#thirdparty
Yahoo - http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html

This will help ensure that mail is delivered & not outright rejected as spam

Just done a lookup on https://www.senderscore.org/ and got a score of 98, looks pretty legit as well

Will also get those set up thanks

 

[p1RATE]

Ive been looking into this a lot recently myself and most of the advise that i have been given is to give up and use a mail service like mail chimp. Despite this not fitting my needs as i don't send out newsletter type emails

I was doing a little reading on amazon SES the other day too.

Most of my business is conducted via email and its a becoming an issue for me now as im looking to expand.

 

[Dynix]

A consideration may be to use Amazon SES and Sendy - as we do to send our newsletters.

Amazon SES: http://aws.amazon.com/ses/
Sendy: http://sendy.co/

Our newsletter has a little over 250,000 subscribers and we have a 750,000 emails/day limit with Amazon SES - allowing us to send 80 emails a second. It costs us around $25 to send 250,000 emails.

Providing your email list is clean, you won't have a problem with Amazon SES. They'll start to kick a fuss when your bounce rate is nearing double-digits or spam records is too high, otherwise they will take care of everything.

Sendy will keep track of any bounced emails and any which are reported as spam. It has a basic API to add/remove a subscriber - it's a great piece of kit.

 

[Beansprout]

http://www.returnpath.com/

But you will fall off your chair, unless you're selling yachts to all your 70,000 clients...