O365 Mailbox sync issues (Migration to O365)

[Roady]

Hi

Long story short, moved/moving my company from on premise (Ubuntu) to online Exchange with O365.

I used the Exchange 'cutover' batch migration tool for some mailboxes (populated O365 from IMAP mailboxes) but had a bunch I didn't which where previously POP3/PST's with some large quantites and files. I think leaving this running may have been causing some of the issue so switched it off yesterday. It was reporting all mailboxes as complete. I have problems with these cutovers but they seem to mostly be ok, so parking those issues until later (not had time to dig into).

The POP/PST based we did no 'pre-migration' with, so everthing was done last Thursday when we switched mail delivery to O365. A couple of users are finding their O365 mailboxes just will not reach a sync'd state with their local Outlook 365 .OST cache. Support/vendor we're working with has said it might take 4-5 hours per mailbox. Some of these are now 5+ days in and still not reach sync, they're at a loss why. Two of them here I've actually turned off the local cache so they can see 'new' mail. I think with this turned off they'll not be updating O365 so never reach a syncd state?

Process we followed was having the 2 accounts loaded on Outlook and dragging mail from one 'account' (technically Data store - their old PST) to the other. One user (with 17,000 in her Inbox) seemed to be ok, but the day after had no new mail and her Inbox. So I turned off cache but she then 'lost' all the older content in her sub folders. Another (FD!) had an ever increasing message 'count', even with lots of deleting and moving mails to sub folders. Turned his cache off and suddenly all the mails he's deleted/filed over the last 3-4 days are back in his Inbox.

Is this something people have seen? I mean some of these previous PST's are not upto nightmare levels of size, 'only' 4-5gb. Should they really take that long to synchronise to O365?

 

[#Chri5#]

In my previous job, I think we tried the drag 'n' drop between accounts (data sources) in Outlook to move content once but I think it was PITA and I vaguely remember problems, so never went down that route again. I have a memory of a colleague using the method of uploading the PSTs to Azure, then creating an import job which uses an Excel file to map the PSTs to a user's mailbox.

https://docs.microsoft.com/en-us/mi...g-pst-files-to-office-365?view=o365-worldwide

https://docs.microsoft.com/en-us/mi...g-pst-files-to-office-365?view=o365-worldwide

Another memory is an on-prem Exchange to O365 migration which hammered the companies Internet connection for days as Outlook sync'd the O365 mailbox to the OST. Do you have any monitoring of your traffic? I could look in PRTG to see the Internet was getting was running flat out.

PS. I've just had a quick cry at somebody with 17,000 e-mails in their Inbox. Nooooooooooooooooooo!

 

[Roady]

In my previous job, I think we tried the drag 'n' drop between accounts (data sources) in Outlook to move content once but I think it was PITA and I vaguely remember problems, so never went down that route again. I have a memory of a colleague using the method of uploading the PSTs to Azure, then creating an import job which uses an Excel file to map the PSTs to a user's mailbox.

Hi Chris, thats good to hear at least it's no an isolated case! Part of the problem is my experience with AD's & Exchange pretty much finished in 2010 (used to do support) until this migration at my current workplace. We moved from a Workgroup to AD around 2 years ago as part of a company wide system change (which went well) but we left our email migration until now.

I'd heard of that PST Azure upload and hadn't gone that route. The volume of PST's it just didn't seem worthwhile (less than probably 50gb). The vast majority of (large) mailboxes I used the 365 IMAP migration tool for (including one of them who previously had a 30gb PST) and I'm happy how that has gone. Although one user with it had issues, he did have 51,000 in his Inbox. I enabled the '365 In-place Archive' for him and it fairly quickly trimmed that amount down to around 10,000 and then got in sync.

Another memory is an on-prem Exchange to O365 migration which hammered the companies Internet connection for days as Outlook sync'd the O365 mailbox to the OST. Do you have any monitoring of your traffic? I could look in PRTG to see the Internet was getting was running flat out.

PS. I've just had a quick cry at somebody with 17,000 e-mails in their Inbox. Nooooooooooooooooooo!

Try 51,000!

Have not noticed the internet getting hammered but we're on a 100mb leased line so that's really not too much of a concern. Equally because of that, the sync taking so long is even more of a mystery. I wouldn't expect a mailbox with a 'corresponding' PST of 5-8gb to take over a week to synchronise!?

Here you go... You'll enjoy this one... Don't forget that's unread emails!

 

[#Chri5#]

I'd agree the OST sync should have completed by now. I'd be inclined to nuke the OSTs tomorrow afternoon and leave them syncing over the weekend without user intervention (if possible).

Is it just the mailboxes manually uploaded from PSTs with the issue?

 

[Roady]

I'd agree the OST sync should have completed by now. I'd be inclined to nuke the OSTs tomorrow afternoon and leave them syncing over the weekend without user intervention (if possible).

Is it just the mailboxes manually uploaded from PSTs with the issue?

Yeah, seems to be these two where we drag & dropped mail from a local PST to the O365 'account' (OST) with issues.

Left one of the PC's on overnight with logout/power off disabled and Outlook left open with cache enabled. From around 2 or 3pm yesterday until around 8 this morning, so 17-18 hours and didn’t reach sync, still showing mail from Tuesday as the newest. Her old PST was 8.6gb so not massive. No reason I can see why that can't have been long enough to sync.

2 other users without issues, I used the Outlook 'Import' tool on day 1 to import their PST's and they've never had a sync issue. 10gb and 14gb, so I'm convinced it's not the size/mail volume is the issue, just the process.

This user I'm happy to nuke the OST and let it rebuild but technically I have already done that (although that was back on the 3rd). I have a backup of her PST from the 27th before we started her migration, so really tempted right now to clear down her O365 mailbox of anything older than the 27th. Then reload it on her Outlook and let it get in sync with very little mail. We then can decide to try the import again using the Import tool, or we just attach her old PST locally for her to use if she needs anything prior to the 27th.

The other mailbox is in more of a mess, so still unsure what the best thing to do. Annoyingly he's our FD so whatever I do I really have to be sure will 'fix' it. Annoyingly the only PST backup of his I have is from the 17th.

 

[Webster]

I always manually import/export, the MS tools never seemed reliable..

 

[Roady]

How? The 'manual' import (click & drag from PST into OST) is what seems to have screwed this up!

The other mailbox is in more of a mess, so still unsure what the best thing to do. Annoyingly he's our FD so whatever I do I really have to be sure will 'fix' it. Annoyingly the only PST backup of his I have is from the 17th.

Although saying that, this one I've left alone that I thought was utterly screwed is now showing old messages in sub folders which where previously filed and not visible. With the cache turned off... So I think it's eventually getting in sync. I'm not touching it as it seems to be sorting itself out!

 

[Webster]

Run the export then import back into outlook, it takes some time to re-sync but never had any issues, some of our mailboxes are pretty large.

Where you have dragged from one to another, instead run an export from account A then import into B, it will ask about duplicates etc and you can choose not to make them.

 

[Roady]

Yeah that's the sequence I've done for at least 2 with original PST's (so no need to export) over 10gb and they've had no issues. Will be the process I follow with the 15-20 users I've got left to do (majority are very low usage so PST's in 1-3gb ranges).

I was just after some tips with sync - some way to get these back working without basically wiping them and starting again. But that's what I've pretty much decided to do for this one (in the process of clearing down the O365 mailbox). The other (my FD) appears to be getting in sync so I'm leaving until it reads 'All folders are up to date' before doing anything more with it.

Thrown myself in the deep end with trying to clear this mailbox with Exchange Powershell. Have not got it working yet... Seems an awkward and overkill way to do what should be a 'simple' task (deleting mail before a certain date).

Search-Mailbox -identity "User" -SearchQuery '(received:01/01/01..23/02/20) AND (kind:email)' -DeleteContent

I toyed with the idea of enabling the Online Exchange with a custom filter and then just clearing that down but doesn't seem to be a way of going into that amount of detail with it's dates. If I can't get powershell figured out I'll be trying to highlight and delete all these messages folder by folder in OWA which I know will be problematic?

Another thought, cache off and sync Outlook (which we know won't complete) from another PC/Profile, yet every time it pulls some mail into the Inbox before my set date, just delete it, gradually working my way through the 17,000+ messages (probably 15+ years of email). But maybe that won't work - as without sync it won't remove from mailbox until it's actually in sync first? Or will that work as sync is only for the cached copy? My brain is fried so parking that until next week...

 

[5tephen]

Done a good few of these. It's slow due to 365's throttling but it has never failed or needed intervention with caches etc.

Bear in mind that the throttling is imposed on your tenant, not just per mailbox, so while importing 1 or 2 mailboxes will be slow as it is, doing multiple of them will be even slower again.

Empty the deleted items out of the PST's. 2 minutes work which will save potentially hours. I believe that throttling is imposed even further for the deleted items folder? I read that somewhere, or got a warning about it when doing an Exchange-to-365 migration in the past I believe.

Always use the Outlook import wizard. That will quite speedily import the PST into the OST file, let that OST file sync up in the background, and keep Outlook responsive while it does so. Messing about with cached mode on/off will not make this happen any quicker. Leave it as it is, i.e. cached. Don't drag and drop, again Outlook is unusable while dropping in, and again it doesn't transfer stuff any quicker.

Bear in mind that while the uploading is happening, Outlook will rarely, if ever, show new emails. We always advise users to use the webmail to receive new mail for a day or two and just keep Outlook running in the background to sync up. I've also noticed that closing and re-opening Outlook during the syncing will sometimes bring down blocks of new mail, if somebody really wants to get something urgent into Outlook.

Always best done on a Friday afternoon, tell people to leave their machines on over the weekend with Outlook open, importing, and turn off auto-standby on the power settings. New mail will come into 365 on their phones over the weekend which keeps most people happy.

 

[Roady]

Any idea why Outlook with cache turned off to a 365 mailbox would show an empty 'Sent Items'? Log into OWA and there's hundreds there (not 1000's) so don't really expect a problem... Not going to spend too much time on it as it's my user I've messed around lots with (who I'm also setting up a new PC for, so trying to get mailbox sorted in sync before setting up new profile).

Done a good few of these. It's slow due to 365's throttling but it has never failed or needed intervention with caches etc.

Bear in mind that the throttling is imposed on your tenant, not just per mailbox, so while importing 1 or 2 mailboxes will be slow as it is, doing multiple of them will be even slower again.

Empty the deleted items out of the PST's. 2 minutes work which will save potentially hours. I believe that throttling is imposed even further for the deleted items folder? I read that somewhere, or got a warning about it when doing an Exchange-to-365 migration in the past I believe.

Good call and a good practice to get into, I hadn't really done this as wasn't expecting mail volume to cause as much issue as it has so will be doing it for any others. Good info on throttling, is there any way to see it's status or load via tenant? We've 61 Office 365 licences if that has any bearing on the tenant allowance? I could then use that to plan these other imports to have the least amount of impact. I've drawn a bit of a blank looking for any kind of 'sync status' from the mailbox side of things so far.

Always use the Outlook import wizard. That will quite speedily import the PST into the OST file, let that OST file sync up in the background, and keep Outlook responsive while it does so. Messing about with cached mode on/off will not make this happen any quicker. Leave it as it is, i.e. cached. Don't drag and drop, again Outlook is unusable while dropping in, and again it doesn't transfer stuff any quicker.

That's my plan, just a shame I've learnt this the 'hard' way. I'm rather annoyed the 3rd party support hasn't mentioned anything like this to me, they're meant to be doing this migration with us but I seem to have drawn the short straw with doing the actual mailboxes while they just did the initial setup...! I'd requested any hints, tips, prerequisites and such from them and didn't really get anything useful.

Thanks for the info and insight. Users here are quite 'light' in terms of PC usage but email data is one thing we seem to have a ton of. I'm quite disappointed Microsoft haven't got more tools for this for people like us coming from outside the Exchange environment. The Exchange batch migration has worked quite well for me (for those users on IMAP), the rest have been so awkward and time consuming I'm half wishing I'd convered them to IMAP before, imported the mail from PST's to our old Ubuntu server and used the batch migration. It would've been easier and quicker to get away from PST files before the migration.

 

[5tephen]

When you click a folder, e.g. the sent items that you mention, it probably says 'updating this folder' at the bottom right of outlook. Sent items is second in line to the inbox and inbox subfolders.
That's a serious amount of PST files dotted around the place. How did anybody let a place grow to that size without Exchange, wow. My experience lately is mainly importing archives into existing mailboxes, or archives into mailboxes while an Exchange-to-365 migration is also running in the background for the tenant. It's slow, but never fails.

After about 10 mailboxes the throttling becomes worse. There's no way to check the throttling status that I know of, but I've never really looked, I just know to work with it. We inherited a new customer with a failed Exchange server, and used third party software to lift mailboxes out of the Exchange EDB file directly into a newly-created 365 tenant. 80 mailboxes. I had to choose 5-6 at a time and prioritise, because running more than that was just taking ages per mailbox. The customer's connection upload was not being stretched at all.

IMAP might have been easier to migrate to Exchange now but it's terrible for anything but small mailboxes. You would have been tortured with PST cache file issues with that many IMAP mailboxes sitting around your estate.

 

[Roady]

When you click a folder, e.g. the sent items that you mention, it probably says 'updating this folder' at the bottom right of outlook. Sent items is second in line to the inbox and inbox subfolders.

Yeah I'd figured there was a priority of some kind and just assumed it was mail 'items' (regardless of folder) then saw about things like Older mail, Deleted Items and Junk having a lower priority I figured it must be by Folder 'name' (or those Exchange kinda defaults).

That's a serious amount of PST files dotted around the place. How did anybody let a place grow to that size without Exchange, wow. My experience lately is mainly importing archives into existing mailboxes, or archives into mailboxes while an Exchange-to-365 migration is also running in the background for the tenant. It's slow, but never fails.

?? Not everyone uses Windows servers!

What's a serious amount of PST files? The 61? That's O365 mailboxes, not PST's - thankfully over the last few years I've changed many of the 'busy' accounts (Sales) over to IMAP and imported the old PST mail back in to them. Batch migration did 32 of those for me.

I do have another PST dump of old Archives to get into 365 at some point... 250gig worth! Got to make sure I've got 'live' across and all the inplace archives working first, so hopefully they won't impact users current sync very much (if at all)... As from what I've read you cannot import directly into the in-place Archive...

Exchange in my experience would have been a good fit here, but the company wouldn't have liked the cost, or administration, until recently. Our systems here where quite backwards until the last couple of years - 4 physical Linux/Ubuntu/Postfix running bespoke databases, mail & backups with only Windows workstations on a Workgroup. Low cost, had easily expanded from 28 'active' network users to the 65 + 15 mobile we have now over the years. The bespoke software was around 25 years old with things like mail and file shares 'bolted on' when required by that provider. With more and more mobile devices and access to data requirements really showed the age of the old bespoke system so we went through a big infrastructure and software change (ERP) to 2 physical hosts and 6 VM's, Windows Servers, AD and now running Dynamics NAV for everything. So that was the switch over to AD and also implementing O365. Roll on to now and I'm completing the email migration to 365.

After about 10 mailboxes the throttling becomes worse. There's no way to check the throttling status that I know of, but I've never really looked, I just know to work with it. We inherited a new customer with a failed Exchange server, and used third party software to lift mailboxes out of the Exchange EDB file directly into a newly-created 365 tenant. 80 mailboxes. I had to choose 5-6 at a time and prioritise, because running more than that was just taking ages per mailbox. The customer's connection upload was not being stretched at all.

IMAP might have been easier to migrate to Exchange now but it's terrible for anything but small mailboxes. You would have been tortured with PST cache file issues with that many IMAP mailboxes sitting around your estate.

IMAP here on our old system has been rock solid, but maybe that's the difference between Exchange and other mail solutions!

I had two users with mailboxes at 30 and 60gb and the batch migration worked well with them (In-place Archive enabled for them!). Majority of the others around 5-8gb it also did well. I had it running for nearly 2 months until we where 'ready' to migrate. Thankfully these 'old' POP3/PST based accounts are generally for light users so the files are 1-2gb, maybe 4gb at the most for me to import.

Really useful info about the throttling thanks, I've got 2 mailboxes here still going (one is 7 days in, the other 3) and hadn't wanted to do much other importing. Although I bit the bullet yesterday and did 3 others, PST's at 1.8gb and smaller. The largest of them with 4800 mail in it's Inbox took around an hour. So that gives me a good estimate for time. I'll be moving forwards with the others now I know I can without realistically hitting any throttling, certainly not for the 1's and 2's I'll be doing at a time, letting them get in sync before then moving onto the next.