1+1=5?

[adolf hamster]

so i'm running into a curious issue here.

i have a drive that is dedicated to work stuff, it's a 2Tb drive and all it contains is work related files (folders, documents, images etc etc)

this is backed up to my works onedrive/sharepoint system which has 1Tb of space.

the work i've been doing has a lot of data, specifically images and tens of thousands of these images led to the total usage going over 1tb.

no problem i thought, i'll compress the datasets that i'm not going to be revisiting any time soon, so i did the ol' send to compressed folder and delete the original folder.

this brought the usage down to 918gb (on disk) which is fine and dandy.

that's the local side, so best i can tell that's what's actually on the drive.

of course onedrive doesn't like having to deal with that and is stuck trying to upload over the 1tb limit before it'll process the operations required to downsize everything.

needless to say, not having backups on the cloud is worrying, that's a lot of data and important documents to have stored on just one disc.

so, my current plan is to disconnect onedrive from the local drive, go into onedrive via the internet and purge everything, then reconnect them so it uploads the 918gb that's currently there.

of course i want to make a backup in case some kind of ****-up leads to me nuking all my work so i'm trying to copy across the 918gb of local files to an external drive.

except apparently the 918gb which file explorer and treesize agree is the entire contents of the drive magically turns into 1.4Tb when trying to copy it across, the destination drive has 1.5tb of free space which should be fine yet windows won't copy it because the drive it's copying from doesn't have enough spare space.

anyone know what windows is smoking to decide this?

 

[JohnG7]

I vote pancake!

 

[Pottsey]

so i'm running into a curious issue here.

i have a drive that is dedicated to work stuff, it's a 2Tb drive and all it contains is work related files (folders, documents, images etc etc)

this is backed up to my works onedrive/sharepoint system which has 1Tb of space.

the work i've been doing has a lot of data, specifically images and tens of thousands of these images led to the total usage going over 1tb.

no problem i thought, i'll compress the datasets that i'm not going to be revisiting any time soon, so i did the ol' send to compressed folder and delete the original folder.

this brought the usage down to 918gb (on disk) which is fine and dandy.

that's the local side, so best i can tell that's what's actually on the drive.

of course onedrive doesn't like having to deal with that and is stuck trying to upload over the 1tb limit before it'll process the operations required to downsize everything.

needless to say, not having backups on the cloud is worrying, that's a lot of data and important documents to have stored on just one disc.

so, my current plan is to disconnect onedrive from the local drive, go into onedrive via the internet and purge everything, then reconnect them so it uploads the 918gb that's currently there.

of course i want to make a backup in case some kind of ****-up leads to me nuking all my work so i'm trying to copy across the 918gb of local files to an external drive.

except apparently the 918gb which file explorer and treesize agree is the entire contents of the drive magically turns into 1.4Tb when trying to copy it across, the destination drive has 1.5tb of free space which should be fine yet windows won't copy it because the drive it's copying from doesn't have enough spare space.

anyone know what windows is smoking to decide this?

Is it something like the external drive is formatted to fat32 or fat32ex as that would explain your problem. If it is assuming you can wipe it, reformat it for NTFS.

 

[adolf hamster]

Is it something like the external drive is formatted to fat32 or fat32ex as that would explain your problem. If it is assuming you can wipe it, reformat it for NTFS.

both drives are ntfs

however formatting it might be possible, i have a third drive with enough free space to back up the current contents of the external (assuming it doesn't pull the same shenanigans).

 

[Tanin'iver]

Interesting one. Couple of ideas - does the original 2 Tb drive currently containing the files have compression enabled in Windows itself?

Secondly, use something else to look at the contents of the drive - something that doesn't respect the normal Windows API, see if it shows stuff on there that Windows isn't showing you. I use a program called FreeCommander when copying over stuff from my main storage drive to backup, since it has a sync feature that lets you see what's present on one of the drives and not the other (or other differences) and only copy over what's required.

 

[adolf hamster]

Interesting one. Couple of ideas - does the original 2 Tb drive currently containing the files have compression enabled in Windows itself?

Nope, it does have some folders which are manually compressed but there isn't compression applied via windows

Secondly, use something else to look at the contents of the drive - something that doesn't respect the normal Windows API, see if it shows stuff on there that Windows isn't showing you. I use a program called FreeCommander when copying over stuff from my main storage drive to backup, since it has a sync feature that lets you see what's present on one of the drives and not the other (or other differences) and only copy over what's required.

I did try the free version of treesize on the whole drive and while it turned up a few gb of temp files outside the folder i'm trying to copy it agreed on the ~918gb size for that folder.

 

[Tanin'iver]

Take a look at FreeCommander too anyway, see what you find. It will show up all kinds of stuff not normally visible, even the strange invisible files inside a recycle bin that Windows uses to uniquely identify it.

Other than that, try something other than Windows to actually do the copying - like FastCopy for example.

 

[adolf hamster]

Take a look at FreeCommander too anyway, see what you find. It will show up all kinds of stuff not normally visible, even the strange invisible files inside a recycle bin that Windows uses to uniquely identify it.

Other than that, try something other than Windows to actually do the copying - like FastCopy for example.

Will give that a try tomorrow, currently backing up the contents of the external drive in case it needs reformatted.

Only thing i can think is either onedrive is hiding files (eg the folders i deleted after compressing them) or windows is deciding instead of copying the zip folder to instead unzip it first?

Doesnt explain why it needs the free space on the drive its copying from in order to copy, surely it's only reading from the drive and writing to the external?

 

[Tanin'iver]

Doesnt explain why it needs the free space on the drive its copying from in order to copy, surely it's only reading from the drive and writing to the external?

's a good question - I don't know what the people who coded the current versions of Windows were smoking when they wrote the file copying code though.

If I had to guess, I'd say maybe with it being external the system is trying to use a temporary folder to copy to first since the destination drive being external it might treat as removable. It still shouldn't need to do that of course, but that's my best guess. FastCopy might work around that since it uses its own copying code instead of the Windows stuff.

 

[thenewoc]

Maybe because your folder on the external drive is not compressed so it's uncompressing the files at source before copying them over?

 

[adolf hamster]

's a good question - I don't know what the people who coded the current versions of Windows were smoking when they wrote the file copying code though.

If I had to guess, I'd say maybe with it being external the system is trying to use a temporary folder to copy to first since the destination drive being external it might treat as removable. It still shouldn't need to do that of course, but that's my best guess. FastCopy might work around that since it uses its own copying code instead of the Windows stuff.

Yeah, i can see the temporary folder element, although that would be kinda dumb as you'd never be able to fill a drive beyond half? Or apparently not even that far given the present situation.

Only other possibility is if onedrive is messing with it? The folder is still within the onedrive folder although onedrive itself was signed out when i was trying this.

All local copies, there shouldnt be any online only files on there as the whole point was to have them in 2 seperate places.

Maybe because your folder on the external drive is not compressed so it's uncompressing the files at source before copying them over?

The compressed folders are full on .zip archives, which should just get treated as a single entity?

 

[adolf hamster]

ok, so looking via freecommander, with the hidden folders turned on it shows the recycle bin folder outside the folder i'm trying to copy, but it's registering as empty.

it seems to be saying 918gb although the right-click properties menu seems to just be using the windows default?

 

[Tanin'iver]

Yeah getting the properties via FreeCommander just uses the Windows default - you'd only actually see files not normally visible when looking inside the view it presents. If you don't see anything unusual in there, then invisible files aren't the cause of your problem.

Did you try FastCopy yet to actually do the copy, any luck with that?

 

[adolf hamster]

Yeah getting the properties via FreeCommander just uses the Windows default - you'd only actually see files not normally visible when looking inside the view it presents. If you don't see anything unusual in there, then invisible files aren't the cause of your problem.

Did you try FastCopy yet to actually do the copy, any luck with that?

fastcopy has been running all day, up to about 800gb transferred now.

will see if it starts running over as well.

 

[adolf hamster]

so fastcopy seems to have done a decent job, it has moved the bulk of the files over.

it did return a bunch of errors in the copy process which is worrying.

i appear to be missing 338 files, which depending on what they are could be anywhere from not a problem to a really big freakin deal.

edit: ok, fastcopy is pretty damn neat, took seconds to fill in the missing files.

windows is now giving the same state for size/number of folders/number of files so hopefully that's accurate because i ain't sorting 750,000+ items to figure out if anything's missing....

 

[Tanin'iver]

Sounds good If you want to be sure everything has been taken over, use FreeCommander again - bring up the original location in one side and the new location in the other, then use the Synchronise feature - it will look for any differences.

If you want to be REALLY sure you can click the "by CRC" checkbox and it will actually read every file and compute the CRC32 to compare it between the two locations to make sure the file contents are identical. That would take a long time though since it would have to read all the data.

 

[adolf hamster]

Sounds good If you want to be sure everything has been taken over, use FreeCommander again - bring up the original location in one side and the new location in the other, then use the Synchronise feature - it will look for any differences.

If you want to be REALLY sure you can click the "by CRC" checkbox and it will actually read every file and compute the CRC32 to compare it between the two locations to make sure the file contents are identical. That would take a long time though since it would have to read all the data.

i've skipped ahead at this point to trying to kill the original onedrive folder.

needless to say merely unlinking onedrive, uninstalling the entirety of office365, signing out of the account on this machine, restarting several times and formatting the local drive still isn't enough to kill it.

i'm starting to think investing in a NAS might not be the dumbest idea.....

 

[Tanin'iver]

i've skipped ahead at this point to trying to kill the original onedrive folder.

needless to say merely unlinking onedrive, uninstalling the entirety of office365, signing out of the account on this machine, restarting several times and formatting the local drive still isn't enough to kill it.

i'm starting to think investing in a NAS might not be the dumbest idea.....

Ick. I don't use OneDrive personally, I don't trust cloud services with my data. Local backup to external drive, updated once per month.

Glad to hear you managed to get the data copied though!

 

[adolf hamster]

Ick. I don't use OneDrive personally, I don't trust cloud services with my data. Local backup to external drive, updated once per month.

Glad to hear you managed to get the data copied though!

i suppose sharepoint would probably be more accurate, it's run through my workplace at least. i mainly use it to sync between machines and to have a backup, it's all work data and nothing sensitive, i'm just paranoid about losing it.

currently copying back the backup to the original drive, gonna sort through as there was a few folders i missed that could be compressed so i'm gonna do that before moving it back to the onedrive folder.