Root disk out of space?!?

[reefoid]

I've logged in to my Win7 machine this morning to find my Samba shares are unaccessible. After some investigation I've found that the root disk on my headless Debian server has 0% free space. I know for a fact I had around 5GB of free space yesterday so I'm a bit confused.

I'm a bit of a newcomer to command line Linux (this is my first headless server) and I've spent the last hour and a half going through various articles on the net trying to find what is using all the space but I'm not getting anywhere. I've used various du commands but I just can't find anything that has used all this space.

The only change I made yesterday was to move my backup mount (which was on the root disk) to a newly installed disk. To do this, I renamed the old directory, created a new directory and amended fstab to point to the new disk and directory and removed the old entry. I tested my backups last night and they all went onto the new disk as expected. However, the /backups directory is not getting mounted now. Is this becaus of the lack of space on root?

So, can anyone advise on how I can find what is using all this space?

Thanks

 

[deadite66]

try
sudo du -h --max-depth=1 /

 

[reefoid]

92G     /data
22M     /etc
4.0K    /opt
du: cannot access `/proc/1940/task/1940/fd/4': No such file or directory
du: cannot access `/proc/1940/task/1940/fdinfo/4': No such file or directory
du: cannot access `/proc/1940/fd/4': No such file or directory
du: cannot access `/proc/1940/fdinfo/4': No such file or directory
0       /proc
94M     /lib
4.0K    /selinux
4.6M    /sbin
40K     /home
15M     /boot
0       /tmp
4.0K    /backups
4.0K    /srv
76K     /root
386M    /var
152K    /dev
16K     /lost+found
95G     /mnt
12K     /media
1.8G    /usr
5.1M    /bin
0       /sys
188G    /

/data is a separate disk and /mnt has a Windows shared disk mounted. But there is nothing else to suggest where the space is being used.

I've also tried:

du -h /foldername

for all of the directories above but that shows nothing obvious either.

 

[memyselfandi]

Is /backup actually mounted now or not? (whats the output of df -k).

I would say that it's likely that something has written to the mount point directory when the disk isn't mounted (thus using space in /) and then been over mounted later.

 

[reefoid]

No, /backups wasn't mounted, but it is now and I can access all the files.

Looking at syslog, I think I can see what has happened. I have a cron job that rsyncs the /data mount onto a shared Windows drive. The Windows machine is usually on overnight but was powered down last night. I can see in syslog that when the cron job ran at 8:00 this morning there is a cifs_mount failure and then disk space warnings. So I'm guessing it couldn't find the Windows drive so used the root disk instead?

So, now I know what caused the issue, how can I find out where this data is?

Cheers.

edit: this is the rsync cron job:

#!/bin/bash
mount -t cifs \\\\192.168.1.3\\deb-backup /mnt/backup -o username=xxxx,password=xxxxxxxx
rsync -az /data /mnt/backup

Where would this data go if the mount is unavailable?

 

[reefoid]

I'm really stuck here. I just can't find anything that is eating up all the free space. I've searched for directories over 1GB in size, nothing unusual shows. I've searched for any files and directories that were created around the time of the rsync failure, nothing unusual shows.

I've been through countless rsync articles to find out what it does if it can't find a mount but I can't find anything.

Any other suggestions?

Thanks

 

[reefoid]

OK, all sorted. In case anyone is wondering, because the remote mount wasn't available it had mounted a new directory, still under /mnt/backup. However, when I restarted the Win7 machine this morning, the original mount overwrote the new one, but leaving the files there. After unmounting the remote share, I could then access /mnt/backup and remove the rogue files. Remounted the remote share and everything is back to normal.

 

[KIA]

memyselfandi was correct.