Downloading from multiple FTP's simultaneously.

  • Thread starter Thread starter X82
  • Start date Start date

X82

X82

X82

Associate
Joined
24 May 2010
Posts
212
I have a list of about 500 FTP addresses. The same user and password for each. Each FTP contains a single file, data.txt. I'm currently using Excel to download these files, but it does it one at a time and it takes 90 minutes. Is there anyway I can speed up this process? The issue is, I need this doing on a works PC, so installing 3rd party software is limited. Although I managed to install WinSCP.
It needs to be simple/idiot proof. Any ideas how to achieve this?
 
Try Cygwin / Putty, then use wget (put it in a shell script for ease).

If you can't install Cygwin or putty, then I think wget can work in Windows too...
 
Try using an FTP client, ideally FlashFXP but if there are concerns about licensing/trial software then go for Filezilla which is freeware.
 
Disco Boy said:
Try Cygwin / Putty, then use wget (put it in a shell script for ease).

If you can't install Cygwin or putty, then I think wget can work in Windows too...
Click to expand...

Putty is already on our systems. But I have no clue how to use it.
The end game would be a single file/shortcut on the desktop, which when clicked would run this and download the files. But how I get there is beyond me.
 
I looked into that before, but I couldn't find a way of downloading files simultaneously. So far, one at a time is the best I can do when scripting.
 
I'll look into that, I just hope it's not out of my depth.
It's a shame there is no way of doing many at the same time. Currently installed at work is Putty, obviously command prompt and WinSCP. Which supports scripting but I believe the problem is the same, line by line. I'm open to other suggestions!
 
You could do simultaneous wget using something like this batch file, where wget.txt contains a list of your addresses, e.g.:

Code: 
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt
ftp://ak:[email protected]/data.txt

Code: 
@echo off
for /F "tokens=*" %%A in  ( wget.txt) do  (
   ECHO Processing %%A.... 
   start /B "" "c:\Program Files (x86)\GnuWin32\bin\wget.exe" "%%A" >NUL 2>&1
   rem start "cmd /c pause"
)
@echo on

(that'll just dump over the same file so would need tweaking)

Would be far far easier to use a proper scripting language and/or a simple C# program though. I'd gladly look at knocking you one up if you were allowed to run it.
 
If you could that would be amazing. I've been talking with the powers that be and they agree with me that this function is needed for our workplace. They would authorise any installation, as long as there is no license issues etc.

But if you could knock one up, that would be phenomenal.
Also, for clarity.

My current excel script will take a column, say A, which contains the IP's. Then it will download said file using column B for a path, say C:/PowerReadings/Monday. Then it would use column C to rename the file. So instead of data.txt it is, AMAEE11.txt and so on. This works but as mentions, only does them one at a time. Renaming the files is essential. Sorry to make things more complicated.
 
Last edited:
Are they 500 separate FTP servers or just 500 accounts on the same server?

If they are just accounts can you not log into the root account and simply copy every sub-folder?
 
Last edited:
So I got bored and wrote a little test program to see if it's possible. 500 simultaneous FTP connection threads back to localhost reading a small txt file (which will be miles, miles quicker than real-world stuff though and mostly thanks to FileZilla Server being super speedy) finished in under a second :o.

X82FTP.png


Might try and finish it off tomorrow if I get time :D.

Is your path your local path or server path, btw?
 
Wow, that looks fantastic! :eek:

I don't know if this will be of any help, but I will clarify a few things anyway.
Currently, there are 5 buttons on the excel sheet. Download Monday, Tuesday etc. This will download each of the files to a directory based on the day, on the local drive. C:/Power Readings/Monday etc, through to Friday.
Here is the file I made to show what is required, hope it makes sense. Don't take the data in the file as 100%, it's an old file. But its a blueprint to show you how I mean.

The end game would be, 5 buttons or icons on the desktop which would download all the files, put them in the directory and rename them.
 
Doesn't soudnd too tricky then. I'll see what I can do later.

If this works then the cake is on your company's expenses :p.
 
Don't worry, I'll make sure the company does something :D
If you need any clarity on anything else, let me know. As I said, the previous file I posted is only a guide, I can obtain the proper file tonight when I go back to work.
 
Sorry to be a further pain. But I just wanted to let you know that this programme would need to be easily editable by me. As in, I will need to add/remove some ip's over time, the same goes to the rename name. I hope this doesn't cause too much trouble!
I'll buy you a drink for this!
 
This would be really easy to do using SAS or something similar. Would even read in the text files as you specify and create an easy to use dataset. Ripoff though.

http://www.ats.ucla.edu/stat/sas/faq/readftp.htm

You'd do something similar to above but use a macro to repeat it for each address.

I'm sure something similar can be done within vba, seems like something which you would think to allow.
 
Last edited:
X82 said:
Sorry to be a further pain. But I just wanted to let you know that this programme would need to be easily editable by me. As in, I will need to add/remove some ip's over time, the same goes to the rename name. I hope this doesn't cause too much trouble!
I'll buy you a drink for this!
Click to expand...

Sure, was going to have it read from a file like the one you provided anyway.

Probably not going to get around to finishing it until Friday now though :(.
 
You must log in or register to reply here.
Back
Top Bottom