Downloading from multiple FTP's simultaneously.

  • Thread starter Thread starter X82
  • Start date Start date

X82

X82

Associate
Joined
24 May 2010
Posts
212
I have a list of about 500 FTP addresses. The same user and password for each. Each FTP contains a single file, data.txt. I'm currently using Excel to download these files, but it does it one at a time and it takes 90 minutes. Is there anyway I can speed up this process? The issue is, I need this doing on a works PC, so installing 3rd party software is limited. Although I managed to install WinSCP.
It needs to be simple/idiot proof. Any ideas how to achieve this?
 
Try Cygwin / Putty, then use wget (put it in a shell script for ease).

If you can't install Cygwin or putty, then I think wget can work in Windows too...

Putty is already on our systems. But I have no clue how to use it.
The end game would be a single file/shortcut on the desktop, which when clicked would run this and download the files. But how I get there is beyond me.
 
I looked into that before, but I couldn't find a way of downloading files simultaneously. So far, one at a time is the best I can do when scripting.
 
I'll look into that, I just hope it's not out of my depth.
It's a shame there is no way of doing many at the same time. Currently installed at work is Putty, obviously command prompt and WinSCP. Which supports scripting but I believe the problem is the same, line by line. I'm open to other suggestions!
 
If you could that would be amazing. I've been talking with the powers that be and they agree with me that this function is needed for our workplace. They would authorise any installation, as long as there is no license issues etc.

But if you could knock one up, that would be phenomenal.
Also, for clarity.

My current excel script will take a column, say A, which contains the IP's. Then it will download said file using column B for a path, say C:/PowerReadings/Monday. Then it would use column C to rename the file. So instead of data.txt it is, AMAEE11.txt and so on. This works but as mentions, only does them one at a time. Renaming the files is essential. Sorry to make things more complicated.
 
Last edited:
Wow, that looks fantastic! :eek:

I don't know if this will be of any help, but I will clarify a few things anyway.
Currently, there are 5 buttons on the excel sheet. Download Monday, Tuesday etc. This will download each of the files to a directory based on the day, on the local drive. C:/Power Readings/Monday etc, through to Friday.
Here is the file I made to show what is required, hope it makes sense. Don't take the data in the file as 100%, it's an old file. But its a blueprint to show you how I mean.

The end game would be, 5 buttons or icons on the desktop which would download all the files, put them in the directory and rename them.
 
Don't worry, I'll make sure the company does something :D
If you need any clarity on anything else, let me know. As I said, the previous file I posted is only a guide, I can obtain the proper file tonight when I go back to work.
 
Sorry to be a further pain. But I just wanted to let you know that this programme would need to be easily editable by me. As in, I will need to add/remove some ip's over time, the same goes to the rename name. I hope this doesn't cause too much trouble!
I'll buy you a drink for this!
 
I've managed to get all the data I need for the programme. If that is of any help. Again, all work is much, much appreciated.
 
Thats amazing work to put it mildly. I'm just about to test it now, but one thing. How do I add a username and password for the ftp? The username and password is the same for everyone. would I add this to the csv?

Again, this looks fantastic. Truly appreciate it.
 
I've been testing it this mornign and its amazing. It takes 60 seconds to do a task which took 60 minutes. However, there is one issue. It manages to download nearly all 444 items except about 9. Which say "The Remote server returned an error: (421) service not available, closing control connection". checking this IP manually shows no errors.
It's always the same 9. It's the last 8 in the list and one a few before. But as I said, manually going to this ftp in IE shows no issues. I've tried the download process 5 times and it produces the error every time. And always on the same IP's.

Apart from this, it works beautifully. Any ideas?

Update:

If I remove all the ip's from the CSV which download OK, then the programme will download about 7 of the ones it previously couldn't. however, there is still one IP which it refused to download even though I can download it myself through IE. An Odd one.

Update 2:

solved the issue. There was a couple of duplicate IP's in my list. Re-arranged the CSV and now it completes with no errors. Fantastic work.
Because I have such little experience with programming, how would I go about making changes to this? For example, removing the details window or adding a done notification sound and dialogue. Can you point me in the right direction with software etc? Very much appreciated, it works great.
 
Last edited:
I would love a few changes, but I feel guilty for asking these requests.
Heres my suggestions, feel free to disregard them.
I love how it looks now, with IP, rename to and status. I don't need save path shown. Would it also be possible to change "Rename To" to read "Reference".

As for notification etc, that would be a plus, but it takes such little time to complete I'm wondering if its worth it.

Perhaps a larger download button and drop down?

Sorry to be such a pain with all this. I feel a bit like a kid in a candy store!

One final query, is it possible to incorporate VBA scripts? The reason I ask is at the moment this programme downloads 440+ txt files. Then a seperate excel spreadsheet pulls certain figures from the text file and enter it in a spreadsheet. The VBA script is all written and works fine, but can it be used with this?

Again, sorry to be a pain. All your help so far has been fantastic.
 
Thank so you much for helping out like this. I will test it tomorrow at work and also post the vba code also. However, the link you have given appears to be broken. Any chance of a new one?
 
Here is the VBA code in question. There are 5 of these for each day.
It will take a certain value (there are 4 variations of the txt file) from each file and put it into a column in excel. I then match over the data myself manually. It's not an issue to get this incorporated into this programme, but would be amazing. Again, all help is appreciated!


Code:
Sub Moncom()
    Dim myDir As String, fn As String, txt As String
    Dim myMax As Double, myName As String, myLoc As String, n As Long
    Dim myMatch As Object, x, y, temp As String, i As Long, ii As Long
    myDir = "c:\Power Readings\Monday\"
    fn = Dir(myDir & "*.txt")
    If fn <> "" Then
        n = n + 1
        Sheets(1).Cells(n, 1).Resize(, 4).Value = _
        [{"FileName","Name","Location","Max"}]
        With CreateObject("VBScript.RegExp")
            .IgnoreCase = True
            Do While fn <> ""
                myName = Empty: myLoc = Empty: myMax = 0
                txt = CreateObject("Scripting.FileSystemObject") _
                .OpenTextFile(myDir & fn).ReadAll
                .Global = True
                .Pattern = "[^\n]+"
                Set myMatch = .Execute(txt)
                .Global = False
                For i = 0 To myMatch.Count - 1
                    .Pattern = "\b(I(out)?Max|Ph ?I(.A)?)(?=\t)"
                    If .test(myMatch(i)) Then
                        temp = .Execute(myMatch(i))(0)
                        x = Application.Match(temp, Split(myMatch(i), vbTab), 0)
                        myMax = Val(Split(myMatch(i + 1), vbTab)(x - 1))
                        Exit For
                    Else
                        .Pattern = "\b(IMax Ph1)(?=\t)"
                        If .test(myMatch(i)) Then
                            temp = .Execute(myMatch(i))(0)
                            x = Application.Match(temp, Split(myMatch(i), vbTab), 0)
                            For ii = x - 1 To x + 1
                                myMax = myMax + Val(Split(myMatch(i + 1), vbTab)(ii))
                            Next
                    Else
                        .Pattern = "\b(IMax B1)(?=\t)"
                        If .test(myMatch(i)) Then
                            temp = .Execute(myMatch(i))(0)
                            x = Application.Match(temp, Split(myMatch(i), vbTab), 0)
                            For ii = x - 1 To x
                                myMax = myMax + Val(Split(myMatch(i + 1), vbTab)(ii))
                            Next
                            Exit For
                            End If
                        End If
                    End If
                Next
                .Pattern = "(\d{2}(?:[/\.])){2}\d{4}\t(\d{2}:){2}\d{2}\t[^\t]+\t([^\t]+)\t([^\t]+)"
                If .test(txt) Then
                    myName = .Execute(txt)(0).submatches(2)
                    myLoc = .Execute(txt)(0).submatches(3)
                End If
                n = n + 1
                Sheets(1).Cells(n, 1).Resize(, 4).Value = _
                Array(fn, myName, myLoc, myMax)
                fn = Dir
            Loop
        End With
    Else
        MsgBox "No file found"
    End If
    
    MsgBox "Compiling Completed Successfully"
    
End Sub



Private Sub Worksheet_SelectionChange(ByVal Target As Range)

End Sub
 
Sorry for the late response.
I agree, implementing this all in one would be too tricky when I can currently open excel and click a button. As it stands, it's working better than I hoped.
As most offices are full of some sort of politics, mine being no different, I am disappointed in the bosses responses to this. Lacklustre is one way to describe it. But I am more than happy that me and my 3 shift mate can complete a task which took us about 5 hours before, in 2 minutes. If I do find anything that needs modification, would it be ok to post here? Or would you rather I learn for myself?
 
Back
Top Bottom