1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Shell scripting, help me with what should be a simple task?!

Discussion in 'HTML, Graphics & Programming' started by Maccy, Oct 1, 2018.

  1. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    I need to create a lot of files for testing, however I've failed at the first hurdle! Here's what I've set out to achieve and broken down into tasks:
    1. Read a CSV file containing a few thousand rows of data (only one column)
    2. Create a directory with a unique name (in the format X_DD-MM-YYYY_HH-MM-SS) eg. 1_24-06-2018-00-00-01 - hoping I can increment the time value to make the dir unique?
    3. Grab the value from the CSV and store it as a variable
    4. Create a .json file with the above variable used once in one of the json fields and the directory name in one of the json fields
    5. Save the .json file in the above dir
    6. Copy a PDF file from another dir (I should be able to do this using the 'cp' function)
    7. Zip up the new folder with the .json file and the PDF file
    I've got this so far but when I run it, nothing happens :(

    Code:
    while IFS="," read -r f1
    do
        #create a unique int that increments through each loop
        int i=1
    
        #print the case number from each row
        echo $f1
    
        #create the directory for each zip file using mkdir
        mkdir 1_24-06-2018-00-00-$i
    
        #cd into the new directory
        cd 1_24-06-2018-00-00-$i
    
    done < CCD_CaseList_01-10-2018_1only.csv
    I have cd'd into the correct dir before the above code btw.

    Can anyone help please?!
     
  2. Vince

    Soldato

    Joined: Oct 30, 2003

    Posts: 6,386

    Location: Essex

    I have a vb.net solution for xml which you could probably adapt, it's doing similar with xml files and json serialization. I built it to add a reporting layer to an app that is buiIt on xml but offered no reporting. I can dig around and share it later if you run into a block on this shell script.

    Fwiw a .net solution can easily do all of the above and I can't see anything in there that is particularly difficult to write.
     
    Last edited: Oct 1, 2018
  3. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    I have run into a block :(
     
  4. Vince

    Soldato

    Joined: Oct 30, 2003

    Posts: 6,386

    Location: Essex

    you got any examples and ill throw it together for you? Shouldn't take that long at all, everything bar zipping up I already have code snippets for. Do you have access to visual studio?

    By examples I mean an example of what the csv and finished json might look like?

    point 2 you can use the clock to randomise but you could just increment using a flat file to store current file directory number and increment from there, everything else is trivial, im sure even zipping is just a system.io call but have just never needed it.

    Also how often do you need to do the task? Edit: In fact it doesn't really matter ill throw something together later this evening/tomorrow morning and post it up you can then change it and away you go.
     
    Last edited: Oct 1, 2018
  5. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    The CSV is many lines of just numbers, it's data that I need to put into the json file, I will PM you an example of it because I can't post it on here.
     
  6. Vince

    Soldato

    Joined: Oct 30, 2003

    Posts: 6,386

    Location: Essex

    Got it mate. Give me a few hours and ill throw something together.
     
  7. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    Appreciate it mate, cheers.
     
  8. Stelly

    Don

    Joined: Oct 5, 2005

    Posts: 10,956

    Location: Liverpool

    Code:
    while IFS=, read -r a1
    do
        #create a unique int that increments through each loop
        i=1
    
        #print the case number from each row
        echo $a1
    
        #create the directory for each zip file using mkdir
        mkdir 1_24-06-2018-00-00-$i
    
        #cd into the new directory
        cd 1_24-06-2018-00-00-$i
    done < no.csv
    That works for me mate... let me know if it doesn't tested in Linux (ubuntu)

    Stelly
     
  9. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    Nope, nothing happens in terminal (running on Mac OS Mojave).
     
  10. Stelly

    Don

    Joined: Oct 5, 2005

    Posts: 10,956

    Location: Liverpool

    Code:
    #!/bin/sh
    while IFS=, read -r a1
    do
       #create a unique int that increments through each loop
        i=1
    
        #print the case number from each row
        echo $a1
    
        #create the directory for each zip file using mkdir
        mkdir 1_24-06-2018-00-00-$i
    
        #cd into the new directory
        cd 1_24-06-2018-00-00-$i
    done < no.csv
    Also, can you try giving execute rights to the script:

    chmod u+x <whateverscriptname.sh>

    Is it giving you no feedback at all? It is the reason I'm thinking its not executing at all...

    Stelly
     
  11. Stelly

    Don

    Joined: Oct 5, 2005

    Posts: 10,956

    Location: Liverpool

    When I run it I get the following:

    Code:
    stelly@DORA:/mnt/d/Maccy$ sh code.sh
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    stelly@DORA:/mnt/d/Maccy$ ls
    1_24-06-2018-00-00-1  code.sh  no.csv
    Stelly
     
  12. AHarvey

    Sgarrista

    Joined: Mar 6, 2008

    Posts: 8,929

    Location: Stoke area

    Does it have to be a shell script? Open to other solutions?
     
  13. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    Yeh no feedback at all. I changed the script to just a simple mkdir command and that worked so not sure what's going on.

    @AHarvey doesn't have to be, no, but it's the only scripting language I'm somewhat familiar with (or apparently not) that I can use on my Mac.
     
  14. Vince

    Soldato

    Joined: Oct 30, 2003

    Posts: 6,386

    Location: Essex

    Not far off of an .exe solution or at least a full .sln file you can play with, question... the csv file... loads of columns or just a single row/column of data?

    My apologies I have only just started tbh, this morning I ended up doing an order of service for a family funeral on thurs :(
     
  15. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    I do have a Windows 10 Parallels instance installed. And the CSV is just a single column with thousands of rows.

    Appreciate the help mate, sorry to hear that :(
     
  16. Vince

    Soldato

    Joined: Oct 30, 2003

    Posts: 6,386

    Location: Essex

    K i'm just building some modules and stuff and if nothing comes up it shouldn't bee too long.
     
  17. AHarvey

    Sgarrista

    Joined: Mar 6, 2008

    Posts: 8,929

    Location: Stoke area

    @Maccy, can you trust me over an example csv and json file?

    As it's a mac s python script could do the same thing without the need for an exe.

    Am I right in thinking that it's a custom folder, with a single json file made from the csv, a pdf copied over and then zipped?

    we'r e not talking 1 csv row = 1 json file = one zip?
     
  18. Stelly

    Don

    Joined: Oct 5, 2005

    Posts: 10,956

    Location: Liverpool

    Okay, here is some powershell you can run in Windows 10 mate:

    $p.Number = Number is the name of the column within the csv...

    Code:
    $p = Import-Csv D:\Maccy\no.csv
    
    $i=0
    
    foreach ($no in $p.Number)
    {
        Write-Host $no
    
        New-Item -ItemType directory -Path "D:\Maccy\1_24-06-2018-00-00-$i"
    
        Set-Location -Path "D:\Maccy\1_24-06-2018-00-00-$i"
    
        $i++
    }
     
  19. Maccy

    Commissario

    Joined: Nov 23, 2004

    Posts: 33,605

    Location: Back in Blighty

    I'll PM you details.

    And yes, it's one zip per line of data in the csv containing the json file (with the 1 bit of data from csv) and pdf.
     
  20. Vince

    Soldato

    Joined: Oct 30, 2003

    Posts: 6,386

    Location: Essex

    well I have quite a lot of it done, reading the csv into an array etc is all done. now it's just a case of formatting the json and pushing the zip etc out. This would end up basically being a form where you set your input and output directory and away you go. Do i still finish it? Right now it just pops up txt boxes at each step with the csv location, full array, chopped up array. The rest is fairly easy stuff.

    Code:
    'Set up our imports
    #Region "Imports directives"
    Imports System.IO
    Imports System
    Imports System.Text
    #End Region
    Public Class Form1
        Dim CSVFileCtnts As String
        Dim MainRoot As String
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            ReadCSV()
            ReadCSVFileToArray()
        End Sub
        Private Sub ReadCSV()
            'Variable Declerations
            'Our root directory where everything starts
            Dim root As New DirectoryInfo("C:\tempTest")
            'we can use root along with files to pick any csv in the folder - We shall do this so we don't have to be specific.
            'Files is the files available that are of type csv in the root
            Dim Files As FileInfo() = root.GetFiles("*.csv")
            Dim FileName As FileInfo
            Dim sourceFile As String
            Dim tempFile As String = ("c:\tempTest\temp.txt")
            Dim outputFile As String = ("C:\tempTest\output.txt")
            Dim currentRow As String = ""
            Dim StreamWriter As StreamWriter
            For Each FileName In Files
                Try
                    'Try loop - There is only one csv file in the temp directory - This grabs the csv file of any name and reads everything
                    'into a single string, we will chop it up later.
                    sourceFile = (root.ToString & "\" & FileName.ToString)
                    MainRoot = sourceFile
                    MsgBox(MainRoot)
                    Using streamReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(sourceFile)
                        currentRow = streamReader.ReadToEnd()
                        'Close the streamReader
                        streamReader.Close()
                    End Using
                    'This is where we split each entry and store the output into an array
             
                    'Write the current line to the streamwriter
                    StreamWriter = File.AppendText(outputFile)
                    StreamWriter.WriteLine(currentRow)
                    CSVFileCtnts = currentRow
                    'Close the stream writer.
                    StreamWriter.Flush()
                    StreamWriter.Close()
                Catch ex As Exception
                End Try
                MsgBox(CSVFileCtnts)
            Next
        End Sub
        Private Sub ReadCSVFileToArray()
            Dim strfilename As String
            Dim num_rows As Long
            Dim num_cols As Long
            Dim x As Integer
            Dim y As Integer
            Dim strarray(1, 1) As String
            ' Load the file.
            strfilename = MainRoot
            'Check if file exist
            If File.Exists(strfilename) Then
                Dim tmpstream As StreamReader = File.OpenText(strfilename)
                Dim strlines() As String
                Dim strline() As String
                'Load content of file to strLines array
                strlines = tmpstream.ReadToEnd().Split(Environment.NewLine)
                ' Redimension the array.
                num_rows = UBound(strlines)
                strline = strlines(0).Split(",")
                num_cols = UBound(strline)
                ReDim strarray(num_rows, num_cols)
                'Debug
                ' Copy the data into the array.
                For x = 0 To num_rows
                    strline = strlines(x).Split(",")
                    For y = 0 To num_cols
                        strarray(x, y) = strline(y)
                    Next
                Next
                'Debug
                ' Display the data in messageboxes
                For x = 0 To num_rows
                    For y = 0 To num_cols
                        MsgBox(strarray(x, y))
                    Next
                    MsgBox(Environment.NewLine)
                Next
            End If
        End Sub
    End Class
    


    I will carry on but if you prefer a powershell solution over a .net app then let me know. Obviously I would give you all the source code and either pre compile or you can compile yourself... Im easy going... Also I love the way the code box has annihilated what was relatively nicely formatted code :(

    edit: it now makes a directory, json file etc... gonna head home for the day but will finish it when I get home. Literally I think all I need to do is zip at the end.
     
    Last edited: Oct 2, 2018