Shell scripting, help me with what should be a simple task?!

Maccy

Maccy

Maccy

Commissario
Joined
23 Nov 2004
Posts
42,338
Location
Herts
I need to create a lot of files for testing, however I've failed at the first hurdle! Here's what I've set out to achieve and broken down into tasks:
  1. Read a CSV file containing a few thousand rows of data (only one column)
  2. Create a directory with a unique name (in the format X_DD-MM-YYYY_HH-MM-SS) eg. 1_24-06-2018-00-00-01 - hoping I can increment the time value to make the dir unique?
  3. Grab the value from the CSV and store it as a variable
  4. Create a .json file with the above variable used once in one of the json fields and the directory name in one of the json fields
  5. Save the .json file in the above dir
  6. Copy a PDF file from another dir (I should be able to do this using the 'cp' function)
  7. Zip up the new folder with the .json file and the PDF file
I've got this so far but when I run it, nothing happens :(

Code: 
while IFS="," read -r f1
do
    #create a unique int that increments through each loop
    int i=1

    #print the case number from each row
    echo $f1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i

done < CCD_CaseList_01-10-2018_1only.csv

I have cd'd into the correct dir before the above code btw.

Can anyone help please?!
 
I have a vb.net solution for xml which you could probably adapt, it's doing similar with xml files and json serialization. I built it to add a reporting layer to an app that is buiIt on xml but offered no reporting. I can dig around and share it later if you run into a block on this shell script.

Fwiw a .net solution can easily do all of the above and I can't see anything in there that is particularly difficult to write.
 
Last edited:
Maccy said:
I have run into a block :(
Click to expand...

you got any examples and ill throw it together for you? Shouldn't take that long at all, everything bar zipping up I already have code snippets for. Do you have access to visual studio?

By examples I mean an example of what the csv and finished json might look like?

point 2 you can use the clock to randomise but you could just increment using a flat file to store current file directory number and increment from there, everything else is trivial, im sure even zipping is just a system.io call but have just never needed it.

Also how often do you need to do the task? Edit: In fact it doesn't really matter ill throw something together later this evening/tomorrow morning and post it up you can then change it and away you go.
 
Last edited:
Vince said:
you got any examples and ill throw it together for you? Shouldn't take that long at all, everything bar zipping up I already have code snippets for. Do you have access to visual studio?

By examples I mean an example of what the csv and finished json might look like?

point 2 you can use the clock to randomise but you could just increment using a flat file to store current file directory number and increment from there, everything else is trivial, im sure even zipping is just a system.io call but have just never needed it.

Also how often do you need to do the task? Edit: In fact it doesn't really matter ill throw something together later this evening/tomorrow morning and post it up you can then change it and away you go.
Click to expand...
The CSV is many lines of just numbers, it's data that I need to put into the json file, I will PM you an example of it because I can't post it on here.
 
Code: 
while IFS=, read -r a1
do
    #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

That works for me mate... let me know if it doesn't tested in Linux (ubuntu)

Stelly
 
Stelly said:
Code: 
while IFS=, read -r a1
do
    #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

That works for me mate... let me know if it doesn't tested in Linux (ubuntu)

Stelly
Click to expand...
Nope, nothing happens in terminal (running on Mac OS Mojave).
 
Code: 
#!/bin/sh
while IFS=, read -r a1
do
   #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

Also, can you try giving execute rights to the script:

chmod u+x <whateverscriptname.sh>

Is it giving you no feedback at all? It is the reason I'm thinking its not executing at all...

Stelly
 
When I run it I get the following:

Code: 
stelly@DORA:/mnt/d/Maccy$ sh code.sh
1
2
3
4
5
6
7
8
9
10
stelly@DORA:/mnt/d/Maccy$ ls
1_24-06-2018-00-00-1  code.sh  no.csv

Stelly
 
Stelly said:
Code: 
#!/bin/sh
while IFS=, read -r a1
do
   #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

Also, can you try giving execute rights to the script:

chmod u+x <whateverscriptname.sh>

Is it giving you no feedback at all? It is the reason I'm thinking its not executing at all...

Stelly
Click to expand...

Yeh no feedback at all. I changed the script to just a simple mkdir command and that worked so not sure what's going on.

@AHarvey doesn't have to be, no, but it's the only scripting language I'm somewhat familiar with (or apparently not) that I can use on my Mac.
 
Not far off of an .exe solution or at least a full .sln file you can play with, question... the csv file... loads of columns or just a single row/column of data?

My apologies I have only just started tbh, this morning I ended up doing an order of service for a family funeral on thurs :(
 
Vince said:
Not far off of an .exe solution or at least a full .sln file you can play with, question... the csv file... loads of columns or just a single row/column of data?

My apologies I have only just started tbh, this morning I ended up doing an order of service for a family funeral on thurs :(
Click to expand...
I do have a Windows 10 Parallels instance installed. And the CSV is just a single column with thousands of rows.

Appreciate the help mate, sorry to hear that :(
 
Maccy said:
I do have a Windows 10 Parallels instance installed. And the CSV is just a single column with thousands of rows.

Appreciate the help mate, sorry to hear that :(
Click to expand...

K i'm just building some modules and stuff and if nothing comes up it shouldn't bee too long.
 
@Maccy, can you trust me over an example csv and json file?

As it's a mac s python script could do the same thing without the need for an exe.

Am I right in thinking that it's a custom folder, with a single json file made from the csv, a pdf copied over and then zipped?

we'r e not talking 1 csv row = 1 json file = one zip?
 
Okay, here is some powershell you can run in Windows 10 mate:

$p.Number = Number is the name of the column within the csv...

Code: 
$p = Import-Csv D:\Maccy\no.csv

$i=0

foreach ($no in $p.Number)
{
    Write-Host $no

    New-Item -ItemType directory -Path "D:\Maccy\1_24-06-2018-00-00-$i"

    Set-Location -Path "D:\Maccy\1_24-06-2018-00-00-$i"

    $i++
}
 
AHarvey said:
@Maccy, can you trust me over an example csv and json file?

As it's a mac s python script could do the same thing without the need for an exe.

Am I right in thinking that it's a custom folder, with a single json file made from the csv, a pdf copied over and then zipped?

we'r e not talking 1 csv row = 1 json file = one zip?
Click to expand...
I'll PM you details.

And yes, it's one zip per line of data in the csv containing the json file (with the 1 bit of data from csv) and pdf.
 
well I have quite a lot of it done, reading the csv into an array etc is all done. now it's just a case of formatting the json and pushing the zip etc out. This would end up basically being a form where you set your input and output directory and away you go. Do i still finish it? Right now it just pops up txt boxes at each step with the csv location, full array, chopped up array. The rest is fairly easy stuff.

Code: 
'Set up our imports
#Region "Imports directives"
Imports System.IO
Imports System
Imports System.Text
#End Region
Public Class Form1
    Dim CSVFileCtnts As String
    Dim MainRoot As String
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        ReadCSV()
        ReadCSVFileToArray()
    End Sub
    Private Sub ReadCSV()
        'Variable Declerations
        'Our root directory where everything starts
        Dim root As New DirectoryInfo("C:\tempTest")
        'we can use root along with files to pick any csv in the folder - We shall do this so we don't have to be specific.
        'Files is the files available that are of type csv in the root
        Dim Files As FileInfo() = root.GetFiles("*.csv")
        Dim FileName As FileInfo
        Dim sourceFile As String
        Dim tempFile As String = ("c:\tempTest\temp.txt")
        Dim outputFile As String = ("C:\tempTest\output.txt")
        Dim currentRow As String = ""
        Dim StreamWriter As StreamWriter
        For Each FileName In Files
            Try
                'Try loop - There is only one csv file in the temp directory - This grabs the csv file of any name and reads everything
                'into a single string, we will chop it up later.
                sourceFile = (root.ToString & "\" & FileName.ToString)
                MainRoot = sourceFile
                MsgBox(MainRoot)
                Using streamReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(sourceFile)
                    currentRow = streamReader.ReadToEnd()
                    'Close the streamReader
                    streamReader.Close()
                End Using
                'This is where we split each entry and store the output into an array
         
                'Write the current line to the streamwriter
                StreamWriter = File.AppendText(outputFile)
                StreamWriter.WriteLine(currentRow)
                CSVFileCtnts = currentRow
                'Close the stream writer.
                StreamWriter.Flush()
                StreamWriter.Close()
            Catch ex As Exception
            End Try
            MsgBox(CSVFileCtnts)
        Next
    End Sub
    Private Sub ReadCSVFileToArray()
        Dim strfilename As String
        Dim num_rows As Long
        Dim num_cols As Long
        Dim x As Integer
        Dim y As Integer
        Dim strarray(1, 1) As String
        ' Load the file.
        strfilename = MainRoot
        'Check if file exist
        If File.Exists(strfilename) Then
            Dim tmpstream As StreamReader = File.OpenText(strfilename)
            Dim strlines() As String
            Dim strline() As String
            'Load content of file to strLines array
            strlines = tmpstream.ReadToEnd().Split(Environment.NewLine)
            ' Redimension the array.
            num_rows = UBound(strlines)
            strline = strlines(0).Split(",")
            num_cols = UBound(strline)
            ReDim strarray(num_rows, num_cols)
            'Debug
            ' Copy the data into the array.
            For x = 0 To num_rows
                strline = strlines(x).Split(",")
                For y = 0 To num_cols
                    strarray(x, y) = strline(y)
                Next
            Next
            'Debug
            ' Display the data in messageboxes
            For x = 0 To num_rows
                For y = 0 To num_cols
                    MsgBox(strarray(x, y))
                Next
                MsgBox(Environment.NewLine)
            Next
        End If
    End Sub
End Class



I will carry on but if you prefer a powershell solution over a .net app then let me know. Obviously I would give you all the source code and either pre compile or you can compile yourself... Im easy going... Also I love the way the code box has annihilated what was relatively nicely formatted code :(

edit: it now makes a directory, json file etc... gonna head home for the day but will finish it when I get home. Literally I think all I need to do is zip at the end.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom