Shell scripting, help me with what should be a simple task?!

Maccy · 1 Oct 2018 at 16:33

I need to create a lot of files for testing, however I've failed at the first hurdle! Here's what I've set out to achieve and broken down into tasks:

Read a CSV file containing a few thousand rows of data (only one column)
Create a directory with a unique name (in the format X_DD-MM-YYYY_HH-MM-SS) eg. 1_24-06-2018-00-00-01 - hoping I can increment the time value to make the dir unique?
Grab the value from the CSV and store it as a variable
Create a .json file with the above variable used once in one of the json fields and the directory name in one of the json fields
Save the .json file in the above dir
Copy a PDF file from another dir (I should be able to do this using the 'cp' function)
Zip up the new folder with the .json file and the PDF file

I've got this so far but when I run it, nothing happens

Code:

while IFS="," read -r f1
do
    #create a unique int that increments through each loop
    int i=1

    #print the case number from each row
    echo $f1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i

done < CCD_CaseList_01-10-2018_1only.csv

I have cd'd into the correct dir before the above code btw.

Can anyone help please?!

Maccy · 1 Oct 2018 at 18:09

I have run into a block

Maccy · 2 Oct 2018 at 09:13

Vince said:
you got any examples and ill throw it together for you? Shouldn't take that long at all, everything bar zipping up I already have code snippets for. Do you have access to visual studio?

By examples I mean an example of what the csv and finished json might look like?

point 2 you can use the clock to randomise but you could just increment using a flat file to store current file directory number and increment from there, everything else is trivial, im sure even zipping is just a system.io call but have just never needed it.

Also how often do you need to do the task? Edit: In fact it doesn't really matter ill throw something together later this evening/tomorrow morning and post it up you can then change it and away you go.

The CSV is many lines of just numbers, it's data that I need to put into the json file, I will PM you an example of it because I can't post it on here.

Maccy · 2 Oct 2018 at 10:53

Vince said:
Got it mate. Give me a few hours and ill throw something together.

Appreciate it mate, cheers.

Maccy · 2 Oct 2018 at 12:19

Stelly said:

Code:

while IFS=, read -r a1
do
    #create a unique int that increments through each loop
    i=1

    #print the case number from each row
    echo $a1

    #create the directory for each zip file using mkdir
    mkdir 1_24-06-2018-00-00-$i

    #cd into the new directory
    cd 1_24-06-2018-00-00-$i
done < no.csv

That works for me mate... let me know if it doesn't tested in Linux (ubuntu)

Stelly

Nope, nothing happens in terminal (running on Mac OS Mojave).

Maccy · 2 Oct 2018 at 14:07

Stelly said:
Code:

#!/bin/sh while IFS=, read -r a1 do #create a unique int that increments through each loop i=1 #print the case number from each row echo $a1 #create the directory for each zip file using mkdir mkdir 1_24-06-2018-00-00-$i #cd into the new directory cd 1_24-06-2018-00-00-$i done < no.csv

Also, can you try giving execute rights to the script:

chmod u+x <whateverscriptname.sh>

Is it giving you no feedback at all? It is the reason I'm thinking its not executing at all...

Stelly

Yeh no feedback at all. I changed the script to just a simple mkdir command and that worked so not sure what's going on.

@AHarvey doesn't have to be, no, but it's the only scripting language I'm somewhat familiar with (or apparently not) that I can use on my Mac.

Maccy · 2 Oct 2018 at 15:01

Vince said:
Not far off of an .exe solution or at least a full .sln file you can play with, question... the csv file... loads of columns or just a single row/column of data?

My apologies I have only just started tbh, this morning I ended up doing an order of service for a family funeral on thurs

I do have a Windows 10 Parallels instance installed. And the CSV is just a single column with thousands of rows.

Appreciate the help mate, sorry to hear that

Maccy · 2 Oct 2018 at 15:44

AHarvey said:
@Maccy, can you trust me over an example csv and json file?

As it's a mac s python script could do the same thing without the need for an exe.

Am I right in thinking that it's a custom folder, with a single json file made from the csv, a pdf copied over and then zipped?

we'r e not talking 1 csv row = 1 json file = one zip?

I'll PM you details.

And yes, it's one zip per line of data in the csv containing the json file (with the 1 bit of data from csv) and pdf.

Maccy · 2 Oct 2018 at 17:19

Wow fantastic thanks @Vince , the directory name I could populate a CSV and provide that maybe?

Maccy · 3 Oct 2018 at 15:50

Well, with the excellent assistance from @Vince and @Stelly I've managed to put together a working solution:

Code:

#---- DECLARE VARIABLES HERE ----#
 
workingDir=/location/of/your/working/directory/
zipLocation=/location/of/your/working/directory/zip_files/
pdfFile=1111001.pdf
pdfLocation=/location/of/your/pdf/file/
csvFile=converted-CSV-file.csv
 
#---- END OF VARIABLE DECLARATION ----#
 
#create a unique int that increments through each loop
var=1
 
while IFS=, read -r f1
do
    #cd to the working directory
    cd "$workingDir"
 
    #print the case number from each row
    echo "The case number from the CSV is $f1 that we are using"
 
    #create the unique folder name using the looped var and required folder structure name
    case=${f1}
    folder="_24-06-2018-00-00-00"
    caseFolder=$var$folder
 
    #create the directory for each zip file using mkdir
    mkdir "${caseFolder}"
    echo "Created dir with name $caseFolder"
 
    #cd into the new directory
    cd "${caseFolder}"
 
    #create the metadata.json file with the required content (template and case number var)
    cat >./metadata.json <<EOF
    {
      "case_number": "${case}",
      "rest of json":"goes here"
    }
EOF
 
    #copy the required PDF file into the new dir
    cp ${pdfLocation}${pdfFile} ${pdfFile}
 
    #move back up a dir
    cd ..
 
    #zip the folder
    zip -r ${caseFolder}.zip ${caseFolder}
 
    #move the zip file to a separate folder to collate all data in one location
    mv ${caseFolder}.zip "${zipLocation}"
 
    #increase the value of i
    var=$((var+1))
 
done < ${csvFile}

Only problem was that I had to convert the CSV file to a DOS formatted CSV file otherwise importing the case_number added a new line to the JSON file which messed everything up!

Maccy · 3 Oct 2018 at 16:03

Stelly said:
case=$(printf $f1)

Try that mate... see if it works

Stelly

Afraid not, line break is still present.

Maccy · 5 Oct 2018 at 12:16

I finished it yesterday guys, Stelly provided the final missing piece to fix the line break:

Code:

#---- DECLARE VARIABLES HERE ----#
 
workingDir=/my/working/dir/
zipLocation=/my/working/dir/zip_files/
pdfFile=pdf_file.pdf
pdfLocation=/location/of/pdf/
csvFile=file.csv
jurisdiction=someData
 
#---- END OF VARIABLE DECLARATION ----#
 
#create a unique int that increments through each loop
var=1
 
while IFS=, read -r f1
do
    #cd to the working directory
    cd "$workingDir"
 
    #print the case number from each row
    #echo "The case number from the CSV is ${f1} that we are using"
 
    #create the unique folder name using the looped var and required folder structure name
    #case=$(printf ${f1})
    case="$(echo "$f1"|tr -d '\r')"
    folder="_24-06-2018-00-00-00"
    caseFolder=$var$folder
 
    #create the directory for each zip file using mkdir
    mkdir "${caseFolder}"
    echo "Created dir with name $caseFolder"
 
    #cd into the new directory
    cd "${caseFolder}"
 
    #create the metadata.json file with the required content (template and case number var)
    cat >./metadata.json <<EOF
    {
      "case_number": "${case}",
      "jurisdiction": "${jurisdiction}",
      "zip_file_name": "${caseFolder}.zip",

****REST OF JSON GOES HERE, DELETED INTENTIONALLY****

    }
EOF
 
    #copy the required PDF file into the new dir
    cp ${pdfLocation}${pdfFile} ${pdfFile}
 
    #move back up a dir
    cd ..
 
    #zip the folder
    zip -r ${caseFolder}.zip ${caseFolder}
 
    #move the zip file to a separate folder to collate all data in one location
    mv ${caseFolder}.zip "${zipLocation}"
 
    #remove the zip folder we created earlier as we no longer require them
    rm -rf ${caseFolder}
 
    #increase the value of i
    var=$((var+1))
 
done < ${csvFile}