Using sed or perl to replace multiple lines with leading spaces?

Feek · 5 Jul 2019 at 18:33

Hi folks.

I'm not a programmer but I can manage to do many things simply by looking into the command formats and working it out but this is causing me an issue. I'm trying to replace multiple lines in a .js file which have leading spaces but can't manage it with either sed or perl

This is what the source looks like

And this is what I want it to be

I need to change the 000000 to 0000DD, add a new line with lineDash below it and change the width to 2.

I can't just search for 000000 and change it to 0000DD because there are more than one instances of that within the file and the same goes for the width. There is only one instance of var ringStyle so that's the point I have to start with.

I've spent the last couple of hours trying padding the perl command with spaces and using \n for the line separators just to try and get this done but I'm failing badly. I'm sure there's a way to do this but I'm hotspured if I can work it out. I've got the command in a format which doesn't error at me any more but it doesn't actually do anything! Here's what I'm trying (this is a long command with the correct number of spaces). This is just to try and replace the 000000 with 0000DD and then I was going to tack the extra bits on the end.

Code:

sudo perl -i -0 -pe 's/\Q                var ringStyle = new ol.style.Style({\n                        fill: null,\n                        stroke: new ol.style.Stroke({\n                                color: '#000000',\n\E/\Q                var ringStyle = new ol.style.Style({\n                        fill: null,\n                        stroke: new ol.style.Stroke({\n                                color: '#0000DD',\n\E/g' script.js

Can anyone help please? If so, please can you explain the format of the command so I can understand what I'm doing.

Thanks.

Randomface74 · 5 Jul 2019 at 18:42

try \r\n instead of \n

bremen1874 · 5 Jul 2019 at 18:46

Worrying about the white space is just being picky. It'll run just fine without making the formatting pretty.

Feek · 5 Jul 2019 at 18:51

billysielu said:
try \r\n instead of \n

Tried that, no different.

bremen1874 said:
Worrying about the white space is just being picky. It'll run just fine without making the formatting pretty.

Right, so it doesn't care about the leading spaces? How would I do this then?

MonkeyBasher · 5 Jul 2019 at 22:47

What he is saying is that to run the JS code with crap formatting will be fine. The whitespace is just to make it easier for you.
You can fix any whitespace issues after solving your problem by using a linter. This will reformat the entire file for you to look pretty.

perl -0777 -i.original -pe "s/'#000000',\n\s*width: 1/'#0000DD',\nlineDash:[4,4]\nwidth: 2/igs" derpy.js

derpy.js is your input file and will also be your output file.
your original is copied to derpy.js.original

The above replaces

Code:

'#000000',\n\s*width: 1

with

Code:

'#0000DD',\nlineDash:[4,4]\nwidth: 2

where \s* matches any amount of whitespace before the width statement (this could be why your attempt failed as you hardcoded X spaces)
\s is whitespace (tabs and spaces)
* is 0 or more

sed is designed for line by line processing unless you delve deep into its scripting functionality which is why it will not match any commbination of stuff with a newline in the middle.

Randomface74 · 5 Jul 2019 at 22:59

if I was doing this I'd be using a few nested string.Split() loops, break it up in the right places and insert your changes in the gaps.

Feek · 6 Jul 2019 at 07:54

MonkeyBasher said:
perl -0777 -i.original -pe "s/'#000000',\n\s*width: 1/'#0000DD',\nlineDash:[4,4]\nwidth: 2/igs" derpy.js

Thanks. This does work, sort of. The trouble is that there are multiple occurrences of the color and width entry within the file and I only want to change the one in the var ringstyle section.

With the example you've given, I thought it would be simple enough to build in the lines above to the command but it's not proving as straightforward as I though. Even just adding the whitespace to the color line isn't working.

One step at a time. First I added the whitespace and the additional text at the start of the color line.

perl -0777 -i.original -pe "s/\s*color: '#000000',\n\s*width: 1/\ncolor: '#0000DD',\nlineDash:[4,4],\nwidth: 2/igs" derpy.js

Works fine.

Then when I get to the next line, I'm introducing characters that are included in regex so I knew I was going to get problems.

perl -0777 -i.original -pe "s/\sstroke: new ol.style.Stroke({\s*color: '#000000',\n\s*width: 1/\nstroke: new ol.style.Stroke({\ncolor: '#0000DD',\nlineDash:[4,4],\nwidth: 2/igs" derpy.js

doesn't work because of the ({ characters.
Unmatched ( in regex; marked by <-- HERE in m/\sstroke: new ol.style.Stroke( <-- HERE {\s*color: '#000000',\n\s*width: 1/ at -e line 1.

I think I need to use \Q and \E as start and end terminators to enclose the ({ but this is where I'm struggling now.

I've been playing with this for an hour this morning and now I have a headache!

MonkeyBasher · 6 Jul 2019 at 09:54

Code:

perl -0777 -i.original -pe  "s/ringStyle.*\}\)/ringStyle = new ol.style.Style\(\{\nfill: null,\nstroke: new ol.style.Stroke\(\{\n color: '#0000DD',\n lineDash:[4,4],\nwidth: 2\n \}\)/igs"  derpy.js

You needed to escape your brackets and braces with a \ (on both your input and output)

This command replaces

Code:

ringStyle.*\}\)

With whatever

Where .* matches "anything" 0 or more times
followed by a single brace and a single bracket

This only matches . with \n because of the /s at the end of the command that makes perl think the entire string is one line when it is not.

I havent used this tool https://github.com/beautify-web/js-beautify but it looks like it would tidy up your code for human readability. It seems to have a web gui if you dont want to use the command line and its active so has been updated recently.

Feek · 6 Jul 2019 at 15:52

Thanks, we're so nearly there!

There are other instances of ringstyle within the file so running that command does what is needed on the first instance but destroys the file following the second instance. I've tried removing the g from the /igs but that doesn't seem to stop it. The file goes from 1900 lines long down to a little under 800.

It's also behaving a bit funky. I can see exactly what it's supposed to be doing. As you say it looks for ringstyle and then replaces everything between ringstyle and }) with the following text. Except that it's not quite right.

It does as it's supposed to but then truncates the next line from }); to a single }

It goes from this:

to this

Which it shouldn't do.

Here's the exact command I'm running

Code:

perl -0777 -i.original -pe  "s/ringStyle.*\}\)/ringStyle = new ol.style.Style\(\{\nfill: null,\nstroke: new ol.style.Stroke\(\{\n color: '#0000DD',\n lineDash:[4,4],\nwidth: 2\n\}\)/igs" derptastic.js

If I change the final 2\n\}\) to be 2\n\}\)feek then the output above will have })feek on the penultimate line and the last line will be }

:confused:

MonkeyBasher · 6 Jul 2019 at 16:08

I have not tested this one as im omw out but try this.

Code:

perl -0777 -i.original -pe  "s/var\s*ringStyle.*\)\};/var ringStyle = new ol.style.Style\(\{\nfill: null,\nstroke: new ol.style.Stroke\(\{\n color: '#0000DD',\n lineDash:[4,4],\nwidth: 2\n\}\)\n\}\);\n/igs" derptastic.js

It replaces the entire

Code:

var ringstyleSTUFF)};

with

Code:

NEW STUFF

and then puts an extra newline on the end

This should alter every instance of ringstyle being declared in the file but will not touch the assignments.

Feek · 6 Jul 2019 at 19:48

MonkeyBasher said:
This should alter every instance of ringstyle being declared in the file but will not touch the assignments.

Thank you. It runs, creates a backup and exits without any errors.

But doesn't do anything to the file. I feel we're really close though.

MonkeyBasher · 6 Jul 2019 at 21:54

I just ran it on test file containing

Code:

derpy derp derp derp her
herp derp s derp
{
        cheese;
        var ringStyle = new ol.style.Style({
                fill: null,
                stroke: new ol.style.Stroke({
                colour: '#000000',
                width: 1
                )}
        )};
        cake;
}

and it produced this

Code:

derpy derp derp derp her
herp derp s derp
{
        cheese;
        var ringStyle = new ol.style.Style({
fill: null,
stroke: new ol.style.Stroke({
 color: '#0000DD',
 lineDash:[4,4],
width: 2
})
});

        cake;
}

If its not mathcing for you and giving no errors then the input pattern is wrong

Code:

var\s*ringStyle.*\)\};

I cant see any way to make this any simpler and still match. Also I think this pattern is too loose and is matching too much and replacing extra stuff.

chroniclard · 6 Jul 2019 at 22:14

Hmm can you not refactor so don't have tons of duplicate code....

Feek · 6 Jul 2019 at 22:31

Sussed it - Well partly.

The brackets were the wrong way round in the command. I've corrected that and it runs. I now go from:

Code:

                var ringStyle = new ol.style.Style({
                        fill: null,
                        stroke: new ol.style.Stroke({
                                color: '#000000',
                                width: 1
                        })
                });

to

Code:

                var ringStyle = new ol.style.Style({
fill: null,
stroke: new ol.style.Stroke({
 color: '#0000DD',
 lineDash:[4,4],
width: 2
})
});

}

But. See that there's a final } there.

My new file is 787 lines compared to 1907 for the original. That } has not been appended by the command, it's just at the end of a huge chunk of code that's been removed.

MonkeyBasher said:
Also I think this pattern is too loose and is matching too much and replacing extra stuff.

Yes. We're nearly there, it's just this bit.

MonkeyBasher · 6 Jul 2019 at 23:26

Code:

perl -0777 -i.original -pe  "s/var\s*ringStyle{1}\s*=\s*new\s*ol\.style\.Style{1}\(\{\n\s*fill:\s*null,\n\s*stroke:\s*new\s*ol\.style\.Stroke{1}\(\{\n\s*colour{1}:\s*'#000000',\n\s*width{1}:\s*1\n\s*\}\)\n\s*\}\);{1}/var ringStyle = new ol.style.Style\(\{\nfill: null,\nstroke: new ol.style.Stroke\(\{\n color: '#0000DD',\n lineDash:[4,4],\nwidth: 2\n\}\)\n\}\);\n/igs" derpy.js

fixed by making sure there were no

Code:

.*

and also using

Code:

{1}

to make sure we match against only 1 occurence of a literal string.

Rock and Roll Saturday night, HashBrown Perl - Living The Dream

Feek · 7 Jul 2019 at 11:37

Thanks. I did have a quick look after you posted that last night but didn't do anything with it as I was full of beer! This morning I've had a good breakfast and am back on the case.

No error.

No change in the file. Which means that it's not finding a match.

To test further, rather than run it over my original source, I've created derpy.js and put this into it:

Code:

             var ringStyle = new ol.style.Style({
                        fill: null,
                        stroke: new ol.style.Stroke({
                                color: '#000000',
                                width: 1
                        })
                });

Run the command

Code:

sudo perl -0777 -i.original -pe  "s/var\s*ringStyle{1}\s*=\s*new\s*ol\.style\.Style{1}\(\{\n\s*fill:\s*null,\n\s*stroke:\s*new\s*ol\.style\.Stroke{1}\(\{\n\s*colour{1}:\s*'#000000',\n\s*width{1}:\s*1\n\s*\}\)\n\s*\}\);{1}/var ringStyle = new ol.style.Style\(\{\nfill: null,\nstroke: new ol.style.Stroke\(\{\n color: '#0000DD',\n lineDash:[4,4],\nwidth: 2\n\}\)\n\}\);\n/igs" derpy.js

No change.

It looks to me as though it should work, the format looks right. I'm stumped.

MonkeyBasher · 7 Jul 2019 at 12:46

Ok. I have spotted the problem. Its a common one when programming. I am not running my JS code which is why I did not hit it.

You need to remove 1 character from the input pattern.

Here is a clue.

https://www.youtube.com/watch?v=qXuvFtcKHKc

Feek · 7 Jul 2019 at 15:59

Aww man, I've spent more time examining that than a sixteen year old spent studying the underwear section of the Grattan catalogue in the early 80s looking for a hint of areola

I can't find it and the clue didn't help (although I did waste half an hour looking at old adverts).

MonkeyBasher · 7 Jul 2019 at 17:57

Feek said:
Aww man, I've spent more time examining that than a sixteen year old spent studying the underwear section of the Grattan catalogue in the early 80s looking for a hint of areola

I can't find it and the clue didn't help (although I did waste half an hour looking at old adverts).

Your code uses the american spelling of color. In my test code and my fix I used the queens english spelling of colour.
Alter the fix so that the input pattern uses "color" instead of "colour".
I have just tested on a massive test file with random indents and repeated matches and it works ok.

Code:

perl -0777 -i.original -pe  "s/var\s*ringStyle{1}\s*=\s*new\s*ol\.style\.Style{1}\(\{\n\s*fill:\s*null,\n\s*stroke:\s*new\s*ol\.style\.Stroke{1}\(\{\n\s*color{1}:\s*'#000000',\n\s*width{1}:\s*1\n\s*\}\)\n\s*\}\);{1}/var ringStyle = new ol.style.Style\(\{\nfill: null,\nstroke: new ol.style.Stroke\(\{\n color: '#0000DD',\n lineDash:[4,4],\nwidth: 2\n\}\)\n\}\);\n/igs" derpy.js

Feek · 7 Jul 2019 at 18:20

Oh for goodness sake!

Yes, you're right. I've tested it and it's perfect. Thanks so much for your help and I've learned from it as well.