PHP Duplicates

meghatronic · 20 Oct 2006 at 16:21

I have a set of data like below that contains duplicates.

I'd like to remove all the duplicates - leaving just one "version" of the line...

example data: (not duplicate)

xx_453845.1 1500
xx_453845.1 1500
xy_455456.1 1600
etc...

Surely there is an easy way to do this?

Would idealy like input to be text area and output to be echo, for now.

(aside and unrelated, how do you get php to output tabs.. at the moment it outputs as above: blah_number, how do I get blah_tab_number)

shine · 20 Oct 2006 at 16:24

PHP:

$tab = "A tab is \t";

should do it.

Augmented · 20 Oct 2006 at 16:25

What is the source of the data set? If you have an array, you can strip duplicates with array_unique().

As for tabs, you use the control character "\t" e.g.

Code:

$tabs = "$blah\t$number";

Make sure that \t is within double quotes, so it is not treated as a literal string.

meghatronic · 20 Oct 2006 at 16:29

Augmented said:
Make sure that \t is within double quotes, so it is not treated as a literal string.

Bingo.

as for the the source, it is a text file at the moment.... I wanted to just cut and paste into text area....

something like array_unique($data); where the $data is contents of file/textarea....

Augmented · 20 Oct 2006 at 16:34

If it's a file with each value on a new line, then you can use the file() function to load it straight into an array, and then run array_unique() on it

.

meghatronic · 20 Oct 2006 at 16:45

<?php
$list = file('data.dat');
$list = array_unique($list);
file_put_contents('data.dat', implode('', $list));
?>

Doesn't work....

"Fatal error: Call to undefined function: file_put_contents()"

Shall I guess I have to use fopen, fclose... now?

robmiller · 20 Oct 2006 at 16:46

jdickerson said:
<?php
$list = file('data.dat');
$list = array_unique($list);
file_put_contents('data.dat', implode('', $list));
?>

Doesn't work....

"Fatal error: Call to undefined function: file_put_contents()"

Shall I guess I have to your fopen, fclose... now?

Yep, file_put_contents is PHP5 only.

meghatronic · 20 Oct 2006 at 17:18

robmiller said:
Yep, file_put_contents is PHP5 only.

RAGE.

I thought I had php5 - oh no wait.. I think I do on the pc at home, not the host..

Bah. Just did it on my box instead.

Not sure if it has worked 'properly' though....

meghatronic · 22 Oct 2006 at 02:45

Hmmmm...

Example data is:

Code:

NP_057850.1 155030 
NP_057850.1 155030 
NP_579876.2 155030 
NP_057852.2 155807 
NP_057853.1 155871 
NP_579876.2 155030 
NP_057854.1 155908 
NP_057854.1 155908 
NP_057854.1 155908 
NP_057854.1 155908 
NP_057854.1 155908 
NP_057855.1 155945 
NP_057855.1 155945 
NP_057855.1 155945 
NP_579881.1 155030 
NP_057852.2 155807 
NP_579876.2 155030 
NP_579876.2 155030 
NP_579876.2 155030 
NP_705928.1 155348 
NP_705928.1 155348 
NP_579880.1 155030 
NP_579876.2 155030 
NP_579883.1 155030 
NP_579883.1 155030 
NP_057853.1 155871 
NP_579883.1 155030 
NP_579883.1 155030 
NP_057853.1 155871 
NP_579883.1 155030 
NP_579883.1 155030 
NP_579883.1 155030 
NP_579876.2 155030 
NP_579876.2 155030 
NP_057853.1 155871 
NP_579876.2 155030 
NP_579876.2 155030 
NP_057853.1 155871 
NP_057850.1 155030 
NP_057850.1 155030 
NP_057855.1 155945 
NP_579876.2 155030 
NP_579876.2 155030 
NP_057853.1 155871 
NP_579881.1 155030 
NP_057852.2 155807 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_579876.2 155030 
NP_579876.2 155030 
NP_057853.1 155871 
NP_579883.1 155030 
NP_579883.1 155030 
NP_579876.2 155030 
NP_579876.2 155030 
NP_057853.1 155871 
NP_579876.2 155030 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057854.1 155908 
NP_579881.1 155030 
NP_057852.2 155807 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_579880.1 155030 
NP_579876.2 155030 
NP_057850.1 155030 
NP_705928.1 155348 
NP_579876.2 155030 
NP_057850.1 155030 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057853.1 155871 
NP_057850.1 155030 
NP_057853.1 155871 
NP_579876.2 155030 
NP_579880.1 155030 
NP_057850.1 155030 
NP_057851.1 155459 
NP_579883.1 155030

How could I remove duplicate lines from this?

ABove method seems to remove more than duplicates...

robmiller · 22 Oct 2006 at 03:10

Code:

fwrite($fh=fopen('data2.txt','w'),join('',array_unique(file('data.txt'))));fclose($fh);

Works for me :k

meghatronic · 22 Oct 2006 at 11:32

robmiller said:
Code:

fwrite($fh=fopen('data2.txt','w'),join('',array_unique(file('data.txt'))));fclose($fh);

Works for me :k

I'm just paranoid it is taking away more than it should

Cheers rob!

meghatronic · 22 Oct 2006 at 15:49

that appears to just blank the file??