Team 10 News Generator - No more slacking at work for dekez!

Soldato
Joined
16 Dec 2005
Posts
14,443
Location
Manchester
As the title says, I have created a Team 10 News Generator! :D

It isn't quite finished yet as I need to iron out a couple of kinks and add some team info. However, below is a sample output of the script so far [this is what gets spat out to the console, the final script will also create spreadsheets and a text document you can copy-paste into the forum]

Currently the script works as follows:

1: Stanford daily_user_summary.txt file is downloaded

2: File is parsed for the first 250 Team 10 usernames
This will currently grab anyone who has more than 80K points. It can be adjusted in future, but parsing takes quite considerable time, so the fewer Team 10 users we grab, the quicker the file will be parsed.

3: With the collection of usernames, the script generates urls to the EOC stat XML feeds and downloads each one.
An example: http://folding.extremeoverclocking.com/xml/user_summary.php?un=Biffa&t=10

4: The script reads each XML file downloaded and creates a new object for each user.
For the non-programmers, an object is basically a collection of data that represents something, in this case a F@H User.

5: The first 250 users contains a lot of slackers/inactives, so the script prunes those out.

6: The script then loads a previously created XML file containing the calculated data from the previous running of the script. Comparisons are then made between each user.

7: The latest stats are written to a new XML file, for using a week later. The info is also printed to the console.

I quite like how it has turned out but there is a rather large issue. Because it is relying on EOC stats, the news wouldn't be entirely accurate if it was still done on a Thursday. This is because a week for EOC is Sunday 00:00 to Saturday 23:59. If we use the script in the current form, the News would have to be on Saturdays to be the most accurate.

I have dabbled with generating my own stats from scratch and not relying on the EOC stats, but that will take some more tinkering! :D

There will still have to be some manual formatting and tinkering, so dekez isn't completely out of a job, but the script serves to remove a lot of the copy/paste malarky and spending hours working out rank changes etc.

Here is the output I promised earier [whitespace collapsing by the forums makes it look awful, but you get the idea!]:

Team 10 News Generator said:
TOP CRUNCHERS

Change Username Points this Week
------ -------- ----------------
0 ChrisBrid 200704
0 Biffa 211345
0 Speed 187614
0 Team_Muck_Spreader 139166
0 dekez 46866
0 Bigstan 6165
0 ReaVerUK 53876
0 SiriusB_[OcUK] 47865
0 AHameed 44540
0 Brown_Sparrow 57376
0 emailiscrap 22557
0 Peter439921 25415
1 datalol-jack 24924
-1 miniyazz 18751
0 Culinia 27022
1 LizardKing[OcUK] 11112
-1 Sir-Les-MP 12590
0 Richy1204 20760
1 lemonman 14993
-1 Pilgrim57 13212
0 Rincin 11949
0 LigerZero 5501
0 Emilystoodinsomedogdotoday 9386
0 JonathanJDavis 7410
0 PHIL21191 6612
0 rich99million 4146
0 steve258 8247
New vertica 1919
0 Theophany 1279
1 verbal 11724
-1 polandro 2722
0 br83taylor 1753
0 Rooney 1757
0 loudbob 2014
0 dr.who 1021
1 snapsh0t 2162
-1 FatRakoon 1579
1 Chris_Davison 1575
-1 vent 1009
0 jeremysmith123 962
0 Kazz_101 988
0 KALIBR(OcUK) 518
New ShaneOsborne 264
-1 feriso 276
-1 Babyface_UK 303
New Arachnae 895
-2 Montiyouk 291
-2 Saetia 173
New davejuk 209
New deej 74


STOMPY STOMP STOMP

Username Stomps
-------- ------
ChrisBrid 2
Biffa 0
Speed 0
Team_Muck_Spreader 0
dekez 0
Bigstan 0
ReaVerUK 0
SiriusB_[OcUK] 0
AHameed 2
Brown_Sparrow 0
emailiscrap 0
Peter439921 0
datalol-jack 5
miniyazz 0
Culinia 1
LizardKing[OcUK] 0
Sir-Les-MP 0
Richy1204 1
lemonman 0
Pilgrim57 0
Rincin 0
LigerZero 0
Emilystoodinsomedogdotoday 0
JonathanJDavis 1
PHIL21191 0
rich99million 0
steve258 0
vertica New
Theophany 0
verbal 0
polandro 0
br83taylor 0
Rooney -1
loudbob 0
dr.who 0
snapsh0t 0
FatRakoon 0
Chris_Davison 0
vent 0
jeremysmith123 0
Kazz_101 0
KALIBR(OcUK) 0
ShaneOsborne New
feriso 0
Babyface_UK 0
Arachnae New
Montiyouk 0
Saetia 0
davejuk New
deej New


TOP 20 PRODUCERS

Change Username PPD
------ -------- ---
0 ChrisBrid 147655
0 Biffa 116634
0 Speed 89672
0 Team_Muck_Spreader 59017
0 dekez 32096
0 Bigstan 31379
0 ReaVerUK 29007
0 SiriusB_[OcUK] 21269
0 AHameed 20328
0 Brown_Sparrow 18681
0 emailiscrap 13984
0 Peter439921 13227
1 datalol-jack 11751
-1 miniyazz 10940
0 Culinia 9728
1 LizardKing[OcUK] 7340
-1 Sir-Les-MP 7287
0 Richy1204 7264
1 lemonman 7169
-1 Pilgrim57 7144

TIFFY YAY!

1,000,000 - ChrisBrid
24,000,000 - Biffa
7,000,000 - Speed
12,000,000 - Team_Muck_Spreader
9,000,000 - dekez
16,000,000 - Bigstan
2,000,000 - ReaVerUK
3,000,000 - SiriusB_[OcUK]
500,000 - AHameed
500,000 - Brown_Sparrow
400,000 - emailiscrap

So let me know what you think! I will try and have a finished product in a few days! :)
 
Nice one sirius, looks like it works a charm, I was contemplating knocking something up in PHP but haven't yet found the time, and I'm a little rusty which wouldnt have helped
 
SB this is fantastic :D, when I first started doing the news I looked into creating a website to automate the stats but needed to learn PHP and SQL and just didn't have the time.

This is great though, and I don't think the timing would be too much of an issue. I could just set this to run as a scheduled task on my home server at whatever time it needs to be done, then pick up the file and do the formatting at some point after.

Only slight problem is that if it's done on a Saturday I'm sometimes away over the weekends, but so long as people don't mine waiting until Sunday night / Monday morning it is not an issue. Actually come to think of it if I run it on my server I can just remote in to grab the file anyway.

One minor point I would say, could you crop out the usernames where people have 0 stomps, then sort the list so the top stompers are at the top? It would make it a bit neater.

Thanks for this it's really appreciated, looking forward to giving this a go :)
 
I'd run it on the day that makes most sense from a stats perspective, no need to make your life harder than it is. Then post it at your earliest convenience.

Maybe you could script it to post it to the forum automatically :)
 
Any formatting changes are easy enough to implement. I will probably throw in a bunch of options too so you can tweak it how you want. Exporting to CSV will also mean you can rearrange data if the script doesn't order it quite how you want.

Setting it to run as a Scheduled Task is a great idea. The script needs no user interaction and Powershell scripts are easily scheduled. I have a couple running on my own server :D. Probably the best time for it to run is between 2100 and 0000 Saturday nights. Then you can post it on a Sunday or whenever.

Automatic posting to the forums could be interesting. Might give that a go just for the new things to learn! Not sure Spie would like it though. :p
 
Im sure spie wouldnt mind as its for a good cause. I'll quickly have a look as I remember looking into auto posting with PHP
 
Learning how to autopost could be a bad idea with you lot! Within days there will be new threads appearing all the time with automatic Parp and Stomp alerts! :D
 
:D PHP is more confusing now that 2 years ago when I first learnt it:confused:

Yeah I know what you mean, I taught myself html and css using online tutorials and build my website using notepad. I thought to myself - yeah it should be easy to learn a bit of PHP and knock up a stats website - no chance! :p
 
Im fairly efficient with PHP, I wrote a fully fledged CMS a couple of years back for my dads website, lost all of it not long after the website was closed down and the sata controller on the servers mobo died and took a couple of drives with it.

Looks like automated posting is a no go due to vbulletins bot catching systems. Looking at the packets sent when you post a thread there is a security token that im guessing updates in each page load and I cant seem to find it, however I can use a PHP script to send email through the form on my dads Plumbing website. Which reminds me, I need some antibot protection on there.
 
I wrote a mini-CMS in PHP for the SMP Folding Guide that Sir-Les is now hosting. I haven't done much with PHP for a couple of years either. I really should as it is a handy language to know.

Hmm, I suspected auto-posting would be a no go, as it would effectively be a bot and I am sure there are measures in place to detect/block them. Still, it's no bother - Copy-Paste takes all of two seconds and dekez, you could always automatically email the data to someone else - just in case you can't make a post in a timely manner!

I think the script is almost finished, but I want to do a real live test. So today I am going to grab some data and then again in two or three days' time and see what the output is like, then I will have it run again while I am away at the weekend - see how robust it is on its own! :p

If I am happy with the results I will pass the script on to you dekez for you to play with. I need to devise a way of protecting the previous weeks' stats xml - because if that goes missing a lot of the rank changes will go missing [no big deal but it wouldn't be PERFECT! :mad: :p].
 
With a little more effort and looking into implementing cookies etc I may be able to do something, on the otherhand I can now spam my dads email account if he annoys me. I found it extremely handy Ive even used it for wake-on-lan and other strange things as well.

If I post up the packets that are sent during login and post you may be able to make something of it Sirius
 
Not as slick as you guys are after probably but you could perhaps use something like iMacros to help with autoposting? As it goes through Firefox so hopefully no bot problems.
 
I have just written a short script that works using I.E. I don't think it will be counted as a bot since it is using a browser. Instead of me clicking a button, I am just telling I.E. that a particular button is to be considered clicked! :D
 
Is the OcUK stats site still working? Perhaps whoever runs it can generate the news for ya, since they have the DBs and such set up already.
 
Back
Top Bottom