Pied Piper Project IV: A New Bot

Pages

AuthorTopic: Pied Piper Project IV: A New Bot
Law Bringer
Member # 2984
Profile Homepage #0
This topic might come a bit out of the blue, because this time it wasn't preceded by an "annual board purge" topic. But it is always a good idea to prepare ahead of time - especially since the dark UBB'Nigurath has taken to scheduling unannounced purges itself.

Unlike Endeavorbot, Piperbot doesn't really do much. Aside from me doing some manual archiving (and some tedious restoring when MoS was devoured a year ago), nothing has been added to the archives since PPP3 in January 06. Unfortunately, this means the Magic Flute (my spidering script) is also the oldest PHP script on the Ermarian Network going by updates.

So I have taken the time to rewrite it, integrating it with the new Drupal site. There's not much more to say, really, except this:

http://pied-piper.ermarian.net/topic/1/4166

The archiving of one single topic works. What remains to be done is the coordination: In order to keep a more or less up-to-date archive that is actually effective against unexpected topic glitches, topics need to be re-archived once they're posted in. Combining it with the RSS feed should do the trick - although that feed itself badly needs an update.

But anyway, PPP4 is about to launch, and if it goes right, it will be the last PPP that has a number - from then on, archives should be continuous.

Edit: And yes, I realize character entities are broken. I'm pretty sure it's a display bug though; so the data itself isn't corrupt.

[ Tuesday, May 22, 2007 03:04: Message edited by: Dr. Johann Georg Faust ]

--------------------
Encyclopaedia ErmarianaForum ArchivesForum StatisticsRSS [Topic / Forum]
My BlogPolarisI eat novels for breakfast.
Polaris is dead, long live Polaris.
Look on my works, ye mighty, and despair.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Raven v. Writing Desk
Member # 261
Profile Homepage #1
Can we all please stop titling stuff in reference to the Star Wars franchise? :P

--------------------
Slarty vs. DeskDesk vs. SlartyTimeline of ErmarianG4 Strategy Central
Posts: 3560 | Registered: Wednesday, November 7 2001 08:00
Guardian
Member # 2080
Profile #2
Question, what happens the the really old stuff on that archive? I'm noticing a distinct lack of stuff from early 2003 and 2002(among other dates).

--------------------
"I don't understand a word you just said. Try speaking American. It's the only language I understand."
Posts: 1918 | Registered: Sunday, October 13 2002 07:00
Councilor
Member # 6600
Profile Homepage #3
Originally by Slartucker:

quote:
Can we all please stop titling stuff in reference to the Star Wars franchise? :P
What do you propose we title stuff after instead? :P

Dikiyoba.
Posts: 4346 | Registered: Friday, December 23 2005 08:00
Infiltrator
Member # 4248
Profile #4
The Matrix Trilogy!

...no, wait, the abominable photothread got that one, I think.

Well, there is still Terminator...

[ Tuesday, May 22, 2007 07:05: Message edited by: Frozen Feet ]

--------------------
I have nothing more to do in this world, so I can go & pester the inhabitants of the next one with a pure concscience.
Posts: 617 | Registered: Tuesday, April 13 2004 07:00
? Man, ? Amazing
Member # 5755
Profile #5
I recommend naming things as they were done in medieval times.

PPP1: The exhalted one.
PPP2: The conqueror.
PPP3: El gordito.
PPP4: ?

--------------------
quote:
Originally written by Kelandon:

Well, I'm at least pretty sure that Salmon is losing.


Posts: 4114 | Registered: Monday, April 25 2005 07:00
Raven v. Writing Desk
Member # 261
Profile Homepage #6
Clearly, the Star Trek franchise. :P

...although PPP IV: The Voyage Home would lead to some great ads involving Aran and whales.

--------------------
Slarty vs. DeskDesk vs. SlartyTimeline of ErmarianG4 Strategy Central
Posts: 3560 | Registered: Wednesday, November 7 2001 08:00
Lifecrafter
Member # 7331
Profile Homepage #7
PPPIV: The Motion Project

--------------------
You Shall Die Laughing: http://www.worfthecat.ermarian.net/converted

The Roost: www.roost01.proboards104.com. Birds of a feather flock together.
Posts: 794 | Registered: Thursday, July 27 2006 07:00
Law Bringer
Member # 335
Profile Homepage #8
PPPIV: Quest of the PiperBot

—Alorael, who hopes the bot has mastered virtues as well as archiving.
Posts: 14579 | Registered: Saturday, December 1 2001 08:00
Law Bringer
Member # 6785
Profile #9
Asimov robot books:

I, Bot
Posts: 4643 | Registered: Friday, February 10 2006 08:00
Law Bringer
Member # 2984
Profile Homepage #10
quote:
Originally written by Yama:

Can we all please stop titling stuff in reference to the Star Wars franchise? :P
I think I'm entitled; I more or less started it. (Well, it was started by PPP2, before I had charge of it) :P

quote:
Originally written by Bandit Keith:

Question, what happens the the really old stuff on that archive? I'm noticing a distinct lack of stuff from early 2003 and 2002(among other dates).
PPP is not as old as the dawn of time. The first concerted effort to make complete and systematic (though manual) copies of all threads - as opposed to saving a few you liked - was the Pied Piper Project in 2004. (By a funny coincidence, exactly three years ago yesterday, which I only realized just now.)

There was at least one purge that never got archived. On General, this covered all the topics to the low 800s (although a few older topics survived because they were still being posted in then).

[ Tuesday, May 22, 2007 14:07: Message edited by: Dr. Johann Georg Faust ]

--------------------
Encyclopaedia ErmarianaForum ArchivesForum StatisticsRSS [Topic / Forum]
My BlogPolarisI eat novels for breakfast.
Polaris is dead, long live Polaris.
Look on my works, ye mighty, and despair.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Guardian
Member # 2080
Profile #11
But what you missed is the fact that the last time I checked it(a couple of weeks ago) it did have stuff from the time period I referenced, but now it doesn't. I was just curious as to why.

--------------------
"I don't understand a word you just said. Try speaking American. It's the only language I understand."
Posts: 1918 | Registered: Sunday, October 13 2002 07:00
Law Bringer
Member # 2984
Profile Homepage #12
Are you sure? Which ones are those?

There are several topics that I've had to put on ice because the files can't be parsed until I improve the parser (it hangs on deleted members), but that's all many months old...

--------------------
Encyclopaedia ErmarianaForum ArchivesForum StatisticsRSS [Topic / Forum]
My BlogPolarisI eat novels for breakfast.
Polaris is dead, long live Polaris.
Look on my works, ye mighty, and despair.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Raven v. Writing Desk
Member # 261
Profile Homepage #13
Really.... PPP2, The Wrath of Imban.

--------------------
Slarty vs. DeskDesk vs. SlartyTimeline of ErmarianG4 Strategy Central
Posts: 3560 | Registered: Wednesday, November 7 2001 08:00
Veteran*
Member # 5
Profile #14
I was just thinking... I've really been at these boards since 2001? That's 6 years. *shutter*
Posts: 455 | Registered: Tuesday, May 17 2005 07:00
Warrior
Member # 4710
Profile #15
It's scary when I checked this old unmarked folder and found the remains of my scattered effort in PPP2 just around the same time I was reading this topic... The ghost of the past haunts us all.

[ Wednesday, May 23, 2007 20:08: Message edited by: JMAN123 ]

--------------------
Life is short. Use a Mac.
Posts: 85 | Registered: Wednesday, July 14 2004 07:00
Law Bringer
Member # 2984
Profile Homepage #16
Oh really? What did you save?

I never did collect all of PPP2. What I do have will take a while to line up for "digitizing" into the database from the files I have, but I'd very much like to have the complete archives before I start doing that. :)

Edit: Time for a stats table again.

+------+--------+--------+
| Era | Topics | Posts |
+------+--------+--------+
| PPP1 | 765 | 25440 |
| PPP2 | 247 | 8067 |
| PPP3 | 6980 | 102312 |
| PPP4 | 10547 | 148739 |
+------+--------+--------+
PPP1 and PPP2 aren't completely in the database. The former due to errors I haven't fixed yet, the latter because I don't yet have everything.

PPP4 is much bigger than PPP3 because it covers the whole forum both times, but only General got purged in the meantime. There is a huge amount of redundancy, but with the space available, that's better than trying to figure out which threads have had new posts (or edited posts) in the meantime.

Also, PPP4 isn't actually archived. What is shown above is merely the result of the indexing I just completed. Since there is no urgency, I had this idea of archiving the topics of PPP4 asynchronously, with one topic every five minutes (avoiding overloading the server). This will take over a month, so be patient (I'm not Google :P ).

[ Wednesday, May 23, 2007 23:52: Message edited by: Dr. Johann Georg Faust ]

--------------------
Encyclopaedia ErmarianaForum ArchivesForum StatisticsRSS [Topic / Forum]
My BlogPolarisI eat novels for breakfast.
Polaris is dead, long live Polaris.
Look on my works, ye mighty, and despair.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Shaper
Member # 73
Profile #17
Question: How will this automated PPPbot handle edited and/or deleted and/or moved posts?

--------------------
My Myspace, with some of my audial and visual art
The Lyceum - The Headquarters of the Blades designing community
The Louvre - The Blades of Avernum graphics database
Alexandria - The Blades of Exile Scenario database
BoE Webring - Self explanatory
Polaris - Free porn here
Odd Todd - Fun for the unemployed (and everyone else too)
They Might Be Giants - Four websites for one of the greatest bands in existance
--------------------
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Posts: 2957 | Registered: Thursday, October 4 2001 07:00
Off With Their Heads
Member # 4045
Profile Homepage #18
quote:
Originally written by Dr. Johann Georg Faust:

(I'm not Google :P ).
Pssh, only for lack of effort. :P

--------------------
Arancaytar: Every time you ask people to compare TM and Kel, you endanger the poor, fluffy kittens.
Smoo: Get ready to face the walls!
Ephesos: In conclusion, yarr.

Kelandon's Pink and Pretty Page!!: the authorized location for all things by me
The Archive of all released BoE scenarios ever
Posts: 7968 | Registered: Saturday, February 28 2004 08:00
Law Bringer
Member # 2984
Profile Homepage #19
quote:
Originally written by The Almighty Do-er of Stuff:

Question: How will this automated PPPbot handle edited and/or deleted and/or moved posts?
Topic-specific changes are not yet fully taken care of. PPP archives all pages in a flat file cache, but also parses them and stores the posts in a database.

The files get overwritten of course. What happens to edited posts in the database - whether they get stored as new revisions or overwritten - isn't yet implemented. Up till now (and for quite a while yet) every topic only gets saved once, so I haven't had to deal with that.

--------------------
EncyclopaediaArchivesMembersRSS [Topic / Forum] • BlogPolarisNaNoWriMo
Look on my works, ye mighty, and despair.
I have a love of woodwind instruments.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Lifecrafter
Member # 375
Profile #20
quote:
Originally written by Yama:

Really.... PPP2, The Wrath of Imban.
IMBAAAAAAAAAAAN!!!11
I couldn't help it, I'm sorry.
But hey, these archives can bring up some gems:
quote:
Originally written by Majordomo:

But ADoS sounds like something that 1337 guy ezAndy would do!
H4 h4 h4 h4, y0u h4v3 n0 ch4nc3 70 b3 |337, m4k3 y0ur 71m3.

--------------------
"I liked white better."
Posts: 771 | Registered: Tuesday, December 11 2001 08:00
Shaper
Member # 73
Profile #21
quote:
Originally written by Majordomo:

But ADoS sounds like something that 1337 guy ezAndy would do!
I don't get it. Maybe if it was in context it would make more sense.

Aran, make sure that if posts are deleted PPP doesn't delete them from the archive, because then that would cause bad stuff.

--------------------
My Myspace, with some of my audial and visual art
The Lyceum - The Headquarters of the Blades designing community
The Louvre - The Blades of Avernum graphics database
Alexandria - The Blades of Exile Scenario database
BoE Webring - Self explanatory
Polaris - Free porn here
Odd Todd - Fun for the unemployed (and everyone else too)
They Might Be Giants - Four websites for one of the greatest bands in existance
--------------------
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Posts: 2957 | Registered: Thursday, October 4 2001 07:00
Law Bringer
Member # 2984
Profile Homepage #22
Well, I have no chance of stopping the file from being overwritten, but the files are only kept for safety anyhow. The database holds it, and in the database the deleted posts would stay around.

--------------------
EncyclopaediaArchivesMembersRSS [Topic / Forum] • BlogPolarisNaNoWriMo
Look on my works, ye mighty, and despair.
I have a love of woodwind instruments.
Posts: 8752 | Registered: Wednesday, May 14 2003 07:00
Lifecrafter
Member # 375
Profile #23
Whoops, forgot about the context.
http://pied-piper.ermarian.net/topic/1/1296/p3
http://pied-piper.ermarian.net/topic/1/1296/p4
I came back, said 1337 for no reason at all, and then I left again. I also got an "omg, ezAndy".
It's just one of those moments that make no sense when you look at it in retrospect. Gives you that feeling of warm, fuzzy nostalgia.

--------------------
"I liked white better."
Posts: 771 | Registered: Tuesday, December 11 2001 08:00
Lifecrafter
Member # 7557
Profile #24
quote:
Originally written by Kelandon:

quote:
Originally written by Dr. Johann Georg Faust:

(I'm not Google :P ).
Pssh, only for lack of effort. :P

lol seconded :)

--------------------
The force is with ME! I'm not so sure about YOU!

Trust me, you do not want to click this link
Posts: 942 | Registered: Sunday, October 8 2006 07:00

Pages