I'd bet fetching was a lot more rewarding when it was first implemented...

From: baconbits | Posted: 08/04/2006 7:35:50 PM
I imagine fetching 500 topics with 400+ new users was probably a pretty rewarding experience.

I wonder if it'd be possible to get a copy of the source of this whole fetch system, so I could play around with fetching, lol.
From: Lord of the Monkeys | Posted: 08/04/2006 8:39:01 PM
Yes, fewer users are being found these days. Though as long as Gamefaqs is signing new users up then there will always be new users to fetch. It was quite rewarding, yes, to fetch several topics on a board and get 100+ new users on it. Ahhhhhh good times
From: The Island of Zeal | Posted: 08/04/2006 9:47:39 PM
At first I didn't know how to login, and I randomly pressed the fetch button on a 500 topic and found 7 users. :( So, I should have 7 more users to my fetch count, but oh well.
From: baconbits | Posted: 08/04/2006 10:00:26 PM
I didn't log in for the first ten or so of mine. :(

I really wish I had a copy of the fetch stuff, :)
From: ActJef1077 | Posted: 08/04/2006 11:44:19 PM
I had about 1000 already fetched when YE started keeping track of how many you fetched
also, I once forgot to log in for an hour or so and fetched like 500 anonymously
From: RockMFR | Posted: 09/04/2006 12:23:44 AM
I probably fetched 15000 before the feature was added.
The Ohio State University: We are so much better than m*ch!g@n.
From: Ryan Dwyer | Posted: 09/04/2006 1:13:35 AM
I can't give you the code, but I can explain in detail what it does.

First, it checks the URL with eregi because I don't know how to use preg_match. If the URL isn't a valid GameFAQs URL, it goes and gives you that nice error message, shows the footer, and stops the script. If it's valid, it will sort the board and topic IDs out of the URL and load them into variables.

Next, it uses CURL to fetch genmessage.php?board=$board&topic=$topic&page=0. It sends my GameFAQs MDAAuth cookie along with it, so when it fetches it's logged into my account and can therefore fetch from boards such as Iconia and LUE. Lots of error checking here. It checks with strpos to see if error messages exist, and/or if the actual user posting data exists. Closed topics show an error message but are still fetchable. If there is an error message and there's no user posting data on the page, it simply echoes the error, shows the footer and stops the script.

It must loop to fetch each page of the topic. It counts the number of posts to determine if it needs to fetch the next page or not. You'll notice if a topic has 50, 100, 150 etc posts it will fetch the next page but find no users.

In each page fetched, it loops through, doing strpos's for the "<A HREF=user.php?user=" code. It finds the user's ID from that link to their profile page, and their name in the say way, loading them both into variables. It attempts to load each user into the database, which may or may not be successful because the user ID field is a unique key.

Keep in mind it uses my GameFAQs account to do this. If I log out and back in, I have to change the MDAAuth value in the script for it to keep working. I can't change my posts per page setting from 50 or it will do strange things. But at least I still get karma if I don't use the boards for a day :P
I will not use abbrev.
From: ActJef1077 | Posted: 09/04/2006 2:17:24 AM
First, it checks the URL with eregi

I stopped trying to understand your post there
From: RockMFR | Posted: 09/04/2006 2:36:00 AM
If I log out and back in, I have to change the MDAAuth value in the script for it to keep working.

That's not necessary. It allows old MDAAuth values to be used, even if you've logged out. I don't know how long any given MDAAuth value remains valid (TEG claims they do eventually expire), but I know they remain valid for a while.
The Ohio State University: We are so much better than m*ch!g@n.

Edited by RockMFR at 09/04/2006 2:39:07 AM

From: baconbits | Posted: 09/04/2006 4:48:33 AM
That's pretty awesome :)

I'm currently coding a board hunter/cataloguer...I'll be sure to let you folks know when I figure it out.