[Forum Support] I want an archive of my posts

[The-Bleh-Bleh]

I'd like to have a local archive of my posts. Is anyone aware of a simple way to retrieve them? If it comes down to it I'll have to write a web crawler to do the job.

I'd appreciate any help. There's some sentimental value, along with an interest in programmatically analyzing the data.

[Jeff]

http://forums.filefront.com/search.p...duser&u=161621

This can be found on your user profile under the [statistics] tab. You can see your profile by clicking your username in the top right corner in the cell referencing your last login and any private messages you may have.

[The-Bleh-Bleh]

Right but I'm looking for a way to download them. The reason is that forums disappear all the time. Considering the hours users put into posting I'm surprised it's not more commonly requested/built in to VBulletin. Or is it?

[G.R.A.E.M.E.]

Beside the above mentioned statistics tab, you can also find your posts by clicking the number of posts underneath your avatar – it's a link that takes you to that page.

As far as I know, there's no standard way of downloading a collection of posts. The only thing that makes use of that functionality is the private messages, which you can download in several formats, such as plain text and XML.

[FileTrekker]

Originally Posted by The-Bleh-Bleh

Right but I'm looking for a way to download them. The reason is that forums disappear all the time. Considering the hours users put into posting I'm surprised it's not more commonly requested/built in to VBulletin. Or is it?

I'm not why a collection of out of context posts would be worth anything, even to the author?

Get it in RSS format and save it?

[The-Bleh-Bleh]

Originally Posted by FileTrekker

I'm not why a collection of out of context posts would be worth anything, even to the author?

I want to be able to look back at how my writing and thought process progressed over the years. Plus I can do silly stuff like go through it and look at trends like post length, frequency of swearing, etc. I just think it would be interesting.

Now I'm trying to work out how to write a program to get them. Think it would hurt FFGF if I downloaded a few thousand pages of content? I don't know if that would put a dent anything ._.

[FileTrekker]

Originally Posted by The-Bleh-Bleh

I want to be able to look back at how my writing and thought process progressed over the years. Plus I can do silly stuff like go through it and look at trends like post length, frequency of swearing, etc. I just think it would be interesting.

Now I'm trying to work out how to write a program to get them. Think it would hurt FFGF if I downloaded a few thousand pages of content? I don't know if that would put a dent anything ._.

It probably wouldn't. Google do it every day. Just as long as it doesn't request everything too quickly?

[G.R.A.E.M.E.]

Like FT said, make sure the requests aren't made too quickly or it might seem like an attempt at DoS (not that one would succeed...)

Build a delay in it per request. 100 to 150 ms should be fine as an amount without having to suffer from it.

[MrFancypants]

Well judging by our robots.txt you are not supposed to crawl the website. Not sure how old that file is though. The robots.txt on break seems to allow crawling, so I'd go with that.

The problem with the statistics tab is that it only shows the last few hundred posts. You can google for the syntax of the search reqeusts for vbulletin, then divide the task into smaller bits (e.g. find my posts for the last month, download, restart with the next month).

[The-Bleh-Bleh]

Thanks. I've written a working script to do a search request. It's just a matter of breaking it up now. I tried limiting results by date but that wasn't enough. I've figured it out though.

Spoiler:

The search query:
- Advanced search for my username
- Get the threads with at least X replies
- Sort ascending by replies

If I set X to 0 that gives me 300 results. Throw away the results near the end that have the greatest reply count. Now set X to that maximum reply count and get the next 300 results... So on. Do that until I have all of my posts.

Nothing would be skipped.

Spoiler:

Unless the user had posted in >= 300 threads that have the exact same reply count...

I'll probably get to writing it this weekend. When I'm done I'll put up the source in case anyone wants it.