hello forums!

Post by **cowsmanaut** » Tue Jun 01, 2004 8:55 am

why is it MSNBOT.Search or whatever then?

with bot in the name and search in there too.. makes me thing it's a search bot going through every single bit of page and catalogueing it.

moo

Post by **Gambit37** » Tue Jun 01, 2004 3:05 pm

Hmmm... Good point. I'm not sure. It's a bit confusing as AWStats lists msnbot64061.search.msn.com in the Hosts information, and it's far too massive to be a search bot -- it *must* be a host.

I'll see what I can find out.

BTW, I implemented the prevention of remote linking images. Tymaul, you will need to take a copy of your avatar and host it somewhere else for the Dungeon Maker forums as it will no longer show up there.

Post by **Gambit37** » Tue Jun 01, 2004 3:20 pm

Did some searching and found a number of issues, which I think are all related to our problem:

1) MSN has a new search facility coming out, and a new spider has been doing the rounds recently eating up tons of the entire webs bandwidth to build their database -- found lots of people complaining about it

2) Spiders can get locked into loops on dynamic sites (such as these forums) because of session IDs -- found something about a hack for phpBB that can disable session IDs for certain search bots

However, the fact that I have put a robots.txt at the root should prevent any further massive bandwidth eating because nothing will be allowed to search the actual topics data -- they will all stop at the forum index page.

I personally think this is OK -- I don't think it's necessary for a search to index every page of these forums. As long as users can find the index page, they can browse themselves.

However, if you want the individual pages to still show up in search engines, then this robots.txt will have to go, and we'll have to suffer massive bandwidth eating...

The whol exercise has been useful for me too -- I ran some tests on my own site and found that people were remote linking to my Tomb Raider Xtra images and eating up loads of bandwidth there too. I've disabled remote linking now... ah, I love .htaccess tricks!

EDIT: I've confirmed that the msnsearch thing is definitely a bot, no idea why it shows up in the Hosts list. Let's give it a week or so to see if robots.txt has an effect. I have read a few sites saying that msnbot doesn't honour it! If we still have problems, we can ban it using IP address in the .htaccess file.

Post by **cowsmanaut** » Tue Jun 01, 2004 5:53 pm

well between the two of them.. MSN should be paying my bill for their exclusive ass romp on my site..

Sheesh!

Thanks Gambit!

Post by **Lunever** » Tue Jun 01, 2004 6:56 pm

Ok, thanx! Seems to work now.

Post by **cowsmanaut** » Wed Jun 02, 2004 1:30 am

ok how about this as a mod.. it if exists? This is a cast back to my BBS days. Show all new posts. It should be like the search button where it shows you just those topics you've flaged. You see, it already knows where all the new posts are since your last visit.. and the code to display individual posts is there as well as the search function.. so everything is there code wise.. they just need to talk to eachother.

Anyway, that would save people loading each page just for 1 or two new posts. To be honest I'm surprised this isn't already there..

moo

Post by **ChristopheF** » Wed Jun 02, 2004 9:05 am

Cows, are you speaking about the "View posts since last visit" option available on top of every page? I am nearly exclusively using this to browse these forums.

Post by **Zyx** » Wed Jun 02, 2004 9:11 am

I browse from this option too.

Post by **cowsmanaut** » Wed Jun 02, 2004 3:22 pm

well I'm virtually blind it seems! Can we make it a bigger link? and move it to the left along with all the actual topic headings? being in fine print and far to the right made me ignore it.

Post by **Gambit37** » Wed Jun 02, 2004 3:24 pm

I only recently started using that too! It's very handy -- if you click on the tiny little arrow next to the Last Post name it takes you straight to the post (as it does from the index page). If you also return here each time, it updates as the posts change so you never have to go back to the index until you've read all new posts.

I use Codexdark and the link is in the Login panel at the top of the page. Is it different in subSilver? It would be easy to add it as a link elsewhere on the page.

Post by **Lunever** » Wed Jun 02, 2004 3:34 pm

Ok, but the "view all new" button appears only at the forum index page. Can we add it to the top of all forum pages please?

Post by **cowsmanaut** » Wed Jun 02, 2004 5:38 pm

the point is you don't need it anywhere else. You come you see new posts and you click it .. it takes you to every one of them automatically.. you don't need to surf the subforums

Post by **Gambit37** » Wed Jun 02, 2004 5:44 pm

Actually, I'm working on this as we speak. You really *do* need it on other pages, because once you've used it to get to a topic, there's no easy way to get back to the View New Posts page.

Currently kind of working on Chronicles and subSilver if you want to check it out.

Post by **cowsmanaut** » Wed Jun 02, 2004 5:47 pm

funny.. I just hit the back button.. it works for me. and if I've posted then I use the little pulldown and go to the one labeled "search" which is the result of the new posts. Works fine.. perhaps it's just my method. I almost always do things this way.. takes much longer for the pages to reload otherwise.

Post by **Lunever** » Wed Jun 02, 2004 6:43 pm

But if you had a link to the "view new posts" you'll have it updated in respect of what posts you've already read/replied and what posts you still have to read. I know, this is a very very minor and unimportant detail, but I'd prefer to have it that way instead of choosing the search page from the pulldown-back-menu.

Post by **Gambit37** » Wed Jun 02, 2004 7:13 pm

It should all be working now in all template styles. I agree with Lunever which is why I've added the following:

Forum Index Page has a new bar at the top with the following options:
View your posts, View posts since last visit, View unanswered posts, Mark all forums read

Topic Summary Page has a new bar at the top:
Goto page(x), View posts since last visit, Mark all topics read

Topic page has an updated bar which includes the View posts since last visit option

I also tidied up some other stuff and moved a few things around. Currently working on Quick Reply mod -- until I post that it's working, please don't use it.

Post by **Gambit37** » Wed Jun 02, 2004 7:30 pm

Quick Reply is now added to all styles/templates. If you don't need any of the fancy features of the full reply page, then please get into the habit of using it as it will help cut our bandwidth usage.

Post by **Gambit37** » Fri Jun 04, 2004 1:00 pm

Seems to have gone very quiet since I made these changes. Did I break something?

If not, comments on the new features are appreciated!

andyboy_uk · Post by **andyboy_uk** » Fri Jun 04, 2004 1:38 pm

Im still here Gambit, and the QuickReply seems to work just fine

A

sucinum · Post by **sucinum** » Fri Jun 04, 2004 2:17 pm

i didn't dare to try it

adds much to comfort imho, next to the bandwidth-thingie

Post by **linflas** » Fri Jun 04, 2004 5:09 pm

quick reply just rocks ! but if you click preview, you get the normal reply form with all (unuseful in that case) previous posts.
so there shouldn't be a preview button for quick reply... imho

EDIT:

something i just think about to save bandwidth : when you have submitted a reply (quick or not), you get the choice between viewing your message or getting back to the forum. this page loads all the graphics just for thos 2 questions and then redirects to your posted message after a few seconds. why not forget this page and get back to the thread directly ?

Post by **cowsmanaut** » Fri Jun 04, 2004 5:24 pm

like the quick reply. works well.. very convenient.. Good job!

Post by **Lunever** » Fri Jun 04, 2004 5:41 pm

I agree with Linflas: I'd prefer a direct return to the thread. Buttons to return to the main forum are provided on any page anyway.

Post by **Gambit37** » Fri Jun 04, 2004 5:48 pm

As you say, when you've posted your message, you get the choice to either view it or return to the forum. I would prefer to leave this to the user for a number of reasons:

1) It's in the forum code and I don't wanna keep messing around with that (rather than a template which is easy to change)
2) Everything on that page is already cached, and the text data is reasonably small so it shouldn't make much difference
3) Sometimes I might wanna return to the thread, sometimes not. That page makes it easy.

Post by **Lunever** » Fri Jun 04, 2004 5:53 pm

Gambit:

1) Good reason, so keep it. But 3) is irrelevant, because whether you click on the forum choice of that little cached in-between-site or whether you click on the forum link in the thread you have been returned to makes absolutely no difference.

Post by **Gambit37** » Fri Jun 04, 2004 6:00 pm

No, they're two different things. The choices given are to "View your Message" or "Return to Forum". The first option takes you back to the message itself, which usually takes you back to the post that you just made. The second option takes you back to the forum index for the current forum.

The reason the first option is useful is in cases where several people may be replying to the same thread at the same time, so by the time you've written and made your post, there may be other posts already in the thread above yours...

I agree about the Preview thing on Quick Reply, and I'll get rid of it.

beowuuf · Post by **beowuuf** » Fri Jun 04, 2004 8:47 pm

I love this new quick reply thing - genius! : )

Post by **Zyx** » Fri Jun 04, 2004 8:55 pm

yep!

Post by **Gambit37** » Sun Jun 06, 2004 1:45 pm

Just having looked at the data usage for the last week, it seems that MSNbot is still gobbling up data, but on average much less than in May. Having said that, it's still a large proportion compared to other robots. So far, Googlebot has only eaten 282Kb, whereas the MSN bot (which for some reason does not show up in the robots list, but the Hosts list), has eaten 15.6MB.

I'm therefore not sure how the fixes I put in place are working for MSN bot, if at all. I'll leave it for the whole month before making a final judgment as things could change drastically.

It may be that the MSN Bot is the thing that was causing the infinite processes to be created on the server. If so, and if we can ban it, a server move may not be necessary.

Let's sit tight and see what happens.

Post by **cowsmanaut** » Sun Jun 06, 2004 9:54 pm

yeah.. and remember that's just in 6 days. we still have the rest of the month to go.