*hug* i've missed you!
umm, i mean...
*runs*
hello forums!
Forum rules
Please read the Forum rules and policies before posting.
Please read the Forum rules and policies before posting.
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
yeah 1.53 gigs on PHP alone!!! already this month. add to that avatars and forum skins we went a little too easily over my 2gig limit.
I don't know what to think of all this.. it means increased reading of the forums from various different locals. Means we're popular.. but it also means that I've been going up a gig per year in traffic. And that comes from my pocket now :/
not sure how to fix this.. we're now set to 3gigs this month and I've just paid however much extra to get the extra gig.
hmmm...
what to do.. if it were just images that were the issue I can deal with that. Force people to host their own avatars. Limit the number of skins on our forums to just one or two styles.. make sure they are more barren.. but it's not the images taking the bulk.. its the PHP.. how the heck do I get around 1.53gigs of PHP in 24 days??
moo
I don't know what to think of all this.. it means increased reading of the forums from various different locals. Means we're popular.. but it also means that I've been going up a gig per year in traffic. And that comes from my pocket now :/
not sure how to fix this.. we're now set to 3gigs this month and I've just paid however much extra to get the extra gig.
hmmm...
what to do.. if it were just images that were the issue I can deal with that. Force people to host their own avatars. Limit the number of skins on our forums to just one or two styles.. make sure they are more barren.. but it's not the images taking the bulk.. its the PHP.. how the heck do I get around 1.53gigs of PHP in 24 days??
moo
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
Yeah, it's a problem once you become popular!
You said that you just paid for the extra space, but aren't you moving the forums somwehere else anyway? How does the payment affect the move? You could perhaps add a PayPal Donate button to the forums -- I'm sure many people here would be happy to pay to keep the forums up -- I know I would, and I contributed quite a lot to the RTC ezBoard forums (In fact, there's still $20 of mine locked in there that was never used, but I digress...)
There's not a lot we can do about the size of the PHP. But the big new shiny version is supposed to make it more compact and also the styling is simpler and cleaner. But when?? Been waiting for this for I don't know how long!
Just looking at the stats, there is 66MB of not viewed traffic. (Not viewed trafic is trafic generated by robots, worms or answers with special HTTP status code). An 'Unknown' Robot has used 35MB of our allocation, and Alexa has used 23MB!!! Google 'only' used 5MB. By adding a robots.txt file and limiting the ability of search engines to spider the pages we could cut these right down, but it will mean that content of threads won't appear in web searches -- just the front page. I'll look into it.
57% of visits are for less than 2minutes. A lot of people are coming to the site just for a few seconds to see if there's any new posts. Great for popularity, hell for bandwidth! I wouldn't want to ask people to come less frequently though!
280MB is jpgs/gifs. There may still be a fair amount of work that can be done on compressing these (though quality will suffer, Cow!) Again, I can look into this. There was 1 hit on a BMP file -- what on earth is that? The PHP generated 1.53GB. I'll see if there's anything that can be done about streamlining it a bit. I seem to recall Chronicles and Subsilver embed their stylesheets which would make the index page a lot bigger. I'll check it out. You're right too, we could limit the styles used.
I noticed that the viewtopic.php and posting.php pages are the largest on average. We could edit the posting page to remove the IFRame that contains the thread your replying to. This might help. But it does mean that you can't refer to previous posts directly when composing your reply (and I do that all the time!). Other small changes coud be made, such as removing less used columns from certain pages, such as 'No of times viewed' etc. If nobody really needs those, it might help reduce the generated HTML a bit.
On average, Mondays seem to generate the most traffic, which is a bit odd. May 24 was 212MB of data transfer alone, the highest this month. Busiest times are between 4pm-8pm GMT.
There are a few small things that could be done. You can see now why I delete needless quoting!
You said that you just paid for the extra space, but aren't you moving the forums somwehere else anyway? How does the payment affect the move? You could perhaps add a PayPal Donate button to the forums -- I'm sure many people here would be happy to pay to keep the forums up -- I know I would, and I contributed quite a lot to the RTC ezBoard forums (In fact, there's still $20 of mine locked in there that was never used, but I digress...)
There's not a lot we can do about the size of the PHP. But the big new shiny version is supposed to make it more compact and also the styling is simpler and cleaner. But when?? Been waiting for this for I don't know how long!
Just looking at the stats, there is 66MB of not viewed traffic. (Not viewed trafic is trafic generated by robots, worms or answers with special HTTP status code). An 'Unknown' Robot has used 35MB of our allocation, and Alexa has used 23MB!!! Google 'only' used 5MB. By adding a robots.txt file and limiting the ability of search engines to spider the pages we could cut these right down, but it will mean that content of threads won't appear in web searches -- just the front page. I'll look into it.
57% of visits are for less than 2minutes. A lot of people are coming to the site just for a few seconds to see if there's any new posts. Great for popularity, hell for bandwidth! I wouldn't want to ask people to come less frequently though!
280MB is jpgs/gifs. There may still be a fair amount of work that can be done on compressing these (though quality will suffer, Cow!) Again, I can look into this. There was 1 hit on a BMP file -- what on earth is that? The PHP generated 1.53GB. I'll see if there's anything that can be done about streamlining it a bit. I seem to recall Chronicles and Subsilver embed their stylesheets which would make the index page a lot bigger. I'll check it out. You're right too, we could limit the styles used.
I noticed that the viewtopic.php and posting.php pages are the largest on average. We could edit the posting page to remove the IFRame that contains the thread your replying to. This might help. But it does mean that you can't refer to previous posts directly when composing your reply (and I do that all the time!). Other small changes coud be made, such as removing less used columns from certain pages, such as 'No of times viewed' etc. If nobody really needs those, it might help reduce the generated HTML a bit.
On average, Mondays seem to generate the most traffic, which is a bit odd. May 24 was 212MB of data transfer alone, the highest this month. Busiest times are between 4pm-8pm GMT.
There are a few small things that could be done. You can see now why I delete needless quoting!
Last edited by Gambit37 on Wed May 26, 2004 1:25 pm, edited 1 time in total.
-
ChristopheF
- Encyclopedist
- Posts: 1538
- Joined: Sun Oct 24, 1999 2:36 pm
- Location: France
- Contact:
Just an idea...
The "Quick Links" drop down list is present on every page of the forum. Its size is around 3330 bytes.
Looking on the statistics access of this site, I see that in May 2004, about 53225 pages have been viewed already (and the month is not finished)
So it represents 53225 * 3330 bytes, which is around 170 MB
So removing these links from the forum pages and putting them on the main home page only can reduce trafic.
Another way would be to put them in an external file (a javascript) that could be cached by client browsers and just leave a function call in place of the full list in the php forum page.
We can also try to optimize the pages by removing all the useless spaces, tabulation and carriage returns they contain.
There are tools that can perform such work automatically on lots of pages. They become far less readable, but you can have huge improvements in bandwidth.
You told that you wanted to switch to another provider, will you have the same bandwidth limits there?
Looking on the statistics access of this site, I see that in May 2004, about 53225 pages have been viewed already (and the month is not finished)
So it represents 53225 * 3330 bytes, which is around 170 MB
So removing these links from the forum pages and putting them on the main home page only can reduce trafic.
Another way would be to put them in an external file (a javascript) that could be cached by client browsers and just leave a function call in place of the full list in the php forum page.
We can also try to optimize the pages by removing all the useless spaces, tabulation and carriage returns they contain.
There are tools that can perform such work automatically on lots of pages. They become far less readable, but you can have huge improvements in bandwidth.
You told that you wanted to switch to another provider, will you have the same bandwidth limits there?
Christophe - Dungeon Master Encyclopaedia
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
Good idea about the quick links too, I can implement that straightaway. It's actually part of the main header for the forum software, which is rendered on every page, but would be easy to make external.
Also, I'm looking at the template code as we speak. A lot of the background stuff could also be called from Javascripts, but I would have to hard-code some of the phpBB parameters which means that you can't edit the theme using the Admin panel. I doubt this matters though -- don't think anyone else looked at the theming except me!
Also, I'm looking at the template code as we speak. A lot of the background stuff could also be called from Javascripts, but I would have to hard-code some of the phpBB parameters which means that you can't edit the theme using the Admin panel. I doubt this matters though -- don't think anyone else looked at the theming except me!
-
andyboy_uk
- On Master
- Posts: 647
- Joined: Thu Feb 20, 2003 1:51 am
- Location: London, UK
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
An interesting question. It depends on how you use the forums currently. If you tend to click the "Dungeon-Master.com Forum Index" link a lot to get back to the front page, that increases the amount of data downloaded, as each time the index page is updated with your list of read/unread posts, it refreshes the php generated HTML. You could perhaps get in to the habit of using your browser's back button more, so that you're not continually loading a fresh page. But this can get confusing, as you the icon indicators are then lying to you....
Also, when you post a message, you are normally automatically taken back to your post. This is kind of unnecessary too, and adds to bandwidth usage. Perhaps we could change this bit of code so that the auto thing doesn't work?
I dunno -- I'm not keen on limiting users navigation options. I would prefer to make some of the changes I listed above.
Also, when you post a message, you are normally automatically taken back to your post. This is kind of unnecessary too, and adds to bandwidth usage. Perhaps we could change this bit of code so that the auto thing doesn't work?
I dunno -- I'm not keen on limiting users navigation options. I would prefer to make some of the changes I listed above.
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
I've added a robots.txt file which disallows indexing of everything from the root, except the homepage and the homepage of the forums. This shopuld mean that www.dungeon-master.com and www.dungeon-master.com/forum/ will still be indexed by search engines, but nothing else.
It'll take a few days if not weeks to see any effect.
It'll take a few days if not weeks to see any effect.
interesting - i tend to use the back button alot and also interrupt the auto-load of the typed message to keep navigating
Of course, i also visit a hell of a lot, so don't feel as smug : (
Also, i imagine the chroniclas skin is a larger drain since it has the spellbook icons and the textured green background
Of course, i also visit a hell of a lot, so don't feel as smug : (
Also, i imagine the chroniclas skin is a larger drain since it has the spellbook icons and the textured green background
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
I got the Chronicles images folder down from 157Kb to 67.9KB using quite a bit of image compression. Not ideal, but I don't think most people will notice the difference in quality, but this could make a fair difference to the overall download rate. SubSilver and CodexDark are already pretty well optimised, not really worth making further gains there. I got the CodexDark style sheet down from 10.6k to 7.7K just by taking out extra spaces,tabs,retruns and comments. I also reduced the DM Java avatar pics folder from 118Kb to 39Kb.
I've also changed the quicklinks to use an external javascript file as Christophe suggested.
Any comments on the other suggestions I made at the top?
I've also changed the quicklinks to use an external javascript file as Christophe suggested.
Any comments on the other suggestions I made at the top?
-
ChristopheF
- Encyclopedist
- Posts: 1538
- Joined: Sun Oct 24, 1999 2:36 pm
- Location: France
- Contact:
Gambit, I'd prefer to keep the IFrame on the posting page, I find it very useful.
Maybe we can try to optimize the viewtopic.php and posting.php pages by removing all the unnecessary spaces and crlf from them.
Maybe we can try to optimize the viewtopic.php and posting.php pages by removing all the unnecessary spaces and crlf from them.
Christophe - Dungeon Master Encyclopaedia
-
sucinum
- Pal Master
- Posts: 872
- Joined: Wed Apr 18, 2001 1:00 am
- Location: Karlsruhe, Germany
- Contact:
hmhmhm - i got 20 gb traffic für 3€ in my webspace-packet. is that more expensive elsewhere?
1 idea here: a quick-answer-field, that's a phpbb-hack which allows to answer without having to hit the reply-button, this could safe a bit of bandwith and would also add to comfort.
http://www.phpbbhacks.com/viewhack.php?id=540
http://www.phpbbhacks.com/viewhack.php?id=522
http://www.phpbbhacks.com/viewhack.php?id=586
another maybe useful hack:
http://www.phpbbhacks.com/viewhack.php?id=1548
another thing i stumbled across (nothing to safe bandwidth, though, sounds still useful):
http://www.phpbbhacks.com/viewhack.php?id=424
i didn't check if the hacks a compatible with the actual board-version and maybe you got the idea to look there by yourselves already...
hope i could help
1 idea here: a quick-answer-field, that's a phpbb-hack which allows to answer without having to hit the reply-button, this could safe a bit of bandwith and would also add to comfort.
http://www.phpbbhacks.com/viewhack.php?id=540
http://www.phpbbhacks.com/viewhack.php?id=522
http://www.phpbbhacks.com/viewhack.php?id=586
another maybe useful hack:
http://www.phpbbhacks.com/viewhack.php?id=1548
another thing i stumbled across (nothing to safe bandwidth, though, sounds still useful):
http://www.phpbbhacks.com/viewhack.php?id=424
i didn't check if the hacks a compatible with the actual board-version and maybe you got the idea to look there by yourselves already...
hope i could help
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
well, I plan to be moving these forums soon yes.. and my paying has no effect on that. However, if the forum had steady support then it's possible that we could do away with extreme limitations we are looking at and not need to suggest people like fuzz face here should skimp on their enjoyment of the forums so I can keep bandwidth down.
In addition to that.. the plans I had hoped to address in the future such as uploading maps and sounds etc... for DM could be done as well.
I need to discuss it with Craig and perhaps even ZPC (if you're reading.. what was that deal you were offering again?) with regards to space and bandwidth.
My issue is responsibility for the forums staying up.. and currently they've been down a few times. I hate it when it happens since I'm the one who convinced you all to bring everything here.. So for every minute to day to week it might be down I feel absolute crap for it. So, I need to be sure we have space, bandwidth, and reliablitiy...
Which is why I'd like to go with someone I know.. who has their own server.. so I can contact them directly if there is an issue.
moo
In addition to that.. the plans I had hoped to address in the future such as uploading maps and sounds etc... for DM could be done as well.
I need to discuss it with Craig and perhaps even ZPC (if you're reading.. what was that deal you were offering again?) with regards to space and bandwidth.
My issue is responsibility for the forums staying up.. and currently they've been down a few times. I hate it when it happens since I'm the one who convinced you all to bring everything here.. So for every minute to day to week it might be down I feel absolute crap for it. So, I need to be sure we have space, bandwidth, and reliablitiy...
Which is why I'd like to go with someone I know.. who has their own server.. so I can contact them directly if there is an issue.
moo
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
LMAO!
i thought i was going mad, since it worked last night yet was back at work this morning...i kept my avatar since it was offsite, but changed back to sub-silver!
can you view bandwidth usage of a person? i'd be interested to see how much gets used with this rather than not in a normal few days browsing
i thought i was going mad, since it worked last night yet was back at work this morning...i kept my avatar since it was offsite, but changed back to sub-silver!
can you view bandwidth usage of a person? i'd be interested to see how much gets used with this rather than not in a normal few days browsing
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
Ok, I will host my avatar myself.
Please, do not remove automatic return to the topic you've posted to, I like to see confirmed that my post is properly posted.
Please do not remove viewtopic, I find it very useful.
Please do not remove chronicles, for a site devoted to an rpg I find it much more atmospheric than ugly subsilver.
Yes, please do add a paypal button.
Cows: Since we do not have a paypal button yet, I will contribute something along with the payment for the Chaos figurine when you have mine ready for shipping.
Please, do not remove automatic return to the topic you've posted to, I like to see confirmed that my post is properly posted.
Please do not remove viewtopic, I find it very useful.
Please do not remove chronicles, for a site devoted to an rpg I find it much more atmospheric than ugly subsilver.
Yes, please do add a paypal button.
Cows: Since we do not have a paypal button yet, I will contribute something along with the payment for the Chaos figurine when you have mine ready for shipping.
Parting is all we know from Heaven, and all we need of hell.
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
Yeah ! Good idea, since therea are probably also looking for email addresses to spam.cowsmanaut wrote:___________________________pages_HITS__Bandwidth___LASTVISIT________
msnbot64061.search.msn.com 12245 12245 621.19 MB 29 May 2004 - 05:47
msnbot64062.search.msn.com 10556 10556 577.84 MB 27 May 2004 - 12:45
uh.. this is over a gig of traffic from these two bots.. can we block THESE two?
Gambit: I've uploaded my avatar gif that Cows had made for me to my own site ( http://lunever.bei.t-online.de/daten/lunever.gif ), but although it is the same file that I had downloaded from the site where Cows had hosted it before, the profile now complains "The avatar must be less than 80 pixels wide and 80 pixels high" and I didn't save the link where Cows had it hosted before. I suppose some forum parameter has changed somehow. Can you adapt the file so it will work again as an avatar?
Parting is all we know from Heaven, and all we need of hell.
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
Let me know when you've grabbed this one, and I'll delete it:
http://www.dungeon-master.com/forum/ima ... ever2a.jpg
http://www.dungeon-master.com/forum/ima ... ever2a.jpg
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
-
cowsmanaut
- Moo Master
- Posts: 4378
- Joined: Fri Jun 30, 2000 12:53 am
- Location: canada
-
Gambit37
- Should eat more pies
- Posts: 13715
- Joined: Wed May 31, 2000 1:57 pm
- Location: Location, Location
- Contact:
The msn data you looked at isn't bots, but in fact hosts (ISPs). That list shows how much data is transferred to users connected through that ISP. So you don't need to worry about it.
Encouragingly, the robots.txt file is getting successful hits already -- assuming I did it right, we should monitor June and compare it to May to see what the difference is. I'll keep the data for comparison.
EDIT: I have just looked at the latest logs using software called Nihuo WQeb Log Analyser (free for 30 days). Unfortunately, our host only keeps raw logs for 1 day, so I can only look at stats for Sunday/Monday.
It's interesting though, as I discovered that Tymaul's avatar that he's using in the Dungeon-Maker forums links here, and some chap on a German forum is linking to your Linflas avatar. Also, another problem is that Google's cache links to images here and that also steals our bandwidth as it's piped through their systems.
I can do the following if you wish:
1) Prevent remote image linking
2) Request that any pages from this site be purged from Google's cache. This takes a couple of weeks to be fulfilled.
This software is more detailed. It also tells me that someone in Zimbabwe has used the most bandwidth. Further investigation shows me that it's not infact Zimbabwe but part of the RIPE network, and it's in the range of numbers assigned to a French host/registrar called Proxad, who own the free.fr host.
There's lots more info like this available. Would take some time to analyse it in detail. What would be very useful is if we could get a months worth of raw logs to view, rather than the processed logs that we can only view using cpanel.
Encouragingly, the robots.txt file is getting successful hits already -- assuming I did it right, we should monitor June and compare it to May to see what the difference is. I'll keep the data for comparison.
EDIT: I have just looked at the latest logs using software called Nihuo WQeb Log Analyser (free for 30 days). Unfortunately, our host only keeps raw logs for 1 day, so I can only look at stats for Sunday/Monday.
It's interesting though, as I discovered that Tymaul's avatar that he's using in the Dungeon-Maker forums links here, and some chap on a German forum is linking to your Linflas avatar. Also, another problem is that Google's cache links to images here and that also steals our bandwidth as it's piped through their systems.
I can do the following if you wish:
1) Prevent remote image linking
2) Request that any pages from this site be purged from Google's cache. This takes a couple of weeks to be fulfilled.
This software is more detailed. It also tells me that someone in Zimbabwe has used the most bandwidth. Further investigation shows me that it's not infact Zimbabwe but part of the RIPE network, and it's in the range of numbers assigned to a French host/registrar called Proxad, who own the free.fr host.
There's lots more info like this available. Would take some time to analyse it in detail. What would be very useful is if we could get a months worth of raw logs to view, rather than the processed logs that we can only view using cpanel.