Save prescaled cache

Messages are moved here (should anyone ever want to see them again) once they are no longer applicable to the current version (e.g. suggestions that have been implemented or bugs that have been fixed).

Moderator: George Gilbert

Forum rules
Please read the Forum rules and policies before posting.
Post Reply
PeyloW
Novice
Posts: 29
Joined: Wed Jan 03, 2001 6:00 pm

Save prescaled cache

Post by PeyloW »

I asume the pre-scaled bitmaps enters some kind of cache system. Could not that cache be dumped to disc and then "fast loaded" if present. That way we would not have to wait for the prescale everytime we start.

I myself uses a Pentium 166 and it works fine for playing and gives me time to make coffee when starting up :)
User avatar
amaprotu
Adept
Posts: 211
Joined: Thu Jan 25, 2001 9:47 pm
Location: California, USA
Contact:

Re: Save prescaled cache

Post by amaprotu »

That is a good idea I dont' think was previously mentioned. As long as the dungeon being played was the last dungeon.... Perhaps best would be three options
saveprescale=no
saveprescale=last (save the prescale file for the last played dungeon, overwrites it if a new dungeon is played)
saveprescale=all (potentially require more HD space but save a different prescale file for each dungeon)

In any case i think this should be a seperate file from the dungeon save file. <p>Amaprotu
Mahkahl Darkwolfe "Avvisione"
Flezz Fuu
</p>
- Amaprotu
User avatar
Gambit37
Should eat more pies
Posts: 13766
Joined: Wed May 31, 2000 1:57 pm
Location: Location, Location
Contact:

Re: Save prescaled cache

Post by Gambit37 »

The 'presecale' file that you mention doesn't actually exist as such - everything is just dumped into memory. Windows determines whether or not that's real memroy or virtual memory (swap file). George mentioned this before.

There probably is a way of doing this but it would mean re-engineering a large portion of the code which I suspect is not the priority at the moment. I think George is more keen to get a working version of v1.0 out before doing these kinds of enhancements.
User avatar
amaprotu
Adept
Posts: 211
Joined: Thu Jan 25, 2001 9:47 pm
Location: California, USA
Contact:

Re: Save prescaled cache

Post by amaprotu »

actually when the stuff is prescaled it is then written to memory prescaled, it should be possible without too much extra work to write it to a disk file at the same time, and load from a disk file if it exists. It might make it take a little longer to do the initial prescale but subsequent ones should be better. However I would not know if the advantage gained from not prescaling the images would be lost in the writing to disk as well as memory and the reading from memory in the future. <p>Amaprotu
Mahkahl Darkwolfe "Avvisione"
Flezz Fuu
</p>
- Amaprotu
User avatar
amaprotu
Adept
Posts: 211
Joined: Thu Jan 25, 2001 9:47 pm
Location: California, USA
Contact:

Re: Save prescaled cache

Post by amaprotu »

Um George can you please let us edit our own posts?

I meant in that last line, that the advantage gained might be enough to make it worth the cost of writing to disk as well, and also that reading it from *disk* might negate some of the advantage of having it prescaled. However now that I think about it, it has to read from disk anyways so....

<p>Amaprotu
Mahkahl Darkwolfe "Avvisione"
Flezz Fuu
</p>
- Amaprotu
User avatar
Gambit37
Should eat more pies
Posts: 13766
Joined: Wed May 31, 2000 1:57 pm
Location: Location, Location
Contact:

Re: Save prescaled cache

Post by Gambit37 »

George isn't around until at least October, so I doubt that the ability to edit posts will happen for a while.

The prescale issue is more complex than simply writing out a file, especially while developing new dungeons. The bitmaps that are scaled are determined by what's used in the dungeon. Each time you edit a dungeon and add a new item, new bitmaps have to be pre-scaled. So while it might make sense to have a 'prescale cache' for *completed* RTC files, during development there would currently be no advantage, unless the cache used separate files for each image.

Perhaps another way would be to offer the option to the designer of dungeons? So that when they have a complete RTC file, a small utility is run to create all the pre-scale images. This is then bundled with the RTC file and RTC can read the information from that, reducing the startup time. This would obviously make the download a lot bigger, and I seem to recall that George mentioned somewhere that the pre-scale data could be tens of megabytes big... in which case this isn't an option.

I don't know about all the technicalities, but it does seem weird that the pre-scaling takes so long when the data itself is relatively small.
User avatar
cowsmanaut
Moo Master
Posts: 4380
Joined: Fri Jun 30, 2000 12:53 am
Location: canada

moo moo..

Post by cowsmanaut »

the issue of prescaling and memory and VM etc.. this has all got me thinking.

How did DM do it.. well, as far as I can remember from observation back playing it on my A500.. It would access only on going up or down levels (stairs and pits) except if it was moving monsters which was a quick read (not even a second).

So it would load to memory only what was to be within the level. Now with the speed of hardrives and the amount of ram an many computers today. I'm sure something like this can be used as a way to assure greater speed. After all there isn't much point in having all the sounds and graphics in memory if you don't need them.. shuffling all that crud around is kinda pointless if it's not liable to be used for the next few hours.

loading just what is required for that level and loading from the drive at critical points or preloading when the player gets close to a pit or stairs.. (5 tiles distance or so?)

I also noticed that the game loads far faster if loading bitmaps, rather than pulling and scaling from the archive. Why not have a structured directory where then game can load the first time and dump the bitmaps into as you would have placed them for and override.. this would also make it easier to figure out what and how you were to override when you wanted since you could just go in and replace or edit those that were there.

So it loads up, and reads from it's prescaled dirs and loads only what it needs. many people running with VM will say that providing your are only loading small bits at a time (as you woudl here) it's barely noticable.. it's when you need to constantly load that it chugs. So this is where a fail safe should fall into the code to provide for those who come down the stairs and say oh $#!T a dragon and then go back up.. prepare a poisonball or what not then go back down then run back up then run back down(got the idea?) You would need to hold additional relevent graphics and sounds for a moment.. (that within a range of pit/stairs thing is good here) or within a few steps or what ever..

There is really nothing that can be lost by optimising in this way it's better to have superfast loading and playing than to find it chug here and there. For the person with the 91.7ghrz with 512megs of ram it may seem pointless but for the poor shmoe with 8 megs of ram and a p60 .. well.. you know..

this just happens to be one of those occasions where the graphics don't really need to be hidden away for copyright reasons.. there are already at anyones disposal and so having them in a compressed file is more about size of download and size on disk. So make it an option to set at the first time it's loaded. Much like other programs that ask for your preffrences the first time you load them. it looks for the init.txt file (generated by RTC the first time it's loaded, instead of being int he archive) and if there is none it runs you quickly through the set up. you can choose to dump the graphics and sounds to the override Dir (to speed up loading on most machines) and then it says please wait and writes to the dirs. set up your other stuff and away you go..

anyway, I could keep going.. but I'll shut up now
User avatar
Gambit37
Should eat more pies
Posts: 13766
Joined: Wed May 31, 2000 1:57 pm
Location: Location, Location
Contact:

Re: moo moo..

Post by Gambit37 »

Actually, DM Amiga dumped everything into memory at startup - there are no further disk accesses once the game is started (other than the 'fake' accesses used to scare the player). Not sure about ST or Apple versions though.

However you're right when you noticed a small delay going up some some stairs or falling down a pit. Looking at the CSB code that Paul Stevens made available, there appears to be a GetGraphics() routine that moves graphics from storage memory to operational memory during these occasions - I think that's what causes the tiny delay (that, and computing what is happening on every tile on the new level).

I also think that part of the problem is that regardless of what bit-depth the source graphics are (16 colours or 16-million) they are converted to 16-million during the startup and prescale routines, and this makes even the tiniest graphic take up much more space.

Who knows what the solution is - probably have to wait for George to come back to comment on this.
User avatar
amaprotu
Adept
Posts: 211
Joined: Thu Jan 25, 2001 9:47 pm
Location: California, USA
Contact:

Re: moo moo..

Post by amaprotu »

I could be wrong but I think George said something about levels below and above being active... or did he say they weren't? I can't remember, and don't have the patience to look for it.

You are correct, I was thinking about prescaling only in terms of those playing completed dungeons. I really don't know but it seems a prescale file might be prohibitivly large if you have one for each dungeon, if you have a lot of dungeons.

In any case it should definatly be able to be dissabled so that while you are working on designing the dungeon it doesn't createthe file everytime. However it would be nice I think if it created one when you played a game (and had the flag set) and checked for an existing one to load on subsequent runs. This would only hold the advantage if you are playing the same dungeon time after time, which I am likely to be doing since it will probably take me many many saves to get through the psychotic puzzles I know you guys capable of dreaming up.

Essentially we are trying to get the advantage of prescaled graphics durring the game and of not prescaling em durring boot up, at the cost of hard drive space. If it can be done, sweet. :)

As for loading things when you go up and down stairs.... I know this was discussed earlier but i'm still too lazy to go looking. Maybe it was decided since many of the monsters and textures are going to be on many levels it actually saves time not to reload everything for the level every time you use some stairs but to do it all at the begining.

And I didn't realize that about George being gone, i guess we could talk about him behind his then! er if we could go back and edit our posts before he got back, oh wait ... er.... :P

<p>Amaprotu
Mahkahl Darkwolfe "Avvisione"
Flezz Fuu
</p>
- Amaprotu
User avatar
beowuuf
Archmastiff
Posts: 20687
Joined: Sat Sep 16, 2000 2:00 pm
Location: Basingstoke, UK

Re: Save prescaled cache

Post by beowuuf »

I really hate loading between levels
The cool thing about DM loading everything up at the start was that you were just walking around, up and down stairs
I don't know, having the hang going down stairs would just remind me 'oh, it's the next level' rather then 'ok, what's down here...'

In the Amiga version of EOTB2, it had 5 disks, and only kept two levels in memory at a time
So, going up or down new stairs required lots of loading and at least four disk swaps (2, then 5 then 4 then 2 then 4 or soemthing stupid).
Was unable to (permenantly) kill off ian_scho (Haynuus), Ameena, oh_brother (Westian), money (Falkor), raixel (Petal) and Lord_Bones (Aurek) in the DM D&D game Time's Champions!

CONGRATULATIONS TO THOSE WHO MADE THE GAME WHAT IT WAS - GREAT!
User avatar
cowsmanaut
Moo Master
Posts: 4380
Joined: Fri Jun 30, 2000 12:53 am
Location: canada

dm loading

Post by cowsmanaut »

you were required to have 1 meg to run the game. on the amiga500 it was 512k of fast ram (which is meant mostly for storing executable information) and 512k chip ram (meant for storing graphics and sound)

This disk contains about 720k of information however the graphics.dat is a heavily compressed file not just encrypted. I'm reasonably certain that it contained over a meg of real data. I don't think all the graphics were capable of being stored in the chip ram at one time along with the sounds. There was loading of sounds for sure at the very least. it also accessed the disk for reasons more than just to make you paranoid. As near as I can tell that's just a rumor since it did seem to coincide with monster movements, however I can't for the life of me figure out what it would be getting from the disk at that time.

at any rate.. I think that something needs to be done about load time in RTC and prescaled set is a decent option..
User avatar
Gambit37
Should eat more pies
Posts: 13766
Joined: Wed May 31, 2000 1:57 pm
Location: Location, Location
Contact:

Re: dm loading

Post by Gambit37 »

One of the over-riding principals behind the development of DM, according to Wayne Holder, was that "It should be a real-time experience; nothing should interrupt the gameply such as disk accesses or slow screen re-draws." In the same interview he also confirmed the paranoia disk accesses: "We know the players are smart, we put it in there to keep them on their toes to make them wonder what was being loaded into the corridor ahead."

Although I still have this text, I'm not sure which interview it came from. I think it was one from ST/Amiga Action around about 1989.
User avatar
cowsmanaut
Moo Master
Posts: 4380
Joined: Fri Jun 30, 2000 12:53 am
Location: canada

all the more reason to get that web page up!

Post by cowsmanaut »

Well, I'd like to read that interview since I have very little development information on the project.. not from wayne himself anyway..

it still doesn't adress the fact that the graphics and sounds would likley exceed the available chip ram.. granted they could have a large amount of the graphics in there I don't think they could have all at once. There was pauses between several of the levels where you *did* have to wait for the game to finish loading something. That at the very least was not paranoia loading.. it couldn't have been. It would defeat the point of quick loading. it may be that it stored 1/3 to 1/2 of the required stuff and only loaded new stuff when needed. If I were less lazy I would pull out my miggy and dust of DM and have a go at it.. My memory may be foggy but I think there was at least some actual disk access rather than just 100% ghost accessing..

I suppose we could always ask MR who dissassembled the st version of CSB.. he could assure me as to if I'm a moron or I really did see that.. anyway.. I'll shut up now :wink:
User avatar
Gambit37
Should eat more pies
Posts: 13766
Joined: Wed May 31, 2000 1:57 pm
Location: Location, Location
Contact:

Re: all the more reason to get that web page up!

Post by Gambit37 »

Yeah, I agree, it still seems that there is more data on the disk than would fit in memory. However, the FTL guys were WAY ahead of their time and were doing some pretty funky stuff with compression / memory managament etc, so it's conceivable they found a solution.

My memory is hazy too, but I certainly don't recall any other disk accesses other than the 'paranoia' thing.
dbucher
Neophyte
Posts: 6
Joined: Sat Sep 22, 2001 8:17 pm

How it was on Amiga

Post by dbucher »

On Amiga, I am positive something was happening
sometimes. OK, maybe not load, but memory decompressing, maybe ?

When falling into a pit, it often took 4-6 seconds.

Personnally I would think the following :
DM was keeping in memory 1-2 levels above and
1-2 levels under. Sometimes when taking stairs to
the next level I think there was some loading.

Why not suppose there was one "file" per level,
and only 'differences' were stored in memory ?

I suppose that knowing MORE on the original DM
would help us very much.

WHO HAS SEEN SOURCES ? Someone here says
he saw the CSB sources, do you have them ???
dbucher
Neophyte
Posts: 6
Joined: Sat Sep 22, 2001 8:17 pm

WHAT THE HELL is prescaling ???

Post by dbucher »

Ok now my next question :
WHAT THE HELL is prescaling ???

Yes, could someone (or the honored author)
explain us was *exactely* is prescaled ?
And why exactely 431 ???
And are monsters prescaled or complete
screens ?

And yes, the worse, why not prescale them to
disk, and the following time only rescale the
'new' ones ???

Thanks for any reply :-)
User avatar
amaprotu
Adept
Posts: 211
Joined: Thu Jan 25, 2001 9:47 pm
Location: California, USA
Contact:

Re: WHAT THE HELL is prescaling ???

Post by amaprotu »

I am sure that what prescaling is has been discussed in another thread, but as must be obvious by now I'm way too lazy to go look for it.

Basically (if I recall correctly) most everything in the dungeon (items and monsters) are only drawn to one scale (one square away). Things must then be scaled so they are smaller when further away. The option to prescale goes through all the items in the dungeon and does the scaling and saves the images to memory, this makes the game run faster durring the play but load slower.

The reason this is even an issue (even though it wasn't for the original DM running on an Atari ST or amiga) has to do with the quality of the graphics. Its all explained on the main webpage for RTC somewhere.

The reason for this thread is the hopes of getting the advantages of prescalling with a shorter load time, at least sometimes. <p>Amaprotu
Mahkahl Darkwolfe "Avvisione" (48th level Enchanter [EQ])
Flezz Fuu (27th Level Rogue [EQ])


Deja Moo - The distinct feeling you have heard this bullshit before.</p>
- Amaprotu
User avatar
cowsmanaut
Moo Master
Posts: 4380
Joined: Fri Jun 30, 2000 12:53 am
Location: canada

prescale

Post by cowsmanaut »

dungeon master has a good 500+ images in graphics.dat

George has eliminated a few of those through direct scaling of them real time in the engine. Things like mirrors wall hanging etc.

What the prescale is, is scaling all the graphics from the orriginal DM to make them from 320x200x8bit to 640x400x24/32bit. There is a lot of graphics so scaling them all up takes a while. this is why we are suggesting one prescale to disk as it takes a looong time to decompress and prescale everything each time we load.

If you load the graphics directly from disk it takes nearly half the time. This can be done by using the override funtion and supplying the required graphics.
User avatar
cowsmanaut
Moo Master
Posts: 4380
Joined: Fri Jun 30, 2000 12:53 am
Location: canada

oh yeah..

Post by cowsmanaut »

the sources to CSB are around. they have been decomplied from ST and rearranged they are not exactly what you would call easy reading.

however if you follow the trail to the CSB for PC off of the DM encyclopedia you should find the sources. Look for the latest ones as they are more documented through out and a lot cleaner.
dbucher
Neophyte
Posts: 6
Joined: Sat Sep 22, 2001 8:17 pm

Prescaling

Post by dbucher »

Ok now I understand what prescaling is. Therefore I can conclude that scaling these bitmaps at each load is totally useless, they are always the same.

Therefore scaling them only the first time would be
ok. And the question at installation would be like most
games nowadays :

Which installation do you want :
(O) Minimal (this would be scaling each time)
(O) Medium (Maybe some partial scaling)
(O) Full (Only scaled once and saved on disk)

:-))
_pharaoh

Re: all the more reason to get that web page up!

Post by _pharaoh »

cowsmanaut wrote:Well, I'd like to read that interview since I have very little development information on the project.. not from wayne himself anyway..

it still doesn't adress the fact that the graphics and sounds would likley exceed the available chip ram.. granted they could have a large amount of the graphics in there I don't think they could have all at once. There was pauses between several of the levels where you *did* have to wait for the game to finish loading something. That at the very least was not paranoia loading.. it couldn't have been. It would defeat the point of quick loading. it may be that it stored 1/3 to 1/2 of the required stuff and only loaded new stuff when needed. If I were less lazy I would pull out my miggy and dust of DM and have a go at it.. My memory may be foggy but I think there was at least some actual disk access rather than just 100% ghost accessing..

I suppose we could always ask MR who dissassembled the st version of CSB.. he could assure me as to if I'm a moron or I really did see that.. anyway.. I'll shut up now :wink:
With the ST, there was no division between types of RAM. RAM was RAM, and that was it. However, it's true that, at least on lower-memory machines, it simply could not hold all the code and data in memory. When I was a teen and got my 520STfm (for you non-ST people, it had 512k), and CSB to go along with it (having already gotten hooked on DM on my brother's ST), it would chug hard every time I hit the stairs. When I later upgraded to a meg of RAM, it didn't hit the disk nearly as often. Sometimes I could go several levels before it did, and I suspect those were more "paranoia accesses", as they seemed a lot quicker than when I was running a half-meg.

I think the random disk accesses served another purpose, too- if it loaded the game into RAM all at once, and never touched the drive, you could conceivably pass the disk over to your buddy, and run two games concurrently. If it kept saying "THAT'S NOT THE MASTER DISK!", however, it would encourage your cheapass friend to buy his own copy. ;)
Post Reply