Migration of Encyclopaedia forums

Post by **ChristopheF** » Mon Nov 03, 2003 12:34 am

I spent my week end working on the migration of my forums at dmweb.free.fr/Forum (using phorum software) to this place (using phpbb).
I have now a method of migration and a home made script. I still have some details to fix.

I plan to do the migration during the next week end. During the migration, both forums will be shut down and unavailable. I think I will need half a day, probably on saturday.

Is it OK for everybody?

Note: the migration will include user accounts (and passwords) and all posts from all forums except general discussion and uploads forum. I will post the details later.

Post by **cowsmanaut** » Mon Nov 03, 2003 4:25 am

sounds good. Make sure you work with Gambit on this to be sure there is no duplication of users if we can help it. or accidental over writting of users here.

moo

Post by **ChristopheF** » Mon Nov 03, 2003 8:59 am

There won't be any user duplication. For users who have accounts in both forums, the posts will be attributed to the correct existing user account here. That was one of the problem I had to solve.

Post by **Gambit37** » Mon Nov 03, 2003 11:43 am

Have you sorted out what happens with user groups? Even though we don't use them on here, you have to assign one to each user or most stuff doesn't work...

andyboy_uk · Post by **andyboy_uk** » Mon Nov 03, 2003 12:11 pm

Sounds alright to me boss, good luck with the migration gents. Hope it all goes well

MadMunky · Post by **MadMunky** » Mon Nov 03, 2003 12:50 pm

Sweet, sounds good.

All website linking to this main forum, so get quite busy

PicturesInTheDark · Post by **PicturesInTheDark** » Mon Nov 03, 2003 12:57 pm

Great you found the time - it's a shame the encyclopedia has suffered so much lately due to the posting problem, but it's still the best overall resource out there, so I'm glad we finally get one place where the major action takes place without losing information.

Good luck, schedule is fine with me.

Regards, PitD

Post by **Lunever** » Mon Nov 03, 2003 5:31 pm

Great!

beowuuf · Post by **beowuuf** » Mon Nov 03, 2003 6:39 pm

I have no problems with that, and a big thanks Christophe for doing the work!

And weird, all four admins are around at the same time - hello!

Post by **ChristopheF** » Tue Nov 04, 2003 12:58 am

Gambit, my script will create a group for each migrated user and put him inside the group. I don't know the use of these groups, but I am just replicating the same config for new users as the config for existing users.

In both forums (here and at the encyclopaedia), there are posts made by unauthenticated people. These posts do not count in the total number of posts of user accounts, even if it is the same username. I plan to fix that to include all posts of registered users in their post count.

Do you agree with this?
For example, Gambit, there are already 153 posts on this board that you made (probably on other migrated forums) but that are not counted in your total.

Post by **Gambit37** » Tue Nov 04, 2003 3:02 am

Hi Christophe. It sounds like you've created a good Phorum --phpBB converter that lots of people have been asking for -- out of interest, will you make it available to others? I remember when I first looked into this, there weren't any good Phorum converters...

As for post counts, if you want to consolidate them, feel free. The reason I didn't bother is that I ended up doing a lot of the work manually -- by changing my system clock before making each post. The only way I could do that was if the users all were classed as Guests-- which is why they have a + in front of their name.

If you have generated a script that RELIABLY can match guest posts with authenticated users, then I would say YES, go ahead and do it. Rather you than me -- I got in to an awful mess with it which is another reason why I didn't do it. There may also be users that have similar guest post names that match one authenticated user. For example, I originally had names of Gambit, Gambit37, MHill and Proteus. Obviously, I could match these manually because I remembered my user names; you won't necessarily have that knowledge.

So, feel free to do it if you are confident of a good result, but personally I think it's a lot of work for little gain.

I assume that you will do the work offline, then upload a new, merged database?

sucinum · Post by **sucinum** » Tue Nov 04, 2003 3:19 am

i'm very glad to hear that

don't waste time in fixing postcounts and details, it's ok to merge only the surely duplicate users (even though it took me about 1 yr to find out how and where to register at the encyclopedia-forum, i didn't see the button before - lol).

Post by **ChristopheF** » Tue Nov 04, 2003 8:55 am

I don't plan to release my script, because it is very ugly, does not perform good error checking (if there are errors, I just delete the database and restore the previous database. there is no rollback, no transactions), and moreover it is rather specific to these particular forums.
And the user migration is not fully automated because there are name conflicts and differences between the two forums. I manually make some changes to the phorum tables before the migration to avoid a few doubles.
I can send you the script if you want to have a look (it is written in PHP), but I doubt it would be very useful to someone else.

For post counts, I will do the update, but only for manually selected users (those with names that I am sure I can match, like Gambit and gambit37). Then the update will be automatic, so I can do it for any number of users.

I will work offline as you say.
First, I will password protect access to this forum with an .htaccess file.
I'll make a backup copy of the original DB.
Then I will work offline in merging the forums. I'll make some tests offline to check that everything is fine.
Then I will upload the new DB, replacing the existing one.
I will do some more tests in real conditions. As I will be the only one to know the password, I will be the only one able to connect and this will prevent visitors from messing in the DB while I check (and maybe fix) things.
Then when all is OK, I'll remove the .htaccess file and we will enjoy the merged forums!

I will keep my script so that I can use it again when phpbb will support attachments in posts.

PicturesInTheDark · Post by **PicturesInTheDark** » Tue Nov 04, 2003 10:18 am

I don't mind the post counts if that eases your work, but I'd be glad if posts made at the encyclopedia would be seen as made with the same user I'm using here since I'm always using the same name and email adress anyway ;o)

Again, good luck,

regards, PitD

Post by **ChristopheF** » Tue Nov 04, 2003 8:42 pm

The posts you made with your "PITD" user at dmweb.free.fr will be linked to your existing account here.

PicturesInTheDark · Post by **PicturesInTheDark** » Wed Nov 05, 2003 9:23 am

I hope you are talking about the "PicturesInTheDark" ones and not "PitD" - if I remember correctly I only use the registered abbreviation three times, everything else was with the (non-registered) longer name - it's so much easier to find older posts if they all run under the same name should the forums get migrated...

Regards, PitD

Post by **Gambit37** » Wed Nov 05, 2003 12:38 pm

Christophe, your plan seems very sound. It's similar to what I did with the ezBoard migration, except for the fact I did no live testing -- it was before I understood what the .htaccess file was so I didn't know how to do the password protection.

As for the script, don't worry about sending it to me -- I probably wouldn't understand it anyway! You're much more a programmer than me -- I can do basic stuff, but prefer to work on design rather than scripting.

Out of interest, what do you do for a living?

Post by **ChristopheF** » Fri Nov 07, 2003 6:56 pm

Gambit, I am employed by a French IT services company and currently working for a corporate customer. I am what we could call a specialist in some Microsoft products and technologies, mainly Windows 2000/2003 and Active Directory, Exchange (messaging) and ISA server (firewall/proxy).
And I can also use some non-Microsoft technology like PHP/MySQL when needed

... on top of the IIS web server of course!

)
I am not a very skilled developer, but scripting never frightens me (VBscript, Javascript, PHP, ASP).
And I am no designer at all... You probably already noticed that on the encyclopaedia

For the forum migration, I do not use the phpbb functions. My script performs all the changes at the database level (using lots of SQL requests). Then I checked that the behavior was OK while browsing the forum.
So the script would probably stop working with newer versions of phpbb if they change their database scheme.

At this level, it is relatively easy to convert posts from unregistered users to posts from registered users. The only issue is matching user accounts with unregistered posters. I did that manually.

PicturesInTheDark · Post by **PicturesInTheDark** » Sun Nov 09, 2003 1:19 pm

Christophe, two questions: first, is the migration still going to happen this weekend? It seems like a close call already... Second: Did I misunderstand you when you said which posts you are going to assign to my nickname here or is there no easy way to do that? Can I help in any way ? Never mind the post-count - but I'd like to be able and find them again under my own nickname...

Regards, PitD

Post by **ChristopheF** » Sun Nov 09, 2003 7:06 pm

I have not been able to perform the migration this week end because I could not backup the database of my forum at dmweb.free.fr: the phpmyadmin interface (to manage the database) of my ISP is not working.

I spent some more time on the migration script and fixed some problems with bbcodes in posts (this will migrate bold, italic, underlined texts as well as urls, images, quoted text, etc...).
I have also found a script to update the search tables, so it will allow full search on all posts on this board (that is not the case today, I think migrated posts from ezboard and maybe others are not indexed yet).
As a result of the migration, the size of the database will increase by around 11 megabytes.

I will perform the migration as soon as I can access my database, I hope this will be tomorrow evening.
You will know that "The Grand Merger" has started because you will not be able to access both forums during the operation.

Again, sorry for the delay,

PicturesInTheDark · Post by **PicturesInTheDark** » Sun Nov 09, 2003 9:46 pm

Thanks for the detailed explanation and again: good luck.

Regards, PitD