Oblivion

Changes to forum structure

  • Comment
As you might know we've been struggling with our forum server after the past few weeks and have been working towards getting them on the same Cloud setup that's worked so well for us with the main sites themselves.

However, we'd struggled to understand why, all of a sudden, the load times on the forums went from fine to absolutely awful in little to no time at all. There wasn't any sort of traffic spike and the servers weren't being constantly DDoS'd so there was no real reason for it. We knew it wasn't our database because the forums are running on the same database cluster that the sites are using; so the sites would have been slow as well.

We came to the conclusion that it must be related to how heavily we've integrated our forum software, Invision Board, with the sites themselves. All membership information, profiles, file comments and discussions and bug reports, image comments, video comments and Premium Membership information is handled and called through Invision Board and each time we add a new game to the site we were creating 17 new forum categories to support it, to the point where we had close to 3,000 forum categories on the forums. It seems Invision Board doesn't play nice with growth like that. At all. For the technically minded, Invision Board seems to store things like your read topics for every single forum inside variables before every single page load, so if there's 3,000 forums, that's a lot of server resources being used on each page load.

We've been experimenting with a number of different tweaks and changes to the forum servers over the weekend. Because it was the final weekend of the Steam Summer sale we once again saw record numbers of users on the sites (36,000 new members this weekend alone and 7,500 page views a minute). We were very pleased to see the sites held excellently during this time with page load times well within normal ranges and the traffic helped us to not only benchmark potential issues with the sites in light of a Fallout 4 release that's not too far away but also to analyse what would help relieve the load on the forum servers.

Today we've made some relatively large changes to the forum category structure. Most notably, almost all games on the network now only make use of a single forum for all discussion (with the exception of the big games like Skyrim, Fallout 3 and so on). We've done this not because we particularly want to but because the forum software itself cannot handle having 3,000 categories (and growing each time we add a new game to the network). All file, image, video and article comment topics now take up a single category each (as opposed to a category for each game on the network) and all the old topics and posts for files already uploaded have been moved to these single categories. While you can still access the "forum view" mode for file and image comments you will no longer be able to access the file and image comment forum category directly. Because all the files and image comment topics for all games on the network are now mingled together in a single forum category it seems pointless to open up the category for viewing; you'll never be able to find what you're looking for by browsing through that category anyway.

Making this small change has not only reduced our forum category count from 3,000 to 1,200 categories (and we haven't finished doing spring cleaning yet which will likely drop that figure to below 1,000) but it's drastically (DRASTICALLY) reduced the load times on the forum server. Essentially, the forums are now back to being quick and, in fact, better than ever.

We'll now manually add more forums (like forums for "mod requests") as and when they are needed. As it is, a lot of the games we support have no need for forums here and they're added automatically as a token gesture. They might be used, they might not. Point is, they're there, and we can always add more categories for popular games as and when they're necessary.

We've still got some spring cleaning to do and we're still going to go ahead with moving over to the cloud setup for the forum server. The cloud setup has showed its worth to us many times already, but case-in-point, this weekend the NMM virtual machine that handles logins and retrieving metadata for the Nexus Mod Manager became overloaded due to the sheer number of requests being made of it. As soon as we found out about the issue we were able to double the RAM and CPU cores available to the virtual machine, on the fly, within 5 minutes. Problem gone.

Aaaaaand, as I type this, I've just been informed of a DDoS taking place against our servers. It's being looked in to as we speak and I can't speak highly enough of our friends (well, actually my friends, two of whom I lived with in Reading after university and one of whom is a childhood friend from my early secondary school days) at Krystal.co.uk who handle a lot of our hosting requirements for the sites and are always there for the sites whether it's 3pm or 3am if we're having issues. If you're a Brit who needs hosting or has a crap hosting provider and is looking for reliable hosting with great service you really can't do better than these guys. And I wasn't paid for that little advertisement, they just deserve the exposure for the times they're there for the servers, fighting off the kiddies out there who want to ruin it for everyone else (they were also up at 3am on Friday morning helping us to bring back the forum server that decided it didn't want to wake up from a restart).

114 comments

Comments locked

A moderator has closed this comment topic for the time being
  1. Thandal
    Thandal
    • Moderator
    • 183 kudos


    I wonder. Since Browsers are stopping the use of NPAPI plugin Adobe Flash and Shockwave will that effect your www.nexusmods.com page that display advertisements causing loads to be slower trying to filter the advertisements?


     
    We can only hope that Flash dies a quick death now that it's on everyone's hit list.
  2. Deleted54170User
    Deleted54170User
    • account closed
    • 49 kudos
    I wonder. Since Browsers are stopping the use of NPAPI plugin Adobe Flash and Shockwave will that effect your www.nexusmods.com page that display advertisements causing loads to be slower trying to filter the advertisements?
     
    I see the advertisement banners and boxes depend upon the Adobe Flash Player and Shockwave one for their video clips
     
    https://support.google.com/chrome/answer/6213033
  3. speedingpullet
    speedingpullet
    • supporter
    • 0 kudos
    I'd love a way to keep my attribute choices from one visit to another. Every time I come here I have to click 'NMM Compatible" and couple of others - because I've yet to find a way to make them active as part of my personal page setup.
  4. mcgoy
    mcgoy
    • premium
    • 28 kudos
    i suggest putting a link to the forum page for the game associated with the mod at the top of each game (ex. http://forums.nexusmods.com/index.php?c=293,296,301,578)  I think this would make it easier for user to know where to go to get help.
     
    There was a player on another forum who said that he didn't get any help on Nexus when he posted his question.  We had already answered his (simple) question so I gave him the link above so that his questions will be seen by more people in the future.
     
    I'm not sure if these are totally unrelated events or a side effect of the change to the forum structure.  For my own mods, I don't get notifications of new comments anymore.  They show up in the counter, but I never get the 'new' identifier.   I thought I would get some notification of new comments when I 'track' mod, but I haven't received notification and my Nexus account profile shows that I'm not tracking anything.
     
    Thanks
  5. Loveblanket
    Loveblanket
    • supporter
    • 51 kudos
    I know this is off topic, but any ETA on the next version of NMM?
  6. RealAntithesis
    RealAntithesis
    • premium
    • 54 kudos
    Getting back to the loss of the Uploaded Files forum, I also used this a lot to keep myself updated on mod discussions as well as being a way to come across older mods that I don't have and which people are saying good things about (or otherwise). It was a good way to keep up with the current status of a lot of mods easily.

    Suggestion:
    If there's a way to get a list of mods sorted by last comment, clicking on which will redirect to that mod's comment section, that should be sufficient I think. The file list should also be viewable in compact list form rather than the larger image tiles as it is currently.
  7. Deleted54170User
    Deleted54170User
    • account closed
    • 49 kudos


     

     

    I'm lmao, because the problem was the Remember Me switch all this time.


    As many times as I have seen this problem and it never once occurred to me that this was the cause.  ohgeez-1.gif
     


     
    Yup!  This one time that I stopped trying to log in and just sat there staring at my monitor without being signed in reading the latest news about the "Remember me" switch, I GOT IT!  and Voila'!
     
    And I did not un-tick the, "Remember Me" switch again!      I'm Still connected!   
     
    (IMPORTANT Point here: even if we are not signed in we can read the forums posts.)
  8. bben46
    bben46
    • premium
    • 781 kudos

     

    I'm lmao, because the problem was the Remember Me switch all this time.


    As many times as I have seen this problem and it never once occurred to me that this was the cause.  ohgeez-1.gif
  9. Deleted54170User
    Deleted54170User
    • account closed
    • 49 kudos
            All this time I have been unticking the "Remember me"!
     
      Just because I was concerned about the possible bad that might happen, because I had two computers connected to my Modem Router before and they were set up to share. 
     
    I did not reset the LAN sharing on my 2 computers; after; reformatting both of the Hard Drives (needed it anyway) reinstalling the OS on both, (Yup! Needed that too! ), even though I did not reset them so they could share the LAN connection; AND I "Unticked the box next to 'Remember me'" I've been getting signed out like I had a Slooooow Internet connection.
     
    I'm so spooked, I'm afraid the link will drop my connection any second now, after I left the check mark in the tick box next to 'Remember me' alone.
     


     

    I'm lmao, because the problem was the Remember Me switch all this time.

  10. jlevicolley23
    jlevicolley23
    • account closed
    • 35 kudos
    Alright I seem to have figured it out here, when I log in to the forums I'm supposed to tick the box next to remember me for some reason because it seems that if I try without ticking the box it won't let me log in, problem solved - thanks for the help guys, it is very much appreciated