Wednesday, March 05, 2008

Armchair Programmers 4TW!

My hat is off to other developers whether you work on games, websites, business apps or whatever. Programming takes a certain kind of mindset but its both fun and challenging.

However, comments like this make me want to slap someone with a trout.


I'm a programmer and if their code is so poorly written they can't segregate and thread that logic then... well... that's just piss poor programming. Object design mehodology has been around for a lot of years now. It's only the terms that have changed the last few years. And given the other SOE titles out there it's crazy to think they don't have the code already and simply need to plug and play -- once again if they can't -- it's piss poor programming.


First off, Object Oriented Programming has NOTHING to do with multi-threading. Also, whether or not an application is a good candidate for multiple processes/threads/cores has nothing to do with how well it is written. So making a statement that EQ2 is made up of piss poor programming because it doesn't fully make use of multiple processors kind of makes me angry.

I'll be the first person to stand up and tell you that the EQ2 code-base isn't perfect. Find me an application that *IS* perfect, with over a million lines of source code, is over 6 years old and has been worked on by countless people who have come and gone, and I will gladly hand in my resignation. Especially an application like an MMO that is constantly growing and changing as new features are added.

All that aside, we are working very hard to improve the performance of the EQ2 client. It might not be in the form of multiple-cores, but we are looking into all possibilities.

Any programmer that thinks you can flip a switch or "plug and play" some code from another application and you magically have a high performing multi-threaded application seriously needs to find another line of work.

24 comments:

Toldain said...

As a professional developer myself, all I can say is the folks who say that probably are VB hackers. Or have they now turned into C# hackers?

Anonymous said...

It's pretty easy for someone to claim they're a programmer to gain some credibility for their criticism. Multi-threading is bar-none, hands-down, unquestionably the hardest thing to do in the programming world.

Akely said...

Regarding re-doing the whole code to support SMP/Multi-core:

I've read about tools that help out with this. Not being a programmer myself ... is such a tool any jelp at all for something so large/complex as a MMO?

I also seem to remember some brass over at Intel stating that they where working on such tools, the reason being that many programmers just had no skill/experience with multi-core programming. Whats your experience with this? (state of programmers, not uour multi-core preogramming experience).

Greg Spence said...

I'm aware of tools out there such as OpenMP to handle parallel processing. We've actually written our own implementation of OpenMP, and by "we" I mean Autenil. :)

Our initial tests with splitting processing using this type of tool yielded very small performance gains which would probably go unnoticed by most people.

The problem is that most of the work that the client is doing requires that it wait on the previous batch of work. This is the main killer of using multiple cores. Even if you farm a process to another core, if you've got the first core waiting for it to be done, you haven't gained anything.

Rather than focus heavily on splitting the CPU load across cores, I would guess that our biggest performance gains would be accomplished by completely removing some of the processing from the CPU and moving it to the GPU. Unfortunately EQ2 was written to support Geforce3 video cards which only support Shader v1.1. So all of our shaders are written in ASM and would have to be upgraded to HLSL which is a huge amount of work.

Akely said...

Very interesting!

In CPU tests some applications seem to scale perfectly (two cores means twice the speed, three means thrice etc. From what I remember these are video encoding programs and the like.

Do they scale like that becouse there are bot other work to wait for? That is all the cores can help with the same job.

I've read Valve's testing about this too, there (dare I say?) hype about multi-core was that their Steam engine (update) would scale perfectly. But tests show that it did not. In fact is was not impressive at all.

Would it be feasible to move work not needed (if there is such) to other threads? Like (and I'm just guessing here) network communication, chat, G15 support (thanks btw), or perhaps move all CPU bound graphics to one thread?

Interesting tidbit about the shader version support. I did not know that. But now I do realize why so many graphics settings seem to affect the CPU so much!

Thanks Greg!

Greg Spence said...

Someone pointing out an article that I saw today about MS's Flight Simulator being modified to use multiple cores, and even with lots of different areas getting re-written, they only report about a 20% increase in performance.

Our network library is already multi-threaded and on top of that the game client spends less than 1% of its processing time in network overhead, so nothing to gain there either.

Features like the G15 support also don't really do anything to performance thats noticeable. The switching between processes and locking memory would probably take more time than it currently does.

I hope it doesn't sound like I'm being combative, we've just looked at a lot of options, and there really isn't any low-hanging fruit that we've found.

I think the reason encoding/decoding scales so well is because you can take whatever you're encoding, divide it up into chunks, and have each core encode a chunk. Then you piece it back together.

Regarding the shader info, if everyone knew the reasoning why things were done they were, I think they'd be more understanding. When EQ2 went into development, they wanted to support GeForce3, and wanted the game to look just as good on lower hardware. So the only choice was to do many things on the CPU since the graphics card couldn't do it. The idea at the time was that as processors got faster, the performance would improve over time. Unfortunately science didn't lead us in that direction. So if you want to run the game on a GeForce3 today, it will still look awesome! :)

Akely said...

The 20% increase is a common number I've heard in regard to modifications from single to multi-core support.

I do not get the impression you are being combative at all. Rather I get the impression you would like multi-core support, if only the yields where better.

Regarding the shader info, if everyone knew the reasoning why things were done they were, I think they'd be more understanding.

I've been thinking about doing a blog or... something about the technical side of EQ2. Stuff like how-to graphics settings, recommended hardware... stuff like multi-core support and the shader info you mentioned would be an important part of that... Let me think about it for a while longer.

Anonymous said...

Greg based on reading your information it seems to be a complete waste of money to buy new extream computers for EQII since the technology will remain the same or will the code change at sometime to take advantage of these latest greatist V.Cards?

Greg Spence said...

We're working towards improving client performance on new systems but don't have any concrete information or dates to release at the moment.

Currently, the following hardware upgrades may improve EQ2 performance.

Increase RAM (up to 4gb)
Increase hard disk access times
Increase CPU speed

CPU's with multiple cores and SLI setups don't currently offer significant performance benefits for EQ2.

Anonymous said...

Are there any plans for an EQ3? If so , what lessons were learned from the epic failure of a CPU- based game Engine (EQ2) ?

Greg Spence said...

I think you're being a bit harsh calling it an epic failure. The game runs great on my three machines at home. I know lots of people that play EQ2 who have no complaints over performance. Sure, I admit that frame rates could be better and that the client design doesn't make use of some of the latest technologies, but EQ2 is still one of the most successful MMO's out there right now.

As with anything, I'm sure any future projects will take what was learned from the past and put it to good use.

Anonymous said...

Greg,

Some great honest feedback, much appreciated. I am in the process of building a next gen system. THe main game I play is eq2 but I also play bf2142, Crysis and world in conflict.

I am planning on ocerclocking the new Intel processor, thinking of using the new 790i Nvidia chipset, ddr 3 memory, etc.

I am worried on the video card. In a perfect world I wanna go sli on the 8800gts g92 or even maybe the gtx or ultra. From what I have read I need to disable SLI for the game to function (eq2).

I currently have the original 8800gts and get the stuttering that is well documented but the game is playable. I tried an 8800 gtx in my current system and the game would crash and lock up so much it was unplayable. It looked good though int he games I mentioned above plus Vanguard and vista.

If you were building a system to be able to run the next gen games as good as possbile (crysis world inc onflict) and still be able to play eq2 (with no budget contraitns) what would you do.

One more point of interest I generally game in 1900 by 1200, and usually running add on sound card with Klip surround speakers.

I currently have 5 eq2 accounts and 4 are played everyday.

Once again your open honest replies here are very much appreciated even if we don;t like to hear them. (aka all the tasks on the processor)

Philip said...
This comment has been removed by the author.
Philip said...

multi threaded programming isn't that hard for a new development but is vastly more complex to do effectively in an existing code base particularly when not all users have the required hardware to take advantage of it.

As to the original blog - if there are really so many programmers whom never write bad code and whom would get fired for such shoddy work or work on defence systems that HAVE to be correct and so on then I really got no idea where they hide themselves as I never met one.

I have though had the displeasure to work with people whom think they write perfect code - mostly graduates - and they never lasted long (on DB designer boasted to me that when I saw his design I woudl be 'stunned by his brilliance'. He lasted 2 minutes longer than the db go-live, crash and burn)

All in all I have never met an experienced, professional developer whom would make such sweeping & judgemental statements without having studied and understood the code. I have advised a programmer to seek another career following a code review, but never before.

Novusod said...

You are probably going to want to slap me with a trout after I say this but I think it has to be said.

My first experience with dual core systems was actually back in 1998 when Micron Computers was producing high end desk tops with motherboards that had sockets for twin Pentium II processors. No single piece of software I ran on that computer was programmed for multi-threading yet the hardware and the OS/BIOS was able to split the single threaded programs across the two cores resulting in considerable performance over a single Pentium II.

I have been a fan of multi-core systems ever since then and every single computer I have owned has been a multi-core machine. When I first started playing EQ2 in early 2006 it was on multi-core machine that was able to get much better performance than it does today.

I had an old trick back then that let me run EQ2 on both cores by changing the preset affinity for eq2.exe in the windows XP task manager. The difference between one core and two cores was night and day. I know you will tell me until you are blue in the face that your single thread program could never run on two cores but I know what I saw. My hardware and OS was able to split the eq2.exe 70/30 across two cores. Splitting single threaded executables is something I have been witnessing for the better part of a decade and EQ2's massive thread was no exception.

About six months after I have been playing EQ2 your programmers decided to close the loophole so I could no longer reset the affinity to two cores. Since then my rig has suffered massive amounts of lag, stuttering, and slide show FPS which made it very unplayable.

The solution is simple and indeed low hanging fruit for you to pick. Give me back the ability to reset the affinity to two cores. My hardware and OS will do the rest.

Greg Spence said...

Its not possible to undo the "fix" we made awhile back. Many times the high-resolution timers on each core are not in sync with each other and return slightly different timing values. If the application isn't handling the synchronizing between cores, the difference in timing values wreaks havoc on the game code. This resulted in speed hacks and all kinds of problems.

Anonymous said...

What about documenting the server interface, develop an alternative client with minimal functionality but _with_ a multi-threading architecture, open source it and leave the rest to the "armchair programmers" out there ?

It may take some time, but eventually it might evolve into something.

It would open up hacking potentials, but one could limit it to a special test server and re-integrate/re-publish a closed source version for live when its ready.

Mike said...

Greg, I appreciate your honesty here, but I have to say that I really don't care how much work it takes. Make it happen. EQ2 is now getting to the point that you have no choice. Either make your code SMP aware, and utilize multiple GPU's, or start losing players to other games that do. I, for one, am sick of having a top end machine, try to show off the "greatness that is EQ2" to someone, just for them to laugh at the lag when they play games that are just as "beautiful" at 2 to 3 times the framerate, on a lesser machine, too.
Oh, and DROP the support for Geforce 3 if that is part of the problem!! Anyone still using that old piece of junk is in DESPERATE need of a hardware upgrade. Hell, they can buy a card 100 times better for 20 bucks now!

Get back to the core of the code. Rewrite it. Screw expansions if you must. Many more players would prefer performance over content at this point. Fix the code, then get back to making content.

Greg Spence said...

Mike, I appreciated your honesty as well, but telling me to "make it happen" isn't going to get you anywhere. This decision would come from higher up than myself.

Also keep in mind that while rewriting the game engine might be your top priority, every other customer doesn't share the same sentiments. While skipping 1 or 2 expansion cycles might be fine for you, it would not be for everyone. Not to mention the fact that the same people that build content for expansions aren't the same people that would rewrite the engine anyway.

The point is, this is a huge undertaking. Dropping support for the GeForce3 sure does sound like an easy thing to do right? We probably just need to go in and deleted a line of code. Sorry, its not that easy. Support for older cards means that all of the shaders were written in ASM for shader version 1.1. We have thousands and thousands of shaders that would all need to be rewritten by artists in order to work on a new shader system.

It always amuses me when people compare EQ2 to the latest and greatest game out there and expect that it should perform identically. I'm sorry, but the reality is EQ2 is a 7 year old game when you consider the development cycle. Its still a fun game, and we can still create great content for it, but you can't compare it to a brand new MMO.

Realistically what you're likely to see is incremental improvements to the client. I wouldn't hold your breath for a full engine rewrite. We have a new developer starting soon who we will be putting on client-related performance issues.

Improving performance is definitely one of our top priorities, but let us decide how we go about it.

Oh, and I've tried AoC. Its certainly on par with EQ2 in terms of graphics quality, but not at 2-3 times the frame rate. It still has its performance problems.

Anonymous said...

Good day !.
You re, I guess , perhaps very interested to know how one can manage to receive high yields .
There is no initial capital needed You may start to receive yields with as small sum of money as 20-100 dollars.

AimTrust is what you haven`t ever dreamt of such a chance to become rich
The company incorporates an offshore structure with advanced asset management technologies in production and delivery of pipes for oil and gas.

Its head office is in Panama with affiliates everywhere: In USA, Canada, Cyprus.
Do you want to become a happy investor?
That`s your chance That`s what you really need!

I`m happy and lucky, I started to take up real money with the help of this company,
and I invite you to do the same. It`s all about how to select a proper partner who uses your savings in a right way - that`s AimTrust!.
I take now up to 2G every day, and my first deposit was 1 grand only!
It`s easy to join , just click this link http://vufikylyl.100megsfree5.com/pihylyh.html
and lucky you`re! Let`s take this option together to become rich

Anonymous said...

Hello !.
might , perhaps very interested to know how one can reach 2000 per day of income .
There is no initial capital needed You may start earning with as small sum of money as 20-100 dollars.

AimTrust is what you need
The company represents an offshore structure with advanced asset management technologies in production and delivery of pipes for oil and gas.

It is based in Panama with structures around the world.
Do you want to become a happy investor?
That`s your choice That`s what you desire!

I`m happy and lucky, I started to take up income with the help of this company,
and I invite you to do the same. It`s all about how to choose a correct partner who uses your funds in a right way - that`s it!.
I take now up to 2G every day, and my first investment was 500 dollars only!
It`s easy to start , just click this link http://atyjyneryt.lookseekpages.com/ajinasa.html
and lucky you`re! Let`s take our chance together to become rich

Anonymous said...

Hi there!
I would like to burn a theme at this forum. There is such a thing, called HYIP, or High Yield Investment Program. It reminds of financial piramyde, but in rare cases one may happen to meet a company that really pays up to 2% daily not on invested money, but from real profits.

For several years , I make money with the help of these programs.
I don't have problems with money now, but there are heights that must be conquered . I get now up to 2G a day , and I started with funny 500 bucks.
Right now, I managed to catch a guaranteed variant to make a sharp rise . Turn to my blog to get additional info.

http://theblogmoney.com

Anonymous said...

Hello!
You may probably be very curious to know how one can make real money on investments.
There is no need to invest much at first.
You may commense to get income with a money that usually goes
on daily food, that's 20-100 dollars.
I have been participating in one company's work for several years,
and I'll be glad to let you know my secrets at my blog.

Please visit my pages and send me private message to get the info.

P.S. I make 1000-2000 per day now.

[url=http://theinvestblog.com] Online investment blog[/url]

Ozymandias said...

What you have not said, but I imagine to be true is that even if you get a 5% improvement on a quad-core processor by doing a six-month rewrite. The consequence might be a 10% drop in performance for people with single core machines.

We all should know that a small boost in performance is soon forgotten by the user base. But a drop in performance is never forgiven.