Major Stress Major Stress

[MOD] The Sins Optimization Project version 1.0 (updated 1/25/15)

[MOD] The Sins Optimization Project version 1.0 (updated 1/25/15)

Because all good things must come to an end.

It is difficult to explain the issues of Sins have without going into too much "technical detail" which some may not understand. The main problem is that Sins is an older generation 32 bit game. With a two gigabyte memory limit. This is nothing new. All 32 bit games have a 2 gig memory limit. However Sins is NOT like other 32 bit games. Most games have an ability to load their assets "on the fly", or only as needed. Some games can also unload assets that are not needed. This is how they prevent themselves from reaching the 2 gig limit. Some games have been patched to where they can use up to 4 gigs of ram on 64 bit systems.

Sins also can load only what is needed at game start up, but Sins does NOT unload what it does not use. Any assets that are loaded remain loaded throughout the entire game! If you use all three factions (Six in the case of Rebellion), Then Sins will load ALL of its game assets at start up. Entrenchment, and Diplomacy in some games loaded between 1.7 to 1.9 gigs worth of assets at game start up! Graphic settings, The size of the Map, and the amount of Players, or AI opponents also determine how much ram Sins will use in any given session.

In 2006 when Sins was first conceived 32 bit Windows XP was the standard OS of the time. Single core CPU's were also still the standard. Though dual core CPU's were starting to become mainstream, and 2 gigs was considered to be "plenty" of memory. So most game developers at the time didn't have to worry about hitting that dreaded 2 gig brick wall. As time went on Computers, and Sins itself evolved into their present state which is now Rebellion, and soon to be Windows 10 running in multiple core 64 bit systems. With 16+ gigs of ram. Things have changed so much over the years. However Sins still remained the 32 bit game as it was designed in 2006. That uses only a single core of your CPU. With a two gigabyte hard coded memory limit. Sins WILL CRASH when the game reaches that 2 gig limit.

There is some misunderstanding about the 2 gig crash issue. The dump happens when Sins reaches 2 gigs of Ram Usage. That means the ram that Sins uses by itself. NOT what your entire operating system, and/or other programs you are running combined uses. Only the ram that Sins uses. No matter how powerful your system is. If it is 32, or 64 bit, or if it is Windows XP, Vista, 7, 8, etc. etc. Sins WILL CRASH when it hits its 2 gig ram limit. It is hard coded into the game.

Nobody was aware that this would be a major problem back when the game was first designed, because Sins was designed based on the hardware standards of 2006-2007. Before 64 bit OS's, Quad Core cpu's, and 4+ gig ram systems became the standard that they are now. There is nothing that can be done about it. Except to re-write Sins in 64 bit. Which is out of the question, because it is too cost prohibitive for the developers to do this (Ironclad said so themselves). Sins is a great game, and it is a shame that this is happening. With the Entrenchment, and Diplomacy expansions Ironclad had unintentionally pushed the game over its own hard coded limitations with all of the new added content.

Over the years as of the release of Diplomacy 1.3, and Rebellion. Stardock, and Ironclad did indeed fix all of the major game crashing issues of Sins. Using the fix's from This Project as a guide. However, NONE of these fix's ever made it into Entrenchment, or Original Sins! While the Stardock, and Ironclad developers have fixed most of these issues over years. They can not do anything to change the game itself short of reprogramming Sins over from scratch. The developers can do no more, but there is something that WE can do as modders to prevent Sins from hitting that dreaded 2 gig brick wall, and improve the performance of Sins "just a little bit more". I have taken the initiative, and applied to Sins what i have learned from my Homeworld Modding days. I present to you....

 

The goal of this project is to improve your Vanilla Sins gaming experience, and to eliminate the 2 gig crash issue. No changes will be made to Original Sins, or any of its Expansions in game play at all. No stat changes will be made. Nothing "new" will be added. There are other mods that can do that. Original Sins will remain Original Sins, and the Expansions will remain as they are. What we are doing is taking what game assets that Sins already has, and we are "optimizing" them to improve the performance, stability, and to moderate the ram usage in game. The side benefit will be that some lower end systems can enjoy almost the same gaming experience as the high end users do (Especially Laptop Users).

The project initially started out as an idea to reduce the texture sizes, and poly counts of some models. Not just arbitrary reductions. The goal was to maintain the quality of Sins in game. To make it look as if nothing has been changed at all.

We started with optimizations to the Planet's, and Skybox's, because they were the "worst offenders" as far as ram usage, and performance was concerned. Unfortunately my modeling skills left much to be desired in Softimage XSI at the time, and there were un-fixable mesh errors on the optimized planet meshes. Despite that the "proof of concept" was established. A lot of ram was saved, and there was a very noticeable increase in performance.

Then we studied the Particle files, and Textures. Many of the particle textures in our opinion were way too high in resolution for the purpose that they served. A reduction in texture resolution by HALF showed no visible difference in game at all! There were some exceptions where the particle resolution reduction difference was very noticeable in game, and those textures were reverted back to their original resolutions. Remember the goal was NOT to reduce in game quality. Also all of the particle textures were changed from TGA format to DDS format, because DDS uses a little less memory, and is much more efficient, because DDS format loads directly into video memory. The particles can also utilize the mip mapping features of DDS format. Where TGA format does not. Therefore enhancing in game video performance (increased FPS). As of version 0.07 no TGA format textures load in TSOP. With the exceptions of the Mouse Cursors, and Scenario Pictures. This is due to those textures being controlled by the hard coded exe. When you change the texture format those textures do not show up in game.

The developers mentioned something about a Memory Leak. I heard that a memory leak can be caused by a file searching for something over, and over again if it is not there. Example a mesh file calling for a texture that does not exist. Vanilla Sins had MANY particle, and mesh files that had that issue. We removed all of the entries that pointed to non existent textures in the mesh, and particle files. We also removed the entries that have lines like for example "C;\g\main\datasource\textures\effects\aura4.dds", and just used "aura4.dds". This will default the search to Sins, or the Mods "Textures" folder, and speed things up a bit. Also various typo's were corrected in some mesh, and particle files. Some ships had mesh nulls that were Miss-Labeled. For example the Akkan had nulls, and flairs that were miss-labeled. The engine trails, and flairs didnt show up in game. the Marza was missing its bomb nulls. etc. etc. All of this has been corrected as well. If there is a ships mesh file in the mods "Mesh" folder then it had something wrong with it that was fixed. The Soundata, and Galaxydef files had errors too. Regardless if any of these issues caused any memory leaks, or not these issues MUST be fixed! (and they were as of Diplomacy 1.3, but NOT for Original Sins, or Entrenchment).

After the Meshes, Particles, Textures, and the User Interface were optimized, and the very obvious issues were fixed there was a tremendous boost in game performance. Sins ram usage was greatly reduced. As of TSOP 0.07 I could run a game of Entrenchment, or Diplomacy ALL DAY, and I NEVER used more than 1.5 gigs of ram! This was using Maximum Settings. Including Bloom enabled. Tested on Huge Random Multi Star maps with 9 AI's. The 2 gig crash problem has been solved! Our primary goal with this project has been achieved! Without having to use any external memory increasing programs! Stardock, and Ironclad took notice, and they used TSOP as a guide in the Diplomacy 1.3 patch, and for developing Rebellion.

Now that the Stardock, and Ironclad have fixed the issues of the game as of Diplomacy 1.3, and Rebellion. TSOP has been focusing more on Original Sins, and Entrenchment. Since the Ironclad/Stardock fixes never made it into Original Sins, or Entrenchment. However, That is not stopping us from optimizing Diplomacy, or Rebellion "just a little bit more" ;)

You can also use Large Address Aware to give Sins an extra gig to play with, but in vanilla Original Sins, and Entrenchment the Endless Search Loop issues that were never fixed will cause the game to eventually go over that extra gig (Run vanilla Entrenchment, and watch your ram usage keep going up, and up, and up). TSOP is a solution that will stop that issue.

____________________________________________________________________________________________________________________

Downloads

The Sins Optimization Project v1.0

For Original Sins 1.195

Sins Optimization Project 1.0 for Original Sins


For Entrenchment 1.055

Sins Optimization Project 1.0 for Entrenchment


For Diplomacy 1.37

Stardock, and Ironclad fixed all of the issues that compelled us to make TSOP. Therefore there is no TSOP for Diplomacy


For Rebellion 1.82

As with Diplomacy, Stardock, and Ironclad fixed all of the issues that TSOP pointed out, and then some in the case of Rebellion. Sins: Rebellion is as optimized as any Sins game is going to get.



The Sins Optimization Project for Rebellion

The Sins Optimization Project for Rebellion

NOTE: TSOP 0.07 is still hosted on ModDB, and it still works in the current versions of Trinity (OS, E, and D), but you will be playing as if Sins was patched to an older version of the game. Plus some Icons and Buttons wont be visible.


Changes in TSOP 1.0

TSOP 1.0 updated to the latest versions of Original Sins, and Entrenchment.

The User Interface is completely overhauled to use pure DDS format.

Trade Ships,  Strike Craft, Mines, and Constructor Unit Meshes, and Textures are Optimized.

 Starbase Textures  are optimized mimicking the Stardock Rebellion Optimizations.

Sounds, and Music are optimized.

All TGA format Textures are replaced with DDS format, and TSOP "force" loads them. The only TGA textures that remain are the Mouse Cursers, and Scenario Pictures. This is due to hard code in the EXE.

 "Strategic Texture Reductions" for the Particles return using newer methods to create a better looking texture.

(Original Sins, and Entrenchment)  Many Mesh Null issues, and various File Errors were corrected, and Unnecessary Text Entry's were removed in both the Mesh, and Particle files. Entry's pointing to Non Existent Textures in both the mesh, and particle files removed. See the various "Fixed Files" texts in the mods Read Me for details.

(Original Sins, and Entrenchment)  The Diplomacy 1.3  graphic fix's have been incorporated into the Original Sins, and Entrenchment versions of TSOP (Pipeline Effects, "White Line" error fix, and Corrected Mesh files).

 TSOP 1.0 for Trinity will see the return of Optimized Planets using the planet meshes from Rebellion! but with our Optimized Original Sins Textures!

There will be no version of TSOP made for Diplomacy, or Rebellion, because there is no need for it. Only Original Sins, and Entrenchment need TSOP, because the 2 gig issue, and various other issues were never fixed in those two versions of the game.

This is the FINAL version of TSOP that i will make. If the community wants to take over the project they are more than welcome to.

 

Community input, and contributions are more than welcome, and they are encouraged! This is a Community Project! All can pitch in!

Note: TSOP does not fix the known issue of "Late Game Lag". It does help prolong the lag until further into the game in some cases, but sooner, or later the lag will happen. The late game lag issue is a CPU issue due to the fact that Sins (including Rebellion) is optimized to use only single core processors. Therefore Sins will use only one core in multi core CPU's. To help with the late game lag issue we recommend the following:

Playing on smaller maps.

Using fewer AI opponents.

Disabling the Trade Ship icons (this actually helps out a lot)

Stacking the icons in the empire tree (same as with disabling trade ship icons the less icons rendered the better).

Using a Strike Craft Reduction mod (The many strike craft in the game are a big CPU hog. There are some good strike craft mods that reduce the number, but maintain the balance. Search for them).

You do not need to download all versions of TSOP. You only need to download the version of TSOP for the specific version of Sins you wish to play. The Retail version of Trinity is just all three versions of Sins bundled into one package. Installation for Trinity versions of Sins are no different than if you purchased each version of Sins separately.

_____________________________________________________________________________________________________________________

Other mods are more than welcome to use TSOP. No questions asked! In fact we encourage it! Learn from it! Use it as guide, and an example! Merge it into your projects!

TSOP is intended to be a Stand Alone mod. Run with no other mods enabled. There are issues enabling TSOP with other mods. Enabling with other "Graphic Enhancement" mods is NOT recommended, because the graphic enhancement mod will defeat the entire purpose of TSOP. All of our optimized files will be overridden by that enhancement mods "enhanced" files. Increasing your ram usage instead of decreasing it. Some enhancement mods like Distant Stars are incorporating elements of TSOP into their future updates.

Read the included READ ME text's for installation instructions.

2,232,671 views 1,134 replies
Reply #101 Top

hi, i was forced against my will to actually set up a forum account just to get on this thread and say "look man, i really want to play with your mod cause it might single handly allow me to actually finish a game but i cant because your shit link to filef$ggot randomly stops the download due to ridiculous errors".

 

on a side note how big is this file suppose to be, hope it wasn't 4 mgs like the first download attempt or 12 mgs like the second or 7 mgs like the third.

 

please get that file somewhere reliable, try fileplanet or something.

Reply #102 Top

I feel your pain. The last 2 host i used were much worse, and to add insult to injury the downloads were corrupt. Filefront, and rapidcrap. Fileplanet will be more of the same crap, and they will force you to make an account just to download. Believe me if there was something more reliable out there that was free it would be there.

Reply #103 Top

Ask Danman he might be able to support you with that, think he hosts downloads for all the mods the community does over at the forum there... though I haven't been there in a while so I don't know what's going on... unless someone from there can confirm with what I am saying

Reply #104 Top

Mediafire is pretty good with instant downloading.  Only issues are a lot of ads, and a 100MB limit on filesize.

Reply #105 Top

my frustration with mediafire is that if you forget the password, then you CAN NOT LOG IN AGAIN nor reset the password. BLOODY frustrating harpo

Reply #106 Top

Quoting SpardaSon21, reply 104
Mediafire is pretty good with instant downloading.  Only issues are a lot of ads, and a 100MB limit on filesize.

That part is why a lot of mod makers don't use that place... lol notice the enlarged bold letters? THAT is why xD

Reply #107 Top

No pressurebut do you have a rough idea when the last update here will be ready?

Reply #108 Top

Quoting Major, reply 102
I feel your pain. The last 2 host i used were much worse, and to add insult to injury the downloads were corrupt. Filefront, and rapidcrap. Fileplanet will be more of the same crap, and they will force you to make an account just to download. Believe me if there was something more reliable out there that was free it would be there.

Stress I would recomment A DRIVE ,  We used that for Soa2 0.4E series and I know that dolynick and a few other modders use this service as well.  It is free and rather fast. I also believe I set a account up for the Soa2 modteam , I will check and confirm and send you the account login  info .  Hope this helps

thanks jtaylorpcs

Reply #109 Top

Quoting Devilize, reply 101
i really want to play with your mod ... but i cant because your shit link to filef$ggot randomly stops the download due to ridiculous errors...

...on a side note how big is this file suppose to be..

please get that file somewhere reliable, try fileplanet or something.

Ok, i have host the file on my own server... will remain there until i need the room or if Major Stress ask me to remove it...

Uncompressed, the mod take 296mb... the Major Stress rar archive 132.4mb ... since these mod is related to optimization, i have use 7ZIP for compress it... result is a archive who is only 37.9mb...

LINK : http://www.love-from-russia.be/Sins_Optimization_Project_0.02.7z

EDIT: i have download and test it, the archive is good... and sorry but the server max speed is only around the 3mb/s...

Reply #110 Top

If I want to play another mod with this activated, in what order do I stack them?

 

Nevermind, I found a helpful post.

Reply #111 Top

Majos Stress, you really should get a job as Sins 2 developer, or tester, or whatever. What you've been doing for the past years is way beyond what some pro devs do for money.

Reply #112 Top

One thing to note here. While your optimization project is a noble goal, you can eliminate nearly all Sins ram cap crashes by simply making the executable large address aware. This will boost the cap from two gigs of ram to nearly three. If you have 4+ gigs of ram and not a ton of crap running in the background Sins should be able to stay under 3gigs easily and give you no problems.

You'd be surprised how many games hit the 2gig limit and crash, and they've been around for years. I remember Supreme Commander, released even back in January 2007, would easily hit 2gigs usage. Simply goggle for a 3gig enabler that adds a large address aware tag, I've seen them for oblivion, stalker, supcom, etc. It shouldn't even matter which one you pick, as long as you have the exe named right it will add the tag. Note that 32bit xp users and the like will need to do a few more tinkers to ensure the large address aware tag is used.

This project will indeed make Sins run faster though, so go for it.

Reply #113 Top

The LARGEADDRESSAWARE tag really only works for 64-bit systems with more than 4 gigs of RAM.  32-bit systems like XP won't see much benefit from it.

Reply #114 Top

What you say is true, but even 32bit systems with 3-4gigs of ram gain some headroom as long as the pc isn't bloated. 32bit systems will have to set windows virutal address space higher as well as adding the tag.

Reply #115 Top

deleted because off-topic

 

Reply #116 Top

How much ram your OS can, or cannot use is totally irrelevent here. It doesnt matter if you can make XP see 4 gigs of ram if Sins crashes when it hits 2 gigs anyway. The 2 gig problem is a Sins problem not an OS problem. Making your OS see the 4 gigs is a topic for another discussion.

32 bit games in general have a 2 gig ram limit. Sins stands out because the game loads most if not all of its assets at startup, and dumps when it reaches 2 gigs of ram usage. I am talking about how much ram Sins itself uses. Not Sins plus what else your OS is running. So Sins problem is a universal problem, and it doesnt matter if you use a 32 bit, or 64 bit OS, or how much ram you have. Sins will still dump when it reaches the 2 gig brick wall. THAT is why this project exists.

Please take OS debates to another topic.

0.03 will be uploaded later today with fighter, and trade ship tweaks. Plus another tweak which i am probably grasping at straws with, but hey its something, and every little bit helps.

Inside the mesh and particle files you will see the entrys for which texture is used for a particular mesh or particle. Example: texturename " C:\gs\main\DataSource\Effects\shockwave4.tga". Why all of that text? when it can simply be texturename "shockwave4.dds". Correct me if i am wrong, but the less code the game reads the faster it will perform right? You guys are the code monkeys not me. Like i said i may be just reaching with this, but there is a LOT of this extra text in the particle and mesh files. We will see.

Reply #117 Top

I'm not sure you're understanding here Major. Yes, 95% of games, or any application, will crash at 2gigs of addressed memory. This happens much, MUCH more often than people realize. But, as I said, a large address aware tag is easy to add and the new 3gig addressed memory limit on the application pretty much completely eliminates these sort of crashes.

The previous discussion wasn't about how much ram you can get your OS to use. That's simple. A 32bit OS will recognize up to 3.5gigs of total ram. A 64bit OS can recognize, well, more ram than you have. The only situation where OS comes into play with the large address aware tag is that 32bit OS's have a 2gig ram page limit by default, although this can be set higher, which one will need to do if using a large address aware tag on a 32bit os with 4gigs of ram. Note that it will not let the os recognize more than 3.5gigs of your ram, but it will make more available to applications, especially with a lightweight OS and a noncluttered system.

So no, you're wrong. How much ram your OS can recognize, how much ram your OS can make available to applications, and how much ram a single application can use (2gig -> 3gig), are ALL relevant.

The only reason OS's came up is because it is just about the only thing to talk about. Adding the tag takes 1 second and works for everybody without further tinkering UNLESS they're running 32bit xp with 4 gigs of ram. Perhaps you should try it yourself Major Stress, you will never crash at the addressed ram limit again.

Reply #118 Top

So, for the sake of argument, let's say this large address tag is a viable option.  I currently run 32-bit Vista on my laptop.  What you're saying is that I can find this special program that actual tells the Sins application it can use up to 3 gigs of RAM instead of the 2 gigs that's a hardcoded limit, right?  If it's feasible, how would I go about implementing this (just the basics)?

 

Reply #119 Top

I assume you have 32bit vista with 4gigs of ram Mystic.

 

Simply follow the instructions detailed here:

http://www.tesnexus.com/downloads/file.php?id=23254

The same little program will work for Sins, all it does is add a large address aware tag to anything named oblivion.exe, just change the name of your sins exe, follow instructions and run it, then rename it back to what the sins exe was originally named. It's really, REALLY easy.

Directions:

First you need to tell XP/Vista that it can use more than 2GB of RAM for applications.
Manually edit your boot.ini file by adding " /3GB /Userva=2900" to your boot.ini (no quotes).
Your boot.ini is normally read only and you will need to change that temporarily while you edit it.
To do this, right-click the file and untick "Read Only". After editing the file, restore the Read-only status.
To Edit it manually you can find it on the root of your system drive or
go to system properties/advanced tab, settings under 'startup and recovery' section, then hit edit.

Mine looks like: multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP Professional" /FASTDETECT /NOEXECUTE=OPTIN /3GB /Userva=2900

Yours maybe different, the only thing you should change is add " /3GB /Userva=2900".

So, change that in vista, run the little program, and you're good to go.

Not however, that your 32bit vista will only recognize 3.5gig of your ram regardless, so try to eliminate any background programs to free up as much ram as you can. If you have 4gig of ram I'd suggest you just switch to 64bit vista, it's on the same disc. Although, if you can try to get your hands on 64bit windows 7, it's a lot "slimmer" than vista and you can use all the ram you can get.

 

 

Reply #120 Top

Thanks!  I'll look into it when i get a chance.

Reply #121 Top

I understand what you are trying to do Luftin, and i appreciate it. However not everyone here is as computer savvy as you. Many cant even open a simple Rar file, or are even literate enough to read any mods Read Me. Nor can they follow simple directions if you look at the past "how do i do this" posts on these boards :/

So we did this project to avoid anyone having to risk altering their computer, or having to download special software to "force" sins to use more than 2 gigs (which i thought it was hardcoded to dump at 2 gigs by the dev's). If your trick works (if people use it correctly) then great. The people with high end machines wont need to use this project, and can play sins at max settings without worry.

There are still some people out there (like me) that cant afford a new windows 7, 6 core, 6 gig ram system with gods own GPU just yet, and have to make due with what they still have. Those are the ones that will benefit from this project the most.

Reply #122 Top

I'm not trying to demean your mod project at all. Improving performance is always a great thing, and there are very real un-optimized problems such as strikecraft.

I had noticed that your initial mod blurb put very heavy emphasis on the 2gig limit, so I posted a very simple workaround that essentially eliminates the 2gig limit crash.

The only people who would not benefit from this fix are those that have only 2gigs of ram, and their solution is to skip lunch and put the $10 into doubling their system's ram.

Get rid of background processes and this is a very viable fix for even 32bit systems with 4gigs of ram.

Reply #123 Top

@Luften The fix IS useful because it fixes the crash, when reaching the 2GB.

BUT it does NOT let the game use more than 2GB of RAM.

http://img39.imageshack.us/img39/4491/ramwx.png

@Major Stress Yes it is hardcoded and there is nothing we can do about it, thats the reason why we need your project.

Great thanks for your work so far. It allowed me to get some serious action going on without reaching 2GB. Keep it up!

Reply #124 Top

The app here location can do the large file bit twiddle to any executable (and it even backs up the executable before modifying it).

Reply #125 Top

Update taking longer than expected, because i found a new, and interesting issue...Typos in the mesh files. (re-converted original sins meshes to confirm this, and make sure it wasnt a fluke). I noticed after the betas some of the tech ships wre missing their Flairs (nav lights), and such. I thought ironclad just removed them. I found the null points are still there in the mesh files, but are named wrong. I have corrected the names, and now the flairs work again, and the misslabled weapon null on the tech battleship which is just an extra null that shouldnt be there. so i deleted it, and the bomb point on the vasari scout capital labled "planetsiege" when it should have been labled "Bomb", and added a Bomb point to Marza which it was MISSING.  I will go over all of the mesh files, correct any error i find, and include them in the next release.

Why hasnt anyone noticed this stuff before??