AI (Advent Rebel + TEC Loyalist + Vasari Rebel) and their choice of starter capital ship

If someone who remembers statistics 101 better than me would be so kind to test if these observations differ from chance (assuming uniform distribution), that would be nice to know.

Well we'd expect 16.5 for each entry in that field if it was uniform. So if I'm remembering this correctly, our good friend the chi squared test says that only the Halcyon and the Rapture had values that were sufficiently different to not likely be due to random chance at a p-value of 0.05.

If I'm interpreting my own math right, only the fortifier AI seemed to show any sort of statistically significant bias towards what capitalship it picked, probably because of those 27 Halcyon. If I were to guess this is likely just an outlier.

Link to my math.

well this is the reason I asked people here as I am no statistician either But those high and low numbers indicated to me there might be something different than simply chance.

I chose 99 test samples because it is sufficiently enough to see at least smaller trends. Of course these results may be totally affected and only by chances but as the person, I think W(insertrandomnumbershere) said somewhere he thought AI prioritizes colony capital ships.

So additional information that might help solving this issue:

1. capital ships having highest number here for one type of AI were almost built in every game of that type, and several times not just one was built.

2. Lowest numbers, in like half of those 11 games the AI's did not even build one of those, in the other games 1, rarely 2

3. There were some spikes, but only with capital ships already having higher numbers. Aggressor had 5 Raptures in one game, Fortifier had 4, 4, and 5 Halcyons in 3 games, Researcher had no spike, Economist built 4,4,4  Radiance, Discord, Revelation in 3 different games.

I will try to find out how to upload a picture here so you may see these numbers as well, I have these numbers in an XLS file but I think a simple screenshot of it would be the easiest.

I did another round, now with TEC Loyalist, to see if they have anything interesting, well, they had.

The results are in the updated OP.

To me it always seemed like the AI picked their capships based on a random number generator. It never really seemed like the AI actually tried to counter enemy fleet composition using a specific choice of capital ships. This is more or less confirmed by your tests (and Goa's spreadsheet).

While it is true in most cases further results indicate it's not entirely true for all cases. I think reality is more like what each AI DOES NOT BUILD too often (have significantly lower weight), while the other 5 capital ships all have the same weight.

Vasari Rebel tests are finished, updating OP now.

To be honest I don't think the other factions would have a very different outcome than this, but I will make these tests for Advent Loyalist to see if I'm right.

I guess everyone in this thread failed their AP Stats class?

Yeah, they may have weight values to determine build chance, similar to how the fleet beacon abilities can.

By all accounts this is what appears to be happening. The TEC and Vasari data are even less varied than the Advent one. Thus we can not disprove that the AI is picking capitalships at random.

The only interesting thing about this as a modder is that the AI does have issues if you do not give it a colony capitalship. Which is an odd restriction if the AI is only picking at random, but these numbers definitely don't support the AI favoring them.

I think they only have negative ones based on these numbers, so for some types one or two capital ships are not favoured, and the remaining have the same value.

TBH I would expect every capital ship to be built on average 1 or two times in each game if every capital ship had random chances (of course reaching this tendency after several games), but for the low numbers it was not like that. In many games there were none, remaining games only one, and 1-2 games when 2 or 3. Highest numbers had strange rows as well.

The Vasari Rebel Aggressive one had this:

in consecutive 11 games how many Skirantras were built by the 9 AI's: 0,1,0,3,0,0,0,1,1,1,0

Radiance by fortifier AR: 0,2,0,1,1,1,0,0,1,1,1

Rapture by Economist AR: 0,1,2,1,0,0,1,1,1,1,0

Sova by researcher TL: 1,0,0,2,1,0,3,1,0,0,1

Marza by economist TL: 0,1,0,0,1,0,1,2,3,1,0

These are not really the 1212121212 sequence I would expect if all ships had equal initial chances. 1 player has 16,7% chance to get any capital ship, leading to 222111 in a (perfect) game, so if we think along this line of thoughts each capital ship should have a 11222121211212 line, and the lack of 2's in the lines above is more than suspicious to me, and the presence of several high numbers in the rows below as well.

There were other strange things, look:

Fortifier TL Kol 2,1,2,3,1,2,4,2,3,1,0

Economist TL Corsev 4,4,6,0,2,2,0,0,1,2,3

Economist VR Antorak 3,3,2,2,2,2,2,5,0,0,2

Fortifier VR Vulkoras 1,1,2,0,4,1,4,2,1,2,2

Aggressive AR Rapture 2,2,3,1,3,1,3,2,3,2,2

Fortifier AR Halcyon 2,1,4,4,0,1,2,2,3,3,5

Such a wide variety of numbers, not even near the 12121212 line someone would expect.

Random number generation and statistical testing thereof isn't as simple as that. Even GoaFan77's testing, which has the basics right, is a bit wrong in that it doesn't do multiple test correction. You can get seemingly positive results by just doing a lot of statistical tests. I'll elaborate on this once I find more time...

In the mean time, you could read the Wikipedia article on Pearson's chi-squared test  for the basic test that GoaFan did and Familywise error rate (for the old school correction method[s]; the most widely used of which is the Bonferroni) as well as False discovery rate for the post-1990 methods (the most widely used of which is the Benjamini–Hochberg–Yekutieli). Those wiki articles tend to be a bit impenetrable if you know nothing about the topic beforehand (because they usually lack examples, which alas is done on purpose on most Wikipedia articles per WP:NOTTEXTBOOK), so a statistics textbook might be a better choice in that case. I don't suppose you have R installed [it's free], but if you do [install it], you can find examples how to do those [latter] tests in this book for example (pp. 109-111)

From my AI observations, the only thing the AI "sees" whenever it builds a ship or a module is the RoleType. If you give the AI two frigates that both only have one frigateRoleType of Light, it will not be able to differentiate between the two, and will build (roughly) equal amounts of both as a result. Transporting this logic to capital ships, all capital ships except for the colony ones have a roleType of Invalid, so there should be no reason why the AI would be able to differentiate between those capital ships. Go ahead, try changing the stats of a particular capital ship drastically and see if you see similarly drastic changes in distribution. I didn't, and I tried everything from health amount to strike craft count to damage (I paid special notice to strike craft count, since someone, I think Lavo, reported seeing the AI prefer ships with strike craft over ones without in one of his mods).

Colony capital ships are where things might get interesting. Since their roleType is COLONY, which is different from the other capital ships' roleType, it is possible that the AI can differentiate it from the others. The odd thing is that unlike the other colony capital ships, the Jarrasul doesn't have a COLONY roleType in unmodded Sins, it has a roleType of Invalid. However, I'm not seeing any significant variation between the Jarrasul's stats compared to Akkan's and Progenitor's stats (you doing additional tests now armed with this information might result in confirmation bias poisoning your conclusions, possibly even your method), so if the AI can differentiate between the two, it likely still has no effect on the first capital ship built.

Added note on random distribution: you must take into account the possibility of a non-ideal scenario taking place even when tests are repeated ad infinitum. According to my calculations, the possibility of a non-ideal distribution at countably infinite runs is ((n - 1) / n)^n, limit n to total number of states; for this system, it will be (5/6)^6, or about 33.49% (when number of states approaches infinite, the probability approaches 1/e, or about 36.79%). Note that this does not indicate what the non-ideal distribution might be, only that it will be non-ideal. Caveat: I can mathematically prove this fact for systems where all probabilities are rational numbers, but I cannot when at least one probability is an irrational (or complex) number. Since I haven't found any proofs online, I cannot be confident that this theorem holds true for any system that has at least one probability whose value is irrational or complex.

Except the basic rules about random numbers say that you should not expect 12121212, even when the probability of selection of both 1 and 2 are the same. Just like flipping a coin 100 times, you'd fairly often get clusters where you get 4 or 5 heads or tails in a row, but in the long run they cancel each other out to a 1:1 ratio, which is more or less what your tests are showing.

Yeah I'm sure we could drag our statistics books out and do all sorts of things to this data, but even the basics mostly suggest that this is probably a random distribution. Most of the rest is to further ensure that your data is not due to random chance, I don't know of anything that could make this MORE likely to be a non-random data set.

We could always ask a dev.

What happens if the AI doesn't have a colony cap?

First of all I don't really know anything about statistics, I just use my common sense here. From the Skirantra line my guess would be that from the next game 0 or 1 Skirantras will be built, and not the supposed one or two.

Uhm, thanks I guess, but these things are horrible even in Hungarian LOL, once our teachers in university tried to teach us these things, I can tell you without any major success

Yes I am aware of something like this that's why I chose a higher number of test samples (99 cases from 11 games for each AI type), and my interpretation of this randon generation dictates that there should be a tendency after so many games that reaches the trend of 121212. And this is not the case in many capital ships, instead they have high fluctuations.

So can the line of Skirantra still be considered acceptable even though it clearly has a different trend with different numbers? Can someone tell me if the Skirantra has the same chance as the other capital ships than how can you make me understand it from the results of it? (if possible without highlevel statistics)

Looks like the Invalid and Colony roletypes has nothing to do here, if there is some weight here it is clearly based on other thing(s).

End game statistics can differentiate capital ships, what is this based upon do you know? Cause I don't.

With a great number of test cases the total proportion should reflect the actual percentage of selection yes. However, you are wrong about the high fluctuations. Thanks you your high sample size, the chi-squared test I did shows almost all of your tests have a p-value of greater than 0.05 (I didn't show this in the spreadsheet but you use the chi-squared number to look it up in a fancy table). The default point for rejecting a theory (that the AI randomly selects capitalships) requires a value less than that.

The biggest fluctuation is actually the Halcyon with that crazy 27 that the fortifier AI built, which eyeballing it looks like 2-3 standard deviations above the mean of 16.5. This high number caused both the p-value for both the halcyon overall and the fortifier AI to be statistically significant. However, in statistics you have to watch out for outliers as occasionally really unlikely things happens with random numbers. That p-value of 0.05 actually means that there is only a 5% chance that the results are due to chance alone, so I would guess that was one of those cases where chance produced a truly unexpected result. Especially as the other carriers did not get similar bias.

There are two types of roles for ships in Sins. One is the role type Delnar mentions, which the AI uses in game to know for example what ships can colonize or capture extractors or are heavy cruisers. Then there is the stat type, which is used for the end game statistics and does not appear to be used by the AI very much if at all. Having some key role types missing can cause the game to crash but you can give everything the same stat type and be fine I think.

In unmodded Sins, the Jarrasul (Vasari colony cap) has the same roleType as all other caps ("Invalid"), while the other two races' colony caps have a roleType of "COLONY". Unless the AI deep-analyzes abilities (meaning it would have to identify colonize abilities that ships might have before it even builds those ships, something highly unlikely given its reliance on the Colony, ResourceCapturer, AntiMine, and Scout frigateRoleTypes instead of simply identifying frigates with those types of abilities), the Vasari AI in unmodded Sins effectively plays like an AI without a colony cap.

Hmm, nem gondoltam volna, hogy találok itt egy másik magyart. Mindenesetre a lényeg az, hogy annak ellenére, hogy meglehetősen nagy a variáció a számok közt, valójában statisztikailag teljesen lehetségesek véletlen eloszlásra. Több ezer, esetleg több száz ezer, próbát kéne tenni, hogy kisimuljon, így kb. 100 próbával teljesen elképzelhető az egyenlőtlen eloszlás.

Vigyázat a "confirmation bias"-re (nem tudom a pontos magyar nevét, Google Translate megerősítési torzítást dob ki): mivel egyenlőtlen eloszlást keresel, torzítottan látod a saját adataidat. Az egy darab fura eloszlás mellett ne feletkezz el a maradék 17 eloszlásról, ami mind teljesen elfogadható. Ha további próbák után is szokatlan eredményeket kapsz, elkezdeném vizsgálni, hogy pontosan mi alapján különböztet meg két hajót az AI (fregattokra és épületekre koncentráló vizsgálásaim arra mutatnak, hogy csak roleType alapján különböztet).

statCountType alapján működik az, mint ahogy a fregattoknál és épületeknél. AI vizsgálásaim alapján elég korán megállapítottam, hogy teljesen ignorálja a statCountType-ot (össze-vissza változtattam őket, de AI viselkedése nem változott akkor sem, amikor pl. felderítőknek nehéz fregatt statCountType-ot adtam).

For everyone else, sorry about the Hungarian post; I know it's impolite, but in case of misunderstandings or debates, I find it's often better to talk in a person's mother tongue to avoid mistranslations and misinterpretations. I'm fortunate enough to have two mother tongues, so you needn't worry about me. Since Hungarian machine translations are so poor because of the language's agglutinative nature, I'll roughly summarize: first paragraph states that though the distribution is uneven, it's completely viable considering the small sample size, second paragraph states to watch out for confirmation bias and remember the 17 regular distributions when considering the one odd one, third paragraph states that post-game stats work with the statCountType parameter, which seems to be completely ignored by the AI in-game.

"If you get strange results after more tests, I would start to examine how AI differentiates those ships"

So what do you suggest here? Should I make some more tests with the AI types that had those strange low values to see if this trend stands in further games and not just results of randomness? How many tests would confirm these are not accidents btw?

And btw thanks for the statistical things, were easy to understand not like the Wiki and other things.

Én valahol láttam valamit ami arra utalt hogy magyar vagy, de elfelejtettem rövid idő alatt, van már egy szlovák és egy magyar is akit ismerek ezen a fórumon a sok angol nyelvű meg random nép fia között.

Szerintem maradjunk itt az angolnál, hogy a többiek is értsék. [Let's speak English so others will understand.]

To test whether or not an AI can "see" a certain parameter, I usually record a few "control" cases where nothing is changed, then run a few test cases where I either change a parameter drastically (eg. eliminate it entirely or multiply it by 10) or swap the parameter values of two different ships (eg. swapping the weapons of a utility frigate and a long-range frigate). If the AI notices, its build pattern will shift accordingly (its average total build amount on the same preset, faction, and map at the same time mark with no pirates or random events will differ noticeably; the map needs to be both perfectly symmetrical and fair).

For capital ships, my suggestion would be to do the same: pick a parameter, change it in a large way, and see if the AI's behavior changes compared to the control case (this can be either the theoretical even distribution or the distributions you found through a hundred or so runs, just remember to give more leeway when comparing stats derived from smaller amounts of runs). Remember to do this in a loaded mod though: it's better to copy over the game files to a mod folder and make modifications there than to modify the base game files, possibly mess something up, and be forced to reinstall the game as a result. Through process of elimination, you will find all stats that the AI will pick up on when choosing what ships to build. If the AI does not differentiate between any stats, then you've deduced through process of elimination that the first capital ship the AI builds is completely random.

I already put quite a lot of time behind something similar back in January when looking into how I could alter AI fleet composition. I eventually found that the AI relies on only one parameter type: frigateRoleType for frigates, roleType for strike craft, and planetModuleRoleType for buildings (there weren't enough allowed roleType values to properly test capital ships). Everything I did afterwards further confirmed this: I've successfully crippled AIs by making them have no ships with the Scout frigateRoleType, made them produce carriers with bombers instead of antimodule frigates by slapping an AntiModule frigateRoleType onto the carrier (frigates can thankfully have more than one frigateRoleType), made them build and use siege frigates instead of LF's by swapping their frigateRoleTypes, and even cut down on their turrets by changing its planetModuleRoleType to that of a repair platform.

So you suggest modding things about capital ships. Unfortunately I am not a great modder, my skill level in this only reaches to be able to change planet stats and maps and remove certain things from game. I do not even know anything about ships, structures, abilities and the rest.