Maximum threads per process – where is the bottleneck??

Recently I was doing an OPC compliance test on one of the IO severs that we developed.
There is a tool for this compliance test from OPCFoundation.org. By default tool tries to test using 10 servers, 10 groups per server and 10 items per group kind of configuration.

But the test case asked for compliance with 100 servers each with 10 groups or 10 servers each with 100 groups. Though the first case doesn’t appeal much, second case definitely does. But this IO server failed to cope up with that kind of load and started crashing after adding around 70 odd servers. Kind soul who logged the bug actually gave me all scripts so that I could recreate the condition at ease.

Suggestions came from many circles to check for resource usage. Isn’t that obvious!!!
A process which is so excessively hungry dies while eating itself. So the idea led to capture memory and thread consumption. All that I could observe is 1980 odd threads and sad demise. So it was not difficult guess to think of magical limit of 2048 as limit on number of threads per process.

Now I started hitting google to substantiate the limit I observed. Google gave me a page from Microsoft site. So here is what Microsoft got to explain about the limit.

Windows operating System allocates one mega bytes of memory as default stack size per thread. So in this case 2048 threads swallowed 2 Gigabytes of memory. Unfortunately that is the limit on user space virtual memory allocation per process. As 4 GB is max per process 2GB goes on user side and remaining 2 GB goes to kernel side.

Isn’t an MB of memory per thread too much!!!!!!

Posix threads can address this situation with stacksize member of pthread_attr_t. But there is catch here too, if you don’t specify this parameter, pthread library tries to assign it with operating system’s default. And Another member of pthread_attr_t is guardsize which has a default value of size of a page (4KB).