Why did an application get 30 times slower start-up time when we moved it from XenApp on 2008R2 to 2012R2?
I was aware of the disadvantage of having a file share on a server with another OS version than the client OS. This is due to SMB negotiating to the version of the oldest OS involved, ending up with a slower SMB transfers that you would get that if both machines were on same OS. But I did not realize that the huge impact of SMB negotiation overhead if you open and close a file multiple times sequentially. Like you do when working with INI files.
First, INI files is something that applications should not use anymore, but I still see a lot of legacy apps using INI files for configuration and user settings. In this case, the application stored all user settings in an INI file on a file share. This is not a good idea, but that’s how the application was designed and the INI path was hardcoded. Running the application on windows 2008R2, the application startup time was less than 10 seconds. Moving it to 2012R2 it was suddenly 4 minutes. Using process monitor we could see that the application was opening and closing the ini file thousands of times. The same happening on 2008R2, but faster. The INI file contained 3000 settings!
Opening the file in a text editors takes just a few milliseconds. Why does it take so much time to open an INI file in the application? The answer is how INI files works. Every time you read or write to an INI file, the file is opened by the GetPrivateProfileString API. So instead of reading out all settings in the file in one operation, the file is opened and closed 3000 times. By why is this slower with 2012R2? I created a small tool to learn about this. The tool will create a 3000 line INI file on a file share, then read 1 by one setting out and take the time to complete the operation. The tool showed the same result as the application. But in the test tool I could change the INI path to a 2012R2 file server, now forcing us to use SMB3.1 instead of SMB2.0. Now the time it took to read through the INI file was the same as with 2008R2 in both client and server. So the difference is not the SMB version being used, but that the SMB version is being negotiated. It appears to me that the SMB negotiation process happens every time a file is opened.
Reading about how SMB is negotiating when you open a file it becomes clearer, SMB negotiation generates an overload for each INI setting that is read out.
To verify my theory I ran the small tool across all SMB versions and here is my result:
On the top is SMB 3.02 to SMB 2.1. Why? Probably because of SMB3 secure negotiation has more overhead http://blogs.msdn.com/b/openspecification/archive/2012/06/28/smb3-secure-dialect-negotiation.aspx
The conclusion is; INI files on a file share is generally a bad idea, but if you have to do it, make sure the client OS and server OS is on the same version.
Generally I would recommend to always have the same OS level on file servers and clients.
Later we also discovered that the 2012R2 server also was using E1000 NIC in VMWare. The general network performance was extremely bad with this setting until we turned of Receive side scaling on the NIC’s. With VMXnet adapter this was enabled by default. This was increasing the general network speed, but still there was a huge difference with SMB negotiantions involved.
If you want to test for yourself, here is a small vb script that will simulate x number of “open file” operations over SMB. Try first with local disk and then towards file shares on different Windows versions.
Set fso = CreateObject("Scripting.FileSystemObject") filename="\\fileserver\c$\test.txt" operations=10000 Set objFile = fso.CreateTextFile(filename,True) objFile.Write "test" & vbCrLf objFile.Close starttime=now for i = 0 to operations Set listFile = fso.OpenTextFile(filename) listFile.ReadLine() listfile.close() next stoptime=now wscript.echo operations & " operations took " & DateDiff("s", starttime, stoptime) & " seconds"