Strange case of mixed 32 bit and 64 bit installers in single Suite installer

March 20, 2017

TL;DR; Do not mix 32 bit msi with 64 bit msi installers in single Suite installer. 

Suite installer is an installer that allows orchestrating multiple msi installers into single user experience. A customer reported that our Suite installer fails randomly when run from Chef scripts in AWS but is fine when run from GUI. To make things more complex, the installation would fail only around 50% of the times. Of course we could not get hold of Cloud Formation templates the customer used and we could not access the customer environment because it ran in AWS Gov Cloud and none of us is US citizen.

windows_package 'AppAndPrerequisiteSuite' do
 source 'AppAndPrerequisiteSuite.exe'
 installer_type :custom
 options '/debuglog c:/consolidatedinstaller.log'
end

First thing we asked was to enable individual installers logging. (https://support.microsoft.com/en-us/help/223300/how-to-enable-windows-installer-logging)

This yielded :

Error 1935.An error occurred during the installation of assembly component {6F6751F2-43FB-4C77-9791-B24EB154D406}. HRESULT: 0x800703FA. assembly interface: IAssemblyCacheItem, function: Commit, assembly name: Company.Systems.Services,processorArchitecture=”amd64″,publicKeyToken=”45d8b3bf5a7dd114″,type=”win32″,version=”1.0.0.73″

Hmm. HRESULT: 0x800703FA? A quick Internet search says that it is “Illegal operation attempted on a registry key that has been marked for deletion”. But which key has been deleted? We are only installing at this stage, not uninstalling, right? We tried to set up procmon.exe to capture all DeleteRegKey operations but failed because Chef keeps track of processes it runs and will not continue until the process is finished.

In the mean time customer has modified their Chef scripts to install individual msi installers, and reproduced the problem, hence removing Suite installer out of the equation.

windows_package 'Prerequisite' do
 source 'Prerequisite.msi'
 installer_type :custom
 options '/qn ACCEPT_EULA=1'
end
windows_package 'APP' do
 source 'App.msi'
 installer_type :custom
 options '/qn ACCEPT_EULA=1'
end

Customer sent log files for failed and successful installations but comparison did not really bring any useful information. One thing we noticed, however. The installer log did not include all of the debugging information, so we asked the customer to append /l*vx logfile.log options to get all the information we possibly could:

windows_package 'Prerequisite' do
 source 'Prerequisite.msi'
 installer_type :custom
 options '/qn ACCEPT_EULA=1 /l*vx c:\Prerequisite.log'
end
windows_package 'APP' do
 source 'App.msi'
 installer_type :custom
 options '/qn ACCEPT_EULA=1 /l*vx c:\APP.log'
end

This got us a little bit further:

MSI (s) (D0:C4) [20:29:06:113]: Note: 1: 1935 2: {6F6751F2-43FB-4C77-9791-B24EB154D406} 3: 0x800703FA 4: IAssemblyCacheItem 5: Commit 6: Company.Systems.Services,processorArchitecture=”amd64″,publicKeyToken=”45d8b3bf5a7dd114″,type=”win32″,version=”1.0.0.73″
MSI (s) (D0:C4) [20:29:06:113]: Assembly Error (sxs): Please look into Component Based Servicing Log located at %windir%\logs\cbs\cbs.log to get more diagnostic information.

So after another 24 hour round trip due to time difference we got hold of the %windir%\logs\cbs\cbs.log file which had quite interesting things in it:

2017-03-07 20:34:24, Info                  CBS    Trusted Installer is shutting down because: SHUTDOWN_REASON_AUTOSTOP

2017-03-07 20:34:24, Info                  CBS    TiWorker signaled for shutdown, going to exit.

2017-03-07 20:34:24, Info                  CBS    Ending the TiWorker main loop.

2017-03-07 20:34:24, Info                  CBS    Ending the TrustedInstaller main loop.

2017-03-07 20:34:24, Info                  CBS    Starting TiWorker finalization.

2017-03-07 20:34:24, Info                  CBS    Starting TrustedInstaller finalization.

2017-03-07 20:34:24, Info                  CBS    Ending TrustedInstaller finalization.

2017-03-07 20:34:56, Info                  CBS    TI: — Initializing Trusted Installer —

2017-03-07 20:34:56, Info                  CBS    TI: Last boot time: 2017-03-07 20:10:19.488

2017-03-07 20:34:56, Info                  CBS    Starting TrustedInstaller initialization.

2017-03-07 20:34:56, Info                  CBS    Ending TrustedInstaller initialization.

2017-03-07 20:34:56, Info                  CBS    Starting the TrustedInstaller main loop.

2017-03-07 20:34:56, Info                  CBS    TrustedInstaller service starts successfully.

2017-03-07 20:34:56, Info                  CBS    No startup processing required, TrustedInstaller service was not set as autostart

2017-03-07 20:34:56, Info                  CBS    Startup processing thread terminated normally

2017-03-07 20:34:56, Info                  CBS    Starting TiWorker initialization.

2017-03-07 20:34:56, Info                  CBS    Ending TiWorker initialization.

2017-03-07 20:34:56, Info                  CBS    Starting the TiWorker main loop.

2017-03-07 20:34:56, Info                  CBS    TiWorker starts successfully.

2017-03-07 20:34:56, Info                  CBS    TiWorker: Client requests SxS store object.

2017-03-07 20:34:56, Info                  CBS    Universal Time is: 2017-03-07 20:34:56.619

2017-03-07 20:34:56, Info                  CBS    Loaded Servicing Stack v6.3.9600.18384 with Core: C:\Windows\winsxs\amd64_microsoft-windows-servicingstack_31bf3856ad364e35_6.3.9600.18384_none_fa1d93c39b41b41a\cbscore.dll

2017-03-07 20:34:56, Info                  CSI    00000001@2017/3/7:20:34:56.621 WcpInitialize (wcp.dll version 0.0.0.6) called (stack @0x7ffe61937fa9 @0x7ffe62624924 @0x7ffe6262415b @0x7ff7d6e3d69a @0x7ff7d6e3e1f8 @0x7ffe743420f3)

2017-03-07 20:34:56, Info                  CBS    Could not load SrClient DLL from path: SrClient.dll.  Continuing without system restore points.

2017-03-07 20:34:56, Info                  CBS    SQM: Initializing online with Windows opt-in: False

2017-03-07 20:34:56, Info                  CBS    SQM: Cleaning up report files older than 10 days.

2017-03-07 20:34:56, Info                  CBS    SQM: Requesting upload of all unsent reports.

2017-03-07 20:34:56, Info                  CBS    SQM: Failed to start upload with file pattern: C:\Windows\servicing\sqm\*_std.sqm, flags: 0x2 [HRESULT = 0x80004005 – E_FAIL]

2017-03-07 20:34:56, Info                  CBS    SQM: Failed to start standard sample upload. [HRESULT = 0x80004005 – E_FAIL]

2017-03-07 20:34:56, Info                  CBS    SQM: Queued 0 file(s) for upload with pattern: C:\Windows\servicing\sqm\*_all.sqm, flags: 0x6

2017-03-07 20:34:56, Info                  CBS    SQM: Warning: Failed to upload all unsent reports. [HRESULT = 0x80004005 – E_FAIL]

2017-03-07 20:34:56, Info                  CBS    NonStart: Set pending store consistency check.

2017-03-07 20:34:58, Info                  CBS    Failed to open the registry root: n/a, key: COMPONENTS. [HRESULT = 0x800703fa – ERROR_KEY_DELETED]

2017-03-07 20:34:58, Info                  CBS    Failed loading COMPONENTS hive with ERROR_SHARING_VIOLATION, retry #1

(…)

2017-03-07 20:35:08, Info                  CBS    Failed to load the COMPONENTS hive from ‘C:\Windows\System32\config\COMPONENTS’ into registry key ‘HKLM\COMPONENTS’. [HRESULT = 0x80070020 – ERROR_SHARING_VIOLATION]

2017-03-07 20:35:08, Info                  CBS    Failed to load component store [HRESULT = 0x80070020 – ERROR_SHARING_VIOLATION]

2017-03-07 20:35:08, Error                 CSI    00000002 (F) c0000425 [Error,Facility=(system),Code=1061 (0x0425)] #11# from Windows::Rtl::SystemImplementation::DirectRegistryProvider::SysOpenKey(flg = (AllowKeyNotFound), key = {provider=NULL, handle=0, name= (“null”)}, da = (KEY_READ|KEY_WOW64_64KEY), oa = @0x2c3dc1d7a0->OBJECT_ATTRIBUTES {s:48; rd:NULL; on:[28]”\Registry\Machine\COMPONENTS”; a:(OBJ_CASE_INSENSITIVE)}, disp = Unmapped disposition: 1036114208 (0x3dc1d920))[gle=0xd0000425]

2017-03-07 20:35:09, Error                 CSI    00000003@2017/3/7:20:35:09.270 (F) base\wcp\sil\merged\ntu\ntsystem.cpp(3813): Error c0000425 [Error,Facility=(system),Code=1061 (0x0425)] originated in function Windows::Rtl::SystemImplementation::DirectRegistryProvider::SysOpenKey expression: (null)

[gle=0x80004005]

2017-03-07 20:35:17, Info                  CBS    Added C:\Windows\Logs\CBS\CBS.log to WER report.

2017-03-07 20:35:17, Info                  CBS    Could not get active session for current session file logging [HRESULT = 0x80004003 – E_POINTER]

2017-03-07 20:35:18, Info                  CBS    Not able to add pending.xml to Windows Error Report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2017-03-07 20:35:18, Info                  CBS    Not able to add pending.xml.bad to Windows Error Report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2017-03-07 20:35:18, Info                  CBS    Not able to add SCM.EVM to Windows Error Report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2017-03-07 20:35:19, Info                  CBS    Failed to open regkey configuration. [HRESULT = 0x800703fa – ERROR_KEY_DELETED]

2017-03-07 20:35:19, Info                  CBS    Failed to query Consent Key. [HRESULT = 0x800703fa – ERROR_KEY_DELETED]

2017-03-07 20:35:19, Error                 CSI    00000004 (F) c0000425 [Error,Facility=(system),Code=1061 (0x0425)] #10# from Windows::Rtl::SystemImplementation::CSystemIsolationLayer_IRtlSystemIsolationLayerTearoff::OpenRegistryKey(flags = (KeyNotFoundIsOk), da = (KEY_READ), kn = [ml:58{29},l:56{28}]”\Registry\Machine\COMPONENTS”, key = NULL, disp = Invalid)[gle=0xd0000425]

2017-03-07 20:35:44, Info                  CBS    Ending TiWorker finalization.

Voila!!! We see here some kind of race condition between two TiWorker processes. Second TiWorker  starts initialization before the first one finishes finalization. Looks like a bug in Microsoft Windows Module Installer. 

To buy us some time we decided to add waits before every msi installation using SysInternal’s pslist:

execute "Wait until TiWorker is gone" do
 command "c:/Sysinternals/pslist /accepteula TiWorker"
 retries 1000
 returns 1
 retry_delay 10
 timeout 10
end

This had hope of success because TrustedInstaller.exe launches TiWorker.exe process at some point during installation but its execution can outlive the installation itself.

Customer no longer saw failures but we still needed to get Suite installer working and we could not just add waits there. Further, we observed that the Suite installer was designed in a way that installs various msis in multiple transactions. First thing we did was to wrap all the installations in single transaction. But this resulted in the unexpected: when one of the installers was executed it reported that there is installation already in progress and it cannot continue! We have just configured it to run in a single transaction, so what is going on?

As it turned out some of the installers were 32 bit (as we could say by looking at Summary filed of msi in Installshield) and some of them were 64 bit. Even though internally Installshield always creates 32 bit installers this mix resulted in spawning two concurrently running installations, one for 32 bit and one for 64 bit. How strange?!

Changing the offending msis to 64 bit (by changing Summary field to AMD64) caused it to not report another installation in progress any more and as it turned out, also resolved the original HRESULT: 0x800703FA problem.

 

Debug symbols in separate files on Linux via boost bjam

July 12, 2015

Surprisingly many developers I have met do not realize the significance of debug symbols and how they should  be handled. This is especially true when it comes to release builds.

Those who do realize the need of resymbolication of customer crash dumps have a problem of how to generate them to separate files. They need to be in separate files because you don’t want to ship debug symbols with your software (because they are big and for other reasons).

Boost bjam does not support this out of the box. Weather the symbols will be stripped (or generated in the first place) in bjam depends on <debug-symbols> feature value. This feature is off for release builds by default but can be enabled explicitly.

Assuming that debug symbols are kept in shared object file or executable we can separate debug symbols, strip and add pointer to the binary so that debugger finds them later. This can be done by leveraging objcopy and is described here.

To get this working in bjam we need to modify gcc.jam file.

First we need to register .so.debug and .debug types:

# Should be called PDB on Windows. Also prefixes should not be set on windows or cygwin
type.register SO_DEBUG : so.debug ;
type.set-generated-target-prefix SO_DEBUG : gcc : lib ;
type.register DEBUG : debug ;

Then we need to modify generator so that it produces two files: .debug and executable and so.debug and shared library:


generators.register
[ new gcc-linking-generator gcc.link
: LIB OBJ
: DEBUG EXE
: gcc ] ;
generators.register
[ new gcc-linking-generator gcc.link.dll
: LIB OBJ
: SO_DEBUG SHARED_LIB
: gcc ] ;

Now, we modify the actions to add additional steps to save debug information to .debug file before stripping. We also add gnu link so that debuggers can find external debug information bits:

actions link bind LIBRARIES
{
"$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,$(RPATH) -Wl,-rpath-link$(SPACE)-Wl,"$(RPATH_LINK)" -o "$(<[-1])" $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS)
objcopy --only-keep-debug "$(<[-1])" "$(<[-1]).debug"
strip --strip-debug --strip-unneeded "$(<[-1])"
objcopy --add-gnu-debuglink="$(<[-1]).debug" "$(<[-1])"
}

And the same for shared library:


# Differs from 'link' above only by -shared.
actions link.dll bind LIBRARIES
{
"$(CONFIG_COMMAND)" -L"$(LINKPATH)" -Wl,$(RPATH_OPTION:E=-R)$(SPACE)-Wl,$(RPATH) "$(.IMPLIB-COMMAND)$(<[1])" -o "$(<[-1])" $(HAVE_SONAME)-Wl,$(SONAME_OPTION)$(SPACE)-Wl,$(<[-1]:D=) -shared $(START-GROUP) "$(>)" "$(LIBRARIES)" $(FINDLIBS-ST-PFX) -l$(FINDLIBS-ST) $(FINDLIBS-SA-PFX) -l$(FINDLIBS-SA) $(END-GROUP) $(OPTIONS) $(USER_OPTIONS)
objcopy --only-keep-debug "$(<[-1])" "$(<[-1]).debug"
strip --strip-debug --strip-unneeded "$(<[-1])"
objcopy --add-gnu-debuglink="$(<[-1]).debug" "$(<[-1])"
}

Here is the link to differences in github.

Problems with the above solution: 

Note that above changes will cause linking with rpath to misbehave. If you know how to fix it, please let me know. I tried reversing EXE and DEBUG (and changing -1 to -2) but this caused problems with relink rule of exe. Not sure how to fix this one other than setting <install-type>EXE to narrow down targets. Also relinking will cause .debug files to be created next to executable anyway so this hack really needs some honing.

Explanation of Xcode 4 templates

May 20, 2014

Explanation of Xcode 4 templates

The Xcode plugin for PNaCl that i was trying to write is not even semi-automatic. It has lots of manual stuff that needs to be set (like PNaCl SDK paths).  I have no idea how to debug (P)NaCl code from within Xcode and i am not even sure if this will be possible. Apparently google folks are working on the IDE for PNaCl.  That are the reasons that i have not done much about it recently – i do not think it would ever be useful to anybody. There is one more reason i have not done much about it: it allows to compile C++ code to PNaCl from within Xcode, so i was busy porting the code i wanted to port. But today i wanted to play with the Portable SIMD Vectors in PNaCl and i needed a standalone project to do this. And it turned out that i did not write any templates, so i  am going to play with templates to ease one-off test projects. Fortunately there was a guy who did the hard work so i can “stand on the shoulder of giants” 😉

Xcode 5 custom compiler

January 15, 2014

Chrome 32 ships with PNaCl enabled by default. It has begun.

Ok but what to do if You want to jump into deep water and port an SDK consisting of quite large C/C++ codebase to PNaCl ( everybody understand that the possibility to run native code with almost native performance directly in the browser is cool )? Which tools to use?

I started with QMake. I thought it would be easy. Create new MkSpec, set paths to pnacl-clang and that is it. Well it is not that easy, but possible.

For some reason, though, I am not really into makefiles. These days i would expect that a click on the error or warning  generated by the compiler taking me  directly on the line of source code that caused problem (or at least compiler thinks so). Wait a second… Our SDK is already ported to iOS. And we have nice Xcode project and C++ unit test, all running. And Xcode used to support GCC side by side with LLVM compiler. Maybe there is a way to tweak Xcode and add custom compiler that would compile PNaCl code in Xcode?

First i looked around and found a few attempts (Erlang Xcode plugin, C# Mono plugin). I did not realize that these plug-ins stopped development in Xcode 3.0 stage. Nevertheless i tried. First by copying : 

/Applications/Xcode.app/Contents/PlugIns/Xcode3Core.ideplugin/Contents/SharedSupport/Developer/Library/Xcode/Plug-ins/Clang LLVM 1.0.xcplugin

to 

/Applications/Xcode.app/Contents/PlugIns/Xcode3Core.ideplugin/Contents/SharedSupport/Developer/Library/Xcode/Plug-ins/PNaCl.xcplugin

and renaming compiler specification file Contents/Resources/Clang LLVM 1.0.xcspec to PNaCl.xcspec and then modifying ExecPath to point to pnacl-clang (and also Identifier, and a few others) after rerunning Xcode (actually there were few other steps involved)   – voila, the PNaCL compiler appear on the list of the compilers in Xcode. So I copied one of the existing targets in the SDK project, named it SDK-PNaCl and clicked Build. The actual pnacl-clang command was run, but it failed because of -F option which was not understood. After a few hours of searching and thinking i decided that there is no way to override this behavior just by modifying the xcspec, while the class is still XCCompilerSpecificationClang.

I needed a custom class. I tried to copy the implementation of compiler specification from Erlang Xcode plugin’ but the method 

-computeDependenciesForInputFile:ofType:variant:architecture:outputDirectory:inTargetBuildContext:

was not called no matter what i did. After some time i realized that in Xcode 4+ this method has changed to: 

- computeDependenciesForInputNodes:ofType:variant:architecture:outputDirectory:withMacroExpansionScope:

But, wtf? I was able to link to /Applications/Xcode.app/Contents/OtherFrameworks/DevToolsCore.framework so there must be another version of this framework. A quick search under /Application/Xcode.app revealed that there is another set of frameworks, for Xcode 5, namely here: 

/Applications/Xcode.app/Contents/PlugIns/Xcode3Core.ideplugin/Contents/Frameworks/DevToolsCore.framework

which were correct for Xcode 5. I used class dump to generate headers out of the binary versions of framework. Class dump is an awesome tool but it has some flaw which means that generated headers need to be manually fixed. One of them is that it tries to include “NSObject.h” where it should include <Foundation/Foundation.h>. 

Having obtained the correct header i tried to go through them and try to understand but this will take me another i don’t know how many sleepless hours. Currently i am able to compile a file into an object file but i am not sure how to substitute the linker/librarian. 

Open questions. Just a few: 

  1. How to substitute linker/librarian? Does product node play a role in this process? Does a file type play a role? 
  2. How to set output file type of the node of the C/C++ file node to newly created pnacl-o, as opposed to mach-o?
  3. How to get rid of unnecessary Build options?
  4. Would it be beneficial to define PNaCl.platform as a platform, and each of  pepper_32, pepper_30, pepper_canary as their SDK? Is it possible?
  5. Very naive: is there a documentation pointing out the Xcode build process, role of specifications, etc?

UPDATE:Added references with very important link. I should have read it thoroughly earlier since it might have a solution for the linker thing ready.
References:

Juniper Network Connect won’t connect on MacOS X.

June 17, 2013

If You came across problem with Juniper Network Connect not connecting to VPN on MacOS then You might try this. In my case this was not enough. Every time i update Safari by autoupdate it  stops working again (this happened to me when updating from  6.0.3 to 6.0.4 and to 6.0.5) for no reason. If it happened to You as well You might go to java test plugin site and click on the plugin. For unknown reasons this reenabled my Juniper Network Connect to connect. 

Please note that this problems might have been fixed in newer version of Juniper Network Connect but our network administrators can’t be bother to update. 

I am using 10.7 Lion, Safari 6.0.5, Java 1.7 Update 21. 

Application Unit Testing from command line on iPhoneSimulator.

June 6, 2013

It’s hard to tell the reasons behind not supporting application-hosted tests on IPhoneSimulator and the infamous warning message:

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests:81: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).

But people need to do this and there are solutions to this problem on the Internet, especially here. The solution seem to work in Xcode 4.5 but things seem to be changed slightly in Xcode 4.6 (4.6.2).

In version 4.6.2 script :

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/
Tools/RunPlatformUnitTests

seems to have bash script method RunTestForApplication redefined:

### Simulator app testing isn't supported, so redefine RunTestsForApplication.
RunTestsForApplication() {
Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."
}

So in addition to the solution linked above this method needs to be commented out in this file. Also please make sure that You somehow append -RegisterForSystemEvents flag to the test command run. You can do this either in a way described in the cited article or by adding the following line:

export OTHER_TEST_FLAGS=-RegisterForSystemEvents

after line 124 of
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests,
otherwise You will end up with the following error:

Couldn't open shared capabilities memory GSCapabilities (No such file or directory)
Couldn't open shared capabilities memory GSCapabilities (No such file or directory)
Couldn't open shared capabilities memory GSCapabilities (No such file or directory)
Terminating since there is no system event server.
(Run the EventPump or pass the argument "-RegisterForSystemEvents" if you want to run without SpringBoard.

Inception

June 4, 2013

Today I made my first move to have a blog. I plan to write about the stuff I do. Because I am a computer geek there might be something technical stuff. Because I like fishing, there might be some fishing stories as well.

There were a few reasons to do start a blog:

  1. I recently tried to solve a problem at my work and could not find a solution. I spent some time trying to solve and I think I would like to share what I have done.
  2. To promote myself. There are some companies and some jobs that require You to have a solid fingerprint in the Internet.
  3. Improve my writing in English skills. Since English is not my first language (I love this euphemism, used by natives in situations where my English really didn’t shine) it would be nice to exercise it a bit. Feel free to correct me (although spell checker does a good job, I believe).

I hope this blog will thrive and will be updated from time to time and that I will have motivation to do something more than putting under construction sign. This happens to too many sites, expressing short lived zeal.

It would be nice to have a commitment at this point. Ok. I will write a subpage introducing myself ( can’t promise it will be interesting ) and I will present a solution on how to integrate unit tests that use CPPUnit framework in Xcode (SenTesting framework).