Problems with parallel builds and wait step

[bwilder]

I'm currently running into 3 issues while attempting to set up a build farm w/parallel script:

Configuration:

• 3 build servers (64bit Win7) and a file server(W2k8),
• All build servers are using VBP8 64bit, installed in the same path,
• 2 build scripts (Master and slave): Master script is on 1 server, Slave script is on the other 2 build servers. All exist in the same directory, c:\TEST.
• Master script has steps to cover all scenarios outlined below. Some steps will need to be enabled/disabled to function as stated in the scenario.
• Slave scripts must be modified to write separate completion files.

Scenario 1: Set up a master/slave configuration. Master build script kicks off 2 parallel builds on remote servers, does a little work of its own, then waits for the parallel processes to complete before finishing itself.

Problem:
The wait step hangs indefinitely because the PIDs for the parallel processes disappeared right after kicking off the parallel processes. The 'Wait for completion' checkbox in the Advanced tab on the parallel steps is disabled and both parallel process steps have scripts similar to the following enabled, causing temporary variables to be created to store the PIDs:

Sub vbld_StepDone()

	' store this step's process ID in another temporary macro
	vbld_TempMacros().Add "SLAVE1_ID", vbld_TempMacro("RUNPROGRAM_PROCESSID").Value

End Sub

Questions:

• Is this working as designed?
• Are the processes started on the master supposed to die after psexec kicks off the remote scripts?

Scenario 2: Set up the same master/slave configuration. Master build script kicks off 2 parallel builds on remote servers, does some work of its own, then waits for completion text files to be written by the parallel processes. (This is a workaround for scenario 1)

Problem:
The wait step in master script, which has 'Wait for all files to be created or modified or processes to exit' enabled, hangs indefinitely when the completion files do not exist prior to the master getting to the wait step. If the completion files from the slaves already exist when the master gets to its wait step and then the slaves complete their work and update the files, the master works properly and recognizes the changes.

Questions:

• Why doesn't the wait step recognize when the named file is created?
• I want to start out with no completion scripts, then have the slave(s) write a script when it finishes its work. The slave(s) may finish before or after the master gets to its wait step, but the master should recognize the existence of the file, regardless. How do I do that?

Scenario 3:
Same setup as scenario 2. Master must wait for the slaves to complete before it can finish, but the slaves must also wait for specific steps in the master to finish before they can do certain things. The master has a step about half way through it that writes a completion file for the slaves to check.

Problem:
The slaves hang indefinitely if the master writes its completion file before the slaves get to their wait step. If the slaves get to their wait step first and the file is created after, the slaves complete as expected. This is the opposite behavior of what was encountered in scenario 2.

So, it would seem that the master's wait only works if the completion files it needs exist beforehand are updated the wait step is reached. Meanwhile, the slave's wait only works if the completion file they need do NOT exist beforehand and is only created after the wait step is reached.

This is crazy confusing and can't be working as designed, can it?

Questions:

• Why is the behavior for a slave different than the behavior for the master?
• I want to start out with no completion script, then have the master write a script when it finishes the work the slaves need. The master may finish before or after the slave(s) get to their wait step, but the slave(s) should recognize the existence of the file, regardless. How do I do that?

Thanks!

Just for reference:
Wait action - http://www.kinook.com/VisBuildPro/Ma...waitaction.htm
VisBuildPro action - http://www.kinook.com/VisBuildPro/Ma...projaction.htm
Chain build example - http://www.kinook.com/VisBuildPro/Ma...hainsample.htm

[kinook]

Originally Posted by bwilder

Scenario 1: Set up a master/slave configuration. Master build script kicks off 2 parallel builds on remote servers, does a little work of its own, then waits for the parallel processes to complete before finishing itself.

Problem:
The wait step hangs indefinitely because the PIDs for the parallel processes disappeared right after kicking off the parallel processes. The 'Wait for completion' checkbox in the Advanced tab on the parallel steps is disabled and both parallel process steps have scripts similar to the following enabled, causing temporary variables to be created to store the PIDs:

Questions:

• Is this working as designed?
• Are the processes started on the master supposed to die after psexec kicks off the remote scripts?

It's working as designed, but if not waiting for completion, the local psexec process will not wait for the remote process to complete, so waiting on those process IDs won't do much good. We don't currently support the scenario of kicking off processes on a remote machine without waiting to complete, and then later checking to see if those processes have completed. We will add to our todo list to see what can be done to support that in the future. For now, you might be able to use WMI and VBScript to do something like that:
http://social.technet.microsoft.com/...8-03fde4d51113

Scenario 2: Set up the same master/slave configuration. Master build script kicks off 2 parallel builds on remote servers, does some work of its own, then waits for completion text files to be written by the parallel processes. (This is a workaround for scenario 1)

Problem:
The wait step in master script, which has 'Wait for all files to be created or modified or processes to exit' enabled, hangs indefinitely when the completion files do not exist prior to the master getting to the wait step. If the completion files from the slaves already exist when the master gets to its wait step and then the slaves complete their work and update the files, the master works properly and recognizes the changes.

Questions:

• Why doesn't the wait step recognize when the named file is created?
• I want to start out with no completion scripts, then have the slave(s) write a script when it finishes its work. The slave(s) may finish before or after the master gets to its wait step, but the master should recognize the existence of the file, regardless. How do I do that?

The wait step does recognize the file(s) being created in our tests. The FindFirstChangeNotification API (used by the Wait action) can have problems with network shares (see http://support.microsoft.com/kb/188321), so make sure to use a local path (same machine as where the Wait step builds) for the signal files. Check the Don't wait if files already exist when step starts option on the Wait action for the master to recognize the file having been created before its Wait step builds (need to uncheck the Delete files before waiting option to enable).

Scenario 3:
Same setup as scenario 2. Master must wait for the slaves to complete before it can finish, but the slaves must also wait for specific steps in the master to finish before they can do certain things. The master has a step about half way through it that writes a completion file for the slaves to check.

Problem:
The slaves hang indefinitely if the master writes its completion file before the slaves get to their wait step. If the slaves get to their wait step first and the file is created after, the slaves complete as expected. This is the opposite behavior of what was encountered in scenario 2.

So, it would seem that the master's wait only works if the completion files it needs exist beforehand are updated the wait step is reached. Meanwhile, the slave's wait only works if the completion file they need do NOT exist beforehand and is only created after the wait step is reached.

This is crazy confusing and can't be working as designed, can it?

Questions:

• Why is the behavior for a slave different than the behavior for the master?
• I want to start out with no completion script, then have the master write a script when it finishes the work the slaves need. The master may finish before or after the slave(s) get to their wait step, but the slave(s) should recognize the existence of the file, regardless. How do I do that?

You need to check the Don't wait if files already exist when step starts option to not wait if the signal file was created before the Wait step began. And use local paths for waiting on file creation/modification.

[bwilder]

Thanks for the feedback.

Your suggestions regarding scenario 2 seem to have worked out best for me.

Thanks!