Occasional build freeze

[petrhlavka]

Hello,

I've using Visual Build for two years without any problems. However in last month or two, some of my builds run through VisBuildCmd.exe start freezing. The freezes occur in various steps. The last step in the log file is usually running external command through cmd.exe, building with Visual Studio .NET 2003 SP1 or 'Set Macro' step.

Didn't somebody encounter similar problems? I'm trying to find what is causing this. I have the same problems on two computers, where no major OS/installation changes/updates weren't done for half a year.

Thanks for the replies.

Visual Build Professional 6.6
Registered to: GRISOFT - Brno, Czech Republic (1-site license)
Windows Version: 5.2.3790.2.0
Install path: C:\Program Files (x86)\VisBuildPro6
SftTree_IX86_U_50.dll version 5.06
unins000.exe version 51.47.0.0
VisBuildCmd.exe version 6.6.0.0
VisBuildPro.exe version 6.6.0.1
VisBuildBld.dll version 6.6.0.1
VisBuildBurn.dll version 6.6.0.0
VisBuildCore.dll version 6.6.0.0
VisBuildDotNET.dll version 6.6.0.0
VisBuildExt.dll version 6.6.0.0
VisBuildLog.dll version 6.6.0.0
VisBuildMisc.dll version 6.6.0.0
VisBuildMS.dll version 6.6.0.0
VisBuildMS2.dll version 6.6.0.0
VisBuildNet.dll version 6.6.0.0
VisBuildSvr.dll version 6.6.0.0
VisBuildSvr.Interop.dll version 1.0.0.0
VisBuildVCS.dll version 6.6.0.0

[kinook]

Is VisBuildCmd.exe definitely hung (can it be aborted via Ctrl+C or Ctrl+Break?)? Is it using any CPU cycles? Is the process that it launched for the current step (devenv for VS.NET) still running, and if so, does killing it allow VBP to continue? Are there any recent errors reported by VBP in the Windows Event Viewer? Could you try re-downloading and installing again from http://www.kinook.com/Download/VisBuildProEval.exe?

Please ZIP and send to support@kinook.com:
1) The .bld file
2) A VS solution that manifests the problem
3) A build log file

Thanks.

[petrhlavka]

VisBuildCmd.exe seems to me to be hung, it's not using any CPU cycles only single thread is being switched into context. The started process is always finished. The build is started through sheduler, so I don't know how to break it. I can only kill it. There is nothing suspicious in event log. The builds have over 200 steps, if I reorder some of them or remove any, the build starts freezing at another step so unfortunately it's hard to narrow the problem.

I will try to reinstall the VisualBuild, as the freezes occur only occasionally (however, on the same steps of build) and recently, it seems to me that some other software or OS updates can be casing this problem.

[NathanGaida]

I've been experiencing the same thing. OS is Server2003. After upgrading to VBP6.6 our builds have been locking up randomly. Generally we run them w/ GUI, and what we see when we RDP to the machine where the build is locked is that VBP is up and running, but the program window is white and does not respond.

Ironically we have 5 of these machines, and only 2 of them are experiencing the lockup issue (randomly), the other 3 have not.

We have seen a similar issue in the past, and that was resolved by a "stealth" update Kinook did. Ie the version was still 6.4, but they updated one of the DLLs to 6.4.0.1.

[petrhlavka]

Hmmm. It seems that there is a new 6.6, release December 31, 2007. The previous one came from September, there is no Changelog/Annoucement in the forum. I will try it.

[NathanGaida]

I think I've been using the latest 6.6. I did a compare between one of the machines that is working and the one that isn't. I thought I had updated all of my machines to 6.6, but apparently that isn't true. The working machines are running 6.4, the random lockup machines are running 6.6.

[kinook]

Originally posted by NathanGaida
I've been experiencing the same thing. OS is Server2003. After upgrading to VBP6.6 our builds have been locking up randomly.Which version did you upgrade from? At what point in the build does it typically lock up? For instance, does it always hang on an action that runs an external program? Can you ZIP and send (to support@kinook.com) the .bld file and a build log file?

Generally we run them w/ GUI, and what we see when we RDP to the machine where the build is locked is that VBP is up and running, but the program window is white and does not respond.Does building via the console app instead of the GUI make a difference?

Ironically we have 5 of these machines, and only 2 of them are experiencing the lockup issue (randomly), the other 3 have not.Please ZIP and send the VBP Help | About | Install Info details for each of the machines (indicating which versions go with the good and problem machines).

We have seen a similar issue in the past, and that was resolved by a "stealth" update Kinook did. Ie the version was still 6.4, but they updated one of the DLLs to 6.4.0.1.I don't recall us having fixed an issue similar to this (we did fix a hang issue with the Replace in File action that you reported in 2003, but not in an interim build [there was a workaround]), but we do sometimes patch the download (without a main version number change) to fix a problem reported by one user that isn't likely to affect other users. We haven't made any changes to resolve the problem you're reporting now, but you can try downloading and installing the latest build if you don't have it (see http://www.kinook.com/Forum/showthread.php?threadid=3056 for details).

Do you have any additional software installed on the build box (anti-virus, anti-spyware, etc.)? What sort of hardware are you running on? Is there anything from VBP in the Event Viewer? If you can narrow the problem down to a reproducible case that we can build here, please ZIP and send it. Thanks

[NathanGaida]

I would have been upgrading from 6.4.

Where does it lock up:
I've looked at the past 3 log files where it has frozen, and each one had it lock up right as it was compiling a VC2003 DLL.

We don't run the console app, so I don't know. And as it happens randomly (actually, it seems to happen at the worst possible times, freezing the nightly build when QA is really needing something in the morning) I can't comment on the difference.

Re: the stealth update. The one I referred to in my post wasn't the bug fix on replace files. I ran into a similar freeze issue with a recent version of Visbuild6 and searched the forums. Someone else mentioned that there was a new update of visbuild6 with a dll version# increment that fixed their problem, and it fixed ours as well (at the time).

Nothing else running on our machines that would interfere like that.

I'll have an email sent to support shortly.

[NathanGaida]

As a followup, the previous 6.6 I had installed was downloaded on the 17th of Dec, before your update on the 31st of Dec.

[kinook]

Originally posted by NathanGaida
We don't run the console app, so I don't know. And as it happens randomly (actually, it seems to happen at the worst possible times, freezing the nightly build when QA is really needing something in the morning) I can't comment on the difference.Are the builds being started from a Scheduled Task, or how are they started?

Re: the stealth update. The one I referred to in my post wasn't the bug fix on replace files. I ran into a similar freeze issue with a recent version of Visbuild6 and searched the forums. Someone else mentioned that there was a new update of visbuild6 with a dll version# increment that fixed their problem, and it fixed ours as well (at the time).Which forum thread?

[NathanGaida]

Scheduled task, which just starts a batch file.

As for the other forum thread, I don't know which it was. It was probably 6months ago, and I don't remember what I searched for to find it.

[score]

I'm seeing this problem too. I installed VBP 6.6 in the beginning of the year and am having it hang. It seems to be random. I am also running on windows 2003 server. I use it through remote desktop. The build normally takes about 4 hours, when I come back to check on it after an hour all I see is a white screen. Most of the time the build finishes without error. However it has happened twice today. This is getting frustrating. Management types are getting on my back!

[NathanGaida]

score,

I'd recommend downgrading to 6.4 (dunno about 6.5? but going back to 6.4 fixed my problem for now.)

Luckily I'm not worried about a long term fix, as they are out-sourcing us and in 2 weeks I no longer have to care about it working or not. :)

[score]

how do we get an older version?

[NathanGaida]

That I don't know. I happened to have forgotten to upgrade a machine from 6.4, so it was convenient for me. (That was how i discovered that 6.6 was the problem. The 6.4 machine always worked, and I just assumed I had updated them all)

[score]

Thanks, I'll see if I can get it. I have version 4.0c version 5.1, and 6.6... no luck so far..

[kinook]

You indicated that you're accessing the build box via Remote Desktop. Is the RD connection remaining open, or does this occur after disconnecting and reconnecting? Are the builds launched manually, from a Scheduled Task, or something else? Does the problem occur when using the console app rather than GUI app? Please ZIP and send to support@kinook.com:
1) The info from Help | About | Install Info
2) The .bld file
3) A build log file from a hanging build

Thanks.

[score]

Ok, thanks for the location of the previous versions. As for you other questions:

I have only used this version thru the gui app. I can run it from the command line if you think that would help. Isn't it just the same .exe but giving the bld file as an argument?

The remote desktop stays longed in, however the screen saver puts me into the userid/password page. So when I log in it refreshes the screen and that is when I see the white screen UI hung.

I'll get you the other info on the next hang..zip it up and send it to you.

thanks

[Zoid]

I've been seeing this for a couple weeks. It consistently hung during our nightly build that we ran from the task scheduler. We downgraded to 6.4 and it no longer hangs.

We run builds from the command line.

The odd thing is I've seen the hang when VisBuildPro was executing a compile through devenv and also when executing one of our own tools. It just stops responding and seems hung just as it excuted the command. Terminating the command VisBulidPro spawned does not resume VisBuildPro, it remains hung. I end up having to terminate VisBuildPro itself.

We'll be sticking with 6.4 for now until this problem has been addressed in newer builds. Fortunately we are not dependent on new features of the later versions.

[kinook]

We have implemented a change that may resolve this issue. Please download the latest from http://www.kinook.com/Download/VisBuildProEval.exe to try it.

Also try launching all VisBuildPro.exe/VisBuildCmd.exe instances with the /mta flag (http://www.visualbuild.com/Manual/threading.htm).

And something that apparently helped one other user was to add a Run Script (VBScript) step like this to the project:

Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000
Application.Options.EnableEventErrorLogging = False

[score]

Ok, I downloaded version 6.4 and it consitently hangs after the same step. I was about to post the information, but then saw this new "test" fix. So I installed it and ran it. The first time it ran in the UI it hung on an earlier step :(

[score]

Ok, I sent you the .bld file and info files. Hopefully we can see why it is hanging.

[score]

Well I installed version 6.7 and running VisBuildPro.exe with the /mta option seems to do the trick. At least in the last 3 tries it did not hang. It was hanging every time at the same step. It was interesting that it only hung whan I ran it thru the UI and not while running it from the command line (with the /s /b switch).

I don't know what the /mta switch does though. You might want to try it.

[score]

I'm unfortunately still seeing hangs running this tool. I was wondering if any progress has been made on identifying the cause and fixing this freeze.

[kinook]

We have not been able to reproduce the problem (and not for a lack of trying), but as far as we know, one or more of the following resolved the problem for others who have reported it:

1) Updating to v6.7a

2) Adding a Run Script (VBScript) step like this at the beginning of with this code:

Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000

3) Removing the VBP DEP exclusions (Control Panel -> System -> Advanced -> Data Execution Prevention -> Turn on DEP for all programs and services except those I select, Remove Visual Build Professional and Visual Build Professional 6 Console App from the list, OK, and restart Windows) and then configuring Windows to 'Turn on DEP for essential Windows programs and services only' (in that same dialog) and restart again.

4) Adding a Wait step (5 seconds) at the end of a chained (child) build or before and/or after a step that hangs.

If none of those help, please provide the following information:

Are you building from a Scheduled Task, starting manually, or? Any ideas on anything that may have changed when the problem returned? Please ZIP and send:
1) The info from Help | About | Install Info (after updating to v6.7a)
2) A screen shot of System properties (Windows+Break)
3) A reproducible test case, including VBP project(s) that demonstrate the problem.

Thanks.

[score]

Thanks, i will try those steps. I am at 6.2a right now. I will upgrade. I tried the 7a before, with worse luck last time. But I'll start again. The 7a version i have is a different size then the one on the download site now. I'll let you know.

[jameskoch]

We've been experiencing something along these lines too. The exact behavior is best described at this site, which I found while Googling for a solution:

http://jira.public.thoughtworks.org/browse/CCNET-1258?page=com.atlassian.jira.plugin.system.issueta bpanels:all-tabpanel

Others in this thread had builds freezing on steps, whereas our builds complete, but the EXE never closes. This seems to have only become an issue in the last few weeks... Unfortunately I can't track down any specific changes in our build environment (Windows Updates could be the culprit).

[score]

Ok, I upgraded to v6.7a and it has hung twice in two runs. So know I will do the other steps. I got to step 2 which is adding the
Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000

Is this setting how long to wait for a process to end? is it in minutes or seconds? Some of the processes take over an hour to run. Could that be it?

It does seem to always hang after a long running step.

thanks



Originally posted by kinook
We have not been able to reproduce the problem (and not for a lack of trying), but as far as we know, one or more of the following resolved the problem for others who have reported it:

1) Updating to v6.7a

2) Adding a Run Script (VBScript) step like this at the beginning of with this code:

Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000

3) Removing the VBP DEP exclusions (Control Panel -> System -> Advanced -> Data Execution Prevention -> Turn on DEP for all programs and services except those I select, Remove Visual Build Professional and Visual Build Professional 6 Console App from the list, OK, and restart Windows) and then configuring Windows to 'Turn on DEP for essential Windows programs and services only' (in that same dialog) and restart again.

4) Adding a Wait step (5 seconds) at the end of a chained (child) build.

If none of those help, please provide the following information:

Are you building from a Scheduled Task, starting manually, or? Any ideas on anything that may have changed when the problem returned? Please ZIP and send:
1) The info from Help | About | Install Info (after updating to v6.7a)
2) A screen shot of System properties (Windows+Break)
3) A reproducible test case, including VBP project(s) that demonstrate the problem.

Thanks.

[kinook]

Originally posted by score
I got to step 2 which is adding the
Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000

Is this setting how long to wait for a process to end? is it in minutes or seconds? Some of the processes take over an hour to run. Could that be it?

It does seem to always hang after a long running step.It is the time, in milliseconds, to wait for output from a called process's standard output pipe. This wait occurs in a loop until the process completes (or the build is canceled). The setting just changes how long to wait within each loop iteration, but one user indicated that it may have made a difference for them.

Looking at the files you submitted in April, it appears the problem occurs when invoking devenv (VS2008) from a Run Program action. Some other things to try if the previous suggestions don't help:

1) Change the Run Program action command to:

%DOSCMD% original cmd here >%TEMP%\out.txt

and Read program output from 'A file' of %TEMP%\out.txt

2) Replace the Run Program action with a Make VS 2008 action (which calls MSBuild instead of devenv by default).

Any information on what might have changed from when it was working and then stopped working (you indicated in April that using the /mta switch w/ v6.7 resolved the problem) or details on how to reproduce the problem (we've run numerous tests with long-running Run Program steps on Windows 2003 SP2 virtual machine guests without any problems) could be helpful.

Also, yesterday we made one tweak to VBP v6.7a which could conceivably make a difference. If Help | About | Install Info does not show

VisBuildPro.exe version 6.7.1.1

try re-downloading and installing http://www.kinook.com/Download/VisBuildProEval.exe to get this update.

Also, back in April you indicated:

VisBuildPro.exe" /mta /s /b coleto.bld (no hang)
VisBuildPro.exe" /s /b coleto.bld (no hang)
VisBuildPro.exe" /mta coleto.bld (no hang)
VisBuildPro.exe" coleto.bld (hangs)

Is that still the case, or under what conditions does it now hang?

Thanks.

[score]

The step that consistantly fails is after all the devenv/msbuild lines are built. One thing we build is a database tool and we execute that which takes about 1.5 hours. The process seems to run to completion. I say this because built tool outputs "BuildPublishBOs completed" to the log and the process is no longer show using ps. But the next step in the .bld file is never run. I'm rerunning the build with the added

Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000


lines, but after your explaination, i don't hold out much hope. I can upgrade to your latest tweek and run again. Turn around for me is about 4 hours to get to this point.

I'll let you know what happens.

[score]

Ok, It actually went through the whole build when I added the statement:

Application.Options.RunProgramOutputWaitTimeout = 1000
Application.Options.RunProgramInputWaitTimeout = 1000

which was exciting. So I ran it again this morning, but it hung one step after the last hang. So it seems like it didn't "fix" it. May have moved it.

I'll try the next two steps. However I am not sure what is meant but a "chained" child. (other than what i would like to do to my 2 yr old sometimes :). Is a chained child a subsequent call to another .bld file? If so I don't have one so wont do step 4.

doing step 3:

3) Removing the VBP DEP exclusions (Control Panel -> System -> Advanced -> Data Execution Prevention -> Turn on DEP for all programs and services except those I select, Remove Visual Build Professional and Visual Build Professional 6 Console App from the list, OK, and restart Windows) and then configuring Windows to 'Turn on DEP for essential Windows programs and services only' (in that same dialog) and restart again.

4) Adding a Wait step (5 seconds) at the end of a chained (child) build.

[kinook]

If it's a Run Program step that is hanging, try changing the command to:

%DOSCMD% original cmd here >%TEMP%\out.txt

and Read program output from 'A file' of %TEMP%\out.txt

Also, back in April you indicated:

VisBuildPro.exe" /mta /s /b coleto.bld (no hang)
VisBuildPro.exe" /s /b coleto.bld (no hang)
VisBuildPro.exe" /mta coleto.bld (no hang)
VisBuildPro.exe" coleto.bld (hangs)

Is that still the case, or under what conditions does it now hang?

Are you able to reproduce a hang with the attached project (or some variation of it)?

[score]

Ok, ran the test.zip project a few times and it did not hang. I also did set 3) "Removing the VBP DEP exclusions " and it still hung.

As stated early, i do not know what is meant by a "chained" child. However I don't think i have that.

With the latest version of vbp (67a) I was not using the /mta switch. I can give that a try.

Is there any way I can get you more information so you can debug this? Do you output any debug log? or get a call stack or perhaps let you log into this system via webex

[score]

I tried it with the /mta option on the command line and it didn't help. Still hung.