Archive

Archive for June, 2008

Using MOM to Restart a BizTalk Server 2006 Host Instance Based on a Problem

June 18, 2008 1 comment

A project that was deployed at work has a terrible tendency to freeze a host instance. Once the problem occurs, the host instance sits in a zombie state until it is restarted. The causes, which can be credited to 2 or 3 different bugs (out of our direct control), are being investigated (some are described in my earlier posts), but we have at least 1 or 2 out there that haven’t yet been solved (and may not be). So, we’d like the known errors to trigger a Microsoft Operations Manager (MOM) alert, which would have one of its responses be the execution of a script.

This sounds simple, right? Well, not really. I started with the RestartBizTalkHostInstances.vbs script installed with BizTalk Server (e.g. C:\Program Files\Microsoft BizTalk Server 2006\SDK\Samples\ApplicationDeployment\VisualStudioHostRestart). Running the script didn’t work. So I then did some searching and found this helpful article. It taught me a thing or two, and I modified the script to avoid certain statements, like WScript.Echo. Great.

But the script still didn’t work. This is where things get interesting. It turns out that MOM runs on the BizTalk Server (or any server for that matter) as the “Local System”. So, to test a script, or find out what’s really wrong with it (other than some useless error in your event log), you need more information. You can start out by running a command like this, ‘at “09:37” /interactive cmd.exe’. This command, as written, will open up a command prompt at 9:37am that can be used to run other commands. In this example, 9:37 happened to be 1 minute ahead of the then-current time. After the new command line opens, you can now run scripts simulating the credentials MOM uses (this simulates the “local system” credentials, so potentially knowing this might help you in other problems you may have in the future). In my case, I received another error, which wasn’t particularly helpful.

So, what I did next was run ‘wbemtest’, which opens up a WMI testing utility. This utility allows you to try executing WMI queries and the like, and instead of doing so via scripting, you have a GUI that aids in the process.

Wbemtest Tool

By pressing ‘Connect’ I can enter the namespace to connect to. In my case the WMI script I’m trying to run shows:

Set objWMIService = GetObject(“winmgmts://./root/MicrosoftBizTalkServer”)

In the GUI you enter something slightly different…

Specify WMI Namespace

After connecting, you can then try to simulate your WMI script. Mine needs to run a query, so I press the Query button and copy from the WMI script:

When pressing “Apply” I got this error:

WMI Script Error

If I press “More Information” and scroll down a bit, I find a property called “Description.” Double clicking it shows this error:

BizTalk Server cannot access SQL server.  This could be due to one of the following reasons:
1. Access permissions have been denied to the current user.  Either log on as a user that has been granted permissions to SQL and try again, or grant the current user permission to access SQL Server.
2. The SQL Server does not exist or an invalid database name has been specified.  Check the name entered for the SQL Server and database to make sure they are correct as provided during SQL Server installation.
3. The SQL Server exists, but is not currently running.  Use the Windows Service Control Manager or SQL Enterprise Manager to start SQL Server, and try again.
4. A SQL database file with the same name as the specified database already exists in the Microsoft SQL Server data folder.

Internal error from OLEDB provider: "Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'."

The interesting part of course is the last section, where it becomes apparent that anonymous credentials are trying to be used to run the script (and of course this is not allowed). So what now?

If you go to the MOM folder in a command window, e.g. C:\Program Files\Microsoft Operations Manager 2005, and type ‘setactionaccount <MOM Mgmt Group> -query’, you will see the credentials used for the action account (if nothing shows up, you are running as local system). You can change this. You can try setting the credentials using the ‘set’ option of the same setactionaccount command. This may or may not work depending on the MOM setup. It didn’t work for me.

The other way to set this, is to reconfigure the primary MOM server configuration (in my case managed by a different department). There you can find the agent computer and specify the action account to be used for running scripts (and the like). After doing this, you should be able to repeat the same comand, ‘setactionaccount <MOM Mgmt Group> -query’ and you should see the new credentials that were set. You may have to wait a minute for the change to take effect.

By the way, if you don’t know the <MOM Mgmt Group> you can find this by looking at the Console Settings of the Administrator or Operator console of MOM.

Once I did all of this, things starting working like a charm! I hope this helps someone out there – let me know.

Advertisements

BizTalk Singleton Orchestration Design

Here’s a white paper from Microsoft on creating sequential FIFO orchestrations (this applies to singletons as well), as a follow up to the blog I wrote a few days ago on poorly written singletons. If you read the paper carefully, and pay attention to the “warning” sections, you’ll see that it’s actually very hard to create an orchestration that neatly ends with zero risk of losing messages. This is because a message might come in after the listen shape, but before the orchestration has terminated.

To test this, I created a simple singleton orchestration that adds a deliberate wait of 2 minutes before finishing.

I then deployed the orchestration, and starting sending in messages one by one (by the way, “Do Something” simply writes the one field in the message to the debugger).

So the message with value 1 was sent first, here’s the debugger output:

12880] Field was: 1

Then the message with value 2 was sent in, here’s the output:
[12880] Field was: 2

Then, i waited about 45 seconds, just long enough to get us past the first Delay shape in the orchestration (of 30 seconds).  I submitted a message with value 3, but received no output (as expected).  I waited until the remaining time had finished and saw the orchestration suspend with this error:

The orchestration was not resumable.  I then sent in a message with value 4, here’s the output:
[12880] Field was: 4

So, what does this prove?  If a message comes in before your orchestration has had the time to complete (and you’re no longer waiting to receive a message), you will have unprocessed messages in the orchestration when goes to complete.  I guess the good news is that you can see what that message was by clicking on the message tab, so if you’re willing to run this risk, you might go ahead with this decent, but not perfect, design.

To be absolutely fail-proof, the paper offers a few suggestions, such as stopping the receive location via wmi script as part of the shutdown process.  This is fine and all, except for this one question – how is it supposed to be turned back on? If you do this as part of the same orchestration, you have the same problem you started with!  I guess the one way this could be done would be by adding a “Start Orchestration” shape, which begins with a delay, and then enables the receive location again (the delay is to allow time for the calling orchestration to finish with no risk of losing messages).

Good luck!

Categories: BizTalk Server

Help w/BizTalk Server 2006/SQL Server Errors

I’d like to see if someone out here has seen any of these errors before… I couldn’t find anything on the internet so far, but I imagine they have a common cause. Ideas?

Severity: Critical Error
Status: New
Source: BizTalk Server 2006
Name: Critical Error: A stored procedure call failed.
Description: The following stored procedure call failed: ” { call [dbo].[bts_GetServiceStaticTrackingInfo]( ?)}”. SQL Server returned error string: “”.
Severity: Critical Error
Status: New
Source: BizTalk Server 2006
Name: Critical Error: A stored procedure call failed.
Description: The following stored procedure call failed: ” { call [dbo].[bts_InsertDynamicStateInfo_BizTalkServerApplication]( ?, ?, ?, ?, ?, ?, ?, ?)}”. SQL Server returned error string: “”.
Severity: Critical Error
Status: New
Source: BizTalk Server 2006
Name: Critical Error: A stored procedure call failed.
Description: The following stored procedure call failed: ” { call [dbo].[bts_GetDynamicStateInfo_BizTalkServerApplication]( ?, ?, ?)}”. SQL Server returned error string: “”.
Severity: Critical Error
Status: New
Source: BizTalk Server 2006
Name: Critical Error: A stored procedure call failed.
Description: The following stored procedure call failed: ” { call [dbo].[bts_CreateSubscription_ACCISHost]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}”. SQL Server returned error string: “Cannot create new connection because in manual or distributed transaction mode.”.
Severity: Critical Error
Status: New
Source: BizTalk Server 2006
Name: Critical Error: A stored procedure call failed.
Description: The following stored procedure call failed: ” { call admsvr_SendPortToPEP( ?)}”. SQL Server returned error string: “”.
Categories: BizTalk Server