I looked at many of the other performance monitors hoping for other insight, but didn’t really find much – the host instance wasn’t throttling and other counters seemed normal.

The environment generally operates pretty well – unfortunately most benchmarking and testing done doesn’t always involve running the application for long times with the same load that production will encounter. I’ll admit that this could have been done better. Thanks for the comment Mike.

To address Bembeng’s point, I believe a white paper was written on how to finish an orchestration without risking losing messages that might be queued up – I agree that this is something I’d definitely be sure of before putting a system in production. I’ll find that white paper and add a link. Thanks for the comment.

I actually had a quite similar singleton issue a month before.

My Orchestration acts as a message distribution controller from several receive locations to some orchestrations using direct binding to MsgBox.

However, same like yours, the messages were consumed but never released by the singleton orchestration which I guess by design the messages will only be cleaned up after the instance has been completed.

In load test, we found that by using singleton orchestration with this kind of scenario where we have high freq and large messages, the performance kept decreasing (hourly) and affecting the other orchestrations as well.

In the end, we didn’t proceed with this singleton orchestration and found other solution for our requirements which is having more consistent processing performance.

I have thought a logic to exit (complete) the singleton orchestration, however there is a slight chance that if you have one or more queued messages for the singleton orchestration and we exit (complete) at the current instances. The queued messages may not be consumed until a new instance is created, I may be wrong at this, but the queued messages are not able to summon a new orchestration instances, only new messages can do that.

Nice article, can I add the following comments for anyone reading this as they may also help anyone new to these kind of issues.

1. This is one of the many examples of why monitoring software such as MOM with a good BizTalk management pack is so important. If you had MOM in place here I would expect it to detect this problem situation (probably from the spool size?) and warn you that this was happening

2. When troubleshooting these kind of problems one of the first things to check are the Message Agent and Host/Messagebox performance counters. This can give you a decent picture of what is happening in your BizTalk environment and help you work out what is happening over a period of time. Very useful in these cases

3. There is lots of info around these days relating to performance and ways of optimising BizTalk, but probably the first place to check is your orchestration/code. Also in my experience if you have done some benchmarking on the environment before you start you can mitigate a lot of the potential problems with your infrastructure setup.