Home > BizTalk Server > BizTalk Disaster Recovery Planning

BizTalk Disaster Recovery Planning

I agree with Nick Heppleston’s post entirely in that you had better test your DR plan in advance of the real disaster if you really want it to work!  I can speak from experience here – my company’s previous plan sounded perfectly fine to me when I worked on it.  In fact, when I discussed the plan with Tim Wieman and Ewan Fairweather they also agreed that it sounded reasonable (to their credit they didn’t get to see a written plan and they both cautioned that I had better test things to be sure).  But, as you can probably already guess, when tested, the plan didn’t work.  In this post I’ll talk about what I did (that didn’t work) so that you can avoid doing the same thing, and then point out a shortcoming in the MSDN documentation, as I see it.

Here’s what I attempted to do.  Since our test/QA environment resides in another data center, far away from our production environment, we decided to use the test environment as our DR environment in the case of a lasting failure in the normal production site.  In a few words, we were planning for a scenario where we’d lose both the BizTalk application servers and the  SQL Server databases.  We thought this was reasonable since many disasters could easily prevent both sets of hardware to stop functioning for an extended period of time.

Here’s a high-level overview of the steps we had in place before the disaster:

  1. We had configured regular backups with transaction log shipping on the destination (DR) database server.
  2. We had the IIS web directories and GAC of the BizTalk application servers backed up to the DR BizTalk application servers.
  3. We had a backup of the master secret available on the DR BizTalk application servers.

Here’s a high-level overview of the steps we had after the disaster:

  1. Recover the production binaries and production web directories onto the test servers.
  2. Restore the production database using the DR instance.
  3. Unconfigure the test BizTalk application servers.
  4. Run the UpdateDatabase.vbs script and UpdateRegistry.vbs script on the test BizTalk application servers.
  5. Reconfigure the test BizTalk application servers, joining the SSO and BizTalk Groups.
  6. Recover Enterprise Single Sign-On.

In early November, we tested the scenario.  To make a long story short, it didn’t work.  Microsoft, also uncertain why things didn’t work (at first anyway), suggested a “hack” to get things working.  That hack would then be reviewed by the product team, which would hopefully provide their final blessing.  The hack involved editing the SSOX_GlobalInfo table (part of the SSODB), updating gi_SecretServer with the “new” test server that would become the master secret server.  It appeared that it worked at first, but a deeper look into things showed that we were wrong (it only appeared to be working because the real production master secret server had been brought up by that point – since the DR exercise was taking too long – so that both BizTalk groups were running at the same time).

The problem that we faced can be summarized as follows: the MSDN documentation makes an assumption, which wasn’t true in our case.  Specifically, Microsoft expects the new master secret server (the test BizTalk app server in our case), to have the same name as the previous master secret server OR for the new master secret server to already be a part of the group prior to the disaster (of course it wouldn’t have been the master secret server prior to the disaster).  These limitations don’t really have anything to do with BizTalk, but rather with the Enterprise Single Sign-On product.  I think they are unrealistic expectations, but limitations nonetheless.

In conclusion, you had better test out your DR scenario if you haven’t already, or be prepared for some ugly surprises.  Please share your experiences with me.  What does your DR plan assume?  Have you had a nasty experience?

Advertisements
Categories: BizTalk Server
  1. January 7, 2010 at 6:55 am

    Hi Victor,

    Some good points here that I hadn’t picked-up – thanks for the post!

    Nick.

  2. Sean
    August 14, 2010 at 11:19 am

    sso is the weak link in biztalk… Forces you to think of one node as special.

    S

  3. Derek
    September 29, 2010 at 11:25 am

    SSO continues to be a pain to work with. After discussions with Microsoft, we setup 2 different Biztalk groups. One in each datacenter and was going to log ship from one to the other. As we were implementing this, we started to raise questions on how this could work. We spoke to different Microsoft people who told us that both sets of servers in both datacenters must be in the same Biztalk group, including all in the same SSO group.

    Otherwise the encryption key is different and you have issues. So, now we have 4 application servers (two in each datacenter) pointing to 1 sql cluster in the primary DC. We log ship to the backup SQL cluster in to 2nd DC. To recover from a disaster, we recover the DB’s on the backup sql cluster and point the 2 remaining application servers in the same DC to the backup sql cluster.

  4. Martin Bring
    May 23, 2011 at 6:38 am

    I am trying to test recovery from a disaster, but feel the SSO part is lika Catch 22. I cannot get it to work.

    We have a DB cluster where SSO is also a clustered service, just like the documentation recommends.

    I want to be able to recover from a DB/SSo cluster disaster. So the we have a third DB server for log shipping. On that server we also have a “waiting” SSO service to take over as Master.

    The problem is to make that server the SSO Master.

    We cannot run “ssomanage -updatedb NewServer.xml” if the old DB is down. We cannot “ssoconfig -restoresecret secret.bak”, since it is not the Master.

    How did you do step 6, 6.Recover Enterprise Single Sign-On? Is it possible to have our setup?

    Regards
    Martin Bring

    • May 31, 2011 at 10:02 am

      I have sent you an email with some additional information.

      • December 27, 2011 at 3:37 pm

        We are also not able restore the SSO in DR using “ssoconfig -restoresecret secret.bak”.
        do you able to confiigure SSO in DR?

  5. Andy Dang
    November 11, 2012 at 4:44 pm

    Hi Fehlberg,

    Thanks for this great tip. However, we have issues in planning for Step 6. Recover Enterprise Single Sign-On? as mentioned by Martin in earlier post. Could you please share me how this is done?

    Thanks,
    Andy Dang
    6.Recover Enterprise Single Sign-On.

    • November 12, 2012 at 9:46 am

      Hi Andy,

      I’ll forward you the message I sent to Martin.

      Thanks,

      Victor

  6. venu
    December 8, 2012 at 10:09 am

    Hi Fehlberg,

    Our client machine configuration has 2 PROD App servers(Load Balancing Cluster), 2 PROD SQL servers(Failover Clustered), 1 DR App server, 1 DR Sql Server.

    1.Can we use the standard Log shipping steps to configure Log shipping in this kind of set up? When we run the bts_ConfigureBizTalkLogShipping stored proc , what should we give for nvcMgmtServerName parameter ? is it SQL cluster name ?

    2. My ex-colleague has already Log shipping with all the host instances created on PROD copied to DR site. We want to bring DR online , shut down PROD servers and test the DR.We stopped the Log shipping jobs, restored the databases, changed the SSO master secret server, changed SSO server name in all databases and registries. Still we see the host instances pointing to the PROD server names.

    Please suggest us on how to bring DR online when PROD goes down in case of disasters.

    Hope to see your reply.

    Thanks
    Venu

    • December 10, 2012 at 11:13 am

      I’ll send an email to you with more information that will help.

  7. Dan
    July 28, 2013 at 8:18 pm

    Hi Victor

    We are putting together a BTS 2013 HA + DR environment. Any updated information you have that can help would be much appreciated!

    Cheers
    Dan

    • July 29, 2013 at 9:39 am

      I don’t have anything updated for BizTalk 2013. By the time my company implements something we’re usually one version behind from the get-go. 😦

  8. Alli
    July 30, 2013 at 10:17 am

    Victor can you make the answers to the question public?

  9. HugoPB
    November 14, 2013 at 3:34 am

    Hi Victor,

    I’m planning Biztalk 2010 Disaster Recovery (DR) with this configuration:
    – Production: 2 BTS Servers with Network Load Balancing (NLB) and 1 SQL Cluster
    – DR: 2 BTS Servers with Network Load Balancing (NLB) and 1 SQL Cluster

    The SQL team is configuring the Log Shipping according to the Microsoft instructions.
    My plan is to have the DR servers in the same Biztalk Group with services disable, and in case of disaster:
    – Shut Down Production BTS and SQL
    – Stop Log shiping and recover the DB in DR
    – Point the DR BTS servers to the DR Databases
    – Activate the BTS services in DR

    I have some questions, about SSODB:
    – Can i do log shipping of this DB to DR and restore it in DR?
    – How can i change the Master Secret server to the DR infrastructure and DB?
    – If the BTS are configured in the same Biztalk Group as production, i must use the same Master Secret SSODB? Or i can create a new one in DR?

    Is this plan correct or do you suggest other strategy?

    Best regards
    Hugo

    • November 14, 2013 at 9:32 am

      Hi Hugo,

      The actual process is quite complicated and more complex than I think you realize. I’m willing to share my actual procedures though. Let me see if I can copy and paste successfully here. Okay, that kind of worked. Formatting isn’t perfect, but hopefully this will help you. You’ll notice that the SSO DB is indeed log shipped, and you’ll see how we restore the master secret.

      4.0 GENERAL
      4.1 Abbreviations
      Abbreviation Definition
      DBA SQL Server Database Administrator
      DR Disaster Recovery
      MSI Microsoft Installer
      SO System Owner
      4.2 Definitions
      Term Definition

      5.0 PROCEDURES PRIOR TO THE DISASTER
      5.1 Setup a Scheduled Task to Backup the GAC and Web Apps – SYSADMIN
      5.1.1 On dr-server-01 and dr-server-02, ensure that the Global Assembly Cache (GAC) of prod-app-server-01 is being backed up to the D:\Disaster Recovery folder. The run as account must have administrative access to dr-server-01, dr-server-02 and prod-app-server-01. A command like this can be used:
      xcopy “\\prod-app-server-01\c$\windows\assembly\gac_msil” “d:\Disaster Recovery\prod-app-server-01 GAC Backup\” /S /C /Y
      5.1.2 Similarly, ensure a backup of the IIS web applications on prod-app-server-01 is taking place daily on dr-server-01 and dr-server-02. Put the backup inside the D:\Disaster Recovery folder. The run as account must have administrative access to dr-server-01, dr-server-02 and prod-app-server-01. A command like this can be used:
      xcopy “\\prod-app-server-01\c$\Inetpub\wwwroot” “d:\Disaster Recovery\IIS Backup\” /S /C /Y
      5.2 Configure the SQL Server Job to Backup BizTalk Server – DBA
      5.2.1 On db-server\biztalkprd_r2, click Start, click Programs, click Microsoft SQL Server 2005, and then click SQL Server Management Studio.
      5.2.2 In the Connect to Server dialog box, specify db-server\biztalkprd_r2 and the appropriate authentication type, and then click Connect.
      5.2.3 In SQL Server Management Studio, double-click SQL Server Agent, and then click Jobs.
      5.2.4 In the details pane, right-click Backup BizTalk Server (BizTalkMgmtDb), and then click Properties.
      5.2.5 In the Job Properties – Backup BizTalk Server (BizTalkMgmtDb) dialog box, under Select a page, click Steps.
      5.2.6 In the Job step list, click BackupFull, and then click Edit.
      5.2.7 On the General page, in the Command box, edit the command, and then click OK. The syntax for the command is:
      exec [dbo].[sp_BackupAllFull_Schedule] ‘d’ /* Frequency */, ‘BTS’ /* Name */, ” /* location of backup files */, 0 (default) or 1 /* ForceFullBackupAfterPartialSetFailure */, ‘BackupHour’ /* local time hour for the backup process to run */
      E.g.
      exec [dbo].[sp_BackupAllFull_Schedule] ‘d’, ‘BTS’, ‘M:\MS_SQL_Backups\BizTalkPRD_R2\BizTalkRepl’
      5.2.8 In the Job step list, click MarkAndBackupLog, and then click Edit.
      5.2.9 On the General page, in the Command box, edit the command, and then click OK. The syntax for the command is:
      exec [dbo].[sp_MarkAll] ‘BTS’ /* Log mark name */, ” /* location of backup files */ 0 (default) or 1 /* use local time stamps for the backup log */
      E.g.
      exec [dbo].[sp_MarkAll] ‘BTS’, ‘M:\MS_SQL_Backups\BizTalkPRD_R2\BizTalkRepl’
      5.2.10 In the Job step list, click Clear Backup History, and then click Edit.
      5.2.11 On the General page, in the Command box, change DaysToKeep= to the number of days you want to keep the backup history, and then click OK twice to close the Job Properties – Backup BizTalk Server (BizTalkMgmtDb) dialog box.
      E.g. exec [dbo].[sp_DeleteBackupHistory] @DaysToKeep=14
      5.2.12 In the details pane, right-click the Backup BizTalk Server job, and then click Enable.
      Note: In the Enable Jobs dialog box, the status changes to Success.
      5.3 Configure Transaction Log Shipping on the Destination (DR) Server – DBA
      IMPORTANT: If you move the full or log backups for a source database from the file path location in which the Backup BizTalk Server job put them (e.g. source database uses d:\log backups and DR server uses e:\logs), you should update the associated row for that database in the bts_LogShippingDatabases table on the destination (DR) system by setting the LogFileLocation or DBFileLocation to the new location where the destination system should read the full/log backup files. This table is populated when you run the bts_ConfigureBtsLogShipping stored procedure. By default, these columns are set to null, which indicates that the destination system should read the backup files from the location stored in the adm_BackupHistory table.
      5.3.1 On dr-db-server\biztalkprd_r2, click Start, click Programs, click Microsoft SQL Server 2005, and then click SQL Server Management Studio.
      5.3.2 In the Connect to Server dialog box, specify dr-db-server\biztalkprd_r2, and then click Connect to connect to the appropriate SQL Server.
      5.3.3 In SQL Server Management Studio, click File, click Open, and then click File.
      5.3.4 In the Open File dialog box, browse to the following SQL script:
      D:\Program Files\Microsoft BizTalk Server 2006\Schema\LogShipping_Destination_Schema.sql
      5.3.5 Click the Query menu, and then click Execute.
      NOTE: The LogShipping_Destination_Schema script drops and recreates the tables used for restoring the source databases on the destination system. This includes tables to store the list of databases being recovered, copies of the backup history imported from the source system’s BizTalkMgmtDb database, and information about SQL Server Agent jobs configured to run against the source databases.
      5.3.6 In SQL Server Management Studio, click File, click Open, and then click File.
      5.3.7 In the Open File dialog box, browse to the following SQL script:
      D:\Program Files\Microsoft BizTalk Server 2006\Schema\LogShipping_Destination_Logic.sql
      5.3.8 Click the Query menu, and then click Execute.
      5.3.9 In SQL Server Management Studio, click New Query.
      5.3.10 In the query window paste the following command:
      exec bts_ConfigureBizTalkLogShipping @nvcDescription = ‘BizTalk_ShippingSolution’,
      @nvcMgmtDatabaseName = ‘BizTalkMgmtDb’,
      @nvcMgmtServerName = ‘db-server\biztalkprd_r2’,
      @SourceServerName = null, — null indicates that this destination server restores all databases
      @fLinkServers = 1 — 1 automatically links the server to the management database
      Important: Before you execute this statement, you must enable the Ad Hoc Distributed Queries configuration option on the destination system.
      5.3.11 Click the Query menu, and then click Execute.
      Important: If the query fails, after you fix the problem with the query, you must start over from step 5.3.1 of this procedure to reconfigure the destination system.
      5.3.12 On dr-db-server\biztalkprd_r2, in SQL Server Management Studio, double-click the appropriate server, double-click SQL Server Agent, and then double-click Jobs. In the details pane, you will see three new jobs:
      1. BTS Log Shipping Get Backup History
      Note: The BizTalk Server Log Shipping Get Backup History job moves backup history records from the source to the destination. It is scheduled by default to run every minute. This job runs as frequently as possible in order to move history records from the source to the destination. In the event of a system failure to the source system, the server that you identified as the destination system will continue to process the history records that have already been imported.
      2. BTS Server Log Shipping Restore Databases
      Note: The BizTalk Server Log Shipping Restore Databases job restores backup files for the given databases for the source to the destination server. It is scheduled by default to run every minute. This job runs continuously without completing as long as there are backup files to restore. As an extra precaution, you can run this job an additional time to ensure that it is complete.
      3. BTS Log Shipping Restore To Mark
      Note: The BizTalk Server Log Shipping Restore To Mark job restores all of the databases to a mark in the last log backup. This ensures that all of the databases are in a transactionally consistent state. In addition, this job re-creates all of the SQL Server Agent jobs on the destination system that had been on the source system.
      5.4 Prepare Custom Databases for Backup – DBA
      5.4.1 On prod-app-server-01, build the custom database objects in the old production database, db-server\biztalkprd_r2, by browsing to the D:\Program Files (x86)\Microsoft BizTalk Server 2006\Schema directory, and then run Backup_Setup_All_Procs.sql and Backup_Setup_All_Tables.sql against all the custom databases.
      Note: This creates the necessary procedures, table, and role and assigns permissions to the stored procedures. This step must be run by an account with sufficient permissions on the database.
      5.4.2 Using SQL Server Management Studio, in the BizTalk Management (BizTalkMgmtDb) database, modify the adm_OtherBackupDatabases table to include a row for each of the custom databases.
      Type the new server and database names in the corresponding columns, as shown in the following table.
      Column Value
      DefaultDatabaseName The friendly name of your custom database.
      DatabaseName The name of your custom database.
      ServerName The name of the computer running SQL Server.
      BTSServerName The name of the BizTalk Server. This value is not used, but it must contain a value nonetheless
      Note: The next time the Backup BizTalk Server job runs, the custom databases will be backed up.
      5.5 Configure the DR Application Server Configuration File – SYSADMIN
      5.5.1 Browse to the following folder on the dr-server-01: D:\Program Files (x86)\Microsoft BizTalk Server 2006\Bins32\Schema\Restore
      5.5.2 Right-click SampleUpdateInfo.xml, and then click Edit.
      5.5.3 Replace all instances of “SourceServer” with db-server\biztalkprd_r2, and then replace all instances of “DestinationServer” with dr-db-server\biztalkprd_r2
      Important: Include the quotation marks around the name of the source and destination systems (typical XML notation).
      5.5.4 Uncomment the XML section and uncomment the following BAM databases:
      • OldPrimaryImportDatabase
      • PrimaryImportDatabase
      • ArchivingDatabase
      • StarSchemaDatabase
      5.5.5 Add custom databases under the section, e.g.:

      5.5.6 Save the file and exit.
      Copy SampleUpdateInfo.xml to the other application servers in the group, i.e. dr-server-02, in the same location, D:\Program Files (x86)\Microsoft BizTalk Server 2006\Bins32\Schema\Restore.
      5.6 Backup the Master Secret – SYSADMIN
      5.6.1 Using prod-app-server-01 of the BizTalk Group, click Start, click Programs, click Microsoft Enterprise Single Sign-On, and then click SSO Administration.
      5.6.2 In the scope pane of the ENTSSO MMC Snap-In, expand the Enterprise Single Sign-On node.
      5.6.3 Right-click System, and then click Back up Master Secret.
      5.6.3.1 Save the file using the password for the BizTalk service account, svc-biztalk-host. Set the password reminder to indicate ‘Password for svc-biztalk-host’.
      5.6.3.2 Place the backed up master secret file on dr-server-01 and dr-server-02 and prod-app-server-01 and prod-app-server-02 in the following location: D:\Master Secret\Prod_R2\machine_name_master_secret.bak, replacing italicized names with the appropriate information.
      5.6.3.3 Repeat 5.6.1 through 5.6.3.2 using dr-server-01 the test application server, so that the test environment master secret is backed up on dr-server-01 and dr-server-02 and prod-app-server-01 and prod-app-server-02.
      5.7 Store System DNS setting – BizTalk SO
      5.7.1 Save system DNS configuration setting for each entries
      5.8 Store BizTalk binding files – BizTalk SO
      5.8.1 Store binding files from production environment for each projects in DR site
      5.9 Backup the BAM data maintenance Data Transformation Service packages – DBA
      5.9.1 On dr-server-01, Click start, click Programs, Click Microsoft SQL Server 2005 and run SQL Server management Studio as BizTalk service account.
      5.9.2 Select “Integration Service” as Service Type and type “dr-db-server-2” as server name and click connect.
      5.9.3 Expand MSDB folder under Stored Packages.
      5.9.4 Take backup of the all data maintenance DTS by repeating following steps
      Right click on data maintenance DTS and select export. Select D:\Disaster Recovery\BDOMAIN\TEST DTS backup as Package path.
      5.10 Prepare update reference of BAM data analysis Data Transformation Service packages – DBA
      5.10.1 On prod-app-server-01 server, Click Start, click Programs, click Microsoft SQL Server 2005 and, and then click SQL Server Business SYSADMINligence Development Studio and run as BizTalk service account.
      5.10.2 In SQL Server Business SYSADMINligence Development Studio, create a new project. Click File, click New, and then click Project.
      5.10.3 In the New Project dialog box, in Templates, click Integration Services Project, and then click OK.
      5.10.4 In the Integration Services Project dialog box, in Solution Explorer, right-click SSIS Packages, and then click Add Existing Package.
      5.10.5 In the Add Copy of Existing Package dialog box, in the Server drop-down list box type db-server
      5.10.6 In Package Path, click the ellipses button.
      5.10.7 In the SSIS Package dialog box, select the BAM_AN package, click OK, and then click OK.
      5.10.8 The package is now listed in Solution Explorer.
      5.10.9 In Solution Explorer, double-click the BAM_AN package. In Connection Managers, double-click database number 3 (MSDB database).
      5.10.10 In the Connection Manager dialog box, in the Server name box, enter the name of the MSDB server as dr-db-server/bistalkprd_r2, and then click OK.
      Click the Package Explorer tab, double-click the Variables folder, and then update the values for the primary import server name and start schema server name as dr-db-server\biztalkprd_R2. Update the values for AnalysisServer as dr-db-server.
      5.10.11 Click File, and then click Save All.
      5.10.12 Repeat section 5.9 for all DTS packages with prefix BAM_AN and save resulted BAM_AN…dtsx file to \\dr-server-01\d$\Disaster Recovery\BDOMAIN\DR DTS
      5.11 Prepare update reference of BAM data maintenance Data Transformation Service packages – DBA
      5.11.1 On prod-app-server-01 server, Click Start, click Programs, click Microsoft SQL Server 2005, and then click SQL Server Business SYSADMINligence Development Studio.
      5.11.2 In SQL Server Business SYSADMINligence Development Studio, create a new project. Click File, click New, and then click Project.
      5.11.3 In the New Project dialog box, in Templates, click Integration Services Project, and then click OK.
      5.11.4 In the Integration Services Project dialog box, in Solution Explorer, right-click SSIS Packages, and then click Add Existing Package.
      5.11.5 In the Add Copy of Existing Package dialog box, in the Server drop-down list box type db-server
      5.11.6 In Package Path, click the ellipses button.
      5.11.7 In the SSIS Package dialog box, select the BAM_DM package, click OK, and then click OK.
      5.11.8 The package is now listed in Solution Explorer.
      5.11.9 In Solution Explorer, double-click the BAM_DM package. In Connection Managers, double-click database number 3 (MSDB database).
      5.11.10 In the Connection Manager dialog box, in the Server name box, enter the name of the MSDB server as dr-db-server/bistalkprd_r2, and then click OK.
      5.11.11 Click the Package Explorer tab, double-click the Variables folder, and then update the values for the primary import server name and ArchivingServer as dr-db-server\biztalkprd_R2 .
      5.11.12 Click File, and then click Save All.
      5.11.13 Repeat section 5.9 for all DTS packages with prefix BAM_DM save resulted BAM_DM…dtsx file to \\dr-server-01\d$\Disaster Recovery\BDOMAIN\DR DTS

      6.0 PROCEDURES AFTER THE DISASTER
      6.1 Prepare for the Exercise – DBA
      Note: Only execute this section if this is a DR exercise and not a real disaster.
      6.1.1 Perform a complete backup of dr-db-server-2\biztalktst_r2.
      6.2 Prepare for the Exercise – BizTalk SO
      Note: Only execute this section if this is a DR exercise and not a real disaster.
      6.2.1 On dr-server-01, stop all host instances from the BizTalk Admin Console.
      6.2.2 On prod-app-server-01, start up HAT (Health Activity Monitoring tool) and take screenshot of send and receive query for reference.
      6.2.3 On prod-app-server-01, stop all host instances from the BizTalk Admin Console.
      6.2.4 On prod-app-server-01 and prod-app-server-02 run the following command:
      iisreset /stop.
      6.2.5 Stop the Enterprise Single Sign-On Windows service on prod-app-server-01 and prod-app-server-02.
      6.3 Prepare for the Exercise – SYSADMIN
      6.3.1 Log in to dr-app-server-01 and launch VERITAS Cluster Manager – Java Console.
      6.3.2 Select File, New Cluster…
      6.3.3 In the Host name field, type uswa-tsql-cls01.
      6.3.4 Log in using cluster credentials.
      6.3.5 In the left-hand tree, right-click sqltest01 and select Freeze, Persistent.
      6.3.6 In the left-hand tree, expand sqltest01, expand SQLServer2005, and select sqltest01-SQLServer2005-R2.
      6.3.7 Select the Properties tab and confirm that the Instance states BIZTALKTST_R2.
      6.3.8 Select the Status tab and note down the System Name that is Online. This is the active node in the test/DR cluster.
      6.3.9 Log in to the active node, if not already logged in, and stop the Windows service “SQL Server (BIZTALKTST_R2).”
      6.3.10 Log in to usto-pdbx-sql03 and launch VERITAS Cluster Manager – Java Console.
      6.3.11 Select File, New Cluster…
      6.3.12 In the Host name field, type usto-psql-cls01.
      6.3.13 Log in using cluster credentials.
      6.3.14 In the left-hand tree, right-click sqlbztp01 and select Freeze, Persistent.
      6.3.15 In the left-hand tree, expand sqlbztp01, expand SQLServer2005, and select sqlbztp01-SQLServer2005-R2.
      6.3.16 Select the Properties tab and confirm that the Instance states BIZTALKPRD_R2.
      6.3.17 Select the Status tab and note down the System Name that is Online. This is the active node in the production cluster.
      6.3.18 Log in to the active node, if not already logged in, and stop the Windows service “SQL Server (BIZTALKPRD_R2).”
      6.4 Redirect DNS –Network admin
      6.4.1 Redirect DR (Test) URL : http://biztalkr2test.CompanyName.com
      to Production URL : http://biztalkr2prod.CompanyName.com
      6.5 Ensure Rights of the Service Account – SYSADMIN
      6.5.1 Ensure svc-biztalk-prd is a member of the administrator group on the dr-server-01 and dr-server-02, adding if necessary.
      6.5.2 Ensure svc-biztalk-prd is a member of the administrator group on dr-app-server-01 and dr-app-server-02, adding if necessary.
      6.6 Ensure Right of the database account – DBA
      6.6.1 Ensure svc-biztalk-prd has the server roles: db creator, sys admin, and security admin rights on dr-db-server\biztalkprd_r2, adding as necessary.
      6.7 Ensure the production binaries are in place on the DR servers – BizTalk SO
      Note: Only execute this section if this is a real disaster.
      6.7.1 Drag and drop into the Global Assembly Cache (GAC) of dr-server-01 all of the CompanyName-generated binaries backed up in Section 5.1.1 (the CompanyName-generated binaries are those that start with the word “CompanyName”).
      6.7.2 Copy the backed up IIS folders (the CompanyName-generated web directories, e.g. web sites, etc.) described in Section 5.1.2 to C:\inetpub\wwwroot on dr-server-01, overwriting folders as necessary.
      6.8 Restore the BizTalk Server Production Environment – DBA
      6.8.1 Increase the memory allocated to dr-db-server\biztalkprd_r2 to a minimum of 16GB, if needed.
      6.8.2 If this is for a DR exercise, from usto-psql-vccs02, run windows service BztlkCopyFiles to copy Production database back up file to DR site once and disable this windows service.
      6.8.3 Using a computer with SQL Server Management Studio 2005 installed, click Start, click Programs, click Microsoft SQL Server 2005, and then click SQL Server Management Studio.
      6.8.4 In the Connect to Server dialog box, specify dr-db-server\biztalkprd_r2 and then click Connect.
      6.8.5 In Microsoft SQL Server Management Studio, double-click the appropriate server, double-click SQL Server Agent, and then double-click Jobs.
      6.8.6 In the details pane, right-click and run BTS Log Shipping – Get Backup History
      6.8.7 In the details pane, right-click BTS Log Shipping – Get Backup History, and then click Disable. In the Disable Jobs dialog box, the status changes to Success.
      6.8.8 In the details pane, right-click BTS Log Shipping – Restore Databases, and then click Disable. In the Disable Jobs dialog box, the status changes to Success.
      6.8.9 In the details pane, right-click BTS Log Shipping – Restore To Mark, and then click Start Job at Step….
      6.8.10 When the Start Job on dialog appears, click Step ID 1 (it will be selected by default) and then click Start.
      6.8.11 The Start Job on dialog will close, leaving the Start Jobs – dialog open. This dialog will show progress and status of the running job. When the job finishes, check that the Status is Success, and then click Close.
      Note: If the Status is Error, click the link in the Message field to obtain more information about the nature of the problem. If the job is successful, SQL Server Agent jobs and BizTalk Server databases are restored to the destination (DR) system.
      6.8.12 Run script or using GUI, restore following database
      • BAMArchive
      • BAMAnalysis
      6.9 Unconfigure the BizTalk Application Server – BizTalk SO
      6.9.1 Open BizTalk Server Configuration on dr-server-01.
      6.9.2 Choose the option to Unconfigure Features.
      Note: This step is required because there is not a BizTalk application server that is still part of the database group nor is there an identically-named server that we are recovering on.
      Note: This will put the database pointed at by the BizTalk application servers in a difficult-to-recover state (if not impossible). It is important that the application servers point at the test database (which should be stopped and fully backed up) prior to this step, and not a live database.
      6.9.3 Select all the features and press OK. Finish the wizard.
      6.9.4 Close the BizTalk Configuration wizard.
      6.9.5 Check the Windows services to see if any services begin with “BizTalk”. If so, run sc Delete from the command line where is the name of the service found. Repeat for each service found starting with “BizTalk”.
      6.9.6 If any services were deleted in the previous step, restart dr-server-01.
      6.9.7 On dr-server-01 check HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk Server\3.0\Administration for the presence of the following keys:
      • BizTalkAdminNTGroup
      • BizTalkGroupName
      • BizTalkGroupName
      • MgmtDBServer
      If found, delete the keys.
      6.10 Update Database References – BizTalk SO
      Note: The steps in this section should be executed an administrative account on dr-server-01 that also has db creator, sys admin, and security admin permissions on dr-db-server\biztalkprd_r2, e.g. svc-biztalk-prd.
      6.10.1 On dr-server-01 and dr-server-02 ensure that 5.5 has been completed. If not, do so now.
      6.10.2 On dr-server-01 open a command prompt and navigate to the directory D:\Program Files (x86)\Microsoft BizTalk Server 2006\Bins32\Schema\Restore. Type: cscript UpdateDatabase.vbs SampleUpdateInfo.xml
      Note: This script updates all tables that store information about the location of other databases.
      Note: You only need to run UpdateDatabase.vbs on one server in the BizTalk group.
      6.11 Join the SSO Group – BizTalk SO
      Note: The steps in this section should be executed an administrative account on dr-server-01 that also has db creator, sys admin, and security admin permissions on dr-db-server\biztalkprd_r2, e.g. svc-biztalk-prd.
      6.11.1 Using dr-db-server\biztalkprd_r2, update the following table of the SSODB database by running (and committing) this SQL command:
      update SSOX_GlobalInfo set gi_SecretServer=’DR-SERVER-01’
      6.11.2 On dr-server-01, click Start, click Programs, click Microsoft BizTalk Server 2006, and then click BizTalk Server Configuration.
      6.11.3 Choose Custom Configuration, enter the database server name as dr-db-server\biztalkprd_r2, and enter the service credentials for svc-biztalk-prd. Press Configure.
      6.11.4 In Microsoft BizTalk Server 2006 Configuration, in the console tree, click Enterprise SSO.
      6.11.5 In the details pane, select Enable Enterprise Single Sign-On on this computer, and then click Join an existing SSO system.
      6.11.6 In Data stores, enter dr-db-server\biztalkprd_r2 for the SQL Server name enter SSODB for the name of the SSO database.
      6.11.7 In Windows service, enter the user name and password for the SSO service account, svc-biztalk-prd.
      Note: You can use a different account, but it must be a member of the Single Sign-On Administrators group.
      6.11.8 Press Apply Configuration and finish the wizard.
      6.11.9 Close the BizTalk Configuration wizard.
      6.12 Recover the BizTalk Application Servers – BizTalk SO
      Note: The steps in this section should be executed an administrative account on dr-server-01 that also has db creator, sys admin, and security admin permissions on dr-db-server\biztalkprd_r2 and is a member of the Single Sign-On Administrators group, e.g. svc-biztalk-prd.
      6.12.1 On dr-server-01 click Start, click Run, type cmd, and then click OK.
      6.12.2 At the command prompt, type:
      cd c:\Program Files\Common Files\Enterprise Single Sign-On
      6.12.3 At the command prompt, type:
      ssoconfig -restoreSecret “d:\Master Secret\Prod_R2\prod-app-server-01_master_secret.bak”
      6.12.4 Enter the password when prompted. If correct, the following message appears : The operation completed successfully.
      6.12.5 Open the BizTalk Admin Console, expand BizTalk Server 2006 Administration and right click on BizTalk Group and choose properties.
      6.12.6 Update SSO Server Name to dr-server-01 and press OK.
      6.12.7 Restart the Enterprise Single Sign-On service found in Windows Services.
      6.12.8 Click Start, click Programs, click Microsoft BizTalk Server 2006, and then click BizTalk Server Configuration.
      6.12.9 Click Group. Add a checkmark to Enable BizTalk Server Group on this computer and select the radio button to Join an Existing BizTalk Group.
      6.12.10 Enter the server name as dr-db-server\biztalkprd_r2.
      6.12.11 Click on Business Rules engine. Add a checkmark to Enable the feathers. Verify the Server Name is correct, updating as necessary.
      6.12.12 Update the Rule Engine Update Service to run as svc-biztalk-prd.
      6.12.13 Click on BAM Tools. Add a checkmark to Enable BAM Portal. Select the checkbox next to Enable Analysis Services for BAM aggregations. Ensure that the Server Name for BAM Primary Import Database, BAM Archive Database, BAM Star Schema Database is set to dr-db-server\biztalkprd_r2, updating as necessary.
      6.12.14 Ensure that the Server Name of the BAM Analysis Database is set to dr-db-server, updating as necessary.
      6.12.15 Click on BAM Portal. Ensure the BAM Management Web Service user and BAM Application Pool Account is set to svc-biztalk-prd.
      6.12.16 Ensure the BAM Portal Users is set to Everyone.
      6.12.17 Press Apply Configuration and finish the wizard.
      6.12.18 Close the BizTalk Configuration wizard.
      6.12.19 On dr-server-01, restart all of the Windows services listed below (if they exist):
      • BizTalk Base EDI Service
      • BizTalk Service BizTalk Group:
      • Enterprise Single Sign-On Service
      • Rule Engine Update Service
      • Windows Management Instrumentation
      6.13 Update references to the BAM databases – BizTalk SO
      Note: The steps in this section should be executed only for the real disaster situation
      6.13.1 On dr-server-01, update D:\Program Files (x86)\Microsoft BizTalk Server 2006\BAMPortal\BamManagementService\Web.Config:

      Set the value of BamServer to dr-db-server\biztalkprd_r2 and set the value of BamDatabase to BAMPrimaryImport, as necessary.
      6.13.2 Update D:\Program Files (x86)\Microsoft BizTalk Server 2006\BAMPortal\BamQueryService\Web.Config:
      Set the value of BamServer to dr-db-server\biztalkprd_r2 and set the value of BamDatabase to BAMPrimaryImport.
      6.13.3 Open the BizTalk Admin Console, expand BizTalk Server 2006 Administration, expand BizTalk Group, expand Platform Settings and highlight Host Instances. Restart all host instances.
      6.13.4 Stop all host instances.
      6.13.5 In Microsoft SQL Server Management Studio, click Connect.
      6.13.6 Click Integration Services and Type uswa-tapp-vcs02 as a server name, double-click Stored Packages, click MSDB, right-click the BAM_DM package, and then click Import Package.
      6.13.7 In the Import Package dialog box, in Package location, select File System.
      6.13.8 In Package Path, navigate to your saved project in D:\Disaster Recovery\BDOMAIN\DR DTS, select the BAM_DM*.dtsx file, and then click Open.
      6.13.9 Click inside the Package Name box to automatically populate the box.
      6.13.10 Click OK, and then click Yes to overwrite.
      6.13.11 Repeat 6.12.5 to 6.12.9 for all BAM_DM*.dtsx files available in file location
      6.13.12 Click Integration Services, double-click Stored Packages, click MSDB, right-click the BAM_AN package, and then click Import Package.
      6.13.13 In the Import Package dialog box, in Package location, select File System.
      6.13.14 In Package Path, navigate to your saved project in D:\Disaster Recovery\BDOMAIN\DR DTS, select the BAM_AN*.dtsx file, and then click Open.
      6.13.15 Click inside the Package Name box to automatically populate the box.
      6.13.16 Click OK, and then click Yes to overwrite.
      6.13.17 Repeat 6.12.11 to 6.12.15 for all BAM_AN*.dtsx files available in file location

      6.14 Review Configurations – BizTalk SO
      Note: Only execute 6.13.5, 6.13.6, 6.13.7 and 6.13.8 if this is a real disaster.
      6.14.1 Run Microsoft BizTalk Server 2006 Configuration on dr-server-01 and verify the values set for each of the screens, correcting values if required.
      6.14.2 Verify that all of the BizTalk-related services in the Windows Services console are running as svc-biztalk-prd, updating as needed.
      6.14.3 Verify that each of the application pools in IIS are running as the updated svc-biztalk-prd, or other valid account, updating as necessary.
      6.14.4 Use the BizTalk.Tools.SSOStorage.exe tool to browse each of the production applications and update references from the old production server(s) to the new DR server(s), as needed.
      6.14.5 On dr-server-01, update D:\Program Files (x86)\Microsoft BizTalk Server 2006\BTSNTSvc.exe.config and D:\Program Files (x86)\Microsoft BizTalk Server 2006\BTSNTSvc64.exe.config as needed, to change references from the old production server(s) to the new DR server(s).
      6.14.6 Use the BizTalk Admin Console to view all Send Ports under All Artifacts. Update any references from the old production server(s) to the new DR server(s), creating folders, HTTP information, retype password etc. as necessary. If needed use Production back up binding files.
      6.14.7 Use the BizTalk Admin Console to view all Receive Locations under All Artifacts. Update any references from the old production server(s) to the new DR server(s), creating folders, HTTP information, retype password etc. as necessary. If needed use Production back up binding files.
      6.14.8 Update System DNS with Production information as necessary.
      6.15 Recreate the host instances – BizTalk SO
      6.15.1 Delete all existing host instances
      6.15.2 Instantiate each host on dr-server-01 using the BizTalk Admin Console.
      6.15.3 Start the host instances associated with dr-server-01 using the BizTalk Admin Console.
      Note: At this point dr-server-01 is now ready for use (one server of the BizTalk production environment has been restored). For DR exercises the RTO can be recorded.
      6.16 Notify the Application Owners – BizTalk SO
      6.16.1 For those applications with components deployed into BizTalk, notify the system and business owners that BizTalk has failed over to the DR site. Note: Load balancers used by those applications will require host name changing.
      6.17 Restore the Secondary Server – BizTalk SO
      Note: For all steps below, repeat the step indicated; however, for all references to dr-server-01 in those sections, use dr-server-02 instead.
      6.17.1 Repeat 6.5 using dr-server-02.
      6.17.2 Repeat 6.7 using dr-server-02.
      6.17.3 Repeat 6.9.2 – 6.9.9 using dr-server-02.
      6.17.4 Repeat 6.10.6 – 6.10.15, then 6.10.18 – 6.10.20 using dr-server-02.
      6.17.5 Repeat section 6.11 using dr-server-02.
      6.17.6 Repeat section 6.12.1 – 6.12.3, then 6.12.5 using dr-server-02.
      6.17.7 Repeat section 6.13 using dr-server-02.
      7.0 COMPLETING THE DISASTER EXERCISE
      If this is a DR exercise (and not a real disaster) this section should be executed.
      7.1 Restore the Original Environments – SYSADMIN
      7.1.1 On dr-server-01 and dr-server-02, remove svc-biztalk-prd service account from the administrator group.
      7.1.2 Start the SQL Server and SQL Server Agent Windows services on the active node corresponding to dr-db-server-2\biztalktst_r2.
      7.1.3 Unfreeze the Veritas Service Group sqltest01.
      7.1.4 Start the SQL Server and SQL Server Agent Windows services on the active node corresponding to db-server\biztalkprd_r2.
      7.1.5 Unfreeze the Veritas Service Group sqlbztp01.
      7.2 Restore the Original Environments – DBA
      7.2.1 Using a computer with SQL Server Management Studio 2005 installed click Start, click Programs, click Microsoft SQL Server 2005, and then click SQL Server Management Studio.
      7.2.2 In the Connect to Server dialog box, specify dr-db-server\biztalkprd_r2 and then click Connect.
      7.2.3 In Microsoft SQL Server Management Studio, double-click the appropriate server, double-click SQL Server Agent, and then double-click Jobs.
      7.2.4 Delete newly created SQL jobs and only leave SQL jobs needed for DR site. BTS Log Shipping – Get Backup History, BTS Log Shipping – Restore Databases and BTS Log Shipping – Mark and Restore
      7.2.5 Run SQL stored procedure called BTS_LogShippingRestoreFull (Script name is BTS_LogShippingRestoreFull.sql)
      7.2.6 Enable windows service BztlkCopyFiles from usto-psql-vccs02
      7.2.7 In the details pane, right-click BTS Log Shipping – Get Backup History, and then click Enable.
      7.2.8 In the details pane, right-click BTS Log Shipping – Restore Databases, and then click Enable.
      7.2.9 Disconnect from dr-db-server\biztalkprd_r2 and connect to the server usto-psql-vcs01\biztalkprd_r2. Double-click SQL Server Agent, and then double-click Jobs.. In the details pane, right-click the Backup BizTalk Server job, and then click Enable.
      7.3 Restore DNS –Network admin
      7.3.1 Restore URL redirection to original value
      DR (Test) URL : http://biztalkr2test.CompanyName.com
      Production URL : http://biztalkr2prod.CompanyName.com

      7.4 Restore the Original Test environment – BizTalk SO
      Note: The steps in this section should be executed an administrative account svc-biztalk-host

      7.4.1 Stop the Enterprise Single Sign-On Windows services on dr-server-01 and dr-server-02.
      7.4.2 On dr-server-01 and dr-server-02 open a command prompt and navigate to the following directory: D:\Program Files (x86)\Microsoft BizTalk Server 2006\Bins32\Schema\Restore
      Type: cscript UpdateRegistry.vbs SampleUpdateInfo.DRtoTest.xml
      Note: This step must run on all BizTalk servers in the group.
      7.4.3 On dr-server-01 and dr-server-02 reconfigure all Windows services to run as domain\svc-biztalk-host (and not domain\svc-biztalk-prd).
      Eg.
      Enterprise Single Sign-On
      Business Rule Engine
      7.4.4 On dr-server-01 and dr-server-02 restart the Windows service Enterprise Single Sign-On.
      7.4.5 On dr-server-01 click Start, click Run, type cmd, and then click OK.
      7.4.6 Navigate to C:\Program Files\Common Files\Enterprise Single Sign-On and run the command:
      ssoconfig -restoreSecret “d:\Master Secret\Test_R2\dr-server-01_master_secret.bak”
      7.4.7 Enter the password when prompted. If correct, the following message appears: The operation completed successfully
      7.4.8 Restart the Enterprise Single Sign-On service found in Windows Services on dr-server-01 and dr-server-02.
      7.4.9 On dr-server-01 expand Host instances within the BizTalk Admin Console. Delete any host instances with status unavailable.

      If this step is executed during DR exercise and one of the test server is not available from the BizTalk admin console, ask SQL DBA to restore BizTalkMgmt database SQL database backup.

      7.5 Restore the Original Prod environment – BizTalk SO
      7.5.1 On prod-app-server-01 and prod-app-server-02, start following items:
      • Enterprise Single Sign On (Windows service)
      • All host instances (can be run from either machine – seen in the BizTalk Admin Console).
      7.5.2 On prod-app-server-01 and prod-app-server-02, run the following command:
      iisreset /start

      • HugoPB
        November 14, 2013 at 11:44 am

        Thanks Victor!!
        I’m have a detailed plan (with some open subjects, like the ENTSSO), i’m reviewing my plan to see if it’s according to yours.

        I will get back here to share my plan and the obstacles in the implementation.

  10. HugoPB
    November 15, 2013 at 3:07 am

    Hi Victor,

    In the step by step the paths are for BTS 2006 and SQL 2005 (example: “D:\Program Files (x86)\Microsoft BizTalk Server 2006\Bins32\Schema\Restore” and “, Click Microsoft SQL Server 2005”), is the same for BTS 2010 and SQL 2008R2?

    BR Hugo

    • November 15, 2013 at 10:07 am

      I don’t know. I presume it’s very similar but not sure.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: