FeedAgg.com Logo
Your Account | Sign In | Sign Up

Add Feed | Search | Home | Help | Contact | Blog

Feed: IT Solution Braindumps - AggScore: 16.1



Summary: IT Solution Braindumps


IT solutions and frustrations from real life including Exchange, Sharepoint, TMG, virtualization etc.

Antispam agents in Exchange 2010 SP1 /Hosting mode


Hi,

If you have played with unfortunate Exchange 2010 SP1 /Hosting mode installation or you are a real hoster that runs it in the production, then you maybe had problems with getting Antispam agents to work on your Hub transport server.

This special mode of Exchange 2010 SP1 installation has some limitations when compared to the regular Exchange 2010 installation. One of them is that you cannot use Exchange 2010 Edge Server role with EdgeSync. This means that you will have to use install-AntispamAgents.ps1 Powershell script to enable Antispam agents on your Hub transport servers.

The script works well, transport agents get installed and enabled. But you will soon notice that e-mail messages don't get stamped with SCL even if you have correctly configured the Content Filtering agent. Furthermore, if you have enabled Recipient Filtering agent too, you will see that all recipient addresses, even the existing ones in your tenant organizations will get rejected with "user unknown" message.

This is becase the Antispam agents simply do not work with Exhange 2010 SP1 /Hosting mode installation. Don't let this article fool you as it did me: http://technet.microsoft.com/en-us/library/ff923278.aspx

Here is an example of a behaviour you will notice with Content Filtering agent. The Get-AgentLog Powershell output on your Hub transport will show this:

Normal 0 21 false false false HR X-NONE X-NONE MicrosoftInternetExplorer4
RunspaceId      : 018024ed-1c5a-498a-8c15-087b1c81ed2e
Timestamp       : 1/24/2012 7:07:28 AM
SessionId       : 08CEA824BF825A40
IPAddress       : 111.222.111.222
MessageId       :
P1FromAddress   : Dinko.Fabricni@span.hr
P2FromAddresses : {Dinko.Fabricni@span.hr}
Recipients      : {dfabricni@example.net}
Agent           : Content Filter Agent
Event           : OnEndOfData
Action          : AcceptMessage
SmtpResponse    :
Reason          : SCL
ReasonData      : 0
Diagnostics     :

You can see that agent does its thing, but when you check the message headers in Outlook you will see that actually no SCL is provided:

Normal 0 21 false false false HR X-NONE X-NONE MicrosoftInternetExplorer4
Return-Path: Dinko.Fabricni@span.hr
X-MS-Exchange-Organization-AuthSource: EXHUBEXT01.cloud.local
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-Exchange-Organization-PRD: span.hr
X-MS-Exchange-Organization-SenderIdResult: None
Received-SPF: None (EXHUBEXT01.cloud.local: Dinko.Fabricni@span.hr does not
designate permitted sender hosts)
X-MS-Exchange-Organization-Antispam-Report: SCLNotProvided

To make it even worse, ALL messages, even those that are 100% spam messages will receive SCL rating of 0 and you will see the SCLNotProvided in message headers.

We did raise this incident to Microsoft and got a confirmation that Antispam agents really do not work and that the documentation on Technet website (the article I linked) is misleading.

Date Published:



Writing to ULS log from Sharepoint e-mail event receiver


In case you are trying to use Sharepoint event logging infrastructure and its SPDiagnosticsService class from Sharepoint e-mail event receiver to write to the Windows Event Log or Sharepoint ULS log you might get this error: 

The source was not found, but some or all event logs could not be searched.  Inaccessible logs: Security

Or even worse, if you are not using Visual Studio 2010 debugger, but only rely on Event Viewer, you could see this error in Applications and Services Logs > Microsoft > Sharepoint Products > Shared > Operational:

An error occurred while processing the incoming e-mail file C:inetpubmailrootDrop3692ac8b01cc95a800000006.eml. The error was: Object reference not set to an instance of an object.

As you might know, SPEmailEventReceiver is running in owstimer.exe process which is actually "Sharepoint 2010 Timer" service. This service by default runs under NETWORK SERVICE account which does not have enough permissions to the Security log and this is the reason why SPDiagnosticsService class methods WriteTrace and WriteEvent will fail.

The solution I found is to add the NETWORK SERVICE account to the "Performance Log Users" Windows Security group and then restart the Sharepoint 2010 Timer service.

Date Published: Oct 28, 2011 - 2:37 pm



Use powershell to document your Hyper-V installation


I want the share this little script I wrote that helped me to document Hyper-V R2 cluster installation. This particular installation had more than 30 virtual machines and I had to document virtual machines properties and disk configuration.

The benefit of this script is that it merges output from two powershell commands, Get-VM and Get-VirtualHardDisk.

$vmmserver = Get-VMMServer -ComputerName SCVMMSERVER

$vms =  Get-VM | Sort-object Name | where {$_.HostGroupPath -like "All Hosts*"}

foreach ($vm in $vms)
{
    $disks = $vm | Get-VirtualHardDisk
        foreach($d in $disks) {

        new-object PSObject -prop @{
        VMName = $vm.Name
        Type = $d.VHDType
        "Disk name" = $d.Name
        "Size GB" = [math]::truncate($d.Size / 1GB)
        }

        }

}

The script generates this output:

Size GB  Type                  Disk name               VMName
-------  ----                  ---------               ------
     20  FixedSize             9002D_SERVER1_disk_1    9002D-SERVER1
      9  DynamicallyExpanding  9021D-TESTVM_disk_1     9021D-MGMTSERVER
     40  FixedSize             _CBBMSERVER
     40  FixedSize             HQVM-PRINTSERVER_C      PRINTSERVER
     20  FixedSize             HQVM-PRINTSERVER_D      PRINTSERVER
      6  DynamicallyExpanding  TEST_disk_1             TEST
     40  FixedSize             _CTest
     15  FixedSize             HQVMAC01_HQVMAC01_C     HQVMAC01
     21  FixedSize             HQVMAC01_D              HQVMAC01
     15  FixedSize             HQVMAC02_C              HQVMAC02
     21  FixedSize             HQVMAC02_D              HQVMAC02



With a little modification to the script you could include additional columns in the output. Let's say that you need CPU count and Memory included in the output:

$vmmserver = Get-VMMServer -ComputerName SCVMMSERVER

$vms =  Get-VM | Sort-object Name | where {$_.HostGroupPath -like "All Hosts*"}

foreach ($vm in $vms)
{
    $disks = $vm | Get-VirtualHardDisk
        foreach($d in $disks) {

        new-object PSObject -prop @{
        VMName = $vm.Name
        Type = $d.VHDType
        "Disk name" = $d.Name
        "Size GB" = [math]::truncate($d.Size / 1GB)
        Memory = $vm.Memory
        "CPU Count" = $vm.CPUCount
        }
        }
}

Note that I have added two rows in the script:

        Memory = $vm.Memory
        "CPU Count" = $vm.CPUCount

The output now looks like this:

Size GB   : 9
CPU Count : 1
Memory    : 1024
Type      : DynamicallyExpanding
Disk name : 9021D-TESTVM_disk_1
VMName    : 9021D-SERVER1

Size GB   : 40
CPU Count : 1
Memory    : 1024
Type      : FixedSize
Disk name : _C
VMName    : HQVM-BBM

Size GB   : 40
CPU Count : 1
Memory    : 1024
Type      : FixedSize
Disk name : HQVM-PRINTSERVER_C
VMName    : HQVM-PRINTSERVER

Size GB   : 20
CPU Count : 1
Memory    : 1024
Type      : FixedSize
Disk name : HQVM-PRINTSERVER_D
VMName    : HQVM-PRINTSERVER

Size GB   : 6
CPU Count : 4
Memory    : 512
Type      : DynamicallyExpanding
Disk name : TEST_disk_1
VMName    : TEST

Size GB   : 40
CPU Count : 1
Memory    : 1024
Type      : FixedSize
Disk name : _C
VMName    : Test

Size GB   : 15
CPU Count : 1
Memory    : 1024
Type      : FixedSize
Disk name : HQVMAC01_HQVMAC01_C
VMName    : HQVMAC01

Size GB   : 21
CPU Count : 1
Memory    : 1024
Type      : FixedSize
Disk name : HQVMAC01_D
VMName    : HQVMAC01

Size GB   : 15
CPU Count : 1
Memory    : 2048
Type      : FixedSize
Disk name : HQVMAC02_C
VMName    : HQVMAC02

Size GB   : 21
CPU Count : 1
Memory    : 2048
Type      : FixedSize
Disk name : HQVMAC02_D
VMName    : HQVMAC02


This is good, but not exactly the table we expected and it's hard to convert it into a configuration table that we could copy/paste into installation document.

So the solution is to create a function from our script like this:

function Global:Get-VMConfiguration()
{
$vmmserver = Get-VMMServer -ComputerName SCVMMSERVER
$vms =  Get-VM | Sort-object Name | where {$_.HostGroupPath -like "All HostsProduction*"}

foreach ($vm in $vms)
{
    $disks = $vm | Get-VirtualHardDisk
        foreach($d in $disks) {
        new-object PSObject -prop @{
        VMName = $vm.Name
        Type = $d.VHDType
        "Disk name" = $d.Name
        "Size GB" = [math]::truncate($d.Size / 1GB)
        Memory = $vm.Memory
        "CPU Count" = $vm.CPUCount
        }
        }
}
}


And call that function:

Get-VMConfiguration | Format-Table -Property vmname,memory,"cpu count","disk name",type,"size gb"

This would give you a nicely formatted table that you could paste into a Word document.
Or even better, you could use export-csv function:

Get-VMConfiguration | export-csv -Path c:Export.csv
Now you can import this comma separated file into Excel and format it as you wish.

Date Published: Oct 18, 2011 - 1:16 pm


Sharepoint 2010 - Cross site collection navigation


Sharepoint 2010 comes with a pretty good navigation out-of-the-box, but there are still some areas where it's not sufficient and that is navigation that crosses site collection boundaries.

If you are designing a large Sharepoint 2010 site architecture you are probably using site collections because they represent security boundary and can also be placed in a separate content database which allows you to physically segment Sharepoint 2010 content across multiple SQL databases.

Fortunately, there is a pretty simple solution to implement cross site collection navigation which makes navigating accross multiple site collections to feel more like having a site structure in a single site collection.

To achieve this we need to create a custom navigation provider in Sharepoint 2010 web application that pulls its data from xml file which represents our site collection structure. This way we can have flat site collection design where all site collections are created in the same URL path level, but also have them hierarchically organised to better represent our organizational structure.

 Here are the steps to implement custom navigation based on xml file.

Copy/paste the following xml in notepad and name it CustomNavigation.sitemap.


 
   
     
     
   
   
       
       
       
   
 

Now open the Internet Information Services Manager, expand your Sharepoint web application, right-click the _app_bin folder and click Explore.

alt

 Copy the CustomNavigation.sitemap file to this folder.


alt


Now you need to register the new site map navigation provider by modifying the web.config file of your Sharepoint web application.

Open web.config file (it's located one level above from where you pasted CustomNavigation.sitemap) and enter the following line just below the rest of the already registered providers:

   
     
    ....
    ....

       
     
   

Now, maybe you've read on some other blog that you should first copy the following line:

       

and then rename it and just modify siteMapFile path. But in my case this did not work very well. Everything was okay on the root portal site, let's say it's http://sps. But on the http://sps/sites/products I would get wrong relative links: http://sps/sites/products/sites/hardware instead of http://sps/sites/hardware. Modifying the xml file with full URL did not work either so when I tried using the System.Web.XmlSiteMapProvider everything worked as it supposed to do. I guess it's because Sharepoint sees http://sps/sites/products as root of the site and not http://sps. The XmlSiteMapProvider does not have this problem.

The next step is to link our custom navigation provider with the top link navigation bar on the Sharepoint site. We need a Sharepoint Designer 2010 to do this.

Browse to the site collection root site where you want to modify navigation and click on Site Actions > Edit in Sharepoint Designer. Keep in mind that you will have to modify all site collections this way to implement consistent navigation.

We will modify the v4.master so that all pages that are based on v4.master will have the same navigation.

alt

Double click on v4.master and click on "Edit file".

alt

Select the "Split" view then right click on the top link navigation bar and click "Zoom to Contents".

Modify the following:

   
      ID="TopNavigationMenuV4"
      Runat="server"
      EnableViewState="false"
      DataSourceID="topSiteMap"
      AccessKey=""
      UseSimpleRendering="true"
      UseSeparateCss="false"
      Orientation="Horizontal"
      StaticDisplayLevels="2"
      MaximumDynamicDisplayLevels="1"
      SkipLinkText=""
      CssClass="s4-tn"/>
   
       
           
              ShowStartingNode="False"
              SiteMapProvider="SPNavigationProvider"
              id="topSiteMap"
              runat="server"
              StartingNodeUrl="sid:1002"/>
       
   

To look like this:

   
      ID="TopNavigationMenuV4"
      Runat="server"
      EnableViewState="false"
      DataSourceID="topSiteMap"
      AccessKey=""
      UseSimpleRendering="true"
      UseSeparateCss="false"
      Orientation="Horizontal"
      StaticDisplayLevels="1"
      MaximumDynamicDisplayLevels="1"
      SkipLinkText=""
      CssClass="s4-tn"/>
           
              ShowStartingNode="False"
              SiteMapProvider="CustomXmlContentMapProvider"
              id="topSiteMap"
              runat="server"
              StartingNodeUrl="/"/>

Save the v4.master and open your site in a web browser.

Here are the results:

alt

You'll notice that I've also modified the StaticDisplayLevels property from 2 to 1. This represents to which level Sharepoint will display links from the xml file visible on the top link navigation bar and when will the links be placed in a drop-down menu when you hover over the link. The default setting of 2 will place all links from the xml file visible on the top link bar like this:

alt

So there you go, cross site collection navigation as simple as it can be.

Date Published: Oct 18, 2011 - 2:13 am


Sharepoint 2010 - Change Domain Membership


Hello,

Couple of days ago I had to change domain membership of a single server Sharepoint 2010 farm with separate database server. Browsing the Internet I found little data about this and most recommendations were based on backup/reinstall/restore procedure.

However, I thought I would give it a try with plain domain membership change and with switching Sharepoint 2010 service accounts. I encountered many problems and I will mentioned some of them here. Here's the procedure:

  • Perform full backup of Sharepoint 2010 farm from Sharepoint Central Administration
  • Create new service accounts in the destination domain
  • Make sure you know your Sharepoint 2010 farm passphrase which you entered when you initially provisioned the farm
  • Change the domain membership of  SQL database server first (in my case the SQL database was running under LocalSystem account so I had no issues with that)
  • Give the future farm account from the new domain sysadmin permissions to the SQL database engine (actually only security admin and dbcreator permissions are necessary)
  • At this point your Sharepoint is not working
  • Run stsadm -o setconfig db with the -connect switch to connect to your Sharepoint configuration database. You will have to use your new domain farm credentials here.
  • After this step, the Central Administration site should be working, however, your Sharepoint box is still in the old domain. In my case I had the domain trust established between the old and new domains.
  • Create the new Sharepoint managed accounts by selecting the accounts from the new domain: Central Administration > Security > General Security > Configure Managed Accounts
  • Change the service accounts to reflect the newly added managed accounts: Central Administration > Security > Change Service Accounts
  • Add your farm account to the local administrators group on the Sharepoint server
  • Change Sharepoint box domain membership
  • At this point, your Sharepoint sites should be accessible. However, in my case they were not working and I received 404 not found message. I realized that after I reconnected the Sharepoint farm to the configuration database, custom solutions that these sites were using were not available any more. Thankfully, I had a full farm backup and managed to restore only the farm solutions. I redeployed the solutions from the Central Administration and the sites worked!
  • At one point, after a couple of iisresets and server restarts I received "The trial period has expired" error message when I opened the Sharepoint sites. Running Sharepoint Configuration Wizard again solved this issue.
  • Looking at the "Central Administration > Manage services on service" I saw only a couple of services listed while I know there should be more. Running Install-SPService from Powershell re-registered these services. This is important step for Sharepoint Service Applications to work properly.
  • Almost all Service Applications were started and I could access the management pages for them except the two most important ones, User Profile Synchronization service and Search service. No matter what I did I could not fix them or even restore them. I ended up creating and provisioning the new services from powershell. There aren't any user generated data in these services so recreating them was not a big issue.
 Here are a few links that helped me solve problems with provisioning new service applications:

This one helped me to solve Sharepoint Server Search instance reporting "Service is offline" when trying to start/provision.
http://msdnrss.thecoderblogs.com/2011/06/unable-to-create-a-search-service-application-errors-were-encountered-during-the-configuration-of-the-search-application/

This is actually about multitenancy, but has some excellent code snippets that helped me provision User Profile Synchronization and Search service.
http://www.harbar.net/articles/sp2010mt5.aspx

I trully hope that these steps will help someone avoid the pain I suffered :)

Regards,
Dinko

Date Published: Sep 30, 2011 - 3:52 am


Sharepoint 2003 to 2007 upgrade with stsadm -o attachdb fails with error - Full-text crawl manager has not been initialized


Hi,

Recently I was engaged by my colleague to assist him in upgrading an aging Sharepoint 2003 installation to Sharepoint 2010. Since the full blown Sharepoint Portal Server 2003 was still in use there was no direct upgrade path, but I had to deploy temporary MOSS 2007 farm first to upgrade database from Sharepoint 2003 to 2007 version. After this step I could upgrade to Sharepoint 2010.

I've run the pre-scan tool on Sharepoint 2003 which did gave me some trouble saying that the site I was trying to scan is not extended with WSS v2 and that I should extend it first. I've fixed this problem with running a stsadm -o upgrade which, I guess, wrote a new database version in the portal content database because the process took only a couple of second. After that the pre-scan run without any errors.

Now, onto the problem...

I've created a new web application on MOSS 2007 and used the following command on the MOSS 2007 box:

stadm -o attachdb -url http://sps -databasename SPSERVER_SITE -databaseserver -SQLSERVER

This returned an error and the database was not attached to the web application.

Upgrade completed with errors.  Review the upgrade.log file located in C:Progra
m FilesCommon FilesMicrosoft SharedWeb Server Extensions12LogsUpgrade.log.
  The number of errors and warnings is listed at the end of the upgrade log file
.

After examining the log files I've found this:

[DropFullTextSearch] [3.0.2.0] [DEBUG] [9/6/2011 8:23:35 PM]: Begin Rollback()
[DropFullTextSearch] [3.0.2.0] [DEBUG] [9/6/2011 8:23:35 PM]: End Rollback()
[DropFullTextSearch] [3.0.2.0] [DEBUG] [9/6/2011 8:23:35 PM]: Begin Dispose()
[DropFullTextSearch] [3.0.2.0] [DEBUG] [9/6/2011 8:23:35 PM]: End Dispose()
[DropFullTextSearch] [3.0.2.0] [DEBUG] [9/6/2011 8:23:35 PM]: Elapsed time: 00:00:00.1405800.
[SPManager] [ERROR] [9/6/2011 8:23:35 PM]: Upgrade [SPContentDatabase Name=SPSERVER_SITE2007 Parent=SPDatabaseServiceInstance] failed.
[SPManager] [ERROR] [9/6/2011 8:23:35 PM]: Full-text crawl manager has not been initialized. Any crawl started before the crawl manager was fully initialized will need to be restarted. Please restart SQL Server and retry the command. You should also check the error log to fix any failures that might have caused the crawl manager to fail.
[SPManager] [ERROR] [9/6/2011 8:23:35 PM]:    at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean async)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Microsoft.SharePoint.Utilities.SqlSession.ExecuteNonQuery(SqlCommand command)
   at Microsoft.SharePoint.Upgrade.SPDatabaseSequence.ExecuteDataDefinitionMethodCore(SqlSession sqlSession, ISqlSession isqlSession, String sqlscript, SPSqlCommandFactory sqlcmdFactory, String[] strTables, Int32[] nThroughputs, SPLog logGlobal)
   at Microsoft.SharePoint.Upgrade.SPDatabaseWssAction.ExecuteDataDefinitionMethod(SqlSession sqlSession, String sqlscript)
   at Microsoft.SharePoint.Upgrade.DropFullTextSearch.Upgrade()
   at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()

 
It then occurred to me that back in the old days Sharepoint was using SQL full text search catalogs for its search feature, but just for WSS v2 search feature if memory serves me correctly.
When I installed the new SQL 2008 R2 I did not check the Full-Text Search component because the MOSS 2007 and Sharepoint 2010 are not using SQL Full-Text Search anymore. But it seems that for upgrade purposes the Full-Text Search component is required because the upgrade process tried to drop the existing full text catalog.

At this moment I've also reviewed the logs when I restored the Sharepoint 2003 database from SQL 2000 to SQL 2008 R2 and found this:


Log Name:      Application
Source:        MSSQLSERVER
Date:          6.9.2011. 20:15:13
Event ID:      3633
Task Category: Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SQL2000.domain.local
Description:
The operating system returned the error '3(failed to retrieve text for this error. Reason: 15100)' while attempting 'ClearTree' on 'FullTextDefaultPathix_SPSERVER_SITE' at 'fulltext.cpp'(1816).


The solution was to simply install the Full-Text Search component on the new SQL Server 2008 R2 and run the database attach process again.

Date Published: Sep 06, 2011 - 1:01 pm


Writing to log files is only allowed to jobs that are owned by sysadmin


Hello everyone,

I am experiencing an issue with SQL Server Agent and its output file. I have a job that runs daily and I need to store its output to txt file on a shared disk in a clustered environment. The job runs the simple script "EXEC sp_help_revlogin" which is used to generate "CREATE LOGIN" scripts for each SQL user account created on that instance of SQL server (domain or local). The goal of this script is to have current SQL login information in case of a disaster recovery scenario. There is another SQL instance on the remote site to which the databases are replicated using SQL Database Mirroring, but  Database Mirroring  is not replicating SQL user accounts because they are actually stored in the master database which is not supported in Database Mirroring configuration.

The problem I am experiencing is that the job is not creating any output file. The output should contain the SQL script generated by the sp_help_revlogin function where each local login is scripted so it can be ran on the remote SQL server instance. The error message I am getting is:

Message
Executed as user: DOMAINuser. Warning: cannot write logfile E:DR_SCRIPTSDR_CREATE_LOGIN.txt. Writing to log files is only allowed to jobs that are owned by sysadmin. Please consider writing log to table.  The step succeeded.

The user that owns this job is member of the sysadmin server role and it is even a member of the local administrators group on the SQL cluster nodes. So my guess is that this is simply a bug in the SQL Server 2008 (for the reference, build number we are running is 10.0.4000). Even the SQL Server Agent service is running under the account that is a member of the sysadmin role and is the local administrator on both of the SQL cluster nodes.

The solution that worked for me was to change the owner of that SQL Server Agent job to the sa account. After I did that, the job successfully created the output file to the destination folder.

If someone has some deeper explanation why is this happening, I would greatly appreciate it.

Dinko

Date Published: May 17, 2011 - 6:52 am


Exchange 2010 SP1 DAC in a single AD site


With the release of SP1 for Exchange 2010 it is now possible to have Datacenter Activation Coordination mode (DAC) enabled in a single AD site. This is perfect for smaller environments as it is now possible to have Exchange 2010 disaster recovery possible where two DAG members are separated across two system rooms in the same building or maybe separate building but still in the same AD site.

The configuration would usually consist of the following:
  • Primary Datacenter with Exchange Data Availability Group (DAG) member and a file share witness which is usually located on Exchange CAS or HUB transport server
  • Secondary Datacenter with second Exchange DAG member
This underlying cluster mechanism which is Failover Cluster now has three votes - two DAG members and a file share witness. In this setup we can loose any single machine but still have the Exchange databases online.

However if the disaster strikes and we loose our primary data center it means two out of three votes are lost and the Failover Cluster mechanism will bring the entire cluster down. This situation leaves the Exchange databases unavailable to users.

 

If you had databases mounted on EXMBX01 they would have status of "Disconnected and Healthy" on EXMBX02 and if there were databases mounted on EXMBX02 they would have status of "Dismounted".


Note: For the following procedure to work you should have already enabled the Datacenter Activation Coordination mode using the following command:

Set-DatabaseAvailabilityGroup -Identity dagname -DatacenterActivationMode DagOnly


To mount databases on the Exchange DAG member in the secondary data center you will need to type the following powershell commands on the second DAG member:

Stop-Service clussvc

Stops the Failover Cluster service

Stop-DatabaseAvailabilityGroup –Identity EXDAG01 –MailboxServer EXMBX01 -ConfigurationOnly

Stops the DAG on the failed Exchange DAG member. Since the DAG member is down and unavailable we are using ConfigurationOnly switch

Restore-DatabaseAvailabilityGroup –Identity EXDAG01 –AlternateWitnessServer EXHUBCAS02 –AlternateWitnessDirectory C:EXDAG01_W02

This command sets up the new Failover Cluster which now consists of only one cluster member (EXMBX02) and file share witness which we will be placed on EXHUBCAS02 which is located in the secondary data center. After the command finishes we should have our cluster online and all databases mounted on EXMBX02. We can ommit AlternateWitnessServer and AlternateWitnessDirectory switches if we have previously set this up on the properties of DAG. We can set this using Set-DatabaseAvailabilityGroup cmdlet.

If we now open the Failover Cluster management console we should see only EXMBX02 as a member of the cluster and file share witness should point to EXHUBCAS02.

But what when the primary data center comes back online? This is when DAC magic comes into play. If EXHUBCAS01 and EXMBX01 are brought online but the WAN link between the sites is still down, DAC mode will not allow for the quorum to be formed even though two out of three votes are available. This is because in DAC mode each DAG member must successfully contact all other DAG members or at least DAG member which has the Active Manager bit of 1 stored in memory. Since the EXMBX01 cannot contact EXMBX02 it will not form a cluster and database will not be mounted thus preventing the split brain scenario.

I recommend you read Scott Feltmann's blog for more information on how this Active Manager bit works.

When everything is back online again you should follow this steps to put everything as it was before the disaster:

Start-DatabaseAvailabilityGroup –Identity EXDAG01 –ActiveDirectorySite Default-First-Site-Name

This command essentially ads EXMBX01 back into cluster. Databases are now still mounted on EXMBX02 but the replication should be resumed and all passive database copies on EXMBX01 should be healthy. If not you will use Update-MailboxDatabaseCopy cmdlet to remedy this.

At this point file share witness is still set on EXHUBCA02. We can confirm this by opening Failover Cluster management console but if we look at the DAG properties using Get-DatabaseAvailabilityGroup cmdlet we will see that it shows EXHUBCAS01 as file share witness. All we need to do is run this cmdlet:

Set-DatabaseAvailabilityGroup –Identity EXDAG01

Failover Cluster management console now shows that EXHUBCAS01 is file share witness.

There is one more thing to do and that is to move Active databases to EXMBX01 and everything is now the way it was before the disaster.






Date Published: Feb 18, 2011 - 12:51 pm


Finding the private key for your certificate with FindPrivateKey tool


Hi,

I had a need to find the private key file for the certificate that was stored inside the Local Computer Certificates store. I needed to set the correct permissions for the private key so that system service could access it.

When the specific Windows service started it logged the following event in my Windows Application log:

System.ArgumentException was unhandled

Message="The certificate 'CN=localhost' must have a private key that is capable of key exchange. The process must have access rights for the private key."
Source="System.ServiceModel"



The private key file actually exists but the user account under which the service runs does not have the permission for it.

Private keys for the certificates on Windows 7 and Windows Server 2008 operating systems are stored here:

C:ProgramDataMicrosoftCryptoRSAMachineKeys

One example of the private key filename is:

1b2c85ebc7a8c0d84076e7da2c608e29_d6b7bae8-7a97-4c58-b63b-5f0381938d6d

And there could be tens of private key files in that directory so you do not know which private key corresponds to your certificate.

The FindPrivateKey tool solves this as it helps you to pinpoint the exact file that you are looking for.

Run the command like this:

FindPrivateKey.exe My LocalMachine -n  "CN=yourservername.domain.local"

And you will get output like this:

Private key directory:
C:ProgramDataMicrosoftCryptoRSAMachineKeys
Private key file name:
02d5c2fdc0f71d522f6011ca8b3b1493_d6b7bae8-7a97-4c58-b63b-5f0381938d6d

So now you know on which file you need to set permissions to.

If your FindPrivateKey output return this:

FindPrivateKey failed for the following reason:
Unable to obtain private key file name

Then the user account which you use to run the FindPrivateKey tool does not have permissions to read that specific private key file you are looking for. Try running the tool with "Run as Administrator" if you are using UAC or try to run your command prompt session with LocalSystem account.

You can do this by using psexec tool from Sysinternals:

psexec -s -i cmd

The new command prompt opens that runs under LocalSystem account and you can run FindPrivateKey tool from that window.

If you still get the "Unable to obtain private key file name" then you are just out of luck :)
You will need to try and pinpoint to the correct private key file manually because neither you or the system has access to it.

Dinko

Date Published: Feb 11, 2011 - 4:48 am


How to create a certificate for Ironport Email Security Appliance


This is a procedure to generate a certificate that you can import to your Ironport Email Security appliance.

Here are the basic steps:

  • Generate a certificate request using OpenSSL for Windows. Change the command line to your liking. The important thing is to change the "ironport.domain.com" to the URL that you want to use to access your Ironport appliance.
openssl req -new -newkey rsa:2048 -nodes -out ironport_domain_com.csr -keyout ironport_domain_com.key -subj "/C=HR/ST=Grad Zagreb/L=Zagreb/O=Organization/OU=IT/CN=ironport.domain.com"
  • Sign the request file (CSR) using Windows CA. You can use web application (https://servername/certsrv) of your Issuing CA and then paste the CSR there and use the Web Server template.
  • Convert the output CER file to PEM file  
openssl.exe x509 -in ironport.cer -inform der -out ironport.pem -outform pem
  • Generate a P12 file that includes private and public keys
openssl.exe pkcs12 -export -out ironport.p12 -in ironport.pem -inkey ironport_domain_com.key
  • Import the P12 file to your Ironport using the web GUI (Network > Certificate > Add Certificate)

Date Published: Jan 28, 2011 - 4:01 pm


Always send a read receipt in Outlook


I wrote a custom Group Policy Administrative Template that sets the "Always send a read receipt" in Outlook clients. This custom ADM template is useful if you want to set this property to all or subset of users in a domain but still want to give them the ability to control other properties in Tracking options in Outlook that would otherwise be disabled if you use Office administrative templates downloaded from Microsoft site.

Just copy/paste the following code in notepad and save the file with ADM extension. Import the file in Group Policy object under "User Configuration".


CLASS USER

CATEGORY "Outlook 2003 Receipt Response"

POLICY "Always send response"
KEYNAME SoftwareMicrosoftOffice11.0OutlookOptionsMail
 PART "Always send response for read receipt requests" CHECKBOX
   VALUENAME "Receipt Response"
   VALUEON NUMERIC 0 DEFCHECKED
   VALUEOFF NUMERIC 1
 END PART
 END POLICY

END CATEGORY

CATEGORY "Outlook 2007 Receipt Response"

POLICY "Always send response"
KEYNAME SoftwareMicrosoftOffice12.0OutlookOptionsMail
 PART "Always send response for read receipt requests" CHECKBOX
   VALUENAME "Receipt Response"
   VALUEON NUMERIC 0 DEFCHECKED
   VALUEOFF NUMERIC 1
 END PART
 END POLICY

END CATEGORY


CATEGORY "Outlook 2010 Receipt Response"

POLICY "Always send response"
KEYNAME SoftwareMicrosoftOffice14.0OutlookOptionsMail
 PART "Always send response for read receipt requests" CHECKBOX
   VALUENAME "Receipt Response"
   VALUEON NUMERIC 0 DEFCHECKED
   VALUEOFF NUMERIC 1
 END PART
 END POLICY

END CATEGORY






Date Published: Jan 28, 2011 - 2:57 pm


How to properly issue a certificate for Forefront TMG Standalone Arrays in a workgroup


Hello,

Due to the problems and pain we have encountered in making Forefront TMG 2010 Standalone Array in a workgroup to work on VMware ESX 3.5 Update 5 I will detail the steps for creating and importing certificates to TMG certificates store and point out to the problems with TMG Control service dependencies.

This is the environment we had:
  • Two Forefront TMG 2010 Enterprise Servers in a workgroup configured in Standalone Array with one TMG configured as Array Manager and another configured as Array Member
  • Windows Server 2008 R2 Standard
  • Virtual machines on VMware ESX 3.5 Update 5

During the implementation we have experienced the problem with Forefront TMG Control service taking 10 minutes to start after a server restart. The service would eventually start but the other Forefront services that depend on it will fail to start.


The problem was solved implementing the following fixes:

I do not imply with this article that the problem with TMG Control service hang is related only to VMware ESX but we only experienced it on this platform. When we encountered the problem we did some tests on Hyper-V environment and on separate VMware ESX 3.5 Update 5 environment and there was no problem, however on this particular environment the service would not start once the TMG array was configured.

So I would recommend for anyone to get these dependencies fixed even if you do not encounter this problem now. Regarding the problem with certificates, I have already blogged about it here but I also wrote a procedure how to properly issue Server Authentication certificate for TMG arrays in a workgroup.


How to issue a proper "Server Authentication" certificate

Prerequisites:
  • Access to any Windows Server 2008 IIS 7.0 web server
  • Access to Enterprise or Standalone Windows Sever 2008 Certification Authority (Windows 2003 CA is also okay)

1. Open the IIS Manager, click on server name node from the left pane and click on "Server Certificates" from the middle pane


2. Click on the "Create Certificate Request" from the right pane

3. In the "Common name" field type the FQDN of the TMG server that will act as an Array Manager. In this example we will use "tmg01.company.hr". Fill the remaining fields so that you best describe your organization.


4. Choose "Microsoft RSA SChannell Cryptographic Provider" for the "Cryptographic service provider" and choose 2048 for the "Bit lenght".

5. Save the certificate request as C:tmg01.req.

6. Navigate to the Issuing or Root CA web site such as https://yourservername/certsrv and click on "Request a certificate"


7. Click on "advanced certificate request"

8. Click on "Submit a certificate request by using  a base-64-encoded CMC or PKCS #10 file, or submit a renewall request  by using a base-64-encoded PKCMS #7 file".

9. Paste the contents of the tmg01.req file that you have created earlier from IIS to the "Base-64-encoded certificate request" field. In case you have a drop-box with Certificate Templates list, select "Web Server" template.


10. Your certificate request is now submitted to the CA. In case the "Request Handling" property of your CA is set to automatically issue certificates you will be presented with the following page where you have the possibility to download your issued "cer" file. Click on "Download certificate" and save the file as C:tmg01.cer. Go to the step number 15.

In case the "Request Handling" is set to manually issue the certificates by the administrator then you will have to perform the following steps.

11. Open the "Certification Authority" console on your Issuing CA server and click on "Pending Requests". You should see your request in the right pane.

12. Right click on the request and select All Tasks > Issue.


13. Browse to the CA web site again (https://yourservername/certsrv) and click "View the status of the pending certificate request". There should be your "Saved-Certificate Request" listed.


14. You are now presented with the same page as in step number 10. Download the "cer" file as described in step 10 and proceed to step 15.

15. Now return to the IIS Manager console from which you have created the certificate request and now select "Complete Certificate Request".

16. In the "Specify Certificate Authority Response" screen browse to the "cer" file you  have downloaded from the CA and enter a friendly name for the certificate. I usually type the same name as common name.


You have now completed the procedure of issuing the "Server Authentication" certificate. If you open the "Local Computer" Certificates store on the server where you have requested the certificate you should see the certificate in the Personal > Certificates folder. The certificate icon should have a little yellow key pictured which means that you have both private and public key. We must export the certificate with private and public keys so that we can import it on our TMG server.

17. Right click on the certificate and click All Tasks > Export.


18. Select "Yes, export the private key".

19. "Personal Information Exchange - PKCS #12 (.PFX)" should be selected. Unmark all the checkboxes and click Next.


20. Type the password that you will need to type when you import the certificate to the TMG computer.

21. Save the certificate as C:tmg01.pfx.

Now that we have our certificate ready for import there is still one thing we must do. Since we are creating TMG array in a workgroup mode we must import the root certificate of the CA that issued the certificate to all of the TMG servers that will participate in array. But first we must export the root CA certificate from a computer that has it.

22. Open the "Local Computer" Certificates store on the Issuing CA computer or on some other computer which is a domain member in a domain where CA resides.

23. Navigate to the Trusted Root Certification Authorities > Certificates, right-click on the root certificate from the CA which issued your certificate and select All Tasks > Export.


24. Select "DER encoded binary X.509 (.CER)" and click Next.

25. Save the "cer" file to disk. In our example it is C:CompanyRootCA.cer.

Now we have both the PFX file which contains our public and private keys for the TMG computer certificate and a CER file that contains a public key from our root CA. The next thing we must do is to import the root certificate to each TMG server that will participate in the array and to import the "Server Authentication" certificate.

Note: It is good practice to create "Server Authentication" certificate for all TMG servers so that if Array Manager fails you can promote some other Array Member to Array Manager.

26. Open the "Local Computer" Certificates store on each TMG server and import the root certificate "cer" file to the "Trusted Root Certification Authorities".

27. Now open the "Forefront TMG Management" console on the TMG server that will act as an Array Manager. Expand "Forefront TMG" in the left pane and click on System node. Click on the TMG server name in the center pane and click on the "Install Server Certificate" in the right pane.


28. Now browse to the "pfx" file you have exported from the web server computer and type a password for the file. Unmark the checkbox "Automatically create the root CA certificate on this array manager." To my experience leaving this checkbox marked always resulted in an error even though the pfx file contained the root CA certificate. Click OK.


Now if you open the Certificates store for the Windows service named ISASTGCTRL you should see the imported certificate with the private key in the Personal store.
 


So why is important to use Forefront TMG Management console to import the certificate? You could just import the certificate in the Local Computer Certificates store, right? Well the answer is yes and no. If you do it this way the ISASTGCTRL service will not have enough permissions to read private key file that is stored in the C:ProgramDataMicrosoftCryptoRSAMachineKeys and you would get Schannel errors in the Windows event log and TMG Control service would not be able to communicate with the ADAM service (ISASTGCTRL) on the Array Manager computer. If you use Forefront TMG Management console to import the certificate all the necessary permissions are added to the private key file including:
  • SYSTEM
  • NETWORK SERVICE
  • fwsrv
Of course you could manually update the permissions on the private key file if you knew which private key it is and things would work but it is not the proper way to do this.

Test the connection

Now there is only thing left and that is to test the secure LDAP connection to the Array Manager server. We will use ldp.exe for this. You should be able to run it from your TMG servers.

Open ldp.exe and click on Connection > Connect. Type FQDN of your TMG server that will act as Array Manager and type 2172 for the port number as this is the port on which ISASTGCTRL service listens. Click on the SSL and click Connect.


If the connection is successful you will see the screen like the following:


And that is all there is to it! Make sure to complete the procedure for all TMG servers that will participate in the array for the already mentioned reason and that is so that another Array Member can become Array Manager in case the Array Manager fails.

Date Published: Jan 26, 2011 - 4:56 am


Forefront TMG and persistent cookies problem


Hi,

I just want to point out to a problem I have just resolved. Customer reported a problem when accessing Sharepoint 2010 site that is published using Forefront TMG 2010. Sharepoint listener is configured to use forms-based authentication with Persistent Cookies enabled for both public and private computers. For those who wants to know how this works you can read my article here.

The problem was that when the Private computer was selected in the authentication form when accessing site it worked fine but when the Public or Shared computer was selected the user could not be authenticated. The TMG log showed the following error:

12302 - The server denied the specified Uniform Resource Locator (URL). Contact the server administrator.

When the listener was modified to use Persistent Cookies only for the Private computers then the authentication worked also for Public computers but then the desired authentication cookie was not saved to the local client cache. The problem appeared on IE, Firefox and on different client computers so it was not related to the local client cache or something like that. TMG and Sharepoint servers were restarted but it also did not help.

The problem was resolved only by recreating the Sharepoint listener on TMG using all the same properties as before. Now both the Public and Private computers work with Persistent Cookies enabled.

Dinko

Date Published: Jan 23, 2011 - 3:21 am


Demistifying "The Cloud"


Hello,

For the last two days I was attending Private Cloud workshop in Microsoft office in Warsaw. The correct name of the workshop was actually Government Private Cloud Computing but the workshop actually gave great overview about types of "clouds" and the types of services that can be delivered through the cloud. It also covered every aspect of building a cloud solution from technology to process management, automation and billing. What strike me the most is actually how easily the specific cloud type can be defined. I am reading about the cloud infrastructure for a while now and I have even been architecting and implementing public cloud solutions based on Exchange 2007/2010 and Sharepoint 2007/2010 and the workshop helped me to sort in my head everything I have learned so far. The main goal of the workshop was to teach us how to have a conversation with a customer company that plans to go to "the cloud" and how to ask some basic questions that will help us define the cloud or the service that is to be offered through the cloud.

In the next couple of lines I will try to summarize the different service delivery types and the different types of cloud and map those to the actual technology or a scenario based on my personal experience.

Let us first cover the service delivery types:
  • Infrastructure as a Service (IaaS)
  • Platform as a Service (PaaS)
  • Software as a Service (SaaS)

Infrastructure as a Service

IaaS
The blue boxes shows what Service Provider actually provides to the customer. We can see that those elements are Datacenter, Networking, Computers and Virtualization. In practice that would mean that the provider company takes care of system rooms, electricity and cooling (Datacenter), networking components such as switches and routers (Networking), servers that comprise the monitoring, provisioning, billing and virtualization infrastructure (Computers) and manages virtualization technology such as Hyper-V or VMware clusters. Everything above can be dynamically created, self-serviced and pooled by the customer. In this certain case it would mean that the customer can self-provision virtual machines with the operating system of choice, pool the amount of storage he needs for the virtual machine and install the applications he needs for his business. In a Microsoft world this functionality could be provided by System Center VMM Self-Service Portal 2.0.


Platform as a Service

PaaS
Now let us move a few boxes upwards. If the operating system and the storage is controlled by a service provider company and the customer can deploy it's own applications that run on the underlying platform of choice then this model is called Platform as a Service or PaaS. Middleware box is grey area here. Let us consider a scenario where the customer gets presented by an empty virtual machine where he can deploy his own engine like Oracle Application Server and build his own applications on top of that. Then the middleware would be controlled by a customer. Or we could provide the customer with Microsoft Sharepoint 2010 site collection and give him the ability to deploy his own web parts or workflows. The customer would not have access to the operating system in this case. He would only have access to his own little isolated area on Sharepoint and anything he deploys there is relevant only to him and it does not affect other customers. Sharepoint 2010 has sandboxed solutions integrated so this would make for a perfect example of Platform as a Service.

Software as a Service

SaaS
If the service provider company has the total control of the stack and the customer only consumes services such sending and receiving of e-mail, upload documents to sharepoint or scheduling and running voice conferences than this is called Software as a Service. Customer only sees the service he uses and has no control of any aspect of the underlying platform. Self-service in this case would mean that the customer administrator can provision a new mailbox for an employee inside his own company or could enable the user for voice conferencing so that he can share video and audio with his colleagues. Examples of these in a Microsoft world are of course Exchange, Sharepoint, OCS and CRM.

There are few other aspects of the service delivery models that must be met so that we could call our solution a "cloud solution". The workshop I have attended specified the following characteristics which I will try do describe in my own words:
  • Resource Pooling - the ability to share resources such as network, memory and processing power and provide the resources when needed to the workload that needs them
  • Measured Service - the ability to measure service utilization per customer such as number of mailboxes he used during the last month, the amount of GBytes he uses on the storage for mailbox, documents, databases etc. or bandwith that was consumed. The most important thing is that the customer can be charged only by what he spent the last month. If the number of mailboxes decreased or the bandwith utilization dropped from the previous month, the customer should be charged accordingly!
  • Broad Network Service - the service should be possible to be accessed from anywhere using any Internet connection
  • Rapid Elasticity - the service should be possible to scale out or scale up in the shortest amount of time. Consider adding new virtualization host to the infrastructure to increase processing power or adding new disk shelf to increase storage space or increase performance. The service should also be able to rapidly scale down if resources are not used. An example for this is moving running virtual machines to the smaller number of hosts and then shutting down hosts with no load.
  • On-Demand Self Service - the customer should be able to self-provision the service such as create a new mailbox, add additional disk space to the virtual machine or enable an employee for Sharepoint or OCS access

Now when we have defined the service delivery models and cloud service characteristics we will cover the cloud deployment models. There are four deployment models:

  • Private cloud - this cloud infrastructure is operated solely for a specific organization. It can be managed by an organization itself or by a service provider and may exist on premise or off premise. Owner of the infrastructure components can be either the customer or the service provider. I can think of a scenario in Microsoft Online Services and e-mail hosting. If you have large enough company with plenty of e-mail users, Microsoft would provide a dedicated infrastructure just for you and it would be managed and operated by Microsoft for your company only. Or there could be another reason for a private cloud infrastructure. An organization security policy could specify that data must somehow be  separated and isolated from other organizations which would require a dedicated e-mail server just for the specific organization or a dedicated hardware infrastructure.
  • Community cloud - the cloud infrastructure that is shared by several organizations and supports a specific community that has shared concerns such as mission, security requirements, policy and compliance considerations. A good example for this scenario are government ministries or agencies that could share the same common infrastructure.
  • Public cloud -  this cloud infrastructure is made for general public and owned and maintained by an organization (service provider) that sells cloud services. Examples would be e-mail services, shared or dedicated Sharepoint servers, virtual machines etc.
  • Hybrid cloud - composition of two or more models (private, community or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability.


Technology

So where does the technology fits in? Almost every major vendor out there today is delivering its own solutions that are cloud enabled or will help you to get to the cloud. Microsoft naturally has its own set of solutions that I will try to cover briefly here.

  • Hardware - hardware selection is totally up to you. You can choose server, storage or network vendor you like or already work with
  • Virtualization - there are at least three out there that I would recommend, Hyper-V, VMware and XenServer, they all do their jobs excellent
  • Deployment - System Center Configuration Manager is a perfect tool for deploying new operating systems, either provisioning new desktops for the customer or new Hyper-V hosts for the service provider or patching the existing infrastructure
  • Monitoring - System Center Operations Manager - used for monitoring the entire environment and alerting in case something goes wrong or you just want to be notified about. It can also be used as a billing tool to show your customer how much "resources" have been used for the past month.
  • Automation - Microsoft has a new member of the System Center family and it is called Opalis. It is used for Datacenter automation. It enables you to automate certain Datacenter processes and helps you get "elasticity" for your cloud infrastructure. One example of automation could be installing new OS on the server, joining it to virtualization cluster, deploying new patches and updates and deploying SCOM agent for monitoring. You can do that with this tool. Microsoft also provides already mentioned System Center Virtual Machine Manager Self-Service Portal 2.0. It can be used for automating virtual machine deployment and it is also customizable in a way it can be further developed or branded for each customer. 
  • Provisioning - provisioning application in the cloud infrastructure is used in two different ways. If you work for service provider company then you would probably use it to quickly provision new customer and give him access to your services. If you are customer, you will probably use the provisioning application (usually a web interface) to quickly create new virtual machines or create new mailboxes for your employees. Microsoft has no unified software that would deal with all scenarios but here are some examples. Exchange 2010 with SP1 has multi-tenancy feature integrated and it provides the so-called Exchange Control Panel or ECP. If you are a customer that is provisioned on a shared or hosted Exchange 2010 server you would probably get access to ECP and have a possibility to manage mailboxes for your employees only. Other customers on that same Exchange server are completely isolated from your users. However, the provisioning application for the service provider company in regard to Exchange 2010 server does not really exist. There is only a powershell interface you can use or develop your own. But luckily there are companies out there who already have a software for that purpose.

Well I hope that I have helped you to understand some of the basics of the cloud infrastructure and that the cloud paradigm is not so cloudy to you anymore :)

Please feel free to comment on this post and give your view about the terms and definitions about "the cloud". I have to be honest and say that I am by no means an expert on this field because the topic is so huge and it would probably take years of experience to understand it completely but I can definitely say that the workshop I have attended helped me to get some basic understanding.

Regards,
Dinko



Date Published: Jan 19, 2011 - 11:09 am


Forefront TMG Enterprise Standalone Array does not start after server reboot


Last couple of days me and my colleague where troubleshooting a brand new installation of Forefront TMG Enterprise Standalone Array consisting of two nodes. The problem we had was that after a server restart, Forefront TMG Control service would not start. It would hang in a starting state for about 10 minutes after which it would eventually start but the TMG Firewall service and all other TMG services that depend on TMG Control service did not start because of this timeout. After that we could manually start the services and the TMG Array worked with no problem. It was only the problem after server reboot.

The environment:

  • Two Windows Server 2008 R2 Standard virtual machines on VMware ESX 3.5 Update 5 environment
  • Forefront TMG 2010 Enterprise SP1 Software Update 1
  • Forefront Standalone Array in Workgroup mode with one node designated as Array Manager and the other one as Array Member
  • Each node had a server certificate installed in local computer store with Extended Key Usages for Server Authentication and Client Authentication

Here are some of the errors we were seeing in the event log:

The Microsoft Forefront TMG Control service hung on starting. 

The Microsoft Forefront TMG Firewall service depends on the Microsoft Forefront TMG Control service which failed to start because of the following error:
After starting, the service hung in a start-pending state.

The Microsoft Forefront TMG Managed Control service depends on the Microsoft Forefront TMG Control service which failed to start because of the following error:
After starting, the service hung in a start-pending state.

The Microsoft Forefront TMG Job Scheduler service depends on the Microsoft Forefront TMG Control service which failed to start because of the following error:
After starting, the service hung in a start-pending state.

Log Name:      Application
Source:        Windows Error Reporting
Date:          14.1.2011. 14:57:24
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      *************
Description:
Fault bucket , type 0
Event Name: ServiceHang
Response: Not available
Cab Id: 0

Problem signature:
P1: isactrl
P2: mspadmin.exe"
P3: 0.0.0.0
P4: 10
P5: 2
P6:
P7:
P8:
P9:
P10:

Attached files:

These files may be available here:

Analysis symbol:
Rechecking for solution: 0
Report Id: 363731e9-1fe6-11e0-846f-00155d274102
Report Status: 0



To better understand the problem here are some technical details.

TMG members in the Standalone array communicate with the array manager which has AD LDS (Active Directore Lightwight Directory Services) installed which provides configuration storage for the entire array. The array manager would first save the configuration to its local AD LDS instance and the rest of the array members connect to it using Secure LDAP which require server certificate with "Server Authentication" key usage. Actually, only the Array Manager requires the certificate and the Array Members require root certificate from the Certification Authority that signed the certificate located in Trusted Root Certification Authorities Store so that they would trust Array Managers certificate. But in case the Array Manager fails you would have to manually promote one of the Array Members to Array Manager and he would then require server certificate installed. So we have installed server certificate on both TMG computers.

Here comes the problem.... Since Array Member had its own server certificate with Extended Key Usage and Intended Purposes set to Server Authentication and Client Authentication, when authenticating to remote AD LDS service it would present its client certificate and this process is known as mutual authentication or MTLS. Well it seems that TMG Control service does not like this behavior and it times out for about 10-15 minute after which none of the TMG services start. The problem happened on both TMG computers even though one TMG was Array Manager, he still needed to connect to local AD LDS instance but he too tried to mutually authenticate to the local AD LDS service.

Well, the solution was quite simple in fact, but very hard and frustrating to find because there were almost no relevant logs to look at. On each TMG computer the certificate properties should be modified to include only Server Authentication for Intended Purpose.

Here is how to do it:
  • Open Certificates MMC snap-in and connect to Local Computer
  • Navigate to Personal > Certificates > your_computer_certificate (the certificate should have common name of FQDN of your TMG computer)
  • Double click on the certificate, click on the Details tab and click Edit Properties
  • Choose "Enable only the following purposes" radio button and check Server Authentication
  • Restart your computer and see if TMG services start normally




Of course, if your certificate only has Server Authentication in Extended Key Usage field then you will not experience this issue.

Microsoft also had something to say about this issue in the following article, but not directly related to this problem.


Client logon is slow and server certificates used for Web publishing are configured with the default purpose settings "Server Authentication" and "Client Authentication"
Issue: When Windows Server 2003 detects the default purpose setting of "Client Authentication", the operating system attempts to perform TLS with mutual authentication to the domain controller. The mutual authentication process requires ISA Server to have access to the private key of the server certificate with the "Client Authentication" setting enabled, and ISA Server does not (and should not) have this access.
Solution: Ensure that all server certificates do not have the default "Client Authentication" purpose enabled. You can disable this setting on the property pages of the relevant server certificate as follows:
Disable Client Authentication purpose on a certificate
1.     Open the Certificates Microsoft Management Console (mmc) snap-in. To add the Certificate Manager to the mmc, do the following:
·         Click Start, and then click Run.
·         Type mmc and then press ENTER.
·         Select the File menu, and then select Add/Remove Snap-in.
·         In the Add/Remove Snap-in box, and then click Add.
·         Double-click the Certificates snap-in, select Computer Account, and then click Finish.
·         Select Local Computer, and then click Finish.
·         Close the dialog boxes.
2.     In the Certificates mmc, click to expand the Certificates node, and then expand Personal.
3.     Right-click the relevant certificate and then click Properties.
4.     On the Details tab, click Edit Properties.
5.     Select Enable only the following purposes, and clear the Client Authentication purpose.


Link to the entire article here.

While troubleshooting this issue, out of pure frustration we even replicated the entire environment on Windows Server 2008 SP2 and later even on Hyper-V as the virtualization platform to eliminate any compatibility issues but finally it seems that this little setting did the trick.

We have also tried, read this carefully, Rollup 1 and Rollup 2 for Software Update 1 for Service Pack 1 for TMG 2010 just to be sure we had the entire environment patched and read numerous blogs that talked about TMG Control service dependency issues that would arise after installation of TMG updates and rollups but none of those worked.

I really hope this article will someday save a lot of time to someone :)

P.S.
Here is a link to a blog article that describes some other startup issues that you may have with TMG related to service dependency ordering.

Date Published: Jan 14, 2011 - 11:38 am


 
Visitor Rating: 1 (1) (Rate)

Story Clicks: 20

Feed Views: 489

Lenses (Add|?)

Comments (Log in to add)

Feed Details
Date Added: 02/08/2011
Date Approved: 02/08/2011
By:
Search FeedAgg.com




8774 serv 1.0447 seconds to generate.