Summary: IT Solution Braindumps
IT solutions and frustrations from real life including Exchange, Sharepoint, TMG, virtualization etc.
Hi,
If you have played with unfortunate Exchange 2010 SP1 /Hosting mode
installation or you are a real hoster that runs it in the
production, then you maybe had problems with getting Antispam
agents to work on your Hub transport server.
This special mode of Exchange 2010 SP1 installation has some
limitations when compared to the regular Exchange 2010
installation. One of them is that you cannot use Exchange 2010 Edge
Server role with EdgeSync. This means that you will have to use
install-AntispamAgents.ps1
Powershell script to enable Antispam agents on your Hub transport
servers.
The script works well, transport agents get installed and enabled.
But you will soon notice that e-mail messages don't get stamped
with SCL even if you have correctly configured the Content
Filtering agent. Furthermore, if you have enabled Recipient
Filtering agent too, you will see that all recipient addresses,
even the existing ones in your tenant organizations will get
rejected with "user unknown" message.
This is becase the Antispam agents simply do not work with Exhange
2010 SP1 /Hosting mode installation. Don't let this article fool
you as it did me:
http://technet.microsoft.com/en-us/library/ff923278.aspx
Here is an example of a behaviour you will notice with Content
Filtering agent. The
Get-AgentLog
Powershell output on your Hub transport will show this:
Normal 0 21 false false false HR X-NONE X-NONE
MicrosoftInternetExplorer4
RunspaceId
: 018024ed-1c5a-498a-8c15-087b1c81ed2e
Timestamp
: 1/24/2012 7:07:28 AM
SessionId
: 08CEA824BF825A40
IPAddress
: 111.222.111.222
MessageId
:
P1FromAddress :
Dinko.Fabricni@span.hr
P2FromAddresses :
{Dinko.Fabricni@span.hr}
Recipients
: {dfabricni@example.net}
Agent
: Content Filter Agent
Event
: OnEndOfData
Action
: AcceptMessage
SmtpResponse
:
Reason
: SCL
ReasonData
: 0
Diagnostics
:
You can see that agent does its thing, but when you check the
message headers in Outlook you will see that actually no SCL is
provided:
Normal 0 21 false false false HR X-NONE X-NONE
MicrosoftInternetExplorer4
X-MS-Exchange-Organization-AuthSource:
EXHUBEXT01.cloud.local
X-MS-Exchange-Organization-AuthAs:
Anonymous
X-MS-Exchange-Organization-PRD:
span.hr
X-MS-Exchange-Organization-SenderIdResult:
None
designate
permitted sender hosts)
X-MS-Exchange-Organization-Antispam-Report:
SCLNotProvided
To make it even worse, ALL messages, even those that are 100% spam
messages will receive SCL rating of 0 and you will see the
SCLNotProvided in message headers.
We did raise this incident to Microsoft and got a confirmation that
Antispam agents really do not work and that the documentation on
Technet website (the article I linked) is misleading.

Date Published:
In case you are trying to use Sharepoint event logging
infrastructure and its SPDiagnosticsService class from Sharepoint
e-mail event receiver to write to the Windows Event Log or
Sharepoint ULS log you might get this error:
The source was not found, but
some or all event logs could not be searched. Inaccessible
logs: Security
Or even worse, if you are not using Visual Studio 2010 debugger,
but only rely on Event Viewer, you could see this error in
Applications and Services Logs > Microsoft > Sharepoint
Products > Shared > Operational:
An error occurred while
processing the incoming e-mail file
C:inetpubmailrootDrop3692ac8b01cc95a800000006.eml. The error was:
Object reference not set to an instance of an
object.
As you might know, SPEmailEventReceiver is running in owstimer.exe
process which is actually "Sharepoint 2010 Timer" service. This
service by default runs under NETWORK SERVICE account which does
not have enough permissions to the Security log and this is the
reason why SPDiagnosticsService class methods WriteTrace and
WriteEvent will fail.
The solution I found is to add the NETWORK SERVICE account to the
"Performance Log Users" Windows Security group and then restart the
Sharepoint 2010 Timer service.

Date Published: Oct 28, 2011 - 2:37 pm
I want the share this little script I wrote that helped me to
document Hyper-V R2 cluster installation. This particular
installation had more than 30 virtual machines and I had to
document virtual machines properties and disk configuration.
The benefit of this script is that it merges output from two
powershell commands, Get-VM and Get-VirtualHardDisk.
$vmmserver = Get-VMMServer -ComputerName SCVMMSERVER
$vms = Get-VM | Sort-object Name | where {$_.HostGroupPath -like "All Hosts*"}
foreach ($vm in $vms)
{
$disks = $vm | Get-VirtualHardDisk
foreach($d in $disks) {
new-object PSObject -prop @{
VMName = $vm.Name
Type = $d.VHDType
"Disk name" = $d.Name
"Size GB" = [math]::truncate($d.Size / 1GB)
}
}
}
The script generates this output:
Size GB
Type
Disk name
VMName
------- ----
---------
------
20 FixedSize
9002D_SERVER1_disk_1 9002D-SERVER1
9 DynamicallyExpanding
9021D-TESTVM_disk_1 9021D-MGMTSERVER
40 FixedSize
_CBBMSERVER
40 FixedSize
HQVM-PRINTSERVER_C PRINTSERVER
20 FixedSize
HQVM-PRINTSERVER_D PRINTSERVER
6 DynamicallyExpanding
TEST_disk_1
TEST
40 FixedSize
_CTest
15 FixedSize
HQVMAC01_HQVMAC01_C HQVMAC01
21 FixedSize
HQVMAC01_D
HQVMAC01
15 FixedSize
HQVMAC02_C
HQVMAC02
21 FixedSize
HQVMAC02_D
HQVMAC02
With a little modification to the
script you could include additional columns in the output. Let's
say that you need CPU count and Memory included in the output:
$vmmserver = Get-VMMServer -ComputerName SCVMMSERVER
$vms = Get-VM | Sort-object Name | where {$_.HostGroupPath -like "All Hosts*"}
foreach ($vm in $vms)
{
$disks = $vm | Get-VirtualHardDisk
foreach($d in $disks) {
new-object PSObject -prop @{
VMName = $vm.Name
Type = $d.VHDType
"Disk name" = $d.Name
"Size GB" = [math]::truncate($d.Size / 1GB)
Memory = $vm.Memory
"CPU Count" = $vm.CPUCount
}
}
}
Note that I have added two rows in the script:
Memory = $vm.Memory
"CPU Count" = $vm.CPUCount
The output now looks like this:
Size GB : 9
CPU Count : 1
Memory :
1024
Type :
DynamicallyExpanding
Disk name :
9021D-TESTVM_disk_1
VMName :
9021D-SERVER1
Size GB : 40
CPU Count : 1
Memory :
1024
Type :
FixedSize
Disk name : _C
VMName :
HQVM-BBM
Size GB : 40
CPU Count : 1
Memory :
1024
Type :
FixedSize
Disk name : HQVM-PRINTSERVER_C
VMName :
HQVM-PRINTSERVER
Size GB : 20
CPU Count : 1
Memory :
1024
Type :
FixedSize
Disk name : HQVM-PRINTSERVER_D
VMName :
HQVM-PRINTSERVER
Size GB : 6
CPU Count : 4
Memory :
512
Type :
DynamicallyExpanding
Disk name : TEST_disk_1
VMName :
TEST
Size GB : 40
CPU Count : 1
Memory :
1024
Type :
FixedSize
Disk name : _C
VMName :
Test
Size GB : 15
CPU Count : 1
Memory :
1024
Type :
FixedSize
Disk name : HQVMAC01_HQVMAC01_C
VMName :
HQVMAC01
Size GB : 21
CPU Count : 1
Memory :
1024
Type :
FixedSize
Disk name : HQVMAC01_D
VMName :
HQVMAC01
Size GB : 15
CPU Count : 1
Memory :
2048
Type :
FixedSize
Disk name : HQVMAC02_C
VMName :
HQVMAC02
Size GB : 21
CPU Count : 1
Memory :
2048
Type :
FixedSize
Disk name : HQVMAC02_D
VMName :
HQVMAC02
This is good, but not exactly the table we expected and it's hard
to convert it into a configuration table that we could copy/paste
into installation document.
So the solution is to create a function from our script like
this:
function Global:Get-VMConfiguration()
{
$vmmserver = Get-VMMServer -ComputerName SCVMMSERVER
$vms = Get-VM | Sort-object Name | where {$_.HostGroupPath -like "All HostsProduction*"}
foreach ($vm in $vms)
{
$disks = $vm | Get-VirtualHardDisk
foreach($d in $disks) {
new-object PSObject -prop @{
VMName = $vm.Name
Type = $d.VHDType
"Disk name" = $d.Name
"Size GB" = [math]::truncate($d.Size / 1GB)
Memory = $vm.Memory
"CPU Count" = $vm.CPUCount
}
}
}
}
And call that function:
Get-VMConfiguration | Format-Table -Property vmname,memory,"cpu count","disk name",type,"size gb"
This would give you a nicely formatted table that you could paste
into a Word document.
Or even better, you could use export-csv function:
Get-VMConfiguration | export-csv -Path c:Export.csv
Now you can import this comma separated file into Excel and
format it as you wish.

Date Published: Oct 18, 2011 - 1:16 pm
Sharepoint 2010 comes with a pretty good navigation out-of-the-box,
but there are still some areas where it's not sufficient and that
is navigation that crosses site collection boundaries.
If you are designing a large Sharepoint 2010 site architecture you
are probably using site collections because they represent security
boundary and can also be placed in a separate content database
which allows you to physically segment Sharepoint 2010 content
across multiple SQL databases.
Fortunately, there is a pretty simple solution to implement cross
site collection navigation which makes navigating accross multiple
site collections to feel more like having a site structure in a
single site collection.
To achieve this we need to create a custom navigation provider in
Sharepoint 2010 web application that pulls its data from xml file
which represents our site collection structure. This way we can
have flat site collection design where all site collections are
created in the same URL path level, but also have them
hierarchically organised to better represent our organizational
structure.
Here are the steps to implement custom navigation based on
xml file.
Copy/paste the following xml in notepad and name it
CustomNavigation.sitemap.
Now open the Internet Information Services Manager, expand your
Sharepoint web application, right-click the _app_bin folder and
click Explore.
Copy the CustomNavigation.sitemap file to this folder.
Now you need to register the new site map navigation provider by
modifying the web.config file of your Sharepoint web
application.
Open web.config file (it's located one level above from where you
pasted CustomNavigation.sitemap) and enter the following line just
below the rest of the already registered providers:
....
....
Now, maybe you've read on some other blog that you should first
copy the following line:
and then rename it and just modify siteMapFile path. But in my case
this did not work very well. Everything was okay on the root portal
site, let's say it's http://sps. But on the
http://sps/sites/products I would get wrong relative links:
http://sps/sites/products/sites/hardware instead of
http://sps/sites/hardware. Modifying the xml file with full URL did
not work either so when I tried using the
System.Web.XmlSiteMapProvider everything worked as it supposed to
do. I guess it's because Sharepoint sees http://sps/sites/products
as root of the site and not http://sps. The XmlSiteMapProvider does
not have this problem.
The next step is to link our custom navigation provider with the
top link navigation bar on the Sharepoint site. We need a
Sharepoint Designer 2010 to do this.
Browse to the site collection root site where you want to modify
navigation and click on Site Actions > Edit in Sharepoint
Designer. Keep in mind that you will have to modify all site
collections this way to implement consistent navigation.
We will modify the v4.master so that all pages that are based on
v4.master will have the same navigation.
Double click on v4.master and click on "Edit file".
Select the "Split" view then right click on the top link navigation
bar and click "Zoom to Contents".
Modify the following:
ID="TopNavigationMenuV4"
Runat="server"
EnableViewState="false"
DataSourceID="topSiteMap"
AccessKey=""
UseSimpleRendering="true"
UseSeparateCss="false"
Orientation="Horizontal"
StaticDisplayLevels="2"
MaximumDynamicDisplayLevels="1"
SkipLinkText=""
CssClass="s4-tn"/>
ShowStartingNode="False"
SiteMapProvider="SPNavigationProvider"
id="topSiteMap"
runat="server"
StartingNodeUrl="sid:1002"/>
To look like this:
ID="TopNavigationMenuV4"
Runat="server"
EnableViewState="false"
DataSourceID="topSiteMap"
AccessKey=""
UseSimpleRendering="true"
UseSeparateCss="false"
Orientation="Horizontal"
StaticDisplayLevels="1"
MaximumDynamicDisplayLevels="1"
SkipLinkText=""
CssClass="s4-tn"/>
ShowStartingNode="False"
SiteMapProvider="CustomXmlContentMapProvider"
id="topSiteMap"
runat="server"
StartingNodeUrl="/"/>
Save the v4.master and open your site in a web browser.
Here are the results:
You'll notice that I've also modified the StaticDisplayLevels
property from 2 to 1. This represents to which level Sharepoint
will display links from the xml file visible on the top link
navigation bar and when will the links be placed in a drop-down
menu when you hover over the link. The default setting of 2 will
place all links from the xml file visible on the top link bar like
this:

So there you go, cross site collection navigation as simple as it
can be.

Date Published: Oct 18, 2011 - 2:13 am
Hello,
Couple of days ago I had to change domain membership of a single
server Sharepoint 2010 farm with separate database server. Browsing
the Internet I found little data about this and most
recommendations were based on backup/reinstall/restore
procedure.
However, I thought I would give it a try with plain domain
membership change and with switching Sharepoint 2010 service
accounts. I encountered many problems and I will mentioned some of
them here. Here's the procedure:
- Perform full backup of Sharepoint 2010 farm from Sharepoint
Central Administration
- Create new service accounts in the destination domain
- Make sure you know your Sharepoint 2010 farm passphrase which
you entered when you initially provisioned the farm
- Change the domain membership of SQL database server
first (in my case the SQL database was running under LocalSystem
account so I had no issues with that)
- Give the future farm account from the new domain sysadmin
permissions to the SQL database engine (actually only security
admin and dbcreator permissions are necessary)
- At this point your Sharepoint is not working
- Run stsadm -o setconfig db with the -connect switch to
connect to your Sharepoint configuration database. You will have
to use your new domain farm credentials here.
- After this step, the Central Administration site should be
working, however, your Sharepoint box is still in the old domain.
In my case I had the domain trust established between the old and
new domains.
- Create the new Sharepoint managed accounts by selecting the
accounts from the new domain: Central Administration >
Security > General Security > Configure Managed Accounts
- Change the service accounts to reflect the newly added
managed accounts: Central Administration > Security >
Change Service Accounts
- Add your farm account to the local administrators group on
the Sharepoint server
- Change Sharepoint box domain membership
- At this point, your Sharepoint sites should be accessible.
However, in my case they were not working and I received 404 not
found message. I realized that after I reconnected the Sharepoint
farm to the configuration database, custom solutions that these
sites were using were not available any more. Thankfully, I had a
full farm backup and managed to restore only the farm solutions.
I redeployed the solutions from the Central Administration and
the sites worked!
- At one point, after a couple of iisresets and server restarts
I received "The trial period has expired" error message when I
opened the Sharepoint sites. Running Sharepoint Configuration
Wizard again solved this issue.
- Looking at the "Central Administration > Manage services
on service" I saw only a couple of services listed while I know
there should be more. Running Install-SPService from Powershell
re-registered these services. This is important step for
Sharepoint Service Applications to work properly.
- Almost all Service Applications were started and I could
access the management pages for them except the two most
important ones, User Profile Synchronization service and Search
service. No matter what I did I could not fix them or even
restore them. I ended up creating and provisioning the new
services from powershell. There aren't any user generated data in
these services so recreating them was not a big issue.
Here are a few links that helped me solve problems with
provisioning new service applications:
This one helped me to solve Sharepoint Server Search instance
reporting "Service is offline" when trying to start/provision.
http://msdnrss.thecoderblogs.com/2011/06/unable-to-create-a-search-service-application-errors-were-encountered-during-the-configuration-of-the-search-application/
This is actually about multitenancy, but has some excellent code
snippets that helped me provision User Profile Synchronization and
Search service.
http://www.harbar.net/articles/sp2010mt5.aspx
I trully hope that these steps will help someone avoid the pain I
suffered :)
Regards,
Dinko

Date Published: Sep 30, 2011 - 3:52 am
Hi,
Recently I was engaged by my colleague to assist him in upgrading
an aging Sharepoint 2003 installation to Sharepoint 2010. Since the
full blown Sharepoint Portal Server 2003 was still in use there was
no direct upgrade path, but I had to deploy temporary MOSS 2007
farm first to upgrade database from Sharepoint 2003 to 2007
version. After this step I could upgrade to Sharepoint 2010.
I've run the pre-scan tool on Sharepoint 2003 which did gave me
some trouble saying that the site I was trying to scan is not
extended with WSS v2 and that I should extend it first. I've fixed
this problem with running a stsadm -o upgrade which, I guess, wrote
a new database version in the portal content database because the
process took only a couple of second. After that the pre-scan run
without any errors.
Now, onto the problem...
I've created a new web application on MOSS 2007 and used the
following command on the MOSS 2007 box:
stadm -o attachdb -url http://sps
-databasename SPSERVER_SITE -databaseserver -SQLSERVER
This returned an error and the database was not attached to the web
application.
Upgrade
completed with errors. Review the upgrade.log file located in
C:Progra
m FilesCommon FilesMicrosoft SharedWeb
Server Extensions12LogsUpgrade.log.
The number of errors and warnings
is listed at the end of the upgrade log file
.
After examining the log files I've found this:
[DropFullTextSearch] [3.0.2.0] [DEBUG] [9/6/2011
8:23:35 PM]: Begin Rollback()
[DropFullTextSearch] [3.0.2.0] [DEBUG]
[9/6/2011 8:23:35 PM]: End Rollback()
[DropFullTextSearch] [3.0.2.0] [DEBUG]
[9/6/2011 8:23:35 PM]: Begin Dispose()
[DropFullTextSearch] [3.0.2.0] [DEBUG]
[9/6/2011 8:23:35 PM]: End Dispose()
[DropFullTextSearch] [3.0.2.0] [DEBUG]
[9/6/2011 8:23:35 PM]: Elapsed time:
00:00:00.1405800.
[SPManager] [ERROR] [9/6/2011 8:23:35
PM]: Upgrade [SPContentDatabase Name=SPSERVER_SITE2007
Parent=SPDatabaseServiceInstance] failed.
[SPManager] [ERROR] [9/6/2011 8:23:35
PM]: Full-text crawl manager has not been initialized. Any crawl
started before the crawl manager was fully initialized will need to
be restarted. Please restart SQL Server and retry the command. You
should also check the error log to fix any failures that might have
caused the crawl manager to fail.
[SPManager] [ERROR] [9/6/2011 8:23:35
PM]: at
System.Data.SqlClient.SqlConnection.OnError(SqlException exception,
Boolean breakConnection)
at
System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject
stateObj)
at
System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior,
SqlCommand cmdHandler, SqlDataReader dataStream,
BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject
stateObj)
at
System.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String
methodName, Boolean async)
at
System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult
result, String methodName, Boolean sendToPipe)
at
System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at
Microsoft.SharePoint.Utilities.SqlSession.ExecuteNonQuery(SqlCommand
command)
at
Microsoft.SharePoint.Upgrade.SPDatabaseSequence.ExecuteDataDefinitionMethodCore(SqlSession
sqlSession, ISqlSession isqlSession, String sqlscript,
SPSqlCommandFactory sqlcmdFactory, String[] strTables, Int32[]
nThroughputs, SPLog logGlobal)
at
Microsoft.SharePoint.Upgrade.SPDatabaseWssAction.ExecuteDataDefinitionMethod(SqlSession
sqlSession, String sqlscript)
at
Microsoft.SharePoint.Upgrade.DropFullTextSearch.Upgrade()
at
Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()
It then occurred to me that back in the old days Sharepoint was
using SQL full text search catalogs for its search feature, but
just for WSS v2 search feature if memory serves me correctly.
When I installed the new SQL 2008 R2 I did not check the Full-Text
Search component because the MOSS 2007 and Sharepoint 2010 are not
using SQL Full-Text Search anymore. But it seems that for upgrade
purposes the Full-Text Search component is required because the
upgrade process tried to drop the existing full text catalog.
At this moment I've also reviewed the logs when I restored the
Sharepoint 2003 database from SQL 2000 to SQL 2008 R2 and found
this:
Log
Name: Application
Source:
MSSQLSERVER
Date:
6.9.2011. 20:15:13
Event ID:
3633
Task Category: Server
Level:
Error
Keywords:
Classic
User:
N/A
Computer:
SQL2000.domain.local
Description:
The operating system returned the error
'3(failed to retrieve text for this error. Reason: 15100)' while
attempting 'ClearTree' on 'FullTextDefaultPathix_SPSERVER_SITE' at
'fulltext.cpp'(1816).
The solution was to simply install the Full-Text Search component
on the new SQL Server 2008 R2 and run the database attach process
again.

Date Published: Sep 06, 2011 - 1:01 pm
Hello everyone,
I am experiencing an issue with SQL Server Agent and its output
file. I have a job that runs daily and I need to store its output
to txt file on a shared disk in a clustered environment. The job
runs the simple script "EXEC sp_help_revlogin" which is used to
generate "CREATE LOGIN" scripts for each SQL user account created
on that instance of SQL server (domain or local). The goal of this
script is to have current SQL login information in case of a
disaster recovery scenario. There is another SQL instance on the
remote site to which the databases are replicated using SQL
Database Mirroring, but Database Mirroring is not
replicating SQL user accounts because they are actually stored in
the master database which is not supported in Database Mirroring
configuration.
The problem I am experiencing is that the job is not creating any
output file. The output should contain the SQL script generated by
the sp_help_revlogin function where each local login is scripted so
it can be ran on the remote SQL server instance. The error message
I am getting is:
Message
Executed as user: DOMAINuser. Warning: cannot write
logfile E:DR_SCRIPTSDR_CREATE_LOGIN.txt. Writing to log files is
only allowed to jobs that are owned by sysadmin. Please consider
writing log to table. The step succeeded.
The user that owns this job is member of the sysadmin server role
and it is even a member of the local administrators group on the
SQL cluster nodes. So my guess is that this is simply a bug in the
SQL Server 2008 (for the reference, build number we are running is
10.0.4000). Even the SQL Server Agent service is running under the
account that is a member of the sysadmin role and is the local
administrator on both of the SQL cluster nodes.
The solution that worked for me was to change the owner of that SQL
Server Agent job to the sa account. After I did that, the job
successfully created the output file to the destination folder.
If someone has some deeper explanation why is this happening, I
would greatly appreciate it.
Dinko

Date Published: May 17, 2011 - 6:52 am
With the release of SP1 for Exchange 2010 it is now possible to
have Datacenter Activation Coordination mode (DAC) enabled in a
single AD site. This is perfect for smaller environments as it is
now possible to have Exchange 2010 disaster recovery possible where
two DAG members are separated across two system rooms in the same
building or maybe separate building but still in the same AD
site.
The configuration would usually consist of the following:
- Primary Datacenter with Exchange Data Availability Group
(DAG) member and a file share witness which is usually located on
Exchange CAS or HUB transport server
- Secondary Datacenter with second Exchange DAG member
This underlying cluster mechanism which is Failover Cluster
now has three votes - two DAG members and a file share witness. In
this setup we can loose any single machine but still have the
Exchange databases online.
However if the disaster strikes and we loose our primary data
center it means two out of three votes are lost and the Failover
Cluster mechanism will bring the entire cluster down. This
situation leaves the Exchange databases unavailable to users.
If you had databases mounted on EXMBX01 they would have status of
"Disconnected and Healthy" on EXMBX02 and if there were databases
mounted on EXMBX02 they would have status of "Dismounted".
Note: For the following procedure to work you should have
already enabled the Datacenter Activation Coordination mode using
the following command:
Set-DatabaseAvailabilityGroup -Identity dagname
-DatacenterActivationMode DagOnly
To mount databases on the Exchange DAG member in the secondary
data center you will need to type the following powershell
commands on the second DAG member:
Stop-Service clussvc
Stops the Failover Cluster service
Stop-DatabaseAvailabilityGroup –Identity EXDAG01 –MailboxServer
EXMBX01 -ConfigurationOnly
Stops the DAG on the failed Exchange DAG member. Since the DAG
member is down and unavailable we are using ConfigurationOnly
switch
Restore-DatabaseAvailabilityGroup –Identity EXDAG01
–AlternateWitnessServer EXHUBCAS02 –AlternateWitnessDirectory
C:EXDAG01_W02
This command sets up the new Failover Cluster which now consists
of only one cluster member (EXMBX02) and file share witness which
we will be placed on EXHUBCAS02 which is located in the secondary
data center. After the command finishes we should have our
cluster online and all databases mounted on EXMBX02. We can ommit
AlternateWitnessServer and AlternateWitnessDirectory switches if
we have previously set this up on the properties of DAG. We can
set this using Set-DatabaseAvailabilityGroup cmdlet.
If we now open the Failover Cluster management console we should
see only EXMBX02 as a member of the cluster and file share
witness should point to EXHUBCAS02.
But what when the primary data center comes back online? This is
when DAC magic comes into play. If EXHUBCAS01 and EXMBX01 are
brought online but the WAN link between the sites is still down,
DAC mode will not allow for the quorum to be formed even though
two out of three votes are available. This is because in DAC mode
each DAG member must successfully contact all other DAG members
or at least DAG member which has the Active Manager bit of 1
stored in memory. Since the EXMBX01 cannot contact EXMBX02 it
will not form a cluster and database will not be mounted thus
preventing the split brain scenario.
I recommend you read Scott Feltmann's
blog for more information on how this Active
Manager bit works.
When everything is back online again you should follow this steps
to put everything as it was before the disaster:
Start-DatabaseAvailabilityGroup
–Identity EXDAG01 –ActiveDirectorySite
Default-First-Site-Name
This command essentially ads EXMBX01 back into cluster. Databases
are now still mounted on EXMBX02 but the replication should be
resumed and all passive database copies on EXMBX01 should be
healthy. If not you will use Update-MailboxDatabaseCopy cmdlet to
remedy this.
At this point file share witness is still set on EXHUBCA02. We
can confirm this by opening Failover Cluster management console
but if we look at the DAG properties using
Get-DatabaseAvailabilityGroup cmdlet we will see that it shows
EXHUBCAS01 as file share witness. All we need to do is run this
cmdlet:
Set-DatabaseAvailabilityGroup –Identity EXDAG01
Failover Cluster management console now shows that EXHUBCAS01 is
file share witness.
There is one more thing to do and that is to move Active
databases to EXMBX01 and everything is now the way it was before
the disaster.

Date Published: Feb 18, 2011 - 12:51 pm
Hi,
I had a need to find the private key file for the certificate that
was stored inside the Local Computer Certificates store. I needed
to set the correct permissions for the private key so that system
service could access it.
When the specific Windows service started it logged the following
event in my Windows Application log:
System.ArgumentException was unhandled
Message="The certificate
'CN=localhost' must have a private key that is capable of key
exchange. The process must have access rights for the private
key."
Source="System.ServiceModel"
The private key file actually exists but the user account under
which the service runs does not have the permission for it.
Private keys for the certificates on Windows 7 and Windows Server
2008 operating systems are stored here:
C:ProgramDataMicrosoftCryptoRSAMachineKeys
One example of the private key filename is:
1b2c85ebc7a8c0d84076e7da2c608e29_d6b7bae8-7a97-4c58-b63b-5f0381938d6d
And there could be tens of private key files in that directory so
you do not know which private key corresponds to your
certificate.
The FindPrivateKey tool solves this as it helps you to pinpoint the
exact file that you are looking for.
Run the command like this:
FindPrivateKey.exe My LocalMachine -n
"CN=yourservername.domain.local"
And you will get output like this:
Private key directory:
C:ProgramDataMicrosoftCryptoRSAMachineKeys
Private key file name:
02d5c2fdc0f71d522f6011ca8b3b1493_d6b7bae8-7a97-4c58-b63b-5f0381938d6d
So now you know on which file you need to set permissions to.
If your FindPrivateKey output return this:
FindPrivateKey failed for the following reason:
Unable to obtain private key file name
Then the user account which you use to run the FindPrivateKey tool
does not have permissions to read that specific private key file
you are looking for. Try running the tool with "Run as
Administrator" if you are using UAC or try to run your command
prompt session with LocalSystem account.
You can do this by using psexec tool from Sysinternals:
psexec -s -i cmd
The new command prompt opens that runs under LocalSystem account
and you can run FindPrivateKey tool from that window.
If you still get the "Unable to obtain private key file name" then
you are just out of luck :)
You will need to try and pinpoint to the correct private key file
manually because neither you or the system has access to it.
Dinko

Date Published: Feb 11, 2011 - 4:48 am
This is a procedure to generate a certificate that you can import
to your Ironport Email Security appliance.
Here are the basic steps:
- Generate a certificate request using OpenSSL for Windows.
Change the command line to your liking. The important thing is to
change the "ironport.domain.com" to the URL that you want to use
to access your Ironport appliance.
openssl req -new -newkey rsa:2048 -nodes -out
ironport_domain_com.csr -keyout ironport_domain_com.key -subj
"/C=HR/ST=Grad
Zagreb/L=Zagreb/O=Organization/OU=IT/CN=ironport.domain.com"
- Sign the request file (CSR) using Windows CA. You can use web
application (https://servername/certsrv) of your Issuing CA and
then paste the CSR there and use the Web Server template.
- Convert the output CER file to PEM file
openssl.exe x509 -in ironport.cer -inform der -out
ironport.pem -outform pem
- Generate a P12 file that includes private and public keys
openssl.exe pkcs12 -export -out ironport.p12 -in ironport.pem
-inkey ironport_domain_com.key
- Import the P12 file to your Ironport using the web GUI
(Network > Certificate > Add Certificate)

Date Published: Jan 28, 2011 - 4:01 pm
I wrote a custom Group Policy Administrative Template that sets the
"Always send a read receipt" in Outlook clients. This custom ADM
template is useful if you want to set this property to all or
subset of users in a domain but still want to give them the ability
to control other properties in Tracking options in Outlook that
would otherwise be disabled if you use Office administrative
templates downloaded from Microsoft site.
Just copy/paste the following code in notepad and save the file
with ADM extension. Import the file in Group Policy object under
"User Configuration".
CLASS USER
CATEGORY "Outlook 2003 Receipt Response"
POLICY "Always send response"
KEYNAME SoftwareMicrosoftOffice11.0OutlookOptionsMail
PART "Always send response for read receipt requests"
CHECKBOX
VALUENAME "Receipt Response"
VALUEON NUMERIC 0 DEFCHECKED
VALUEOFF NUMERIC 1
END PART
END POLICY
END CATEGORY
CATEGORY "Outlook 2007 Receipt Response"
POLICY "Always send response"
KEYNAME SoftwareMicrosoftOffice12.0OutlookOptionsMail
PART "Always send response for read receipt requests"
CHECKBOX
VALUENAME "Receipt Response"
VALUEON NUMERIC 0 DEFCHECKED
VALUEOFF NUMERIC 1
END PART
END POLICY
END CATEGORY
CATEGORY "Outlook 2010 Receipt Response"
POLICY "Always send response"
KEYNAME SoftwareMicrosoftOffice14.0OutlookOptionsMail
PART "Always send response for read receipt requests"
CHECKBOX
VALUENAME "Receipt Response"
VALUEON NUMERIC 0 DEFCHECKED
VALUEOFF NUMERIC 1
END PART
END POLICY
END CATEGORY

Date Published: Jan 28, 2011 - 2:57 pm
Hello,
Due to the problems and pain we have encountered in making
Forefront TMG 2010 Standalone Array in a workgroup to work on
VMware ESX 3.5 Update 5 I will detail the steps for creating and
importing certificates to TMG certificates store and point out to
the problems with TMG Control service dependencies.
This is the environment we had:
- Two Forefront TMG 2010 Enterprise Servers in a workgroup
configured in Standalone Array with one TMG configured as Array
Manager and another configured as Array Member
- Windows Server 2008 R2 Standard
- Virtual machines on VMware ESX 3.5 Update 5
During the implementation we have experienced the problem with
Forefront TMG Control service taking 10 minutes to start after a
server restart. The service would eventually start but the other
Forefront services that depend on it will fail to start.
The problem was solved implementing the following fixes:
I do not imply with this article that the problem with TMG Control
service hang is related only to VMware ESX but we only experienced
it on this platform. When we encountered the problem we did some
tests on Hyper-V environment and on separate VMware ESX 3.5 Update
5 environment and there was no problem, however on this particular
environment the service would not start once the TMG array was
configured.
So I would recommend for anyone to get these dependencies fixed
even if you do not encounter this problem now. Regarding the
problem with certificates, I have already
blogged about it here but I also wrote a
procedure how to properly issue Server Authentication certificate
for TMG arrays in a workgroup.
How to issue a proper "Server Authentication"
certificate
Prerequisites:
- Access to any Windows Server 2008 IIS 7.0 web server
- Access to Enterprise or Standalone Windows Sever 2008
Certification Authority (Windows 2003 CA is also okay)
1. Open the IIS Manager, click on server name node from the left
pane and click on "Server Certificates" from the middle pane
2. Click on the "Create Certificate Request" from the right
pane
3. In the "Common name" field type the FQDN of the TMG server that
will act as an Array Manager. In this example we will use
"tmg01.company.hr". Fill the remaining fields so that you best
describe your organization.
4. Choose "Microsoft RSA SChannell Cryptographic Provider" for the
"Cryptographic service provider" and choose 2048 for the "Bit
lenght".
5. Save the certificate request as C:tmg01.req.
6. Navigate to the Issuing or Root CA web site such as
https://yourservername/certsrv and click on "Request a
certificate"
7. Click on "advanced certificate request"
8. Click on "Submit a certificate request by using a
base-64-encoded CMC or PKCS #10 file, or submit a renewall
request by using a base-64-encoded PKCMS #7 file".
9. Paste the contents of the tmg01.req file that you have created
earlier from IIS to the "Base-64-encoded certificate request"
field. In case you have a drop-box with Certificate Templates list,
select "Web Server" template.
10. Your certificate request is now submitted to the CA. In case
the "Request Handling" property of your CA is set to automatically
issue certificates you will be presented with the following page
where you have the possibility to download your issued "cer" file.
Click on "Download certificate" and save the file as C:tmg01.cer.
Go to the step number 15.
In case the "Request Handling" is set to manually issue the
certificates by the administrator then you will have to perform the
following steps.
11. Open the "Certification Authority" console on your Issuing CA
server and click on "Pending Requests". You should see your request
in the right pane.
12. Right click on the request and select All Tasks > Issue.
13. Browse to the CA web site again
(https://yourservername/certsrv) and click "View the status of the
pending certificate request". There should be your
"Saved-Certificate Request" listed.
14. You are now presented with the same page as in step number 10.
Download the "cer" file as described in step 10 and proceed to step
15.
15. Now return to the IIS Manager console from which you have
created the certificate request and now select "Complete
Certificate Request".
16. In the "Specify Certificate Authority Response" screen browse
to the "cer" file you have downloaded from the CA and enter a
friendly name for the certificate. I usually type the same name as
common name.
You have now completed the procedure of issuing the "Server
Authentication" certificate. If you open the "Local Computer"
Certificates store on the server where you have requested the
certificate you should see the certificate in the Personal >
Certificates folder. The certificate icon should have a little
yellow key pictured which means that you have both private and
public key. We must export the certificate with private and public
keys so that we can import it on our TMG server.
17. Right click on the certificate and click All Tasks >
Export.
18. Select "Yes, export the private key".
19. "Personal Information Exchange - PKCS #12 (.PFX)" should be
selected. Unmark all the checkboxes and click Next.
20. Type the password that you will need to type when you import
the certificate to the TMG computer.
21. Save the certificate as C:tmg01.pfx.
Now that we have our certificate ready for import there is still
one thing we must do. Since we are creating TMG array in a
workgroup mode we must import the root certificate of the CA that
issued the certificate to all of the TMG servers that will
participate in array. But first we must export the root CA
certificate from a computer that has it.
22. Open the "Local Computer" Certificates store on the Issuing CA
computer or on some other computer which is a domain member in a
domain where CA resides.
23. Navigate to the Trusted Root Certification Authorities >
Certificates, right-click on the root certificate from the CA which
issued your certificate and select All Tasks > Export.
24. Select "DER encoded binary X.509 (.CER)" and click Next.
25. Save the "cer" file to disk. In our example it is
C:CompanyRootCA.cer.
Now we have both the PFX file which contains our public and private
keys for the TMG computer certificate and a CER file that contains
a public key from our root CA. The next thing we must do is to
import the root certificate to each TMG server that will
participate in the array and to import the "Server Authentication"
certificate.
Note: It is good practice to create "Server Authentication"
certificate for all TMG servers so that if Array Manager fails you
can promote some other Array Member to Array Manager.
26. Open the "Local Computer" Certificates store on each TMG server
and import the root certificate "cer" file to the "Trusted Root
Certification Authorities".
27. Now open the "Forefront TMG Management" console on the TMG
server that will act as an Array Manager. Expand "Forefront TMG" in
the left pane and click on System node. Click on the TMG server
name in the center pane and click on the "Install Server
Certificate" in the right pane.
28. Now browse to the "pfx" file you have exported from the web
server computer and type a password for the file. Unmark the
checkbox "Automatically create the root CA certificate on this
array manager." To my experience leaving this checkbox marked
always resulted in an error even though the pfx file contained the
root CA certificate. Click OK.
Now if you open the Certificates store for the Windows service
named ISASTGCTRL you should see the imported certificate with the
private key in the Personal store.
So why is important to use Forefront TMG Management console to
import the certificate? You could just import the certificate in
the Local Computer Certificates store, right? Well the answer is
yes and no. If you do it this way the ISASTGCTRL service will not
have enough permissions to read private key file that is stored
in the C:ProgramDataMicrosoftCryptoRSAMachineKeys and you would
get Schannel errors in the Windows event log and TMG Control
service would not be able to communicate with the ADAM service
(ISASTGCTRL) on the Array Manager computer. If you use Forefront
TMG Management console to import the certificate all the
necessary permissions are added to the private key file
including:
- SYSTEM
- NETWORK SERVICE
- fwsrv
Of course you could manually update the permissions on the
private key file if you knew which private key it is and things
would work but it is not the proper way to do this.
Test the connection
Now there is only thing left and that is to test the secure LDAP
connection to the Array Manager server. We will use ldp.exe for
this. You should be able to run it from your TMG servers.
Open ldp.exe and click on Connection > Connect. Type FQDN of
your TMG server that will act as Array Manager and type 2172 for
the port number as this is the port on which ISASTGCTRL service
listens. Click on the SSL and click Connect.
If the connection is successful you will see the screen like the
following:
And that is all there is to it! Make sure to complete the procedure
for all TMG servers that will participate in the array for the
already mentioned reason and that is so that another Array Member
can become Array Manager in case the Array Manager fails.

Date Published: Jan 26, 2011 - 4:56 am
Hi,
I just want to point out to a problem I have just resolved.
Customer reported a problem when accessing Sharepoint 2010 site
that is published using Forefront TMG 2010. Sharepoint listener is
configured to use forms-based authentication with Persistent
Cookies enabled for both public and private computers. For those
who wants to know how this works you can read my article
here.
The problem was that when the Private computer was selected in the
authentication form when accessing site it worked fine but when the
Public or Shared computer was selected the user could not be
authenticated. The TMG log showed the following error:
12302 - The server denied the
specified Uniform Resource Locator (URL). Contact the server
administrator.
When the listener was modified to use Persistent Cookies only for
the Private computers then the authentication worked also for
Public computers but then the desired authentication cookie was not
saved to the local client cache. The problem appeared on IE,
Firefox and on different client computers so it was not related to
the local client cache or something like that. TMG and Sharepoint
servers were restarted but it also did not help.
The problem was resolved only by recreating the Sharepoint listener
on TMG using all the same properties as before. Now both the Public
and Private computers work with Persistent Cookies enabled.
Dinko

Date Published: Jan 23, 2011 - 3:21 am
Hello,
For the last two days I was attending Private Cloud workshop in
Microsoft office in Warsaw. The correct name of the workshop was
actually Government Private Cloud Computing but the workshop
actually gave great overview about types of "clouds" and the types
of services that can be delivered through the cloud. It also
covered every aspect of building a cloud solution from technology
to process management, automation and billing. What strike me the
most is actually how easily the specific cloud type can be defined.
I am reading about the cloud infrastructure for a while now and I
have even been architecting and implementing public cloud solutions
based on Exchange 2007/2010 and Sharepoint 2007/2010 and the
workshop helped me to sort in my head everything I have learned so
far. The main goal of the workshop was to teach us how to have a
conversation with a customer company that plans to go to "the
cloud" and how to ask some basic questions that will help us define
the cloud or the service that is to be offered through the
cloud.
In the next couple of lines I will try to summarize the different
service delivery types and the different types of cloud and map
those to the actual technology or a scenario based on my personal
experience.
Let us first cover the service delivery types:
- Infrastructure as a Service (IaaS)
- Platform as a Service (PaaS)
- Software as a Service (SaaS)
Infrastructure as a Service
|
|
IaaS
|
The blue boxes shows what Service Provider actually
provides to the customer. We can see that those elements are
Datacenter, Networking, Computers and Virtualization. In practice
that would mean that the provider company takes care of system
rooms, electricity and cooling (Datacenter), networking components
such as switches and routers (Networking), servers that comprise
the monitoring, provisioning, billing and virtualization
infrastructure (Computers) and manages virtualization technology
such as Hyper-V or VMware clusters. Everything above can be
dynamically created, self-serviced and pooled by the customer. In
this certain case it would mean that the customer can
self-provision virtual machines with the operating system of
choice, pool the amount of storage he needs for the virtual machine
and install the applications he needs for his business. In a
Microsoft world this functionality could be provided by System
Center VMM Self-Service Portal 2.0.
Platform as a Service
|
|
PaaS
|
Now let us move a few boxes upwards. If the operating
system and the storage is controlled by a service provider company
and the customer can deploy it's own applications that run on the
underlying platform of choice then this model is called Platform as
a Service or PaaS. Middleware box is grey area here. Let us
consider a scenario where the customer gets presented by an empty
virtual machine where he can deploy his own engine like Oracle
Application Server and build his own applications on top of that.
Then the middleware would be controlled by a customer. Or we could
provide the customer with Microsoft Sharepoint 2010 site collection
and give him the ability to deploy his own web parts or workflows.
The customer would not have access to the operating system in this
case. He would only have access to his own little isolated area on
Sharepoint and anything he deploys there is relevant only to him
and it does not affect other customers. Sharepoint 2010 has
sandboxed solutions integrated so this would make for a perfect
example of Platform as a Service.
Software as a Service
|
|
SaaS
|
If the service provider company has the total control of
the stack and the customer only consumes services such sending and
receiving of e-mail, upload documents to sharepoint or scheduling
and running voice conferences than this is called Software as a
Service. Customer only sees the service he uses and has no control
of any aspect of the underlying platform. Self-service in this case
would mean that the customer administrator can provision a new
mailbox for an employee inside his own company or could enable the
user for voice conferencing so that he can share video and audio
with his colleagues. Examples of these in a Microsoft world are of
course Exchange, Sharepoint, OCS and CRM.
There are few other aspects of the service delivery models that
must be met so that we could call our solution a "cloud solution".
The workshop I have attended specified the following
characteristics which I will try do describe in my own words:
-
Resource Pooling - the ability to share resources such
as network, memory and processing power and provide the
resources when needed to the workload that needs them
-
Measured Service - the ability to measure service
utilization per customer such as number of mailboxes he used
during the last month, the amount of GBytes he uses on the
storage for mailbox, documents, databases etc. or bandwith that
was consumed. The most important thing is that the customer can
be charged only by what he spent the last month. If the number
of mailboxes decreased or the bandwith utilization dropped from
the previous month, the customer should be charged accordingly!
-
Broad Network Service - the service should be possible
to be accessed from anywhere using any Internet connection
-
Rapid Elasticity - the service should be possible to
scale out or scale up in the shortest amount of time. Consider
adding new virtualization host to the infrastructure to
increase processing power or adding new disk shelf to increase
storage space or increase performance. The service should also
be able to rapidly scale down if resources are not used. An
example for this is moving running virtual machines to the
smaller number of hosts and then shutting down hosts with no
load.
-
On-Demand Self Service - the customer should be able to
self-provision the service such as create a new mailbox, add
additional disk space to the virtual machine or enable an
employee for Sharepoint or OCS access
Now when we have defined the service delivery models and cloud
service characteristics we will cover the cloud deployment models.
There are four deployment models:
-
Private cloud - this cloud infrastructure is operated
solely for a specific organization. It can be managed by an
organization itself or by a service provider and may exist on
premise or off premise. Owner of the infrastructure components
can be either the customer or the service provider. I can think
of a scenario in Microsoft Online Services and e-mail hosting.
If you have large enough company with plenty of e-mail users,
Microsoft would provide a dedicated infrastructure just for you
and it would be managed and operated by Microsoft for your
company only. Or there could be another reason for a private
cloud infrastructure. An organization security policy could
specify that data must somehow be separated and isolated
from other organizations which would require a dedicated e-mail
server just for the specific organization or a dedicated
hardware infrastructure.
-
Community cloud - the cloud infrastructure that is
shared by several organizations and supports a specific
community that has shared concerns such as mission, security
requirements, policy and compliance considerations. A good
example for this scenario are government ministries or agencies
that could share the same common infrastructure.
-
Public cloud - this cloud infrastructure is made
for general public and owned and maintained by an organization
(service provider) that sells cloud services. Examples would be
e-mail services, shared or dedicated Sharepoint servers,
virtual machines etc.
-
Hybrid cloud - composition of two or more models
(private, community or public) that remain unique entities but
are bound together by standardized or proprietary technology
that enables data and application portability.
Technology
So where does the technology fits in? Almost every major vendor out
there today is delivering its own solutions that are cloud enabled
or will help you to get to the cloud. Microsoft naturally has its
own set of solutions that I will try to cover briefly here.
-
Hardware - hardware selection is totally up to you. You
can choose server, storage or network vendor you like or
already work with
-
Virtualization - there are at least three out there that
I would recommend, Hyper-V, VMware and XenServer, they all do
their jobs excellent
-
Deployment - System Center Configuration Manager is a
perfect tool for deploying new operating systems, either
provisioning new desktops for the customer or new Hyper-V hosts
for the service provider or patching the existing
infrastructure
-
Monitoring - System Center Operations Manager - used for
monitoring the entire environment and alerting in case
something goes wrong or you just want to be notified about. It
can also be used as a billing tool to show your customer how
much "resources" have been used for the past month.
-
Automation - Microsoft has a new member of the System
Center family and it is called Opalis. It is used for
Datacenter automation. It enables you to automate certain
Datacenter processes and helps you get "elasticity" for your
cloud infrastructure. One example of automation could be
installing new OS on the server, joining it to virtualization
cluster, deploying new patches and updates and deploying SCOM
agent for monitoring. You can do that with this tool. Microsoft
also provides already mentioned System Center Virtual Machine
Manager Self-Service Portal 2.0. It can be used for automating
virtual machine deployment and it is also customizable in a way
it can be further developed or branded for each customer.
-
Provisioning - provisioning application in the cloud
infrastructure is used in two different ways. If you work for
service provider company then you would probably use it to
quickly provision new customer and give him access to your
services. If you are customer, you will probably use the
provisioning application (usually a web interface) to quickly
create new virtual machines or create new mailboxes for your
employees. Microsoft has no unified software that would deal
with all scenarios but here are some examples. Exchange 2010
with SP1 has multi-tenancy feature integrated and it provides
the so-called Exchange Control Panel or ECP. If you are a
customer that is provisioned on a shared or hosted Exchange
2010 server you would probably get access to ECP and have a
possibility to manage mailboxes for your employees only. Other
customers on that same Exchange server are completely isolated
from your users. However, the provisioning application for the
service provider company in regard to Exchange 2010 server does
not really exist. There is only a powershell interface you can
use or develop your own. But luckily there are companies out
there who already have a software for that purpose.
Well I hope that I have helped you to understand some of the basics
of the cloud infrastructure and that the cloud paradigm is not so
cloudy to you anymore :)
Please feel free to comment on this post and give your view about
the terms and definitions about "the cloud". I have to be honest
and say that I am by no means an expert on this field because the
topic is so huge and it would probably take years of experience to
understand it completely but I can definitely say that the workshop
I have attended helped me to get some basic understanding.
Regards,
Dinko

Date Published: Jan 19, 2011 - 11:09 am
Last couple of days me and my colleague where troubleshooting a
brand new installation of Forefront TMG Enterprise Standalone Array
consisting of two nodes. The problem we had was that after a server
restart, Forefront TMG Control service would not start. It would
hang in a starting state for about 10 minutes after which it would
eventually start but the TMG Firewall service and all other TMG
services that depend on TMG Control service did not start because
of this timeout. After that we could manually start the services
and the TMG Array worked with no problem. It was only the problem
after server reboot.
The environment:
- Two Windows Server 2008 R2 Standard virtual machines on
VMware ESX 3.5 Update 5 environment
- Forefront TMG 2010 Enterprise SP1 Software Update 1
- Forefront Standalone Array in Workgroup mode with one node
designated as Array Manager and the other one as Array Member
- Each node had a server certificate installed in local
computer store with Extended Key Usages for Server Authentication
and Client Authentication
Here are some of the errors we were seeing in the event log:
The Microsoft Forefront TMG
Control service hung on starting.
The Microsoft Forefront TMG
Firewall service depends on the Microsoft Forefront TMG Control
service which failed to start because of the following error:
After starting, the service hung in a start-pending
state.
The Microsoft Forefront TMG
Managed Control service depends on the Microsoft Forefront TMG
Control service which failed to start because of the following
error:
After starting, the service hung in a start-pending
state.
The Microsoft Forefront TMG
Job Scheduler service depends on the Microsoft Forefront TMG
Control service which failed to start because of the following
error:
After starting, the service hung in a start-pending
state.
Log
Name: Application
Source: Windows Error
Reporting
Date:
14.1.2011. 14:57:24
Event ID: 1001
Task Category: None
Level:
Information
Keywords: Classic
User: N/A
Computer: *************
Description:
Fault bucket , type 0
Event Name: ServiceHang
Response: Not available
Cab Id: 0
Problem signature:
P1: isactrl
P2: mspadmin.exe"
P3: 0.0.0.0
P4: 10
P5: 2
P6:
P7:
P8:
P9:
P10:
Attached files:
These files may be available here:
Analysis symbol:
Rechecking for solution: 0
Report Id: 363731e9-1fe6-11e0-846f-00155d274102
Report Status: 0
To better understand the problem here are some technical
details.
TMG members in the Standalone array communicate with the array
manager which has AD LDS (Active Directore Lightwight Directory
Services) installed which provides configuration storage for the
entire array. The array manager would first save the configuration
to its local AD LDS instance and the rest of the array members
connect to it using Secure LDAP which require server certificate
with "Server Authentication" key usage. Actually, only the Array
Manager requires the certificate and the Array Members require root
certificate from the Certification Authority that signed the
certificate located in Trusted Root Certification Authorities Store
so that they would trust Array Managers certificate. But in case
the Array Manager fails you would have to manually promote one of
the Array Members to Array Manager and he would then require server
certificate installed. So we have installed server certificate on
both TMG computers.
Here comes the problem.... Since Array Member had its own server
certificate with Extended Key Usage and Intended Purposes set to
Server Authentication and Client Authentication, when
authenticating to remote AD LDS service it would present its client
certificate and this process is known as mutual authentication or
MTLS. Well it seems that TMG Control service does not like this
behavior and it times out for about 10-15 minute after which none
of the TMG services start. The problem happened on both TMG
computers even though one TMG was Array Manager, he still needed to
connect to local AD LDS instance but he too tried to mutually
authenticate to the local AD LDS service.
Well, the solution was quite simple in fact, but very hard and
frustrating to find because there were almost no relevant logs to
look at. On each TMG computer the certificate properties should be
modified to include only Server Authentication for Intended
Purpose.
Here is how to do it:
- Open Certificates MMC snap-in and connect to Local Computer
- Navigate to Personal > Certificates >
your_computer_certificate (the certificate should have common
name of FQDN of your TMG computer)
- Double click on the certificate, click on the Details tab and
click Edit Properties
- Choose "Enable only the following purposes" radio button and
check Server Authentication
- Restart your computer and see if TMG services start normally
Of course, if your certificate only has Server Authentication in
Extended Key Usage field then you will not experience this
issue.
Microsoft also had something to say about this issue in the
following article, but not directly related to this problem.
Client logon is slow
and server certificates used for Web publishing are configured
with the default purpose settings "Server Authentication" and
"Client Authentication"
Issue: When Windows Server
2003 detects the default purpose setting of "Client
Authentication", the operating system attempts to perform TLS
with mutual authentication to the domain controller. The mutual
authentication process requires ISA Server to have access to the
private key of the server certificate with the "Client
Authentication" setting enabled, and ISA Server does not (and
should not) have this access.
Solution: Ensure that all
server certificates do not have the default "Client
Authentication" purpose enabled. You can disable this setting on
the property pages of the relevant server certificate as
follows:
Disable Client
Authentication purpose on a certificate
1. Open the Certificates Microsoft
Management Console (mmc) snap-in. To add the Certificate Manager
to the mmc, do the following:
·
Click Start, and
then click Run.
·
Type mmc and then
press ENTER.
·
Select the File
menu, and then select Add/Remove Snap-in.
·
In the Add/Remove
Snap-in box, and then click Add.
·
Double-click the
Certificates snap-in, select Computer Account, and
then click Finish.
·
Select Local
Computer, and then click Finish.
·
Close the dialog
boxes.
2. In the Certificates mmc, click to
expand the Certificates node, and then expand
Personal.
3. Right-click the relevant certificate
and then click Properties.
4. On the Details tab, click
Edit Properties.
5. Select Enable only the following
purposes, and clear the Client Authentication
purpose.
Link to the entire article
here.
While troubleshooting this issue, out of pure frustration we even
replicated the entire environment on Windows Server 2008 SP2 and
later even on Hyper-V as the virtualization platform to eliminate
any compatibility issues but finally it seems that this little
setting did the trick.
We have also tried, read this carefully, Rollup 1 and Rollup 2 for
Software Update 1 for Service Pack 1 for TMG 2010 just to be sure
we had the entire environment patched and read numerous blogs that
talked about TMG Control service dependency issues that would arise
after installation of TMG updates and rollups but none of those
worked.
I really hope this article will someday save a lot of time to
someone :)
P.S.
Here is a
link to a blog article that describes some other
startup issues that you may have with TMG related to service
dependency ordering.

Date Published: Jan 14, 2011 - 11:38 am