Update - Creating SharePoint 2010 user profile exclusion filters using Powershell

This post is actually a follow-up on my previous post on user profile exclusion filters.

A quick recap of that post:

• I needed to script the creation of a user profile service application, along with some profile exclusion filters.
• Unfortunately I couldn't find any existing reference on script the creation of user profile exclusion filters (only guides on how to do this in the central admin pages).
• Took a peek at the code SharePoint uses to create exclusion filters in central admin, only to see they use internal/hidden classes & methods to create these filters.
• In the end I couldn't find any public API/powershell cmdlet to create these filters, so ended up writing my own workaround: a small .NET 3.5 project that will invoke the internal/hidden SharePoint 2010 methods using reflection.

Still, for me there was something missing: an easy way to use this workaround in PowerShell. Cause we all know: Powershell rocks! :p

First attempt

Originally I wanted to translate the complete .NET 3.5 workaround into one (or more) ps1 scripts. This would eliminate all need for external assemblies and is just easier to read/fix should something happen.
I managed to translate 99% of the .NET code, which was actually easier than I thought it would be. It was a nice exercise in the usage of reflection and generics within PowerShell 2, so I might make a specific blogpost on this subject later on :)

However, 99% is not 100%. I just couldn't get the final step working: how to assign the FilterSet objects (the actual filters) to the user profile connection.
Internally SharePoint 2010 uses the SetExclusionFilters method - something I had to call using reflection. This worked like a charm from within a .NET project, but I couldn't get PowerShell to cast the objects to the correct internal classes (despite the fact that Get-Member did mention it was casted correctly).
As a result, I kept getting the following error ... no matter what I tried

Exception calling "Invoke" with "2" argument(s): "Parameter count mismatch."
At Functions.ps1:96 char:35
+ $setExclusionFiltersMI.Invoke <<<< ($connection, @($filterSetList)) + CategoryInfo : NotSpecified: (:) [], MethodInvocationException + FullyQualifiedErrorId : DotNetMethodException

When I find the time, I'll post the ps1 scripts too ... maybe one of you can solve this mystery.

First Second attempt

Soooo translating it all into ps1 scripts didn't work, but no worries! Time for a workaround .. for a workaround?!
This time I just wrote another .NET project, which defines some custom cmdlets. These cmdlets will then invoke the .NET code that I described in my previous post. On the bright side: the cmdlets can be easily used from within PowerShall and it actually works! :)
(But you're still stuck with a bunch of .NET assemblies that you cannot change on the fly ... can't have it all, I guess.)

Before I start a lengthy description of the cmdlets I developed, I'll post provide a quick overview. Use these links if you can't be a***d / bothered to read manuals :)

• Detailed cmdlet overview
• How to use this in SharePoint 2010's management shell
• Downloads
• Examples

The project defines these cmdlets

• Get-FilterSetCondition
• Get-FilterSet
• Set-ExclusionFilter
• Get-FilterAttributes
• Get-FilterSetConditionOperators
• Get-FilterSetOperators

Get-FilterSetCondition
Use this cmdlet to create a new condition for your exclusion filter. These conditions are internally stored as FilterSetCondition objects, hence the naming. You can find an example of this in Example1.ps1 and Example2.ps1
Syntax:

Get-FilterSetCondition -Connection $connection `
                       -ConditionType "user" `
                       -Attribute "userAccountControl" `
                       -Operator "Bit_on" `
                       -Value "2"

Parameters:

• Connection
  The connection on which you want to create an exclusion filter (Microsoft.Office.Server.UserProfiles.Connection object)
• ConditionType
  Whether this condition applies to a user-related attribute or a group-related attribute (just like you have separate windows in the UI to specify user and group filters)
• Attribute
  The attribute on which you want to filter, e.g. company or userAccountControl. Use the Get-FilterAttributes cmdlet to get an overview of all available attributes.
• Operator
  The operator that you want to use in this condition, e.g. "Bit_on" or "Equals". Use the Get-FilterSetConditionOperators cmdlet to get an overview of all available operators.
• Value
  The string-value you want to use in this condition.

Get-FilterSet
Use this cmdlet to create a new exclusion filter, based on conditions you've created using the Get-FilterSetCondition cmdlet. Do not mix user-based conditions with group-based conditions though! You can find an example of this in Example1.ps1 and Example2.ps1
Syntax:

Get-FilterSet -FilterType "user" `
              -Operator "or" `
              -Conditions $userConditions

Parameters:

• FilterType
  Whether this filter contains user-related conditions or a group-related condition
• Operator
  The operator that you want to use to combine the conditions. Either supply "or" or "and" as value.
• Conditions
  An array of FilterSetCondition objects (which you created using the Get-FilterSetCondition cmdlet)

Set-ExclusionFilter
Use this cmdlet to assign exclusion filters (created using the Get-FilterSet cmdlet) to an existing user profile connection. You can either assign a user-based exclusion filter, a group-based exclusion filter or both.
You cannot however assign multiple user-based exclusion filters or multiple group-based exclusion filters. You'll have to combine these into a single user-based exclusion filter or a single group-based exclusion filter. An example of this cmdlet can be found in Example1.ps1 and Example2.ps1
Syntax:

Set-ExclusionFilter -Connection $connection `
                    -UserFilter $userFilter `
                    -GroupFilter $groupFilter

Parameters:

• Connection
  A reference to a user profile connection (Microsoft.Office.Server.UserProfiles.Connection object)
• UserFilter
  The user-related exclusion filter you want to set on the connection. Don't supply this parameter if you don't want to set a user-related exclusion filter.
• GroupFilter
  The group-related exclusion filter you want to set on the connection. Don't supply this parameter if you don't want to set a group-related exclusion filter.

Get-FilterAttributes
Use this cmdlet to get a list of all available user-related or group-related attributes on a given user profile connection. Each of the returned strings can be used in the Get-FilterSetCondition cmdlet to create a condition on. You can find an example of this in Example3.ps1
Syntax:

Get-FilterAttributes -Connection $connection -User
Get-FilterAttributes -Connection $connection -Group

Parameters:

• Connection
  A reference to a user profile connection (Microsoft.Office.Server.UserProfiles.Connection object)
• User
  A switch parameter. Supply this if you want to list all user-related attributes on the connection. Cannot be used in combination with the -Group switch
• Group
  A switch parameter. Supply this if you want to list all group-related attributes on the connection. Cannot be used in combination with the -User switch

Get-FilterSetConditionOperators
Lists all available operators that you can use in Get-FilterSetCondition

Get-FilterSetOperators
Lists all available operators that you can use in Get-FilterSet

That's brilliant ... but how do I use this in SharePoint's management shell?!

1. Download either the complete package or the standalone archive. Look in the download section for a link ...
   
   • The standalone package only contains the necessary assemblies and some example scripts.
   • The complete package also contains the source-code I used, so you could make your own modifications. The source-code is located in the src folder, but I've also copied the necessary assemblies and examples in the distribution folder for easy access
2. Whatever you downloaded, copy the VNTG.UserProfileFilters.PowerShell.dll and VNTG.UserProfileFilters.dll assemblies to a folder on your SharePoint 2010 server.
3. Open a new SharePoint 2010 Management Shell on that server
4. Load the cmdlets using Import-Module ".\VNTG.UserProfileFilters.PowerShell.dll"
5. That's it! You can now use the cmdlets to create exclusion filters. Don't forget to check out the Example*.ps1 scripts that are also present in the downloaded packages. These scripts provide you with some real-life scenarios to create exclusion filters using the new cmdlets.

Disclaimer and downloads

Like I mentioned in the first part of this post: the workaround is based on reflection, in order to call the internal classes/methods that SharePoint 2010 uses in its Central Admin pages.
I'm sure they're hidden for a reason, so this workaround is definately not officially supported!

USE IT AT YOUR OWN RISK!

Neither I nor my bosses at Ventigrate can be held reliable if something goes wrong!

(The download is hosted at their codeplex project, so they forced me to add this disclaimer ... :p)

If you're interested in the complete source: use this download. This zip-archive contains the source of both the .NET workaround mentioned in the first part, as well as the powershell cmdlets. It also comes with additional examples to clarify its usage.

If you're not interested in any source code, but just want to use this workaround in your SharePoint 2010 Management Shell: use this download. This archive only contains the necessary .NET assemblies and some example ps1 scripts on how to use the cmdlets in your Powershell 2 console.

Examples

I'll list the same example that was mentioned in my original post - only this time using the custom cmdlets. This example is also available as Example1.ps1 in the downloads.
This code will create a user-attribute based exclusion filter and a group-attribute based exclusion filter and assign it to a user profile connection.

• The user-attrib exclusion filter will filter on "userAccountControl bit on equals 2" or "company equals ventigrate"
• The group-attrib exclusion filter will filter on "name equals testgroup"

Check the Example2.ps1 script if you want an example that only adds a user-attribute based filter.

#region STEP 1: load the necessary references
#--------------------------------------------
# Load the sharepoint powershell cmdlets
Add-PSSnapin "Microsoft.SharePoint.PowerShell" -ErrorAction SilentlyContinue | Out-Null

# Load the custom userprofile cmdlet dll
Import-Module ".\VNTG.UserProfileFilters.PowerShell.dll"
#endregion

#region STEP 2: get the user profile connection on which you want to create exclusion filters
#--------------------------------------------------------------------------------------------
# Get a reference to the service application called 'User Profile Service Application' (very original)
# and a connection called Example 1.
# Change these parameters to whatever user profile service app & connection you want to change on 
# your environment...
$serviceAppName = "User Profile Service Application"
$connectionName = "Example 1"

$profileApp = Get-SPServiceApplication | ? {$_.DisplayName -eq $serviceAppName}
$profileContext = [Microsoft.SharePoint.SPServiceContext]::GetContext(
          $profileApp.ServiceApplicationProxyGroup, 
          [Microsoft.SharePoint.SPSiteSubscriptionIdentifier]::Default)
          
# Get a new UserProfileConfigManager
$configManager = New-Object Microsoft.Office.Server.UserProfiles.UserProfileConfigManager($profileContext)

# Get the connection called "Example 1" in the service application
$connection = $configManager.ConnectionManager | ? {$_.DisplayName -eq $connectionName}

#endregion

#region STEP 3: create the conditions and bind them into an user-filter and a group-filter
#-----------------------------------------------------------------------------------------
# Execute Create-FilterSetCondition to create 2 user-related conditions and 1 group-related condition
# - userCondition1 = userAccountControl bit on equals 2
# - userCondition2 = company equals ventigrate
# - groupCondition = name equals testgroup
$userCondition1 = Get-FilterSetCondition -Connection $connection -ConditionType "user" -Attribute "userAccountControl" -Operator "Bit_on" -Value "2"
$userCondition2 = Get-FilterSetCondition -Connection $connection -ConditionType "user" -Attribute "company" -Operator "Equals" -Value "ventigrate"
$groupCondition = Get-FilterSetCondition -Connection $connection -ConditionType "group" -Attribute "name" -Operator "Equals" -Value "testgroup"

# Combine the user-related conditions in a single user FilterSet, using an OR operand
$userConditions = @($userCondition1, $userCondition2)
$userFilter = Get-FilterSet -FilterType "user" -Operator "or" -Conditions $userConditions

# "Combine" the group condition in another group FilterSet, using an AND operand (though it doesn't matter as it's a single condition ;))
$groupConditions = @($groupCondition)
$groupFilter = Get-FilterSet -FilterType "group" -Operator "and" -Conditions $groupConditions

#endregion

#region STEP 4: assign the filters to the connection
#---------------------------------------------------
# Assign the newly created filters to the connection
Set-ExclusionFilter -Connection $connection -UserFilter $userFilter -GroupFilter $groupFilter

#endregion

I think this code is pretty self-explaining (lots of comments in there).
As always: leave a comment if something isn't 100% clear ... or if you just want to post some constructive feedback :)

SharePoint 2010 - Creating user profile exclusion filters using code

Ah, user profiles in SharePoint 2010. When I initially read about them, I was really impressed by the concept: no more SSP, nicely separated service applications, etc. But now that I had the honour to experience them first-hand in some real-life scenarios … well, let’s say they sometimes want me to unleash my inner hulk.

Anyway I digress. In this post I’d like to cover a little black spot in the user-profile API. As you might know, SharePoint 2010 ships with a ****load of PowerShell commands and API functions that allow you to script your entire farm set-up. If you Google around, you’ll find plenty of blogs on how to script various pieces of your SharePoint farm.
Nevertheless there’s something I just can’t seem to find any info on: how to script the creation of user profile connection exclusion filters. There’s even no mention of them in the TechNet resources. Plenty of guides on how to set them up using the UI though, but nothing scriptable.

Setting up filters in the SP2010 UI - Step1

Setting up filters in the SP2010 UI - Step 2

Unfortunately my current employer wants to script everything regarding their farm installation, so I had to get my own hands dirty.

Fire up the disassemblers!

This time I didn’t rely on my old companion Reflector (see my earlier post for more details), but I tried out three other products that attempt to fill the void that Reflector left:

• ILSpy
• JetBrains dotPeek
• Telerik JustDecompile

I’ll probably post a small review on all three later on. Using these babies I loaded up the code-behind from the /_layouts/editconnectionfilters.aspx page, which is used to set the exclusion filters with the UI. Its code behind is the Microsoft.SharePoint.Portal.UserProfiles.AdminUI.EditConnectionFilters class in the Microsoft.SharePoint.Portal assembly.

<insert random curse />

Apparently Microsoft uses internal classes to create these exclusion filters. There goes plan A: call the undocumented public API to create the filters. Also, it doesn’t seem like these internal classes are referenced in any existing SharePoint PowerShell cmdlets, so out went plan B too: call some undocumented cmdlets.

… and in comes plan C

So with all easy options gone, it’s time to bring out the big guns: use reflection to invoke the internal classes in a similar way as it’s done in the UI’s code-behind (of course they just reference the objects without reflection).

Warning: calling internal classes with reflection is certainly not supported in any way! I’m sure Microsoft had their reasons to keep them internal. Or maybe they were just too busy trying to get everything working before the product’s shipping date & forgot to write a public API for these filters. :)
In any case: use the following code at your own risk. I’m not sure if Microsoft will void your warranty for this, but a warned man/woman …

I'm not going to delve too deep into the internal workings of SharePoint and how they handle the creation of exclusion filters ... but briefly summarized they use the following internal classes:

• FilterSetCondition (located in the Microsoft.Office.Server.UserProfiles.Synchronization assembly)
  This class represents a single “rule” in the filter, e.g. [userAccountControl bit on equals‘2’]
• FilterSet (also located in the Microsoft.Office.Server.UserProfiles.Synchronization assembly)
  This represents a set of rules (FilterSetCondition objects), combined together using either an “All apply” operator (AND) or an “Any apply” operator (OR) [just like the UI]. User-related rules are combined in one FilterSet, group-related rules are combined in another FilterSet object.
• Both FilterSet objects are then assigned to the user profile sync connection using the internal SetExclusionFilters method on the connection object.

I'm in a lazy mood today, so I suggest you browse the internal classes yourselves to get a clear view on how it all ties together. (+ I don't want to anger Microsoft too much by pasting bits of their code)

Reflection workaround

You can find the complete source of this workaround on codeplex. The code should also give you some clues on how things are handled internally in SharePoint. The codeplex download consists of two VS2010 projects:

• a class library project with the actual workaround: VNTG.UserProfileFilters
• a console application with some examples on calling the class library objects: VNTG.UserProfileFilters.Examples

I tried to properly document everything in that codeplex project, so I hope it's all self-explanatory.
UPDATE: This solution is mainly aimed at Visual Studio 2010 (.NET) projects. If you're looking for a PowerShell-friendly implementation of this workaround, check out part 2 of this post

Once again, use this code at your own risk as it's not officially supported! 

just covering my ass ;)

Just as a teaser I'll also paste one of the examples here. It should give you an idea on how easy it is to add those exclusion filters with the workaround.

Example 1

In this example we’ll add 2 user-based exclusion filters (any apply) and a single group-based exclusion filter to a user profile connection called Local AD:

• User exclusion filter: ["userAccountControl bit on equals '2'" OR "company equals 'ventigrate'"]
• Group exclusion filter: ["name equals 'test'"]

// Get a reference to the first available user profile service app
SPFarm localFarm = SPFarm.Local;
var upServiceproxy = SPFarm.Local.Services.Where(s => s.GetType().Name.Contains("UserProfileService")).FirstOrDefault();

if (upServiceproxy != null)
{
 var upServiceApp = upServiceproxy.Applications.OfType<SPIisWebServiceApplication>().FirstOrDefault();
 if (upServiceApp == null)
 {
  Console.WriteLine("No User profile svc app found");
  return;
 }

 SPServiceContext ctx = SPServiceContext.GetContext(upServiceApp.ServiceApplicationProxyGroup, SPSiteSubscriptionIdentifier.Default);
 Microsoft.Office.Server.UserProfiles.UserProfileConfigManager upConfigManager = new Microsoft.Office.Server.UserProfiles.UserProfileConfigManager(ctx);

 // Let's create some user filters for the sync connection called "Local AD"
 var connection = upConfigManager.ConnectionManager["Local AD"];                
 if (connection != null)
 {
  //----------------------------------------------
  // Create a set of user-related rules, any apply
  //----------------------------------------------
  Filters userFilter = new Filters("user");
  var fsCondition1 = userFilter.CreateFilterSetCondition(connection, "userAccountControl", Filters.FilterOperator.Bit_on, "2");
  var fsCondition2 = userFilter.CreateFilterSetCondition(connection, "company", Filters.FilterOperator.Equals, "ventigrate");

  // get the actual filterset based on an OR between all conditions
  List<object> conditions = new List<object>();
  conditions.Add(fsCondition1);
  conditions.Add(fsCondition2);
  var userFilterSet = userFilter.GenerateFilterSetForConditions(conditions, Filters.FilterSetOperator.Or);

  //------------------------------------------------------
  // Create a set of group-related rules, all apply (=AND)
  //------------------------------------------------------
  Filters groupFilter = new Filters("group");
  var groupCondition1 = groupFilter.CreateFilterSetCondition(connection, "name", Filters.FilterOperator.Equals, "test");

  conditions = new List<object>();
  conditions.Add(groupCondition1);
  var groupFilterSet = groupFilter.GenerateFilterSetForConditions(conditions, Filters.FilterSetOperator.And);

  //---------------------------------------------
  // Add the created filtersets to the connection
  //---------------------------------------------
  Filters helper = new Filters("");
  helper.SetExclusionFilters(connection, userFilterSet, groupFilterSet);
 }
}

Example code

Lets dissect this example. First part of the code is to retrieve the user profile connection itself. This is done using the public API of SharePoint 2010 (no reflection at all).

SPFarm localFarm = SPFarm.Local;
var upServiceproxy = SPFarm.Local.Services.Where(s => s.GetType().Name.Contains("UserProfileService")).FirstOrDefault();

if (upServiceproxy != null)
{
 var upServiceApp = upServiceproxy.Applications.OfType<SPIisWebServiceApplication>().FirstOrDefault();
 if (upServiceApp == null)
 {
  Console.WriteLine("No User profile svc app found");
  return;
 }

 SPServiceContext ctx = SPServiceContext.GetContext(upServiceApp.ServiceApplicationProxyGroup, SPSiteSubscriptionIdentifier.Default);
 Microsoft.Office.Server.UserProfiles.UserProfileConfigManager upConfigManager = new Microsoft.Office.Server.UserProfiles.UserProfileConfigManager(ctx);

 // Let's create some user filters for the sync connection called "Local AD"
 var connection = upConfigManager.ConnectionManager["Local AD"];                
 

Then we’ll create the necessary FilterSetCondition objects. This is done with reflection in the CreateFilterSetCondition method of the Filters helper class. You need to specify in the Filters constructor whether you want to build  exclusion rules for user-related attributes (using the "user" parameter) or whether you want to build exclusion rules for group-related attributes (using the "group" parameter).
Finally we group the different user-related and group-related rules together in their own FilterSet objects by calling the GenerateFilterSetForConditions method. In this method you also specify if you want the different rules to be combined as "All apply" (=AND) or as "Any apply" (=OR).

//----------------------------------------------
// Create a set of user-related rules, any apply
//----------------------------------------------
Filters userFilter = new Filters("user");
var fsCondition1 = userFilter.CreateFilterSetCondition(connection, "userAccountControl", Filters.FilterOperator.Bit_on, "2");
var fsCondition2 = userFilter.CreateFilterSetCondition(connection, "company", Filters.FilterOperator.Equals, "ventigrate");

// get the actual filterset based on an OR between all conditions
List<object> conditions = new List<object>();
conditions.Add(fsCondition1);
conditions.Add(fsCondition2);
var userFilterSet = userFilter.GenerateFilterSetForConditions(conditions, Filters.FilterSetOperator.Or);

//------------------------------------------------------
// Create a set of group-related rules, all apply (=AND)
//------------------------------------------------------
Filters groupFilter = new Filters("group");
var groupCondition1 = groupFilter.CreateFilterSetCondition(connection, "name", Filters.FilterOperator.Equals, "test");

conditions = new List<object>();
conditions.Add(groupCondition1);
var groupFilterSet = groupFilter.GenerateFilterSetForConditions(conditions, Filters.FilterSetOperator.And);

Finally, we’ll add both FilterSet objects to the connection that we fetched in the first step.

Filters helper = new Filters("");
helper.SetExclusionFilters(connection, userFilterSet, groupFilterSet);

Result in SharePoint UI

As you can see, the rules provided in the code are now listed on the user profile connection page in the UI. For me this worked like a charm on my dev-machines, I hope it does the same on yours.
Once again: feel free to post a comment if you do have questions on using this...

Download link to the .NET code
Once again: if you're looking for a Powershell-friendly solution -> check my update

[UPDATED: moved the inline code to a codeplex project & rewritten some bits for extra clarity]