Should You Depend on Content Deployment?

Content Deployment is a great new feature of MOSS that can be used to create a very nice staging environment. It can also help you with deployment and in maintaining sensible data in your test and development environments.

In short it has a great and promising feature set.

Unfortunately it is a very bug ridden feature.

I made it work and addressed a number of issues in an earlier post. Looking at the number of comments and additional bugs that I encountered later on it seems that this feature is simply broken. And that is not counting all the bugs that you’re likely to encounter when you go into production with this. Another scary post worth a read by Maurice Prather on the issue.

To balance things Stefan Goβner (nice guy by the way, he personally helped me) has made a valiant effort with some great advice on how you can/should use the beast.

Did any of you guys ever take this into production?

I decided not to use it for any of our portal sites as it is simply too uncertain whether or not it will work any given day and if your next code deployment breaks something new. If something goes wrong, how can you ensure that it’s even fixable? Almost anything else that goes wrong can be fixed by some code that performs some changes through the OM, but content deployment is really a big black box.

I would rather code my own – less ambitious – content deployment than rely on a barely tested module that I got no means to fix. Or more probable I would design the platform so I didn’t rely on it.

If you do decide to go with the feature you should a) ensure that you can survive it breaking down for a period (a few months) and b) ensure that you got a premier support license with Microsoft, so they should be able to help you when you hit that rock.

Rumors

Let’s start some

Apparently Redmond is very much aware of this problem and a large number of their hotfixes addresses various issues with content deployment. And they are probably aware of a whole lot more that we haven’t discovered yet – after all you should expect many if you try to go into production with it.

So, they are working on a large patch (service pack?) that addresses these issues. The bad news is that it is nowhere ready yet. My (educated) guess would be that they will announce it at the SharePoint conference in March in Seattle.

 

Lock Down Security: My Best Practices (and restoring usability)

I’m sure many of you are need to establish and implement a security matrix on your SharePoint sites. What do you need to do to ensure that your users can actually do the stuff they need to do? After all they are the business they are bringing the actual money home.

This is my first of probably many blog articles about SharePoint security. It is a rather complex and annoying issue that most of us don’t really want to think about too much. Generally most people solve it in one of three ways:

1. Grant everybody some standard permissions and forget about it
2. Grant (too) many administrative rights and trust them not to mess it up
3. Lock down security and accept the loss of usability as many features stop working

I tend to go with either 1 or 3. For my more advanced setup the standard groups (option 1) doesn’t quite cut it and I’m stuck with option 3. That being said I do take pains to ensure that as much functionality as possible continue to work – the very purpose of this article.

I usually think of (and use) SharePoint Groups as roles and permission levels as role definitions. Note that the security setup has changed substantially since WSS 2.0, so you might need to read up on it (see bottom).

Designing the Security Matrix

When you need to implement security you quickly discover that there really is no way to import/export the security definitions and you are stuck with manual configuration. Sorry. It’s on my to-do list of tools that I would like (someone) to build. I know of only one way to “copy” the security setup from one environment to the next and that is a stsadm backup/restore of the site collection. It can be useable once or twice if you plan it right.

So what you need to do is to document and plan it thoroughly before you start. Nobody likes to do the same work 5 times over which you will otherwise end up doing. And sadly you’ll need to involve your client/customer/user and get them to buy in on whatever security you end up implementing. If you are like me you’ll want them to be able to maintain it after the project is delivered and then you need them to understand what you are doing, why and how they should use it in the future.

I prefer to use excel with a simple sheet with the site hierarchy as the first column and subsequent user groups/roles on the next columns. In the cells you simply write something the places where you’ll break the security inheritance and invent some labels that describe what that particular group should be able to do at that particular site. Don’t be too creative and don’t use a 20 letter encoding of the SharePoint permission levels. Use a label and map that to a custom permission level later (client probably don’t care about that).

It almost goes without saying that you should plan for as much inheritance as possible so the spreadsheet should be fairly sparse. If it contains lists or individual list items you should consider other options as that will certainly go awry in a very short time.

Deciding on AD or SharePoint Groups

So should you use SharePoint groups or should you use AD security groups (I’ll skip the discussion of AD group scope here – domain local, universal etc.)? You’ll probably end up with both as there are benefits and tradeoffs with both:

AD Security Group	SharePoint Group
Contains both users and other AD security groups	Contains both users and other AD security groups, but not other SharePoint groups
Must be maintained in the AD, usually by IT department and/or IT administrators	Permission to maintain groups can be assigned within SharePoint
Can be used (directly) within SharePoint for permissions	Can be used for permission assignment (obviously)

And there are many more that I didn’t consider important here. Sharp readers will notice that I omitted server local groups here. Don’t ever use them. They are simply confusing, impossible to maintain and they don’t go well with a multi-server farm.

I usually prefer to have the AD deliver as much as possible, e.g. at least domain membership, probably department membership. I then assign those groups to a number of SharePoint groups that I use as roles and don’t worry too much about those. E.g. if you want to assign every user in the domain reader rights and a few (10%) some additional rights, then 90% of the users will be correctly assigned after adding “Domain Users” to your reader SharePoint group.

One word of warning: If your organization has been very creative in their domain setup and use excessively deep nesting in AD it might not work within SharePoint. It has been seen to happen. If you can assign that group properly to file share on the SharePoint server you should be quite certain that it also works in SharePoint.

The most important missing feature of the SharePoint groups is that they cannot be nested. It is a shame and the consequence is that you might need to add a given user to, say, 4 different SharePoint groups to make everything work. If it requires somebody to grant permissions on individual basis somewhere on the portal chances are that you should rethink your design.

Be sure to write down what is needed to grant a new user access to your portal that can be used as a reference for the SharePoint business owners at a later stage.

Managing Membership

One very helpful tip in managing the memberships of the SharePoint groups is that each group is assigned a owner that can adjust the memberships (by default), but it will also work if you assign a SharePoint group that role.

Go to the edit group page to modify it and set your custom SharePoint administrator group as owner of the group. From that point on your membership handling can be delegated to somebody without excessive rights e.g. farm administrators.

Implementing the Security Matrix

One thing that I’ll always recommend: Don’t use the standard SharePoint permission levels. They are well designed and therefore you should take a copy of them and assign the copy to your groups where you need it. That will enable you to later modify the permission levels in one place (as opposed to going through everywhere you assigned permission to that group) without compromising the integrity of the standard permission levels. Nothing is worse than a Contributor permission that actually grants administrative rights…

Ensure that the right people can administrate various group memberships by making sure who the owner of the SharePoint groups is.

When you implement (click through it) your matrix make extensive use of the “View Group Permission” feature (from settings menu when viewing group memberships, people.aspx). Another thing for my to-do list is to extend this feature to handle users as well – it’s not as hard as it sounds.

In the following I’ll cover some specific quirks that you might need to remember to ensure that your users can still use the site after you did your work (oh, they will resent you for it…)

Making the Galleries Work (Again)

The web part, list and site template galleries might need special permissions (if you try to limit access to the site collection root), to continue to work. To go to the permission setup, go to the actual galleries through Site settings, then gallery settings and finally the permissions.

• Web Part Template Gallery: You should assign read permissions to this gallery to everybody (their group!) that should be able to add webparts somewhere. Otherwise the Add Web Part popup will be rather empty
• List Template Gallery: Users that need to create lists from saved list templates obviously needs read access and the people creating the list templates should have contribute/write permissions.
• Site Template Gallery: Users that need to create sites from the templates require read access. Users creating the site templates need contribute permissions on the gallery as well as the Site permission “Manage Permissions” (available on the permission level settings) on the target site that they want to save

Reuseable content

On a publishing web site, i.e. publishing or collaboration site, there is a list in the site collection root named “Reuseable content”.

If you want the functionality to work on any subsites you need to assign all editors contributor right to the list and all other users read permissions. If some users lack the permissions they simply won’t see that particular item of content on your publishing page. There will be no sign that they do not see the complete page, which tend to be pretty confusing to end users.

Content Types

Sadly there is no list or permission level that governs the customization of content types. You can’t prevent this through permissions if you also require some users to be able to manage your site permissions, i.e. add users.

You will want to prevent “people” (administrators) from changing all the content types that you defined in xml through feature deployment as that will disconnect your content type (see my two posts on that, 1 and 2) from the underlying xml source.

You can mark your content type as either read only or sealed with following tradeoffs

• Read only: Will prevent accidental changes and allow inherited content types to delete/hide inherited columns. Administrators can actually override this setting if they really want to (but then they are not “in good faith”)
• Sealed: Will prevent all changes through the GUI, but will also lock child content types so they can only hide inherited columns, not delete them

Both cases will not affect your ability to update the underlying xml files, i.e. it is actually not supported to change them after initial deployment. You can do it, but then you’ll need to update all inherited content types to preserve or restore their “inheritance” (see post). (A proper fix for this is another item on my to-do list)

Master Pages, Page Layouts and Styles

Just maintain the default “Style Readers” group and you should have no problems with these lists. If you do change the style readers group too much you need to ensure that

• all users can read from the master pages catalog (“/_catalogs/”), otherwise they are completely blocked from your pages
• all users can read from “/Style Library”

Final Words

I hope that these guidelines will help you a little bit. They helped me, but don’t for a minute consider them to be complete.

There are a number of features that require some special permission here and there if you tighten security on something else. It’s bound to happen and it is tedious and hard to test thoroughly.

Additional Resources

You can find some general information from Microsoft that defines all the basics and provides some guidelines here.

As usual Joel Oleson has something worthwhile to read on the subject.

 

Fixing Uninstalled Activated Features

[Updated October 11 2008]

I uploaded a precompiled project with suitable visual studio project to google code (I’m fed up with wordpress file type restrictions by now), you can download the complete zip (includes wsp) here. Many thanks to Anthony Sumner for taking the time to create the package.

(Mine is sort of souped into a number of other stuff that I’m not at liberty of sharing freely)

[/Update]

In my former post on content deployment I had to cleanup old removed features before the content deployment would work. This post presents the tool that I build to fix those features.

The problem in a nutshell is that when a feature is activated (at some scope) and later uninstalled it still lingers in the content database. The feature doesn’t do much; it’s uninstalled, it doesn’t work, but it is still registered as being activated at that particular site, site collection, web application or farm. As far as I can see an uninstalled but activated feature only affects the affects the content deployment.

The procedure for finding where uninstalled features are activated is fairly straightforward:

1. Go through every site collection and site recursively
2. For each SPWebApp, SPSite and SPWeb
   1. Iterate through the *.Features collection
   2. For each feature, if the definition is null it is uninstalled but still activated

The way I’ve done it is to add a new webpage to the central administration site that will run through a given web application and list all the features activated at each level. It presents a “Remove” and “Remove (force)” next to each feature. Specifically all problematic features are listed at the very top of the page, so they can easily be removed (only forced removal work for these features) without looking through the entire list (it can be quite long).

I should probably stress that “remove” in this context simply means to deactivate the feature at the particular level.

It looks like this (with a couple of “bad” features):

Sorry for the beeps…

Adding the Link to Central Admin

Add a feature with the following feature.xml file (shortened a bit as I have a lot more elements in it):

<?xml version="1.0" encoding="utf-8" ?>
<Feature xmlns="http://schemas.microsoft.com/sharepoint/"
Id="47240D13-6F84-4F43-9FCA-0FF456D36B95"
Scope="Farm"
Title="Carlsberg Administration Utilities"
Description="Adds a number of pages to the Utilities section on the Operations tab."
AutoActivateInCentralAdmin="TRUE"
Hidden="TRUE">
<ElementManifests>
<ElementManifest Location="WebConfigModifications.xml" />
<!-- additional element manifest files for other features... -->
</ElementManifests>
</Feature>

And “WebAppFeatureReporter.xml” file that will add a link to the webpage under the “Web Application Configuration” section on the Administration tab:

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
<CustomAction
Id="WebAppFeaturesList"
GroupId="WebApplicationConfiguration"
Location="Microsoft.SharePoint.Administration.ApplicationManagement"
Sequence="100"
Title="Web Application Features List" >
<UrlAction Url="/_admin/WebAppFeatureReporter.aspx"/>
</CustomAction>
</Elements>

Adding the Code

Finally we just need to add the actual code. There is only one file as I added the code directly to the aspx file, name it “WebAppFeatureReporter.aspx” and make sure that it’ll be copied to the /_admin folder under the template folder when the feature is deployed (your manifest.xml of the containing solution).

I’m satisfied with the functionality and layout of this page, but not the coding style within it. It was a rather quick hack that proved to work perfectly and stayed unchanged after that.

(note: If wordpress formatting messed something up – very likely – you can download the file here. Just rename file back to .aspx)

<%@ Assembly Name=”Microsoft.SharePoint.ApplicationPages, Version=12.0.0.0,

<%@ Assembly Name="Microsoft.SharePoint.ApplicationPages, Version=12.0.0.0,
Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>

<%@ Page Language="C#" Inherits="Microsoft.SharePoint.WebControls.LayoutsPageBase"
MasterPageFile="admin.master" %>
<%@ Import namespace="System.Reflection"%>
<%@ Import namespace="Microsoft.SharePoint.Administration"%>

<%@ Import Namespace="Microsoft.SharePoint.ApplicationPages" %>

<%@ Register TagPrefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls"
Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" %>
<%@ Register TagPrefix="Utilities" Namespace="Microsoft.SharePoint.Utilities"
Assembly="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c" %>
<%@ Import Namespace="Microsoft.SharePoint" %>

<%@ Register TagPrefix="wssuc" TagName="LinksTable" src="/_controltemplates/LinksTable.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="InputFormSection"
src="/_controltemplates/InputFormSection.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="InputFormControl"
src="/_controltemplates/InputFormControl.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="LinkSection"
src="/_controltemplates/LinkSection.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="ButtonSection"
src="/_controltemplates/ButtonSection.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="ActionBar"
src="/_controltemplates/ActionBar.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="ToolBar"
src="/_controltemplates/ToolBar.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="ToolBarButton"
src="/_controltemplates/ToolBarButton.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="Welcome" src="/_controltemplates/Welcome.ascx" %>

<asp:Content ID="Content1" ContentPlaceHolderID="PlaceHolderPageTitle" runat="server">
</asp:Content>
<asp:Content ID="Content2" ContentPlaceHolderID="PlaceHolderPageTitleInTitleArea"
runat="server">
Web application feature reporter
</asp:Content>
<asp:Content ID="Content3" ContentPlaceHolderID="PlaceHolderPageImage" runat="server">
</asp:Content>
<asp:Content ID="Content4" ContentPlaceHolderID="PlaceHolderAdditionalPageHead"
runat="server">
</asp:Content>
<asp:Content ID="Content5" ContentPlaceHolderID="PlaceHolderMain" runat="server">

<table width="100%">
<tr><td align="right">
<SharePoint:WebApplicationSelector id="selector" runat="server"
OnContextChange="OnContextChange"
AllowAdministrationWebApplication="false"/>
</td></tr>
</table>

<asp:Literal ID="litMessage" runat="server" />
<asp:PlaceHolder ID="phMain" runat="server" />

</asp:Content>

<script runat="server" language="c#">

Hashtable parentObjects = new Hashtable();
string message = "";

protected override void OnLoad(EventArgs e)
{
base.OnLoad(e);
EnsureChildControls();
}

protected override void OnPreRender(EventArgs e)
{
base.OnPreRender(e);

if (selector.CurrentItem == null)
{
message = "Please select a web application to modify";
}
litMessage.Text = "<span style='text-color:red;'>" + message + "</span>";
}

protected void OnContextChange(object sender, EventArgs e)
{
//Recreate the controls
CreateControls();
}

protected override void CreateChildControls()
{
base.CreateChildControls();
CreateControls();
}

private void CreateControls()
{
//Rebuild controls
phMain.Controls.Clear();

PlaceHolder pnlWebApp = new PlaceHolder();
PlaceHolder pnlSite = new PlaceHolder();
PlaceHolder pnlWeb = new PlaceHolder();
PlaceHolder pnlInvalid = new PlaceHolder();

SPWebApplication oApp = selector.CurrentItem;

if( oApp != null )
{
SPSecurity.RunWithElevatedPrivileges(delegate { ListFeatures(pnlWebApp,
pnlSite, pnlWeb, pnlInvalid, oApp); });
}

if (pnlInvalid.Controls.Count > 0)
{
AddHtml(phMain, GetHeaderByTitle("Invalid scope or installed \"missing\""
+ " feature definitions"));
AddHtml(phMain, "<TABLE cellpadding='2' cellspacing='2' width='100%'>");
phMain.Controls.Add(pnlInvalid);
AddHtml(phMain, "</TABLE>");
}

phMain.Controls.Add(pnlWebApp);
phMain.Controls.Add(pnlSite);
phMain.Controls.Add(pnlWeb);
}

static void AddHtml(Control parent, string html)
{
LiteralControl lit = new LiteralControl();
lit.Text = html;
parent.Controls.Add(lit);
}

private void ListFeatures(
PlaceHolder pnlWebApplication,
PlaceHolder pnlSite,
PlaceHolder pnlWeb,
PlaceHolder pnlInvalid,
SPWebApplication webApp)
{

ListFeaturesAux("Web application", pnlWebApplication, pnlInvalid, webApp.Features);

foreach(SPSite site in webApp.Sites ){
ListFeaturesSite( pnlSite, pnlInvalid, site);
ListFeaturesWeb( pnlWeb, pnlInvalid, site.RootWeb );
}
}

private void ListFeaturesWeb(PlaceHolder pnlWeb, PlaceHolder pnlInvalid, SPWeb web)
{
ListFeaturesAux( "Web: " +  web.Url, pnlWeb, pnlInvalid, web.Features);

foreach( SPWeb subweb in web.Webs)
{
ListFeaturesWeb( pnlWeb, pnlInvalid, subweb);
}
}

private void ListFeaturesSite(PlaceHolder pnlSite, PlaceHolder pnlInvalid, SPSite site)
{
ListFeaturesAux( "Site collection: " +  site.Url, pnlSite, pnlInvalid, site.Features);
}

private void ListFeaturesAux(string header, PlaceHolder pnlContent,
PlaceHolder pnlInvalid, SPFeatureCollection features ){

AddHtml(pnlContent, GetHeader(header));
AddHtml(pnlContent, "<TABLE cellpadding='2' cellspacing='2' width='100%'>");

foreach (SPFeature feature in features)
{
if( feature.Definition != null )
{
//Feature Ok, add
GetRow(pnlContent, feature.Parent, feature.Definition);
}
else
{
AddHtml(pnlInvalid, "<tr><TD class='ms-vb2' style='font-weight: bold;'>");
AddHtml(pnlInvalid, "Missing feature definition (feature id:"
+ feature.DefinitionId + "), Scope: " + header );
AddHtml(pnlInvalid, "</td><td align='right'>");
AddUninstallButton(pnlInvalid, feature.Parent, feature.DefinitionId, false);
AddUninstallButton(pnlInvalid, feature.Parent, feature.DefinitionId, true);
AddHtml(pnlInvalid, "</td><tr>");
}
}

AddHtml(pnlContent, "</TABLE>");
}

protected static string GetHeader(string scope)
{
return GetHeaderByTitle("Features scoped at the level of the " + scope);
}

protected static string GetHeaderByTitle(string title)
{
StringBuilder sb = new StringBuilder();

sb.Append("<TABLE cellpadding='2' cellspacing='2' width='100%' class='ms-toolbar'>");
sb.Append("<TR><TD class='ms-toolbar' nowrap='true'>");
sb.Append("<TABLE cellpadding='1' cellspacing='0' border='0'><TR>");
sb.Append("<TD class='ms-toolbar' nowrap style='padding: 2px'>");
sb.Append( title + "</TD></TR></TABLE></TD></TR></TABLE>");

return sb.ToString();
}

protected void GetRow(PlaceHolder pnl, object parent, SPFeatureDefinition feature)
{

AddHtml( pnl, "<TR><TD class='ms-vb2' style='font-weight: bold;'>");
AddHtml( pnl, "<B>" + feature.DisplayName + "</B>  ("
+  feature.Id + ")</TD>");

AddHtml(pnl, "<TD align='right'>");
//Create uninstall buttons
AddUninstallButton(pnl, parent, feature.Id, false);
AddUninstallButton(pnl, parent, feature.Id, true);

AddHtml( pnl, "</TD></TR>");
AddHtml( pnl, "<TR><TD class='ms-vb2' colspan='2'><I>"
+ feature.GetDescription(new System.Globalization.CultureInfo("en-us"))
+ "</I></TD>");
AddHtml( pnl, "<TR><TD class='ms-vb2' colspan='2'>");

if( feature.ActivationDependencies.Count>0 ) {
AddHtml( pnl, "Activation dependencies:");
AddHtml(pnl, "<UL>");
foreach (SPFeatureDependency dep in feature.ActivationDependencies) {
SPFeatureDefinition depFet = GetFeature(dep.FeatureId);
AddHtml(pnl, "<LI>" + depFet.DisplayName + " (" + depFet.Id + ")</LI>");
}
AddHtml(pnl, "</UL>");
}
AddHtml(pnl, "</TD>");
AddHtml(pnl, "</TR>");
}

private void AddUninstallButton(PlaceHolder pnl, object parent, Guid featureId, bool force)
{
//Add the link to the parent object for this feature
string buttonId = GetFeatureContextId( parent, featureId );
parentObjects[ buttonId] = parent;

Button b = new Button();
b.ID = "remove_" + buttonId + "_" + force;
b.Text = "Remove" + (force ? " (force)" : "" );
b.CommandArgument = buttonId; //featureId.ToString();
b.CommandName = force ? "forceremove" : "remove";
b.Command += new CommandEventHandler(UninstallCommand);

pnl.Controls.Add( b );
}

/// <summary>
/// remove a given feature with or without force
/// </summary>
/// <param name="sender"></param>
/// <param name="e">CommandArgyment is the id of the feature, syntax: "parent name|feature guid"</param>
void UninstallCommand(object sender, CommandEventArgs e)
{
try
{
string operation = e.CommandName;
string globalFeatureId = e.CommandArgument.ToString();
Guid featureId = new Guid(globalFeatureId.Split('|')[1]);

//Should NEVER be null
object parent = parentObjects[globalFeatureId];

SPSecurity.RunWithElevatedPrivileges(delegate
{
//Could have used reflection to invoke member, but this is more
// readable and less error prone.
SPFeatureCollection features;
if (parent is SPWebApplication)
{
features = ((SPWebApplication) parent).Features;
}
else if (parent is SPSite)
{
features = ((SPSite) parent).Features;
}
else if (parent is SPWeb)
{
features = ((SPWeb) parent).Features;
}
else
{
throw new ApplicationException("Unsupported parent:" + parent);
}

switch (operation)
{
case "remove":
features.Remove(featureId);
message = "Removed feature:" + featureId;
break;
case "forceremove":
message = "Removed feature:" + featureId + " (forced)";
features.Remove(featureId, true);
break;
default:
throw new ApplicationException(
"Failed to remove feature, unknown operation");
}
});

//Do a refresh (ok to do at this time
//  since only one command event will be fired per request)
CreateControls();
}
catch( Exception exc )
{
message = exc.ToString();
}
}

///<summary>
/// Get a feature definition from the feature id
///</summary>
protected static SPFeatureDefinition GetFeature(Guid featureId)
{
SPFeatureDefinition retval = null;
foreach (SPFeatureDefinition feature in SPFarm.Local.FeatureDefinitions)
{
if (feature.Id == featureId)
{
retval = feature;
break;
}
}
return retval;
}

///<summary>
/// Get a feature reference that includes the parent context.
/// Result is "parent id|feature id".
///</summary>
private static string GetFeatureContextId(object parent, Guid featureId)
{
if( parent is SPSite )
{
return ((SPSite)parent).ID + "|" + featureId.ToString("N");
}
else if( parent is SPWeb )
{
return ((SPWeb)parent).ID + "|" + featureId.ToString("N");
}
else if( parent is SPWebApplication)
{
return ((SPWebApplication)parent).Id + "|" + featureId.ToString("N");
}
throw new ApplicationException("Unknown feature parent:" + parent);
}
</script>

Final Comments

I will accept no critic of the actual code in the web page as it was compiled rather quickly by cut/paste from a few other custom admin pages, to solve a specific task as quickly as possible. I do like the way it’s added to the central administration site (thanks to Joel Oleson for that), it works, it looks fairly good, the code style is crap, let’s leave it at that

I actually don’t remember where the original layout for this page came about but it was either something from Joels Oleson blog or a feature from GotDotNet or a mixture thereof.

The breadcrumb on the page is specified in another xml file, not shown here. See Joel’s blog for that subject, as it is not without problems and worth a proper discussion.

Hope it helps all of you guys to get an overview of where your features are used whether or not you use content deployment.

 

Propagate Site Content Types to List Content Types

[Aka: Make Content Type Inheritance Work for XML Based Content types]

[Note:Updated Nov 6]

If you are working with XML based content types, you’ll sooner or later fall into a trap well hidden by Microsoft. Simply put Content Type inheritance don’t work for XML based content types. SharePoint does not check what fields you’ve added/deleted/changed since you last deployed your content type feature, so you don’t have the “luxury” (it bloody well should work!) of choosing whether or not to propagate changes down to inherited content types as in the web interface or object model.

I found MS mention it briefly here.

The Role of List Content Types

I’ll take a small detour here to explain the concept of list content types. Whenever you assign a content type to a list (e.g. the pages document library for publishing webs) a new list content type is created which inherits from the site content type that you are assigning to the list and with the same name.

Actually the inheritance copies the site content type definition to the list content type in its entirety and assigns an “inherited” content type id to the new list content type.

If you later modify the site content type, through the web or object model, you have the option of propagating those changes to inherited content types, which in particular includes all the list content types on the site.

Don’t Disconnect the Content Type

Another small detour before I’ll present what to do about this mess. It should be stressed that any modification of the site content type through the web or object model, will disconnect the site content type from the underlying xml metadata file. If it happens you my reestablish the link as described here. This is very bad news. It not only means that you should be careful about what you do to your content types, but it also means that there is no simple way to propagate the changes code wise – we can’t just update the site content type (re-add fields) and have the changes propagate through the object model.

The Code to Propagate

Finally I’m ready to present the code I used to propagate changes to my content types.

The procedure is:

1. Locate the site content type in question (run it multiple times if needed)
2. Start at the root web site
   1. Go through all lists on the web
   2. If the list is associated with the site content type (actually the inherited list content type)
      1. Compare every field on the site content type with the list content type
      2. Add, remove or change the field in question on the list content type
   3. Go recursively through every subweb and continue from step a

I created the following code as a new extension to stsadm, called like this:

stsadm -o cbPropagateContentType -url <site collection url> -contenttype <contenttype name> [-verbose] [-removefields] [-updatefields]

For instance:

stsadm -o cbPropagateContentType -url http://localhost -contenttype “My Article Page” –verbose

The “removefields” switch specifies whether or not fields found in the list content type that are not in the site content type should be removed or not. Default is not. New fields in the site content type will always be added to the list content types.

Note: I’ve not tried to create an “update” field. I’ve not had a use for that yet and it will also require considerably more testing to ensure that it works correctly. You really don’t want to break your list content types in case of errors…

Updated (Nov 6): An update option has now been added you should consider that option as beta.

Note 2: If the job stops prematurely no harm is done. Restart it and it will continue from where it stopped – it will examine and skip the webs/lists that it already has processed.

Finally something tangible:

Save the following as “stsadmcommands.CBPropagateContentType.xml” and save it into “\Config” in the root of the SharePoint install folder (remember to update the assembly reference to whatever you compile the code into):

<?xml version="1.0" encoding="utf-8" ?>
<commands>
  <command name="cbpropagatecontenttype"
          class="Carlsberg.SharePoint.Administration.STSAdm.CBPropagateContentType,
          Carlsberg.SharePoint.Administration, Version=1.0.0.0, Culture=neutral,
          PublicKeyToken=55c69d084ac6678f"/>
</commands>

And finally the code you need to compile to an assembly that the xml file should specify:

Updated (Nov 6): Code has been updated a bit. Some mistakes with display name/internal name have been fixed and the update option has been added. I’m not yet satisfied with the testing of the update method so consider it to be beta.

using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Text;
using Microsoft.SharePoint;
using Microsoft.SharePoint.StsAdmin;

namespace Carlsberg.SharePoint.Administration.STSAdm
{
    ///
    /// A custom STSAdm command for propagating site content types to lists
    /// content types.
    ///
    /// The code is provided as is, I don't take any responsibilty for
    /// any errors or data loss you might encounter.
    ///
    /// Use freely with two conditions:
    /// 1. Keep my name in there
    /// 2. Report any bugs back to http://soerennielsen.wordpress.com
    ///
    /// Enjoy
    /// Søren L. Nielsen
    ///
    ///
    class CBPropagateContentType : ISPStsadmCommand
    {
        #region Input parameters
        private string providedUrl;
        private string contentTypeName;
        private bool removeFields = false;
        private bool verbose = false;
        private bool updateFields = false;

        private bool UpdateFields
        {
            get { return updateFields; }
            set { updateFields = value; }
        }

        private bool Verbose {
            get { return verbose; }
            set { verbose = value; }
        }

        private bool RemoveFields {
            get { return removeFields; }
            set { removeFields = value; }
        }

        private string ContentTypeName {
            get { return contentTypeName; }
            set { contentTypeName = value; }
        }

        private string ProvidedUrl {
            get { return providedUrl; }
            set { providedUrl = value; }
        }
        #endregion

        ///
        /// Runs the specified command. Called by STSADM.
        ///
        /// The command.
        /// The key values.
        /// The output.
        ///
        public int Run(string command, StringDictionary keyValues,
                       out string output) {
            //Parse input
            // make sure all settings are valid
            if (!GetSettings(keyValues)) {
                Console.Out.WriteLine(GetHelpMessage(string.Empty));
                output = "Required parameters not supplied or invalid.";
            }

            SPSite siteCollection = null;
            SPWeb rootWeb = null;

            try {
                // get the site collection specified
                siteCollection = new SPSite(ProvidedUrl);
                rootWeb = siteCollection.RootWeb;

                //Get the source site content type
                SPContentType sourceCT =
                             rootWeb.AvailableContentTypes[ContentTypeName];
                if (sourceCT == null) {
                    throw new ArgumentException("Unable to find "
                        + "contenttype named \"" + ContentTypeName + "\"");
                }

                // process the root website
                ProcessWeb(rootWeb, sourceCT);

                output = "Operation successfully completed.";
                Log( output, false );
                return 0;
            }
            catch (Exception ex) {
                output = "Unhandled error occured: " + ex.Message;
                Log(output, false);
                return -1;
            }
            finally {
                if (rootWeb != null) {
                    rootWeb.Dispose();
                }
                if (siteCollection != null) {
                    siteCollection.Dispose();
                }
            }
        }

        ///
        /// Go through a web, all lists and sync with the source content
        /// type.
        /// Go recursively through all sub webs.
        ///
        ///
        ///
        private void ProcessWeb(SPWeb web, SPContentType sourceCT) {
            //Do work on lists on this web
            Log("Processing web: " + web.Url);

            //Grab the lists first, to avoid messing up an enumeration
            // while looping through it.
            List lists = new List();
            foreach (SPList list in web.Lists) {
                lists.Add(list.ID);
            }

            foreach (Guid listId in lists) {
                SPList list = web.Lists[listId];

                if (list.ContentTypesEnabled) {
                    Log("Processing list: " + list.ParentWebUrl + "/"
                         + list.Title);

                    SPContentType listCT =
                                         list.ContentTypes[ContentTypeName];
                    if (listCT != null) {
                        Log("Processing content type on list:" + list);

                        if (UpdateFields) {
                          UpdateListFields(list, listCT, sourceCT);
                        }

                        //Find/add the fields to add
                        foreach (SPFieldLink sourceFieldLink in
                                               sourceCT.FieldLinks) {
                          if (!FieldExist(sourceCT, sourceFieldLink)) {
                            Log(
                              "Failed to add field "
                              + sourceFieldLink.DisplayName + " on list "
                              + list.ParentWebUrl + "/" + list.Title
                              + " field does not exist (in .Fields[]) on "
                              + "source content type", false);
                          }
                          else {
                            if (!FieldExist(listCT, sourceFieldLink)) {
                              //Perform double update, just to be safe
                              // (but slow)
                              Log("Adding field \""
                                 + sourceFieldLink.DisplayName
                                 + "\" to contenttype on "
                                 + list.ParentWebUrl + "/" + list.Title,
                                   false);
                              if (listCT.FieldLinks[sourceFieldLink.Id]
                                                                != null) {
                                listCT.FieldLinks.Delete(sourceFieldLink.Id);
                                listCT.Update();
                              }
                              listCT.FieldLinks.Add(new SPFieldLink(
                                      sourceCT.Fields[sourceFieldLink.Id]));
                              listCT.Update();
                            }
                          }
                        }

                      if (RemoveFields) {
                            //Find the fields to delete
                            //WARNING: this part of the code has not been
                            // adequately tested (though
                            // what could go wrong?   ... )

                            //Copy collection to avoid modifying enumeration
                            // as we go through it
                            List listFieldLinks =
                                                  new List();
                            foreach (SPFieldLink listFieldLink in
                                                     listCT.FieldLinks) {
                                listFieldLinks.Add(listFieldLink);
                            }

                            foreach (SPFieldLink listFieldLink in
                                                        listFieldLinks) {
                                if (!FieldExist(sourceCT, listFieldLink)) {
                                    Log("Removing field \""
                                       + listFieldLink.DisplayName
                                       + "\" from contenttype on :"
                                       + list.ParentWebUrl + "/"
                                       + list.Title, false);
                                    listCT.FieldLinks.Delete(
                                                        listFieldLink.Id);
                                    listCT.Update();
                                }
                            }
                        }
                    }
                }
            }

            //Process sub webs
            foreach (SPWeb subWeb in web.Webs) {
                ProcessWeb(subWeb, sourceCT);
                subWeb.Dispose();
            }
        }

      ///
      /// Updates the fields of the list content type (listCT) with the
      /// fields found on the source content type (courceCT).
      ///
      ///
      ///
      ///
      private void UpdateListFields(SPList list, SPContentType listCT,
                                    SPContentType sourceCT) {
        Log("Starting to update fields ", false);
        foreach (SPFieldLink sourceFieldLink in sourceCT.FieldLinks) {
          //has the field changed? If not, continue.
          if (listCT.FieldLinks[sourceFieldLink.Id]!= null
               && listCT.FieldLinks[sourceFieldLink.Id].SchemaXml
                  == sourceFieldLink.SchemaXml) {
            Log("Doing nothing to field \"" + sourceFieldLink.Name
                + "\" from contenttype on :" + list.ParentWebUrl + "/"
                + list.Title, false);
            continue;
          }
          if (!FieldExist(sourceCT, sourceFieldLink)) {
            Log(
              "Doing nothing to field: " + sourceFieldLink.DisplayName
               + " on list " + list.ParentWebUrl
               + "/" + list.Title + " field does not exist (in .Fields[])"
               + " on source content type", false);
            continue;

          }

          if (listCT.FieldLinks[sourceFieldLink.Id] != null) {

            Log("Deleting field \"" + sourceFieldLink.Name
                + "\" from contenttype on :" + list.ParentWebUrl + "/"
                + list.Title, false);

            listCT.FieldLinks.Delete(sourceFieldLink.Id);
            listCT.Update();
          }

          Log("Adding field \"" + sourceFieldLink.Name
              + "\" from contenttype on :" + list.ParentWebUrl
              + "/" + list.Title, false);

          listCT.FieldLinks.Add(new SPFieldLink(
                                     sourceCT.Fields[sourceFieldLink.Id]));
          //Set displayname, not set by previus operation
          listCT.FieldLinks[sourceFieldLink.Id].DisplayName
                      = sourceCT.FieldLinks[sourceFieldLink.Id].DisplayName;
          listCT.Update();
          Log("Done updating fields ");
        }
      }

      private static bool FieldExist(SPContentType contentType,
                                                     SPFieldLink fieldLink)
        {
            try
            {
                //will throw exception on missing fields
                return contentType.Fields[fieldLink.Id] != null;
            }
            catch(Exception)
            {
                return false;
            }
        }

        private void Log(string str, bool verboseLevel) {
            if (Verbose || !verboseLevel) {
                Console.WriteLine(str);
            }
        }

        private void Log(string str) {
            Log(str, true);
        }

        ///
        /// Parse the input settings
        ///
        ///
        ///
        private bool GetSettings(StringDictionary keyValues) {
            try {
                ProvidedUrl = keyValues["url"];
                //test the url
                new Uri(ProvidedUrl);

                ContentTypeName = keyValues["contenttype"];
                if (string.IsNullOrEmpty(ContentTypeName)) {
                    throw new ArgumentException("contenttype missing");
                }

                if (keyValues.ContainsKey("removefields")) {
                    RemoveFields = true;
                }

                if (keyValues.ContainsKey("verbose")) {
                    Verbose = true;
                }

                if (keyValues.ContainsKey("updatefields"))
                {
                    UpdateFields = true;
                }
                return true;
            }
            catch (Exception ex) {
                Console.Out.WriteLine("An error occuring in retrieving the"
                    + " parameters. \r\n(" + ex + ")\r\n");
                return false;
            }
        }

        ///
        /// Output help to console
        ///
        ///
        ///
        public string GetHelpMessage(string command) {
            StringBuilder helpMessage = new StringBuilder();

            // syntax
            helpMessage.AppendFormat("\tstsadm -o {0}{1}{1}", command,
                                                 Environment.NewLine);
            helpMessage.Append("\t-url " + Environment.NewLine);
            helpMessage.Append("\t-contenttype " + Environment.NewLine);
            helpMessage.Append("\t[-removefields]" + Environment.NewLine);
            helpMessage.Append("\t[-updatefields]" + Environment.NewLine);
            helpMessage.Append("\t[-verbose]" + Environment.NewLine);

            // description
            helpMessage.AppendFormat("{0}This action will propagate a site"
                + " content type to all list content types within the "
                + "site collection.{0}Information propagated is field "
                + "addition/removal.{0}{0}", Environment.NewLine);
            helpMessage.AppendFormat("{0}Søren Nielsen (soerennielsen."
                + "wordpress.com){0}{0}", Environment.NewLine);

            return helpMessage.ToString();
        }
    }
}

Final Comments

I’ve made zero attempts to optimize the code. It doesn’t really matter how long it takes, does it? Give it 10 minutes till a couple of hours for huge site collection (I’ve tested with about 400 sub sites).

I recommend that you use the verbose flag and pipe the output to a file, so that you can review that it did everything correctly.

The code does not handle site content types on sub sites I’ll probably add it fairly soon if I need it or time permits (does it ever?)

License

Use the above code freely, with two conditions:

1. Leave my name and link in there
2. Report bug and improvements back to me

 

Convert “virtual” content types to “physical”

What do you do, if you in a fit of madness/desperation/stupidity created the content types used throughout your site, through the web interface and you now want to do the “right” thing and place them in xml files packaged as a feature?

Well this is description on how to convert the existing “virtual” content type to that xml file, while maintaining the integrity of your existing site and content. Warning: I’m modifying the SharePoint content database directly – use at your own risk!

The basic idea:

1. Create a content type xml file and package it in a feature (don’t deploy it yet) as you would if you started in a blank environment
2. “Steal or copy” the content id for “virtual” content type from the database and use it in your xml files. In other words the existing content id that is used throughout your existing SharePoint database in the inheritance hierarchy, will remain unchanged
3. Modify the database so that SharePoint sees your content type as being feature based instead of “database based”
4. Deploy your new content type feature. You can now update that content type as if you had started it out xml based to begin with

It seems fairly straightforward doesn’t it? It actually is.

Howto

Information on creating xml based content type can be found here (and on many other sources), it’s really not that hard. Your deployment will be much easier after this.

Right about now would be a good time to do a backup of your content database

Step 1: Steal the Content Type ID

Your content type will need a very specific ID that the SharePoint created for you when you created your new content type in the first place (either through the web frontend or API). It looks like “0×0101……” and will probably be a rather long string. You need to grab this id from the content database:

1. Connect to the content database in question, probably named wss_content_XXXX (if you didn’t choose a database name the XXXX will be a guid)
2. Execute the following query to find the right content typeselect ResourceDir, ContentTypeId, Definition
   

   from dbo.ContentTypes
   

   where ResourceDir like ‘%Article Page%’

   Obviously substitute your own content type name, note that the web interface might have appended some trailing numbers to the name, so you’ll have to do a “like” selection

3. Copy the ContentTypeID and insert it into the xml file. You might also want to verifiy that the definition corresponds to your that in your xml file (or just copy it over)

Step 2: Connect the Content Type to the XML File

Now you need to go into the database and modify the ContentType table to make SharePoint see it as a feature based content type as opposed to those solely in the database.

1. Connect to the content database again (you might just have kept the window open)
2. Execute begin tran once, just to give you an undo option
3. Execute the following SQL statementUpdate dbo.ContentTypes
   

   Set Size = 16, Definition = null, IsFromFeature = true
   

   where ContentTypeId = 0×010100C5…..
   

   It should only modify one row

4. If the name “ResourceDir” has been mangled by the web interface, you might want to take the opportunity to fix that too now
5. If you are satisfied with the update execute commit tran, otherwise rollback tran, do not forget this as you are locking the table for the duration (btw: Isn’t that a neat trick?)

Caveats

I will not take any responsibility if you lose your databases, however I would like to know if you find flaws with the procedure

If you have many environments this technique only works if they have the same content type id for the same type across the farms. They will have if you did a backup/restore or content deployment from one to the other. They won’t if you created them through the web on both servers. Then you either choose which one is the master of the content or you are out of luck.

Note that if you update/change the content type xml files at a later time, the changes will only apply to the site scoped content type, not the actual list content types that the system created for every list where the type is enabled. This is very bad news, but not to worry I’ll post the fix for that in a few days (give me a bit of time).

If you modify the content type through the web interface after deployment it will once again be disconnected from the xml source, and you’ll have to complete Step 2 (only) to reconnect it.

 

Top 10 Indispensible Tools

Every blog is only complete with some sort of top 10 list present – this is my version , listing ten (and then some) of the applications I use and value the most. Some are freeware, most are not. I don’t care I’ll happily pay for any of the applications listed here.

I don’t list Word, Excel, Outlook or Visual Studio here as they come with the furniture, I can’t do my job without any of them, but what makes the difference and provide the extra 50% efficiency are these tools:

1. AutoHotkey
   This absolutely invaluable tool is a hotkey and macro management program. You can literally program it to anything you can think of, though the syntax is a bit archaic, the functionality is staggering. I constantly modify my hotkey file to suit me you can download my version here (rename file extension to “ini”. read through the file to find keys it defines).
   Example 1: Ctrl + w: Bring firefox to the front if it’s running, otherwise start it.
   Example 2: Ctrl + Alt + a: Remember current window, Check if KeePass (see below) is running, If not start it and wait the main window to show up, Switch back to the original application and send the shortcut to KeePass that will autotype the password for the current window (KeePass will look at the window title and find the correct password in the database if possible).
   Example 3: Ctrl + G: Generate and insert a Guid into the active window.The list goes on and on … Brilliant.
2. OneNote (2007)
   This is part of MS Office 2003 and 2007 in some of the licenses (please don’t ask me to explain the licensing model – I don’t think that anyone understands it). As the name implies this is a program for notetaking. It replaces the numerous little txt files, notes within outlook or however you prefer to do it. After a few test runs you are addicted to it. I think I spend at least 30% of my time here.Brief list of the features I use:Deep linking – You can create links directly to specific pages within OneNote (e.g. your main todo page) and save them as windows shortcuts. That’ll enable you to create hotkeys through windows to find the page quickly. Of course creating that hotkey through Autohotkey is a lot faster and the way I do it Outlook integration – Create tasks in OneNote and have them synchronized with outlook (status info is synced two-ways, description only one way)

   Aggregation of flags – You can use a number of flags/tags for your notes that you can aggregate in a task pane, i.e. “show me all todo tagged notes in all pages”

   Cached file mode – This program probably features the most advanced file system available in any program I’ve ever seen. It basically opens your onenote database from whatever location you point it to and creates a cached version of the file. Whenever the original file is available it’ll sync your changes with that file. You can compare it to the way outlook works in offline mode, this time there’s no exchange/mail server, just the actual file itself. I place my notebooks on a USB stick to carry between computeres and I can still use my laptop without the USB most of the time – it just works

   Calculations – Simple but clever. It can do simple calculations, pretty much like google does. Write “Sin(Pi/2)=” and it will calculate that value for you. In effect replaces the calculator (which I used all the time)

   Copy text from pictures – The screenshot capturing is nice, but one really cool feature is that it has a function to “copy text from picture”, i.e. it’ll perform OCR on any picture you paste into it. It’s pretty far from perfect, but works reasonably well for ordinary screenshots, i.e. not for a scan of a traffic sign

   Drawing/handwriting recognition – This is the feature that Microsoft primarily highlights which enables you to use a Tablet PC with OneNote to do handwritten notes that are searchable etc. We all know the success of the Tablet PC and this particular feature seems useless to the 99% of us that prefer to use a reliable keyboard instead. Whether it actually works or not I can’t say.

   Search – obviously…

   To finish off I’ll recommend reading David Rasmussens blog about OneNote he has some valuable technical insights.

3. Google Desktop
   Can’t find a thing on my computer without this one being enabled, I primarily use it to search my emails as the amount quickly goes beyond normal search/categorizing within Outlook.Simple, fast, efficient and elegant, hit Ctrl twice, enter “from:torben SLA SharePoint” and the mail Torben send with that content is found. Google Desktop is a simpler product than MS desktop search offering, it has fewer advanced options, but I find it to be quicker to search and especially a lot faster to actually use.It comes with a number of extra crap, like Google Toolbar, Google Sidebar/gadgets/widgets/whatever they are called. Don’t care, don’t steal my screen real estate, just disable it
4. FireFox
   No one can do without FireFox. It’s simply a faster, better browser that you can tweak to your liking. I use the following extensions all the time:FireBug – Absolutely invaluable tool for every developer that are forced to look into html/css/javascript once in a while. Was hell before this extention

   Del.icio.us – It handles my bookmarks and syncs them between all my various machines. I don’t care much about the much touted social bookmarking aspect, I don’t really find it all that useful

   All in One Mouse Gestures – Web browsing is usually best done with a mouse, this alleviates you from having to touch the keyboard too often during browsing. Simple saves ½ a second when you need to navigate, open links in new windows etc. I just use the 4-6 simplest gestures, there are lots defined, most are impossible to remember or actually perform

   Selenium studio – I haven’t actually used Selenium studio for anything yet, but it seems an invaluable tool for doing web based unit testing, that is, recording them for later execution in a proper environment. I think I’ll try to use this in my next project. Everyone loves unit testing I just haven’t seen that many good tools for doing browser based unittesting, though it’s by far the most interesting to do

5. KeePass
   Just a small simple password remember program. I gave up on remembering all my passwords six months ago, there was simply too many. I’ve used KeePass since and it’s now contains around 100 passwords for various systems (many are web sites). Now I can use all that extra brainpower for something useful I’ll argue that a program like KeePass actually increases the security as it enables you to use/generate distinct random password for every system instead of reusing a couple of passwords for all of them.One useful feature is the autotype feature where it’ll type the appropriate password into your current window with a hotkey (it figures out what password you need based on the title of the current window). Of course I extended this functionality a bit with AutoHotkey, so my shortcut actually starts the program if it’s not running, wait for it to be ready and then executes autotype.The one thing I miss is actually a FireFox extension to integrate the two – replace FireFox’ insecure password remember with KeyPass automatically (sorry, I don’t have the time to build it).
   Note: I’ve only used version 1.xx not 2.xx – it’s a complete redesign where the new version is based on .Net.
6. Ceedo
   This is an application that will revolutionize your usage of USB drives. Nothing less. It enables you to install and execute applications on the USB disk on all host machines. So you can have your favorite text editor installed on the USB stick and be able to use it on any computer, with no installation on the host machine.The unique thing with ceedo (over competitors like MojoPac and U3) is that it enables you to install practically anything (with exceptions of course) also applications that were never designed for portability. It creates a “virtual windows environment” that the applications execute within, so when the applications writes to the registry that is stored in Ceedo’s registry not the host machine and when it reads from it Ceedo will overlay the host registry with its own. Works great for most applications, specific exceptions are Java, .Net., Visual Studio and MS Office – I suppose they change too much in windows for Ceedo to capture it all.This tool is indispensible when you routinely work with security restricted windows computers – now I don’t need to install programs all my programs on the host machines to be able to use my favorites.
   Note: On my installation AutoHotkey is of course added to Ceedo’s startup folder (yup it emulates normal windows behavior). Incidentally if another version of AutoHotkey is already running, the new one takes over (both run, the latest temporarily overwrites the first hotkey definition). That’s exactly what you want so the version on the USB stick can add hotkeys for the programs installed on that disk.
7. VMWare Workstation
   Well you can’t really develop any serious SharePoint solutions without using virtual machines, often lots of them. I routinely use VMWare Workstation/Player/Server where the last two are freeware. Workstation is brilliant, fast, reliable and a snapshot tree feature to kill for (Player and Server does not really support snapshots properly – but they are free(!) ).I believe VPC is also a usable tool, though I feel that VMWare are still a couple of years ahead in terms of features and support for many operating systems. You can find many blogs with performance comparison between the two and the one clear conclusion is that the difference is negligible; you should base your choice on the features needed and pricing.
8. Daemon tools
   Mount ISO files (many types) as virtual CD/DVDs. It works.I still don’t know why this is not a standard windows feature, it really should be. The funny thing is that MS distributes many of their products as ISO files, but they have no proper support for it (there is actually one MS program for it, lousy by the hearsay).
   Note: Be sure to deselect the Daemon tools search bar during installation. You don’t want another commercial internet explorer bar, do you?
9. ReSharper
   An add-in to Visual Studio that will make everything better, easier and faster. I’m never going back. When Visual Studio 2005 was released a number of the refactorings supported by ReSharper were also supported by the base product. ReSharper still does it better though, so the refactoring capabilities are still a selling point along with countless other improvements.Very soon you’ll code both much faster and less error-prone (it catches a number of possible bugs even without compilation). The simplest feature I use the most is for to automatically add using statements when you start using a new class within your code, just type “Hashtable”, hit Ctrl+Alt, and continue coding knowing that the using statement has now been added (color coding will help you here).
   Note: I have only tested version 1.x and 2.x, not the “new” version 3. Yet.
10. Reflector for .Net
    An excellent tool for disassembling .Net code. Let’s face it sometimes we all learn better with some inspiration from others’ code. To me it’s an indispensible source of information for SharePoint 2007 in figuring out what some of the errors/exception actually mean and how to counter them. Most of SharePoint 2007 can be disassembled, which is huge improvement over the locked down SharePoint 2003. Bits and pieces are still “obfuscated” and cannot be disassembled – obviously some of the best parts that we need the most.

Runner ups

While trying to keep the list short I also considered two more:

1. FastStone capture
   If you ever need to do screen capture this program is your friend. It supports capturing a window with scrolling and some simple drawing on the captured image. An ellipses around the button to press is always nice… OneNote also has a nice screen capture utility, it’s only lacking the easy editing capabilities
2. Paint.Net
   Delete and bury windows Paint once and for all. Finally. This is a high quality freeware image handling program that falls into the category of people that needs some basic image handling (transparent background for instance) for which Paint doesn’t cut it and PhotoShop is complete overkill (not to mention the learning curve and pricing).
   And yes: There is a file plugin that will read and write photoshop files (not working for huge files though – 100mb and above is not a good idea)

Final words

This is my list of must have applications that I use all the time. Your mileage may vary. In two months time some of them will have changed again – I always look for ways to improve my setup (and I also change the primary tasks in my job once in a while).

To that end, feel free to share your favorites in the comments.

 

Find the needle in the SQL database

[AKA: SQL Grep]

[Updated Nov 11 2008]

[Updated March 10 2009]

A few years ago I was debugging an annoying SharePoint 2003 issue with some wrong usernames after renaming them. It turned out that I had to dig deep into the SQL database tables to figure out how everything worked (nobody cares about the actual result anymore).

To do that I had to write a piece of SQL scripting that could search the database like: “search for a string in any column in any record in any table in the current database”. The script was actually quite useful (and hard to write) so I’ve kept it for a few years.

The other day I had another strange problem with the new SharePoint server and had to search the database for a specific guid. I dusted off my old script and it turned out to be very useful again. I updated it slightly and figure that it might be useful for you too.

So here it is.

There are really two parameters that you need to provide. “@searchText” is the search string and “@datatype” is the datatype to search (i.e. what types of columns that should be searched).

“@datatypes” values (select system_type_id, name from sys.types order by name):

Datatype value	SQL DataType
null	All types of strings*
-1	All types of numbers*
127	Bigint
173	Binary
104	Bit
175	Char
61	Datetime
106	Decimal
62	Float
34	Image
56	Int
60	Money
239	Nchar
99	Ntext
108	Numeric
231	Nvarchar
59	Real
58	smalldatetime
52	Smallint
122	Smallmoney
98	sql_variant
231	Sysname
35	Text
189	Timestamp
48	Tinyint
36	uniqueidentifier
165	Varbinary
167	Varchar
241	Xml

* I added special support for the first two options

Remarks:

• The script is not actually a function or a stored procedure because I usually use it by copying the SQL into a SQL query window and execute. I don’t want to create a new function/stored procedure in the databases that I’m peeking into.
• You can only search datatypes that are convertible to a “nvarchar” column (and only the first 4096 characters are searched), i.e. pretty much everything barring “image” and “timestamp” (see table on http://msdn2.microsoft.com/en-us/library/ms187928.aspx)
• Every value that is searched is converted to a “nvarchar” column by using default options, i.e. dates and thousand separators will follow your database settings
• The reason I convert everything is that if you search for a number you probably want to search all number like columns, not just “short” or “long” etc., likewise you’ll want to search every type of text column, not just the one of “char”, “varchar”, “nvarchar”, “text” etc.
• Is it slow? Absolutely. I wanted to find the string I was searching for, not a query to be part of any production systems
• Add “%” to search for wildcards (i.e. before and after your string)
• No it does not search every database only the current one. It’s easy/possible to add a search through all databases, but I didn’t really need it
• Output is a list of found values with table name, column name and the actual value found (might be interesting if you used wildcards)

Finally! The actual script (copy/paste + modify params):

declare @searchText varchar(255)
set @searchText = '%mystring%'

declare @datatype int
set @datatype = null

/*Set type of coloumn to search

null All types of strings*
-1 All types of numbers*
127 Bigint
173 binary
104 Bit
175 Char
61 datetime
106 decimal
62 Float
34 image
56 Int
60 money
239 nchar
99 ntext
108 numeric
231 nvarchar
59 Real
58 smalldatetime
52 smallint
122 smallmoney
98 sql_variant
231 sysname
35 text
189 timestamp
48 tinyint
36 uniqueidentifier
165 varbinary
167 varchar
241 xml
*/

declare @innerSQL varchar(4000)

DECLARE tables CURSOR READ_ONLY
FOR (
	select so.name as TableName, sc.name as ColName, ss.name as sch
	from syscolumns sc left join sysobjects so
	on sc.id = so.id
	left join sys.all_objects ao
	on so.id = ao.object_id
	left join sys.schemas ss
	on ao.schema_id = ss.schema_id
	where OBJECTPROPERTY(so.id, N'IsUserTable') = 1
		and (sc.xtype = @datatype
		or (@datatype is null and collation is not null)
		or (@datatype = -1 and sc.xprec>0))) -- to include text only

create table ##found ( tablename varchar(255), colname varchar(255), val nvarchar(4000) )

DECLARE @tablename varchar(255)
declare @colname varchar(255)
declare @sch varchar(255)

OPEN tables

FETCH NEXT FROM tables INTO @tablename, @colname, @sch

WHILE (not @@fetch_status = -1)
BEGIN
	IF (not @@fetch_status = -2)
	BEGIN
		--print 'table: ' + @sch + '.' + @tablename + ', column: '+ @colname
		set @innerSql = 'insert into ##found (tablename, colname, val) select '''
		+ @sch + '.' + @tablename + ''', ''' + @colname + ''', Convert(nvarchar(4000),['
		+ @colname + ']) from ' + @sch + '.[' +@tablename + '] where Convert(nvarchar(4000),['
		+ @colname + ']) like ''' + @searchText + ''' '

		exec( @innerSql )

	END
	FETCH NEXT FROM tables INTO @tablename, @colname, @sch
END

CLOSE tables

DEALLOCATE tables

--Output result
select *
from ##found
drop table ##found

I hope some of you will find it useful, if not at least I have a place to find it the next time I need it

Enjoy.

Download

[Added March 10 2009]

For your convenience I’ve added download links

Sql Grep (2005 version)

Sql Grep (2000 version) – Thanks Sootie!

 

Tool for Deployment of SSP search settings

I recently had the dubious honor to transfer search settings from one SSP to another. Going through every managed property, content source, search scope etc. just wasn’t something I looked forward to. On top of that – in the near future I will have to do it again when we deploy another SharePoint site to production.

Searching the net I found a tool created by Sahil Malik that could create the managed properties for me (link), provided that you manually merged some xml dumps of crawled and managed properties. Thanks Sahil for that great start – I needed something more therefore this post.

I modified Sahils code to suit my additional needs. It took me two full days to complete and test the code and in the end I guess that about 30% of the code base is Sahils original code.

I now have a tool that can import/export content sources, crawled properties, managed properties and (shared) search scopes – and it works!

I designed the import procedures so that they create, or synchronize, the destination SSP search settings with the xml files given, but do not delete anything not in those files, i.e. it will synchronize/create all the managed properties in the input xml file but not tough the existing ones not mentioned in the input file.

Ok, here are the details for the various operation types. The order listed here is the order that they should be imported in a complete SSP import.

Content Sources

Type, name, schedules, start addresses etc. are all handled. As far as I know that is everything, I’ve not been able to test the Exchange and BDC content sources, but they should work.

If you are transferring settings between two servers you probably want to correct the search start addresses as they are likely wrong. I’ve not tried to do anything fancy with automatic recognition of the local farm address and the like as the risk of error is too great, I wanted to keep the focus on the SSP settings not the various sites and their access mappings etc. Sorry for that you can’t have everything.

There is an option to start – and wait – for a full crawl after the import (“-FullCrawl”). This will allow the indices to be built and crawled properties will automatically be added for the crawled content. This is the “normal” way to create crawled properties.

Currently the program will wait a maximum of two hours for the crawl to complete, it will probably be configurable in the future (if I need it).

Crawled Properties

It is possible to import as well as export these. I should stress that the import operation should be considered experimental.

Why would you want to import crawled properties? They are usually created by the crawler and are available for use in managed properties immediately afterwards. However if the content in question have not yet been created (e.g. you are deploying a site to a new farm) or if you don’t want to wait for a full crawl before you create the managed properties, you might want to import them.

I’m not really using this feature myself so I don’t consider my testing to be conclusive enough.

Managed Properties

The code to import and export managed properties is originally from Sahil Malik, though considerable redesigned and bug fixed. It is now possible to dump all managed properties from one site and import them to another – there is no need to extract the standard system managed properties from your own custom (you are welcome if you want to), all can be imported with no changes.

The import will fail if one of the managed properties maps to an unknown crawled property, then you might need to either schedule a full crawl to create the crawl properties or import them too.

The “remove excess mappings” option (“-RemoveExcessMappings”)can be used to delete mappings from existing managed properties to crawled when those properties exists in the input xml file with other mappings, i.e. using this option will ensure that the SSP managed properties are exactly the same as those in the xml file after the import.

Search Scopes

The shared search scopes (those defined in the SSP) are fully supported – settings and rules are all transferred. The import will prune the scope rules to match the import xml file.

The import will fail for scopes that use property rules if the managed properties used has not been defined or marked for use in scopes (the “allow this property to be used in scopes” switch. Import of the managed property includes this setting).

The option “-StartCompilation” starts a scope compilation after the import but not wait for completion (not much point in waiting for that).

The one thing is missing from the scope import is scope display groups. They are of used on sites to populate the search scope dropdown (and some of my own search webparts as well) and are quite important for the end user search experience. You will have to set those yourself as I limited the scope (sorry for the pun) of the program to the setting stored in the SSP. Should be fairly easy for a site collection administrator to enter them however. In a similar vein any site specific search scopes are not handled. I don’t use that feature at all so there’s no support. Perhaps a topic for future improvement.

How to use

Usage: sspc.exe -o <operation> <target type> <parameters>

Operation = Export|Import

Target type = ContentSources|CrawledProperties|ManagedProperties|SearchScopes

Parameters = -url <ssp url> -file <input/output file name> [-FullCrawl|-RemoveExcessMappings|-StartCompilation]

Note all arguments are case insensitive.

This post is quite long enough as is so if you want to see the exact xml format needed download the code and run the export.

Sample Export

SSPC.exe -o export ContentSources -url http://moss:7000/ssp/admin -file output_contentsources.xml

SSPC.exe -o export CrawlProperties -url http://moss:7000/ssp/admin -file output_crawlproperties.xml

SSPC.exe -o export ManagedProperties -url http://moss:7000/ssp/admin -file output_managedproperties.xml

SSPC.exe -o export SearchScopes -url http://moss:7000/ssp/admin -file output_searchscopes.xml

I created a batch file for a full export (excluding crawled properties):

“Export SSP settings.bat” http://moss:7000/ssp/admin

which will create the output files “output_contentsources.xml”, “output_managedproperties.xml” and “output_searchscopes.xml”.

Sample Import

SSPC.exe -o import ContentSources -fullcrawl -url http://moss:7002/ssp/admin -file input_contentsources.xml

SSPC.exe -o import CrawlProperties -url http://moss:7002/ssp/admin -file input_crawlproperties.xml

SSPC.exe -o import ManagedProperties -removeexcessmappings -url http://moss:7002/ssp/admin -file input_managedproperties.xml

SSPC.exe -o import SearchScopes -startcompilation -url http://moss:7002/ssp/admin -file input_searchscopes.xml

The corresponding batch import file:

“Import SSP settings.bat” http://moss:7002/ssp/admin

which assumes the presence of input files “output_contentsources.xml”, “output_managedproperties.xml” and “output_searchscopes.xml” generated above.

Code Design Notes

Sahil Malik named the program SSPC (supposedly short for “Shared Services Provider Property Creation”) and the corresponding project name on the codeplex site is SSSPPC (“Sharepoint Shared Services Search Provider Property Creation”). It’s a mess and now that I’ve expanded the scope of the program considerably the name is even more misleading now.

Just to avoid further confusion I’ve refrained from renaming the program.

Sahil Malik spent some time doing a proper code design for the initial version. I personally think that he did go a bit over the top (sorry Sahil), but I’ve nevertheless retained most of the basic design.

He split up the code in a number of layers (we all love that) where each layer is a different class-library project. I kept that design and therefore the download will contain a number of dll files as well as the actual exe file. Just keep them all in the same directory and all should be well.

Some comments:

• I did not change the naming of the existing projects (i.e. they are all named “Winsmarts.*” though I did change a lot of the code) but the ones I added are named “Carlsberg.*”
• I redesigned/recoded the managed property import section as I simply hate duplicated code and deleted the duplicated BO classes that were present in the old “MAMC project” (now moved to “Winsmarts.SSPC.ManagedProperies”).
• The import code is now always present in the same project that performs the export.
• The managed property import/export is now complete in the sense that it can now export and import everything including the system properties. No need to sort through it all and find the ones you are responsible for (though it might still be a good idea to sift through and ensure that old test data are removed)
• I renamed a number of the classes as some of the BO objects were named as their SharePoint counterparts and the code was quite a bit harder to read than it needed to be.
• Version number of all (sub) projects has been changed to 1.1.0.0.
• Error handling is still pretty basic so you’ll get an exception with a stack trace in the console if anything is amiss

[Updated]

My code changes has now been merged into the main code base at the codeplex site. These changes breaks everything in the original code, so you will need to update xml and script files…

Future Improvements

This is the list of future improvements I’ve noted that might be added if I find the time and need for it.

• [Updated: Done] The code could be cleaned up somewhat (there shouldn’t be any todo’s in released code)
• Perhaps site scopes should be added
• Scope display groups might be added (requires some connection from SSP to the sites)
• It might make sense to add these commands to the list of operations supported by stsadm, which is fairly easy to do (see Andrew Connells excellent post for a sample)
• [Updated: Done] I’m not too fond of the serialization classes – basically the same piece of code is copied four times with minimal changes. I always consider duplicated code as a bug

Downloads

[Updated]

The code has now been merged with the existing code base at codeplex, so head over there for the latest download.

Codeplex/SSSPPC

References

Sahil Maliks original post

The current Codeplex site

A couple of useful MS articles: Creating Content Sources to Crawl Business Data in SharePoint Server 2007 Enterprise Search and Creating and Exposing Search Scopes in SharePoint Server 2007 Enterprise Search

 

The Long Path to Content Deployment

Recently I’ve had the pleasure to use the content deployment feature of MOSS between some of our SharePoint environments.

It was not a smooth ride and I believe that I have now met every single obstacle and bug in that feature. This posting is the result of several months’ worth of frustration.

We got three environments Dev, Test and Prod and we want to use content deployment to move content from the production servers to the development and test servers, simply to ensure current test data.

As a side node, you might want to read the Microsoft brief “Team-based Development in Microsoft Office SharePoint Server 2007” for another good reason to make content deployment one of your skills.

Well, without beating around the bush anymore the following are the steps needed to do a content deployment (with some minor obvious exceptions) in order of appearance.

Setup path

Content deployment requires you to first setup a path and then one or more jobs for that path. Simply put the path is the server to server connection configuration, the job is a specification of what to do and when.

I try to use security best practices so all my both service users are just standard domain users, any special rights they might need are assigned by MOSS during the configuration wizard. In the same vein my farm administrator accounts are standard domain users that are assigned extra permissions only through SharePoint.

 

The first problem with creating a path is an access denied problem during the path setup page (/_admin/DeploymentPath.aspx), when you select the source web application and site collection. For the destination site collection you are required to provide an explicit user with sufficient rights.

Figure 1: The “Access denied” message

It seems that your logged in user needs to be able to read the IIS metabase when you select a web application. There are many ways to grant access, I choose to add my farm administrator to the IIS_WPG local security group on the server. In my opinion the SharePoint team forgot to impersonate the call to read the IIS metadata. Will hopefully be fixed in some future service pack.

The second problem (on the very same page) occurs if you connect to another farm using SSL (which you should!) – You get the exception “Requested registry access not allowed” when you submit the page.

After some tracing I’ve learned that the problem is that the server tries to store the SSL key for your destination server in the registry hive for the system user, which is the correct one, but apparently the SharePoint configuration wizard tightened the security on the keys in question.

To get around this: Grant your user (farm admin) membership to the local WSS_RESTRICTED_WPG security group and grant that group “full control” to HKEY_USERS\.DEFAULT\Software\Microsoft\SystemCertificates. You could also opt for granting your user direct access to the key.

To sum it up:

1. Your logged in user should be a farm administrator
2. Should be a member of either the local IIS_WPG or WSS_WPG groups
3. Grant access to HKEY_USERS\.DEFAULT\Software\Microsoft\SystemCertificates
   1. Add user to WSS_RESTRICTED_WPG group
   2. Grant that group “full control” access to the key

That’s it! You should now be able to setup a path.

Setup Job

Next setup the job to your liking, choose whatever options you desire.

Run the job. If it succeeds then stop reading now and save a few minutes of your time. No? Carry on then…

Hotfix deployment

At this point I got the following exception:

User cannot be found. at Microsoft.SharePoint.SPUserCollection.GetByID(Int32 id) at Microsoft.SharePoint.SPWeb.get_Author() at Microsoft.SharePoint.Deployment.WebSerializer.GetDataFromObjectModel(Object obj, SerializationInfo info, StreamingContext context) at Microsoft.SharePoint.Deployment.DeploymentSerializationSurrogate.GetObjectData(Object obj, SerializationInfo info, StreamingContext context) at Microsoft.SharePoint.Deployment.XmlFormatter.SerializeObject(Object obj, ISerializationSurrogate surrogate, String elementName, Boolean bNeedEnvelope) at Microsoft.SharePoint.Deployment.XmlFormatter.Serialize(Stream serializationStream, Object topLevelObject) at Microsoft.SharePoint.Deployment.ObjectSerializer.Serialize(DeploymentObject deployObject, Stream serializationStream) at Microsoft.SharePoint.Deployment.SPExport.SerializeObjects() at Microsoft.SharePoint.Deployment.SPExport.Run()

Looking like this:

Figure 2: Deployment error – no creator/owner of site

The exception occurs fairly quickly during the preparation phase. It obviously indicates that a creator of a (sub) site it not to be found in the SharePoint user database.

In my case it happened because the farm was originally deployed using a site collection backup/restore from another AD domain, the creators of various sites would then be users in the original SharePoint farm which would be unknown in the new (which is now my source). I suppose you might see this error if you deleted some users as well.

There is nothing you can do about this error; Microsoft however, has a hotfix for this (which also solves a few other bugs). Hotfix number is 313183 and the knowledge base article you are trying to address is kb936867. At the moment of writing this is private hotfix that can only be obtained by contacting MS support. Sucks but there is a way at least.

The hotfix solves a total of 11 bugs including one more in relation to Content Deployment entitled “Violation of PRIMARY KEY” – it seems I avoided one snag after all.

Feature problems

Next error in line occurred right after the last during the export phase. I received the very informative exception:

Failed to compare two elements in the array.
at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)
at System.Collections.Generic.ArraySortHelper`1.Sort[TValue](T[] keys, TValue[] values, Int32 index, Int32 length, IComparer`1 comparer)
at System.Collections.Generic.ArraySortHelper`1.Sort(T[] items, Int32 index, Int32 length, IComparer`1 comparer)
at System.Array.Sort[T](T[] array, Int32 index, Int32 length, IComparer`1 comparer)
at System.Collections.Generic.List`1.Sort(Int32 index, Int32 count, IComparer`1 comparer)
at System.Collections.Generic.List`1.Sort(IComparer`1 comparer)
at Microsoft.SharePoint.Deployment.WebSerializer.GetDataFromObjectModel(Object obj, SerializationInfo info, StreamingContext context)
at Microsoft.SharePoint.Deployment.DeploymentSerializationSurrogate.GetObjectData(Object obj, SerializationInfo info, StreamingContext context)
at Microsoft.SharePoint.Deployment.XmlFormatter.SerializeObject(Object obj, ISerializationSurrogate surrogate, String elementName, Boolean bNeedEnvelope)
at Microsoft.SharePoint.Deployment.XmlFormatter.Serialize(Stream serializationStream, Object topLevelObject)
at Microsoft.SharePoint.Deployment.ObjectSerializer.Serialize(DeploymentObject deployObject, Stream serializationStream)
at Microsoft.SharePoint.Deployment.SPExport.SerializeObjects()
at Microsoft.SharePoint.Deployment.SPExport.Run()
*** Inner exception:
Object reference not set to an instance of an object.
at Microsoft.SharePoint.SPFeature.EnsureProperties()
at Microsoft.SharePoint.SPFeature.get_TimeActivated()
at Microsoft.SharePoint.Deployment.WebSerializer.ExportFeatureComparer.System.Collections.Generic.IComparer .Compare(ExportObject exportObject1, ExportObject exportObject2)
at System.Collections.Generic.ArraySortHelper`1.QuickSort[TValue](T[] keys, TValue[] values, Int32 left, Int32 right, IComparer`1 comparer)

Going to the source server I tried to pinpoint the error by running “stsadm –o export …” on the base site collection url (same error) and then the first level of sub sites (all exported fine).

That command is exactly the same as a content deployment just without the transfer and import part on the destination end (you are responsible for transfer and then use the import command).

The exception basically means that some of the features activated at the site collection level no longer exist on disk. Their feature definition files have probably been deleted. This can easily occur if you delete some features from your solution packs without deactivating/reactivating all features on deployment (and who cares to do that?).

If you know exactly what features are the problem (might be more than you know) I suppose you might be able to reinstall, deactivate and then uninstall to fix the problem. You might also be able to create dummy features with the correct ids and then try to install, deactivate and uninstall.

I did neither; code had to be written

What you need to do is:

1. Go recursively through your web application / site collection / root web / web
2. Each of these “parent” objects have a Feature collection
3. Each of these features in the feature collection should be examined
   1. If there is no feature.Definition (== null) then this is one of the faulty features.Simple remove it by executing parent.Features.Remove( id, true ). The force parameter is needed since the feature is not properly installed anymore, so you just remove it without any knowledge of deactivation event handlers etc.

I wrote an aspx page for this that is installed as a feature in the Central Administration site:

Figure 3: My page to list and possibly remove features from web app / site / web

Pretty cool

In due time, I will clean up the code and publish another article about this administration feature along with a few others that I’ve developed and since found indispensable.

Server Name Problem

Finally the export phase can be completed. Next problem is the transfer phase, which simply moves the exported files from the server assigned the task of performing the content deployment job to the destination server. In the path specification the location of the Central Administration site on the destination server is specified along with the destination web application and site collection.

The source server tries to export the file directly to the destination server handling content deployment jobs, which might or might not be the same server that’s running the Central Administration site. It does so by resolving the FQDN (Fully Qualified Domain Name) of the server, which might very likely be a problem to you.

If you deploy content between servers in separate network segments this won’t work out of the box, e.g. the source server can probably not find your destination server by the name “my_dest_server.my_domain”, which is only known within the immediate local AD domain of that server.

There’s no reason to think too deeply about these names – just try to do a content deployment and if it fails during the transport phase it’ll report “The remote upload Web request failed” along with the name it’s trying to resolve. A similar event log entry is also created:

Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: Timer
Event ID: 6398
Date: 6/12/2007
Time: 9:48:07 PM
User: N/A
Computer:
Description:

The Execute method of job definition Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJobDefinition (ID 2f94ff2b-2aa1-498b-96ba-649c2e75ada7) threw an exception. More information is included below.

The remote name could not be resolved: ‘servername.dev.local’

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The fix is simple:

1. Open your host file, usually located at c:\windows\system32\drivers\etc\hosts
2. Add a new line (don’t delete existing lines): “qqq.xxx.yyy.zzz servername.dev.local“. Be sure to use the correct ip address and name for your destination server. Remember that it should be an ip address that works from the source server (in some cases internal other external ip). You can probably get away with specifying the same ip address you used for the Central Administration site in that path definition – if you used a dns name then just ping the dns and grab the ip address.
3. Save the file
4. Retry your content deployment. No need to restart any services the fix is immediate.

Feature problems – part II

We’re now in the import phase and almost through. During the import SharePoint will check whether all the activated features at the source site collection (and sub sites) are also available at the destination site collection.

If not you’ll get an informative error, similar to: “Content deployment job ‘Remote import job for job with sourceID = 71bb6ada-762c-4e78-8bc3-2a105bbe5988′ failed.The exception thrown was ‘Microsoft.SharePoint.SPException’ : ‘Could not find Feature xxxxxx.’

The solution is obvious – install the same solutions/features at the destination web application as on the source.

Specified Name is already in use

If you get an error similar to “Content deployment job ‘Remote import job for job with sourceID = 71bb6ada-762c-4e78-8bc3-2a105bbe5988′ failed.The exception thrown was ‘Microsoft.SharePoint.SPException’ : ‘The specified name is already in use. A list, survey, discussion board, or document library cannot have the same name as another list, survey, discussion board, or document library in this Web site. Use your browser’s Back button, and type a new name.’“

You forgot to read the manual (ok, blogs) that specifies that the destination site collection should be a brand new blank site collection – in my case I tried to export to a newly created publishing site collection.

The Small Print

Finally your content deployment ought to be complete

If your site looks a bit strange it’s probably because the master page settings wasn’t copied, so you’ll have to assign the correct master page through the site settings. Might also be the case for the welcome page, though I haven’t confirmed it.

During the export every file of your site will be copied to the deployment package so if any files have gone missing you’ll get appropriate warnings during the export, but it’ll still complete.

This is actually a great way to detect if any aspx files have been deleted by accident, e.g. an AllItems.aspx page for a custom list might have gone missing if somebody changed the list definition (probably deployed within a feature).

Phew! At least it works for me now…

(Updated) Other bugs

This is a small list of other bugs I’ve heard/read about

1. (Thanks Harry!) The Cab-files are not always deleted and remain in the folder C:\WINDOWS\Temp\ContentDeployment. Regardless of the content deployment setting of the number of jobs to keep, they will not be deleted. Roll your own workaround and delete the files. You could schedule a job to delete everything older than 1 day (as I’m sure your job can complete in less time than that)

And of course: Move the directory location to a non-system drive

2. (Thanks Harry!) Specific sites within the site collection: I select only the language-variations but not the root. After Test Job or Run Now the root has been always added to the Scope (Fix 937208 apparently solves the problem)

3. Some characters are mangled after deployment, specifically “nbsp;” in html fields (probably a lot of others too). Update: This has been fixed with hotfix 938536 (private hotfix for now – sorry)

4. Missing disk space on destination server will be reported (on the source server) as
Failed to read package file. at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request) at Microsoft.SharePoint.Deployment.SPImport.Run() *** Inner exception: Failure writing to target file at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation) at Microsoft.SharePoint.Deployment.ImportDataFileManager.<>c__DisplayClass2.<Uncompress>b__0() at Microsoft.SharePoint.SPSecurity.CodeToRunElevatedWrapper(Object state) at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.<RunWithElevatedPrivileges>b__2() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)

5. If you choose to copy “all” security information between servers not in the same domain you might get the following error (copying role definitions only works fine)
A duplicate name “62c4fcbb-7ff7-4cc3-842e-17476b2e6219″ was found. at Microsoft.SharePoint.SPFieldCollection.AddFieldAsXmlInternal(String schemaXml, Boolean addToDefaultView, SPAddFieldOptions op) at Microsoft.SharePoint.Deployment.ListSerializer.CreateOrUpdateField(SPList list, String fieldName, XmlNode fieldNode) at Microsoft.SharePoint.Deployment.ListSerializer.UpdateListFields(SPList list, Dictionary`2 listMetaData) at Microsoft.SharePoint.Deployment.ListSerializer.SetObjectData(Object obj, SerializationInfo info, StreamingContext context, ISurrogateSelector selector) at Microsoft.SharePoint.Deployment.XmlFormatter.ParseObject(Type objectType, Boolean isChildObject) at Microsoft.SharePoint.Deployment.XmlFormatter.DeserializeObject(Type objectType, Boolean isChildObject, DeploymentObject envelope) at Microsoft.SharePoint.Deployment.XmlFormatter.Deserialize(Stream serializationStream) at Microsoft.SharePoint.Deployment.ObjectSerializer.Deserialize(Stream serializationStream) at Microsoft.SharePoint.Deployment.ImportObjectManager.ProcessObject(XmlReader xmlReader) at Microsoft.SharePoint.Deployment.SPImport.DeserializeObjects() at Microsoft.SharePoint.Deployment.SPImport.Run()

On hotfixes: Please note that hotfixes are cumulative and later numbers super seeds earlier ones – so the game is to get the highest number of all  

(Updated) Conclusion (sort of)

Some of you have asked (in the comments) whether or not it all worked in the end. It actually did work! All of the above steps where enough to solve our immediate problems with the content deployment between our three environments.

Great! So do we use it? No.

We really really wanted to use this feature and all looked so well after our the latest hotfix deployment that we planned to use it for the existing publishing site as well as for a some additional upcoming site collections we’re developing.

One particular annoying feature (sorry for the pun) is breaking the content deployment which I have yet to find a solution for. The new sites are based on custom site definitions and a number of features that creates custom content types and site columns from a number of xml files. Some of these site columns are lookup columns that which cannot be created with xml files the same way as every other site column, because they need to refer to an existing list by list id (in the xml file) but the list will be assigned a dynamic id by the system when it’s created (by another feature that the site columns is dependent on).

To get around that problem a feature activation handled is executed that creates the lookup site columns using custom code that find the dynamic list id from the name. That code is roughly based on code found on codeplex (here). Some minor differences: I fixed some bugs with internal/display name mixup, reactivation problems and a missing “webid” (had to be dynamic as well) in the constructed field. The missing “webid” did cause content deployment to fail immediatly. For some reason I have yet to track down it now fails if the content type using one of the lookup site columns is in use, i.e. a list item of that type exists anywhere in the site collection. For lookup site columns created through the web interface there are no problems. Bummer.

The choice we’re facing (barring that I manage to solve the problem before long) is

1. Either: We can define the content type through xml files

2. Or: we can use content deployment and create content types manually through the web interface – they will be copied as part of the content deployment)

We chose the first option as content deployment at the current level of maturity seems too unstable. Furthermore we also decided to use only one strategy and therefore not use the content deployment for the first site collection that this article originally targetted (one where all content types where created through the web interface).  

How do we do it now? We use “stsadm -o backup/restore” to deploy new versions of the site. It’s essentially a database backup with all the benefits and drawbacks of such. It’s very stable. Specifically history of all items are maintained, you get a messed up user database where you’ll (eventually) find users from all environments, you need to explicitly set new ownership (to a valid user that can be found in the relevant environments AD), you get the luxury of copied security permissions sets and groups (which I might still build a tool to import/export).

 

Fixing those pesky DCOM event log error 10016 in a SharePoint farm environment

I’m responsible for a couple of SharePoint 2007 (MOSS) farms where all SharePoint servers showed the following error in the system event log:

Event Type: Error
Event Source: DCOM
Event Category: None
Event ID: 10016
Date: 1/17/2007
Time: 4:31:48 AM
User: <DOMAIN>\sa_adm
Computer: <SERVER>
Description:

The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID

{61738644-F196-11D0-9953-00C04FD919C1}

to the user <DOMAIN>\sa_adm SID (S-1-5-21-162740987-2502514208-3469184634-1119). This security permission can be modified using the Component Services administrative tool.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

The error would show up at regular intervals in clusters (4-12 at roughly the same time) and there would be a few more with other usernames and other class id’s. I had two fully functional farms with 3 SharePoint servers each and a number of standalone development machines. They all exhibited similar behavior.

The error listed above is that the user running the Central Administration web application doesn’t have access to activate (instantiate) the IIS WAMREG admin Service object (search the registry for the CLSID).

Strangely enough I didn’t observe any functional errors in the farms as a result of these errors – nothing seemed amiss (plenty of stuff didn’t work but none directly related to this).

An important note here is that the service users used in the farm are all standard domain accounts and only given additional local rights by the SharePoint installer and Central Administration (The one exception is that “aspnet_regiis -ga IIS_WPG” was executed after SharePoint install and initial configuration).

The following procedure removes the errors from the event log without compromising the entire security setup (yes, assign administrative rights for the service users would do the trick too) and has been verified by Microsoft consulting services.

On each SharePoint server do the following:

1. Click Start, Click Run, type “dcomcnfg” and click ok
2. Expand Component Services / Computers / My Computer / DCOM Config

   

3. Right click IIS WAMREG admin Service and choose Properties
4. Click the Security tag
5. Click Edit under Launch and Activation Permissions

   

6. Click Add
7. In the Select Users, Computers or Groups type computername\WSS_WPG and
   computername\WSS_ADMIN_WPG

   

8. Click ok
9. In the Permissions for UserName list, click to select the Allow check box

   

10. Click Ok twice.
11. Go back to the main Component Services window, right click the “netman” node and select Properties
12. Click the security tab
13. Click Edit under Activation Permissions
14. Click Add on the Launch Permissons Dialog
15. Enter “NETWORK SERVICE” in the edit box
16. Click Ok
17. Enable all the checkboxes for the NETWORK SERVICE

    

18. Click Ok twice
19. Finally, run “IISReset”

That should be it!

A little less event log errors to worry about – there are plenty left on a reasonable complex SharePoint farm…

As a side note: The above error also shows up in other applications as well – I’ve heard about it for exchange servers as well and more applications are probably affected. In that case you’ll need to search the registry for the actual DCOM application and assign the rights to another local group (or username as a last resort).