Find/Remove Obsolete Resource Labels


This is part 3 of 4 in a series on how to improve the way we usually work with resource (resx) files. At least the way my team and I work with them.

I generally like to – and do – use resource files for all string constants that are shown to end-users, however I do feel that it is needlessly cumbersome, therefore these posts:

  1. A Good Way to Handle Multi Language Resource Files
  2. Verify that your Resource Labels Exists
  3. Find Obsolete Resource Labels (this one)
  4. Supercharge your (Resource) Efficiency with Macros

Generally these issues are generic to .NET and not specific to SharePoint, though that is where I’m spending my time writing this.

So – Near the End of the Project – are those Resource Entries Still in Use?

The issue at hand is that you have hundreds of source files of various flavors and they are sprinkled with references to a number of resource files. When code is refactored or just deleted what happens to old resource labels? Likely nothing at all.

Are you happy with a ton of useless resource entries no longer in active use? What if you had to translate it to a couple of languages?

Quite obviously this is no biggie code/quality wise, but still…

The answer is that you run the PowerShell script below, that’ll check – and optionally fix – it 😉

The Script

I made a small script to check and remove the excess resource entries to slim down those resource files a bit.

The script will, given a starting location (i.e. the root folder for your solution)

  1. Go through every code file (to be safe every file, except a list of binary extensions) and look for resource labels of the form “$Resources:filename,label_key
  2. Search recursively for resx files
  3. For every one of those resx files it will look through the resource labels in use and flag those that it cannot find
  4. (Optionally) Do a “safemode” check where every file is searched for the resource label, i.e. necessary if you are using multiple/other resource lookup methods then the $Resource moniker
  5. (Optionally) If you choose you may remove them automatically but do make a dry run first to sanity check that you got the paths right and that you have all the source files to be searched

Usage:

    PS> & VerifyResxLabels.ps1 “path to solution dir” [-remove] [-safemode]

(Tip: Download the script to somewhere, write an ampersand (&) and then drag the ps1 file into the PowerShell window and then drag the solution folder to the window.)

You’ll definitely want to pipe the output to a file.

Limitations

There are obviously some limitations

  • It will not: Check out the resx files from source control (but it will show the file error in the output)
  • It will not: Respect commented out code – it’s simple pattern matching so commented out code will be treated as actual code (hardly an issue)
  • Safemode is very slow of necessity and will likely find false positives, i.e. it will play it safe and keep entries that exists in some files, even though the same label may be used in a completely different context
    • It’s an (O(n*m) algorithm, with number of files and number of labels) – My test with 1000 unique labels, 28 resx files and 2400 files it takes a night

Download the script here

SharePoint Advanced Large Scale Deployment Scripting – “Dev and QA gold-plating” (part 3 of 3)


I have been battling deployments for a while and finally decided to do something about it.

😉

The challenge is to handle a dozen WSP packages on farms that host 20-50 web applications (or host header named site collections) to complete the deployments in a timely manner, ensure that the required features are uniformly activated every time while minimizing the human error element of the poor guy deploying through the night.

The deployment scripts require PowerShell v2 and are equally applicable to both SharePoint 2007 and 2010 – the need is roughly the same and the APIs are also largely unchanged. Some minor differences in service names are covered by the code.

To keep this post reasonable in length I’ve split the subject into three parts:

Part 1 is the main (powershell) deployment scripts primarily targeted at your test/QA/production deployments

Part 2 is the scripts and configuration for automatic large scale feature activations

Part 3 (this part) is about gold-plating the dev/test/QA deployment scenario where convenience to developers is prioritized while maintaining strict change control and records

Note: This took a long time to write and it will likely take you, dear reader, at least a few hours to implement in your environment.

If you have not done so already read the two other parts first.

The Challenge

When you have a number of developers working on the same service line you need to be able to keep strict control of your QA and test environment.
So either you have one guy as the gatekeeper or you automate as much as possible. I’m the last type of guy so I’ve developed a “one-click” deployment methodology where the focus is
1. Simple and consistent deployment
2. Easy collection of solutions for next production deployment – you should deploy the exact code that you tested not a “fresh” build(!)

The Solution

Our build server is set up to produce date labelled zip files with the wsp and manual installer (exe) within folder of the same name.
I’m sure yours are different therefore the scripts work with a solution drop directory that accepts both zip files (that includes a wsp file) and also plain wsp files.
The folder structure is up to you I’ll just search recursively for zip and wsp.

The script files are:

03 DeployAndStoreDropDir.bat: The starting batch file that will execute the deployment process as a specific install user. That way the developers can login with their personalized service accounts and use a specialized one for deployment purposes. You need to change the user name in this batch file. The first time you run this you need to enter the password for your install account subsequent runs will remember it  (“runas /savecred”).

03b DeployAndStoreDropDirAux.bat: The batch file doing half the work. It’ll archive old log files in “.\ArchivedDeploymentLogFiles” (created if not present) and it execute “QADeploymentProcess.ps1” that is doing the hard work. At the end it’ll look for error log and write an alert that you should look for it. Saves a lot of time – I don’t go looking in log files if I don’t get that message.

SharePointLib\QADeploymentProcess.ps1: The script that is handling the unzipping and storage of the zip/wsp files and executing both the deployment and feature activation scripts. The goal is to have a simple drop dir where for WSP/zips and a single storage location, “SolutionsDeployed”, for the currently deployed code. The “SolutionsDeployed” folder will be updated at every deployment, so that there is only the latest version of each wsp deployed. You do not need to drop all wsps in the drop dir every time; it does not just delete the SolutionsDeployed folder at every deployment.

This is how the files are moved around for a couple of WSPs and a zip file:

In short:

  1. If you want to start it with a specific install user, just execute “03 DeployAndStoreDropDir.bat”
  2. If you want to start it in your own name, execute “03b DeployAndStoreDropDirAux.bat” and perhaps delete the other bat file.
  3. If you want to modify how it works go to the “QADeploymentProcess.ps1” file.
  4. If you want to work with zip files they can have the name and internal folder structure that you like. The one important thing is that they should only contain one WSP file. It is pretty smart; at the next deployment your zip file may be named differently (e.g. with a date/time in the name) and it will still remove the old version from “SolutionsDeployed” and store your new version with the new folder/zip name.
  5. At the end of your build iteration SolutionsDeployed will contain the latest version of your deployed code (provided that you emptied it at the last production deployment)

Closing Notes

It is not a big challenge to connect the remaining dots and have your build server do the deployment at certain build types and have a “zero-click” deployment, however I opted to not do it. It should be a conscious decision to deploy to your test and QA environments, not something equivalent to a daily checkin. Your milage may wary.

I would generally provide these scripts to everyone in the team and encourage them to use them on their own dev box once in a while – it always pays to have your dev environment resemble the actual production environment this makes it feasible to keep the code (almost) in sync on all dev environments.

The challenge may be the complexity of the scripts and the effort to understand them – you shouldn’t rely too much on stuff you don’t understand 😉

Feel free to adjust the script a bit (likely QADeploymentProcess.ps1) to suit your environment.

If you make something brilliant then let me know so I can include it in my “official” version. Please also leave credits in there so it’ll be possible for your successors to find the source and documentation.

Download

Grab the files here – part 1, 2 and 3 are all in there. I’ll update them a bit if I find any errors or make improvements.

Note: Updated Aug 29 2011, minor stuff.

Don’t forget to change the user name in “03 DeployAndStoreDropDir.bat” to your “install” account.

SharePoint Advanced Large Scale Deployment Scripting – “Features” (part 2 of 3)


I have been battling deployments for a while and finally decided to do something about it.

😉

The challenge is to handle a dozen WSP packages on farms that host 20-50 web applications (or host header named site collections) to complete the deployments in a timely manner, ensure that the required features are uniformly activated every time while minimizing the human error element of the poor guy deploying through the night.

The deployment scripts require PowerShell v2 and are equally applicable to both SharePoint 2007 and 2010 – the need is roughly the same and the APIs are also largely unchanged. Some minor differences in service names are covered by the code.

To keep this post reasonable in length I’ve split the subject into three parts:

Part 1 is the main (powershell) deployment scripts primarily targeted at your test/QA/production deployments

Part 2 is the scripts and configuration for automatic large scale feature activations

Part 3 is about gold-plating the dev/test/QA deployment scenario where convenience to developers is prioritized while maintaining strict change control and records

Note: This took a long time to write and it will likely take you, dear reader, at least a few hours to implement in your environment.

If you have not done so already read “Part 1” first and excuse me for a bit of duplicated text.

It’s all about the Features

This part is about features and managing their (de-)activations.

What you can do with this is:

  • Maintain a set of templates (“FeatureSets”) that defines what features should be activated on a given type of site – the scripts then ensures that they are properly activated
  • Set actions for features to be either activate, deactivate or reactivate through the configuration file
    • It safely handles web.config updates so that the scripts do not enter a race condition with the SharePoint timer service
  • Selectively override the default behavior and reactivate certain features when you need to

In other words you can ensure that your features are properly activated across sites and farms.

A note on reactivation is in order here. Reactivation is (force) deactivation and followed by activation which is particular useful for features that updates files in document library, e.g. masterpages. Some features require it when they have been updated but it is very rare that a feature always require reactivation.

Therefore I usually do not specify any features to always be reactivated in the configuration file, however if I know that we updated the masterpages I’ll update the batch file, or supply an additional one, where that feature is forced reactivated. The safe option of always reactivating is simply too slow if many files are deployed (or lengthy feature receivers) are involved.

The Configuration

The scripts work based on a configuration file that is shared between your dev, test, QA and production environments.

The procedure is:

  1. Identify all valid Sites/URLs in the local farm (i.e. always executed on one of the servers in the farm)
  2. For each site/URL go through all Feature sets and ensure activations, deactivations and reactivations required
    1. Keep track of reactivations so same feature is only reactivated once at each scope, i.e. overlap between FeatureSets are allowed as long as they specify the same action
    2. Report inconsistencies as errors, but do continue with the next features so that you can have the complete overview of all errors at the end of the run

A sample configuration file is (note: I just picked some more or less random WSPs from codeplex):

<?xml version="1.0" ?>
  <Config>
    <Sites>
      <!-- note: urls for all environments can be added, the ones found in local farm will be utilized -->
      <!-- DEV -->
      <Site url="http://localhost/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      <Site url="http://localhost:8080/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="MySite" />
      </Site>
      <Site url="http://localhost:2010">
        <FeatureSet name="CA" />
      </Site>
      <!-- TEST -->
      <Site url="http://test.intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
      <!-- PROD -->
      <Site url="http://intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
    </Sites>

    <FeatureSets>
      <FeatureSet name="BrandingAndCommon">
        <Solution name="AccessCheckerWebPart.wsp" />
        <Feature nameOrId="AccessCheckerSiteSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="AccessCheckerWebPart" action="activate" ignoreerror="false" />
        <Solution name="Dicks.Reminders.wsp" />
        <Feature nameOrId="DicksRemindersSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="DicksRemindersTimerJob" action="activate" ignoreerror="false" />
        <!-- Todo - all sorts of masterpages, css, js, etc. -->
      </FeatureSet>
      <FeatureSet name="Intranet">
        <Solution name="ContentTypeHierarchy.wsp" />
        <Feature nameOrId="ContentTypeHierarchy" action="activate" ignoreerror="false" />
      </FeatureSet>
      <FeatureSet name="MySite">
      </FeatureSet>
      <FeatureSet name="CA">
      </FeatureSet>
    </FeatureSets>

    <Solutions>
      <!-- list of all solutions and whatever params are required for their deployment -->
      <Solution name="AccessCheckerWebPart.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="ContentTypeHierarchy.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="Dicks.Reminders.wsp" allcontenturls="true" centraladmin="false" upgrade="false" resettimerservice="true" />
    </Solutions>
  </Config>

Where the interesting bits are that Sites contain a number of named FeatureSets. The FeatureSets are gathered in their own section (for reusability) and defines a number of solutions to be deployed and the features to be handled. It is good practice to list the solutions for the features that you are working on so you can be sure that the solution deployment matches your features. Do not spend time on noting all the SharePoint system features to be activated as they are normally handled just fine by the Site Definitions.

Each feature activation line looks like:

<Feature nameOrId=”AccessCheckerWebPart” action=”activate” ignoreerror=”false” />

Where nameOrId is either the guid (with or without braces, dashes etc.) or the name of the feature, i.e. the internal name in the WSP packages, not the display name. When possible I always opt for flexibility 😉

The action attribute is either “activate”, “deactivate” or “reactivate”.

The ignoreerror is simply a boolean true/false switch that determines how the error should be logged if the feature fails to be activated. It is quite usable for the occasional inconsistencies between environments, e.g. if you are removing a feature then it might be present in production but no longer in the test environment. The script continues with next feature in case of errors regardless of this switch.

It is important to note that web scoped features are not supported here, “only” farm, web app or site scope is allowed.

The features are activated in the order that they are present in the config file.

The Scripts

There are two sections here:

  1. A number of SharePoint PowerShell scripts that include each other as needed
  2. A few batch files that start it all. I generally prefer to have some (parameter less) batch files that executes the PowerShell scripts with appropriate parameters

I have gold-plated the scripts quite a bit and created a small script library for my SharePoint related PowerShell.

The PowerShell scripts are too long to write in the post, they can be downloaded here. The list is (\SharePointLib):

*(Part 1) DeploySharePointSoluitions.ps1: Start the WSP deployments. Parse command line arguments of config file name and directory of WSP solutions

*EnsureFeatureActivation.ps1: Start the feature activation scripts (need command line arguments)

*(Part 3) QADeploymentProcess.ps1: Deployment process for dev, test and QA environments (need command line arguments)

SharePointDeploymentScripts.ps1: Main script that defines a large number of methods for deployment and activations.

SharePointHelper.ps1: Fairly simple SharePoint helper methods to get farm information

*RestartSharePointServices.ps1: Restart all SharePoint services on all servers in the local farm

Services.ps1: Non-SharePoint specific methods for handling services, e.g. restarts

Logging.ps1: Generic logging method. Note that there are options for setting different log levels, output files, etc.

The ones with an asterisk are the ones that you are supposed to execute – the others just defined methods for the “asterisk scripts”.

The batch files (same download) (root of zip download):

(Part 1) 01 Deploy Solutions.bat: Start the solution deployment. It will deploy all the WSP files dropped in “\SolutionsDropDir\” that are also specified in the “\DeploymentConfig.xml”:

02 Activate Features.bat: Works with the “\DeploymentConfig.xml” file and ensures that all features are activated.

In other words you’ll execute “02 Activate Features.bat” to ensure that all features are activated in your farm.

The important line in 02 Activate Features.bat file is

powershell.exe -File “SharePointLib\EnsureFeatureActivation.ps1” “%config%” “” >>ActivateFeatures.log

And if you need to reactivate your branding features (masterpages) you can just copy the batch file and change the line to something like:

powershell.exe -File “SharePointLib\EnsureFeatureActivation.ps1” “%config%” “BrandingMasterPages,BrandingCSS,BrandingPageLayouts” >>ReactivateFeatures.log

to force the three (made up) features “BrandingMasterPages”, “BrandingCSS” and “BrandingPageLayouts” to be reactivated.

By default you will get one log file from each batch file named after the operation, one log file with a timestamp and in case of any errors an additional error log with the same timestamp. In other words if the error log file is not created then no errors occurred.

Runtimes

The scripts are quite fast and the time to execute is determined by the time it takes to activate the features within SharePoint. Therefore if the script has nothing to do, i.e. all features are activated as they should, then it’s normally less than 1 minute to check everything and return.

What usually takes time is features with many files to be added to document libraries (branding) and features that modify the web.config as we wait for the timer job to complete (assuming that the API is used to do this).

Closing Notes

The scripts write quite a bit of logging that can be daunting (especially if you set it to verbose) but don’t worry – I rarely look at them anymore. Instead I look for the EnsureFeatureActivation_date_error_log. If it’s not there then no error occurred and no reason to look into the log files.

These scripts have been a huge benefit for our deployment in order to reduce the post-deployment errors and fixes. It is however also the first to be blamed for errors whenever some feature does not behave as intended (“the unghosted file is not change that must be the fault of the deployment scripts”) and it rarely is the source of the error.

They can be blamed for being a bit complicated though and that is a fair point. If you don’t understand what they do and how you shouldn’t use them.

If you make some brilliant changes then let me know so I can include it in my “official” version.

Please also leave credits in there so it’ll be possible for your successors to find the source and documentation.

Download

Grab the scripts here.

Note: Updated Aug 29 2011.

SharePoint Advanced Large Scale Deployment Scripting – “Deploy” (part 1 of 3)


I have been battling deployments for a while and finally decided to do something about it.

🙂

The challenge is to handle a dozen WSP packages on farms that host 20-50 web applications (or host header named site collections) to complete the deployments in a timely manner, ensure that the required features are uniformly activated every time while minimizing the human error element of the poor guy deploying through the night.

The deployment scripts are equally applicable to both SharePoint 2007 and 2010 – the need is roughly the same and the APIs are largely unchanged. Some minor differences in service names are covered by the code.

To keep this post reasonable in length I’ve split the subject into three parts:

Part 1 is the main (powershell) deployment scripts primarily targeted at your test/QA/production deployments

Part 2 is the scripts and t for automatic large scale feature activations

Part 3 is about gold-plating the dev/test deployment scenario where convenience to developers is prioritized while maintaining strict change control and records

Note: This took a long time and countless iterations to develop and it will likely take you, dear reader, at least a few hours to implement in your environment.

The Requirements

My design goals were to solve

  • Automatic solution deployments (including timer jobs resets)
  • Consistent activation of features across the farm while allowing for different types of sites
  • Easy, fast and reliable execution
  • Not too awkward configuration of solutions and features for farms with many sites – it will never be a no-brainer
  • Exact same configuration (file) to be re-used across Dev, Test, QA and Prod tiers

The cost is some fairly complex scripts that took quite a while to write and test.

The same script works equally well for both 2007 and 2010.

There is almost no difference between SharePoint 2007 and 2010 in this area – 2010 comes with some new CmdLets for PowerShell but they are often just convenient access to the API. I’m taking the extra effort to hit the API directly which is essentially unchanged for deployment purposes. One of the really cool new features in 2010 is the ability to control solution upgrade in detail which is of course supported by these scripts simply by choosing the upgrade option for a given solution.

The Solution

The idea is that you have the same configuration file for all your app tiers, and

  1. When you need to deploy a new version of some of your WSPs you just provide them and execute the scripts
  2. Script will scan the local farm and map the URLs with the provided WSPs and deploy them
  3. Run through every identified Site and match feature activations with the configuration file and fix any discrepancies

The features are

  • Deploy WSP files regardless of whether or not they were deployed beforehand (uninstall followed by install is the default installation mode for me)
    • If solution deployment goes wrong the behavior is to restart all SharePoint related services on all servers in farm and retry (a few times). It surprisingly often solves problems like locked files etc.
    • If a solution contains timer jobs the timer service is restarted on all servers in farm. Finally
  • Upgrade existing WSP files
  • (Part 2) Scan the farm and ensure that required features are activated (or perhaps deactivated) on specified sites
    • Farm and WebApp scoped features are by default deactivated after an uninstall/install – this script reactivates them
    • Features are always (de)activated with the “force” parameter just for good measure
    • Yes, it’ll wait for web config modifications jobs when required
  • (Part 2) Re-activate selected features i.e. deactivate first then activate
  • (Part 3) Automatic deployment and activations for dev, test and QA environments

The Configuration File

The hearth of the scripts is the configuration file.

The one configuration file can be re-used for both handling solution deployment and feature activations across all tiers of your farms. You can also conceivable use the same config file for different types of farms, e.g. the intranet and extranet, however I feel that in those scenarios it is better to keep those things separate.

The configuration file contains:

  • A number of “Sites” which is the URLs of either (host header named) site collections or web applications
    • The local farm will be scanned and the sites found in the config file that are present in the local farm is the ones that we work on – the rest are ignored (e.g. running in test it’ll skip all the production URLs)
    • Each site node contains a number of FeatureSets but no list of features or solutions
  • A number of “FeatureSets” that groups WSPs and features. Features are either “activated”, “deactivated” or “reactivated”
    • (The feature thingie is in part 2)
    • For a given site the union of all the solutions specified in all the FeatureSets associated with the sites defines what solutions should be deployed to that site.
  • A number of “Solutions” that list the WSP names and their individual options, i.e. deploy to central admin, all content URLs, upgrade or (re)install, does it contain timer jobs (which requires timer service to be restarted on all farms)
    • Note: The scripts deploy all the solutions in a given “drop” directory and ignore all the configuration nodes without a matching WSP file

To demo it I’ve downloaded a couple of WSPs from CodePlex and created the following configuration file for my dev box:

<?xml version="1.0" ?>
  <Config>
    <Sites>
      <!-- note: urls for all environments can be added, the ones found in local farm will be utilized -->
      <!-- DEV -->
      <Site url="http://localhost/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      <Site url="http://localhost:8080/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="MySite" />
      </Site>
      <Site url="http://localhost:2010">
        <FeatureSet name="CA" />
      </Site>
      <!-- TEST -->
      <Site url="http://test.intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
      <!-- PROD -->
      <Site url="http://intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
    </Sites>

    <FeatureSets>
      <FeatureSet name="BrandingAndCommon">
        <Solution name="AccessCheckerWebPart.wsp" />
        <Feature nameOrId="AccessCheckerSiteSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="AccessCheckerWebPart" action="activate" ignoreerror="false" />
        <Solution name="Dicks.Reminders.wsp" />
        <Feature nameOrId="DicksRemindersSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="DicksRemindersTimerJob" action="activate" ignoreerror="false" />
        <!-- Todo - all sorts of masterpages, css, js, etc. -->
      </FeatureSet>
      <FeatureSet name="Intranet">
        <Solution name="ContentTypeHierarchy.wsp" />
        <Feature nameOrId="ContentTypeHierarchy" action="activate" ignoreerror="false" />
      </FeatureSet>
      <FeatureSet name="MySite">
      </FeatureSet>
      <FeatureSet name="CA">
      </FeatureSet>
    </FeatureSets>

    <Solutions>
      <!-- list of all solutions and whatever params are required for their deployment -->
      <Solution name="AccessCheckerWebPart.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="ContentTypeHierarchy.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="Dicks.Reminders.wsp" allcontenturls="true" centraladmin="false" upgrade="false" resettimerservice="true" />
    </Solutions>
  </Config>

Note: I more or less picked some random WSPs from CodePlex for demo purposes.

Note 2: The scripts will respect the order given in the config file. The Sites are processed in order, the FeatureSets listed first within a “Site” node is handled in top-down order. The solutions deployed to the farm are deployed in the order they are listed in the “Solutions” node.

Note 3: XML is case sensitive; therefore, all the named sections are case sensitive. The one exception is that I accept different casing between the WSP file names and the name they are given within the config file.

The Scripts

There are two sections here:

  1. A number of SharePoint PowerShell scripts that include each other as needed
  2. A few batch files that start it all. I generally prefer to have some (parameter less) batch files that executes the PowerShell scripts with appropriate parameters

I have gold-plated the scripts quite a bit and created a small script library for my SharePoint related PowerShell.

The PowerShell scripts are too long to write in the post, they can be downloaded here. The list is (\SharePointLib):

*DeploySharePointSoluitions.ps1: Start the WSP deployments. Parse command line arguments of config file name and directory of WSP solutions

*(Part 2) EnsureFeatureActivation.ps1: Start the feature activation scripts (need command line arguments)

*(Part 3) QADeploymentProcess.ps1: Deployment process for dev, test and QA environments (need command line arguments)

SharePointDeploymentScripts.ps1: Main script that defines a large number of methods for deployment and activations.

SharePointHelper.ps1: Fairly simple SharePoint helper methods to get farm information

*RestartSharePointServices.ps1: Restart all SharePoint services on all servers in the local farm (see runtime notes below)

Services.ps1: Non-SharePoint specific methods for handling services, e.g. restarts

Logging.ps1: Generic logging method. Note that there are options for setting different log levels, output files, etc.

The ones with an asterisk are the ones that you are supposed to execute – the others just defined methods for the “asterisk scripts”.

The batch files (same download) (root of zip download):

01 Deploy Solutions.bat: Start the solution deployment. It will deploy all the WSP files dropped in “\SolutionsDropDir\” that are also specified in the “\DeploymentConfig.xml”:

02 Activate Features.bat (Part 2): Works with the “\DeploymentConfig.xml” file and ensures that all features are activated.

In other words you’ll execute “01 Deploy Solutions.bat” to deploy all your solutions.


By default you will get one log file from each batch file named after the operation, one log file with a timestamp and in case of any errors an additional error log with the same timestamp. In other words if the error log file is not created then no errors occurred.

Note: Scripts must be executed on one of your SharePoint servers which sufficient rights. That is more or less local admin and owner on the individual databases (or local admin on the SQL box as well).

Note 2: I write quite a bit of logging it is invaluable in times of trouble. You can change the amount in the logging.ps1 file by adjusting the loglevel variable.

Note 3: In case of deployment errors the default behavior is to restart every SharePoint service on every server and then retry. These service restarts are performed async, so it should take no more than a minute or two, no matter how many servers you have (almost).

Note 4: Any WSP found in the drop dir that has not been specified in the config is considered an error – therefore you know that all the wsps provided are deployed if no errors are reported.

Runtimes

The time it takes to search the local farm for relevant site collections / web applications is negligible so you shouldn’t worry unduly about impact on runtime to add 300 URLs to cover all your possible environments including every developer’s individual preferred setup.

But there is no magic and it simply runs in the time it takes to deploy solutions to your environment. If it takes about 3 minutes for a full retract, remove, add, deploy then it takes 30 minutes for 10 WSPs.

The default way of handling deployment errors is to restart all SharePoint services in the farm. The clever way of doing that is to send the stop/start commands to all services first and then wait for the operations afterwards. That way the time taken to do restarts is about the time of restarting the slowest service in the farm once – it usually takes 1-2 minutes regardless of the number of servers. In my opinion this is quite cool 🙂

Closing notes

If you are wondering about my exception handling note that the scripts were originally made for PowerShell 1.0 and later I decided to only target 2.0. Some statements may be written prettier without me taking the time to do so.

These scripts save us a ton of time and prevents a lot conversations of the sort “oh I forgot to activate that feature, 2 secs, there you go” – but of course quite some time has been spent creating the deployment scripts and fixing errors within them.

Some of the scripts are quite usable for other purposes. For instance, the remote service restart script I have not seen anywhere else – this is simpler than installing PowerShell remoting (WinRM?) on every server.

If you make some brilliant changes then let me know so I can include it in my “official” version.

Please also leave credits in there so it’ll be possible for your successors to find the source and documentation.

Download

Grab the scripts here (updated with part 2 and 3).

Note: Updated Aug 29 2011.

Automatic Fix of Pre Upgrade Issues (SharePoint 2007 to 2010)


Ever tried to run the PreUpgradeCheck command at your MOSS farm and received a frightening long list of issues that takes forever to solve manually?

If you are like me you refuse to do any manual work that takes more than 2 hours (and you’ll have to re-do it for every farm) – so I’ve tried to script all the required fixes. This is one of those posts that I’ve been working on for months so I hope you appreciate it – I’m a bit proud of the script at the end of the post.

The goal is to have a PowerShell (v1.0) script fix every one of those blocking issues you see in the PreUpgradeCheck report. On my own very messy (intentionally so!) farm with 40+ databases and site collections and a large number of solutions I’ve managed to fix all but the one about my operating system needing an upgrade to Server 2008. That one I’ll not work on 😉

Going from:

To this:

Note: It will only work of you have installed SP2 and cumulative update from October 2009. At least. I recommend that you work of as new a version as possible.

WARNING

The script will remove invalid “stuff” from your sites so be absolutely sure that every solution that you care about is deployed to the farm – otherwise all features, files and webparts from missing solutions will be removed from the sites.

The script will also publish unpublished pages if they contain some of these invalid webparts.

I give you no warrenty – backup your farm first. I’m pretty sure I won’t mess it up though…

What it does

The script will check a number of things in your farm by working on the output of the “stsadm –o enumallwebs” command.

I use some direct SQL to get information out that is not possible through the API, but all modifications are done through the SharePoint 2007 API in a supported way only.

The script currently does 5 things:

Remove Invalid Webparts

(AKA: “The following web part(s) are referenced by the content, but they are not installed on the web server“)

This is the method I started with and by far the biggest timesaver.

It uses the “stsadm –o enumallwebs” to get a list of webparts that’s currently missing in your farm. They have been uninstalled at some point and are now those webpart with “Fatal Errors” on some of your pages. Some of the missing webparts are not even listed as failing with fatal errors (they have socalled “bad markup”) they are just listed on the page with and ID and no description (i.e. if you go to a given page and add “?contents=1” to go to the webpart maintenance page).

Trouble is that there can be lots of these and there is no easy way to find all the pages.

This function will query the SQL for the pages where these webparts are (both the one with fatal errors and the ones with “bad markup”) and then remove them through the API. The offending pages will be published (if not already), whatever checkouts they may have been there is overridden, the webparts are removed and the page re-published.

It will take all pages not just the one in document libraries (i.e. those that came through features)

Note: I’m only handling the case of shared webparts I did not have any user specific webparts in my test farm so I’m uncertain how that will be handled.

Note2: I’ve copied the PublishListItem from some of Gary Lapointes excellent extentions and converted them to powershell. It seems that no one has been as thorough handling special cases as he.

Remove Orphaned Site Collections

(AKA: “Issue : Orphaned site collections“)

Site collections can get orphaned in a number of ways; the fix is simply to delete those.

You will not lose anything (new) as those sites were not accessible before anyway.

Remove Uninstalled/missing Features Activated at Web scope

Features that have been uninstalled at some point often linger in the content databases because they are still activated in some web or site. That activation is not deleted when the feature is removed though the feature will have no actual function anymore.

I wrote quite a bit on this a while ago.

The script will run through all those SPWeb’s and deactivate the offending features.

Note that I’m not crawling all webs in the site I only access those with missing features.

Remove Uninstalled/missing Features Activated at Site Collection Scope

(AKA: “The following feature(s) are referenced by the content, but they are not installed on the web server“)

Similarly features may be activated at site collection level and forgotten. In this case these features are not reported by the “stsadm –o enumallwebs” command and I have to run through every Site Collection in the farm to locate the features.

Remove Missing Setup Files

(AKA: “The following setup file(s) are referenced by the content, but they are not installed on the web server“)

After features have been uninstalled it is quite common that they forget to remove the files they installed on the farm. I suppose it’s just the way SharePoint works. If a feature installs a file in a document library somewhere changes are it will remain.

What are those files? Some examples are stuff in the style library, webpart gallery, master pages, page layouts, etc…

This method uses SQL to find where the missing files are still references and deletes them from the lists. It even deletes files that have been unghosted which could actually still be accessed because SharePoint has made a customized copy of it.

I’m still considering if it would be better to only remove the ones that are still ghosted.

Note: In my tests some page layouts could not be removed because they were still being referenced in the farm. That is clearly an error in my farm, however the script will not fix that. It really does require special handling by a human to decide what to do. I suppose it’s a good thing that they are not removed in this case as some things might otherwise break.

What it does not no

Out of scope of script:

  1. Fixing the OS version
  2. Fixing the database version

Sorry 😉

How to Execute

First of all the script requires you to be site collection owner on every site collection, farm admin and have owner rights in every content and configuration database. Pretty much all you can get.

In the beginning of the script there is a few switches that you should know:

  • The logging level, set it to 3 to get lots of info: $verboseLevel = 3. Lower is less.
  • The “$whatif” ´switch. Set it to “$true” to have the script simulate a run but not change anything in your farm
  • The “$targetdb” that can limit the script to a single content database only. Blank equals all databases a name indicates one (note: Not the site name)

I’ve built in a lot of error handling and logging to let you know what’s going on.

Note that STSADM must be included in the path setting.

Recommended sequence of execution

Everything need to run on the SharePoint 2007 server and it must have (at least) PowerShell 1.0 installed.

  1. Backup all content databases. This is not optional. Easiest to use stsadm –o backup
  2. Use stsadm -o preupgradecheck
  3. Run script on one or all databases with the “whatif” statement set to true
  4. Check and recheck the output and verify that it looks correct to you.
    1. You may want to reduce the logging level a bit
  5. Run with $whatif set to $false, i.e. let it work!
    1. Depending on your confidence in me you can ask to work on only one database at a time
  6. Check the output!
    1. Errors and warnings will be written both to the output stream (what is piped down in your logfile) and on the screen
  7. Run preupgradecheck again and see the result J

Note to run the script use something like “powershell -file FixUpgradeIssues.ps1 >>fixupdateoutput.txt”, i.e. always pipe the output to a file that you can investigate later.

Runtime is about 5 minutes on my 40+ database test system so reasonably ok given the task.

I’ve been working on this script for a couple of months and have fixed a huge number of issues and funny edge cases; however I’ve likely missed a few. Let me know in the comments.

Good luck J

Download the Actual Script

All 850 lines of powershelling are not pretty in a blog post, so you can download it here.

[Updaet 25-12-2010: Now with correct file]

How to install a SharePoint 2010 Complete (Dev) Server without AD


This post details how you do install a SharePoint 2010 as a “complete” install without an AD, which is very useful to me as a development server (with or without Visual Studio).

This applies to both virtual and physical machines but I always work with VMs because traditionally SharePoint dev environments need to be re-installed once in a while and that’s easier with VMs.

Why would you want this?

  • A development server with all components is likely to resemble your test/QA/production environment a lot more than the alternative standalone install
  • A server with a local install of SharePoint with non-AD accounts can be run with or without an AD domain – you can even run the VM as a domain server disconnected from the actual domain e.g. at home or the commute
    • Alternative 1: Make your server an AD server but that changes all sorts of stuff with user management and will definitely not resemble any of your production servers
    • Alternative 2: Create two VMs one being the AD one being the actual development server connected on the same VLAN and waste a lot of resources for (almost) nothing.
  • To eliminate any need for rogue AD servers on your network that some develop accidently connected directly to the network and running DHCP, DNS etc. Don’t trust your developers or external consultants to care about your network!
  • I want a full SQL Server!

Why would you not want this?

  • This is not a supported development environment from Microsoft – they support installing a so-called standalone development environment without any of the frills. It’s easier and it’s officially supported. It’s even doable on Windows 7.
    • Why anyone would want to develop SharePoint on a Windows 7 machine is beyond me, the runtime environment for your code will always be a server 2008 so why not develop and test it directly on such a box? Surely you develop only in VMs so that you are able to create a clean dev environment easily once in a while…

How to

The procedure is fairly simple except for the final steps. Note that you can (and should) use whatever tools you can to help you out, I’ll point at the promising AutoSPInstaller at CodePlex.

Procedure:

  1. Create / install a Server 2008 (R2) 64 bit with
    1. Visual Studio 2010
    2. SQL Server 2008 (use a local user as service account)
    3. … and whatever other tools you are fond of …
    4. (Remember to sysprep/snapshot it at this stage)
  2. Install SharePoint 2010 with all prerequisites
    1. Scripted or not – do not run Config Wizard yet (It would result in “Local accounts should only be used in standalone mode” error)
  3. Create the farm by (trick #1)
    1. Start the SharePoint PowerShell
    2. Create a local service account
    3. Create a farm by running the “New-SPConfigurationDatabase” cmdlet and supply parameters for the service account, DB name, DB server and passphrase (thanks to Neil ‘The Doc’ Hodgkinson for that)
    4. After it finishes start the Config Wizard (interactive or not) and configure your server with all components
  4. Configure the farm services as you like
    1. I usually just use the wizard in Central Admin to configure all the Service Applications with some fairly useful values it works well enough
  5. Enterprise Search doesn’t work to fix it see below… (trick #2)

The Trouble with Search

Search will fail with a number of errors and in the search administration the Query Component will remain stuck in the initializing state:

The other bunch of event log errors etc. is listed at the end of this post for the benefit of Google.

As far as I can conjecture the problem is that the timer service is trying to setup a network share for every query component where the crawlers can dump their data. It is trying to setup that share with a domain account that happens to be a local user instead in this case and fails with either an “Access Denied” error or a “System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated”.

The share name it’s trying to use is the same as the query role, i.e. “Guid-query-0″ pointing to (if using default locations) “C:\program files\Microsoft Office Servers\14.0\Data\Office Server\Applications” with change permissions for the “WSS_WPG” group.

Unfortunately it does not help to just create the share for it apparently the query components insist on waiting for the timer job to complete successfully. L

The Search Fix

The fix is fairly simple and almost completed by Gary Lapointe whom I owe great thanks for doing most of the hard work in his post on scripting the Enterprise Search installation and the comments below his post (thanks to Marco van Wieren).

The fix is simply to create and configure all the enterprise search components from PowerShell as it allows you to set a few more options, specifically the share name for the query components so that you are then allowed to create them yourself.

The script was originally made for configuring search components across an entire farm and therefore a bit more complicated than it strictly has to be. I left it in there while adding support for single server install as well. Gary’s script was made for beta 2 and I’ve fixed a few simple errors/typos, corrected the few API changes between beta 2 and RTM and finally added the share name support.

The script is quite long a not suitable for pasting into a blog – download instead.

The script needs a configuration file with something like this:

<Services>
    <EnterpriseSearchService ContactEmail="no-reply@SharePointDev1.com"
                             ConnectionTimeout="60"
                             AcknowledgementTimeout="60"
                             ProxyType="Default"
                             IgnoreSSLWarnings="false"
                             InternetIdentity="Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)"
                             IndexLocation="C:\Program Files\Microsoft Office Servers\14.0\Data\Office Server\Applications"
                             PerformanceLevel="PartlyReduced"
                             Account="localhost\saservice"
                             ShareName="SearchShare">

        <EnterpriseSearchServiceApplications>
            <EnterpriseSearchServiceApplication Name="Enterprise Search Service Application"
                                                DatabaseServer="localhost"
                                                DatabaseName="SharePoint_Search"
                                                FailoverDatabaseServer=""
                                                Partitioned="false"
                                                Partitions="1"
                                                SearchServiceApplicationType="Regular">
                <ApplicationPool Name="SharePoint Enterprise Search Application Pool" Account="localhost\saservice" />
                <CrawlServers>
                    <Server Name="localhost" />
                </CrawlServers>
                <QueryServers>
                    <Server Name="localhost" />
                </QueryServers>
                <SearchQueryAndSiteSettingsServers>
                    <Server Name="localhost" />
                </SearchQueryAndSiteSettingsServers>
                <AdminComponent>
                    <Server Name="localhost" />
                    <ApplicationPool Name="SharePoint Enterprise Search Application Pool" Account="localhost\saservice" />
                </AdminComponent>
                <Proxy Name="Enterprise Search Service Application Proxy" Partitioned="false">
                    <ProxyGroup Name="Default" />
                </Proxy>
            </EnterpriseSearchServiceApplication>
        </EnterpriseSearchServiceApplications>
    </EnterpriseSearchService>
</Services>

Remarks:

  • I replace “localhost” with the actual computer name in the script
  • The Share Name (here “SearchShare”) will be created by the script as well, so whatever you call it doesn’t matter
  • The config file shown can be reused on every machine provided that the local service account “saservice” has been created before

To continue and complete step 5 in the procedure above (sorry for the numbering wordpress is messing up the html):

  1. Start PowerShell shell (I will load the SharePoint snapin if it’s not a SharePoint Management Shell)
    1. Load the “SetupEnterpriseSearch.ps1” script (just drag the file into the shell and execute) which will define the required functions
    2. Execute “Start-EnterpriseSearch “<path>\searchconfig.xml””
    3. Wait for a few minutes and watch for errors
  2. Go to the Search Administration and verify that your new search topology works
    1. It should look something like this:

    2. If you configured search in step 4 you will have two
    3. If you have two you can safely go back to “Manage service applications” and delete the one named “Search Service Application 1” (and associated databases) – the one created by the script is “Enterprise Search Service Application”
  3. Try it! Go to a local SharePoint site and search for something
    1. Before the search would return a server error 500 so anything else than that can be considered a success
    2. I like to add a few documents and have them show up in the search before I call it a success…

Caveats / Fast Search

Don’t know if it fair to call it a caveat however only the Enterprise Search is demonstrated here, the Fast Search behaves similarly in respect to the “share trouble” and will probably need the same fix as the enterprise search. I’ve not found the time or need to poke around with that just yet, but it should be doable in less than a day given the foundation above (for someone skilled in SharePoint and powershell).

Conclusions

It works; I’ll use it from now on 🙂

… and I hope the nice chaps at AutoSPInstall will include this fix in their tool.

Scared of being in unsupported land?

  • It’s only your dev server and it did move a lot closer to production that the standalone dev machine option
  • It also protected your network from rogue AD servers that might potentially kill half your network if you are unlucky

So how well is this tested? Quite well for a single server install and not at all for a farm install (not by me at least). Trust it with the former and test it yourself if you need the latter.

Observed Errors

I got a lot of different errors, here they are for the benefit of Google.

Event log entry after completing the configuration wizard in Central Administration:

Log Name: Application
Source: Microsoft-SharePoint Products-SharePoint Server Search
Date: 11-06-2010 22:20:17
Event ID: 2579
Task Category: Administration
Level: Error
Keywords:
User: SHAREPOINTDEV1\saservice
Computer: SharePointDev1
Description:
Component a61ca0ca-194f-4cf0-bb5c-8ca998178935-query-0 of search application ‘Search Service Application’ has failed to execute transition sequence ‘initialize with empty catalog’ due to the following error: System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String& sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage(). It is now in state Uninitialized.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Server Search” Guid=”{C8263AFE-83A5-448C-878C-1E5F5D1C4252}” />
<EventID>2579</EventID>
<Version>14</Version>
<Level>2</Level>
<Task>14</Task>
<Opcode>0</Opcode>
<Keywords>0×4000000000000000</Keywords>
<TimeCreated SystemTime=”2010-06-11T20:20:17.723875000Z” />
<EventRecordID>3926</EventRecordID>
<Correlation ActivityID=”{B1431F7E-1D0C-4CB7-B690-F0F016447FE4}” />
<Execution ProcessID=”956? ThreadID=”3484? />
<Channel>Application</Channel>
<Computer>SharePointDev1</Computer>
<Security UserID=”S-1-5-21-452889701-636363473-2591022535-1012? />
</System>
<EventData>
<Data Name=”string0?>a61ca0ca-194f-4cf0-bb5c-8ca998178935-query-0</Data>
<Data Name=”string1?>Search Service Application</Data>
<Data Name=”string2?>initialize with empty catalog</Data>
<Data Name=”string3?>System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String&amp; sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage()</Data>
<Data Name=”string4?>Uninitialized</Data>
</span
</EventData>
</Event>

And from the ULS log:

06/11/2010 22:20:17.72         OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server Search         Administration         fea9        Critical        Component a61ca0ca-194f-4cf0-bb5c-8ca998178935-query-0 of search application ‘Search Service Application’ has failed to execute transition sequence ‘initialize with empty catalog’ due to the following error: System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated. Parameter name: sddlForm at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm) at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm) at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String& sddl) at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path) at Microsoft.S…        b1431f7e-1d0c-4cb7-b690-f0f016447fe4
06/11/2010 22:20:17.72*        OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server Search         Administration         fea9        Critical        …harePoint.Administration.SPServer.CreateFileShare(String name, String description, String path) at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component) at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage(). It is now in state Uninitialized.        b1431f7e-1d0c-4cb7-b690-f0f016447fe4
06/11/2010 22:20:17.72         OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server         Unified Logging Service         2m1i        Verbose         Adding event 2579 (category: Administration, product: SharePoint Server Search) to spam monitoring list        b1431f7e-1d0c-4cb7-b690-f0f016447fe4
06/11/2010 22:20:17.72         OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server Search         Administration         djs2        Medium         SearchApi (): executing SetQueryComponent(d355048f-d4fa-4f31-88b0-342b5ed48e5c, null, null, null, null, Uninitialized, Uninitialized, null, -1, Failed, null, False, null, null, False, null)        b1431f7e-1d0c-4cb7-b690-f0f016447fe4

And another event log:

Log Name: Application
Source: Microsoft-SharePoint Products-SharePoint Foundation
Date: 12-06-2010 20:40:26
Event ID: 6398
Task Category: Timer
Level: Critical
Keywords:
User: SHAREPOINTDEV1\saservice
Computer: SharePointDev1
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.CrawlReportJobDefinition (ID 9529aace-a679-4fc9-ab8d-325780484cf0) threw an exception. More information is included below.
The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component ’3147b99c-8f3a-41e9-a08b-296f930af877' in search application ‘Enterprise Search Service Application’ is in a good state and try again.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6FB7E0CD-52E7-47DD-997A-241563931FC2}” />
<EventID>6398</EventID>
<Version>14</Version>
<Level>1</Level>
<Task>12</Task>
<Opcode>0</Opcode>
<Keywords>0×4000000000000000</Keywords>
<TimeCreated SystemTime=”2010-06-12T18:40:26.553054700Z” />
<EventRecordID>4159</EventRecordID>
<Correlation ActivityID=”{6CED0041-2038-43E3-AB79-4DEFBB4216B3}” />
<Execution ProcessID=”1324? ThreadID=”1532? />
<Channel>Application</Channel>
<Computer>SharePointDev1</Computer>
<Security UserID=”S-1-5-21-452889701-636363473-2591022535-1012? />
</System>
<EventData>
<Data Name=”string0?>Microsoft.Office.Server.Search.Administration.CrawlReportJobDefinition</Data>
<Data Name=”string1?>9529aace-a679-4fc9-ab8d-325780484cf0</Data>
<Data Name=”string2?>The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component ’3147b99c-8f3a-41e9-a08b-296f930af877' in search application ‘Enterprise Search Service Application’ is in a good state and try again.</Data>
</EventData>
</Event>

And one for foundation search:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          …
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:
User:          …
Computer:      …
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.QueryTopologyActivationJobDefinition (ID de8eac2b-57db-4069-896d-747ae4fb35ed) threw an exception. More information is included below.
Topology activation was aborted because of System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String& sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage().
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6fb7e0ce-52e7-47dd-997a-241563931fc2}” />
<EventID>6398</EventID>
<Version>14</Version>
<Level>1</Level>
<Task>12</Task>
<Opcode>0</Opcode>
<Keywords>0×4000000000000000</Keywords>
<EventRecordID>10895</EventRecordID>
<Correlation ActivityID=”{6E239D20-A2CD-45B4-AC87-4477A82558BB}” />
<Execution ProcessID=”2016? ThreadID=”2288? />
<Channel>Application</Channel>
<Computer>id1314</Computer>
<Security UserID=”S-1-5-21-30024279817-590149927-1659320300-1003? />
</System>
<EventData>
<Data Name=”string0?>Microsoft.Office.Server.Search.Administration.QueryTopologyActivationJobDefinition</Data>
<Data Name=”string1?>de8eac2b-57db-4069-896d-747ae4fb35ed</Data>
<Data Name=”string2?>Topology activation was aborted because of System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String&amp; sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage().</Data>
</EventData>
</Event>

And finally from Gary’s blog (Marco van Wieren):

Component: 3b609311-67da-4df8-8c12-e597e9228dd3-crawl-0
Details:
The system cannot find the file specified. 0x80070002Propagation for search application 3b609311-67da-4df8-8c12-e597e9228dd3-crawl-0: failed to communicate with query server 3b609311-67da-4df8-8c12-e597e9228dd3-query-0.

Automatic Setup of Blank Sites across Farm


This is another PowerShell automation blog post 🙂

What

Setup a blank site (that is: “Sorry site is down for maintenance”) on all the front end servers and stop all existing web applications with a minimum of work. Likewise take it down easily.

By taking it down I mean deleting it so there is no risk that “someone” (the ghost in the server) might activate it by accident.

One script running on one server will install the blank site to multiple front-end servers.

Why

When you need to take down your sites for maintenance the right thing to do is to setup some temporary site – otherwise your visitors might easily see a semi-functioning site or a huge ugly exception when you’re in the middle of something.

This is not a big task to do manually, however it is tedious if you have many front-end servers and/or many web applications.

How

This is a bunch of powershell scripts that is started by two batch files – one for installing the blank site and one for uninstalling. The tasks performed during installation/setup of blank site are:

For each web server (defined as arguments to the PowerShell script within the batch file):

  1. Stop all running websites
  2. Setup a new web site (named “Blank Site” on port 80, path “c:\inetpub\wwwroot\blanksite” – customize yourself)
  3. Copy all the files (html, aspx, css, etc..) that goes into the blank site to the newly created directory

When you uninstall the blank site the same tasks are executed in reverse order. If there is more than one blank site on the server (i.e. you happened to run the _InstallBlankSite.bat file twice) they will all be deleted by the _UninstallBlankSite.bat script.

Notes

These scripts are used by me on my SharePoint farms; however it can be used on any type of (IIS) web server setup. I’ve tested it with IIS 6 but I believe it should work with IIS 5 and upwards.

You need to edit the two batch files with the correct servernames for your farm. You may add as many as you like (separate with spaces)

I recommend that you run the scripts on a server with PowerShell installed that is not an actual front-end server. I consider it fairly bad practice to install PowerShell on the front-end servers – I only install it on the backend CA/index server.

I use one of MS’ vbs scripts from IIS6 to do interact with the IIS. Another option is to use WMI, but only if you are either only accessing local server or using PowerShell 2.0. I’ve included that script in my package, so there is no requirement that the server executing the PowerShell scripts has an IIS server installed. Not sure if it’s actually legal, but I have a hard time seeing the harm done.

The uninstall script will start all stopped/paused websites regardless if they were actually running before you started the whole thing.

If you can’t get it to work at all, check your execution policy.

Download

The following scripts are included

_InstallBlankSite.bat: Main script to execute the dependent scripts. It will start powershell, and you need to add your server names here.

_UninstallBlankSite.bat: Main script to remove blank site. You need to add your servernames here.

InstallBlankSiteOnServers.ps1: Given a blank site and list of servers do the magic and call dependent scripts.

StopAllWebsites.ps1: Stop all running websites on a given server.

CreateBlankSite.ps1: Create a new website on server with the given local path (does not copy files).

UninstallBlankSiteOnServers.ps1: Remove a blank site from a list of servers. Assumes a local path to files.

StartAllWebSites.ps1: Start all stopped websites on a (remote) server.

DeleteBlankSite.ps1: Removes the blank site(s) from the IIS on the specified server (UninstallBlankSiteOnServers.ps1 deletes the files).

BlankSite: (Directory) the files that constitute your blank site. You’ll want to make it prettier but you get the point from what’s in there.

IISWeb.vbs: IIS script file for IIS6 from Microsoft that performs all IIS management tasks. It has not been altered in any way, it’s merely there for convenience. You’ll find it in system32 on your webservers.

The package can be found here

Fixing the Timer Service when everything breaks down


This is a pure troubleshooting post (that will save you days!) if you experience any of the following problems:

  • You have trouble deploying solutions to a your farm (more than one node) where nothing happens the deployment never completes
    • You may be able to complete deployments by executing “stsadm –o execadmsvcjobs” on every server in the farm
  • You experience frequent CPU spikes of 100% every few minutes on your frontend servers and they are all but unresponsive at those times
  • You find one of the following lines in the ULS logs (the SharePoint logs)
    • OWSTimer.exe, w3wp.exe (Id: 888k) : “File system cache monitor encountered error, flushing in memory cache”
    • OWSTimer.exe (Id: 8xqx): “Exception in RefreshCache. Exception message :Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.”
    • OWSTimer.exe (5utx): “The timer service could not initialize its configuration, please check the configuration database. Will retry later.”
  • “Some” of the administrative pages in the central administration occasionally fails with the error
    • “Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information”
  • The timer and topology cache is never updated (see next section…)
  • PSConfig (GUI wizard or not) fails to configure your server and the log file reveals the “LoaderExceptions” error above.

The above problems will probably affect all servers in the farm.

Note: The fusion log troubleshooting will be applicable to all .NET loader errors, not just SharePoint specific stuff.

First shot: Clear the Config Cache

A little known fact is that SharePoint maintains a disk cache (and memory) based configuration on every server that contains the topology information and timer job definitions. Go have a look at “C:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\<guid>\”.

Sometimes old stuff can get stuck in there and you can kick start the timer by clearing it.

The procedure is simple (do it for every server):

  1. Stop the Administration and Timer service
  2. Delete all XML files in the directory (not the folder itself)
  3. Open “Cache.ini” and write the number 1 instead of the existing number (you might want to make a note of it)
  4. Start the services again
  5. Wait for a minute or two and see if the folder starts to fill up with xml files. It is likely that it will contain less than before clearing it.
  6. Check the cache.ini file. If it’s accessible and the number is considerable greater than 1 your cache has been properly initialized and chances are that your problems are now fixed. It didn’t fix my problem, so you may need to read on… (if you didn’t have the “888k” log entry mentioned above you probably have it now)

The above procedure is grabbed from tomblog’s excellent post (that will also help if you actually did delete the folder too)

[Updated] Or you can run this batch file. [/Updated]

If the procedure didn’t fix the problem you’ll notice that the xml files are updated (timestamp) every few minutes coinciding with the CPU spikes.

Second shot: Digging in (Using the fusion log)

The core problem in my case was that some .NET class/assembly could not be loaded as the message “Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information” strongly hints. It may be a little surprising how many times an assembly, completely unrelated to the task at hand (e.g. deployment), is actually loaded. To fix the problem all we have to do is identify the assembly and “make it available”.

Enabling Fusion Log

You need to debug the failed assembly bindings you need to look into the fusion log. Fusion is an ancient (.Net 1.0) codename for the GAC or assembly loader (I think).

To enable the log you need to add/change three keys in the registry:

HKLM\Software\Microsoft\Fusion\LogPath    REG_SZ    (path to local directory)

HKLM\Software\Microsoft\Fusion\LogFailues    DWORD     1

HKLM\Software\Microsoft\Fusion\EnableLog    DWORD    1

It is not strictly necessary to restart anything, but I recommend that you now restart your timer service in order for it to log any binding errors that occurs. Try starting it a couple of times with 10 minutes in between. That should reveal any binding error patterns.

(Refer to Brad Adams for more info)

Interpreting the Log

Finally we will use the “fuslogvw.exe” program that is part of the .NET SDK to view the actual logs. Your development machine will have this file; copy the executable to your server.

It is not a very good program. It gets the job done, but it’s hard to figure out, you can’t order errors by date you can’t resize the window, etc..

Hopefully your window will contain a lot less entries (in my case the offending entry is the highlighted one. It also failed with similar errors from the psconfig wizard and powershell).

Look for patterns using the timestamps. Did you get a group of binding errors a few minutes after you started the timer service? Or are there just some errors that look spurious? I’ll recommend skipping the core internal microsoft dll’s to begin with (msvcm80 is the c runtime library version 8 that .Net 2.0 uses).

So what goes into the log? Every assembly binding failure, which is (at least):

  • If an assembly cannot be located (it will show you where it searched and any assembly redirects)
  • If the dll load methods within the assembly throws an exception – e.g. a dependent assemly could not be found – it sadly looks exactly like the dll file could not be found (other exceptions are possible but unlikely for .NET people). The one way to distinguish is to look for the line “LOG: GAC Lookup was unsuccessful.” if it is not there then it was (probably) found in the GAC and a dependent assembly failed.

The last bullet means that if assembly A depends on B and B could not be found then both the binding for A and B fails (at the same time). To distinguish the two I’ll recommend that you look at the “Calling Assembly” in the bind log.

In my case:

  1. The Nintex.Workflow dll failed to load and the calling assembly was Microsoft.SharePoint
  2. The Nintex.Workflow.Charting.dll failed to load and the calling assembly was Nintex.Workflow. Aha, so the Nintex.Workflow.dll was actually found but failed to find the dependent Nintex.Workflow.Charting assembly.
  3. Found the assembly on a development machine and copied it to the server. Retried. Added another missing assembly
  4. And everything worked!

I should stress that neither the error type nor the troubleshooting is Nintex workflow specific.

The bind log failure from step 2 above was:

*** Assembly Binder Log Entry (1/13/2009 @ 4:34:15 PM) ***

The operation failed.

Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll

Running under executable C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN\STSADM.EXE

— A detailed error log follows.

=== Pre-bind state information ===

LOG: User = ….

LOG: DisplayName = Nintex.Charting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=913f6bae0ca5ae12

(Fully-specified)

LOG: Appbase = file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/

LOG: Initial PrivatePath = NULL

LOG: Dynamic Base = NULL

LOG: Cache Base = NULL

LOG: AppName = NULL

Calling assembly : Nintex.Workflow, Version=1.0.0.0, Culture=neutral, PublicKeyToken=913f6bae0ca5ae12.

===

LOG: This bind starts in default load context.

LOG: No application configuration file found.

LOG: Using machine configuration file from C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\config\machine.config.

LOG: Post-policy reference: Nintex.Charting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=913f6bae0ca5ae12

LOG: GAC Lookup was unsuccessful.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting.DLL.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting/Nintex.Charting.DLL.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting.EXE.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting/Nintex.Charting.EXE.

LOG: All probing URLs attempted and failed.

Note: To grab a dll from the GAC on one machine you need to use a shell (cmd, bash or powershell) to go into the c:\windows\assembly\gac_msil\… folder structure and copy it.

Why did it go wrong in the First Place?

It’s always a useful exercise to figure out what went wrong and why instead of just fixing the error at hand.

The root cause of the error was that the dll’s in questions were distributed through a wsp solution file which was referenced in other code. When that particular solution was undeployed the dll’s were removed (from the GAC) and the new updated wsp file only contained the new versions. Suddenly something went missing 😦

It can also happen if you use some auto wsp packaging procedures like vbs scripts or the ubiquitous WSPBuilder. What if one of the SharePoint system dll’s are marked as “copy local” on one (of the others’) local develop environment and automatically included in the wsp file? It will be deployed, no problems, but when undeployed everything stops as the SharePoint system dll just got removed from the GAC. Oops. Prevention is obviously to use a build server to make clean builds, to educate your people and to have the means to troubleshoot when it didn’t work out.

Automatic Configuration of Asp.NET Ajax Extension 1.0


I’m currently testing the world famous RadEditor for MOSS from Telerik and come across a rather annoying deployment issue. Don’t get me wrong the RadEditor is a fantastic product; however it relies on the Asp.NET Ajax Extension 1.0 from Microsoft that really is a royal pain to install and configure.

Installation of the Ajax extension is fairly straight forward, just run through the installer.

Configuring a given website (here it’s a SharePoint site with a lengthy web.config file to begin with) is a rather cumbersome task, described in detail at Mike Ammerlaan’s nice blog post. In total there are about 20 additional tags that need to be inserted into your web.config file in just the right places.

It’s a tedious and error prone process that I’m unwilling to go through with 10+ websites replicated in 2 different environments. Googling for an hour revealed no real solution; it seems that no one has properly automated the process.

I’ve used a small tool ConfigMerge with bit of powershell to do the magic. I can now just run a script that will deploy the configuration settings to all my web.config files at my SharePoint sites (have to run it on every server) 🙂

Read on…

Web.Config Modifications

I’ve gathered the required modifications in a xml file that resembles the structure of a “normal” web.config file, which are then merged into the real web.config files. The file is “ajax35.config” in the downloadable zip file.

Please note that the following applies to Asp.NET Ajax Extensions 1.0 for a .NET 3.5 web sites, slight variations are likely for 2.0 and 3.0 sites.

Auto Configuration

[Updated 23-12-2008]

The key component in the auto configuration is the ConfigMerge program found on CodeProject that will merge ajax config file with the existing web.config file. If the nodes are already there they will be updated, not duplicated.

ConfigMerge uses a small list of attributes to identify similar nodes which I’ve had to extend just a bit for our needs (look for my comment on the CodeProject page near the bottom). The point is that it is now idempotent – if you run it more than once, no changes are performed.
I’ve included both the new binary and the modified source in the zip package.

The usage of the ConfigMerge utility is:

ConfigMerge.exe ExistingConfigFile ChangesConfigFile OutputFile

However I prefer to use powershell to call it, and perform (necessary) backup of web.config.

Note: If you don’t like powershell stop reading now and just use the above line in your own bat files and you’ll be good.

Powershelling

To get started with powershell read my old post, though this time I’m actually not accessing the SharePoint API.

Usage for the EnableAjax.ps1 script is (from within powershell):

EnableAjax.ps1 WebConfigPath [AjaxConfigFile]

Alternatively from a batch file/cmd use:

Powershell –command EnableAjax.ps1 WebConfigPath [AjaxConfigFile]

WebConfigPath is the path where the script should search for web.config files. It’ll go recursively down from the specified directory. If you only want to work on a single web application then use the root dir for that web application, if you want to run on all SharePoint applications you can use something like “c:\inetpub\wwwroot\wss\virtualdirectories”.

AjaxConfigFile is the name of the config file you want to merge in. It defaults to “ajax35.config” in the same dir as EnableAjax.ps1.

When the script is about to fix a web.config file it performs a backup first named with the current time, e.g. “web_ajax_2008_12_21 21_06_09.backup”.

Finally here is the actual script (EnableAjax.ps1):

# Apply Asp.NET Ajax Extensions to a number of web.config files easily.
#
# 21/12-2008 Søren L. Nielsen (soerennielsen.wordpress.com)
#
$ajaxconfig = "./ajax35.config"
$path = "."
if( $args.length -eq 2 ){
    $path = $args[0]
    $ajaxconfig = $args[1]
}
elseif ( $args.length -eq 1 ){
    $path = $args[0]
}
else {
    Write "Usage: ./EnableAjax.ps1 [WebConfigPath [AjaxConfigFile]]"
    Write ""
    Write "WebConfigPath - Path where the script should look for web.config files"
    Write "                Script will search for web.config files recursively and "
    write "                modify all it finds."
    Write ""
    Write ("AjaxConfigFile - Default is " + $ajaxconfig + " set it to a xml config")
    Write "                 file with the same structure as a 'normal' web.config file"
    Write "                 but only containing the nodes that should be added/updated"
    Write "                 in the web config files."
    Write "                 The script is build for Asp.Net Ajax extensions it will however"
    Write "                 be useful in many settings, e.g. manipulating dev, QA, Prod "
    Write "                 environment settings."
    Exit
}
write ("Modifying web.configs found in " + $args[0])
foreach( $f in Get-ChildItem $args[0] -filter web.config -recurse ){
    $realname = $f.FullName
    $backup = $f.FullName.ToLower().Replace(".config", "_ajax_" +
[DateTime]::Now.ToString("yyyy_MM_dd HH_mm_ss") + ".backup"
    Write ("Updating " + $f.FullName )
    Write ("Backup file " + $backup )
    $f.MoveTo( $backup )
    ./ConfigMerge.exe $backup $ajaxconfig $realname
}

As the web.config is fairly critical for your application you should be thorough when you test this script, I’ll recommend using WinMerge when comparing before and after config files.

Or you could just trust me an know that I take absolutely no responsibilty for any harm that comes your way 😉

[Updated 23-12-2009] Note: You need to run the script for every server, however if you use UNC paths as input to the script you can run it for every server from just one. I’ll recommend that you only install powershell on your backend servers not your frontends (if you’re on Server 2003).

Note 2: You need to rerun the script if you provision new web applications, or if you add a new server to the farm (on that server). The web.config modifications performed by SharePoint should not interfere with the script.

Download (Source, Scripts and Binaries)

Auto Asp Net Ajax Config

Final Notes

Please note that the files provided here only configures Asp.NET Ajax Extentions 1.0; it does install or configure the telerik RadEditor. It requires two more keys documented in the install guide (and a number of other small steps), that you can easily add through the SharePoint API (WebConfigModifications).

I should stress again that the version of ConfigMerge used here has been modified a bit. If you use the unmodified version you risk that some of the existing tags are “reused”/changed for the Ajax configuration and then SharePoint might well be in trouble.

”Not enough storage” event log error


[Note: Updated Feb. 22 2008, solution at the bottom] 

I’m responsible for a couple of SharePoint 2007 (MOSS) farms where all SharePoint servers showed a number of annoying errors in the application event log.

Every minute the following three errors show up in the event log:


Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: Timer
Event ID: 6398
Date: 3/6/2007
Time: 11:47:58 AM
User: N/A
Computer:
Description:
The Execute method of job definition Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob (ID 371548ff-a05e-41f0-90da-6f2d25fbb483) threw an exception. More information is included below.


Not enough storage is available to process this command.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

And


Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 7076
Date: 3/6/2007
Time: 11:47:58 AM
User: N/A
Computer:
Description:
An exception occurred while executing the Application Server Administration job.


Message: Not enough storage is available to process this command.
Techinal Support Details:
System.Runtime.InteropServices.COMException (0x80070008): Not enough storage is available to process this command.


Server stack trace:
at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
at System.DirectoryServices.DirectoryEntry.Bind()
at System.DirectoryServices.DirectoryEntry.get_IsContainer()
at System.DirectoryServices.DirectoryEntries.CheckIsContainer()
at System.DirectoryServices.DirectoryEntries.Find(String name, String schemaClassName)
at Microsoft.SharePoint.AdministrationOperation.Metabase.MetabaseObjectCollection`1.Find(String name)
at Microsoft.SharePoint.AdministrationOperation.Metabase.MetabaseObjectCollection`1.get_Item(String name)
at Microsoft.SharePoint.AdministrationOperation.SPProvisioningAssistant.ProvisionIisApplicationPool(String name, ApplicationPoolIdentityType identityType, String userName, SecureString password, TimeSpan idleTimeout, TimeSpan periodicRestartTime)
at Microsoft.SharePoint.AdministrationOperation.SPAdministrationOperation.DoProvisionIisApplicationPool(String name, Int32 identityType, String userName, String password, TimeSpan idleTimeout, TimeSpan periodicRestartTime)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)
[…continues…]

And

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 7076
Date: 3/6/2007
Time: 11:47:58 AM
User: N/A
Computer:
Description:
An exception occurred while executing the Application Server Administration job.


Message: Not enough storage is available to process this command.


Techinal Support Details:
System.Runtime.InteropServices.COMException (0x80070008): Not enough storage is available to process this command.


Server stack trace:
[…continues…]

Sometimes they are replaced with another three that has the same event id and source, but with the error message being “Old format or invalid type library” instead. The stack trace will differ a little bit.

Obviously it’s a timer job scheduled every minute that fails. The one in question is “Application Server Administration Service Timer Job”, which apparently is in charge of ensuring that the IIS application pools are in sync (or something like that). Nothing seems to be broken in the farm by the job failures.

And quite annoyingly: They all disappear when you reboot the server and will not reappear until after “a few days” in my case. I’m sure that the re-surface time will differ between systems.

I’ve read many proposed solutions for this error, including adding ram, disc etc., often claimed to work. I seriously doubt that any of these solutions actually work. My SharePoint servers are equipped with no less than 6gb of ram with default settings for all the application pools (I know that the ram is hardly utilized with these settings, but in my world it’s sometimes cheaper to go for a “standard server” where you only utilize 70% than one customized for your needs), plenty of disc etc.

As a side note you can also find references to this error in connection with SQL server 2005, the fix below possibly also work for that as well.

The solution turns out to be quite easy – the patch you are looking for is kb923028. It is an update for an error in the .NET 2.0 remoting subsystem, and has actually nothing to do with SharePoint at all. Reading the description it is quite hard to glean that it’ll solve your problem. MS support pointed me to it and it seems to work just fine.

Caveat: I’ve seen multiple versions of this file, the one I have working (until proven wrong) is “NDP20-KB923028-X86.exe” (1,936,224 bytes). I’ve tested another one with the same name with a filesize of 1,963,440 bytes that didn’t work.

The server has been chugging along for some time now (about a week) without the bug so let’s hope it’s all done.

That didn’t work…

Update: Finally a resolution

You need to look at hotfix KB946517, which will fix the problem. It is a private hotfix, so you’ll need to contact MS acquire it.

The servers have been running for about a week now, they are still ok and other people are also reporting success.

For the third time I’m confident that the problem has been solved – guess I don’t learn from experience 😉