Find/Remove Obsolete Resource Labels


This is part 3 of 4 in a series on how to improve the way we usually work with resource (resx) files. At least the way my team and I work with them.

I generally like to – and do – use resource files for all string constants that are shown to end-users, however I do feel that it is needlessly cumbersome, therefore these posts:

  1. A Good Way to Handle Multi Language Resource Files
  2. Verify that your Resource Labels Exists
  3. Find Obsolete Resource Labels (this one)
  4. Supercharge your (Resource) Efficiency with Macros

Generally these issues are generic to .NET and not specific to SharePoint, though that is where I’m spending my time writing this.

So – Near the End of the Project – are those Resource Entries Still in Use?

The issue at hand is that you have hundreds of source files of various flavors and they are sprinkled with references to a number of resource files. When code is refactored or just deleted what happens to old resource labels? Likely nothing at all.

Are you happy with a ton of useless resource entries no longer in active use? What if you had to translate it to a couple of languages?

Quite obviously this is no biggie code/quality wise, but still…

The answer is that you run the PowerShell script below, that’ll check – and optionally fix – it 😉

The Script

I made a small script to check and remove the excess resource entries to slim down those resource files a bit.

The script will, given a starting location (i.e. the root folder for your solution)

  1. Go through every code file (to be safe every file, except a list of binary extensions) and look for resource labels of the form “$Resources:filename,label_key
  2. Search recursively for resx files
  3. For every one of those resx files it will look through the resource labels in use and flag those that it cannot find
  4. (Optionally) Do a “safemode” check where every file is searched for the resource label, i.e. necessary if you are using multiple/other resource lookup methods then the $Resource moniker
  5. (Optionally) If you choose you may remove them automatically but do make a dry run first to sanity check that you got the paths right and that you have all the source files to be searched

Usage:

    PS> & VerifyResxLabels.ps1 “path to solution dir” [-remove] [-safemode]

(Tip: Download the script to somewhere, write an ampersand (&) and then drag the ps1 file into the PowerShell window and then drag the solution folder to the window.)

You’ll definitely want to pipe the output to a file.

Limitations

There are obviously some limitations

  • It will not: Check out the resx files from source control (but it will show the file error in the output)
  • It will not: Respect commented out code – it’s simple pattern matching so commented out code will be treated as actual code (hardly an issue)
  • Safemode is very slow of necessity and will likely find false positives, i.e. it will play it safe and keep entries that exists in some files, even though the same label may be used in a completely different context
    • It’s an (O(n*m) algorithm, with number of files and number of labels) – My test with 1000 unique labels, 28 resx files and 2400 files it takes a night

Download the script here

Advertisements

SharePoint Advanced Large Scale Deployment Scripting – “Dev and QA gold-plating” (part 3 of 3)


I have been battling deployments for a while and finally decided to do something about it.

😉

The challenge is to handle a dozen WSP packages on farms that host 20-50 web applications (or host header named site collections) to complete the deployments in a timely manner, ensure that the required features are uniformly activated every time while minimizing the human error element of the poor guy deploying through the night.

The deployment scripts require PowerShell v2 and are equally applicable to both SharePoint 2007 and 2010 – the need is roughly the same and the APIs are also largely unchanged. Some minor differences in service names are covered by the code.

To keep this post reasonable in length I’ve split the subject into three parts:

Part 1 is the main (powershell) deployment scripts primarily targeted at your test/QA/production deployments

Part 2 is the scripts and configuration for automatic large scale feature activations

Part 3 (this part) is about gold-plating the dev/test/QA deployment scenario where convenience to developers is prioritized while maintaining strict change control and records

Note: This took a long time to write and it will likely take you, dear reader, at least a few hours to implement in your environment.

If you have not done so already read the two other parts first.

The Challenge

When you have a number of developers working on the same service line you need to be able to keep strict control of your QA and test environment.
So either you have one guy as the gatekeeper or you automate as much as possible. I’m the last type of guy so I’ve developed a “one-click” deployment methodology where the focus is
1. Simple and consistent deployment
2. Easy collection of solutions for next production deployment – you should deploy the exact code that you tested not a “fresh” build(!)

The Solution

Our build server is set up to produce date labelled zip files with the wsp and manual installer (exe) within folder of the same name.
I’m sure yours are different therefore the scripts work with a solution drop directory that accepts both zip files (that includes a wsp file) and also plain wsp files.
The folder structure is up to you I’ll just search recursively for zip and wsp.

The script files are:

03 DeployAndStoreDropDir.bat: The starting batch file that will execute the deployment process as a specific install user. That way the developers can login with their personalized service accounts and use a specialized one for deployment purposes. You need to change the user name in this batch file. The first time you run this you need to enter the password for your install account subsequent runs will remember it  (“runas /savecred”).

03b DeployAndStoreDropDirAux.bat: The batch file doing half the work. It’ll archive old log files in “.\ArchivedDeploymentLogFiles” (created if not present) and it execute “QADeploymentProcess.ps1” that is doing the hard work. At the end it’ll look for error log and write an alert that you should look for it. Saves a lot of time – I don’t go looking in log files if I don’t get that message.

SharePointLib\QADeploymentProcess.ps1: The script that is handling the unzipping and storage of the zip/wsp files and executing both the deployment and feature activation scripts. The goal is to have a simple drop dir where for WSP/zips and a single storage location, “SolutionsDeployed”, for the currently deployed code. The “SolutionsDeployed” folder will be updated at every deployment, so that there is only the latest version of each wsp deployed. You do not need to drop all wsps in the drop dir every time; it does not just delete the SolutionsDeployed folder at every deployment.

This is how the files are moved around for a couple of WSPs and a zip file:

In short:

  1. If you want to start it with a specific install user, just execute “03 DeployAndStoreDropDir.bat”
  2. If you want to start it in your own name, execute “03b DeployAndStoreDropDirAux.bat” and perhaps delete the other bat file.
  3. If you want to modify how it works go to the “QADeploymentProcess.ps1” file.
  4. If you want to work with zip files they can have the name and internal folder structure that you like. The one important thing is that they should only contain one WSP file. It is pretty smart; at the next deployment your zip file may be named differently (e.g. with a date/time in the name) and it will still remove the old version from “SolutionsDeployed” and store your new version with the new folder/zip name.
  5. At the end of your build iteration SolutionsDeployed will contain the latest version of your deployed code (provided that you emptied it at the last production deployment)

Closing Notes

It is not a big challenge to connect the remaining dots and have your build server do the deployment at certain build types and have a “zero-click” deployment, however I opted to not do it. It should be a conscious decision to deploy to your test and QA environments, not something equivalent to a daily checkin. Your milage may wary.

I would generally provide these scripts to everyone in the team and encourage them to use them on their own dev box once in a while – it always pays to have your dev environment resemble the actual production environment this makes it feasible to keep the code (almost) in sync on all dev environments.

The challenge may be the complexity of the scripts and the effort to understand them – you shouldn’t rely too much on stuff you don’t understand 😉

Feel free to adjust the script a bit (likely QADeploymentProcess.ps1) to suit your environment.

If you make something brilliant then let me know so I can include it in my “official” version. Please also leave credits in there so it’ll be possible for your successors to find the source and documentation.

Download

Grab the files here – part 1, 2 and 3 are all in there. I’ll update them a bit if I find any errors or make improvements.

Note: Updated Aug 29 2011, minor stuff.

Don’t forget to change the user name in “03 DeployAndStoreDropDir.bat” to your “install” account.

SharePoint Advanced Large Scale Deployment Scripting – “Features” (part 2 of 3)


I have been battling deployments for a while and finally decided to do something about it.

😉

The challenge is to handle a dozen WSP packages on farms that host 20-50 web applications (or host header named site collections) to complete the deployments in a timely manner, ensure that the required features are uniformly activated every time while minimizing the human error element of the poor guy deploying through the night.

The deployment scripts require PowerShell v2 and are equally applicable to both SharePoint 2007 and 2010 – the need is roughly the same and the APIs are also largely unchanged. Some minor differences in service names are covered by the code.

To keep this post reasonable in length I’ve split the subject into three parts:

Part 1 is the main (powershell) deployment scripts primarily targeted at your test/QA/production deployments

Part 2 is the scripts and configuration for automatic large scale feature activations

Part 3 is about gold-plating the dev/test/QA deployment scenario where convenience to developers is prioritized while maintaining strict change control and records

Note: This took a long time to write and it will likely take you, dear reader, at least a few hours to implement in your environment.

If you have not done so already read “Part 1” first and excuse me for a bit of duplicated text.

It’s all about the Features

This part is about features and managing their (de-)activations.

What you can do with this is:

  • Maintain a set of templates (“FeatureSets”) that defines what features should be activated on a given type of site – the scripts then ensures that they are properly activated
  • Set actions for features to be either activate, deactivate or reactivate through the configuration file
    • It safely handles web.config updates so that the scripts do not enter a race condition with the SharePoint timer service
  • Selectively override the default behavior and reactivate certain features when you need to

In other words you can ensure that your features are properly activated across sites and farms.

A note on reactivation is in order here. Reactivation is (force) deactivation and followed by activation which is particular useful for features that updates files in document library, e.g. masterpages. Some features require it when they have been updated but it is very rare that a feature always require reactivation.

Therefore I usually do not specify any features to always be reactivated in the configuration file, however if I know that we updated the masterpages I’ll update the batch file, or supply an additional one, where that feature is forced reactivated. The safe option of always reactivating is simply too slow if many files are deployed (or lengthy feature receivers) are involved.

The Configuration

The scripts work based on a configuration file that is shared between your dev, test, QA and production environments.

The procedure is:

  1. Identify all valid Sites/URLs in the local farm (i.e. always executed on one of the servers in the farm)
  2. For each site/URL go through all Feature sets and ensure activations, deactivations and reactivations required
    1. Keep track of reactivations so same feature is only reactivated once at each scope, i.e. overlap between FeatureSets are allowed as long as they specify the same action
    2. Report inconsistencies as errors, but do continue with the next features so that you can have the complete overview of all errors at the end of the run

A sample configuration file is (note: I just picked some more or less random WSPs from codeplex):

<?xml version="1.0" ?>
  <Config>
    <Sites>
      <!-- note: urls for all environments can be added, the ones found in local farm will be utilized -->
      <!-- DEV -->
      <Site url="http://localhost/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      <Site url="http://localhost:8080/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="MySite" />
      </Site>
      <Site url="http://localhost:2010">
        <FeatureSet name="CA" />
      </Site>
      <!-- TEST -->
      <Site url="http://test.intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
      <!-- PROD -->
      <Site url="http://intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
    </Sites>

    <FeatureSets>
      <FeatureSet name="BrandingAndCommon">
        <Solution name="AccessCheckerWebPart.wsp" />
        <Feature nameOrId="AccessCheckerSiteSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="AccessCheckerWebPart" action="activate" ignoreerror="false" />
        <Solution name="Dicks.Reminders.wsp" />
        <Feature nameOrId="DicksRemindersSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="DicksRemindersTimerJob" action="activate" ignoreerror="false" />
        <!-- Todo - all sorts of masterpages, css, js, etc. -->
      </FeatureSet>
      <FeatureSet name="Intranet">
        <Solution name="ContentTypeHierarchy.wsp" />
        <Feature nameOrId="ContentTypeHierarchy" action="activate" ignoreerror="false" />
      </FeatureSet>
      <FeatureSet name="MySite">
      </FeatureSet>
      <FeatureSet name="CA">
      </FeatureSet>
    </FeatureSets>

    <Solutions>
      <!-- list of all solutions and whatever params are required for their deployment -->
      <Solution name="AccessCheckerWebPart.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="ContentTypeHierarchy.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="Dicks.Reminders.wsp" allcontenturls="true" centraladmin="false" upgrade="false" resettimerservice="true" />
    </Solutions>
  </Config>

Where the interesting bits are that Sites contain a number of named FeatureSets. The FeatureSets are gathered in their own section (for reusability) and defines a number of solutions to be deployed and the features to be handled. It is good practice to list the solutions for the features that you are working on so you can be sure that the solution deployment matches your features. Do not spend time on noting all the SharePoint system features to be activated as they are normally handled just fine by the Site Definitions.

Each feature activation line looks like:

<Feature nameOrId=”AccessCheckerWebPart” action=”activate” ignoreerror=”false” />

Where nameOrId is either the guid (with or without braces, dashes etc.) or the name of the feature, i.e. the internal name in the WSP packages, not the display name. When possible I always opt for flexibility 😉

The action attribute is either “activate”, “deactivate” or “reactivate”.

The ignoreerror is simply a boolean true/false switch that determines how the error should be logged if the feature fails to be activated. It is quite usable for the occasional inconsistencies between environments, e.g. if you are removing a feature then it might be present in production but no longer in the test environment. The script continues with next feature in case of errors regardless of this switch.

It is important to note that web scoped features are not supported here, “only” farm, web app or site scope is allowed.

The features are activated in the order that they are present in the config file.

The Scripts

There are two sections here:

  1. A number of SharePoint PowerShell scripts that include each other as needed
  2. A few batch files that start it all. I generally prefer to have some (parameter less) batch files that executes the PowerShell scripts with appropriate parameters

I have gold-plated the scripts quite a bit and created a small script library for my SharePoint related PowerShell.

The PowerShell scripts are too long to write in the post, they can be downloaded here. The list is (\SharePointLib):

*(Part 1) DeploySharePointSoluitions.ps1: Start the WSP deployments. Parse command line arguments of config file name and directory of WSP solutions

*EnsureFeatureActivation.ps1: Start the feature activation scripts (need command line arguments)

*(Part 3) QADeploymentProcess.ps1: Deployment process for dev, test and QA environments (need command line arguments)

SharePointDeploymentScripts.ps1: Main script that defines a large number of methods for deployment and activations.

SharePointHelper.ps1: Fairly simple SharePoint helper methods to get farm information

*RestartSharePointServices.ps1: Restart all SharePoint services on all servers in the local farm

Services.ps1: Non-SharePoint specific methods for handling services, e.g. restarts

Logging.ps1: Generic logging method. Note that there are options for setting different log levels, output files, etc.

The ones with an asterisk are the ones that you are supposed to execute – the others just defined methods for the “asterisk scripts”.

The batch files (same download) (root of zip download):

(Part 1) 01 Deploy Solutions.bat: Start the solution deployment. It will deploy all the WSP files dropped in “\SolutionsDropDir\” that are also specified in the “\DeploymentConfig.xml”:

02 Activate Features.bat: Works with the “\DeploymentConfig.xml” file and ensures that all features are activated.

In other words you’ll execute “02 Activate Features.bat” to ensure that all features are activated in your farm.

The important line in 02 Activate Features.bat file is

powershell.exe -File “SharePointLib\EnsureFeatureActivation.ps1” “%config%” “” >>ActivateFeatures.log

And if you need to reactivate your branding features (masterpages) you can just copy the batch file and change the line to something like:

powershell.exe -File “SharePointLib\EnsureFeatureActivation.ps1” “%config%” “BrandingMasterPages,BrandingCSS,BrandingPageLayouts” >>ReactivateFeatures.log

to force the three (made up) features “BrandingMasterPages”, “BrandingCSS” and “BrandingPageLayouts” to be reactivated.

By default you will get one log file from each batch file named after the operation, one log file with a timestamp and in case of any errors an additional error log with the same timestamp. In other words if the error log file is not created then no errors occurred.

Runtimes

The scripts are quite fast and the time to execute is determined by the time it takes to activate the features within SharePoint. Therefore if the script has nothing to do, i.e. all features are activated as they should, then it’s normally less than 1 minute to check everything and return.

What usually takes time is features with many files to be added to document libraries (branding) and features that modify the web.config as we wait for the timer job to complete (assuming that the API is used to do this).

Closing Notes

The scripts write quite a bit of logging that can be daunting (especially if you set it to verbose) but don’t worry – I rarely look at them anymore. Instead I look for the EnsureFeatureActivation_date_error_log. If it’s not there then no error occurred and no reason to look into the log files.

These scripts have been a huge benefit for our deployment in order to reduce the post-deployment errors and fixes. It is however also the first to be blamed for errors whenever some feature does not behave as intended (“the unghosted file is not change that must be the fault of the deployment scripts”) and it rarely is the source of the error.

They can be blamed for being a bit complicated though and that is a fair point. If you don’t understand what they do and how you shouldn’t use them.

If you make some brilliant changes then let me know so I can include it in my “official” version.

Please also leave credits in there so it’ll be possible for your successors to find the source and documentation.

Download

Grab the scripts here.

Note: Updated Aug 29 2011.

SharePoint Advanced Large Scale Deployment Scripting – “Deploy” (part 1 of 3)


I have been battling deployments for a while and finally decided to do something about it.

🙂

The challenge is to handle a dozen WSP packages on farms that host 20-50 web applications (or host header named site collections) to complete the deployments in a timely manner, ensure that the required features are uniformly activated every time while minimizing the human error element of the poor guy deploying through the night.

The deployment scripts are equally applicable to both SharePoint 2007 and 2010 – the need is roughly the same and the APIs are largely unchanged. Some minor differences in service names are covered by the code.

To keep this post reasonable in length I’ve split the subject into three parts:

Part 1 is the main (powershell) deployment scripts primarily targeted at your test/QA/production deployments

Part 2 is the scripts and t for automatic large scale feature activations

Part 3 is about gold-plating the dev/test deployment scenario where convenience to developers is prioritized while maintaining strict change control and records

Note: This took a long time and countless iterations to develop and it will likely take you, dear reader, at least a few hours to implement in your environment.

The Requirements

My design goals were to solve

  • Automatic solution deployments (including timer jobs resets)
  • Consistent activation of features across the farm while allowing for different types of sites
  • Easy, fast and reliable execution
  • Not too awkward configuration of solutions and features for farms with many sites – it will never be a no-brainer
  • Exact same configuration (file) to be re-used across Dev, Test, QA and Prod tiers

The cost is some fairly complex scripts that took quite a while to write and test.

The same script works equally well for both 2007 and 2010.

There is almost no difference between SharePoint 2007 and 2010 in this area – 2010 comes with some new CmdLets for PowerShell but they are often just convenient access to the API. I’m taking the extra effort to hit the API directly which is essentially unchanged for deployment purposes. One of the really cool new features in 2010 is the ability to control solution upgrade in detail which is of course supported by these scripts simply by choosing the upgrade option for a given solution.

The Solution

The idea is that you have the same configuration file for all your app tiers, and

  1. When you need to deploy a new version of some of your WSPs you just provide them and execute the scripts
  2. Script will scan the local farm and map the URLs with the provided WSPs and deploy them
  3. Run through every identified Site and match feature activations with the configuration file and fix any discrepancies

The features are

  • Deploy WSP files regardless of whether or not they were deployed beforehand (uninstall followed by install is the default installation mode for me)
    • If solution deployment goes wrong the behavior is to restart all SharePoint related services on all servers in farm and retry (a few times). It surprisingly often solves problems like locked files etc.
    • If a solution contains timer jobs the timer service is restarted on all servers in farm. Finally
  • Upgrade existing WSP files
  • (Part 2) Scan the farm and ensure that required features are activated (or perhaps deactivated) on specified sites
    • Farm and WebApp scoped features are by default deactivated after an uninstall/install – this script reactivates them
    • Features are always (de)activated with the “force” parameter just for good measure
    • Yes, it’ll wait for web config modifications jobs when required
  • (Part 2) Re-activate selected features i.e. deactivate first then activate
  • (Part 3) Automatic deployment and activations for dev, test and QA environments

The Configuration File

The hearth of the scripts is the configuration file.

The one configuration file can be re-used for both handling solution deployment and feature activations across all tiers of your farms. You can also conceivable use the same config file for different types of farms, e.g. the intranet and extranet, however I feel that in those scenarios it is better to keep those things separate.

The configuration file contains:

  • A number of “Sites” which is the URLs of either (host header named) site collections or web applications
    • The local farm will be scanned and the sites found in the config file that are present in the local farm is the ones that we work on – the rest are ignored (e.g. running in test it’ll skip all the production URLs)
    • Each site node contains a number of FeatureSets but no list of features or solutions
  • A number of “FeatureSets” that groups WSPs and features. Features are either “activated”, “deactivated” or “reactivated”
    • (The feature thingie is in part 2)
    • For a given site the union of all the solutions specified in all the FeatureSets associated with the sites defines what solutions should be deployed to that site.
  • A number of “Solutions” that list the WSP names and their individual options, i.e. deploy to central admin, all content URLs, upgrade or (re)install, does it contain timer jobs (which requires timer service to be restarted on all farms)
    • Note: The scripts deploy all the solutions in a given “drop” directory and ignore all the configuration nodes without a matching WSP file

To demo it I’ve downloaded a couple of WSPs from CodePlex and created the following configuration file for my dev box:

<?xml version="1.0" ?>
  <Config>
    <Sites>
      <!-- note: urls for all environments can be added, the ones found in local farm will be utilized -->
      <!-- DEV -->
      <Site url="http://localhost/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      <Site url="http://localhost:8080/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="MySite" />
      </Site>
      <Site url="http://localhost:2010">
        <FeatureSet name="CA" />
      </Site>
      <!-- TEST -->
      <Site url="http://test.intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
      <!-- PROD -->
      <Site url="http://intranet/">
        <FeatureSet name="BrandingAndCommon" />
        <FeatureSet name="Intranet" />
      </Site>
      ...
    </Sites>

    <FeatureSets>
      <FeatureSet name="BrandingAndCommon">
        <Solution name="AccessCheckerWebPart.wsp" />
        <Feature nameOrId="AccessCheckerSiteSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="AccessCheckerWebPart" action="activate" ignoreerror="false" />
        <Solution name="Dicks.Reminders.wsp" />
        <Feature nameOrId="DicksRemindersSettings" action="activate" ignoreerror="false" />
        <Feature nameOrId="DicksRemindersTimerJob" action="activate" ignoreerror="false" />
        <!-- Todo - all sorts of masterpages, css, js, etc. -->
      </FeatureSet>
      <FeatureSet name="Intranet">
        <Solution name="ContentTypeHierarchy.wsp" />
        <Feature nameOrId="ContentTypeHierarchy" action="activate" ignoreerror="false" />
      </FeatureSet>
      <FeatureSet name="MySite">
      </FeatureSet>
      <FeatureSet name="CA">
      </FeatureSet>
    </FeatureSets>

    <Solutions>
      <!-- list of all solutions and whatever params are required for their deployment -->
      <Solution name="AccessCheckerWebPart.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="ContentTypeHierarchy.wsp" allcontenturls="false" centraladmin="false" upgrade="false" resettimerservice="false" />
      <Solution name="Dicks.Reminders.wsp" allcontenturls="true" centraladmin="false" upgrade="false" resettimerservice="true" />
    </Solutions>
  </Config>

Note: I more or less picked some random WSPs from CodePlex for demo purposes.

Note 2: The scripts will respect the order given in the config file. The Sites are processed in order, the FeatureSets listed first within a “Site” node is handled in top-down order. The solutions deployed to the farm are deployed in the order they are listed in the “Solutions” node.

Note 3: XML is case sensitive; therefore, all the named sections are case sensitive. The one exception is that I accept different casing between the WSP file names and the name they are given within the config file.

The Scripts

There are two sections here:

  1. A number of SharePoint PowerShell scripts that include each other as needed
  2. A few batch files that start it all. I generally prefer to have some (parameter less) batch files that executes the PowerShell scripts with appropriate parameters

I have gold-plated the scripts quite a bit and created a small script library for my SharePoint related PowerShell.

The PowerShell scripts are too long to write in the post, they can be downloaded here. The list is (\SharePointLib):

*DeploySharePointSoluitions.ps1: Start the WSP deployments. Parse command line arguments of config file name and directory of WSP solutions

*(Part 2) EnsureFeatureActivation.ps1: Start the feature activation scripts (need command line arguments)

*(Part 3) QADeploymentProcess.ps1: Deployment process for dev, test and QA environments (need command line arguments)

SharePointDeploymentScripts.ps1: Main script that defines a large number of methods for deployment and activations.

SharePointHelper.ps1: Fairly simple SharePoint helper methods to get farm information

*RestartSharePointServices.ps1: Restart all SharePoint services on all servers in the local farm (see runtime notes below)

Services.ps1: Non-SharePoint specific methods for handling services, e.g. restarts

Logging.ps1: Generic logging method. Note that there are options for setting different log levels, output files, etc.

The ones with an asterisk are the ones that you are supposed to execute – the others just defined methods for the “asterisk scripts”.

The batch files (same download) (root of zip download):

01 Deploy Solutions.bat: Start the solution deployment. It will deploy all the WSP files dropped in “\SolutionsDropDir\” that are also specified in the “\DeploymentConfig.xml”:

02 Activate Features.bat (Part 2): Works with the “\DeploymentConfig.xml” file and ensures that all features are activated.

In other words you’ll execute “01 Deploy Solutions.bat” to deploy all your solutions.


By default you will get one log file from each batch file named after the operation, one log file with a timestamp and in case of any errors an additional error log with the same timestamp. In other words if the error log file is not created then no errors occurred.

Note: Scripts must be executed on one of your SharePoint servers which sufficient rights. That is more or less local admin and owner on the individual databases (or local admin on the SQL box as well).

Note 2: I write quite a bit of logging it is invaluable in times of trouble. You can change the amount in the logging.ps1 file by adjusting the loglevel variable.

Note 3: In case of deployment errors the default behavior is to restart every SharePoint service on every server and then retry. These service restarts are performed async, so it should take no more than a minute or two, no matter how many servers you have (almost).

Note 4: Any WSP found in the drop dir that has not been specified in the config is considered an error – therefore you know that all the wsps provided are deployed if no errors are reported.

Runtimes

The time it takes to search the local farm for relevant site collections / web applications is negligible so you shouldn’t worry unduly about impact on runtime to add 300 URLs to cover all your possible environments including every developer’s individual preferred setup.

But there is no magic and it simply runs in the time it takes to deploy solutions to your environment. If it takes about 3 minutes for a full retract, remove, add, deploy then it takes 30 minutes for 10 WSPs.

The default way of handling deployment errors is to restart all SharePoint services in the farm. The clever way of doing that is to send the stop/start commands to all services first and then wait for the operations afterwards. That way the time taken to do restarts is about the time of restarting the slowest service in the farm once – it usually takes 1-2 minutes regardless of the number of servers. In my opinion this is quite cool 🙂

Closing notes

If you are wondering about my exception handling note that the scripts were originally made for PowerShell 1.0 and later I decided to only target 2.0. Some statements may be written prettier without me taking the time to do so.

These scripts save us a ton of time and prevents a lot conversations of the sort “oh I forgot to activate that feature, 2 secs, there you go” – but of course quite some time has been spent creating the deployment scripts and fixing errors within them.

Some of the scripts are quite usable for other purposes. For instance, the remote service restart script I have not seen anywhere else – this is simpler than installing PowerShell remoting (WinRM?) on every server.

If you make some brilliant changes then let me know so I can include it in my “official” version.

Please also leave credits in there so it’ll be possible for your successors to find the source and documentation.

Download

Grab the scripts here (updated with part 2 and 3).

Note: Updated Aug 29 2011.

Automatic Fix of Pre Upgrade Issues (SharePoint 2007 to 2010)


Ever tried to run the PreUpgradeCheck command at your MOSS farm and received a frightening long list of issues that takes forever to solve manually?

If you are like me you refuse to do any manual work that takes more than 2 hours (and you’ll have to re-do it for every farm) – so I’ve tried to script all the required fixes. This is one of those posts that I’ve been working on for months so I hope you appreciate it – I’m a bit proud of the script at the end of the post.

The goal is to have a PowerShell (v1.0) script fix every one of those blocking issues you see in the PreUpgradeCheck report. On my own very messy (intentionally so!) farm with 40+ databases and site collections and a large number of solutions I’ve managed to fix all but the one about my operating system needing an upgrade to Server 2008. That one I’ll not work on 😉

Going from:

To this:

Note: It will only work of you have installed SP2 and cumulative update from October 2009. At least. I recommend that you work of as new a version as possible.

WARNING

The script will remove invalid “stuff” from your sites so be absolutely sure that every solution that you care about is deployed to the farm – otherwise all features, files and webparts from missing solutions will be removed from the sites.

The script will also publish unpublished pages if they contain some of these invalid webparts.

I give you no warrenty – backup your farm first. I’m pretty sure I won’t mess it up though…

What it does

The script will check a number of things in your farm by working on the output of the “stsadm –o enumallwebs” command.

I use some direct SQL to get information out that is not possible through the API, but all modifications are done through the SharePoint 2007 API in a supported way only.

The script currently does 5 things:

Remove Invalid Webparts

(AKA: “The following web part(s) are referenced by the content, but they are not installed on the web server“)

This is the method I started with and by far the biggest timesaver.

It uses the “stsadm –o enumallwebs” to get a list of webparts that’s currently missing in your farm. They have been uninstalled at some point and are now those webpart with “Fatal Errors” on some of your pages. Some of the missing webparts are not even listed as failing with fatal errors (they have socalled “bad markup”) they are just listed on the page with and ID and no description (i.e. if you go to a given page and add “?contents=1” to go to the webpart maintenance page).

Trouble is that there can be lots of these and there is no easy way to find all the pages.

This function will query the SQL for the pages where these webparts are (both the one with fatal errors and the ones with “bad markup”) and then remove them through the API. The offending pages will be published (if not already), whatever checkouts they may have been there is overridden, the webparts are removed and the page re-published.

It will take all pages not just the one in document libraries (i.e. those that came through features)

Note: I’m only handling the case of shared webparts I did not have any user specific webparts in my test farm so I’m uncertain how that will be handled.

Note2: I’ve copied the PublishListItem from some of Gary Lapointes excellent extentions and converted them to powershell. It seems that no one has been as thorough handling special cases as he.

Remove Orphaned Site Collections

(AKA: “Issue : Orphaned site collections“)

Site collections can get orphaned in a number of ways; the fix is simply to delete those.

You will not lose anything (new) as those sites were not accessible before anyway.

Remove Uninstalled/missing Features Activated at Web scope

Features that have been uninstalled at some point often linger in the content databases because they are still activated in some web or site. That activation is not deleted when the feature is removed though the feature will have no actual function anymore.

I wrote quite a bit on this a while ago.

The script will run through all those SPWeb’s and deactivate the offending features.

Note that I’m not crawling all webs in the site I only access those with missing features.

Remove Uninstalled/missing Features Activated at Site Collection Scope

(AKA: “The following feature(s) are referenced by the content, but they are not installed on the web server“)

Similarly features may be activated at site collection level and forgotten. In this case these features are not reported by the “stsadm –o enumallwebs” command and I have to run through every Site Collection in the farm to locate the features.

Remove Missing Setup Files

(AKA: “The following setup file(s) are referenced by the content, but they are not installed on the web server“)

After features have been uninstalled it is quite common that they forget to remove the files they installed on the farm. I suppose it’s just the way SharePoint works. If a feature installs a file in a document library somewhere changes are it will remain.

What are those files? Some examples are stuff in the style library, webpart gallery, master pages, page layouts, etc…

This method uses SQL to find where the missing files are still references and deletes them from the lists. It even deletes files that have been unghosted which could actually still be accessed because SharePoint has made a customized copy of it.

I’m still considering if it would be better to only remove the ones that are still ghosted.

Note: In my tests some page layouts could not be removed because they were still being referenced in the farm. That is clearly an error in my farm, however the script will not fix that. It really does require special handling by a human to decide what to do. I suppose it’s a good thing that they are not removed in this case as some things might otherwise break.

What it does not no

Out of scope of script:

  1. Fixing the OS version
  2. Fixing the database version

Sorry 😉

How to Execute

First of all the script requires you to be site collection owner on every site collection, farm admin and have owner rights in every content and configuration database. Pretty much all you can get.

In the beginning of the script there is a few switches that you should know:

  • The logging level, set it to 3 to get lots of info: $verboseLevel = 3. Lower is less.
  • The “$whatif” ´switch. Set it to “$true” to have the script simulate a run but not change anything in your farm
  • The “$targetdb” that can limit the script to a single content database only. Blank equals all databases a name indicates one (note: Not the site name)

I’ve built in a lot of error handling and logging to let you know what’s going on.

Note that STSADM must be included in the path setting.

Recommended sequence of execution

Everything need to run on the SharePoint 2007 server and it must have (at least) PowerShell 1.0 installed.

  1. Backup all content databases. This is not optional. Easiest to use stsadm –o backup
  2. Use stsadm -o preupgradecheck
  3. Run script on one or all databases with the “whatif” statement set to true
  4. Check and recheck the output and verify that it looks correct to you.
    1. You may want to reduce the logging level a bit
  5. Run with $whatif set to $false, i.e. let it work!
    1. Depending on your confidence in me you can ask to work on only one database at a time
  6. Check the output!
    1. Errors and warnings will be written both to the output stream (what is piped down in your logfile) and on the screen
  7. Run preupgradecheck again and see the result J

Note to run the script use something like “powershell -file FixUpgradeIssues.ps1 >>fixupdateoutput.txt”, i.e. always pipe the output to a file that you can investigate later.

Runtime is about 5 minutes on my 40+ database test system so reasonably ok given the task.

I’ve been working on this script for a couple of months and have fixed a huge number of issues and funny edge cases; however I’ve likely missed a few. Let me know in the comments.

Good luck J

Download the Actual Script

All 850 lines of powershelling are not pretty in a blog post, so you can download it here.

[Updaet 25-12-2010: Now with correct file]

How to install a SharePoint 2010 Complete (Dev) Server without AD


This post details how you do install a SharePoint 2010 as a “complete” install without an AD, which is very useful to me as a development server (with or without Visual Studio).

This applies to both virtual and physical machines but I always work with VMs because traditionally SharePoint dev environments need to be re-installed once in a while and that’s easier with VMs.

Why would you want this?

  • A development server with all components is likely to resemble your test/QA/production environment a lot more than the alternative standalone install
  • A server with a local install of SharePoint with non-AD accounts can be run with or without an AD domain – you can even run the VM as a domain server disconnected from the actual domain e.g. at home or the commute
    • Alternative 1: Make your server an AD server but that changes all sorts of stuff with user management and will definitely not resemble any of your production servers
    • Alternative 2: Create two VMs one being the AD one being the actual development server connected on the same VLAN and waste a lot of resources for (almost) nothing.
  • To eliminate any need for rogue AD servers on your network that some develop accidently connected directly to the network and running DHCP, DNS etc. Don’t trust your developers or external consultants to care about your network!
  • I want a full SQL Server!

Why would you not want this?

  • This is not a supported development environment from Microsoft – they support installing a so-called standalone development environment without any of the frills. It’s easier and it’s officially supported. It’s even doable on Windows 7.
    • Why anyone would want to develop SharePoint on a Windows 7 machine is beyond me, the runtime environment for your code will always be a server 2008 so why not develop and test it directly on such a box? Surely you develop only in VMs so that you are able to create a clean dev environment easily once in a while…

How to

The procedure is fairly simple except for the final steps. Note that you can (and should) use whatever tools you can to help you out, I’ll point at the promising AutoSPInstaller at CodePlex.

Procedure:

  1. Create / install a Server 2008 (R2) 64 bit with
    1. Visual Studio 2010
    2. SQL Server 2008 (use a local user as service account)
    3. … and whatever other tools you are fond of …
    4. (Remember to sysprep/snapshot it at this stage)
  2. Install SharePoint 2010 with all prerequisites
    1. Scripted or not – do not run Config Wizard yet (It would result in “Local accounts should only be used in standalone mode” error)
  3. Create the farm by (trick #1)
    1. Start the SharePoint PowerShell
    2. Create a local service account
    3. Create a farm by running the “New-SPConfigurationDatabase” cmdlet and supply parameters for the service account, DB name, DB server and passphrase (thanks to Neil ‘The Doc’ Hodgkinson for that)
    4. After it finishes start the Config Wizard (interactive or not) and configure your server with all components
  4. Configure the farm services as you like
    1. I usually just use the wizard in Central Admin to configure all the Service Applications with some fairly useful values it works well enough
  5. Enterprise Search doesn’t work to fix it see below… (trick #2)

The Trouble with Search

Search will fail with a number of errors and in the search administration the Query Component will remain stuck in the initializing state:

The other bunch of event log errors etc. is listed at the end of this post for the benefit of Google.

As far as I can conjecture the problem is that the timer service is trying to setup a network share for every query component where the crawlers can dump their data. It is trying to setup that share with a domain account that happens to be a local user instead in this case and fails with either an “Access Denied” error or a “System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated”.

The share name it’s trying to use is the same as the query role, i.e. “Guid-query-0″ pointing to (if using default locations) “C:\program files\Microsoft Office Servers\14.0\Data\Office Server\Applications” with change permissions for the “WSS_WPG” group.

Unfortunately it does not help to just create the share for it apparently the query components insist on waiting for the timer job to complete successfully. L

The Search Fix

The fix is fairly simple and almost completed by Gary Lapointe whom I owe great thanks for doing most of the hard work in his post on scripting the Enterprise Search installation and the comments below his post (thanks to Marco van Wieren).

The fix is simply to create and configure all the enterprise search components from PowerShell as it allows you to set a few more options, specifically the share name for the query components so that you are then allowed to create them yourself.

The script was originally made for configuring search components across an entire farm and therefore a bit more complicated than it strictly has to be. I left it in there while adding support for single server install as well. Gary’s script was made for beta 2 and I’ve fixed a few simple errors/typos, corrected the few API changes between beta 2 and RTM and finally added the share name support.

The script is quite long a not suitable for pasting into a blog – download instead.

The script needs a configuration file with something like this:

<Services>
    <EnterpriseSearchService ContactEmail="no-reply@SharePointDev1.com"
                             ConnectionTimeout="60"
                             AcknowledgementTimeout="60"
                             ProxyType="Default"
                             IgnoreSSLWarnings="false"
                             InternetIdentity="Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)"
                             IndexLocation="C:\Program Files\Microsoft Office Servers\14.0\Data\Office Server\Applications"
                             PerformanceLevel="PartlyReduced"
                             Account="localhost\saservice"
                             ShareName="SearchShare">

        <EnterpriseSearchServiceApplications>
            <EnterpriseSearchServiceApplication Name="Enterprise Search Service Application"
                                                DatabaseServer="localhost"
                                                DatabaseName="SharePoint_Search"
                                                FailoverDatabaseServer=""
                                                Partitioned="false"
                                                Partitions="1"
                                                SearchServiceApplicationType="Regular">
                <ApplicationPool Name="SharePoint Enterprise Search Application Pool" Account="localhost\saservice" />
                <CrawlServers>
                    <Server Name="localhost" />
                </CrawlServers>
                <QueryServers>
                    <Server Name="localhost" />
                </QueryServers>
                <SearchQueryAndSiteSettingsServers>
                    <Server Name="localhost" />
                </SearchQueryAndSiteSettingsServers>
                <AdminComponent>
                    <Server Name="localhost" />
                    <ApplicationPool Name="SharePoint Enterprise Search Application Pool" Account="localhost\saservice" />
                </AdminComponent>
                <Proxy Name="Enterprise Search Service Application Proxy" Partitioned="false">
                    <ProxyGroup Name="Default" />
                </Proxy>
            </EnterpriseSearchServiceApplication>
        </EnterpriseSearchServiceApplications>
    </EnterpriseSearchService>
</Services>

Remarks:

  • I replace “localhost” with the actual computer name in the script
  • The Share Name (here “SearchShare”) will be created by the script as well, so whatever you call it doesn’t matter
  • The config file shown can be reused on every machine provided that the local service account “saservice” has been created before

To continue and complete step 5 in the procedure above (sorry for the numbering wordpress is messing up the html):

  1. Start PowerShell shell (I will load the SharePoint snapin if it’s not a SharePoint Management Shell)
    1. Load the “SetupEnterpriseSearch.ps1” script (just drag the file into the shell and execute) which will define the required functions
    2. Execute “Start-EnterpriseSearch “<path>\searchconfig.xml””
    3. Wait for a few minutes and watch for errors
  2. Go to the Search Administration and verify that your new search topology works
    1. It should look something like this:

    2. If you configured search in step 4 you will have two
    3. If you have two you can safely go back to “Manage service applications” and delete the one named “Search Service Application 1” (and associated databases) – the one created by the script is “Enterprise Search Service Application”
  3. Try it! Go to a local SharePoint site and search for something
    1. Before the search would return a server error 500 so anything else than that can be considered a success
    2. I like to add a few documents and have them show up in the search before I call it a success…

Caveats / Fast Search

Don’t know if it fair to call it a caveat however only the Enterprise Search is demonstrated here, the Fast Search behaves similarly in respect to the “share trouble” and will probably need the same fix as the enterprise search. I’ve not found the time or need to poke around with that just yet, but it should be doable in less than a day given the foundation above (for someone skilled in SharePoint and powershell).

Conclusions

It works; I’ll use it from now on 🙂

… and I hope the nice chaps at AutoSPInstall will include this fix in their tool.

Scared of being in unsupported land?

  • It’s only your dev server and it did move a lot closer to production that the standalone dev machine option
  • It also protected your network from rogue AD servers that might potentially kill half your network if you are unlucky

So how well is this tested? Quite well for a single server install and not at all for a farm install (not by me at least). Trust it with the former and test it yourself if you need the latter.

Observed Errors

I got a lot of different errors, here they are for the benefit of Google.

Event log entry after completing the configuration wizard in Central Administration:

Log Name: Application
Source: Microsoft-SharePoint Products-SharePoint Server Search
Date: 11-06-2010 22:20:17
Event ID: 2579
Task Category: Administration
Level: Error
Keywords:
User: SHAREPOINTDEV1\saservice
Computer: SharePointDev1
Description:
Component a61ca0ca-194f-4cf0-bb5c-8ca998178935-query-0 of search application ‘Search Service Application’ has failed to execute transition sequence ‘initialize with empty catalog’ due to the following error: System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String& sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage(). It is now in state Uninitialized.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Server Search” Guid=”{C8263AFE-83A5-448C-878C-1E5F5D1C4252}” />
<EventID>2579</EventID>
<Version>14</Version>
<Level>2</Level>
<Task>14</Task>
<Opcode>0</Opcode>
<Keywords>0×4000000000000000</Keywords>
<TimeCreated SystemTime=”2010-06-11T20:20:17.723875000Z” />
<EventRecordID>3926</EventRecordID>
<Correlation ActivityID=”{B1431F7E-1D0C-4CB7-B690-F0F016447FE4}” />
<Execution ProcessID=”956? ThreadID=”3484? />
<Channel>Application</Channel>
<Computer>SharePointDev1</Computer>
<Security UserID=”S-1-5-21-452889701-636363473-2591022535-1012? />
</System>
<EventData>
<Data Name=”string0?>a61ca0ca-194f-4cf0-bb5c-8ca998178935-query-0</Data>
<Data Name=”string1?>Search Service Application</Data>
<Data Name=”string2?>initialize with empty catalog</Data>
<Data Name=”string3?>System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String&amp; sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage()</Data>
<Data Name=”string4?>Uninitialized</Data>
</span
</EventData>
</Event>

And from the ULS log:

06/11/2010 22:20:17.72         OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server Search         Administration         fea9        Critical        Component a61ca0ca-194f-4cf0-bb5c-8ca998178935-query-0 of search application ‘Search Service Application’ has failed to execute transition sequence ‘initialize with empty catalog’ due to the following error: System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated. Parameter name: sddlForm at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm) at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm) at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String& sddl) at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path) at Microsoft.S…        b1431f7e-1d0c-4cb7-b690-f0f016447fe4
06/11/2010 22:20:17.72*        OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server Search         Administration         fea9        Critical        …harePoint.Administration.SPServer.CreateFileShare(String name, String description, String path) at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component) at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage(). It is now in state Uninitialized.        b1431f7e-1d0c-4cb7-b690-f0f016447fe4
06/11/2010 22:20:17.72         OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server         Unified Logging Service         2m1i        Verbose         Adding event 2579 (category: Administration, product: SharePoint Server Search) to spam monitoring list        b1431f7e-1d0c-4cb7-b690-f0f016447fe4
06/11/2010 22:20:17.72         OWSTIMER.EXE (0x03BC)         0x0D9C        SharePoint Server Search         Administration         djs2        Medium         SearchApi (): executing SetQueryComponent(d355048f-d4fa-4f31-88b0-342b5ed48e5c, null, null, null, null, Uninitialized, Uninitialized, null, -1, Failed, null, False, null, null, False, null)        b1431f7e-1d0c-4cb7-b690-f0f016447fe4

And another event log:

Log Name: Application
Source: Microsoft-SharePoint Products-SharePoint Foundation
Date: 12-06-2010 20:40:26
Event ID: 6398
Task Category: Timer
Level: Critical
Keywords:
User: SHAREPOINTDEV1\saservice
Computer: SharePointDev1
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.CrawlReportJobDefinition (ID 9529aace-a679-4fc9-ab8d-325780484cf0) threw an exception. More information is included below.
The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component ’3147b99c-8f3a-41e9-a08b-296f930af877' in search application ‘Enterprise Search Service Application’ is in a good state and try again.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6FB7E0CD-52E7-47DD-997A-241563931FC2}” />
<EventID>6398</EventID>
<Version>14</Version>
<Level>1</Level>
<Task>12</Task>
<Opcode>0</Opcode>
<Keywords>0×4000000000000000</Keywords>
<TimeCreated SystemTime=”2010-06-12T18:40:26.553054700Z” />
<EventRecordID>4159</EventRecordID>
<Correlation ActivityID=”{6CED0041-2038-43E3-AB79-4DEFBB4216B3}” />
<Execution ProcessID=”1324? ThreadID=”1532? />
<Channel>Application</Channel>
<Computer>SharePointDev1</Computer>
<Security UserID=”S-1-5-21-452889701-636363473-2591022535-1012? />
</System>
<EventData>
<Data Name=”string0?>Microsoft.Office.Server.Search.Administration.CrawlReportJobDefinition</Data>
<Data Name=”string1?>9529aace-a679-4fc9-ab8d-325780484cf0</Data>
<Data Name=”string2?>The search service is not able to connect to the machine that hosts the administration component. Verify that the administration component ’3147b99c-8f3a-41e9-a08b-296f930af877' in search application ‘Enterprise Search Service Application’ is in a good state and try again.</Data>
</EventData>
</Event>

And one for foundation search:

Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          …
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:
User:          …
Computer:      …
Description:
The Execute method of job definition Microsoft.Office.Server.Search.Administration.QueryTopologyActivationJobDefinition (ID de8eac2b-57db-4069-896d-747ae4fb35ed) threw an exception. More information is included below.
Topology activation was aborted because of System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String& sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage().
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6fb7e0ce-52e7-47dd-997a-241563931fc2}” />
<EventID>6398</EventID>
<Version>14</Version>
<Level>1</Level>
<Task>12</Task>
<Opcode>0</Opcode>
<Keywords>0×4000000000000000</Keywords>
<EventRecordID>10895</EventRecordID>
<Correlation ActivityID=”{6E239D20-A2CD-45B4-AC87-4477A82558BB}” />
<Execution ProcessID=”2016? ThreadID=”2288? />
<Channel>Application</Channel>
<Computer>id1314</Computer>
<Security UserID=”S-1-5-21-30024279817-590149927-1659320300-1003? />
</System>
<EventData>
<Data Name=”string0?>Microsoft.Office.Server.Search.Administration.QueryTopologyActivationJobDefinition</Data>
<Data Name=”string1?>de8eac2b-57db-4069-896d-747ae4fb35ed</Data>
<Data Name=”string2?>Topology activation was aborted because of System.ArgumentException: The SDDL string contains an invalid sid or a sid that cannot be translated.
Parameter name: sddlForm
at System.Security.AccessControl.RawSecurityDescriptor.BinaryFormFromSddlForm(String sddlForm)
at System.Security.AccessControl.RawSecurityDescriptor..ctor(String sddlForm)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateShareSecurityDescriptor(String[] readNames, String[] changeNames, String[] fullControlNames, String&amp; sddl)
at Microsoft.SharePoint.Win32.SPNetApi32.CreateFileShare(String name, String description, String path)
at Microsoft.SharePoint.Administration.SPServer.CreateFileShare(String name, String description, String path)
at Microsoft.Office.Server.Search.Administration.QueryComponent.CreatePropagationShare(QueryComponent component)
at Microsoft.Office.Server.Search.Administration.QueryComponent.ExecuteCurrentStage().</Data>
</EventData>
</Event>

And finally from Gary’s blog (Marco van Wieren):

Component: 3b609311-67da-4df8-8c12-e597e9228dd3-crawl-0
Details:
The system cannot find the file specified. 0x80070002Propagation for search application 3b609311-67da-4df8-8c12-e597e9228dd3-crawl-0: failed to communicate with query server 3b609311-67da-4df8-8c12-e597e9228dd3-query-0.

Automatic Setup of Blank Sites across Farm


This is another PowerShell automation blog post 🙂

What

Setup a blank site (that is: “Sorry site is down for maintenance”) on all the front end servers and stop all existing web applications with a minimum of work. Likewise take it down easily.

By taking it down I mean deleting it so there is no risk that “someone” (the ghost in the server) might activate it by accident.

One script running on one server will install the blank site to multiple front-end servers.

Why

When you need to take down your sites for maintenance the right thing to do is to setup some temporary site – otherwise your visitors might easily see a semi-functioning site or a huge ugly exception when you’re in the middle of something.

This is not a big task to do manually, however it is tedious if you have many front-end servers and/or many web applications.

How

This is a bunch of powershell scripts that is started by two batch files – one for installing the blank site and one for uninstalling. The tasks performed during installation/setup of blank site are:

For each web server (defined as arguments to the PowerShell script within the batch file):

  1. Stop all running websites
  2. Setup a new web site (named “Blank Site” on port 80, path “c:\inetpub\wwwroot\blanksite” – customize yourself)
  3. Copy all the files (html, aspx, css, etc..) that goes into the blank site to the newly created directory

When you uninstall the blank site the same tasks are executed in reverse order. If there is more than one blank site on the server (i.e. you happened to run the _InstallBlankSite.bat file twice) they will all be deleted by the _UninstallBlankSite.bat script.

Notes

These scripts are used by me on my SharePoint farms; however it can be used on any type of (IIS) web server setup. I’ve tested it with IIS 6 but I believe it should work with IIS 5 and upwards.

You need to edit the two batch files with the correct servernames for your farm. You may add as many as you like (separate with spaces)

I recommend that you run the scripts on a server with PowerShell installed that is not an actual front-end server. I consider it fairly bad practice to install PowerShell on the front-end servers – I only install it on the backend CA/index server.

I use one of MS’ vbs scripts from IIS6 to do interact with the IIS. Another option is to use WMI, but only if you are either only accessing local server or using PowerShell 2.0. I’ve included that script in my package, so there is no requirement that the server executing the PowerShell scripts has an IIS server installed. Not sure if it’s actually legal, but I have a hard time seeing the harm done.

The uninstall script will start all stopped/paused websites regardless if they were actually running before you started the whole thing.

If you can’t get it to work at all, check your execution policy.

Download

The following scripts are included

_InstallBlankSite.bat: Main script to execute the dependent scripts. It will start powershell, and you need to add your server names here.

_UninstallBlankSite.bat: Main script to remove blank site. You need to add your servernames here.

InstallBlankSiteOnServers.ps1: Given a blank site and list of servers do the magic and call dependent scripts.

StopAllWebsites.ps1: Stop all running websites on a given server.

CreateBlankSite.ps1: Create a new website on server with the given local path (does not copy files).

UninstallBlankSiteOnServers.ps1: Remove a blank site from a list of servers. Assumes a local path to files.

StartAllWebSites.ps1: Start all stopped websites on a (remote) server.

DeleteBlankSite.ps1: Removes the blank site(s) from the IIS on the specified server (UninstallBlankSiteOnServers.ps1 deletes the files).

BlankSite: (Directory) the files that constitute your blank site. You’ll want to make it prettier but you get the point from what’s in there.

IISWeb.vbs: IIS script file for IIS6 from Microsoft that performs all IIS management tasks. It has not been altered in any way, it’s merely there for convenience. You’ll find it in system32 on your webservers.

The package can be found here