How to Make List Items Visible to Anonymous Users (in Search)


I had a funny little issue with showing list items from a custom list (with a custom view form) to anonymous users on a publishing site.

My good colleague Bernd Rickenberg insisted that I blogged the resolution since he had found quite a few post detailing the problem but no viable solutions ;-)

The Issue

You want to show a custom list to anonymous users. In our setting it was through search, your use case is likely different, but the issue remains the same with or without search.

Quite simply forms (List forms) are generally not accessible to anonymous users when you have the lockdown feature enabled (ViewFormPagesLockdown). It is a critical feature to have enabled otherwise you expose way too much information to SharePoint savvy users. Many of the solutions to this issue suggest turning it off.

It is fairly simple to test if you have this problem. Start Firefox, assuming that you have not enabled the NTLM auto login setting, and hit the …/DispForm.aspx page for a specific item. If you receive the login prompt as anonymous in Firefox and sail through when logged in this blog is for you.

If the ordinary publishing pages are also not viewable then this blog is not for you.

Note: Never use IE to test for anonymous access or not, I can’t count the number of times it has tricked consultants into thinking it works because they are tricked by the auto login feature while on the corporate network.

The Resolution

Is fairly simple.

The basics for search is that the list must be made searchable (it is by default) in the list settings, anonymous users must have access rights to the site and the lockdown feature should(!) be enabled.

What the lockdown feature does is that it changes the anonymous permission mask at the root site (which is inherited by all by default). The mask is basically the permission level assigned to all anonymous users and is similar to the normal permission sets – but is not editable in the UI.

The “View Application Pages” permission level is removed, see the image below on where to find it (for non-anonymous users):

Permission level settings page

Permission level settings page – not available for anonymous users

The best option, security wise, is to break the permission inheritance for your particular list and then add the “View Application Pages” permission to the anonymous users. Do not do that at the web level as you do not want to expose e.g. All site content etc.)

The Script

You need to run the following the PowerShell commands on one of your servers (replace url and list name):

$web = get-spweb "http://yoursiteurl/subweb/subsubweb" 
$list = $web.Lists["ListName"]
$list.BreakRoleInheritance($true)
$list.AnonymousPermMask = $list.AnonymousPermMask -bor ([int][Microsoft.SharePoint.SPBasePermissions]::ViewFormPages) #binary or adding the permissions
$list.Update()
 

(Note: Do be careful if copying directly from this page to PowerShell – you need to re-type the quotes as WordPress mangles them)

Fixing Event Error 7888: “User not found”


We’ve been having an annoying recurring event log error every minute for a while on some of our farms. Recently we digged deep into the issue and finally fixed it J

This applies to SharePoint 2007 and SQL Server 2005. It’s likely fixed in other versions.

The error message is “Windows NT user or group ‘domain\user‘ not found” returned from the SQL server when a SSP synchronization timer jobs tries to sync service account access to the databases. It executes “sp_grantlogin ‘domain\user’” even though the account is already granted login in the databases (content and config).

After a lot of digging and code disassembling it turns out that the database stored procedure is case sensitive on the user name and SharePoint provides the user name in whatever casing you originally supplied to SharePoint (when creating app pools). I’m quite surprised by this and not sure if it’s always the case. We are using a SQL Server 2005 with proper case insensitive collation, but still…

Interestingly the domain name is not case sensitive.

The Implications

This error seems to be only an annoyance in the event log and nothing more as the user the job tries to add to the database is actually already there. Nothing is really broken by this.

The Fix

Fortunately it can easily be fixed by an administrator.

Procedure:

  1. Go to Central Administration / Operations / Service Accounts (/_admin/FarmCredentialManagement.aspx)
  2. Choose “Web Application Pool” and “Windows SharePoint Services Web Application”

  3. Go through every application pool and verify that the user name on the page is using the correct casing
    1. What is the correct casing? Lookup the user in your AD and use the value for sAMAccountName
  4. Click Ok

That’s it!

For good measure I restarted the timer services on every SharePoint server in the farm.

The Symptoms

You get the following lines in the ULS log:

12/08/2010 07:15:31.75     OWSTIMER.EXE (0x01D0)                       0x1FB8    Office Server                     Office Server General             6pqn    High        Granting user 'domain\user' login access to server 'cbdks173\mosstest,1433'.     
12/08/2010 07:15:31.77     OWSTIMER.EXE (0x01D0)                       0x1FB8                                      484                               880i    High        System.Data.SqlClient.SqlException: Windows NT user or group 'domain\user' not found. Check the name again.     at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)     at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)     at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)     at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)     at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)     at Syste...     
12/08/2010 07:15:31.77*    OWSTIMER.EXE (0x01D0)                       0x1FB8                                      484                               880i    High        ...m.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)     at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)     at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()     at Microsoft.Office.Server.Data.SqlSession.ExecuteNonQuery(SqlCommand command)     
12/08/2010 07:15:31.77     OWSTIMER.EXE (0x01D0)                       0x1FB8                                      484                               880j    High        SqlError: 'Windows NT user or group 'domain\user' not found. Check the name again.'    Source: '.Net SqlClient Data Provider' Number: 15401 State: 1 Class: 11 Procedure: 'sp_grantlogin' LineNumber: 49 Server: 'cbdks173\mosstest,1433'     
12/08/2010 07:15:31.77     OWSTIMER.EXE (0x01D0)                       0x1FB8                                      484                               880k    High           at Microsoft.Office.Server.Data.SqlServerManager.GrantLogin(String user)     at Microsoft.Office.Server.Administration.SharedResourceProvider.SynchronizeConfigurationDatabaseAccess(SharedComponentSecurity security)     at Microsoft.Office.Server.Administration.SharedResourceProvider.SynchronizeAccessControl(SharedComponentSecurity sharedApplicationSecurity)     at Microsoft.Office.Server.Administration.SharedResourceProvider.Microsoft.Office.Server.Administration.ISharedComponent.Synchronize()     at Microsoft.Office.Server.Administration.SharedResourceProviderJob.Execute(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result)       
12/08/2010 07:15:31.77     OWSTIMER.EXE (0x01D0)                       0x1FB8                                      484                               880l    High        ConnectionString: 'Data Source=cbdks173\mosstest,1433;Initial Catalog=master;Integrated Security=True;Enlist=False;Pooling=False'    ConnectionState: Open ConnectionTimeout: 15     
12/08/2010 07:15:31.77     OWSTIMER.EXE (0x01D0)                       0x1FB8                                      484                               880m    High        SqlCommand: 'sp_grantlogin'     CommandType: StoredProcedure CommandTimeout: 0     Parameter: '@loginame' Type: NVarChar Size: 128 Direction: Input Value: 'domain\user'
   

Combined with this application event log error every minute:

Log Name:      Application
Source:        Office SharePoint Server
Date:          08-12-2010 08:32:31
Event ID:      7888
Task Category: (1516)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <computer>
Description:
The description for Event ID 7888 from source Office SharePoint Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

Windows NT user or group 'domain\user' not found. Check the name again.
System.Data.SqlClient.SqlException: Windows NT user or group 'domain\user' not found. Check the name again.
 at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
 at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
 at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
 at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
 at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
 at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
 at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
 at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
 at Microsoft.Office.Server.Data.SqlSession.ExecuteNonQuery(SqlCommand command)
 at Microsoft.Office.Server.Data.SqlServerManager.GrantLogin(String user)
 at Microsoft.Office.Server.Administration.SharedResourceProvider.SynchronizeConfigurationDatabaseAccess(SharedComponentSecurity security)
 at Microsoft.Office.Server.Administration.SharedResourceProvider.SynchronizeAccessControl(SharedComponentSecurity sharedApplicationSecurity)
 at Microsoft.Office.Server.Administration.SharedResourceProvider.Microsoft.Office.Server.Administration.ISharedComponent.Synchronize()

Automatic Fix of Pre Upgrade Issues (SharePoint 2007 to 2010)


Ever tried to run the PreUpgradeCheck command at your MOSS farm and received a frightening long list of issues that takes forever to solve manually?

If you are like me you refuse to do any manual work that takes more than 2 hours (and you’ll have to re-do it for every farm) – so I’ve tried to script all the required fixes. This is one of those posts that I’ve been working on for months so I hope you appreciate it – I’m a bit proud of the script at the end of the post.

The goal is to have a PowerShell (v1.0) script fix every one of those blocking issues you see in the PreUpgradeCheck report. On my own very messy (intentionally so!) farm with 40+ databases and site collections and a large number of solutions I’ve managed to fix all but the one about my operating system needing an upgrade to Server 2008. That one I’ll not work on ;-)

Going from:

To this:

Note: It will only work of you have installed SP2 and cumulative update from October 2009. At least. I recommend that you work of as new a version as possible.

WARNING

The script will remove invalid “stuff” from your sites so be absolutely sure that every solution that you care about is deployed to the farm – otherwise all features, files and webparts from missing solutions will be removed from the sites.

The script will also publish unpublished pages if they contain some of these invalid webparts.

I give you no warrenty – backup your farm first. I’m pretty sure I won’t mess it up though…

What it does

The script will check a number of things in your farm by working on the output of the “stsadm –o enumallwebs” command.

I use some direct SQL to get information out that is not possible through the API, but all modifications are done through the SharePoint 2007 API in a supported way only.

The script currently does 5 things:

Remove Invalid Webparts

(AKA: “The following web part(s) are referenced by the content, but they are not installed on the web server“)

This is the method I started with and by far the biggest timesaver.

It uses the “stsadm –o enumallwebs” to get a list of webparts that’s currently missing in your farm. They have been uninstalled at some point and are now those webpart with “Fatal Errors” on some of your pages. Some of the missing webparts are not even listed as failing with fatal errors (they have socalled “bad markup”) they are just listed on the page with and ID and no description (i.e. if you go to a given page and add “?contents=1″ to go to the webpart maintenance page).

Trouble is that there can be lots of these and there is no easy way to find all the pages.

This function will query the SQL for the pages where these webparts are (both the one with fatal errors and the ones with “bad markup”) and then remove them through the API. The offending pages will be published (if not already), whatever checkouts they may have been there is overridden, the webparts are removed and the page re-published.

It will take all pages not just the one in document libraries (i.e. those that came through features)

Note: I’m only handling the case of shared webparts I did not have any user specific webparts in my test farm so I’m uncertain how that will be handled.

Note2: I’ve copied the PublishListItem from some of Gary Lapointes excellent extentions and converted them to powershell. It seems that no one has been as thorough handling special cases as he.

Remove Orphaned Site Collections

(AKA: “Issue : Orphaned site collections“)

Site collections can get orphaned in a number of ways; the fix is simply to delete those.

You will not lose anything (new) as those sites were not accessible before anyway.

Remove Uninstalled/missing Features Activated at Web scope

Features that have been uninstalled at some point often linger in the content databases because they are still activated in some web or site. That activation is not deleted when the feature is removed though the feature will have no actual function anymore.

I wrote quite a bit on this a while ago.

The script will run through all those SPWeb’s and deactivate the offending features.

Note that I’m not crawling all webs in the site I only access those with missing features.

Remove Uninstalled/missing Features Activated at Site Collection Scope

(AKA: “The following feature(s) are referenced by the content, but they are not installed on the web server“)

Similarly features may be activated at site collection level and forgotten. In this case these features are not reported by the “stsadm –o enumallwebs” command and I have to run through every Site Collection in the farm to locate the features.

Remove Missing Setup Files

(AKA: “The following setup file(s) are referenced by the content, but they are not installed on the web server“)

After features have been uninstalled it is quite common that they forget to remove the files they installed on the farm. I suppose it’s just the way SharePoint works. If a feature installs a file in a document library somewhere changes are it will remain.

What are those files? Some examples are stuff in the style library, webpart gallery, master pages, page layouts, etc…

This method uses SQL to find where the missing files are still references and deletes them from the lists. It even deletes files that have been unghosted which could actually still be accessed because SharePoint has made a customized copy of it.

I’m still considering if it would be better to only remove the ones that are still ghosted.

Note: In my tests some page layouts could not be removed because they were still being referenced in the farm. That is clearly an error in my farm, however the script will not fix that. It really does require special handling by a human to decide what to do. I suppose it’s a good thing that they are not removed in this case as some things might otherwise break.

What it does not no

Out of scope of script:

  1. Fixing the OS version
  2. Fixing the database version

Sorry ;-)

How to Execute

First of all the script requires you to be site collection owner on every site collection, farm admin and have owner rights in every content and configuration database. Pretty much all you can get.

In the beginning of the script there is a few switches that you should know:

  • The logging level, set it to 3 to get lots of info: $verboseLevel = 3. Lower is less.
  • The “$whatif” ´switch. Set it to “$true” to have the script simulate a run but not change anything in your farm
  • The “$targetdb” that can limit the script to a single content database only. Blank equals all databases a name indicates one (note: Not the site name)

I’ve built in a lot of error handling and logging to let you know what’s going on.

Note that STSADM must be included in the path setting.

Recommended sequence of execution

Everything need to run on the SharePoint 2007 server and it must have (at least) PowerShell 1.0 installed.

  1. Backup all content databases. This is not optional. Easiest to use stsadm –o backup
  2. Use stsadm -o preupgradecheck
  3. Run script on one or all databases with the “whatif” statement set to true
  4. Check and recheck the output and verify that it looks correct to you.
    1. You may want to reduce the logging level a bit
  5. Run with $whatif set to $false, i.e. let it work!
    1. Depending on your confidence in me you can ask to work on only one database at a time
  6. Check the output!
    1. Errors and warnings will be written both to the output stream (what is piped down in your logfile) and on the screen
  7. Run preupgradecheck again and see the result J

Note to run the script use something like “powershell -file FixUpgradeIssues.ps1 >>fixupdateoutput.txt”, i.e. always pipe the output to a file that you can investigate later.

Runtime is about 5 minutes on my 40+ database test system so reasonably ok given the task.

I’ve been working on this script for a couple of months and have fixed a huge number of issues and funny edge cases; however I’ve likely missed a few. Let me know in the comments.

Good luck J

Download the Actual Script

All 850 lines of powershelling are not pretty in a blog post, so you can download it here.

[Updaet 25-12-2010: Now with correct file]

Fixing the Timer Service when everything breaks down


This is a pure troubleshooting post (that will save you days!) if you experience any of the following problems:

  • You have trouble deploying solutions to a your farm (more than one node) where nothing happens the deployment never completes
    • You may be able to complete deployments by executing “stsadm –o execadmsvcjobs” on every server in the farm
  • You experience frequent CPU spikes of 100% every few minutes on your frontend servers and they are all but unresponsive at those times
  • You find one of the following lines in the ULS logs (the SharePoint logs)
    • OWSTimer.exe, w3wp.exe (Id: 888k) : “File system cache monitor encountered error, flushing in memory cache”
    • OWSTimer.exe (Id: 8xqx): “Exception in RefreshCache. Exception message :Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.”
    • OWSTimer.exe (5utx): “The timer service could not initialize its configuration, please check the configuration database. Will retry later.”
  • “Some” of the administrative pages in the central administration occasionally fails with the error
    • “Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information”
  • The timer and topology cache is never updated (see next section…)
  • PSConfig (GUI wizard or not) fails to configure your server and the log file reveals the “LoaderExceptions” error above.

The above problems will probably affect all servers in the farm.

Note: The fusion log troubleshooting will be applicable to all .NET loader errors, not just SharePoint specific stuff.

First shot: Clear the Config Cache

A little known fact is that SharePoint maintains a disk cache (and memory) based configuration on every server that contains the topology information and timer job definitions. Go have a look at “C:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\<guid>\”.

Sometimes old stuff can get stuck in there and you can kick start the timer by clearing it.

The procedure is simple (do it for every server):

  1. Stop the Administration and Timer service
  2. Delete all XML files in the directory (not the folder itself)
  3. Open “Cache.ini” and write the number 1 instead of the existing number (you might want to make a note of it)
  4. Start the services again
  5. Wait for a minute or two and see if the folder starts to fill up with xml files. It is likely that it will contain less than before clearing it.
  6. Check the cache.ini file. If it’s accessible and the number is considerable greater than 1 your cache has been properly initialized and chances are that your problems are now fixed. It didn’t fix my problem, so you may need to read on… (if you didn’t have the “888k” log entry mentioned above you probably have it now)

The above procedure is grabbed from tomblog’s excellent post (that will also help if you actually did delete the folder too)

[Updated] Or you can run this batch file. [/Updated]

If the procedure didn’t fix the problem you’ll notice that the xml files are updated (timestamp) every few minutes coinciding with the CPU spikes.

Second shot: Digging in (Using the fusion log)

The core problem in my case was that some .NET class/assembly could not be loaded as the message “Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information” strongly hints. It may be a little surprising how many times an assembly, completely unrelated to the task at hand (e.g. deployment), is actually loaded. To fix the problem all we have to do is identify the assembly and “make it available”.

Enabling Fusion Log

You need to debug the failed assembly bindings you need to look into the fusion log. Fusion is an ancient (.Net 1.0) codename for the GAC or assembly loader (I think).

To enable the log you need to add/change three keys in the registry:

HKLM\Software\Microsoft\Fusion\LogPath    REG_SZ    (path to local directory)

HKLM\Software\Microsoft\Fusion\LogFailues    DWORD     1

HKLM\Software\Microsoft\Fusion\EnableLog    DWORD    1

It is not strictly necessary to restart anything, but I recommend that you now restart your timer service in order for it to log any binding errors that occurs. Try starting it a couple of times with 10 minutes in between. That should reveal any binding error patterns.

(Refer to Brad Adams for more info)

Interpreting the Log

Finally we will use the “fuslogvw.exe” program that is part of the .NET SDK to view the actual logs. Your development machine will have this file; copy the executable to your server.

It is not a very good program. It gets the job done, but it’s hard to figure out, you can’t order errors by date you can’t resize the window, etc..

Hopefully your window will contain a lot less entries (in my case the offending entry is the highlighted one. It also failed with similar errors from the psconfig wizard and powershell).

Look for patterns using the timestamps. Did you get a group of binding errors a few minutes after you started the timer service? Or are there just some errors that look spurious? I’ll recommend skipping the core internal microsoft dll’s to begin with (msvcm80 is the c runtime library version 8 that .Net 2.0 uses).

So what goes into the log? Every assembly binding failure, which is (at least):

  • If an assembly cannot be located (it will show you where it searched and any assembly redirects)
  • If the dll load methods within the assembly throws an exception – e.g. a dependent assemly could not be found – it sadly looks exactly like the dll file could not be found (other exceptions are possible but unlikely for .NET people). The one way to distinguish is to look for the line “LOG: GAC Lookup was unsuccessful.” if it is not there then it was (probably) found in the GAC and a dependent assembly failed.

The last bullet means that if assembly A depends on B and B could not be found then both the binding for A and B fails (at the same time). To distinguish the two I’ll recommend that you look at the “Calling Assembly” in the bind log.

In my case:

  1. The Nintex.Workflow dll failed to load and the calling assembly was Microsoft.SharePoint
  2. The Nintex.Workflow.Charting.dll failed to load and the calling assembly was Nintex.Workflow. Aha, so the Nintex.Workflow.dll was actually found but failed to find the dependent Nintex.Workflow.Charting assembly.
  3. Found the assembly on a development machine and copied it to the server. Retried. Added another missing assembly
  4. And everything worked!

I should stress that neither the error type nor the troubleshooting is Nintex workflow specific.

The bind log failure from step 2 above was:

*** Assembly Binder Log Entry (1/13/2009 @ 4:34:15 PM) ***

The operation failed.

Bind result: hr = 0×80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll

Running under executable C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN\STSADM.EXE

— A detailed error log follows.

=== Pre-bind state information ===

LOG: User = ….

LOG: DisplayName = Nintex.Charting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=913f6bae0ca5ae12

(Fully-specified)

LOG: Appbase = file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/

LOG: Initial PrivatePath = NULL

LOG: Dynamic Base = NULL

LOG: Cache Base = NULL

LOG: AppName = NULL

Calling assembly : Nintex.Workflow, Version=1.0.0.0, Culture=neutral, PublicKeyToken=913f6bae0ca5ae12.

===

LOG: This bind starts in default load context.

LOG: No application configuration file found.

LOG: Using machine configuration file from C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\config\machine.config.

LOG: Post-policy reference: Nintex.Charting, Version=1.0.0.0, Culture=neutral, PublicKeyToken=913f6bae0ca5ae12

LOG: GAC Lookup was unsuccessful.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting.DLL.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting/Nintex.Charting.DLL.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting.EXE.

LOG: Attempting download of new URL file:///C:/Program Files/Common Files/Microsoft Shared/web server extensions/12/BIN/Nintex.Charting/Nintex.Charting.EXE.

LOG: All probing URLs attempted and failed.

Note: To grab a dll from the GAC on one machine you need to use a shell (cmd, bash or powershell) to go into the c:\windows\assembly\gac_msil\… folder structure and copy it.

Why did it go wrong in the First Place?

It’s always a useful exercise to figure out what went wrong and why instead of just fixing the error at hand.

The root cause of the error was that the dll’s in questions were distributed through a wsp solution file which was referenced in other code. When that particular solution was undeployed the dll’s were removed (from the GAC) and the new updated wsp file only contained the new versions. Suddenly something went missing :-(

It can also happen if you use some auto wsp packaging procedures like vbs scripts or the ubiquitous WSPBuilder. What if one of the SharePoint system dll’s are marked as “copy local” on one (of the others’) local develop environment and automatically included in the wsp file? It will be deployed, no problems, but when undeployed everything stops as the SharePoint system dll just got removed from the GAC. Oops. Prevention is obviously to use a build server to make clean builds, to educate your people and to have the means to troubleshoot when it didn’t work out.

PowerShell – Friend to every MOSS developer


Recently I’ve made the effort to integrate PowerShell with MOSS in my development environment. I recommend that every one of you do the same :-)

Why?

Because you get the ability to jump right into your local farm and use the object model directly – no need to write small programs, and jump into the debugger, just to inspect if an SPItem’s folder object is null or not (note: It will be null for non-folder type items). What that means is:

  • It’s scripting that understands .Net. If you’ve ever had to do “normal” scripting you will love this. Every .Net object is accessible and usually very fast to prototype against.
  • You can very easily explore your farms stranger aspects and change what you don’t like, i.e. did you know that a number of lists are hidden and that you can easily make them visible by just flicking the switch?
  • It is a super fast way to prototype and/or troubleshoot errors
  • What is the current query for a given view?
  • What does your content type really look like?
  • What is really stored in a given items metadata? Was it null or was it an empty string? How should a lookup value look?
  • As with all scripting you get pretty much unlimited power. You can chain operations as you like, as complex as needed. You can write small procedures that batch updates pretty much whatever you like – but please be careful with this…
  • Objects can be manipulated through chaining, combine that with filtering/selection, and you got something that pretty much resembles Linq for SharePoint.

Why not? Because it’s hard. The learning curve is rather steep at first, which is why I’m posting some of my learnings and tips here. It will be much slower the first few times and it will be frustrating at times.

I’m currently using PowerShell only in my various dev and test environments, I do not do any batch scripting in my production environments. I simply don’t feel that confident (“with great power comes great responsibility” ;-) ) and besides it shouldn’t be necessary. Pretty much the same argument that restricts me from just accessing the content database directly and update the faulty content type on a couple of thousand documents though I know I can do it. What could possibly go wrong?

Either pessimism or experience…

Now I’ll not claim to be an expert PowerShell user, very far from it, and I will not create a full PS primer and explain everything to you. When you hit a problem google it and the help will surprisingly often be right there. How do I select the items where the creator is from my local domain? I used to not know, now I know that there is a “where-object” statement that will help me.

I suggest that you follow my simple steps below to get a crack at accessing the SharePoint OM and then head off to some of the PowerShell references at the bottom that will explain all the things I neglected (or simply don’t know).

Anyway, let’s get started with the first baby steps in PowerShell and then move on to the few scripts I created to make my life that much easier.

Getting Started

Install the damn thing. It’s easy, it’s a small download and it’s unfortunately not a portable application (my main complaint). Install from here.

PowerShell scripts use the “.ps1″ extension and be default PowerShell will not execute any script for you (what the hell?). To remedy that:

  1. Start powershell
  2. Type “Set-ExecutionPolicy RemoteSigned”

Now it will execute all local scripts and only remote scripts that have been signed. The long story is available by executing “get-help Set-ExecutionPolicy”.

Accessing MOSS

PowerShell knows .Net and as such is able to execute code to get you a SPFarm, SPSite or SPWeb in much the same way you do in C# (there are some syntax differences). The caveat here is that you’ll have to load/reference the SharePoint assembly into PowerShell first. It’s hard to remember how, but the correct statement is:

[Reflection.Assembly]::Load("Microsoft.SharePoint,
   Version=12.0.0.0,
   Culture=neutral,
   PublicKeyToken=71e9bce111e9429c")

The good news is that you don’t have to remember it, I’ll save it in the default profile so it’s always available, when I start PowerShell. Let’s also stuff a number of WSS specific methods in there while we’re at it.

What is the statement doing? It’s simply accessing the static Load method of the Reflection.Assembly object with the full assembly name of the Microsoft.SharePoint.dll, which is usually invoked for you by the .Net framework when you reference the assembly in your project. You can also use some of the other load methods, e.g. LoadWithPartialName, if you like that better.

Next I’ve stolen a script from Darrin Bishop that I heard speak at the SharePoint Conference in Seattle ’08. You really need to read his post that goes with the scripts as he explains a lot of powershell details that I’ve neglected:

[Reflection.Assembly]::Load("Microsoft.SharePoint,
     Version=12.0.0.0,
     Culture=neutral,
     PublicKeyToken=71e9bce111e9429c")
function Get-LocalFarm{
  return [Microsoft.SharePoint.Administration.SPFarm]::Local
}
filter Get-WebService{
   $webServices = new-object
           Microsoft.SharePoint.Administration.SPWebServiceCollection($_)
   $webServices
}
filter Get-WebApplication
   $_.WebApplications
}
filter Get-SiteCollection {
   $_.Sites
}
filter Get-Web {
   $_.AllWebs
}

These few, but rather obscure for the unenlightened, allows you to access all of the SharePoint object model in a somewhat convenient way. To use it you need to chain the methods (using the script pipe operator “|”), e.g. to get all the SPWebs in your site execute:

get-localfarm |get-webservice | get-webapplication | get-sitecollection | get-web

It will return an array of SPWeb objects and dump all their properties. It will be slow (but speed is hardly an issue here) and it will be very long. You can also filter it to get only the webs that has anonymous access enabled and format the result with a table with title, url and the site template used:

Get-localfarm |get-webservice | get-webapplication | get-sitecollection | get-web | Where-Object{$_.allowAnoymousAccess -eq $true} | format-table title, url, WebTemplate

Whew! That’s some PowerShell statement to just throw out. Key notes: 1) Use Format-table to save you to look through hundreds of lines and only show what you want to see, 2) Use Where-Object {} to query and use “$_” to access the current object in the script block. I haven’t listed any of the options where-object supports, but be careful not to use “=” for equality. In our case it would enable anonymous access on all webs. SharePoint fortunately (in this case) use the pattern of calling Update() when you need to commit changes, which you obviously should not call in this case. As I didn’t store the SPWebs I actually cannot even do that.

PowerShell 101

One of the best things to learn in PS is to use variables to save you from typing the same over and over all the time and to enable you to change properties through later Update() calls.

To store all the SPWebs from the previous example:

$webs = get-localfarm |get-webservice | get-webapplication | get-sitecollection | get-web

If the statement returns more than one SPWeb $webs will be an array. To get the first SPWeb type:

$webs[0]

To get a list of all methods and properties that the SPWeb supports, type:

$webs[0] | get-members

To iterate over each object in the pipeline use:

$webs | foreach-object { $_.Lists} | Format-table Name, Url

This will go through all the webs and all their lists and finally output the name and URLs in a simple table.

Combine the foreach statement with the where statement and then you can solve pretty much any batch update task. One fairly easy and useful task would be to write a one-liner that will publish all draft documents in every pages library. You might want to learn a bit more PowerShell first though ;-)

Accessing MOSS 2

I created the following 4 functions that I continue to use every time I need to access my local farm:

function Find-Webs([string] $urlpattern) 
   Get-localfarm | get-webservice | get-webapplication |
         get-sitecollection |
         get-web  | Where-Object{$_.url -like "$urlpattern"}
}
function Get-Site([string] $url){
    $site = New-Object Microsoft.SharePoint.SPSite($url)
    return $site
}
function Get-Web([string] $url){
    $u = New-Object Uri($url)
    $site = Get-Site($url)
    $web = $site.OpenWeb($u.AbsolutePath)
    return $web
}
function Select-ItemFields($item) {
    #primitive formatting
    $item.Fields | ForEach-Object { new-object
       System.Collections.DictionaryEntry($_.title, $item[$_.Title] ) }
}

Find-Webs will locate any web that matches the url pattern, so to get any of my “news” webs (in the entire farm):

$news = Find-Webs(“*/news”)

Usually I use it with a specific enough url to get at only one specific site. It is a bit slow because it relies on the above methods from Darrin Bishop that iterates through all web apps/site collections/sites and it does require you to be local admin on your farm (which you probably are anyway).

Get-Site and Get-Web will obviously get a SPSite or SPWeb object given a full url. It is faster and almost as easy to use as Find-Webs.

Select-ItemFields is a small method that will list every metadata field of a given SPListItem. I simply got tired of writing

$item["field1"]

to read specific fields ($item holds the SPListItem). I found no really easy way to iterate over the values in the default indexer of a given object, so I just wrote the few lines required. I’m sure there is a PowerShell wizkid out there that will say, Idiot! Use the built-in in Select-XXX instead. Please speak up; I’ll like to know…

These four functions actually cover my MOSS PowerShell needs. I keep coming back to them to get the first hook into the OM and then perform the task at hand (often a small test) on a given list item, content type, field etc.

Your mileage may vary, so by all means add 10 more convenience methods.

Note: To test my method, just paste the content of the entire script file into your PowerShell window – you don’t need to create fancy scripts or setup profiles.

Making it Convenient

Finally I like stuff to be convenient and easy.

To that end I’ve stuffed all of the above scripting in my profile file, so it’s always loaded when I start PowerShell, in particular the Microsoft.SharePoint.dll is loaded (download my profile script file here. Note: Strip away the “.gif” extension – crappy wordpress restrictions).

To add it to your profile:

  1. Go to your “My Documents” folder
  2. If there is no directory named “WindowsPowerShell” create it and navigate to it
  3. Create/edit a file named “profile.ps1″ and paste my script into it

That’s it! When you launch PowerShell the script should be loaded. To verify, look for the assembly loaded confirmation message, when you start PowerShell. And of course: Execute some of the functions!

One last tip: F7 is your friend. Try it.

References

Darrin Bishops original scripts. He has also written an additional blog post that will help you with filtering.

The basics are coveraged better than I could possibly do on webmat’s blog.

Deeper coverage, when you need to know what’s really going on: Learn PowerShell in Your Launch Break Day 1 and Day 2 (and just keep going…)

And finally, given a problem or a difficult task ask the experts ;-)

Audit your Content Type Hierarchy


Shortly after you deploy a SharePoint site you probably start to wonder how your carefully designed content types are actually being used. And abused. More importantly I want to know if any of my feature based content types have been disconnected (unghosted) from the XML source file.

To that end I looked into extracting and visualizing the content type hierarchy to get the overview – and I found that it was already built by some nice chaps at codeplex (can be found here)! The community works like never before ;-)

I changed it in a number of ways (in my mind improved it):

  • The original only showed the content types declared at the site collection root, now I use a completely different way of finding inherited content types through the SPContentTypeUsage classes, to be able to show derived content types in sub sites as well
  • Using the SPContentTypeUsage will also give you derived list content types, so they can be displayed as well. There might be a large number of lists, so you can switch the list content type display on and off. Default is off.

    Caveat: To figure out whether a given (list) content type is a direct descendent of another I have to open the actual list content type and check. Some lists are special (like _catalogs) and will be rendered slightly off in the hierarchy (hierarchy “missing” but the list content type location will be there)

  • Added my very own detection of the ghosting/unghosting status of any given content type. There are three states: ghosted (connected to xml source), unghosted (disconnected from xml source) and “DB only” (created through the UI or OM, not from a feature)

At the end of the day pretty much all of the code has been changed, while keeping the layout. I’m still grateful for the very nice first step provided and I will post this code back to the codeplex site. Hopefully the guys there will accept the modified code.
[Updated April 14: Patch posted now]

Detecting Ghosted/Unghosted State

I did a lot of digging to figure out how to get at the ghosting status of a content type. You can have a look in the wss content database (table “ContentTypes”) to see if your content types are disconnected or not (column “definition” is null – see my other blog on this). There’s even a dedicated stored procedure that will tell you the ghosting state of a given content type (named “proc_IsContentTypeGhosted”).

I really did not want to directly call stored procedures on the database, so I had a good long hard look at the properties available through the SPContentType class (and many others through reflector) and came up with the following conditions for detecting the state:

  • Is it a feature based content type, by checking “FeatureID != null”, if not mark it as “DB only”. Note that this is actually a private field, so I have to go the long way around and fetch it through reflection.
  • If it is feature based on the version > 0 then it has been unghosted. The version number seem to work well for this. It will always be 0 for ghosted content types and be incremented by one every time somebody modifies (and thereby unghost it). According to the xml schema you can specify this field in your content type definition, but it will (fortunately) not make it to the database so the detection seems sound.

    If you make new versions of the xml file, the server won’t really notice (see post here).

Disclaimer: This algorithm works well for me, seems reasonable sensible, but there might be an edge case that I’m not aware of.

The Result

Here is what it looks like:

conenttypehierarchy.png

The page is rather long – you are seeing about half of it. It’s also slow, but that’s hardly a performance issue you should care to fix ;-)

The Code

You need to create a feature to deploy the page, the feature.xml I use:

<Feature xmlns="http://schemas.microsoft.com/sharepoint/" Id="2D9921AE-D263-4e2b-B4F7-ABDD6223C8B0"
   Scope="Site"
   Title="My Administration Utilities (Site collection)"
   Description="My adminsitration tools at the site collection level">

   <ElementManifests>
      <ElementManifest Location="ContentTypeHierarchy.xml" />
   </ElementManifests>
</Feature>

The referenced ContentTypeHierarchy.xml file (will create a link in the Gallery section of site settings):

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
<CustomAction Id="ContentTypeHierarchy"
   GroupId="Galleries"
   Location="Microsoft.SharePoint.SiteSettings"
   Sequence="0"
   Title="Site Content Types Hierarchy"
   Description="Displays a hierarchy of content types.">

   <UrlAction Url="_layouts/ContentTypeHierarchy.aspx"/>
</CustomAction>
</Elements>

And finally the magic page, ContentTypeHierarchy.aspx (deployed to the TEMPLATE\Layouts folder) (you can download it here, you’ll have to rename it back to aspx):

<%@ Page Language="C#" MasterPageFile="~/_layouts/application.master" %>

<%@ Assembly Name="Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral,PublicKeyToken=71e9bce111e9429c" %>
<%@ Import Namespace="Microsoft.SharePoint" %>
<%@ Import Namespace="Microsoft.SharePoint.WebControls" %>
<%@ Import Namespace="System.Data.SqlClient" %>
<%@ Import Namespace="System.Reflection" %>
<%@ Register TagPrefix="wssuc" TagName="ToolBar" Src="~/_controltemplates/ToolBar.ascx" %>
<%@ Register TagPrefix="wssuc" TagName="ToolBarButton" Src="~/_controltemplates/ToolBarButton.ascx" %>
<asp:Content ID="Content2" runat="server" ContentPlaceHolderID="PlaceHolderPageTitleInTitleArea">
	Content Types Hierarchy
</asp:Content>
<asp:Content ID="Content3" runat="server" ContentPlaceHolderID="PlaceHolderPageDescription">
	This page shows all Site Content Types and their hierarchical relationships
</asp:Content>
<asp:Content ID="Content4" runat="server" ContentPlaceHolderID="PlaceHolderMain">

	<table border="0" width="100%" cellspacing="0" cellpadding="0">
		<tr>
			<td id="mngfieldToobar">
				<wssuc:ToolBar id="onetidMngFieldTB" runat="server">
					<template_buttons>
					  <wssuc:ToolBarButton runat="server" Text="<%$Resources:wss,multipages_createbutton_text%>" id="idAddField" ToolTip="<%$Resources:wss,mngctype_create_alt%>" NavigateUrl="ctypenew.aspx" ImageUrl="/_layouts/images/newitem.gif" AccessKey="C" />

						<table cellpadding="1" cellspacing="0" border="0">
							<tr>
								<td class="ms-toolbar" nowrap style="padding:0px">
									<asp:CheckBox runat="server" CssClass="ms-toolbar" Text="Show lists" id="cbList" AutoPostBack="true" />
								</td>
							</tr>
						</table>                

						<table cellpadding="1" cellspacing="0" border="0">
							<tr>
								<td class="ms-toolbar" nowrap style="padding: 0px">
									<asp:CheckBox runat="server" CssClass="ms-toolbar"  Text="Show descriptions" id="cbDesc" AutoPostBack="true" Checked="true" />
								</td>
							</tr>
						</table>
					</template_buttons>
				</wssuc:ToolBar>
			</td>
		</tr>
	</table>
	<asp:PlaceHolder ID="phHierarchy" runat="server" />
</asp:Content>

<script runat="server">
    Hashtable contentTypesShown = new Hashtable();

    protected override void OnLoad(EventArgs e)
    {
		base.OnLoad(e);

		try {
			EnsureChildControls();
			cbDesc.CheckedChanged += new EventHandler(CheckedChanged);
			cbList.CheckedChanged += new EventHandler(CheckedChanged);
			BuildHierarchy();
		}
		catch (Exception x) {
			OutputHtml(x.Message);
		}
	}

	void CheckedChanged(object sender, EventArgs e) {
		BuildHierarchy();
	}

	public void BuildHierarchy() {
		EnsureChildControls();
		phHierarchy.Controls.Clear();
        contentTypesShown.Clear();
		SPSite site = SPControl.GetContextSite(Context);
		SPContentTypeCollection types = site.RootWeb.ContentTypes;

		SPContentTypeId id = types[0].Id;
		OutputHtml("<table style='font-size:10pt' border='0'" + " cellpadding='2' width='100%'><tr><td><ol>");
		ShowContentType(types[0]);
		OutputHtml("</ol></td></tr></table>");
	}

	public void OutputHtml(string html) {
		Literal l = new Literal();
		l.Text = html;
		phHierarchy.Controls.Add(l);
	}

	public void ShowContentType(SPContentType currenttype) {
		try {
			OutputHtml("<li><a class='ms-topnav'" +
				" href=\"" + currenttype.Scope.TrimEnd('/') +
				"/_layouts/ManageContentType.aspx?ctype=" +
				currenttype.Id.ToString() + "\">" + currenttype.Name +
				"</a>" + ExtraInfo(currenttype) +
				(cbDesc.Checked ? "<span class='ms-webpartpagedescription'>" + currenttype.Description + "</span>" : "") +
				"</li>");

			SPContentTypeId id = currenttype.Id;

			OutputHtml("<ol>");

			System.Collections.Generic.IList<SPContentTypeUsage> usages = SPContentTypeUsage.GetUsages(currenttype);

			int listCount = 0;
            foreach (SPContentTypeUsage usage in usages)
            {
                if (!contentTypesShown.ContainsKey(usage.Url + "|||" + usage.Id))
                {
                    contentTypesShown.Add(usage.Url + "|||" + usage.Id, true);

                    if (!usage.IsUrlToList)
                    {
                        using (SPWeb subSite = SPControl.GetContextSite(Context).OpenWeb(usage.Url))
                        {
                            SPContentType subType = subSite.ContentTypes[usage.Id];
                            if (subType.Parent.Id == id && subType.Parent.Id != subType.Id)
                            {
                                ShowContentType(subType);
                            }
                        }
                    }
                    else if (usage.IsUrlToList && cbList.Checked)
                    {
                        //1. Find web + list
                        //2. Open content type
                        listCount++;

                        Match m = Regex.Match(usage.Url, "^(?<web>.*?/)(Lists/)?(?<list>[^/]+)$");
                        if (m.Success)
                        {

                            try
                            {

                                using (SPWeb subSite = SPControl.GetContextSite(Context).OpenWeb(m.Groups["web"].Value))
                                {
                                    SPContentType subType = subSite.Lists[m.Groups["list"].Value].ContentTypes[usage.Id];
                                    if (subType.Parent.Id == id && subType.Parent.Id != subType.Id)
                                    {
                                        OutputHtml("<li>(List) <a class='ms-topnav'" +
                                            " href=\"" + m.Groups["web"] + "_layouts/ManageContentType.aspx?ctype=" +
                                            usage.Id.ToString() + "&list=" + subType.ParentList.ID.ToString() + "\">" + subType.Name + "</a> " + usage.Url + "</li>");
                                    }
                                }
                            }
                            catch (Exception e)
                            {
                                //exceptions occur for some of the "special" list, e.g. catalogs, then just list the list ;-)
                                OutputHtml("<li>(List) " + usage.Url + "</li>");
                            }
                        }
                    }
                }
            }

			OutputHtml("</ol>");
		}
		catch (Exception x) {
			OutputHtml("<b>Error:" + x.ToString() + "</b>");
		}

	}

	string ExtraInfo(SPContentType type) {
		string info = "";

		//Ghosted?
		// if the content type came from a feature and version > 0 then it's ghosted (as
		// far as I've been able to determine).
		// Version will not be greater than 0 for non-ghosted content types.
		// Alternative is to query DB directly (proc_IsContentTypeGhosted).

		//Need to find the featureId (if any) through reflection, since it's not a
		// public property)
		System.Reflection.PropertyInfo featureIdProp = type.GetType().GetProperty("FeatureId", BindingFlags.NonPublic | BindingFlags.Instance, null, typeof(Guid), Type.EmptyTypes, null);
		object featureId = featureIdProp.GetValue(type, null);

		if (type.Version > 0 && featureId != null && ((Guid)featureId) != Guid.Empty) {
			info += "Storage: <b>UNghosted (DB)</b>|";
		}
		else if (featureId != null && ((Guid)featureId) != Guid.Empty) {
			info += "Storage: Ghosted (Disk)|";
		}
		else {
			info += "Storage: DB only|";
		}

		if (type.ReadOnly) {
			info += "ReadOnly|";
		}
		if (type.Sealed) {
			info += "Sealed|";
		}
		info += "Scope:\"" + type.Scope + "\"|";

		info = info.TrimEnd('|').Replace("|", ", ");
		if (info.Length > 0) {
			info = " <i>(" + info + ")</i> ";
		}
		return info;
	}
</script>

Remember to reference the files in your solution manifest file and you should be good to go :-)

Summary

It works and I’m very happy with it. It is an invaluable tool to debug all sorts of content type related problems. Big thanks to the guys at codeplex for letting me hit the ground running on this.

What’s not in it? Site columns. Didn’t find a good way for it and the need seems limited.

Finally understand that this is not really an audit in the normal SharePoint sense, so you should also consider to enable auditing on content type changes as well. Even when you have locked down security properly accidents do happen (only for your fellow admins of course).

”Not enough storage” event log error


[Note: Updated Feb. 22 2008, solution at the bottom] 

I’m responsible for a couple of SharePoint 2007 (MOSS) farms where all SharePoint servers showed a number of annoying errors in the application event log.

Every minute the following three errors show up in the event log:


Event Type: Error
Event Source: Windows SharePoint Services 3
Event Category: Timer
Event ID: 6398
Date: 3/6/2007
Time: 11:47:58 AM
User: N/A
Computer:
Description:
The Execute method of job definition Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob (ID 371548ff-a05e-41f0-90da-6f2d25fbb483) threw an exception. More information is included below.


Not enough storage is available to process this command.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

And


Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 7076
Date: 3/6/2007
Time: 11:47:58 AM
User: N/A
Computer:
Description:
An exception occurred while executing the Application Server Administration job.


Message: Not enough storage is available to process this command.
Techinal Support Details:
System.Runtime.InteropServices.COMException (0×80070008): Not enough storage is available to process this command.


Server stack trace:
at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail)
at System.DirectoryServices.DirectoryEntry.Bind()
at System.DirectoryServices.DirectoryEntry.get_IsContainer()
at System.DirectoryServices.DirectoryEntries.CheckIsContainer()
at System.DirectoryServices.DirectoryEntries.Find(String name, String schemaClassName)
at Microsoft.SharePoint.AdministrationOperation.Metabase.MetabaseObjectCollection`1.Find(String name)
at Microsoft.SharePoint.AdministrationOperation.Metabase.MetabaseObjectCollection`1.get_Item(String name)
at Microsoft.SharePoint.AdministrationOperation.SPProvisioningAssistant.ProvisionIisApplicationPool(String name, ApplicationPoolIdentityType identityType, String userName, SecureString password, TimeSpan idleTimeout, TimeSpan periodicRestartTime)
at Microsoft.SharePoint.AdministrationOperation.SPAdministrationOperation.DoProvisionIisApplicationPool(String name, Int32 identityType, String userName, String password, TimeSpan idleTimeout, TimeSpan periodicRestartTime)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)
[…continues…]

And

Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared Services
Event ID: 7076
Date: 3/6/2007
Time: 11:47:58 AM
User: N/A
Computer:
Description:
An exception occurred while executing the Application Server Administration job.


Message: Not enough storage is available to process this command.


Techinal Support Details:
System.Runtime.InteropServices.COMException (0×80070008): Not enough storage is available to process this command.


Server stack trace:
[…continues…]

Sometimes they are replaced with another three that has the same event id and source, but with the error message being “Old format or invalid type library” instead. The stack trace will differ a little bit.

Obviously it’s a timer job scheduled every minute that fails. The one in question is “Application Server Administration Service Timer Job”, which apparently is in charge of ensuring that the IIS application pools are in sync (or something like that). Nothing seems to be broken in the farm by the job failures.

And quite annoyingly: They all disappear when you reboot the server and will not reappear until after “a few days” in my case. I’m sure that the re-surface time will differ between systems.

I’ve read many proposed solutions for this error, including adding ram, disc etc., often claimed to work. I seriously doubt that any of these solutions actually work. My SharePoint servers are equipped with no less than 6gb of ram with default settings for all the application pools (I know that the ram is hardly utilized with these settings, but in my world it’s sometimes cheaper to go for a “standard server” where you only utilize 70% than one customized for your needs), plenty of disc etc.

As a side note you can also find references to this error in connection with SQL server 2005, the fix below possibly also work for that as well.

The solution turns out to be quite easy – the patch you are looking for is kb923028. It is an update for an error in the .NET 2.0 remoting subsystem, and has actually nothing to do with SharePoint at all. Reading the description it is quite hard to glean that it’ll solve your problem. MS support pointed me to it and it seems to work just fine.

Caveat: I’ve seen multiple versions of this file, the one I have working (until proven wrong) is “NDP20-KB923028-X86.exe” (1,936,224 bytes). I’ve tested another one with the same name with a filesize of 1,963,440 bytes that didn’t work.

The server has been chugging along for some time now (about a week) without the bug so let’s hope it’s all done.

That didn’t work…

Update: Finally a resolution

You need to look at hotfix KB946517, which will fix the problem. It is a private hotfix, so you’ll need to contact MS acquire it.

The servers have been running for about a week now, they are still ok and other people are also reporting success.

For the third time I’m confident that the problem has been solved – guess I don’t learn from experience ;-)

Follow

Get every new post delivered to your Inbox.