Use CSOM from PowerShell!!!

The SharePoint 2010/2013 Client Object Model (CSOM) offers an alternative way to do basic read/write (CRUD) operations in SharePoint. I find that it is superior to the “normal” server object model/Cmdlets for

  1. Speed
  2. Memory
  3. Execution requirements – does not need to run on your SharePoint production server and does not need Shell access privileges. Essentially you can execute these kind of scripts with normal SharePoint privileges instead of sys admin privileges

And to be fair inferior in the type of operation you can be perform. It is essentially CRUD operations, especially in the SharePoint 2010 CSOM.

This post is about how to use it in PowerShell and a comparison of the performance.

How to use CSOM

First, there is a slight problem in PowerShell (v2 and v3); it cannot easily call generics such as the ClientContext.Load method. It simply cannot figure out which overloaded method to call – therefore we have to help it a bit.

The following is the function I use to include the CSOM dependencies in my scripts. It simply loads the two Client dlls and creates a new version of the ClientContext class that doesn’t use the offending “Load<T>(T clientObject)” method.

I nicked most of this from here, but added the ability to load the client assemblies from local dir (and fall back to GAC) – very useful if you are not running on a SharePoint server.

$myScriptPath = (Split-Path -Parent $MyInvocation.MyCommand.Path) 

function AddCSOM(){

     #Load SharePoint client dlls
     $a = [System.Reflection.Assembly]::LoadFile(    "$myScriptPath\Microsoft.SharePoint.Client.dll")
     $ar = [System.Reflection.Assembly]::LoadFile(    "$myScriptPath\Microsoft.SharePoint.Client.Runtime.dll")
    
     if( !$a ){
         $a = [System.Reflection.Assembly]::LoadWithPartialName(        "Microsoft.SharePoint.Client")
     }
     if( !$ar ){
         $ar = [System.Reflection.Assembly]::LoadWithPartialName(        "Microsoft.SharePoint.Client.Runtime")
     }
    
     if( !$a -or !$ar ){
         throw         "Could not load Microsoft.SharePoint.Client.dll or Microsoft.SharePoint.Client.Runtime.dll"
     }
    
    
     #Add overload to the client context.
     #Define new load method without type argument
     $csharp =     "
      using Microsoft.SharePoint.Client;
      namespace SharepointClient
      {
          public class PSClientContext: ClientContext
          {
              public PSClientContext(string siteUrl)
                  : base(siteUrl)
              {
              }
              // need a plain Load method here, the base method is a generic method
              // which isn't supported in PowerShell.
              public void Load(ClientObject objectToLoad)
              {
                  base.Load(objectToLoad);
              }
          }
      }"

    
     $assemblies = @( $a.FullName, $ar.FullName,     "System.Core")
     #Add dynamic type to the PowerShell runspace
     Add-Type -TypeDefinition $csharp -ReferencedAssemblies $assemblies
}

And in order to fetch data from a list you would do:

AddCSOM()

$context = New-Object SharepointClient.PSClientContext($siteUrl)

#Hardcoded list name
$list = $context.Web.Lists.GetByTitle("Documents")

#ask for plenty of documents, and the fields needed
$query = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery(10000, 'UniqueId','ID','Created','Modified','FileLeafRef','Title') 
$items = $list.GetItems( $query )

$context.Load($list)
$context.Load($items)
#execute query
$context.ExecuteQuery()


$items |% {
          Write-host "Url: $($_["FileRef"]), title: $($_["FileLeafRef"]) "
}

It doesn’t get much easier than that (when you have the AddCSOM function that is). It is a few more lines of code than you would need with the server OM (load and execute query) but not by much.

The above code works with both 2010 and 2013 CSOM.

Performance Measurement

To check the efficiency of the Client object model compared to the traditional server model I created two scripts and measured the runtime and memory consumption:

Client OM:

param 
(
[string]$listName = $(throw "Provide list name"),
[string] $siteUrl = $(throw "Provide site url")
)

AddCSOM

[System.GC]::Collect()
$membefore = (get-process -id $pid).ws

$duration = Measure-Command {

          $context = New-Object SharepointClient.PSClientContext($siteUrl)
         
          #Hardcoded list name
          $list = $context.Web.Lists.GetByTitle($listName)
         
          #ask for plenty of documents, and the fields needed
          $query = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery(10000, 'UniqueId','ID','Created','Modified','FileLeafRef','Title') 
          $items = $list.GetItems( $query )
         
          $context.Load($list)
          $context.Load($items)
          #execute query
          $context.ExecuteQuery()
         
         
          $items |% {
                  #retrieve some properties (but do not spend the time to print them
                  $t = "Url: $($_["FileRef"]), title: $($_["FileLeafRef"]) "
          }
         
}

[System.GC]::Collect()
$memafter =  (get-process -id $pid).ws

Write-Host "Items iterated: $($items.count)"
Write-Host "Total duration: $($duration.TotalSeconds), total memory consumption: $(($memafter-$membefore)/(1024*1024)) MB"

Server OM:

param 
(
[string]$listName = $(throw "Provide list name"),
[string] $siteUrl = $(throw "Provide site url")
)

Add-PsSnapin Microsoft.SharePoint.PowerShell -ea SilentlyContinue

[System.GC]::Collect()
$membefore =  (get-process -id $pid).ws

$duration = Measure-Command {
          $w = Get-SPWeb $siteUrl 
          $list = $w.Lists[$listName]

          $items = $list.GetItems()
          $items |% {
                  #retrieve some properties (but do not spend the time to print them
                  $t = "url: $($_.Url), title: $($_.Title)"
          }
}

[System.GC]::Collect()
$memafter =  (get-process -id $pid).ws

Write-Host "Items iterated: $($items.count)"
Write-Host "Total duration: $($duration.TotalSeconds), total memory consumption: $(($memafter-$membefore)/(1024*1024)) MB"

And executed them against a document library of 500 and 1500 elements (4 measurements at each data point).

The Results

Are very clear:

OMChart

As you can see it is MUCH more efficient to rely on CSOM and it scales a lot better. The server OM retrieves a huge number of additional properties, but it has the benefit of going directly at the database instead of the webserver. Curiously the CSOM version offers much more reliable performance where the Server OM various quite a bit.

In addition you get around the limitation of Shell access for the powershell account and the need for server side execution. Might be convenient for many.

Conclusions

The only downside I can see with the CSOM approach is that it is unfamiliar to most and it does require a few more lines of code. IF your specific need is covered by the API of course.

It’s faster, more portable, less memory intensive and simple to use.

Granted, there are lots of missing API’s (especially in the 2010 edition) but every data manipulation need is likely covered. That is quite a bit after all.

So go for it J

Advertisements

About Søren Nielsen
Long time SharePoint Consultant.

13 Responses to Use CSOM from PowerShell!!!

  1. maxui says:

    Hi

    I am trying to use CSOM to download a file from SharePoint Online doc Library with no success. everything works fine up until the actual download portion
    I wonder if you would be so kind as to have a look over on StackOverflow and give me a pointer or two? http://stackoverflow.com/questions/18866958/downloading-a-file-from-sharepoint-online-with-powershell

    would be Much appreciated

    • Certainly.

      I’ve failed to find a good way to do it in the CSOM, but then again the solution was not that hard.
      I used the code below, to retrieve a bunch of files (metadata from CSOM queries) to a folder:

      #$siteUrlString site collection url
      #$outPath path to export directory
      #$collection hashtable where values are (CSOM) listitems

      $siteUri = [Uri]$siteUrlString
      $client = new-object System.Net.WebClient
      $client.UseDefaultCredentials=$true

      if ( -not (Test-Path $outPath) ) {
      New-Item $outPath -Type Directory | Out-Null
      }

      $collection.Values |% {
      $url = new-object Uri($siteUri, $_[“FileRef”])
      $fileName = $_[“FileLeafRef”]
      $outFile = Join-Path $outPath $fileName
      Write-Host “Downloading $url to $outFile”

      try{
      $client.DownloadFile( $url, $outFile )
      }
      catch{
      #one simple retry…
      try{
      $client.DownloadFile( $url, $outFile )
      }
      catch{
      write-error “Failed to download $url, $_”
      }
      }
      }

      If your list of files to download originate from somewhere else, you’ll have to build the full url yourself (the $url part above).

  2. Soeren,
    Do you believe the PowerShell CSOM method could work against SharePoint Online (wave 15/SPO 2013)? I’ve been looking for a way to programmatically modify properties of document libraries in SharePoint Online. Thank you for your time and efforts.

  3. Mark says:

    Thanks for this example Soeren….It works great from a 2010 SharePoint machine where the CSOM assemblies are loaded from the GAC. I am however running into trouble when trying to run this on a non-SharePoint with the CSOM assemblies in the same directory as the Powershell script. Add-Type is failing because it can’t find the Metadata files for the CSOM assemblies couldn’t be found. Research suggests switching the reference assemblies to use the path on disk vs. their FullName didn’t yield any results either.

    Did you run into this? Have any advice?

    • yes, you are on the right path. The solution us to simply use the disk path for the reference assemblies. however if you lack any of their dependencies you end up with more or less the same error 😦
      In that case you need to look into assembly bind failures. An old alias for the GAC is the fusion log, therefore you need to enable fusion log to see what dlls windows is searching for (and where) and add them to your package. It’s probably enough to copy them to the same location as the other dlls. if they still cannot be found you can add these dependent dlls as referenced assemblies before the sharepoint ones.
      the tool i use for this is usually “fuslogvw.exe”. search and copy it from your dev machine to the executing machine. One word of caution: when you are done don’t forget to disable the logging.

    • Anatoli says:

      Hi guys,

      same problem here? Did you solve it and if yes, how?

      Greetings…

  4. Jesper says:

    This performance comparison here is in favor of CSOM because it executes a caml query whereas the server object model does not.
    SPList.GetItems does take a SPQuery argument. It would execute a DB query on less fields and lower the memory consumption.

    • I disagree.

      I use no query fields. There is no “where” in the caml query so it should return all elements in the list.

      The difference comes from CSOM only loads the specific fields I ask for and the OM returns the full object with every property initialized. If I had used a caml query on the server that would have been used to select all items, but when they are fetched again all props are initialised.

      I could of course just opt for using CSOM even when I’m running code on the server to reap the benefits – and I sometimes actually do just that.

      • Mr Harrison says:

        I agree with Jespers opinion. To really perform the same task in both APIs, you would need to pass the viewfields for the GetItems Method in the Server OM as well.
        Furthermore the item count should be less than 10000, otherwise you should create an SPQuery in the Server OM and set the Itemlimit also to 10000.

        The the results would be compareable.
        Are you sure that your methode to measure memory consumption is a valid method?

  5. Pingback: SPCAF | SharePoint Online Client Object Model and PowerShell: Three tips

  6. Pingback: SharePoint Online Client Object Model and PowerShell: Three tips by @mattein

  7. Pingback: SharePoint Online Client Object Model and PowerShell: Three tips - Rencore

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: