Posted by: Eric Schrader | January 11, 2016

SharePoint DevOps Part 1 – Setup CentOS with Ansible

This is the first post in a series for creating a Linux host with Ansible to control Windows machines and install SharePoint.

Let me preface this with I have no clue how far I will get. I expect this to take a few weeks if it’s possible. Ansible seems to be growing, and this all might not be fully baked out. We will see! Feel free to comment.

  1. Part 1- Setup CentOS with Ansible (This post)
  2. Part 2- Setup Ansible with Windows Machines (Coming soon)
  3. Part 3- Using Ansible to prepare Windows Machines for SQL Server (Coming soon)
  4. Part 4- Using Ansible to install SharePoint using AutoSPInstaller (Coming soon)
  5. Part 5- Using Ansible to Maintain SharePoint Machines (Coming soon)

Introduction

There are various technologies I have attempted to use in the past for Windows Deployment SharePoint automation using AutoSPInstaller, such as:

  1. Chocolatey– I could not find enterprise worthy packages for SQL Server and SharePoint. I also couldn’t figure out a way to provision Azure VMs from it.
  2. PowerShell DSC – Way to overkill with push/pull when I just wanted to configure a new SharePoint farm
  3. Chef– Have not tried it. I personally can’t get over the names of things relating to items in the kitchen. But I believe this is a standard. Check it out
  4. Azure WebHooks– Runs PowerShell, possibly limited, on Azure VMs. Note each VM needs an endpoint for each runbook. I have 60 dev VMs/farms that I manage, and I wanted a different runbook for different parts of the SharePoint configuration process, this wasn’t going to happen.
  5. Azure RunBooks– Lots of great tutorials for building a SP Farm, but the farm was not fully configured. I prefer AutoSPInstaller and couldn’t figure out a way to merge them.

Now my above experience is very limited, as when I would hit a wall I would pretty much see how far the technology is and if it’s not supported I would give up. Also, the above experiences are a hybrid of on-premises SharePoint installation needs and Azure IaaS hosting SharePoint needs. So if this does not work, I will be going back to the drawing board or the above list. I have not used Microsoft System Center Orchestrator, as I rarely provision new Hyper-V VMs, and heard it is a ton of work to configure.

My Goal with Automation

I have a very well defined steps I have documented to install SharePoint 2013 farms with one or more servers, for dev/qa or production. These can be in Azure or on-premises in Hyper-V. I want something lightweight that I can provision a new VM, join it to the domain, create a domain if needed, run Windows Update, configure OS rules, install SQL Server (different versions), install SharePoint via AutoSPInstaller, patch SharePoint, and configure services. I would also like to set services and verify those rogue developers did not change any system settings (just kidding team). I know automation does not replace planning, but I hope to turn documentation and PowerShell scripts into a custom deployment tool I can use for provisioning future environments as well as maintaining existing environments. When I read the Phoenix Project, they brought up a good point of “If a family dog gets injured, you nurse it back to health, but if one of your cattle get injured, you will be having beef for dinner”, which basically means why are you spending 100 hours fixing an environment, when you can just recreate or replace it with another? Of course, developers and users change system settings, but the core of the machine can be recreated via a script in less than a day, rather than troubleshooting something and possibly not solving the issue.

Getting Started with Ansible

I picked up a book on Amazon called Ansible for DevOps by Jeff Geerling but shortly into the book realized it was only for Linux based machines. Ansible does support Windows targets, but the commands must be ran from a Linux OS. Dang. I don’t know Linux. So now I am writing this blog post. And it begins. Let’s see what Ansible can do for Windows targets.

Installing Linux – CentOS on a Windows Hyper-V host

  1. Download Cent OS7
    1. https://www.centos.org/download/
    2. I choose Torrent option, as the mirrors were pretty slow. Torrent went at 6mbps download and finished 4GB in about 10 minutes.
  2. Create new VM

    1. Choose a location for the VM file

    2. Choose Generation1

    3. Choose a fixed amount of ram

    4. Choose your Hyper V NIC

    5. Choose your VHD path/info for a new blank VHD to be created

    6. Choose the CentOS ISO file downloaded from the first step

    7. Turn on the VM!
    8. Boot to Cent OS install

Install CentOS on VM (then Python/Ansible)

  1. Select Language


  1. Choose software selection (Choose Server with GUI unless you know how to use Linux terminal well) I also selected the Development/Security tools, and MariaDB (I saw MariaDB in the Ansible book example and figure this will save me some steps later for Ansible testing)


  2. Choose disk


  3. Enable Ethernet and choose a hostname for the computer


  4. Verify everything looks good:


  5. While the OS is installing, configure a root and local user account. Root is like a local server admin password, which we will be using. The user account is a username and password, which you will be logging into each time you start the VM


  6. Select the blue Reboot option when the install is complete.

Log In to Cent OS

  1. Log in to CentOS
    1. License Agreement
      1. Hit 1 to read it, 2 to accept, c to continue, and c again to continue (I kinda struggled with this part)


    2. Sign in using the username and password you set up:


      1. Accept language, keyboard layout, and skip cloud accounts if desired. Then click Start using Linux!


  2. Run Terminal


Install Python/Ansible on CentOS

  1. Install Python
    1. Type SU then hit Enter in terminal to enter the root admin window


    2. Enter root password (different from user password, you entered it in setup)
    3. Once in root, install Python.
      1. Type: sudo yum install epel-release


      2. Hit Y to continue (twice)
      3. Verify complete

  2. Install Ansible
    1. Type sudo yum install ansible


    2. Select Y to continue (twice)

    3. Test ansible command to verify install is complete:
      1. Type: ansible –version


      2. You should get back a version number.

RESOURCES

Thanks to this article for the Python install help I was able to figure out how to install it without errors on CentOS7. Here are the same commands over again, just together without screenshots:

http://stackoverflow.com/questions/32048021/yum-what-is-the-message-no-package-ansible-available

$ su

$ sudo yum install epel-release

$ sudo yum install ansible

If you are a SharePoint admin or use, you have probably seen this error message on your SharePoint 2013 MySite Newsfeed:

“We’re still collecting the latest news. You may see more if you try again a little later.”

sharepoint 2013 mysite company newsfeed error -Were still collecting the latest news

Hopefully this article will help explain a few common scenarios I have ran into, and how to resolve any errors. Please post if you have any tips or suggestions, as I am always looking for thoughts from the community on this.

This article will address:

  • Common reasons for newsfeed data not displaying
  • What is distributed cache?
  • Distributed cache configuration
  • Shutdown/Reboot WFE procedure for maintenance so you don’t lose your cache
  • Repopulating cache (if server stopped unexpectedly)
  • References

Common reasons for newsfeed data not displaying

  1. Someone rebooted all your Distributed Cache servers at the same time
    1. Check the task manager uptime to see how long the servers with distributed cache have been running, or if they were rebooted unexpectedly
    2. Fix: Repopulate cache using PowerShell, or maybe wait a long time for new news
    3. Prevent it from happening again: Shut down one server at a time, stopping the cache first, rebooting, and then starting the cache again.
  2. “Everyone” is empty because it only keeps company conversations for 14 days by default.
    1. Fix: Increase the retention time, or encourage people to post to the company newsfeed (not site newsfeeds). See this article for more information on what appears in site newsfeeds vs company newsfeeds. https://support.office.com/en-za/article/What-items-appear-in-your-newsfeed-bd3d9268-0408-4ad4-bc51-2e4ec5406e16#__toc327280723
  3. Distributed cache is not configured right
    1. Fix: Configure it right J this one is so simple, yet so difficult I find. See configuration below.

What is distributed cache?

Distributed cache is a framework Microsoft uses to quickly host social information in SharePoint within the SP servers ram. This can be enabled on one or many SP servers in your farm.

Official definition can be found on this poster, https://www.microsoft.com/en-us/download/details.aspx?id=35557

What uses distributed cache?

Pretty much anything social, but some of the social data comes from content databases and user profile databases. Company newsfeed posts are stored in distributed cache.

  • Newsfeeds
  • Authentication
  • OneNote client access
  • Security Trimming
  • Page load performance

Distributed cache configuration

Note: Run any scripts/commands logged in as the SPFarm account, and be sure to run SharePoint Management Shell as administrator if you have UAC enabled (like a good administrator)

Caution: configuration deletes the cache, so you will need to repopulate the cache after configuring it.

Determine servers to host the distributed cache service

Usually the WFE servers, not servers running search or excel. AutoSPInstaller has a limit of 4 servers, but typically it does not configure distributed cache correctly.

Configuring Distributed Cache

There is a good article here on these commands, https://technet.microsoft.com/en-us/library/JJ219613.aspx and probably better than this article I am writing. But it’s very long so I wrote this article to get Admins 90%-100% of the way there.

Here is how I have been configuring distributed cache. Thanks Jon for the help!

  1. Use Central Admin or PowerShell to start/stop the SharePoint Distributed Cache service on the desired servers in your farm (usually WFE’s).
    1. Or you can use PowerShell to get/start-spserviceinstance of Distributed Cache on the desired servers. I like to use PowerShell to see what servers are running Distributed Cache within my farm:
      1. Get-SPServiceInstance | where-object {$_.typename -ilike “*distributed*”}
  2. Verify Cache service is running on desired servers:
    1. Use-CacheCluster
      1. Note, this command doesn’t configure the server, but just connects the current PowerShell session to manage the cache cluster it’s joined for the PowerShell management session. It’s actually an alias command for Connect-AFCacheClusterConfiguration.
    2. Get-CacheHost
    3. You should see each server running distributed cache listed above. If not, there might be more work to configure the cache cluster I may have missed in this post. Let me know!
  3. Get current configuration
    1. Get-AFCacheHostConfiguration -ComputerName wfe01 -CachePort “22233”
    2. The Cache Size can be updated, see guide here https://technet.microsoft.com/en-us/library/JJ219613.aspx#memory . For 16GB of ram on our WFE servers, we go with 819MB (~5% of 16GB). Note, changing this requires the distributed cache service to be stopped on the computer you are changing it on. Update-SPDistributedCacheSize -CacheSizeInMB 819
  4. Export config, verify service account for distributed cache, as well as servers.
    1. Export-CacheClusterConfig -Path C:\test.xml
      1. Check max cache size (default is 5% -, no more than 4GB – size depends on services on the server)
      2. Check servers – ensure only WFE (or desired servers are in the cluster)
      3. Check service account – ensure all servers use the same service account (spservice)
      4. Check ports
  5. Warning: After configuration is complete do not ever run Add-SPDistributedCacheServiceInstance or Remove-SPDistributedCacheServiceInstance. It reconfigures the cluster (and usually incorrectly)

Shutdown/Reboot WFE procedure

If you have to do reboots on the WFE for Windows Updates, etc., you might be expecting to lose your Newsfeed cache. Here is the proper procedure to retain the cache.

Summary: shutdown the cache one server at a time, reboot that server, add the server back into the cache cluster. Repeat on next server.

  1. Verify Cache service is running on desired servers (more than one server too is key):
    1. Use-CacheCluster
    2. Get-CacheHost
  2. Reboot SQL Server first if needed. Get this out of the way.
    1. Wait for SQL Server to come back online
  3. Reboot WFE1
    1. Perform these commands on WFE1
    2. Verify Cache service is running on desired servers (more than one server too is key):
      1. Use-CacheCluster
      2. Get-CacheHost
    3. Run Stop-spdistributedcacheserviceinstance -graceful
    4. Verify Cache service is stopped on WFE1. Ensure it is stopped before proceeding:
      1. Get-CacheHost
    5. Reboot WFE1
    6. Verify Cache service is running on WFE1:
      1. Use-CacheCluster
      2. Get-CacheHost
      3. If not, go to Central Admin Services on Server and start Distributed Cache service on WFE1, or use PowerShell.
  4. Reboot WFE2
    1. (Repeat above Step #3, but replace WFE1 with WFE2)
  5. Verify it is running
    1. Verify Cache service is running on desired servers (more than one server too is key):
      1. Use-CacheCluster
      2. Get-CacheHost

Repopulating cache (if server stopped unexpectedly)

Replace URL with your mysite URL. This script will populate each user’s cache using Update-SPRepopulateMicroblogFeedCache and the entire user profile newsfeed cache using Update-SPRepopulateMicroblogLMTCache. I am not sure if I stole this script from anywhere, but part of it is from various user profile scripts adapted to fix the users feed cache.

Note: Run any scripts/commands logged in as the SPFarm account, and be sure to run SharePoint Management Shell as administrator if you have UAC enabled (like a good administrator). Otherwise you will get a .ctor error that will drive you crazy.

Download the script from here: http://pastebin.com/K9yR2pEk

$proxy
= Get-SPServiceApplicationProxy | ? {$_.Name -ilike
“User Profile Service Application*”}

Update-SPRepopulateMicroblogLMTCache -ProfileServiceApplicationProxy $proxy

[System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.Office.Server”)

[System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.Office.Server.UserProfiles”)

$url
=
http://mysiteurl.domain.com

$contextWeb
=
New-Object
Microsoft.SharePoint.SPSite($url);

$ServerContext
= [Microsoft.Office.Server.ServerContext]::GetContext($contextWeb);

$UserProfileManager
=
New-Object
Microsoft.Office.Server.UserProfiles.UserProfileManager($ServerContext);

$Profiles
=
$UserProfileManager.GetEnumerator();

foreach ($oUser
in
$Profiles ) {


if ($oUser.item(“SPS-PersonalSiteCapabilities”).Value -eq 14 ){


$personalurl
=
$url
+
$oUser.item(“personalspace”).Value


Write-Host
$oUser.item(“AccountName”).Value

Update-SPRepopulateMicroblogFeedCache -ProfileServiceApplicationProxy $proxy -accountname $oUser.item(“AccountName”).Value


#-siteurl $personalurl

}

}

$contextWeb.Dispose()

After running the script on each WFE where distributed cache runs, wait 15 minutes for the Newsfeed data to populate.

Then test newsfeed.

References

http://consulting.risualblogs.com/blog/2014/04/01/export-impor-distributed-cache-configuration-in-sharepoint-2013/

http://sharepoint.stackexchange.com/questions/125798/userprofileapplicationnotavailableexception-logging-userprofileapplicationpro

http://netwovenblogs.com/2014/03/11/the-newsfeed-is-not-working-on-mysite-in-sharepoint-2013/

Posted by: Eric Schrader | August 3, 2015

SharePoint PowerShell to audit list columns

Recently, I had to list find where all of the SharePoint site columns are used in lists across many site collections. I wrote some quick PowerShell to get all SharePoint list columns and write the site collection, list, and column.

Columns: I listed all site columns/list column display names I was seeing if they are used

Sites: all site collections, except my sites

I am sure it can be cleaned up, but it’s quickly to audit the list columns to get a report. I tried to export to CSV but decided it was more time than formatting the few results that came back in the window.

Results: (can be cleaned up easily, but I don’t have time for this quick script)

AuditColumns.ps1

SPSite Url=https://client/sites/docs, Accounting Documents, Document Type

SPSite Url=https://client/sites/docs, Meeting Minutes, Document Type

SPSite Url=https://client/sites/docs, Documents, Document Type

SPSite Url=https://client/sites/docs, Announcements, Document Type

Script:

[system.reflection.assembly]::loadwithpartialname("microsoft.sharepoint")
$sites = @("https://client",
"https://client/sites/docs");

$cols = @("Document Type",
"Another Column Name",
"My Column",
"Find Me");
foreach($siteurl in $sites){
$site = get-spsite $siteurl
$web=$site.OpenWeb()
$lists=$web.Lists
foreach($list in $lists) {
$fields = $list.Fields
foreach($col in $cols){
if($fields.title -contains $col){
write-output "$site, $list, $col"
}
}
}

I recently had an issue with a new Windows 8.1 laptop not able to open Visio files from our SharePoint 2013 intranet. The error is similar to the 32bit/64bit mixed environment errors. My co-worker Rod and I found a workaround that might not be best practices, but resolved the issue. Please post any comments if there is an industry best standard to resolve the error aside from reformatting.

SharePoint 2013 Intranet error with Office Pro Plus on Visio 2013 64bit: Microsoft Visio- Sorry, we can’t perform this action. Incompatible Office products are installed on your machine. If you have an administrator, please contact them for help. OK

Microsoft Visio- Sorry, we can't perform this action. Incompatible Office products are installed on your machine. If you have an administrator, please contact them for help. OK

Terminology:

  • OneDrive for Business- SharePoint file sync tool
    • Blue cloud icon
  • OneDrive (Personal)- Not discussed anywhere in this post. Ignore personal OneDrive here, its 100% separate from SharePoint.
    • White cloud icon

My setup:

  • OS: Windows 8.1 Enterprise 64 bit
  • Computer: Dell Latitude E5550 touch screen laptop, Intel i7 2.6GHz, 8GB Ram
  • Office: Office 365 ProPlus 15.0.4727.1003
  • Visio Version: Visio Professional 2013 15.0.4569.1506
  • OneDrive for Business Updates:
  • Internet Explorer Version: Internet Explorer 11 Version: 11.0.9600.17842
    • SharePoint intranet site is added as Local Intranet, default security settings.

Short term solution: I noticed if I ended the OneDrive for Business process in task manage, I would then end the Microsoft Office Upload Center process. After these were ended (in that order only) I could open Visio files from the intranet.

Before I did this short term fix, other users at my company with the same application versions would try to launch OneDrive for Business and get the same “Sorry, we can’t perform this action” message with OneDrive for Business, but they could open Visio files from the intranet. However, I could launch only OneDrive for Business, but not Visio from the Intranet file link. So when I did this fix, I could launch Visio from the SharePoint 2013 intranet file and view my Visio file, and when I would open OneDrive for Business, I would get the same error as other users, “Sorry, we can’t perform this action”. Good, now I am following the company standard. BUT, as soon as I reboot, same issue comes back and I can’t open Visio files from the SharePoint intranet.

Long term solution: (NOTE, this might not be best practices, but it solves my issue. If you have a Microsoft supported alternative, please post it here in the comments). Instead of ending the processes above, go to Startup tab of the Task Manager and disable OneDrive for Business. This might also disable the child Office Document Cache under it, but it resolves my issue. Summary: OneDrive for Business conflicts with Microsoft Office Document Cache and it might be a 64bit/32bit issue, not sure, but disabling OneDrive for Business on startup resolves the issue for me. I will post if there are any noticeable side effects of file offline caching, stale cache, or other errors. I think this happened to me only because I configured OneDrive for Business to resolve this issue when I first got my laptop, not sure why the laptop originally had the issue. Again, if there’s a better way, please comment below. Thanks!

Posted by: Eric Schrader | June 23, 2015

Azure VM PowerShell audit VHD disk information

I was having trouble getting a list of VHDs that are in use by my Azure VMs for OS and Data disks. I wanted to list them in a single line CSV to audit each VMs configuration. The issue was I had 3 commands with different objects:

  • Get-AzureVM
  • Get-AzureOSDisk
  • Get-AzureDataDisk

I wanted each output on the same line. I was able to rename the conflicting properties and output them to a single CSV, with each VM on a new line.

Script: (download)

$vms = Get-AzureVM -ServiceName mycloudservicenamehere
foreach($vm in $vms) {

#get-azurevm
$obj1 = $vm | Select-Object Name,InstanceSize,AvailabilitySetName,DNSName,deploymentname
#get VM Data disk, rename same properties to prevent output conflicts
$obj2 = $vm | Get-AzureDataDisk | Select-Object @{Name=’DataDiskName’; Expression={$_.DiskName}},@{Name=’DataDiskLabel’; Expression={$_.DiskLabel}},@{Name=’DataMediaLink’; Expression={$_.MediaLink}},@{Name=’DataDiskSizeGB’; Expression={$_.LogicalDiskSizeInGB}},@{Name=’DataLUN’; Expression={$_.Lun}}
#same as above VM data disk, but with OS disk
$obj3 = $vm | Get-AzureOSDisk | Select-Object  @{Name=’OSDiskName’; Expression={$_.DiskName}},@{Name=’OSMediaLink’; Expression={$_.MediaLink}},@{Name=’OSDiskLabel’; Expression={$_.DiskLabel}}, OS

#combine these outputs into a single row CSV per VM
$combined = New-Object -TypeName PSObject
$obj1,$obj2,$obj3|%{
$CurObj = $_;
$_|gm|?{$_.MemberType -match “NoteProperty”}|%{
$NewMember = $_.Name;
$Combined|Add-Member -MemberType NoteProperty -Name $NewMember -Value $CurObj.$NewMember
}
}
#write each VM output to CSV
$Combined|Export-CSV -Path c:\Users\administrator\Desktop\output.csv -NoTypeInfo -Append -Force
}

Output csv:

Output properties: Notice how I renamed conflicting disk properties to be prefixed with “Data” or “OS”.

  1. AvailabilitySetName
  2. DeploymentName
  3. DNSName
  4. InstanceSize
  5. Name
  6. DataDiskLabel
  7. DataDiskName
  8. DataDiskSizeGB
  9. DataLUN
  10. DataMediaLink
  11. OS
  12. OSDiskLabel
  13. OSDiskName
  14. OSMediaLink

Now I can look at VM Instance Size, Labels, OS/Data disk information, etc. and make any changes. Our team has about 40+ SharePoint development VMs in multiple cloud services, so provisioning/management can be a bit of work. In a perfect world I would use PowerShell DSC or Azure Automation, but that is quite a bit of work to setup for VMs we scrap every few months once projects are complete.

Next steps:

  • Split VMs into unique cloud services so multiple developers can start/stop VMs at the same time (worker process conflict) (using same site-to-site VPN)
  • Audit OS/Data disk configuration and possible move between storage accounts
  • Set affinity groups for disks/VMs

Links:

Older Posts »

Categories

Follow

Get every new post delivered to your Inbox.

Join 185 other followers