Access Denied When trying to crawl MySites

This happened to me recently. I was configuring a SharePoint 2013 farm with an auto installer. Had the configuration almost complete, and then I noticed that Search was throwing an error crawling my sites by looking at the crawl log.

sps3://[MySiteUrl]

Access is denied. Verify that either the Default Content Access Account has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled. ( HttpStatusCode Unauthorized The request failed with HTTP status 401: Unauthorized. )

I took a peak around the site and everything looked ok. I then looked at the Manage Web Applications screen in Central Admin. To do this, select the web application, then click on User Policy in the ribbon. Make sure that the search crawl account has a full read on all web applications that you want it to index. In my case, even this was set correctly.

I then started poking around some more and went to the User Profile Service application and I found my answer. To do this, go to the service applications screen, highlight the User Profile Service application (not the proxy), by clicking to the right of the link. Once highlighted, click on the Administrators link in the ribbon, then add the search crawl account to this list and give it the 'Retrieve People Data for Search Crawlers'. Hit ok and do a full crawl. Your People search should work now. Happy SharePointing.

How to Correctly Backup and Restore Term Store DB, Content Type Hub and Content Database

I have run across this scenario a few times so I thought I would write a guide for others to follow. The need for this comes up when companies have multiple environments such as Prod, Staging, Test, Dev etc. Almost invariably, I have seen that the Prod environments are the most up to date, not only in terms of content but also in terms of configuration of the SharePoint farm.

So lets assume that your organization is using the Content Type Hub to syndicate and push content types to the different subscribing site collections. This involves the use of the Managed Metadata Service which directs the publishing of the content types and also allows for the term sets to be used in the various SharePoint web sites. Lets also assume that your sites use Managed Metadata columns that are mapped to the term sets in the term store. Every once in a while, you want to pull data back to your other environments for periodic refreshes and testing. I have been asked how to do this by a few clients so I will lay it out here.

To do this correctly, you will need to back up at least 3 databases from your production farm, depends on how many databases your hub and content sites are using. I am assuming 1 per web application here. So, backup the Hub database, the content database or databases which you want to bring back, and the managed metadata database that contains all the term sets and terms. What is the managed metadata database, you ask. To find that out, you have to go to Service Applications, highlight the Managed Metadata Service application and then click properties. You will see the name of the database on this screen.

Bring those backed up databases to the target environment SQL Server, ready to be restored. As a best practice first I make backups of all three or more databases in the target environment that I am about to replace. Then I turn of the SP Timer service to make sure the timer jobs don't try to push stuff while I am restoring term sets and the Hub. Then I override the MMS destination database with the MMS database I just backup from Prod. This brings back all the term sets and terms. I also disconnect the Hub and the target web application content databases using CA or stsadm or powershell.

Note: To view this go back to the properties screen we were just on for the Managed Metadata Service Application. This brings back the connection to the content type hub in production, which is what we don't want. We want this URL to point to our target environment (Staging, Test etc). It is not possible to change this URL from the UI, we have to change it using powershell.

The Powershell command is as follows:

Set-SPMetadataServiceApplication -Identity "<ServiceApplication>" -HubURI "<HubURI>"

Once the managed metadata service looks good and all the term sets and terms have come through, then we can move on to the hub and content databases. Restore the hub and content databases as different names on the target farm so they dont override. Make sure that the permissions match on the newly restored database and the old database that we just disconnected. This is just a practice I follow. Then attach the hub web application to the newly restored hub database from prod. You might have to redo permissions on the newly restored site collections through CA so you can access them.

Finally, restore and attach all the content databases that you need to. After all this is done, do an IISReset and start the timer service. This will start the two timer jobs that push and subscribe to the content types from the new hub.

There you go, all done! Verify to make sure that your MMS columns work correctly in your web applications and that your content types are being pushed down correctly from your local hub.

Remember password with ISA Server

Here was a scenario I ran into recently.  A client had an ISA server 2006 deployment and used forms authentication against AD for logging in their customers. A long time request had been that users did not want to enter their username and password every time, they wanted the browser to remember their credentials.

After talking the customer through the fact that this was not a good idea from a security standpoint and not succeeding in dissuading them, I started to look for the solution. We took a look at the ISA settings on the server, but couldn't find anything in this regard. I was chatting to a colleague about this who had been making modifications to the ISA server login page to customize the HTML the end user would see, and he pointed out that there is a form option on the login page that turns auto complete on or off. Here is the piece of HTML that causes this:

form autocomplete="off" id="logonForm" method="post" action="/CookieAuth.dll?Logon"

Setting autocomplete="on" then allows the browser to save the username and password credentials.

A pattern for building custom forms in SharePoint 2010

Recently, I have been working on a pretty large enterprise intranet project on SharePoint 2010 that involves building a few modules that meet the business needs. These modules each involve a few related content types. The way we architected these was to have a main core content type for each module – and have a few other content types have a lookup column back to that core content type. There were some one-one relationships as well as one-many relationships in the mix. Given that there were these related entities, there is no OOB way to allow for a seamless experience for an end user to go through the entire process of creating the core entity and all related entities without needing to know how they are stored in custom lists. There were also event receivers that needed to be created in these modules that would do sub-tasks, set permissions on items, send email etc.

One of our key design goals from the onset was to make the user experience as friendly as possible while at the same time accounting for any changes/additions to fields to the content types after the project was delivered. So we did not want to build fully custom forms, because those would be tied strongly to the schema as we knew it during development and would need a UI change and deployment to add any new field to the custom pages

.

Here is an example of content types in one specific (relatively simple) module:

The approach we took was making extensive use of list field iterators in SP 2010. These are available in the SP API and are fairly easy to use once you get the hang of these. So these would allow us to point the ListFieldIterator to a list in SharePoint and set its mode to New, Edit or Display. What this would do is open up the list item in one of those modes. The iterator iterates through all the fields in that list and surfaces them to you, just like in an OOB SharePoint list form. If we needed to edit or display we would also bind the list item ID to the iterator and set the corresponding mode. If the user did not have permissions to edit the list item, we would show the iterator in the display mode.

Here is how the new/edit form looked with the iterators.

We used the Tab control in the AjaxControlToolkit to manage the UI using different iterators in the tabs. We also employed the modal framework quite extensively to manage sub types as shown below. Here is the management screen for audit tracking.

On clicking new audit tracking entry, the user would get this screen as show below. Notice the Tab control and how the result looks like an OOB list form. It is easy to bind the iterator to the list and the item and set its mode (New, Edit or Display). The buttons below are custom and thus the saving has to be done by you as well.

You can click the new Audit Finding link to create a new sub type. Here is that screen.

As you can see above, this really is a SharePoint modal dialog showing yet two more sub content types in their own lists and related to the main core Audit content type. The modal framework is really handy for doing these sort of things and improving the user experience vastly. Imagine if the user had to go to 5-6 different lists to create a new audit, to say that would not fly would be an understatement!

Here is some code that highlights how to bind these. Here is the code to declare the iterator.

In the code behind, this is how you bind it.


SPList AuditTrackingList = SPContext.Current.Web.Lists[Constants.AUDIT_CORE_LIST_NAME];
ListFieldIteratorAuditTracking.ListId = AuditTrackingList.ID;
//Set the mode as well depending on whether you are opening an existing one to display or new one. lets go with new for here
ListFieldIteratorAuditTracking.ControlMode = SPControlMode.New;
//Could also be SPControlMode.Edit or SPControlMode.Display

Voila, your iterator binds to the list item and shows it just like SharePoint would.

To update/save, in your Button_Save event (obviously this is oversimplified):

 protected void ButtonSave_Click(object sender, EventArgs e)
        {
            if (this.Page.IsValid)
            {

                 SPItem item = ListFieldIteratorAuditTracking.Item;
                  item.Update();
            }
        }

Enjoy!!




 

Blog Comments were not posting

My apologies, I just realized there were many blog comments that were awaiting moderation. For some reason, blogger did not email me about these so I could not reply. I will try to answer the ones that are still relevant, and I am going to take off moderation for now until I start getting spam on the comments.

SharePoint Conference 2011 - Final Thoughts

So here I am, sitting at the airport going back to Denver. It’s been a whirlwind week here in Anaheim, CA. We had the RBA consulting private party last night which was attended by quite a few clients as well as Microsoft folks. Jared Spataro, who delivered the keynote at the conference and Matt Berg who ran the conference, was also in attendance which was pretty cool. I did chat for quite a while with Bill Baer from Microsoft, who owns the SharePoint Foundation product. We talked about Remote Blob Storage strategies (he had just presented earlier that evening on that topic), as well as where SharePoint is and where it is going. I also had the opportunity to talk with Ted Pattison, Andrew Connell and Todd Baginskiwho apart from being MVP’s have been involved with SharePoint for a while.

So here are my personal thoughts about the writing on the wall at this year’s conference. The next wave of SharePoint is going to be a revolution, not an evolution in terms of how we do things and think about the development paradigm and practices. Code/customizations you do in the current release will continue to work in the next release, but they will not utilize the benefits of the new platform. So what technologies can you sharpen your skills on to get ready for the next release? The few areas in my mind are:

1.      SharePoint Online + Azure

There were some good sessions on SharePoint Online and its integration with Azure. With Azure, you can host your custom applications and databases up in the Microsoft cloud and access them that way. Another major advantage of the Azure model is that it has a service bus that can interact with data into your organization’s data center. This, coupled with the ability of SharePoint Online to talk to Azure provides you a model to get back into your data sitting back in your datacenter. This also leads into #2 below, because the way to connect with Azure is through client side object model (sandboxed code cannot call outside the site collection). Another big announcement was that BCS will be supported in the public cloud soon.

2.       jQuery/Javascript and other client side technologies.

jQuery is a literally a set of Javascript libraries that allow you to do things way more efficiently than just using plain Javascript. jQuery also works across browsers seamlessly – so you write it once and it works as expected. There were a few sessions that talked about jQuery (with HTML5, best practices etc.)

3.       Social Aspects

There were a few sessions that talked about the social features of SP 2010. I didn’t have the opportunity to attend any of these but this is one area that Microsoft will continue to invest heavily in – so this will keep getting better and better.

A note on HTML5 vs Silverlight

Ted Pattison covered a session on jQuery with HTML5. HTML5 looks pretty promising with its ability to build pretty cool UX elements – but in terms of the browser experience it still has quite ways to go. The HTML5 specifications are not fully developed yet, so you have to embed different tags for every browser. Plus, if you want to target older browsers then it’s even more effort required. So bottom line, this is coming but not ready for primetime for the browser experience.

SharePoint Conference 2011 Day 2

Day 2 of the SharePoint conference was more of the same as day 1, except that it was jam packed with sessions all day. There were three notable sessions that I attended that I want to talk about.

The first session was our very own Phil Wicklund’s session on SharePoint in the Cloud: Architecture. This session was very informative and talked about the different cloud architectures supported by Microsoft, from the SharePoint Online shared to SharePoint Online Dedicated (which is hosted by Microsoft), to a hybrid scenario where some of your SharePoint installation could be hosted on premise and some in the cloud. Phil made very interesting points about which scenario you would address your business needs. He also talked about the search experience and how to store your user profiles and tag clouds so they are aggregated and available. Great session to understand what feature set is supported in each scenario. Tomorrow Phil presents on managing identity in SharePoint online so that will be a great session as well.

Another session I attended was Ted Pattison’s JQuery and HTML5. This session also contained good information about how to get started in jQuery and provided some examples of how effective it is. Ted had a demo built out of different things that can be done with it and showed some code samples for the same. The interesting part is that jQuery works on every browser just the same so you only have to code it once. Ted also talked about HTML5 and where this is heading vis-à-vis Silverlight. From the session I gathered that HTML5 is not ready for prime time yet, there is work required to make it work with the different latest browsers (due to lack of standards), but it is even more work to make it function in older browsers. So, Silverlight is still the way to go for a delightful user experience, but HTML is the far future in terms of browsers. Using HTML5 for building mobile apps across the different mobile platforms, that is another question altogether..

Another session I attended was Todd Bleeker’s Building a custom service application in SharePoint 2010. I have to admit it was one of the most animated technical presentations I have ever attended. He was very excited about what he was presenting, so much so that it was really hard to not get caught up in it. He did a great job explaining the details and the plumbing behind building service applications in SP 2010. It was a very technical session and I thoroughly enjoyed the content and the presentation of that content.  I would definitely recommend checking Todd out for understanding the service application model and building a custom service application.

Going to the SPC 2011 conference at Disneyland tonight (the happiest and most magical place on earth). More till later.

SharePoint Conference 2011

We are at the SharePoint Conference 2011 this week, also known as SPC 2011. It is hosted in Anaheim this year. It is a mid-release conference so we didn’t expect to hear many surprises at the keynote or at the sessions. The keynote was hosted by Jeff Spataro, who is the senior director of SharePoint Product management. Though an engaging speaker, the one thing that struck me about the keynote was that there were no surprises, no announcements etc. The most interesting part of the keynote was a demonstration of new failover capabilities of SharePoint 2010 and SQL Server “Denali”.

After the keynote I attended a couple of sessions that day. One of them was titled Documents are boring but document solutions are not. It was a pretty good session and the speaker showed off information panel within a word document with an Infopath form embedded in the document. He also showed how fields in the word document can be tied to columns on the document library, and Word will update these everytime it is opened. The interesting part is that these fields could also be bound to information from external sources via the BCS.  The speaker also showed content controls, quick parts, co-authoring of documents and how Word detects and informs users that another person is editing the document. Another good technique is to always render a field code with a link to a picture or chart stored in the library. This way the image/chart is not stored in the document but can be refreshed from the library at any time.

The other engaging session I attended was Andrew Connell’s Out of the sandbox and into the cloud: Build you next SharePoint app on Azure. It does seem like that is one of the directions Microsoft is not only investing heavily in but also starting to push to the developer community. It was a real engaging session and AC talked about the different ways of interfacing Azure data in SharePoint 2010. The session was very informative and engaging. Definitely recommend checking out his presentation over video if you have the time. Scott Key from Microsoft also presented on the next topic the very next day. The key takeaway is that you can store your custom applications and custom databases in Azure (it is easier and cheaper) and build hooks into SharePoint to consume/surface that.

Todd Baginski also presented that evening about building SharePoint apps for the Windows phone. That was also a highly engaging session and Todd covered a lot of material and shared what he has been working on lately. If you are interested in that topic definitely recommend checking out his talk. He also laid out the process of interfacing with Twitter/Facebook/Linkedin etc fromyour custom apps.

So all in all, it was a good first day. There were tons of parties/happy hours/ packed together so we were out till the wee hours of the morning. More on the conference soon.

View Application Error on SharePoint page

When working with SharePoint 2010, when there is an error in the application you may come across the generic SharePoint error page that states to turn the customErrors tag to RemoteOnly or Off in the web.config. So you diligently locate the web.config file for the web app, and turn the customErrors to one of those settings based on your level of comfort.

The generic page still does not go away, and you can't see the error. What gives? Well, it appears that there is another web.config file in the LAYOUTS directory that has the customErrors set to On. This is overriding the web application web.config setting. So setting this customErrors to Off or RemoteOnly will show you the real error so you can take action on it!

After Installing SharePoint 2010 Farm, it says Web Server needs to be upgraded

I came across this a couple of times recently so I figured I would blog about it. I've been working quite a bit on SharePoint 2010 projects, and usually I am called upon to do farm installs. I recommend that clients always do least privileged installs, so that means advising them on the accounts they need to create for their (dev/test/uat/prod farms) and best practices regarding those. Here is the list of accounts recommended for use on technet for a least privileged install.

Account	Purpose	Requirements
SQL Server service account	The SQL Server service account is used to run SQL Server. It is the service account for the following SQL Server services: MSSQLSERVER SQLSERVERAGENT If you do not use the default SQL Server instance, in the Windows Services console, these services will be shown as the following: MSSQL$InstanceName SQLAgent$InstanceName	Use either a Local System account or a domain user account. If you plan to back up to or restore from an external resource, permissions to the external resource must be granted to the appropriate account. If you use a domain user account for the SQL Server service account, grant permissions to that domain user account. However, if you use the Network Service or the Local System account, grant permissions to the external resource to the machine account (domain_name\SQL_hostname$). The instance name is arbitrary and was created when Microsoft SQL Server was installed.
Setup user account	The Setup user account is used to run the following: Setup SharePoint Products Configuration Wizard	Domain user account. Member of the Administrators group on each server on which Setup is run. SQL Server login on the computer that runs SQL Server. Member of the following SQL Server security roles: securityadmin fixed server role dbcreator fixed server role If you run Windows PowerShell cmdlets that affect a database, this account must be a member of the db_owner fixed database role for the database.
Server farm account or database access account	The server farm account is used to perform the following tasks: Configure and manage the server farm. Act as the application pool identity for the SharePoint Central Administration Web site. Run the Microsoft SharePoint Foundation Workflow Timer Service.	Domain user account. Additional permissions are automatically granted for the server farm account on Web servers and application servers that are joined to a server farm. The server farm account is automatically added as a SQL Server login on the computer that runs SQL Server. The account is added to the following SQL Server security roles: dbcreator fixed server role securityadmin fixed server role db_owner fixed database role for all SharePoint databases in the server farm

Ok, makes sense. So you login to the app server with the setup account, and install SharePoint. Then you do the same on one or more Web Servers. Now go back to the app server and run the Configuration Wizard and set up the farm and Central Admin. After some configuration (setting up services, the first site collection), you come back to the Web Servers and attempt to add them to the farm. The whole process goes smoothly and the Web Server connects to the farm. Great, now you go to the Servers in Farm page in Central Admin, and to your dismay the Web Server line item shows an error and that it needs upgrading.

What?? So you run the 'stsadm -o localupgradestatus' on the Web Server and it shows you as everything being ok. Hmm..

Resolution
The way I found to get around this is to remove the Web Server from the farm, then log off the Web Server. Log back on using the Server farm account, not the setup account. Now run the Configuration Wizard and add the Web Server back to the farm. This works well and now when you go to the 'Servers in Farm' page, everything looks good!