Scale Out a SharePoint 2010 Farm From Two-Tier to Three-Tier By Adding A Dedicated Application Server

Hi All,
In this post we will discuss how to scale out from two-tier to three-tier in sharepoint 2010.
Here is a screen shot of all of the service applications that run on the SharePoint 2010 server in a two-tier farm when you install SharePoint Server 2010 Enterprise edition and run the out-of-the-box Configure Your SharePoint Farm Wizard and choose to provision all service applications:

(Note: for this post, I am working under the assumption that you have used the SharePoint 2010 “Configure your SharePoint Farm” wizard and have opted for it to provision all of the SharePoint Server 2010 **Enterprise Edition** service applications)

Your goal is to add a third server to the SharePoint 2010 farm and have it take over running all of the service applications in the list above, with the exception of the three that have been circled.  The three that have been circled in the screen shot are the ones that are necessary for the original server to function as a dedicated WFE with query processing.

The Search Query and Site Settings Service and some of its associated functionality in the SharePoint Server Search Service are technically not required on a WFE, but it is the best place to put them.  The reason is that this is the process that takes the user’s search query and looks it up in the indexes.  The indexes are files that the query processor needs local access to and are stored on the file system of the server(s) that is running the query service, not in SQL Server.

So, for best performance it is recommended to run the Search Query and Site Settings Service on the WFEs that are serving the pages.  The crawling and index process is a separate process whose job it is to build the indexes and push them up to the query servers.

The Search Topology configuration settings in SharePoint 2010 dictate what functionality of the SharePoint Server Search Service runs on what server in the farm.  So, while the SharePoint Server Search Service needs to run on both the WFE and the Application Server in this example, it will be possible break out the functionality that it performs on each.  We will want it to perform query-related functionality on the WFE and crawling/indexing functionality on the Application Server.  Later in this post I will show you how to do this.

Now, on to the actual steps to doing the work:

Step by Step: Scaling SharePoint 2010 to Three Tiers

Step 1 – Build a new SharePoint Server with exactly the same software

I’m talking about taking a fresh physical or virtual server that has Windows Server 2008 (R1 or R2) running on it, and installing all the same SharePoint Server 2010 software on it that is installed on the existing SharePoint 2010 server in your existing farm.  That includes the full RTM Enterprise edition, whatever patches have been applied in your farm since RTM, and any other separate products that have been installed on your existing server such as the Office 2010 Web Applications and its patches.

Step 2 – Run the SharePoint 2010 Products Configuration Wizard on the new server and join the existing farm

I recommend installing all RTM software and all patches that have previously been applied to the farm BEFORE running the SharePoint 2010 Products Configuration Wizard from the new server’s Start menu.  This means that you will want to respond NO to the prompt to automatically run the wizard until you have installed all software packages on the new server.  This will save you from having to run the wizard multiple times.  Run it once – after you have installed all software and patches on the new server.
When you do run the SharePoint 2010 Products Configuration Wizard, you will run it on the new server that will be your application server.  The wizard is going to help you join the server to the farm and get all of the software configured and running that you installed in Step 1.
Here are what the pages of the wizard look like as you go through the process:












Oops, you forgot to install a piece of software on this new server that is already installed on the other server.  The wizard has caught your error and is not going to let you proceed until you get this done.
Exit the wizard and go install the software – in this case, the Microsoft Office 2010 Web Apps.
OK, you got the missing software installed and have restarted the wizard.  The next screen asks you for the Farm PassPhrase.  This is a special password you created when you originally created the farm.  You have to enter it here in order to join this server to the farm:
















If you click on Advanced Settings above, the next page asks whether or not you want to use this server to host the Central Administration website (sort of implying that you could move it from your existing SharePoint 2010 server to the new one).

I haven’t tried selecting the second option in SharePoint 2010.  In MOSS 2007, according to this blog post you needed to remove the Central Administration web application from the original server before you got to this step on the new server. In the context of scaling out by adding an application server, that is probably what you would want do.  If you choose to go this route, just make sure you have good backups before you delete the Central Admin site from the existing server.

For this walkthrough, you are going to leave Central Administration on the existing server:







Now the server has been joined to the farm and is a full-fledged farm member.  But, the Configure Your SharePoint Farm Wizard in Central Administration needs to run to add the service applications that exist in the farm to this new server.  So, it automatically fires up your browser and asks you to run the Farm Configuration Wizard:




After you start the wizard, it will just run for a while without any input from you and return this page if everything was successful:

Step 3 – Verifying that everything is running properly on the new server

It’s a good idea at this point to go verify that the new server is showing up as a member of the farm with a healthy status.  To do that go to Central Administration > System Settings > Manage Servers In This Farm and find the new server and verify that it has a “No action required” status:

Take a moment to breathe deep and pat yourself on the back.  You have done a lot of work to get to this point.  You now have a three-tier SharePoint 2010 farm. 

But, there is more work to be done so that your three-tier farm has only the web page serving and query processing services running on the WFE and all of the other service applications running only on the Application Server.  Until you get that accomplished, the job is not done.

(Note: the farm will work and be fully functional if you stop here.  You will have the same Service Applications running on multiple servers and SharePoint 2010 will automatically use this topology as a load balancing technique for the Service Applications.  There may be some environments where this is desired.  But, most organizations will want to separate the web-serving services and the application-serving services to provide a better balance for the farm as a whole as opposed to just load balancing the Service Applications.)

Step 4 – Re-configure the servers to run the services that are appropriate for their individual roles

You want the Web Front-End to run these (and only these) services:

1. Microsoft SharePoint Foundation Web Application (this is what turns IIS into a SharePoint “page-serving” machine)
2. Search Query and Site Settings Service (the process that takes the user’s query string and looks it up in the index)
3. SharePoint Server Search Service (but just the functionality that is necessary for the query processor)
4. Central Administration (assuming you didn’t decide to move it to the Application Server)

You want the Application Server to run these (and only these) services:

1. Access Database Service
2. Application Registry Service
3. Business Data Connectivity Service
4. Excel Calculation Services
5. Managed Metadata Web Service
6. Microsoft SharePoint Foundation Incoming E-mail
7. Microsoft SharePoint Foundation Workflow Timer Service
8. PerformancePoint Service
9. Secure Store Service
10. SharePoint Server Search (but just the scheduled content crawling and indexing building functionality)
11. User Profile Service
12. Visio Graphics Service
13. Web Analytics Data Processing Service
14. Web Analytics Web Service
15. Word Automation Services
16. Word Viewing Service

If you can get this done and everything works properly, you will have achieved your overall goal.
(Important Note: Step 1 above is really the only step in the process that can be done during normal working hours.  Everything else has the potential to impact the availability of the system to the users.  If everything goes smoothly, it is possible to do Step 2 through Step 4 in two to four hours.  Of course, it is highly recommended to have solid backups in place before starting Step 2.)

For the most part, the re-configuration of the services involves stopping a lot of services on the WFE server (using the Services on Server page in Central Admin) and verifying that they are running on the new server (which they probably are because the Configure Your SharePoint Farm wizard started them up when you ran it in Step 2).  Then, you will want to make one last pass over the list of services running on the Application Server and make sure that the Microsoft SharePoint Foundation Web Application Service and the Search Query and Site Settings are not running on it.

Adjusting the Search Application Topology

The exception to the statements of the previous paragraph is the search-related services:  SharePoint Server Search Service and Search Query and Site Settings Service.  Search is complicated enough that it has its own topology configuration settings.   You need to use this capability to place the query functionality of the SharePoint Server Search Service on the WFE and to place the crawling\indexing functionality of the service on the Application Server.

Since this is a little more complicated than the other Service Applications, go ahead and do this one first.
Navigate to the Search Administration home page in Central Administration.  Scroll down to the bottom of the page until you see the section titled Search Application Topology:



This part of the page shows you what servers the following four components of the Search service are running on:

• Search Administration component
• Crawling component (this is the crawling engine that crawls your content and builds full-text indexes from it)
• Database component (as the crawling engine crawls through the content, it stores the full-text indexes in SQL Server.  It also compiles the full-text indexes into special non-SQL files that can be propagated up to the WFE)
• Query component (this is the component that receives the user’s query and looks up the results in the special files that have been propagated to the hard drive of the WFE)

The Server Name column shows that the Search Administration, Crawl, and Query components are currently running on the existing server (SPS-INTRANET in the example).  The search-related databases are running on the SQL Server.
You want to do the following:

1. Move the Search Administration component to the new Application Server
2. Move the Crawl component to the new Application Server
3. Leave the Database component running on the SQL Server
4. Leave the Query component running on the WFE

To accomplish this, click on the Modify button to go to the Topology for Search Service Application page:



By hovering your mouse over the component lines, you can bring up a drop down menu and select Edit Properties for the components you want to move to the new server. 
Do this now for the Search Administration component:



Now do it the same way for the Crawl component (screen shot is the same as the one above).

Once you have changed the server assignments for these two components, you need to kick of the actual transfer of responsibilities by clicking on Apply Topology Changes:



The actual transfer of responsibilities begins:


When it is finished, you will be returned to the Search Administration home page and you should see that the components have been transferred as directed and all of the search-related servers should have a status of “Online”:

Note:  I am not sure why, but this page never shows anything in the Status column for the Databases component.  So, it is normal for that column to be blank for that component.

Transferring the remaining Service Applications

All that is left is to use the Services on Server page in Central Administration to make sure the list of services running on each server matches your master list from above:
You want the Web Front-End to run these (and only these) services:

1. Microsoft SharePoint Foundation Web Application (this is what turns IIS into a SharePoint page-serving machine)
2. Search Query and Site Settings Service (the process that takes the user’s query string and looks it up in the index)
3. SharePoint Server Search Service (only the functionality that is necessary for the query processor)
4. Central Administration (assuming you didn’t decide to move it to the Application Server)

You want the Application Server to run these (and only these) services:

1. Access Database Service
2. Application Registry Service
3. Business Data Connectivity Service
4. Excel Calculation Services
5. Managed Metadata Web Service
6. Microsoft SharePoint Foundation Incoming E-mail
7. Microsoft SharePoint Foundation Workflow Timer Service
8. PerformancePoint Service
9. Secure Store Service
10. SharePoint Server Search (only the scheduled content crawling and indexing building functionality)
11. User Profile Service
12. Visio Graphics Service
13. Web Analytics Data Processing Service
14. Web Analytics Web Service
15. Word Automation Services
16. Word Viewing Service

To do this, you use the Server drop-down control to select the server you want to adjust, and then use the Start/Stop link in the Action column to turn on/off the services.
Here is what your Services on Server page should look like once each has been properly adjusted fore each server:
For the Web Front-End (SPS-INTRANET in this example):



For the Application Server (SPS-APPSVR in this example):


If you navigate to the Servers in Farm page of Central Administration, you will see a more succinct view of your new farm topology:

Step 5 – Testing and Verifying

Even though you are ready to head out the door and head home since you are probably doing this on a night or weekend, it is really important to fight the urge to leave too soon. 

You really need to do some basic testing and verification before you leave.  It will be a lot better to find out about any problems now rather than when the next business day has already started.

Here is what I recommend doing before you leave:

1. Browse to each of your SharePoint web applications and log in with your user account and make sure you can hit the home page of each of them.
2. While you are there, try to open up and edit a document in the browser using one of the Office 2010 Web Apps (Word, PowerPoint, Excel or OneNote).
3. Browse to your My Site and verify that everything is working normally.
4. Add a unique phrase to a test page somewhere in one of your Sites (I always use the phrase “riderrocky”) and then go run an incremental Search crawl from Central Administration.  After the crawl completes, go back to your Site Collection and search for the phrase.  Verify that it comes up in the results.
5. Run an incremental User Profile Synchronization from the User Profile Administration page.  While it is running, logon to the desktop of the new Application Server, and find this program and run it:  c:\program files\microsoft office servers\14.0\synchronization service\uishell\miisclient.exe.  This is the Forefront Identity Management (FIM) client application that you can use to see the details of the AD synchronization process.  Several jobs will be run by FIM.  Verify that they all complete successfully with no error messages.
6. In Central Administration, go into Manage Service Applications and click on Managed Metadata Service and select Manage in the ribbon.  Verify that the Term Store management interface loads and that you can add/change/delete a Term Set and some Terms.
7. Finally, reboot your WFE and Application Server.  When they come back up, check your Windows System and Application event logs on those servers and verify that there are no SharePoint-related critical or warning events that you haven’t seen before you scaled out to three tiers.
8. Browse to your primary web application one more time before you head out the door.

I hope this blog post is a good resource for those SharePoint Server Administrators who find themselves needing to scale out to the next level!!

Regards,
Viral Shah