Windows as a Service in the Enterprise Table of Contents – this link contains a list of blogs covering different parts of the solution.

Windows as a Service in the Enterprise Overview Part 2

In Windows as a Service in the Enterprise Overview Part 1, we talked about the challenges that Windows 10 brings to the Enterprise and covered off ‘why’ we needed to do something and also ‘what’ we are doing to solve the problem. In Part 2, we are going to talk about the ‘how’ the solution fits together and along with some of the technical details. We also plan on writing other detailed blogs on particular parts of the solution that you will find listed in the Windows as a Service Table of Contents.

Our number 1 objective is to be able to Minimize Risk and Maximize Velocity. With that, we set a goal to create an extensible, modular and reusable framework. What we came up with was a gated process to maximize success and improve the end user experience. We only wanted to start the upgrade (or inform the user they could opt-in) if everything was ready to go, content was cached and we were 99% sure the upgrade was going to be successful. We also wanted to minimize the in-place upgrade time for the end users that would be experiencing the upgrade on their own time. Systems need to be patched and ready to go after the upgrade, hence we did not want them to be upgraded and then have to sit through another 30-60 minute cumulative update (see Optimizing Win10 OS Upgrade WIM Sizes for more information on how we accomplish this).

Multi-phase, gated approach consisting of the following phases: Pre-assessment, Pre-cache/Compat Scan, Ready for Scheduling, Pre-flight, In-place Upgrade.

Considerations and Challenges:
Different device types – when we talk about building a modular process, we wanted it to include all possible device types that are running Windows 10 that need to be upgraded. This goes beyond the traditional office worker laptop and desktop systems. We also needed to cover kiosk devices, devices that display information in public view and VDIs to name a few. What we did not want was a different process/method for upgrading each of these since this would become unbearable to keep up with the maintenance.
Different types of users – mobile users are quite common these days, especially ones that work remotely on the road or at home over VPN. It was extremely important that the solution work for this type of user since we have a large population of them. We also have users in various time zones around the world with different working hours and need to be sure not to disrupt their working day. We still have some groups that prefer to do managed deployments – meaning they do not want the end user of the system involved at all. They want the upgrade to happen at night without any user interaction. On the flip side, we wanted to also see how many users would opt into upgrading at their convenience before the deadline. Initially, we were told that we would be lucky to get 15% opt-in, but so far that percentage is more like 65% out of all users including the ones that do not get the TS notification (so it is actually an even higher percentage). It is a new workforce – not the same as it was 20 years ago or even 10 years ago. People upgrade the OS on their smart phone all the time without IT holding their hand. Make it available and people will install it on their schedule.
Improve the end user experience – this is something that we wanted to do better from the last upgrade which was based on deferrals. I’ll admit, I am not a fan of the deferral process. It causes issues with reporting. It is not a true deferral. Depending on when the system is added to the collection could mean that it runs at different times. I remember getting my deferral notification from the previous upgrade. It was about 3 PM in the afternoon and I was still in the middle of something. I deferred the upgrade thinking it would be a 24 hour deferral before asking again. Nope! At 8 PM that night it attempted again and since I was not in front of my system to click the defer button the upgrade ran. So we decided to use native functionality in Configuration Manager (with some modifications but more on that later). Users would know the required deadline from Software Center and get reminder notifications the closer it got to the deadline. We also customized some of the screens so that end users knew that something was happening (there is nothing worse than an end user rebooting the system in the middle of the upgrade).
Minimize the business impact – there are many parts to this, but loss of productivity and maximizing success are the main ones. The purpose of the business is to well…run the business, not spend time running OS upgrades. So the goal of doing as much of the prep work ahead of time so that a) the upgrade is able to run (eliminating things that are going to block the upgrade from running), b) that it runs as fast as possible and c) that it has a high chance of success. By maximizing the 1st time success rates (and tracking them for data that we can use in our annual review), we minimize the impact to the business and also minimize technician time and the amount of touches they need to do.
Collect metrics for better reporting and future use – how do you think some of the large companies that you buy from seem to know you so well? They collect data and (among other things) use this to provide a better experience in the future. We did the same thing by collecting information that we will be able to analyze and use to create an even better experience for the next upgrade.
Content distribution and updated WIM – this helps speed things up. Start pre-caching content if you are not already and I highly recommend using peer to peer technologies (especially BranchCache because it is great at deduping WIM files among other types of content). Otherwise, you might not become best friends with your network team.
Change Control process – this is probably everyone’s favorite that has to deal with change control. The solution that we come up with has to meet the change control requirements.
Lastly, automate as much as possible. There is going to be times where you need to troubleshoot an issue by digging through log files, running process monitors or something else to help diagnose the problem. So the more automation, the more time it frees up to do things that are not easily automated.

Backend Setup

Collections

There are certain collections that are tied to the various phases of the process. Not everyone can see all of these collections as we want the automation moving devices through the process based on the rules that have been defined.
NOTE: With the extended support on the Fall release, we will likely move to a yearly cadence (which is what we wanted anyway) and skip the Spring release all together and we will likely drop the Spring collections and the Fall text from the Fall collections.

Ready for Pre-assessment
The entry point into the process is the Ready for Pre-assessment collections. These are nothing more than place holders that are used when the deployment team imports devices into the process. They are scoped to see their respective collection (we have multiple groups that manage their devices so it makes it a little more complex than the average environment). These collections are called:
OSD_W10_{Fall/Spring}_Ready_for_PreAssessment_xxx
Fall indicates the xx09 releases, whereas Spring indicates the xx03 releases of Windows 10. You could also use xx09/xx03 or H1/H2 to differentiate between the two sets of WaaS collections.

Pre-assessment
Once the first WaaS job runs (currently it runs twice a day), it moves the machines from the Ready for Pre-assessment collections into the Pre-assessment collection. These collections are called:
OSD_W10_{Fall/Spring}_PreAssessment
Once in the Pre-assessment collection, this is where another backend job runs once a day and evaluates all of the Pre-assessment rules mentioned in Part 1 against the systems in the collection using inventory data. Devices that pass all of the rules are moved into the Pre-cache/Compat Scan collection once a day at night. Those that do not pass, stay in the Pre-assessment collection until they pass or are removed from the WaaS process.

Pre-cache/Compat Scan
The Pre-cache/Compat Scan collection is the first place in the process that something is being run on the device. These collections are called:
OSD_W10_{Fall/Spring}_Precache_Compat_Scan
This collection has the Pre-cache/Compat Scan Task Sequence deployed to it. It runs completely hidden and the intent is to cache the content that will be used during the actual in-place upgrade and also run the Windows 10 compatibility scan to uncover any hard blockers or other issues that we do not already know about. Once we discover issues, we create a Pre-assessment rule so that any other devices with this problem do not make it past Pre-assessment until the issue is resolved. Devices that pass Pre-cache/Compat Scan get moved to Ready for Scheduling. Devices that do not pass, stay in the Pre-cache/Compat Scan collection until they pass, are moved back to Pre-assessment, or are removed from the WaaS process.

Ready for Scheduling
The Ready for Scheduling collection is another place holder collection that contains the devices that have passed every phase up to this point and are ready to be scheduled for the upgrade. These collections are called:
OSD_W10_{Fall/Spring}_Ready_for_Scheduling
When the deployment team runs the Scheduling Wizard, it reads the devices from this collection.

In-place Upgrade
There are 31 daily collections for In-place Upgrade, 1 per every day of the month. For the months with fewer than 31 days, those collections are not used those months. These collections are called:
OSD_W10_{Fall/Spring}_Day_{01-31}_8PM
8 PM indicates the time that the corresponding maintenance window opens and also the required time of the deployment. It is possible to have more than one deployment time per day if required, but it does increase complexity. We also have certain scenarios where 8 PM does not work for some business groups and handle those slightly different (see Next Available Maintenance Window). We also have the mandatory collection where some devices go that missed the 7 required night deployment times. These collections are called:
OSD_W10_{Fall/Spring}_Mandatory
This collection has a required deployment without any restrictions and will run the next time the device comes on the network. Other devices go into a needs remediation collection so that a technician can investigate why the device did not upgrade during the 7 required night deployment times. These collections are called:
OSD_W10_{Fall/Spring}_Needs_Remediation
Although the 8 PM deadline works for most scenarios, we did have a few cases where the device could not run until later (like midnight). These devices already had maintenance windows defined (in one case from 12 AM to 5 AM). Since the daily collections also have maintenance windows, they were combining to create an 8 PM to 5 AM window. Since these devices did not qualify for user opt-in and already had a maintenance window, we created another scheduling option called Next Available Maintenance Window. These collections are called:
OSD_W10_{Fall/Spring}_Next_Available_MW
These collections do not have a maintenance window on them. When a device is scheduled, the wizard checks to see if a device already has an assigned maintenance window. For those that do, they get place in the Next Available Maintenance Window collection. The next window could be that night or the upcoming weekend.
For the tech led, on-demand scenarios, we have collections that have the On-demand Task Sequence deployment. These collections are called:
OSD_W10_{Fall/Spring}_Available_On_Demand
The deployment team has access to this collection via LOB specific collections mentioned below and can add machines into it since there is only an available deployment targeted to it.

Exclusion collections
There are multiple exclusion collections that are configured for the various deployment teams. They can add devices to their collection so that it does not appear in the WaaS process. These collections are called:
OSD_W10_{Fall/Spring}_Exclude_xxx

On Demand
There are multiple on demand collections that are configured for the various deployment teams. These are for the manual tech led upgrades and techs can add devices to their collection so that the available in-place upgrade deployment can be run from software center. Devices in this collection may or may not go through Pre-assessment first. They do run a task sequence that contains both the Pre-cache/Compat Scan and In-place Upgrade Task Sequences. These collections are called OSD_W10_{Fall/Spring}_Available_On_Demand_xxx and roll into the main OSD_W10_{Fall/Spring}_Available_On_Demand collection mentioned above.

Mandatory In-place Upgrade
After the opt-in period and seven nightly attempts (at 8 PM), certain devices that have not run go into a collection that has a mandatory deployment without any restrictions on it. This means it will run as soon as the device comes online. These collection are called OSD_W10_{Fall/Spring}_Mandatory.

Needs Remediation
After the opt-in period and seven nightly attempts (at 8 PM), certain devices that have not run or devices that have run but failed for some reason go into a needs remediation collection so that deployment team member can investigate either the failure or determine if the device should be rescheduled again. These collections are called OSD_W10_{Fall/Spring}_Needs_Remediation_xxx.

Next Available Maintenance Window
There were certain devices that already had defined maintenance windows that were more restrictive than the night WaaS 8 PM to 4 AM OSD maintenance windows that are set up on all of the day collections. Some of them did not start until midnight or the weekend. So for scheduling purposes, the wizard was modified to determine if a device was already in collection with a defined maintenance window and if it was, it would be placed in the next available maintenance window collection. This way the upgrade would only happen during the time period that the device owner had previously defined. These collections are called OSD_W10_{Fall/Spring}_Next_Available_MW.

Collection Membership Rules
With the exception of the Exclude collections, all membership rules are Direct Membership rules. This is so there can be granular control over individual devices. Also, we do not need to set an aggressive collection evaluation cycle and beat on the collection evaluator. Direct Membership rules are pretty efficient since it will only evaluate when a rule is added or removed. There were several internal team discussions on the best way to handle this and both Stephen Owen and Keith Garner came up with some really efficient methods for bulk adding and bulk removing devices from a collection.
The WaaS process is state based, meaning that devices should only be in one state (collection) at a time and follow the flow of the process. The deployment teams are only scoped to see some of the collections (Exclude, On-demand, Ready for Pre-assessment and Ready for Scheduling), the rest is hands off – collection membership is all done by the automation framework (WaaS Jobs).

WaaS Jobs
There are a few WaaS Jobs that are configured to perform the automation behind the scenes. This can be done using various methods, however, our automation team has selected SMA. Hopefully they will publish some blogs on the technical aspects of how these jobs run.

Job 1
This first checks for any devices in the Ready for Pre-assessment collection (devices that were placed here by the Import Wizard) and moves them into the Pre-assessment collection (adds the devices to the Pre-assessment collection and then removes them from the Ready for Pre-assessment collection). This job runs twice a day at 3 PM and 7 PM.

Job 2.1
This job runs the pre-assessment rules against the devices in the Pre-assessment collection. The devices that do not pass pre-assessment will stay in the Pre-assessment collection. This job runs twice a day at 1:30 PM and 9:30 PM.

Job 2.2
This job moves the devices that passed pre-assessment into the Pre-cache/Compat Scan collection (adds the devices to the Pre-cache/Compat Scan collection and then removes them from the Pre-assessment collection). This job runs once per day at 10:30 PM so that pre-caching starts at night time during the off hours for the devices that are online. Originally jobs 2.1 and 2.2 were combined but were later split out so that pre-assessment could be run more than once per day in the event that a device’s pre-assessment issue was resolved the same day.

Job 3
This job moves devices that have successfully completed the Pre-cache/Compat Scan phase into the Ready for Scheduling collection (adds devices to the Ready for Scheduling and then removes them from the Pre-cache/Compat Scan collection). The ones that do not finish or complete the Pre-cache/Compat Scan phase stay in the collection. This job runs once per day at 7 AM.

Job 3.5
This job moves the devices that we submitted from the Scheduling Wizard into the correct day collection or Next Available Maintenance Window collection. If a device was scheduled for the 31st of the month then it would get placed into the Day 31 collection. This job runs once per day at 9 PM. This way any changes in content from the time the device was pre-cached will start to download at night if the device is online.

Job 4
This job does collection clean up. It looks at the n-7 day collection and determines which devices are done and removes them from the process (i.e. successful upgrade), which devices should go into the Mandatory collection, and which devices should go into the Needs Remediation collection. It also looks at the Next Available Maintenance collection, Needs Remediation collection, the On Demand and the Mandatory collection and pulls out any devices that were successfully upgraded.

Maintenance Windows
The OSD_W10_{Fall/Spring}_Day_{01-31}_8PM have maintenance windows on them. The maintenance window is from 8 PM to 4 AM and runs for 7 days from the starting day. The reason for this is to prevent required deadline deployments from running during the day. If a device was simply offline or powered off, it will not get hit with the upgrade the next morning once it is powered on. Most of the devices fall into the category where there is not a pre-defined maintenance window, however, there are certain devices that do have pre-defined maintenance windows. These get scheduled into the OSD_W10_{Fall/Spring}_Next_Available_MW collection (as mentioned above) and will run during the device’s next scheduled maintenance window time frame. Maintenance windows do not apply to Opt-in (user invoked upgrade before the deadline) or On Demand Deployments.

Deployments
There is one required deployment for each day that recurs for 7 days. Rerun if previously failed is configured in order to account for Pre-flight failures since there is not an easy way to currently perform pre-flight checks before running a task sequence that still adheres to maintenance windows (i.e. run another program before running the task sequence does not honor the maintenance windows and will make it confusing for end users). The deployments are also configured to allow users to run independently of assignment which allows the end users to opt-in ahead of the required deadline. For the daily deployments, task sequence progress is displayed. This can be selectively enabled and disabled starting in CB 1706. The deployments are configured to download all content locally before starting the task sequence. This is why it is important to keep the number of reference packages to a minimum. For driver packages, we use a dynamic method that downloads the correct driver package during the running task sequence. Also the deployments are configured to use remote distribution points and the default boundary group. We use peer to peer technology and do not use boundary groups for content location (which would be a nightmare in our environment). Lastly, we rename the deployment “advertisement name” for better/cleaner report filtering.

$NewTS = Get-CMTaskSequenceDeployment –TaskSequenceId $TSID | where AdvertisementID –eq $DPID
$NewTS.AdvertisementName = 'Windows 10 In-Place Upgrade Fall - Day 01 PM'
$NewTS.Put()

Pre-cache/Compat Scan Task Sequence
There are two goals to the Pre-cache/Compat scan phase – the first one is to pre-cache the contents (including dynamically downloading the correct driver package), and the second one is to see if the system will actually pass the Windows 10 compatibility scan. If it does not, the user is not even disrupted or aware that anything is happening at this point since it is configured to run silently in the back ground. There is no point sending the actual upgrade to a system that fails the compatibility scan. That just leads to end user frustration. Also, it helps us identify previously unknown hard blockers (applications that prevent the in-place upgrade from running). This was invaluable for a few reasons – we discovered applications that we did not know about and were able to add those back to our pre-assessment rules and secondly, we found out that old install binaries would prevent the upgrade as well and these simply just needed to be deleted from the flagged system. If there is only one thing that you adopt from this WaaS process, it should be this – running a pre-cache/compat scan ahead of time. This alone will increase your first time success rates.

In addition to the pre-caching and the compat scan, we also wanted to gather some other information, so the task sequence also collects the following information by writing to the registry: start and stop time, start and stop time for downloading content (drivers), check readiness failure reason, compat scan results. This information is then collected by extending the hardware inventory.

One interesting thing that we found out was that running a compat scan only (i.e. using the TS Upgrade Operating System step with the Perform Windows Setup compatibility scan without starting upgrade option) put the client in provisioning mode. This was bad since we were running the Pre-cache/Compat Scan TS completely silent and sometimes a device would be powered off during this process effectively leaving the client in provisioning mode. We had put some fail safes in the TS so that it would pull it back out after a period of time and we also created a User Voice item. This is now fixed in CB 1806, the client now does not go into provisioning mode when only running a compat scan. My colleague Gary Blok has a post called WaaS – Post 1 – PreCache Compat Scan TS that goes into the details of the Pre-Cache/Compat Scan TS and even provides a sample that can be downloaded and imported.

Pre-Flight
Originally we tried to get the Pre-flight checks to run as a script before kicking off the In-place Upgrade Task Sequence, but we quickly found out that it does not adhere to maintenance windows. So for the laptop user that had their device off the night before who powered up to check some emails while running on battery would all of the sudden get a notification to plug their laptop in so the upgrade could run – but it would not run since it was out of the maintenance window and it led to a very confusing experience. Therefore, we had to move the Pre-flight phase into the task sequence, which is something that I absolutely despise. Since there is no abort exit code on a task sequence, you are left with sending either a failure or success. If you send a success for a pre-flight failure, you have a false positive since the machine did not upgrade and if you send a failure, you have a pre-flight failure that is hard to tell apart from a real failure, skewing your metrics and results. The Task Sequence engine could use some improvement in this area and hopefully we will see something some day in a future CB release (hint hint, DJam or Rob if you are reading this post).

As for the actual Pre-flight checks, these are mainly the same checks that we did for the Pre-assessment but they are done on the client at run time and not against inventory data. We also added some execution checks, like running on battery, emergency kill switch, MP connectivity, pending reboot and VPN status. There are two steps in the Task Sequence for this – one that will create pop-up messages using ServiceUI if a user is logged on and one that will run silently if no user is logged on. Pop up notifications are to inform the user if there is a failure, the reason and what needs to be done to remediate the failure. It is designed to auto re-check the rules every 5 minutes and continues if the issue(s) is resolved. It retries 12 times (60 mins total) and then fails the step. If the Pre-flight fails, it skips the Main TS and continues to the end where it records the failure(s). The results are captured in a PreFlight.log in the ccm\logs directory and also in the registry like the Pre-cache/Compat Scan metrics.

This enables easier identification of Pre-flight failures.

In-Place Upgrade Task Sequence
The In-Place Upgrade Task Sequence is similar to the Pre-cache/Compat Scan Task Sequence, as it also sets variables along the way to keep track of metrics. This Task Sequence also makes heavy use of nested Task Sequences. The goal was to make things modular, so for various parts of the Task Sequence we split up parts that could be re-used in other Task Sequences. Tracking some of the same metrics was important so that we could know how many attempts for first time success rates, run time, if a user was logged on, if a user invoked the upgrade (opt-in) and a variety of other things. Just like the other phases, this is written to the registry, inventoried and used in reporting.

Gary also has a detailed blog on a sample In-Place Upgrade Task Sequence and modules that can be downloaded and imported on his blog WaaS – Post 2 – In Place Upgrade TS.

On Demand Task Sequence
We had another requirement for technicians to be able to run tech-led in-place upgrades (think executives). They needed to be able to kick these off from Software Center while sitting in front of the system. We did not want to have to maintain a completely separate task sequence for this use case, so we decided to use nested Task Sequences. It is a very simple Task Sequence that combines the Pre-Cache/Compat Scan and In-Place Upgrade Task Sequence into 1 Task Sequence. It sets a variable called SMSTS_OnDemand to TRUE. Then enables the TS Progress UI so that the Pre-cache/Compat Scan Task Sequence progress is visible by setting the variable TSDisableProgressUI to FALSE. Step three is the Pre-cache/Compat Scan Task Sequence and step four is the In-place Upgrade Task Sequence. It has a single, available deployment to the OSD_W10_Fall_Available_On_Demand collection with the Pre-download content for this task sequence enabled. This way, as soon as devices are added to the on demand collection, they should start downloading the OS Upgrade Package and be ready for when the tech shows up to do the upgrade. They could also be run through the Pre-assessment, Pre-cache/Compat Scan phases as well, but it is not a requirement. Lastly, the deployment is configured like the other deployments to ‘download all content before starting’.

User Experience
The user experience is always a fun thing to deal with and decide how to handle. There are a lot of creative ideas on the internet from the community that provide different front ends, notification screens and deferrals. However, we wanted to use native Configuration Manager functionality as much as possible. Plus, if the CM Product Team does not hear otherwise, they assume that everyone is satisfied with the out of the box functionality. The goal should be to let them know how we want or think the product should work and provide that feedback using User Voice, Twitter, User Groups and conferences.

We decided to go with the built in notifications and instantly found some bugs and limitations. For some of our devices and lines of business, there should be no pop up notifications. The upgrade should just run off hours without any notification or user involvement. For some of our other devices and lines of business we wanted pop up notifications and reminders so that they would have the ability to opt-in to the upgrade before the deadline. The challenge here is that notifications are enabled on the Task Sequence. Having to duplicate the Task Sequences seemed a bit crazy. So we used a little local policy trick and enabled notifications on only those devices that should get notifications. This was covered at MMSMOA in the Windows 10 OS Deployment – MVP Showcase session. It is on the list of WaaS items to be blogged, so stay tuned on how we do this…

The notification screens and text were another point of concern (bug and limitation). Depending on how it was launched – from the pop-up notification or from Software Center, different screens, fonts, and font colors would be used.

A support case was opened for this issue and it is mostly fixed in 1806. The text boxes are still very ridged on what can be put in each one and there is not the ability to simply provide a hyper link to an internal support page. Also, the from the pop-up notification, the default selection cannot be customized and we would like the ability to specify the default or have it changed to Snooze and remind me: Later. Although our opt-in numbers were nice and high for 1709, we suspect the usual human behavior of just clicking OK kicked off a fair number of the upgrades.

Another thing that was done to make sure users knew an upgrade was in process was we customized the lock and logon screens. We thought about actually preventing users from logging in but instead we just warned them that an upgrade was in process. Depending on what step the Task Sequence is running, the progress notification may have already happened prior to log on and if it is in the middle of the OS Upgrade step, an end user logging on will not see the progress UI. So this helps prevent issues of that happening.

Reporting
We built from scratch reports for Pre-assessment and In-place Upgrades and also modified some existing reports as well to only show data for machines currently in the WaaS process. I am sure that we will be blogging about some of these in the future and we are also hoping to get more Power BI style dashboards for daily status.

Troubleshooting
There is always a need for troubleshooting. For most of our unexplained errors, I blame them on our 3rd party security software. These are extremely difficult to pinpoint and show the actual cause. Microsoft recommends removing 3rd party anti-virus/anti-malware if you are having issues with upgrades. However, we structured things so that each phase could use various reports, status messages, and log files to help find the issue at hand. For certain failures, logs were automatically zipped up and copied to a log share so they could be further analyzed. After we figured out how to incorporate the Dynamic Updates into the OS Upgrade Package, systems that were not upgrading previously, upgraded without any issues. Unfortunately, we did not figure out how to do this until the tail end of our upgrade. Gary has a blog on it called IPU & Offline Dynamic Updates. Special mention to Adam Gross (http://www.asquaredozen.com/), David Segura (https://www.osdeploy.com/), and Marc Graham (https://blog.ctglobalservices.com/author/mag/) – as we have tested this over and over until we finally got it to work. Follow them on Twitter and read their blogs as they have some nice scripts and solutions for building and optimizing install.wim files and OS Upgrade Packages.

WaaS 1809
We have some things planned for WaaS 1809 that we did not get to in the first iteration or things we thought of since then that we want to incorporate. We have some exciting things planned around new and not so new technology (Ledbat++, Delivery Optimization and BranchCache) that will make pre-caching content even better. We plan on continuing to share our knowledge, best practices, tips and tricks and ways of doing things with the community so that you might find that one thing that makes your upgrade that much easier and that much more successful. Hopefully you enjoyed this detailed overview of our WaaS process and have been able to take away something useful. Stay tuned for more posts and good luck with your Windows 10 upgrades.

Originally posted on https://miketerrill.net/