3/18/2020
By now IT departments are scrambling to get as many users as possible to work from home as a result of the COVID-19 outbreak. What they are finding out is that Microsoft patches chew up a lot of bandwidth when these clients can download the patches directly from Microsoft Update (yet still be managed by Configuration Manager). I have seen a few blog posts on the topic that ultimately end up leading to more questions than they answer. So hopefully I can make this as complete as possible and answer as many of those outstanding questions as possible.
To set the stage, I am not going to be talking about scenarios that involve CMG (I am going to assume that you are already ahead of the game and do not face this challenge). This is more for the customers on the trailing edge that have not (been able to) adopt the cloud strategy and are stuck with distribution points on the corpnet. The goal is to work with your VPN team so that they configure it for split tunneling. I will not go into this part as each VPN configuration is unique, however, I will help provide you with the necessary URLs that are needed to be excluded from coming back through the corpnet. Also be sure to factor in other things like proxy servers or other apps that inspect/filter web traffic as they will need to exclude this traffic as well so it does not come back through corpnet.
Everything starts with boundaries and if you know me, I have never been a fan of boundaries for content location (p2p FTW!). The only boundaries that I configure for content location is when I need to protect a DP in a build center where I do not want other clients outside of the build center leaching off the build center DP. Other than that, who has time to manage boundaries that are constantly changing? Plus, in my environment I could not even tell you how many subnets we have let alone pretend to get it right.
The other goal of this is to keep the operational aspect as simple as possible. Meaning, don’t expect the Software Update person to now configure a bunch of different software update deployments just to allow the VPN clients to get their updates from MU.
This is hopefully going to be a simple example to get you up and running (plus I can’t really show our production environment, so don’t ask). IP Ranges are your friend. Ever since the CM Team optimized the queries for client location requests, big honking IP Ranges are the way to go. Forget IP Subnets and AD Sites (unless you really like to cause yourself pain). So think big, like 0.0.0.0 – 255.255.255.255. However for this example I am going to keep it simple. The following are my three ranges:
Boundary Groups are pretty simple as well:
Corpnet Boundary Group Properties
Uses the Data Center DP:
In this example, every IP range is accounted for so I have not defined a relationship to the Default Site Boundary Group (or any other Boundary Groups). However, your configuration may be different:
And I am not using peer cache (BranchCache FTW!) and we do not want our corpnet devices going out to MU:
VPN Boundary Group Properties:
VPN Boundary Group uses the dedicated VPN DP(s):
Not making any assumptions, I like to explicitly state that the VPN Boundary Group should never fallback to another boundary group’s distribution point (in case an admin screws up a check box on a deployment). And if your MP(s) and SUP(s) are in the Default BG, then you will want the VPN clients to be able to get to them:
Once again I am not using peer cache (BranchCache FTW!). The “Prefer cloud based scenarios over on-premise sources” is an interesting checkbox. Since we have everything pretty much protected, it would not hurt to check it however it isn’t necessary. It could be a good safe guard in the event that someone screws up and distributes Microsoft patches to the VPN DP Group:
Be sure to set up dedicated DP Groups. You will only want to distribute Microsoft patches to the Data Center Distributions Point Group (Corpnet) and not the VPN Distribution Points Group. However, 3rd Party Updates will need to be staged on both DP Groups (and for third party updates check out Patch My PC):
IMPORTANT: When you set up the Software Update Deployment configure it exactly as follows. Since we hopefully have defined all possible IP Ranges (remember I said think big and carve up 0.0.0.0 – 255.255.255.255 accordingly), every client should have either a DP to get content from without falling back or in the case of VPN clients and Microsoft patches – Microsoft Update:
On the clients, you are going to want to check out two logs. The first one will be the CAS.log:
And the second one will be the ContentTransferManger.log:
And remember, just because it says it is getting it from Microsoft Update does not necessarily mean it is getting directly from MU. It could still be going back through the corpnet because the split tunnel was not set up correctly or a proxy is re-directing traffic. So work closely with these teams. From a CM stand point mission accomplished.
Now for the URLs. There is not a public Microsoft doc/KB (at least that I know of) that says these are exactly the URLs that are required for this feature when defining Client -> MU traffic. However, most of them are similar to what the SUP uses when it downloads the content.
(from Software Updates)
Office 365 Updates will be further down the page:
(from Manage Office 365)
Lastly, Windows 10 Updates have a slightly different URL:
(from Windows 10 servicing)
The download location can be found in the meta data for each patch:
Plus you can run a query in SQL to find it:
select top 1000 SourceUrl
from vSMS_CIContentFiles
Hopefully this helps in getting the Microsoft Update traffic off of your VPN links. If you have any suggestions or other useful tips, please leave them in the comment section below.
Originally posted on https://miketerrill.net/
Mike's Tech Blog 
Pingback: How to find software update deployments enabled with download content from Microsoft update for clients from VPN CMG internet connected | All about Microsoft Endpoint Manager
Really appreciate this post. I currently use sites as BG’s. Would it be more advantageous to switch those to IP address ranges to complement this procedure? Also would opening up the VPN clients to MU bring all updates including feature updates?
As far as the AD Sites vs. IP Ranges, I prefer the IP Ranges because I can be in total control and I can explicitly define the entire range. With AD Sites, that is not something I control nor is it easy to define the entire range. By allowing the VPN to split tunnel, you are just allowing the traffic to go through the individual’s ISP to the Internet vs. going back through the VPN tunnel to the corpnet and then out to the Internet. ConfigMgr will control the policies if this is how you have it configured.
Thanks a lot. We use this method during two years. but today we see that SSU 03.2020 is not downloaded.
We checked that MS link dl.delivery.mp.microsoft.com is not available to access from our SCCM server. Your article saved our time!
Interesting – which OS Version is the SSU you are having the download problems?
Pingback: System Center Mart 2020 Bülten – Sertaç Topal
Anything to add for clients who are on Direct Access? Gotcha’s when it comes to ADRs? Boundaries / Groups? Do’s and Don’t?
Sorry – it has been ages since I have used Direct Access and I don’t have an environment to test it.
nice, but you have overdone it.
it is much easier to configure:
\Software Library\Overview\Windows 10 Servicing\Windows Update for Business Policies
and deploy it to VPN device collection
This policy that you mention is for Windows Update for Business. Configuration Manager does not use WUfB and you would need to split your managed clients up (I wouldn’t call this easy, especially for clients that go back and forth between the office and home).
Mike, Thanks for the info, with this setup – does this mean I will have to set up 2 deployments for Windows\Office 365 updates (1 for corp & 1 for VPN)?
One other thing I was going to mention, I’m currently using 1E Nomad, v6.3.201 I have set the compatibility flags to allow it to source from MU – will this work with your set-up. Cant remember if BigBank is still using this. thanks for any light you can shed on this.
I think they finally fixed this in a later release and also a 6.3 hotfix. I just tested without the compat flags setting and Nomad failed and it fell back to using the default provider (which is able to successfully download from MU).
Yes – Nomad 6.3.201 will download from MU using the configuration Mike has very comprehensively set out in this blog. You need hotfix 20267 (released December 2018) or later and enable download from MU in CompatibilityFlags as you have done.
If you don’t care if CM downloads it, then you don’t need to worry about the compatflags as Nomad will just fail and then CM will get it.
Hi Vern, if everything is defined, meaning that it is either corpnet or vpn, then you can configure the deployment like the screen shot and only have one to cover everything.
Hi Mike, Could you suggest what i should use as a VPN DP? is this an existing DP or would i need to create a new one?
It depends on your current hierarchy and how many DPs you already have. If you can easily add another DP (or DPs), then that might be the easiest way to go. Otherwise, if you take an existing DP (or DPs), and you want to follow my ease of operational guidance (by only managing one deployment), then you will want to remove all of the MSFT updated from this DP (or DPs).
Thanks Mike!
i will be taking the approach of using an existing DP and just to clarify, the deployment packages need to be removed from the existing DP which will trigger downloading the updates from microsoft updates?
Yes, that is correct. You could use the “Prefer cloud based sources over on-premise sources” if you don’t mind that some might come back to the DP. Otherwise, if they are not on the DP, then you do not need to worry that they would ever pull MSFT patches from it since it will never be returned as a content location. 😉
Hi, we don’t have a separate boundary group for our VPN clients (which is a split tunnel configuration), nor a dedicated distribution point, nor a cloud distribution point, or CMG, as it was originally such a small scope that handled 5 to 10 users a few days a week.
Now, we have increased that scope to almost 500 (our entire remote workforce, a relatively small organisation), and have 99 percent of our employees working from home with their SCCM management devices.
My question is, can we just set our ADRs to not create a distribution group, and set the deployment properties to use Microsoft Update, thereby forcing *all* clients (whether corporate or VPN) to go direct to the internet? We use a cloud based web proxy as well, so only corporate traffic comes through the corporate VPN.
Do you think this was work? If so, I think this would be a simpler approach during the COVID-19 pandemic to have all clients get updates from the Internet.
Thanks.
This highly depends on how your VPN is configured (and what it is capable of). If the only traffic that comes back through your VPN is corpnet traffic, then things might just work for you by enabling MU. But double check with your VPN team/vendor and also do some network traces (using something like WireShark). This should give you a better idea on the traffic flow.
Pingback: Creating a collection of VPN devices – GivingSomethingBack
Good sharing . Seems it is not a new feature . We used this feature to for a while .
Thanks – yes, this has been around since the CM 2012 days.
we’re basically one location (other sites are Gbit connected in the same city), hence one DP for all locations would be sufficient.
To implement your method, I would need a DP just for the purpose of VPN?
So it really depends on what you are after. If you do not mind that some clients might come back via the VPN to get patches in the event that they cannot get them from MU for some reason, then there is the option to set “Prefer cloud based sources over on-premise sources” on the Options tab of the Boundary Group Properties. For the intent of my post, I wanted to always have VPN clients use the cloud sources. Unfortunately there isn’t an option to ‘Only use cloud based sources over on-premise sources’.
from the naming, I actually expect “Prefer cloud based sources over on-premise sources” to do exactly that.
Assuming everything is set up correctly, it should use MS to download updates. For everything else using the DP over VPN, right?
Key word – assuming. In my case I want to always pull from MSFT.
Hi,
I have this set up and the clients are trying to download from Microsoft. I can see in contenttransfermanager log it’s downloading from dl.delivery.mp.microsoft.com.
However, after a while the download gets cancelled and then it falls back to my DP or, if I set it to neverfallback it tries to download again from Microsoft and cancels again. This continues to repeat and happens to all clients and also on different updates.
It seems to cancel after the download completes because it takes longer to cancel depending on the size of the update and I can find the update file in ccmcache increasing in size.
I am unsure why it’s doing this. Can you help?
Thanks.
It is hard to say where the problem is in your situation. Your best bet would be to use Wireshark and other network tools so that you can see not only where it is going but also how it is routing there. If you need outside help, I know that CT Global has been doing a lot of these engagements lately and would be happy to do some consulting.
Thanks for your reply. After trawling through the log files I’m getting the below error from Windows Update.
*FAILED* [80240033] ISusInternal:: GetEulaText
This only happens when it’s trying to download the CU from Microsoft. The servicing stack downloads fine.
We have several DPs but we need to isolate our VPN for MS Updates. However, we may need to push out application updates as well. If I have this correct, I need to setup an individual VPN DP, as well as setup never fallback on all DPs including the VPN DP itself, is that correct?
Yes – as I mentioned to Roland, it really depends on what you are after. If you do not mind that some clients might come back via the VPN to get patches in the event that they cannot get them from MU for some reason, then there is the option to set “Prefer cloud based sources over on-premise sources” on the Options tab of the Boundary Group Properties.
I have that set as well as an IP address range for it. Even though we do not have fallback on any boundary groups, except explicitly stated for the VPN boundary, is an explicit rule as you mentioned, needed on the other site DPs?
Just curious why the rule on the additional boundary groups when the VPN boundary group is limited to a specific IP range and no fallback. I’d think if a DP were to go down we’d want the onsite devices able to reach another DP.
Additionally, are you suggesting separate MU deployments for VPN users?
As for other clients falling back to another DP, that is completely possible and will depend on your CM design (and DP capacity).
The way that I have the deployments configured in the blog is that you do not need a separate MU deployment for VPN users – “The other goal of this is to keep the operational aspect as simple as possible. Meaning, don’t expect the Software Update person to now configure a bunch of different software update deployments just to allow the VPN clients to get their updates from MU.”
Hi Materrill, Thanks for the good post. I have already followed above.but failed to achive the desired output. I wanted to ask few questions.
1. Do we need to allow the urls above from client computers side(As we have allowed from SUP server side).
2.when connected VPN I am able to ping the primary site from client. but not able to ping the client from Primary site.
3.Network team perspective VPN Split tunnelling already enabled.
4. we have a DP without April patch content.still clients are not going to WU to get patches
Looking forward for hear answers from you.
It is hard to say where the problem is in your situation. Your best bet would be to use Wireshark and other network tools so that you can see not only where it is going but also how it is routing there. If you need outside help, I know that CT Global has been doing a lot of these engagements lately and would be happy to do some consulting.
Hi Materril, The last time our company check boxed “download content from Microsoft Updates” in the ADR, some of our machines received the feature update, which upgraded the operating system from WIN10 1809 to 1903. Any suggestions on how to stop this?
I would double check the ADR to make sure you aren’t allowing the feature updates.
Pingback: LockDown Diary - How I used DJOIN to Build Test Machines over VPN - A Square Dozen
We’re seeing the initial communication to officecdn.microsoft.com being split tunneled, however, the actual download of the Office 365 patch is being handed off to one of the many xxx.deploy.static.akamaitechnologies.com CDNs. As a result, the download is traversing the VPN tunnel. We can look at adding xxx.deploy.static.akamaitechnologies.com to the list of domains being split tunneled, however, I’m curious if anyone else seeing this behavior?
Sounds like the issue is with the VPN configuration. Unfortunately, I don’t believe you will ever be able to list all of the CDNs to deny them from hitting the VPN if that is the approach.
We also noticed that the Windows updates are being downloaded from a range of IP addresses owned by Microsoft, however, the IP addresses aren’t resolvable to any domain names. We opened a ticket with Microsoft earlier this week and asked if they could provide the URLs and IP ranges needed for split-tunneling the download of both Windows and Office 365 updates, however, I suspect they will be hesitant to commit to an answer as the CDN networks could change over time. Another concern is that the CDNs are most likely Geo-based and could be a factor here. I agree, the issue is with the VPN configuration. The bigger question is has anyone else successfully split tunneled Windows / Office 365 updates and if so, how did they accomplish it? You mentioned you don’t believe that I’ll ever be able to list all of the CDNs if that’s my approach, however, what approach should I be attempting here?
Pingback: All My Devices Left Me. I’m Scared. What Do I Do Now? – Dam Good Admin
Thank you, we have recently deployed this internally as well, with great success!!!
We have recently noticed that while the content is downloading directly from MS, the VPN connection must persist for the entire download duration.
Is this something we can change?
Not sure but something doesn’t sound quite right. I would review network traces to make sure the traffic is indeed going from the local host to MSFT. I would think that even if the VPN connection was broken during a download, the CM Client would still continue to download the content it is pulling.
Pingback: How to convert the CMG cloud service from PKI to Public cert | How to redeploy the CMG service | All about Microsoft Endpoint Manager
We only have one single, standalone, dp/main site; how would I go about forcing VPN clients to pull updates from MU; we already have a split-tunnel VPN and only provide routes for our corpnet so no other traffic will come through it. We have enough bandwidth to support office machines pulling updates direct from MU, but I don’t see how we can configure things the way you describe without creating a new DP just for VPN and then just not deploying updates to it. Thoughts?
It is all going to depend on how the traffic is routed. You do have the option to use the “Prefer cloud based sources over on-premise sources”. Or just not even download the patches to begin with and let everything pull from MU. Also, in MEMCM 2006 there is a new VPN boundary type – https://docs.microsoft.com/en-us/mem/configmgr/core/plan-design/changes/whats-new-in-version-2006#vpn-boundary-type
Ok…our windows uppdates is working fine…but NOT our Office-patches =( Ideas?
You probably need to make sure the Office CDNs are defined for your split tunnel traffic.
We have no split tunnelin…we can download the updates to sccm-server…but our clients do not download them….windows updates works but not office =(
It slipped my mind that Office is a bit strange in the way it handles updates. You might need to have a look at how you configured it. When creating your config.xml (see config.office.com), you have the option to select Office Content Delivery Network (CDN), Local source, or Microsoft Endpoint Configuration Manager. It only allows the selection of one and yours is likely set to MEMCM.
Hi materrill, thanks for great article. But if you say “do not install update” options for both. then internal clients are getting data from MU too.
Yes – good catch! That top option on the Download Settings tab should be “Download software updates from distribution point and install”. I will get that screen shot corrected – thanks!
Hi –
fantastic article, one that ive been trying to implement currently.
Our VPN has all traffic coming back into the corpnet.
MSUpdates is blocked by firewall (except for WSUS/CM)
Ive split my ADRs to deploy patches on Laptops as above, forcing them all to essentially go out to MU for patches. This is to stop them from consuming the VPN traffic.
I had “do not install for both options”, but in my testing found that they were coming back through the VPN and going out to MSUpdates (same issue the last person commented)
I guess my question is, if split tunnel is not an option, will this scenario work? (laptops get MSupdates when off the VPN)
What would be a better solution if changing the “Download software updates from distribution point and install”.” doesnt resolve my current dilemma.
hopefully that makes sense…
Hi Tim, I think it is going to depend on how your local firewall is configured to behave when it detects not being on the VPN and not on the corpnet (assuming this is where your firewall is blocking the traffic) and if it can resolve the address (meaning it is able to use a non-corp dns when not VPN’ed in). If so, then this might work for the time when not on VPN. Hope this helps.
-Mike
Hello. Deployed the scheme Primary site – management point, distribution point, and additional distribution point with System update point. Configured the boundaries for the VPN client (test client). the test client is looking at an additional distribution point. When trying to trigger an update, the logs are empty. can you tell me which way to look?
I would double check your boundaries and boundary groups.