Nomad Software Update LsZ File Purge aka whack-a-mole

In short, there is an issue in Configuration Manager CB 1806 where an additional file (*express.cab) gets included with some software updates. This causes Nomad to generate a LsZ file that contains download instructions for two files as seen below:

; Format=0003 (UNICODE) with alt hash
; Package Version = 1       .
; Data Format = 0   (ORIGINAL)
; Generated by “CM02” from “D:\SCCMContentLib bc72d1d6-2283-4df9-bfe5-7f1d7f0e5ca8”. 11/13/2018 21:17:11(UTC) 0x1d47b963e1d6877
; BlockSize=00008000
; RDC data generation was enabled
0: “bc72d1d6-2283-4df9-bfe5-7f1d7f0e5ca8_1.LsZ” ln=0x0 dt=0
1: “Windows10.0-KB4467686-x64-express.cab” ln=0x1879155 dt=1d47b85de243723 CRC=50E7B98AAA4EC667600179BFB237E3A441E52F7B8FCD8397C77DC86DF0535581
; Large Hash 11/13/2018 21:17:17
; 11/13/2018 21:17:17..
; Large Hash complete 11/13/2018 21:17:29
785: “Windows10.0-KB4467686-x64.cab” ln=0x36EF89D2 dt=1d47b857f42088b CRC=FDAC6287883BB398EFA9BBBBB3D5DB307B2EB501F5E7680440571F02D0124A7A
; 947329831 bytes in 2 Files (blks=28913)
; LastModifiedTime 0x1d47b85de243723 11/13/2018 19:19:58(UTC)
; Hash B8EA224C75FCBEB721408B332FEB253FEF26C264
; HashV4 CB7188D42C59B4469BABFA321972604E49134D2A3A5BF3F50D3D5C82DC0EF939
; List Complete – 1

This CB 1806 issue is described in KB4465865:

For software updates that contain express installation files, Configuration Manager synchronizes the Express.cab file and distributes it to the client. This behavior occurs even if the client does not require the Express.cab file for a given deployment.

This behavior is mainly cosmetic. It does not involve the complete set of express installation files, only the Express.cab file.

Note Installing the hotfix will stop the unconditional download attempts for express CAB files for new updates. After the hotfix is installed, downloading will occur only if express installation file support is enabled for Windows 10 updates.

This will work fine until either this hotfix is applied or Configuration Manager is upgraded to a newer version like 1810. The old LsZ files will not automatically get updated and since Nomad peer to peers these files, it makes it difficult to get rid of the bad files. The problem will arise when a peer needs a patch and is running the newer version of CM. It will request the patch and corresponding LsZ file and it could potentially get a bad LsZ file (one that contains two files instead of one) from a peer. Nomad does not have a way to configure it to always pull LsZ files directly from the DPs and not peers, so only the first client in a subnet will download the LsZ from the DP. At this point, as far as Nomad is concerned, it has been instructed to download two files as part of the patch. Once Nomad hands the job back to the CM Client, the CM Client fails the hash check since it gets two files back when now it is only expecting one file.

Thus, there are two things that need to be done – first thing is to ‘clean’ the ‘bad’ LsZ files from all clients in order to force clients that need the content to pull the LsZ file from the DP. The second is to ‘clean’ the ‘bad’ LsZ files from all of the DPs after they are upgraded to the hotfix listed above or CB 1810.

For the client cleanup, we can use a CI/Baseline to perform this action. Since Nomad may not be on all systems, create an Application type CI called “Nomad Software Update LsZ Purge” and use the following PowerShell script for the Detection Method:

<# .SYNOPSIS Confirm if NomadBranch 6.x .DESCRIPTION For Detection Logic for a Configuration Item, specifically for Nomad 6.x version .NOTES 2019-1-3 Mike Terrill #>
$ErrorActionPreference = 'SilentlyContinue'
$key = 'HKLM:\SOFTWARE\1E\NomadBranch'
if ( ((Get-ItemProperty -path $key -Name ProductVersion).ProductVersion).Substring(0,2) -eq '6.') {
    write-host ((Get-ItemProperty -path $key -Name ProductVersion).ProductVersion).Substring(0,3)
}

For the Settings, use the same name (Nomad Software Update LsZ Purge) and for the Discovery script, use the following PowerShell script:

$NomadCachePath = get-itempropertyvalue -path 'HKLM:\Software\1E\NomadBranch' -name 'LocalCachePath' | Write-Output
$LsZFiles = Get-ChildItem $NomadCachePath*.LsZ -Name
$SoftwareUpdatesLsZFiles = $LsZFiles | Select-String -Pattern '^[\dA-F]{8}-(?:[\dA-F]{4}-){3}[\dA-F]{12}_1.Lsz$' | write-output
If ($SoftwareUpdatesLsZFiles.count -ge 1) {
    Write-Host "Non-compliant"
}
ElseIf ($SoftwareUpdatesLsZFiles.count -eq 0) {
    Write-Host "Compliant"
}

Since this only applies to the LsZ files for Software Updates, we only need to delete these LsZ files and not any others. Since Software Updates start with a GUID, we can filter on them using the pattern (thanks to Keith Garner and Gary Blok). Packages, OS Upgrade Packages, OS Images, Boot Images, etc. will show up at XXXYYYYY_Z (where XXXYYYYY is the site code and package ID from where the content was generated and Z is the version). Application DTs will be in the form of Content_GUID_Z.

For the Remediation script, use the following PowerShell script:

$NomadCachePath = get-itempropertyvalue -path 'HKLM:\Software\1E\NomadBranch' -name 'LocalCachePath' | Write-Output
$LsZFiles = Get-ChildItem $NomadCachePath*.LsZ -Name
$SoftwareUpdatesLsZFiles = $LsZFiles | Select-String -Pattern '^[\dA-F]{8}-(?:[\dA-F]{4}-){3}[\dA-F]{12}_1.Lsz$' | write-output

ForEach ($SoftwareUpdatesLSZFile in $SoftwareUpdatesLSZFiles)
{
    Write-Host "Deleting: $NomadCachePath$SoftwareUpdatesLSZFile" -ForegroundColor Red
    Remove-Item $NomadCachePath$SoftwareUpdatesLSZFile -Force
}

And for the Compliance Rule, use “The value returned by the specified script equals Compliant” and select “Run the specified remediation script when this setting is noncompliant”.

Add this to a Baseline as Optional and deploy it to a set of test clients to make sure it is operating as intended and deleting the correct LsZ files. When clients request the LsZ from a peer, it will get a file not found error, cycle through a few retry attempts and then disqualify that peer before going onto the next. It may take a while to completely rid the “bad” LsZ files from the environment, hence why I refer to it as “Whack-a-mole”, as soon as you get a batch clean another pops up.

The second thing that needs to be done is to clean up the “bad” LsZ files from the DPs once they have been upgraded to the hotfix or CB 1810. This can be done by running the following PowerShell script on each DP:

$NomadCachePath = get-itempropertyvalue -path 'HKLM:\Software\1E\NomadBranch' -name 'LocalCachePath' | Write-Output
$NomadDPLSZFILESPath = $NomadCachePath + "LSZFILES\"
$LsZFiles = Get-ChildItem $NomadDPLSZFILESPath*.LsZ -Name
$SoftwareUpdatesLsZFiles = $LsZFiles | Select-String -Pattern '^[\dA-F]{8}-(?:[\dA-F]{4}-){3}[\dA-F]{12}_1.Lsz$' | write-output

ForEach ($SoftwareUpdatesLSZFile in $SoftwareUpdatesLSZFiles)
{
    Write-Host "Deleting: $NomadDPLSZFILESPath$SoftwareUpdatesLSZFile" -ForegroundColor Red
    Remove-Item $NomadDPLSZFILESPath$SoftwareUpdatesLSZFile -Force
}

Once this is done, any new requests coming in *should* result in Nomad generating a new LsZ file for the Software Update that only contains the single file and not one that has the express.cab file included. Otherwise, contact the vendor if you continue to have issues.

Originally posted on https://miketerrill.net/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.