Many of us that write and use PowerShell regularly are familiar with the PowerShell Gallery (PSGallery) package feed and the PackageManagement/OneGet modules that provides access from the shell. But did you know you can use the Package Management extension of VSTS as a private PowerShell module feed?
You may want to do this if:
- Your organization are already VSTS users and you want a convenient, OneGet-compatible way of distributing your PowerShell modules to users within the organization.
- You want an automated release pipeline for PowerShell code but PowerShell Gallery is not an appropriate choice for hosting the modules. It might be too public.
In this blog post I’m going to show you how to use VSTS to host your PowerShell source code and define a build process that will publish that code to a private, OneGet-compatible package feed.
|Tool||Version used in this post|
|VSTS with Package Management extension|
|A PowerShell module (including well-formed module manifest)|
Step 1: Use source control
Advanced PowerShell users understand that scripting and automation are subject to the same software development best practices that traditional developers have been following for many years. Chief amongst these best practices is the use of source control. The benefits of using proper source control for PowerShell are vast and beyond the scope if this post, but our first step is to commit a PowerShell module to a Git repository. We will then define a build pipeline that triggers on any future commit to this repository, which will publish our module to the VSTS package feed for consumption elsewhere.
Since we’re using VSTS as the package feed I’m also going to use it to host the Git repository. But if you prefer GitHub, or any other repository that a VSTS build can trigger off, feel free to use that instead. If you’re already using source control, skip to Step 2 below.
I’m going to demonstrate everything with my PSPerceptron module from: Developing a neural network in PowerShell. For the purposes of this blog post, it doesn’t matter at all what the PowerShell module being published actually does, as long as it had a valid module manifest (.psd1 file). Just follow along with your own module changing any module descriptions, etc, as needed.
From the VSTS homepage, click the New Project button and complete the name and description fields. We want to use Git for version control. Click Create when done.
On the next page we want to copy the repository’s URI to the clipboard so that we can clone the repository using our local Git client. Click the copy icon as shown below.
On your development machine, open a shell/terminal and clone the Git repository with the following command, replacing the working directory for your usual Git repo directory and the URI for your own VSTS repository:
cd ~/source/repos/ git clone https://anchorloop.visualstudio.com/_git/PSPerceptron
You will probably get asked for your credentials as part of the clone operation. If you’re new to Git this will create a new directory with the same name as your VSTS repository, which is your local copy of the source code repository.
Our first job is just to get our module into source control, so copy your module directory into the cloned repository. The directory structure you’re probably looking to make is as follows:
~/source/repos/PSPerceptron |_ PSPerceptron |_ PSPerceptron.psd1 |_ PSPerceptron.psm1
This is the typical directory structure that the Publish-Module cmdlet expects when we come to publishing the module to a package feed.
Finally, commit your module and push it back to VSTS:
git add * git commit -m "Initial commit" git push
You should now be able to browse the module code in VSTS.
Step 2: Define a continuous publishing process
Now that we have our module in source control, we need to define a build process which will automatically publish the module to the VSTS package feed whenever we push code to our source control repository.
The best way to define a CI pipeline for PowerShell modules in VSTS is with YAML configuration files. The reason I prefer these to conventional UI-driven build definitions in VSTS is that the YAML file is easy to branch to many PowerShell modules that you might want to build and publish this way. If you had a hundred PowerShell modules that you wanted to host in a package feed, would you want to define a build pipeline for each individually via the UI? That’s a lot of work. I would rather VSTS derive the build process dynamically from a YAML file I can branch to each module as needed.
First, enable YAML build configurations in your VSTS instance. Click your user icon and select Preview features, choose for this account and toggle Build YAML definitions on.
Now, create a new file in the local Git repository for your module called: .vsts-ci.yml. Note that the filename begins with a dot. This is where we’re going to define the publishing process. After creation, open it up in an editor of your choice and paste in the following:
queue: Hosted VS2017 trigger: - master steps: - task: NuGetToolInstaller@0 displayName: "Install nuget.exe and add to PATH" inputs: versionSpec: 4.6.* - powershell: | Write-Host "PowerShell Module publishing code goes here." displayName: "Publish PowerShell module" failOnStderr: true
Let’s go through this element-by-element so that we understand what everything means.
- queue: Describes what build queue is going to handle the process. Here we’re using the Visual Studio 2017 build machines hosted by Microsoft. As far as hosted choices go, it’s usually between Visual Studio on Windows, Linux or Mac. We want the machine to be Windows for access to PowerShell 5+ (PackageManagement modules included) rather than for access to Visual Studio in particular, so this queue is the best choice for that.
- trigger: Defines a list of branches that should trigger the build process when a change is pushed to them. We only want to republish the module for every checkin to the master branch, so ours is a list with one entry.
- steps: Defines a list of sequential build steps that make up the build process.
- task: Describes a VSTS build task in the format Name@MajorVersionNumber. It’s important to include the major version numbers because the same task is not usually backwards compatible with a different major version of itself. This format is mandatory, even if there is only one major version number of a given task (with with NuGetToolInstaller which is still at v0). For further details see: https://github.com/Microsoft/vsts-agent/blob/master/docs/preview/yamlgettingstarted-tasks.md
- displayName: A string to display in the UI for each build step.
- inputs: A list of parameters for each VSTS build task. Each task expects different parameters, you can check what valid inputs are for a task by viewing its task.json definition on GitHub. E.g. https://github.com/Microsoft/vsts-tasks/blob/master/Tasks/NuGetToolInstallerV0/task.json
- powershell: Defines a PowerShell script that should run as part of the build.
- failOnStderr: A boolean that determines whether the powershell step should fail if anything is emitted on stderr during execution. Otherwise the pass/fail state will be based on $LASTEXITCODE.
In a nutshell, the steps install a specific version of NuGet on %PATH%, then executes some PowerShell, which at the moment is just a Write-Host. The NuGetToolInstaller task is mainly to ensure nuget.exe is available to use without us having to figure out a path. We don’t really want to go hunting around for it on a hosted build machine, which could change periodically.
Why is easy access to nuget.exe even needed? The Publish-Module cmdlet that we are going to use to publish our PowerShell module to the VSTS package feed depends on nuget.exe being available and configured correctly for publishing to VSTS. OneGet doesn’t do a particularly good job of wrapping nuget.exe, it needs to be available and pre-configured to work with VSTS or Publish-Module will just barf.
Save the YAML file but don’t push it to master yet. Let’s flesh out the PowerShell step to do the publishing first.
Step 3: Publishing to the VSTS package feed
Here’s the PowerShell that we’re going to integrate into our build pipeline:
$patUser = 'email@example.com' $patToken = 'INSERT PAT TOKEN HERE' $securePat = ConvertTo-SecureString -String $patToken -AsPlainText -Force $credential = New-Object System.Management.Automation.PSCredential($patUser, $securePat) Register-PSRepository -Name 'VSTS' -SourceLocation 'https://anchorloop.pkgs.visualstudio.com/_packaging/Packages/nuget/v2/' -PublishLocation 'https://anchorloop.pkgs.visualstudio.com/_packaging/Packages/nuget/v2/' -InstallationPolicy Trusted -Credential $credential -Verbose nuget.exe sources add -name 'VSTS' -Source 'https://anchorloop.pkgs.visualstudio.com/_packaging/Packages/nuget/v2/' -UserName $patUser -Password $patToken Publish-Module -Path ".\$env:BUILD_REPOSITORY_NAME" -Repository 'VSTS' -Credential $credential -NuGetApiKey 'VSTS' -Verbose
This code does the following:
- Constructs a PSCredential from a VSTS personal access token (PAT), which will be used to register VSTS as a repository and publish the module. Think of PATs as access tokens – they’re safer than embedding your user credentials in scripts because you can lock each PAT down such that it only has access to the specific thing you want to use it for. Another user would not be able to use this PAT to log in as you. We’ll create a PAT for this script in the next section.
- We register VSTS as both a PowerShell repository with OneGet and a NuGet source separately. As I mentioned earlier, OneGet doesn’t wrap NuGet very well (even though it depends on it) and it won’t register the repository as a NuGet source for us, unfortunately. It would be nice if it did in future. Note that OneGet uses the v2 endpoints of the package feed. If you’re hunting around NuGet.config files for your endpoint URI, you’re likely to find the v3 endpoint. Just replace ‘v3/…’ and everything after it in the URI with ‘v2/’.
- Finally, we publish the module to VSTS with the Publish-Module cmdlet. Note that we need to supply the PSCredential and a NuGetApiKey, which for VSTS can be any string (it’s pretty much ignored, but is a mandatory parameter of the cmdlet). The Path parameter is the path to the module directory in the Git repo. For simplicity I tend to always make the module name the same as the Git repo name, so I can just refer to this build variable in the script and it works when I branch the YAML file into other module repos. But you could do this any way you like.
Once you’ve modified the above for your VSTS user account, package feed URIs, etc., paste it into your YAML file in place of the existing Write-Host placeholder. Now all we need to do is generate a PAT for the script.
Step 4: Generating a PAT for the publishing script
In VSTS, click your user icon along the top bar then Security. You’ll be taken to your Personal access tokens page. Click Add.
Name it sensibly, give it an expiry date and select the Packaging (read and write) scope only. This locks the PAT down such that it can only be used to publish packages and nothing else.
Click Create Token. VSTS will display the PAT to you this one time only, so make sure you copy it and paste it into the PowerShell script as the value of the $patToken variable.
Now we’re ready to go for a test drive.
Step 5: Pushing the CI definition
Push the YAML build definition with the following commands:
git add .vsts-ci.yml git commit -m "Add .vsts-ci.yml" git push
Now in VSTS, if you browse to Build and Release for your project you will see that a new build definition has been created and the first build should be in progress. All being well, your build job should pass.
Any future extensions you might want to make to the build process, for example running some Pester tests before publishing, can be done by extending the YAML build definition that is now alongside your module in source control.
Advice for consuming packages from the VSTS feed
I thought I would close with some advice for consuming PowerShell modules from your private VSTS package feed. First, you need to register VSTS as a trusted PSRepository on any machine you wish to consume packages on. This is the same line of code as in the publishing script:
Register-PSRepository -Name 'VSTS' -SourceLocation 'https://anchorloop.pkgs.visualstudio.com/_packaging/Packages/nuget/v2/' -PublishLocation 'https://anchorloop.pkgs.visualstudio.com/_packaging/Packages/nuget/v2/' -InstallationPolicy Trusted -Credential $credential -Verbose
You’ll need to assemble a PSCredential again to do this, every interaction with VSTS needs to be authenticated. I suggest rather than reusing the PAT we generated for publishing packages, that you generate a separate one with only the Packaging (read) scope to lock down consumers even further. You don’t need the write permission to use packages from the feed.
To consume packages from the feed, just use Install-Module:
Install-Module -Name PSPerceptron -Repository VSTS -Credential $credential
Note that if you have multiple repositories installed, and you likely will considering you will still have access to PSGallery, Install-Module will expect you to clarify which repository you intend to use with the Repository parameter. This becomes mandatory with multiple repositories registered.