Dan Wanek's notes when first developing the ruby implementation of NTLM.

This week the WinRM ruby gem version 1.6.0 was released and there is really just one new feature it delivers but it is significant: NTLM/Negotiate authentication. Simply stated this provides a safe, low friction authentication and encryption mechanism long available in native Windows remote management tooling but absent in cross platform tools that implement the WinRM protocol.

In this post I'll highlight why I think this is a big deal, share a little history behind the feature release and discuss why authentication/encryption over WinRM has historically been a problem in the cross platform ecosystem.

Tell me again? Why is this awesome?

This is awesome because it means that assuming you are connecting to a Windows Server OS of Windows 2012 R2 or later, you can now securely connect from either a Windows or Linux application leveraging the ruby WinRM gem without any preconfiguration of the target machine. Client OS SKUs (like Windows 7, 8 and 10) and server versions 2008 R2 and prior still need to have WinRM explicitly enabled.

No SSL setup and no WinRM configuration that compromises its security. Just to be clear SSL is still a good idea and encouraged for production nodes, but if you are just trying to get your feet wet with Windows remote execution using tools like Chef or Vagrant, it is now much easier to accomplish and keep the security bar at a sane level.

How did this come to be?

I am a developer at Chef and we use the WinRM gem inside of Knife-Windows in order to execute remote commands on a Windows node from either Windows or Linux. We use a monkey patch gem called winrm-s to provide Negotiate authentication and encryption BUT it only works from windows workstations because it leverages the native Win32 APIs available only on Windows. Also its just monkey patches on top of the winrm and httpclient gems and therefore quite fragile.

My initial objective was to simply port the patches in winrm-s to winnrm and possibly the httpclient gem so that we could provide the same functionality but the implementation would live downstream where it belongs.

Well I remembered I had seen a PR in the WinRM repo that mentioned providing Negotiate auth implementation. It was submited by Dan Wanek, the original author of the WinRM gem. The PR was from late 2014 and I never really looked at it but filed it away in my mind as something worth looking at when the time came to drop winrm-s. So the time came and I quickly noticed that it did not seem to use any native Win32 APIs but leveraged another gem, rubyntlm (also authored by Dan Wanek), which appeared to be a pure ruby implementation of NTLM/Negotiate. That means it would work on Linux in addition to Windows which would be really really great.

It was pretty straight forward to get working and needed just a small amount of tweaking to be production ready - impressive since it had been dormant for over a year. Once it was up and running I verified that it indeed worked both from Windows and Linux. Nice work Dan!

The cross platform Windows remote execution landscape

I've written several posts covering different aspects of WinRM. Like this one and this one and also this one. Its a protocol that many native windows users likely take for granted and perhaps even forget they are using it when they are using its more full featured cousin: Powershell Remoting. However those who dont have access to direct Powershell because they either run on other operating systems that need to talk to Windows machines or are on Windows but use tools that are portable to non windows platforms are likely using a library that implements the WinRM protocol.

There are many such libraries:

The ruby WinRM gem (covered in this post)
WinRM for GO
pywinrm python library
Xebialabs Overthere Java library for remote execution

This is just a list of the most popular libraries but there are many more. These are used by well known "DevOps" automation tools such as Chef, Ansible, Packer, Vagrant and others.

WinRM is a simple SOAP based client/server protocol. So the above libraries merely implement a web service client that issues requests to a WinRM Windows service and interprets the responses. Basically these exchanges result in:

Creating a Shell
Creating a Command
Requesting Command Output and Exit Code

The output can include both Standard Output and Standard Error streams.

Kind of like SSH but not.

What about Powershell Remoting

I'll save the details for another post but be aware there is a more modern protocol called the Powershell Remoting Protocol (PSRP). There is no cross platform implementation that I am aware of. It uses the same SOAP based wire protocol - MS-WSMV (Web Services Management Protocol Extensions for Windows Vista). I just love the "shout out" to Vista here.

PSRP is more feature rich but more difficult to implement. I've been playing with a ruby based partial implementation and will blog more details soon. You can run powershell commands with WinRM but you are shelling out to Powershell.exe from the traditional "fisher-price" Windows command shell.

Securing the transport

If you are using native WinRM on Windows (likely via Powershell Remoting) the most popular methods of authentication and encryption are (but are not limited to):

Negotiate

To quote the Windows Remote Management Glossary, Negotiate Authentication is defined:

A negotiated, single sign on type of authentication that is the Windows implementation of Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO). SPNEGO negotiation determines whether authentication is handled by Kerberos or NTLM. Kerberos is the preferred mechanism. Negotiate authentication on Windows-based systems is also called Windows Integrated Authentication.

Mmmmmmm. Lets all take a minute or two and think about just what this means and how it might guide our relationships and shape our perspectives on global social injustice.

Yeah its complex right? Anyways its also secure. Its much better than emailing your password to the contacts in your address book.

Kerberos

You know this Windows Remote Management Glossary is pretty darn smart. Lets just see what it has to say about Kerberos:

A method of mutual authentication between the client and server that uses encrypted keys. For computers running on a Windows-based operating system, the client account must be a domain account in the same domain as the server. When a client uses default credentials, Kerberos is the authentication method if the connection string is not one of the following: localhost, 127.0.0.1, or [::1].

Takeaways here are its "mutual", "encrypted" and importantly "for computers." Again, better than emailing your password to the contacts in your address book.

Basic Authentication

Secrets are hard. Not only technically but emotionally as well. There is no small effort involved hiding the truth. Well thank goodness Basic Authentication makes it easy to share our secrets easily. Ah sweet freedom. Here credentials are transmitted in plain text. I like to think of it as a "giving" protocol. Sure we all think our secrets are worth being kept but are they? Really? Just use Basic Authentication and you might find out.

SSL

This really isn't so much an authentication mechanism but it is a familiar means of securely transporting data from one point to another where the contents are only to be accessible to the sender and receiver. WinRM communication can use either HTTP or HTTPS (SSL). HTTP is the default but HTTPS provides an added layer of encryption.

One of the critical keys to securely using SSL is having a valid certificate issued by a reputable certification authority that serves to ensure that those on either side of the communication are who they say they are. Without this, for example using a non validated self signed certificate, you run the risk of a "Man in the Middle Attack". Not a nice man who offers to fix your tire, provide financial assistance or offer grievance counseling after the loss of a loved family member or pet. Rather a mean man who wants to take what you have and not give it back. He can do that because the authenticity of your certificate cannot be validated and therefore he can stand in the middle between you and the remote windows machine pretending to be that machine.

The dilemma of sane encryption using cross platform libraries

Some of the dilemmas I'll mention are shared in both native windows and cross platform libraries. For example, the friction of getting SSL up and running is pretty much the same on both sides. The key difference is that cross platform libraries may not (usually don't in fact) have access to all the authentication mechanisms listed above or getting them installed and configured is far less than clear.

Good news: SSL is Everywhere, Bad News: SSL is a pain to setup everywhere

Of course this statement is somewhat relative. There are those that are familiar with the basic rules of computer security and work with the knobs and levers of these mechanisms frequently enough that it is straight forward for them to setup. Even among the technically savvy, this is NOT the majority and its especially true in the world of WinRM vs. SSH. Not because SSH users are smarter, but because in Linux land, SSH is just so ubiquitous and pervasive its hard not to deal with regularly enough for it to be unfamiliar.

Here are some points to highlight the friction and pitfalls of SSL over WinRM:

Its not on by default and there are several steps to set it up
Without a "valid" certificate its still not secure (but better than Basic Authentication over HTTP) and valid certificates take effort to obtain.
Bootstrapping problem: How do I get the valid certificate onto the remote machine before establishing a secure connection? There are several ways to do this depending on your cloud provider or image prep system but that assumes you have a cloud provider or an image prep system.

Kerberos: I cant tell you in a paragraph how to set it up and get it working

First just getting the right library can be the worst part - one that's compatible with your OS, architecture and the language runtime of your WinRM library. Most cross platform libraries support it but its less than trivial to get working.

Negotiate is likely not implemented

Well on ruby it is now, however, its the only non native Windows implementation I am aware of (which does not mean that there are not others). If it is implemented, and again - it is on Ruby, its secure and it "Just Works."

Basic auth is available everywhere, horrible everywhere and slightly painful to setup but easier than SSL

If you dig into almost all of the Readme files of the cross platform WinRM libraries, they will all tell you how to run horrible commands on your computer that put out the Welcome Mat for the bad guys. I've listed these in a few posts and for once I will not do so here.

Not only are running these commands a bad idea in general (certainly on production nodes) but again they represent a bootstrapping problem. Before you can successfully talk to the machine they have to be run. There are ways to accomplish this but again rely on cloud APIs or pre-baking images.

Next steps, caveats and how can we make this even better

Its totally awesome that NTLM/Negotiate authentication is now available as a cross platform option but it only lays down the foundation.

Its not the default and consuming applications need to "turn it on"

When using the WinRM gem, consumers must specify which authentication transport they want to use. There is no default. So today applications that specify ":plaintext, basic_auth: true" will continue to use basic authentication.

Chef and Test-Kitchen support coming very soon, and Vagrant support on the way

I am a developer at Chef and my PR for porting this into Knife-Windows is imminent. I will also be porting to Test-Kitchen's winrm transport and I plan to submit a PR to do the same for Vagrant.

Update: PRs to knife-windows, test-kitchen and vagrant are all submitted.

Still no support outside of Ruby

For instance Ansible (Python) and Packer (GO) - both very popular tools that help manage a data center have yet to have working Negotiate Auth implementations available.

The fact is its hard and tedious to implement something like an encryption specification solely from following a specification PDF. Further, there are other ways to get secure so its not a "show stopper." However as more become aware of the rubyntlm library, referencing such a library makes it much easier to implement.

Its been well over a year since Salim Afiune started the effort of making Test-Kitchen (a popular infrastructure testing tool) compatible with Windows. I got involved in late 2014 to improve the performance of copying files to Windows Test instances from Windows or Linux hosts. There was a lot of dust flying in the air as this work was nearing completion in early 2015. Shawn Neal nicely refactored some of the work I had submitted to the winrm gem into a spin off gem called winrm-fs. At the same time, Fletcher Nichol was feverishly working to polish off the Test-Kitchen work by Chef Conf 2015 and was making some optimizations on Shawn's refactorings and those found their way into a new gem winrm-transport.

Today we are releasing a new version of the winrm gem that pulls in alot of this work into the core winrm classes and will make it possible to pull the rest into winrm-fs, centralizing ruby winrm client implementations and soon deprecating the winrm-transport gem. If you use the winrm gem, there are now some changes available that improve the performance of cross platform remote execution calls to windows along with a couple other new features. This post summarizes the changes and explains how to take advantage of them.

Running multiple commands in one shell

The most common pattern for invoking a command in a windows shell has been to use the run_cmd or run_powershell_script methods of the WinRMWebService class.

endpoint = 'https://other_machine:5986/wsman'
winrm = WinRM::WinRMWebService.new(endpoint, :ssl, user: 'user', pass: 'pass')
winrm.run_cmd('ipconfig /all') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end

winrm.run_powershell_script('Get-Process') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end

Under the hood both run_cmd and run_powershell_script each make about 5 round trips to execute the command and return its output:

Open a "shell" (equivalent of launching a cmd.exe instance)
Create a command
Request command output (potentially several calls for long running streamed output or long blocking calls)
Terminate the command
Close the shell

The first call, opening the shell, can be very expensive. It has to spawn a new process and involves authenticating the credentials with windows. If you need to make several calls using these methods, a new shell is spawned for each one. And things are even worse with powershell since that spawns yet another process (powershell.exe) from the command shell incurring the cost of creating a new runspace on each call. Stay tuned for a future winrm release (likely 1.6) that implements the powershell remoting protocol (psrp) to avoid that extra overhead.

While there is currently a way to make several calls in the same shell, its not very friendly or safe (you could easily end up with orphaned processes running on the remote windows machine). Typically this is not such a big deal because most will batch up several calls into one larger script. But here's the kicker: you are limited to a 8000 character command in a windows command shell (again stay tuned for the psrp implementation to avoid that). So imagine you are copying a 100MB file over the command line. Well, you will have to break that up into 8k chunks. While this may provide an excellent opportunity to review the six previous Star Wars episodes as you wait to transfer your music library, its far from ideal.

Using the CommandExecutor to stream commands

This 1.5 release, exposes a new class, CommandExecutor that you can use to make several commands from the same shell. The CommandExecutor provides run_command and run_powershell_script methods but these simply run a command, collect the output and terminate the command. You get a CommandExecutor by calling create_executor from a WinRMWebService instance. There are two usage patterns:

endpoint = 'https://other_machine:5986/wsman'
winrm = WinRM::WinRMWebService.new(endpoint, :ssl, user: 'user', pass: 'pass')
winrm.create_executor do |executor|
  executor.run_cmd('ipconfig /all') do |stdout, stderr|
    STDOUT.print stdout
    STDERR.print stderr
  end
  executor.run_powershell_script('Get-Process') do |stdout, stderr|
    STDOUT.print stdout
    STDERR.print stderr
  end
end

This yields an executor to a block that uses the executor to make calls. When the block completes, the shell opened by the executor will be closed.

The other pattern:

endpoint = 'https://other_machine:5986/wsman'
winrm = WinRM::WinRMWebService.new(endpoint, :ssl, user: 'user', pass: 'pass')
executor = winrm.create_executor

executor.run_cmd('ipconfig /all') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end
executor.run_powershell_script('Get-Process') do |stdout, stderr|
  STDOUT.print stdout
  STDERR.print stderr
end

executor.close

Here we are responsible for the executor and this closing the shell it owns. Its important to close the shell because if it remains open, that process will continue to live on the remote machine. Will it dream?...we may never know.

Self signed SSL certificates and ssl_peer_fingerprint

If you are using a self signed certificate, you can currently use the :no_ssl_peer_verification option to disable verification of the certificate on the client:

WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass, :basic_auth_only => true, :no_ssl_peer_verification => true)

This is not ideal since it still risks "Man in the Middle" attacks. Still not completely ideal but better than completely ignoring validation is using a known fingerprint and passing that to the :ssl_peer_fingerprint option:

WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass, :ssl_peer_fingerprint => '6C04B1A997BA19454B0CD31C65D7020A6FC2669D')

This ensures that all messages are encrypted with a certificate bearing the given fingerprint. Thanks here is due to Chris McClimans (HippieHacker) for submitting this feature.

Retry logic added to opening a shell

Especially if provisioning a new machine, it's possible the winrm service is not yet running when first attempting to connect. The WinRMWebService now accepts new options :retry_limit and :retry_delay to specify the maximum number of attempts to make and how long to wait in between. These default to 3 attempts and a 10 second delay.

WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass, :retry_limit => 30, :retry_delay => 10)

Logging

The WinRMWebService now exposes a logger attribute and uses the logging gem to manage logging behavior. By default this appends to STDOUT and has a level of :warn, but one can adjust the level or add additional appenders.

winrm = WinRM::WinRMWebService.new(endpoint, :ssl, :user => myuser, :pass => mypass)

# suppress warnings
winrm.logger.warn = :error

# Log to a file
winrm.logger.add_appenders(Logging.appenders.file('error.log'))

If a consuming application uses its own logger that complies to the logging API, you can simply swap it in:

winrm.logger = my_logger

Up next: PSRP

Thats it for WinRM 1.5 but I'm really looking forward to productionizing a spike I recently completed getting the powershell remoting protocol working, which promises to improve performance and opens up other exciting cross platform scenarios.

Hurry Up and Wait!

Tales from an automation engineer

Sane authentication/encryption arrives to ruby based cross platform WinRM remote execution / January 23, 2016 by Matt Wrock