Monday, October 30, 2017

Removing Adsense

About a dozen years ago I signed up for AdSense on this blog.  Here is an odd thing about AdSense; They have a $100 payment threshold.  As of today I'm $0.80 cents away from that magical threshold for the first time.  As soon as I get that check I'll be removing the ads.  One more month should do it.  Thank you for putting up with them for so long.

I will occasionally post links to stuff on Amazon that is relevant, but I'm done with intrusive advertising after this. 

Thursday, October 12, 2017

Fixed: ACK/RST intermittently connecting to RPC/Kerberos/LDAP ports on a Server 2012 R2 Domain Controller

I worked an interesting case this week.  My customer had widespread domain controller outages on their Windows Server 2012R2 DCs.  A network trace showed that connections to the domain controllers were intermittently being refused with an ACK/RST (acknowledgement/reset) in response to the initial SYN (TCP Hello) packet.

It only happened under load.
It wasn't a firewall.
It wasn't the AV software.
It wasn't a laundry list of other things.

Here is the odd bit.  In a netsh trace on the DC, we could see the SYN packet was getting to the DC, but it wasn't getting to the RPC endpoint mapper, lsass, or netlogon.

Why?

That took the weekend and Monday to figure out.  What we discovered was there was a TDI filter driver in the networking stack that was, for cause unknown, slowing down network connections to the DC.

TDI "Transport Driver Interface" is an older Windows technology that lets software drivers hook between the TCP/IP driver stack and applications.




Further digging led us to find this KB article from Trend Micro that indicated a problem with the VmWare NSX Network Introspection Driver (vnetflt.sys) was causing the trouble.  This driver is installed with the VmWare Tools Package on the DCs, and it was several years out of date.

The solution from the KB worked.  We disabled that driver via the registry and the problem stopped immediately after a reboot.

Specifically, in the registry:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vnetflt\
Key: Start
Type: REG_DWORD
Value: 4

Magic.

It's interesting to note that I haven't seen or heard of any other similar issues with this.  I assume it was an interaction between the ancient VmWare tools on the box and some of their security software or a Software Update, but I'm not sure.  They wanted to do that root cause analysis in their lab independently.  I asked them to cc me on the root cause, and I'll update this blog post if they do.

Lessons Learned:
  • You really should keep your VmWare tools updated.  
    • Like all drivers, a fire and forget mentality can seriously bite you.
  • You can convert a netsh network trace ETL file to a big text file with this command:
    • netsh trace convert input="input.etl" output="output.log" 
  • You can capture a netsh network trace ETL and add in extra ETL providers, like the TCPIP driver messages, with this command: 
    • netsh trace start capture=yes overwrite=yes maxsize=2048  tracefile=c:\output.etl scenario=netconnection provider={EB004A05-9B1A-11D4-9123-0050047759BC} keywords=0xffffffffffffffff level=0xff provider="Microsoft-Windows-TCPIP" keywords=0xffffffffffffffff level=0xff
  • The Potbelly Sandwich Shop in Normal, Illinois makes a fantastic grilled cheese sandwich.

Hola Voyager 2


NASA has a cool page that lets you see what the (Deep Space Network) is talking to at any given time here: Deep Space Network Now

Today they are chatting with Voyager 2, a spacecraft that launched the year after I was born. It has been flying for almost 40 years without any hands-on maintenance or repair. That’s an achievement.

I’m awestruck that the spacecraft is still alive, but they are still doing useful science with it. Right now, today, they are communicating with it to download science data.

So how does this communication compare to the WiFi in my house?
 The received signal strength on Earth is 0.000 000 000 000 000 000 000 133 watts of power.
 The WiFi in my house is 0.075 watts.

 On my WiFi I can download 150 Megabytes per second.
 Voyager 2 is transmitting at 159 bits per second.

This blog post, not counting the images and links is 1,294 8-bit bytes. For us to transmit this blog entry to Voyager it would take over a minute just to send the message. Then it would take 32 hours for the message to travel the 17,300,000,000 kilometers to get to the spacecraft and the “OK” message to travel back to us on Earth.

Godspeed, Voyager 2. Please tell them we’re coming; It’s just taking a bit longer than planned.