Thursday, October 12, 2017

Fixed: ACK/RST intermittently connecting to RPC/Kerberos/LDAP ports on a Server 2012 R2 Domain Controller

I worked an interesting case this week.  My customer had widespread domain controller outages on their Windows Server 2012R2 DCs.  A network trace showed that connections to the domain controllers were intermittently being refused with an ACK/RST (acknowledgement/reset) in response to the initial SYN (TCP Hello) packet.

It only happened under load.
It wasn't a firewall.
It wasn't the AV software.
It wasn't a laundry list of other things.

Here is the odd bit.  In a netsh trace on the DC, we could see the SYN packet was getting to the DC, but it wasn't getting to the RPC endpoint mapper, lsass, or netlogon.


That took the weekend and Monday to figure out.  What we discovered was there was a TDI filter driver in the networking stack that was, for cause unknown, slowing down network connections to the DC.

TDI "Transport Driver Interface" is an older Windows technology that lets software drivers hook between the TCP/IP driver stack and applications.

Further digging led us to find this KB article from Trend Micro that indicated a problem with the VmWare NSX Network Introspection Driver (vnetflt.sys) was causing the trouble.  This driver is installed with the VmWare Tools Package on the DCs, and it was several years out of date.

The solution from the KB worked.  We disabled that driver via the registry and the problem stopped immediately after a reboot.

Specifically, in the registry:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vnetflt\
Key: Start
Value: 4


It's interesting to note that I haven't seen or heard of any other similar issues with this.  I assume it was an interaction between the ancient VmWare tools on the box and some of their security software or a Software Update, but I'm not sure.  They wanted to do that root cause analysis in their lab independently.  I asked them to cc me on the root cause, and I'll update this blog post if they do.

Lessons Learned:
  • You really should keep your VmWare tools updated.  
    • Like all drivers, a fire and forget mentality can seriously bite you.
  • You can convert a netsh network trace ETL file to a big text file with this command:
    • netsh trace convert input="input.etl" output="output.log" 
  • You can capture a netsh network trace ETL and add in extra ETL providers, like the TCPIP driver messages, with this command: 
    • netsh trace start capture=yes overwrite=yes maxsize=2048  tracefile=c:\output.etl scenario=netconnection provider={EB004A05-9B1A-11D4-9123-0050047759BC} keywords=0xffffffffffffffff level=0xff provider="Microsoft-Windows-TCPIP" keywords=0xffffffffffffffff level=0xff
  • The Potbelly Sandwich Shop in Normal, Illinois makes a fantastic grilled cheese sandwich.

Hola Voyager 2

NASA has a cool page that lets you see what the (Deep Space Network) is talking to at any given time here: Deep Space Network Now

Today they are chatting with Voyager 2, a spacecraft that launched the year after I was born. It has been flying for almost 40 years without any hands-on maintenance or repair. That’s an achievement.

I’m awestruck that the spacecraft is still alive, but they are still doing useful science with it. Right now, today, they are communicating with it to download science data.

So how does this communication compare to the WiFi in my house?
 The received signal strength on Earth is 0.000 000 000 000 000 000 000 133 watts of power.
 The WiFi in my house is 0.075 watts.

 On my WiFi I can download 150 Megabytes per second.
 Voyager 2 is transmitting at 159 bits per second.

This blog post, not counting the images and links is 1,294 8-bit bytes. For us to transmit this blog entry to Voyager it would take over a minute just to send the message. Then it would take 32 hours for the message to travel the 17,300,000,000 kilometers to get to the spacecraft and the “OK” message to travel back to us on Earth.

Godspeed, Voyager 2. Please tell them we’re coming; It’s just taking a bit longer than planned.

Wednesday, September 20, 2017

Fixed: "The application has failed to start because its side-by-side configuration is incorrect."

It's been a fun week of AppCompat work for me so far.  I took an ancient proprietary "Runs only on XP" application and cajoled it up into running on Windows 7.  That was fun!

Here is one cool thing I ran into along the way.  One of the application's executables would start and immediately die with this error:

"The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail."

In the event log we faced a mostly gibberish error that indicated it couldn't find x86_microsoft.vc90.crt AKA the 32-bit Visual C++ 9.0 runtime AKA Visual C++ 2008. 

The solution was to download and install the Visual C++ 2008 SP1 runtime from Microsoft.

If that link happens to rot away, the search keywords to find that package are "Microsoft Visual C++ 2008 SP1 Redistributable Package (x86) Download"

Viola! The application worked!

Monday, August 21, 2017

Eclipses and Volcanoes

Today, like many other Americans, I saw an eclipse. This is my second; the first was when I was in elementary school in Madison Tennessee. I understand the physics behind today’s eclipse much more than I did as a child, but I still have the same wonder that I had then.

I think I’d like to see a volcano.

Thursday, August 10, 2017

Fixed: Fusion 360 streamer.log is huge and taking up all of my disk space

I had a low disk space problem on my PC, which is odd because I just archived off a probably unhealthy amount of industrial robot pictures.

Digging around, I found that the Fusion 360 Streamer.Log was 31GB, yeouch!

According to the fine folks at AutoDesk this is a known issue and they will have a fix for it soon.  In the interim, you can fix it to get the disk space back.

To fix it:
  • Use Task Manager to kill streamer.exe
  • Delete
    • C:\Users\<your username here>\AppData\Local\Autodesk\autodesk.webdeploy.streamer.log
Fusion 360 still works and I can bring back some of my robots.  Life is good.


Sunday, June 11, 2017

Printing in ABS on the Monoprice MP Select Mini

Today I did my first successful ABS prints on the MP Select Mini.

The successful print was with SoliDoodle ABS Filament printed at 240C and the Bed at 70C.  This was printed on the glass bed I wrote about previously with Elmers Washable Disappearing Purple Glue Stick applied to promote adhesion and I used a raft.  I blocked the fan duct with masking tape to prevent warping.  I also found I needed the fan set at 90% for the entire print to move some of the heat away from the heat block so the radiant heat from the block doesn't distort my overhangs.

Removing the print from the bed was easy.  I slid a paring knife under the still-warm raft and it popped right off.

This part is a replacement door slider for my Brother's Arrow Brand metal shed.  At 1.75mm layers, 4 Perimeters, 50% infill the part is much sturdier than the heavily weathered part it is replacing.  These parts are commercially available at $12 for a set of four.  I modeled the part in Fusion 360 and the STL is available on Thingiverse at

I had several issues getting the ABS to stick, and I won't consider the problem fully resolved until I can print without the raft.  I tried a number of combinations that did not work for me.

  • 3M Scotch Contractor Grade Masking Tape #2020CG 240C Ext, 50C Bed
  • Masking Tape, 240C/60C
  • Masking Tape, 240C/70C*
  • Masking Tape and Elmer's Washable disappearing Purple Glue Stick (Amazon Link) 240C/50C
  • Masking Tape and Glue Stick 240C/60C
  • Masking Tape and Glue Stick 240C/70C*
  • Masking Tape and Glue Stick 240C/80C**
  • Bare Glass 240/70C*

* The Control panel only lets you set the temperature to 60C, but you can set it higher than this via G-Code.

** The printer took forever to reach the 80C bed temperature.  I helped it along by putting a piece of cloth (the square pad you use to fetch hot things from the oven) on the bed while it was preheating, then pulling it off as it started to print.  If future testing shows that higher temperatures help with this I'll have to insulate the bed.

Wednesday, June 7, 2017

Glass Bed upgrade for The Monoprice Select IIIP Mini 3d printer

Today I installed a glass bed on my Monoprice Select Mini 3d printer.  I did it a bit differently from most of the conversions I've seen online.

I purchased 3/32" inch (2.38mm) glass from Home Depot, and cut it to size with a carbide glass cutter. Next I used a knife sharpening stone and some vegetable oil to grind down the cut and factory edges so they'd be safe to handle.  I rounded off the corners a bit too.  With that done, I peeled up the bed tape from my aluminum bed, scraped off the high points of the bed's dings and scratches with a razor, and set the bed in place.  I held the glass in place with 4 Wal-Mart 1/2 inch Mini Binder clips.

This is, so far, fairly standard stuff.  If you were following most online tutorials the next step would be to install a clip around the X axis gantry to raise the Z axis home position by the height of the glass.  I did something different.  I raised the hot end instead.  The hot end is held in place by a 1.5mm grub screw.  I removed the fan, loosened the grub screw, took out the print head, removed the PTFE spacer block, reinstalled the print head, raised it well above its old height, and re-tightened the screw.  Then I carefully homed the printer, loosened the screw, lowered the printhead into the correct position, and retightened it.  Viola, Glass print bed and no spacer required.

There is a downside to this approach.  That PTFE tube spacer helps filament feed smoothly back into the print head if you retract it too far.  I'll be replacing this very soon as part of my NinjaFlex fully-constrained-filament-path upgrade.

It's worth noting that I didn't intentionally choose to do my glass bed conversion this way.  I printed a glass bed spacer from Thingiverse ( ) and disassembled the printer to install it.  Unfortunately I found that it didn't snap on to my x gantry properly, and was "sprung" into position such that it would never be accurate on my printer.  That's probably a printer calibration problem on my side, and not a problem with the model.  This way appears to be working, so I'm happy with it.

Bonus content:

Tiny gnomes stole one of my fan clips while I had this apart.  I foiled their evil plot though; I fabricated a serviceable replacement from a paper-clip.