Windows 10 1809 RDP “black screen of death”

Recently I noticed some serious issues connecting from my Windows 10 1809 laptop to another Windows 10 1809 hosts via RDP over VPN (OpenVPN over TCP).
The RDP server replied correctly to the initial connection and asked for login, but after entering username and password… nothing, only a deep, dark, freightening black screen of death :(

No problems with RDP server with other previous Windows versions (Windows 7, Server 2012, Server 2016), looking to my OpenVPN server I found no errors in the logs, but on my OpenVPN client (v 2.4.7-I603) I found some interesting log records:

Mon Feb 25 15:13:58 2019 us=737935 openvpnclient/xxx.xxx.xxx.xxx:48404 MULTI: packet dropped due to output saturation (multi_process_incoming_tun)

Looking through OpenVPN forums and ML I found no help, a few old issues with this kind of errors but no solution at all, and nothing that makes sense with my setup.
So I started to look around for Windows issues or new features introduced with 1809 service pack (forgive me but I still call those updates in the old way :P ) and I found some interesting news, MS introduced RDP UDP Transport Extension Version 2 and enabled by default UDP protocol for the RDP client.
Seems like this cause some fragmentation isssue on UDP protocol that cause the VPN connection to drop.

Actually there’s no patch to solve this problem and seems like the few threads on Technet didn’t have any official reply from MS, for now the only workaround is to enable  “Turn Off UDP On Client” using local group policies:

  • start group policy management with gpedit
  • navigate to computer configuration\Admin Templates\ Windows Components\Remote Desktop Services\ Remote Desktop Connection Client
  • open “Turn Off UDP On Client”, enable it and apply changes

Now without any reboot you should be able to connect to 1809 hosts without the ugly “black screen of death”.


Edit php files via WebDAV

One thing I really like is to give developers exactly the right access they need for work, nothing more nothing less.

For example for LAMP stacks I usually avoid using unsecure protocols like ftp or network shares to give developers access to files, I hate unnecessary products (for example management web interfaces) which add complexity instead of reduce it, and I prefer not to use scp because I don’t like to add new shells to the system and chroot is usually a pain in the ass.

One protocol that usually fits perfectly these needs imho is WebDAV, it’s easy to setup, it’s robust, it’s well known, it’s network friendly (it uses the same protocol and ports used to render sites, which means also no troubles with file ownership), it can be easily used via TLS, and don’t need chroot or other complex setups.
Obviously It’s not a good idea to activate WebDAV to production or at least to virtualhosts accessible from web, but in general I think it’s a very covenient way to give people the right access they need to work.

There’s only one issue regarding this method, usually LAMP setups require that php files will be handled by php itself, that create problems even for a simple GET, which returns an empty file.

One easy way to fix this issue with Apache is to add “SetHadler none” directive to prevent php from handling requests for those files.
Obviously you can do this inside a specific virtualhost not used for rendering content, otherwise php will never be processed and pages will never work…

After reloading the webserver configuration webdav requests start working.


Wind Infostrada FTTC

Dopo tanto tempo ho deciso di scrivere uno post in italiano sperando che possa agevolarne la diffusione tra gli utenti del nostro Paese che si accingono ad aderire alle varie offerte di connettività propinate dai diversi operatori tlc.

Nel mio caso si tratta di Wind, operatore con il quale ho iniziato ad avere rapporti nel 2007 dopo una iniziale esperienza con NGI.
A onor del vero devo confessare di non aver mai avuto problemi catastrofici, la mia vecchia linea ADSL (seppur limitata a 4Mbps in downstream) si è sempre dimostrata stabile, affidabile e con una latenza decisamente superiore alla media.
A parte il feedback tecnico (variabile da zona a zona a causa dell’infrastruttura spesso in stato di abbandono) quello che mi ha favorevolmente impressionato è la flessibilità di Wind nell’applicare profili di connettività differenti da quelli proposti come standard, nel mio caso alzando la banda di upstream da 128Kbps a 512Kbps.

Poco meno di un anno fa scopro di avere copertura FTTC, viste la tariffe vantaggiose e il buon rapporto maturato decido di rimanere con questo operatore, tutto perfetto fino allo scorso agosto, quando noto un sensibile rallentamento nel downstream e un drastico aumento nella latenza.
Provo a lanciare qualche test con Speedtest.net utilizzando il server per me più affidabile (Fastweb Milano) e ottengo un ping aumentato da 5/6 a 25 ms e una banda in downstream diminuita da 60/70Mbps a 15/20Mbps.

Ripeto il test ogni giorno e ottengo lo stesso risultato, schedulo un test ogni 15 minuti utilizzando l’ottimo speedtest-cli da cui emerge un evidente problema di saturazione nelle ore serali, con qualche anticipo pomeridiano nel week end,  questo è il risultato, giudicate voi.




Quick check multipath status

Recently I had a huge activity in a customer’s datacenter, moving rack cabinets around for some works on the power supply lines. I love working on the hardware or inside datacenters, some people consider it a low profile work but I always found it very inspiring and it gives me so much satisfaction, sadly it happens rarely :( BTW, I managed to complete this tasks without shutting down anything and without any downtime thanks to power redundancy on almost any device (server, blade chassis, network or storage switch/device), a couple of 32A extension wires and a very precise action plan. Winner winner chicken dinner! One critical aspect was the storage, we had a lot of systems which extensively use SAN over FC interfaces, and some of the SAN FC switches had only one PSU, any storage path had redundancy but cutting down half of your storage devices on production systems require to be very careful and test everything. If you have a lot of servers with different environments (GNU/Linux, Windows Server and Vmware ESX) and you need to cut off and restore paths multiple times, you need to be very precise in checking paths status to avoid storage losses and potential data corruptions. Here is some quick hints to check your multipath devices on those environments, thanks to command line interface you’ll be able to check many systems with very few commands, save a lot of time and avoid a lot of headheaches.


On GNU/Linux checking multipath status is very easy, you’ll only need to run “multipath -ll” and you’ll get the status of each path for every multipath device on your server. Regarding HBA all you need to know is under /sys/class/fc_host directory where you’ll find one host* directory for each device, inside those directories you’ll find port_name and node_name with WWPN and WWN. With basic bash skills and ssh you can easily grab those information on each server, this is a trivial example.


The only requirement is the fantastic and free PsExec utility from Mark Russinovich Edit a text file with a list of all your server’s ip or hostnames, one per line (server.txt). PsExec @server.txt -e -u <USERNAME> mpclaim -s -d <DEVICE ID> If you want to see all the details (for example node number and port number) of your HBA launch Get-InitiatorPort command on a Powershell instance with superuser grants. PsExec @server.txt -e -u <USERNAME> powershell Get-InitiatorPort

Vmware ESXi

First of all you must enable ssh daemon on each Vmware host (follow this Vmware KB article), if you want to login with ssh keys follow this KB article. For checking multipath status you must run this command “esxcli storage nmp device list”, the output is quite verbose so it’s better to grab only the information we need adding a nice “| grep Working”, each line shows the paths for every datastore on the Vmware server. You can find WWN and WWPN with “esxcli storage core adapter list” As for GNU/Linux server you can easily cycle through your Vmware servers using ssh and bash to grab those information with a single script.


http request and tcpdump

If you work with http reverse proxy one of the most frequent problem is that people working on the backend systems complain about things they expect but they don’t see coming from your frontend service.
Working with Tivoli Access Manager this happen to me every time I pass some value to the backend services like iv-user, iv-remote-address or LtpaToken… every time people open the browser, press F12 and expect to find those data into the http request exchanged with the browser… NO!!! It does not work like that! :\

In these moments the only way to close the case is sniff some packets and put them in front of their nose with a giant red arrow showing the damn data they are expecting and which is perfectly exchanged between TAM and backend servers.
You can do this in many ways, the fastest and simple imho is by one of the most important tool for problem solving and analysis, the swiss knife of every sysadmin: tcpdump.

In this case the syntax of tcpdump is a bit “esoteric”, here it is:

sudo tcpdump -nn -i <interface> -A -s 0 '<protocol> port <port> and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

For example: sudo tcpdump -nn -i eth0 -A -s 0 ‘tcp port 80 and (((ip[2:2] – ((ip[0]&0xf)<<2)) – ((tcp[12]&0xf0)>>2)) != 0)’

If you want to grab the dump and open with some other software (for example the great Wireshark) you must add “-w /path/to/dump/dump.dmp”

That’s all.

« Post precedenti