31/07/2020

Dell iDrac java patch

Every now and then people ask me which is my favorite server producer, and every time I honestly don’t know how to reply because they all work pretty well.
What really changes between competitors are technical support and some of the small bits that many people consider irrelevant, but Imho they are very important, one of them, maybe the most important, it the lights-out management interface (LOM).
Every server producer has it’s own LOM interface, but my favorite (and one of the reasons why I prefer Dell servers) is the Dell Drac.

One of the most common problems with Dell Drac is the virtual console which requires Java JRE and obviously this makes people angry because… well basically because people are lazy, most of the time leave the brain turned off and don’t read errors and exceptions…

If you search online “dell drac java error” you’ll find a whole bunch of forums, thread, reddit posts, also useless Chrome extentions for make the damn virtual console work, sometimes those sources are crap, sometimes they contain small bits of the solution, which is changing because there are several versions of Drac devices and obviously they evolved during the years.
These errors always came from the java.security settings, Drac encrypt data transmissions, and old Drac cards use old encryption protocols and cypher suites, so I decided to make a simple patchfile for the java.security file for a quick change and rollback (it’s not a good idea to turn on old unsecure protocols for you JRE).

First of all you have to identify your java.security file, which is inside you JAVA_HOME/lib/security, after that apply this java.security patchfile.

After that open you java settings and add the url of your Drac web interface to the “Security > site exception” list.

That’s all, now you’ll be able to open the vitual console even on an old Drac 5 with the latest JRE (tested right now with JRE 1.8.0_261).

05/05/2020

Eve Online SOTA part 1

It’s been a while since a wrote something about Eve, and to be honest this is not a State Of The Alliance (SOTA) because… well I’m not an alliance executor and not even a corp CEO, I’m just a regular grunt, as my Twitter bio says I’m just another kender exploring New Eden.

This is not only a post about Eve, in some sense this is a milestone for me because my main character reached the symbolic objective of 100.000.000 skill points; If I sum all the skill points of all my toons I’m reaching about 400.000.000 SP, but you know… the first char is always the most beloved.

Last time I wrote something about Eve I was starting to train one of my biggest and most precious objective, the JF pilot, It was exactly 4 years ago, I reached that goal and despite all the other big objectives (dread and carrier pilots well trainer, two maxed rorqual toons, one almost maxed industrial an reprocessing toon, 6 maxed pi toons, etc etc…) that JF was one of the sweetest and maybe the most precious.

The Italian Alliance I started with died almost 6 months after I joined because of too many elite-pvp players inside, with the core group we joined CO2 back in Tribute, I was part of one of the epic siege of M-OEE8, we fought hard, we lost the first real fought Keepstar citadel and became history.
Then with the entire alliance I moved to Catch and Impass, where we suffered the biggest betrayal in the history of Eve aka “the Judgement Day”.

Thanks to some good friends (and awesome human beings) we moved to Test Alliance Please Ignore in Esoteria, one of the most remote regions of the Eve universe.
We spent almost a year in Test and I have to admit that I loved it, my time in CO2 was great, but the alliance really changed during the years, it became more obsessed on pvp and revenge, almost closed to new players, at some point CO2 became the only alliance without allies and against everyone, and honestly I didn’t like it…
Test was really different, even today I think it’s the most organized alliance I ever seen, awesome wiki, doctrines extremely well documented, very friendly for noobs, everything was great for regular grunts like me, can’t say the same thing for our CEO because Test leadeship seems quite… how can I say… tricky.

After that experience our corp moved into another historic alliance, The Initiative. I always heard about them as smaller but more pvp focused alliance, some kind of elite pvp group inside the Imperium coalition.
At first I was not sure at all and was tempted to leave my corp and stay in Test, but I decided to have faith in our former CEO (I repeat an awsome person) so I jumped into Init.
We started living between Querious (which was the first null region when I started playing Eve) and Fountain, It was tough at first because we lost our habits (we made a lot of huge industrial production in Esoteria) but then I found an equilibrium and things started to work pretty well.

After about six months into the trial alliance Initiative Mercenaries we were promoted as full members of The Initiative, and I have to admit that I was really proud of it, we were parte of one of the most skilled and active groups in the history of Eve.
With Init we made history again, we archived something that everyone in the game considered impossibile, something called the “siege of Rage” which took an entire year of work and preparation and concluded with the destruction of the first Keepstar citadel ever built in the game, we made history, again.

Living in Init is quite different from every other alliance I lived before, in Init I found great fleet commanders, awesome people always helpful and willing to do everything, but it’s a more mature alliance, you must be able to get your stuff, you must be more independent from a logistic point of view, don’t expect the alliance will run for you providing everything you need, you asked to be part of Init, not the opposite.

Now I continue to play, yesterday I lost my first capital into a huge brawl (I used them several times during the years but never lost one) that made almost 1 trillion isk in lost ships, It was awsome.
During the last year I had moments when I really never played, when I thought to leave the game, months spent mainly doing PI, putting skills in the queues and nothing else; it was not an alliance fault, nor my corp fault, it was simply the consequence of huge CCP mistakes, but that’s a different story for a new SOTA rant.

o7

03/04/2020

AWS EC2 instance migration

Recently I received some complanings about load problems on an AWS EC2 t2.medium instance with CentOS 7, despite being a development environment it was under heavy load.
I checked logs and monitoring and excluded any kind of attack, after a speech with the dev team it was clear that the load was ok for the applications running (some kind of elasticsearch scheduled bullshit).

The load was 100% from cpu but I noticed some interesting behavior since a couple of weeks with a lot of steal load.

Looking to EC2 CPU Credits it was crystal clear that we ran out of cpu credits, which turned on some heavy throttling.

Since the developers can’t reduce the load from the applications and the management won’t move from EC2, the solution I suggested was to move to a different kind of instance specifically designed for heavy computational workloads and without cpu credits.

So I made some snapshots and launched a new C5 instance, piece of cake, right?
Well no… as soon as I started the new instance it won’t boot, and returned “/dev/centos/root does not exist” on the logs. :\

So what’s going on here?
Pretty simple, there are significant hardware differences between each type of EC2 instance, for example EC2 C type instances have NVMe SSD storage which require a specific kernel module, same for the network interface with ENA module.

The goal here is to make a new init image with these two modules inside, so during the boot the kernel could use these devices, and find a usable volume for boot and nic for network; the only problem is that we can’t simply boot the system using a live distro and build a new init image with those modules already loaded, remember we’re on AWS not on a good old Vmware instance (sigh…).

First of all I terminated the new instance, it was basically useless, and got back to the starting T2 instance.
Check which kernel version you’re using with “uname -a” and build a new init image including nvme and ena modules using mkinitrd, for example:

mkinitrd -v --with=nvme --with=ena -f /boot/initramfs-3.10.0-1062.18.1.el7.x86_64-nvme-ena.img 3.10.0-1062.18.1.el7.x86_64

Using lsinitrd you can check that your new init image has nvme and ena module files inside.

Now you have to edit your grub config file (/boot/grub2/grub.cfg) and change your first menu entry switching from the old init image to the new one.

Save /boot/grub2/grub.cfg file, CHECK AGAIN YOU HAVE A GOOD SNAPSHOT OR AN AMI, and reboot, nothing should have changed.

Now you can make a new snapshot or AMI and build a new instance from it, choose a C type instance and now it should be able to boot properly.

As you can see the new C5 instance have different storage device names, it has a new nic driver (ena) and it has ena and nvme modules loaded.

Life should be easier without the cloud… again.

24/12/2019

SPID… grazie ma no grazie

Negli ultimi anni (in particolare in questo 2019 che si sta concludendo) uno dei temi su cui la PA ha più martellato è SPID, il cosidetto Sistema Pubblico di Identità Digitale promosso dal Mise insieme con Agid e il celebre team per la trasformazione digitale del governo.

In questo post vorrei esporre i motivi per cui a mio modestissimo parere non solo SPID è una colossale regressione rispetto al precedente sistema di autenticazione “strong”, ma anche perchè è PERICOLOSO per i soggetti interessati, aziende e soprattutto privati cittadini.

Cominciamo col definire cos’è appunto SPID, trattasi di un sistema di autenticazione strong, ovvero che permette di certificare che il soggetto che si registra è stato verificato da un altro soggetto considerato affidabile (che da ora in poi chiameremo identity provider o idp, in questa sede prenderemo in esame i tre principali, Aruba, TIM e Poste, gli altri si comportano in modo del tutto identico).
Già qui cominciano i distinguo, ovvero effettuare la login tramite SPID non certifica che il soggetto che si sta autenticando sia il soggetto titolare delle credenziali SPID, nulla vieta al soggetto Pippo di consegnare le credenziali SPID al soggetto Pluto, e costui non avrà alcun problema ad autenticarsi come se fosse Pippo.
Si obbietterà che questo vale per qualsiasi altro sistema di autenticazione remota, ed è assolutamente vero a meno che non siano previsti fattori biometrici, ma è sempre bene ricordarlo.

In secondo luogo chiariamo un dettaglio che può sembrare lessicale ma è SOSTANZIALE e che riprenderemo successivamente, l’acronimo SPID viene pubblicizzato come Sistema Pubblico di Identità Digitale, niente di più falso, semmai sarebbe il caso di ribattezzarlo Sistema PRIVATO di Identità Digitale in quanto di pubblico in SPID non c’è proprio nulla.
Già perchè ogni volta che vi loggate ad un servizio pubblico con SPID, in realtà effettuate una login presso un servizio PRIVATO, di una società PRIVATA, perchè tali sono gli identity providers, non ci credete? Controllate voi stessi direttamente sul sito Agid.

Già signore e signori, non esiste un idp pubblico, non è mai esistito dal lancio del servizio e non ci sono informazioni che facciano pensare al lancio di un idp pubblico (non mancherebbero di certo le risorse alla PA per istituirne almeno uno…), quindi semmai non ve ne foste resi conto le vostre credenziali SPID non stanno affatto sul sito del vostro comune, regione, provincia, INPS o Agenzia delle Entrate, ma risiedono fisicamente nelle banche dati di queste società private, e indovinate un po’… quei dati danno accesso a TUTTI I VOSTRI DATI PIU’ SENSIBILI E PIU’ PERSONALI: fascicolo sanitario con tutta la vostra storia clinica, fascicolo previdenziale, dichiarazioni dei redditi, multe, pagamenti, rette scolastiche e chi più ne ha più ne metta, tutto in mano a un unico soggetto che ha (giustamente) l’obbiettivo di fare utili, non certo di fornire un servizio pubblico.

Qualcuno obbietterà che SPID prevede una autenticazione di secondo livello, ovvero quella che si definisce 2FA (autenticazione multifattore), ovvero un ulteriore dato oltre alle classiche username e password per autenticarsi; questo è vero, è previsto ed è fornito tramite app mobile (es Aruba e Poste) o SMS (es TIM).
Pensateci un attimo però, chi fornisce i fattori di autenticazione di secondo livello (2FA), nel caso di Aruba è una app sviluppata da Aruba stessa, idem per Poste, nel caso di TIM il mittente dell’SMS è TIM stessa; ovvero il soggetto che detiene i primi due fattori di autenticazione (username e password) è lo stesso che fornisce il terzo fattore di autenticazione, non proprio una best practice in termini di sicurezza e riservatezza dei dati, non trovate?

Alquanto curiosa come soluzione considerando che esistono degli standard specifici per i dispositivi per l’autenticazione multifattore, ed esiste una discreta quantità di dispositivi software e hardware che le implementano (Google Authenticator, Microsoft Authenticator, Authy, FreeOTP, tanto per citare alcune delle più celebri); usare queste non avrebbe comportato alcun costo e avrebbe permesso di scorporare i fattori di autenticazione riducendo drasticamente la criticità rappresentata dagli idp.

Si può obbiettare che gli idp sono soggetti selezionati e che danno garanzie ben precise in termini di professionalità, onorabilità e sicurezza.
Ebbene, ricordiamoci che il pericolo può provenire dall’interno di queste organizzazioni, ma soprattutto dall’esterno, un data breach su questi dati e una manomissione dei meccanismi di autenticazione multifattore avrebbero conseguenze CATASTROFICHE, sia quantitativamente(si stanno concentrando questi dati ipercritici in pochissimi soggetti) e qualitativamente (come dicevo poco sopra, accesso ai dati più sensibili e personali che esistano).

Quali garanzie forniscono Agid e idp? Un pezzo di carta, e mi si lasci aggiungere, che vale meno della carta su cui è scritto.
Già perchè di fatto l’unica garanzia fornita all’utente è puramente formale, una convenzione stipulata tra idp e Agid (qui un esempio) che non varrà assolutamente nulla in termini pratici una volta che il danno sarà stato fatto (le informazioni trafugate non torneranno magicamente indietro con qualche risarcimento).

Passiamo ora ad un aspetto sicuramente meno importante, SPID in fin dei conti è gratis, a caval donato non si guarda in bocca, giusto?
Beh sì… ni… no.
La creazione di un account SPID viene pubblicizzata come gratuita, in realtà è in promozione gratuita a certe condizioni, ovvero solo per privati e solo se ci si identifica con un altro meccanismo di autenticazione strong (es CRS/CNS), in caso contrario in alcuni casi occorre recarsi fisicamente presso uno sportello dell’idp (es Poste o TIM) oppure identificarsi tramite webcam ad un costo che varia mediamente tra le 14 e le 20 €,
Sia chiaro che se si vogliono attivare ulteriori meccanismi di autenticazione (es livello 2 con token fisico o livello 3) occorre sborsare altra pecunia.

Qualcuno obbietterà che quella spesa (soldi o tempo perso allo sportello) è ben poca cosa rispetto ai vantaggi di SPID.
E se vi dicessi che in realtà tutto questo esisteva già da tempo (2007, forse prima) grazie alla Carta Regionale dei Servizi e Carta Nazionale dei Servizi?
E se vi dicessi che CRS/CNS:

  • sono erogate esclusivamente da soggetti pubblici
  • prevedono meccanismi di autenticazione materialmente diversi (certificato nella smartcard fisica e PIN)
  • prevedono il possesso da parte dell’utente di entrambi i meccanismi di autenticazione (il PIN può essere modificato a piacimento da parte dell’utente)
  • sono gratuite

L’unica criticità storica di CRS/CNS è legata al dispositivo materiale (smartcard) che richiede un lettore dedicato e del relativo software, criticità vere con le quali mi sono scontrato io stesso, ma facilmente risolvibili cambiando il formato del dispositivo (da smartcard a pendrive usb? Magari micro o type-C e comunque compatibile con smartphone?) o con un miglior supporto software, tutte cose che avrebbero un costo irrisorio rispetto alla montagna di soldi spesi con SPID per rifare (MALE) qualcosa che già esisteva e funzionava (BENE).

Last but not least faccio presente che tutti i principali servizi pubblici, zitti zitti stanno gradualmente bandendo ogni meccanismo di autenticazione strong differente da SPID, resiste ancora qualche baluardo con CNS/CRS ma le amministrazioni stanno facendo di tutto per nasconderlo agli utenti e prevedono di eliminarlo a breve (es Comune di Roma): RESISTANCE IS FUTILE!

Lasciatemi concludere questa piccola rassegna degli orrori di SPID con un simpatico scambio che ho avuto su Twitter con Team Digitale, alla mia richiesta di chiarimento sulle criticità relative a sicurezza, riservatezza dei dati, fattori di autenticazione etc etc mi è stato risposto che: SPID è sicuro perchè gli identity providers verificano tramite webcam la registrazione degli utenti, non vi sentite più sicuri ora? :\

05/08/2019

On containers and orchestration

I don’t know what do you think about containers and orchestration tools, but for me it’s been a while since I started discussing on those topics on forum and various platforms, and I can’t say It’s an easy discussion.
To be honest I’m quite tired of repeating the same concepts so I thought this blog could be help me to express my point of view, at least in future I only have to copy and paste this post url and not waste time anymore :)

For someone my point of view on containers and microservices could sound a bit grumpy, but I can assure that I’m not against them, or think they are bad or simply some nasty trend.

First of all It’s imperative to distinguish between two main scenarios: development/test and production environment.

For the first one containers are wonderful, they are the Nirvana and Shangri-La all together, they give developers the opportunity to setup dev environment in no time, no setups, no hard requirements, everything is perfectly consistent and works in the same way as the tools they are familiar with (think about git or other versioning softwares), everything is perfectly reproducible on whatever platform, their work pc, home pc, a server, a laptor, everything.

For production… well It’s not the same.
First of all: the most important requirement for a production environment is reliability, then comes security, then everything else.
Literature, research, experience and logic taught us that these two basic requirements could be reached only following the KISS principle (Keep It Simple Stupid) which is not a joke, It’s real, It’s true and It works, period.
Second, when your application moves from development to production It moves from developers to sysadmins, It’s the sysadmin the guy which must maintain It, that must provide resources and guarantee that the application (and all its requirements) are working, are reliable and kept secure.
Third: if you think about the lifecycle of an application most of It is in production, an application could require a few months of development but will remain online for years, and usually the more It will require for the development the more It will remain in production.

Following the KISS principle means REDUCE COMPLEXITY, containers ADD COMPLEXITY :\
This may sound strange to containers users because “well it’s a piace of cake, launch a couple of commands and you’re ready to go”, well stop for a moment and think about it.

  • On a “legacy” environment you have, applications (your site, your database, etc etc) on top of services (webservers, application servers, rdbms, etc etc shared by several applications) on top of an OS (shared by several services) on top of some sort of hardware (which could be very complex in production, and could be shared between several OSs if you’re using virtual machines).
  • With containers and orchestrators (like Kubernetes or Openshift) you have a LOT more complexity, you still have your applications, on top of services (which are the same as before for instance…), on top of containers, on top of a container environment (for example Docker), on top of orchestrators, on top of a OS (usually installed on a vm, so add also the hypervisor complexity) on top of hardware.

More complexity means less reliability and less security, or a least a lot more work and variables to manage to reach the same level of those requirements.

Like everything there are pros and cons, and someone could argue that beside that added complexity containers and orchestrators give you a lot of benefits, mainly:

  1. reproducibility
  2. horizontal scalability (adding more “nodes” and distribute load across them).

We already talked about the first before, It’s awsome for a developer, but for a sysadmin on a production environment?
Well, not really, simply we don’t need it because move an application between different environments is really rare (in 20+ years of IT consultant work It happened to me only one time, moving and Oracle 10g rdbms from Windows Server 2003 to RHEL) and usually any service have its own backup and restore procedures to accomplish this goal.

Scalability is another story, in theory it’s an awsome thing, in real world it’s not a big deal in 99% of companies or services.
The idea of having some black magic that will add more and more instances of your application and distribute the load on those is good and managers love it, they simply think this is the solution for every problem because, let’s be honest, they don’t understand the complexity behind an application and they think that all the problems come from lack of resources.

In the real world any experienced sysadmin can confirm that usually resources are enough and when an application have problems it’s all about some bug, exception, unmanaged situations (for example the application use a third party service which doesn’t work and the application does not manage it), all those things can lead to a slow or unresponsive application even if you have a lot of resources available.
Adding more and more application instances and balance load across them will lead to a simple result: more and more exceptions.

Ok, let’s ride our fantasy horse and think about a bug free application, can we scale up with containers with no worries?
Sure you can, but do you really need it?
For 99% of companies nowadays it’s really rare that an application don’t have enough resources, or have such a huge amount of requests to run out of resources.
If you are Google, Facebook, Netflix, Amazon or any other global huge company maybe you really need horizontal scalability, so orchestrators and containers are very useful (it’s not surprising that one of the most popular orchestrators, Kubernetes, came from Google), otherwise… well no, you don’t need it.

So that’s all?
No, there are a lot of smaller pros and cons (most cons to be honest imho…) on this topic, like security, access on third party resources, logging, service redundancy vs consolidation and many more.
Most of these are huge problems with containers (and most of the times they are ignored) while with a traditional architecture they simply aren’t, even a simple stdout append on a log is a pain in the ass with containers, and require to add a lot of complexity to reach this simple goal (remember KISS principle).

Let me add a small personal thought on this subject that can be extended in many other areas of the IT universe.
As you probably understood I think that containers are a really good tool for developers and not a good one for sysadmins, containers are born from developers for developers.
Why people always think that a development tool should work for production?
Why people don’t think from a production perspective when they develope tools for production?

If you are a developer and you’re working on a tool that will be used in production, please ask your sysadmin what does he think about it, what’s its requirements, what are the problems that he usually have in production and he need to fix.
Maybe if we start to develope tools from the right perspective we’ll have better results, otherwise we will continue to have always the same problems.

 

« Post precedenti