Mugurel Tudor

Category: Technology

  • WSL2 for DevOps / SysOps

    Intro

    Back in the day, it used to be both a very simple and a very complicated decision to pick the OS for your working station. You were a linux administrator? Then you would use linux and suffer the pain of reading your emails from Exchange. The calendar wouldn’t work but nobody would summon you in a meeting so that was OK. Years would pass and then suddenly meetings are far more important, you really need to make use of that damn Exchange calendar and Joy to the world! you discover MacBooks. Now you can use Outlook for Mac, have a working calendar AND you still have your awesome command line interface. If you did your scripts in a smart way, you don’t even need to point them to your new $HOME.

    And while you happily hack away with ansible, terraform, vagrant, python and other tools that you “brew” yourself in the morning over a coffee, you read about “that last thing Microsoft did”: WSL2, Windows Subsystem for Linux. OK, you didn’t tried version 1 but now everybody mentions how great this one works so… you start wondering. Do you FINALLY have a choice for the OS of your working station?

    Sure, by now you now no longer call yourself a linux administrator. Not even a SysOp, maybe. Could be that you are a DevOp. Or a Reliability Engineer. Or any other random name they use now for basically the same thing. But, the question is: does WSL2 fit as a solution for you as a… SysOp? DevOp? SRE? Linux Administrator?

    Well, as they say on Facebook… it’s complicated. Keep reading to find out if it fits for you.

    So what is WSL?

    In a nutshell, Windows Subsystem for Linux is…. Linux in your Windows. It sounds dumb as an explanation, but that’s what it is. You install your favorite Linux distribution (or several linuxes) in your Windows 10, via Microsoft Store. And then you click it on them, and they run. If you now remember how you installed your first linux (from scratch), and how many days it took, and you cry inside, that’s ok: it’s called progress.

    But leaving that aside: while WSL1 was “not quite a linux”, as it didn’t had a full linux kernel / full system call compatibility, this has been fixed in WSL2. Now it runs via HyperV and it really is a full, functional, performant and integrated with Windows… Linux.

    Resource usage

    I admit that, before anything, I was mostly interested in the resource usage. How much RAM does it take? CPU? How fast does it start? Do you hear the CPU fan when you use it? I mean, if you used VMWare at some point, those are legitimate questions.

    And I admit I was… bamboozled. You click on it, and it starts. In a second. RAM usage? As a proud owner of a 32GB of RAM, non-existent. CPU usage? Non existent. Disk usage? Well that takes a bit, but not that significant: for an Ubuntu 20.04, between maybe 1.5GB and 2GB, depending on what you installed on it.

    If you have even a vaguely modern computer, you can discard the resource usage, as it’s a non issue.

    OS integration

    I can only say… wow! This is all integrated in such a nice way:

    • you can run arbitrary linux commands from windows cmd if you prefix them with wsl. So you want to use a random thing like… ipcalc? You start your lame windows cmd and you run “wsl ipcalc <whatever>”. It will run. If you want to control on which linux to run that, it’s via “–distribution <your distro name>”. You can also set a default distribution.
    • in terms of filesystem integration, from within linux, you have /mnt/c/whatever to access windows files, or /mnt/<random drive>/<random file>

    And best of all: I don’t know what code editor you use, but I use Microsoft Visual Studio Code. And boy, does this one integrates the shit out of WSL. You need to install the WSL extension, and then from the folder inside the WSL where you have your code checked out you do a “code .” and… tadaaa! Visual Studio Code will open with that folder imported. Now everything you edit and run inside this project is inside your WSL but controlled from your Windows.

    Man, at this point I was completely sold. Shut up and take my money! Everything will work! This is great!

    * voice from the studio: “But it wasn’t”.

    AWS

    Not much to cover here. You use AWS? You have a choice of two AWS CLI clients: 1 and.. surprise! 2. Version 1 most probably you can install via linux with an apt-get or a yum. But the second version is with their lame install.

    The biggest surprise is if you use AWS SSO and therefore awscli v2. Man… The surprise of running inside Linux “aws sso login” and having the browser open in Windows 10 for a confirmation… Priceless! Did I mention I was sold already at at the OS resource usage and OS integration? By now, I’m already triple sold!

    Terraform

    That’s an easy take: it works like a charm. Do yourself a favor and install it via the linux repository made available by Hashicorp recently.

    If you’re thinking “brew for linux” + “hashicorp tap” think again: Hashicorp offers that tap for MacOS only, so stir away from that.

    Or, if you are one of those guys having a million different repositories, all requiring a million different terraform versions, install it via tfenv, that one worked OK as well.

    Now, for the actual usage, I haven’t noticed anything out of the ordinary in those few days I’ve used it. Everything works as expected.

    Packer

    You build images with external “builders” like vsphere-iso or amazon-ebs? Everything should work as expected. You build it with some local builder, like virtualbox? Congratulations, you find the first problem. From here, it starts to be downhill.

    WSL2 uses HyperV, and because of this no other hypervisor can run on the same machine. VirtualBox has started recently to add HyperV support in its software, but for the moment is buggy and works somehow randomly.

    So here the verdict is: it depends.

    Ansible

    And it doesn’t get any better from here: ansible in itself works decently (with some caveats), but if you develop Ansible code, you would normally use Vagrant to test your code. Vagrant works best with VirtualBox and you’re back on the previous problem: I wasn’t able to to have Vagrant working in this setup.

    I tried also to use Vagrant with HyperV, but a lot of the things that you expect to “automagically work” are not working anymore. Could be that this is fixed in the future, but it’s not for now. See also this and this for more.

    For me, this is the point where I stopped my evaluation. Is it possible to still use Ansible? Of course. Are you able to work efficiently? Not quite.

    Later edit: it turns out I didn’t considered remote Vagrant providers. It’s possible to use Vagrant with Vsphere or AWS providers (see this plugin and this plugin), so if you use any of these, you can still make it fly. While they are a bit restricting in terms of images you can select, you can still do your job with them. Vagrant seems to support even more platforms through plugins (DigitalOcean and Azure comes to mind) but I didn’t test those yet.

    Conclusions

    While it’s a huge step in the right direction, it’s still a new technology and unfortunately some things are not yet working as expected. If you don’t use Ansible / Vagrant combination, this might already work out great for you. If you use Ansible, you can still use this but with some extra work and restrictions, and a Linux or a MacOS workstation are still a better choice for this type of work.

    Did I switched to Windows? No. Do I intend to? Maybe, if these problems are fixed at one point. At least I’ll consider it. Did you switched? Then let me know in the comments how did it work for you.

  • Streaming MySQL database backups

    Let’s say you want to backup a MySQL database but instead of going with the standard “dump the backup on the same server and send it to the backup server afterwards”, you’d like to save yourself an extra step and directly stream it remotely to the backup server. Or you’re in one of those funny scenarios where “yo database is so big” that you don’t have enough space for a full dump on the same server.

    Enter innobackupex and xbstream, from Percona XtraBackup.

    If you didn’t do that already, install percona-xtrabackup on both your target and backup servers. On Ubuntu, that would be an easy “apt-get install percona-xtrabackup”.

    Now, if you’re an one-liner type of guy, you can just go like this, from your backup server:

    ssh BACKUP_USER@MYSQL_SRV "sudo innobackupex --defaults-file=DEFAULTS_FILE \
  --stream=xbstream TEMP_DIR" | xbstream -x -C LOCAL_BACKUP_DIR

    So what happens here? In a nutshell, you run innobackupex from your backup server, remotely via ssh on the database server, which streams the backup to STDOUT, as an xbstream stream, and from there back to the backup server, where it gets decoded back to a mysql backup via xbstream utility. MySQL credentials are handled in this example via defaults-file, but you can have many approaches here.

    Normally, if you wouldn’t specify TEMP_DIR, innobackupex would make use of your default temp folder (most probably /tmp), so if your tables are small enough / your tmp folder is big enough, you can omit this parameter. Otherwise, make sure you specify a big enough TEMP_DIR.

    Please note that the above one-liner is very basic and usually you would want to do at least two other things:

    • compress the backup with –compress
    • encrypt the backup with –encrypt=AES256 –encrypt-key-file=YOUR_ENCRYPTION_KEY

    In both cases, you can specify the number of threads via –compress-threads / –encrypt-threads / –parallel.

    As always, make sure you read the fine manual before use.

  • Fixing permissions with find “the quick way”

    Probably I’m not the only one having to fix some permissions on an entire tree and being annoyed by the fact that, sometimes, you have to do it twice: one time to fix the file permissions and second time to fix folder permissions.

    Let’s say you want to grant “read” to “all”. Running “chmod -R a+r” doesn’t help with the folders, which require an extra “+x” so you could enter them. But setting “chmod -R a+rx” make also the files executable, and you don’t want that.

    So you may end up with idiotic things like this:

    chmod -R a+r /path
find /path -type d -exec chmod 755 {} \;

    This is not only stupid, but also slow. Fortunately, there’s also a quick way to do it:

    chmod -R a+rX /path

    where X means “set execute only if the file is a directory or already has execute permission for some user”. And most of the time, that’s exactly what you need. Shorter to type and faster to run.

  • Hello, world!

    So I decided to have a tech blog. Why oh why, you may ask, aren’t there already millions of tech blogs on the mighty internet? Yes, there are. But sometimes I feel the need to share this awesome thing I just found. Sometimes I just have to take a note of it, so I won’t forget. After all, I might need it again in a couple of years. And sometimes… well… I’m just bored.

    Let’s see how it goes.