Andrew Hutchings (aka LinuxJedi) has worked on many of the Open Source software projects that make up the Internet. He now works from home in the UK for MariaDB developing their MariaDB ColumnStore engine.
For the last few days I had my laptop connected to an external monitor which has a 2560×1600 resolution. Whilst having a few websites open and doing a video chat the laptop completely ground to a halt with the fan going full speed.
My laptop is a Lenovo ThinkPad X260 with an i7 CPU and 16GB RAM, so not really a lightweight which was why this was so unexpected. After trying some random things I have solved this. So I thought I would detail it here if only so I remember in the future.
After a little more investigating I found the root cause of my problem. I have a Firefox profile I have been copying around for a while and in it layers.acceleration.force-enabled and gfx.canvas.azure.accelerated both enabled. It turns out that these appear to eat massive CPU resources and can cause performance issues with Glamor. I turned them off, switched back to Glamor rendering and we are back to normal performance again.
This is the more ideal scenario because as we move towards Wayland being the default display server we will also be moving away from Xorg graphics drivers.
Recent versions of Xorg have switched the Glamor as a graphics driver. This basically uses 3D acceleration to draw your 2D desktop. From what I can tell for most uses this is very performant. With just my laptop screen I have no complaints. When using high resolution external monitors however it appears to be really struggling.
When I switched to the Intel native Xorg driver I could suddenly use the external monitor effortlessly. CPU usage was way down and it has even appeared to increase battery life when disconnected from the external screen. I’ve done a bit of searching as to why this would be but haven’t come up with any hard evidence so far.
To do this switch in my Kubuntu 18.10 installation I created a file called /etc/X11/xorg.conf.d/20.intel.conf with the following contents:
That did get me thinking as to whether or not Kubuntu with the backports PPA is as good and as stable to use as KDE Neon.
I therefore did what I always do when I want to try distros out, installed it on a VM first. Once installed along with the PPA I was using the latest stable KDE Plasma with Kubuntu 18.10 as the base OS. This gives me the kind of bleeding edge I’m comfortable with and also the KDE Plasma features that I have been excited about using. I’ve never had a good experience with rolling distributions (sudden unbootable devices after a system update is not fun). So this gives me the latest stable of everything I want without being totally rolling.
The desktop theme is a little different to KDE Neon’s. The most noticeable difference is the main panel is using a dark theme instead of the default lighter theme. This is fine for me as I’d actually switched the panel to the dark theme in Neon. It also adds the “show desktop” widget to the right of the panel. Something that I had also instinctively added straight away in KDE Neon.
After some more playing with it I actually figured it was good for prime time. So… long story short Kubuntu with the backports PPA is now the main distro on my laptop.
I’ve been using it for several days and it is great. The kernel changes between the Ubuntu 18.04 and 18.10 base have meant the battery life is actually a lot better and on-par with the Fedora 29 GNOME setup I was using previously.
This solves my concern with KDE Neon, that it would get stale very quickly whilst pinned to an LTS base. Don’t get me wrong, I think LTS bases are great for enterprise and servers. But on desktops you end up having to wait for so nifty features such as power saving enhancements and better drivers. For me I also like testing in the more up-to-date compilers as every compiler release adds more useful warnings.
I’ve even started using Kubuntu on a new toy that came from China recently, more about that in an upcoming post.
To Max’s original question, for many desktop users I think Kubuntu may be a better choice than KDE Neon. But KDE Neon is a fantastic showcase of KDE Plasma and applications the way the developers had designed them to look and feel. Certainly a good introduction to the KDE world.
Thank you Max for the question!
Featured Image credit: Energy Arc by Blaise Frazier
At the beginning of the year I gave KDE Plasma a try as the primary desktop on one of my devices. It wasn’t my primary laptop but I still used it heavily in that time. I enjoyed it but there were still some niggles that meant I wouldn’t have been happy with it being my primary desktop at the time.
A week ago I happened to come across the announcement for KDE Plasma 5.14. The thing that really caught my eye there was the “Display Configuration widget”. This led me to giving KDE & Plasma another chance, another week of testing. Again this is on my primary laptop, a highly-sepecced Lenovo ThinkPad X260.
At the time of writing Fedora did not have a KDE spin that included Plasma 5.14 beyond Rawhide and I didn’t want to try Rawhide on my laptop. I’ve been hitting a lot of paper-cut style bugs with Fedora 29 anyway including not being able to update my laptop at all due to TeX Live package issues. I therefore decided to try KDE Neon, I figured this distro would show Plasma as intended by the people behind KDE. For anyone not familiar with KDE Neon, it is based on Ubuntu’s current LTS release (18.04) and has the latest KDE packages on top. There are a few options depending on how bleeding edge you want to go, I went for the “stable” option.
There were two interesting parts to the new display configuration widget that really drew me in. The first is the fact you have five easy to select buttons for instant monitor configuration. This is extremely handy for me at conferences and meetings where I need to quickly setup to give a presentation. The second is a rather innocuous switch that was actually my main draw “Enable Presentation Mode”. This mode works a lot like the popular Caffeine extension in GNOME Shell. It inhibits the screen blanking / locking for when you are giving presentations. I actually also use this during conference calls when I’m using my laptop as a second screen to my desktop that is making the call and don’t want it to go to sleep in the couple of minutes that I’m not reading.
Everything is Better!
In my previous post I broke things up into Good/Bad/Ugly. I can’t do that this time because every bug I hit, every crash I had… It is all fixed. I’m someone that almost always installs software from the command line but I even found “Discover” a joy to use!
KMail works pretty well, I’m using it as my primary mail client now. Whilst configuration can be a bit fiddly with it once it is up and running it is a pleasure to use. I’ve been having issues with performance with Thunderbird 60 onwards, no matter what machine I used, scrolling thousands of emails is painfully slow. I was looking to switch mail clients anyway so this is a refreshing change.
Things I Discovered
The screenshot tool is called “Spectacle” and it is amazing. You have much more control over what you are taking a screenshot of and you can even use a magnifier for area selections. It will even connect to a few screen recording applications to do video capture.
Dolphin, the file manager, has always been a very powerful tool. But I found out there are integrations such as Dropbox and Git available. I’ve only used these plugins briefly but they appear to work very well.
Idle power usage is insanely low. When the screen is off (and the system is not yet in suspend) it is as if the battery isn’t being used at all. I’ve left the laptop overnight open with the screen off (I disable auto-suspend) and the battery drain has been tiny.
That being said the battery drain appears to be a little higher than GNOME Shell when using a web browser. It appears a process called “kwin_x11” uses a lot of CPU time. I think with heavy usage I’ll only get 10-12 hours instead of 16-18 that I would get with GNOME Shell at the moment. But I’m OK with that. When using other applications the battery usage seems to be on-par with GNOME Shell.
My Workflow Changing
One of the big differences for me between GNOME Shell and KDE Plasma is task switching. As I said in my previous post I often have a lot of terminals open at any time and I like the fact that “Alt-Tab” in GNOME Shell brings them all to the front in one go. Plasma doesn’t have something like this, but it is OK.
Konsole’s multi tab and split screen is awesome. I found myself gravitating towards that and using Firefox containers instead of Chrome’s multi-window user profiles for personal/work Google account separation.
I have also created an “Activity” for development work which is separate from the one that has email, web, etc… on it. So I can work for a while pretty much distraction free.
This means I have less windows open at any time to do the same things I was doing before. I feel comfortable with this workflow.
Things I Would Like To See
KCalc is really useful when I’m doing base conversion work. But I would like to see a history there like other calculators do, I would also love if “Numeric System Mode” showed a preview of the result in other bases like GNOME Calculator’s “Programming Mode”. I think basically I just want a KDE version of GNOME Calculator.
I still miss having automatic timezone switching that is integrated into GNOME Shell. I have a trip in a few weeks where I will be spending a few hours in four different timezones. It would be nice for Plasma to see use the libraries that can see that and adjust accordingly. There are workarounds using scripting but I’ll adjust this manually for now.
I’m also missing slightly newer versions of software such as GIMP that can be found in Fedora. For now it isn’t too much of an issue, but waiting for 2 years for such things to be updated (the next LTS) may become an issue for me. If it were possible to have Neon against the latest Ubuntu releases I think I would be very happy. But this would put huge demands on the KDE team so LTS is a good compromise.
I have gone from not liking the workflow changes in my previous post to easily adjusting my workflow accordingly now. In hindsight I think the many paper-cut style bugs were dampening the experience for me. But now? Things are great. KDE Neon is the primary driver on my laptop and at least for now it is there to stay.
I may in the future also change my primary desktop computer’s OS to Neon, but this is a much larger task so I want to continue on my laptop for at least another few weeks before I try it.
When I moved into my current house one of the bigger technical challenges has been to get WiFi that works throughout my house reliably. Since we can only get regular phone lines here I’m on a VDSL2 internet connection which runs at 80/20mbit. The problem is this comes in via my office which is in the corner of the house. My kids bedrooms are in the opposite corner of the house so getting WiFi to reach that far becomes a technical challenge.
The first solution I tried a couple of years ago was the Linksys WRT1900ACS running in my office. The range on this device is pretty good and I moved it to a place where it pretty much reached the whole house. The problem with it was that it was unstable, often dropping connections or freezing up completely. After a bit of digging I found other users having similar issues with no real resolution to the problems. So I decided to move on.
The next solution was to use a spare machine I had as a pfSense and OPNsense router. I’d switch between the two regularly depending on which I was having the most trouble with at the time. I then used an Ubiquity UniFi AC Pro wireless access point which I mounted on a wall roughly in the centre in the downstairs of the house. The good thing about UniFi gear is it is Power-over-Ethernet (PoE) so you just need to run one cable which I could run under the office door and along the top of the wall in the hallway to make it less messy.
Unfortunately the router software had issues that were difficult to solve (timed profiles in particular) and there was still one room in the house the UniFi couldn’t reach. My next attempt was Google WiFi. Google WiFi uses a mesh system so that only one of the devices needs to be physically wired to your network, the rest connects to this one or others in a chain using antennas. I thought this would be a great solutions as I didn’t want to run hundreds of meters of Ethernet along walls.
I bought three Google WiFi hockey puck style devices, put one in the hallway downstairs, one in the hallway upstairs (both roughly centre of the house) and the one acting as the router in my office. This gave fantastic coverage and the UI is really easy to use, but there is a big problem. All the Apple devices in the house would regularly forget their IP address and couldn’t use the network, every time this happened (and this can be every few minutes) it could only be solved by turning the WiFi on the device off and on again. I found a forum post where a lot of people were having the same issue.
I gave it a bit of time hoping that Google would come up with a fix. I also plugged the UniFi back in, gave it a new network name and got all Apple devices to connect to that. But this was not idea and had the coverage issues again.
After some digging I found that Google WiFi uses 802.11r roaming to communicate with clients about which access point to use. This cannot be turned off. It turns out Apple devices have a very poor 802.11r implementation (this took some digging to find out) and can sometimes try to get an IP address too early. Other mesh WiFi systems were having the same problem.
I had Google WiFi for 9 months at this point and it wasn’t getting any better so I hatched a plan in phases:
Get a UniFi Cloud Key and get the UniFi AC Pro to use this, before this I was just using the phone app to provision it
Ceiling mount the AC Pro in my office (this covered the whole downstairs much better than wall mounting in the hallway)
Add another AC Pro upstairs in the hallway wired in using a TP-Link AV2000 Ethernet over Power line kit
When my house is asleep, swap the WiFi networks so my primary one is running from the UniFi kit
Since I already had the TP-Link and one AC Pro the total cost of the new parts was about as much as the Google WiFi had cost in the first place, so I recouped most of the cost putting the Google WiFi up on eBay.
Very basic block diagram of the new setup
For the first time in the 2 years since I moved into this house I finally have fast, reliable WiFi working throughout the whole house.
Gource is a tool which can take a source code tree and create beautiful visualisations out of it. I’ve used it a few times before for various projects. This weekend I spent a little bit of time playing with it and applying it to MairaDB Server to see what it would produce.
In this visualisation you can see every file in the source code as a coloured dot. The dots are clustered in directories which are linked together in the directory tree using the lines. Git users swam around the files and spray code into them. It gives you a real sense of just how much work goes into a project.
To create this I got the MariaDB tree from GitHub and switched to branch 10.3. I observed the first MariaDB 5.1 tag was on the date 26th October 2009 so used that as a starting point. I could have gone right back to the beginning of MySQL here but the video would have been a lot longer!
I used my Lenovo ThinkStation S30 to generate the video, this is a 6-core Ivy-Bridge Xeon with 64GB RAM and a GeForce 1050Ti graphics card. The reason I used this over my more powerful machines is the GeForce can be used both for the OpenGL requirements of Gource and to hardware encode the video using FFmpeg.
Gource and FFmpeg are in most Linux distribution repositories. I was using Fedora for this with the RPM Fusion repositories to give me the proprietary NVidia drivers and CUDA based H.264 encoding support.
Breaking this down we are telling Gource to generate at 1080p starting at 2009-10-26, every day in git should take 0.1 seconds so that this video doesn’t go on for hours. The ‘-a’ flag auto skips if nothing happens for a whole second. We are hiding filenames and usernames simply because there are so many that they would cover up the visualisation. The final part of the Gource command tells it to stream our a PPM video to the pipe at 60 FPS.
For the FFmpeg command we tell it to receive a 60 FPS PPM video feed, add a PNG overlay of the MariaDB logo to the bottom right and pipe this though the NVidia H.264 encoder to generate our output mp4 file. The rest of the settings are to set the quality quite high so that the image is still relatively crisp at the end (YouTube uploading/re-encoding will have reduced the quality a little).
Whilst encoding you get to see the visualisation on the screen, make sure you don’t move your mouse over it as this will bring up context information which will also appear on the video. With FFmpeg using the GeForce to encode the video this is encoded in real time. Before this I tried piping the data over SSH to my larger 16 core HP Z620 workstation to encode but my 1GBit network was only fast enough to do 18 FPS.
I later added a Creative Commons licensed music track on YouTube just to add a bit of ambience.
You may notice the date jumping backwards a little bit due to things such as branches being merged. I haven’t yet seen a way to flatten this out.
Everything here is Open Source and east to tweak for your own software project. I’d be interested to see what others can do with these tool.
My go-to Linux distribution when I’m setting up my workstations is Fedora. I have been using it since Fedora Core 1 and RedHat 9 before that. I can’t even remember what RedHat version I started with now. I often try other distributions out and sometimes use them for a few months but I always end up jumping back to Fedora.
Fedora defaults to the Gnome desktop environment so I have used Gnome as my primary desktop environment for many years. Again I’ve tried others such as Cinnamon and XFCE but I always end up back in Gnome. In recent months I’ve heard a lot of good things about KDE Plasma so I thought I would give it a shot for a week. This is a report of my findings.
I’ve tried KDE Plasma many times in the past but every time I’ve hit blockers very quickly which has stopped it becoming my primary desktop environment. Typically really bad crashes that would lose work. With KDE Plasma 5.11 I figured these kind of bugs should have been ironed out and there were a lot of positive reviews, so it was time to experiment.
I first installed the Fedora KDE spin on my test laptop which is a Lenovo ThinkPad x220 and spent an evening getting familiar with it. I was quite happy with it. Fedora’s dnfdragora was a bit annoying and didn’t seem to work correctly but I could hide that away. I decided to install it on my primary laptop, a Lenovo ThinkPad x260 for a week to see how it would fare with my day to day tasks.
This was actually going to be a tough week for testing. I had to go to Finland for some meetings and give a presentation. I was a little concerned about jumping in the deep end like this, especially as it would be difficult to roll back whilst I was away but it would be a really good testing outside of my comfort zone.
Some of my findings may be due to the fact I was using the Fedora spin of KDE Plasma rather than another distribution. But I think it is worth putting them down anyway. In my tests I tried to use KDE applications as much as possible. I know I could have used Gnome/GTK applications instead but I wanted to test as much of the native stack as possible.
For the most part it worked and it worked really well. KDE Plasma runs much smoother than I remember, it is also more keyboard friendly than I remember (I use a lot of keyboard shortcuts in Gnome). I was able to mostly customise it the way I wanted to and KDE’s native applications seem to work pretty well together.
Visually it is amazing. It has come a long way since the KDE 4 days and apart from a few icon choices I think everything is well polished. I’ve even used KDE’s cursor theme (Breeze) on my Gnome desktops for a while.
The battery icon gave a much finer-grained detail of how much power I had left (I was going whole-days on battery). Gnome typically only has 3 bars in its battery icon, although you can make it show percentages. I’ve always liked Konsole and it still seems to be pretty good. Many of the KDE apps just worked as good replacements for their Gnome counterparts.
Dolphin is a really good file manager. It if no secret that not many people enjoy using Gnome’s Nautilus in its current state. Although I don’t have much of a problem with Nautilus I really like Dolphin.
I used Okular to give the presentation from a PDF export of the slides I had. I liked the fact it had presentation options. It has options for transitions but these appear to be a bit slow and clunky to be usable, you also cannot control the transition time so I just turned them off. In addition it did crash once during transitions whilst I was testing. When it came to give the presentation it worked although it dimmed my screen and the mirrored projector image when idle. In gnome if I was presenting this kind of thing would be inhibited automatically.
In Gnome I love SimpleScan it is great for scanning multi page documents, cropping and saving them as a PDF or just one-off JPEGs of a receipt for work. KDE appears to have a tool called Skanlite. I didn’t get a chance to test this fully but I couldn’t see a way to handle multi page documents well in it. Maybe it works. There are a lot of good features in it though.
I was doing a lot of hex/decimal conversion work in the last week and KCalc helped with that, although I just didn’t find the UI quite as intuitive as Gnome’s. If I want to copy part of a number to clipboard I am out of luck.
I have a Samsung curved 29″ ultra wide monitor on my main desk in my home office. Fedora 27 in general hates this monitor, especially Wayland but Xorg isn’t happy either. When waking from DPMS it appears to disconnect and then reconnect which causes GDM to crash, especially when I connect it to the NVidia 1050Ti card on my main desktop. In KDE Plasma it didn’t crash but triggered several screen re-configurations before I could actually log in using it. I was impressed that it appeared to dynamically add/remove monitors, KDE Plasma has never been able to do that in my previous tests. But the colours were off, it looks like it wasn’t using the correct colour profile for the monitor. Blues were purple and it just wasn’t a good experience.
KDE Plasma has an interesting concept called “Activities”. They are pretty much like virtual desktops but appear to have a little more isolation. I tried using one for all the terminal sessions I had open for vim and ssh sessions and one for web and email. This didn’t work out too well as the background would glitch badly after transitioning until I dragged a window all over the background. It was very distracting. Luckily KDE Plasma supports virtual desktops too so I enabled those and all was well.
The battery on the x260 was definitely drained quicker in KDE than Gnome, I got a few hours less out of it. This could be subjective but my laptop feels like it is running hotter most of the time. I don’t have any scientific readings for this though. I hear this is going to improve with the next release.
Crashes! Still so many crashes! Now, Plasma desktop didn’t crash for me, this is the first time I tried KDE Plasma and not have the whole desktop crash. But some of the bundled applications were very unstable. The “System Settings” application would crash about 50% of the time I did a search inside it. The software centre, also called “Discover”, crashed a lot and reported applications in a quantum state of both installed and not installed at the same time.
I tried to configure KMail with one of my mail accounts. Once I got this configured no folders would show up. I closed it and reopened it, the mail was there but when I clicked on a folder the list only showed the date field, not sender or subject. Apparently those fields were enabled but I couldn’t find a way of changing the size of them to make them visible. So I gave up on that and went to my trusted Thunderbird.
I travel quite a bit for work. I have been to Finland 3 times in the last 4 months. I travel via Amsterdam due to where my local UK airport is located. This means I use my laptop in a total of 3 timezones when I travel. Which is why I appreciate the fact that Gnome will detect my timezone when I travel and correct itself accordingly. I couldn’t find any way of doing this in KDE and it is a pain to do this manually when every other device I own can manage this automatically.
One of my biggest gripes with KDE Plasma will be a minor thing to many but it breaks my workflow quite a bit. I often have several SSH sessions open in several terminals at once whilst testing during my day job. I need to enter something in one and see log output in another two for example. Gnome’s alt-tab will bring all terminals to the foreground at once when I use it to task switch, if for example I also have a full screen web browser open. KDE’s won’t so I have to individually switch to each one to bring it forward. I couldn’t find any way to fix this and it is a big deal for me.
I really like the way KDE Plasma is heading. I will definitely be keeping it on my x220 which will be used as a demo machine for a workshop I’m giving at MariaDB M|18. But it isn’t quite there yet to be used as a daily driver for me. The x260 will go back to Gnome in the next few days.
All software has bugs. Even if you could possibly write the perfect bug free software all the layers down have bugs. Even CPUs as can be seen with the recent Meltdown and Spectre bugs. This means unfortunately sometimes software will crash. When this happens it is useful to capture as much information as possible to try and stop it happening again.
One of the first things I did when coming back to work from the holiday break is code a new crash dump handler to be used in MariaDB ColumnStore. This will spit out a stack trace for the current thread into a file upon a crash. It is very useful for daemons to try and find the root cause of a problem without running through a debugger.
The first thing you will want to do is enable useful debugging symbols and frame pointers to your binary compilations. This may add a tiny overhead to binary execution, a few percent at most but it is worth it to be able to run a postmortem on crashes. The useful options are “-g” and “-fno-omit-frame-pointer”.
This is a basic crash handler, it will dump the crash data into a file with the filename of the PID of the process in /tmp. You will likely want to expand on this to add more information and error handling. The important thing is to try and avoid mallocs as much as possible:
This opens the file, writes the current time/date into it as well as the signal number that generated the crash. It then gets the backtrace and writes it into the file. We then reset the signal handler to default. You’ll need some more headers than this example, but execinfo.h, which is part of glibc, provides the backtrace functionality.
Adding to Application
Somewhere near the beginning of your ‘main’ function you need to add signal handler hooks, you’ll need to include ‘signal.h’ for this to work:
Once compiled and running an easy way to test this is to send a signal to an application to tell it that it has crashed. You can do this with “kill -11 <PID>”. You should find the crash dump in /tmp.
The crash dump file will have a list of function calls and address offsets. This may be useful but you can use the same binaries to generate source line numbers. The following is an example from a MariaDB ColumnStore binary:
With MariaDB ColumnStore 1.1 Beta now released I quickly checked Twitter today to see what the response so far has been. I noticed that someone had posted up a benchmark comparison of MariaDB ColumnStore against a couple of other databases with data that doesn’t quite add up.
You can see the blog post here. I can’t say much for the other two databases but the results and testing itself for ColumnStore did not seem right so I thought I would try them myself here.
The author was using a Ubuntu 14.04 VM with 8GB RAM. To make this test as fair as possible I decided to take my 2012 Intel i5 Lenovo X220 and run the VM I gave to participants at a recent QuruLabs workshop I gave (ignore the cancelled status, it was an issue with Meetup). The VM is CentOS 7 with 6GB RAM and only configured to use only 1 CPU core with a stock MariaDB ColumnStore 1.0.11 RPM install. This is pretty crippled and we don’t recommend running it on such a low specification.
The data set used is the MovieLens database which is about 1GB of data uncompressed (although the parts of the test used is only about 2/3rds of it). This is an incredibly small data set for MariaDB ColumnStore, it is designed for tens/hundreds of GB and TB ranges of data. But when giving laptop demos on limited amounts of RAM it is not a bad data size.
Now, the schemas used by the author are a little strange. DECIMAL types when INTs should be used for example, and one of the tables the author created has a column which isn’t in the CSV files. So for this test I used the following (the original schemas won’t make much difference, this just seems more sane):
For my testing I used cpimport but LOAD DATA would have also worked with a small performance hit. Loading the four tables took 24.83 seconds. Much lower than the author’s 4 minutes.
I then ran the four queries the author posted. I modified the first query to be the AVG on “rating” since “fact_id” isn’t even a column and I believe that is what was intended. The rest of the queries are the same:
-- Query 1
SELECT AVG(rating) FROM fact_ratings;
-- Query 2
SELECT rating, count(*) from fact_ratings group by rating order by rating;
-- Query 3
select dim_movies.title, avg(fact_ratings.rating) from dim_movies join fact_ratings on dim_movies.movieId = fact_ratings.movieID group by dim_movies.title DESC order by dim_movies.title limit 20;
-- Query 4
select dim_movies.title, fact_ratings.rating, dim_links.imdbid from dim_movies left outer join fact_ratings on dim_movies.movieID = fact_ratings.movieID left outer join dim_links on dim_movies.movieID = dim_links.movieID where fact_ratings.rating > (select avg(fact_ratings.rating) from fact_ratings) order by fact_ratings.rating DESC limit 20;
This gives the following resulting times:
Query 1: 1.65 seconds
Query 2: 2.56 seconds
Query 3: 16.74 seconds
Query 4: 25.16 seconds
Now, these times are not great, but I’m doing this on a very limited VM inside 5 year old hardware and with no optimisation/tuning. We also ran them on a modern i7 laptop with more cores enabled and more RAM, these times were around 1/3rd of the above. But the original author was observing times of more than 5 minutes for the last two queries.
So why did the original author observe such high execution times? My best guess is they didn’t set the engine to ColumnStore on CREATE TABLE so it defaulted to using InnoDB (I haven’t tried with InnoDB to compare). Also we don’t publish an Ubuntu 14.04 build so they may have created their own debug build instead which does give a performance penalty. There are not enough details to see how they built/installed it (or even what version they are running).
The moral of this story is don’t trust other people’s benchmarks. Yes, I may be biased so have deliberately tried to cripple myself where it counts. MariaDB ColumnStore definitely should not be that slow on that setup. I highly recommend trying MariaDB ColumnStore for themselves to see how well it works with your analytical use case. MariaDB has fantastic consulting and support teams if you need help setting up and optimising your database and queries.
Unlike most storage engines, MariaDB ColumnStore does not store its data files in the datadir. Instead these are stored in the Performance Modules in what appears to be a strange numbering system. In this post I will walk you through deciphering the number system.
If you are still using InfiniDB with MySQL, the system is exactly the same as outlined in this post, but the default path that the data is stored in will be a little different.
The default path for the data to be stored is /usr/local/mariadb/columnstore/data[dbRoot] where “dbRoot” is the DB root number selected when the ColumnStore system was configured.
From here onwards we are looking at directories with three digits ending in “.dir”. Every filename will be nested in similar to 000.dir/000.dir/003.dir/233.dir/000.dir/FILE000.cdf.
Now, to understand this you first need to understand how ColumnStore’s storage works. As the name implies every column of a table is stored separately. These columns are broken up into “extents” of 2^15 (roughly 8M) entries either 1 or 2 extents (depending on how much data you have) will make up a segment file. Each segment file is given a segment ID and a collection of four segments is given a partition ID. In addition to all this every column is given an “Object ID”.
You can find the object ID for every column using the information_schema.columnstore_columns table and details about every extent, including the partition and segment IDs using the information_schema.columnstore_extents table. This will be useful when working out the file names.
The following is how to work out a filename from an object ID. It should be noted that object IDs are 32bit and the output of each of these parts is converted to decimal:
Part 1: The top byte from the object ID (object ID >> 24)
Part 2: The next byte from the object ID ((object ID & 0x00ff0000) >> 16)
Part 3: The next byte from the object ID ((object ID & 0x0000ff00) >> 8)
Part 4: The last byte from the object ID (object ID & 0x000000ff)
Part 5: The partition ID
Part 6 (the filename): The segment ID
Each part here apart from the final part is a directory appended with “.dir”. The filename is prepended with FILE and appended with “.cdf”. There is of course a much easier way of finding out this information. The information_schema.columnstore_files table will give you the filename for each object/partition/segment combination currently in use as well as the file size information.
I have a friend who was hit by a workforce reduction at a company I used to work at (as was his entire department). He is a brilliant engineer who has worked on some massive projects. But is struggling to find somewhere new in-part due to the insane way the recruitment process works in our industry.
I’ve had a lot of conversations about recruitment in the last few months for seemingly random reasons and unfortunately there are still lessons that appear to need to be learnt. This post is to share my opinions on hiring in tech, specifically Open Source remote workers but some can be applied to other roles. My opinions here may not be popular but feel free to comment.
Degrees vs Experience
There is a case my wife witnessed many years ago where there were two candidates for a specific job. The first had many years experience in that role, the second had an unrelated degree and had no experience. The company jumped straight onto the second candidate. This wasn’t a tech job but unfortunately the same thing is happening in tech.
In my case I do not have a degree, but I have many years experience. I have been a Principal Software Engineer for a large tech company. I have been the team lead on large software projects with engineers working for me (one project that failed three times under other people but succeeded under me). I have been a keynote speaker at an O’Reilly conference. My point is I am probably not too bad at what I do (that isn’t for me to judge).
Yet I know of HR department in certain companies that would instantly throw my CV in the bin because a degree isn’t listed on there. I’m not so concerned for me right now but this is happening to other really good engineers who would be an asset to a company.
Don’t get me wrong, a degree is a great way into this industry but it should never automatically trump experience. Both should be accessed with some balance. Some of the best engineers that I have worked with have had degrees, but some have also come from military backgrounds or were not fortunate enough to be able to go to university and had to work hard in other ways.
A few years ago I noticed that a local friend who was a chef was gifted enough to understand code. I helped train him up in how to be a good engineer and he got a job as a junior PHP developer. Today he is a team lead at a software company in a nearby city. He doesn’t have a degree but is extremely dedicated, a very fast learner and very good at what he does.
I have tweeted about this a few times recently, for example:
New dictionary entry request: Whiteboarding: like waterboarding but for bad tech job interviews.
For those who don’t know, whiteboarding is an interview where you will be given a certain coding task to do on a whiteboard. For example, code a binary tree, or code X sorting algorithm.
I have failed these tests every time (apart from one company where I told them that they had failed the interview). They are almost never relevant for the job. You don’t have your tools, resources and colleagues with you. Sometimes the job is to work in a language you might not even know yet.
I had an instance where I needed to hire a new engineer for a job to work for me. I picked the guy who probably didn’t have the most experience but had a clear history of having an ability to learn quickly and was a better personality fit for the team. A manager above me gave the two candidates a whiteboarding interview and overruled me, hiring the other person. I actually left that department soon after that.
This process might be a good way of weeding out recent graduates but I can say with some certainty that many of my past colleagues who have worked on software that is pretty much the backbone of the Internet would (and do) fail these.
I have literally heard these interviews referred to as hazing processes or a way for recent graduates to try and outsmart experienced engineers. Whilst I like to defer to Hanlon’s Razor (“Don’t assume bad intentions over neglect and misunderstanding”) I can see where these views are coming from.
Companies that interview in this way are throwing away talent in favour of people who can memorise interview cheat books and algorithms from the 1970s. Almost no engineer ever looks at these again after they graduate.
What does work?
I’m gearing this part more to remote work Open Source engineers because that is what I know, but some of this will also apply to other tech jobs too.
Many experienced engineers have contributed to projects on GitHub. Unfortunately GitHub doesn’t always have the right UI to show you this. Make sure you look at contribution history and those projects as well as their repositories. My main projects work using developer branches inside the projects so they do not show up in my repositories (and often don’t show up much at all because master isn’t updated often).
If you really do need to test their development abilities give them a write-in test. Give them a code sample to debug. But not in a face-to-face setting. The experienced candidate will have certain tools and processes for how they work. Throwing a Mac with a US keyboard layout in front of them and asking them to fix something when they only used Linux with a UK layout is going to struggle.
You could get the candidate to look over the source code of your project in their time and get them to answer questions on how they think some parts work. You may not always get right answers and that isn’t bad, people are sometimes wrong and being wrong sometimes is a good thing (that is probably for another blog post). But you get a clearer picture of how they communicate ideas. Also really important that such tests are vetted by your team first.
The key things I hire for are experience in the industry relevant to the role (a junior probably won’t have much experience), the ability to learn and a personality that will work well with our team. I often hire home workers which also require a certain discipline to get the job done which I look for, it isn’t for everyone.
Drilling down into experience and ability to learn. The candidate doesn’t have to have any experience in the language that we use. Often candidates will come to me with experience in several different programming languages and just like human languages, once you have a few under your belt it is pretty easy to pick up more. As long as the candidate is showing willingness and ability to learn.
This is the kind of industry that you are forever learning in. There is new technology all the time, the field grows exponentially. I don’t (and couldn’t possibly) memorise the entire syntax for every language and library I work with, but I have memorised where to quickly look them up when I need them and which is the right tool for the job. Having this index card type ability is way more important than how to write a sort algorithm off the top of your head that is pretty much built-in and highly optimised in every language’s standard library.
In my own experience I have made hiring mistakes before. Most of the time for not following my own rules and trying to get a candidate to fill a role quickly rather than finding the right person for the job. I have learnt from them and hopefully it won’t happen again. If you hire someone that doesn’t fit then it might be possible to move them to a department where they will, or as a worst case scenario there is the probationary period.
I know my opinions in this post will not be shared by everyone. There are many big companies who’s hiring techniques are the polar opposite of what I have said here. I do hope as an industry we can improve things so that the right people get the right jobs. But it is never going to be an easy task. Some mistakes will be made along the way.
Please comment on your own experiences and thoughts on the subject. I would be really interested to learn beyond my own bubble.