Assaf Muller is associate manager of the Red Hat OpenStack Networking engineering team. He was formerly a neutron core reviewer, member of the drivers team and responsible for the project’s testing. Follow this blog where he shares his thoughts and experience on OpenStack.
Leadership to me seems largely revolving around the topic of persuasion.
How do you get people to do something important but not exciting? (e.g. documenting a feature). How do you reposition the unattractive activity in to a norm, and make people want to do it?
How do you nudge people’s mentality or attitude? Spotting an overarching problem with an employee (e.g. snubs his/her nose at writing end to end tests), how do you get them to come around and eventually even lobby others?
How do you communicate a significant decision? (e.g. going with one technology over the other). You won’t be able to please everyone, but you can minimize pushback and resentment with some.
How do you handle yourself when the person you are talking with has a polar opposite interest to you, and a decision has to be made nevertheless? (e.g. you’re trying to decide which team will on board a new hire).
All people naturally fall somewhere on the scale. Even looking at an individual, you may come down harder on one side at any given day depending on how well you slept, if you have indigestion, or if you just happen to feel impatient. People respond to these persuasion styles differently, and like with anything I’ve found that people gravitate towards others like them, and this is when adjusting to your audience comes in to play. Culture also factors in to this. Coming in to an international organization, ignoring outliers, the difference between, say, an Israeli and an English employee is almost comically apparent.
Defining the aggressive and measured persuasion styles
To elaborate on the aggressive and measured approaches, let’s use an example of a manager involved in a technical decision. The manager in this scenario is exposed to information earlier than his team members, and a decision has to be made regarding the high level architecture of a future project: Which technology stack should be used: Tried and true A or new and exciting B? To drive home the point, in this scenario the manager develops a strong opinion favoring A, based on thorough exploration, past experience and strong expertise in the problem domain. Now the question is – How do you as a manager gain deep acceptance of A and minimize pushback within the team?
The aggressive approach would be to expose the manager’s opinion early in the process, fully embracing an inherent managerial influence, and highlighting the benefits of option A. An even more aggressive approach would be to avoid presenting B as an option in the first place, or not framing the conversation as a discussion but as an announcement.
The measured approach, even in the case of the manager’s strong preference for option A, is to allow a full and honest discussion to take place. The manager would present their opinion late in the process or perhaps never at all. The manager would either stay out of the discussion entirely, or help facilitate an investigation led by the team itself, looking in to the benefits and drawbacks of options A and B. The manager would be presenting both options in a completely neutral way, as if the manager holds no opinion on the matter. The manager does not present A and B, rather the options present themselves by an external quasi-Buddhist selfless conscious creature. The advantage of course is that a decision that comes from within the team has a much stronger stickiness. From the perspective of the manager, a possible risk is that the team can land on B. At that point the manager has to choose their next move, especially in the case that the manager is convinced that choosing B will cause real and irreversible harm to the organization.
Different but equal?
This is when I tell you that while both approaches are different, one is not better than the other. Well, no. What wasn’t obvious to me is that it is not merely a matter of style. The measured approach is in fact common to all senior and effective members of my organization and is therefore implicitly encouraged, to the extent that this topic becomes relevant when it is time to talk about bonuses and promotions. While the lack of tolerance for a different tendency was surprising to me, it is easy to understand why in an Engineering organization, a measured approach would be beneficial. To explain why I think that an organization rewarding measured persuasion might not be fully optimized, I claim that it is reasonable to draw a line from persuasion style to a wider discussion about personality traits. For example, the ability to regulate emotions like passion or agitation, or even something like a person’s tolerance to disagree in public forums.
Point being, if you tell me that a person has an extremely measured persuasion style, I will be able to tell you with reasonable certainty about other character traits that that person may possess.
A natural tendency to avoid stating their opinion on controversial issues
Shying away from being the sole and visible decision maker (and in extreme cases: cover your tush syndrome)
Not being willing to cut a discussion short when it becomes clear that further debate will not change anything
When an unscientific method is preferred in an Engineering organization
My claim is that at times, an aggressive decision maker is what you’re after. Sometimes timing is what will make a difference to your success and you cannot afford the unfortunate cost of inclusivity or the uncertainty the decision making process introduces to your organization. A team or organization filled with hot shots will be too quick to make mistakes or too slow to identify one. An organization filled with academic style deliberates will be too busy to ensure that the problem was looked at from all angles to notice that a more nimble competitor has already gained market share. An organization that explicitly favors this type of mentality may implicitly create a homogeneous echo chamber and conflate an attitude that uses disagreement as a tool to find truth  with those that are just unwittingly difficult. Sometimes it is unfair to decide by committee, or to involve people impacted by the decision, and to be unwilling to make a hard decision yourself. Sometimes we are too quick to consciously decide not to make a decision and let the scenario play itself out, rationalizing for inclusivity and transparency, thereby ignoring the human cost to the people involved by a persistent uncertainty. I would go further and say that the line between an overly measured demeanor and cowardice is thin.
I’d try and reframe this discussion from one where one approach is clearly superior to the other, to a discussion about diversity of thought. While as a rule of thumb I prefer measured persuasion, I’ve found that both approaches have their merits, and that a team flourishes with a healthy mixture of attitudes. I would encourage people in a position of influence to try and create an atmosphere where both approaches are ultimately tolerated, and in hiring stages I would be wary of biases that lead us to recruit people just like us.
Edit: The position has been filled, thank you everyone.
I’m looking for a Software Engineer to join the Red Hat OpenStack Networking team. I am presently looking to hire in Europe, Israel and US East. The candidate may work from home or from one of the Red Hat offices. The team is globally distributed and comprised of talented, autonomous, empowered and passionate individuals with a healthy work/life balance. The candidate will work on OpenStack Octavia and LBaaS. The candidate will write and review code while working with upstream community members and fellow Red Hatters. If you want to do open source, Red Hat is objectively where it’s at. We have an institutional culture of open source at all levels and this has a ripple effect on your day to day and your career at the company.
Adding new configuration options has a cost, and makes already complex projects (Hi Neutron!) even more so. Double so when we speak of architecture choices, it means that we have to test and document all permutations. Of course, we don’t always do that, nor do we test all interactions between deployment options and other advanced features, leaving users with fun surprises. With some projects seeing an increased rotation of contributors, we’re seeing wastelands of unmaintained code behind left behind, increasing the importance of being strategic about introducing new complexity.
I categorize the introduction of new OpenStack configuration options to two:
There’s two or more classes of operators that have legitimate use cases and a configuration option would enable those use cases without hurting cloud interoperability
For example: Neutron DVR, is essentially a driver for router implementations, that changes your L3 architecture substantially while abstracting details via an API. DVR has various costs and benefits and letting operators make that choice, especially as the feature matures, makes sense to me.
Developers that don’t fully understand the choice they are making and pass the complexity down to the operators to figure out. This results in options that are often never changed from their defaults because operators don’t have access to sufficient documentation that explains the rationale for choosing a specific value, as well as a misuse of time and energy because developers sometimes focus on use cases that are off center or don’t exist.
For example: neutron.conf:DEFAULT:send_events_interval: Number of seconds between sending events to nova if there are any events to send. Unless you grep through the code, how is an operator supposed to know if they should increase or decrease the value? Even then, shouldn’t developers take responsibility of that choice, and test for the best value? If problems are found under load, shouldn’t the value be calculated as a function of some variable? Instead of distributing the work to thousands of operators, wouldn’t we as a community like to to do the work in one place?
When contemplating adding a new option, ask yourself:
Is it possible that you don’t fully understand the use case, and in lieu of making a choice, you’re letting the operator bear the burden?
Are you, and your replacement, prepared to own the cost of the new option indefinitely?
In continuation to a previous blog post https://assafmuller.com/2016/12/02/upstream-contribution-give-up-or-double-down/, I presented a version with new content at the recent OpenStack Boston summit. The session had a fair amount of participation and discussions, where we talked about the journey of a new contributor in OpenStack through the lens of data gathered from Gerrit and Stackalytics. We even had members of the Technical Committee that were interested in concrete action items they could take on their part – How do we make it easier for new contributors? What behaviors work in the quest to get your patches merged quicker?
The video, slides and code used to generate the data are below:
Every once in a while I have to recognize the beauty that is a high level language.
Consider this code snippet:
l = [1, 2, 3, 4]
for x in map(square, l):
And this one:
from multiprocessing import Pool
l = [1, 2, 3, 4]
for x in Pool().map(square, l):
Discounting the import line (Because I’m trying to make a point here, dammit!), the diff is a total of 7 characters. Of course, if you recall your Computer Science operating systems 101, the complexity that is hidden here is breath taking. And yet, Python multiprocessing exposes an API that is identical to Python’s good old ‘map’. It does so and manages to keep it boring, without exposing the caller to messy internals and endless complications that are usually involved whenever we dive in to the world of multi processing.
Even more impressive is the fact that I exploited this abstraction in a real world app that does honest to Zeus useful things, and it “just worked”. The processing time was cut by half and I didn’t need to worry about forking, sockets, PIDs, creating and managing processes or any of that mess. All I had to do was focus on the subject matter and not the underlying code. That is the beauty of a high level language. You don’t have to use up cognitive load thinking about memory management, the syntax of templates meta programming, or the depth of pointer dereferencing required, instead you can focus on getting something useful done.
* Herein lies a vim vs. Emacs type disclaimer: Every tool has its place. I would never dream of using Python for a demanding mobile game (Or a cloud operating system, for that matter), but wow is Python fun when you’re parsing and processing tens of thousands of JSON files.
Tomorrow morning, February 1st, OpenStack contributors will receive emails to elect their new PTLs. OpenStack Networking is having an interesting election this cycle. Please, have some fun with your morning coffee, spend a few minutes and read Ihar and Kevin’s nomination emails.
The position has been filled, thank you for all of the applications!
I’m looking for a Software Engineer to join the Red Hat OpenStack Networking team. I am presently looking to hire in Europe and Israel. The candidate may work from home or from one of the offices listed here. The team is globally distributed and comprised of talented, autonomous, empowered and passionate individuals with a healthy work/life balance. The candidate will work on OpenStack networking projects such as Neutron and LBaaS. The candidate will write and review code while working with upstream community members and fellow Red Hatters. If you want to do open source, Red Hat is objectively where it’s at. We have an institutional culture of open source at all levels and this has a ripple effect on your day to day and your career at the company.
One of my personal highlights of the recent Barcelona Summit was a session by Mirantis engineers Elena and Oleg titled “Is OpenStack Neutron Production Ready for Large Scale Deployments?”. In the session they outline a comprehensive control and data plane testing effort, run on two labs, one with 200 nodes and run of the mill hardware, and the other with 378 and top of the line hardware, all running the Mirantis distribution based off Mitaka with standard ML2/OVS, DVR, L2POP and VXLAN. In the session they show near line-rate speed for east/west and north/south routing with jumbo frames and VXLAN offload enabled. They were also able to spawn 24,500 VMs across 125 networks without errors and low CPU consumption.
Turning our eyes to adoption, the OpenStack Foundation conducts a usage survey every 6 months. Looking at the April 2016 user survey, we can see that ML2 with Open vSwitch and Linux Bridge dwarf other solutions.
Examining the openstack/neutron project via Stackalytics we see that ML2/OVS has a rich and robust community with 20 companies contributing over 5 patches in the Newton time frame. 779 people have contributed at least 1 patch to Neutron since its inception, 215 of which during the Newton timeframe. Some of the effort targeted the base Neutron platform, e.g. configuration options, database work, versioned objects, quotas or other in-tree ML2 drivers such as SRIOV. Looking at the contribution of everyone who has committed at least 5 patches in the Newton cycle, we are left with 50 authors, 42 of which contributed at least 1 patch to ML2/OVS.
Looking at the interactive version of the OpenStack user survey we can see that ML2/OVS is the most popular choice by an order of magnitude regardless of deployment size. And so to answer the question: “Is OpenStack Neutron ML2/OVS Production Ready for Large Scale Deployments?”. Yes, it is, of course it is. It has been for some time now.
Ever since I’ve been involved with OpenStack people have been complaining that upstream is hard. The number one complaint is that it takes forever to get your patches merged. I thought I’d take a look at some data and attempt to visualize it. I wrote some code that accepts an OpenStack project and a list of contributors and spits out a bunch of graphs. For example:
How long does it take to merge patches in a given project over time? Looking back, did governance changes affect anything?
Is there a correlation between the size of a patch and the length of time it takes to merge it? (Spoiler: The answer is… Kind of)
Looking at an author: Does the time to merge patches trend down over time?
Looking at the average length of time it takes to merge patches per author, how does it look like when we graph it as a function of the author’s number of patches? Reviews? Emails? Bug reports? Blueprints implemented?
The data suggestes answers for many of those questions and more.
I’ve observed a persistent theme across valuable and successful CI systems, and that is actionable results.
A CI system for a project as complicated as OpenStack requires a staggering amount of energy to maintain and improve. Often times the responsible parties are focused on keeping it green and are buried under a mountain of continuous failures, legit or otherwise. So much so that they don’t have time to focus on the following questions:
How do you determine that a job failed?
How are the results presented to the relevant developers?
Can developers do anything about a failure?
To bring it to concrete terms let’s take a look at how Rally is used in upstream jobs. This is not a criticism of the Rally project itself, which I’m a big fan of, but rather how it’s used upstream. It uses the standard upstream CI infrastructure, which is a miracle of engineering when it comes to correctness tests. The infrastructure spins up VMs from a node pool comprised of many clouds. It then uses devstack-gate and devstack to install OpenStack and runs several Rally scenarios. When the result of a CI run is True or False, the variance of hardware and congestion levels is irrelevant. However, when you’re trying to measure performance, variance matters. You can try setting a maximum, and any result over the maximum is declared as a failure, however with a variance sufficiently large setting up SLAs is an exercise in futility.
Let’s look at a recent Linux Bridge change  that cannot impact Rally results (The Rally job is setup to run against Open vSwitch). Consecutive runs would ideally show the same results. However, looking at the results of patchsets 10, 11, 13 and 14, we can see that the total length of the job runs between 60 and 83 minutes. The full duration of the create_and_list_ports flow runs from 1517 seconds to 1887 seconds. The average for a single create_and_list_ports execution runs between 4.58s and 5.29s. What am I supposed to do with the results of the next run? What can I learn from it? I’d argue: Nothing. The results of the job are not actionable. The result is that the job has been non-voting ever since its introduction and worse yet, none of the engineers I work with look at its results.
The next step would be to give up on the idea of gating or blocking performance regressions and instead detect them after the fact. We can do that by persisting historical results, graphing them and spotting trends. It’s clear that with a variance this large, the results would not be actionable either. To demonstrate this, let’s turn to the fantastic openstack-health project. Looking at the Neutron API test with the longest average run time  we can see that at the time of writing, the test ran 249 times in the past month so we get a great sample size. However, the run time graph looks like a Jackson Pollock painting, with a min of just under 5s and a max of just over 9s. Looking at the graph it’s clear we can’t clean up the data via statistical Jiu Jitsu either. When consistency matters, I don’t think you can get around a dedicated bare metal setup.
The Gerrit interface does a great job of presenting CI results, and a failing voting job forces developers to look at its results. However, I don’t know many engineers who look at CI results as a form of amusement. Post-merge and periodic CI runs in to these issues – They burn your favorite form of fossil fuels and drain the life force of the fine folks who maintain it but the results are often not presented in a consumable manner. Running the tests reliably is as important as making sure the intended audience is aware of the results. One solution could be to make sure the relevant developers subscribe to a mailing list, triggering a mail on failures filtered after distracting infrastructure issues. Periodic CI can only be valuable if it’s actionable and developers are held accountable and demonstrate a persistent urgency to failures.