PEOPLE FEB 28, 201913 MIN READ

7 steps to building an
engineering competency
matrix
Justin Cowperthwaite
Engineering Manager

Every engineer deserves a clear growth path so they can understand, plan, and execute on meaningful
career growth. Providing a framework for this growth (we call ours a competency matrix; it’s also known
as a career ladder, or professional development ladder) is important work, and the responsibility of any
organization that wants to nurture and grow its employees. Back at the beginning of 2018, we had 32
developers and a plan to double throughout the year, we already had a competency matrix, but it was
woefully outdated. It focused on our more junior levels, maxing out at a level which some developers had
already reached. It was also misaligned with the skills our organization had grown to value, which meant in
practice, we often ignored it. It was time for a re-design.

Building a new competency matrix was a learning process, and a lengthy one, taking about eight months to
complete. Along the way we discovered things we valued, as well as what the keys steps to building a
career ladder are (and which ones are wasteful). While every matrix is different, and will reflect the values
of the organization that wrote it, the process of producing a succinct career ladder to guide your team is
consistent.

When we published our new Engineering competency matrix in December, we received many emails from
teams saying they were working on similar systems. Because of this feedback, I want to share the steps we
went through, and the lessons we learned, to help teams reach a productive conclusion with much less
waste, and in much shorter time, than trying to figure it out from scratch.

If you want to provide your employees and reports with a clear, agreed-upon, and well-defined path for
growth within your organization, then this blog post is for you.
Step 1: Make this someone’s top priority

In retrospect, this was the biggest factor in our lengthy redesign process. I had initially taken on this project
as one of my many side projects. The only time I had to dedicate to the matrix were early mornings, late
nights, and weekends. This was a passion project for me, and I loved working on it, but I was not able to
give it the care it needed.

When Lena Reinhard, our new Director of Engineering, joined, she took this on as her first big project. We
worked together closely after this, but the fact that she now “owned” it made it immediately obvious to me
how much my limited availability had been holding this project back. If I had continued on driving this
during my free time, I don’t know if we would have seen it completed in 2018.
If you are taking this project on, give it the attention it deserves by giving it to someone as their #1 job.

Step 2: Agree on what you are building

Until all stakeholders agree on what your goals are, every attempt at implementation will stall.

A competency matrix is a powerful tool to set a cultural tone and direction, so in designing your matrix,
you’ll have many impactful choices to make. Each one has consequences, and you’ll only get through them
if the team is aligned. One of the questions that came up repeatedly for us was: is this a codification of the
status quo or is this aspirational? (It took us about halfway through the process to explicitly agree it was a
blend of both.)

Another key point of agreement is: who is going to be affected by this? In our case this question hinged on
whether or not our new matrix would affect our Site Reliability Engineers (we decided it would). Maybe
you are building a matrix for your whole company, or maybe you are building a matrix for your frontend
development team. The breadth of roles affected will greatly change the competencies you choose to
codify, and the level of abstraction you need to use. So ask these questions, and get explicit agreement.

Finally, we needed to agree on the goals of the matrix. We decided the primary goal was performance
evaluation and growth planning with individual engineers at CircleCI. Secondarily, we wanted to use it to
influence our hiring process and communicate externally what being an engineer at various levels meant.
Knowing the potential uses, and the priority of those, guided certain decisions.

Step 3: Guide by your values

This, for me, was the most fun part of this entire process. This is the part where you sit down and debate
“what matters to us?” We had some help from our excellent Head of HR, David Mann, who came in with
note cards detailing out ~100 behavior traits that are valuable to have as a professional. These were not
engineering-specific, and ranged from Communication to Political Awareness. We also utilized other
publicized competency matrices to seed ideas of what couldbe in the running.

We also enjoyed coming up with our own values as a team. The one that sticks out most in my mind
is Economic Thinking, something the management team continually discussed as being one of the key
skills that differentiates good developers from great ones. We didn’t borrow that competency from any HR
cliff notes or other matrices we consulted, but from the get go this was a key competency we knew would
be in the final cut.

Start wide, dream big! It’s better to cast a wide net up front and merge and cut later.
Once you have a list, doing a few passes to merge similar competencies is a good idea, as it will save you
time later in the process. For example, we merged Pragmatism and Economic Thinking, as we realized the
manifestation of these competencies resulted in the same behaviors.

Step 4: Define your levels

Before you start to actually fill out the content its good to define the x-axis of the matrix: the growth in
terms of title and responsibility. At CircleCI, we already had the existing set of titles we wanted to use, (E1
/ Associate Software Engineer to E6 / Principal Software Engineer). Then we explicitly agreed upon on
generalized responsibilities for each title.

We broke it down into two categories, both focusing on individual contributors: E1, E2, and E3 focus on
mastering the skill of software engineering and becoming a highly effective IC. E4, E5, and E6 focus on
utilizing those skills to scale impact and create leverage across larger and larger groups of people. Once we
had this high-level guidance, we broke it down further, assigning scope to each of those execution levels.
Before we started creating content, we had the follow guiding principles:

E1 - E3: Execution of Work

E1: Within Task
E2: Within Project
E3: Within Team
E4 - E6: Utilizing skills to scale and generate leverage
E4: To team
E5: To set of related teams
E6: To engineering department

Of course, these are guidelines, not rules. For example, we expect E4s, E5s, and E6s to contribute to
organizational Practices and Processes at differing frequencies. Career development, like all human
pursuits, is messier in practice than in theory. However, by setting clear guidelines up front, you can make
the low resistance paths easy, and use your time focusing on those harder, messier bits.

Step 5: Create the content

This step is definitely the most laborious. However, if you have followed steps 1-4, it should make this step
much easier for you than it was for us. I tried to tackle this step multiple times throughout the process, but
without the correct framework to shape my approach, I often felt aimless and scattered. Once the
framework was in place, filling out the content became much more straightforward.

Given the meatiness of this step, I’m going to break it down further into sub steps. Following this
sequencing should minimize any wasted work.

1. Define a single level for all competencies

2. Look for opportunities to merge competencies
 3. Fill out the rest of the levels
4. Wordsmith

Step 5.1: Define a single level for all competencies

The first step I would strongly recommend is to define one level, such as your Senior Software Engineer,
for all competencies. Rather than trying to wordsmith the nuanced differences between what it means to
exhibit Delivering Feedback as an E2 vs an E3, reifying what you mean by each competency by defining a
single level is a very enlightening and aligning process.

We approached this slightly differently, by defining two tiles: E3 (top level individual contributor) and E4
(first position of scale and leverage). Given our bifurcated approach to executing vs scaling, this gave us a
better picture of what growth looked like through a pivotal step in a developer’s career at CircleCI.

Step 5.2: Look for opportunities to merge competencies

An amazing thing will happen while you are defining that single level for every competency: you’ll realize
some of the values point at the same behaviors. This is an opportunity to merge!

After we had defined single levels, we noticed that we had two tiles that were eerily similar: Self Starting
and Delivery Accountability. Although these competencies had appeared different, the behaviors we had
codified were similar enough that we merged them into a single tile: Reliability and Delivery
Accountability.

Early on, we made the explicit agreement that we wanted our matrix to be a collectively exhaustive
definition of what growth looked like for a developer at CircleCI while also being as simple as possible.
Ultimately, we ended up with 27 competencies. However when we started 5.1 we had almost 50. Defining
the behaviors for each showed us where we had opportunities to consolidate.

Step 5.3: Fill out the rest of the levels & wordsmith

Now that you know you have the behaviors and competencies you want in your matrix, it’s time to fill out
the rest of the cells! This step is a lot of work, but an important step as it is the core of the matrix.

During this step would be the best time to start wordsmithing. While you are in the throes of defining the
nitty gritty details of what it means to scale Economic Thinking to one team vs many is the best time to be
critical about whether or not the language you are using properly conveys the intent. I would strongly
advise against wordsmithing before this, but this would be a great time to revisit the tiles you defined in
your first pass.

Step 5.4: Polish & consistency

I have no doubt during your run through defining each individual tile you will learn things that you want to
apply to tiles you have already defined. About halfway through the process, Lena and I discovered that we
used “often” and “usually” to mean the same thing at about a 50/50 rate. Since consistency was a big focus
of ours, we decided on “usually” and put it in a proverbial parking lot to apply later.
I would suggest a similar approach. As you find opportunities for polish (such as replacing all occurrences
of “often” with “usually”), write them down, and use that as your todo list for a final pass. That way if you
have learnings that contradict each other, you can resolve those conflicts before application, minimizing
churn and ensuring greater consistency.

Step 6: Involve the people it will affect

Great, you have a competency matrix, you’re done! Actually, not quite. To us the idea that Lena and I
would sit down, write up a bunch of values and behaviors, get it signed off by our VP of Engineering, and
then present it to the world as final was a laughable proposition. This was going to affect our culture, our
hiring, and most importantly, shape career growth for all the people we work with and whose careers we
care about. We had been testing different versions and aspects of the matrix throughout the months it took
to design it, but now it was time to really put it to the test. With our first pass done, we set out to garner
feedback.

Lena orchestrated an excellent feedback session in the format of a focus group, made up of individual
contributors of all levels, managers, and HR, that had both asynchronous and synchronous portions. There
are many ways to gather feedback, such as having a diverse portion of individual contributors complete
mock self evaluations with their manager. Regardless of how you do it, the important thing is ensuring you
gather feedback from the people this will affect.

The feedback we received was extremely valuable for two reasons: it surfaced good questions, which led
us to tweak the matrix so it conveyed what we intended. This process also ensured we had codified the
values of the organization, not just a few managers, and that the value progression was representative of
how our engineers saw their career growing. This step was extremely important in generating confidence
that what we had developed represented what we wanted, and would be well received.

Step 7: Ship it!

The best step of all - ship it! You have a finished product that has been wordsmithed, stress-tested, and has
buy in from respected individuals in your organization. However, even after you following a thorough
process, it will never be perfect. As people start to use and test what you have created they will find slight
inconsistencies and opportunities for improvement. Be ready for feedback now, and on an ongoing basis.

There are two mechanisms you should put in place now that you have your matrix: a way to gather
feedback and an agreed-upon timeline for addressing it. Constantly iterating on your competency matrix
would be a bad idea, because it means constantly moving the goal posts for career development at your
organization. However, pretending that it is perfect and will never change is foolhardy and unrealistic.
We’ve developed a mechanism for collecting feedback (spoiler alert: it’s a Google Doc) and will be
revisiting the matrix at the six month mark to address what we’ve learned. This provides a level of stability
that is important, while also allowing for iteration and evolution over time. I suggest you provide a
mechanism to achieve the same goals.

Congratulations on your new competency matrix! Hopefully this guide allowed the process to be quicker
and easier than it was for us.

If you have your own thoughts, experiences, or questions about building a competency matrix, I’d love to
hear them! Feel free to reach out to me via email (justin@circleci.com) or via Twitter (@JustinC474).

P.S. Want to be a part of a culture that values individual growth and career progression? We’re hiring!

Read more:

What we’ve learned about hiring engineering managers

Why we re-designed our engineering career paths at CircleCI
Designing a package manager from the ground up: an inside look at CircleCI orbs