The Sorting Hat
Amy Writes Words #12
When I tell the story of why I became a software engineer, it goes like this: I was 23. My now-husband was taking an algorithms class at Harvard Extension School. One day, it was time to study heapsort. Who knows why, I guess because I like puzzles, I started puzzling over heapsort. I puzzled so long that I managed to figure out how to write pseudocode that would, not especially elegantly, first heapify and then sort an array of numbers. I thought this was so cool that I wanted to do puzzles like this all the time. I wanted to learn to program!
So I did. I took an intro to programming class at Harvard Extension, and then I took that same algorithms class, and I wrote better, real code for the heapsort, both the iterative version and the recursive version, and I learned how to talk about the kind of time it took to use heapsort as opposed to the kind of time it took to use other kinds of sorts. Then I got a job programming and I also TA’d that algorithms class and I corrected many other peoples’ heapsorts. Then the semester ended and I decided I did not want to TA any more because grading problem sets was soul-sucking, and then, in the 20+ years of my career in tech that followed, I never once had occasion to even use a sort that I knew was a heapsort again, I just used array.sort in whatever language I was writing in, so that by the time someone (not unreasonably, really) asked me in an interview about my professed love for the heapsort, I could no longer in fact remember any details about it, and I certainly could not implement it anymore.
The puzzles I’d been solving weren’t anything like the heapsort, it turned out. They were more likely to turn out to be about a background process that was quietly changing the salt on an accounts table, thereby locking users out. And then, later, when I started managing people, most of the problems that I sought to solve weren’t puzzles at all, because the problems of leadership are not puzzles.
*
I remain constantly amused by the existence of a book about engineering management entitled An Elegant Puzzle: Systems of Engineering Management, as if managing systems of people working together with software systems is either elegant or a puzzle. All human systems are messy and insoluble. They are alive, and they are recursively complex, or, if you prefer, they are bigger on the inside.1
*
Early this July, past midnight, insomniac, I decided to remember the heapsort. I did not decide to look it up, I decided to recall it. And, over the next two hours, as painfully documented in the notebook I scribbled in that whole time, I did indeed remember it.2
I remembered why you first heapify the numbers, because if you know that the top of the heap is the smallest (or largest) number, and that this is true on all the branches going all the way down to the bottom of the heap, then you know that the amount of time it will take you to then sort the heap, using the most basic ‘swap places with your neighbor till there are no more swaps left to do’ algorithm, that that amount of time will be limited by the depth of the heap, rather than the length of the array of numbers you are trying to sort. Your worst-case heapsort will take, then, O(n log n) time, if you want to be technical.
O(n log n) time, and you better be able to write all of this up and explain it in half an hour, on a whiteboard, if you want the sorting hat to sort you into the pile of people called “technical enough” to be '“real engineers”.
I proved to myself that I remembered how to write a heapsort, both iteratively and recursively (the recursive version took me a bit, I’d forgotten how mind-bending recursion can seem, and also it was 1 am and I was manic). But yes, I did both.
I mean, there are a bunch of other algorithms you need to remember though, to pass a whiteboard test, to pass the kind of engineering interview that is assumed in the pages of Cracking the Coding Interview (if you have three months of your own time to study up again every time you change jobs), or, if you are exceptionally lucky, the kind of interview that you can just learn to ace by taking the class at Stanford whose sole purpose is to teach you to pass those interviews, called Problem-solving for the CS technical interview.
Like the SAT, also a Stanford production (an explicitly eugenicist one, btw), these kinds of interviews mostly measure one thing very well: how much privilege you have in the world.
*
Once, I was rejected for a job managing engineers. There’d been several parts of the interview process that were ‘technical’ in nature. I was rusty writing code, since I’d been busy recruiting, hiring, and managing engineers for several years already. One part of the interview was a code review. These are touted as a better kind of interview than the whiteboard one, and perhaps they are. In this case, however, I was put in a small windowless room with a man who was a last-minute swap for an interviewer who was out sick. The man did not want to be in the room. He handed a computer to me with a pull request on it, and asked me to write comments on the PR while he sat there and took notes. The PR had no readme and no context around it. “Well the first thing I would do is to ask the developer to explain what problem they are trying to solve, because it’s really hard to read and understand code out of context,” I said. “Can you tell me about the context here?” I asked. “Nah, I just pulled this code off of github kind of randomly,’ said my interviewer. I tried to make reasonable comments given both the lack of context and the surliness of my interviewer, but the laptop’s trackpad worked the wrong direction for me, the room was small, and my interviewer unpleasant.
Later the recruiter said I’d been rejected from the job because even though I’d actually done fine on the ‘technical’ parts of the interview, I came across as put-out that I’d been asked to do them. Specifically, he said, I “sighed a lot.”
That interview had not tested anything at all related to my ability to read and comment on code. It had only tested my nerves, and my ability to control them adequately, and in that I had been found wanting.
Some days I sigh all day long, for no reason at all. I sigh because there is a very large weight on my chest, I sigh because I’m exhausted by whatever new tortures my mind has devised for me, be it hypochondria or intrusive thoughts or debilitating anxiety or deep despair or simply a wordless screaming in my head that cannot be shushed. I sigh because I’m testing my own nerves.
Other days I sigh because I am having my nerves tested by someone else.
*
When I excavated all of this information about heapsorts from the depths of my mind, I felt proud. Since I learned heapsort I’ve been through 20+ years of life. I’ve given birth to two children, I’ve been through many depressive episodes, some so severe that I sought out any cure I could — I’ve had electroconvulsive therapy, which might or might not wipe out parts of your memory, depending on who you ask and exactly what was done to you. I’ve crammed a whole lot more information into my brain and I’ve learned a whole lot of stuff the hard way, by fucking it up. And still somehow in the dark of night I was able to search my mind and retrieve heapsort.
I could never have done it in an interview though. That kind of information retrieval requires silence, requires calm, requires time.
I felt proud that I still remembered something that I had loved when I had discovered it. And yes, some piece of me gave a big Fuck You to anyone who’s ever questioned my ‘technical’ ability, from the candidate who thought I knew a lot of sql “for an HR person” to the greeter who assumed I was on a business trip in the capacity of girlfriend, rather than customer, to the recruiter who said ‘so you can’t pass a coding test’ when I stated that it would be a waste of my time and his company’s money if he were to hire me, at this point in my career, to spend any time at all writing code for them.
“Everyone including the CEO writes code here at <<big tech co>>” said the recruiter. As I said, sir, that would be a waste of my time and your money. As I didn’t say, sir, that CEO is just causing problems.
It isn’t just that engineering leaders at larger companies shouldn’t write code because their leverage is much greater doing something else, such as leading. They shouldn’t write code also because the more power they have the more negative the impact of their carelessly dropping a pull request on some team all of the members of which have vastly less power than they do. They’re not keeping their skills fresh and maintaining a handle on the code base, they’re just causing stress for no good reason at all.
So that they maintain the respect of their engineers for being still ‘technical’ — when it is they who have told their engineers that still being ‘technical’ is important, and told them exactly what that looks like — “writing code”, and taught them to distrust and look down upon leaders who appear to be insufficiently ‘technical’.
As if there’s any content at all to the word ‘technical’ in its current usage in tech, as if it’s not entirely empty of meaning, as if the meaning of the word were even the relevant thing about the word, which it is not.
The relevant thing about the word technical is that it serves a purpose. Some words mean things. Others do things. “Technical” does a thing, and what it does is sort. It is a sorting hat used to uphold the order of things, or rather, the order of people, in the world of tech capitalism.
Whiteboarding a sorting algorithm itself serves as a sorting algorithm, and people will talk themselves blue in the face about why it’s a fair and reasonable sorting algorithm to use, even if doing it well is apparently a skill so difficult and specialized that Stanford students take a class in how to do it well, and even practicing software engineers writing code daily feel the need to study hard for some interviews.
Well gee, you wouldn’t have to study so hard something you used every day? That was relevant to your job?
*
Just because you think your particular sorting hat is not designed to uphold your privilege and justify your rejection of candidates who have not had that privilege doesn’t mean that it is not, in fact, designed, at least in part, to do exactly that.
*
I’m good at leading engineers. I don’t write code anymore, I haven’t done it in a while, it’s not a good use of my time. Do not tell me you want to hire me to lead engineers and yet put me through an interview process that will force me to excavate more than just heapsort from my memory, that I have to study for.
Let me be clear: I don’t mind studying for an interview. I mind studying for an interview when the studying is irrelevant to the actual work I would be doing, I mind make-work just to prove a point, just to make sure that I am one of the people deemed sufficiently ‘technical’ to do a job that does even not require that particular ‘technical’ skill.
If I do not think your sorting hat is relevant or fair, I will not sit down and let you put it on my head.
I do not want to work at your company that sorts people this way.
*
Of course we need some way of choosing who to hire. I’m not saying hire at random. (Actually, there’s a whole movement devoted to doing essentially exactly that: Open Hiring) I am not even going to sit here and tell you that I know the best way to interview and hire; I do not. There’s no particular best way; every interview process sucks in various ways, and they are all biased, and that includes every interview process I have myself been responsible for and every process I have gone through as a candidate. The question is not “is this a perfect, unbiased process?” The questions are: “how is this processed biased?” “how can we make it less biased?” “what might we miss with this process, who might we dismiss, how can we both work toward less bias through objective standards and at the same time respect each person as an individual, with individual strengths and weaknesses and experiences and perspectives?”
I have fucked this up. I have passed over good people. I have sometimes watched good people be passed over and not spoken up more strongly for them. I remember many occasions where in retrospect I see that I was running a broken process, that I was too harsh, that I was in a bad mood, or I was too judgmental or I didn’t question myself enough on why the ‘fit’ didn’t feel right. I remember times someone else’s nerves were being tested and I did not extend them grace. I have watched myself make excuses for people who did not deserve my excuses and refuse to excuse other people who did.
Any manager who says they aren’t haunted by some of the interviewing they’ve done and the hiring choices they’ve made is either lying or needs to seriously sit down and do some reflection. You’ve been wrong, my dude. I promise you you’ve been wrong.
*
Okay Amy, then what are some non-terrible ways to hire people (not just engineers!)?
I am not an expert in this. This post is not a be-all end-all, it’s mostly been an opportunity to talk about sorting hats because I was so very pleased at my metaphor and to talk about the word technical because I care so very much that people understand that how it’s used today is as a word that does something rather than means something.
So you should do some googling and some reading. You should read more stuff about interviewing and hiring from people who think about it more than I do, and a lot of it should be from people who are not white. But here are a few of the many things you could think about:
Stop believing that just because the big companies or the ones that have status hire in a certain way that that is the right way. Stop believing the way you’ve always hired is the right way.
Focus on assessing what the job actually requires someone to do on a daily basis, roughly speaking, while acknowledging that everybody doesn’t have to have exactly the same skill set — that there are many people who can excel in a role each doing it in a different way bringing different strength and perspectives to the table. That is: assess what you need them to do and suss out what they excel in at the same time. Give folks a chance to shine.
Lower candidates’ stress levels as much as you can. What causes stress? Lack of information, lack of control, and uncertainty. So give people information: what is the process? What should they expect? When will they hear from you? Who will they meet? What will they be asked to do? And give them control when you can: that means options. Maybe they can pick between two projects, or a take-home project and a presentation, or having their interview all in one day vs broken up into two days. Uncertainty is inherent in the process since neither the hiring manager nor the candidate know what the outcome will be — but you can reduce uncertainty through clear communication, making decisions as quickly as you can, and reminding the candidate that they are making a choice just as you are — that is, that they are also responsible for the outcome.
Ask folks if they need adjustments of any sort in the hiring process, and then provide them if you can. Don’t force them to ask for accommodations in the legal sense.
Learn about proven ways to reduce bias in hiring and implement them. This includes stuff like wording in job descriptions, anonymous assessments, appropriate training of your interviewers, and openly discussing and dealing with bias as you or members of your team exhibit or notice it. (For example, in a debrief about a woman engineering candidate “She seemed over-prepared”.)
Remember that you can have a great hiring process that still turns out to be a bad experience for an individual, for example, when someone’s out sick and an untrained or just plain doesn’t want to be there interviewer subs in. Hell, there are some phone screens that I can look back at and say I really have no idea how that person would be at the job because I was in a bad mood and I threw them off their game because of it. These days I hope I have more awareness about how I’m heading into an interview and can consciously take a few moments to center myself so I don’t do that to a stranger.
Remember that all sorting processes do involve choices. If you are doing your level best to meet the needs of the business and being as respectful and fair as you can to your candidates, then give yourself some grace as you make the best choice you can with the available information and learn what you can from it.
Enough about sorting, here are some reflections on soldiering on
Last week I went deep into my history of soldiering on at work through some truly nutballs personal challenges. Now I want to complicate that story a bit. (If you haven’t read that post, you might want to read it before reading on here).
It’s true that I’m a workaholic and it’s true that some part of me doesn’t really believe I deserve to be alive if I’m not delivering value in the world, from a capitalist production standpoint. It’s true that sometimes I’ve pushed so hard that my body or others have had to make me stand down.
It is also true that work for me has also been at times a genuine source of solace, social support, self-efficacy, and structure in my life. Work has distracted me from the screaming in my head. Work has given me a place to be every day, and people to be with. Work has provided me with some problems I could solve, some people I could help, some things I could be part of making. It’s given me money that I’ve turned around and used to make other parts of my life easier or more fun. It’s been a small opportunity to exert some positive influence in the world, through sharing my own struggles and making my own workplace a more humane place to be, and having the opportunity to have a platform to help others make their workplaces more humane too. I have loved my work, I have loved so many of the people I have worked with, I have clung to work as solid ground when my mind is buffeted by its own storms. I do not think that I would have been better off if I had not gone to work in the weeks after my friend committed suicide, or if I had not worked during the spring of 2017 when I wanted to die so badly, every day.
What I mean to say that since it’s such a very difficult thing to say stop, enough, I can’t right now, that generally speaking we should believe people when they say it, and support them in doing it.3
I do not think I would be better off if I took a mental health day every time I woke up with the screaming in my head, because that would be a lot of days, that would be ceding a lot of my life to my mental illness, and I truly do not want to cede that much of my life. Not-working will not make me happy any more than working will — happiness is not my goal, even.
What I am trying to do is to live a life that I value, that I think is worthwhile to live. I am optimizing for that rather than optimizing for the absence of pain. There is no absence of pain. I have worked many days through my pain because I wanted to be doing something I valued, despite the pain. I will continue to work many days through my pain because I want to do something I value rather than succumbing to pain.
The question becomes, then, though, what do I value? What is the work (broadly speaking) that I have to do now, in this moment in my life, in the life of the world, that I value enough to keep doing through the pain?
Tell me, what else should I have done?
Doesn't everything die at last, and too soon?
Tell me, what is it you plan to do
with your one wild and precious life?Self-Promotion Area
Got thoughts about what you read here? Smash that reply button and send them directly to me, or maybe share your thoughts on social! I love it when people share their thoughts about my writing on social, even if trolls then show up to say that I’m “amusing at best”.
I’m still offering office hours for women and non-binary folk in software development who want career or engineering management advice, sign up here. Please send this information along to people you think might benefit. I <3 doing office hours it fills me up!
If you would like me to write or speak or teach or consult about engineering leadership, mental illness, or boots, just smash that reply button and ask, or slide into my twitter DMs, or email me at amy.newell@gmail.com.
If you would like to hire me full-time to lead your engineering team or some part of it, please check back in September; I’m on hiatus from full-time employment. You can find current updates on my availability at www.amynewell.com.
If you like my instagram and would like to collaborate on creative projects (yes I model, style, and do all my own photography, and am open to collabs as a photographer, a model, or a stylist), let’s talk! More info over at www.amywearsboots.com
If you like my writing and want to pay me to write my memoir/handbook for living, tentatively titled Many Minor Apocalypses, definitely get in touch because that’s probably the thing I’m supposed to be doing right now. Just smash that reply button and ask, or slide into my twitter DMs, or email me at amy.newell@gmail.com.
If you like my writing and would like to support it financially but you can’t pay me to write a whole book, there’s always venmo @amy-isikoff-newell — yes I’ll be offering a ‘supporter’ subscription program at some point but I’m still figuring out the pricing and packaging so it’s just straight up tips for the moment.
If you like my writing and do not want to pay me for more of it, you can at least share this newsletter with people you know so that more people read my words and eventually someone will pay me.
If you are reading this and you are not already subscribed, what are you waiting for? Look at this cute purple button Substack gave me for you to smash and sign up! Go ahead! Smash the button!
P.S. If you’re wondering where the Taylor Swift reference is, check the footnotes. I love footnotes almost as much as David Foster Wallace did.
Please note that I don’t claim the book isn’t valuable. In fact I highlighted many interesting ideas in many chapters when I read it, it’s got a ton of value. I only claim that to title a book about human systems An Elegant Puzzle is to have in my view a pretty significant misunderstanding of how humans and human systems work.
This was pages and pages of scribbled numbers and pseudocode like this:
