Some thoughts on job searches

I recently had a very successful end to a particularly stressful job search and I thought it might be useful to share some of my experiences.

I don’t need to tell you all that we’re pretty up there in hype cycle land (from Gartner):

Which of course has come along with declarations that both all jobs will be AI and all jobs will be replaced by AI. But this comes at a kind of strange time when many major tech companies have been laying off large portions of their staff.

So which is it? Is this boom or bust times?

The landscape of AI jobs

Long gone are the years of data scientists being sexy, as Vicki Boykis has lamented. A proliferation of bootcamps and certificate programs has churned out waves of candidates with AI, writ large, credentials. This matched the rising demand that peaked in 2022, but since then there have been fewer jobs available in the field. And technology hasn’t exactly stood still.

Data science has become more focused on implementation, leading to a rise in the the number of postings for “machine learning engineers”. As with Data Scientists, this position is also vaguely defined, but I’ve seen it best described as a kind of “bridge” between the DS and Software Engineer worlds. It also happens to be, on average, the highest paying position in tech.

Why this change? It likely comes from many things, but what I’ve noticed is an increasing focus on how much of a ML system exists outside of the model development itself. When a system moves out of the lab, considerations like scalability, consistency and security become the determining factors for success. I always think of the Netflix prize story here - where the company paid $1 million for an algorithm it couldn’t use due to its complexity. Today, recommendation engines are usually a combination of many different systems with multiple layers of failsafes.

Some of the technologies I see in many ML Engineer and Data Science roles:

• Experience with cloud architecture (e.g. AWS, GCP)
• DS workflow development (e.g. Airflow, Kubeflow)
• General familiarity with containerization and CI/CD workflows

This goes along with the usual experience with machine learning, statistics and coding (e.g. python, SQL).

What you can (and cannot) do

One of the main inspirations for this post came from Eric Ma’s discussion about what a candidate can control in about the interview process. To the extent budget considerations or pre-existing preferences impact the process, the candidate has very little control or insight. I know at least one of the opportunities I was most excited about ended up having to be put on pause because of budget. I’m sure there were others that had similar issues which took the form of delays or rejections.

Eric puts the portion of the process that is in the candidate’s control at 30-40%. This is not a small portion and he encourages candidates to focus on building out a portfolio of interesting applications that employ sensible engineering practices. I feel this links back to what I said in the previous section; hiring managers (like Eric) are looking for engineering skills along with methods chops.

I ensured that, as a candidate, I had a set of projects to point to that demonstrate my experience dealing with the thorniness of implementation. This is because I know that when I’ve been in the hiring manager position, the focus of my questions is around the goals of the projects, the issues encountered and how the final approach was built around these obstacles.

If a candidate has mostly worked with clean datasets and a simple set of requirements (e.g. improving performance on standard benchmarks), it puts them in a particular category of practitioner, in my mind. For some positions, that’s fine. But having no engineering exposure is likely not going to make a candidate particularly appealing.

Final thoughts

Personally, I found this search one of the hardest yet. I imagine there was a lot of competition for a narrow set of opportunities, but I also imagine hiring managers struggling under a set of urgent but poorly-specified demands around generative AI. I found it helpful to outline what, specifically, I was looking for in a new role and assign each of those criteria a weight. If an opportunity was above a particular score - it’s worth pursuing. If not - well, I had the luxury of not being in a particular rush.

I want to note that my experience is different from a lot of other candidates out there. As mentioned, I have a lot of privilege in this process. So as with all of my advice - YMMV.

More to come!