What does a Test Engineer do?
The Test Engineering team (TestEng) is a new and exciting team embedded within Engineering. We are accomplished engineers on a quest for higher quality. Collectively, we have built complex and impressive software systems at startups and big tech, written test frameworks and program analysis tools; and even applied formal verification and automated reasoning. We have done all of those things and more while obsessing about correctness and performance, yet never quite achieving nirvana.
In short, TestEng is responsible for driving the quality of CockroachDB. Our mission is software validation through automated testing and program analysis.
At Cockroach Labs, quality is everyone’s responsibility. Our engineers are obsessed with (functional) correctness, performance, and user experience. Collectively, they own subsystems which make up CockroachDB, including their quality. TestEng owns correctness and performance validation methodologies as well as the test infrastructure. Each engineer on TestEng is dedicated to improving quality through automation, exploration, and performance scaling.
How does TestEng fit within the rest of the organization to drive results?
TestEng collaborates with all other code owners to establish quality gates, test strategies, and best practices. Partnering with other departments is essential to the success of our mission. We work with:
- Product to align planning and anticipate new focus areas.
- Customer Success to better understand customer workloads and help us drive test strategies.
- Marketing to share our knowledge externally; to help disseminate original content that is intuitive.
- HR and Recruiting to help grow and support our amazing team.
How Stan got into Test Engineering
My fascination with computing started at an early age when I was an internationally ranked chess player, competing in tournaments. Witnessing an early generation of chess computers beat top players and win tournaments blew my mind. I continued playing chess competitively and studying mathematics up until college at which point I became obsessed with building software.
After graduating with a CS degree, I worked in the industry for several years, building a web-based trading system and customizable GUI widgets. It was my first experience witnessing how a lack of strong engineering culture can quickly erode software quality and even lead to large (financial) losses; e.g., Knight Capital (years later) lost $460 million in 45 minutes due to a series of bugs in their high-frequency trading system.
I pursued a Ph.D. in CS to study how to reason about software correctness. Software verification through automated reasoning is a fascinating field. My dissertation was on verifying functional correctness of Java code annotated with specs. (i.e., pre/post-conditions) via decision procedures made sound by a formal logic. After my defense, I felt more prepared to tackle large software projects, so I went back to the industry.
Startups have been compared to competitive sports and graduate schools. For me, the transition from academia felt very natural. The aspect of bootstrapping an idea from virtually zero to a fully-functional product has many similarities to research. The stark difference is the speed of execution. Over the last decade I’ve built complex distributed systems as well as engineering teams and organizations. Stepping into a leadership role has enabled me to collaborate cross-functionally, helping shape not only the software but also how it’s built and evolved, i.e., its organization.
I knew of CockroachDB’s accomplishments and strong engineering culture, mostly through high-quality engineering blog posts and presentations. When I learned about an open director role to help shape Test Engineering at Cockroach Labs, I immediately saw it as a unique opportunity. While my formal training is in verification, through my career in software engineering, I’ve honed my skills to apply pragmatic approaches such as testing techniques and scalable program analysis. CockroachDB is a highly complex software system. It’s built by some of the brightest and most talented engineers in the field. The prospect of working together to ensure the database continues to scale both in performance and reliability is what led me to my current role. To this day, it is the most exciting aspect of my job.
How does Cockroach Labs help you grow?
Prior to joining, I already knew of the strong engineering culture. But there is a lot more to this culture than can be experienced by reading a blog post. The company culture of full transparency, mutual respect and ownership is extremely unique. This open, collaborative culture truly amazes me on a daily basis; I have not seen such a level of collaboration across all departments at any of the previous places I worked. At the end of the day, this is what drives growth and innovation.
I felt empowered to be afforded the trust and respect to, first, survey all the open problems in TestEng, and only then focus on solving those which matter. The leadership team deeply understands what it takes to nurture creative minds in order to produce a great software product. Everyone I’ve met across dozens of teams has been supportive, open-minded, always willing to help. Even the most opinionated engineers (including myself) don’t waste a breath on arguments of subjective matter; arguments are invariably resolved with data and respect for everyone’s opinion. All of these things have helped me grow in my role and continue to motivate me to do my best.
Why should someone join the Test Engineering team?
The Test Engineering organization was born recently out of the quality initiative across engineering to form dedicated teams of engineers focused on reliability and performance. The impact to improve quality is huge; it can be felt across all departments. By joining in the early days, there is ample opportunity to shape not only TestEng but to grow in your own career path. Our current structure is aligned with three focus areas–infrastructure, exploratory testing, and performance. Each focus area has a diverse set of challenging problems to solve.
There is a large misconception about Test Engineering, in part because its role isn’t always elevated to the ranks of software engineering. Rest assured, that’s not how we see it at Cockroach Labs. Without Test Engineering, CockroachDB would simply not be the product that it is today. Currently, each PR runs ~80K unit, integration, acceptance, and metamorphic tests as well as microbenchmarks; nightly runs include ~9K SQL logic tests and ~400 end-to-end tests and performance benchmarks. All this infrastructure was built by engineers, and every engineering team owns their tests. At our scale, Test Engineering is inextricable from software engineering. By being a member of TestEng you are a software engineer working with other engineers to help scale the infrastructure and develop new test and validation strategies.
