---

Constructing and working
A massive object storage service, Amazon S3.

For over three decades, I have dedicated myself to the development of computer systems software, with a focus on operating systems, virtualization, storage solutions, network architectures, and cybersecurity strategies that ensure robust security measures are in place throughout my entire professional career. Despite my significant experience with Amazon’s Easy Storage Service (S3) over the past six years, I have been compelled to think more expansively about approaches than I ever have previously. Within a typical week, my responsibilities span the gamut from scrutinizing intricate details of disk mechanics, firmware, and storage medium properties at one end, to developing customer-focused efficiency expertise and crafting expressive APIs at the other. And beyond the technical perimeters, I’ve had the opportunity to support engineering teams’ accelerated adoption, collaborated with finance and hardware groups to establish cost-effective enterprises, and partnered with clients to craft breathtaking applications across domains such as video streaming, genomics, and generative AI.

I’d like to express my genuine astonishment at the innovative storage solutions that are rapidly emerging, which truly impress me. While building something akin to Amazon’s S3, it was crucial to address several intriguing subtleties, with key takeaways and often surprising insights garnered during my tenure within that system.

17 years in the past, on a college campus far, distant…

Which implies that he turned 17 this year. For engineers starting their professional journey, it’s disheartening to acknowledge that S3 has been a stalwart web storage service for nearly the entirety of your career in computer systems. Seventeen years ago, as I completed my PhD at Cambridge University. As part of the team responsible for developing Xen, an open-source hypervisor, I worked in the lab where this technology was first utilized by several companies, including Amazon, to build the initial public clouds. Following the success of our team in completing the Xen challenge at Cambridge University, we transitioned to establish XenSource, a startup with a distinct mission: to capitalize on the potential of Xen by marketing it as an enterprise-grade solution rather than developing a public cloud infrastructure using this technology. We might have overlooked a slight chance there. As XenSource expanded and ultimately merged with Citrix, I found myself immersed in the intricacies of building and growing an organization, including navigating complex commercial leases and tackling minor yet crucial IT infrastructure tasks, such as server room HVAC maintenance, which were not part of my graduate school curriculum.

At one point in my life, I had always aspired to become a college professor, rather than being content with my current situation. I secured various college positions and ultimately found one at UBC, which proved to be an excellent fit due to my spouse’s existing job in Vancouver, allowing us to relocate and enjoy the city. I impetuously scaled up my laboratory to 18 undergraduate students in a misguided attempt to assert myself in the academic sphere, a decision I would strongly advise novice assistant professors to avoid at all costs. Thrilled by the prospect of guiding a large team of talented graduate students, I must admit that managing such a diverse group proved an overwhelming task, leaving me feeling utterly fatigued and uncertain if my efforts were truly effective. In the midst of my time in the esteemed analysis lab, a vibrant community of collaborators emerged, allowing us to tackle complex problems and produce work that still brings me pride today. We authored numerous engaging papers on topics ranging from security and data storage to virtualization and networking, our collective efforts yielding tangible results and lasting impact.

Two years into my tenure as a professor at the University of British Columbia (UBC), I joined forces with some of my college students to launch another entrepreneurial venture. We launched Coho Information, capitalizing on the early adoption of NVMe SSDs and programmable Ethernet switches to develop a high-performance, scale-out storage solution. As we scaled Coho to 150 individuals across four international locations, I was afforded the opportunity to learn from unexpected sources, such as the load-bearing capacity of second-floor server room flooring and analytics workflows in Wall Street hedge funds – experiences that challenged my training as a computer science researcher and educator. The company’s Coho division, a highly specialized and intellectually rigorous endeavour, unfortunately proved unsustainable in its current form, prompting the decision to discontinue operations.

As I sat alone in my mostly vacant office at UBC, a sense of solitude suddenly enveloped me. As I celebrated the graduation of my last PhD student, I couldn’t help but question whether I possessed the requisite vigor to start building a research lab anew. To gain a deeper understanding of how the cloud functions and better equip myself for teaching students about this topic, I decided it was essential to obtain hands-on experience with its inner workings.

I interviewed at several cloud providers, and I must say that my conversation with Amazon’s leaders was exceptionally memorable and left me eager to join their team. That’s where I’m currently employed. Based primarily in Vancouver, I am a mechanical engineer with expertise spanning Amazon’s entire suite of storage products. Thus far, a substantial portion of my time has been dedicated to Amazon S3.

How S3 works

Upon joining Amazon in 2017, I decided to spend the majority of my initial day on the job with Seth Markle. One of the earliest engineers at Amazon was Seth, who spent six hours in a conference room with me, meticulously detailing the inner workings of S3 on a whiteboard.

It was superior. As we pored over the footage, I posed one query after another in a relentless stream, yet Seth remained unfazed, consistently anticipating my next question with ease. It was an utterly exhausting experience, yet one that unfolded with a meticulous attention to detail. Although Amazon’s Simple Storage Service (S3) is a massive system, at first glance, its architecture might seem similar to other storage solutions you’ve encountered in broad strokes.

S3 is a cloud-based object storage service that offers a simple and secure way to store and retrieve data in the form of objects, providing a robust and scalable architecture through its RESTful API. A complex system comprises a frontend fleet, a REST API, a namespace service, a storage fleet utilizing exhaustible disks, and a dedicated fleet for executing background operations seamlessly. Within an enterprise setting, we would refer to these supporting tasks as “knowledge companies,” such as data replication and tiering initiatives. Notably, upon scrutinizing the high-level block diagram of S3’s technical design, it’s striking to see that AWS has a tendency to release its organizational chart alongside its architectural blueprints. While often employed with disdain, this phrase takes on an intriguing quality in this instance. All individual components comprise the S3 group. Everyone typically has a lead developer, with various teams contributing to the project’s success. Were we to delve further into this diagram’s structure, expanding one of these containers outward to reveal its constituent components, we would uncover that each nested part operates as a distinct entity, boasting its own fleet and functioning, in essence, as an independent business unit.

All things considered, the current architecture of S3 comprises numerous microservices arranged thusly. Interactions between these groups are essentially API-level contracts, with modularity being often misunderstood, leading to inefficient and clunky team-level interactions – a cumbersome labour to rectify, yet an integral part of both software development and the construction of software teams themselves.

Two early observations

Prior to my work at Amazon, I had honed my skills developing software for analysis, working extensively with widely adopted open-source software, and crafting enterprise software and hardware solutions used in large-scale manufacturing operations within prominent corporations. Despite its flaws, the software program was nonetheless a product we conceptualized, developed, tested, and released. It was the packaged software program and the software program we delivered. Despite encountering escalations and help requests, as well as mounting bugs that required patched updates, our team persevered to successfully deliver a functioning software program. Unlike interacting with a world storage service like S3, which operates as a dynamic and autonomous entity. Every component, from the coders crafting lines of code that underpin the software pyramid, to the engineers installing fresh storage arrays in our data centers, to users fine-tuning applications for peak performance, each element forms a cohesive, perpetually adapting ecosystem. Given the assumption that S3’s customers don’t purchase software but instead look for a reliable service, they expect the experience of using that service to be consistently and predictably exceptional – free from surprises or disruptions?

The primary concern was that. This didn’t merely imply scaling enthusiasm for software development to encompass thousands of microservices that comprise Amazon S3; rather, it signified scaling to also include everyone who designs, builds, deploys, and maintains all that code. It’s ultimately one element, and you won’t really comprehend it just by treating it like a software application. It’s a software program, backed by robust hardware, and an ever-growing community of individuals, constantly innovating and evolving over time.

While the original whiteboard diagram may have provided a rough outline of the group and software program, it was equally misleading due to its complete disregard for the scale of the system. Each container holds a distinct collection of scaled-out software entities, often comprised of multiple company aggregations.

What drives the technology behind our daily interactions with digital storage? The answer lies in the realm of physics, where the principles of thermodynamics, magnetism, and electrical engineering converge to enable the seamless transfer, retrieval, and preservation of data.

As we navigate the complexities of storage systems, let us explore how these fundamental forces shape the scale at which we store and access our digital assets.

In the early days of computing, magnetic storage was the primary means by which data was preserved. The principles of magnetism allowed for the creation of rigid disks that could store vast amounts of information.

Notably enormous, Amazon’s S3 is built upon a colossal infrastructure comprising vast amounts of hard drives. Hundreds of thousands of them. When discussing Amazon S3, it’s worthwhile to take a moment to consider the characteristics of traditional hard disk drives themselves? Exhausting drives have always been exceptional, and it’s little surprise that they’ve consistently received high praise over time.

In 1968, Jacob Rabinow, a researcher at the precursor to the National Institute of Standards and Technology (NIST), designed the primary exhausting drive. Rabinow, a seasoned expert in both magnetism and mechanical engineering, had been tasked with designing a device that could facilitate magnetic storage on flat sheets of media, akin to pages in a book. To simplify the design, he borrowed inspiration from floppy disk technology used in the gaming industry, creating an array of spinning magnetic disks accessible to a single read/write head. To ensure a seamless transition, he carved a small notch on each disk that would fit onto the intended platter. The analogy by Rabinow aptly describes his experience of reading about magnetic storage without actually using it, likening it to studying a guidebook without opening it. It wasn’t until 1956 that the first commercially available hard disk emerged, with IBM’s launch of the 305 RAMAC computer system, which marked the beginning of a new era in data storage. We’ll be back at the RAMAC soon enough.

Sixty-seven years since the initial industrial push, the world now relies extensively on numerous hard drives. As global storage capacity on hard drives increases annually, their functionality is noticeably waning. As cloud storage and memory-rich devices become increasingly prevalent, our reliance on traditional hard drives is diminishing for many applications. As modern technology advances, solid-state drives (SSDs) have become the norm for both personal devices and enterprise storage solutions, replacing traditional hard disk drives with their faster performance and increased reliability. Jim Grey predicted this trajectory in 2006, foreseeing the decline of tape technology with his astute observation: “Tape is lifeless.” Disk is Tape. Flash is Disk. While the phrase “RAM Locality is King” has gained popularity in recent years for its emphasis on flash storage, the significance of disk locality is equally intriguing.

Outdated disk technologies no longer occupy the same space in the market, due to their immense physical size, sluggish speed, and vulnerability compared to modern storage solutions. Flash storage consistently outperforms traditional alternatives in most scenarios. Despite their many limitations, hard drives remain remarkable achievements in technological progress, and for the tasks they excel at, they are truly exceptional. One notable strength is value effectiveness, which is particularly pertinent in a massive system like Amazon S3, where designers must navigate unique trade-offs between individual hard disk constraints.

As I prepared to present at FAST, I asked Tim Rausch if he could help me revisit the classic aircraft flying over blades of grass hard drive example. A pioneering researcher in the field of heat-assisted magnetic recording, Tim earned his PhD from Carnegie Mellon University, contributing to the early development of HAMR drives alongside a cohort of innovative peers. With a storied career spent laboring on exhausting drives, particularly HAMR, Tim has developed a profound understanding of their intricacies. We concurred that the aircraft analogy, wherein we amplify the top of a challenging drive to the scale of a jumbo jet and discuss the relative proportions of its constituent parts, effectively illustrates the complexity and mechanical precision inherent in HDDs. Here’s our model for 2023:

Visualize a rugged hard drive head as a mighty Boeing 747 soaring effortlessly above a lush green expanse at a speed of 75 miles per hour. Is there a sufficient gap beneath an airplane’s belly to accommodate approximately two stacked pages of paper? Assuming a consistent metaphor, the revised text would be: Now, if we equate disk bits with individual blades of grass, the observed width could span roughly 4.6 blades, while each bit itself is equivalent to a single blade. Given that an aircraft flies at a significant altitude, it is extremely unlikely that it would be affected by individual blades of grass; therefore, I will assume this text is intended to convey a metaphorical or humorous message rather than a literal one.

That’s a remarkably low error rate of 1 in 10^15 requests, indicating exceptional system reliability and efficiency. In reality, blades of grass are frequently overlooked – a phenomenon we must consider in Season Three.

Here was the dawn of the digital age, when machines began to store information in a way that would change the world forever? The specifications for this particular factor include:

Now let’s assess it against the largest HDD you could buy at publication time, specifically the Western Digital Ultrastar DC HC670, boasting an astonishing 26TB capacity. Given the monumental advancements in computing technology since the dawn of the RAMAC era, it’s astonishing to consider that the original device was capable of storing data 7.2 million times more effectively than its physical counterpart, which has shrunk an incredible 5,000-fold in size? At an inflation-adjusted cost of just six billion times less per byte. Despite these advances, search occasions – the time it takes for a random query to retrieve a desired piece of information from storage – have increased by a staggering 150 times. Why? Because their mechanical nature dictates it. Despite advancements, we still rely on manual adjustments, awaiting the plate’s rotation, with mechanical components that haven’t significantly evolved at a comparable cost. If you engage in rapid, indiscriminate read-write activities on a drive, you can expect approximately 120 operations per second. The sales figures remained remarkably consistent around 2006, mirroring the performance of the S3 launch; similarly, they had maintained this level a full decade prior.

The tension between HDDs’ increasing capabilities and stagnant efficiency levels has had a profound impact on the design of S3. To effectively manage our data storage, we need to rapidly adopt larger-capacity drives and migrate our byte volumes accordingly. In today’s landscape, the largest hard drive capacities have reached an astonishing 26 terabytes. Moreover, industry projections suggest that we’re on track to reach a remarkable 200 terabytes within the next decade – an era of 200TB drives. When optimizing for random access, it’s generally recommended to spread reads and writes evenly across the entire dataset, thereby allowing for approximately one input/output operation per second for every 2 terabytes of stored data.

Although S3 doesn’t currently offer 200TB drives, I’d be happy to inform you that our team anticipates taking advantage of these larger storage capacities when they become available. The entire spectrum of drive sizes lies in between that point and this one.

Optimizing Thermal Comfort: Insights into Strategic Placement and Energy Efficiency

Considering the sheer magnitude of data storage, one of the most intriguing technical challenges I’ve faced lies in orchestrating I/O demands across an enormous array of hard drives to ensure seamless performance. In Amazon S3, we refer to this concept as heat management.

By throughput, I infer the diverse array of requests impinging on a specific disk within a finite timeframe. When we fail to manage warmth effectively, our system becomes overwhelmed by concentrating an excessive volume of requests on a solitary disk drive, thereby generating hotspots caused by limited I/O access from that single disk. To optimize our storage, we transform it into a mathematical challenge that seeks to identify the most efficient method for distributing knowledge across our disk arrays while minimizing hotspot density.

Hotspots refer to limited areas of excessive drive activity within a system, leading to performance degradation and ultimately impacting overall system efficiency as requests are bottlenecked by these overutilized drives? When experiencing a burning issue, technical problems persist, but as you accumulate requests, the customer experience falters. Unbalanced loads can cause requests that might be ready on busy drives to stall, which is exacerbated through layers of the software storage stack. The impact is amplified by dependent I/Os for metadata lookups or erasure coding, ultimately leading to a small proportion of high-latency requests – commonly referred to as “stragglers”. As specific disk hotspots emerge, they can lead to elevated tail latencies, ultimately impacting overall request latency if left unchecked.

As our S3 platform scales, we aim to distribute warmth consistently and enable individual users to leverage as much of the HDD fleet as feasible. Because the uncertainty surrounding when or how knowledge will be accessed at the time of writing makes it challenging to determine the optimal placement, we must consider this factor in our decision-making process. Prior to joining Amazon, I delved into developing predictive models and thermal management strategies for small-scale storage systems, such as local hard drives or enterprise arrays, but it proved challenging to achieve significant success? Despite its similarities in concept, the sheer magnitude and multi-tenancy of S3 result in a fundamentally distinct system.

As workload demands increase on our S3 infrastructure, the diverse requests from individual users become increasingly decoupled from one another. While some storage workloads can be characteristically bursty, it’s true that many storage workloads remain largely idle for extended periods before experiencing sudden, intense spikes in demand as data is rapidly accessed? The reality is that peak demand far surpasses what’s initially suggested. As workload volumes grow to thousands and thousands, a counterintuitive phenomenon emerges: the aggregated demands become surprisingly smoother and more predictable. Indeed, upon closer examination of this phenomenon, it becomes clear that there exists a threshold beyond which even the most substantial modifications to workload will have no discernible impact on the overall mixture’s peak performance. By aggregating demands and flattening their distribution, we must transform this relatively straightforward demand metric into a corresponding level of demand across all our disks, ensuring the temperature of each workload is harmoniously balanced.

Replication: knowledge placement and sturdiness

In storage methodologies, redundancy schemes are typically employed to safeguard data from hardware failures, yet this approach also proves effective in managing heat. They offer a load-out and provide a chance to redirect site visitors away from potential hotspots. Consider replication as a straightforward approach to encoding and preserving knowledge. Replication safeguards knowledge against disk failure by maintaining multiple identical copies on distinct, separate disks. Notwithstanding this, the software also grants you the freedom to learn from any of the disks. Once we consider replication from a capability perspective, it’s indeed costly to undertake. From a purely I/O-driven perspective, replication could actually prove to be surprisingly environmentally friendly in the context of knowledge acquisition.

We avoid the expense of replicating every piece of stored data, which is why we utilize erasure coding in conjunction with Amazon S3. Utilizing a comparable algorithm, we fragment the object into a collection of “id” shards through a process of decomposition. We subsequently create additional sets of m parity shards. As long as a sufficient number of the ok plus m complete shards remain accessible, we will be able to learn from the article. This approach enables us to mitigate capability overhead while sustaining the same number of failure survivability.

The impact of scale on a knowledge placement technique?

Redundancy schemes enable us to segment our data into additional entities, thereby allowing for easier entry and storage, which in turn enables us to avert sending requests to overwhelmed disk drives, thus preventing unnecessary heat generation. To expand the deployment of most recent objects across our entire disk infrastructure. To optimize storage and access efficiency, we strategically distribute distinct objects across numerous drives, thereby ensuring each customer’s queries span a vast array of disk drives.

Two significant benefits arise from distributing the contents of each bucket across numerous heaps and disks:

1. With minimal data storage required per user, the benefits of workload isolation are achieved through the distribution of individual workloads across multiple disks, thereby preventing hotspots and ensuring efficient performance.
2. The processing demands of specific individuals can overwhelm even the most robust storage architectures, which can be notoriously difficult and expensive to build as a standalone solution.

Viewing the accompanying graph for further clarification. Considering the need for rapid evaluation from a multitude of Lambda features, it is likely that a genomic buyer, or burst, will perform a parallel assessment to minimize any potential delays. The data demands of that burst could be met by a storage array comprising more than one million individual hard drives. That’s not an exaggeration. Today, we’re dealing with tens of thousands of consumers managing S3 buckets that span across thousands of drives. As I delved into S3’s architecture, my initial excitement gave way to humility as I grasped the sheer scope of storage construction at this scale. However, as I gained a deeper understanding of the system, I realized that it was the confluence of consumers and workloads that enabled its scalability, allowing even the most demanding applications to burst into unprecedented efficiency, rendering smaller-scale constructions impractical.

The human components

Beyond the technical expertise required to operate S3, there exist crucial human elements that transform this complex system into a cohesive entity. At Amazon, one fundamental principle is that engineers and teams should be allowed to fail quickly and safely. While we encourage them to exhibit boldness in their approach as builders, they must still prioritize delivering exceptionally robust storage solutions with unwavering dedication. To ensure effective quality control in S3, our team employs a proven method called “sturdiness evaluations.” This human-centric approach, while not bound by strict statistical models (i.e., 11/9), is equally crucial to our process.

When engineers make adjustments that could impact our structural stability, we conduct a thorough structural integrity assessment. The method draws on a concept from safety analysis, specifically the risk model. The aim is to provide an overview of the transformation, followed by a comprehensive catalog of potential risks, and subsequently elucidate on the transformative change’s inherent resistance to these hazards? By creating a risk model, you’re forced to think like an attacker and envision all the potential malicious actions they might take against your system. During a thoroughness assessment, we advocate for a proactive approach that considers “what could possibly go awry,” and indeed, we urge engineers to think outside the box when reviewing their own code. The method effectively addresses two key concerns:

1. The revised text reads: It urges authors and reviewers to think critically about the threats we need to defend against.
2. This approach enables a clear distinction between threats and countermeasures, allowing for focused conversations about each aspect individually.

While conducting robustness assessments, we employ the robustness risk model, subsequently evaluating whether our existing countermeasures and safeguards effectively mitigate potential risks. As we develop safeguards, our focus shifts to designing broad-based “guardrails”. These straightforward mechanisms provide robust protection against a wide range of potential threats. Rather than obsessing over minute details and crafting bespoke solutions for individual threats, we prefer straightforward and comprehensive approaches that provide robust protection against a wide range of potential issues.

A further example of a broad technique was demonstrated in a challenge we launched several years ago, which aimed to reimplement the lowest layer of S3’s storage stack – specifically, the component responsible for managing data at the individual disk level. The newly developed storage layer, ShardStore, underwent a significant overhaul, with the decision to rebuild it from the ground up. To ensure quality and reliability, we implemented a rigorous strategy called “lightweight formal verification” to safeguard against potential issues. To enhance the robustness of our system, our team decided to migrate the implementation to Rust, leveraging its inherent focus on memory safety and compile-time type checking to detect bugs earlier, and developed custom libraries that integrate these benefits for use in on-disk data structures. We developed a reduced prototype of ShardStore’s logic in Rust, mirroring its production counterpart, and committed it to the same repository as the actual implementation for verification purposes.

This mannequin simplified the intricacies of on-disk storage layers and hard drives, functioning instead as a concise yet executable blueprint. The scaled-down prototype, approximately 1% the size of the original system, surprisingly enabled thorough testing on a manageable scale, which would have been virtually impossible against a robust drive boasting 120 accessible IOPS. We even managed to .

By leveraging our proficiency in crafting tools and embracing cutting-edge methodologies such as property-based testing, we can effectively create test cases that verify the seamless alignment between an implementation’s behavior and its corresponding specification. What made this project truly impressive was the harmonious fusion of designing ShardStore and leveraging advanced formal verification techniques. We successfully transitioned to an “industrialized” verification approach, leveraging innovative research-based strategies for program correctness, and enabling non-experts with no PhDs in formal verification to contribute to sustaining specifications by incorporating these methods into code, thereby allowing us to employ our tools with each subsequent software decision. Verification’s incorporation as a guiding principle has fostered trust among team members, empowering them to work more efficiently and sustainably even as fresh talent is onboard.

We approach scalability in our S3 framework by considering both sturdiness evaluations and lightweight formal verification, reflecting a holistic perspective that encompasses both human and organizational aspects. The formal verification tools we developed and integrated require technical expertise. Moreover, our robustness evaluations provide a framework for the team to systematically consider robustness, thereby ensuring consistent accountability for achieving high standards of reliability. As we explore diverse manifestations of teamwork within complex systems, it’s captivating to observe how a paradigmatic shift can catalyze experimentation and innovation in how teams collaborate and operate, mirroring the creative problem-solving processes used to build and deliver their products.

Scaling myself: Mastering exhausted issue resolution hinges on cultivating a sense of “Possession”.

The last but not least instance I’d like to share with you is a personal one. As a budding entrepreneur and a college professor, I ventured into the realm of Amazon. I had led a team of over 150 graduate students in constructing an engineering crew at Coho. Throughout my tenure at the college and various startups, I relished the opportunity to combine technical expertise with creative vision, designing innovative solutions and leading high-performing teams, while continuously expanding my skill set. Having never held a role like this in terms of software, people, or businesses, I was suddenly faced with the challenge at Amazon.

As a CS professor, I found teaching the methods seminar course to graduate students to be one of my absolute favorite responsibilities. In this course, we would engage in lively discussions while studying and exploring various types of research papers covering fundamental analytical methods. One of my favourite aspects of teaching that course was that roughly half the time, we’d delve into learning the. As I eagerly anticipated numerous papers in the course, I particularly looked forward to the one where we analyzed the Dynamo paper, since it stemmed from a real-world manufacturing process that students could readily connect with and apply their knowledge to. Amazon boasted a seamless checkout experience, courtesy of Dynamo’s innovative functionality. It’s always a pleasure to discuss analytical work when people can relate it to real-world problems within their specific field of expertise.

However, engaging in debates with Dynamo was intellectually stimulating because his perspectives were often unconventional, making it possible for me to challenge my own thinking and arrive at alternative conclusions that might not have been otherwise considered.

I cherished this spot because it allowed us to concentrate on your actual manufacturing processes when Dynamo was unavailable. When a customer attempts to place an order only to discover that the desired item has already sold out? Wouldn’t you want to investigate the scene and gather evidence before deciding on a course of action? The shopper anticipated a sufficient supply.

While this instance could have potentially expanded the narrative of the Dynamo paper, it ultimately led to a satisfying conclusion. Scholars often invest considerable time and effort in formulating technical software solutions through extensive dialogue. However, someone might argue that this was far from meeting expectations. At last, rare conflicts arose, and you would resolve them by engaging concerned employees and making a compassionate decision.

In a short span of time, if the system functioned as intended, you would encourage participants to shift their focus from mere categorization to an engaged discussion about trade-offs and design considerations in software development methods, thereby fostering an appreciation for the complexity of the system that goes beyond its initial scope. It is probable that the entity in question is a comprehensive organization or corporation, potentially among its peers, still operational.

After years of working at Amazon, I’ve gained a deeper understanding that my initial perception wasn’t far off from the truth – Amazon is much more than just its software infrastructure. Upon further reflection, I’ve come to realize that there’s more to the concept than initially met my eye while reading the text. Amazon intensely focuses on the notion of “ownership,” a concept that frequently arises in discussions about accountability, such as “who is responsible for driving the success” of a product or service?

Giving attention to possession enables us to gain insight into numerous organisational structures and engineering methods employed within Amazon, specifically in S3. To achieve speed and excellence in quality, groups should consider taking ownership of their work. Developers must craft robust API contracts with distinct methods that align with their service interactions, holding themselves accountable for stability, efficiency, and availability. Moreover, they must be prepared to intervene promptly, addressing any issues that arise, including sudden bugs that impact availability as early as three in the morning. However, they must be equipped with the autonomy to replicate and refine their solution, ensuring that the issue is thoroughly resolved and does not recur in the future. True possession involves a delicate balance between responsibility and trust, as relinquishing control necessitates granting individuals or teams the autonomy to make decisions that align with their unique vision for delivering the service. By allowing individuals and groups to own and personalize software applications, they are empowered to become genuinely invested in their work and drive meaningful innovation forward. The consequences of misplacing possession are strikingly significant.

Encouraging possession in others

I’ve invested considerable time at Amazon fascinated by the imperative role that focus on possession plays in driving business success, as well as its seamless integration when collaborating with engineers and teams to develop innovative solutions. Recognizing and embracing my own sense of ownership has proven to be a highly effective tool in various professional capacities I’ve held. Here’s the improved text: As a newly minted professor at UBC, I recall grappling with the challenge of identifying promising research topics for my fledgling laboratory and mentoring my first batch of graduate students. I distinctly recall having a conversation with a colleague who was also a relatively new professor at another institution. When asked about how they identify analysis issues with their college students, they were taken aback. They’d a surprisingly annoyed response. I’m unable to make a determination on this matter whatsoever. I have five initiatives that I’d like college students to take on. I’ve written them up. They hesitate and debate, selecting a solution that ultimately falls short. I’d probably take care of these initiatives faster on my own than trying to teach them how.

What this individual accomplished was truly exceptional – they excelled in their work, achieved impressive results, and published notable research before going on to make a significant impact with a leading organization. When speaking with graduate college students who had collaborated with their mentors, I discovered that they often shared a common sentiment: “I just couldn’t become passionate about that project.” It wasn’t my concept.”

For me, that moment was a turning point in my academic career. Since then, I have made a concerted effort to genuinely engage with college students by asking thoughtful questions, actively listening, and radiating genuine excitement and enthusiasm. However, my most profitable analytical initiatives were never my own. I was fortunate enough to have played a role in their education as they were once my college students. It wasn’t until much later that I realized the crucial factor that contributed to the success of these group-based initiatives at Amazon: the students’ genuine ownership and personal stake in their projects. When college students were empowered to explore their own ideas, developing a personal stake in the project’s outcome, it became effortless to motivate them to invest time and effort into bringing those concepts to fruition. To truly resonate with their audience, they simply needed to make it more relatable and authentic by adding specific anecdotes or examples that shared a common human experience.

In truth, the Amazon role that’s been a top priority for me has been one that I’ve thoroughly examined and striven to excel in with unparalleled dedication. As a highly respected senior engineer, with a wealth of experience and unwavering conviction in my technical expertise, I possess strong opinions shaped by my profound understanding of the field. When attempting to bridge the gap between technical and non-technical minds by simplistically conveying complex ideas, we often find ourselves exhausted from the lack of synergy in our efforts to succeed together. While being emotionally detached from an idea is sometimes beneficial, it’s actually more productive and meaningful to have a personal stake in something. After collaborating with groups, I’ve found that my most profound ideas emerge from collective thinking rather than personal insight alone? As I take the initiative to thoroughly address challenges, I dedicate significantly more time to crafting well-articulated solutions rather than pitching potential options. I invest considerable time in observing how these alternatives spark creativity, effortlessly guiding individuals to pinpoint the most effective path for instilling a sense of urgency and accelerating action. Despite this, I have found it genuinely fulfilling to scale myself as an engineer, measured by driving profitability for various teams and individual engineers, supporting them through personal challenges, and celebrating their triumphs.

Closing thought

Upon arriving at Amazon, I expected to tackle the development of a complex and advanced storage software solution. As I reflected on my role, I discovered that every aspect exceeded my initial understanding by a significant margin. As I’ve come to understand, the technological scope of this system is truly massive, with its burden, creation, and maintenance being fundamentally distinct from those of smaller projects I’d worked on before, necessitating a paradigm shift in my approach and expertise. Realizing it wasn’t enough to solely focus on the software program, I understood “the system” encompassed not only its programming but also the operational dynamics of the service, the organization managing it, and the interdependent customer code working in tandem with it. The group’s inherent complexity, as an integral component of the system, presented its own scaling difficulties and equally daunting puzzles to solve, alongside opportunities for creative disruption. To ultimately achieve success in my role, I came to understand the importance of focusing on defining problems rather than proposing solutions, and identifying ways to empower robust engineering teams to take ownership of those problems.

I’m barely comfortable figuring anything out, but I genuinely feel like I’ve learned quite a bit so far. Thank you for taking the time to discuss this with me.

Associated posts (and papers)