SRE - Measuring and reliability

 SRE - Measuring and reliability

Goals of measuring:

• IT teams and business can understand the current status of the service. 
• Teams can analyze data and can take necessary action to improve the status
• Make better decision based on the collected metrics

Measuring everything makes your decisons data driven and one can make better decisons based on operational data that is collected.

SREs look for:

1. Measuring Reliability (make sure reliability percentage is high)
2. Measuring Toil (make sure toil percentage is decreasing)
1. Count tickets
2. Count alerts
3. Collect Statistics
3. Monitoring (What to monitor)
1. Alerts per cause will lead to a. lot of noise and spam alerts.
2. Capacity alerts (scaling helps)
3. Latency
4. Traffic
5. Errors
6. Saturation

SRE Skills and training:

What skills to train (for engineers moving to SRE or who are already working as an SRE)

• Operations and Software Engineering
• Monitoring systems
• Production Automation
• System Architecture
• Troubleshooting
• Culture of trust (shared ownership requires a culture of trust)
• Incident Management

Types of SRE Team implementations:

• Kitchen Sink (or Everything SRE)
• Scope is unbounded. 
• Good place to start SRE in an Organization
• For Orgs with few applications and need one SRE
• Since one SRE team, no coverage gaps.
• Infrastructure
• Helps makes other teams jobs easier
• Maintains shared services
• For Orgs with multiple developer teams.
• Disadvantage is that the improvements made by this team may not be directly linked with customer experience.
• Tools
• Builds tools for monitoring etc
• Does not work for customer experience
• Risk of increase of toil.
• Product Application
• Works to improve reliability of an application or product.
• Provides clear focus and link between business priorities and team effort expenditure.
• Embedded
• SREs embedded with Developers.
• Project or time bound
• Less time for mentoring
• Consulting
• Less hands on
• Not recommended until org complexities is large.
• Can help to scale and add to an existing SRE team.
• Risk is lack of sufficient context

Previous

• Introduction
• SRE Practices
• SRE concepts and automation

SRE - Concepts and Automation

 SRE - Concepts and Automation

As discussed before, implementing gradual changes reduces the cost of failure. 

SREs look at ways to implement changes to few users before it is released to a wider audience. e.g Implement feature flags to turn on/off features per customer. Using these flags, SREs can implement changes to a small set of customers thereby reducing the cost of failure. Gradually the feature flag can be set to true for all customers.

For an SRE, change is best when it is small and frequent.

SREs focus on CI/CD and canarying.

CI/CD (Continuous Integration/Continuous Delivery)

Continuous Integration refers to building, integrating and testing code within the development environment. Main goal is to enable engineers to work on code and test more often.

Continuous delivery means that one can frequently push code to production. This stage involves continuous integration, testing automation and deployment automation.

How does this help?

• Minimize code integration challenges
• Promotes higher code quality
• Easier to recover (e.g turn off feature flag)
• Time to market is shorter
• Provides more metrics to work on

Canarying

There is a phrase "Canary in a coal mine". Canaries are birds which are small and breathe faster than humans. Coal miners would take canaries with them to detect dangerous gases. If the canaries died, it would imply the miners were in danger.

To simplify:

• We have something larger that we don't want to risk (human lives)
• We can risk a smaller object (canaries)
• Smaller object detects danger 

In SRE terms:

• Canarying is deploying a change in service to a small set of users who are unaware of it.
• Evaluate the impact to the group
• Decide how to proceed based on the metrics collected
• If it was a buggy release, the impact was to a small set of users.
• Buggy release can be rolled back quickly

Note that the canary population should be small enough so it does not endanger the whole service if broken.

TOIL Automation

As per SREs "if a human operator has to touch your system during normal operations, you have a bug".

What is TOIL?

TOIL is work directly related to a service that is:

• Manual
• Repetitive
• Automatable
• Without enduring value 
• Scales linearly as the service grows

SREs look to reducing toil and scale services. This is the Engineering aspect of SREs.

SREs look to reduce TOIL by automation.

Automation provides the following values:

• Consistency (manual human actions are prone to error)
• Quick resolutions
• Time saved

Previous:

• Introduction
• SRE Practices

Next:

• SRE Measuring and Reliability

SRE Practices

Lets look at some of the SRE practices: 

Accept failure as normal with blameless postmortems

• Mistakes are inevitable when working at high velocities and development operations. Accepting failures as normal is one of the pillars of DevOps philosophy.

But how does one implement this in their organization?

Experienced SREs: 

• are comfortable with failures knowing that incidents and site outages are going to occur even if all the necessary precautions are taken care of.
• build monitoring to eliminate ambiguity 
• build observability and document processes for incident response and handoff.

After an outage it is important to know/understand why an incident occurred and take steps to ensure it doesn't repeat. 

• This is done by conducting a blameless postmortem (also called retrospective meetings or retros)
• Systematic approach is taken so that the team learns from every incident.
• Components of a blameless postmortem:
• Details of the incident and its timeline.
• The actions taken to mitigate or resolve the incident
• The incidents impact
• Its trigger or root-cause 
• The follow up actions taken to prevent it to re-occur.
• Do note that no particular individual or team is blamed for the incident and hence the name blameless postmortem.
• Intent of the postmortem:
• Ensure the root causes is understood by the entire team.
• Define or take effective action to ensure the incident doesn't occur again.
• Reduce the likelihood of outages
• Avoid multiplying complexities
• Learn from your mistakes and those of others.

Blamelssness can have an immense positive effect in your organization and create a culture of psychological safety for our teams.

In work environments with low psychological safety:

• Team members keep their ideas/concerns to themselves
• It can stifle learning and innovation

Good organizations do not focus on people when an outage or incident occurs but focuses more on the system and processes. 

Service Level Objectives (SLOs) and Error Budgets

Developers generally aren't familiar with the system on which their code runs and operations team have to deal with unstable code at times. And as we discussed before, due to agile methodology, the code is churned out at high velocity and that makes it tougher for the operations team who prefer slow progress to ensure consistency and stability.

As we read before, SRE tries to break down the silos between development teams and Operations and promotes shared ownership. These fundamentals helps teams maintain reliability of their services.

What is an Error budget?

We define service reliability as a fraction of Good Interactions v/s Total Interactions.

[ (Good Interactions) / (Total Interactions) ] = Fraction of real users who experience a service that is working and available.

Error budget is the amount of unreliability that you are willing to tolerate 

If we strive for 100% reliability, we will slow down new service features which will impact business. This is where error budget comes in. As long as our reliability is > the error budget, new releases can be pushed out.

Essentially, error budget is an agreement that helps prioritize engineering work.

It helps to find the right balance between innovation and reliability.

SLOs

The performance of our system relative to SLOs should guide our business decisions.

SLOs are precise numerical targets for system reliability.

How to define SLOs?

Service level Indicator (SLI) tells you at any moment in time how well your service is doing. Its a quantifiable measure of service reliability.

SLI is expressed as a ratio of:

Good events divided by the number of valid events multiplied by 100%. This gives us a range between 0% and 100%. 0% indicates nothing works and 100% indicates nothing is broken.

SLI should map to users expectations like response time, latency and quality.

An SLO is your target for SLI aggregated over time.

SLO should generally be just short of 100% like 99.999% etc.

Anything below your SLO will result in an unhappy customer whose expectations for a reliable service is not being met. 

Previous: Introduction

Next: SRE concepts and automation 

Site Reliability Engineering (SRE)

 What is Site Reliability Engineering (SRE)? 

Site Reliability Engineering, or SRE, is both a practice and a job role, where engineering directly supports software operations. At Google, SRE is described by its founder as "what happens when you ask a software engineer to design an operations team."

SRE involves Ops work (tickets, on call and manual tasks) + development (internal tools, SRE tools and building automatic systems).

Need to track metric for the time spent on Ops work and Development work.

Over time, the Ops percentage should decrease.

Development and Operations teams have conflicting priorities and continue to work in silos.

Standardising practices to balance velocity (of development features) with the risk to reliability - these practices combined with a culture to support, forms the core of SRE.

Since SRE principles are closely aligned with DevOps practices, lets understand what is DevOps and why we need DevOps.

DevOps:

• An IT team consists of Developers and Operations.
• Developers are responsible for writing code for the system.
• Operations team is responsible to ensure those systems operate reliably.
• Ideally both the teams strive to make the end consumer/customer happy with the product.
• Developers follow an agile model to deploy and push code as quickly as possible. So, developers want to work faster.
• Operations team prefer to work slower since they have to keep the system reliable. They focus on reliability and consistency.
• Overtime, this sometimes resulted in Operations team inheriting code with less understanding of how it would run in production. 
• This would eventually hurt business needs since the priorities were not aligned.
• How to close the gap between Developers and Operations? Enter "DevOps"
• Reduce Organizational silos
• Implement gradual change (reduce cost of failure)
• Automate manual work
• Measure everything
• Remember that DevOps is a philosophy (its not a development methodology or technology). It highlights critical ways for IT teams to operate, it doesn't define on how an organization should implement practices to become successful. That's where SRE comes in.
• SREs are engineers who are responsible for Operations.
• SRE is a practice and a role.
• SREs share ownership of production with Developers. This helps to reduce Organizational silos.
• Together, SREs and Developers define Service Level Objectives (SLOs).
• They together share responsibility to determine reliability and prioritize work.
• This promotes shared vision and knowledge.
• SREs aim to reduce cost of failure by rolling out changes to a small percentage of users before rolling it world wide. This can be achieved by implementing code changes like feature flags which can be turned on/off for a small percentage of users and/or removed completely to implement worldwide. This also promotes more design thinking and proto-typing.
• SREs also focus on automation to reduce the amount of manual repetitive work.
• SREs measure everything related to automation, reliability and the health of their systems. This leads to transparency and data driven decision making.
• Do remember the GOAL of the SRE is to serve the business and the end user.

SRE Mission:

To protect, provide for and progress software and systems with consistent focus on availability, latency, performance and capacity.

Next: SRE Practices

ElasticSearch (Introduction)

Q. What is ElasticSearch?

ElasticSearch is an open source analytics and full text search engine.

Often used to enable search functionality for applications. One can built complex search functionality using Elastic Search. Can also be used to aggregate data and analyze results.

Data is stored as documents (JSON Object) in ElasticSearch.

e.g A document in ElasticSearch corresponds to a row in a relational database (for better understanding).

This document contains fields which corresponds to columns in a relational database row.

e.g

{

    "Name": "James Bond",

    "DOB": "01-01-1999",

    "EmployeeID":"10203"

}

Data is stored in Nodes. There are multiple nodes and each node stores a part of the data.

Node is an instance of ElasticSearch. A machine can have many nodes. To store large datasets, we use multiple machines, each having multiple nodes. Best to have a node per machine.

Set of nodes is called a Cluster.

Every document in ElasticSearch is stored within an index (all documents need to be indexed).

e.g A document containing a person's details like name, country, DOB etc may be stored in an index names "people_index". 

Indexes are broken into shards which resides on nodes. Based on the capacity of node(s), shards are created and they reside on the nodes. Sharding helps in improving performance since queries can now be run in parallel on multiple shards.

By default, an index has one shard and we can configure to increase/decrease the # of shards using split/shrink API.

But what if a node/shard fails? Will we lose data? 

If there is no copy of the data, we will lose data. Replication is enabled by default in ElasticSearch. Default value is 1.

Replication is configured at the index level. We can choose the # of replicas we need while creating an index (default is 1 as mentioned above). Copies of shards are created. These copies are known as "replicas" or "replica shards".

The main shard is known as the "primary shard" and a replication group (primary and replicas) is created.

As obvious the replicas will be stored in different nodes to avoid a single point of failure. 

But what if I have only one node?

Replication in one node makes no sense since failure of node will lead to loss of data. Need to have a minimum of 2 nodes for replication to be effective.

How is replication distributed?

1 node:

• Replication ineffective.

2 nodes:

• With the default configuration of 1, the replicas will be placed in the other node (other than the primary)
• With a configuration > 1, all replicas will be placed in the other node (other than the primary)

> 2 nodes:

• With a configuration > 1, all replicas will be distributed across multiple nodes to improve availability and fault tolerance.

What does replication achieve?

• High availability (replica available in case of downtime)
• Throughput (since search queries can be routed to replica as well and hence improve performance)

Can we take backups of the data?

Snapshots can be taken at any point in time (at index level or cluster level). Can be used to restore data as well.

 

Cassandra (High Level details)

What is Cassandra?

• Open source
• Distributed Database
• NoSQL Database
• Can add lots of nodes in a cluster hence highly scalable
• High availability since the nodes can be spread across DC's.
• Fault tolerant, durable with no single point of failure.
• Note that in Cassandra distributed architecture, there are multiple nodes. These nodes can be replicated across multiple data centers to avoid a single point of failure.

All the nodes have the same functionality and no one node is a master node. We do not have a master slave node architecture. All nodes have the same functionality.

If all nodes have the same functionality, how do the nodes know about the other nodes in the cluster?
Ans. Snitch.

What is a Snitch?

As per official documentation: https://cassandra.apache.org/doc/latest/operating/snitch.html?highlight=snitch

In Cassandra, a snitch has two functions:

1. Teaches Cassandra enough about your network topology to route requests efficiently
2. Allows Cassandra to spread replicas around your cluster to avoid correlated failures. It does this by grouping machines into “data centers” and “racks.” Cassandra will do its best not to have more than one replica on the same “rack” (which may not actually be a physical location).

There are various snitch classes. Read more about them in the above link.
SimpleSnitch is suitable for only a single DC.

In PropertyFileSnitch, one will put the IP address of all the nodes in a cluster and also specify the rack and DC. Data is entered in the cassandra-topology.properties file.
This is fine if we had lesser # of nodes. But if we had 1000 nodes, it would be very tedious to keep creating and modifying entries for the nodes.

What is Gossip?
Gossip is how the nodes in a cluster communicate with each other.
Every second, a node sends information about itself and the other nodes information (that it has) to at-least 3 other nodes.
Thus, as mentioned before, there is no master slave configuration. Eventually all nodes have information of all the other nodes and they keep sending/sharing information every 1 sec.
Do note that this is internal communication between nodes in a cluster.

Data is distributed across nodes (so rows in a table can be in different nodes).

But if the rows in a table are in different nodes and if a node goes down, would we lose data?
We have an option to replicate data by setting a replication factor. If the replication factor is 1, then data is not replicated and you may lose data. But if the replication factor is increased to 2 or 3 etc, the data is replicated across nodes and we have high availability of data even if nodes go down.

MapReduce example

 Q. Given a data set of movie ID, ratings and user ID, find the # of ratings + popular movies.

MapReduce code:

Use MR Job and have the following script:

class MovieRatings(MRJob):

    def steps(self):

        return [

            MRStep(mapper=self.mapper_get_ratings,

                   reducer=self.reducer_count_ratings)

        ]

    def mapper_get_ratings(self, _, line):

        (userID, movieID, rating, timestamp) = line.split('\t')

        yield rating, 1

Here we are splitting the data (delimiter is tab) into a tuple.

Return a 1 for every rating.

    def reducer_count_ratings(self, key, values):

        yield key, sum(values)

For the reducer we use a sum with the keys being the movie ratings.

if __name__ == '__main__':

    MovieRatings.run()

The main run part.

Execute the same on local or on Hadoop cluster to get a break down.