Database Sharding

 What is Database Sharding?

Any application or website that sees significant growth will need to scale to take care of the increase in traffic. Some organisation's choose to scale their databases dynamically.

Sharding is a database architecture pattern related to horizontal partitioning.

• Separating a table's rows into multiple different tables (which are known as partitions)
• Each partition will have the same table and schema but the rows would be different
• Data in each of the rows is different
• Data is broken up into smaller sets known as shards.
• These shards are then pushed into multiple distributed database nodes.
• All the data collectively in all these shards represent the entire dataset.
• This is also known as scaling out.
• We can add more machines as the load increases.
• Leads to faster processing.

Vertical partitioning

• Columns are separated and put into new tables
• Data is unique but there is a common column in all tables to align the data
• Known as scaling up
• Upgrade hardware, CPU and memory.

Benefits

1. As the table grows, queries slow down.
2. With horizontal sharding, queries go through fewer rows and hence are faster.
3. If there is an outage, only a single shard may get impacted and while it may impact the application, the overall impact would be less as compared to the whole DB being down.

Drawbacks

1. Complex
2. Very difficult to return to unsharded/original form
3. Unbalanced shards
1. In case the sharding was done by name e.g. A-H, I-P and S-Z
2. If the names are more in A-H and less in S-Z, it would lead to unbalanced shards
3. Application queries to A-H will slow down
4. Thus, A-H shard is known as database hotspot.
4. Not supported by all DBs like PostgreSQL

Sharding Architectures

Some of the Architectures that are important for all to be aware of:

Range Based sharding

• Sharding data based on a range (obvious by the name)
• Relatively simple to implement
• Every shard has unique set of data
• Read and write from the shard where the range falls in.
• As expected, this can lead to unbalanced data as well.

Directory Based sharding

• Create a lookup table that will contain a mapping of a Key and shard ID.
• Do note that the key here should have low cardinality (less # of possible values)
• Application needs to have another query to get shard ID from the lookup table.
• Can the lookup table become a point of failure? Yes, it can.

Key Based Sharding

• Also known as Hash Based Sharding
• Creates a hash value from a key and that hash value represents the Shard ID
• The shard key value should be static and not change with time
• If data grows and more shards need to be added, effort would be needed to add corresponding hash and do remapping of existing values.

Introduction to Image Generation

 

Introduction to Image Generation

Note: The below is from my learning from https://www.cloudskillsboost.google/course_templates/541 (Introduction to Image Generation).

While many approaches have been implemented for image generation, the following model families have looked more promising over time:

• Variational auto encoders

• Encode images to a compressed size and then decode back to the original size while learning the distribution of the data itself.

• Generative adversarial models (GANs)

• Pit two neural networks against each other.

• One neural network, the generator creates images.

• The other neural network, the discriminator predicts, if the image is real or fake.

• Over time, the discriminator gets better and better at distinguishing between real and fake, and the generator gets better and better at creating real looking fakes.

• Auto regressive models

• Generate images by treating an image as a sequence of pixels

Lets discuss Diffusion Models.

Unconditioned diffusion models, where models have no additional input or instruction, can be trained from images of a specific thing, such as faces and it will learn to generate new images of that thing.

Conditioned diffusion models like text to image where we can generate 

• An image from a text prompt 
• e.g. Batman with a cat face
• Image-in painting
• e.g. Remove the apple from the image
• Text guided image to image where we can remove or add things, we can edit the image itself.
• e.g. Horse in space with glowing headbands

How do Diffusion models work?

The idea is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. Really, this is going to be adding noise iteratively to an image. We then learn a reverse diffusion process that restores structure in the data, yielding a highly flexible and tractable generative model of the data. 

In other words, we can add noise to an image iteratively, and we can then train a model that learns how to de-noise an image, thus generating novel images.

Process:

1. We start with a large dataset of images.
2. For an image we add some noise in an iterative way.
3. So if we do this over and over, iteratively adding more noise, we need to think about how many times do we perform that operation.
4. By the end of it, we should reach a phase of pure noise.
5. By this point, all structure in the initial image is completely gone
6. The challenging part is how do we go from a noisy image to a relatively less noisy image?
7. This process is called the "reverse diffusion process"
8. Do note, that every step that we add noise, we also learn the reverse diffusion process. That is, we train a machine learning model that takes in as input the noisy image and predicts the noise that's been added to it.
9. Over time, after seeing enough examples, this model gets very very good at removing noise from images.
10. How do we generate images with it?
1. We can just start with pure, absolute noise and send that noise through our model that is trained. 
2. We then take the output, the predicted noise and subtract it from the initial noise. 
3. And if we do that over and over and over again, we end up with a generated image.

Generative AI Studio

What is Generative AI?

It is a type of artificial intelligence that generates content for you.

Note: The below is from my learning from https://www.cloudskillsboost.google/course_templates/556 (Introduction to Generative AI Studio).

What type of content?

Any type of content like text, images etc.

How does it generate this content?

It generates this content from so many existing content already available. The process of learning from these existing content is called training. Through this it creates a foundational model. e.g of foundational model is LLM (Large language model). The foundational model can be used to generate content and and tasks such as content extraction.

One can add new data sets to the above foundational model for a specific task and thus creating a new model.

How can I create a new model from a foundational mode? Is it easy?

Using Google Cloud tool called Vertex AI. Vertex AI is an end-to-end ML development platform on Google Cloud that helps you build, deploy, and manage machine learning models.

What is Generative AI Studio?

Generative AI Studio allows a user to quickly prototype and customize generative AI models with no code or low code. Generative AI Studio supports language, vision, and speech.

Language - Tune Language models

Vision - Generate images based on prompts

Speech - Generate text from speech or vice versa.

Best practices for prompt design

What is a prompt? 

A prompt is your text input that you pass to the model

Best practices for prompt design:

• Be concise
• Be specific and well-defined
• Ask one task at a time
• Ask to classify instead of generating (e.g. "is X better to learn?" instead of "what is better to learn"?)
• Include examples (Adding examples tends to yield better results)

There are a few model parameters once can experiment with to try to improve the quality of responses:

1. Temperature
2. Top P 
3. Top K

Temperature is a number used to tune the degree of randomness.

Low temperature means to select the words that are highly possible and more predictable.

High temperature implies more random, unexpected and some may say "creative" responses.

Top K lets the model randomly return a word from the top K number of words in terms of possibility. For example, top 2 means you get a random word from the top 2 possible words.

Top P allows the model to randomly return a word from the top P probability of words.

Conversations

Before we try to create conversations, we need to specify the conversation context.

Context instructs how the model should respond. 

We can add words that the conversation can or cannot use. Same goes for the topic to focus on or avoid.

Tune a Language Model

Prompt design allows for fast experimentation and customization.

However, we have to understand that changes in the prompt wordings can impact the model significantly.  Hence, we look to tune the model.

 

Generative AI - Introduction to Large Language Models (LLM)

  Note: The below is from my learning from https://www.cloudskillsboost.google/course_templates/536 (Introduction to Generative AI).

Large Language Models (LLMs) are a subset of Deep Learning.

They can be pre-trained and then fine tuned for specific purposes.

What do we mean by pre-trained and fine tuned?

Assume in our everyday life we train dogs basic commands such as sit/stand/walk etc. This is basic training. But if need to train a dog to be a police dog, we need more fine training apart from the basic ones. This is the difference between pre-trained and fine tuned.

Similar idea applies to LLMs.

LLMs are trained to solve common language problems like document summarization, text classification, text generation etc.

They can then be tailored to solve specific problems in the field of finance, retail etc.

Benefits of using LLMs:

• Single model can be used for various purposes
• Fine tuning a LLM requires minimum field data
• Performance grows continuously as more data and parameters are added.

Note: the above image has been taken from the internet.

If we think of an example of getting answers for questions, the question answering model is able to search for an answer from a large document. Depending on the model used, the answer will either be extracted from a document or a new answer will be generated.

e.g. see the answer from Google's AI chatbot BARD which can be access by going to bard.google.com

In the above example, we gave the prompt (question) to get the desired answer.

Prompts involve instructions and context passed to the language model to achieve the desired result.

There are three main LLMs (and each need prompting in a different way).

• Generic (or Raw) Language Model
• Predict the next word
• Instruction Tuned
• Predict a response
• Dialog Tuned
• Have a dialog by predicting the next response

Generative AI (Introduction)

 Note: The below is from my learning from https://www.cloudskillsboost.google/course_templates/536 (Introduction to Generative AI).

What is Generative AI?

Generative AI is a type of artificial Intelligence technology that can produce various types of content like text, images, speech, audio etc.

What is AI?

AI has to do with the theory and methods to build machines that think and act like humans.

Machine learning (ML), which is a subfield of AI, is a program or system that trains a model from input data. Trained model can make useful predictions. ML gives the computers the ability to learn without programming. 

Two of the most common classes of machine learning models are unsupervised and supervised (Labeled data) ML models.

What problem can a supervised ML model solve?

Supervised ML model implies the model has labelled data. If we have historical labelled data of bill amount and how much different people tipped based on order type, the model learns from past examples to predict future values.

If you want to look at tenure and income and then group or cluster employees to see whether someone is on the fast track, would be a case for Unsupervised ML model. Unsupervised problems are all about discovery, about looking at the raw data and seeing if it naturally falls into groups.

What is Deep Learning?

Deep learning is a type of machine learning that uses artificial neural networks, allowing them to process more complex patterns than machine learning. Deep learning models typically have many layers of neurons, which allows them to learn more complex patterns than traditional machine learning models.

Gen AI is a subset of deep learning. Large language models are also a subset of deep learning.

Deep learning models can be divided into two types, generative and discriminative.

A discriminative model is a type of model that is used to classify or predict labels for data points. Discriminative models are typically trained on a data set of labeled data points. Once trained, it can be used to predict the label for new data points.

A generative model generates new data instances based on a learned probability distribution of existing data. Thus generative models generate new content.

A discriminative model, given input dog classifies it as a dog and not a cat.

The generative model learns and predicts the conditional probability that this is a dog and can then generate a picture of a dog.

So generative models can generate new data instances while discriminative models discriminate between different kinds of data instances. 

In generative AI, we as users can generate our own content, whether it be text, images, audio, video etc, for example models like PaLM or Pathways Language Model (PaLM API lets you test and experiment with Google's large language models and gen AI tools) ingest very, very large data from the multiple sources across the internet and build foundation language models we can use simply by asking a question using a prompt.

Given the above, lets now formally define Generative AI.

• Generative AI is a type of artificial intelligence that creates new content based on what it has learned from existing content.
• The process of learning from existing content is called training and results in the creation of a statistical model when given a prompt.
• AI uses the model to predict what an expected response might be and this generates new content.

Generative AI Studio helps developers create and deploy Generative AI models by providing a variety of tools and resources that make it easy to get started.

Generative AI App Builder lets you create gen AI apps without having to write any code. It has a drag and drop interface.

Make a talking Avataar in 5 min (using AI)

Yes, you read it right!

Make an AI talking Avataar in 5 min.

How?

Step 1 - Go to https://playgroundai.com/ and make your own avataar. (You need to sign in). Alternatively you can skip this and go to 2 and 3 (in 3 you can choose from existing avataars)

Step 2 - Go to https://beta.elevenlabs.io/ and convert text to Audio

Step 3 - Go to https://studio.d-id.com/ and create your own video

• Take the Avataar from 1 (or already existing one in this site)
• Upload the audio from 2
• Click Generate video.

Example below:

Kafka - Basics

 Hello All!!!

Starting a new series on learning Kafka (from basics)!!!

What is Kafka?

Apache Kafka allows you to decouple your data streams and your systems. So your source system will feed data into Kafka and target system will take data from Kafka.

Why Apache Kafka?

• Open Source
• Resilient Architecture
• Distributed
• Fault Tolerant
• Horizontally scalable
• High performance with low latency

Why Apache Kafka?