https://www.kai-waehner.de/blog/2023/09/15/how-to-build-a-real-time-advertising-platform-with-apache-kafka-and-flink/
Wednesday, 17 July 2024
Intergrating Kafka with your API
Integrating Kafka with your application API without making direct changes to the API itself can be achieved by using various techniques and tools. This approach typically involves setting up an intermediary service or using a middleware that intercepts API requests and forwards the necessary data to Kafka. Here are a few strategies to achieve this:
Example - You want to know what are the ad request you are making . Example want to know which songs users are playing on spotify
Strategy 1: Sidecar Pattern
The sidecar pattern involves running a companion service alongside your main application that handles Kafka integration. This can be particularly useful in containerized environments like Kubernetes.
Steps to Implement Sidecar Pattern
Create a Sidecar Service:
- Develop a separate service that consumes API logs or data and sends it to Kafka.
Deploy the Sidecar:
- Deploy the sidecar service alongside your main application.
Example Implementation
Sidecar Service (Python):
pythonimport requests
from confluent_kafka import Producer
import json
# Kafka Producer configuration
conf = {
'bootstrap.servers': 'localhost:9092',
'client.id': 'sidecar-producer'
}
producer = Producer(**conf)
def send_to_kafka(topic, message):
producer.produce(topic, json.dumps(message).encode('utf-8'))
producer.flush()
print(f"Message sent to Kafka topic {topic}")
def intercept_and_forward(url, headers, params=None):
response = requests.get(url, headers=headers, params=params)
data = response.json()
send_to_kafka('your_topic', data)
return data
# Example usage
if __name__ == "__main__":
url = "http://example.com/api/data"
headers = {"Authorization": "Bearer token"}
data = intercept_and_forward(url, headers)
print(data)
Kubernetes Deployment Example
Kubernetes Pod Definition:
yamlapiVersion: v1
kind: Pod
metadata:
name: api-with-sidecar
spec:
containers:
- name: main-app
image: your-main-app-image
ports:
- containerPort: 80
- name: sidecar
image: your-sidecar-image
env:
- name: KAFKA_BOOTSTRAP_SERVERS
value: "localhost:9092"
Strategy 2: API Gateway
Using an API gateway allows you to manage, secure, and monitor API traffic, as well as intercept and forward data to Kafka without modifying the backend API.
Steps to Implement API Gateway Integration
Set Up API Gateway:
- Use an API gateway solution like Kong, Apigee, or Amazon API Gateway.
Configure Plugins or Middleware:
- Use plugins or middleware to intercept requests and forward data to Kafka.
Example with Kong Gateway
Install Kong and Kafka Plugin:
- Follow the Kong installation guide.
- Install the Kafka logging plugin for Kong.
Configure Kafka Plugin:
yamlservices:
- name: example-service
url: http://example.com
routes:
- name: example-route
paths:
- /api
plugins:
- name: kafka-log
config:
bootstrap_servers: "localhost:9092"
topic: "your_topic"
producer_config:
acks: "all"
- Apply Configuration:
bash# Assuming you have a Docker-based setup
docker-compose up -d
Strategy 3: Log-Based Integration
You can also use a log-based approach where application logs are collected and processed to send data to Kafka.
Steps to Implement Log-Based Integration
Set Up Log Forwarder:
- Use a log forwarder like Filebeat, Fluentd, or Logstash.
Configure Log Forwarder to Send Data to Kafka:
Filebeat Configuration Example:
yamlfilebeat.inputs:
- type: log
paths:
- /var/log/myapp/*.log
output.kafka:
hosts: ["localhost:9092"]
topic: "your_topic"
codec.json:
pretty: true
- Run Log Forwarder:
bashfilebeat -e -c filebeat.yml
Summary
These strategies allow you to integrate Kafka with your application API without making direct changes to the API code:
- Sidecar Pattern: Run a companion service alongside your application.
- API Gateway: Use an API gateway to intercept and forward data.
- Log-Based Integration: Use log forwarders to process and send logs to Kafka.
Choose the strategy that best fits your architecture and operational requirements.
Monday, 21 August 2023
Interview Preparation Topics - Manager
https://igotanoffer.com/blogs/tech/facebook-engineering-manager-interview
- People Management
- Project Management
- Culture Fit
- System Design
- Coding Interview
Tuesday, 6 June 2023
Leadership and Growth Mindset
https://sites.google.com/deliveroo.com/tech-leadership/engineering-manager-prep/leadership-growth-mindset
Behaviour Interview Prep - Manager
https://sites.google.com/deliveroo.com/tech-leadership/engineering-manager-prep/behavioural
Resolving Conflicts
Principle of conflict resolution
- everyone is right in their eyes
- avoid false dichotomies (Black or white)
- people generally present a position , but rarely
- that is the issue, the person resolving issue need to understand the real issue by using listening skills
- Dont dig in, that is dont try to explain why you think you are right, i am right because it leads to escalations, look for sound of argument
- dont use exagerated statement - you always
- seek unity of purpose
- focus on where we are going
- its not person but process -- how can we make sure
- such conflicts do not appear in future example RACI Framework
- look for win wins mutual benefits
- long term -- develop a culture where people listen to each other
When it comes to creating resolving conflicts a few of these communication skills come in Handy.
- Empathy , when you empathise with people it allows them to move over things , otherwise you will hear people saying the same things in many conversations
- Acknowlegement - When listening to someone acknowlegement like nodding , saying yes motivates the speaker, absense of these kinds of motivation leads to demotivation
- Reflecting or Rephrasing -- Listening is more than Hearing, we need to understand what the other person is feeling and rephrase it. Reflecting is showing the other person mirror. This is what i think you said
- Brainstorming - Is allowing people to come up with ideas without Judging. Note that a idea might not be good but it might lead someone to come up with better idea on those lines
Framewok for conflict resolution
- Find Agreement
- Clarify disagreement
- express empathy
- seek alternatives
- Agree on criteria for solution
- Combine and create
- Reach consensus
Project Walkthrough for Interview in STAR Format
Overview of Project
Company had roadmap to create a Financial Data lake. Software engineering team Was tasked with creating infrastructure for this data lake
- that is the actual data lake to store the data
- ETL tool which would connect to this datalake on which jobs needs to be created
- Tools to query this data lake
Data Engineering team which I was leading was tasked with Migrating data pipeline to datalake, using tools developed by Software engineering team. Wireframes on how this tool would look like was shared with team and data Engineering team waited for Software engineering tool to complete
Their work so we can start working on pipeline migration.
STAR Format follows
Situation (Challenges you faced)
- 6 months prior to the data lake project completion date, My senior manager discusses the project and asks me to get started. At this time his understanding is the same that a tool would be developed.
- After discussing with Software engineering team, we understand that they have hit roadblocks and entire effort is delayed, They have not even started the tool devleopment and have not completed the Datalake infrastructure development
- I escalated it to senior management clarifying that Data Engineering project cannot start until ETL tool development is completed.
- After multiple hard discussions with Senior management , Software engineering team accepted that they have completed missed timeline and there is no time available for Data engineering to migrate.
- Software engineering team will not work on creation of tool, instead they will only work on Creation of Data lake API which will be used by Data engineering team in their solution
- We were to run the development efforts in Parallel with Data lake API Development
- Data engineering will work on migration using AWS platform and Data lake API
Tasks (Decision you took )
- Were tasked with creating a approach for migration, with aim of completing it in next 6 months.
Action (How you worked with your team and cross functional peers)
- Discussed with team came up with few approaches how we could do the project
- Had POC done on multiple approaches in a weeks timeframe , presented pro and cons to management
- Decided on approach to follow
- Lead development effort on solution proposed by team , During development optimized the current pipeline which lead to improvement in SLA
- Had to deal with mutliple issue as Data lake API were not fully matured and had bugs which had potential to derail the roadmap
- Migrated the ETL data pipeline in next 6 months
- Created a resuable solution which could be used by mulitple teams
Result (Metrics that was impacted)
- Were able to migrate the pipeline before anticipate time, allow ample of time for use testing and overall success of Datalake
- Reduce time for data delay from 6 hours to 1 hour
- Showcased to organization best practices around ETL pipeline development. Infrastructure as code , Unit test cases for Data pipelines. Performance optimizations
Monday, 5 June 2023
Wednesday, 31 May 2023
Manager Training Resources
Both the Links should be used together
Facillitaor guide is better than slides has detailed description
https://docs.google.com/document/d/16aDChcr2W6r683bCUeOD-4FEP3sx754_n37-bSHPFw4/edit
Manager Handbook For Training
This has Conversion Guide for Getting to know employees
Python Tricky Questions
https://prepinsta.com/competitive-advanced-coding/
https://prepinsta.com/interview-preparation/technical-interview-questions/most-asked-coding-questions-in-placements/
https://prepinsta.com/python-program/find-largest-sum-of-contiguous-subarray/
# Find the larget substring with largets sum
#Array = [-1, 8, 1, -7, -1, 5, 1, -3]
#Sub array which will give maximum sum [8, 1]
#Maximum sum = 9
list_a = [1,2,3,4,-4,-5,3,0,-1]
len_list = len(list_a)
max_sum = -1
ans = []
for i in range(len_list):
for j in range(i,len_list):
sum_v = sum(list_a[i:j])
if sum_v > max_sum:
max_sum = sum_v
ans = list_a[i:j]
print(ans)
Tuesday, 30 May 2023
Google Project Oxygen - Managers Matter
* Apart from Coaching Googles Project oxygen adds few more points
“Pure” Coaching
Actively listen
Ask powerful questions
Raise awareness around “blind spots”
Challenge thinking and deeply held beliefs
Make requests for new action
Google Project Oxygen Definition
Provide specific & timely feedback
Balance positive (motivational) & negative (developmental) feedback
Understand unique strengths & development areas of each person
Tailor coaching to the individual & situation ( What is coaching refer above)
Suggest solutions
Have regular 1:1s
Another Coaching Defination
Coaching is unlocking a person's potential to maximize their own performance. It is helping them to learn rather than teaching them.
When should i Coach vs Tell how to do a task
When i should coach
When working with high-potentials
Coaching fosters their long-term development
When working with knowledge workers
The “expert” leadership role has limitations when workers may be more knowledgeable than the boss
When commitment trumps control
When securing employees’ commitment and intrinsic motivation is more important than controlling them, coaching is the best approach
When the issue is around managing relationships
e.g. stakeholder management, how to motivate others
When Should Coaching be Avoided
When dealing with serious underperformers
Coaching is not a performance plan
When you do have the answers. If you know exactly how work must be done, direct instruction is better than inquiry
If a question has just one answer, people feel quizzed
When task control is more important than commitment
In doing routine tasks for example, this may be the case
If the coach believes the coachee cannot achieve the goal
An important aspect of coaching is having a growth mindset. If this is the case, question your assessment before acting.
- When there are safety or legal risks involved
How to Be a Good Coach
Be fully present for and focused on the coachee
Be aware of your own mindset and that of the coachee
Practice empathic listening (aka active or reflective listening)
- Ask open-ended questions to facilitate coachee’s own insight
“Focused attention on coaching is more important than the time spent. What matters most is listening so the other person feels heard. Eliminate distractions and cultivate a sense of presence in the moment.”
Wednesday, 17 May 2023
Role of Manager in Agile Team
Agile Manager
- Servent leader - lead by serving team. Create a environment where team can succeed
-- if there are any blockers for team i help unblock it
Adaptive management
- open for change. We have been doign things this way and we always look for ways of improving
Coaching and mentoring
- coach and mentor team so team becomes self organized
Focus on what not on How
--
Build trust in team
-- create environmet where we focus on problem rather than who is raising it
-- more focus on workflows and producing a output
outcome over output
-- what value we have achieved
-- move team from activities to custom focus
sustainable pace
-- team should not be burned out
12 Principle of Agile
https://agilemanifesto.org/principles.html
Agile Development Manifesto
https://www.atlassian.com/agile/manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Tuesday, 16 May 2023
Team Management Notes 2
Responsibilities on Team
The formal leader on the team
-- the facillitator
-- Scribe --
-- timekeeper
-- Share the responsibities with team as they mature
Agenda
-- People should come to meeting with clear topics that we cover, What is expected outcome
-- Order of things -- Time allocated to topic
-- recognition since last meeting, you can ask whole team
-- action plan
Performance management
-- Define performance , what you want to achieve
-- then either skill (cant do ) or Motivation (Wont do)
-- Is it a skill -- requires practice training
-- Is it knowledge -- requires education
How to define a skill training
-- Take a skill - Break it into component parts. It has to be pinpointed behaviour
--
Motivations
-- What is the highest form of motivation
- Purpose , Meaningfulness -- focus on society
-- social -- Relationship bonding -- focus on team and community
-- situational - rewards and motivations - Focus on self
-- if you can connect to people with higher purpose , that is highest form of motivation
Facillitations
-- getting collective wisdom
-- given a problem its a group solution, without individual egos
-- so idea is to come up with solution that was not possible with one person
Facillitatins skills
-- Set the agenda
-- tell how many items are in meeting, what we want to do
-- share outcomes
-- share expectations
Decision styles
-- command decisions
-- consultive decisions
-- concesus type decision
Conflict resolution
--Everyone is right in their eyes, idea is for to look at other persons idea
--
Lean thinking
-- focus on values
-- eliminate waste
-- evidence based
-- enpowed people
--
Becoming a Influencing manager
-- maintain a Adult ego state , (Parent,child)
-- ADISCO -- Adult, direct honest and polite , i statement (Take ownership) , solution oriented, calmly expression yourself,
o - ownership of own behavior
Vision
Do you team know what job is Individually
Do they know what a good job looks like
Do they know how to measure good performance. Do they know what a good week looks like
Work ith Smart Goals
- Specific - really clear goals
- M - measurable goal , there is proper metric that tells how far we are from achieving goal
- A - Achievalble goals
-- R - Relevant
- T - Time bound
Involve team in Setting goals
-- involve team in identifying the behaviours
Influencing attitudes
- icek Azben
- Is it the right thing to do (show why its the right thing to do )
- Is everyone doing it ( show all are following it )
- can i do it ( Any blockers for doing this )
Is is the right thign to do
-- has the decisoin come by colloboration
-- State the benifits
- Regular team meetings on why this behaviours are important
- one to ones - repeating the behaviours
- repeatedly stress who these relate to goals
- appeal to personal values
- Reward
What everyone is doing
-- showcase the behaviour
-- dont ignore distructive behaviour
-- Intervene quickly
-- Encourage team to intervene
-- Publicise success
Can i do it
-- provide a good training
-- Support training before and after
-- Ensure processes support behaviour you wont
-- provide authority
-- provide opportunity
Communications p
-- words , Tones , Body language
Influencing your boss
Is it right thing to do
-- what is their kpi
-- what is their personal motivations
-- aligning your goals with their motivationns and Goals
What everyone else is doing
- best practice industry standards
can i do it
-- prevent as evolution than revolution - so that they will adopt
-- look for example that this has happened
-- show a credible plan
Influencing your peers
-- what formed the intentions to behave
-- is it good --- what are their drivers, their own KPI,
-- is everyone doing it
-- can i do it
dealing with dificult people
-- opposing goal or narrative
-- someone having a different goal then you
-- someone havin a different narrative, how they perceive and talk about
-- For narrative - making sure that your purpose is very clear
Personality Clashes
-- individual difference
- extraversion
- Agreeableness -- go along with flow
-conscientous -- how anxious we are to get ti right
- emotional stabiliy
- intelligence
Leadership styles
-- Authoratian - Its my way or highway
-- collaborative - lets work toghether to identify the best way
-- delegative - i will let you make the decision
Heartbeat of role
-- stable and predictable
What do you do on day on day basis
-- Team meeting
-- Management meeting
-- email sessions
-- checking KPI
-- scheduled daily/weekly/monthly checks
-- one to ones
-- Research and personal learning
Managing performance
-- Understanding specific standard and level of performance
-- be clear on required outcome
-- be clear on what the standard is
-- provide tools , training and opportunity
-- provide a clear time frame
-- smart instruction -- specific , measurable , achievalble , relevant, timebound
Feedback when standard is reached
--
Feedback when standard is not reached
-- explain what was below the standard
-- ask for their analysis
-- remind them of what was required
-- explore training and other needs
Delegate
-- Giving authority for someone else to do part of you job
-- example - Get the information and we can decide what to do,involve people in decision making
-- to get delegation right , make sure to match person to task , level of expertise ,
-- set smart goals
Sql Tricky Question - Consequetive Wins
https://www.sqlteam.com/articles/detecting-runs-or-streaks-in-your-data
GameDate Result 1/1/2000 W 1/12/2000 L 1/15/2000 W 1/17/2000 W 1/22/2000 W 2/1/2000 L 2/5/2000 W 2/8/2000 L 2/16/2000 W 2/19/2000 L 2/25/2000 L 2/28/2000 L 3/15/2000 L 3/19/2000 W 3/25/2000 W
- Do a self join with table - do a cross join with all the previous dates where result is not matching - count number of rows above a record where record is not matching - This becomes the run group
SELECT GameDate, Result, (SELECT COUNT(*) FROM GameResults G WHERE G.Result <> GR.Result AND G.GameDate <= GR.GameDate) as RunGroup FROM GameResults GR
GameDate Result RunGroup 1/1/2000 W 0 1/12/2000 L 1 1/15/2000 W 1 1/17/2000 W 1 1/22/2000 W 1 2/1/2000 L 4 2/5/2000 W 2 2/8/2000 L 5 2/16/2000 W 3 2/19/2000 L 6 2/25/2000 L 6 2/28/2000 L 6 3/15/2000 L 6 3/19/2000 W 7 3/25/2000 W 7
-- Now we need to group by Result and Run Group and find how many games are played
SELECT Result, MIN(GameDate) as StartDate, MAX(GameDate) as EndDate, COUNT(*) as Games FROM (SQL 1) A GROUP BY Result, RunGroup ORDER BY Min(GameDate)
Result StartDate EndDate Games W 1/1/2000 1/1/2000 1 L 1/12/2000 1/12/2000 1 W 1/15/2000 1/22/2000 3 L 2/1/2000 2/1/2000 1 W 2/5/2000 2/5/2000 1 L 2/8/2000 2/8/2000 1 W 2/16/2000 2/16/2000 1 L 2/19/2000 3/15/2000 4 W 3/19/2000 3/25/2000 2
Tuesday, 9 May 2023
Team Management Notes 1
Leaders define , create that energy.
Why we do what we do, if you cant communicate that you will loose energy.
Manager
- allow people to showcase that how they are making the impact
Leaders are not about controling people,
Managers are not about controlling people but controlling processes..
As a manager i Model people behaviour to our values
- defining Standard of work
- to coach to give feedback
- to motivate both individual and team
Process
- 95% of problems are in process and not in nature of person
- set of activities that lead to tranformation of set of inputs to output
-- process can be measure by output or ratio of input to output
How do you evaluate a team
- Team Purpose
-- Does team has well defined purpose
-- team knows its customers
-- team knows how to run its meetings and general execution behaviour
-- Teams know about its desired performace and how it contributes to company goals
Team meetings
- Does team have regular scheduled team meetings (Monthly)
- Action plans in team meeting have been noted down and worked upon
- Members display truste inside and outside the meeting
Process Management
- Team know the steps in process
- Team has done improvement in process
-- it means teams knows the process clearly
-- can analyze how the process is working
-- can improve the process
- Team knows how to measure a process and can talk about variation in process
Customer focus
- Team has defined its customers and identified improvement areads from customer feedback
-- This means team knows its internal and external customers
-- know how to get feedback
-- how to analyze that feedback
- Team is able to improve customer satisfaction
Problem solving and decision making
-
Team Purpose and Charter 1 Team Purpose
2 Customer Satisfaction
3 Team Norms
4 Performamnce
Team Meeting Management 1 Team Meetings
2 Facilitation
3 Decision Styles
4 Team Trust
Process Management 1 Process Mapping
2 Process Improvements
3 Balanced Scorecard
4 Waste Ellimination
5 Process Variances
Customer Focus 1 Customer Focus
2 Customer Satisfaction
3 Customer Feedback
Problem Solving and DM 1 Problem Solving Tools II
2 Problem Solving
3 Problem Solving Meetings
4 Problem Soling Tools I
Vision
-- Where are we goin
Values
-- What is important
-- what gets rewarded for , what is punished for
-- what companies values are these have to be in line
Values worth considering
- Absolute honesty and integrity
- Respect for people
-- Engaging people in problem solving demostrates respect in people
- Continous improvement of the processes they own
-- Measurement of data using facts
-- learn to look at data , understand the facts ,
-- unity of people and process
-- Challenge
Communication
-- Horizon or vertical is building trust
-- Communiation equals trust
-- Listen more , Always ask for feedback on what has been communicated
-- succesful communication is more about expectation,
--- perception is relative to expectations
Vertical communication
-- You should have key performance measures. You should clarify this with your measure
and team should be responsible for this
-- Any changes in process that may impact output
-- Always present problems with solutions -- partner for solutions
-- Any improvement in process share with your manager
In addition to above the below is responsible for team
-- Company events
-- Stratergy
-- policy changes
Delegation and Assigment of Tasks
-- When delegation is difficult
-- When delegation is easy
-- when delegates are motivated
-- When they are expert in their process
-- time for completion do checkout
Core Guiding principles
-- Respect for Individuals
-- Service for customers
-- Striving for excellence
Disciple and Justice
-- Well disciple team, people know what to do , Behaviour according to desired expectations
Justin
-- justice is fair and equal treatment to all individuals
-- people want others behaving in a manner similar
Human performance analysis
-- provide reinforcement behaviour
-- 4 positive for 1 negetive , recognition vs critisism
Standard work
-- you cannot have continous improvement unless you have standard work
-- Is it bad execution or its Bad standard work
What is standard work for leader , this should be well defined
Teams
-- create a high performing team, Dont lead CATS ( individual people)
-- Having multi skill teams , it becomes easier for team members
Defining structure of your team
--
Optimizing performance for your team
-- What information
-- what skills
-- what tools , Software , computers
-- What motivations - Awards , Rewards , Praise
Stages of Team Development
-- Forming - Figuring team dynamics
-- Storming -- people rebel about responsibilities
-- Norming - People settle into roles
-- performing -- things become fluid , people perform at high level
Who decides and who controls
-- depends on maturity of the team
-- as team has demostrated responsibility , you give the control to the team
-- So first we need to show team what a good behaviour is
Team charter
-- What is the purpose , Who is on the team ( or may be what aare the jobs)
-- who is the sponsor of the team
-- Define process -- what is input , what is output . Set the boundries of the team
-- what are the performance responsibilities of team
-- what is code of conduct -- example showing up on time,
-- what is communication resposibility - who do we communicate and how
Subscribe to:
Posts (Atom)