Tuesday, 6 June 2023

Leadership and Growth Mindset

 https://sites.google.com/deliveroo.com/tech-leadership/engineering-manager-prep/leadership-growth-mindset

Behaviour Interview Prep - Manager

 https://sites.google.com/deliveroo.com/tech-leadership/engineering-manager-prep/behavioural



Resolving Conflicts

Principle of conflict resolution 
  • everyone is right in their eyes 
  • avoid false dichotomies (Black or white) 
  • people generally present a position , but rarely 
  • that is the issue, the person resolving issue need to understand the real issue by using listening skills
  • Dont dig in, that is dont try to explain why you think you are right, i am right because it leads to escalations, look for sound of argument 
  • dont use exagerated statement - you always 
  • seek unity of purpose 
  • focus on where we are going 
  • its not person but process -- how can we make sure
  • such conflicts do not appear in future example RACI Framework
  • look for win wins mutual benefits 
  • long term -- develop a culture where people listen to each other

When it comes to creating resolving conflicts a few of these communication skills come in Handy. 

  • Empathy , when you empathise with people it allows them to move over things , otherwise you will hear people saying the same things in many conversations 
  • Acknowlegement - When listening to someone acknowlegement like nodding , saying yes motivates the speaker, absense of these kinds of motivation leads to demotivation 
  • Reflecting or Rephrasing -- Listening is more than Hearing, we need to understand what the other person is feeling and rephrase it. Reflecting is showing the other person mirror.  This is what i think you said 
  • Brainstorming - Is allowing people to come up with ideas without Judging. Note that a idea might not be good but it might lead someone to come up with better idea on those lines


Framewok for conflict resolution 
  • Find Agreement 
  • Clarify disagreement 
  • express empathy 
  • seek alternatives 
  • Agree on criteria for solution 
  • Combine and create 
  • Reach consensus

Project Walkthrough for Interview in STAR Format

 Overview of Project 

Company had roadmap to create a Financial Data lake. Software engineering team Was tasked with creating infrastructure for this data lake

  • that is the actual data lake to store the data
  • ETL tool which would connect to this datalake on which jobs needs to be created
  • Tools to query this data lake 

Data Engineering team which I was leading was tasked with Migrating data pipeline to datalake, using tools developed by Software engineering team. Wireframes on how this tool would look like was shared with team and data Engineering team waited for Software engineering tool to complete

Their work so we can start working on pipeline migration. 


STAR Format follows 

Situation (Challenges you faced)

  • 6 months prior to the data lake project completion date, My senior manager discusses the project and asks me to get started. At this time his understanding is the same that a tool would be developed. 
  • After discussing with Software engineering team, we understand that they have hit roadblocks and entire effort is delayed, They have not even started the tool devleopment and have not completed the Datalake infrastructure development
  • I escalated it to senior management clarifying that Data Engineering project cannot start until ETL tool development is completed. 
  • After multiple hard discussions with Senior management , Software engineering team accepted that they have completed missed timeline and there is no time available for Data engineering to migrate.
  • Software engineering team will not work on creation of tool, instead they will only work on Creation of Data lake API which will be used by Data engineering team in their solution
  • We were  to run the development efforts in Parallel with Data lake API Development 
  • Data engineering will work on migration using AWS platform and Data lake API 


Tasks (Decision you took ) 

  • Were tasked with creating a approach for migration, with aim of completing it in next 6 months.

Action (How you worked with your team and cross functional peers) 

  • Discussed with team came up with few approaches how we could do the project 
  • Had POC done on multiple approaches in a weeks timeframe , presented pro and cons to management 
  • Decided on approach to follow 
  • Lead development effort on solution proposed by team , During development optimized the current pipeline which lead to improvement in SLA
  • Had to deal with mutliple issue as Data lake API were not fully matured and had bugs which had potential to derail the roadmap
  • Migrated the ETL data pipeline in next 6 months 
  • Created a resuable solution which could be used by mulitple teams 
  •  

Result (Metrics that was impacted) 

  • Were able to migrate the pipeline before anticipate time, allow ample of time for use testing and overall success of Datalake 
  • Reduce time for data delay from 6 hours to 1 hour 
  • Showcased to organization best practices around ETL pipeline development. Infrastructure as code , Unit test cases for Data pipelines. Performance optimizations 


Monday, 5 June 2023

Wednesday, 31 May 2023

Your Role as Manager

 




Manager Training Resources

 Both the Links should be used together 

Facillitaor guide is better than slides has detailed description 

https://docs.google.com/document/d/16aDChcr2W6r683bCUeOD-4FEP3sx754_n37-bSHPFw4/edit

Manager Handbook For Training 

https://docs.google.com/presentation/d/1DpxBRSE2FwxNazRMUOozf-MVRChCqCr7m7p1Fl_gx4U/edit#slide=id.g6271fc939_00

This has Conversion Guide for Getting to know employees 





Why Decision Making is Hard

 





















 
 










Giving Feedback

 

















Python Tricky Questions

 https://prepinsta.com/competitive-advanced-coding/

https://prepinsta.com/interview-preparation/technical-interview-questions/most-asked-coding-questions-in-placements/



https://prepinsta.com/python-program/find-largest-sum-of-contiguous-subarray/

# Find the larget substring with largets sum
#Array = [-1, 8, 1, -7, -1, 5, 1, -3]
#Sub array which will give maximum sum [8, 1]
#Maximum sum = 9

list_a = [1,2,3,4,-4,-5,3,0,-1]
len_list = len(list_a)
max_sum = -1
ans = []

for i in range(len_list):
    for j in range(i,len_list):
        sum_v = sum(list_a[i:j])
        if sum_v > max_sum:
            max_sum = sum_v
            ans = list_a[i:j]
       
   
print(ans)
           

Tuesday, 30 May 2023

Google Project Oxygen - Managers Matter

 https://rework.withgoogle.com/blog/the-evolution-of-project-oxygen/#:~:text=Back%20in%202008%2C%20an%20internal,into%20our%20manager%20development%20programs.



* Apart from Coaching Googles Project oxygen adds few more points


“Pure” Coaching

  • Actively listen

  • Ask powerful questions

  • Raise awareness around “blind spots” 

  • Challenge thinking and deeply held beliefs

  • Make requests for new action


Google Project Oxygen Definition

  • Provide specific & timely feedback

  • Balance positive (motivational) & negative (developmental) feedback 

  • Understand unique strengths & development areas of each person

  • Tailor coaching to the individual & situation ( What is coaching refer above)

  • Suggest solutions

  • Have regular 1:1s


Another Coaching Defination
Coaching is unlocking a person's potential to maximize their own performance. It is helping them to learn rather than teaching them.

When should i Coach vs Tell how to do a task

When i should coach

  • When working with high-potentials

Coaching fosters their long-term development


  • When working with knowledge workers

The “expert” leadership role has limitations when workers may be more knowledgeable than the boss 


  • When commitment trumps control

When securing employees’ commitment and intrinsic motivation is more important than controlling them, coaching is the best approach


  • When the issue is around managing relationships

e.g. stakeholder management, how to motivate others


When Should Coaching be Avoided

  • When dealing with serious underperformers

Coaching is not a performance plan


  • When you do have the answers. If you know exactly how work must be done, direct instruction is better than inquiry

If a question has just one answer, people feel quizzed


  • When task control is more important than commitment

In doing routine tasks for example, this may be the case


  • If the coach believes the coachee cannot achieve the goal

An important aspect of coaching is having a growth mindset. If this is the case, question your assessment before acting.


  • When there are safety or legal risks involved
How to Be a Good Coach 

  • Be fully present for and focused on the coachee

  • Be aware of your own mindset and that of the coachee

  • Practice empathic listening (aka active or reflective listening)

  • Ask open-ended questions to facilitate coachee’s own insight


“Focused attention on coaching is more important than the time spent. What matters most is listening so the other person feels heard. Eliminate distractions and cultivate a sense of presence in the moment.”







Read this presentation for More on Coaching - https://docs.google.com/presentation/d/13-rj7ZqlyuHNli9Ybd5D8g-qqUNtG81t3JLq7JV5c2o/edit#slide=id.gc65ffdba2_0_2656



Wednesday, 17 May 2023

Role of Manager in Agile Team

 Agile Manager 

- Servent leader - lead by serving team. Create a environment where team can succeed

-- if there are any blockers for team i help unblock it 


Adaptive management 

- open for change. We have been doign things this way and we always look for ways of improving 


Coaching and mentoring 

- coach and mentor team so team becomes self organized 


Focus on what not on How 

-- 


Build trust in team 

-- create environmet where we focus on problem rather than who is raising it 

-- more focus on workflows and producing a output 


outcome over output 

-- what value we have achieved 

-- move team from activities to custom focus 


sustainable pace 

-- team should not be burned out 


12 Principle of Agile

 https://agilemanifesto.org/principles.html


Agile Development Manifesto 

https://www.atlassian.com/agile/manifesto


Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Tuesday, 16 May 2023

Team Management Notes 2

 Responsibilities on Team 

The formal leader on the team 

-- the facillitator 

-- Scribe -- 

-- timekeeper 

-- Share the responsibities with team as they mature 


Agenda 

-- People should come to meeting with clear topics that we cover, What is expected outcome 

-- Order of things -- Time allocated to topic 

-- recognition since last meeting, you can ask whole team 

-- action plan 


Performance management 

-- Define performance , what you want to achieve 

-- then either skill (cant do ) or Motivation (Wont do)

-- Is it a skill -- requires practice training 

-- Is it knowledge -- requires education 


How to define a skill training 

-- Take a skill - Break it into component parts. It has to be pinpointed behaviour

-- 


Motivations

-- What is the highest form of motivation 

- Purpose , Meaningfulness -- focus on society 

-- social -- Relationship bonding -- focus on team and community 

-- situational - rewards and motivations - Focus on self 

-- if you can connect to people with higher purpose , that is highest form of motivation


Facillitations 

-- getting collective wisdom 

-- given a problem its a group solution, without individual egos 

-- so idea is to come up with solution that was not possible with one person 


Facillitatins skills 

-- Set the agenda

-- tell how many items are in meeting, what we want to do 

-- share outcomes 

-- share expectations 



Decision styles 

-- command decisions 

-- consultive decisions 

-- concesus type decision 



Conflict resolution 

--Everyone is right in their eyes, idea is for to look at other persons idea 

-- 


Lean thinking 

-- focus on values 

-- eliminate waste 

-- evidence based 

-- enpowed people 

-- 


Becoming a Influencing manager 

-- maintain a Adult ego state , (Parent,child)

-- ADISCO -- Adult, direct honest and polite , i statement (Take ownership) , solution oriented, calmly expression yourself, 

o - ownership of own behavior 


Vision 


Do you team know what job is Individually 

Do they know what a good job looks like 

Do they know how to measure good performance. Do they know what a good week looks like 



Work ith Smart Goals 

- Specific - really clear goals 

- M - measurable goal , there is proper metric that tells how far we are from achieving goal 

- A - Achievalble goals 

-- R - Relevant 

- T - Time bound 


Involve team in Setting goals 

-- involve team in identifying the behaviours 



Influencing attitudes 

- icek Azben 

- Is it the right thing to do (show why its the right thing to do )

- Is everyone doing it ( show all are following it )

- can i do it ( Any blockers for doing this )



Is is the right thign to do 

-- has the decisoin come by colloboration 

-- State the benifits 

- Regular team meetings on why this behaviours are important 

- one to ones - repeating the behaviours 

- repeatedly stress who these relate to goals 

- appeal to personal values 

- Reward 


What everyone is doing 

-- showcase the behaviour 

-- dont ignore distructive behaviour 

-- Intervene quickly 

-- Encourage team to intervene 

-- Publicise success 


Can i do it 

-- provide a good training 

-- Support training before and after 

-- Ensure processes support behaviour you wont 

-- provide authority 

-- provide opportunity 


Communications p

-- words , Tones , Body language 



Influencing your boss 


Is it right thing to do 

-- what is their kpi 

-- what is their personal motivations 

-- aligning your goals with their motivationns and Goals 


What everyone else is doing 

- best practice industry standards 


can i do it 

-- prevent as evolution than revolution - so that they will adopt 

-- look for example that this has happened 

-- show a credible plan 


Influencing your peers 

-- what formed the intentions to behave 

-- is it good --- what are their drivers, their own KPI, 

-- is everyone doing it 

-- can i do it 


dealing with dificult people 

-- opposing goal or narrative 

-- someone having a different goal then you 

-- someone havin a different narrative, how they perceive and talk about 


-- For narrative - making sure that your purpose is very clear 


Personality Clashes 

-- individual difference 

- extraversion 

- Agreeableness -- go along with flow 

-conscientous -- how anxious we are to get ti right 

- emotional stabiliy 

- intelligence 


Leadership styles 

-- Authoratian - Its my way or highway 

-- collaborative - lets work toghether to identify the best way 

-- delegative - i will let you make the decision 


Heartbeat of role 

-- stable and predictable 

What do you do on day on day basis 

-- Team meeting 

-- Management meeting 

-- email sessions 

-- checking KPI 

-- scheduled daily/weekly/monthly checks 

-- one to ones 

-- Research and personal learning 


Managing performance 

-- Understanding specific standard and level of performance 

-- be clear on required outcome 

-- be clear on what the standard is 

-- provide tools , training and opportunity 

-- provide a clear time frame 

-- smart instruction -- specific , measurable , achievalble , relevant, timebound



Feedback when standard is reached 

-- 


Feedback when standard is not reached 

-- explain what was below the standard 

-- ask for their analysis 

-- remind them of what was required 

-- explore training and other needs 


Delegate 

-- Giving authority for someone else to do part of you job 

-- example - Get the information and we can decide what to do,involve people in decision making

-- to get delegation right , make sure to match person to task , level of expertise , 

-- set smart goals 


Sql Tricky Question - Consequetive Wins

 https://www.sqlteam.com/articles/detecting-runs-or-streaks-in-your-data

GameDate    Result
1/1/2000    W
1/12/2000   L
1/15/2000   W
1/17/2000   W
1/22/2000   W
2/1/2000    L
2/5/2000    W
2/8/2000    L
2/16/2000   W
2/19/2000   L
2/25/2000   L
2/28/2000   L
3/15/2000   L
3/19/2000   W
3/25/2000   W
- Do a self join with table 
- do a cross join with all the previous dates where result is not matching
- count number of rows above a record where record is not matching
- This becomes the run group 
SELECT GameDate, 
  Result, 
  (SELECT COUNT(*) 
   FROM GameResults G 
   WHERE G.Result <> GR.Result 
   AND G.GameDate <= GR.GameDate) as RunGroup 
FROM GameResults GR
GameDate    Result    RunGroup
1/1/2000    W         0
1/12/2000   L         1
1/15/2000   W         1
1/17/2000   W         1
1/22/2000   W         1
2/1/2000    L         4
2/5/2000    W         2
2/8/2000    L         5
2/16/2000   W         3
2/19/2000   L         6
2/25/2000   L         6
2/28/2000   L         6
3/15/2000   L         6
3/19/2000   W         7
3/25/2000   W         7
-- Now we need to group by Result and Run Group and find how many games are played
SELECT Result, 
  MIN(GameDate) as StartDate, 
  MAX(GameDate) as EndDate, 
  COUNT(*) as Games
FROM (SQL 1) A
GROUP BY Result, RunGroup
ORDER BY Min(GameDate)
Result      StartDate   EndDate     Games
W           1/1/2000    1/1/2000    1
L           1/12/2000   1/12/2000   1
W           1/15/2000   1/22/2000   3
L           2/1/2000    2/1/2000    1
W           2/5/2000    2/5/2000    1
L           2/8/2000    2/8/2000    1
W           2/16/2000   2/16/2000   1
L           2/19/2000   3/15/2000   4
W           3/19/2000   3/25/2000   2

Tuesday, 9 May 2023

Team Management Notes 1

 Leaders define  , create that energy. 

Why we do what we do, if you cant communicate that you will loose energy. 


Manager 

- allow people to showcase that how they are making the impact 


Leaders are not about controling people, 


Managers are not about controlling people but controlling processes..


As a manager i Model people behaviour to our values 

- defining Standard of work 

- to coach to give feedback 

- to motivate both individual and team 


Process 

- 95% of problems are in process and not in nature of person 

- set of activities that lead to tranformation of set of inputs to output

-- process can be measure by output or ratio of input to output


How do you evaluate a team 

- Team Purpose 

-- Does team has well defined purpose 

-- team knows its customers 

-- team knows how to run its meetings and general execution behaviour 

-- Teams know about its desired performace and how it contributes to company goals


Team meetings 

- Does team have regular scheduled team meetings (Monthly) 

- Action plans in team meeting have been noted down and worked upon 

- Members display truste inside and outside the meeting 


Process Management 

- Team know the steps in process 

- Team has done improvement in process 

-- it means teams knows the process clearly 

-- can analyze how the process is working 

-- can improve the process 

- Team knows how to measure a process and can talk about variation in process



Customer focus 

- Team has defined its customers and identified improvement areads from customer feedback

-- This means team knows its internal and external customers 

-- know how to get feedback 

-- how to analyze that feedback 

- Team is able to improve customer satisfaction 


Problem solving and decision making 



Team Purpose and Charter 1 Team Purpose

2 Customer Satisfaction

3 Team Norms

4 Performamnce

Team Meeting Management 1 Team Meetings

2 Facilitation

3 Decision Styles

4 Team Trust

Process Management 1 Process Mapping

2 Process Improvements

3 Balanced Scorecard

4 Waste Ellimination

5 Process Variances

Customer Focus 1 Customer Focus

2 Customer Satisfaction

3 Customer Feedback

Problem Solving and DM 1 Problem Solving Tools II

2 Problem Solving

3 Problem Solving Meetings

4 Problem Soling Tools I


Vision 

-- Where are we goin 



Values 

-- What is important 

-- what gets rewarded for , what is punished for 

-- what companies values are these have to be in line 


Values worth considering 

- Absolute honesty and integrity 

- Respect for people 

-- Engaging people in problem solving demostrates respect in people 

- Continous improvement of the processes they own 

-- Measurement of data using facts 

-- learn to look at data , understand the facts , 

-- unity of people and process 

-- Challenge 



Communication 

-- Horizon or vertical is building trust 

-- Communiation equals trust 

-- Listen more , Always ask for feedback on what has been communicated 

-- succesful communication is more about expectation, 

--- perception is relative to expectations 



Vertical communication 

-- You should have key performance measures. You should clarify this with your measure 

and team should be responsible for this 

-- Any changes in process that may impact output 

-- Always present problems with solutions -- partner for solutions 

-- Any improvement in process share with your manager 


In addition to above the below is responsible for team 

-- Company events 

-- Stratergy 

-- policy changes 


Delegation and Assigment of Tasks 

-- When delegation is difficult 

-- When delegation is easy 

-- when delegates are motivated 

-- When they are expert in their process 

-- time for completion do checkout 



Core Guiding principles 

-- Respect for Individuals 

-- Service for customers 

-- Striving for excellence 


Disciple and Justice 

-- Well disciple team, people know what to do , Behaviour according to desired expectations


Justin

-- justice is fair and equal treatment to all individuals 

-- people want others behaving in a manner similar 


Human performance analysis

-- provide reinforcement behaviour 

-- 4 positive for 1 negetive , recognition vs critisism 



Standard work 

-- you cannot have continous improvement unless you have standard work 

-- Is it bad execution or its Bad standard work 


What is standard work for leader , this should be well defined 


Teams 

-- create a high performing team, Dont lead CATS ( individual people)

-- Having multi skill teams , it becomes easier for team members


Defining structure of your team

-- 


Optimizing performance for your team

-- What information 

-- what skills 

-- what tools , Software , computers 

-- What motivations - Awards , Rewards , Praise 


Stages of Team Development 

-- Forming - Figuring team dynamics 

-- Storming -- people rebel about responsibilities 

-- Norming - People settle into roles 

-- performing -- things become fluid , people perform at high level 


Who decides and who controls 

-- depends on maturity of the team

-- as team has demostrated responsibility , you give the control to the team 

-- So first we need to show team what a good behaviour is 


Team charter 

-- What is the purpose , Who is on the team ( or may be what aare the jobs)

-- who is the sponsor of the team 

-- Define process -- what is input , what is output . Set the boundries of the team 

-- what are the performance responsibilities of team 

-- what is code of conduct -- example showing up on time, 

-- what is communication resposibility - who do we communicate and how

Friday, 28 April 2023

HUDI

 Question on Hudi 

What is Copy on write and Merge on Read 

What is Serialization in Athena table Serde 

  • serde tells us format of input file. If we have json file and we load into hive table. serde tells it the underlying file is in json format. 

What happens if we remove or add a column in Glue 

- Materialized views and joining tables 

-- how hudi knows something is latest record 

-- Hudi Snapshot and Incremental API 

How data skews are handled in Spark 

  • SALT Technique in spark for Handling Skews - Adding a new field to join key 

-- How do we partition table in S3 or HUDI , what partition keys do we use. What are indexes in HUDI 


How Parquet stores data -


Monday, 24 April 2023

Data Streaming

 There are various data streaming platforms 

- KAFKA

- Apark Streaming 

- Flink 

- Glue for Streaming 


Amazon Kinesis Data Streams 


Understanding AVRO 

- https://www.confluent.io/blog/avro-kafka-data/

- Avro has metadata in JSON and data is stored in Binary. Each file has header which is human readable and rest is binary 

-- Parquet and ORC - Are both suitable for Write once and Read heavy. ORC is optimized for hive with Hadoop. Both are columunar formas 

-- Parquet works best with Spark 

How Parquet Stores Data.Good Article  - https://www.linkedin.com/pulse/all-you-need-know-parquet-file-structure-depth-rohan-karanjawala/

- Both ORC and Parquet squeeze the data - ORC ( Optimized Row Columunar) 

https://www.upsolver.com/blog/the-file-format-fundamentals-of-big-data#:~:text=The%20ORC%20file%20format%20stores,reduce%20read%20and%20decompression%20loads.

How to Create Data streaming job 

https://aws.amazon.com/blogs/big-data/crafting-serverless-streaming-etl-jobs-with-aws-glue/

Blog on Stream Processin 

https://www.upsolver.com/wp/stream-processing-ebook?submissionGuid=d36d39a3-91ac-4f92-9a9b-f41cfd7eb305

Message Broker / Stream Processor 

- Apache Kafka 

- Kinesis Data streams 

Stream Processing Tools 

- AWS Kinesis 

- Apache Spark Streaming 

- Apache Flink 

- Kafka streaming API 

- Amazon Kinesis Data analytics allows you to process data in streams and use analytics over it. It uses apache flink

--------------------------------------------

For Loading data 

- Kinesis Firehose  -Firehose also has some tranformation capabilities . Firehose has dynamic partition capabilities and can partition data based on keys


Kafka Consumer API 

-- this is not streaming API 

-- this allows to read kafka streams and validate them or apply a logic