Wednesday, 19 April 2023
Python Notes
Apache Airflow Notes
Notes on apache airflow
with DAG( dag_id = ) as dag :
- Operators in Airflow - default python , bash the default ones which come with it
- https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/operators/index.html
- When interactin with any third party provides we install those providors
- Example AWS , Snowflake , Databricks
- Sensors -
- Task group - Airflow utils has task group ids which can be used for grouping
Interview Use cases to talk about
Engineering Challenges solved
Building Data lake
- A system built over time was moved to Datalake
- Initial system has multiple redshifts / Compaction issues / Multiple s3 paths from which data was consumed by customers
- Ownership was divided among teams but not logically
- access was not controlled , redundant access
STAR format - How was the impact measured
- SLA improvement -- Team was able to redesign pipeline during migration to datalake and take our redundant steps improving sla 6 hours to 3 hours
- Cost saving of 200k by moving processing
People Challenges solved
Hiring and Recruiting Issues solved
Big Projects Handled
Appraisal Ratings
Cost Saving
- What is cost of Redshift
- What is cost of EMR
- Cost of Athena
- Number of nodes
Ra3.16x Large - Reserved instance - 75,000 - 48vcpu , 384 ram , 128TB space , Scales up to 16 petabytes
DS2- 8x large - 16TB , HDD , 244gb memory , -- We had 50 Node cluster - DS2 is deprecated - 30 thousand
DC2-8x - 32vcpu ,244 gb memory, 2.5TB SSD,
EMR Type used - R5d - 48 VCPU, 512 memory - Supports upto EMR 6.3
Tuesday, 18 April 2023
Responsibilites of Manager
Responsibilites of Manager
- Business impact
- For teams in initial stages - Documenting user feedback on How Data teams have helped business indirectly greatly helps. Example Product owners were able to show a new widget which boosted sales. Understanging user behaviour helped them to come up with widgets
- For teams in advanced stages - Integrating data reports with business operations. Infrastructure savings
- Productivity
- Prioritizing work , unblocking teams members.
- Building trust within team so that they can voice their concerns, Running retros. Running team bonding activities
- Monitoring daily scrum calls
- Making sure work is aligned with career goals of people on the team.
- Building work life balance by setting up processes around work intake, completion timelines.
- Setting ownership expectations
- System Health
- Making sure alarms, disaster recorvery are in place. Setting up ownership for projects
- planning for version updates , migrations
- Team Health
-
- Stakeholder Happiness
- separate post on stakeholder happiness
Delivering Results
Team Builing
Collaboration
Vision
Professional Goals
Stakeholder Management
Earn trust of Stakeholder and make sure you are not loosing trust
Clearly defining ways of working of the team with SLA for resolving a particular type of issue
* Data Quality issues
* New data onboarding requests, How will new request come in
* Adhoc data requests
* Office hours and how to get expert to help stakeholders
* Clearly defining Data SLA
* What will the team helping with (Example team wont be writing queries for user questions) . May be only during office hours team can help
* Clearly defining ways to onboard customers
* How will stakeholder receive communication in case of issues/delays
* A clear onboarding process for stakeholders with clear defination of what to expect.
* In case of indirect dependencies stakeholder is creating and are critical business functions. Making sure the downstream customers are aware of the SLA , processes from upstream team
* Making sure access is not spiralling out and only those who are required are having access
Have seen a lot of stakeholder issues as it was not clear how a stakeholder is consuming data. The consumption pattern were randomly developed by Stakeholders without consulting data team.
Having Dashboards to clearly show and document
* SLA
* Data Quality
* Incident requests
* Updates and Backfill delays
* Centere of excellence - In case of bigger issues what was done in past