Friday, 28 February 2014

Parrallel hint in RAC making ETL run slow

First Question is when we should use parallel hints ??

Without parallel it makes the query wait on the indexes data to be read. So we are just waiting for data to be read , so parallel hint makes sense.

 Link to Article on Understanding Parallel Explain plans




When you write parallel hint your session becomes coordinator processor and there are several slave processes which are created. Slave read data and give to coordinator which does the loading.Below is a scenario in which Parallel hint was actually slowing the query .Below analysis is for DML statements. Inserts were taking time

select sid, state,event,SECONDS_IN_WAIT,BLOCKING_SESSION,BLOCKING_INSTANCE,
row_wait_obj#,
       row_wait_file#,
       row_wait_block#,
       row_wait_row#,p1,p2
from v$session
where username = 'ABC' and
OSUSER='kkk'
and event = 'enq: TX - row lock contention'

Note -- There are 66 parallel session on Measure val

select event, total_waits, (time_waited*10)/1000 tw_ms,
       average_wait*10 aw_ms, max_wait*10 mw_ms
from v$session_event
where sid in (
select sid from v$session
where username = 'ABC' and
OSUSER='kkk' )
order by 3 desc





There are two events which point highly in the favour of this .
    1) PX Deq Credit: send blkd---- This means the slave are waiting for coordinator process to be free to supply their data
    2) PX Deq: Execution Msg---This means once slave processes have supplied their data they are waiting for more request from coordinator process.
    3) So both of them point that the coordinator is not able to handle the data provided by slaves, which means there are more slave processes that the coordinator can handle


Note -- On average every parallel read wait quarter of second to read

select name, sum(value)
from v$sesstat s, v$statname n
where n.statistic# = s.statistic# and
sid in (
select sid from v$session
where username = 'ABC' and
OSUSER='kkk' )
group by name
order by 2 desc

Note - The interconnnect bytes seems to be a large number















 

 
To do insert in parallel we need to have parallel DML enabled which enables the slaves to do the insert other wise only coordinator processor does the insert and it will be bottleneck you will all the slaves waiting for one coordinator process do the load


Below is analysis for Report query which was taking time due to degree of parallelism

Analysis for  logic in report

1) Changing the parallel degree in report  logic from parallel(8,1) to parallel. Brings down the query time from 100 to 11 seconds

    1. PX Deq: Execution Msg---This means once slave processes have supplied their data they are waiting for more request from coordinator process.
    2. So both of them point that the coordinator is not able to handle the data provided by slaves, which means there are more slave processes that the coordinator can handle

select sid, state,event,SECONDS_IN_WAIT,BLOCKING_SESSION,BLOCKING_INSTANCE,
row_wait_obj#,
       row_wait_file#,
       row_wait_block#,
       row_wait_row#,p1,p2
from v$session
where username = 'ABC' and

OSUSER='kkk'












ETL load running slow- How to identiy issue

Hi All,

V$session is all you need to resolve your ETL load issue. It tells you what your sql is waiting for and what is happening with sql. Below are few notes that i found helpful. In my case parallel hint was causing too many slave processes and query coordinator was taking time to read all the messages from different slaves.

Below are very good articles by Arup Nanda which i found helpful


 Basics about v$session table , V$session_event and Active session history





select sid, state,event,SECONDS_IN_WAIT,BLOCKING_SESSION,BLOCKING_INSTANCE,
row_wait_obj#,
       row_wait_file#,
       row_wait_block#,
       row_wait_row#,p1,p2
from v$session
where username = 'ABC'

Now either you can check what is the blocker on either from row_wait_obj#, which is the object id in the dba objects or you can check with block id which is p1 and file id p2 which can be checked in dba_objects


select * from dba_objects
where object_id= 85988
-------------------------------------------------------------------------------------------------------------------------------------------------------------

select session_id, sample_time, session_state, event, wait_time, time_waited, sql_id, sql_child_number CH#
from v$active_session_history
where user_id =1000
and sample_time between
    to_date('27-FEB-14 09.00.00 AM','dd-MON-yy hh:mi:ss PM')
       and
    to_date('27-FEB-14 11.0.00 AM','dd-MON-yy hh:mi:ss PM')


select event, total_waits, (time_waited*10)/1000 tw_ms,
       average_wait*10 aw_ms, max_wait*10 mw_ms
from v$session_event
where sid in (
select sid from v$session
where username = 'ABC' and
OSUSER='KKKK' )
order by 3 desc




Saturday, 1 February 2014

Reporting and Network Latency issues


100Mbps notice the b is small it means Bits . 8 Bits make a byte so 100Mbs/8 = 12.5 MBps( 12 Mega bytes per second--size of music file)

In a local LAN network maximum speed is about 100 Mbps which means 12.5MB per second. Considering you are using the entire connection to your self.

Best possible speed of Copper wire in LAN is 1000Mbps which means 125MB per second.

Fiber optic cable maximum speed upto 100Gbps -- 12.5 GB ( Giga byte per second -- Entire movie trilogy in one second)

Note --Above is theoritical numbers, we will have delays due to network card and other latency issues

If I connect from my machine in india using reporting tool to database server in US .It takes more time 300second compared to 30 seconds when developer in US is connecting from local machine reporting tool to database.

Answer --- This is due to delay of 300ms when connecting from india to US db server. For details how 300ms delay causes page to wait for 6 min you need to understand below

Basics - Signal is transmitted from india to US via a under sea link. Or in some cases through satelite link. There are many companies which operate under sea link (check in google) and point from where these undersea link start in india.

Note -- propogation delay would be there even if you only person using this link.

Propagation delay - distance /speed of signal in copper wire or fibre optic in KM ---
so from india to US ----- 5400/197863 = 23ms delay

Serialization delay --- packet size in bits/transmission rate in bits per second
Usually packet sizes are in KB so only if your packet size is big in MB. It will be a big issue OR if your transmission rate is low that is you are having 50kbps phone modem link. It will be a MAJOR issue

Latency due to TCP ---Transmission control protocol . Key point how it works are
  1. Handshake between source and destination
  1. Received acknowledgement
  1. Requesting missing bits
  2. Windowing technique -- Adjusting the transmission rate depending on the ability of network to handle it. So for high latency network though bandwidth is there it will send lesser data in a window

Most import point -- So when TCP is transfering, if there is a delay of 100 ms (it is always round trip delay) because TCP always waits for acknowledgment before next packet is sent.

Hey , All this is in milliseconds not even seconds. So why should we bother??

Link having 50 ms (milli seconds)  gives page in 3 seconds
Link having 300ms gives page in 11 seconds.

So user will not be pleased with 11 seconds

How to check the round trip time for connecting from your computer

Ping your database server and notice the RTT round trip time. You want it to be within 100ms at the maximum

Now check the RTT round trip time from your application server


Question -- Sql server returns data in few minutes but reporting tool takes 10 min to return full data.

Ans -- TCP protocol adjust the transmission speed on the speed of the tool that is how fast is it able to take the data supplied by the source. So since sql developer does no processing of its own . It just stores file in excel . The output is much faster compared to reporting tool which builds its own file. The speed at which tool is able to build file affects the speed at which data is transferred from source