Wednesday, 10 July 2013

Sql tuning with Bitmap indexes and Star schema transformation

Hi Guys

This Article is relavent only if you are using OLAP (Star schema/Snowflake) .Not if you are using OLTP systems(Online transcation processing) 

General note on Indexes -- B tree index (ones that we use on primary keys) works best when you have unique values. like emp_no . When you dont have unique values they still work good if your query is such that it retrieves less than 20% of data. (eg select * from emp where gender = Male and 20% of your staff is Male so only 20% of rows are retrieved )

If you query retrieves more than 20% data then oracle decides to use full scan instead of indexes. Bitmap indexes are slightly better because oracle still decides to use them even if your query retrieves 40% of total number of rows.

If you query is such that it retrieves more than 40% of rows then oracle will skip bitmap and go for full scan. The advantage bitmap have is rowid are sorted so it know which blocks of data to easily pick


You should remember that bitmap stores only bit value with rowid and the rowid are sorted
  
Rowid
Value of row
Rowid 1
Rowid 2
Rowid 3
Rowid 4
Rowid  5
male
1
1
0
1
0
Female
0
0
0
0
1

When to use B tree or Bitmap indexes

The comparison is not straight forward and it depends on the distribution of data obviously . Like a table which is having emp id (which is unique) distributed randomly wil perform poorly with a B tree index for range scan because of clustering factor . Because of random distribution of data it has to fetch multiple blocks .

Note - the word range scan is used because for equality predicate anyway it has to fetch a single block so it does not make a difference whether bitmap or b tree index

So here is one more question so why this random distribution of data does not apply to bitmap index and it works well . Even bitmap has to get rowid and then go to table to fetch blocks with those row id what is different.

Contrary to popular belief consider a table with 1 million entries and a column for male and female with bitmap index on it. 1/2 million male  and 1/2 million female and you filter by = male will oracle use bitmap index. Most probably it will go for  full scan because even with bitmap it has to get rowid and then fetch those blocks from table

For B tree index oracle uses 5-20% rule that is the data retrieved is within 20% it will go for index otherwiser full table scan is good . Same is case for bitmap but the range is more may be 50%. But consider out of 1 million 800 thousand are males then oracle will prefer full scan instead of bitmap.

So even the male female example for bitmap is not 100% true. You need to understand the data.


Bitmap index are even applied for null values whereas b tree indexes are not applied


If there are two bitmap indexes they can be combined to reduce number of records and then the records can be fetched this is the concept of star transformation

Found some interesting articles on Oracle capabilities for your datawarehouse .How to improve performance

Oracle document on Datawarehouse query improvement

Will write later after implementation.Planning to implement solution

Good Article on Bitmap and B-Tree indexes


http://www.oracle.com/technetwork/articles/sharma-indexes-093638.html

Hints Like /*+ STAR TRANFORMATION */  has huge impact 

Point to be remembered -- Bitmap indexes work together in finding the row required. You cannot have the same columsn with bitmap index and same column with B tree index and compare performance. 

If you have bitmap index on all your dimension seq key in your fact then when you filtered by dimension it can combine those bitmap indexes in finding the correct row. In case of B tree indexes it cannot combine them to get correct row. The more bitmap indexes you have and the more filters you apply on those column result will be faster.

It combines result of each index for zooming to output. 

 
 

Firewall and Tunnelling basics

Hi Guys ,

Few days back i got a mail saying firewall are being replaced. I hardly understood what is a firewall.

Few basics that i knew 

I knew windows has firewall which does not allow you access to ports for someone trying to get access from other machine.Similarly also knew oracle communicate on 1515 port and My sql on some other port .Port 80 for http and firewall monitors these that it .With tunneling you can safely communicate with other machines using ports you like .Like sending http request on some other port so that you can access facebook from inside private networks (office/College) .Nothing much in detail

Below is some more understanding 

Check out this video.It tells you all that firewalls are capable of doing

http://www.youtube.com/watch?v=3_wGDeQOsDE

will write later when time permits 

Horizontal scroll bar for list with 3 columns Fixed

Hi Guys,

Below is a functionality that i am trying . A list with first 3 columns fixed and a scroll bar for remaining 20 columns.This can be achieved by using jquery UI .Data table functionality.

Will be updating details later

Saturday, 6 July 2013

Difference between Linux and Unix

Hi Guys


Linux is free operating system whereas Unix is not free.To understand this completely we need to understand we need some basics below.

Linux have some version which cost some money.Unix does not have any free versions.Unix is used mainly for servers ( Solaris (oracle) ,AIX (IBM) are unix based OS.

Linux is Unix like operating system.Unix was developed by Bell labortories(those that gave you phone and transistor) and linux is based on similar structure so its called Unix like.


An operating system (OS) is a collection of software that manages computer hardware resources and provides common services for computer programs.For hardware functions such as input and output and memory allocation, the operating system acts as an intermediary between programs and the computer hardware.

A Unix kernel — the core or key components of the operating system — consists of many kernel subsystems like process management, memory management, file management, device management and network management.

Its free - You get the code for kernel freely available apart from code from other stuff you can build your system on that

Popular example of systems built on linux are Android(For mobiles ).Red hat,Google chrome OS.

Some other points

Linux and unix are both know to be virus free hardly about 200 virus till date for both of them combined together

Unix OS is mainly used for servers , mainframes.There are different flavours of unix Solaris (Oracle) (Mostly being promoted for datacenters for clouds) ,AIX(IBM),and Apple OSX.

So what are flavours .We are not having ice Cream here ??.

The kernel remains the same and you can build on top of it as you like. Like for ice cream the base may be plain white milk and cream and you can add chocalate or strawberry flavour to it.

Linux for desktop pc.Mobiles (Androids)

Note:-They say using oracle solari as OS for datacentres(servers with oracle db will provide good scalability and great performance as its designed that way .

The black screen most datawarehousing guys see

Please remember telnet does not have a UI in unix .We use putty to connect to unix server(Solaris).For GUI to unix we can use WINSCP a free software.

Because we have command screen for telnet does not mean you have command screen for everything .It has a proper GUI only telnet is command screen.

Some other basic points


In 1969, UNIX was developed by a group of AT&T employees at Bell Labs and Dennis Ritchie. It was written in “C” language and was designed to be a portable, multi-tasking and multi-user system in a time-sharing configuration

Some Linux based OS

 Android,Ubuntu, Fedora, Red Hat, Debian, Archlinux, etc are all linux OS.Google chrome OS is linux based OS.

Some open source softwares

Many of open source softwares are Apache http sever ,JBOSS (its a application server like Apache but supports web services, and java beans), Mozilla , Mysql (database) .You can get the code for these things .


http://cognossimplified.blogspot.com/2013/08/linux-basics.html

Tuesday, 25 June 2013

Microstrategy for Cognos Developers

Hi Guys,

I am a Cognos developer.Recently i got a chance to make a POC in MSTR.It was for creation of dashboard.Below is my learning on how things are different and how to proceed

Disclaimer - These are my personal opinion based on my limited knowledge on MSTR.

Below are some screenshot of MSTR

Web MSTR  

(Its has a desktop installable Called MSTR Desktop which is not there in Cognos)



 Desktop in MSTR(Report Studio of Cognos)




MSTR Architect (Framework manager of Cognos)



Attributes in MSTR 

Prompts in MSTR




Cognos V/s Microstrategy

Hi Guys,

We have seen lot of post on web on Microstrategy V/S Cognos, Saying why Microstrategy is better and all that .Now recently i got a chance to make a POC in microstrategy .

In involves creation on dashboard in MSTR.So i learned a lot from that.What is different in MSTR from Cognos .How it works and What are some of feature missing in it from Cognos .

Disclaimer - Below thought are my personal opinion based on my limited Knowledge of MSTR.

I will also be posting a article titles .MSTR for Cognos developers .

Below are some of the points 

FM in Cognos <---> Architect in MSTR

Major Disadvantages

Issue 1

Architect assumes that you have a star schema model in your database from which you are making  your report.Consider you have OLTP system and you are making report on that it will be next to impossible to do so in MSTR without going for cubes

Now i am not in favour of cubes due to their maintainance issues. Cognos allows you do all types of join in FM very very easily .Consider simple left outer join.Its very difficult to achieve in MSTR.

Issue 2 

MSTR will create temporary tables in your schema.you need to have a schema where you have create table priviledges.So you should have Schema A from where you access your main schema B .Because in schema A you will need priviledges to create tables .

Whereas in cognos you do not require write priviledges on your schema .All optimisation is handled by cognos server .

So MSTR will suggest to go for Cubes in order to improve performance in case of large data.

Issue 3

MSTR has something called Free Form sql .It allows you to write your own sql.But that is also very limited.You can have prompts in this but it does not allow you to change prompt values once report is run.

Consider below scenario .You have a cognos report in which you have a prompt on report page and not on prompt page .Like product name .You want to change the product name once report is run to see a different product .

In MSTR you cant pass prompt value to SQL again now becaue it has already retrieved teh data.Only way to do this is to rerun the report.Very sad .


Issue 4 

MSTR cannot handle factless fact queries . In Datawarehousing a big disadvantage .Consider you need to pull data from two dimensions without involving any fact .Then you cant do it in MSTR without involving a fact

I wont say MSTR is great for making reports .But it has very Good Dashboarding Capabilityand i am assuming that you are using cubes are source for those  dashboards.Good performance and a Lot of options


Now having said this Some Good points on MSTR

1) Its the King of dashboarding .Cognos has a limited dashboarding capability.Panels are fixed.In MSTR you can actually draw your dashboard how you want it like in paint.

2) It has lot of widgets for dashboard Fisheye , Timelines .Those capabilities are not there in cognos dashboards



I will continue to update more points as i come across .Do read my post on Microstrategy for Cognos developers .Explains components in MSTR and their Counterparts in Cognos




1) Bullet chart not available directly in MSTR.Need to do a lot of formating to make them appear as bullets

2) Left outer joins are not availabel directly in architect.

3) Does not do any server processing

4)Creates tables in oracle schema intermediate tables you should have create priviledges

5) The way it handles factless fact queries.If a fact is not there in a query it will give a crossjoin


6) you cannot have a prompt on report page .All prompts have to submitted before a report is run .you cant do any prompt based filtering once report is run 

Tuesday, 14 May 2013

Difference between Apache and Apace tomcat

Hi Guys,

Apache and Apache Tomcat are two different servers.

Apache server is used for static HTML content ,means someone else does the processing.Like you have cognos application server which does the processing of reports and sends html content to apache for displaying .It is not a java virtual machine (JVM)

So what is virtual machine .Consider you have windows on your laptop and you want to run a program which runs only in unix .Now you dont want to install 2 separate OS on your machine .So you install a program which lets you install one more OS .So it will be like a separate pop up in which you see a different OS.so you have windows and in that VMware in that unix .

Most cloud based machines use virtual machines.

So Coming back to JVM  (sorry for deviating from topic)

So JVM is a environment where you can run your java code which you cant run directly on windows because it does not have the files so you either dowload JRE in programs running on windows ( Java run time environment) or for web based have a web server which has JVM like apache tomcat.