Hi
Covering some basiscs of unstructured data annd its challenges
Covering some basiscs of unstructured data annd its challenges
Saturday, 3 August 2013
Databases -Relational and Non relational
Hi Guys,
We have worked on relational database (oracle , sql server , mysql , teradata) .So with the coming of hadoop into picture i started analysis no relational databases.Non relatonal databases are used for storin unstructured data . Like your windows file system c drive , d drive and all
We can store email in our database stables but then our databases table will serve as file systems .
I will write more on this topic as i get time.
We have worked on relational database (oracle , sql server , mysql , teradata) .So with the coming of hadoop into picture i started analysis no relational databases.Non relatonal databases are used for storin unstructured data . Like your windows file system c drive , d drive and all
We can store email in our database stables but then our databases table will serve as file systems .
I will write more on this topic as i get time.
Basics about Servers
Hi Guys,
Since most of us dont make it to server rooms daily .so here are some basics about server.We do use them daily. but we dont know much
details
I will format this article later.But do go through you tube videos by ELI .they are 60 minute long but very very good
Basics of servers
http://www.youtube.com/watch?v=CDxaRfwzFrs
Server Hardware Good Video
http://www.youtube.com/watch?v=QYzJl0Zrc4M
Basically any PC can act as a server.But here we are discussing enterprise level servers which are more robust and are built not to
be easily stopped.Consider this like even if the power supply burns .If hard disks fail ,If RAMS fail,If Fans fail.It has redundency
built for all of these scenarios.
When you buy window server edition software it has certain services like exchange server(MS outlook) ,VOIP server software built in
it.It has different set of options then your windows desktop software.
Consider a Linux server.it will just give you a terminal screen (black screen) and you need to navigate only using commands.Dont
confuse it with ubuntu a desktop OS supplied by linus which has GUI.Linux servers are very very stable and will go on for years
without even requiring reboot.
It wont have issues of memory leaks like windows server which would require a reboot.
Some general Hardware of servers(Not absolute necessary)
server has ECC ram ( not usually neccessary).It checks RAM status on start up
xeon processor
redundant power supply
RAID- ( redundant array of integrated disk).It like multiple hard disk connected together which are hot swappable that is they can be
removed when server is running.Check the below video
SAN -- storage area network will is like RAID but we are using physical boxes.data is redundant that is stored inmulitple location
RAID
http://www.youtube.com/watch?v=X1x9EMd5ywY
what is esata... It a type of external sata (Generally our hard disk are SATA) .I have a article with basics of some hardware.Check
august articles
Some casing
1 u server cases--Check out the images at end of articles.Just to give idea how things look
4 u server cases---Check out the images at end of articles.Just to give idea how things look
ATX case--- Desktop PC case
PCI slots ??? for RAID cards
ITX cases
-------------------------------------------
http://www.youtube.com/watch?v=Kiftbm1L_eQ
virtualization of infrastructure
------------------------------------
Read article on cloud computing to learn more about virtulization
Virtulization software like XEN make it possible for virutalization. so you have one server( any PC) on that instead of windows you
have XEN there is one more PC which acts as your XEN Master machine which know the status of your server.
So suppose your server fails it will migrate the application to other server which now how it does it(Eg ESXI (slave) and Vsphere
(Management softwaer on other machine).Basically when you install any virtualization software it installs operating system on that
software so basically its like a file sitting on that software it can directly copy paste that file.That is you can migrate a OS
directly like a music file.
-------------------------------------
How DB,Webserver,Active directory servers are managed
The idea of separating out instances under different operating system is to make sure that you have less ports as possible
Generally your webserver , active directory and db server will not sit on single machine(virtual machine).which makes it more secure
if one of the Virtual machine is hacked.The db server can communicate only on single port DB port and only to application server.So
you cant sent it a http request to db server.
Like database server how firewall is set up
we have a apache on different machine and mysql on other machine both are virtual machines we have set up the machine where we have
mysql database in such a manner taht it communicates only on port 1512 say(db port) and not on 80. So the security can be easily
controlled as outside hacker cannot get to this server by other port
-----------------------------------------
How server failure is managed
then we have number of physical server connected so in case of failure.so with server management software like a vm sphere
automatically it will bring up the server which is in hibernate software whenver load is there suppose for active directory
There are two Types of RAID --Hardware and software .
In hardware Raid there is a raid controller so for operating system disk appear as if they are single disk
Software raid can be configure with linux and windows server.
RAID isn’t just a single way of combining disks. There are multiple RAID levels that provide different levels of performance and redundancy. All RAID levels have one thing in common: they combine multiple physical disks into a single logical disk that is presented to the operating system.
http://www.scottklarr.com/topic/23/how-raid-5-really-works/
RAID 5 in detail
RAID 5 provides a very redundant fault tolerance in addition to performance advantages allowing data to be safeguarded while only
sacrificing the equivalent of one drive's space. RAID-5 requires at least three hard drives of the same size; The total storage space
available with a RAID-5 array is equal to { (number of drives - 1) * size of smallest drive }. So if you use three 120gb hard drives,
you will have 240gb of actual usable space. If you use five 120gb hard drives, you would have 480gb of usable space. The more drives
you use, the more efficient your storage space becomes without losing any redundancy
Your data can survive a complete failure of one hard drive, however if two drives fail at the same time, ALL data will be lost. It is
very important to have an extra drive on hand so if a drive fails, you can replace it immediately for data rebuild. The RAID-5 array
can actually still be used with one drive completely missing or not working, but performance is degraded as the data must be rebuilt
on the fly. However, if you do not have an extra drive to plug in right away when one fails, it would be wise to keep the computer
and all drives powered off until you can replace the failed drive. You may think, "oh it will only be a couple days before the new
drive arrives," but ask yourself this: Is not having access to the data on these drives for only a couple days worse than taking the
risk of losing it all forever if another drive happens to fail? Probably not.
striping & Parity
Data is "striped" across the hard drives, with a dedicated parity block for each stripe. A, B, C, and D represent data "stripes."
Each stripe segment per drive can vary in size; I believe anywhere from 4kb to 256kb per stripe is normal and can be set during setup
to adjust performance. The blocks with a subscript P are the parity blocks which are a representation of the sum of all other blocks
in that stripe (explained in more detail below). The parity is responsible for the data fault tolerance and is also the reason why
you lose the amount of space equivalent to one drive. Taking notice of figure 1, let's say that the second drive fails. When a new
hard drive is put in its place the RAID controller would rebuild the data automatically. The data in segments A1 and A3 would be
compared to the AP parity block, which would allow the data for A2 to be rebuilt. This would take place on each stripe until the
entire drive is up to speed, so to speak. Parity blocks are determined by using a logical comparison called XOR (Exclusive OR) on
binary blocks of data which will be explained down further.
Performance
RAID 5 offers accelerated read performance because the data stream is accessed from multiple drives at the same time. Referring to
figure 1, let's say that stripe A was a single file. Normally on a single drive when you open that file, the whole thing would be
streamed from the one hard drive bit by bit - thus the one hard drive's max read speed is going to become a bottleneck. BUT, with a
RAID-5, that one file can be accessed in 1/3 of the time because it will be read from all 3 drives at once; block 1 has the first 1/3
of the file, block 2 has the second 1/3 section of the file, and the block 3 has the last part of the file. This, in a perfect
situation, causes your read speed to be tripled - with even more performance potential in RAID-5 arrays containing additional hard
drives!
The downfall to this is that there is an increased overhead when writing to the drives caused from parity calculation. Every single
bit written to the drives must be compared and processed to create a parity block. If your intended use involves a lot of data
writing (such as video recording, high traffic server, etc) RAID-5 would not be the most ideal choice.
XOR Comparison
Data is stored and processed at the very lowest levels in the form of binary which is of course 0s and 1s. There are methods of
comparing binary bits called operators. The one that does the magic of parity creation is called XOR, or Exclusive OR. If you have
experience in lower level programming or electronics, you probably already know what an XOR is
Basically, an XOR comparison will take two binary bits, compare them, and output a result of 0 or 1. It will return a 1 ONLY IF the
two inputs are different. If both bits are 0, the output is 0; If both bits are 1, the output is 0; If one bit is 0 and the other bit
is 1, the output is 1.
parity is distributed acroos drives so that there is no single disk which handles parity so if parity disk fails than what
Xor is exclusively or
00 0
01 1
10 1
11 0
So even if a drive fails we can rebuild the data
(Drive 1) XOR (Drive 2) = (0100) XOR (0101) = (0001)
(Result) XOR (Drive 3) = (0001) XOR (0010) = (0011)
Recovering data
(Drive 2) XOR (Drive 3) = (0101) XOR (0010) = (0111)
(Result) XOR (Drive 4) = (0111) XOR (0011) = (0100)
Since most of us dont make it to server rooms daily .so here are some basics about server.We do use them daily. but we dont know much
details
I will format this article later.But do go through you tube videos by ELI .they are 60 minute long but very very good
Basics of servers
http://www.youtube.com/watch?v=CDxaRfwzFrs
Server Hardware Good Video
http://www.youtube.com/watch?v=QYzJl0Zrc4M
Basically any PC can act as a server.But here we are discussing enterprise level servers which are more robust and are built not to
be easily stopped.Consider this like even if the power supply burns .If hard disks fail ,If RAMS fail,If Fans fail.It has redundency
built for all of these scenarios.
When you buy window server edition software it has certain services like exchange server(MS outlook) ,VOIP server software built in
it.It has different set of options then your windows desktop software.
Consider a Linux server.it will just give you a terminal screen (black screen) and you need to navigate only using commands.Dont
confuse it with ubuntu a desktop OS supplied by linus which has GUI.Linux servers are very very stable and will go on for years
without even requiring reboot.
It wont have issues of memory leaks like windows server which would require a reboot.
Some general Hardware of servers(Not absolute necessary)
server has ECC ram ( not usually neccessary).It checks RAM status on start up
xeon processor
redundant power supply
RAID- ( redundant array of integrated disk).It like multiple hard disk connected together which are hot swappable that is they can be
removed when server is running.Check the below video
SAN -- storage area network will is like RAID but we are using physical boxes.data is redundant that is stored inmulitple location
RAID
http://www.youtube.com/watch?v=X1x9EMd5ywY
what is esata... It a type of external sata (Generally our hard disk are SATA) .I have a article with basics of some hardware.Check
august articles
Some casing
1 u server cases--Check out the images at end of articles.Just to give idea how things look
4 u server cases---Check out the images at end of articles.Just to give idea how things look
ATX case--- Desktop PC case
PCI slots ??? for RAID cards
ITX cases
-------------------------------------------
http://www.youtube.com/watch?v=Kiftbm1L_eQ
virtualization of infrastructure
------------------------------------
Read article on cloud computing to learn more about virtulization
Virtulization software like XEN make it possible for virutalization. so you have one server( any PC) on that instead of windows you
have XEN there is one more PC which acts as your XEN Master machine which know the status of your server.
So suppose your server fails it will migrate the application to other server which now how it does it(Eg ESXI (slave) and Vsphere
(Management softwaer on other machine).Basically when you install any virtualization software it installs operating system on that
software so basically its like a file sitting on that software it can directly copy paste that file.That is you can migrate a OS
directly like a music file.
-------------------------------------
How DB,Webserver,Active directory servers are managed
The idea of separating out instances under different operating system is to make sure that you have less ports as possible
Generally your webserver , active directory and db server will not sit on single machine(virtual machine).which makes it more secure
if one of the Virtual machine is hacked.The db server can communicate only on single port DB port and only to application server.So
you cant sent it a http request to db server.
Like database server how firewall is set up
we have a apache on different machine and mysql on other machine both are virtual machines we have set up the machine where we have
mysql database in such a manner taht it communicates only on port 1512 say(db port) and not on 80. So the security can be easily
controlled as outside hacker cannot get to this server by other port
-----------------------------------------
How server failure is managed
then we have number of physical server connected so in case of failure.so with server management software like a vm sphere
automatically it will bring up the server which is in hibernate software whenver load is there suppose for active directory
There are two Types of RAID --Hardware and software .
In hardware Raid there is a raid controller so for operating system disk appear as if they are single disk
Software raid can be configure with linux and windows server.
RAID isn’t just a single way of combining disks. There are multiple RAID levels that provide different levels of performance and redundancy. All RAID levels have one thing in common: they combine multiple physical disks into a single logical disk that is presented to the operating system.
- RAID 0: Unlike other RAID levels, RAID 0 provides no redundancy. However, RAID 0 allows you to increase performance using multiple disks. When you use RAID 0, data your computer writes to a hard disk is split across two (or more) hard drives evenly. For example, if your computer writes a 100MB file, 50MB will be written to one hard drive and 50MB will be written to the other hard drive. When the computer needs to read the file back, it can read 50MB from one hard drive and 50MB from the other hard drive at the same time — this will be faster than reading 100MB from a single hard drive. However, if any of the hard drives in the RAID array dies, you’ll lose your data. When you use RAID 0, your multiple disks appear to be a larger and faster hard disk — but they’re much more fragile.
- RAID 1: In RAID 1, two disks are configured to mirror each other. When your computer writes 100MB of data to its disks, it will write the same 100MB to both hard disks. Each disk contains a complete copy of the data. This ensures that, if one of the disks ever fails, you will always have a complete, up-to-date copy of your data.
- RAID 2, 3, and 4: These RAID levels are little-used and often considered obsolete.
- RAID 5: To use RAID 5, you will need at least three disks. RAID 5 uses striping to divide data across all hard drives, with additional parity data divided across all disks. If one of the hard drives dies, you won’t lose any of your data. RAID 5 offers data redundancy with less storage cost than RAID 1 — for example, if you had four 1TB hard drives, you could create two separate RAID 1 arrays (1TB each for a total of 2TB storage space) or a single RAID 5 array with 3TB of storage space.
- RAID 6: RAID 6 is similar to RAID 5, but adds an additional parity block, writing two parity blocks for each bit of data striped across the disks. You lose storage capacity, but RAID 6 provides additional protection from data loss. For example, if two hard drives die in a RAID 5 configuration, you’ll lose your data. If two hard drives die in a RAID 6 configuration, you’ll still have all your data.
- RAID 10: Also known as RAID 1+0, RAID 10 divides data between primary disks and mirrors this data to secondary disks. In this way, it attempts to provide the advantages of RAID 0 (dividing data across multiple disks for a performance increase) with the advantages of RAID 1 (redundancy).
http://www.scottklarr.com/topic/23/how-raid-5-really-works/
RAID 5 in detail
RAID 5 provides a very redundant fault tolerance in addition to performance advantages allowing data to be safeguarded while only
sacrificing the equivalent of one drive's space. RAID-5 requires at least three hard drives of the same size; The total storage space
available with a RAID-5 array is equal to { (number of drives - 1) * size of smallest drive }. So if you use three 120gb hard drives,
you will have 240gb of actual usable space. If you use five 120gb hard drives, you would have 480gb of usable space. The more drives
you use, the more efficient your storage space becomes without losing any redundancy
Your data can survive a complete failure of one hard drive, however if two drives fail at the same time, ALL data will be lost. It is
very important to have an extra drive on hand so if a drive fails, you can replace it immediately for data rebuild. The RAID-5 array
can actually still be used with one drive completely missing or not working, but performance is degraded as the data must be rebuilt
on the fly. However, if you do not have an extra drive to plug in right away when one fails, it would be wise to keep the computer
and all drives powered off until you can replace the failed drive. You may think, "oh it will only be a couple days before the new
drive arrives," but ask yourself this: Is not having access to the data on these drives for only a couple days worse than taking the
risk of losing it all forever if another drive happens to fail? Probably not.
striping & Parity
Data is "striped" across the hard drives, with a dedicated parity block for each stripe. A, B, C, and D represent data "stripes."
Each stripe segment per drive can vary in size; I believe anywhere from 4kb to 256kb per stripe is normal and can be set during setup
to adjust performance. The blocks with a subscript P are the parity blocks which are a representation of the sum of all other blocks
in that stripe (explained in more detail below). The parity is responsible for the data fault tolerance and is also the reason why
you lose the amount of space equivalent to one drive. Taking notice of figure 1, let's say that the second drive fails. When a new
hard drive is put in its place the RAID controller would rebuild the data automatically. The data in segments A1 and A3 would be
compared to the AP parity block, which would allow the data for A2 to be rebuilt. This would take place on each stripe until the
entire drive is up to speed, so to speak. Parity blocks are determined by using a logical comparison called XOR (Exclusive OR) on
binary blocks of data which will be explained down further.
Performance
RAID 5 offers accelerated read performance because the data stream is accessed from multiple drives at the same time. Referring to
figure 1, let's say that stripe A was a single file. Normally on a single drive when you open that file, the whole thing would be
streamed from the one hard drive bit by bit - thus the one hard drive's max read speed is going to become a bottleneck. BUT, with a
RAID-5, that one file can be accessed in 1/3 of the time because it will be read from all 3 drives at once; block 1 has the first 1/3
of the file, block 2 has the second 1/3 section of the file, and the block 3 has the last part of the file. This, in a perfect
situation, causes your read speed to be tripled - with even more performance potential in RAID-5 arrays containing additional hard
drives!
The downfall to this is that there is an increased overhead when writing to the drives caused from parity calculation. Every single
bit written to the drives must be compared and processed to create a parity block. If your intended use involves a lot of data
writing (such as video recording, high traffic server, etc) RAID-5 would not be the most ideal choice.
XOR Comparison
Data is stored and processed at the very lowest levels in the form of binary which is of course 0s and 1s. There are methods of
comparing binary bits called operators. The one that does the magic of parity creation is called XOR, or Exclusive OR. If you have
experience in lower level programming or electronics, you probably already know what an XOR is
Basically, an XOR comparison will take two binary bits, compare them, and output a result of 0 or 1. It will return a 1 ONLY IF the
two inputs are different. If both bits are 0, the output is 0; If both bits are 1, the output is 0; If one bit is 0 and the other bit
is 1, the output is 1.
parity is distributed acroos drives so that there is no single disk which handles parity so if parity disk fails than what
Xor is exclusively or
00 0
01 1
10 1
11 0
So even if a drive fails we can rebuild the data
(Drive 1) XOR (Drive 2) = (0100) XOR (0101) = (0001)
(Result) XOR (Drive 3) = (0001) XOR (0010) = (0011)
Recovering data
(Drive 2) XOR (Drive 3) = (0101) XOR (0010) = (0111)
(Result) XOR (Drive 4) = (0111) XOR (0011) = (0100)
Thursday, 1 August 2013
File systems
Hi
Wanted to cover file systems
NTFS, ext3 , ext4,XFS .How to do and details
a file can be video audio file
it can be formated text
most opersation system has folder directories
drive lettes
how is data writen into storage media is the
thing that operating system manages
EXT2,EXT3 ,NTFS , FAT, HPFS
role of OS is to define API for file access and
the structure (directories names)
OS also provides drivers for physical access
file is a name for group of data.This group of
data
also has attribute like owners, permissions
date/times etc
structure of file can be explicit or implicit
like .exe it executible file and .txt is text
file can have access type like direct sequential
etc
file can have performance caching , organization
,memory management
file system also has directories management like
hierarchical form.
some file system can be across multipl physical
drives
like linux allows you to mount a drive so that
the directory sturcure remains the same
not always structure is hierarchical .like
mainframes the data is separated by using
abc.kkk.aaa.kklkl
reference paths ..
how data is saved on disk
. the OS always know where the root directory is
so its always saved on some posistion
so OS can decide where to start from
so how space is allocated .. usually a fixed
allocation of blocks , there is a directory that
contains a list of blocks for a file
OS access method masks use of non contiguos block
this is called fragmentaion
see how data is saved on ms block
file name 8 ext 3 attribut 1 reserved 10 time2
date 2 1st block 1 and size 4
FAT file allocation table -- the location of
first block will be kept in file allocation table
in windows 98.. the same structure FAT-32 this
was because of larger disks
base 8 ext3 attribute 1 ,NT 1,Sec 1 , creation
3,last accessed 2 , starting block 2 , last write
4 starting block 15.31 , file size 4
in unix i node number2 file name 14
Wanted to cover file systems
NTFS, ext3 , ext4,XFS .How to do and details
a file can be video audio file
it can be formated text
most opersation system has folder directories
drive lettes
how is data writen into storage media is the
thing that operating system manages
EXT2,EXT3 ,NTFS , FAT, HPFS
role of OS is to define API for file access and
the structure (directories names)
OS also provides drivers for physical access
file is a name for group of data.This group of
data
also has attribute like owners, permissions
date/times etc
structure of file can be explicit or implicit
like .exe it executible file and .txt is text
file can have access type like direct sequential
etc
file can have performance caching , organization
,memory management
file system also has directories management like
hierarchical form.
some file system can be across multipl physical
drives
like linux allows you to mount a drive so that
the directory sturcure remains the same
not always structure is hierarchical .like
mainframes the data is separated by using
abc.kkk.aaa.kklkl
reference paths ..
how data is saved on disk
. the OS always know where the root directory is
so its always saved on some posistion
so OS can decide where to start from
so how space is allocated .. usually a fixed
allocation of blocks , there is a directory that
contains a list of blocks for a file
OS access method masks use of non contiguos block
this is called fragmentaion
see how data is saved on ms block
file name 8 ext 3 attribut 1 reserved 10 time2
date 2 1st block 1 and size 4
FAT file allocation table -- the location of
first block will be kept in file allocation table
in windows 98.. the same structure FAT-32 this
was because of larger disks
base 8 ext3 attribute 1 ,NT 1,Sec 1 , creation
3,last accessed 2 , starting block 2 , last write
4 starting block 15.31 , file size 4
in unix i node number2 file name 14
Wednesday, 31 July 2013
Service oriented Architecture (SOA)
Hi
http://www.youtube.com/watch?v=0hyXOuvyq2Q&list=TLsaIHJaZiiPY
http://www.youtube.com/user/fourthmonthllc?feature=watch
Basic of SOA
Simple SOA - Mainframe( has data , business logic and UI , graphics ) and dummy terminal .But issue is scalability
Later came the PC where validation can be done by browser , like validating date this is client server .But updation on all the machine became difficult as the 1000 user updation in an office network will be difficult.
then the client/server gave way to 3 tier architecture A browser a application server and database server.
Now even will 3 tier architectue it becomes difficult to manage so came teh service oriented architecture
So the db , application service and SAP , oracle
first there were function
then there were object and clasees
issue was .net and java were not able to interact
then people started talking in terms of messages
xml messages.. so messages were standedise
then with it camse services
services are logical encapsulation of self contained
business functionality
services communicates in termsn of messages
-----------------------------
few of points that services should be able to do are
services should be to self define themselves
services should be maintained in repository where we can find them
we should be able to orchestrate services that is
we should be able to design our workflow and plac services in them
consider example of security service and order processing services
we shoud be able to first use security service and then order process
that is check if user is valid and them do order processing
or we shoudl be able to process the order first and then do security check
so the order of service can change
But consider the predominant n-tier development environment in use today -- Java EE:
presentation implemented with JSP/servlets
business logic implemented with POJOs or EJBs
data access logic implemented with JPA/Hibernat
first there were function
then there were object and clasees
issue was .net and java were not able to interact
then people started talking in terms of messages
xml messages.. so messages were standedise
then with it camse services
services are logical encapsulation of self contained
business functionality
services communicates in termsn of messages
-----------------------------
few of points that services should be able to do are
services should be to self define themselves
services should be maintained in repository where we can find them
we should be able to orchestrate services that is
we should be able to design our workflow and plac services in them
consider example of security service and order processing services
we shoud be able to first use security service and then order process
that is check if user is valid and them do order processing
or we shoudl be able to process the order first and then do security check
so the order of service can change
But consider the predominant n-tier development environment in use today -- Java EE:
presentation implemented with JSP/servlets
business logic implemented with POJOs or EJBs
data access logic implemented with JPA/Hibernat
first there were function
then there were object and clasees
issue was .net and java were not able to interact
then people started talking in terms of messages
xml messages.. so messages were standedise
then with it camse services
services are logical encapsulation of self contained
business functionality
services communicates in termsn of messages
-----------------------------
few of points that services should be able to do are
services should be to self define themselves
services should be maintained in repository where we can find them
we should be able to orchestrate services that is
we should be able to design our workflow and plac services in them
consider example of security service and order processing services
we shoud be able to first use security service and then order process
that is check if user is valid and them do order processing
or we shoudl be able to process the order first and then do security check
so the order of service can change
But consider the predominant n-tier development environment in use today -- Java EE:
presentation implemented with JSP/servlets
business logic implemented with POJOs or EJBs
data access logic implemented with JPA/Hibernat
Amazon web sevices alllows for
computer , storage and database services
http://www.youtube.com/watch?v=DERzYnthq1s
what is webservice
we are callign a function which is inside a software from other software
software which gives the call is called client and software which receives the request is called server
both these things can be writen in different languages client can be in .net and server function can be in java
in this the format of input and output messages is important
there is 2 ways a web service can be used either the severs provides WSDL file or there is a UDDI universal description discovery and integration on the internet where all service providers register their service and the whoever want can reach UDDI which will list all similar services with their WSDL file
there are RESTful web services using rest protocol and SOAP web services
RESTful web services are accessed by passing URI from the webserve
server would send teh representation fo that service to client this can be in XML
REST ful webservices uses HTTP. they use 4 main http methods
GET retrieve a resource
POST create a resource
PUT update a resource
DELETE deletes a resource
http://www.youtube.com/watch?v=0hyXOuvyq2Q&list=TLsaIHJaZiiPY
http://www.youtube.com/user/fourthmonthllc?feature=watch
Basic of SOA
Simple SOA - Mainframe( has data , business logic and UI , graphics ) and dummy terminal .But issue is scalability
Later came the PC where validation can be done by browser , like validating date this is client server .But updation on all the machine became difficult as the 1000 user updation in an office network will be difficult.
then the client/server gave way to 3 tier architecture A browser a application server and database server.
Now even will 3 tier architectue it becomes difficult to manage so came teh service oriented architecture
So the db , application service and SAP , oracle
first there were function
then there were object and clasees
issue was .net and java were not able to interact
then people started talking in terms of messages
xml messages.. so messages were standedise
then with it camse services
services are logical encapsulation of self contained
business functionality
services communicates in termsn of messages
-----------------------------
few of points that services should be able to do are
services should be to self define themselves
services should be maintained in repository where we can find them
we should be able to orchestrate services that is
we should be able to design our workflow and plac services in them
consider example of security service and order processing services
we shoud be able to first use security service and then order process
that is check if user is valid and them do order processing
or we shoudl be able to process the order first and then do security check
so the order of service can change
But consider the predominant n-tier development environment in use today -- Java EE:
presentation implemented with JSP/servlets
business logic implemented with POJOs or EJBs
data access logic implemented with JPA/Hibernat
first there were function
then there were object and clasees
issue was .net and java were not able to interact
then people started talking in terms of messages
xml messages.. so messages were standedise
then with it camse services
services are logical encapsulation of self contained
business functionality
services communicates in termsn of messages
-----------------------------
few of points that services should be able to do are
services should be to self define themselves
services should be maintained in repository where we can find them
we should be able to orchestrate services that is
we should be able to design our workflow and plac services in them
consider example of security service and order processing services
we shoud be able to first use security service and then order process
that is check if user is valid and them do order processing
or we shoudl be able to process the order first and then do security check
so the order of service can change
But consider the predominant n-tier development environment in use today -- Java EE:
presentation implemented with JSP/servlets
business logic implemented with POJOs or EJBs
data access logic implemented with JPA/Hibernat
first there were function
then there were object and clasees
issue was .net and java were not able to interact
then people started talking in terms of messages
xml messages.. so messages were standedise
then with it camse services
services are logical encapsulation of self contained
business functionality
services communicates in termsn of messages
-----------------------------
few of points that services should be able to do are
services should be to self define themselves
services should be maintained in repository where we can find them
we should be able to orchestrate services that is
we should be able to design our workflow and plac services in them
consider example of security service and order processing services
we shoud be able to first use security service and then order process
that is check if user is valid and them do order processing
or we shoudl be able to process the order first and then do security check
so the order of service can change
But consider the predominant n-tier development environment in use today -- Java EE:
presentation implemented with JSP/servlets
business logic implemented with POJOs or EJBs
data access logic implemented with JPA/Hibernat
Amazon web sevices alllows for
computer , storage and database services
http://www.youtube.com/watch?v=DERzYnthq1s
what is webservice
we are callign a function which is inside a software from other software
software which gives the call is called client and software which receives the request is called server
both these things can be writen in different languages client can be in .net and server function can be in java
in this the format of input and output messages is important
there is 2 ways a web service can be used either the severs provides WSDL file or there is a UDDI universal description discovery and integration on the internet where all service providers register their service and the whoever want can reach UDDI which will list all similar services with their WSDL file
there are RESTful web services using rest protocol and SOAP web services
RESTful web services are accessed by passing URI from the webserve
server would send teh representation fo that service to client this can be in XML
REST ful webservices uses HTTP. they use 4 main http methods
GET retrieve a resource
POST create a resource
PUT update a resource
DELETE deletes a resource
What is Hadoop & BIG data
Below is a good link
http://readwrite.com/2013/05/23/hadoop-what-it-is-and-how-it-works#awesm=~odc8IiyozQvfDZ
also can check out on you tube
apche HDFS , mapreduce , apache hbase
Below is article by Akash Mitra . I liked it
http://www.dwbiconcepts.com/data-warehousing/18-dwbi-basic-concepts/176-what-is-big-data.html
http://www.dwbiconcepts.com/data-warehousing/18-dwbi-basic-concepts/2-map-reduce.html
Vijay Thakorals Blog on Hadoop Name node and its details
http://vijayjt.blogspot.in/2013/02/hadoop-namenode-web-interface.html
Good article on Hadoop from Yahoo developers site
http://developer.yahoo.com/
Mongo db tutorial videos on youtube
http://www.youtube.com/watch?v=bVRqd8mnQ6c&list=PLw2e3dFxewkIS1YjkLcdCUI5BPBg_YMwD
---------------------
GFS -- white paper
Mapreduce --
Big table -- these are google whitepaper
---------------------------------
To read 1 terabyte of data from one machine which
has 4 hard drives so 4 i/o channels each channel
has 100M/B per second speed
240000mb per minute 24gb per minute s0 240 gb
every 10 minutes so 43 or so minutes to read 1
terabyte of data
--------------------------------
Now hadoop allows you to spread that across 10
machines so to read 1 terabyte it takes 4.5
minutes
----------------------------------
name node is the master of file system
and data node is the slave
--------------------------------------
name node has metadata for data like which block
the data is broken into and where the file
resides
-------------------------------
multiple data nodes are having a copy of file so
single file is stored on multiple nodes
-------------------------
name node has a webserver which gives the
information like how many webnodes makes
theserver
---------------------------
what is the format of data in HDFS .like we
usually format our harddisk to ntfs
---------------------------------
for hadoop -- we have ext3 , ext4 ,XFS
------------------------------------
your command will depends on file system you use
like command prompt.
-----------------------------------
how does your racks work like core switches.
Each data node is kept in rack.
------------------------------------
Few thing to note about Hadoop
Hadoop can be deployed on raid configured drives.Raid is just configuring hard drives it not files system .Hadoop can be also deployed on cassandra file system
Apache Hbase -- Big table
Apache Flume -- RDBMS connect to hadoop
OOzie -- used for scheduling
HUE -- graphical interface for hadoop -- all these are offering by cloudera
PIG and Hive i am not sure that they do .
Latency -- it is the measure of time delay
Throughput -- amount of work a computer can do in given time frame.
Hadoop is designed for through put and not latency.Its like a train and not a sports car.It can pull huge data and you can notice it going only when data is really big .
http://readwrite.com/2013/05/23/hadoop-what-it-is-and-how-it-works#awesm=~odc8IiyozQvfDZ
also can check out on you tube
apche HDFS , mapreduce , apache hbase
Below is article by Akash Mitra . I liked it
http://www.dwbiconcepts.com/data-warehousing/18-dwbi-basic-concepts/176-what-is-big-data.html
http://www.dwbiconcepts.com/data-warehousing/18-dwbi-basic-concepts/2-map-reduce.html
Vijay Thakorals Blog on Hadoop Name node and its details
http://vijayjt.blogspot.in/2013/02/hadoop-namenode-web-interface.html
Good article on Hadoop from Yahoo developers site
http://developer.yahoo.com/
Mongo db tutorial videos on youtube
http://www.youtube.com/watch?v=bVRqd8mnQ6c&list=PLw2e3dFxewkIS1YjkLcdCUI5BPBg_YMwD
---------------------
GFS -- white paper
Mapreduce --
Big table -- these are google whitepaper
---------------------------------
To read 1 terabyte of data from one machine which
has 4 hard drives so 4 i/o channels each channel
has 100M/B per second speed
240000mb per minute 24gb per minute s0 240 gb
every 10 minutes so 43 or so minutes to read 1
terabyte of data
--------------------------------
Now hadoop allows you to spread that across 10
machines so to read 1 terabyte it takes 4.5
minutes
----------------------------------
name node is the master of file system
and data node is the slave
--------------------------------------
name node has metadata for data like which block
the data is broken into and where the file
resides
-------------------------------
multiple data nodes are having a copy of file so
single file is stored on multiple nodes
-------------------------
name node has a webserver which gives the
information like how many webnodes makes
theserver
---------------------------
what is the format of data in HDFS .like we
usually format our harddisk to ntfs
---------------------------------
for hadoop -- we have ext3 , ext4 ,XFS
------------------------------------
your command will depends on file system you use
like command prompt.
-----------------------------------
how does your racks work like core switches.
Each data node is kept in rack.
------------------------------------
Few thing to note about Hadoop
Hadoop can be deployed on raid configured drives.Raid is just configuring hard drives it not files system .Hadoop can be also deployed on cassandra file system
Apache Hbase -- Big table
Apache Flume -- RDBMS connect to hadoop
OOzie -- used for scheduling
HUE -- graphical interface for hadoop -- all these are offering by cloudera
PIG and Hive i am not sure that they do .
Latency -- it is the measure of time delay
Throughput -- amount of work a computer can do in given time frame.
Hadoop is designed for through put and not latency.Its like a train and not a sports car.It can pull huge data and you can notice it going only when data is really big .
Subscribe to:
Posts (Atom)