Wednesday, March 23, 2011

LDAP

What is LDAP ?
  • Lightweight Directory Access Protocol
  • Based on X.500
  • Directory service (RFC1777)
  • Stores attribute based data
  • Data generallly read more than written to
    • No transactions
    • No rollback
  • Hierarchical data structure
    • Entries are in a tree-like structure called Directory Information Tree (DIT)

Attribute abbreviations

uid User id
cn Common Name
sn Surname
l Location
ou Organisational Unit
o Organisation
dc Domain Component
st State
c Country

The Lightweight Directory Access Protocol (LDAP) is an application protocol for reading and editing directories over an IP network.
A client starts an LDAP session by connecting to an LDAP server, called a Directory System Agent (DSA), by default on TCP port 389. The client then sends an operation request to the server, and the server sends responses in return. With some exceptions, the client does not need to wait for a response before sending the next request, and the server may send the responses in any order.
A common alternate method of securing LDAP communication is using an SSL tunnel. This is denoted in LDAP URLs by using the URL scheme "ldaps". The default port for LDAP over SSL is 636. The use of LDAP over SSL was common in LDAP Version 2 (LDAPv2) but it was never standardized in any formal specification.

Tuesday, March 22, 2011

Data integration

Data integration is a set of procedures, techniques, and technologies used to design and build processes that extract, restructure, move, and load data in either operational or analytic data stores either in real time or in batch mode.

Metadata is the “data” about the data; it is the business and technical definitions that provide the
data meaning.

A major function of data integration is to integrate disparate data into a single view of information.

ETL Data integration
ETL is the collection and aggregation of transactional data with data extracted from multiple sources to be conformed into databases used for reporting and analytics.
Most of the cost and maintenance of complex data integration processing occurs in the bulk data
movement space. ETL has experienced explosive growth in both frequency and size in the past 15 years. In the mid-1990s, pushing 30GB to 40GB of data on a monthly basis was considered a
large effort. However, by the twenty-first century, moving a terabyte of data on a daily basis was a requirement. In addition to standard flat file and relational data formats, data integration environments need to consider XML and unstructured data formats. With these new formats, along with the exponential growth of transactional data, multi-terabyte data integration processing environments are not unusual.

ETL: Extract, Transform and Load

ETL stands for extract, transform and load, includes reading data from its source, cleaning it up and formatting it uniformly, and then writing it to the target repository to be exploited.
The data used in ETL processes can come from any source: a mainframe application, an ERP application, a CRM tool, a flat file, an Excel spreadsheet—even a message queue.

The processes enable companies to move data from multiple sources, reformat and cleanse it, and load it into another database, a data mart or a data warehouse for analysis, or on another operational system to support a business process.

In OTC, ETL mostly used in transfer Data from Informix to Vertica, move Data inside database, and extract Data report from database. I will explain in my next blog how OTC use Vertica with market data warehouse.

Tuesday, March 8, 2011

How to use mysql in Linux

1. sudo apt-get install mysql-client mysql-server
2. sudo apt-get install php5-mysql
3. mysql -u root -p

Monday, March 7, 2011

JDBC Informix

1. Add external jar for jfxjdbc.jar to Java Build Path.
2. Coding:
package db;
import java.io.IOException;
import java.io.InputStream;
import java.sql.*;
import java.util.*;

public class jdbc_informix
{
public static void main(String[] args) throws IOException
{
Connection conn = null;
String userName = "xx";
String password = "xx";
String url = "jdbc:informix-sqli://db.test.ps:1111/table_name:INFORMIXSERVER=table_n";

try
{
Class.forName("com.informix.jdbc.IfxDriver");

System.out.println("Driver OK");

}
catch (Exception e)
{
System.out.println("FAILED: failed to load Informix JDBC driver.");
}

try
{
conn = DriverManager.getConnection(url, userName, password);
System.out.println ("Database connection established");

Statement s = conn.createStatement ();

s.executeQuery ("select * from table_name");

ResultSet rs = s.getResultSet ();
if (rs.next ())
{
String nameVal = rs.getString 1);
System.out.println(nameVal);
}
else
System.out.println("error");

}
catch (SQLException e)
{
System.out.println("FAILED: failed to connect!"+e);
}

}

private static void dispValue(InputStream value) {
// TODO Auto-generated method stub

}
}

Friday, February 25, 2011

File permission in Linux

File Ownership
1. User
2. Group
3. Other

File Permissions
1. Read permission
2. Write permission
3. Execute permission

How to view file permissions
$ls -l
$ ls -l
total 17
drwxr-xr-x 3 nana writers 80 2005-09-20 21:37 dir
-rw-r----- 1 nana writers 8187 2005-09-19 13:35 file
-rwxr-xr-x 1 nana writers 10348 2005-07-17 20:31 otherfile

d = directory
- = regular file
l = symbolic link
s = Unix domain socket
p = named pipe
c = character device file
b = block device file

Set file permissions - symbolic mode

Which user?
u user/owner
g group
o other
a all
What to do?
+ add this permission
- remove this permission
= set exactly this permission
Which permissions?
r read
w write
x execute


First, you decide if you set permissions for the user (u), the group (g), others (o), or all of the three (a). Then, you either add a permission (+), remove it (-), or wipe out the previous permissions and add a new one (=). Next, you decide if you set the read permission (r), write permission (w), or execute permission (x). Last, you'll tell chmod which file's permissions you want to change.
eg. $ chmod g+x testfile
Add execute permissions for group.

Set file permissions - numeric mode
4 = read (r)
2 = write (w)
1 = execute (x)
0 = no permission (-)

$ chmod 755 testfile
equals to: -rwxr-xr-x
$ chmod 640 testfile
equals to: -rw-r-----

Quickly add your public key to an authorized keys file

This will add your public ssh key to an authorized keys file on a remote server for passwordless login.
1. Generate key on local machine:
ssh-keygen -t dsa
In your local machine:
$cd .ssh
$ls
authorized_keys id_dsa id_dsa.pub id_rsa id_rsa.pub known_hosts
$more id_dsa.pub

2. Ensure that the remote server has a .ssh directory
$ cd ~/.ssh
$ ls
authorized_keys id_rsa id_rsa.pub known_hosts
$vi authorized_keys

3. Add your public key into remote server.
Now you can ssh to the remote server without entering your password.
Keep in mind that all someone needs to login to the remote server, is the file on your local machine ~/.ssh/id_rsa, so make sure it is secure.