Linux / Unix Command: crond

cron - daemon to execute scheduled commands

Description:

Cron should be started from /etc/rc or /etc/rc.local.

Cron searches /var/spool/cron for crontab files which are named after accounts in /etc/passwd; crontabs found are loaded into memory. Cron also searches for /etc/crontab and the files in the /etc/cron.d/ directory, which are in a different format. Cron then wakes up every minute, examining all stored crontabs, checking each command to see if it should be run in the current minute. When executing commands, any output is mailed to the owner of the crontab (or to the user named in the MAILTO environment variable in the crontab, if such exists).

Additionally, cron checks each minute to see if its spool directory's modtime (or the modtime on /etc/crontab) has changed, and if it has, cron will then examine the modtime on all crontabs and reload those which have changed. Thus cron need not be restarted whenever a crontab file is modified.

Cron (a Linux process that performs background work, often at night) is set up by default on your RedHat system. So you don't have to do anything about it unless you would like to add some tasks to be performed on your system on a regular basis or change the time at which cron performs its duties.

Please note that some of the cron work might be essential for your system functioning properly over a long period of time. Among other things cron may:

- rebuild the database of files which is used when you search for files with the locate command,

- clean the /tmp directory,

- rebuild the manual pages,

- "rotate" the log files, i.e. discard the oldest log files, rename the intermediate logs, and create new logs,

- perform some other checkups, e.g. adding fonts that you recently copied to your system.

Therefore, it may not be the best idea to always switch your Linux machine off for the night--in such a case cron will never have a chance to do its job. If you do like switching off your computer for the night, you may want to adjust cron so it performs its duties at some other time.

To find out when cron wakes up to perform its duties, have a look at the file /etc/crontab, for example:

cat /etc/crontab

It may contain something like this:

# run-parts

01 * * * * root run-parts /etc/cron.hourly

02 4 * * * root run-parts /etc/cron.daily

22 4 * * 0 root run-parts /etc/cron.weekly

42 4 1 * * root run-parts /etc/cron.monthly

You can see that there are four categories of cron jobs: performed hourly, daily, weekly and monthly. You can modify those or add your own category. Here is how it works.

The columns in the entries show: minute (0-59), hour (0-23), day of month (1-31), month of year (1-12), day of week (0-6--Sunday to Saturday). The "*" means "any valid value".

Thus, in the example quoted, the hourly jobs are performed every time the computer clock shows "and one minute", which happens every hour, at one minute past the hour. The daily jobs are performed every time the clock shows 2 minutes past 4 o'clock, which happens once a day. The weekly jobs are performed at 22 minutes past four o'clock in the morning on Sundays. The monthly jobs are performed 42 minutes past four o'clock on the first day of every month. The directory with the script file that contain the command(s) to be executed is shown as the last entry on each line.

If you wanted your jobs to be performed at noon instead of 4 in the morning, just change the 4s to 12s. Cron wakes up every minute and examines if the /etc/crontab has changed so there is no need to re-start anything after you make your changes.

If you wanted to add a job to your cron, place a script which runs your job (or a link to your script) in the directory /etc/cron.hourly or cron.daily or /etc/cron.weekly, or /etc/cron.monthly .

Here is an example of an entry in /etc/crontab which causes a job to be performed three times a week (Mon, Wed, Fri):

02 4 * * 1,3,5 root run-parts/etc/cron.weekly

Ubuntu Linux OpenSSH Server installation and configuration

OpenSSH is a FREE version of the SSH connectivity tools that technical users of the Internet rely on. Users of telnet, rlogin, and ftp may not realize that their password is transmitted across the Internet unencrypted, but it is. OpenSSH encrypts all traffic (including passwords) to effectively eliminate eavesdropping, connection hijacking, and other attacks. Additionally, OpenSSH provides secure tunneling capabilities and several authentication methods, and supports all SSH protocol versions.

Install:

# sudo apt-get install openssh-server openssh-client

Diagnostic Report

ps -ef

List of all running processes

top

List the top resource consumers

env

List all environment variables

find -ls

Recursively list all files and directories

df -hk

Display the file system information

uname -Xa

List information about the server

psrinfo -v

List information about the servers processors

jstack

Get a stack trace of Java program to see what logic is running or stuck in.
Use kill -3 if jstack not working due to high cpu usage

http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html

jinfo

Get all Java runtime parameters such as command line options, classpath, etc.

http://download.oracle.com/javase/6/docs/technotes/tools/share/jinfo.html

jmap

Dump Java program memory into file for memory leak analysis.

http://download.oracle.com/javase/6/docs/technotes/tools/share/jmap.html

Data integration

Data integration: retrieve data from different sources and assemble it in a unified way.

Data integration focuses mainly on databases. A database is an organized collection of data. It's similar to a file system, which is an organizational structure for files so they're easy to find, access and manipulate.

There's the common data storage method, also known as data warehousing. Using this method, all the data from the various databases you intend to integrate are extracted, transformed and loaded. That means that the data warehouse first pulls all the data from the various data sources. Then, the data warehouse converts all the data into a common format so that one set of data is compatible with another. Then it loads this new data into its own database. When you submit your query, the data warehouse locates the data, retrieves it and presents it to you in an integrated view.

A data warehouse is a database that stores information from other databases using a common format.

Descriptions of data are called metadata. Metadata is useful for naming and defining data as well as describing the relationship of one set of data to other sets. Data integration systems use metadata to locate the information relevant to queries.

The warehouse must have a database large enough to store data gathered from multiple sources. Some data warehouses include an additional step called a data mart. The data warehouse takes over the duties of aggregating data, while the data mart responds to user queries by retrieving and combining the appropriate data from the warehouse.

One problem with data warehouses is that the information in them isn't always current. That's because of the way data warehouses work -- they pull information from other databases periodically. If the data in those databases changes between extractions, queries to the data warehouse won't result in the most current and accurate views. If the data in a system rarely changes, this isn't a big deal. For other applications, though, it's problematic.

Load balancing

In networking, load balancing is a technique to distribute workload evenly across two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, minimize response time, and avoid overload. Using multiple components with load balancing, instead of a single component, may increase reliability through redundancy. The load balancing service is usually provided by a dedicated program or hardware device (such as a multilayer switch or a DNS server).

F5 Networks originally manufactured and sold some of the very first load balancing product, called BIG-IP. If a server went down or became overloaded, BIG-IP directed traffic away from that server to other servers that could handle the load.

F5's BIG-IP product is based on a network appliance (either virtual or physical), which runs F5's Traffic Management Operating System (TMOS), which runs on top of Linux. This appliance can then run one or more product modules (depending on the appliance selected), which provide the BIG-IP functionality.

LDAP

What is LDAP ?

• Lightweight Directory Access Protocol
• Based on X.500
• Directory service (RFC1777)
• Stores attribute based data
• Data generallly read more than written to
  • No transactions
  • No rollback
• Hierarchical data structure
• Entries are in a tree-like structure called Directory Information Tree (DIT)
  

Attribute abbreviations

uid User id

cn Common Name

sn Surname

l Location

ou Organisational Unit

o Organisation

dc Domain Component

st State

c Country

The Lightweight Directory Access Protocol (LDAP) is an application protocol for reading and editing directories over an IP network.

A client starts an LDAP session by connecting to an LDAP server, called a Directory System Agent (DSA), by default on TCP port 389. The client then sends an operation request to the server, and the server sends responses in return. With some exceptions, the client does not need to wait for a response before sending the next request, and the server may send the responses in any order.

A common alternate method of securing LDAP communication is using an SSL tunnel. This is denoted in LDAP URLs by using the URL scheme "ldaps". The default port for LDAP over SSL is 636. The use of LDAP over SSL was common in LDAP Version 2 (LDAPv2) but it was never standardized in any formal specification.

Data integration

Data integration is a set of procedures, techniques, and technologies used to design and build processes that extract, restructure, move, and load data in either operational or analytic data stores either in real time or in batch mode.

Metadata is the “data” about the data; it is the business and technical definitions that provide the

data meaning.

A major function of data integration is to integrate disparate data into a single view of information.

ETL Data integration

ETL is the collection and aggregation of transactional data with data extracted from multiple sources to be conformed into databases used for reporting and analytics.

Most of the cost and maintenance of complex data integration processing occurs in the bulk data

movement space. ETL has experienced explosive growth in both frequency and size in the past 15 years. In the mid-1990s, pushing 30GB to 40GB of data on a monthly basis was considered a

large effort. However, by the twenty-first century, moving a terabyte of data on a daily basis was a requirement. In addition to standard flat file and relational data formats, data integration environments need to consider XML and unstructured data formats. With these new formats, along with the exponential growth of transactional data, multi-terabyte data integration processing environments are not unusual.

ETL: Extract, Transform and Load

ETL stands for extract, transform and load, includes reading data from its source, cleaning it up and formatting it uniformly, and then writing it to the target repository to be exploited.

The data used in ETL processes can come from any source: a mainframe application, an ERP application, a CRM tool, a flat file, an Excel spreadsheet—even a message queue.

The processes enable companies to move data from multiple sources, reformat and cleanse it, and load it into another database, a data mart or a data warehouse for analysis, or on another operational system to support a business process.

In OTC, ETL mostly used in transfer Data from Informix to Vertica, move Data inside database, and extract Data report from database. I will explain in my next blog how OTC use Vertica with market data warehouse.

How to use mysql in Linux

1. sudo apt-get install mysql-client mysql-server
2. sudo apt-get install php5-mysql
3. mysql -u root -p

JDBC Informix

1. Add external jar for jfxjdbc.jar to Java Build Path.
2. Coding:
package db;
import java.io.IOException;
import java.io.InputStream;
import java.sql.*;
import java.util.*;

public class jdbc_informix
{
public static void main(String[] args) throws IOException
{
Connection conn = null;
String userName = "xx";
String password = "xx";
String url = "jdbc:informix-sqli://db.test.ps:1111/table_name:INFORMIXSERVER=table_n";

try
{
Class.forName("com.informix.jdbc.IfxDriver");

System.out.println("Driver OK");

}
catch (Exception e)
{
System.out.println("FAILED: failed to load Informix JDBC driver.");
}

try
{
conn = DriverManager.getConnection(url, userName, password);
System.out.println ("Database connection established");

Statement s = conn.createStatement ();

s.executeQuery ("select * from table_name");

ResultSet rs = s.getResultSet ();
if (rs.next ())
{
String nameVal = rs.getString 1);
System.out.println(nameVal);
}
else
System.out.println("error");

}
catch (SQLException e)
{
System.out.println("FAILED: failed to connect!"+e);
}

}

private static void dispValue(InputStream value) {
// TODO Auto-generated method stub

}
}

File permission in Linux

File Ownership
1. User
2. Group
3. Other

File Permissions
1. Read permission
2. Write permission
3. Execute permission

How to view file permissions
$ls -l
$ ls -l
total 17
drwxr-xr-x 3 nana writers 80 2005-09-20 21:37 dir
-rw-r----- 1 nana writers 8187 2005-09-19 13:35 file
-rwxr-xr-x 1 nana writers 10348 2005-07-17 20:31 otherfile

d = directory
- = regular file
l = symbolic link
s = Unix domain socket
p = named pipe
c = character device file
b = block device file

Set file permissions - symbolic mode

Which user?
u	user/owner
g	group
o	other
a	all
What to do?
+	add this permission
-	remove this permission
=	set exactly this permission
Which permissions?
r	read
w	write
x	execute

First, you decide if you set permissions for the user (u), the group (g), others (o), or all of the three (a). Then, you either add a permission (+), remove it (-), or wipe out the previous permissions and add a new one (=). Next, you decide if you set the read permission (r), write permission (w), or execute permission (x). Last, you'll tell chmod which file's permissions you want to change.
eg. $ chmod g+x testfile
Add execute permissions for group.

Set file permissions - numeric mode
4 = read (r)
2 = write (w)
1 = execute (x)
0 = no permission (-)

$ chmod 755 testfile
equals to: -rwxr-xr-x
$ chmod 640 testfile
equals to: -rw-r-----



Quickly add your public key to an authorized keys file

This will add your public ssh key to an authorized keys file on a remote server for passwordless login.
1. Generate key on local machine:

ssh-keygen -t dsa

In your local machine:
$cd .ssh
$ls
authorized_keys id_dsa id_dsa.pub id_rsa id_rsa.pub known_hosts
$more id_dsa.pub

2. Ensure that the remote server has a .ssh directory
$ cd ~/.ssh
$ ls
authorized_keys id_rsa id_rsa.pub known_hosts
$vi authorized_keys

3. Add your public key into remote server.
Now you can ssh to the remote server without entering your password.
Keep in mind that all someone needs to login to the remote server, is the file on your local machine ~/.ssh/id_rsa, so make sure it is secure.

More bash shell commands

1. The basic output shows the process ID (PID) of the programs, the terminal (TTY) that they are
running from, and the CPU time the process has used.
$ ps
PID TTY TIME CMD
3081 pts/0 00:00:00 bash
3209 pts/0 00:00:00 ps

2. see everything running on the system
$ ps -ef

3. The kill command allows you to send signals to processes based on their process ID (PID).
$ kill 3940

4. see how much disk space is available on an individual device. The df command allows us to easily see what’s happening on all of the mounted disks
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 6813412 2889540 3577760 45% /
none 1024880 212 1024668 1% /dev
none 1030476 644 1029832 1% /dev/shm
none 1030476 96 1030380 1% /var/run
none 1030476 0 1030476 0% /var/lock

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 6.5G 2.8G 3.5G 45% /
none 1001M 212K 1001M 1% /dev
none 1007M 644K 1006M 1% /dev/shm
none 1007M 96K 1007M 1% /var/run
none 1007M 0 1007M 0% /var/lock

5. The du command shows the disk usage for a specific directory (by default, the current directory). This is a quick way to determine if you have any obvious disk hogs on the system.
t$ du
8 ./testfold
116 .

$ du -c
8 ./testfold
116 .
116 total

$ du -h
8.0K ./testfold
116K .

$ du -s
116 .

6. sort
$ cat test8
8
93
5
6
28
$ sort test8
28
5
6
8
93
$ sort -n test8
5
6
8
28
93

$ sort -M file3
The -n parameter is great for sorting numerical outputs, such as the output of the du command:
$ du -sh * | sort -nr

7.
$ cat > file1
one
two
three
four
five
$ grep three file1
three
$ grep t file1
two
three
$ grep -v t file1
one
four
five
$ grep -n t file1
2:two
3:three
$ grep -c t file1
2
$ grep -e t -e f file1
two
three
four
five
$ grep [tf] file1
two
three
four
five

8. Compressing data
■ bzip2 for compressing files
■ bzcat for displaying the contents of compressed text files
■ bunzip2 for uncompressing compressed .bz2 files
■ bzip2recover for attempting to recover damaged compressed files

$ ls -l
total 4
-rw-r--r-- 1 linna linna 9 2011-02-24 02:20 test1
$ bzip2 test1
$ ls -l
total 4
-rw-r--r-- 1 linna linna 45 2011-02-24 02:20 test1.bz2
$ more test1.bz2
BZh91AY&SYøG²
$ bzcat test1.bz2
test
test
$ bunzip2 test1.bz2
$ ls -l
total 4
-rw-r--r-- 1 linna linna 9 2011-02-24 02:20 test1
$ more test1
test
test

9. The gzip utility
■ gzip for compressing files
■ gzcat for displaying the contents of compressed text files
■ gunzip for uncompressing files

10. Archiving data
By far the most popular archiving tool used in Unix and Linux is the tar command.

File Handling in shell

1. Create file with unique inode number
$ touch test1
$ ls -il test1
1954793 -rw-r--r-- 1 rich rich 0 Sep 1 09:35 test1

2. Create file without inode number
$ touch test1
$ ls -l test1
-rw-r--r-- 1 rich rich 0 Sep 1 09:37 test1

3. Change the file modification time
$ touch -t 200812251200 test1
$ ls -l test1
-rw-r--r-- 1 rich rich 0 Dec 25 2008 test1

4. Copy files
$cp source destination

5. Copy file to a directory
$ cp test1 dir1
To copy a file to the current directory you’re in, you can use the dot symbol.

6. Copy the contents of an entire directory in one command:
$ cp -R dir1 dir2

7. Linking files
If you need to maintain two (or more) copies of the same file on the system, instead of having separate physical copies, you can use one physical copy and multiple virtual copies, called links. A link is a placeholder in a directory that points to the real location of the file.
Hard link: create a separate file that contains information about the original file(test1) and where to locate it. After you create it, you will see the inode for test4 is the same as test1.
$ cp -l test1 test4
$ ls -il
Soft link: the -s parameter creates a symbolic, or soft link.
Tip: Instead of using the cp command, if you want to link files you can also use the ln command. By default the ln command creates hard links. If you want to create a soft link, you’ll still need to use the -s parameter.

Remember that the hard link file uses the same inode number as the original file. The hard link
file maintains that inode number until you remove the last linked file, preserving the data! All the soft link file knows is that the underlying file is now gone, so it has nothing to point to.

8. Renaming files
$ mv test2 test6
$ ls -il test*
Rename test2 to test6. Notice that moving the file changed the filename but kept the same inode number and the timestamp value.

9. Renaming directory
$ mv dir2 dir4

10. Deleting files
$ rm -i test2
rm: remove `test2’? y
$ ls -l

11. Creating directories
$ mkdir dir3
$ ls -il

12. Deleting directories
$ rmdir dir3
$ rmdir dir1
rmdir: dir1: Directory not empty
if you really want to remove a directory, you can use the -r parameter to recursively
remove the directory.

13. Viewing file statistics: The results from the stat command show just about everything you’d want to know about the file being examined:
$ stat test10

14. Viewing the file type: The file command is a handy little utility to have around. It has the ability to peek inside of a file and determine just what kind of file it is:
$ file test1
test1: ASCII text

15. Viewing the whole file
$ cat test1
$ cat -n test1
$ cat -b test1
$ cat -s test1
$ cat -T test1
$ more test1
The more command displays a text file, but stops after it displays each page of data.
The more command allows some rudimentary movement through the text file. For more advanced features, try the less command.
$ less test1

16. Viewing parts of a file
$tail -f
A great way to monitor the system log file in real-time mode.

How to write shell script

1. Type following cat command:
$ cat > first
#
# My first shell script
#
clear
echo "Knowledge is Power"
echo -e "An apple a day keeps away \a\t\tdoctor\n"
$ echo "Today is date"
$ echo "Today is `date`"
Press Ctrl + D to save. Now our script is ready.
2. Set Execute permission for our script first:
$ chmod +x first
3. Run the shell script
$ ./first
Or: $ bash first
Or: $ /bin/sh first

Addition:
1. Type bc at $ prompt to start Linux calculator program, quit to exit
$ bc
After this command bc is started and waiting for you commands, i.e. give it some calculation as
follows type 5 + 2 as
5 + 2
7
Now what happened if you type 5 > 2 as follows
5 < 2
0
0 (Zero) is response of bc, false
2. eg.
$ cat > demo
#!/bin/sh
#
# Script that demos, command line args
#
echo "Total number of command line argument are $#"
echo "$0 is script name"
echo "$1 is first argument"
echo "$2 is second argument"
echo "All of them are :- $*"
executable & run:
$ chmod +x demo
$ ./demo Hello World
3. eg. show file
$ cat > showfile
#!/bin/sh
#
#Script to print file
#
if cat $1
then
echo -e "\n\nFile $1, found and successfully echoed"
fi
Now run it.
$ chmod +x showfile
$./showfile foo
4. eg. delete file
#
# Script to test rm command and exist status
#
if rm $1
then
echo "$1 file deleted"
fi
(Press Ctrl + d to save)
$ chmod +x trmif
$ ./trmfi foo
5. eg. for loop
$ cat > testfor
for i in 1 2 3 4 5
do
echo "Welcome $i times"
done
Run it as,
$ chmod +x testfor
$ ./testfor
6. eg.
n=$1
for i in 1 2 3 4 5 6 7 8 9 10
do
echo "$n * $i = `expr $i \* $n`"
done
Save and Run it as
$ chmod +x mtable
$ ./mtable 7
7. eg. the read statement
$ cat > sayH
#
#Script to read your name from key-board
#
echo "Your first name please:"
read fname
echo "Hello $fname, Lets be friend!"
Run it as follows
$ chmod +x sayH
$ ./sayH
8. eg.
To run two command with one command line.For eg. $ date;who ,Will print today's date followed by users who are currently login.
9. eg. function
$ SayHello()
{
echo "Hello $LOGNAME, Have nice computing"
return
}
Now to execute this SayHello() function just type it name as follows
$ SayHello

Shell

The shell allows you to chain commands together into a single step. You can use semicolon and put both commands on the same line, but in a shell script, you can list commands on separate lines. The shell will process commands in the order in which they appear in the file.

Creating a Script File

1. use a text editor to create a file, then enter the command to the file

2. specify the shell you are using, format: #!/bin/bash

3. give permission to execute the file: $chmod u+x test

4. $./test

Structured Command:
1. if -then
$ cat test1
#!/bin/bash
# testing the if statement
if date
then
echo "it worked"
fi

$ ./test1
Thu Feb 24 15:29:12 EST 2011
it worked

Shell is a user program or it's environment provided for user interaction. Shell is an command language interpreter that executes commands read from the standard input device (keyboard) or from a file.

Shell is not part of system kernel, but uses the system kernel to execute programs, create files etc.

Several shell available with Linux including: BASH, CSH, KSH, TCSH

Tip: To find all available shells in your system type following command:
$ cat /etc/shells

Tip: To find your current shell type following command
$ echo $SHELL

Normally shells are interactive. It means shell accept command from you (via keyboard) and execute them. But if you use command one by one (sequence of 'n' number of commands) , then you can store this sequence of command to text file and tell the shell to execute this text file instead of entering the commands. This is know as shell script.

Following steps are required to write shell script:

(1) Use any editor like vi or mcedit to write shell script.

(2) After writing shell script set execute permission for your script as follows
syntax:
chmod permission your-script-name

Examples:
$ chmod +x your-script-name
$ chmod 755 your-script-name

Note: This will set read write execute(7) permission for owner, for group and other permission is read and execute only(5).

(3) Execute your script as
syntax:
bash your-script-name
sh your-script-name
./your-script-name

Examples:
$ bash bar
$ sh bar
$ ./bar

NOTE In the last syntax ./ means current directory, But only . (dot) means execute given command file in current shell without starting the new copy of shell, The syntax for . (dot) command is as follows
Syntax:
. command-name

Debug a script:

$bash -x script_name

$bash -xv script_name

cd to /etc/init.d and view various system init scripts:

cd /etc/init.d
ls
vi ssh

Display your current PATH:

echo $PATH

Sample outputs:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Customize your PATH variable and remove /usr/games from PATH:

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export PATH=$PATH:/usr/games

The whatis command is used display a short description about command. whatis command searches the manual page names and displays the manual page descriptions for a command:

whatis command-name
whatis date
whatis ifconfig
whatis ping

Change password:
$passwd

Create alias:
alias name='command'
alias name='command arg1 arg2'

Remove alias:
unalias alias-name

Go to root: $su

alias update='apt-get update && apt-get upgrade'

Doing logout and login back operation: $bash

Basic Linux Command

Command	Example	Description
cat		Sends file contents to standard output. This is a way to list the contents of short files to the screen. It works well with piping.
	cat .bashrc	Sends the contents of the ".bashrc" file to the screen.
cd		Change directory
	cd /home	Change the current working directory to /home. The '/' indicates relative to root, and no matter what directory you are in when you execute this command, the directory will be changed to "/home".
	cd httpd	Change the current working directory to httpd, relative to the current location which is "/home". The full path of the new working directory is "/home/httpd".
	cd ..	Move to the parent directory of the current directory. This command will make the current working directory "/home.
	cd ~	Move to the user's home directory which is "/home/username". The '~' indicates the users home directory.
cp		Copy files
	cp myfile yourfile	Copy the files "myfile" to the file "yourfile" in the current working directory. This command will create the file "yourfile" if it doesn't exist. It will normally overwrite it without warning if it exists.
	cp -i myfile yourfile	With the "-i" option, if the file "yourfile" exists, you will be prompted before it is overwritten.
	cp -i /data/myfile .	Copy the file "/data/myfile" to the current working directory and name it "myfile". Prompt before overwriting the file.
	cp -dpr srcdir destdir	Copy all files from the directory "srcdir" to the directory "destdir" preserving links (-p option), file attributes (-p option), and copy recursively (-r option). With these options, a directory and all it contents can be copied to another directory.
dd	dd if=/dev/hdb1 of=/backup/	Disk duplicate. The man page says this command is to "Convert and copy a file", but although used by more advanced users, it can be a very handy command. The "if" means input file, "of" means output file.
df		Show the amount of disk space used on each mounted filesystem.
less	less textfile	Similar to the more command, but the user can page up and down through the file. The example displays the contents of textfile.
ln		Creates a symbolic link to a file.
	ln -s test symlink	Creates a symbolic link named symlink that points to the file test Typing "ls -i test symlink" will show the two files are different with different inodes. Typing "ls -l test symlink" will show that symlink points to the file test.
locate		A fast database driven file locator.
	slocate -u	This command builds the slocate database. It will take several minutes to complete this command. This command must be used before searching for files, however cron runs this command periodically on most systems.
	locate whereis	Lists all files whose names contain the string "whereis".
logout		Logs the current user off the system.
ls		List files
	ls	List files in the current working directory except those starting with . and only show the file name.
	ls -al	List all files in the current working directory in long listing format showing permissions, ownership, size, and time and date stamp
more		Allows file contents or piped output to be sent to the screen one page at a time.
	more /etc/profile	Lists the contents of the "/etc/profile" file to the screen one page at a time.
	ls -al |more	Performs a directory listing of all files and pipes the output of the listing through more. If the directory listing is longer than a page, it will be listed one page at a time.
mv		Move or rename files
	mv -i myfile yourfile	Move the file from "myfile" to "yourfile". This effectively changes the name of "myfile" to "yourfile".
	mv -i /data/myfile .	Move the file from "myfile" from the directory "/data" to the current working directory.
pwd		Show the name of the current working directory
	more /etc/profile	Lists the contents of the "/etc/profile" file to the screen one page at a time.
shutdown		Shuts the system down.
	shutdown -h now	Shuts the system down to halt immediately.
	shutdown -r now	Shuts the system down immediately and the system reboots.
whereis		Show where the binary, source and manual page files are for a command
	whereis ls	Locates binaries and manual pages for the ls command.

Linux Command Related with Process

Following tables most commonly used command(s) with process:

For this purpose	Use this Command	Examples*
To see currently running process	ps	$ ps
To stop any process by PID i.e. to kill process	kill {PID}	$ kill 1012
To stop processes by name i.e. to kill process	killall {Process-name}	$ killall httpd
To get information about all running process	ps -ag	$ ps -ag
To stop all process except your shell	kill 0	$ kill 0
For background processing (With &, use to put particular command and program in background)	linux-command &	$ ls / -R | wc -l &
To display the owner of the processes along with the processes	ps aux	$ ps aux
To see if a particular process is running or not. For this purpose you have to use ps command in combination with the grep command	ps ax | grep process-U-want-to see	For e.g. you want to see whether Apache web server process is running or not then give command $ ps ax | grep httpd
To see currently running processes and other information like memory and CPU usage with real time updates.	top See the output of top command.	$ top Note that to exit from top command press q.
To display a tree of processes	pstree	$ pstree

* To run some of this command you need to be root or equivalnt user.

System Variables in Shell

System Variable	Meaning
BASH=/bin/bash	Our shell name
BASH_VERSION=1.14.7(1)	Our shell version name
COLUMNS=80	No. of columns for our screen
HOME=/home/vivek	Our home directory
LINES=25	No. of columns for our screen
LOGNAME=students	students Our logging name
OSTYPE=Linux	Our Os type
PATH=/usr/bin:/sbin:/bin:/usr/sbin	Our path settings
PS1=[\u@\h \W]\$	Our prompt settings
PWD=/home/students/Common	Our current working directory
SHELL=/bin/bash	Our shell name
USERNAME=vivek	User name who is currently login to this PC

You can print any of the above variables contains as follows:
$ echo $USERNAME

vi cheat sheet

The default editor that comes with the UNIX operating system is called vi (visual editor).
The UNIX vi editor is a full screen editor and has two modes of operation:

1. Command mode commands which cause action to be taken on the file, and
2. Insert mode in which entered text is inserted into the file.
Vi has two modes insertion mode and command mode. The editor begins in command mode, where the cursor movement and text deletion and pasting occur. Insertion mode begins upon entering an insertion or change command. [ESC] returns the editor to command mode (where you can quit, for example by typing :q!).

To start vi:
vi filename edit filename starting at line 1
vi -r filename recover filename that was being edited when system crashed

To Exit vi:
:x quit vi, writing out modified file to file named in original invocation
:wq quit vi, writing out modified file to file named in original invocation
:q quit (or exit) vi
:q! quit vi even though latest changes have not been saved for this vi call

Moving the cursor:
j move cursor down one line
k move cursor up one line
h move cursor left one character
l move cursor right one character
0 move cursor to start of current line (the one with the cursor)
$ move cursor to end of current line
w move cursor to beginning of next word
b move cursor back to beginning of preceding word

Screen Manipulation: (the symbol ^ before a letter means that the key should be held down while the letter key is pressed):
^f move forward one screen
^b move backward one screen
^d move down (forward) one half screen
^u move up (back) one half screen
^l redraws the screen
^r redraws the screen, removing deleted lines

Adding, Changing, and Deleting Text:
u UNDO WHATEVER YOU JUST DID; a simple toggle

Inserting or Adding Text:
The following commands allow you to insert and add text. Each of these commands puts the vi editor into insert mode; thus, the key must be pressed to terminate the entry of text and to put the vi editor back into command mode.
* i insert text before cursor, until hit
I insert text at beginning of current line, until hit
* a append text after cursor, until hit
A append text to end of current line, until hit
* o open and put text in a new line below current line, until hit
* O open and put text in a new line above current line, until hit

Deleting Text:
* x delete single character under cursor
* dd delete entire current line

Cutting and Pasting Text:
yy copy (yank, cut) the current line into the buffer
p put (paste) the line(s) in the buffer into the text after the current line

Log4j

1. download log4j.jar
2. import it to Elicpse
3. it can output various log to console or files.

Examples:
if you file name is testlog4j.java, then...

public class testlog4j {

//using log4j;
private static Logger logger = Logger.getLogger(testlog4j.class.getName());
public static void main(String[] args) throws Exception, IOException {
long limit = 1000000;
//XMLLayout layout1 = new XMLLayout();//
SimpleLayout layout = new SimpleLayout();
FileAppender appender = null;
try {
appender = new FileAppender(layout,"D:\\output.log", false);
} catch(Exception e) {}

logger.addAppender(appender);
logger.setLevel((Level) Level.ALL);
logger.info("Here is some INFO");

}
}

If you want to append log, set
appender = new FileAppender(layout,"D:\\output.log", true);

How to generate random letters in Java

I know there is a java.util.Random class, which provides more flexible ways to generate uniformly distributed random numbers, like:
Random r = new Random();
int i = r.nextInt(int n) Returns random int >= 0 and < i =" r.nextInt()" l =" r.nextLong()" f =" r.nextFloat()">=0.0 and < d =" r.nextDouble()">=0.0 and < 1.0.
boolean b = r.nextBoolean() Returns random boolean (true or false).

But how can you generate random letters for a string?
Here are solutions:

Solution 1:
String q[] = {"A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z"};

Random r = new Random();
String c = q[r.nextInt(25)]+ q[r.nextInt(25)]+ q[r.nextInt(25)] + q[r.nextInt(25)];

Solution 2: using ASC II integer to string convert
Random r = new Random();
int n = 0;
String c = "";
while (n<4) {
int i = 65 + r.nextInt(25);
String q = new Character ((char)i).toString();
c = c + q;
n++;
}

Solution 3: If you want every time you pick up different letters, you can try this:
String q[] = {"A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", "Y", "Z"};

Random r = new Random();
int n = 0;
String c = "";
while (n < 4) {

int i = r.nextInt(25) + n;
String a = q[i];
q[i] = q[0+n];
n++;
c = c + a;
}

Webdriver cheat sheet

▪ Core Classes
WebDriver
WebElement
By

▪ Visit page
webdriver.get(”url”)

▪ get page title, page source string
webdriver.getTitle(), getPageSource()

▪ Find Element(s)
WebElement elm = webdriver.findElement(s)(By byclause)

▪ By selector where selector in { name, id, tagname, name, xpath, cssName, cssSelector }
webdriver.findElement(By.(”selectorString”))

▪ Get text from a node
webElement.getText()

▪ Fill in input boxes
webElement.sendKeys(”value”)

▪ Click input buttons (radio, checkbox, submit, image)
webElement.click()

▪ Select option
webElement.setSelected()

▪ Submit form
webElement.submit()

▪ Waiting for a condition (ajax + dhtml)

ExpectedCondition cond = new ExpectedCondition() {
@Override
public Boolean apply(WebDriver arg0) { }
};
WebDriverWait wait = new WebDriverWait(driver, waitTime);
wait.until(cond);

▪ Configure implicit wait timeouts

driver.manage().timeouts().implicitlyWait(waitTime, TimeUnit.SECONDS);

Some Windows Tools Used During Work

1. mTAIL
TAIL command is well known at the UNIX users. It shows the real-time changes of a text file. Very usefull for log watching operations. A similar to TAIL command wasn’t available for Windows users till now. Here comes mTAIL,a freeware Windows program that emulates the TAIL Unix command.

2. Putty
PuTTY is a free SSH, Telnet and Rlogin client for Windows system.
SSH, Telnet and Rlogin are three ways of doing the same thing: logging in to a multi-user computer from another computer, over a network.
Using public keys for SSH authentication:
Public key authentication is an alternative means of identifying yourself to a login server, instead of typing a password. It is more secure and more flexible, but more difficult to set up.
You generate a key pair, consisting of a public key (which everybody is allowed to know) and a private key (which you keep secret and do not give to anybody). The private key is able to generate signatures. A signature created using your private key cannot be forged by anybody who does not have that key; but anybody who has your public key can verify that a particular signature is genuine.
So you generate a key pair on your own computer, and you copy the public key to the server. Then, when the server asks you to prove who you are, PuTTY can generate a signature using your private key. The server can verify that signature (since it has your public key) and allow you to log in. Now if the server is hacked or spoofed, the attacker does not gain your private key or password; they only gain one signature. And signatures cannot be re-used, so they have gained nothing.
PuTTYgen is a key generator. It generates pairs of public and private keys to be used with PuTTY, PSCP, and Plink, as well as the PuTTY authentication agent, Pageant. PuTTYgen generates RSA and DSA keys.

3. WinSCP
WinSCP (Windows Secure CoPy) is a free and open source SFTP and FTP client for Microsoft Windows. Its main function is secure file transfer between a local and a remote computer. Beyond this, WinSCP offers basic file manager and file synchronization functionality. For secure transfers, it uses Secure Shell (SSH) and supports the SCP protocol in addition to SFTP.

4. nmap
Nmap (Network Mapper) is a security scanner used to discover hosts and services on a computer network, thus creating a "map" of the network. To accomplish its goal, Nmap sends specially crafted packets to the target host and then analyzes the responses. Unlike many simple port scanners that just send packets at some predefined constant rate, nmap accounts for the network conditions (latency fluctuations, network congestion, the target interference with the scan) during the run.Linux is the most popular nmap platform with Windows following it closely.

5. SQuirrel SQL Client
The SQuirreL SQL Client is a database administration tool. It uses JDBC to allow users to explore and interact with databases via a JDBC driver. It provides an editor that offers code completion and syntax highlighting for standard SQL. It also provides a plugin architecture that allows plugin writers to modify much of the application's behavior to provide database-specific functionality or features that are database-independent. It should run on any platform that has a JVM.

6. Tortoise SVN
TortoiseSVN is a really easy to use Revision control / version control / source control software for Windows. It is based on Subversion.
Since it's not an integration for a specific IDE like Visual Studio, Eclipse or others, you can use it with whatever development tools you like.

7. JXPlorer
JXplorer is an open source ldap browser.

SSL vs. SSH

Secure Shell or SSH is a network protocol that allows data to be exchanged using a secure channel between two networked devices. SSH Used primarily on Linux and Unix based systems to access shell accounts, SSH was designed as a replacement for Telnet and other insecure remote shells, which send information, notably passwords, in plaintext, rendering them susceptible to packet analysis.The encryption used by SSH is intended to provide confidentiality and integrity of data over an unsecured network, such as the Internet.

Secure Sockets Layer (SSL), are cryptographic protocols that provide communications security over the Internet.

HTTPS = HTTP + SSL
The main idea of HTTPS is to create a secure channel over an insecure network.
HTTP URLs use port 80 by default, HTTPS URLs use port 443 by default.

The differences between SSH and SSL are:
1. SSH uses port 22 while SSL uses port 443.

2. SSL is used predominantly for securely transmitting critical information like in credit cards and banking. On the other hand, SSH is for securely executing commands across the internet.

3. SSH uses a username/password authentication system to establish a secure connection while SSL does not really bother with it.

4. SSH is more about network tunneling while SSL is more about certificates.

Automation Testing on WebApps with Selenium Free Tool

Selenium WebDriver is a portable, open-source software testing framework, released under the Apache 2.0 license, for web applications testing. Selenium 2.0 integrate the WebDriver API into Selenium-RC. By Java coding in WebDriver, JavaScript framework is actually running in your web browser, like Firefox, IE and Chrome. WebDriver can simulate a user navigating through pages and then assert for specific marks on the pages. So, I think it is perfect for web automation testing.

Let's see how to implement it:
1. Download Selenium server from Selenium website:
selenium-server-standalone-2.0b1.jar
2. Open eclipse, include this jar file to the library. Here assuming your computer installed all necessary software, like JRE, JDK, Ant, eclipse...
3. Start your java programming in eclipse now!

See, it's easy, and powerful!

How does the mobile device change your life

Ten years ago, many companies did not even have a website. Now, the number that people access the internet via mobile phones is already more than

sitting in front of computer. People use mobile phone to check email, book ticket and hotel, even shopping online. What possible business during this

breakneck pace change? Usablenet Inc, the company which I worked between May, 2009 and January, 2011, is playing a important role in this business.

Usablenet provides a dynamic mobile web platform that uniquely leverages a company’s web site and services and extends the functionality to the handhelds and mobile phones of their customers. The result is that users can seamlessly access web site and services from whatever mobile phones they are using in a view specialized for mobile. Usablenet clients include most major airlines, hotel chains in the US as well as a variety or national retailers, city transit agencies, governments, colleges and universities. Any business related to people’s life, any company would like their sites to be accessed via mobile device by customers, Usablenet can customize a mobile platform for you.

During my one and half years working as Quality Assurance Engineer there, I attended the mobile sites development of American Airlines, Expedia, Starwoods, etc. So basically, it will take 6 to 8 weeks to design a new mobile site for the client, based on the restriction of mobile devices, like exclude javascript, flash, which is not supported by most mobile devices until now. The new mobile site has totally different view of their website, but will keep main functions, like booking, login account, manage reservation, and so on. Then after everything is done, client will put a redirect code in their server, launch mobile site to production. When the customer try to access this website by mobile device, the http request will include the user agent information, tell the server, which device it is, then the redirect code will take effect, redirecting the website to pre-customized mobile site, for iphone, blackberry, or andriod phone. Pretty cool, right?