Parallel SSH execution and a single shell to control them all

 Execute commands simultaneously on multiple server using PSSH/CLUSTER

 you can clieck here parallel-ssh

Logrotation in Linux/unix

Log files are the most valuable tools available for Linux system security. The logrotate program is used to provide the administrator with an up-to-date record of events taking place on the system. The logrotate utility may also be used to back up log files, so copies may be used to establish patterns for system use.

logrotate
the logrotate program is a log file manager. It is used to regularly cycle (or rotate) log files by removing the oldest ones from your system and creating new log files. It may be used to rotate based on the age of the file or the file’s size, and usually runs automatically through the cron utility. The logrotate program may also be used to compress log files and to configure e-mail to users when they are rotated.

Configuration File :- 
Files
/var/lib/logrotate.status   >> this file update  status of recent execution of  logrotation.

root@puppet:~/sadeek/big# ls -l /var/lib/logrotate/status
-rw-r--r-- 1 root root 2030 Feb 23 07:55 /var/lib/logrotate/status

root@puppet:/etc/logrotate.d# ls -l /var/lib/logrotate/status

-rw-r--r-- 1 root root 2030 Feb 24 05:57 /var/lib/logrotate/status

Default state file.
/etc/logrotate.conf     by default configuration file of logs
below are the some example for log rotation configuration, you need to create a file and make the entry as below. it will rotate the logs weekly.
/etc/logrotate.d/

 vi samba

/var/log/samba/log.smbd {

        weekly

        missingok

        rotate 7

        postrotate

                reload smbd 2>/dev/null

        endscript

        compress

        notifempty

root@puppet:/etc/logrotate.d# 

for different application you need to create a file and config . we can execute the script with the help of postrotate  you need to mention the script location. 

root@puppet:/etc/logrotate.d#

this job execute by cron.daily  you can cehck the cron.daily entry under the cat /etc/cron.daily file 

 vi test

/root/sadeek/logs/logs.log {

    daily

    rotate 2

    missingok

    copytruncate

    olddir /root/rama/big

#    postrotate

#        /bin/sh /root/sadeek/rotate.sh

#    endscript

}

FOR MORE OPTION YOU CAN CHECK THE man logroate

Difference between Soft Mount & Hard Mount

The directory /nfs   should be created in your node/server. The nfs mount can be mount as a “soft mount” or as a “hard mount” these mount option define the how the nfs client should be handle nfs crash/failure. We will see the difference between hard mount and soft mount.

Soft mount:- suppose you have mount the nfs by using “soft mount’ when a program request a file from nfs server. Nfs demon will try to retrieve the data from the nfs server. If doesn’t get any response from nfs server due to some failure or crash on nfs server. Then nfs client report an error to the process on the client machine requesting the file access the Advantage “fast responsiveness”   it doesn’t wait to the nfs server to respond. The Main Disadvantage of this method is data corruption or loss of data so this is not the recommended option to use.

 [root@sadeek ~]# showmount -e 172.28.50.109

Export list for 192.168.0.105:

/nfs *

Soft mounting (Temporary Mounting):-

[root@sadeek ~]# mount   172.28.50.109:/nfs/  /mnt/

Hard mounting:-if you have mounted the nfs by “hard mount”.  It will repeatly try to connect to the server. Once the server is back online the program will continue to execute undisturbed the state where it was during the crash. We can use the mount option “intr’ which allows nfs request to interrupt if the server goes down or cannot be access able.

Hard Mount (Permanent mounting):-

[root@sadeek~]# mount  -o rw,hard,intr  172.28.50.109/nfs  /mnt

[root@sadeek ~]# vim  /etc/fstab

172.28.50.109:/nfs       /mnt                         nfs    Defaults    0 0

:wq!

We can check nfs share  with the help of below mention command.

[root@sushee ~]# mount –a

Nagios on RHEL 6

For nagios configuration you can click here

Load Average

for this you can  click here 

Linux System Monitoring Tools

Need to monitor Linux server performance? Try these built-in command and a few add-on tools. Most Linux distributions are equipped with tons of monitoring. These tools provide metrics which can be used to get information about system activities. You can use these tools to find the possible causes of a performance problem. The commands discussed below are some of the most basic commands when it comes to system analysis and debugging server issues such as:

1. Finding out bottlenecks.
2. Disk (storage) bottlenecks.
3. CPU and memory bottlenecks.
4. Network bottlenecks

1: top - Process Activity Command

The top program provides a dynamic real-time view of a running system i.e. actual process activity. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds
.

Commonly Used Hot Keys

The top command provides several useful hot keys:

Hot Key	Usage
t	Displays summary information off and on.
m	Displays memory information off and on.
A	Sorts the display by top consumers of various system resources. Useful for quick identification of performance-hungry tasks on a system.
f	Enters an interactive configuration screen for top. Helpful for setting up top for a specific task.
o	Enables you to interactively select the ordering within top.
r	Issues renice command.
k	Issues kill command.
z	Turn on or off color/mono

2: vmstat - System Activity, Hardware and System Information

The command vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity.
# vmstat 3

Sample Outputs:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 2540988 522188 5130400    0    0     2    32    4    2  4  1 96  0  0
 1  0      0 2540988 522188 5130400    0    0     0   720 1199  665  1  0 99  0  0
 0  0      0 2540956 522188 5130400    0    0     0     0 1151 1569  4  1 95  0  0
 0  0      0 2540956 522188 5130500    0    0     0     6 1117  439  1  0 99  0  0
 0  0      0 2540940 522188 5130512    0    0     0   536 1189  932  1  0 98  0  0
 0  0      0 2538444 522188 5130588    0    0     0     0 1187 1417  4  1 96  0  0
 0  0      0 2490060 522188 5130640    0    0     0    18 1253 1123  5  1 94  0  0

Display Memory Utilization Slabinfo

# vmstat -m

Get Information About Active / Inactive Memory Pages

# vmstat -a

3: w - Find Out Who Is Logged on And What They Are Doing

w command displays information about the users currently on the machine, and their processes.
# w username
# w vivek
Sample Outputs:

 17:58:47 up 5 days, 20:28,  2 users,  load average: 0.36, 0.26, 0.24
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    10.1.3.145       14:55    5.00s  0.04s  0.02s vim /etc/resolv.conf
root     pts/1    10.1.3.145       17:43    0.00s  0.03s  0.00s w

4: uptime - Tell How Long The System Has Been Running

The uptime command can be used to see how long the server has been running. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
# uptime
Output:

 18:02:41 up 41 days, 23:42,  1 user,  load average: 0.00, 0.00, 0.00

1 can be considered as optimal load value. The load can change from system to system. For a single CPU system 1 - 3 and SMP systems 6-10 load value might be acceptable.

5: ps - Displays The Processes

ps command will report a snapshot of the current processes. To select all processes use the -A or -e option:
# ps -A
Sample Outputs:

  PID TTY          TIME CMD
    1 ?        00:00:02 init
    2 ?        00:00:02 migration/0
    3 ?        00:00:01 ksoftirqd/0
    4 ?        00:00:00 watchdog/0
    5 ?        00:00:00 migration/1
    6 ?        00:00:15 ksoftirqd/1

To See Threads After Processes

# ps -AlLm

Print All Process On The Server

# ps ax
# ps axu

Print A Process Tree

# ps -ejH
# ps axjf
# pstree

See Every Process Running As User Vivek

# ps -U sadeek -u sadeek u

Display Only The Process IDs of Lighttpd

# ps -C lighttpd -o pid=
OR
# pgrep lighttpd
OR
# pgrep -u sadeek php-cgi

Display The Name of PID 55977

# ps -p 55977 -o comm=

Find Out The Top 10 Memory Consuming Process

# ps -auxf | sort -nr -k 4 | head -10

Find Out top 10 CPU Consuming Process

# ps -auxf | sort -nr -k 3 | head -10


6: free - Memory Usage

The command free displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.
# free Sample Output:

            total       used       free     shared    buffers     cached
Mem:      12302896    9739664    2563232          0     523124    5154740
-/+ buffers/cache:    4061800    8241096
Swap:      1052248          0    1052248

7: iostat - Average CPU Load, Disk Activity

The command iostat report Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS).
# iostat Sample Outputs:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in)  06/26/2009
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.50    0.09    0.51    0.03    0.00   95.86
Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              22.04        31.88       512.03   16193351  260102868
sda1              0.00         0.00         0.00       2166        180
sda2             22.04        31.87       512.03   16189010  260102688
sda3              0.00         0.00         0.00       1615      

  

8: sar - Collect and Report System Activity

The sar command is used to collect, report, and save system activity information. To see network counter, enter:

# sar -n DEV | more
To display the network counters from the 24th:

# sar -n DEV -f /var/log/sa/sa24 | more
You can also display real time usage using sar:

# sar 4 5
Sample Outputs:

Linux 2.6.18-128.1.14.el5 ) 06/26/2009 06:45:12 PM
 CPU %user %nice %system %iowait %steal %idle
06:45:16 PM all 2.00 0.00 0.22 0.00 0.00 97.78
06:45:20 PM all 2.07 0.00 0.38 0.03 0.00 97.52
06:45:24 PM all 0.94 0.00 0.28 0.00 0.00 98.78
06:45:28 PM all 1.56 0.00 0.22 0.00 0.00 98.22
06:45:32 PM all 3.53 0.00 0.25 0.03 0.00 96.19
Average: all 2.02 0.00 0.27 0.01 0.00 97.70

9: mpstat - Multiprocessor Usage

The mpstat command displays activities for each available processor, processor 0 being the first one. mpstat -P ALL to display average CPU utilization per processor:


# mpstat -P ALL

Sample Output:

Linux 2.6.18-128.1.14.el5 (www03.nixcraft.in)   06/26/2009
06:48:11 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
06:48:11 PM  all    3.50    0.09    0.34    0.03    0.01    0.17    0.00   95.86   1218.04
06:48:11 PM    0    3.44    0.08    0.31    0.02    0.00    0.12    0.00   96.04   1000.31
06:48:11 PM    1    3.10    0.08    0.32    0.09    0.02    0.11    0.00   96.28     34.93
06:48:11 PM    2    4.16    0.11    0.36    0.02    0.00    0.11    0.00   95.25      0.00
06:48:11 PM    3    3.77    0.11    0.38    0.03    0.01    0.24    0.00   95.46     44.80
06:48:11 PM    4    2.96    0.07    0.29    0.04    0.02    0.10    0.00   96.52     25.91
06:48:11 PM    5    3.26    0.08    0.28    0.03    0.01    0.10    0.00   96.23     14.98
06:48:11 PM    6    4.00    0.10    0.34    0.01    0.00    0.13    0.00   95.42      3.75
06:48:11 PM    7    3.30    0.11    0.39    0.03    0.01    0.46    0.00   95.69 

 10: iptraf - Real-time Network Statistics

The iptraf command is interactive colorful IP LAN monitor. It is an ncurses-based IP LAN monitor that generates various network statistics including TCP info, UDP counts, ICMP and OSPF information, Ethernet load info, node stats, IP checksum errors, and others. It can provide the following info in easy to read format:

• Network traffic statistics by TCP connection
• IP traffic statistics by network interface
• Network traffic statistics by protocol
• Network traffic statistics by TCP/UDP port and by packet size
• Network traffic statistics by Layer2 address

 General interface statistics: IP traffic statistics by network interface

11: tcpdump - Detailed Network Traffic Analysis

The tcpdump is simple command that dump traffic on a network. However, you need good understanding of TCP/IP protocol to utilize this tool. For.e.g to display traffic info about DNS, enter:

# tcpdump -i eth1 'udp port 53'


To display all IPv4 HTTP packets to and from port 80, i.e. print only packets that contain data, not, for example, SYN and FIN packets and ACK-only packets, enter:


# tcpdump 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
To display all FTP session to 202.54.1.5, enter:


# tcpdump -i eth1 'dst 202.54.1.5 and (port 21 or 20'
To display all HTTP session to 192.168.1.5:


# tcpdump -ni eth0 'dst 192.168.1.5 and tcp and port http'
Use wireshark to view detailed information about files, enter:


# tcpdump -n -i eth1 -s 0 -w output.txt src or dst port 80

12: /Proc file system - Various Kernel Statistics

/proc file system provides detailed information about various hardware devices and other Linux kernel information. See Linux kernel /proc documentations for further details. Common /proc examples:

# cat /proc/cpuinfo
# cat /proc/meminfo
# cat /proc/zoneinfo
# cat /proc/mounts

13: Nagios - Server And Network Monitoring

Nagios is a popular open source computer system and network monitoring application software. You can easily monitor all your hosts, network equipment and services. It can send alert when things go wrong and again when they get better. FAN is "Fully Automated Nagios". FAN goals are to provide a Nagios installation including most tools provided by the Nagios Community. FAN provides a CDRom image in the standard ISO format, making it easy to easilly install a Nagios server. Added to this, a wide bunch of tools are including to the distribution, in order to improve the user experience around Nagios.

A few more tools:

• nmap - scan your server for open ports.
• lsof - list open files, network connections and much more.
• ntop web based tool - ntop is the best tool to see network usage in a way similar to what top command does for processes i.e. it is network traffic monitoring software. You can see network status, protocol wise distribution of traffic for UDP, TCP, DNS, HTTP and other protocols.
• Conky - Another good monitoring tool for the X Window System. It is highly configurable and is able to monitor many system variables including the status of the CPU, memory, swap space, disk storage, temperatures, processes, network interfaces, battery power, system messages, e-mail inboxes etc.
• GKrellM - It can be used to monitor the status of CPUs, main memory, hard disks, network interfaces, local and remote mailboxes, and many other things.
• vnstat - vnStat is a console-based network traffic monitor. It keeps a log of hourly, daily and monthly network traffic for the selected interface(s).
• htop - htop is an enhanced version of top, the interactive process viewer, which can display the list of processes in a tree form.
• mtr - mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.

Diff between ext2 & ext3, ext4

Extended n file system
ext2 :
- Introduced with kernel 1.0 in 1993
- Flexible can handle upto 4TB
- super block feature increase file system performance
- ext2 reserve 5% of disk space for root
- ext2 is popular on USB and other solid-state devices.
  This is because it does not have a journaling function.
  so it generally makes fewer reads and writes to the drive,
  effectively extending the life of the device .
-  NO journalalizm

ext3 :
- Provide all the feature of ext 2 + journaling and backward compatibility .
- can upgrade ext2 to ext3 without loss of data.
- journaling feature speed up the system to recover the state after power-failure
  or improper mount unmount etc.
- Example: In ext2 in an improper unmount or in-between power-off etc.. so in time
  of receiver it checks whole file system .
  But in ext3 it keeps record of uncommitted file transactions and checks applied
  on on them so system will come back up in faster and quicker .
-

ext4:
- Introduced with kernel 2.6.28
- Ext4 is a deeper improvement over Ext3
- support  larger filesystem, faster checking, nanosecond timestamps,
  and verification of the journal through checksums.
- It’s backward and forward compatible with versions 2 and 3, so we can
  mount a ext2 or ext3 filesystem as ext4 .
- The main benefits that ext4 has over ext3 are:
  - faster time-stamping
  - faster file system checking
  - journaling check-sums
  - extents (basically automatic space allocation to avoid fragmentation)

What is Journalism in linux file syatem ?
A journaling is a file system,  that keeps track of the changes that will be made in
a journal (usually a circular log in a dedicated area of the file system) before committing them to
the main file system. In the event of a system crash or power failure, such file systems are quicker
to bring back online and less likely to become corrupted.