Monday, 2 January 2017

/etc/vfstab (Virtual File System Table)

/etc/vfstab (Virtual File System Table)

           Before you can access the files on a filesystem, you need to mount the filesystem. when you mount a filesystem, you attach that file system to a directory(mount point) and make it available to the system. The root(/) filesystem is always mounted. Any other filesystem can be connected or disconnected from the root (/) filesystem. When you mount a filesystem, any files or directories in the underlying mountpoint directory are unavailable as long as thefilesystem is mounted. These files are not permanently affected by the mounting process. They become available again when the filesystem is unmounted. However, mount directories are typically empty because you usually do not want to obscure existing files.
           When you mount the filesystem from the command prompt the mount is temporary and is blown away by a reboot. Inorder to make them permanent accross reboots you need to update  them under /etc/vfstab file.
             The /etc/vfstab file lists all the filesystems to be automatically mounted at system boot time, with the exception of the /etc/mnttab and /var/run filesystems.
            The file format includes seven fields per line etry. By default, a tab seperates each field, but any whitespace can be used for seperators. the dash (-) character is used as a placeholder for fields when text arguments are not appropriate. Commented lines begin with the hash (#) symbol.

To add a line entry, you need the following information:

Device to Mount: The device to be mounted. For example a local ufs filesystem /dev/dsk/c#t#d#s#, or a pseudo filesystem /proc.
Device to fsck: The raw or character device checked by the silesystem check program (fsck) if applicable.
Mount Point: The name of the directory that servers as the attach mountpoint in the Solaris directory hierarchy.
Fs Type: The type of the filesystem to be mounted (ufs,nfs,swafs etc.,)
fsck pass: Indicates whether the filestsem is to be checked by the fsck utility at boot time. A 0 (zero) or a non-numeric in this field indicates no. A 1 in this field indicates that fsck utility gets started for that entry and runs to completion. A number greater than 1 indicates that the device is added to the list of devices to have fsck utility run. The fsck can run on up to eight devices in parallel. This field is ignored by the "mountall" command.
Mount at Boot: Enter "Yes" to enable the "mountall" command to mount the filesystema at boot time. Enter "no" to prevent a filesystem mount at boot time.
Mount Options: A comma-seperated list of options passed to the mount command. A dash (-) indicates the use of default mount options.
Note: Use tabs to seperate fields in the /etc/vfstab while you edit the file as the fields often do not line up under their respective headings.

Tuesday, 28 June 2016

System Scheduler - CRON and AT

System Scheduler - CRON and AT

"Cron" and "at" are used for scheduling tasks to run on a server. The jobs or tasks are referred to as "cron jobs" and "at jobs".
The features include:
1. Permits scheduling on scripts/tasks on a per-user basis via individual cron tables.
2. Permits recurring execution of tasks.
3. Permits one-time execution of tasks via "at"
4. Logs results (exit status but can be full output) of executed tasks.
5. Facilitates restrictions/permissions via - cron.deny, cron.allow, at.deny and at.allow
6. There is a daemon "crond" associated with the jobs and is responsible for launching the cron jobs on schedule.

Why to use Cron Jobs?

Let us look at why we need to use cron jobs that are relevant to server admins:
1. You can send out daily disk usage reports.
2. You can run scheduled checks on the servers that sendout e-mails at regular intervals with the server status.
3.You can schedule long-running tasks to run from a command line script.
4. You can auto-check the file content and have a report e-mailed regularly


Directory Layout for Cron daemon:

/var/spool/cron --> is the main directory where the entries of "aron" and "at" are stored.
/var/spool/cron/atjobs -> houses one-off, atjobs
/var/spool/cron/cronjobs --> houses recurring jobs for users
-username --> these files house recurring tasks for each user. 
/var/cron/logs --> is where the logs of cron are stored

 Cron Command:

 "crontab" is the command that facilitates management of cron table files.
"crontab -l " --> lists the cron table of current user. This reads the /var/spool/cron/crontabs/<username> file and prints the output.
"crontab -e" --> is the command to edit the crontab file of the current user
"crontab -r" --> is the command to remove the crontab file of the current user.
###############Cron Table Format#####################
 m(0-59)   h(0-23)  dom(1-31)  mon(1-12)  dow(0-6)  command
10 3 * * * /usr/sbin/logadm  ---> runs at 3:10 AM everyday
15 3 * * 0 /usr/lib/fs/nfs/nfsfind --> runs at 3:15 AM every Sunday
m(0-59) --> minutes from 0 to 59
h --> hours from 0 to 23
dom --> day of the month from 1 to 31
mon --> month of the year from January to December
dow --> day of the week (0-6) Sunday to Saturday
  • Date/time/command MUST be on a single line.
  • Seperate columns/fields using whitespace or tabs.
  • Always test the command prior to crontab/at submission
  • Sometimes 'contab -e' would not give you the appropriate prompt to edit. So set EDITOR to desired editor.
  • Aim to reference scripts (shell/perl/python/ruby/PHP, etc.) instead of the various characters.
  • Default Solaris install creates 'at.deny' & 'cron.deny', You MUSt not be included in either file to be able to submit 'at' and 'cron' entries. Conversely if cron.allow and at.allow files exist, you MUST belong to either files to submit at or cron entries.

One Time Cron or AT Job:

Closely related to 'cron' is the 'at' command. However 'at' will only run the job once, unless the job reschedules itself.
 The basic syntax is:
 at <time>
 The above invocation will cause at to read STDIN for a list of command to be executed at a specfied time. Alternatively you could use:
at <time> < file
 Where 'file' is a shell script of some list of commands.
            <time> can be in several forms, such as 12 or 24 hour clock time.

Wednesday, 4 May 2016

Compression Utilities

Compression Utilities

The main reason for data compression are conserving disk space and using less bandwidth for network communication.
Compression is a way of reducing disk space of a file on disk using different algorithms and mathematical calculations.
Some of the compression utilities are:
1. gzip/gunzip
2. bzip2/bunzip2
3. zip/unzip
4. tar
gzip is a classic method of compression. The main advantage is speed and compatibility.
a. Create a file with content.
$ ls -l Linux.txt
-rw-rw-r--   1 rakul   staff     123396 May  2 06:09 Linux.txt
b. gzip -c Linux.txt
$ µåêòï2et[ÙÁ¶¬&iT5|ÎBDék¾'@3À}Û¿ïÙÙOÄØ Üî¤\cÚ
Ë»Dè®Üu )q`À53
c --> Compression option. Redirects the output to STDOUT. Sometimes this will corrupt the console as the compressed output looks like above.
c. gzip -c Linux.txt >Linux.txt.gz
Redirects the output to the file and saves the file Linux.txt.gz.
$ gzip -c Linux.txt >Linux.txt.gz
$ ls -l Linux*
-rw-rw-r--   1 rakul   staff     121k May  2 06:09 Linux.txt
-rw-rw-r--   1 rakul  staff      14k May  2 06:15 Linux.txt.gz
d. gzip -l Linux.txt.gz 
$ gzip -l Linux.txt.gz
         compressed        uncompressed  ratio uncompressed_name
              14692              123396  88.1% Linux.txt
-l option lists or enumerates the contents of the file. This returns to the STDOUt the file that is contained with in the compressed file including its compressed footprint and its uncompressed footprint.

 a. zcat Linux.txt.gz

This will dump the contents fo the compressed file to STDOUT without decompressing it.

 a. gunzip Linux.txt.gz
gunzip followed by the name of the file decompresses the compressed file. It asks you to overwrite the existing file with your permission if given "yes".

$ gunzip Linux.txt.gz
gunzip: Linux.txt already exists; do you wish to overwrite (y or n)? y
$ ls -l Linux*
-rw-rw-r--   1 rakul   staff     123396 May  2 06:15 Linux.txt

  Note: Typically compressed files online resemble: 'filename.tar.gz'
 a. bzip2 -c Linux.txt
$ bzip2 -c Linux.txt
bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.
bzip2 with '-c' option complains that it cannot write to the terminal where as gzip does not. You would need to redirect the output to a file insted of STDOUT.

$bzip2 -c Linux.txt > Linux.txt.bz2
$ ls -l Linux*
-rw-rw-r--   1 rakul   staff     123396 May  2 06:15 Linux.txt
-rw-rw-r--   1 rakul   staff      13233 May  3 02:19 Linux.txt.bz2

b. bunzip Linux.txt.bz2
 This will decompresses the file (bunzip has an option: '-d' which can or cannot be used to decompress)
$ bunzip2 Linux.txt.bz2
bunzip2: Output file Linux.txt already exists.

$ ls -l Linux*
-rw-rw-r--   1 rreddy   staff     123396 May  2 06:15 Linux.txt
-rw-rw-r--   1 rreddy   staff      13233 May  3 02:19 Linux.txt.bz2
bunzip2 warns you that a file already exists. Ideally once the file has been compressed the original file needs to be removed to save disk space. There are many options with bunzip2 which you can get by running 'bunzip2 -- help' on your terminal.
 c.bzip2 -d -c Linux.txt.bz2 >Linux.txt2
 bzip2 with -d option will decompress the output and saves in a filename you provide. 

$ bzip2 -d -c Linux.txt.bz2 >Linux.txt2
$ ls -l Linux*
-rw-rw-r--   1 rakul   staff     123396 May  2 06:15 Linux.txt
-rw-rw-r--   1 rakul   staff      13233 May  3 02:19 Linux.txt.bz2
-rw-rw-r--   1 rakul   staff     123396 May  3 05:14 Linux.txt2

As you can see the files Linux.txt and Linux.txt2 are of same size.

 d.bzcat Linux.txt.bz2

bzcat cat reads and redirects the standard output.


zip and unzip are classic compression utilities which use zip algorithm for compression and decompression.

a. zip Linux.txt

$ zip Linux.txt
  adding: Linux.txt (deflated 88%)
$ ls -l Linux*
-rw-rw-r--   1 rakul   staff     123396 May  2 06:15 Linux.txt
-rw-rw-r--   1 rakul   staff      13233 May  3 02:19 Linux.txt.bz2
-rw-rw-r--   1 rakul   staff      14814 May  3 06:15
-rw-rw-r--   1 rakul   staff     123396 May  3 05:14 Linux.txt2

 When we compare zip and gzip, the size of the zip file would be slightly bigger than the gzip file.


 This would replace the original file.

$ unzip
replace Linux.txt? [y]es, [n]o, [A]ll, [N]one, [r]ename:

  c. zcat

This will read the zip algorithm or zip compressed files and dumps the output to the terminal.


tar creates archives. These are untimately represented by a single file but may contain 1 or more files or directories.
The usage of tar is as follows:

a. tar -cvf temp.tar /temp

we can create tar archieve using cvf, specify the name of the archieve and the source.

b. tar -tvf temp.tar

This will enumerate the contents without actually extracting the data.

c. tar -cvzf temp.tar.gz temp/

This will create a tar ball and compresses it.

d. tar -xvf temp.tar.gz 

This will recreate the hierarchy.


Thursday, 28 April 2016

Grep - Global Regular Expression Parser


Grep is regular expression line processor. The features of grep are:
1) Processes lines using regular expressions (searches for normal and meta characters)
2) Returns the entire lines when a keyword is matched.
3) Searches are case sensitive by default (use: '-i' to enable case insensitivity)
4) Metacharacters allows us to search based on meaningful string combinations and positions. we will use symbols to indicate the positions.
Let us look at some of the search patterns using grep:
a. Create a file with content 
$ cat grep.txt
Red Hat Linux
Linux 2015
Linux 2015 Edition
linux contents
Test 1
L inux
Test L inux
LinuxVBT        -----> 2 or more spaces are added after VBT

b. search for a given string in a file.
The basic usage of grep is to search a given string in the file.
$ grep "Linux" grep.txt
Red Hat Linux
Linux 2015
Linux 2015 Edition
The above command searches for the word "Linux" in the file grep.txt and returns all matches for the case "Linux".
c.  grep the words with case insensitivity
Inorder to enable case insensitivity using grep you have to use '-i'. 

$ grep -i "Linux" grep.txt
Red Hat Linux
Linux 2015
Linux 2015 Edition
linux contents

d. Recursive search for a string
If you want to search for a word called "Linux" in all the files use '-r' option 

$ grep -r "Linux" * 

e. search using metacharacters
If you want to search for a match where the line ends with 2015 it can be achieved as below.
$ grep "2015$" grep.txt
Linux 2015
Note: '$' means to search for the content at the end of the line.

If you want to search for a match where the line starts with the string "Linux" it can be achieved as:
$ grep "^Linux" grep.txt
Linux 2015
Linux 2015 Edition

 Note: '^' (carat) represents search  for the content at the beginning of the line.
Both '^' and '$' are called as anchor tags because they indicate the beginning and end of text.
f.  Search for a letter followed by anything
The below search searches for the letter 'L' followed by anything.

$ grep "L.*" grep.txt
Red Hat Linux
Linux 2015
Linux 2015 Edition
L inux
Test L inux
If you wish to search for a pattern where the line begings with 'L' followed by naything you can use:
$ grep '^L.*' grep.txt
Linux 2015
Linux 2015 Edition
L inux

what if if the line contains the letter 'L' alone?. The answer would be even that is printed.
Note: '.*' means zero or more characters
 g. searches using various positions and combinations
If you want to search for a word/string which starts with 'L' and ends with 'VBT'

$ grep -i '^L.*VBT$' grep.txt

 If you want to search for a word/string which starts with 'L' and ends with a space after  'VBT' 
$ grep -i '^L.*VBT $'  grep.txt

 In the above output nothing is returned as our text file does not contain anything whwre VBT ends with a single space.
 If you wish to search for a pattern where the line begins with 'L' and ends with one or more spaces at the end  you can use:
$ grep -i '^L.*VBT.*$' grep.txt
LinuxVBT    ---> The last line of the file where one or more spaces added

  h. character searches 

$ grep -i '[Red|2015]' grep.txt
Red Hat Linux
Linux 2015
Linux 2015 Edition
linux contents
Test 1
Test L inux
The above search uses character classes and allows us to search using a class of characters. This searches for each and every character and returns the lines where a match is found.
I. Display line number for every matching search
$ grep -n "linux" grep.txt
7:linux contents 


Tuesday, 9 February 2016

Basic Architecture of a Disk

Basic Architecture of a Disk

Generally UNIX disk drives are of SCSI (Small Computer System Interface) or IDE (Integrated Drive Electronics). On both disk drives UNIX divides physical disk into logical disks called as partitions/slices. Each partition is a standlone file system.
Each disk device is given it's own major device number and each partition has an associated minor device number which the device driver uses to access the raw filesystem.
Each disk drive is manaed by a controller. Each type of controller can supporta fixed number of drives.
SCSI controllers support upto seven disks per controller or upto 15 disks per controller depepnding upon the SCSi controller type, IDE supports maximum of 2 devices per channel.

Physical Disk Structure:

Disk consists of the following:
  • Tracks -- Concentric rings on each platter
  • Cylinders -- groups of tracks
  • Sectors/Blocks -- 512 byte blocks
UNIX flesystem contains:
Boot block, Super block, Inode block, Data block, Backup super block

Boot block:

Boot block is located in 1-15 sectors of a filesystem. Boot block contains the initial bootstrap program called as BSL (Boot Strap Loader) used to load operating system.

Super block:

Super block is located from 16-31 sectors of the fileystem. It describes the complete state of the filesystem, the total size of the partition, the block size, pointers to a list of free blocks, the inode number of the root directory, magic number, etc.

 Backup Super block:

 Super block maintains identical copies of the data in backup super block. The main purpose of BSP is to maintain data redundancy. If super block gets corrupted we can recover using backup super block with the combination of 'fsck'. It can be located anywhere from 32nd sector.

 Data block:

It contains actual contents of files.

Inode block:

A liner array of inodes (Short for Index Nodes). There is a one to one mapping of files to indoes and vice versa. An inode is identified by its 'inode number', which contains the information needed to find the inode itself on the disk. Users think of files in terms of file names, UNIX thinks of files in terms of inodes.
|          |           | | | | | | | |        |  |  |  |      |
| B. B.    | S. B.     | Inodes  | | | BSP    |  Data Blocks  |
|          |           | | | | | | | |        |  |  |  |      |
An inode contains the following information: 
  • File ownership indication
  • File types (e.g., regular, directory, special device, pipes, etc.)
  • File access permissions. May have setuid(sticky) bit set
  • Time of last access and modification of ifles
  • number of links (aliases) to the file
  • pointers to the data blocks for the file
  • size of the file in bytes (for regular files), major and minor device numbers for special devices.
An integral nunber os inodes fits in a single data block.
Designed By Blogger Templates