GLOBAL ENVIRONMENT

* Environment variables file for all users :
/etc/environment

* scripts to run at startup for the shell of any user :
/etc/profile.d/*.sh

PROFILE

* Reload bashrc :
source ~/.bashrc
OR
. ~/.bashrc

History

Show command history of all users :

getent passwd |
cut -d : -f 6 |
sed 's:$:/.bash_history:' |
xargs -d '\n' grep -s -H -e "$pattern"

System Information

* uname (kernel information)
– a : display all information
Example : uname -a

Linux fooHostname 1.1.1-1111.11.1.el7.x86_64

* lsb_release (distribution-specific information)
-a : display all information
Example : lsb_release -a

LSB Version:    :core-4.1-amd64:core-4.1-noarch 
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.8 (Maipo)
Release:        7.8
Codename:       Maipo

* In /etc/ a file *release*also contain distrib information.
Find it with : ls /etc/*release*

Examples:
– /etc/system-release
– /etc/lsb-release

CPU info

lscpu | grep -E '^Thread|^Core|^Socket|^CPU\(' to get something like :

CPU(s):                16
Thread(s) per core:    4
Core(s) per socket:    1
Socket(s):             16

INIT SYSTEM

What is it ?
In Unix-based OS, init (short for initialization) is the first process started during booting of the computer system. Init is a daemon process that continues running until the system is shutdown. It is the direct or indirect ancestor of all other processes and automatically adopts all orphaned processes. Init is started by the kernel during the booting process; a kernel panic will occur if the kernel is unable to start it. Init is typically assigned process identifier 1.

SysV, Upstart or Systemd ?

– System V style : inherited from Unix System III
– Systemd (most used for now), a software suite, full replacement for init in Linux that includes an init daemon, with concurrent starting of services, service manager, and other features.
– Upstart, a full replacement of init.d/systemV designed to start processes asynchronously. Initiated by Ubuntu and used by them until 2014.

* How to guess its system ?
sudo stat /proc/1/exe

Output st such as :
File: /proc/1/exe -> /lib/systemd/systemd

init.d

It comes from System V but is also available in new systems (compatibility reasons).
init.d is verbose and picky. Don’t use it but if you don’t have any other choices.

Services configurations are stored in files without any extension in the /etc/init.d folder.

Basic command : service

– list all services with their status :
service --status-all

– execute a command on a service:
service foo-service [start, stop, restart, try-restart, reload, force-reload, status]

Upstart

It is available in RHEL 6 and some others old Ubuntu distribs.
Services configurations are stored in files with a conf extension in the /etc/init folder.

Basic command : initctl

– list all services with their status : initctl list

– Action on a service : start, stop, restart, reload, status
Ex : initctl status SERVICE_NAME

Sample configuration file for a service :

description "Foo service"
 
# system state condition to start/stop the service
start on runlevel [2345]
stop on runlevel [!2345]
 
#That is the command to execute
exec tail -f dev/null 
#That allows to restart the command in cases of error (kill/reboot...)
respawn 
 
# tasks to do before that the started event be emitted
post-start script
    /bin/sleep 5
end script

SYSTEMD

SYSTEMCTL

systemctl – Control the systemd system and service manager

1) It manages services and more… We call all of them units.
2) Units have a unit file used to load them.
3) Valid unit names consist of a « name prefix » and a dot and a suffix specifying the unit type.
4) We have multiple unit types : « .service », « .socket », « .device », « .mount », « .automount », « .swap », « .target », « .path », « .timer », « .slice », or « .scope ».

–Some commands–
* List units that systemd currently has in memory (it excludes some not active units).
systemctl
that by default does
systemctl list-units

* Show runtime status information about one or more units, followed by most recent log data from the journal.
If no units are specified, show system status.

systemctl status (unit) or (pid)

example :
systemctl status (for system unit)
systemctl status syslog (for syslog.service)
systemctl status pid123 pid456 (for the services associated to the PIDs. The service may be associated to the parent process of the PID or an older ancestor process)

* Useful flags
– No units exclusion : -all
– specify a unit (t)ype as filter : -t any-unit-type

* enable a service
systemctl enable myService

* disable a service
systemctl disable myService

* edit the service configuration
systemctl edit –full myService

* declare a new service

Create the service file (for example my-jenkins.service) either in /usr/lib/systemd/system, in /etc/systemd/system/ or
still in /lib/systemd/system such as :

[Unit]
Description=Jenkins container
Requires=docker.service
After=docker.service

[Service]
Restart=always
User=jenkins
Group=cicdTools
#  PermissionsStartOnly=true allows that only ExecStart to be executed with the
# user/group provided here. For example RuntimeDirectory creation will per performed
# with root but set with as owner : jenkins:cdcdTools. 
# Other example ExecStartPre will be executed as root 
PermissionsStartOnly=true
#RuntimeDirectory is optional. It is generally used with PermissionsStartOnly=true  
# It ensure that the jenkins directory is created in /var/run 
RuntimeDirectory=jenkins
SuccessExitStatus=100
ExecStart=/usr/bin/docker start -a my-jenkins
ExecStop=/usr/bin/docker stop -t 2 my-jenkins

[Install]
WantedBy=default.target

Then we must enable it and it is done. A symlink is created from /etc/systemd/system/…wants

The « wantedBy » states when the service should be started (if not been started before by a « Require » from another service).

* reload the systemd manager configuration (rerun all generators , reload all unit files, and recreate the entire dependency tree) :
systemctl daemon-reload

* reload the services configuration (ex: httpd.conf for apache) :
systemctl reload

* Return the default target to boot into (valued by the symlink default.target)
systemctl get-default

* Set the default target to boot into
systemctl set-default anyTarget

* list the targets
systemctl -t target


–Debug a service start failure–

* show all props (explicitly + implicitly defined) for a service :
systemctl show fooService

* show a specific prop for a service :
systemctl show fooService -P myProp

* beware of start/restart params:

RestartUSec=...
TimeoutStartUSec=...
StartLimitInterval=...
StartLimitBurst=...
StartLimitAction=...

A not enough high value may make the service to not start or restart correctly.

System Logs

There exists a standard for message logging on Linux: syslog (1980) at the origin.
Then, some improvements of it have emerged. The most common are: syslog-ng (1998)and more recently (2004) rsyslog.
Nowadays, these two are very widespread.
These are started as a service, for example : rsyslog.service

How to read logs?

With journalctl

Go to the next point : journalctl to get more information.

From logs files

In the /var/log/ directory.
Here some common files :

– general messages
On Debian based OS: /var/log/syslog
On RHEL/CentOS : /var/log/message

– authentication logs
/var/log/authlog or /var/log/secure

– boot logs
/var/log/boot.log

– cron logs
/var/log/cron

– device driver logs
/var/log/dmesg (to read with cat or dmesg)

– mail logs
/var/log/maillog

– kernel logs
/var/log/kernel.log

– yum logs
/var/log/yum.log

JOURNALCTL

journalctl – Query the systemd journal

-Useful flags

* Jump to the end of the journal :
-e

* Filter since a slice of time:
--since "1 day/hour/minute/second ago" (or -S for –since)

* Filter since a date:
--since "2020-08-30 14:10:10" (or -S for –since)

* Same thing but with until or --U
Ex:
journalctl -U "2020-05-21 13:00:00"

* Combine since and until : Ex:
journalctl -S « 2020-05-21 09:00:00 » -U « 2020-05-21 13:00:00 »

* Follow :
-f

* Filter for a specific unit :
-u unitName

* Narrow from a specific boot :
-b [ID] (current boot if ID is omitted)

* No pagging :
--no-pager

* last 100 lines:
-n 100 --no-pager

Monitoring

top : process monitoring (memory, time, cpu usage)

CONTENT EXPLANATION
LINE 1 : overall load info

top - 11:35:01 up  4:54,  0 users,  load average: 1,98, 2,11, 2,13

– current datetime
– system uptime : time for which the system is running
– number of active user session
– load average on 1, 5 and 15 minutes
The load average shows the system load time for the last 1, 5 and 15 minutes.
How to interpret : 0.2 means that the system uses 20% of its resources, 1 means 100% and 1.5 means 150%. More than 1 means that the current load exceed the current system working capacity.
Warn : load average has to be interpreted with the number of cores (ex: for 2 cores, 2 means 100% usage).

LINE 2 : tasks (processes)

Tasks: 418 total,   3 running, 353 sleeping,   0 stopped,   1 zombie

Total number of tasks, running tasks, sleeping tasks, stopped tasks, zombie tasks

LINE 3 : Distribution in % of the CPU time

%Cpu(s): 12,0 us, 12,0 sy,  0,0 ni, 74,2 id,  1,2 wa,  0,0 hi,  0,5 si,  0,0 st

us : user tasks
sy : system tasks
ni : nice tasks (manually low priority)
id : idle
wa : wait for IO to complete
hi : hardware interrupt
si : software interrupt
st : steal time (needed time but stolen by other VM tasks)


Columns of the main table
TIME+ : minutes:second since that the process is running.

Actions
– shit-I : toggle Irix Mode (sum all CPUs when off, sum all CPUs / nb CPU when on)
– Z : toggle color mode
– X : hightlight the sorted column
– shift-P : toggle sort by CPU use
– shift-M : toggle sort by Memory use
– shift-N : toggle sort by process id
– shift-T : toggle sort by runtime


mpstat: processors statistics.
Require the sysstat package.
Usage :
mpstat [INTERVAL_IN_SEC] [NUMBER_SNAPSHOT] : cpu’s average stats.
To perform several measures/snapshots : we could pass the two parameters such as :
mpstat 30 3 : perform 3 measures with an interval of 30 seconds.

Flags :
-P CPU_NUMBER,CPU_NUMBER_OTHER,... : cpu stats for each specific cpu number or ALL to means all CPU.
ex: mpstat -P 1,2,5
ex: mpstat -P ALL

Column explanations
%usr – % CPU usage at the user level
%nice – % CPU usage for user processes labeled “nice”
%sys – % CPU usage at the system (Linux kernel) level
%iowait – % CPU usage idling waiting on a disk read/write
%irq – % CPU usage handling hardware interrupts
%soft – % CPU usage handing software interrupts
%steal – % CPU usage being forced to wait for a hypervisor handling other virtual processors
%guest – % CPU usage spent running a virtual processor
%idle – % CPU usage on idle time (no processes, and not waiting on a disk read/write)

IOSTAT command

Syntax : iostat [SEC]
If SEC is passed, it loops the execution every SEC seconds.

Helpful flags :
-x : display extended statistics.
-y : omit first report with statistics since system boot.
-z : omit output for devices without activity during the sample period.
-m : in megabytes
-p partition : execute for a specific partition.

Example : Output every 5 secondes detailed stats for devices with activity:
iostat -xyz 5

Example : Output stats for a specific partition :
iostat -p dm-32

Mapping between LVMs, device names and directories

* list mapping LVMs -> devices names :
ls -l /dev/mapper/*

* find directory where the LVM is mounted :
df /dev/mapper/MY_LVM

SAR
Sar stores metrics on CPU and disks.
It is helpful to analyze disk or cpu usage for a specific day or hours.

* Average cores stats : that is the default.
Similarly to mpstat, by sar outputs only the average of cores cpu stats.

* Specific cores stats : to specify a CPU number or output all CPU stats, we need to specify the -P flag with the same mpstat -P syntax .

* list stats of the day :
sar

* list continuously current stats at second interval:
sar secondInterval
Ex: sar 10 : output the current stats every 10 seconds.

Helpful flag : -A : list all stats
-d : add the report activity for each block (d)evice.
When data are displayed, the device specification dev m-n is generally used
-f : specify a sar data file (26th day of the current month) :
ex: sar -d -f /var/log/sa/sa26

IOTOP

iotop is as the top command but for IO.
So it also gives the command/program that perform the IO.

measure duration of a command :
time myCmd

MEMORY COMMANDS

* display ram/memory info
rss free flags :
-m to get value in Megabytes and without suffix
-g to get value in Gigabytes and without suffix
-h to get human display

For example :
free -g

                  total        used        free      shared      buff/cache     available
Mem:                21           13           2           0               4             6
Swap:               0             0           0

« buff/cache » : the memory used to cache/buffer resources on the FS.
By default, Linux uses unused memory to cache FS resources.

« free » : the current free memory.
It doesn’t mean that if a process requires more memory or that a new process is created, we could not allocate memory beyond which is indicated.

Indeed, « available » is the max memory that the OS may produce if really needed (pressure case).
So in this case, the OS may reduce memory used by shared and buff/cache.

* free cache memory

echo 3 | sudo tee /proc/sys/vm/drop_caches
(we can execute synch before to catch also in progress cache objects.)

3 is a kernel marker to mean objects and pagecache clear. It includes 1 (pagecache) and 2 (objects cache).

* display fine grained memory info (free memory relies on that) :
cat /proc/meminfo

* display processes with among other things their memory info (RSS and VSZ) :
*ps -aux (these are displayed in distinct columns)

Useful flags :
-p fooPid : filter on a specific process
w : specify a width
ww : no width limit

What is RSS ? The Resident Set Size measures how much memory is allocated to a process. It excludes the swapped memory but it includes memory and their pages from shared libraries

What is VSZ ? The Virtual Memory Size measures the total memory that a process can access. It includes the swapped memory, allocated but not used memory and shared libraries memory.

* Sum RSSs for processes with a command matching with a pattern and display it in megabytes:

ps -aux --sort -rss | awk 'NR>1 && $11 ~ /unicorn/ {sum+=$6} END{print "sum="sum/1000}'

Explanations :
NR>1 : skip the first line
$11 ~ /unicorn/ : the 11th column value has to match with the pattern « unicorn »
END {action} : after all iterations and processing, execute that action

DISK COMMANDS

lsblk command (list block devices)

lsblk lists information about all available or the specified block devices.
It reads the sysfs filesystem and udev db to gather information.

Default output :

NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0  99,4M  1 loop /snap/core/11420
loop1     7:1    0  99,4M  1 loop /snap/core/11316
loop2     7:2    0 175,4M  1 loop /snap/postman/133
loop3     7:3    0 175,4M  1 loop /snap/postman/132
sda       8:0    0   160G  0 disk 
├─sda1    8:1    0 119,9G  0 part /
├─sda2    8:2    0    40G  0 part 
├─sda14   8:14   0     4M  0 part 
└─sda15   8:15   0   106M  0 part /boot/efi
sr0      11:0    1  1024M  0 rom

Helpful flags :
-o, --output list : Specify which output columns to print.
Ex: -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT
-O, --output-all : Output all available columns.

fstab command

Static information about the filesystems.
The file fstab (located at /etc/fstab) contains descriptive information about the filesystems the system can mount.
fstab is only read by programs, not written.
it is the duty of the system administrator to properly create and maintain this file.
The order of records in fstab is important because fsck(8), mount(8), and umount(8) sequentially iterate through fstab doing their thing.
fstab content overal description :
– each filesystem is described on a separate line.
– fields are separated by tabs or spaces.
– lines starting with ‘#’ are comments. – blank lines are ignored.

fstab column description :
field 1 : the block special device or remote filesystem to be mounted.
Several possibilities :
device name, LABEL=<label> or UUID=<uuid>
Most robust way : LABEL or UUID.

field 2 : the mount point (target) for the filesystem.

field 3 : the type of the filesystem.

field 4 : the mount options associated with the filesystem.
It is a comma-separated list of options.
It contains at least the type of mount (ro or rw).
Basic filesystem-independent options are:
defaults : use default options: rw, suid, dev, exec, auto, nouser, and async.
noauto : do not mount when « mount -a » is given (e.g., at boot time)
user : allow a user to mount
owner : allow device owner to mount
comment or x- : for use by fstab-maintaining programs
nofail : do not report errors for this device if it does not exist.

field 5 : Does the filesystem need to be dumped. Defaults to zero (don’t dump) .

field 6 : Determine the order in which filesystem checks are done at boot time.
The root filesystem should be specified with a fs_passno of 1. Other filesystems should have a fs_passno of 2.

The following is an example of a fstab file.
It starts two ext4 partitions at boot : one with the label desktop-rootfs as root partition (/) and another with a UUID=6a3… that is mounted on /mnt/workspaces.

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
LABEL=desktop-rootfs  /       ext4    errors=remount-ro   0   1
#/swapfile      none    swap    sw      0       0
LABEL=UEFI      /boot/efi       vfat    defaults        0 0
UUID=6a344595-417f-47f6-94e4-21b722042806 /mnt/workspaces/ ext4    defaults   0   2

e2label command

Change the label on an ext2/ext3/ext4 filesystem.
Ex:
e2label /dev/sda2 workspaces

fdisk command

Manipulate disk partition table tool.
fdisk -l : List the partition tables for all devices
fdisk -l sda sdc .. : List the partition tables for the specified devices

parted command

It is a partition manipulation program, an alternative to fdisk.

Some commands:
– Display unallocated space:
print free
For each partition that does not have all spaces allocated,we have below the partition line, an additional line that indicates how much unallocated space is available.
For example:

3      290MB   178GB   178GB   ntfs         Basic data partition          msftdata
       178GB   178GB   146kB   Free Space

Mount command

Mount command description and standard use

mount – mount a filesystem

All files accessible in a Unix system are arranged in one big tree, the file hierarchy, rooted at /.
These files can be spread out over several devices.
The mount command serves to attach the filesystem found on some device to the big file tree.
Conversely, the umount(8) command will detach it again

The standard form of the mount command is:
mount -t type device dir

Helpful commands :
Mount all filesystems mentioned in fstab except for those whose line contains the noauto keyword :
mount -a [-t type] [-O optlist]

Procedure : mount a partition to a directory

1)Identify partition to mount by listing them :
lsblk -o NAME,FSTYPE,LABEL,SIZE,MOUNTPOINT

2)Create a directory under /mnt that the newly mounted partition will be mounted to (here the worskpaces folder):
sudo mkdir /mnt/workspaces

3)Mount the partition to that new created directory:
sudo mount -t auto -v /dev/sda2 /mnt/workspaces/
Output :
mount: /dev/sda2 mounted on /mnt/workspaces.
Explanations :
-t : type of partition (auto means guess)
-v : verbose

4) To unmout the partition :
sudo umount -l /dev/sda2

Procedure : make one directory or multiple directories to use the mount

That procedure may be seen as the next step after the below procedure : mount a partition to a directory.
Indeed, while mounting a partition to /mnt/fooDir/ provides a concrete way to create files and folder in our partition, it looks undesirable for users and applications to refer to create all data in /mnt/fooDir/.

Scenario 1 : we want that an existing directory to use the new mount : /mnt/workspaces/
Suppose we have an existing folder with much data /home/david/workspace-java that uses another mount that lacks of space
Now We want that the directory content be hosted on the new mount.
To achieve that :
– we move /home/david/workspace-java/ to /mnt/workspaces/ :
mv /home/david/workspace-java/ to /mnt/workspaces/
– From the /home/david/ current directory, we create the sym link workspace-java with as target /mnt/workspaces/workspace-java/ :
pwd -> /home/david/
ln -s /mnt/workspaces/workspace-java workspace-java

And that’s all, we could check :
david@david-Virtual-Machine:~$ ls -la /home/david/workspaces


Scenario 2 : we want that a new directory to use the new mount : /mnt/workspaces/
/home/david/workspace-python is the new directory that is going to use that mount
To achieve that :
– we create a /mnt/workspaces/workspace-python/ directory :
mkdir /mnt/workspaces/workspace-python/
– From the /home/david/ current directory, we create a sym link workspace-python with as target /mnt/workspaces/workspace-python/ :
pwd -> /home/david/
ln -s /mnt/workspaces/workspace-python workspace-python

And that’s all, we could check :
david@david-Virtual-Machine:~$ ls -la /home/david/workspaces

Unmount commands

umount command 

umount MOUNTPOINT
Flags :
-f : force
-l : lazy
Note :
If umount failed with error : Transport endpoint is not connected, we can try with fusermount command as : fusermount MOUNTPOINT -uz

fusermount command 

fusermount MOUNTPOINT
Flags :
-u : unmount
-z : lazy unmount
df tool command(disk filesystem usage)

df helpful flags :
* display filesystems disk usage in (h)uman format:
-h
* display inodes information instead of disk blocks information (the default):
-i
* display partition type:
-T

du command : disk file space usage tool

It computes the size of each directory (recursively size) along the current dir.
Important:
– the size associated to the current dir (.)is the sum of all.

Some common usages:
* display current directory disk usage in (h)uman format for each directory:
du -h
* display disk usage for each recursively directory:
du -h */
Important: – It doesn’t show information for hidden directories
* To show size of hidden directories, we can use this regex expression:
du -hs .[^.]*

du helpful flags:
-s : display only the summarize and not each directory size
-c : display a overall total size at the end
-h : display human format
-d N : max-depth to N (0=current dir, 1=child’s first dir, and so for )

ps COMMANDS

Basic uses

*list every(e) processs/users with full-format listing (f) (includes the ppid):
ps -ef

*list all (a) processes (every users) with user-oriented (u) format and tty restriction excluded (x) :
ps aux

*list processes of the current user with user-oriented (u) format and tty restriction excluded (x) :
ps -ux

*list processes matching with ids :
ps -p « 1 2 »
or
ps -p 1,2

Customize the output

*list e(very) process with customized o(utput) format (here we output elapsed time, rss memory and the command):
ps -eo field1,field2,...
Example:
ps -eo pid,etime,rss,command,nice

Ordering the output

*list processes… and sort them ascendingly :
ps -aux –sort STANDARD FORMAT SPECIFIERS

*list processes… and sort them descendingly :
ps -aux –sort -STANDARD FORMAT SPECIFIERS

Examples :
* list & sort by ascending creation date :
ps -aux --sort start_time

* list & sort by higher memory usage (reserved memory) :
ps -aux --sort -rss (resident set size)

*list & sort by higher cpu usage (reseved memory) :
ps -aux --sort -pcpu (resident set size)

*list & sort by higher memory usage and for pids in the specified values :
ps --sort -rss -u -p 4060 5064 7117 7708 21252

* list processes sorted by higher memory usage and output in MB :
ps -eo pid,etime,rss,command --sort -rss | awk '{print $1 " - " $2 " - " $3/1024 "MB - " $4 $5 $6 $7 $8 $9}' | head -20

Kill command

*kill a process (no force) :
kill pid

*kill multiple processes :
kill pid1 pid2 …

*kill a range of process :
kill {pidFrom..pidTo}

*kill a process (force) :
kill -9 pid
*test whether the process is alive :
kill -0 pid


We can use it like that to wait for a process be effectively killed :

kill -9 $PID # kill but not necessarily right now
while kill -0 $PID; do 
    sleep 1
done

*fill a process from the parent pid process :
pkill -P ppid


* Filter processes which the command contains a specific word (here « chrome ») :
ps -elf | awk ‘$15 ~ /chrome/ {print}’

* Kill processes which command contains a specific word (here autokey)
ps -eo pid,command | grep -o -P '\d+\s+.*autokey' | cut -d ' ' -f1 | awk 'system("kill -9 " $1)'

* List pid of these processes separated by a space
ps -elf | awk ‘$15 ~ /chrome/ {listPid=listPid » « $4} END{print listPid}’

* force kill these processes :
sudo kill -9 $(previousCommand)

NETWORK COMMANDS

Netstat command

Status in the ouput :
ESTABLISHED : the socket has an established connection. CLOSE_WAIT : the remote endpoint (other side of the connection) has closed the connection.
TIME_WAIT : the local endpoint (this side) has closed the connection.
FIN_WAIT1 : The socket is closed, and the connection is shutting down.
FIN_WAIT2 : Connection is closed, and the socket is waiting for a shutdown from the remote end.

*list all processs with port listening on TCP (add sudo to display all processes):
netstat -tnlp
t (tcp) = show only those with TCP protocol (u for UDP protocol)
n (numerical) = numerical addresses instead of resolved symbolic hosts, ports or user names
l (listening) = show only listening sockets. (These are omitted by default.)
p (program) = show the PID and name of the program

*list pid(s) of process that listening on a specific TCP port (example 8095 port) :
lsof -t -i :8095 -s TCP:LISTEN

Nc command

nc or ncat – Concatenate and redirect sockets
It is a feature-packed networking utility which reads and writes data across networks from the command line.

*Test a port on a host :
telnet fooHost 8090
or
nc -zv fooHost 8090

*Test several ports on a host :
nc -zv fooHost 8090 8091 8092
or with a port range :
nc -zv fooHost 8090-8092

*Create a listener socket on the port 9000:
nc -l -k -p 9000 -v
flags:
-v : verbose
-l : listener
-p PORT : port listener

*Connect to a listener socket on the port 9000 of the machine hostOrIp :
nc hostOrIp 9000


tcpdump
Capture and inspect network traffic on the host.

The general syntax of the output :
[Timestamp] [Protocol] [Src IP].[Src Port] > [Dst IP].[Dst Port]: [Flags], [Seq], [Ack], [Win Size], [Options], [Data Length]

The flags definition :

[.] - ACK (Acknowledgment)
[S] - SYN (Start Connection)
[P] - PSH (Push Data)
[F] - FIN (Finish Connection)
[R] - RST (Reset Connection)
[S.] - SYN-ACK (SynAcK Packet)

The sysctrl settings/details

the kernel

– Directory: /proc/sys/kernel

– Console log level:
–query the current console log level:
read the file /proc/sys/kernel/printk

– Set the console log level:
To make the change persistent we must not modify the above file but updating the /etc/sysctl.conf file.
For example:
kernel.printk = 3 4 1 3

– Level and value description

The kernel log levels are:
 
0 (KERN_EMERG)
The system is unusable.
 
1 (KERN_ALERT)
Actions that must be taken care of immediately.
 
2 (KERN_CRIT)
Critical conditions.
 
3 (KERN_ERR)
Non-critical error conditions.
 
4 (KERN_WARNING)
Warning conditions that should be taken care of.
 
5 (KERN_NOTICE)
Normal, but significant events.
 
6 (KERN_INFO)
Informational messages that require no action.
 
7 (KERN_DEBUG)
Kernel debugging messages, output by the kernel if the developer enabled debugging at compile time.

What do the 4 values represent?

console_loglevel: messages with a higher priority than this will be printed to the console
default_message_loglevel: messages without an explicit priority will be printed with this priority
minimum_console_loglevel: minimum (highest) value to which console_loglevel can be set
default_console_loglevel: default value for console_loglevel

The sysctl cmd :

That is used to modify kernel parameters at runtime. The parameters available are those listed under /proc/sys/.
Procfs is required for sysctl support in Linux. You can use sysctl to both read and write sysctl data.

Usages : 

Reload the sys settings without reboot :
sysctl --system

Load in sysctl settings from the file specified or /etc/sysctl.conf if none given.
sysctl -p

Update/add property for the current session :
sysctl -w foo-section.foo-prop=12345
Example to set vm.max_map_count :
sysctl -w vm.max_map_count=262144

Update/add property permanently :
Edit the /etc/sysctl.conf file and add the property such as :
foo-section.foo-prop=12345

Virtual memory tuning

Tuning of the virtual memory (VM) subsystem of the Linux kernel and
the writeout of dirty data to disk : these config files are located in /proc/sys/vm.

Doc : https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Resource Limit for users

The /etc/security/limits.conf file stores them.

The native shell command : ulimit
Provides control over the resources available to the shell and processes it creates.

Syntax :
ulimit [-SHabcdefiklmnpqrstuvxPT] [limit]

Report and help flags :
–help : display help
-a : all current limits are reported

All others are limit modification flags.

FILES AND DIRECTORIES

FILES AND DIRECTORIES : CREATION AND RIGHTS

ln command :
ln [OPTION]... [-T] TARGET LINK_NAME (1st form)
The 1st form, create a link to TARGET with the name LINK_NAME.

* create a symbolic link:
ln -s existingFileOrPath symbolicFile

* delete a symbolic link (without deleting the source):
rm symbolicFile

To skip the confirmation:
rm -f symbolicFile

To delete a symbolic link to folder, don’t suffix the directory with a / such as rm symbolicDir/ because it will fail.
You have to specify rm symbolicDir
useful flag :
-f : (for force) remove existing destination files (It spares us from removing the link).

* add the rwx right on a file/folder for the owner :
chmod u+rwx fooFile

* remove the rwx right on a file/folder for the owner :
chmod u-rwx fooFile

To apply rights to the file’s owner : u
To apply rights to the file’s group : g
To apply rights to others : o
To apply rights to all : a (same as ugo)

* To apply same rights on several targets (ex: owner and group) :
chmod ug+rwx fooFile

* To apply distinct rights on several targets :
chmod g+r,u+rw fooFile

* To apply recursively rights :
-R flag

* To apply on files with a specific extension inside a directory recursively:
-chmod a+rx -R fooFolder/*.sh
* To apply the execution right recursively on a directory for the user and the group:
find myFolder/ -type d -exec chmod u+x,g+x {} \;

FILES AND DIRECTORIES : ATTRIBUTES

lsattr command
List file attributes (a ls like command focused on attributes)

Helpful flags :
-a : List all files in directories, including files that start with `.’
-l : Print the options using long names instead of single character abbreviations
-R : Recursively list attributes of directories and their contents

Output example :

lsattr  -a
 
-----a--------e--- ./chien
-----------I--e--- ./..
----i---------e--- ./chat
--------------e--- ./.

The letters represent the attribute modes : a (append only), e (extent), i (immutable), I (???).
See chattr command details (below we have it) to have a full reference.

chattr command
Change file attributes.

Important things
– requirement : sudoer !
– append-only, immutable modes applied on a directory doesn’t add the attribute recursively while the effect on a directory is also applied recursively

General syntax 
chattr [ -RVf ] [ -v version ] [ mode ] files...
Mode is +-=[aAcCdDeijsStTu]where :
‘+’ causes the selected attributes to be added to the existing attributes of the files
‘-‘ causes them to be removed
‘=’ causes them to be the only attributes that the files have

About ‘aAcCdDeijsStTu’ mode meaning :
– append only (a)
– no atime updates (A)
– compressed (c)
– no copy on write (C)
– no dump (d)
– synchronous directory updates (D)
– extent format (e)
– immutable (i)
– data journalling (j)
– secure deletion (s)
– synchronous updates (S)
– no tail-merging (t)
– top of directory hierarchy (T)
– undeletable (u)

Options :

-R           Recursively change attributes of directories and their contents.
-V           Be verbose with chattr's output and print the program version.
-f           Suppress most error messages.
-v version   Set the file's version/generation number.

Examples :
Add append-only attribute on a folder :
chattr +a foo/.
So rm -rf foo/ will fail for every file/folder inside along the foo/ directory itself  :

rm: cannot remove foo/aa: Operation not permitted
rm: cannot remove foo/a: Operation not permitted

USERS AND GROUPS

* id [option] [user]:

print user and group ids. Default is for current user and display both text and id group.

flag :
-u : print only the effective userid
-g : print only the effective groupid
-G : print all groupids


* List all users and some other informations :
cat /etc/passwd

Example : git:x:998:998::/var/opt/gitlab:/bin/sh

– git is the username
– x means encrypted password
– /var/opt/gitlab is the user home directory
– /bin/sh is the login shell

* list all groups :
cat /etc/group

Example : adm:x:4:syslog,david
– adm is the group
– x means encrypted password
– syslog,david are users of this group

* list groups of the current user :
groups

* list groups of a specific user :
groups user


Add groups or users

Warn :  adduser and addgroup examples are not portable.
These work well with debian-based but may fail with Redhat based because these command differ in terms of options/features according to the OS.
For more portable command, favor useradd and usermod command.

adduser and addgroup commands

adduser and addgroup are interractive wrappers for the commands useradd and groupadd.

* add a user in the system (require a reboot):
sudo adduser username

* add a user to a group (require a reboot):
sudo adduser username groupname

* create a group :
sudo addgroup groupname

**useful flags for adduser and addgroup :
-S : system group or user

usermod commands
Helpful flags :
-a, --append
Add the user to the supplementary group(s). Use only with the -G option.
-G, --groups GROUP1[,GROUP2,...[,GROUPN]]]
Set the list of supplementary groups which the user is also a member of.

* set the shell for the user
sudo usermod -s /bin/bash username  

* add an existing user to a group :
usermod -a -G groupName username
Ex: usermod -a -G docker david

useradd commands

* create a user (without creating the home directory)
useradd foo-user  

* create a user and its home directory
useradd -m foo-user  

passwd command

Set/overwrite interactively the password of an user (do it as root) :
passwd foo-user

userdel commands (low level command)

* delete a user
userdel foo-user  

Helpful flags :
-f, --force
If -r is used, forces the removal of the user account, even if the user is still logged in.
It also forces the remove of the user’s home directory.  
-r, --remove
Files in the user’s home directory will be removed along with the home directory itself and the user’s mail spool.  

chown command : change owners of files and directories

Basic syntax :
sudo chown user:group file

Helpful flags :
-R : recursive application
The recursive flag (-R)  defines also a symbolic link traversal behavior (-P) that we could overrride. We cannot specify more than one (the last one wins) :
-P : do not traverse any symbolic links (default)
-H : if a command line argument is a symbolic link to a directory, traverse it
-L : traverse every symbolic link to a directory encountered


— rights files

* sudoers file : /etc/sudoers
* edit sudoers : sudo visudo
* sudoers.d directory : /etc/sudoers.d

* entry general syntax :

username host1, host2,...      =   (user1, user2... : group1, group2,...)   cmd1, cmd2, ....
   |              |                                 |                             |
   |              |                                 |                             |
user to grant  list of appli-      user and groups the user may use        Commands that 
               cable hostnames       to run as. default : root.            could be run

* common simplified syntax :
username ALL = cmd1, cmd2, ….

Example :

* sudoer right for any commands :
username ALL = NOPASSWD: ALL

* sudoer right for a command for a user :
username ALL = cmd

* sudoer right for a command for a group :
%groupname ALL = cmd

* same thing without password required :
username ALL = NOPASSWD: cmd

Password expiration

chage : change user password expiry information
General syntax : chage [options] USER Useful flags :
-l : list account information
-M NUMBER : set max number of days before password expiration. -1 value means never expire

MISC

* Generate a stacktrace for a Linux process :
gstack PID

* Find file locks :
lslocks (to say list system locks)

Crontab command

Crontab overview

– Jobs executed by user that define them.
– By default the output (std and error) of the job is sent by mail
– If we override the output destination inside (both std and error) a cron job , no mail with the output is sent for that job.

Crontab format

 +---------------- minute (0 - 59)
 |  +------------- hour (0 - 23)
 |  |  +---------- day of month (1 - 31)
 |  |  |  +------- month (1 - 12)
 |  |  |  |  +---- day of week (0 - 6) (Sunday=0 or 7)
 |  |  |  |  |
 *  *  *  *  *  command to be executed

Examples :
– every minutes : * * * * *
– every 15 minutes : */15 * * * *
– every hour : 0 * * * *
– every day at 8h AM : 0 8 * * *
– Every 5 hours : 0 */5 * * *
– every sunday and wednesday at 12h30 AM: 30 12 * * 0,3
Explanations :
* : at each unity (ex: for hours it means every hour)
*/15 : every 15 unities (ex: for minutes it means every 15 minutes)
0 : at each time that 0 occurs (ex: 0 for minutes means every hour)
0,3,6 : at each that 0, 3 or 6 occurs.

Common Crontab commands

Edit crons of the current user :
crontab -e

Edit crons of root user :
sudo crontab -e

List crons of all users :
for user in $(cut -f1 -d: /etc/passwd); do echo $user; crontab -u $user -l; done
Alternative :
cat /etc/passwd | cut -d: -f1 | xargs -I% sh -c "echo 'user %' && crontab -l -u % "

cron logs file:
/var/log/cron.log

Allow or deny the cron execution for some users :
Add the users in :
/etc/cron.allow and or /etc/cron.deny

Common Cron issues

Problem : cron.log logs command execution but not the output (standard and error) produced by the command.
Symptom : we see a info like that in cron.log after the command execution :

CRON[18182]: (CRON) info (No MTA installed, discarding output)

It means that no one Mail Transfer Agent is installed on the system.
Solutions :
– Either override the default cron behavior (that is sending an email) by redirecting both the std and error output.
– Or install a MTA on the system.

Crontab hints

job using the current date (dd-mm-yyyy) as log file of the cron task output :
Task that purge dangling docker images every day at 00h00 and store the log file in the user directory of the user’s cron :
0 0 * * * docker image prune -f > ~/cron-tasks/$(date +\%Y-\%m-\%d) 2>&1
Why escaping the % character ?
Because the crontab doc states that The « sixth » field specifies the command to run and that the entire command portion of the line, up to a newline or % character, will be executed and at last all data after the first % will be sent to the command as standard input.
To avoid % to be interpreted such as, we need to escape it with backslash \.

jobs setting bash as shell
Add in first line :
SHELL=/bin/bash

Crontab examples

Delete archived files that are older 5 days :
SHELL=/bin/bash
30 12 * * 0,3 echo "Execution at : $(date)" &>>/var/log/cron/root-purge-archived-file-in-varlog.txt && find /var/log/ \( -name "*.gz" -or -name "*.gzz" \) -mtime 5 -exec rm -fv {} &>>/var/log/cron/root-purge-archived-file-in-varlog.txt \;

Anacron

Anacron is used to execute commands periodically, with a frequency specified in days.
Unlike cron(8), it does not assume that the machine is running continuously.
Hence, it can be used on machines that are not running 24 hours a day to control regular jobs as daily, weekly, and monthly jobs.

Configuration file : /etc/anacrontab

Configuration Example :

# environment variables
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
RANDOM_DELAY=30
# Anacron jobs will start between 6am and 8am.
START_HOURS_RANGE=6-8
# delay will be 5 minutes + RANDOM_DELAY for cron.daily
1         5    cron.daily          nice run-parts /etc/cron.daily
7         0    cron.weekly         nice run-parts /etc/cron.weekly
@monthly  0    cron.monthly        nice run-parts /etc/cron.monthly

This example shows how to set up an Anacron job similar in functionality to /etc/crontab which starts all regular jobs between 6:00 and 8:00 only. A RANDOM_DELAY which can be 30 minutes at the most is specified.

Logrotate

logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled daily, weekly, monthly, or when it grows too large.

* execute logrotate :
logrotate /etc/logrotate.conf

Classic configuration location of logrotate : /etc/logrotate.conf

Include configuration location of logrotate : /etc/logrotate.conf

Log and temporary files that may occupy spaces

/var/log/journal/ folder
It contains log information displayed by journalctl.
We can query the space used with : journalctl --disk-usage
To gain space right now, reduce the current size to a specific size :
journalctl --vacuum-size=500M

To limit the size of the journal whenever, set that property in the [Journal] section of /etc/systemd/journald.conf :
SystemMaxUse=XXXMB such as : SystemMaxUse=1024MB

Global preferences

Select the default editor for the current user :
select-editor
The results are stored as SELECTED_EDITOR in ~/.selected_editor, which is sourced and used by sensible-editor.
SELECTED_EDITOR is overridden by the VISUAL and EDITOR environment variables.

IPTABLES Command (Agnostic Linux distros)

Flags

-L, --list [chain] : List all rules in the selected chain. If no chain is selected, all chains are listed.

Chains explanation

Type of chains
iptables uses three different chains: INPUT, OUTPUT and FORWARD.
According to the flow direction, iptables selects the relevant chain
INPUT : incoming packets.
OUTPUT : packets emitted by the host.
Their destination is usually another host, but can be the same host via the loopback interface, so not all packets that go through OUTPUT are in fact outgoing.
FORWARD : packets that are neither emitted by the host nor directed to the host.
They are the packets that the host is merely routing.

Relation between chains, rules and target
Each chain is a list of rules which can match a set of packets.
Each rule specifies what to do with a packet that matches.
This is called a target, which may be a jump to a user-defined chain in the same table.
A rule specifies criteria for a packet and a target.
If the packet does not match, the next rule in the chain is examined; if it does match, then the next rule is specified by the value of the target.

Target special value

ACCEPT : let the packet through.

DROP : drop the packet by giving no information to the client.
Recommended way for incoming packets coming from « outside ».

REJECT : drop the packet by giving some information to the client.
Recommended way for incoming packets coming from « inside » (our network).

DROP OR REJECT : Concretely REJECT gives more information to the client than DROP does.
Indeed DROP  produces to the client a response that may leave think that the host is not reachable while it is whereas REJECT is be more transparent : the host will appear as reachable, so a port issue is guessed from the client side.

RETURN : stop traversing this chain and resume at the next rule in the previous (calling) chain.
If the end of a built-in chain is reached or a rule in a built-in chain with target RETURN is matched, the target specified by the chain policy determines the fate of the packet.

Reject details
As seen above, reject gives some information to the client about the error.
To override the error returned, we could specify with REJECT the flag --reject-with foo-type Where foo-type can be:
icmp-net-unreachable
icmp-host-unreachable
icmp-port-unreachable
icmp-proto-unreachable
icmp-net-prohibited
icmp-host-prohibited
icmp-admin-prohibited

Add or delete rules

Two ways :
1) Do a copy of the /etc/sysconfig/iptables file that contains current rules in a file (ex: foo-iptables.txt) and do your changes on that copy.
When changes are finished, you can update the iptables with iptables-restore :
iptables-restore < foo-iptables.txt

2) Use iptables command.
Create a new chain called FOO :
iptables -N FOO

Delete a rule :
sudo iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited

Add a rule :
sudo iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

Rules update persistence

Updating /etc/sysconfig/iptables will make the changes to be visible at runtime and for the current session.
To make these changes persistent after shutdown/startup, ip tables content needs to be stored at a specific place and restored at startup.
To achieve it we need the iptables service.
With that enabled, we persist changes in that way:
service iptables save

Delete all rules

# clear all rules
iptables --flush
# make it persistent
service iptables save
# check that iptables has well be updated with default rules
cat /etc/sysconfig/iptables

Note : if the iptables file is not updated correctly, you can also delete it manually first and reenter the commands.

Software firewall with Debian based

Two ways to manage the firewall :
– either iptables with iptables-services
– either firewalld
Not both ! It cannot work !
Since Centos 7, firewalld is enabled as a service and iptables-services is not installed.
To use the later, we need to disable the firewalld service and to install and enable iptables-services as a service :
systemctl disable firewalld
yum -y install iptables-services
systemctl enable iptables

Software firewall with Debian based

Two ways to manage the firewall :
– either iptables with iptables-services

Backup files and directories with rsync

rsync is a fast, versatile, remote (and local) file-copying tool
Local syntax :
rsync [OPTION...] SRC... [DEST]

Remote shell syntax:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST

Helpful flags :
-z : compress during the transport
-a, --archive : archive mode (shortcut for -rlptgoD flags)
-R, --relative : use relative path names
This means that the full path names specified on the command line are sent to the server rather than just the last parts of the filenames.

--exclude=... --exclude=... : simplified form of the --filter option that defaults to an exclude rule

--include=... --include=... : simplified form of the --filter option that defaults to an include rule

Examples :
* Local sync of some kubernetes directories by keeping the whole tree (-R option to keep var/lib/ and /etc/kubernetes) while excluding the « pods » folder (helpless and potential huge):

rsync --exclude=pods -azR /var/lib/kubelet/ /etc/kubernetes/ /mnt/workspaces/backup-things/

Result :

ls -la /mnt/workspaces/backup-things
total 16
drwxr-x--- 4 root kube 4096 août  27 15:17 .
drwxrwxrwx 7 root root 4096 août  27 14:42 ..
drwxrwxr-x 3 root root 4096 août  19 12:09 etc
drwxr-xr-x 3 root root 4096 avril  4 12:41 var

backup the whole filesystem with mksquashfs

Export the filesystem

1)Export of the filesystem data into a file.
We need to executes the command from the root path because the tool exports permissions relative to the current path where the command is executed.
– The first parameter is the source of the export (by specifying the / path, we export permissions for the whole filesystem)
– the second parameter is the file where the export has to be generated
– The flag -e is path we want to exclude

sudo su
cd /
sudo mksquashfs / /media/temp/root-backup.sqsh -e media dev run mnt proc sys tmp

The export writes in the standard output, this contains summarize of the export but also all warnings and problems encountered during the export processing.
note: even if some problems happen, the processing goes on
Here a snippet of the generated output:

Unrecognised xattr prefix system.posix_acl_access
[=================================================================================================================================================================================-         ] 1101609/1151697  95%
Unrecognised xattr prefix system.posix_acl_access
[=================================================================================================================================================================================|         ] 1101994/1151697  95%
Unrecognised xattr prefix system.posix_acl_access
[==================================================================================================================================================================================-        ] 1102387/1151697  95%
File //var/log/kern.log changed size while reading filesystem, attempting to re-read
[===================================================================================================================================================================================\       ] 1113656/1151697  96%
File //var/log/syslog changed size while reading filesystem, attempting to re-read
[=====================================================================================================================================================================================|     ] 1123360/1151697  97%
File //var/opt/gitlab/postgresql/data/pg_subtrans/0002 changed size while reading filesystem, attempting to re-read
[=========================================================================================================================================================================================- ] 1150430/1151697  99%
Failed to read file //var/opt/gitlab/prometheus/data/chunks_head/000037, creating empty file
[=========================================================================================================================================================================================- ] 1150856/1151697  99%
... ... ... ...
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 25294263.64 Kbytes (24701.43 Mbytes)
	42.14% of uncompressed filesystem size (60024722.29 Kbytes)
Inode table size 10601528 bytes (10353.05 Kbytes)
	28.73% of uncompressed inode table size (36899769 bytes)
Directory table size 10647821 bytes (10398.26 Kbytes)
	39.85% of uncompressed directory table size (26720468 bytes)
Xattr table size 1974 bytes (1.93 Kbytes)
	20.43% of uncompressed xattr table size (9662 bytes)
Number of duplicate files found 259623
Number of inodes 1006289
Number of files 762105
Number of fragments 43422
Number of symbolic links  147294
Number of device nodes 175
Number of fifo nodes 64
Number of socket nodes 13
Number of directories 96638
Number of ids (unique uids + gids) 56
Number of uids 30
	root (0)
	lp (7)
	david (1000)
	gitlab-redis (997)
	gitlab-psql (996)
	git (998)
	gitlab-prometheus (995)
    ...
Number of gids 49
	root (0)
	ssl-cert (110)
	david (1000)
	git (997)
	gitlab-redis (996)
	gitlab-www (998)
	gitlab-psql (995)
	developper (1001)
	...

We can see some warnings about acl. These cannot be exported by this tool, it is a known issue of the tool.
We can bypass this problem by exporting aside the acls

2) If we have some acl, we export the permissions such as:
getfacl -R . > permissions.facl

Import the filesystem

1)Import the filesystem data .

2) If we have exported acls, we reimport them such as:
setfacl --restore=permissions.facl