Aakarsh Nair

Emacs/Evil-mode - A basic reference to using evil mode in Emacs.

2017-03-09T00:00:00+00:00

This cheat sheet will be used to provide a very basic reference to using evil mode , vi emulation layer inside of emacs. For people who would want to still have access to base emacs lisp layer while working with vim’s modal, noun verb editing features. Since evil-mode is trying its best to emulate vim, this post might even serve as a basic reference to vim’s editing features for folks familiar emacs with cursory (no pun intended) curiosity with vim.

Introduction

Evil mode for emacs has been out for a while and is available to clone from

     $ git clone https://github.com/emacs-evil/evil.git

It can be installed by cloning the above repo into your .emacs.d/site-isp and adding it to your load-path:

     (add-to-list 'load-path "~/.emacs.d/site-lisp/evil")
     (require 'evil)
     (evil-mode 1)

Modes and States

If you eval the above code you will see a <N> in the mode line. This reflects the fact that evil mode is running in vim’s equivalent of normal mode. To revert from normal vim emulation to emacs use Ctrl-z. This will put you back into emacs mode you are used to. Typing Ctrl-z again will take you back into evil’s vim normal state. Other list of modes include:

`<N>` - Normal state - for most vims commands.
`<V>` - Visual state - Vim's rich selection sate.
`<R>  - Replace state
`<M>` - Motion state
`<E>` - Emacs state - will be receptive to usual emacs key bindings in the buffer.

Each state has will have its own customization and bindings. Most of the bindings can be seen in evil-maps.el.

Some of the key maps are


evil-emacs-state-map 
evil-ex-completion-map
evil-inner-text-objects-map
evil-insert-state-map
evil-motion-state-map
evil-normal-state-map
evil-operator-state-map
evil-outer-text-objects-map
evil-read-key-map
evil-replace-state-map
evil-visual-state-map
evil-window-map

You can checkout evil-commands.el for list of examples of commands that have already been defined. As you begin to look at the custom macro evil-define-motion used while defining motion types you will see most of the commands take in a count for the number of times the motion needs to be performed. This is a basic implementation of vim’s numeric parameterization of motion commands See references.

;; Simpler way to exit to normal states than pressing <ESC>
(define-key evil-visual-state-map (kbd "C-c") 'evil-normal-state)
(define-key evil-insert-state-map (kbd "C-c") 'evil-normal-state)
(define-key evil-motion-state-map (kbd "C-e") nil)
(define-key evil-visual-state-map (kbd "C-c") 'evil-exit-visual-state)

Additional plugins

While base evil is fairly feature complete I have found the following additional plugins to be useful at times.

Key chord mode

Key chord mode will make it easier to jump between modes which becomes important when we enter the world of modal editing.

(require 'key-chord)
(key-chord-mode 1)

(require 'key-seq)
(key-chord-mode 1)

(key-chord-define evil-normal-state-map ",," 'evil-force-normal-state)
(key-chord-define evil-visual-state-map ",," 'evil-change-to-previous-state)
(key-chord-define evil-insert-state-map ",," 'evil-normal-state)
(key-chord-define evil-replace-state-map ",," 'evil-normal-state)

(key-chord-define evil-normal-state-map "jk" 'evil-force-normal-state)
(key-chord-define evil-visual-state-map "jk" 'evil-change-to-previous-state)
(key-chord-define evil-insert-state-map "jk" 'evil-normal-state)
(key-chord-define evil-replace-state-map "jk" 'evil-normal-state)

(key-chord-define evil-normal-state-map "ee" 'evil-emacs-state)
(key-chord-define evil-insert-state-map "ee" 'evil-emacs-state)
(key-chord-define evil-emacs-state-map "ee" 'evil-normal-state)

This will allow one to jump back into normal mode using the jk keys hit in quick succession. Also hitting ee keys quickly will allow one to exit and leave default emacs mode in the buffer quickly.

Vim Basics

For people new to vim’s philosophy of using here is a quick recap.

Motion Commands

Character Level Motion :

+---+------+
| h | left |
| j | down |
| k | up   |
| l | right|
+---+------+

Are single character motions actions. All of which can be prefixed with numeric arguments. Thus 7j will move down 7 lines in normal mode.

Screen Level Motions :

+---------+------------------+
| Ctrl-F  | back screen      |
| Ctrl-f  | forward screen   |
| Ctrl-B  | page down        |
| Ctrl-b  | page up          |
| Ctrl-U  | half page up     |
| Ctrl-D  | half page down   |
| <n>G    | goto line n      |
| gg      | goto first line  |
| z.      | center to point  |
| zz      | center to point  |
| zt      | center point top |
| zb      | center bottom    |
+---------+------------------+

Motions can also be performed at the page level using Ctrl-F will move back one screen full. Where as Ctrl-U will move up one half page-full. Most motion commands can be given as targets to action commands like copy, paste delete. Thus yCtrl-F will copy a screen full of text back from the cursor position.

Additionally z. , zt, zb can be used to center and scroll screen top and bottom.

Line Level Motions:

+---+-----------------------------+
| 0 | beginning of line           |
| ^ | beginning of non black line |
| $ | end of line                 |
+---+-----------------------------+

To operate on current line some useful commands are shown above. For example y^ will copy the line to beginning current line. and y$ will copy it to the end of the current line. yy will copy the current line.

Line forward and backwards

| f{char} | forward search in line for {char}   |
| F{char} | backward search in line for {char}  |
| t{char} | forward search till {char} in line  |
| t{char} | forward search till {char} in line  |
| T{char} | backward search till {char}in line  |
| ;       | repeat last search forward          |
| ,       | repeat last search backward         | 

While working within lines it may be convenient to jump forward to a character. 2f, will jump forward to the second comma in a line. Previous searches can be repeated forward and backwards using comma and semicolons.

Sentence and Paragraph Motions

| ( | sentence back      |
| ) | sentence forward   |
| { | paragraph forward  |
| | | paragraph backward |

All sentence and paragraph motions can take in numeric arguments.

Search and Replace:

| /                  | search forward            |
| ?                  | search backwards          |
| n                  | continue search forward   |
| N                  | continue search backward  |
| :%/<pat>/<pat2>/gc | search and replace        |

/ and ? allows you to search forward and backwards also allowing you to perform actions as you search. So y?## will copy text till previous heading in markdown and P will allow you to paste after current cursor position. One can perform search and replace using :%/<pat1>/<pat2>/g which will replace every occurance of regex <pat1> with <pat2>.

Marks

| m{char}  | store current position in {char} register  |
| '{char}  | jump to position stored in {char} register |

Much like emacs marks allow one to jump back and forth between a history of positions. ma for example will set a mark in register named a. One can then yank from current line to mark using y'a. Notice the usage of single apostrophe works line wise whereas using back-quote will act line wise.

Macro recordings

| q{char} | record following commands in {char} register |
| @{char} | re-run recorded commands from {char} register|
| .       | re-run last macro                            |

Much like emacs macros qa can start recording a macro into register a. After you are done recording press q to indicate its end. @a can then be used to re-run recorded macro. . can be used to repeat last run commands.

Why bother?

The basic idea is to create as rich of a structured language for editing allowing us to instruct the editor at the level of intentionality using composable editing abstractions. To get a more visual guide to possibilities of evil mode checkout some of thoughtbot’s evil mode/vim videos or bailey ling’s guide. Another extremely useful resource is the vim reference manual which seems to be written in a very deliberate manner by Bram Moolenaar available here.

References

Notes on synchronization in the Linux Kernel

2015-07-23T00:00:00+00:00

Just a small collection of notes on synchronization mechanisms in the Kernel. The underlying reference for which is the Linux Kernel Development book by Robert Love.

Atomic Operations

Indivisible operations - executed without interruption
Keep state consistent across threads of execution
Some architectural support
Atomic Integer Operations
- Use special type atomic_t for int
  - Prevent compiler from optimizing value
  - hide architecture specific implementation differences
  - defined in linux/types.h

    typedef struct {
      volatile int counter;
    } atomic_t;

* Devlopers need to limit the usage of `atomic_t` to `24 bits` to not  break on sparc
* on sparc lock embeded in lower 8 bits (this may no longer be relevant)   * atomic operations in `asm/atomic.h`   * defining atomic

   /* define v */
   atomic_t v;
   
   /* define u and initialize it to zero */
   atomic_t u = ATOMIC_INIT(0);

Some atomic operations :

  /* v = 4 (atomically) */
  atomic_set(&v, 4);
  /* v = v + 2 = 6 (atomically) */
  atomic_add(2, &v);  
  /* v = v + 1 = 7 (atomically) */
  atomic_inc(&v);

  /* will print "7" convert atomic_t to int */
  printk(“%d\n”, atomic_read(&v));

Using atomic_add , atomic_inc ,atomic_dec lighter weight instead of complex locking
int atomic_dec_and_test(atomic_t *v)
- decrement v
  - if v is zero returns true
  - else return false
- List of Atomic Operations

    // Atomic Integer Operation Description
    //At declaration, initialize to i.
    ATOMIC_INIT(int i)
    // Atomically read the integer value of v.
    int atomic_read(atomic_t *v)
    
    // Atomically set v equal to i.
    void atomic_set(atomic_t *v, int i)
    
    // Atomically add i to v.
    void atomic_add(int i, atomic_t *v)
    
    // Atomically subtract i from v.
    void atomic_sub(int i, atomic_t *v)
    
    // Atomically add one to v.
    void atomic_inc(atomic_t *v)
    
    // Atomically subtract one from v.
    void atomic_dec(atomic_t *v)
    
    // Atomically subtract i from v and
    // return true if the result is zero;
    // otherwise false.
    int atomic_sub_and_test(int i, atomic_t *v)
    
    //Atomically add i to v and return
    //true if the result is negative;
    //otherwise false.
    int atomic_add_negative(int i, atomic_t *v)
    
    //Atomically add i to v and return
    //the result.
    int atomic_add_return(int i, atomic_t *v)
    
    //Atomically subtract i from v and
    //return the result.
    int atomic_sub_return(int i, atomic_t *v)
    
    //Atomically increment v by one and
    //return the result.
    int atomic_inc_return(int i, atomic_t *v)
    
    // Atomically decrement v by one and
    // return the result.
    int atomic_dec_return(int i, atomic_t *v)
    
    //Atomically decrement v by one and
    //return true if zero; false otherwise.
    int atomic_dec_and_test(atomic_t *v)

    //Atomically increment v by one and
    //return true if the result is zero;
    //false otherwise.
    int atomic_inc_and_test(atomic_t *v)

Implemented as inline functions & inline assembly
word sized reads always atomic
most architectures read/write of a single byte is atomic( no two simultaneous operations)
read is consistent(ordered either before or after the write)
Thus on most architectures we get

/**
* atomic_read - read atomic variable
* @v: pointer of type atomic_t
*
* Atomically reads the value of @v.
*/
static inline int atomic_read(const atomic_t *v)
{
   return v->counter;
}

Consitent atomicity does not guarantee consistent ordering (use memory barriers for enforced ordering)
64-bit architectures
- atomic_t size cant be different for different architectuers
  - atomic_t is consitently 32-bit accross architectuers
- atomic64_t is used to for 64-bit atomic operations on 64 bit machines
- nearly all atomic operations listed above implement a 64-bit version
- all corresponding functions shown below

typedef struct {
  volatile long counter;
} atomic64_t;
ATOMIC64_INIT(long i)
long atomic64_read(atomic64_t *v)
void atomic64_set(atomic64_t *v, int i)
void atomic64_add(int i, atomic64_t *v)
void atomic64_sub(int i, atomic64_t *v)
void atomic64_inc(atomic64_t *v)
void atomic64_dec(atomic64_t *v)
int atomic64_sub_and_test(int i, atomic64_t *v)
int atomic64_add_negative(int i, atomic64_t *v)
long atomic64_add_return(int i, atomic64_t *v)
long atomic64_sub_return(int i, atomic64_t *v)
long atomic64_inc_return(int i, atomic64_t *v)
long atomic64_dec_return(int i, atomic64_t *v)
int atomic64_dec_and_test(atomic64_t *v)
int atomic64_inc_and_test(atomic64_t *v)

Atomic Bit Operations
- architecture specific code
- asm/bitops.h for details
- operate on generic pointers
- on 32-bit machines
  - bit 31 most significant bit
  - bit 0 least significant bit
- Non-atomic versions of bit operations provided but their name prefixed with __
  - test_bit() atomic version , __test_bit() non-atomic version
  - when dont need locking use non-atomic versions for performance

/**
 * Atomically set the nr -th bit starting from addr.
 */
void set_bit(int nr, void *addr)

/**
 * Atomically clear the nr -th bit starting from addr.
 */
void clear_bit(int nr, void *addr)

/**
 * Atomically flip the value of the nr -th bit starting from addr.
 */
void change_bit(int nr, void *addr)

/**
 * Atomically set the nr -th bit starting from addr and return the previous value.
 */
int test_and_set_bit(int nr, void *addr)

/**
 * Atomically clear the nr -th bit starting from addr and return the
 * previous value.
 */
int test_and_clear_bit(int nr, void *addr)

/**
 * Atomically flip the nr -th bit starting from addr and return the
 * previous value.
 */
int test_and_change_bit(int nr, void *addr)

/**
 * Atomically return the value of the nr -
 * th bit starting from addr.
 */ 
int test_bit(int nr, void *addr)

See example usage setting and clearing bits atomically

unsigned long word = 0;
/* bit zero is now set (atomically) */
set_bit(0, &word);

/* bit one is now set (atomically) */
set_bit(1, &word);

/* will print “3” */
printk(“%ul\n”, word);
/* bit one is now unset (atomically) */
clear_bit(1, &word);
/*bit zero is flipped; now it is unset (atomically) */
change_bit(0, &word);

/* atomically sets bit zero and returns the previous value (zero) */
if (test_and_set_bit(0, &word)) {
/* never true ... */
}

/* the following is legal; you can mix atomic bit instructions with normal C */
word = 7;

Spin Locks

Allow for creation of critical regions in code
spinlock can be held by at most one thread at a time
contending threads busy loop waiting for the lock to unlock
if lock is uncontended acquire lock and proceed instantly
busy loop will "waste" processor time
thus must minmize length of time spinlock is held
advantage that doesnt result in context switches
can be used in contexts which do not support blocking (interrupt context) or (preemption disabled?)
Spin Lock Methods
- Defined in linux/spinlock.h

  
  DEFINE_SPINLOCK(my_lock);
  // acquire the spin lock
  spin_lock(&my_lock);
  ... place critical region here ..
  //release the spin lock
  spin_unlock(&my_lock);

On uniprocessor system spinlocks compile away
- “some people” say they have some side effects, and spinlocs are not noops
- need to verify
- act as markers to enable disable kernel preemption
Spinlocks are not recursive
- trying to acquire the same lock again will hang the thread of execution
Using spinlock in interrupt context - need to disable local interrupts on this processor
- if preemption not disabled may deadlock on the lock this processor already holds
- Thus need to use methods to which will disable interrupts
spin_lock_irqsave
- save the current state of interrupts
- disables interrupts locally
- obtain the spinlock
spin_unlock_irqrestore
- unlock the given lock
- returns interrupts to previous state
  - This is key , if the interrupts were disabled on entry then we need should keep them disabled
  - allows nesting of different spinlocks and interrupt disabling clode
Locks should be associated with data structures which they lock
make association of lock with data clearer by naming conventions

DEFINE_SPINLOCK(my_lock);

unsigned long flags;

spin_lock_irqsave(&my_lock, flags);

/* critical region ... */

spin_unlock_irqrestore(&my_lock, flags);

Kernel Configs to help Spin Lock Debugging
- CONFIG_DEBUG_SPINLOCK
  - using uninitialized spinlock detection
  - unlocking a lock which was never locked
- CONFIG_DEBUG_LOCK_ALLOC
  - debugging lock lifecycles?
Dynamic Allocation of Spinlock
- spin_lock_init() initialize dynamically created spinlock, ie with a pointer
- spin_trylock() - try to obtain lock, if fail return 0, if succed return non-zero

/* Acquires given lock*/
spin_lock()

/** Disables local interrupts and acquires given lock */
spin_lock_irq()

/** Saves current state of local interrupts, disables local inter-
/  rupts, and acquires given lock */
spin_lock_irqsave()

/** Releases given lock */
spin_unlock()

/** Releases given lock and enables local interrupts */
spin_unlock_irq()

/** Releases given lock and restores local interrupts to given pre-
vious state */
spin_unlock_irqrestore()

/** Dynamically initializes given spinlock_t */
spin_lock_init()

/** Tries to acquire given lock; if unavailable, returns nonzero */
spin_trylock()

/** Returns nonzero if the given lock is currently acquired, other-
wise it returns zero */
spin_is_locked()

Spin Locks and Bottom Halfs
- Bottom halfs preempt process contexts
- two tasklets of same type don’t run concurrently
- tasklet never preempts another tasklet on same processor
- interrupt context can preempt bottom halfs and process contexts
- Process context should disable bottom half preemption over contended locks
Reader-Writer Spin Locks
- Dealing with the asymmetrical nature of reading and writing.
- Writing demands mutal exclusion
  - no reading permitted
  - no writing permitted
- Reading requires - write exclusion
  - no writers permitted
  - multiple readers ok
- Reader-Writer spinlocks increase throughput by allowing multiple readers
- Provide separate variants of read and write locks
- one or more readers can hold read locks
- only one writer and no one readers or writers can hold write lock
- alternative terminology: (reader,writer) ,(shared/exlusive) , (concurrent/exclusive)

DEFINE_RWLOCK(mr_rwlock);

// Attain read lock
read_lock(&mr_rwlock);
/* critical section (read only) ... */
read_unlock(&mr_rwlock);

// Attain write lock
write_lock(&mr_rwlock);
/* critical section (read and write) ... */
write_unlock(&mr_rwlock);

read locks cant later be transformed to write locks without first releasing the read lock and then asking for a write lock

    read_lock(&mr_rwlock);
    
    // Dead locks will never get a write lock since the read lock will
    // never get released
    
    write_lock(&mr_rwlock);

reader locks are recursive, same thread can re-obtain same reader locks without deadlock
Thus if only readers in interrupt handlers no need to disable interrupts.
- use read_lock() instead of read_lock_irqsave()
- Must use write_lock_irqsave() in interrupts

// Acquires given lock for reading
 read_lock()

// Disables local interrupts and acquires given lock for reading
 read_lock_irq()

/**
 * Saves the current state of local interrupts, disables local in-
 * terrupts, and acquires the given lock for reading
 */
 read_lock_irqsave()

// Releases given lock for reading
 read_unlock()

// Releases given lock and enables local interrupts
 read_unlock_irq()

/**
 * Releases given lock and restores local interrupts to the
 * given previous state
 */
 read_unlock_ irqrestore()

//Acquires given lock for writing
 write_lock()

/**
 * Disables local interrupts and acquires the given lock for
 * writing
 */
 write_lock_irq()

/**
 * Saves current state of local interrupts, disables local inter-
 * rupts, and acquires the given lock for writing
 */
 write_lock_irqsave()

//Releases given lock
  write_unlock()

//Releases given lock and enables local interrupts
  write_unlock_irq()

/**
 * Releases given lock and restores local interrupts to given
 * previous state
 */
 write_unlock_irqrestore()

/**
 * Tries to acquire given lock for writing; if unavailable, returns
 * nonzero
 */
 write_trylock()

//Initializes given rwlock_t
rwlock_init()

Need to think of appropriate priority for writers. Else lots of readers may starve writer.
Spinlocks are on small timescales (nanosceconds?) For larger timescales blocking semaphores are advisable instead.

Semaphores

For larger timescales where putting process context to sleep is advisable (ms) scale
Sets processor free to execute other code.
Should only be obtained in process context cannot be used in interrupt context, interrupt context is not schedulable.
Cannot hold a spinlock while acquiring a semaphore. Since you may sleep and cause a deadlock on the spinlock (how is this enforced?)
Semaphores don’t disable kernel preemption, don’t hurt affect scheduling latency as much as spinlocks
Counting vs Binary Semaphores
- Unlike spinlocks allow arbitrary number of simultaneous lock holders
- usage count or count - number of permisable simultaneous lock holders
- Binary semaphore/mutex - enforces mutual exclusion - usage count is one
- Counting semaphore - usage count greater than one
- While count is greater than zero acquiring semaphore succeeds.
- asm/semaphore.h
- struct semaphore key structure

    struct semaphore name;
    sema_init(&name, count);
    // or alternatively for delcaring and inializing a staitc mutex
    static DECLARE_MUTEX(name);

* `init_MUTEX(sem)` - initialize a dynamically created semaphore
* To try acquire use `down_*` methods.
* To release use `up_*` methods
* `down_interruptible()`
  * one failure to acquire lock puts calling process in
    `TASK_INTERRUPTIBLE` and sleep
  * If `process` receives a `signal` then wake up and fail by
    returning `-EINTR`
    
* `down()`
  * place task in `TASK_UNINTERRUPTIBLE`, then sleep
  * process will not respond to signals

* prefer `down_interruptible()` over `down()`

* `down_trylock()`
  * acquire semaphore without blocking
  * if lock held return non-zero

   /* define and declare a semaphore, named mr_sem, with a count of one */
   static DECLARE_MUTEX(mr_sem);
   /* attempt to acquire the semaphore ... */
   if (down_interruptible(&mr_sem)) {
   /* signal received, semaphore not acquired ... */
   }
   /* critical region ... */
   /* release the given semaphore */
   up(&mr_sem);

Semaphore Methods

/**
 * Initializes the dynamically created semaphore
 * to the given count
 */
sema_init(struct semaphore *, int)

/**
 * Initializes the dynamically created semaphore
 * with a count of one
 */
init_MUTEX(struct semaphore *)

/**
 * Initializes the dynamically created semaphore
 * with a count of zero (so it is initially locked)
 */
init_MUTEX_LOCKED(struct semaphore *)

/**
 * Tries to acquire the given semaphore and
 * enter interruptible sleep if it is contended
 */
down_interruptible (struct semaphore *)

/**
 * Tries to acquire the given semaphore and
 * enter uninterruptible sleep if it is contended
 */
down(struct semaphore *)

/**
 * Tries to acquire the given semaphore and
 * immediately return nonzero if it is contended
 */
down_trylock(struct semaphore *)

/**
 * Releases the given semaphore and wakes a
 * waiting task, if any
 */
up(struct semaphore *)

Reader-Writer Semaphores

Similar to reader-writer spinlocks
linux/rwsem.h
Static Creation
- static DECLARE_RWSEM(name);
Dynamic Creation
- init_rwsem(struct rw_semaphore *sem)
Define mutual exclusion on writers with usage count 1
Allow simultaneous reads
All readers and writers use uninterruptible sleep

// statically declare  a read only semaphore
static DECLARE_RWSEM(mr_rwsem);

/* attempt to acquire the semaphore for reading ... */
down_read(&mr_rwsem);

/* critical region (read only) ... */
/* release the semaphore */
up_read(&mr_rwsem);

/* attempt to acquire the semaphore for writing ... */
down_write(&mr_rwsem);
/* critical region (read and write) ... */
/* release the semaphore */
up_write(&mr_sem);

For non-blocking lock-acquisition use down_read_trylock() and down_write_trylock()
- return non-zero if lock can be acquired
- return zero if lock cannot bet acquired
downgrade_write() converts a write lock to a read lock

Mutexes

struct mutex
used to simplify common use case of semaphores
Static Definition
- DEFINE_MUTEX(name)
Dynamic Definition
- mutex_init(&mutex)
Locking and Unlocking Mutex

  mutex_lock(&mutex);
  /* critical region ... */
  mutex_unlock(&mutex);

Does not require usage counts

/**
 * Locks the given mutex; sleeps if the lock is
 * unavailable
 */
mutex_lock(struct mutex *)

/**
 * Unlocks the given mutex
 */
mutex_unlock(struct mutex *)

/**
 * Tries to acquire the given mutex; returns one if suc-
 * cessful and the lock is acquired and zero otherwise
 */
mutex_trylock(struct mutex *)

/**
 * Returns one if the lock is locked and zero otherwise
 */
mutex_is_locked (struct mutex *)

Constrains Imposed on mutex usage
- usage count is always one
- single task holds mutex
- only original locker allowed to unlock mutex
- recursive locks and unlocks are not allowed.
- process cannot exit while holding a mutex
- a mutex cannot be acquired by an interrupt handler or bottom half
- Constraints checked via CONFIG_DEBUG_MUTEXES
Usage guidelines
- start with mutex use semaphore if constriants not statisfiable
When to use Spinlocks vs (Semaphore , Mutex) ?

| Low overhead                        | Spin lock              |
| Short lock hold time                | Spin lock              |
| Long lock hold time                 | Mutex is preferred.    |
| Need to lock from interrupt context | Spin lock is required. |
| Need to sleep while holding lock    | Mutex is required.     |

Completion Variables

Event signalling between tasks
One task waits for signal, other task signals on completion
On completion wake up all sleeping tasks
Ex. Completion Variable, wake up parent when child exits
- - See kernel/sched.c and kernel/fork.c
Static
- DECLARE_COMPLETION(mr_comp);
Dynamic
- init_completion()
To wait for completion call wait_for_completion
To signal completion call complete

/**
 * Initializes the given dynamically created
 * completion variable
 */

init_completion(struct completion *)

/**
 * Waits for the given completion variable
 * to be signaled
 */
wait_for_completion(struct completion *)
/**
 * Signals any waiting tasks to wake up
 */
complete(struct completion *)

BKL: The big kernel lock

Spinlock used to ease transition to SMP
global spinlock
Lock dropped on sleep
Is a recursive lock: same process multiple acquisition allowed
Forbidden new users
Transition away from it
lock_kernel acquires lock
unclock_kernel releases recursively
kernel_locked returns
- 0 - lock already held
- non-zero -lock not being held
linux/smp_lock.h
problem with single lock - difficult to determine which two parties actually need to syncrhonize with each other # Sequential Locks

Sequential Locking

Simple mechanism reading/writing shared data
maintains a sequence counter
when sequence counter is odd a write is taking palce
check sequence counter is even prior to and after read to be sure no write was underway

// define a seq lock
seqlock_t mr_seq_lock = DEFINE_SEQLOCK(mr_seq_lock);

// Create a write lock region
write_seqlock(&mr_seq_lock);
/* write lock is obtained... */
write_sequnlock(&mr_seq_lock);

// Read  path
unsigned long seq;
do {
  seq = read_seqbegin(&mr_seq_lock);
  /* read data here ... */
} while (read_seqretry(&mr_seq_lock, seq));

Write lock always succeeds as long as no writers
Favor writers over readers
Ideal cases
- Many of readers
- Few writers
- readers never starve writers
- simple data
Ex : seq_lock on jiffies

u64 get_jiffies_64(void)
{
    unsigned long seq;
    u64 ret;
    do {
       seq = read_seqbegin(&xtime_lock);
       ret = jiffies_64;
    } while (read_seqretry(&xtime_lock, seq)); // if unsuccesful read tries again
    return ret;
}

// update jiffies in timer interrupt
write_seqlock(&xtime_lock);
jiffies_64 += 1;
write_sequnlock(&xtime_lock);

Preemption Disabling

kernel is preemptive
spin locks disable premption for duration held
during modification of per-processor data disable preemption to prevent corrupting in flight data
preempt_disable and preempt_enable can be nested
methods maintain counts of the number of times preeption is enabled or disabled.
If count is 0 kernel is preemptive

/**
 * Disables kernel preemption by incrementing the preemp-
 * tion counter
 */
preempt_disable()

/**
 * Decrements the preemption counter and checks and serv-
 * ices any pending reschedules if the count is now zero
 */
preempt_enable()

/**
 * Enables kernel preemption but does not check for any
 * pending reschedules
 */
preempt_enable_no_resched()

/* Returns the preemption count*/
preempt_count()

Alternatively calling get_cpu() to obtain an index into perprocessor data will disable kernel preemption.
put_cpu() will reenable kernel preemption

Ordering and Barriers

Ensuring ordering of memory loads and stores
Compiler and Processors like to reorder loads and stores
Barriers prevent compiler and processor from reordering instructions
x86 does not do out of order stores
Other processors might do out of order stores
Reorderings can have dependencies :

a = 1
b = a

data dependency between b and a
static reordering : compiler
- Reflected in object code
dynamic reordering: processor
rmb()
- a read memory barrier
- loads before the rmb() will never be reordered to loads after the call
- read_barrier_depends()
  - read barrier for loads
  - only for loads where the subsequent load depends on previous load
  - much quicker on some architectures
  - on some architectures transforms to noop
wmb()
- write barrier
- stores before the wmb() will never be reordered with stores after the call
- x86 does not reorder stores
mb()
- read/write barrier
- (loads and stores) before the wmb() will never be reordered with (loads and stores) after the call
smp_rmb , smp_wmb and smp_read_barrier_depends
- turn memory barrier to compiler barrier on smp
- compiler barrires are nearly free - only preventing static rearrangement

Summary

Notes on memory management in the Linux Kernel

2015-07-23T00:00:00+00:00

Just a small collection of notes on memory management in the Kernel. The underlying reference for which is the Linux Kernel Development book by Robert Love.

Why is memory allocation in the kernel hard ?

Not easy to deal with memory allocation errors.
Kernel often cannot sleep.
Require special primitives different from userspace.

Pages

Physical Pages act as basic unit of memory management
Different from processor’s smallest addressable unit (byte or word)
Hardware provides assistance via the MMU (memory management unit)
virtual memory pages are smallest unit.
Page sizes architecture specific 32-bit 4Kb page size 64-bit Kb page size
Kernel keeps track of pages in struct page structure
pages kept track are the actual physical pages these are not virtual pages.
struct page defined in <linux/mm_types.h>

struct page {
   unsigned long flags;
   atomic_t _count;
   atomic_t _mapcount;
   unsigned long private;
   struct address_space *mapping;
   pgoff_t index;
   struct list_head lru;
   void *virtual;
};

flags stores the status of the page 32-different flags available see <linux/page-flags.h>
- See the page flags enum for list of states

enum pageflags {
	PG_locked,		/* Page is locked. Don't touch. */
	PG_error,
	PG_referenced,
	PG_uptodate,
	PG_dirty,
	PG_lru,
	PG_active,
	PG_slab,
	PG_owner_priv_1,	/* Owner use. If pagecache, fs may use*/
	PG_arch_1,
	PG_reserved,
	PG_private,		/* If pagecache, has fs-private data */
	PG_private_2,		/* If pagecache, has fs aux data */
	PG_writeback,		/* Page is under writeback */
#ifdef CONFIG_PAGEFLAGS_EXTENDED
	PG_head,		/* A head page */
	PG_tail,		/* A tail page */
#else
	PG_compound,		/* A compound page */
#endif
	PG_swapcache,		/* Swap page: swp_entry_t in private */
	PG_mappedtodisk,	/* Has blocks allocated on-disk */
	PG_reclaim,		/* To be reclaimed asap */
	PG_swapbacked,		/* Page is backed by RAM/swap */
	PG_unevictable,		/* Page is "unevictable"  */
#ifdef CONFIG_MMU
	PG_mlocked,		/* Page is vma mlocked */
#endif
#ifdef CONFIG_ARCH_USES_PG_UNCACHED
	PG_uncached,		/* Page has been mapped as uncached */
#endif
#ifdef CONFIG_MEMORY_FAILURE
	PG_hwpoison,		/* hardware poisoned page. Don't touch */
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	PG_compound_lock,
#endif
	__NR_PAGEFLAGS,

	/* Filesystems */
	PG_checked = PG_owner_priv_1,

	/* Two page bits are conscripted by FS-Cache to maintain local caching
	 * state.  These bits are set on pages belonging to the netfs's inodes
	 * when those inodes are being locally cached.
	 */
	PG_fscache = PG_private_2,	/* page backed by cache */

	/* XEN */
	PG_pinned = PG_owner_priv_1,
	PG_savepinned = PG_dirty,

	/* SLOB */
	PG_slob_free = PG_private,
};

_count field represents the usage count for a page
- negative count field indicates page is free for allocation
- access via the page_count() macro, provides locking, atomic read
virtual virtual address of the page.
- the virtual address of page
- for pages in HIGH_MEM which need to be mapped as needed NULL
This structure keeps track of the data about physical pages has less to do with actual pages.
struct page consumes about 40 bytes
- Assuming 4Gb system with 8Kb page size
- Means 524,288 pages
- That is 20 Mb of struct pages in memory

Zones

Allows for non uniform treatment of pages
Zones group pages of similar properties
Example of hardware limitations
- being able to perform DMA (direct memory access) only within certain memory addresses
- large physically addressability vs small virtual addressability
  - allow for pages not permanently mapped into kernel address space
Four primary zones in Linux
- ZONE_DMA - Pages that can do DMA
- ZONE_DMA32 - Pages that can do DMA and accessible on only 32 bit devices
- ZONE_NORMAL - Regularly mapped pages
- ZONE_HIGHMEM - Pages not permanently mapped into kernel address space
Zones defined in linux/mmzone.h
Actual usage of zones are architecture dependent
- ISA on x86-32 limited to DMA on first 16 Mb of memory
- on x86-32 ZONE_DMA 0-16mb
- on x86-32 ZONE_NORMAL 16 mb-896 Mb
- on 32-bit ZONE_HIGHMEM all memory above 896 Mb
- x86-64 No High Mem and all memory is mappable all memory in ZONE_NORMAL
Thus ZONE provides some logical grouping of pages.
Some Key fields in struct zone
lock - spinlock to prevent concurrent modification
watermark - minimum and low values for this zone.
name - null terminated string
Initialized during boot in mm/page_alloc.c

struct zone {
	/* Fields commonly accessed by the page allocator */

	/* zone watermarks, access with *_wmark_pages(zone) macros */
	unsigned long watermark[NR_WMARK];

	/*
	 * When free pages are below this point, additional steps are taken
	 * when reading the number of free pages to avoid per-cpu counter
	 * drift allowing watermarks to be breached
	 */
	unsigned long percpu_drift_mark;

	/*
	 * We don't know if the memory that we're going to allocate will be freeable
	 * or/and it will be released eventually, so to avoid totally wasting several
	 * GB of ram we must reserve some of the lower zone memory (otherwise we risk
	 * to run OOM on the lower zones despite there's tons of freeable ram
	 * on the higher zones). This array is recalculated at runtime if the
	 * sysctl_lowmem_reserve_ratio sysctl changes.
	 */
	unsigned long		lowmem_reserve[MAX_NR_ZONES];

	/*
	 * This is a per-zone reserve of pages that should not be
	 * considered dirtyable memory.
	 */
	unsigned long		dirty_balance_reserve;

#ifdef CONFIG_NUMA
	int node;
	/*
	 * zone reclaim becomes active if more unmapped pages exist.
	 */
	unsigned long		min_unmapped_pages;
	unsigned long		min_slab_pages;
#endif
	struct per_cpu_pageset __percpu *pageset;
	/*
	 * free areas of different sizes
	 */
	spinlock_t		lock;
#if defined CONFIG_COMPACTION || defined CONFIG_CMA
	/* Set to true when the PG_migrate_skip bits should be cleared */
	bool			compact_blockskip_flush;

	/* pfn where compaction free scanner should start */
	unsigned long		compact_cached_free_pfn;
	/* pfn where async and sync compaction migration scanner should start */
	unsigned long		compact_cached_migrate_pfn[2];
#endif
#ifdef CONFIG_MEMORY_HOTPLUG
	/* see spanned/present_pages for more description */
	seqlock_t		span_seqlock;
#endif
	struct free_area	free_area[MAX_ORDER];

#ifndef CONFIG_SPARSEMEM
	/*
	 * Flags for a pageblock_nr_pages block. See pageblock-flags.h.
	 * In SPARSEMEM, this map is stored in struct mem_section
	 */
	unsigned long		*pageblock_flags;
#endif /* CONFIG_SPARSEMEM */

#ifdef CONFIG_COMPACTION
	/*
	 * On compaction failure, 1<<compact_defer_shift compactions
	 * are skipped before trying again. The number attempted since
	 * last failure is tracked with compact_considered.
	 */
	unsigned int		compact_considered;
	unsigned int		compact_defer_shift;
	int			compact_order_failed;
#endif

	ZONE_PADDING(_pad1_)

	/* Fields commonly accessed by the page reclaim scanner */
	spinlock_t		lru_lock;
	struct lruvec		lruvec;

	/* Evictions & activations on the inactive file list */
	atomic_long_t		inactive_age;

	unsigned long		pages_scanned;	   /* since last reclaim */
	unsigned long		flags;		   /* zone flags, see below */

	/* Zone statistics */
	atomic_long_t		vm_stat[NR_VM_ZONE_STAT_ITEMS];

	/*
	 * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
	 * this zone's LRU.  Maintained by the pageout code.
	 */
	unsigned int inactive_ratio;


	ZONE_PADDING(_pad2_)
	/* Rarely used or read-mostly fields */

	/*
	 * wait_table		-- the array holding the hash table
	 * wait_table_hash_nr_entries	-- the size of the hash table array
	 * wait_table_bits	-- wait_table_size == (1 << wait_table_bits)
	 *
	 * The purpose of all these is to keep track of the people
	 * waiting for a page to become available and make them
	 * runnable again when possible. The trouble is that this
	 * consumes a lot of space, especially when so few things
	 * wait on pages at a given time. So instead of using
	 * per-page waitqueues, we use a waitqueue hash table.
	 *
	 * The bucket discipline is to sleep on the same queue when
	 * colliding and wake all in that wait queue when removing.
	 * When something wakes, it must check to be sure its page is
	 * truly available, a la thundering herd. The cost of a
	 * collision is great, but given the expected load of the
	 * table, they should be so rare as to be outweighed by the
	 * benefits from the saved space.
	 *
	 * __wait_on_page_locked() and unlock_page() in mm/filemap.c, are the
	 * primary users of these fields, and in mm/page_alloc.c
	 * free_area_init_core() performs the initialization of them.
	 */
	wait_queue_head_t	* wait_table;
	unsigned long		wait_table_hash_nr_entries;
	unsigned long		wait_table_bits;

	/*
	 * Discontig memory support fields.
	 */
	struct pglist_data	*zone_pgdat;
	/* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */
	unsigned long		zone_start_pfn;

	/*
	 * spanned_pages is the total pages spanned by the zone, including
	 * holes, which is calculated as:
	 * 	spanned_pages = zone_end_pfn - zone_start_pfn;
	 *
	 * present_pages is physical pages existing within the zone, which
	 * is calculated as:
	 *	present_pages = spanned_pages - absent_pages(pages in holes);
	 *
	 * managed_pages is present pages managed by the buddy system, which
	 * is calculated as (reserved_pages includes pages allocated by the
	 * bootmem allocator):
	 *	managed_pages = present_pages - reserved_pages;
	 *
	 * So present_pages may be used by memory hotplug or memory power
	 * management logic to figure out unmanaged pages by checking
	 * (present_pages - managed_pages). And managed_pages should be used
	 * by page allocator and vm scanner to calculate all kinds of watermarks
	 * and thresholds.
	 *
	 * Locking rules:
	 *
	 * zone_start_pfn and spanned_pages are protected by span_seqlock.
	 * It is a seqlock because it has to be read outside of zone->lock,
	 * and it is done in the main allocator path.  But, it is written
	 * quite infrequently.
	 *
	 * The span_seq lock is declared along with zone->lock because it is
	 * frequently read in proximity to zone->lock.  It's good to
	 * give them a chance of being in the same cacheline.
	 *
	 * Write access to present_pages at runtime should be protected by
	 * mem_hotplug_begin/end(). Any reader who can't tolerant drift of
	 * present_pages should get_online_mems() to get a stable value.
	 *
	 * Read access to managed_pages should be safe because it's unsigned
	 * long. Write access to zone->managed_pages and totalram_pages are
	 * protected by managed_page_count_lock at runtime. Idealy only
	 * adjust_managed_page_count() should be used instead of directly
	 * touching zone->managed_pages and totalram_pages.
	 */
	unsigned long		spanned_pages;
	unsigned long		present_pages;
	unsigned long		managed_pages;

	/*
	 * Number of MIGRATE_RESEVE page block. To maintain for just
	 * optimization. Protected by zone->lock.
	 */
	int			nr_migrate_reserve_block;

	/*
	 * rarely used fields:
	 */
	const char		*name;
} ____cacheline_internodealigned_in_smp;

Getting Pages

methods for requesting memory at page size granularity in linux/gfp.h
struct page * alloc_pages(gfp_t gfp_mask, unsigned int order)
- core function to fetch pages
- allocates 2^order or (1 << order) pages
- returns poniter to first page
- Convert to logical address of where page resides using void * page_address(struct page *page)
unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)
- Returns logical address where the page resides
- used when struct page not required
- returned pages are contiguous and follow the first
unsigned long get_zeroed_page(unsigned int gfp_mask)
- useful for pages handed to userspace
- prevent leaking sensitive data
Freeing pages.
- void __free_pages(struct page *page, unsigned int order)
- void free_pages(unsigned long addr, unsigned int order)
- void free_page(unsigned long addr)
- double free is a serious problem in the kernel
All allocations can fail and we can get NULL logical address

unsigned long page;

// allocalte 2^3 or eight pages
// page is the logical address
page = __get_free_pages(GFP_KERNEL, 3);

if (!page) {
/* insufficient memory: you must handle this error! */
return –ENOMEM;
}

/* ‘page’ is now the address of the first of eight contiguous pages ... */

// free the 8 pages.
free_pages(page, 3);

`kmalloc()`

Useful or byte sized allocation
declared in <linux/slab.h>
preferred choice if allocations are not multiples of page size
void * kmalloc(size_t size, gfp_t flags)
returned memory is at least size bytes in length

struct dog *p;

p = kmalloc(sizeof(struct dog), GFP_KERNEL);

if (!p)
   /* handle error ... */

gfp_mask Flags
- flags defined in linux/types.h
- as unsigned int
- Flag Types
  - action modifiers
    - how memory is allocated
    - eg. during interrupt handler allocations should fail but never sleep
  - zone modifiers
    - specify where the memory is allocated
  - types
    - Specify combination of action and zone modifiers
    - GFP_KERNEL
      - for code in process context
Action Modifiers

+------------------------------------------------------------------------+  
| Flag             | Description                                         |
+------------------------------------------------------------------------+
|__GFP_WAIT        | The allocator can sleep.                            |
|__GFP_HIGH        | The allocator can access emergency pools.           |
|__GFP_IO          | The allocator can start disk I/O.                   |
|__GFP_FS          | The allocator can start filesystem I/O.             |
|__GFP_COLD        | The allocator should use cache cold pages.          |
|__GFP_NOWARN      | The allocator does not print failure warnings.      |
|__GFP_REPEAT      | The allocator repeats the allocation if it          |
|                  | fails, but the allocation can potentially fail.     |
|__GFP_NOFAIL      | The allocator indefinitely repeats the allocation.  |
|                  | The allocation cannot fail.                         |
|                  |                                                     |
|__GFP_NORETRY     | The allocator never retries if the allocation       |
|                  | fails.                                              |
|__GFP_NOMEMALLOC  | The allocator does not fall back on reserves.       |
|__GFP_HARDWALL    | The allocator enforces “hardwall” cpuset boundaries.|
|__GFP_RECLAIMABLE | The allocator marks the pages reclaimable.          |
|__GFP_COMP        | The allocator adds compound page metadata           | 
+------------------------------------------------------------------------+

Sample usage

// Sleepable, Disk IO able, file system operationable
ptr = kmalloc(size, __GFP_WAIT | __GFP_IO | __GFP_FS);

kmalloc calls ultimately use the alloc_pages and above grants great flexibility in page allocation
Zone Modifiers

+------------------------------------------------------------+
|Flag          | Description                                 |
+------------------------------------------------------------+  
|__GFP_DMA     | Allocates only from ZONE_DMA                |
|__GFP_DMA32   | Allocates only from ZONE_DMA32              |
|__GFP_HIGHMEM | Allocates from ZONE_HIGHMEM or ZONE_NORMAL  |
+------------------------------------------------------------+

Use GFP_DMA if you must have dma-able memory
__GFP_HIGHMEM can use if needed
- only alloc_pages can return High memory
- since logical address cannot be returned for memory not mapped into kernels virtual address space
Type Flags
- specify combinations of action and zone modifiers
- simpler and less error prone

+--------------------------------------------------------------------+
|Flag          | Description                                         |
+--------------------------------------------------------------------+  
|GFP_ATOMIC        | The allocation is high priority and must
|                  | not sleep. This is the flag to use in interrupt
|                  | handlers, in bottom halves, while holding a spin-
|                  | lock, and in other situations where you cannot sleep.
|                  |
|GFP_NOWAIT        | Like GFP_ATOMIC , except that the call will not
|                  | fallback on emergency memory pools. This increases the
|                  | liklihood of the memory allocation failing.
|                  |
|GFP_NOIO          | This allocation can block, but must not initiate
|                  | disk I/O. This is the flag to use in block I/O code
|                  | when you cannot cause more disk I/O, which might lead
|                  | to some unpleasant recursion.
|GFP_NOFS          | This allocation can block and can initiate disk I/O,
|                  | if it must, but it will not initiate a filesystem
|                  | operation. This is the flag to use in filesystem code
|                  | when you cannot start another filesystem operation.
|GFP_KERNEL        | This is a normal allocation and might block. This is
|                  | the flag to use in process context code when it is safe
|                  | to sleep. The kernel will do whatever it has to do to
|                  | obtain the memory requested by the caller. This flag
|                  | should be your default choice.
|GFP_USER          | This is a normal allocation and might block. This flag is used to
|                  | allocate memory for user-space processes.
|GFP_HIGHUSER      | This is an allocation from ZONE_HIGHMEM and might block. This
|                  | flag is used to allocate memory for user-space processes.
|GFP_DMA           | This is an allocation from ZONE_DMA . Device drivers that need
|                  |DMA-able memory use this flag, usually in combination with one of
|                  |the preceding flags.
+-----------------------------------------------------------------------------------+

GFP_NOFS (__GFP_WAIT | __GFP_IO)
GFP_KERNEL (__GFP_WAIT | __GFP_IO | __GFP_FS)
GFP_USER (__GFP_WAIT | __GFP_IO | __GFP_FS)
GFP_HIGHUSER (__GFP_WAIT | __GFP_IO | __GFP_FS | __GFP_HIGHMEM)
GFP_DMA __GFP_DMA
GFP_KERNEL most frequently used flag
- can block so cannot be used interrupt context
- has high probability of succeeding
- can put caller to sleep , swap inactive pages to disk, flush dirty pages,
GFP_ATOMIC
- most restrictive
- memory allocations which cannot sleep
- if no continguous chunk is available then will not call free instead just fail
- less chance of succeeding
kfree()
- frees block of memory previously allocated
- void kfree(const void *ptr)
- double free is a serious bug.
- kfree(NULL) is checked for and works

char *buf;

buf = kmalloc(BUF_SIZE, GFP_ATOMIC);
if (!buf)
   /* error allocating memory ! */
   
kfree(buf);

`vmalloc()`

Allocates memory which is only virtually continguous unlike kmalloc whose pages are physically contiguous
Acheived by fixing up page tables map memory into contiguos chunks in logical address space
Cannot be use when hardware requires contiguos pages, due to not being behind MMU
All memory would appear to kernel as logically contiguous
most kernel code will still use kmalloc for performance reasons
Uses greater number entries in TLB
vmalloc used rarely when need to allocate large regions which may fail with kmalloc
delcared in linux/vmalloc.h defined in linux/vmalloc.c
void * vmalloc(unsigned long size)
returned pointer is at least size bytes
cannot be used in interrupt context

char *buf;
buf = vmalloc(16 * PAGE_SIZE); /* get 16 pages */
if (!buf)
/* error! failed to allocate memory */
/*
* buf now points to at least a 16*PAGE_SIZE bytes
* of virtually contiguous block of memory
*/
// Free with
vfree(buf);

Slab Layer

Generalization of the idea of free lists of certain granularity of data
Tries to avoid cost of allocation/deallocation
consolidates the idea of free lists in the kernel
Allows kernel global management of free lists
Why do we use the slab allocator ?
- Frequently used data structures are allocated and freed often
- arranging free lists contiguously means less memory fragmentation from frequent alloc/free
- freed objects immediately available for subsequent use
- allocator aware of obj size, page size ,total cache
- using some processor specific memory in slabs , means fewer locks
- NUMA aware allocators be location sensitive in alloc/free
Design of Slab layer
- objects devided into caches
- caches devided into slabs
- slabs composed of one or more contiguous pages (typically single page)
- slab states : full,partial,empty
- requests satisfied from partial slabs
- if no partial slabs then request satisfied from empty slab
- eg. cache to store struct inode from inode_cachep , cache for task_struct
- cache represented kmem_cache with three lists slabs_full,slabs_partial,slabs_empty
- above lists stored in kmem_list3
- slab described by structure in mm/slab.c
Slab Allocator Interface
- int kmem_cache_destroy(struct kmem_cache *cachep)
  - invoked on module shutdown to free cache
  - may sleep dont call from interrupt context
  - caller must be sure that cache is empty , no active slabs
  - caller must ensure synchronization
- struct kmem_cache * kmem_cache_create(const char *name, size_t size, size_t align, unsigned long flags, void (*ctor)(void *))
  - Creates a cache
  - returns a pointer to the cache created
  - /proc/slabinfo to see caches , name shows up there
  - size - size of each cache element
  - align - offset of first element
  - ctor - slab constructor
    - not used but called when new pages added to cache
  - flags
    - SLAB_HWCACHE_ALIGN :
      - This flag instructs the slab layer to align each object within a slab to a cache line
    - SLAB_POISON:
      - fill slab with known value a5a5a5a5 used to catch uninitialized memory
    - SLAB_RED_ZONE
      - use red zones around to detect buffer overruns
    - SLAB_PANIC
      - panic if allocation fails
      - indicate that allocations must not fail
    - SLAB_CACHE_DMA
      - each slab must be in dma’able memory
- void * kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
  - return poniter to object from cache
  - allocates new pages if no free slabs
Example Usage

// cache for task_structs
struct kmem_cache *task_struct_cachep;

// create the cache
task_struct_cachep = kmem_cache_create(“task_struct”,sizeof(struct task_struct),
                                        ARCH_MIN_TASKALIGN,SLAB_PANIC | SLAB_NOTRACK,NULL);

// allocate a task struct from the cache as needed

struct task_struct *tsk;

tsk = kmem_cache_alloc(task_struct_cachep, GFP_KERNEL);

if (!tsk)
   return NULL;



// free a task struct
kmem_cache_free(task_struct_cachep, tsk);

Statically Allocating on the Stack

Kernel stacks are small and fixed
Kernel stack generally 2 pages per process
8Kb on 32-bit or 16 Kb on 64-bit
Sometimes beneficial to deal with single page stacks
- deal with memory fragmentation
- allocation of new process becomes harder , not able to find contiguous pages.
interrupts use kernel stacks of process they interrupted
use interrupt stacks instead - one page per processor
this depends on enablement of single page interrupt stacks.
kernel stacks will overflow into process thread info structure
Thus keep stack allocations minimum and use dynamic allocation.

High Memory Pages

Pages from alloc_pages() and __GFP_HIGHMEM.
Since no permanent, high mem pages might not have logical address
x86-32 all memory beyond 896 Mb is high memory(not permanently mapped to kernel address space)
x86-32 can theoretically map about 2^32(4 Gb) and (63 Gb) with PAE
x86-32 high memory pages get mapped in and out between 3 Gb and 4 Gb
Permanent Mappings
- <linux/highmem.h>
- void * kmap(struct page* page)
  - works on both high and low memory
  - returns the virtual memory if page in low memory
  - if high memory page creates a mapping and returns the page.
  - function may sleep - only works in process context
  - mappings are permanent user responsible for unmapping
  - good to unmap when usage finishes
- void kunmap(struct page *page)
  - unmap the created mapping of high memory
Temporary/Atomic Mappings
- To create mappings in interrupt context and other non-blocking contexts
- void * kmap_atomic(struct page * page , enum km_type type)
  - does not block
  - can be used in non-schedulable contexts
  - Defined in <asm-generic/kmap_types.h>

    enum km_type {
       KM_BOUNCE_READ,
       KM_SKB_SUNRPC_DATA,
       KM_SKB_DATA_SOFTIRQ,
       KM_USER0,
       KM_USER1,
       KM_BIO_SRC_IRQ,
       KM_BIO_DST_IRQ,
       KM_PTE0,
       KM_PTE1,
       KM_PTE2,
       KM_IRQ0,
       KM_IRQ1,
       KM_SOFTIRQ0,
       KM_SOFTIRQ1,
       KM_SYNC_ICACHE,
       KM_SYNC_DCACHE,
       KM_UML_USERCOPY,
       KM_IRQ_PTE,
       KM_NMI,
       KM_NMI_PTE,
       KM_TYPE_NR
    };

* Disables kernel preemption - mappings are processor unique
* `void kunmap_atomic(void *kvaddr, enum km_type type)`
  * Ability to undo mapping at `kvaddr`
  * On most architectures does nothing but enable kernel preemption
  * a Temporary mapping is only valid until next kernel mapping

Per CPU Allocation

On SMP use data unique to a CPU
Per-CPU data stored in an array
Items in array correspond to processor specific data
get_cpu() - get the current cpu and disable kernel preemption
put_cpu() - re-enable kernel preemption

unsigned long my_percpu[NR_CPUS];

int cpu;
/* get current processor and disable kernel preemption */
cpu = get_cpu();

// use the cpu specific data
my_percpu[cpu]++;
printk(“my_percpu on cpu=%d is %lu\n”, cpu, my_percpu[cpu]);

/* enable kernel preemption */
put_cpu();

locking is not required for since data unique to cpu
Problems with Kernel Preemption
- cpu variable will become invalid if kernel is preempted and rescheduled on another processor
- another thread may access now be able to access dirty data structure on same processor
Using get_cpu ensures that kernel preemption on the procssor is disabled

The `percpu` Interface

linux/percpu.h
Definitions in mm/slab.c and asm/percpu.h
DEFINE_PER_CPU(type, name);
- An instance of a percpu variable with type and name
get_cpu_var(name) and put_cpu_var(name)
- disable kernel preemption and get cpu specific value
per_cpu(name, cpu)++;
- fetch another processor per cpu variable
- dangerous method since doesnt disable preemption and doesnt provide locking
Per-CPU data at Runtime
- In <linux/percpu.h>
- void *alloc_percpu(type); /* a macro */
  - Allocate one instace per processor
  - macro around __alloc_percpu
  - alligns at byte boundary
  - __alignof__ - gcc feature to get recomended alignment
  - returned pointer indirectly references dynamically created data
  - get_cpu_var(ptr) fetches cpu specific pointer to dynamically created data.
- void *__alloc_percpu(size_t size, size_t align);
  - number of bytes to allocate and alignment
- void free_percpu(const void *);
  - frees data on all the processors

void *percpu_ptr;

unsigned long *foo;

percpu_ptr = alloc_percpu(unsigned long);

if (!ptr)

/* error allocating memory .. */
foo = get_cpu_var(percpu_ptr);

/* manipulate foo .. */
put_cpu_var(percpu_ptr);

Reasons for using Per-CPU Data

reduction in locking
reduces cache invalidation due to data being modified on other processors
Cannot sleep in the middle of accessing per-CPU data

Summary/Guidelines for picking an allocation methods

Mostly pick between GFP_ATOMIC and GFP_KERNEL for allocations
For free high memory use alloc_pages() since it returns struct page
Use kmap() to map high memory pages
Use vmalloc() when doing large allocations where contiguous memory is not a requirement
If doing lots of creations/destructions of same object type use slab cache, prevent fragmentation,get faster allocations

Programming Language Reference - EmacsLisp (draft)

2015-06-23T00:00:00+00:00

In this post we explore the extension language for the powerful yet very arcane ‘text editor’ emacs. Trying to keep track of various features of emacs lisp along with useful extensions. The main reference for his cheat sheet is the elisp manual which comes with you installation of emacs. See (info "elisp")

The Emacs Lisp

Lisp Data Types

Lisp object data maintained by lisp
objects belongs to at least one type
Fundamental types example
- integer
- float
- cons
- symbol
- string
- vector
- hash-table
- subr
- byte-code function
- buffer (editor specific types)
objects know their types variables point to objects

Printed Representation and Read Syntax

Format from Lisp printer prin1
printed representation also read syntax
For certain objects dont have read syntax
Start with #<
Example

(current-buffer)
  => #<buffer objects.texi>

invalid-read-syntax if starts with #<
Comments
start with ;
continues till end of line
discarded by Lisp reader
#@COUNT skips next COUNT characters

Programmer Types

Integer Types

-536870912 to 536870911 or 2^29 to 2^29 -1
read syntax
- optional sign
- optinoal period at end
For too large read as floating point number

Floating Point Type

Character Type

Symbol Type

: denotes key workd symbol

Sequence Type

Cons Cell Type

Array Type

String Type

Vector Type

Char-Table Type

Bool-Vector Type

Hash Table Type

Function Type

Macro Type

Primitive Function Type

Byte-Code Type

Autoload Type

Programmer Types

####### Buffer Type

The basic object of editing.

Marker Type

A position in a buffer.

Window Type

Buffers are displayed in windows.

Frame Type

Windows subdivide frames.

Terminal Type

A terminal device displays frames.

Window Configuration Type

Recording the way a frame is subdivided.

Frame Configuration Type

Recording the status of all frames.

Process Type

A subprocess of Emacs running on the underlying OS.

Stream Type

Receive or send characters.

Keymap Type

What function a keystroke invokes.

Overlay Type

How an overlay is represented.

Font Type

Fonts for displaying text.

Numbers

Strings and Characters

Lists

Cons Cells

How lists are made out of cons cells.
List-related Predicates
Is this object a list? Comparing two lists.
List Elements
Extracting the pieces of a list.
Building Lists
Creating list structure.
List Variables
Modifying lists stored in variables.
Modifying Lists
Storing new pieces into an existing list.
Sets And Lists
A list can represent a finite mathematical set.
Association Lists
A list can represent a finite relation or mapping.

Sequences Arrays Vectors

Sequence Functions

Functions that accept any kind of sequence.
Arrays
Characteristics of arrays in Emacs Lisp.
Array Functions
Functions specifically for arrays.
Vectors
Special characteristics of Emacs Lisp vectors.
Vector Functions
Functions specifically for vectors.
Char-Tables
How to work with char-tables.
Bool-Vectors
How to work with bool-vectors.
Rings
Managing a fixed-size ring of objects.

Hash Tables

Creating Hash

Functions to create hash tables.
Hash Access
Reading and writing the hash table contents.
Defining Hash
Defining new comparison methods.
Other Hash
Miscellaneous.

Symbols

Evaluation

Control Structures

Sequencing

Evaluation in textual order.
Special form progn

(progn A B C ...)

execute A , B, C in order
body of Function defines implicity progn
implicit in may control structures

    (progn (print "The first form")
           (print "The second form")
           (print "The third form"))
         -| "The first form"
         -| "The second form"
         -| "The third form"
    => "The third form"

progn : evaluates all forms returns value of final form
prog1 : This special form evaluates FORM1 and all of the FORMS, in textual order, returning the result of FORM1.
prog2 : This special form evaluates FORM1, FORM2, and all of the following FORMS, in textual order, returning the result of FORM2.

Conditionals

if, cond, when, unless
if condition then-form else-forms...
- chooses between then-form and else-forms based on conditionals
- else has implicit progn
- unexecuted branches are not executed

  (if nil
     (print 'true)
   'very-false)

when condition then-forms...
- variant of if without else-forms
- implicit progn for then-forms

(when CONDITION A B C)
;; equivalent to
(if CONDITION (progn A B C) nil)

unless condition forms...
- This is a variant of `if’ where there is no THEN-FORM:

(unless CONDITION A B C)
;; equivalent to
(if CONDITION nil
   A B C)

cond clause...
- cond chooses among an arbitrary number of alternatives.
- Each CLAUSE in the cond must be a list
- The CAR of this list is the CONDITION
- If the value of CONDITION is non-nil,
  - the clause “succeeds”; then cond evaluates its BODY-FORMS,
  - the value of the last of BODY-FORMS becomes the value of the cond

  (cond ((numberp x) x)
        ((stringp x) x)
        ((bufferp x)
         (setq temporary-hack x) ; multiple body-forms
         (buffer-name x))        ; in one clause
        ((symbolp x) (symbol-value x)))

Use t for default clause which always passes.

  (setq a 5)
    (cond ((eq a 'hack) 'foo)
          (t "default"))
  => "default"

Combining Conditions

and, or, not. ####### not condition
return t if condition is nil and nil otherwise ####### and conditions...
ensures each condition is t
short-circuits if any condition is nil

(and (print 1) (print 2) nil (print 3))
     -| 1
     -| 2
=> nil

;; another example
(if (and (consp foo) (eq (car foo) 'x))
    (message "foo is a list starting with x"))

####### or conditions...

Requires at least one of the conditions to be true
short cirquits on first non-nil condition
value returned is first non-nil
else returns nil

;; test x is nil or integer 0
(or (eq x nil) (eq x 0))

Iteration

####### while condition forms...

while non-nil condition evaluation, evaluate forms in textual order
exit on nil condition or throw

 (setq num 0)
      => 0
 (while (< num 4)
   (princ (format "Iteration %d." num))
   (setq num (1+ num)))
      -| Iteration 0.
      -| Iteration 1.
      -| Iteration 2.
      -| Iteration 3.
      => nil

support for repeat until loop available

####### dolist (var list [result]) body...

execute body for each element of list
var hods the current element
returns value of result or nil if result ommited

(defun reverse (list)
  (let (value)
    (dolist (elt list value)
      (setq value (cons elt value)))))

####### dotimes (var count [result]) body...

evaluate body from [0,count)
return result

;; lol
(dotimes (i 100)
  (insert "I will not obey absurd orders\n"))

Nonlocal Exits

Transfer control from one point to another
Unbind all variable bindings made by exited constructs

####### Catch and Throw

Allow nonlocal exit on request

(defun foo-outer ()
  (catch 'foo
    (foo-inner)))

(defun foo-inner ()
  ...
  (if x
      (throw 'foo t))
  ...)

throw when executed tranfers to corresponding catch
second argument of throw is return value of catch
first argument used to find matching catch eq comparision
innermost matching catch takes precedence
If binding constructs like let exited then variables get unbund
throw restores buffers and position saved by save-restriction
throw restores window selection saved by save-window-excursion
lexical nesting unnecessary only needs to be chronologically after catch
emacs-lisp uses only throw for non-local exits

####### catch tag body...

extablishes a return point distinguished by tag
tag can be anything but nil
evaluate body in textual order
if corresponding throw executed, exit with throw second argument as value

####### throw tag value

return to previously established catch
if multiple tag matches use innermost
value becomes value returned by catch

####### Examples of Catch

Using catch throw to exit a double nested loop

;; Example exiting a double nested loop
(defun search-foo ()
  (catch 'loop
    (let ((i 0))
      (while (< i 10)
        (let ((j 0))
          (while (< j 10)
            (if (foo i j)
                (throw 'loop (list i j)))
            (setq j (1+ j))))
        (setq i (1+ i))))))

Errors

Cleanups

Variables

Global Variables
- simplest definition
- Instantiate a variable throughout the lifetime of the system

(setq x '(a b))

Gives x the value (a b)
setq special form
does not evaluate first argument
second argument evaluated and bound to first
Constant Variables
- certain symbols that evaluate to themselves
- nil and t
- : symbols tarting with :
- keywordp object
  - object is symbol name starts with :
Local Variables
- values which are scoped
- argument variables toa function
- only in effect during scope
- allows for nesting and superceding of values
- default scoping dynamic scoping
- dynamic scoping current value is the most recent one created regardless of lexical placement
- allows cusomtization of all variables in scope
- let (bindings...) forms..
  - sets up local bindings
  - returns value of last form in forms
  - binding is either (val) where val gets bound to nil
  - or (var value) where variable is bound to value

(setq y 2)
     => 2
;; value of y gets overriden     
(let ((y 1)
      (z y))
  (list y z))
  => (1 2)

let* (bindings...) forms...
- like let but binding available right after computation
- expression in next binding can represent previous binding

(setq y 2)
     => 2

(let* ((y 1)
       (z y))    ; Use the just-established value of `y'.
  (list y z))
     => (1 1)

Void Variables
- if symbol has unassigned value cell
- unassigned value cell not the same as nil assigned
- evaluating results in void-variable error
- makeunbound symbol
  - empties out value cell making variable void
  - return symbol
  - if symbol has dynamic local binding unbinding only has effect over last shadowed local

(setq x 1)               ; Put a value in the global binding.
     => 1
(let ((x 2))             ; Locally bind it.
  (makunbound 'x)        ; Void the local binding.
  x)
error--> Symbol's value as variable is void: x
x                        ; The global binding is unchanged.
     => 1

(let ((x 2))             ; Locally bind it.
  (let ((x 3))           ; And again.
    (makunbound 'x)      ; Void the innermost-local binding.
    x))                  ; And refer: it's void.
error--> Symbol's value as variable is void: x

(let ((x 2))
  (let ((x 3))
    (makunbound 'x))     ; Void inner binding, then remove it.
  x)                     ; Now outer `let' binding is visible.
     => 2

boundp variable
- returns t if variable is not void nil otherwise

(boundp 'abracadabra)          ; Starts out void.
     => nil
(let ((abracadabra 5))         ; Locally bind it.
  (boundp 'abracadabra))
     => t
(boundp 'abracadabra)          ; Still globally void.
     => nil
(setq abracadabra 5)           ; Make it globally nonvoid.
     => 5
(boundp 'abracadabra)
     => t

Defining Variables
- defconst , defvar - signal intent of varable usage
- defconst used for signaling but emacs allows you to change value defined as const
- defconst unconditionally initializes a variable
- defvar initializes only if variable is originally void
- defcustom defines a customizable variable (uses defvar internally)
- defvar symbol [value [doc-string]]
  - defines symbol as a variable
  - symbol is not evaluated
  - variable marked as special always dynamically bound
  - if symbol already has a value then the value is not even evaluated
  - if symbol has buffer local value then defvar acts on buffer-independent value
  - not current (buffer-local) binding
  - C-M-x eval-defun force setting variable unconditionally without testing

    (defvar bar 23
            "The normal weight of a bar.")
               => bar

defconst symbol value [doc-string]
- defines symbol and initializes it
- establishes global value for the symbol
- marked as special always dynamically bound
- marks the variable as risky
- sets the buffer independent value
Tips for Defining
- Some naming conventions as follows (defined by suffix) :
  - -function : defines functions
  - -functions : The value is a list of functions
  - -hook : variable is a hook
  - -form : The value is a form
  - -forms: The value is a list of forms
  - -predicate : The value is a predicate boolean expression
  - -flag : value significant only if not nil
  - -program : The value is a program name
  - -command : the value is a shell command
  - -switches: value is a list of command switches
- For complicated initializations put it all in a defvar

(defvar my-mode-map
       (let ((map (make-sparse-keymap)))
         (define-key map "\C-c\C-a" 'my-command)
         ...
         map)
       DOCSTRING)

file reloading will initialize it the first time but not second time unless C-M-x is used
Accessing Variables
- symbaol-value symbol
  - returns value in symbol value cell
  - value cell holds current (dynamic) value
  - if variable is void throws void-variable error

 (let ((abracadabra 'foo))
   (symbol-value 'abracadabra))
      => foo

Setting Variables
- setq [symbol form]...
  - symbol given value result of form
  - does not evaluate symbol
  - argument gets automatically quoted

(setq x (1+ 2))
     => 3
     
(let ((x 5))
  (setq x 6)        ; The local binding of `x' is set.
  x)
     => 6
x                   ; The global value is unchanged.
     => 3

set symbol value
- puts VALUE in the value cell of symbol
- symbol is evaluated to obtain the symbol to set
- when dynamic binding is in effect same as setq
- when variable is lexically bound set affects dynamic value
- setq affects the current lexical value

(set one 1)
error--> Symbol's value as variable is void: one
(set 'one 1)
     => 1
(set 'two 'one)
     => one
(set two 2)         ; `two' evaluates to symbol `one'.
     => 2
one                 ; So it is `one' that was set.
     => 2
(let ((one 1))      ; This binding of `one' is set,
  (set 'one 3)      ;   not the global value.
  one)
     => 3
one
     => 2

Scoping Rules for Variable Bindings
- by default local bindings are dynamic bindings
- dynamic bindings have indefinite scope
- dynamic bindings have dynamic extent lasts as long as binding construct(let form) is being executed
- lexical bindings optional support in emacs
- lexical bindings any reference to variable must be textually within binding construct
- lexical binding chan have indefinite extent under closures where binding can live even after finshed execution of binding construct
Dynamic Binding
- variables binding is its current binding at during execution of Lisp program.
- The most recently-created dynamic local binding
- indefinite scope and synamic extent
- shown in example below

 (defvar x -99)  ; `x' receives an initial value of -99.

 (defun getx ()
   x)            ; `x' is used "free" in this function.

 (let ((x 1))    ; `x' is dynamically bound.
   (getx))
      => 1

 ;; After the `let' form finishes, `x' reverts to its
 ;; previous value, which is -99.

 (getx)
      => -99

Free reference in getx
within let takes on let’s binding for x
outiside takes global value of x`

(defvar x -99)      ; `x' receives an initial value of -99.

(defun addx ()
  (setq x (1+ x)))  ; Add 1 to `x' and return its new value.

(let ((x 1))
  (addx)
  (addx))
     => 3           ; The two `addx' calls add to `x' twice.

;; After the `let' form finishes, `x' reverts to its
;; previous value, which is -99.

(addx)
     => -98

Simple implementation
Each symbol value cell has current dynamic value
new values assigned are recorded in a stack
when binding construct (eg. let) finishes pops off old value from stack
restores old value in value cell
Proper Use of Dynamic Binding
- powerful since variables not textually defined can be used
- can make programs hard to understand
- keep it simple
  - if no global definition use variables as a local variable in binding cosntruct
  - else use defvar , defconst or defcustom with variable documentation
  - see variable definition using C-h v
Lexical Binding

(let ((x 1))    ; `x' is lexically bound.
  (+ x 3))
     => 4

(defun getx ()
  x)            ; `x' is used "free" in this function.

(let ((x 1))    ; `x' is lexically bound.
  (getx))
error--> Symbol's value as variable is void: x

Each binding construct defines a lexical environment
specify symbols bound in construct
when evaluator wants variable find first lexical environment
lexical bindings can have indefinite extent.
lexical environment can be “kept around” in “closure”
closure created when named or anonymous function with lexical binding enabled.

(defvar my-ticker nil)   ; We will use this dynamically bound
                         ; variable to store a closure.

(let ((x 0))             ; `x' is lexically bound.
  (setq my-ticker (lambda ()
                    (setq x (1+ x)))))
    => (closure ((x . 0) t) ()
          (1+ x))

(funcall my-ticker)
    => 1

(funcall my-ticker)
    => 2

(funcall my-ticker)
    => 3

x                        ; Note that `x' has no global value.
error--> Symbol's value as variable is void: x

lexical environtment deined by let
x locally bound
lambda expression automatically turned into a closure
evaluation will increment x using x in stored lexical environment
symbol-value, boundp and set only retrieve and modify variables dynamic binding
code in body of defun and defmacro cannot reffer to surrounding lexical varaibles.
lexical variables not used as much
opportunities for optimization.
lexical binding enabled if buffer-local variable lexical-binding is not nil
lexical-binding use lexical instead of dynamic binding but special variables still dynamically bound
must be set in the first line of the file
eval has a LEXICAL argument that enables lexical bindings
speical-variable-p SYMBOL a special variable
byte-compiler wll warn usage of free variables
Buffer Local Variables
- binding associated with a particular buffer
- binding in effect only when buffer is current
- if variable set then new value set in buffer-local binding
- ordinary binding called default binding
- variable can have buffer local binding in some buffers
- make-local-varaible affects only current buffer
- make-variable-buffer-local makes the variable local in all buffers and future buffers
- setq-default becomes the only way to change a variable made local to all buffers
- *WARNING* if let has been used to shadow a buffer-local variable but current buffer has changed let binding wont be visible

     (setq foo 'g)
     (set-buffer "a")
     (make-local-variable 'foo)
     (setq foo 'a)
     (let ((foo 'temp))
       ;; foo => 'temp  ; let binding in buffer `a'
       (set-buffer "b")
       ;; foo => 'g     ; the global value since foo is not local in `b'
       BODY...)
     foo => 'g        ; exiting restored the local value in buffer `a',
                      ; but we don't see that in buffer `b'
     (set-buffer "a") ; verify the local value was restored
     foo => 'a

Creating and Deleting Buffer-Local Bindings
- make-local-variable variable
  - starts with original value of variable
  - other buffers remain unaffected
  - The value returned is variable
  - If variable is void it remains void

          ;; In buffer `b1':
          (setq foo 5)                ; Affects all buffers.
               => 5
          (make-local-variable 'foo)  ; Now it is local in `b1'.
               => foo
          foo                         ; That did not change
               => 5                   ;   the value.
          (setq foo 6)                ; Change the value
               => 6                   ;   in `b1'.
          foo
               => 6

          ;; In buffer `b2', the value hasn't changed.
          (with-current-buffer "b2"
            foo)
               => 5

make-variable-buffer-local variable
File Local Variables
Directory Local Variables
Variable Aliases
Variables with Restricted Values

Functions

Macros

Customization

Loading

Byte Compilation

Advising Function

Debugging

Read and Print

Minibuffers

Command Loop

Keymaps

Modes

Files

Buffers

Windows

Frames

Positions

Markers

Text

Searching and Matching

Syntax Tables

Processes

Suprocess Creation

Shell Arguments

Syncronous Processes

Asynchronous Processes

Deleting Processes

Process Information

Input to Processes

Signals to Processes

Display

System Interface

Summary

Unfortunately with great power comes great responsibility. Managing emacs can be something that requires great care and attention as well as requires openness to hacking at its internals. Perhaps this arcane editor is on its last legs or perhaps it is like a phoenix about to be reborn. Only time will be the judge.

Brief introduction to linux socket filtering

2015-04-24T00:00:00+00:00

We give a brief overview of the linux socket filtering framework. Starting with a breif overview of raw sockets we look at the attaching filters to sockets and give a breif overview of the berekely packet filter machine implemented in the kernel.

Introduction.

Linux socket filter is a register based packet filtering machine. It has two components a network tap and a packet filter. The tap handles the copying and delivery, the filter handles decision to copy. The user has the ability to specify how many bytes of the packet he would like to copied for his use. Only bytes needed for filtering are ever referenced.

Raw Sockets

The user can attach filters to sockets via a call to the setsockopt system call specifying either the SO_ATTACH_FILTER or SO_ATTACH_BPF to the socket. Now in our case assuming we created our socket as a SOCK_RAW socket

The entry for SOCK_RAW in the inetsw_array looks like

       {
	       .type =       SOCK_RAW,
	       .protocol =   IPPROTO_IP,	/* wild card */
	       .prot =       &raw_prot,
	       .ops =        &inet_sockraw_ops,
	       .flags =      INET_PROTOSW_REUSE,
       }
// From af_inet.c

// From  raw.c
struct proto raw_prot = {
	.name		   = "RAW",
	.owner		   = THIS_MODULE,
	.close		   = raw_close,
	.destroy	   = raw_destroy,
	.connect	   = ip4_datagram_connect,
	.disconnect	   = udp_disconnect,
	.ioctl		   = raw_ioctl,
	.init		   = raw_init,
	.setsockopt	   = raw_setsockopt,
	.getsockopt	   = raw_getsockopt,
	.sendmsg	   = raw_sendmsg,
	.recvmsg	   = raw_recvmsg,
	.bind		   = raw_bind,
	.backlog_rcv	   = raw_rcv_skb,
	.release_cb	   = ip4_datagram_release_cb,
	.hash		   = raw_hash_sk,
	.unhash		   = raw_unhash_sk,
	.obj_size	   = sizeof(struct raw_sock),
	.h.raw_hash	   = &raw_v4_hashinfo,
#ifdef CONFIG_COMPAT
	.compat_setsockopt = compat_raw_setsockopt,
	.compat_getsockopt = compat_raw_getsockopt,
	.compat_ioctl	   = compat_raw_ioctl,
#endif
};

Going back the function of ip_local_deliver_finish Where the ip header is checked to find out the protocol to dispatch incoming packet to.

We see the function raw_local_deliver is used to made the determination if the packet is meant for delivery to a raw socket.

int raw_local_deliver(struct sk_buff *skb, int protocol)
{
	int hash;
	struct sock *raw_sk;

	hash = protocol & (RAW_HTABLE_SIZE - 1);
	raw_sk = sk_head(&raw_v4_hashinfo.ht[hash]);

	/* If there maybe a raw socket we must check - if not we
	 * don't care less
	 */
	if (raw_sk && !raw_v4_input(skb, ip_hdr(skb), hash))
		raw_sk = NULL;

	return raw_sk != NULL;

}

This will determine if there is a raw socket that is waiting on the protocol. If there is then we must deliver the packet to this socket. The __raw_v4_lookup function is used to look up raw sockets that would be interested in this skb. Attributes of the look up may include protocol, source address, destination address, device interface index. We call skb_clone function to clone the skb and call the raw_rcv on the socket with the cloned packet. raw_rcv will reset the packet headers back to pointing to this IP layer. Finally adding the packet to the socket’s receive queue. The raw_rcv will eventually call sock_queue_rcv_skb. Which will check if the packet ought to be filtered via a call to the sk_filter function.

//from raw.c 
static int raw_v4_input(struct sk_buff *skb, const struct iphdr *iph, int hash)
{
	struct sock *sk;
	struct hlist_head *head;
	int delivered = 0;
	struct net *net;

	read_lock(&raw_v4_hashinfo.lock);
	head = &raw_v4_hashinfo.ht[hash];
	if (hlist_empty(head))
		goto out;

	net = dev_net(skb->dev);
	sk = __raw_v4_lookup(net, __sk_head(head), iph->protocol,
			     iph->saddr, iph->daddr,
			     skb->dev->ifindex);

	while (sk) {
		delivered = 1;
		if ((iph->protocol != IPPROTO_ICMP || !icmp_filter(sk, skb)) &&
		    ip_mc_sf_allow(sk, iph->daddr, iph->saddr,
				   skb->dev->ifindex)) {
			struct sk_buff *clone = skb_clone(skb, GFP_ATOMIC);

			/* Not releasing hash table! */
			if (clone)
				raw_rcv(sk, clone);
		}
		sk = __raw_v4_lookup(net, sk_next(sk), iph->protocol,
				     iph->saddr, iph->daddr,
				     skb->dev->ifindex);
	}
out:
	read_unlock(&raw_v4_hashinfo.lock);
	return delivered;
}

If we had attached a socket filter the raw socket we are going to run this filter at this point.

The packet filter details

The call to sk_filter gets passed in a socket and an skb. Inside the socket if we have attached a socket filter this socket filter struct sk_filter *filter is going to be found. Resulting in a call to SK_RUN_FILTER. The return type of which is amount of packet information the user program is interested in.

// from filter.h
/* Macro to invoke filter function. */
#define SK_RUN_FILTER(filter, ctx) \
(*filter->prog->bpf_func)(ctx, filter->prog->insnsi)

.....

struct bpf_prog {
	u16			pages;		/* Number of allocated pages */
	bool			jited;		/* Is our filter JIT'ed? */
	u32			len;		/* Number of filter blocks */
	struct sock_fprog_kern	*orig_prog;	/* Original BPF program */
	struct bpf_prog_aux	*aux;		/* Auxiliary fields */
	unsigned int		(*bpf_func)(const struct sk_buff *skb,
					    const struct bpf_insn *filter);
	/* Instructions for interpreter */
	union {
		struct sock_filter	insns[0];
		struct bpf_insn		insnsi[0];
	};
};

struct sk_filter {
	atomic_t	refcnt;
	struct rcu_head	rcu;
	struct bpf_prog	*prog;
};

During the creation of the packet filter the program that the user passed in will be copied into the socket filter and jit compiled via bpf_jit_compile or in case no jit is available the program is translated into an optimized interpreter via bpf_migrate_filter.

Understanding the instruction set

The BPF engine contains a low level asm-like filter language. Consisting of the following basic elements

  Element          Description

  A                32 bit wide accumulator
  X                32 bit wide X register
  M[]              16 x 32 bit wide misc registers aka "scratch memory
                   store", addressable from 0 to 15

An instruction would look like

    op:16, jt:8, jf:8, k:32

op is a 16 bit opcode for an instruction, jt and jf are two 8-bit jump targets where jt is jump target if true and jf is jump target if false k is an instruction dependent argument.

Following table lists the instructions based on the above machine model. With A as accumulator, X as register and M as scratch memory.

  Instruction      Addressing mode      Description

  ld               1, 2, 3, 4, 10       Load word into A
  ldi              4                    Load word into A
  ldh              1, 2                 Load half-word into A
  ldb              1, 2                 Load byte into A
  ldx              3, 4, 5, 10          Load word into X
  ldxi             4                    Load word into X
  ldxb             5                    Load byte into X

  st               3                    Store A into M[]
  stx              3                    Store X into M[]

  jmp              6                    Jump to label
  ja               6                    Jump to label
  jeq              7, 8                 Jump on k == A
  jneq             8                    Jump on k != A
  jne              8                    Jump on k != A
  jlt              8                    Jump on k < A
  jle              8                    Jump on k <= A
  jgt              7, 8                 Jump on k > A
  jge              7, 8                 Jump on k >= A
  jset             7, 8                 Jump on k & A

  add              0, 4                 A + <x>
  sub              0, 4                 A - <x>
  mul              0, 4                 A * <x>
  div              0, 4                 A / <x>
  mod              0, 4                 A % <x>
  neg              0, 4                 !A
  and              0, 4                 A & <x>
  or               0, 4                 A | <x>
  xor              0, 4                 A ^ <x>
  lsh              0, 4                 A << <x>
  rsh              0, 4                 A >> <x>

  tax                                   Copy A into X
  txa                                   Copy X into A

  ret              4, 9                 Return

Along with the above instructions we have the following addressing modes for addressing locations in packets, in scratch memory and registers.

  Addressing mode  Syntax               Description

   0               x/%x                 Register X
   1               [k]                  BHW at byte offset k in the packet
   2               [x + k]              BHW at the offset X + k in the packet
   3               M[k]                 Word at offset k in M[]
   4               #k                   Literal value stored in k
   5               4*([k]&0xf)          Lower nibble * 4 at byte offset k in the packet
   6               L                    Jump label L
   7               #k,Lt,Lf             Jump to Lt if true, otherwise jump to Lf
   8               #k,Lt                Jump to Lt if predicate is true
   9               a/%a                 Accumulator A
   10               extension            BPF extension

Some linux extensions to BPF which allow more convenient access to frequently needed data.

Possible BPF extensions are shown in the following table:

  Extension                             Description

  len                                   skb->len
  proto                                 skb->protocol
  type                                  skb->pkt_type
  poff                                  Payload start offset
  ifidx                                 skb->dev->ifindex
  nla                                   Netlink attribute of type X with offset A
  nlan                                  Nested Netlink attribute of type X with offset A
  mark                                  skb->mark
  queue                                 skb->queue_mapping
  hatype                                skb->dev->type
  rxhash                                skb->hash
  cpu                                   raw_smp_processor_id()
  vlan_tci                              vlan_tx_tag_get(skb)
  vlan_pr                               vlan_tx_tag_present(skb)
  rand                                  prandom_u32()

To get a feel for some of these instructions and their usages consider a sample BPF programs.

/* IPv4 TCP packet filter */
ldh [12]           /* not sure */
jne #0x800, drop   /*  If loaded half word(16 bits?) is not #0x800  go to label drop */
                   /*  I think checking for ip. */
ldb [23]           /*  load byte at offset 23  the protocol field of the ip packet */
jneq #6, drop      /*  compare with protocol number for tcp which  0x6 not go to drop */
ret #-1            /*  accept whole packet */
drop: ret #0       /*  drop the packet */

We can now save this into a file say tcp.f , and run the accompanying assembler. We can generate code that can be directly loaded by the bpf_dbg where commands get converted into their op codes.

$ ./bpf_asm tcp.f
6,40 0 0 12,21 0 3 2048,48 0 0 23,21 0 1 6,6 0 0 4294967295,6 0 0 0,

Often one needs to get bpf code in syntax used to specify filters in C code. We can get this by running

$ ./bpf_asm -c tcp.f
{ 0x28,  0,  0, 0x0000000c },
{ 0x15,  0,  3, 0x00000800 },
{ 0x30,  0,  0, 0x00000017 },
{ 0x15,  0,  1, 0x00000006 },
{ 0x06,  0,  0, 0xffffffff },
{ 0x06,  0,  0, 0000000000 },

This code can be used in one’s c code to while attaching to sockets like so

// untested.
// Where the instructionformat is
// bpf.h
struct bpf_insn {
	__u8	code;		/* opcode */
	__u8	dst_reg:4;	/* dest register */
	__u8	src_reg:4;	/* source register */
	__s16	off;		/* signed offset */
	__s32	imm;		/* signed immediate constant */
};

// in user code.
struct bpf_insn insns[] = {
     { 0x28,  0,  0, 0x0000000c },
     { 0x15,  0,  3, 0x00000800 },
     { 0x30,  0,  0, 0x00000017 },
     { 0x15,  0,  1, 0x00000006 },
     { 0x06,  0,  0, 0xffffffff },
     { 0x06,  0,  0, 0000000000 },
};

Since tools like tcpdump use the libpcap library to compile user specified filter commands to bpf they can be helpful aids to quickly generating bpf code. This is where tcpdump options -d , -dd and -ddd are helpful. As shown bellow. We can use -d to see the mnemonic code which the tcpdump will generate for an expression. -dd to generate C-code for an expression and finally -ddd will generate them as decimal numbers loadable directly into bpf_dbg and other tools.

# tcpdump -iwlan0 -d  'tcp'
(000) ldh      [12]
(001) jeq      #0x86dd          jt 2	jf 7
(002) ldb      [20]
(003) jeq      #0x6             jt 10	jf 4
(004) jeq      #0x2c            jt 5	jf 11
(005) ldb      [54]
(006) jeq      #0x6             jt 10	jf 11
(007) jeq      #0x800           jt 8	jf 11
(008) ldb      [23]
(009) jeq      #0x6             jt 10	jf 11
(010) ret      #262144
(011) ret      #0

# sudo tcpdump -iwlan0 -dd  'tcp'
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 5, 0x000086dd },
{ 0x30, 0, 0, 0x00000014 },
{ 0x15, 6, 0, 0x00000006 },
{ 0x15, 0, 6, 0x0000002c },
{ 0x30, 0, 0, 0x00000036 },
{ 0x15, 3, 4, 0x00000006 },
{ 0x15, 0, 3, 0x00000800 },
{ 0x30, 0, 0, 0x00000017 },
{ 0x15, 0, 1, 0x00000006 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000000 },

# sudo tcpdump -iwlan0 -ddd 'tcp
12
40 0 0 12
21 0 5 34525
48 0 0 20
21 6 0 6
21 0 6 44
48 0 0 54
21 3 4 6
21 0 3 2048
48 0 0 23
21 0 1 6
6 0 0 262144
6 0 0 0

One additional tool that my be helpful to mention in the context of the bpf. Is the bpf_dbg tool(in tools/net directory of kernel source) which can be used to debug bpf filters over pcap files.This a bpf debugger allowing us to run bpf , step through the code and more.

// First we capture some traffic in the the pcap format
# sudo tcpdump -s 0 -i wlan0 -w mycap.pcap

// we generate the decimal notation of our desired packet filter.
# sudo tcpdump -iwlan0 -ddd  'tcp' | tr '\n' ','
12,40 0 0 12,21 0 5 34525,48 0 0 20,21 6 0 6,21 0 6 44,48 0 0 54,21 3 4 6,21 0 3 2048,48 0 0 23,21 0 1 6,6 0 0 262144,6 0 0 0,

# sudo ./bpf_dbg
> load pcap mycap.pcap
> load bpf 12,40 0 0 12,21 0 5 34525,48 0 0 20,21 6 0 6,21 0 6 44,48 0 0 54,21 3 4 6,21 0 3 2048,48 0 0 23,21 0 1 6,6 0 0 262144,6 0 0 0,
> run
bpf passes:1716 fails:432
> step
-- register dump --
pc:       [0]
code:     [40] jt[0] jf[0] k[12]
curr:     l0:	ldh [12]
A:        [00000000][0]
X:        [00000000][0]
M[0,15]:  [00000000][0]
-- packet dump --
len: 86
  0: 74 9d dc 8d 68 61 00 22 68 ac 53 2f 86 dd 60 00 
 16: 00 00 00 20 06 40 26 02 03 06 bc ca f1 00 4c 72 
 32: 39 e9 c1 5c 0c 1a 20 01 4d e0 41 01 00 01 00 00 
 48: 00 00 d1 6b c2 58 d2 54 01 bb f0 71 8c f0 3b ef 
 64: 34 b6 80 10 01 ec 51 2b 00 00 01 01 08 0a 00 b6 
 80: 2b 78 81 b3 46 5f

Where the load can be used in to load in captured packets and bpf code which can then be stepped through and debugged.

Other helpful commands available in the debugger are

disassemble    - get the mnemonic code for filter
dump           - dump c like code 
breakpoint #   -  set a line break point
select     #   - select a given packet for running index of 1
step    #      - step and dump registers
quit           - quit the debugger

Complete Example

Finally to wrap up we present a complete example program that uses raw sockets to listen to http requests and responses and prints it out to standard out. This is just a modification of the code presented in the binary tides article for illustration purposes.

/** References : https://gist.github.com/msantos/939154
    http://www.binarytides.com/packet-sniffer-code-in-c-using-linux-sockets-bsd-part-2/
*/
#include <stdio.h>
#include<errno.h>

#include<stdio.h> //For standard things
#include<stdlib.h>    //malloc
#include<string.h>    //strlen

#include<netinet/tcp.h>   //Provides declarations for tcp header
#include<netinet/ip.h>    //Provides declarations for ip header
#include<net/ethernet.h>  //For ether_header

#include<sys/socket.h>
#include<arpa/inet.h>

#include <linux/filter.h>

//sudo tcpdump -A -dd -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
struct sock_filter tcp_filter [] = {
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 27, 0, 0x000086dd },
{ 0x15, 0, 26, 0x00000800 },
{ 0x30, 0, 0, 0x00000017 },
{ 0x15, 0, 24, 0x00000006 },
{ 0x28, 0, 0, 0x00000014 },
{ 0x45, 22, 0, 0x00001fff },
{ 0xb1, 0, 0, 0x0000000e },
{ 0x48, 0, 0, 0x0000000e },
{ 0x15, 2, 0, 0x00000050 },
{ 0x48, 0, 0, 0x00000010 },
{ 0x15, 0, 17, 0x00000050 },
{ 0x28, 0, 0, 0x00000010 },
{ 0x2, 0, 0, 0x00000001 },
{ 0x30, 0, 0, 0x0000000e },
{ 0x54, 0, 0, 0x0000000f },
{ 0x64, 0, 0, 0x00000002 },
{ 0x7, 0, 0, 0x00000005 },
{ 0x60, 0, 0, 0x00000001 },
{ 0x1c, 0, 0, 0x00000000 },
{ 0x2, 0, 0, 0x00000005 },
{ 0xb1, 0, 0, 0x0000000e },
{ 0x50, 0, 0, 0x0000001a },
{ 0x54, 0, 0, 0x000000f0 },
{ 0x74, 0, 0, 0x00000002 },
{ 0x7, 0, 0, 0x00000009 },
{ 0x60, 0, 0, 0x00000005 },
{ 0x1d, 1, 0, 0x00000000 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000000 },
};

void print_tcp_packet(unsigned char* packet, int size);
void process_packet(char* packet, int packet_size);
void print_ip_header(unsigned char* packet, int packet_size);
void print_data (unsigned char* data , int Size);


int main(int argc , char* argv[]){
  unsigned char *packet ;
  int saddr_size ;
  int data_size;
  struct sockaddr saddr;         

  const int max_packet_size = 65536; //Its Big!
  printf("begin: http sniff\n");
  
  packet  = (unsigned char *) malloc(max_packet_size); 
  
  int sock_fd = socket(AF_PACKET,SOCK_RAW,htons(ETH_P_ALL)) ;
  if(sock_fd < 0)    {    
    perror("Socket Error");
    return 1;
  }
  struct sock_fprog fcode = {0};
  fcode.len = sizeof(tcp_filter) / sizeof(struct sock_filter);
  fcode.filter = &tcp_filter[0];
  
  setsockopt(sock_fd,SOL_SOCKET,SO_ATTACH_FILTER,&fcode,sizeof(fcode));
  
  while(1)
    {
        saddr_size = sizeof saddr;
        //Receive a packet
        data_size = recvfrom(sock_fd , packet , max_packet_size , 0 , &saddr , (socklen_t*)&saddr_size);
        if(data_size <0 )
        {
            printf("Recvfrom error , failed to get packets\n");
            return 1;
        }
        process_packet(packet,data_size);
    }
  
  free(packet);
  printf("end: http sniff\n");
  return 0;
}

void process_packet(char* packet, int packet_size)
{
  printf("process_packet: packet size %d \n",packet_size);
  print_tcp_packet(packet, packet_size);
}


void print_ip_header(unsigned char* Buffer, int Size)
{
  //    print_ethernet_header(Buffer , Size);
  struct sockaddr_in source,dest;
    unsigned short iphdrlen;
         
    struct iphdr *iph = (struct iphdr *)(Buffer  + sizeof(struct ethhdr) );
    iphdrlen =iph->ihl*4;

    memset(&source, 0, sizeof(source));
    source.sin_addr.s_addr = iph->saddr;
     
    memset(&dest, 0, sizeof(dest));
    dest.sin_addr.s_addr = iph->daddr;
     
    fprintf(stdout , "\n");
    fprintf(stdout , "IP Header\n");
    fprintf(stdout , "   |-IP Version        : %d\n",(unsigned int)iph->version);
    fprintf(stdout , "   |-IP Header Length  : %d DWORDS or %d Bytes\n",(unsigned int)iph->ihl,((unsigned int)(iph->ihl))*4);
    fprintf(stdout , "   |-Type Of Service   : %d\n",(unsigned int)iph->tos);
    fprintf(stdout , "   |-IP Total Length   : %d  Bytes(Size of Packet)\n",ntohs(iph->tot_len));
    fprintf(stdout , "   |-Identification    : %d\n",ntohs(iph->id));
    //fprintf(stdout , "   |-Reserved ZERO Field   : %d\n",(unsigned int)iphdr->ip_reserved_zero);
    //fprintf(stdout , "   |-Dont Fragment Field   : %d\n",(unsigned int)iphdr->ip_dont_fragment);
    //fprintf(stdout , "   |-More Fragment Field   : %d\n",(unsigned int)iphdr->ip_more_fragment);
    fprintf(stdout , "   |-TTL      : %d\n",(unsigned int)iph->ttl);
    fprintf(stdout , "   |-Protocol : %d\n",(unsigned int)iph->protocol);
    fprintf(stdout , "   |-Checksum : %d\n",ntohs(iph->check));
    fprintf(stdout , "   |-Source IP        : %s\n",inet_ntoa(source.sin_addr));
    fprintf(stdout , "   |-Destination IP   : %s\n",inet_ntoa(dest.sin_addr));

}



void print_tcp_packet(unsigned char* Buffer, int Size)
{
    unsigned short iphdrlen;
     
    struct iphdr *iph = (struct iphdr *)( Buffer  + sizeof(struct ethhdr) );
    iphdrlen = iph->ihl*4;
     
    struct tcphdr *tcph=(struct tcphdr*)(Buffer + iphdrlen + sizeof(struct ethhdr));
             
    int header_size =  sizeof(struct ethhdr) + iphdrlen + tcph->doff*4;
     
    fprintf(stdout , "\n\n***********************TCP Packet*************************\n");  
         
    print_ip_header(Buffer,Size);
         
    fprintf(stdout , "\n");
    fprintf(stdout , "TCP Header\n");
    fprintf(stdout , "   |-Source Port      : %u\n",ntohs(tcph->source));
    fprintf(stdout , "   |-Destination Port : %u\n",ntohs(tcph->dest));
    fprintf(stdout , "   |-Sequence Number    : %u\n",ntohl(tcph->seq));
    fprintf(stdout , "   |-Acknowledge Number : %u\n",ntohl(tcph->ack_seq));
    fprintf(stdout , "   |-Header Length      : %d DWORDS or %d BYTES\n" ,(unsigned int)tcph->doff,(unsigned int)tcph->doff*4);
    //fprintf(stdout , "   |-CWR Flag : %d\n",(unsigned int)tcph->cwr);
    //fprintf(stdout , "   |-ECN Flag : %d\n",(unsigned int)tcph->ece);
    fprintf(stdout , "   |-Urgent Flag          : %d\n",(unsigned int)tcph->urg);
    fprintf(stdout , "   |-Acknowledgement Flag : %d\n",(unsigned int)tcph->ack);
    fprintf(stdout , "   |-Push Flag            : %d\n",(unsigned int)tcph->psh);
    fprintf(stdout , "   |-Reset Flag           : %d\n",(unsigned int)tcph->rst);
    fprintf(stdout , "   |-Synchronise Flag     : %d\n",(unsigned int)tcph->syn);
    fprintf(stdout , "   |-Finish Flag          : %d\n",(unsigned int)tcph->fin);
    fprintf(stdout , "   |-Window         : %d\n",ntohs(tcph->window));
    fprintf(stdout , "   |-Checksum       : %d\n",ntohs(tcph->check));
    fprintf(stdout , "   |-Urgent Pointer : %d\n",tcph->urg_ptr);
    fprintf(stdout , "\n");
    fprintf(stdout , "                        DATA Dump                         ");
    fprintf(stdout , "\n");

    /**
    fprintf(stdout , "IP Header\n");
    print_data(Buffer,iphdrlen);
         
    fprintf(stdout , "TCP Header\n");
    print_data(Buffer+iphdrlen,tcph->doff*4);
    */                           
    fprintf(stdout , "Data Payload\n");    
    print_data(Buffer + header_size , Size - header_size );

    fprintf(stdout , "\n###########################################################\n");
}



void print_data (unsigned char* data , int Size)
{
    int i , j;
    for(i=0 ; i < Size ; i++)
    {
        if( i!=0 && i%16==0)   //if one line of hex printing is complete...
        {
            fprintf(stdout , "         ");
            for(j=i-16 ; j<i ; j++)
            {
                if(data[j]>=32 && data[j]<=128)
                    fprintf(stdout , "%c",(unsigned char)data[j]); //if its a number or alphabet
                 
                else fprintf(stdout , "."); //otherwise print a dot
            }
            fprintf(stdout , "\n");
        } 
         
        if(i%16==0) fprintf(stdout , "   ");
            fprintf(stdout , " %02X",(unsigned int)data[i]);
                 
        if( i==Size-1)  //print the last spaces
        {
            for(j=0;j<15-i%16;j++) 
            {
              fprintf(stdout , "   "); //extra spaces
            }
             
            fprintf(stdout , "         ");
             
            for(j=i-i%16 ; j<=i ; j++)
            {
                if(data[j]>=32 && data[j]<=128) 
                {
                  fprintf(stdout , "%c",(unsigned char)data[j]);
                }
                else
                {
                  fprintf(stdout , ".");
                }
            }
             
            fprintf(stdout ,  "\n" );
        }
    }
}

Summary

This article scratches the surface of understanding the flexibility offered in the linux networking stack. Since this more as a brain dump of reading through the source in the 3.19-rc7 kernel networking subsystem,thus it may be highly unreliable and inaccurate. Users beware! With that fair warning lets try to begin.Any errors in the article are entirely my fault.

Tracing a packet through the linux networking subsystems.

2015-03-16T00:00:00+00:00

In this post we trace the path of a packet from through the linux networking subsystem. We go from a breif overview from packet’s arrival at the network card to its final destination on a socket’s receive queue.

Networking drivers and their woes

We try to take as example a simple networking driver which seems to be widely used as an example the RTL8139 ethernet driver. Its a fairly complicated driver for a ethernet PCI card which I dont pretend to understand. Just to document some key parts. Most of the source code can be found in

<linux-src>/drivers/net/ethernet/realtek/8139too.c

When the driver module is brought up it registers it self with the PCI subsystem in the kernel using the standard call of pci_register_driver. Defining standard interface methods, as shown below.

static struct pci_driver rtl8139_pci_driver = {
	.name		= DRV_NAME,
	.id_table	= rtl8139_pci_tbl,
	.probe		= rtl8139_init_one,
	.remove		= rtl8139_remove_one,
#ifdef CONFIG_PM
	.suspend	= rtl8139_suspend,
	.resume		= rtl8139_resume,
#endif /* CONFIG_PM */
};

As the board initialized via the probe function the method rtl8139_init_one is called. This method will confirm that a board plugged in is belongs to the current vendor. The board gets initialized and the we map the memory regions on the PCI device to and ioaddr.

   	ioaddr = pci_iomap(pdev, bar, 0);

Using the bar register of the pci device. This is so that we can perform memory mapped io with the device. We also initialize the key kernel structure used to describe the network device a huge datastructure called struct net_device.

The netdev data structure also contains a dev->netdev_ops defining device operations.

static const struct net_device_ops rtl8139_netdev_ops = {
	.ndo_open		= rtl8139_open,
	.ndo_stop		= rtl8139_close,
	.ndo_get_stats64	= rtl8139_get_stats64,
	.ndo_change_mtu		= rtl8139_change_mtu,
	.ndo_validate_addr	= eth_validate_addr,
	.ndo_set_mac_address 	= rtl8139_set_mac_address,
	.ndo_start_xmit		= rtl8139_start_xmit,
	.ndo_set_rx_mode	= rtl8139_set_rx_mode,
	.ndo_do_ioctl		= netdev_ioctl,
	.ndo_tx_timeout		= rtl8139_tx_timeout,
#ifdef CONFIG_NET_POLL_CONTROLLER
	.ndo_poll_controller	= rtl8139_poll_controller,
#endif
	.ndo_set_features	= rtl8139_set_features,
};

In the the function rtl8139_open as a key step we define a method to respond to interrupts from the hardware done as follows

   	retval = request_irq(irq, rtl8139_interrupt, IRQF_SHARED, dev->name, dev);

Where the irq to use is obtained from the pci configuration of the device. We also allocate two buffers to which will be mapped to the transfer and receive buffers on the device

	struct rtl8139_private *tp = netdev_priv(dev);
        ....
         ....

	tp->tx_bufs = dma_alloc_coherent(&tp->pci_dev->dev, TX_BUF_TOT_LEN,
					   &tp->tx_bufs_dma, GFP_KERNEL);
	tp->rx_ring = dma_alloc_coherent(&tp->pci_dev->dev, RX_BUF_TOT_LEN,
					   &tp->rx_ring_dma, GFP_KERNEL);

Too delay with highly interrupting devices linux has started to move to newer api called napi which can dynamically switch a device from polling to interrupt mode based on certain policy considerations.

Finally we perform certain device specific initializations in rtl8139_hw_start Like enabling interrupts on the device, setting receive modes and other device specific miscellany.

Having thus set up the device we allow the linux to start using this device to send packets by calling the key method

static inline void netif_start_queue(struct net_device *dev)

netif_start_queue (dev);

There is also a watchdog timer which I am punting on for now.

Now on packet receipt the device is going to raise the interrupt calling our method.

static irqreturn_t rtl8139_interrupt (int irq, void *dev_instance)

If we can schedule the running of napi we do it as shown here

	if (status & RxAckBits){
		if (napi_schedule_prep(&tp->napi)) {
			RTL_W16_F (IntrMask, rtl8139_norx_intr_mask);
			__napi_schedule(&tp->napi);
		}
	}

The receipt of the packets being are processed thus by the napi which will call the specified poll routine.

static int rtl8139_poll(struct napi_struct *napi, int budget)

Passed in a fixed budget which decides to perform the receipt now or later.

The actual method doing the receipt is :

static int rtl8139_rx(struct net_device *dev, struct rtl8139_private *tp,
		      int budget)

If all is well we will allocate a an skb. The key kernel datastructure to hold packets received and being processed up the protocol stack. Thus we now copy the packet from the device receive buffer into an skb. Update some device statistics. Detect the link layer protocl used by the packet and finally call the key method netif_receive_skb with the copied packet.

   netif_receive_skb (skb);

As of reading this its unclear to me if the copy happens in the context of the actual interrupt or in the context of the soft IRQ generated by the napi subsystem.

Either way the netif_receive_skb will take place. The skb is not going to get queued int to a per cpu packet backlog queue. Called softnet_data. Using the function

static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
			      unsigned int *qtail)

....
	__skb_queue_tail(&sd->input_pkt_queue, skb);
.....
	return NET_RX_SUCCESS;
...

A packet arrives

After successfully queuing the packet onto the cpu backlog queue we are going to return NET_RX_SUCCESS to the driver. Now moving away from the driver side of packet receipt to the operating system side of processing the packet. I still need to look into how the process_backlog queue getting invoked.

   static int process_backlog(struct napi_struct *napi, int quota)

Anyway our process function gets called at which point we dequeue the from the per queue

   static int process_backlog(struct napi_struct *napi, int quota)
   .....
		while ((skb = __skb_dequeue(&sd->process_queue))) {
			local_irq_enable();
			__netif_receive_skb(skb);
			local_irq_disable();
			input_queue_head_incr(sd);
			if (++work >= quota) {
				local_irq_enable();
				return work;
			}
		}

 ......

Its now the job of __netif_receive_skb to take the job of processing the skb. After some munging around of the skb our __netif_receive_skb_core will get called which will call the function deliver_skb.

A key step here is to determine the packet type that we are dealing with here. Different protocols register their packet types allowing themselves to become identifiable. The key packet type interface is described as follows :

struct packet_type {
	__be16			type;	/* This is really htons(ether_type). */
	struct net_device	*dev;	/* NULL is wildcarded here	     */
	int			(*func) (struct sk_buff *,
					 struct net_device *,
					 struct packet_type *,
					 struct net_device *);
	bool			(*id_match)(struct packet_type *ptype,
					    struct sock *sk);
	void			*af_packet_priv;
	struct list_head	list;
};

We can see the packet type of ipv4 in net/ipv4/af_inet.c. As shown here

#define ETH_P_IP	0x0800		/* Internet Protocol packet	*/
....
static struct packet_type ip_packet_type __read_mostly = {
	.type = cpu_to_be16(ETH_P_IP),
	.func = ip_rcv,
};

Thus our deliver_skb function is going to match the type of the packet as ip and call ip_rcv.

int ip_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev)

ip_input.c

In the ip_rcv function we are going to parse out the ip header from skb. Determine the length of the packet Update some statistics. Finally ending with the mysterious Netfilter hook which is generally used customize action on packets if we so choose. As shown here

   int ip_rcv(struct sk_buff *skb, struct net_device *dev, struct packet_type *pt, struct net_device *orig_dev)
   ....
	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);
   .....

The key function that is provided to the netfilter hook is the ip_rcv_finish function which is called if netfilter wants to continue the processing of the packet.

static int ip_rcv_finish(struct sk_buff *skb) {
....
}

A packet begins its ascent

The ip_rcv_finish may need to look into the packet , check if it needs to be routed to other machines. I am only going to look at the case that the packet is destined to the current machine.

The ip layer consults the routing table and a routing table cache to find out where the packet is meant to be delivered.

Finally if the packet is to be delivered to the local host it returns a struct dst_entry with its input method set to ip_local_deliver.

The ip_local_deliver gets called we encounter another netfilter hook NF_INET_LOCAL_IN which is called as follows.

int ip_local_deliver(struct sk_buff *skb)
{
....
  return NF_HOOK(NFPROTO_IPV4, NF_INET_LOCAL_IN, skb, skb->dev, NULL,
         ip_local_deliver_finish);
}

Thus finding we can now add a netfilter hook just for packets meant for the local host. Assuming again that netfilter allows for further processing of the packet we are now ready to begin further processing of the packet.

Inside of ip_local_deliver_finish we are now ready to examine the ip protocol to which the packet ought to be delivered. There is some thing about raw delivery which needs to be looked at but currently skipped.

static int ip_local_deliver_finish(struct sk_buff *skb)
{
        ......
        int protocol = ip_hdr(skb)->protocol;
        ....
	ipprot = rcu_dereference(inet_protos[protocol]);
        ...
   	ret = ipprot->handler(skb);
        ......
}

Notice how we look up the protocol in the ip header and then use this protocol look up the inet_protos array for implementing protocol finally calling its handler. These protocl handlers are initialized inet subsystem initialization with a call to inet_inet.

static int __init inet_init(void)
{
        ....
       	if (inet_add_protocol(&icmp_protocol, IPPROTO_ICMP) < 0)
		pr_crit("%s: Cannot add ICMP protocol\n", __func__);
	if (inet_add_protocol(&udp_protocol, IPPROTO_UDP) < 0)
		pr_crit("%s: Cannot add UDP protocol\n", __func__);
	if (inet_add_protocol(&tcp_protocol, IPPROTO_TCP) < 0)
		pr_crit("%s: Cannot add TCP protocol\n", __func__);
#ifdef CONFIG_IP_MULTICAST
	if (inet_add_protocol(&igmp_protocol, IPPROTO_IGMP) < 0)
		pr_crit("%s: Cannot add IGMP protocol\n", __func__);
#endif
        ....
}

There thus we see the initialization of the protocl array with some common ip protocols. The protocol themselves are describe as follows.

static const struct net_protocol tcp_protocol = {
	.early_demux	=	tcp_v4_early_demux,
	.handler	=	tcp_v4_rcv,
	.err_handler	=	tcp_v4_err,
	.no_policy	=	1,
	.netns_ok	=	1,
	.icmp_strict_tag_validation = 1,
};

static const struct net_protocol udp_protocol = {
	.early_demux =	udp_v4_early_demux,
	.handler =	udp_rcv,
	.err_handler =	udp_err,
	.no_policy =	1,
	.netns_ok =	1,
};

static const struct net_protocol icmp_protocol = {
	.handler =	icmp_rcv,
	.err_handler =	icmp_err,
	.no_policy =	1,
	.netns_ok =	1,
};

We see each protocol defining its corresponding handlers.We however are only going to look at the udp handler. To keep it relatively simple.

Home Sweet Socket

UDP much like tcp contains a hash table of sockets that are currently in listening for packets.

/**
 *	struct udp_table - UDP table
 *
 *	@hash:	hash table, sockets are hashed on (local port)
 *	@hash2:	hash table, sockets are hashed on (local port, local address)
 *	@mask:	number of slots in hash tables, minus 1
 *	@log:	log2(number of slots in hash table)
 */
struct udp_table {
	struct udp_hslot	*hash;
	struct udp_hslot	*hash2;
	unsigned int		mask;
	unsigned int		log;
};
extern struct udp_table udp_table;

The details of this are left to the reader to delve into. Assuming that the packet was a udp packet its protocl must have been initialized to

#define IPPROTO_UDP		IPPROTO_UDP
  IPPROTO_IDP = 22,		/* XNS IDP protocol			*/

Now we begin to look up the socket and do simple checksum. The look up takes into account the source and destination and the source port and destination ports. As we see from the the arguments to the look method.

int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
		   int proto)
{
        ....
	sk = __udp4_lib_lookup_skb(skb, uh->source, uh->dest, udptable);
        ....
}

struct sock *__udp4_lib_lookup(struct net *net, __be32 saddr,
		__be16 sport, __be32 daddr, __be16 dport,
		int dif, struct udp_table *udptable)

If there is a socket listening we ought to find it. Finally calling the udp_queue_rcv_skb with the found socket and the skb packet.

   	ret = udp_queue_rcv_skb(sk, skb);

Which is going to finally translate into a call to sock_queue_rcv_skb. In case we are using some sort of socket filtering which I believe is somewhat similar to the Berkeley packet filter we pass the socket and and the skb to that socket filter. The underlying method for this is the sk_filter.

   int sk_filter(struct sock *sk, struct sk_buff *skb)
   ...
   int sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
   {
   ...
   	err = sk_filter(sk, skb);
   ...
   }

We call the skb_set_owner_r to set the skb to have the found socket as its owner. And are now ready to queue this skb into the sockets receive queue.

int sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
{
        ....
        struct sk_buff_head *list = &sk->sk_receive_queue;
        ....
       	__skb_queue_tail(list, skb);
        ....
}

Thus having reached the underlying socket.

Oh Packet, I waited for you so long.

When the inet subsystem gets initialized apart from initializing all sorts of caches and adding to the inet_protos array various ip protocols. We also initialize the socks subsystem with call to sock_register which ads a protocol handlers for various sockets.

   	(void)sock_register(&inet_family_ops);


static const struct net_proto_family inet_family_ops = {
	.family = PF_INET,
	.create = inet_create,
	.owner	= THIS_MODULE,
};

We might recognize the PF_INET as the protocol family that is used during the socket creation step. If we remember our socket programming one of the first steps in the creation of the socket is the socket system call which can be seen in socket.c. Which will thread down the a call to __sock_create. with all the usual.

SYSCALL_DEFINE3(socket, int, family, int, type, int, protocol)
int __sock_create(struct net *net, int family, int type, int protocol,
			 struct socket **res, int kern)
{
...
	sock = sock_alloc();
        ....
       	pf = rcu_dereference(net_families[family]);

        ...
       	err = pf->create(net, sock, protocol, kern);
}

The protocol family that is passed is an integer referencing the net_families array.This is the very protocol family which we had created and initialized at the inet_init. Thus our pf create method is going to result in calling inet_create and the socket system call.

in af_inet.c we actually see the definition of various protocols of the ip family.

static struct inet_protosw inetsw_array[] =
{
	{
		.type =       SOCK_STREAM,
		.protocol =   IPPROTO_TCP,
		.prot =       &tcp_prot,
		.ops =        &inet_stream_ops,
		.flags =      INET_PROTOSW_PERMANENT |
			      INET_PROTOSW_ICSK,
	},
	{
		.type =       SOCK_DGRAM,
		.protocol =   IPPROTO_UDP,
		.prot =       &udp_prot,
		.ops =        &inet_dgram_ops,
		.flags =      INET_PROTOSW_PERMANENT,
       },
       {
		.type =       SOCK_DGRAM,
		.protocol =   IPPROTO_ICMP,
		.prot =       &ping_prot,
		.ops =        &inet_dgram_ops,
		.flags =      INET_PROTOSW_REUSE,
       },
       {
	       .type =       SOCK_RAW,
	       .protocol =   IPPROTO_IP,	/* wild card */
	       .prot =       &raw_prot,
	       .ops =        &inet_sockraw_ops,
	       .flags =      INET_PROTOSW_REUSE,
       }
};

Each protocol defining its operations to common socket operations. Consider for example SOCK_DGRAM which is the the uses the udp_protocol. As we traverse through the inet_create we find that the struct socket which gets created also gets gets assigned a struct sock. If we remember the struct sock was the structure on to whose sk_receive_queue the final packet ended up. Here we are creating the empty queue on to which our sent and received packets will get placed. Still need to look at why the struct socket as used as a encapsulation layer over struct sock. Aneeways, moving on to our next method i.e bind() if we remember will the bind system call is defined in the kernel as follows

SYSCALL_DEFINE3(bind, int, fd, struct sockaddr __user *, umyaddr, int, addrlen)

// Example usage

   
   serv_addr.sin_family = AF_INET;
   serv_addr.sin_addr.s_addr = INADDR_ANY;
   serv_addr.sin_port = htons(portno);
   
   /* Now bind the host address using bind() call.*/
   if (bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0)

Ok since the fd passed to the bind is a regular file descriptor The first thing we must do is convert this regular file descriptor to a struct socket. To do this we look at the current processes list files just as we would for a regular file. This file descriptor entry is had actually gotten added when we created the socket using socket fs. The struct socket for this file descriptor is tucked away in the file’s private data. As seen here :

struct socket *sock_from_file(struct file *file, int *err)
{
	if (file->f_op == &socket_file_ops)
		return file->private_data;	/* set in sock_map_fd */

	*err = -ENOTSOCK;
	return NULL;
}

We can see the cast to struct socket*. Finally having found the socket we are referring to we end up calling the bind of the underlying socket. as shown here.

SYSCALL_DEFINE3(bind, int, fd, struct sockaddr __user *, umyaddr, int, addrlen)
{
...

				err = sock->ops->bind(sock,
						      (struct sockaddr *)
						      &address, addrlen);
...
}

Ah but then one might ask what does the ops of our AF_INET socket which we created with SOCK_DGRAM point to ? I am just going to guess the ops is inet_dgram_ops. Thus perhaps it will be helpful to look at the inet_dgram_ops.

const struct proto_ops inet_dgram_ops = {
	.family		   = PF_INET,
	.owner		   = THIS_MODULE,
	.release	   = inet_release,
	.bind		   = inet_bind,
	.connect	   = inet_dgram_connect,
	.socketpair	   = sock_no_socketpair,
	.accept		   = sock_no_accept,
	.getname	   = inet_getname,
	.poll		   = udp_poll,
	.ioctl		   = inet_ioctl,
	.listen		   = sock_no_listen,
	.shutdown	   = inet_shutdown,
	.setsockopt	   = sock_common_setsockopt,
	.getsockopt	   = sock_common_getsockopt,
	.sendmsg	   = inet_sendmsg,
	.recvmsg	   = inet_recvmsg,
	.mmap		   = sock_no_mmap,
	.sendpage	   = inet_sendpage,
#ifdef CONFIG_COMPAT
	.compat_setsockopt = compat_sock_common_setsockopt,
	.compat_getsockopt = compat_sock_common_getsockopt,
	.compat_ioctl	   = inet_compat_ioctl,
#endif
};
EXPORT_SYMBOL(inet_dgram_ops);

We see a mapping for the bind method to the generic inet_bind. Which is used both by TCP and UDP. Inside inet_bind we see we get the underlying struct sock and now using the struct sockaddr *uaddr. can set it up with relevant information which will be used later.

int inet_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
{
 .....
      
      	struct inet_sock *inet = inet_sk(sk);
 .....
  inet->inet_rcv_saddr = inet->inet_saddr = addr->sin_addr.s_addr;
  ...
  inet->inet_sport = htons(inet->inet_num);
}

Are you listening to the words coming out of my mouth.

While it seems that the fd can be used by any sort of file descriptor reader. I dont know much about it. Instead I shall look into the recvfrom method. An example usage of which could be

// Example Usage
char buffer[549];
struct sockaddr_storage src_addr;
socklen_t src_addr_len=sizeof(src_addr);
ssize_t count=recvfrom(fd,buffer,sizeof(buffer),0,(struct sockaddr*)&src_addr,&src_addr_len);
if (count==-1) {
    die("%s",strerror(errno));
} else if (count==sizeof(buffer)) {
    warn("datagram too large for buffer: truncated");
} else {
    handle_datagram(buffer,count);
}

// System call in the kernel
SYSCALL_DEFINE6(recvfrom, int, fd, void __user *, ubuf, size_t, size,
		unsigned int, flags, struct sockaddr __user *, addr,
		int __user *, addr_len)
{
....
       	err = sock_recvmsg(sock, &msg, size, flags);
....
}

// Calling underlying socket recvmsg
static inline int __sock_recvmsg_nosec(struct kiocb *iocb, struct socket *sock,
				       struct msghdr *msg, size_t size, int flags)
{
	struct sock_iocb *si = kiocb_to_siocb(iocb);

	si->sock = sock;
	si->scm = NULL;
	si->msg = msg;
	si->size = size;
	si->flags = flags;

	return sock->ops->recvmsg(iocb, sock, msg, size, flags);
}

Now switching over to the recvmsg. We see that we can receive messages by calling inet_recvmsg.

int inet_recvmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg,
		 size_t size, int flags)
{
....
       	err = sk->sk_prot->recvmsg(iocb, sk, msg, size, flags & MSG_DONTWAIT,
			   flags & ~MSG_DONTWAIT, &addr_len);
....
}

Which in the end just calls the udp protocols recvmsg . We can see it define all sorts of method that are common to protocols running over IP.

// udp.c
struct proto udp_prot = {
	.name		   = "UDP",
	.owner		   = THIS_MODULE,
	.close		   = udp_lib_close,
	.connect	   = ip4_datagram_connect,
	.disconnect	   = udp_disconnect,
	.ioctl		   = udp_ioctl,
	.destroy	   = udp_destroy_sock,
	.setsockopt	   = udp_setsockopt,
	.getsockopt	   = udp_getsockopt,
	.sendmsg	   = udp_sendmsg,
	.recvmsg	   = udp_recvmsg,
	.sendpage	   = udp_sendpage,
	.backlog_rcv	   = __udp_queue_rcv_skb,
	.release_cb	   = ip4_datagram_release_cb,
	.hash		   = udp_lib_hash,
	.unhash		   = udp_lib_unhash,
	.rehash		   = udp_v4_rehash,
	.get_port	   = udp_v4_get_port,
	.memory_allocated  = &udp_memory_allocated,
	.sysctl_mem	   = sysctl_udp_mem,
	.sysctl_wmem	   = &sysctl_udp_wmem_min,
	.sysctl_rmem	   = &sysctl_udp_rmem_min,
	.obj_size	   = sizeof(struct udp_sock),
	.slab_flags	   = SLAB_DESTROY_BY_RCU,
	.h.udp_table	   = &udp_table,
#ifdef CONFIG_COMPAT
	.compat_setsockopt = compat_udp_setsockopt,
	.compat_getsockopt = compat_udp_getsockopt,
#endif
	.clear_sk	   = sk_prot_clear_portaddr_nulls,
};
EXPORT_SYMBOL(udp_prot);

Thus the finally the call will find us trickling down to the udp_recvmsg.

int udp_recvmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
		size_t len, int noblock, int flags, int *addr_len)
{
    .....
    	skb = __skb_recv_datagram(sk, flags | (noblock ? MSG_DONTWAIT : 0),
				  &peeked, &off, &err);
    .....
}

We we call here which blocks depending on the options and waits for the the skb in anycase. Of course the receive method for datagram is endlessly flexible in ways which we are currently not interested in. But for now we see a a loop which will wait for and assemble packets to be ready to be served to the user, working with various timeout issues as necessary.

struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags,
				    int *peeked, int *off, int *err)
{
...
	do {
        ...
        ..
        } while (!wait_for_more_packets(sk, err, &timeo, last));
....
}

Where wait_for_more_packets will optionally creates a wait queue on which it can wait until a packet arrives

static int wait_for_more_packets(struct sock *sk, int *err, long *timeo_p,
				 const struct sk_buff *skb)
{

	DEFINE_WAIT_FUNC(wait, receiver_wake_function);
       	prepare_to_wait_exclusive(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
        ...
        ..
       	*timeo_p = schedule_timeout(*timeo_p);
}

Enqueue the task to the sk->sk_wq (I think). periodically waking itself up and checking the queue (I think).

On waking up we walk through the sockets sk_receive_queue piking up first skb and return it

struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags,
				    int *peeked, int *off, int *err)
{
        struct sk_buff_head *queue = &sk->sk_receive_queue;
         ....
 	skb_queue_walk(queue, skb) {
        {
         ....
         __skb_unlink(skb, queue);
         ....
         return skb;
        }
}

Now that we have gotten the skb from the network we need to copy it into the msg for the user to consume. This happens in th skb_copy_and_csum_datagram_msg passed in the message header. If the message is too big it may need to be chunked struct iov_iter *to an iterator of the msg.

int skb_copy_and_csum_datagram_msg(struct sk_buff *skb,
				   int hlen, struct msghdr *msg)
{
...
	if (iov_iter_count(&msg->msg_iter) < chunk) {
		if (__skb_checksum_complete(skb))
			goto csum_error;
		if (skb_copy_datagram_msg(skb, hlen, msg, chunk))
			goto fault;
	}
        ....
}

Where skb_copy_datagram_msg copies as much of the data as is required by the underlying application. Using the appropriate __copy_to_user method

__copy_to_user(v.iov_base, (from += v.iov_len) - v.iov_len,
			       v.iov_len),

Thus finally handing the data to user space.

Summary

The primary reference text which contains a lot of the gory details is the extremely detailed Linux Networking Internals which reads kind of like a bible. And for reference to kernel details there is the equally detailed Understanding Linux Kernel. And then there is always the good google search which invariable lands on to a lwn article. As always any comments or suggestions for improvements are always welcome. Nice tutorial on pci cards on tldp.For a more sane introduction to listening to UDP datagrams see UDP server. Lot of the material was discussed in a linux kernel class I am taking at UCSC whose reference site probably contains more accurate information. Clearly the most amazing thing about this is that all this logic really does get executed many times all over the internet and on the localhost for every packet. I guess I am writing this more as a brain dump of reading through the source in the 3.19-rc7 kernel networking subsystem,thus it may be highly unreliable and inaccurate. Users beware! With that fair warning lets try to begin.

Linux System Calls - Using and Implementing Poll

2014-08-11T00:00:00+00:00

Here we try to describe the linux post system call from both the client and the device driver perspective. Implementing post system call can serve as a useful excercise in clarifying lots of introductory device driver and systems programming concepts.

Client side poll

At a basic level the poll system call allows you to wait on several file descriptors for data to become available allowing you perform non-blocking I/O operations on the active file descriptors. As the client for poll one you need to invoke the following system call.

#include <poll.h>

/**
 * fds - the set of file descriptors to be monitored.
 * nfds - size of the set
 * timeout - in milliseconds specified how long to block waiting for
 * data to become available in any one one of the available
 * descriptors
 */
int poll(struct pollfd *fds, nfds_t nfds, int timeout);

The argument the set of file descriptors to be monitored is specified using the an array of structures of the following form.

struct pollfd {
    int   fd;         /* file descriptor */
    short events;     /* requested events */
    short revents;    /* returned events */
};

The events field is a bit mask of requested events that the application program is interested in. While the revents is a an output parameter filled in by the kernel.

The bits in the events can use predefined definitions in ****. Some commonly used definitions are as follows:

Bit Field   | Description   
------------|---------------
POLLIN      | There is data to read.  
POLLOUT     | Writing now will not block.  
POLLERR     | Error condition (output only).  
POLLHUP     | Hang up (output only).  
POLLNVAL    | Invalid request: fd  not  open  (output only).

Upon success the poll system call will return the number of structures which have non-zero revents. A value of 0 simply indicates that timeout was reached with no file descriptor ready. A value -1 indicates that an errono was set for us.

Example usage of poll

One of the commonly cited reasons for using poll is the example of a server which needs to serve multiple clients when data becomes available. For our somewhat contrived example however we will use named pipes as a way to exercises our client. For a more detailed account on named pipes see here

We can create a named pipe and write to it with the following commands:

 $ mkfifo np
 $ tail -f /var/log/messages > np &
 
 $ mkfifo np2
 $ tail -f /var/log/wpa_supplicant.log > np2 &

Thus the output of tail shall becomes available to be read from at the read end of the named pipe np.

We can test then by it out

$ cat np
... output of tail should show up here...

Finally we can write a client with now given multiple file names can poll them for output as and when data becomes available.

  $ ufollow np np2

 Some output read freom np
  ~~~~~~~~~ Last read  bytes from [np] ~~~~~~~~~
 Some ouput read from np2 
  ~~~~~~~~~ Last read bytes from [np2] ~~~~~~~~~
  
  ... interlace outputing from which ever descriptor has data
  available to be read.

Here ufollow is a client program which uses poll system call to open check the file descriptors corresponding to the file names to read data as and when it becomes available.

   // main loop in ufollow.c
	while(1) {
    
		// wait infinitely for data on file descriptors
		// This only works for character devices,named pipes and sockets 
		// For regular files we need to do use the inotify interface.
		int retval = poll(poll_fds, nfiles,-1);
        
		if(retval == -1 ){
			perror("poll");
            break;
		}        

		for(i = 0; i < nfiles; i++) {
            if(poll_fds[i].revents & POLLIN) {
				print_data(files[i].name,files[i].fd,&config);
            }
            poll_fds[i].revents = 0;
		}
	}

One wrinkle in all of this is that poll system call always returns success for regular files. Although i am yet to try it inotify can be used to convert regular file descriptors into file descriptors which will work with the poll system call. Allowing us to preserve the regularity of the the poll interface.

Driver side view of Blocking I/O and wait queues.

Driver side implementation of poll system call.

Typically a users request for a poll an a file descriptor is going to get translated into a call into the appropriate driver associated with that file descriptor. Imagine for instance a simple device file occurring in /dev directory on the linux file system. Typically such device file inodes are associated with a major number and minor number as seen in from the following invocation of the ls command.

$ ls -al /dev | grep  '1,'

crw-rw-rw-.  1 root root      1,   7 Aug 11 15:40 full
crw-r--r--.  1 root root      1,  11 Aug 11 15:40 kmsg
crw-r-----.  1 root kmem      1,   1 Aug 11 15:40 mem
crw-rw-rw-.  1 root root      1,   3 Aug 11 15:40 null
crw-------.  1 root root      1,  12 Aug 11 15:40 oldmem

Here we see that /dev/null device is associated with the major number 1 and the minor number 3. We also notice that that this is the same driver that is used to read and write the memory directly(as witnessed by the shared major number of 1). The ‘c’ at the beginning specifies that the underlying device files are character devices.

Assuming thus that we have created and associated our driver with a major number and minor number appropriately (See LDD3) for details. We now get to the linux kernels internal architecture for simplifying the implementation of the poll system call.

Drivers intending to support the poll system call must start by first implementing

unsigned int (*poll) (struct file *filp, poll_table *wait);

void poll_wait (struct file *, wait_queue_head_t *, poll_table *);

Summary

The primary reference text that I have been using which is tremendously helpful in understanding the implementation of the poll and select interface is the Linux Device Driver Third Edition by the dynamic trio Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman. Available for free here and at Amazon. While the kernel has gone through many modifications since 2.6.10 many parts of the book are still quite relevant and readable. As always any comments or suggestions for improvements are always welcome.

Aakarsh Nair

Emacs/Evil-mode - A basic reference to using evil mode in Emacs.

Introduction

Modes and States

Additional plugins

Key chord mode

Vim Basics

Motion Commands

Character Level Motion :

Screen Level Motions :

Line Level Motions:

Line forward and backwards

Sentence and Paragraph Motions

Search and Replace:

Marks

Macro recordings

Why bother?

References

Notes on synchronization in the Linux Kernel

Atomic Operations

Spin Locks

Semaphores

Reader-Writer Semaphores

Mutexes

Completion Variables

BKL: The big kernel lock

Sequential Locking

Preemption Disabling

Ordering and Barriers

Summary

Notes on memory management in the Linux Kernel

Why is memory allocation in the kernel hard ?

Pages

Zones

Getting Pages

kmalloc()

vmalloc()

Slab Layer

Statically Allocating on the Stack

High Memory Pages

Per CPU Allocation

The percpu Interface

Reasons for using Per-CPU Data

Summary/Guidelines for picking an allocation methods

Programming Language Reference - EmacsLisp (draft)

The Emacs Lisp

Lisp Data Types

Printed Representation and Read Syntax

Comments

Programmer Types

Integer Types

Floating Point Type

Character Type

Symbol Type

Sequence Type

Cons Cell Type

Array Type

String Type

Vector Type

Char-Table Type

Bool-Vector Type

Hash Table Type

Function Type

Macro Type

Primitive Function Type

Byte-Code Type

Autoload Type

Programmer Types

Marker Type

Window Type

Frame Type

Terminal Type

Window Configuration Type

Frame Configuration Type

Process Type

Stream Type

Keymap Type

Overlay Type

Font Type

Numbers

`kmalloc()`

`vmalloc()`

The `percpu` Interface