lua luaipc

Portable inter-process communication for Lua

LuaIPC – Inter-Process Communication for Lua

Introduction

Since Lua is mostly written in portable ISO C it comes with the bare
minimum of batteries. There are some extension libraries that provide
a portable (usually lowest common denominator) interface to some parts
of the operating system not covered by ISO C, like sockets
(LuaSocket), or filesystem operations (LuaFileSystem).
LuaIPC is such a library consisting of multiple modules for
inter-process communication in Lua. It allows you to:

create/read/write shared memory
create/manipulate semaphores
memory-map files
lock/unlock (parts of) files
spawn subprocesses and communicate with them using unnamed pipes

The implementation should be portable to Lua 5.1-5.3 on recent Windows
and POSIX machines (tested on Win 7, Ubuntu Linux, FreeBSD, and OSX).

Reference

In general all functions return a true-ish value on success (true if
there is nothing better to return). This applies only to functions
that do not aim for compatibility with existing Lua functions (like
e.g. the file methods). On error, usually nil, an error message,
and a numeric error code are returned. All handles created by this
library are finalized during garbage-collection, so manually cleaning
up is optional under most circumstances.

The LuaIPC library consists of the following modules:

ipc.shm

local shm = require( "ipc.shm" )

A shared memory segment is a piece of RAM that can be accessed by
multiple processes at the same time. Since Lua does not have any form
of pointer arithmetic, a file-like interface is provided instead.
The shared memory segment is deleted automatically when all its
handles are closed. It is unspecified whether you can still open a
new handle to an existing shared memory segment once its original
creator has closed the handle. On some OSes this module might allocate
more than the requested number of bytes (usually a multiple of the
page size). To be portable you have to agree on the real size (e.g.
by storing it at the beginning of the shared memory segment) and
adjust the file size that this module uses by calling h:truncate()
for all parties that attached to the shared memory (see below).

The module provides the following functions:

shm.create( name, size ) ==> handle
shm.attach( name ) ==> handle

shm.create() creates a new shared memory segment with the given
name and size (in bytes), and returns an attached handle to it. If
a memory segment with that name already exists, this function fails.
shm.attach() on the other hand tries to open an existing shared
memory segment with the given name. In both cases name is not a
file path: you cannot find it on the filesystem (well, on POSIX you
can), and it must not contain any directory separators!

A shared memory handle has all the methods that a normal Lua file
handle has, and additionally:

h:size() ==> bytes
h:addr() ==> pointer
h:truncate( bytes ) ==> true

h:size() returns the size of the shared memory segment in bytes, and
h:addr() returns the starting address as a lightuserdata. Unless you
use some form of FFI, this is probably not very useful for you. The
h:truncate() method can tell this module to use less than the
current shared memory size for its file-like interface.

The file-like methods behave exactly like their Lua file equivalents
with the following exception: Currently the "*n" format specifier of
read/lines is not supported. Since handles are not actually Lua
file objects, io.type() will return nil when called with such a
handle.

ipc.sem

local sem = require( "ipc.sem" )

A semaphore is a shared integer counter that you can atomically
increment or decrement, but its value will never go below zero. If a
decrement operation would cause the value to become negative, the call
blocks (and/or returns false for the non-blocking/waiting calls).
The semaphore is deleted automatically when all its handles are
closed. It is unspecified whether you can still open a new handle to
an existing semaphore once its original creator has closed the handle.

The module provides the following function:

sem.open( name [, n] ) ==> handle

If the initial counter value n is given (and not 0), sem.open
tries to create a new semaphore with the given name. If such a
semaphore already exists, this function fails. If n is zero or
absent, sem.open tries to open an existing semaphore with the given
name. Anyways, name is not a file path: you cannot find it on the
filesystem (well, on POSIX you can), and it must not contain any
directory separators!

A semaphore handle has the following methods:

h:inc() ==> true
h:dec( [timeout] ) ==> boolean
h:close() ==> true

h:inc() increments the semaphore value (if successful). h:dec()
tries to decrement the semaphore value. If the optional timeout (in
seconds, can have a fractional part) is absent or negative, the
function will block until some other process increments the semaphore.
Otherwise h:dec() will wait at most timeout seconds and return
true or false depending on the success (errors still result in
nil, error message, and error code).

ipc.mmap

local mmap = require( "ipc.mmap" )

Memory-mapped I/O is usually faster than normal file I/O, because by
making the kernel buffers available at a certain memory address, you
can save the copy operations to user-supplied buffers. Memory-mapping
also has some draw-backs: Most OSes don’t allow mapping a zero-length
file or very large files (e.g. larger than the address space). Also,
this module does not allow resizing the file while it is mapped, so
you can’t write beyond the initial bounds of the memory-mapped file.

The module provides the following function:

mmap.pagesize
mmap.open( filepath [, mode [, offset [, size]]] ) ==> handle

mmap.open() opens the given filepath and maps the contents into
memory. mode can be "r" (the default), "w", or "rw". On
success a mmap handle is returned. Offsets usually must be given in
multiples of the current pages size (or equivalent). mmap.pagesize
contains this number. Default value for size is 0 which uses the
current size of the file.

An mmap handle has all the methods that a normal Lua file handle has,
and additionally:

h:size() ==> bytes
h:addr() ==> pointer
h:truncate( bytes ) ==> true

h:size() returns the size of the memory map in bytes, and h:addr()
returns the starting address as a lightuserdata. Unless you use some
form of FFI, this is probably not very useful for you. h:truncate()
can shrink the size of the memory mapping, but it’s mostly useful for
the ipc.shm module (see there).

ipc.filelock

local filelock = require( "ipc.filelock" )

This module contains functions for (un-)locking byte ranges of an
opened Lua file. The locks might be advisory (you have to check for
locks yourself) or mandatory (file functions will honor the locks
automatically) depending on the OS. Also, the file locking functions
can’t be used to synchonize different threads in the same process.
There is similar functionality in LuaFileSystem, but in LFS all
locking is non-blocking. It is explicitly not supported to mix the
locking functions from LFS and from this module.

This module provides the following functions:

filelock.lock( file, mode [, offset [, nbytes]] ) ==> true
filelock.trylock( file, mode [, offset [, nbytes]] ) ==> boolean
filelock.unlock( file [, offset [, nbytes]] ) ==> true

filelock.lock() acquires a lock for the given range of nbytes
bytes starting at offset (default 0) in the Lua file object. If
nbytes is absent (or 0), the whole file is locked. If the given
range already is locked by another process, filelock.lock() will
block until the other process releases that lock. filelock.trylock()
is similar, but it will not block. Instead it returns false if another
process already holds a lock on the given byte range.
filelock.unlock unlocks the given byte range that has been locked by
the same process before.

ipc.proc

local proc = require( "ipc.proc" )

On the most common OSes Lua provides the io.popen() function to
spawn a subprocess and capture its output or provide its input via
unnamed pipes. This module can capture stdout and stderr
simultaneously, while still providing input for the spawned command.
To avoid deadlocks as much as possible and to work around platform
differences the interface is callback-based.

This module provides the following functions/fields:

proc.spawn( cmd, options ) ==> handle
proc.EOF ==> lightuserdata

Similar to os.execute() and io.popen(), proc.spawn() takes a
command (cmd) as string and runs it using the shell. The options
table specifies the callback function (field callback), and which
streams to redirect via pipes (fields stdin, stdout, and stderr;
a true value creates a pipe, a false value uses the default streams of
the parent process – you can also specify Lua file objects).

A process handle has the following methods:

h:write( ... ) ==> true
h:kill( "term"/"kill" ) ==> true
h:wait() ==> true/nil, string, number

h:write() accepts strings and enqueues them to be sent to the child
process’ stdin stream when it is ready. You may also pass proc.EOF
to close the stdin stream when all enqueued output has been sent.
The h:kill() function sends either a SIGTERM (requesting graceful
shutdown) or a SIGKILL (for immediate shutdown) to the child
process. h:wait() starts a small blocking event loop that will send
enqueued data to the child process’ stdin, read data from the
child’s stdout/stderr streams, and/or wait for the child process
to exit. The callback given to the proc.spawn() function is called
with the stream name ("stdout" or "stdin") and the received data
(or the usual error values) when output from the child is available.
On Lua 5.2+ you may yield from the callback function. Return values of
h:wait() are the same as for os.execute() on recent Lua versions.

ipc.strfile

local strfile = require( "ipc.strfile" )

This module is not about IPC at all. It allows you to create a
file-like object from a Lua string. It shares most of the code for
file handling with the ipc.shm and ipc.mmap modules, and it is
often useful – that’s why it is here. It has exactly the same methods
as the shared memory or mmap handles (except you are not allowed to
write).

The following function is provided:

strfile.open( str ) ==> handle

Contact

Philipp Janda, siffiejoe(a)gmx.net

Comments and feedback are always welcome.

License

LuaIPC is copyrighted free software distributed under the MIT
license (the same license as Lua 5.1). The full license text follows:

LuaIPC (c) 2015, 2016 Philipp Janda

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHOR OR COPYRIGHT HOLDER BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The sem_timedwait() implementation used on OSX was written by Keith
Shortridge at the Australian Astronomical Observatory. See the
comments in osx/sem_timedwait.c for details.