opensmalltalk vm

Cross-platform virtual machine for Squeak, Pharo, Cuis, and Newspeak.

563

111

The Cog VM source tree

This is the README for the Cog Git source tree:
https://github.com/OpenSmalltalk/opensmalltalk-vm

Builds are tested automatically on each commit, for Windows, macOS, and Linux.
Windows and Linux also have 32-bit targets. Squeak and Pharo VMs are built with
and without JIT, with and without Spur, and with and without Sista, as available
per platform. All build artifacts remain accessible for 90 days for each
workflow run; the latest artifacts are uploaded to a dedicated release tag
(e.g., “latest-build”), overwritten with each successful run. If you wish to
commit without triggering a build, for example if you were to only edit this
documentation, then if you add [ci skip] somewhere in your commit message,
no workflow run will be started for that commit.

Important notice for Developers:

We rely on source file substitutions in the src tree, specifically,
any sq*SCCSVersion.h files anywhere in the tree are processed to replace
$Rev$ , $Date$ , and $URL$ with the current revision (defined as the
timestamp %Y%m%d%H%M of the latest commit), the human readable date of that
commit, and the url of the origin repository these sources were cloned from.

The first time you clone this repository, you must therefore run this command:

./scripts/updateSCCSVersions

This will install filters, post-commit, and post-merge hooks to update the
sq*SCCSVersion.h files anytime you commit or merge.

For easier use, we include the scripts/gitci and scripts/gitciplugins scripts to
commit changes to this branch and changes to the Cross and win32 plugins (which
are shared with the old Squeak interpreter trunk). If you decide not to use
these scripts for checking in sources, you should also run the
updateSCCSVersions script anytime you decide to use git-reset or
git-checkout to make sure the stamps are updated correctly. Failing to do so
will result in incorrect version stamps in your compiled VMs.

Overview

First, opensmalltalk-vm (a.k.a. the Cog VM) is the virtual machine beneath the
Cuis, Pharo and Squeak Smalltalk dialects. For issues related to these systems
that are unrelated to the VM itself, please use their forums:

Second, the core VM, which comprises the execution engine and garbage collector,
and the core plugins, is developed in Smalltalk, using the VM Simulator. This
repository contains the code generated by the Simulator, and the platform support
code for the entire VM, its CI infrastructure and so on. The core VM should
not be developed by editing the generated code. The core VM should be
developed using Smalltalk. The source code repository for the VM is
http://source.squeak.org/VMMaker.html.
You can find scripts to build a Smalltalk image in which to do core VM development
in the image
directory in this repository. You can read about the Simulator here:

https://www.researchgate.net/publication/328509577_Two_Decades_of_Smalltalk_VM_Development_Live_VM_Development_through_Simulation_Tools

Cog is an evolution of the Squeak Back-to-the-future Smalltalk virtual machine
that provides a number of different Smalltalk virtual machines. The VMs are
developed in Smalltalk, using all the dynamic and reflective facilities of the
Squeak Smalltalk system. As such, developing in Cog is a delight. The
Smalltalk framework comprising the various Cog VMs is translated into C by its
Slang component to produce VM source that is combined with platform-specific
support sources and compiled via a C compiler to obtain a fast production VM.
This directory tree includes the output of Slang for various configurations of
“Cog VM” along with the associated platform support code, plus build directories that
can be used to produce production VMs.

This directory tree also includes a directory containing scripts that can be used to create an instance of the Smalltalk Cog development system, suitable for developing the VM in Smalltalk, and for generating new VM sources. See the image subdirectory, described below.

Variants

The “Cog VM” comes in a bewildering variety of forms. The first distinction
is between Stack, Cog and Sista VMs. Stack VMs are those with context-to-stack
mapping that optimise message sending by keeping method activations on a stack
instead of in contexts. These are pure interpreters but are significantly
faster than the standard context-based Interpreter VM. Cog VMs add a JIT to
the mix, compiling methods used more than once to machine code on the fly.
Sista VMs, as yet unrealised and in development, add support for adaptive
optimization that does speculative inlining at the bytecode-to-bytecode level.

Another distinction is between “v3” VMs and Spur VMs. “v3” is the original
object representation for Squeak as described in the back-to-the-future paper.
Spur, as described on the www.mirandabanda.org blog, and in the paper “A partial read barrier for efficient support of live object-oriented programming” by Miranda & B�ra, is a faster object
representation which uses generation scavenging, lazy forwarding for fast
become, a single object header format common to 32 and 64 bit versions, and a
segmented heap that can grow and shrink, releasing memory back to the host OS.
Squeak 5.0, Cuis 5 and Pharo 5 and subsequent releases use Spur.

Another distinction is between normal single-threaded VMs that schedule “green”
Smalltalk light-weight processes above a single-threaded VM, and multi-threaded
VMs that share the VM between any number of native threads such that only one
native thread owns the VM at any one time, switching between threads on FFI
calls and callbacks or on Smalltalk process switches when Smalltalk processes
are owned by threads. This architecture offers non-blocking FFI calls and
interoperability with multiple native threads, but does /not/ provide true
concurrency. This multi-threaded support is as yet experimental.

VM source directories

The Slang output of the various VMs are kept in “vm source” directories. These
C sources define the core VM (the Smalltalk execution engine and the memory
manager), and a substantial set of “plugins” that provide interfaces to various
external facilities via Smalltalk primitive methods. Each vm source directory
is specific to a particular VM, be it Squeak Cog Spur, or V3 Stack, etc.
The plugins can be shared between VMs, choosing the set of plugins to include
in a VM at build time (see plugins.int & plugins.ext in build directories).

The VM source are in directories such as

	src/v3.sista		- Smalltalk Sista V3
	src/spur32.sista	- Smalltalk Sista Spur
	src/spur32.cog 		- Smalltalk Cog Spur
	src/spur64.cog 		- Smalltalk Cog Spur 64-bit
	src/spur32.stack 	- Smalltalk Stack Spur
	src/spur64.stack 	- Smalltalk Stack Spur 64-bit
	src/v3.cog			- Smalltalk Cog V3
	src/v3.stack 		- Smalltalk Stack V3

All plugins are in the directory

	src/plugins

These contain many, but not all, of the plugins available for the VM. Others
can be found in Cog, or in various Monticello packages in various repositories.

Each vm source directory contains several files, a subset of the following:

	cogit.c				- the JIT; a Cogit cooperates with a CoInterpreter.
                          This simply includes a processor-specific JIT file
	cogitIA32.c et al   - relevant processor-specific JIT, selected by cogit.c
	cogit.h				- the Cogit's API, as used by the CoInterpreter
	cogmethod.h			- the structure of a CogMethod, the output of the Cogit
	cointerp.c			- the CoInterpreter's source file
	cointerp.h			- the API of the CoInterpreter, as used by the Cogit
	cointerpmt.c		- the multi-threaded CoInterpreterMT's source file
	cointerpmt.h		- the API of the CoInterpreterMT, as used by the Cogit
	gcc3x-cointerp.c	- cointerp.c massaged to interpret faster if using gcc
	gcc3x-cointerpmt.c	- ditto for cointerpmt.c
	gcc3x-interp.c		- ditto for interp.c
	interp.c			- the StackInterpreter's source file
	interp.h			- defines for the VM configuration, word size, etc
	vmCallback.h		- the structure of the VM's VMCallbackContext

Platform build directories

The current “official” build directories are of the form
building/OS_WordSize_Processor, and include

	building/linux32x86	- uses autoconf, gcc and make
	building/macos32x86	- 32-bit Mac OS X using clang and gmake
	building/macos64x64	- 64-bit Mac OS X using clang and gmake
	building/win32x86		- uses cygwin, gcc and gmake

More can be added as required. In each there is a HowToBuild that describes
the necessary steps to compile a VM.

Within each building/OS_WordSize_Processor directory are a set of build directories
for specific configurations of Cog, and for support code and makefiles. For
example, there exist

	building/macos32x86/squeak.cog.spur   - A Cog JIT VM with Squeak branding,
                                         using the Spur memory manager.
	building/macos32x86/squeak.stack.spur - A Stack interpreter VM with Squeak
                                         branding, and the Spur memory manager.
	building/macos32x86/squeak.cog.v3     - A Cog JIT VM with Squeak branding,
                                         using the old Squeak memory manager.
	building/macos32x86/pharo.cog.spur    - A Cog JIT VM with Pharo branding and
                                         plugins (not yet implemented) using the
                                         Spur memory manager.

etc.

There exist

    building/macos64x64/bochsx86 - Support libraries for the BochsIA32Plugin which
                                is used to develop Cog itself.
    building/macos64x64/bochsx64 - Support libraries for the BochsX64Plugin which
                                is used to develop Cog itself.
    building/macos64x64/gdbarm32 - Support libraries for the GdbARMPlugin which
                                is used to develop Cog itself.
    building/macos64x64/gdbarm64 - Support libraries for the GdbARMv8Plugin which
                                is used to develop Cog itself.

and the intention is to add such directories to contain e.g. support code for
the Pharo Cairo and Freetype plugins, and anything else needed. By placing
support directories in each build directory they can be shared between various
branded VM builds, avoiding duplication.

There exist

	building/macos32x86/common - Gnu Makefiles for building the various branded VMs
	building/macos64x64/common - Gnu Makefiles for building the various branded VMs
	building/win32x86/common   - Gnu Makefiles for building the various branded VMs
	building/win64x64/common   - Gnu Makefiles for building the various branded VMs

And the intention is to add building/linuxNN???/common as soon as possible to
use Gnu Makefiles to build all VMs on all platforms.

The scripts directory contains various scripts for validating and checking-in
generated sources, packaging builds into installable artifacts (tar, msi, zip),
and so on. The linux builds and the packaging scripts produce outputs in the
products directory tree. The packaging scripts may choose to include Smalltalk
source files included in the sources directory.

Other directories

The platforms directory contains the associated platform-specific files that
combine with the Slang-generated source to produce complete VMs. The structure
is

	platforms/Cross/vm
	platforms/Cross/plugins
	platforms/iOS/vm
	platforms/iOS/plugins
	platforms/unix/vm*
	platforms/unix/plugins
	platforms/win32/vm
	platforms/win32/plugins

Each vm directory contains support for the core VM. Each plugin directory
contains run-time and build-time support for various plugins.

The processors directory contains the source for various processor simulators.
The JIT is developed in Smalltalk by using one of these processor simulators
to execute the code the JIT produces. Currently x86 & x86-64 are derived from
Bochs, and ARMv6/v7 & ARMv8 are derived from gdb.

Customization of builds may be done in two main ways, by adapting an existing
build directory, and by using facilities designed to be applied after the fact
to a preexisting build. Such facilities live in the top-level deploy directory
and are documented there; e.g. see deploy/packaging/README.win32.

Finally the image directory contains scripts that will build a “VMMaker” image,
a Squeak Smalltalk image containing all the packages that comprise the Cog
system, suitable for developing the VM and for generating (or updating) the
sources in the vm source directories. There is also a script for generating a
64-bit Spur image from a 32-bit Spur image, and the VMMaker.oscog package includes
code for converting Spur images in either direction. Note that JIT development depends on processor simulations which are included under the top-level processors directory, Bochs providing x86 and x86-64, gdb providing ARMv6 and ARMv8. To do JIT development the processor plugins must be built. These will be built automatically if the simulator libraries are built. In several building top-level subdirectory are four directories, bochsx86, bochsx64, gdbarm32 & gdbarm64, which contain the build scripts. To build a VM suitable for full JIT simulation, first build the support code in the building subdirectory for your platform, and then build a squeak.cog.spur VM in the sibling directory.

Enjoy, and give forward.