MirageOS

From the OCaml Labs wiki
Jump to: navigation, search

Overview[edit]

The cloud has so far mostly been used to consolidate existing operating systems and manage them more conveniently. The interface between guest kernels, applications and VMs are all managed very differently, leading to serious inefficiencies, unreliability and insecurity. MirageOS revisits the library OS concept (built here as Nemesis in the late '90s), and narrows the gap between safe, high-level programming, and low-level systems construction.

Applications are written in high-level OCaml and compiled directly into microkernels that run on the Xen hypervisor. By treating the hypervisor as a stable hardware platform, we can focus on high-performance protocol implementations without worrying about having to support the thousands of device drivers found in a traditional OS. Although Mirage initially targets the Xen hypervisor, other backends, such as a FreeBSD kernel module and Javascript, also exist.

Mirage includes clean-slate functional implementations of protocols ranging from TCP/IP, DNS, SSH, Openflow (switch/controller), HTTP, XMPP and Xen inter-VM transports. MirageOS has grown as a project, and along with OCaml Labs there are many individual contributors to the ecosystem.

Overview of MirageOS

Related Pages[edit]

Recent and Ongoing Projects[edit]

MirageOS consists of numerous libraries that are independently developed and released by a wide community.

Mirage Dashboard[edit]

A dashboard displaying useful data from MirageOS project and its related repositories.

Solo5[edit]

Solo5 is a kernel library used to build unikernels. This version of the Mirage toolstack can build Mirage unikernels that run directly on KVM/QEMU instead of Xen.

Canopy[edit]

Canopy is a git-blogging unikernel that provides a simple blog platform requiring only a Git URL. It has full HTTP and TLS support, is written in OCaml and uses MirageOS and Irmin.

Functoria[edit]

A staged DSL to organize functor applications. The Functoria DSL allows users to describe how to create portable and flexible applications. It allows to pass application parameters easily using command-line arguments either at configure-time or at runtime.

Irmin[edit]

Irmin is a distributed filesystem and block store that follows the same design principles as Git. The design consists of three main components:

  • a low-level immutable and consistent key/value data-store
  • a DAG persisted in that datastore
  • a tag store which associate names to keys of the low-level data-store.

Irmin is written in pure OCaml, and can thus be compiled to a variety of backends (including Javascript, and Mirage microkernels). Unlike the git frontend, applications can directly iterate over the object graph.

The immutability of the low-level block store makes it significantly easier to apply replication and network coding techniques to improve resilience via replication, and to optimise scheduling across many hosts.

OCaml TLS[edit]

Transport Layer Security (TLS) is probably the most widely deployed security protocol on the Internet. It is used to setup virtual private networks, secure various services such as web and email, etc. For MirageOS, we developed a clean slate TLS implementation developed in OCaml. Our motivating goals are reliability, robustness, and API conciseness.

Home Router[edit]

Jan 2015 - Present | GitHub

Now that Mirage 2.0 has support for the ARMv7 platform and the Jitsu virtual machine toolstack, we are building the missing libraries needed to turn a Cubieboard2 into a home router. This includes NAT, uPnP and HTTP proxies built as unikernels.

LVM-compatible block device[edit]

Oct 2014 - Present | GitHub

A logical volume manager allows you to group together multiple physical disks (volumes) and treat them as a pool of disk blocks, from which you can allocate multiple logical disks (volumes).

This pure OCaml implementation uses the same on-disk format as Linux LVM with the following features:

  • constant-time volume manipulation
  • linear-mapped logical volumes

It provides an mlvm command-line interface to use the library under Unix.

Mirage Releases[edit]

Mirage 2.9 Apr 2016 | Announcement | GitHub

The MirageOS 2.9 release includes the mirage command-line tool, which improves the tooling around logging - it's been a long time coming!

  • Add logging support. A new `reporter` parameter to `register` is now available. This parameter defines how to configure the log reporter, using `Logs` and `Mirage_logs`. Log reporters can also be configured at configuration AND runtime using on the new `-l` or `—logs` command-line argument. (#534, by @samoht, @talex5 and @Drup)
  • Allow to disable command-line parsing at runtime. There is a new `argv` parameter to the `register` function to allow to pass custom command-line argument parsing devices. Use `register ~argv:no_argv` to disable command-line argument parsing. (#493, by @samoht and @Drup)


Mirage 2.8 Apr 2016 | Announcement | GitHub | OPAM

The MirageOS 2.8 release was Mindy's first ever release! It included:

  • Define an ICMP and ICMPV4 module type. ICMPV4 is included in, and surfaced by, the STACKV4 module type. The previous default behavior of the IPv4 module with respect to ICMP is preserved by STACKV4 and the tcpip_stack_direct function provided by mirage. (#523, by @yomimono)
  • Explicitly require OCaml compiler version 4.02.3 in opam files for mirage-types and mirage.


Mirage 2.5 Jun 2015 | Announcement | Release

The MirageOS 2.5 release includes first-class support for SSL/TLS. Since the pure OCaml implementation of TLS last summer, we have been working to improve the integration and create a robust framework. You can now easily build and deploy secure unikernel services and we've incorporated numerous bug-fixes and major stability improvements in the network stack.


Mirage 2.0 Nov 2013 - Aug 2014 | GitHub | Announcement

The MirageOS 2.0 release included a number of new features:

  • ARM device support
  • Irmin distributed, branchable storage
  • OCaml TLS
  • Modular communication

There is more detail of all of these activities on the Mirage 2.0 announcement.


Mirage 1.0 Jun 2013 - Nov 2013 | GitHub | Announcement

The first release of Mirage 1.0 has the goal of self-hosting infrastructure running on Xen (ideally on Amazon EC2 or Rackspace), and a set of tools to make it easy to extend the system with new backends and device drivers. See the announcement blog post for more.


Mirage 0.9 Jan 2013 - Jul 2013 | GitHub

The Mirage developer preview aims to be a stable release of the Mirage tools and libraries in OPAM, and a simple (tested) "Getting started guide" on the wiki.

The three target applications are to build and run:

  • a self-hosting website and DNS server (without SSL)
  • stub-domains for xenstore and xenconsoled, demonstrating how mirage can be used to build critical infrastructure
  • simple distributed system using multiple communicating Mirage instances

Previous and Completed Projects[edit]

Charrua[edit]

Charrua - a DCHP core library implementation in OCaml. It provides basically two modules, a DHCP responsible for parsing and constructing DHCP messages and a Dhcp_server module used for constructing DHCP servers.

  • charrua-unix is a Unix DHCP server based on charrua-core.
  • mirage is a Mirage DHCP unikernel server based on charrua-core.

The name charrua is a reference to the, now extinct, semi-nomadic people of southern South America. This project became one of the Mirage Pioneer Projects.

Jitsu Toolstack[edit]

May 2014 - May 2015 | GitHub

Network latency is a problem for all cloud services. It can be mitigated by moving computation out of remote datacenters by rapidly instantiating local services near the user. This requires an embedded cloud platform on which to deploy multiple applications securely and quickly.

Jitsu is a new Xen toolstack that satisfies the demands of secure multi-tenant isolation on resource-constrained embedded ARM devices. It does this by using unikernels: lightweight, compact, single address space, memory-safe virtual machines (VMs) written in a high-level language. Using fast shared memory channels, Jitsu provides a directory service that launches unikernels in response to network traffic and masks boot latency.

The USENIX NSDI paper evaluation shows Jitsu to be a power-efficient and responsive platform for hosting cloud services in the edge network while preserving the strong isolation guarantees of a type-1 hypervisor.

Moana ML[edit]

Aug 2014 - Apr 2015 | GitHub | Summary Blog

Moana, an information-centric middleware for distributed services and applications. Moana offers a shared graph-based storage abstraction through which applications communicate with each other by appending and observing the shared graph.

Moana supports two basic operations of ADD and MAP; add allows an application to persistently extend the global graph, and map provides a dynamically maintained mapping of the global graph to an application specific, internal sub-graph. MoanaML is a pure OCaml implementation of this abstraction to make it suitable for use in MirageOS unikernels.

Polyversal TCP prototype[edit]

Oct 2012 - Feb 2015 | GitHub

Over the last decade TCP has become the de facto "narrow waist" of the Internet; a one-size-fit-all transport that is poorly suited to the needs of modern applications. As middle-boxes have become ubiquitous, it has become nigh impossible for alternative transports to exist, and so application developers have come to view opening a TCP socket as the only reliable way to connect to a server. Some recent proposals circumvent this problem by camouflaging new transports so that they appear like TCP to middleboxes.

Polyversal TCP aims to unify many different strands of TCP to make it suitable to work as an extensible one-size-fits-all transport that can take advantage of heterogenous interconnects such as shared memory or virtual machine channels. This will in turn greatly simplify the construction of complex distributed systems being built elsewhere in OCaml Labs.

Xen/ARMv7 port[edit]

Feb 2014 - Sep 2014 | GitHub | Blog

The ARMv7 architecture introduced the (optional) Virtualization Extensions, providing hardware support for running virtual machines on ARM devices, and Xen's ARM Hypervisor uses this to support hardware accelerated ARM guests.

Mini-OS is a tiny OS kernel designed specifically for running under Xen. It provides code to initialise the CPU, display messages on the console, allocate memory (malloc), and not much else. It is used as the low-level core of Mirage's Xen implementation.

Mirage v1 was built on an old version of Mini-OS which didn't support ARM. For Mirage v2, we have added ARM support to the current Mini-OS (completing Karim Allah Ahmed's initial ARM port) and made Mirage depend on it as an external library. This means that Mirage will automatically gain support for other architectures that get added later. We are currently working with the Xen developers to get our Mini-OS fork upstreamed into Xen 4.5 and 4.6.

TCP/IP Stack[edit]

Apr 2013 - Mar 2014

The TCP/IP stack in Mirage 1.0 was functional, but lacked support for modern TCP extensions. We have now extended the TCP stack to include more congestion controllers, better ACK handling, a more compete state machine, and extensions such as window scaling. These all contribute to making Mirage 2.0 unikernels more robust Internet servers.

kFreeBSD[edit]

Jun 2013 - Dec 2013 | GitHub

This tiny project puts Mirage to the FreeBSD kernel space for evaluation. Also includes DTrace for tracing, profiling and debugging purposes.

Graphing[edit]

Apr 2013 - Aug 2013 | GG library | VG library

Vg is a declarative 2D vector graphics library for OCaml. It's a baseline library with multiple output backends like PDF, SVG or the HTML5 Canvas element. This project aims to build a "Tufte principled" 2D visualization library with some simple graphing widgets for data visualisation. In the longer term, the intention is to build a declarative UI framework also.

Mirari[edit]

Feb 2013 - Jun 2013 | GitHub | Blog

The Mirari prototype has been superseded by the Mirage 2.0 release. This is here for historical reasons.

Mirari takes a single configuration file and splits the lifecycle of a Mirage application into three distinct segments:

  • configuration: it scans the config file, checks for any missing OPAM dependencies, and installs them if missing. It also looks for any filesystem directives and calls mir-crunch to generate the static ML files. All of this is then glued together into an autogenerated main.ml which is the entry point for the application
  • build: it runs the OCaml build (via Vincent's obuild), and then issues any backend-specific commands (such as the specific link script for the Xen backend, or whatever else we dream up). For convenience, a symlink pointing to the compiled binary will be created in the current working directory
  • run: this is being implemented. Running a Mirage application is quite stateful: in the case of Xen, we want to monitor the kernel, attach a console, and so on. Similarly for UNIX, one can imagine the socket version opening up a control channel for Mirari to pass additional configuration directives, such as network interfaces. In the kFreeBSD backend, this would be done via ioctls or other kernel/user interfaces.

Nigori[edit]

Oct 2012 - Jun 2013 | Github | Homepage

Computer users today have a smartphone, a tablet, a laptop and a desktop machine. Consequently, many new computer applications seamlessly synchronise user data between devices using cloud storage as a highly-available intermediary. Whilst the communication link between the user device and cloud storage is often encrypted, user data is typically stored in a form which is readable by the cloud provider and the application developer.

The aim of the Nigori project is to develop a practical, application-neutral, mechanism for storing sensitive user data in the cloud in such a way that the cloud provider and application developer cannot read any of the stored information. There is an initial specification, and an implementation of Nigori for Java and Android. The next effort is to build a JavaScript version suitable for use as a plug-in for Web browsers. One interesting approach to building a Javascript version is to build it in OCaml using Cryptokit and js_of_ocaml. This lets the same functional source code be compiled into fast native code that runs on some mobile devices (e.g. iOS), and also to slower but more portable Javascript. The code could also have aspects of it formally verified for correctness, for example using the Coq theorem prover.

Nigori consists of two components: a datastore and a client library. A Nigori datastore is a service, either run locally on the device alongside the application, or run remotely in the cloud. The client library forms part of the application and runs on a user's device, encrypts data, and manages the user's datastores. A typical application deployment will contain one datastore on each user device and one datastore in the cloud; the application can then use Nigori to keep datastores, and therefore user data, synchronised across all their devices.

Signpost Prototype[edit]

Aug 2012 - Jun 2013

The defacto architecture of today's Internet services all but removes users' ability to establish inter-device connectivity except through centrally controlled "cloud" services. Whilst undeniably convenient, the centralised data silos of the cloud remain opaque and an attractive target for attackers. A range of mechanisms exist for establishing secure peer-to-peer connections, but are inaccessible to most users due to the intricacy of their network configuration assumptions. Users effectively give up security, privacy and (when peers are both on the same LAN) low-latency simply to get something useable. We observe that existing Internet technologies suffice to support efficient, secure and decentralized communication between users, even in the face of the extreme diversity of edge connectivity and middlebox intervention.

Signpost is a Mirage-compatible system that explicitly represents individual users in a network-wide architecture. Signpost DNS servers create a "personal CDN" for individuals, securely orchestrating the many different available techniques for establishing device-to-device connectivity to automatically select the most appropriate. A DNS API gives application compatibility, and DNSSEC and DNSCurve bootstraps secure connectivity.

See the USENIX FOCI 2013 paper for more details.

XEN.org Incubation[edit]

Dec 2012 - May 2013 | Proposal

The Mirage project is being incubated as an official xen.org project. This will give us a neutral community ground to continue long-term development of the libraries and tools required to turn it into a robust, production-grade exokernel.

To date we have:

The next stage is to continue working with ">Lars on the new xen.org properties and public announcements.

In addition, Xen.org has become a Linux Foundation Collaborative Project and MirageOS was prominently mentioned in the press release!

Unikernel Mirage Prototype[edit]

Aug 2012 - Mar 2013

The first Mirage prototype was initially described in the HotCloud 2011 paper, and then built and evaluated in the ASPLOS 2013 paper. During the course of the work, we separated our thinking into the general concept of unikernels (which are specialised, sealed tasks that can be compiled very portably), and the OCaml-based Mirage prototype that comprehensively implemented the solution.

This task resulted in a solid prototype for the evaluation in the paper, but further cleanup and documentation is still required to release a public version. This continues as part of the Xen.org incubation task.