OCaml Platform

From the OCaml Labs wiki
Jump to: navigation, search

The OCaml Platform combines the core OCaml compiler with a coherent set of libraries, tools and documentation. The requirements and direction of the Platform are initially being guided by large users of OCaml such as Jane Street, Citrix and Lexifi. The reason for initially taking direction from industrial users is because these companies have a great deal of experience of using the language at scale (e.g millions of lines of code, thousands of modules and complicated testing requirements). We take a pragmatic view that examining and solving the problems such users have will ultimately result in a large and positive impact on the wider community, through significantly improved tooling, libraries and documentation.

Tooling Improvements[edit]

The current thrust of work carried out by OCaml Labs is to improve the tooling around packaging, build and test infrastructure. Once complete, this will enable us to effectively run over hundreds of community packages and determine selection criteria for the Platform. A key aspect of this selection will be which libraries are already popular and in use, and also how actively maintained and portable they are across different operating systems.

This can be separated into:

  • The OPAM package manager acts as the package management front end
  • The codoc tool replaces the venerable ocamldoc tool with modern, cross-referenced HTML output across several libraries
  • The CISO testing infrastructure (replacing ocamlot) runs continuous integration of OPAM stable and candidate packages across a variety of operating systems and CPU architectures
  • Maintaining and improving popular libraries

See the platform mailing list for discussion about this (archive).


Non-x86 Machines

In order to create a robust and stable Continuous Integration system, we need a pool of machines that reflect the diverse environments where OCaml code is deployed. To this end, we've been assembling a small collection of conventional high-powered machines that can build the whole OPAM package set quite quickly, and also more unusual CPU architectures and operating systems that will help developers that don't have access to them.

So far we have:

  • 12-core AMD64/Linux Debian 64-bit
  • 12-core AMD64/Linux Debian 32-bit
  • ARMv6l/Raspbian Linux 32-bit (Raspberry Pi 512MB)
  • ARMv5tel/Debian Linux 32-bit (Dreamplug)
  • sparc64/FreeBSD (thanks to Jasper Wallace for the donation)
  • macppc/OpenBSD (G4 Powerbook) (thanks to Dave Scott for the donation)
  • AMD64/MacOS X 10.8
  • iMac G5/Debian Linux 64-bit
  • ARM 64BIT SERVER 1U

Current and Ongoing Projects[edit]

Ctypes[edit]

Ctypes is a library for binding to C libraries using pure OCaml. The primary aim is to make writing C extensions as straightforward as possible. The current FFI in OCaml is low-level and difficult to use correctly, so we're keen to improve the ease-of-interoperability in the first Platform release.

The core of ctypes is a set of combinators for describing the structure of C types -- numeric types, arrays, pointers, structs, unions and functions. You can use these combinators to describe the types of the functions that you want to call, then bind directly to those functions -- all without writing or generating any C!

Web Framework[edit]

Our high level goal is to enable OCaml users to build client and server-­side web applications that operate as MirageOS unikernels. In order to encourage other programming language users to use the OCaml framework for web applications, we plan to provide more client-side libraries following the existing OCaml and OPAM workflow.

  • ocaml-aws: an Amazon Web Services SDK for OCaml.
  • ocaml-cohttp: further improvements to the existing library, including a new HTTP parser that can be fed incremental input as it becomes available.
  • ocaml-session: a session manager that handles cookie headers and back-end storage for HTTP servers.
  • ocaml-webmachine: a layer on top of cohttp that implements a state-based HTTP request processor.

Assemblage[edit]

GitHub | Issues

The Assemblage toolbox provides an API and a set of binaries to setup, manage, and use OCaml projects. The library can be used to describe OCaml projects with static dependencies. A project description consists of a list of libraries, binaries and/or toplevels, each containing a list of compilation units with precise (and static) dependency relationships.

Assemblage is still a work-in-progress and has not had a public release.

Docker Builds[edit]

GitHub | Output

As the OPAM package database grows beyond 3000 packages, we need automated ways of building each package. This requires not only an OCaml and OPAM installation, but also some sandboxing techniques to install external dependencies. For Linux, we are using the Docker container manager to perform these builds. The logs from bulk builds are then collected via the Irmin database in order to be classified and rendered to a webpage.

Note that this is a purely Linux-only solution. While it lets us test various distributions such as CentOS, Debian, Ubuntu and RHEL, we still need other solutions for MacOS X, Free/Net/OpenBSD and Windows. That will hopefully get easier as the upstream Docker support for non-Linux platforms matures.

Codoc[edit]

Codoc 0.2.0 Apr 2014 - Feb 2015 | GitHub | Issues | Blog

Codoc aims to provide a widely useful set of tools for generating OCaml documentation. In particular, we are striving to:

  • Cover all of OCaml's language features
  • Provide accurate name resolution and linking
  • Support cross-linking between different packages
  • Expose interfaces to the components we've used to build codoc
  • Provide a magic-free command-line interface to the tool itself
  • Reduce external dependencies and default integration with other tools

We haven't yet achieved all of these at all levels of our tool stack but are getting close. Codoc 0.2.0 is usable today (if a little rough in some areas like default CSS). Please see the blog post for more details about its current status.

OPAM[edit]

OPAM is a source-based package manager for OCaml. It supports multiple simultaneous compiler installations, flexible package constraints, and a Git-friendly development workflow.

Previous and Completed Projects[edit]

Community Governance[edit]

Jul 2014 - Jun 2015 | Blog

As the projects around the OCaml.org domain name are becoming more established, it is important to document how the domain-name itself is being governed. Therefore we have begun to put together a governance framework that clarifies the current stakeholders and documents how the domain is currently managed. The intent is to provide clarity around how things operate in reality, rather than being aspirational. As such we hope it can be adopted without introducing burdensome processes, while ensuring that it grow and develop as the community does.

A brief summary of the growth of the projects and the factors that are part of the governance document can be found here.

OPAM-in-a-box[edit]

Dec 2013 - May 2015 | Issues

As OPAM gets more widely used for teaching, it's important to make it easy to distribute and install on a variety of operating systems. We are firstly generating binary packages for several Linux distributions (via the OpenSUSE Build Service and Ubuntu PPAs). There is also an ongoing effort to package up an OPAM environment using virtualisation tools such as Vagrant and Docker.

See the issues for ongoing progress.

Ctypes Stubgen[edit]

Sep 2013 - Mar 2015 | GitHub | Documentation

The first Ctypes release used dlopen to dynamically access C libraries, but this approach has some limitations that will be addressed by also generating C stub files that can be compiled using the system C compiler. From the Ctypes 0.4 release onwards, there is comprehensive support for C stub generation that is completely independent of the dynamic libffi-based mode (one mode can be used without the other).

As of early 2015, there are a growing set of OCaml C bindings that use Ctypes as their binding generator of choice. These include uses from industrial users such as Citrix, Jane Street and Cryptosense.

OCamlot[edit]

OCamlot 0.2.0 Mar 2013 - Sep 2013 | GitHub | Issues | Travis and OPAM

OCamlot (OCaml Online Testing), is a Continuous Integration and Testing system for packages provided via OPAM. It comprises all the appropriate tools and libraries to enable management and development of the machine infrastructure, for example an OCaml library to interface with Github (available through OPAM). If you submit a package via OPAM, you will be rewarded with regular regression tests across many diverse operating systems and build tool environments. Ultimately, we would also like to integrate benchmarking, constraint exploration, experimental design, complete isolation, and domain testing into this system.

The resourcing and commissioning of the Machine Pool itself is dealt with separately from this software which utilises it.

Current status: Minimal version 0.2.0 released.

OPAMDoc[edit]

Jan 2013 - Nov 2013 | GitHub

The OCaml toolchain has shipped with the ocamldoc tool for a long time. ocamldoc runs over a single OCaml library and generates cross-referenced documentation. It also supports a variety of outputs, such as Latex, HTML, PDF and even manpages. However, it is starting to show its age for large, complex codebases such as Core, and so we are developing a more scalable alternative for the Platform effort.

OPAM-doc consists of two separate commands:

  • bin-doc is a replacement for the OCamldoc lexer (which extracts documentation from source code comments. It uses the OCaml-4.00+ facility for generating .cmt files that contain the typed AST, and generates .cmd files which contain the documentation information. By using a separate file from the AST, we leave open the possibility of having multiple language translations in the future. These .cmd and .cmdi files can be combined with the .cmt files to generate complete documentation directly from the output of the compiler. This command is intended to be temporary, and can be integrated into the upstream in the future.
  • opam-doc takes a set of cmt and cmd files and outputs a single JSON representation of all the files. This JSON can then be post-processed (or directly rendered in Javascript) to create a single documentation repository that reliably renders cross-references across entire libraries. Thus, the entire Platform can have one documentation source rather than having to search across packages.

The ultimate aim is to support the OCaml platform with interactive tutorials using the js_of_ocaml compiler. You can try out the prototype of this in OPAM via opam install opam-doc && opam doc core async. It will start a local webserver on which you can browse the traffic. There is also a snapshot available on the Mirage documentation.