An Advanced Data Analytics Library for the Java Virtual Machine
Fully open source and released under the Apache Software License, Morpheus enables the development of high performance analytical applications involving large datasets for both offline and real-time analysis on the JVM.
Introduction
The Morpheus library is designed to facilitate the development of high performance analytical software involving large datasets for both offline and real-time analysis on the Java Virtual Machine (JVM). The library is written in Java 8 with extensive use of lambdas, but is accessible to all JVM languages.
Motivation
At its core, Morpheus provides a versatile two-dimensional memory efficient tabular data structure called a DataFrame, similar to that first popularized in R. Whiled dynamically typed scientific computing languages like R, Python & Matlab are great for doing research, they are not well suited for large scale production systems as they become extremely difficult to maintain, and dangerous to refactor. The Morpheus library attempts to retain the power and versatility of the DataFrame concept, while providing a much more type safe and self describing collection of interfaces, which should make developing, maintaining & scaling code complexity much easier.
Another advantage of the Morpheus library is that it is extremely good at scaling on multi-core processor architectures given the powerful threading capabilities of the JavaVirtual Machine. Many operations on a Morpheus DataFrame can seamlessly be run in parallel by simply calling parallel() on the entity you wish to operate on, much like with Java 8 Streams.Internally, these parallel implementations are based on the Fork & Join framework, and near linear improvements in performance are observed for certain types of operations as CPU cores are added.
Capabilities
A Morpheus DataFrame is a column store structure where each column is represented by a Morpheus Array of which there are many implementations, including dense, sparse and memory mapped versions. Morpheus arrays are optimized and wherever possible are backed by primitive native Java arrays (even for types such as LocalDate, LocalDateTime etc...)as these are far more efficient from a storage, access and garbage collection perspective. Memory mapped Morpheus Arrays, while still experimental, allow very large DataFrames to be created using off-heap storage that are backed by files.
While the complete feature set of the Morpheus DataFrame is still evolving, there are already many powerful APIs to affect complex transformations and analytical operations with ease. There are standard functions to compute summary statistics, perform various types of Linear Regressions, apply Principal Component Analysis(PCA) to mention just a few. The DataFrame is indexed in both the row and column dimension, allowing data to be efficiently sorted, sliced, grouped, and aggregated along either axis.
Morpheus at a Glance
A Simple Example
Consider a dataset of motor vehicle characteristics accessible here.The code below loads this CSV data into a Morpheus DataFrame, filters the rows to only include those vehicles that have a power to weight ratio > 0.1 (where weight is converted into kilograms), then adds a column to record the relative efficiency between highway and city mileage (MPG), sorts the rows by this newly added column in descending order, and finally records this transformed result to a CSV file.
var url = "https://d3x.s3.amazonaws.com/data/public/morpheus/cars93.csv";
DataFrame.read(url).csv(Integer.class, options -> {
options.setExcludeColumnIndexes(0);
options.setHeader(true);
}).rows().select(row -> {
var weightKG = row.getDouble("Weight") * 0.453592d;
var horsepower = row.getDouble("Horsepower");
return horsepower / weightKG > 0.1d;
}).cols().add("MPG(Highway/City)", Double.class, v -> {
var cityMpg = v.row().getDouble("MPG.city");
var highwayMpg = v.row().getDouble("MPG.highway");
return highwayMpg / cityMpg;
}).rows().sort(false, "MPG(Highway/City)").write().csv(options -> {
options.setFile("/Users/witdxav/cars93m.csv");
options.setTitle("DataFrame");
});
This example demonstrates the functional nature of the Morpheus API, where many method return types are in fact a DataFrame and therefore allow this form of method chaining. In this example, the methods csv(), select(), add(), and sort() all return a frame. In some cases the same frame that the method operates on, or in other cases a filter or shallow copy of the frame being operated on. The first 10 rows of the transformed dataset in this example looks as follows, with the newly added column appearing on the far right of the data frame.
The Morpheus API includes a regression interface in order to fit data to a linear model using either OLS, WLS or GLS. The code below uses the same car dataset introduced in the previous example, and regresses Horsepower on EngineSize. The code example prints the model results to standard out, which is shown below, and then creates a scatter chart with the regression line clearly displayed.
//Load the data
var url = "https://d3x.s3.amazonaws.com/data/public/morpheus/cars93.csv";
DataFrame.read(url).csv(Integer.class, options -> {
options.setExcludeColumnIndexes(0);
options.setHeader(true);
});
//Run OLS regression and plot
var regressand = "Horsepower";
var regressor = "EngineSize";
data.regress().ols(regressand, regressor, true, model -> {
IO.println(model);
var xy = data.cols().select(regressand, regressor);
Chart.create().withScatterPlot(xy, false, regressor, chart -> {
chart.title().withText(regressand + " regressed on " + regressor);
chart.subtitle().withText("Single Variable Linear Regression");
chart.plot().style(regressand).withColor(Color.RED).withPointsVisible(true);
chart.plot().trend(regressand).withColor(Color.BLACK);
chart.plot().axes().domain().label().withText(regressor);
chart.plot().axes().domain().format().withPattern("0.00;-0.00");
chart.plot().axes().range(0).label().withText(regressand);
chart.plot().axes().range(0).format().withPattern("0;-0");
chart.show();
});
return Optional.empty();
});
Visualizing data in Morpheus DataFrames is made easy via a simple chart abstraction API with adapters supporting both JFreeChart as well as Google Charts (with others to follow by popular demand). This design makes it possible to generate interactive Java Swing charts as well as HTML5 browser based charts via the same programmatic interface.
Chart Gallery
PASSION AND A CULTURE OF ENGINEERING EXCELLENCE DRIVES US
To Serve As Your Financial Technology Partner
Thank you for your interest! We will be in touch shortly.
Oops! Something went wrong while submitting the form.
