Salvus: Distributed Scalable Online Mathematical Software

Salvus

Distributed Scalable Online Mathematical Software

William Stein

University of Washington

Screencast of talk (part 1 of 2)

Screencast of talk (part 2 of 2)

Project Goals

Make Sage (and other math software) very robustly available over the internet to a large number of users
Generate substantial revenue to support core sage development: working with UW C4C on license agreement, payment system, etc. They get 20% of revenue and the other 80% goes in the Sage Foundation account.
If successful, will spin off as a separate company in a few years.
If successful, will hire elite developers to work fulltime on the core research-level mathematical algorithms in Sage.

Related Projects

Programming, but no math
- http://c9.io -- "Your code anywhere, anytime... Write, run, and debug your code with our powerful and flexible cloud IDE."
- http://www.heroku.com/ -- "cloud application platform"
- https://www.appsoma.com/ -- "Code & run in the cloud"
Math, but no programming or scalable computation
- http://www.wolframalpha.com/
Sage/Ipython
- http://sagenb.org -- online notebooks
- http://aleph.sagemath.org -- single embeddable sage snippet
- ipython notebook
Cloud hosting: can do everything, but expensive and complicated...
- http://aws.amazon.com/ -- Amazon Web Services
- http://www.linode.com/ -- "Deploy and Manage Linux Virtual Servers in the Linode Cloud."

sage-support mailing list a few days ago

On Wed, Sep 26, 2012 at 9:54 AM, Chris Seberino  wrote:
> I have a Xen Ubuntu Linux 1 Gb RAM virtual machine  from a provider
> called Linode.
>

Linode charges $39.95/month for your 1GB machine.  For a 4GB machine,
which would be more likely to work well, you would have to pay
$159.95/month.

My plan with http://salv.us/ -- which I'm working hard on right now --
is to buy a bunch of hardware (mostly done), host it at UW, write much
better more efficient software to make Sage ("highly") available
through it, and provide enough guaranteed resources for a class like
you have... for *substantially* less than $159.95/month, since the
monthly hosting and network fees I have to pay at UW are super-cheap.

Stay tuned...

 -- William

Longterm Programming Goals (part 1)

Simple starter page: where you can type Sage code in a box (no login required) with tab completion and interact support. If you login, then you can easily browse/search/share all past computations you've ever done. Support for mobile browsers from the start.
Projects: A project is defined by a git repository, which salvus stores. Many users can share the same project. It can be linked to github (or google code, or bitbucket, etc.) Thus a project provides a directory tree of files, and there is a revision history.
A command line: alternative way to browse a project, and will be like the bash command line. Can type "sage" to run Sage here, and "git" to run git, providing the full power of git.

Longterm Programming Goals (part 2)

Editing documents: Salvus will also provide browser-based editing of several file types, starting with .sws files (for compatibility with sage), but also supporting many other file types, and ways of working with documents and data. Multiple users can simultaneously edit a document; changes are broadcast and updated on all clients. Also, editors for .py files, .pyx files, csv files, presentations, etc. Special support for the Sage library as a project.
Publishing: this is like Heroku or Google App Engine; it will support making a persistent Sage-based application available to many users, either with a specific URL, or embedded in another web page, like with aleph.sagemath.org, ee Rob Beezer's page. Paying users can allocate dedicated hosting resources.
Chat/cross-user messaging: useful for teaching, online classes, tutorials, etc.

Approach to Salvus

My entire approach to Salvus is the exact opposite of the monolithic single-point-of-failure Sage notebook:

Take my time and do things right: With the sage notebook, I wrote something that looks and feels a lot like the current version in a sleep-deprived month.
Service-based architecture: Numerous independent components running on dozens of machines communicating securely via messages over a VPN.
NO SPOF! absolutely no single points of failure anywhere in the design of salvus.
Concise: research existing solutions to make new code shorter and clearer, instead of just implementing everything from scratch.

Physical Hardware and Remote VM's

4 3Ghz 16-core 64GB RAM, 1TB disk, 1U servers in Padelford basement
1 8-core 32GB RAM machine in my office running OS X
3 3Ghz 16-core 64GB RAM, 6TB disk, 1U servers at UW tower
2 servers at servedby.net (a cloud hosting started by UW undergrad)
Google Compute Engine: "8 single CPU instances with 100GB of persistent disk space"
Google App Engine: I received $60,000 via a research grant program -- not sure how to use it yet!

Fixed monthly costs

Current monthly fixed cost for hardware above: $60/month (for UW tower hosting)
Amazon (in multiple ways) also told me they would give me free credits, but didn't deliver. So far, too big, confused and confusing.
This should be enough resources to support several thousand paying customers (and many free customers). Do the math.
Google and Servedby resources are free right now, because they want my longterm business

Revenue Model

I've written 100% of the new code in Salvus. It is not open source, though all the third party components it builds on are. I will open source some key pieces of it over time, since they will also be useful for research mathematics applications.

Competitive advantage: better software and hardware resources than other players. Better understanding of end user needs in this space, because of Sage.
Pricing: probably around $10/month per user for the first tier of non-free account, which competes with linnode + sagenb ($150/month?).
Goal for March 2013: Have 1,500 paying users at $10/month, which is what the above hardware should be able to support, done right. $1500*10/month = $15,000/month

Other Longterm Revenue

Purchase higher levels of dedicated resources, e.g., 30 students in a class.
Site licenses with service level and special features for an entire university or company.
Software license for the complete Salvus implementation, so a company with strong security restrictions can deploy their own internal server.
Pay more for access to commercial software via Salvus (e.g., Mathematica, Maple, Matlab, Magma).

The Salvus Stack

Infrastructure: private cloud (libvirt + kvm); public cloud
VPN: tinc
Database: Apache Cassandra
SSL: stunnel
Load balancing: HAproxy
Static HTTP: Nginx
Dynamic HTTP: Tornado
Math/compute: forking tcp-based Sage server
Management: Paramiko + much new code

Architecture Diagram

   Client    Client    Client   Client   Client  Client
     /|\
      |       (sockjs)
   Stunnels ... (https://salv.us)
      |
      |
     \|/
 HAProxy Load Balancers  ...  Nginx static servers ...
 /|\       /|\      /|\      /|\
  |         |        |        |           [tinc vpn]
  |https1.1 |        |        |
 \|/       \|/      \|/      \|/
Tornado  Tornado  Tornado  Tornado    <------------------>   Cassandra   ...
           /|\      /|\                                         /|\
            |        |        -----------------------------------|
   ---------|        |        |
   |                 |        |
   |                 |        |
  \|/               \|/      \|/
 SageServer   SageServer  SageServer   SageServer ...

Client/server communication

SockJS: websocket connection between browser and tornado, when supported; otherwise, falls back to other long-polling approaches
TCP + SSL: standard UNIX Socket connection for non-browser clients, e.g, command line.
HTTP: old-fashioned POST for simple HTTP clients

Virtualization: libvirt + kvm

Virtualization: absolutely essential due to allowing users to remotely execute arbitrary code (ok, with sagenb.org nothing is virtualized, but that is just a ticking bomb, which keeps exploding in my face!!)
Public clouds: many companies (famously Amazon.com) now rent virtual machines to anybody
Private clouds: on your own hardware, start up many virtual machines based on a template with certain parameters, start services running on them. When the machine shuts down, it leaves no trace (except persistent sparse disk images).
The only persistent data: in all of Salvus is in the distributed database. This will sit in a disk partition that is encrypted using the Linux ecryptfs file system and a passphrase that I type in (exactly once from my management interface) when launching the database.

Private Cloud Distributions

OpenStack, CloudStack, OpenNebula, Ganeti, etc. -- war of the private cloud software distributions.
I spent quite a bit of time investigating all this, and it's not useful yet, at least for what I'm doing -- basically a lot of companies and people are making noise about implementing something open source kind of like what Amazon and VMware already did, but with a lot of the truly hard real work being doing by libvirt. (This will likely be useful for something in a year or two, and now if you pay companies money, but now it is worse than useless...)
The real work is done by libvirt, kvm (or xen). Evidently, Redhat put a ton of serious money/development into supporting the creation of high quality virtualization support -- KVM -- that is built into the Linux kernel now. This is what I'm using directly... in the end, to do everything I want nicely, I wrote a few hundred lines of code (vm.py) and learned about how to get around bugs.

VPN: Motivation

VPN = Virtual Private Network, in public, using encryption
By using a VPN, Salvus can easily combine cloud computers, my own computers, etc., and don't have to worry about ip addresses changing.
We don't have to worry about securing communication between the different services or vm's in salvus.
For a long time, I was worrying about things such as implementing SSL + TCP communication between the Tornado web server and backend Sage process; I threw all that code away, and instead just use a VPN and have a nice clean class A address space as a bonus.

VPN I chose: tinc

I'm NOT using openVPN, even though it is enormously popular... because all traffic goes through a master VPN server, a single point of failure!
tinc is "a Virtual Private Network (VPN) daemon that uses tunnelling and encryption to create a secure private network between hosts on the Internet." Key features:
- Public-facing machines on the VPN also acts as servers, so clients can connect to VPN as long as at least one public-facing machine is up.
- It's P2P -- traffic does not all go through one master node, which could be very inefficient.
- All traffic is encrypted.
- Easy/sensible; to add a machine to the VPN, you just generate a public and private key, then add the public key to at least one public facing machine.
- Small C program that builds in seconds.

Database: the memory of Salvus

The database is the only persistent component of Salvus. It stores everybody's data -- it's the memory of what the system is. It's thus an absolutely crucial component of the design.

PostgreSQL is very nice, as is MySQL and even SQLite. I really wanted to use one of them, and tried very, very hard to do so (and wrote a lot of code that I threw away). Then I started thinking seriously about the amount of data I will have to manage with the number of users I plan to have in the long run. And I read books about how to scale website using mysql. What they do is very difficult and ugly, e.g., sharding -- a bunch of different databases based on usernames. (I also read a lot about SQLalchemy, used PostgreSQL a lot, etc.)
Then the Padelford server room blew up for the n-th time, and I firmly decided on no SPOF's as an Axiom.

Database: noSQL

NO SPOFS!: Relational databases like PostgreSQL become even more difficult to use with this constraint. You have to have a master database, at least one other slave databases replicating it, and some sort of automatic failover that is activated when your software decides the master is not working (which is not an easy thing to decide!).

"I'd like to welcome the github ops/dbas to the club of people who've learned the hard way that automated database failover usually causes more downtime than it prevents." http://news.ycombinator.com/item?id=4524340

One of the motivations for the recent wave of "NoSQL databases" is creation of databases that have no single point of failure. They scale up by just adding more small machines, and one can configure how redundant they are. I had learned a lot about MongoDB (a noSQL database) a few years ago, and after getting LMFDB.org to use it a lot (and using it for my own database work), I think it sucks.... for me at least. "Does everyone hate MongoDB?"

Database: Apache Cassandra

NoSQL: I read about and tried out every noSQL database I could find. (I even wrote one: http://code.google.com/p/nosqlite/) For my particular use case, by far the best one is Apache Cassandra (cassandra.apache.org).
Cassandra is an open source database with copyright owned by Apache, and also commercial support from a company. It is written in Java, which I'm no fan of, but still I really like Cassandra. Unlike many options, Cassandra is pretty heavily used by serious companies. E.g, Netflix makes enormous use of Cassandra, as does Disney. Cassandra was originally written by Facebook, who open sourced it, and it gained traction in the open source world.
Cassandra seems to be the only database that satisfied my constraints (mature, distributed, scalable, support for multiple data centers). I've been using it for a while now, and I'm impressed. I started with version 1.1 using CQL3, which looks like a restricted subset of SQL, with NO JOINS or relations at all between tables!

SSL: stunnel

One of the main motivations for rewriting the sage notebook "back in the day" to use twisted was to support SSL, i.e., encrypted communication between the user's browser and sagenb.org. This worked, but we never properly signed the cert, so users got a terrifying warning when they visited sagenb.org, and it was deamed easier to just remove the security than deal with the cert stuff.
Can easily make any webapp using SSL encryption without having to touch the code that defines that website -- just use the program stunnel!
Getting a properly signed SSL certificate setup isn't impossible. (I even bought my cert for $12.99 by some trick... though UW actually offers free sigs.)
With salvus we run a couple of stunnel's and put them all in DNS, so the browser chooses one. When an stunnel fails, the browser will just load a different one. No SPOF.

Load balancing: haproxy

The stunnel process directs all requests to another program called haproxy (which can be running on a different computer or the same computer).

Haproxy is pretty much the canonical open source load balancer program.
Haproxy monitors a bunch of web servers (for salvus, that means nginx servers and tornado servers), makes sure they are working, and balances incoming connections between them.
Static content requests go to nginx.
Dynamic websocket connections go to tornado servers.

Static HTTP: nginx

nginx seems to be the canonical choice for serving a huge number of static webpages efficiently. It was easy to setup and use. It's lightweight and easy to build from source.
nginx is single threaded (but fixed number of multiple server processes). This is absolutely key to its success in serving huge numbers of requests per second. It's far more efficient than the old apache approach, which would spawn a thread or process per request.

Dynamic HTTP: Tornado

http://www.tornadoweb.org/ -- an open source version of the scalable, non-blocking web server
Tornado is basically a clean self-contained nicely coded webserver written in Python that works a lot like twisted (i.e., it is asynchronous), but is much simpler and more targeted.
"While Tornado is similar to existing Web-frameworks in Python (Django, Google's webapp, web.py), it focuses on speed and handling large amounts of simultaneous traffic."
Very complete and self-contained and self-consistent, e.g., has its own template system built in
It was written by some company called Friendfeed founded by a bunch of ex-Googlers that Facebook bought; Facebook then open sourced it.

More about Tornado

Ipython's notebook also uses Tornado
Tornado does well in benchmarks.
Tornado is event driven. This is a basic axiom in the design of Salvus.
One can read all the documentation for tornado in a few hours, and you really know all you need to know, except what you'll learn from the excellent source code.
One concern I have is that right now the way Tornado talks to cassandra is not asynchronous -- I may have to write a better driver. Database queries are typically super fast though, by the nature of cassandra slow queries simply aren't allowed by the language.

Math/compute: forking tcp-based Sage server

I wrote from scratch a TCP server that starts up as a Sage process with various things pre-imported such as graphics, Maxima, etc., listens on a socket (running as root), accepts a connection, forks, lowers privileges, executes blocks of sage code, and streams results back over the connection.
The design is much different than with the Sage notebook, where one process watches another. Here, as code executes, the running process itself sends messages whenever it wants containing whatever it wants. The tornado server receives these messages and routes them on (either directly to a connected client, or to another tornado server to which a client is connected).
All communication uses Google Protobuf2 binary format for speed and clarity. protobuf2 is a brilliant way of *defining* a message protocol. And it is very, very efficient and well supported.

Service Management: approaches

First I tried to do management by writing a Tornado-based management web application that had a database backend. This got painful and ugly quickly, but looked cool and powerful.
I then wrote a big Python program to do component administration (starting and stopping the many programs listed above), using mainly Python Subprocess module, and only got it working on a single machine.
Then -- just as with private clouds, I found that there are tons (and tons!) of programs out there for managing installing and running software on a bunch of machine. I found one called Ansible that I kind of liked at first, but found it to be too immature and not so widely used. There were some other solutions, but they seemed like massive overkill. I spent quite a bit of time investigating them. But for every actual application, there is a bunch of subtle logic that just has to be figured out and coded, and I had a lot of that down from my code mentioned above.

Service Management: Paramiko

I took some of the good design ideas from Ansible, and the really, really good underlying library Ansible uses -- Paramiko, which is a nearly complete re-implementation of ssh in Python. Regarding Paramiko, it is very good code and it has excellent documentation. The documentation is a bit odd, because it visibly looks ugly, unorganized and bad, at first glance. But when I actually started *using* it to get work done, it is incredibly good.
So I adopted some of the code mentioned above, and have something that I like now (which is only 776 lines of Python code total right now!). This is actually where most of the code I've written and kept has gone.

Summary: Salvus Architectural Axioms

NO SPOFS: Absolutely no single points of failure
EVENT DRIVEN: Everything must be event driven; absolutely no polling is allowed anywhere
SERVICES: Scalable robust independent symmetric services; avoid monolithic design as much as possible
CONCISE: Research existing tools and approaches extensively to minimize amount of new code
LOW FIXED COST: For the first year, the monthly fixed costs must minimal

Next Talk

Demos showing how these components actually work...

University of Washington / Oct 4, 2012