akka-pony : Actors are magic

I want to present you a recent contribution I made to help user doing akka based architecture without much pain.

https://github.com/pocamin/akka-pony

myforexidea.com architecture

Myforexidea is my pet project. I wanted to share here how it has been built.

What is myforexidea.com ?

The idea behind myforexidea is to create a tool to help user to create robots using mathematical formula in conditions.

It relies on a basic workflow 

• Wait conditions to open a position
• Open a position
• Wait conditions to close a position
• Close the position and loop

The complexity here for a person wanting to create a robot is to write the conditions and the structure. 

Currently myforexidea creates robots for metatrader 4, a well known platform for trading Forex. It means that the code generated by myforexidea should be written in mql4 : the language of the platform.

The code generation

The code generation is divide in 3 parts 

• Parsing of the formulas belonging to the strategy
• Conversion of the AST into mql4.
• Composition of the final mql4 code given strategy parameters.

The application has been designed to enable compilation into other programming language. That's on purpose since the future of myforexidea is to go further than MT4.

For parsing the formulas I used a ANTLR4 grammar that I have created for formulas. 

The conversion of AST in MQL4 uses the visitor pattern. It is the most easy way to get some control of the generation flow. In case of conversion into other language the visitor should be rewritten accordingly. It normally won't be hard to do

To compose the final mql4 code I use freemarker. It is a good template engine. I just needed to take care about automatic parsing of number but was easily fixable.

Testing MQL4 code 

Testing mql4 code is not an easy task. Metatrader is not an open platform. There is no way to introduce a testing framework into it because you have no control of the expected events of the platform (ticks, bars...) 

Running the code on the platform and doing manual test are also sometimes impossible. Hence it is hard to find data for certain very specific case.

On the other hand, we have quite good testing framework on JAVA and good integration with CI tools. so

👊👊 Let's compile MQL4 into JAVA 👊👊

Creating a perfect compiler from a language like MQL4 to Java takes time. It's a pet project : I had no time. So it will not be perfect ! Who cares ?

I compile only a limited set of language features. Given that I will do only unit test it is ok to have a compiler that doesn't support notion of classes for instance even if I'm using it in the generated MQL4 code. Hence I extracted only some methods of this class that I want to test. It also means that my methods should be designed with limited responsibilities... you want that anyway

At the end it was a winning strategy even if i wonder if I shouldn't have created a JAVA (or other testable language) to MQL4 compiler. It would have ease compilation into other platform than MT4 but it would have been possibly more time consuming.

Frontend

For the frontend most of the things use standard Angular2. I've learnt a lot during this dev phase in particular rx.js. It means that the application is not everywhere clean and I'm focusing now to make it more consistent. Learn rx.js before angular2 it will help you

I had the chance to work with react and redux in the past. Redux was good for state management and I took some ideas of it. Nonetheless I've chosen to keep my strategy object mutable. The benefits of immutability in the context of something that can be only changed by user interaction is not trivial in this case and it has a huge cost (Doing normalization in particular can be overwhelming). 

Back to the time I started, I didn't know about mobx but I will probably use that if I refactor the application later on.

For now instead of an action, I'm sending a Mutator that is a function that change the state. This mutator are created through methods that belong to same module/directory than the state and publish a change through Observable. Easy and no hidden magic here

For the validation of the content of the formula, I could have used javascript version of antlr4 code generator but it is very hard to integrate in a web page (Use some weird require implementation) and it is slow. It's is a very bad point because I would be able to reuse the same grammar.

I end up using peg.js it is fast, feature complete but badly documented. The killing feature is its online live testing tool that I used a lot to create the grammar. Thanks guys you did a very good job overall.

For the editor I use ace.js with its own parser for syntax coloring (Yeah 3 different parsers for formulas !!!!)

I have for a long time searched for the perfect library to create a graph with the different step of the strategy. For me it makes the tool way more usable and easy. If I pretend making the life more easier for non developer, I cannot propose a tool too complex as a solution.

Here is the graph of a strategy

Let face it, there is no one library that permits to do what I wanted.

Finding that out took me almost the same amount of time that's doing all this with a low level svg library (snapsvg). Did this search worth it ? Of course, doing a library by yourself takes normally huge amount of time and on top of that you have to maintain it. If someone would have do that for me obviously I wouldn't have reinventing the wheel.

I also use Bootstrap 4 since I only use css it is stable enough even in alpha.

The icons come either from font awesome or from myself (I'm using Inkscape to do design)

Recently I have discovered primeng. There is lot of components converted from primefaces to angular2. I wasn't convinced by angular material components but here there is a lot of really good ones. I will probably migrate some of my component to it.

User management

I use json web tocken to manage authentification.  JWT offer me the possibility to have a stateless architecture by removing the need of session management. Also it is very versatile as I will show you.

When user land on myforexidea it automaticaly creates a temporary json web token that is stored on browser's localsession. This web token as an audience "temp" that will disable some features. 

Nonetheless myforexidea stays usable to a certain extend for the not logged user and idea are stored on the backend server as for normal user. If the user loses it's JWT, its ideas will stay phantom ideas : nobody will be able to see them anymore. 

When the user connect, the backend server moves the temp user ideas to the users ideas. 

Lost password and account activation uses also a different kind of JWT that permits to do only this action for a short period of time (expiration attribute of JWT)

It is really easy to put in place JWT I didn't use services like auth0 because of temp user but in most of the case it would have been smarter

To manage easily this user I have an AuthHttp object that replace angular2 Http where needed.

Deployment 

That's still a work in process but I describe here the target architecture which follow the idea of microservices and stateless server.

All the application is bundled as docker containers. 

There is 7 types of containers :

• The user container : It is responsible of all user and JWT related operations
• The front server : It's a apache2 container that is the entry point for the user. 
• The asset processing container : It is responsible of converting user assets (image) but possibly other stuff.
• The asset store : It stores user assets and generated robots -> this part may be moved to an s3 server
• The MT4 compiler : It is a simple golang server that compile mql4. Since the mt4 compiler is for window only it also contains wine. (It's open sourced : https://github.com/pocamin/mql4-compiler-docker)
• The Backend server : It has the responsibility to store and convert ideas into mql4 code 
• The Database : a simple postgresql instance.

It is a bit complex as an infrastructure but it is really easy to deploy containers nowadays and brings so much flexibility that it worth to be done.

Some figures 

• Number of containers in prod ~10
• Number of servers : 2
• Number of parsers : 3
• Number of programming languages : 4 (java, golang, typescript, mql4)
• Time taken : ~ 500 hours
• Cup of coffee : ~ 1000 
• Fun to do : 👊👊👊👊👊 
• Number of devs : 1