akka-pony : Actors are magic

I want to present you a recent contribution I made to help user doing akka based architecture without much pain.

https://github.com/pocamin/akka-pony

myforexidea.com architecture

Myforexidea is my pet project. I wanted to share here how it has been built.

What is myforexidea.com ?

The idea behind myforexidea is to create a tool to help user to create robots using mathematical formula in conditions.

It relies on a basic workflow 

• Wait conditions to open a position
• Open a position
• Wait conditions to close a position
• Close the position and loop

The complexity here for a person wanting to create a robot is to write the conditions and the structure. 

Currently myforexidea creates robots for metatrader 4, a well known platform for trading Forex. It means that the code generated by myforexidea should be written in mql4 : the language of the platform.

The code generation

The code generation is divide in 3 parts 

• Parsing of the formulas belonging to the strategy
• Conversion of the AST into mql4.
• Composition of the final mql4 code given strategy parameters.

The application has been designed to enable compilation into other programming language. That's on purpose since the future of myforexidea is to go further than MT4.

For parsing the formulas I used a ANTLR4 grammar that I have created for formulas. 

The conversion of AST in MQL4 uses the visitor pattern. It is the most easy way to get some control of the generation flow. In case of conversion into other language the visitor should be rewritten accordingly. It normally won't be hard to do

To compose the final mql4 code I use freemarker. It is a good template engine. I just needed to take care about automatic parsing of number but was easily fixable.

Testing MQL4 code 

Testing mql4 code is not an easy task. Metatrader is not an open platform. There is no way to introduce a testing framework into it because you have no control of the expected events of the platform (ticks, bars...) 

Running the code on the platform and doing manual test are also sometimes impossible. Hence it is hard to find data for certain very specific case.

On the other hand, we have quite good testing framework on JAVA and good integration with CI tools. so

👊👊 Let's compile MQL4 into JAVA 👊👊

Creating a perfect compiler from a language like MQL4 to Java takes time. It's a pet project : I had no time. So it will not be perfect ! Who cares ?

I compile only a limited set of language features. Given that I will do only unit test it is ok to have a compiler that doesn't support notion of classes for instance even if I'm using it in the generated MQL4 code. Hence I extracted only some methods of this class that I want to test. It also means that my methods should be designed with limited responsibilities... you want that anyway

At the end it was a winning strategy even if i wonder if I shouldn't have created a JAVA (or other testable language) to MQL4 compiler. It would have ease compilation into other platform than MT4 but it would have been possibly more time consuming.

Frontend

For the frontend most of the things use standard Angular2. I've learnt a lot during this dev phase in particular rx.js. It means that the application is not everywhere clean and I'm focusing now to make it more consistent. Learn rx.js before angular2 it will help you

I had the chance to work with react and redux in the past. Redux was good for state management and I took some ideas of it. Nonetheless I've chosen to keep my strategy object mutable. The benefits of immutability in the context of something that can be only changed by user interaction is not trivial in this case and it has a huge cost (Doing normalization in particular can be overwhelming). 

Back to the time I started, I didn't know about mobx but I will probably use that if I refactor the application later on.

For now instead of an action, I'm sending a Mutator that is a function that change the state. This mutator are created through methods that belong to same module/directory than the state and publish a change through Observable. Easy and no hidden magic here

For the validation of the content of the formula, I could have used javascript version of antlr4 code generator but it is very hard to integrate in a web page (Use some weird require implementation) and it is slow. It's is a very bad point because I would be able to reuse the same grammar.

I end up using peg.js it is fast, feature complete but badly documented. The killing feature is its online live testing tool that I used a lot to create the grammar. Thanks guys you did a very good job overall.

For the editor I use ace.js with its own parser for syntax coloring (Yeah 3 different parsers for formulas !!!!)

I have for a long time searched for the perfect library to create a graph with the different step of the strategy. For me it makes the tool way more usable and easy. If I pretend making the life more easier for non developer, I cannot propose a tool too complex as a solution.

Here is the graph of a strategy

Let face it, there is no one library that permits to do what I wanted.

Finding that out took me almost the same amount of time that's doing all this with a low level svg library (snapsvg). Did this search worth it ? Of course, doing a library by yourself takes normally huge amount of time and on top of that you have to maintain it. If someone would have do that for me obviously I wouldn't have reinventing the wheel.

I also use Bootstrap 4 since I only use css it is stable enough even in alpha.

The icons come either from font awesome or from myself (I'm using Inkscape to do design)

Recently I have discovered primeng. There is lot of components converted from primefaces to angular2. I wasn't convinced by angular material components but here there is a lot of really good ones. I will probably migrate some of my component to it.

User management

I use json web tocken to manage authentification.  JWT offer me the possibility to have a stateless architecture by removing the need of session management. Also it is very versatile as I will show you.

When user land on myforexidea it automaticaly creates a temporary json web token that is stored on browser's localsession. This web token as an audience "temp" that will disable some features. 

Nonetheless myforexidea stays usable to a certain extend for the not logged user and idea are stored on the backend server as for normal user. If the user loses it's JWT, its ideas will stay phantom ideas : nobody will be able to see them anymore. 

When the user connect, the backend server moves the temp user ideas to the users ideas. 

Lost password and account activation uses also a different kind of JWT that permits to do only this action for a short period of time (expiration attribute of JWT)

It is really easy to put in place JWT I didn't use services like auth0 because of temp user but in most of the case it would have been smarter

To manage easily this user I have an AuthHttp object that replace angular2 Http where needed.

Deployment 

That's still a work in process but I describe here the target architecture which follow the idea of microservices and stateless server.

All the application is bundled as docker containers. 

There is 7 types of containers :

• The user container : It is responsible of all user and JWT related operations
• The front server : It's a apache2 container that is the entry point for the user. 
• The asset processing container : It is responsible of converting user assets (image) but possibly other stuff.
• The asset store : It stores user assets and generated robots -> this part may be moved to an s3 server
• The MT4 compiler : It is a simple golang server that compile mql4. Since the mt4 compiler is for window only it also contains wine. (It's open sourced : https://github.com/pocamin/mql4-compiler-docker)
• The Backend server : It has the responsibility to store and convert ideas into mql4 code 
• The Database : a simple postgresql instance.

It is a bit complex as an infrastructure but it is really easy to deploy containers nowadays and brings so much flexibility that it worth to be done.

Some figures 

• Number of containers in prod ~10
• Number of servers : 2
• Number of parsers : 3
• Number of programming languages : 4 (java, golang, typescript, mql4)
• Time taken : ~ 500 hours
• Cup of coffee : ~ 1000 
• Fun to do : 👊👊👊👊👊 
• Number of devs : 1 

Boolean logic with events

Hello guys,

I want to share with you one thing that I have faced recently, It was about to do boolean logic with event stream.

My issue was to compute some expression that uses time range sequences like

there as been at least one click during 10 minutes then the user open a new window within 1 minute then he click on button A within 10 secondes or on button B within 30 seconds.

Guess what, it is not easy to compute that.

Here is the solution in java (easily convertible to something different)

https://github.com/pocamin/ComplexEventLogic

Bookmarklet for password

It's friday,


Today we create a stupid bookmarklet for having no more 100 passwords to remind :

javascript:(function () { var ps = document.querySelector("input[type='password']"); var p = ps.value; var h = window.location.hostname.match(/[^.]*\.[^.]*$/)[0]; if (p !== undefined) { var t = ""; for (var i = 0; i < 10; i++) { var e = (p.charCodeAt(i % p.length) ^ h.charCodeAt(i % h.length) ^ i); t += String.fromCharCode(i < 3 ? 97 + e % 26 : i < 5 ? 65 + e % 26 : i < 7 ? 35 + e % 4 : 48 + e % 10); } console.log(t); Array.prototype.forEach.call(document.querySelectorAll("input[type='password']"), function (d) { d.value = t; }); }})();


You type your password on the first password input of a webpage and it generates a hash with the hostname (ex google.com) and the password you initially typed.
The bookmarklet puts the new hashed password on all relevant inputs and on the console if you want to print it on a T-shirt

Continuous development

Hello,

Here is a small trick to enable continuous development with your source control tool (GIT, HG, or many others)

The idea is that maybe your build chain take a bit of time and before having completely polish what you have done you want to verify to everything is working well with you build so far.

Unfortunately if you are doing that, say goodbye to your refactoring that you want to do meanwhile, It will break the running build process.

So what about

1. Do a patch between latest repository version and you current development
2. Clone you repository in another location
3. Apply the patch to the new location
4. Run you build
5. Depending on the success/failure status send a system notification  

And of course everything is shell scripted... 

That's trivial but some developers don't know how to do shell nowaday

Better test : featured random generator (1/3)

EARLY ACCESS please provide feedbacks
As of 2015/09/20 I'm still working on a library that enable this kind of test. That a bit harder than what I suggested here and I will end up providing more implementation details 

In this series of 3 articles, I will propose you two tools to improve or implement your testing framework (mainly IT tests).
The idea behind that is to make your test suite more modular, easier to maintain, rich (lot of tests), faster to run and parallelizable.

• In this first article, I will talk about featured random generator
• In the second article I will talk about forest testing. 
• And in the third article I will propose some implementation details

As usual, I will not propose you a complete implementation. Indeed, this blog is more and more a way for me to share ideas and upgrade them. On top of that, proposing an implementation will discard all not java developers. Finally, I hope you will do this work for me :P

So let's start with :

Featured random generator 

In my previous article (you say that value is not used prove it) I introduced the concept of Random value generator. The idea was for an object used in a test to populate not relevant fields for this test with automatically generated random values.

I really invite you to read that blog entry in order to understand this article

Here we will extend this concept in order to populate all fields with smartly chosen random values.

The issue

You have to test some behavior of your application, somehow it's means that you have to specify behavior of your object and not its characteristics.

For instance let say you want to test what append if you crash a car in a wall at 20mph.
Maybe you would like a tests suite like :

Given a not safe car And you send it to a wall at 20mph Then the driver goes to hospital 

Given a safe car And you send it to a wall at 20mph Then the driver will walk to reach its destination

and not something like

Given a car with seat belt, airbag, thick bumper, compressible motor, no sharp flying element And ... 

Given a car with seat belt, thick bumper, compressible motor, no sharp flying element And ... 

Given a car with compressible motor, no sharp flying element And ...
......

It is a lot of tests (it can have in this case up to 32), It is hard to maintain and hard to figure out the normal behavior. Does it bring your more confidence ? when you write it maybe but in one year after when you will have to do a small change, will you pay attention to all this tests ? Be honest ! I won't

Featured generated objects

You get it ? not safe and safe are what we miss in most our test's object creation. A way to specify behavior, annotations to help our random value generator to smartly fill our fields.

And this annotations give a contract to the object to be created. 

For instance if a car is safe 

• it has a seat belt 
• it should also have either a thick bumper or a compressible motor 
• maybe if it have an airbag we can accept to have sharp flying element ... (I'm not an expert in car security)

or, in an other form :

safe = seat belt & ( compressible motor | bumper > 2cm ) & (airbag | !sharp flying element) 

not safe = !safe 

It is at least for now up to you the way you implement this contract but it is obviously not obvious. The main idea is that you need to use your random value generator to create a matching set of properties given the contract you have 

Also, you have to consider that sometime we need objects that have many features that interact. (That is not hard to do if you solve previous problem I guess)


That's it for today, it is late in the evening and I start to fell asleep.

Your tests take too long to run

On a big application with lot of acceptance tests, integration tests, unit tests, performance tests, whatever you want tests you can wait sometime more than 1 hour to have your results.

YOU SHOULDN'T WAIT

There is not excuse. Whatever your change is, it doesn't justify that you have to waste one hour of your precious time. 

Sometime people says "During this time, I may use my brain to think about next tasks" you know what ? If I have to queue for one hour at the checkout, even if the cashier allow me to think about what I can do when I will be back at home, I just don't care and I want to burn the checkout, the cashier and the shop too.

SO BURN YOUR TESTS

Agree ? you don't want that anymore ? so what next

1. Refactor your test

Profiling tool are not only for your running application. For instance, I gained 50% of time changing a simple configuration in jBehave's steps retrieving in my current project. It wouldn't have happen without profiling.

2. Use crowd testing !!!

You are not alone on your journey there are guys that can help you : your colleagues and your continuous integration server.

Sound weird ? let me explain :

• 90 % of the time you know witch test may have been impacted by your change. (You might want to look at the next chapter at this point but don't do it unless you want to come back here later and it will be all scrambled in your mind)
• Run those tests on your machine. (You should take less than 5 minutes to do so) 
• Run your new crazy maven/make/ant/gradle/yourOwnStuffThatIsSoCool goal that will run only a chunk of the tests (Your are 7 in your team run only 1/5 of all tests for instance)
• Ask all your colleagues to do so.
• Grab the result, fix your test

How to do that ? That's up to you, but some ideas :

Randomly chosen chunk. If you and your colleague run tests quite often maybe it is reasonable to choose randomly your tests to run and hope to have a failure quite soon

Here is some example with different size of chunk and number of builds required to have all tests run at least once.

	chunck size	0.1
Number of build	0	5	10	15	20	30
Chance that a test is run	0%	41%	65%	79%	88%	96%

By running only 1/10 of your tests, you have to run your build 50 times in order to make sure (>99 %) all tests are run. If your team of 7 guys are running tests 5 times a days your get >96% chance that the failing test will be run in one day.

On top of that your tests now take virtually 5 minutes to run

	chunck size	0.2
Number of build	0	5	10	15	20	30
Chance that a test is run	0%	67%	89%	96%	99%	100%

By running only 1/5 of your tests, you have to run your build 20 times in order to make sure (>99 %) all tests are run.

On top of that your tests now take virtually 10 minutes to run  

	chunck size	0.5
Number of build	0	5	10	15	20	30
Chance that a test is run	0%	97%	100%	100%	100%	100%

By running only 50% of your tests, you have to run your build 20 times in order to make sure (>99 %) all tests are run.

On top of that your tests now take virtually 30 minutes to run


The good point of this strategy is that whatever the frameworks you use it should be easy to implement that.


Round Robin chunk

Better but less easy to put in place is to choose your chunk with an incremental system. It means that all test in the chunk you run will be different than for the next build. It is not always easy to implement since it should be stateful. (Some ideas later but you have to continue reading)


Almost Round Robin chunk
 
Each user of your system builds a predefined chunk of tests but in this solution the idea is to insure that if only x% of the guys are required to run all tests. (Yes some guys dare to be sick in my team)

It means that we are doing a more tests than our chunk. For instance if I have a team of 5 guys and if i want to have all tests runs even if 1 guy is absent I have to run 1/5 of the test + 1/4 tests to cover others colleagues chuncks   = 45 % of all tests 

Here is a reminder table :

		Number of colleagues
		4	5	6	7	8	9	10
number of colleagues potentially dead	1	58%	45%	37%	31%	27%	24%	21%
	2	92%	70%	57%	48%	41%	36%	32%
	3		95%	77%	64%	55%	49%	43%
	4			97%	81%	70%	61%	54%
	5				98%	84%	74%	66%
	6					98%	86%	77%
	7						99%	88%
	8							99%

So it's becoming quite interesting and worth investigation with large team.

Also one thing to understand is that you have to put an order to each members of your team and store it somewhere. (As a bash variable for instance)




Continuous distrubuted testing 

We can also think about having one way to have a daemon on each machine that run chunk of tests and communicate with each other to allow full test coverage in an acceptable time.

You can even think about having a predefined time to test and have a time limited chunk. (Chunk should take less than 5 minutes)

Most of the time you can even use your own VCS and commit one single little file that informs others what need to be tested in the next build
Also using your VCS you can easily verify that all the test has been run for a unique commit.

Although even if it s clearly cool, this solution is not easy to implement if you are thinking that you want test processes to be run in parallel

3. Test in priority what need to be tested

Challenge your project to know if it is really useful to run all tests always. does it worth it ? What are the benefits vs cost ? Don't be dogmatic

If you want to segregate your tests there's multiple strategies :
- Isolate business domains in your test (Annotation or other language artifact may help you)
- Create separate goal in your build tool to run only subsets of tests
- Link your test coverage tool with your build and VCS tool Unfortunatly it's your job here to do that, the idea is to say that if you modify one piece of code it should, most of the time, only impact new created test or test that previously cover the chunk of code you have modified.