Strix devlog #7

2018/12/03

9 minute read

A new month a new dev log for the stuff that’s been going on  Huge change yet again for the engine which I will get into later. First order of business is a summary of the new stuff:

Summary

  1. get rid of all but USD pairings for cryptos
  2. reduce symbol listing response size from 1.2M to 0.8M
  3. new scanner: trend start
  4. S/R zones to charts
  5. using TaLib java API
  6. Candle stick patterns scanners
  7. port back over to java w/o spring + hibernate
  8. eclim, idea + X forward, Che adventures
  9. market overview calculations in SQL
  10. store technical calculations in KV store
  11. correlation calculations in technical calculations

Good bye pairings

Some of the alt coins are worth so less that compared with BTC they end up taking 8-9 decimal places. This is a disaster for the display and the layout of the app in general. At first I had thought about displaying these types of currency pairs as satoshis but that was met with high resistance from my previous team members. The pair name is XXX/BTC so you cannot display it in Satoshi’s was the justification. Fair enough I guess. Another fix for this could be decreasing the font size if there are a lot of decimal places, but I have tried this. It sounds like it should work in theory but I’d have to experiment with it to be convinced that it does. So for the time being the easiest solution is to drop pairings with BTC and keep only the pairs that are traded against the USD. All of the big coins are included in this so no big loss their.

slim responses

The autocomplete component of Vuetify requires a list of the items to complete (though I’m pretty sure their should be a version that can do partial searches with AJAX) and that list in the previous version was huge around 1.2 megabytes. The initial delay after the autocomplete trigger was around 2 seconds which made it appear to be not responding. My primary solution to this was to trigger a request to the symbol listing endpoint after the app loaded in the background and since requests are cached the searches would not have that initial lag. This worked out like I thought but the payload size is still more than I cared for so I got rid of some of the fields in the response and change the structure from a JSON object to an array dropping the keys and therefore reducing the size even further.

more scanners A saw a scanner on STB that I wanted to incorporate as I think it’s important: The new trend started scanner. It fires when the ADX crosses the 25 line. Even though ADX lags it’s still useful to know that a trend has begun.

more charts Data visualizations are always cool, you can never get enough of them. This motivated me to add a chart displaying the support and resistance zones calculated using the clustering method I blogged about a couple of weeks back.

I plan on adding charts to each triggered scan with the relevant indicators like MA’s for MA crossovers etc. But it’s a low priority task right now.

New technical analysis library Ta-lib.org is a mature technical analysis library that supports way more indicators that I built into talib4j. I was quite happy with the results I got when using the python wrapper so it’s back in. I plan on doing a write up on the performance vs talib4j as the code is probably transpiled from the C version and impossible to read. The Java API is god awful because of this generated code, which means the C API is just as disgusting. To limit the exposure to this filth I wrapper the API with a custom class and I can just substitute that for talib4j any time if I have all the indicators coded in.

Even more scanners Using Talib also gives me access to candle pattern recognition. I didn’t want to write a scanner for each of the patterns so I had to resort to using the java reflection API to figure out the correct method to call from the scan definition file. So the definition file is something like this:

{
"id": "..."
"name": "some scanner":
"module": "CDL2CROWS"
}

I had all the definitions generated from the documentation on the talib site with a small python script (around 60 different scans) and look up the method from the module attribute of the definition. With the addition of all the candle patterns each run consists now of around 16K scans!

Back to the drawing board Now that I am running 16K scans the python code took 48 hours to complete a days worth of scans. This of course is unacceptable as the scans need to be done quickly. It’s no use to display scans 2 days after the market closes. I poked around with multi threading in the python code but to no avail as there is this thing called the “Global Interpreter Lock” which limits you to the total performance of a single core. Also using multi threaded code consume too much memory - something which I will not have that much of in production. So back to a faster language: Java. But this time I didn’t want all the bloat that comes with spring + hibernate. They consume way too much memory that was one of the reasons for going with python. So this time around it’s a plain old Java application with small utility libraries for database operations. Now 16K scans take 1 hour to complete without multi threaded code. I was initially thinking of keeping the python engine for other stuff like synchronization, and technical calculations etc. but ended up porting all the engine code back to Java.

Have Chromebook will code I got a new toy from the Black Friday sales events so naturally I want to use it all the time. But it’s not buffed in terms of hardware so I needed to find a way of coding Java without running a full blown IDE on my Chromebook. It would probably run the IDE OK but the engine + database would put to much strain and it would start to crawl. So I started coding in Emacs. Plain old Emacs with no packages. It sounds crazy and it actually is quite crazy. It was nice just suspending the machine and reattaching the session to continue where I left off, but no syntax checking, not auto imports makes it a hassle. Not being able to see the parameters of a method call was the worst. So I checked out what packages people were using for coding on Emacs. The most effective one seemed to be Eclim. I had used Eclim before with vim (which is the original purpose of Eclim: Eclipse + vim) but somebody had wrote a wrapper around the binary for Emacs. I tried to get it running on my Debian development server but I could not get it to work. It would just not connect. So I gave up on Eclim and checked out another package called Megahanada. Found that one too complicated and didn’t really find the functions it provided useful. Then I tried X forwarding with Intellij Idea. This felt like home a familiarity that was much appreciated. This lasted for a week or 2 until the inefficient X protocol drove me nuts with the stupid lagging of the UI. I looked around for some alternatives and came across xpra which was supposed to be faster but really wasn’t. And the fonts and graphics were blurry with Xpra so it went out the window. I thought that maybe it was Swing that wasn’t playing nicely so I tried X forwarding with VS Code. Same problems. But on a side note I really liked the Java plugin for VS Code - it’s lightweight and provides all the great functionality that I was searching for. Anyways then I remembered Eclipse Che - A web based IDE. I had tried Che before in it’s early stages and wasn’t really impressed with it’s current state. But know they’ve created a Docker image which is super easy to install and get running and they’ve also added Git support so worst I could use Che to write the code push and compile/test on my development server. But Che workspaces already come with Java + Maven so I could basically do everything I wanted in the Che environment. Another bad thing with X Forwarding was that once the laptop suspended the SSH connection was lost and the applications died on me. Since I have leave the computer to do other stuff (like burping, diaper changes, etc) this was bugging me quite a bit. Now in Che the tab remains open and I can continue where I left off without a problem.

bugs bugs bugs Porting over code is also a great chance to check what I’ve actually written. I figured out major bugs in the technical calculations code which I fixed. I was doing all of these calculations in application code, which is probably slower than doing them on the DB side so I moved the things I could calculate on the DB over there.

few architectural changes I was letting the API calculate some of the data on the symbol detail view on the fly, but now I decided to have these pre-calculated for the last trade date and store them in the key/value store. I really wan’t refrain from having the API do calculations to consolidate all the logic on the engine code base. This meant moving the technical and overview tab data to the key/value store. I also wanted to merge these to requests as they did have a couple of overlapping values. I also decided to store the correlation data in the key/value store as it was taking up a lot of rows in the table. I was keeping a history of the correlated items which is probably not necessary. While at it I also removed the dedicated job for correlated calculations and merged it into the technical calculations.

Wow that was quite a long post - a lot has happened in the past week.