Strix devlog #8

2018/11/20

3 minute read

So I’ve been working on the engine in past 2 weeks every chance I get, which is not a lot these days. I’ve also had to adapt the web client to some of the breaking API changes (mostly fields names, and simple structure changes). With major changes in the database structure, some of the old tables being merged into the key value table, the engine code is a bit clearer now. MYSQL should be able to handle the queries to the KV table with ease as it’s properly indexed and the record cardinality is much lower because most of the data is stored in a JSON structure now. I implemented the top activity module, correlation and the news sync module. The correlation finder takes a long time as python is kind of slow when iterating over a lot of records and each symbol has to be checked against all other symbols to find a correlation. This made me want to switch to something faster but I resisted the urge as the correlation finder can be run maybe every other week and it’s OK if it takes 12 hours to run. I also got rid of the old API code that was hogging memory thanks to spring + hibernate storing tons of classes and garbage in memory. I went with flask which is a simple micro framework for creating API’s. Currently I create a new connection for each request to the database and I need to test if this will scale under the load. What I have read is that the old “connections are expensive” is now a myth with newer databases, but still the network overhead could prove this theory wrong. In the second half of this month I adjusted the web client code to the new API responses and fixed cosmetics here and there. I can probably say I ported all the old code to the new API with maybe a couple of features missing that I will add in the following days. A major change on the client was switching from the Google Charts JS library to static images to display the candle sticks. My initial thought was it would be good to offload the chart creation to the client to lessen the load on the server, but it this turned out to have 2 disadvantages: 1. slower mobile clients take forever to render the chart (my Samsung tablet). 2. a ton of charting data is transferred to the client which slows the page loading. So I struggled for a day with the excellent matplotlib for python to get a nice candle chart with a volume overlay and I think it turned out quite well.

Before this was completed I used a chart from Finviz as a placeholder and inspiration. I also managed to squeeze in the android client build by using an excellent plugin for Vue which was quite painless to setup. I side loaded the app on my phone and tablet and they seem to work great. After loading the apps I realized that some things like pull to refresh were missing. It’s essential to convert to a mobile app and try it out to get a good feel for the user experience, even though I’m actually developing the client in a browser. My plan for the upcoming days are using the app to iron out some more user experience quirks, then I need to get into StockTwits integration and start with marketing stuff. The launch timing seems quite bad as the markets have taken a turn for the worse - or maybe people will be searching for opportunities in this turmoil and can use TechScan to seek out these opportunities?