Strix devlog #10

2018/10/10

5 minute read

summary

Wow, September was a month that I would only describe as “I lost my humanity and became a beast”. After checking the git commit logs I see 480 new commits, touching 26410 lines for the API and Engine, which is quite a bit. There is basically too much stuff that I have done to cover in detail so I’ll just have to go over the most import changes for this month.

First off I want to start with the automation tasks. I was running the BFX, IEX sync tasks, scanner tasks via crontab on the test machine. This was OK for some time, but it wasn’t getting notifications about failed jobs or any information about the task duration. So I moved all these periodic engine tasks to Jenkins. Another added bonus is it’s way easier to manage the task pipeline (which task should run after which task, which can run in parallel) from Jenkins rather than a bunch of ad-hoc shell scripts. I also changed the development cycle from committing code to master to committing first into a development branch, and then asking Jenkins via Hubbot through slack to test and merge and push the changes. This keeps the master branch at a stable state all the time. G. requested that any pushes to the master branch of the web client repository be deployed immediately so I had to do some GitLab trickery to get that working, as GitLab doesn’t allow that for unpaid accounts. But it can be done via web hooks.

The current flow of crons are looking like this

Using spring boot is a MUST if you are developing with Java - the advantages it provides are countless, but it does come with it’s own quirks. I had been getting “transaction manage not found - cannot remove entity” errors on some of the data sync services. The solutions on the internet were all about adding a transactional annotation to the service method and people had accepted those answers. But I guess something changed along the way in spring development as they did not work for me. The solution is was to add the annotation to the repository interface. Sometimes something so simple can eat up a lot of precious development time, but it’s satisfying in the end when you solve it. As I was fiddling around with this I also decided to save the response JSON from the providers instead of parsing them into domain entities and saving those. Now instead of having 1 M daily data points, I have 8 K key value items and I process them in memory. I was caching the domain objects anyways so this spares some extra load on the database.

During development I wanted to get the engine results as fast as possible so I was parallelizing all the operations in the engine. I noticed along the way that “parallelStream” is not the answer to all of the question. I ran into cases where parallelizing would really screw up the cache, and serial processing with a good cache usage is way better (in terms of efficiency) than just using all the cores. I can get more bang for a core than I could when the processing was done in parallel. This decision was also in part due to the fact that I want to run the engine on the same machine as the API (yes, I am basically poor) and I can’t have the engine hogging all the CPU and exhausting RAM. I can’t also have the entire data set in memory (or most of it) and this tends to happen when threads are running at the same time. So with the serial processing using the cache in an optimal way, the scanning process takes 2.5 K seconds on a single core. Running in parallel took 600 seconds on 8 cores.

A prerequisite to serialize the scanners was to improve their performance. I installed the excellent YourKit profiler trail version (which by the way is very expensive - otherwise I would buy it) and tracked down the bottle necks to unnecessary object wrapping (using my own SuperList class with convenience methods for accessing elements, like getLast(N), getTail(..), etc.) and the parsing of string to dates and vice versa. After hunting these down and refactoring the code to get rid of extra layers and work, there was a 3x increase in speed which brings the processing time to acceptable limits.

An integral part of the system is the performance evaluator for scans based on past performance. I had coded this in a hurry and it was a bit disgusting, so I refactored this into it’s own service. The functionality is the same but it’s simplified and runs a bit faster.

I hate the verbosity of Java so I thought I’d look into what was going on in the Java world, which I hadn’t done since the release of Java 8 and the streaming API. I found out that Java 10 was released March 2018 and finally had support for the “var” keyword, which meant less verbosity in assignments. I know it’s kind of trivial but I wanted to give it a go, since spring boot it supposed to support Java 10. So I went ahead and updated the development and test environments to Java 10 and compiled the code. A couple of warnings of unsafe access in spring but otherwise everything seemed OK. That is until I tried running the API. Then a lot of exceptions about Redis not being happy with something (which I don’t remembers, and could not find an easy solution for), so I reverted back to 8, which is way more stable. I have long lines of code, but at least everything works correctly :D