Strix devlog #2

2018/05/30

2 minute read

** summary **

now that the foundation for getting the data and caching properly was complete and I had a couple of scanners going I started on the technical indicator calculations. The first batch contains

At first I thought I’d only store 1 record per symbol containing the most up-to-date data on the instrument, but then I opted for storing the history of the technicals as well. I could be useful to browse them later.

When running the technicals calculations I noticed that the redis cache retrieval code was still not fast enough. I also tweaked that code. Instead of making N parallel requests to redis per stock symbol and parsing and storing them in the local cache, I do a single multi get to redis, then parse the json data in parallel and store them into the cache. This is way faster (I didn’t measure the exact amount). Even after this optimization it was still not fast enough. I can’t wait more than 10 seconds for the cache warm-up. So I checked the next offender which is the json parsing. People on the net did benchmarks on the Gson library performance and it looked like it was one of the slowest libraries around. I thought that maybe using org.json could be better without using any reflection but that is the only library out there that’s slower than Gson. Luckily the library that comes with spring (jackson) is one of the fastest so I switched to using that and the results were amazing.

gson parser took 133s to parse ~6.000 items, while jackson took 1.7s to parse the same items.

After this improvement I could run the technicals without wasting time on the application startup.

Another matter that was annoying was the NaN values that TaLib4J returns. I didn’t really expect to see a NaN value but some of the data from IEX for some stocks contains no opening/closing prices or the stock didn’t trade at all somedays so the prices are all the same. This led to NaN values in the library. I sorted these out.

I also started on the stock data display page gui design. Foundation css provides a nice framework for the layout etc so I’m not using bootstrap.