Several weeks ago, a project which I’d worked on for the better part of two months finally went live across all Jornaya customer campaigns. In many ways, I am prouder of this project than I am of anything else I’ve done in the ten years or so that I’ve been doing web development — in large part because the finished product is just so damn cool.
Because I can’t help myself, I’m going to jump right to the pretty pictures. By leveraging the Insights reporting backend provided by New Relic, Jornaya now has access to a full dashboard that tracks all the errors and debug data associated with our customers' campaign scripts:
Why is this cool?
Thousands of businesses use New Relic or their competitors to monitor their web sites and track errors. Why is what Jornaya did so interesting?
How does it work?
Preliminary Project Design and Initial Prototype
When the project was first conceived, the specific backend where the logging data would be stored was an open question.
Jornaya was already using New Relic for server monitoring and RayGun for application error logging. As a result, either service had the potential to be a good fit for this project. Additionally, I had previous experience with the open source project Sentry and, as a free software alternative to New Relic and RayGun, there were definite cost advantages to going with Sentry. As an additional free software alternative, we considered the ELK stack (ELK stands for Elastic Search, Logstash and Kibana).
With assistance from our first-rate Dev Ops team, I prototyped a backend that injected data into all the services except Elk, which we never tested due to the effort. (The prototype microservice was thankfully thrown away and reimplemented by Senior Engineer Ian Carpenter, which is what you hope for when you build a prototype). Next, we drew up a comparison table showing each choice’s strengths:
To cut the chase, New Relic Insights was very compelling for two reasons:
- The ability to run SQL-like queries against the data and graph the results with very little work
- As a SaaS-based solution, we save large amounts of time versus writing and maintaining our own solution
One of the interesting realizations we had when comparing the chart above with our “technical requirements” was that error data and logging data are fundamentally two different things. Three of the four alternatives are more directly geared toward error data than storing normal application logging data, but New Relic Insights (which is essentially a NoSQL database masquerading as a relational one) proved itself able to be molded to fit both error messages and logging data. A few of the negatives to New Relic noted in the table ended up not mattering after all:
- The 256-character limit on tag length is mitigated by simply spreading your data across multiple fields
- While we relinquish some control by having our error and logging data stored in New Relic’s system, the Insights API allows integrations with this data. (I even wrote a script called insights2csv to easily pull down our data so we can import it into PostgreSQL and elsewhere)
To this end, we can easily pull the data down from New Relic Insights to feed it into other databases to enable more advanced queries. This also facilitates resource sharing with our Data Science department, who are now able to use the data on which consumers encountered errors to exclude that problematic data from the datasets they use to train machine learning and develop scoring models.
Several specific examples of how Jornaya Engineering are using the Insights data are visible in the dashboard graphs above. We are now able to identify specific customer sites which are sending us high volumes of error data. It is far better for a company to proactively identify issues and reach out to its customers than it is for the company to fix the problem only after customers have noticed and complained. Jornaya is now in a position to notice any such errors before our customers do and before the errors have a chance to have a major negative impact on our products.
All in all, the finished Campaign JS Logging system has met or exceeded all the project goals. I had a lot of fun developing the finished solution and it continues to provide value to Jornaya and our customers on a daily basis.