SailsJS: NodeJS Framework Development For Cryptocurrency Data Aggregator

Blockchain-powered cryptocurrencies have taken over the world as the most secure and decentralized form of currency with no financial institution acting as an intermediary. With the world wide web being the only source for cryptocurrency trading, web-based cryptocurrency exchanges consistently produce lots of data for blockchain consulting firms. This data is publicly available for all – waiting to be used for planned investments.

However, cryptocurrencies exchange rates remain highly volatile due to their trade volume based nature. While this raises questions on the legitimacy of Cryptocurrencies, at present it is also generating businesses opportunities for:

• Data aggregators,
• Cryptocurrency exchange analysts
• and blockchain consulting firms

Their task,  to monitor and analyze the cryptocurrencies exchange rates in order to forecast its performance in the stock market. In their attempt to escalate their business and convert cryptocurrency investors into clients, exchange analysts consistently rely on web applications that automatically aggregate data from such exchanges and process it into actionable form.

The Brihaspati Infotech recently had the chance to work with TotalCryptos, a similar cryptocurrency exchange rate analysis firm. Our collaboration followed up with a SailsJS based web application able to aggregate exchange data from 20 different cryptocurrency exchanges and generate whitelisted market indexes to evaluate the cryptocurrency market.

With this article, we aim to share the experiences of our advanced JavaScript developers with:

1. 1. Using the SailsJS framework to develop a NodeJS application
2. 2. Integrating Cryptocurrency exchange data using 20 different 3rd party services
3. 3. Challenges they faced during the development lifecycle

If you are planning to create cryptocurrency data aggregator web application, then read along to discover what the development process looks like for a similar application.

Let’s begin by taking a look at the key features of the web application we developed.

Cryptocurrency Data Aggregator: Key Features

In order to understand the key features of the application, it is essential to have a basic knowledge of a cryptocurrency exchange and what data is generated from it.

What is Cryptocurrency exchange

A cryptocurrency exchange works the same way as our fiat currency exchange outlets in a bank or at an airport. You walk in and exchange your native currency with any other currency. For Cryptocurrencies, these exchanges are done online over a web portal where you can either convert your cryptocurrency into others (for eg: Bitcoin to Ethereum) or convert it into fiat currency (for eg: Bitcoin to USD).

There are several cryptocurrency exchanges available on the world wide web, some major exchanges include GDAX, Bittrex, CoinEgg and, Binance

The fact that each cryptocurrency exchange has its own exchange rate, it’s not easy for cryptocurrency traders to deduce the market index of currency or in which exchange he can get the best rates.

This is where Totalcryptos web application helps traders by fetching exchange data from 20 different cryptocurrency exchanges and generating cryptocurrency market index.

How we retrieved currency exchange data?

The most reliable way to fetch data from any 3rd party source legally is by using their API’s. Being currency exchanges, API’s are generally available as they not just provide the data about currency trade rate but also provide the ability to perform exchanges through their API.

For our use case, we just needed the real-time and historical exchange rate data of cryptocurrencies.

As the currency exchange data updates in real-time after every few minutes, we deployed a custom CRON job tasked to trigger the data PULL request from the API after every couple of minutes.

Whitelisting raw exchange data with visualizations

The aggregated data is then used to create our own data sets by calculating average exchange data per currency. The fetched data is segregated and displayed in tabular and chart form:

1. i) Displaying data in tabular form: Displaying a list of top cryptocurrencies based on top performers for Cryptocurrency to USD exchange, top performers for Cryptocurrency to Cryptocurrency exchange and finally a full list of all the cryptocurrencies.

All the different columns that you notice in the following screenshot are fetched from the data exchange provider’s API except the Charts.

1. ii) Displaying data using Charts:

We also integrated two different chart libraries: C3 Charts and Sparkline charts to represent the individual cryptocurrency performance.

1. 1. C3 Charts were used to display the fluctuation in the price of the exchange rate for a cryptocurrency.
2. 2. Sparkline charts were used to display the TC100 index (price of the top 100 Crypto Currencies by market cap) and how it varied throughout the day.

   

Creating a custom REST API

Finally, all the whitelisted market indexes generated are also available as a JSON data feed through a custom-built REST API.  Total Cryptos REST API provides access to the data in their database as a JSON feed such that anyone can recreate a similar platform with any web development framework by utilizing Total Cryptos data.

Now that we have a brief vision of what all features were built, let’s take a look at the technology stack that was used.

Cryptocurrency Data Aggregator: Technology Stack

The data aggregator app was built using SailsJS (an MVC based NodeJS framework). The front-end template was built in Bootstrap and was deployed in Sails Views using EJS (Embedded JavaScript) template engine.

On the database end, CRUD operations were handled using MySQL.

Why SailsJS: NodeJS Framework?

SailsJS is one of the very few NodeJS frameworks that use an MVC development model. This allowed us to keep the application logic separated from the User interface code and managing the interactions between them in a separate layer.

In addition to this, SailsJS provides the ability to switch the storage layer through its Waterline ORM using which we were able to utilize Mysql and its robust ways of creating and optimizing database queries.

The sole reason to utilize SailsJS was it’s organized development and deployment workflow while not cutting corners in the application’s performance.

Why EJS: Embedded JavaScript templates?

Other than the fact that EJS is the official template engine supported by SailsJS, we utilized it due to its HTML based syntax. Even though we had the option to utilize variously advanced UI frameworks like Angular or Jade, EJS is simply the most suitable language for front-end templating in SailsJS.

We were able to utilize an off the shelf Bootstrap template to kickstart the front-end development and within no time were writing the controller code to display data in views.

Challenges we faced:

1. 1. Maintaining uniformity of exchange data

The Total crypto application was programmed to fetch exchange data from 20 different cryptocurrency exchange providers. The problem with fetching data into a single application from different sources is that each data exchange web service has its own terminology, data feed format and calling nature.

This makes the raw data non-homogeneous and unusable as our front-end views can only accept data in a linear form. Therefore, it is never a straightforward task to fetch the data and save it in the database.

Our Workaround:

To overcome this challenge, we started with creating our own array that will handle the data of all exchanges. We defined a format of our own that will be used by different views throughout the website.

Then to save the fetched cryptocurrency data within the array based on our format, we created multiple SWITCH cases, each case powered with a Foreach loop to convert the raw data into the format of our array.

The Foreach loop processes the conversion for each variable provided by the specific exchange web service and at the end of the Switch case, we had created a custom array of exchange data which is linear and homogeneous.

1. 2. Optimizing data loading time

Keeping the loading time in check is always one of our top priority for any web application that we develop as this directly affects the usability of an app. For a data-heavy web application like TotalCrypto, the loading time was an anticipated concern as the website is handling a data of around 20 different websites.

Loading the entire exchange data on page load in a traditional way was never the best way to load the data and we experienced this first hand. The web app immediately showed high loading times due to the amount of data being loaded on page load.

Our Workaround:

The fix was deployed utilizing asynchronous data loading mechanism using AJAX. This way, every time the page was loaded, an asynchronous call was made to the database while the page was being loaded and HTML/CSS elements were being rendered.

Hence, by the time the page loading was complete, the data was almost loaded and it just took a couple more seconds to completely load the data.

This was more like a time management approach for loading the web application’s content which immediately resulted in optimized loading speed for the application

Final Words

Utilizing the Publicly developed API, we have also extended the development efforts for a WordPress plugin which provides all the cryptocurrency exchange data and unique market indexes as a WordPress widget. With the entire application developed in SailsJS where visual entities are separate from the core logic, this Node.JS application is as extensible as it gets.

The future holds some exciting development opportunities for us working as development partners for Totalcryptos. Stay tuned to our blog for more information about the latest development.

Till then, Contact Us with your queries about this post or if you are looking to hire a web developer to build and launch your web application. Adios!