In our first set of meeting with our client, we worked on breaking down the development of our system into distinct modules which could be developed separately from each other. This was not only done for easier delegation of work but also, mainly, for setting clear deadlines and milestones for the development process. We divided our system into three distinct modules, each of which is described in further detail below.

Downloading and indexing the Hansard data

Hosting and Segmentation of data

Querying the data through a user interface

Our web application is built using node.js. The node app sits on top our search backend which is implemented using ElasticSearch. Our search backend retrieves, pre-processes, indexes, and analyses the Hansard data. The whole thing is hosted on a cluster of Azure servers.

The web application is built using node.js. We use the framework express, which is a popular framework for doing web applications and services. Furthermore, we use the jade library to make templates that generate the HTML for our pages.
The node application’s main purpose is to handle requests from the end-user, query the search back-end, and respond to the end-user with the results from the queries. It communicates with the search back-end through a REST API which is used for performing search and similarity queries.

The main functionality of our application is the indexing and analysis of the Hansard data. We index the data to make it searchable and we use the TF-IDF algorithm to find similarities in the data. We use ElasticSearch as the foundation of our search engine. ElasticSearch allows indexing of data through a REST API.
In order to index the data, we start by synchronising with a repository of xml files containing all the debate data. This data is converted to JSON and preprocessed before indexing. The indexing process consists of two steps. Firstly, we generate an index of ids used for lookups of a paragraphs context (prior and subsequent paragraphs), Secondly, the processed JSON data is indexed for searching using ElasticSearch’s API.

PoliticsMapper is hosted on Azure servers. The servers are connected through a virtual network. Each server has an instance of ElasticSearch and an instance of our node.js application running. ElasticSearch automatically detects other nodes in the network and distributes the load and the data so it’s available if one of the nodes go down. All the servers are accessed through a load balancer. This infrastructure allows us to easily scale the product by simply adding more nodes to the cluster.

One of our main technical achievements is that we successfully have implemented the TF-IDF algorithm for similarity analysis.
TF-IDF is an abbreviation of Term Frequency-Inverse Document Frequency. The idea is basically to count the number of occurrences of each unique term in a document (where the document is a segment of the data). We say this is the term frequency of the term t in a specific document d and we denote this as tft,d where t is our term and d is the specific document. We now have a frequency measure of each term in a document. We want to compare this to the complete collection of documents (the dataset as a whole) in order to see which other documents are similar to this one. Here we could just compare the term frequency of our document to all the others and see which are most similar, but not all terms are going to be relevant. For example, the term frequency of the term “and” will be a high in all documents and is therefore useless as a similarity measure. We denote the document frequency of t as dft, which is defined by the number of documents which contain t. The closer the document frequency of a term is to the total number of documents N, the less important that term is in the similarity calculation. So in order to weigh the term frequency of t in d with the “uniqueness” of the term in the complete document collection, we define the inverse document frequency as idft = log( N / dft). We combine these two measures to tf-idft,d = tft,d×idft. We now have a measure of how relevant a term is in a document.

Another of our main technical achievements is our simple and modular architecture. Even though the functionality of a search engine like PoliticsMapper is fairly complex, we have managed to define an architecture that allows us to easily exchange functional components, scale the server infrastructure. We have been able to do this because of our thorough research and ongoing discussions with our clients.

Node.js is heavily based on the idea of using asynchronous callbacks and calls to functions. This design pattern is used to allow calling parts of code that can be computationally expensive or slow without blocking the thread. By providing a callback function as a parameter in a function call, the result we are interested in is returned whenever the called function is done and sends the result back via the callback function.
This pattern is one of the things that makes our web application fast and responsive.

Internally in our application we use a RESTful API to communicate between our web application front-end and our search back-end. When indexing documents a bunch of POST requests are sent to the back-end and search queries are performed with GET requests. Using a REST API for communication makes the product more modular as each component is more easily replaceable. Most platforms and languages allow for easy integration with a REST API, so if we for example decided to rewrite the front-end, this would be very easy to do.

Our node.js application makes use of the Model-View-Controller pattern through the express framework. This pattern splits the responsibilities of the application up into three parts; a model, a view and a controller. Our view is our HTML templates (using jade) that generate HTML injected with the results from the controller. The model is in this case just the interfacing with the search back-end which provide the model objects. The controller takes requests from the user, sends queries to the backend and send the results to be rendered in the views.
This pattern makes our web application modular internally and makes our code more structured.

One of our testing strategies has been to unit test where possible. This has been done for various functionalities in the node application and the indexing scripts. We have for the node application used the mocha unit test library to do our testing.
An example of one of the vital parts that we have created unit tests for is the context paragraph lookup which is the process of searching our pre-computed index of IDs for a certain ID and returning the IDs immediately prior and after the given ID. The paragraphs corresponding to these ids are then displayed on the view of an individual paragraph (when you press “view” on a result). The search is implemented using a binary search of an ordered list of several million IDs and has been heavily optimised since this is a common operation on a large set of data.
By unit testing a vital functionality like this, we have a significant level of confidence in the reliability of our code.

Another place where testing has been important to us is the quality of the search results. This is a somewhat subjective thing and it is not possible to simply write a unit test for this. We have throughout the course of the project been optimising the results of search queries and similarity results. Then testing process has been an iterative process in which we have tried different approaches to optimising the results by for example pre-processing the documents before indexing and post-indexing filtering of results. We have continually been in contact with our clients and they have come with feedback, expertise, and advice on how we could do better on our search and similarity functionalities.