Why you should consider using asynchronous programming for your eDiscovery work

To most programmers in eDiscovery world, asynchronous programming might sound like something new. However, the concept has existed for decades, and there are many implementations in pretty much every mainstream languages. If you do not know it, you must have heard of multi-threading/multi-processing for concurrency. Async programming is a way of achieving concurrency by only using one thread! Because there is no need to start new thread or new process, there is no need for OS to participate. Everything is done within programming language itself. 

A simple analogy how asynchronous programming works

Let's say you are working on 3 tasks. You can start each task immediately, and the tasks will go on by themselves without needing you to intervene till they finish. Let's assume task A takes 5 mins, task B takes 3 mins, and task C takes 2 mins. The traditional way of finishing them is doing them one by one. Order doesn't matter. You will need 10 mins to wait for them to all finish. In asynchronous world, you would start task A first, and then you immediately start task B and C shortly after. Since task A takes the longest time, by the time when task A is done, your task B and C are both done already. In this case, you will just need slightly more than 5 mins to finish all. Now you see how this asynchronous thing works? In the nutshell, it allows you to do other things while waiting for some long running tasks. 

In our eDiscovery world, we deal with a lot of long waiting tasks like waiting for a web server to respond, waiting for network to send a file, waiting for database to return a result and etc. If we do everything synchronously, we could waste a lot of time waiting. Asynchronous programming is here to rescue.

An example implemented in Python

The is one of many things that I do quite often during my work. I need to write a script to interact with a Restful server. You could imagine it is a Relativity instance, and you are querying via Restful APIs to get some data. In real world, when we do eDiscovery work, we never work with just a few documents, right? we work with hundreds of thousands, sometimes millions documents. In this example, we are querying a dummy RESTful API http://dummy.restapiexample.com/api/v1/employees. It will return a json string containing all the dummy employees. We are curious to know how many employees we have. To simulate this to a real world, we initiate 1000 calls to this endpoint, and time the process. Let us see the time difference between sync and async programming. 

You can see with only 5 workers, the async version is already 10 times faster than sync version. 

Conclusion

Asynchronous programming has already been adopted by Relativity Kepler Framework, and Relativity Service API. I will not be surprised to see more async programming shift in the world of eDiscovery. 

Comments

Post a Comment

Popular posts from this blog

Perform efficient Latent Semantic Index using Python

Image
First of all, Latent Semantic Index(LSI) and Latent Semantic Analysis(LSA) are interchangeable terms. In E-discovery world, we often use LSI but in other fields, researchers often use LSA.  Latent semantic analysis   is widely used in information retrieval to search for similar documents. In E-discovery doc review, finding similar documents can drastically decrease the number of documents case team have to review so it is a powerful and essential tool in today's E-discovery industry. LSI is the thing behind Relativity conceptual analytics .  What is LSI and the intuition The main idea of LSI can be generalized in one simple sentence: Words with similar meaning will occur in similar documents .  Each document can have several topics and several words together can be used to express a topic. In another word, each document consists of a mixture of topics, and each topic consists of a collection of words. A word can appear several topics but the mixtures are dif...
Read more

SVM

Image
SVM has been a quite hot topic in recent E-discovery scene. The reason is Relativity has introduced Active Learning which is based on SVM. Now everybody knows it works especially well on classification. However what on earth is SVM? This blog is meant to take you through it briefly so you can know something more than just how to set up an active learning project in Relativity (: Introduction First of all, SVM is a supervised learning algorithm in general, meaning it is often used to classify data which is fundamentally different from LSI. You will need labelled data to train SVM model to predict on unseen data. In a nutshell, SVM is trying to find the hyperplane that can maximize the separation margins. So in another word, there might be many hyperplanes that can separate your data, but SVM is trying to find the best one for you! Nearly all the SVM tutorials will explain what is hyperplane and what is the margins, so i wont discuss it much about them here. The basic SVM is linear ...
Read more

Open Source to E-discovery

Image
As a data scientist who is working in E-discovery industry, I am glad to be able to use many good tools to work with E-discovery data, i.e. Nuix for processing, Relativity for review and some analytics, and so on. However, the limitation for any hard-coded software is they do not always work well when data is messy and unorganized. Coupled with the high license fees and server fees, those tools are not always a good option. So, is there anything free that you can rely on to perform all the core functionalities that those big tools provide? Well, the answer is yes and no. Yes, there are numerous open source tools that can do what you are looking for, and No, you still need to pay money for hiring a data scientist but not pay money for software. To demonstrate an open source approach to E-discovery, I decide to start writing blogs about this topic. When it comes to data, Python is hands down a very good option. (The other equivalent is R). Python has a very complete and robust standar...
Read more