Open Source to E-discovery


As a data scientist who is working in E-discovery industry, I am glad to be able to use many good tools to work with E-discovery data, i.e. Nuix for processing, Relativity for review and some analytics, and so on. However, the limitation for any hard-coded software is they do not always work well when data is messy and unorganized. Coupled with the high license fees and server fees, those tools are not always a good option. So, is there anything free that you can rely on to perform all the core functionalities that those big tools provide? Well, the answer is yes and no. Yes, there are numerous open source tools that can do what you are looking for, and No, you still need to pay money for hiring a data scientist but not pay money for software. To demonstrate an open source approach to E-discovery, I decide to start writing blogs about this topic.

When it comes to data, Python is hands down a very good option. (The other equivalent is R). Python has a very complete and robust standard library, plus many awesome data-related libraries like Numpy, Pandas, Scikit-learn, Tensorflow, Pytorch and so on. It offers a complete coverage from data cleaning, data mining, data manipulation to advanced areas like machine learning,  deep learning and etc. As a python guy, I will be using Python as my primary tool but I might throw in C++/C, Java or C# when necessary.

For the first blog post, I decide to show how to use Python to analyze two documents, and display the textual difference. The similar thing you might have already seen in Relativity after you run textual near duplicate identification, you can compare two documents within a near duplicate set. In Python, you do not have the limitation. You can compare any two documents. In Python world, there are two popular options: 1) difflib library 2) Google Diff Match Patch library The latter is very likely what Relativity uses ( : However, I would show today is option 1. difflib is one of the standard libraries of Python 3, so if you install Python 3.5+, you should already have it. difflib is very powerful already.

Let me introduce difflib a little bit. The underlying core is SquenceMatcher class. It implements Ratcliff-Obershelp algorithm. If you are interested, you can google and learn more but i am not going to expand on this. I would just like to point out the performance for this algorithm is cubic time for worst case, quadratic time for expected case, and linear time for best case. As you can see, it is not a very fast but acceptable. If you want to compare two very very long texts which differ a lot, it will take quite a long time. It is also the main reason why Relativity only allows you to compare two documents which are considered near duplicates.

Now it is time to code. ( :  Just like most of Python snippets, you can code just a few lines to get the job done. In difflib, there are basically two options to analyze and display the textual difference. 1) difflib.HtmlDiff class which generates a html file with table and coloration to show the difference 2) difflib.context_diff method which shows the difference in text format. I will be demonstrating both ways. I have uploaded all source codes and sample text files in github. Feel free to check it out. The first snippet is below

The result is shown as below:
for context_diff, it prints out a list of difference. Below is part of it just for demonstration


The result shows for text1 from position 17 to 29, text "January" is different from it is in text2 where it says "November". That is why there are ! in front of the characters. The second difference is in text2 from position 584 to 608, where it has extra text (the + sign) "t miss....". For the html file it generates, it provides a much better tabular report like below:

it is much easier to see the difference in the report than in the console printout. However, we notice the comparison is on characters. It works but somehow harder for human to inspect. If we can make the comparison based on words not characters, it will be much easier to spot the difference, right? How do we do that? Well, we need to break the text down into words (we call it tokenization). Fortunately, there are several ways to do that in Python. I choose to use NLTK library. Below is the new snippet, and we simply just tokenize the text into word list before pass them for comparison:

Now you can see the report:

It displays the difference by words. 

To summarize, to perform textual comparison in Python is just piece of cake. We can compare any two documents and have Python to generate a nice html report for us. There are much much more open source tool like Python can do, so stay tuned. 




Comments

Post a Comment

Popular posts from this blog

Perform efficient Latent Semantic Index using Python

Image
First of all, Latent Semantic Index(LSI) and Latent Semantic Analysis(LSA) are interchangeable terms. In E-discovery world, we often use LSI but in other fields, researchers often use LSA.  Latent semantic analysis   is widely used in information retrieval to search for similar documents. In E-discovery doc review, finding similar documents can drastically decrease the number of documents case team have to review so it is a powerful and essential tool in today's E-discovery industry. LSI is the thing behind Relativity conceptual analytics .  What is LSI and the intuition The main idea of LSI can be generalized in one simple sentence: Words with similar meaning will occur in similar documents .  Each document can have several topics and several words together can be used to express a topic. In another word, each document consists of a mixture of topics, and each topic consists of a collection of words. A word can appear several topics but the mixtures are dif...
Read more

SVM

Image
SVM has been a quite hot topic in recent E-discovery scene. The reason is Relativity has introduced Active Learning which is based on SVM. Now everybody knows it works especially well on classification. However what on earth is SVM? This blog is meant to take you through it briefly so you can know something more than just how to set up an active learning project in Relativity (: Introduction First of all, SVM is a supervised learning algorithm in general, meaning it is often used to classify data which is fundamentally different from LSI. You will need labelled data to train SVM model to predict on unseen data. In a nutshell, SVM is trying to find the hyperplane that can maximize the separation margins. So in another word, there might be many hyperplanes that can separate your data, but SVM is trying to find the best one for you! Nearly all the SVM tutorials will explain what is hyperplane and what is the margins, so i wont discuss it much about them here. The basic SVM is linear ...
Read more