Command Line Interface aka CLI

SVD is the core algorithm of Latent Semantic Analysis. I have demonstrated how to perform LSA/LSI using SVD in blog here . However, SVD itself is not an easy thing to understand. To fully understand how it works requires quite a lot of linear algebra knowledge. We all agree math is important but it is not necessary for everyone to understand the math behind SVD in order to use SVD. Is there an intuitive way to see how it works? I mean, without any equations and formulas? Yes, there is. Today, i am going to let you SEE how it works ( : I randomly downloaded a free picture from internet. An adorable parrot bird. In order to convert the image into a matrix that can be applied with SVD, i need to convert it into gray scale first, and then perform SVD. The original gray-scaled image's dimension is 2000 x 3000 (see below) This is how it looks like after we only keep the most important 300 components (2000 x 300). It doesn't look it changes that much, right? T...

First of all, Latent Semantic Index(LSI) and Latent Semantic Analysis(LSA) are interchangeable terms. In E-discovery world, we often use LSI but in other fields, researchers often use LSA.  Latent semantic analysis   is widely used in information retrieval to search for similar documents. In E-discovery doc review, finding similar documents can drastically decrease the number of documents case team have to review so it is a powerful and essential tool in today's E-discovery industry. LSI is the thing behind Relativity conceptual analytics .  What is LSI and the intuition The main idea of LSI can be generalized in one simple sentence: Words with similar meaning will occur in similar documents .  Each document can have several topics and several words together can be used to express a topic. In another word, each document consists of a mixture of topics, and each topic consists of a collection of words. A word can appear several topics but the mixtures are dif...

As a data scientist who is working in E-discovery industry, I am glad to be able to use many good tools to work with E-discovery data, i.e. Nuix for processing, Relativity for review and some analytics, and so on. However, the limitation for any hard-coded software is they do not always work well when data is messy and unorganized. Coupled with the high license fees and server fees, those tools are not always a good option. So, is there anything free that you can rely on to perform all the core functionalities that those big tools provide? Well, the answer is yes and no. Yes, there are numerous open source tools that can do what you are looking for, and No, you still need to pay money for hiring a data scientist but not pay money for software. To demonstrate an open source approach to E-discovery, I decide to start writing blogs about this topic. When it comes to data, Python is hands down a very good option. (The other equivalent is R). Python has a very complete and robust standar...

As a programmer, the first instinct of searching for something in a collection of objects will be looping through it! It is a straight solution and often times very fast, if, and only if the collection is small. However, once the size of collection goes big, it can hurt your program's performance a lot. So when you need to search for a large array, what do you do? 1) Binary search. BS is always a fast and effective way to search if the array is sorted. Regular looping through an array, aka linear search, takes O(n), but binary search takes O(logn), which is much faster. To make it into perspective, please refer to the below chart. As you can see, when the n is very small, linear search might be a little bit faster than binary search, because binary search has to do a little bit more work than a linear search method. However, when n is large, binary search is way faster than linear search. Binary search is also very easy to code up. Let us use a simple Python code to demonstr...