dimaag kehta hain.. :-)

lex_ing and regex_ing

2011-02-28T08:40:00.000-08:00

During engineering, this topic came and went.. w/o bothering me much.. just a small assignment on parser and done! But.. no.. the practical exam threw on my face a problem on designing parser in C (not with lex/yacc or generated one) and then I realized - well.. I had not understood it!

This was like 6+ years back.. and now I am here.. writing regex and grammar to generate parser with lex-yacc! That's life! :D ... And... it's very good!

Just came across a nice tutorial on lex yacc. Find it here. Even if you keep technical details apart and just glance through the tutorial, you would get a new outlook towards lex-yacc - specially regex - writing them and debugging them when they don't behave the way you expect them to..!

A very good and important thing I realized during this exercise is as follows. Parser design can easily be segregated in lex and yacc when you know your application well. Taking some reference from the tutorial link posted, for example - a simplified regex could serve your purpose given that you want to capture occurrences of a pattern; and you need a bit complex regex when you want to validate inputs. (Again from the tutorial,) For example, if you want to list down html tags in the browsed web crawl, all you need is a simple pattern that captures anything between a "<" and a ">". You need not __validate__ that __anything__. The basic assumption here - that you are expecting to get positive (meaning correctly matching with pattern) cases more in such scenario inherently comes from the application - i.e. capturing tags from web crawl. In an application of say - writing a SQL kind of language, you expect to get negative cases as probably as positive ones and just capturing __anything__ would not work. Again, I would say that it depends on how you look at it. You could always write a light weight regex and a comparatively heavy code in back end to validate your verbose input. It's up to you! But, understanding internals of how a regex engine works (given in the tutorial, for many cases), and mainly understanding its complexity and contrasting it with the complexity you would introduce by writing validation in code - is definitely important.

Indeed, a nice bunch of information!

mplayer - not playing files with _space_ characters

2010-09-24T10:39:00.000-07:00

Recently I upgraded all packages on Ubuntu and somehow since then I could not play video files with vlcplayer! Not just that, somehow Realplayer and default movie player lost the codecs they used to previously use! After this miracle, the only option left with me was to try mplayer. I had heard that it supports many common formats and so on. I tried that and faced another issue. It magically could not play files having space characters in it!!

Initially I used to manually rename the file from browser before double clicking it for mplayer. Later on I found this very hectic when following happened. I renamed a long file name replacing some 7-8 spaces by underscores (Note: spaces, it converts to %20 and cannot play such file, underscore works!) and soon realized that I wanted to see another episode (with similar long file name)!!! So, I got irritated a lot and said to my friend that I will soon write a script to rename all files such the spaces are replaced by underscores. Now, I realized that all main / important video files (movies, acad lectures and so on) were on portable hard disk and the list will get augmented day after day as I get something good from some source, I will have a copy. Thus, this method sounded a bit overkill - to rename all file names just like that!

I then did following.

I wrote following lines in a file ~/scripts/mplayer_helper.sh
#!/bin/bash
fname20=`echo $1 | sed 's/ /\\ /g'`
fname=`echo $1 | sed 's/ /_/g'`
mv "$fname20" "$fname"
mplayer $fname
The idea here is simple. Just create a file name (do some trials on command line first as to how exactly "mv" expects the input.) replacing space by underscore and replace file using "mv" command. Once that is done, invoke mplayer on renamed file.
Then I made it executable using
chmod +x ~/scripts/mplayer_helper.sh
Then I created a soft link to this executable inside /usr/bin/ as follows:
cd /usr/bin
sudo ln -s ~/scripts/mplayer_helper.h sudomplayer
Now, when I want to open video from browser, I simply right-click it and say "Open with". Then, I expand the option "Use custom command" and write "/usr/bin/sudomplayer" and say "Open".
This not only opens the video by run-time substituting spaces with underscores but also once opened, adds this sudo application into the list of applications. Additionally, sudomplayer will be the default application shown for that file format then onwards.

Easy! Isn't it!

But, there is a glitch, what if not only the file to be played but also the recursive folders it is contained in have spaces. mplayer fails then as well! But, the solution is simple! Modify the script to take care of that as follows.

#!/bin/bash
xmessage -center -nearmouse Note that this tries to rename the files so this might fail if the file path does not have write permissions
i=2
token=`echo $1 | cut -d"/" -f$i`
more=`echo $1 | cut -d"/" -f$i | grep -i ^$ | wc -l`
currDir=""
while [[ $more -eq 0 ]]
do
        currDir=$currDir/
        token=`echo $1 | cut -d"/" -f$i`
        name20=`echo $token | sed 's/ /\\ /g'`
        dir20=$currDir$name20
        name=`echo $token | sed 's/ /_/g'`
        dir=$currDir$name
        if [[ $dir20 != $dir ]]
        then
                if [[ -a $dir ]]
                then
                        xmessage -center -nearmouse $dir File exists! Aborting!!
                exit
                fi
                mv "$dir20" "$dir"
        fi
        currDir=$dir
        i=`expr $i + 1`
        more=`echo $1 | cut -d"/" -f$i | grep ^$ | wc -l`
done
mplayer $currDir

The idea is here, to go one directory at a time. Try to rename it and then keep track of current directory such that now next level directory can be renamed. But, WARNING!! What if the current directory is inside "/home/xyz/abc/"? One might not have permission to replace "xyz" with "xyz" inside /home/. There are two ways to look at it. First, mostly the filenames with space character won't appear in such so_called __system_prohibited__ paths and even if they appear, it's better not to mess with it (at least automatically!) and thus, the initial "xmessage" serves the purpose. Another WARNING! is - what if we are trying to rename directory "dir with spaces" and there already exists a directory called "dir_with_spaces". To prevent such collisions, we will terminate the script once we find such case. Work around to this problem is to use some weird set of characters instead of "_", say like "_-_" or so; but, then if this script is run quite frequently, then your filesystem (portable hdd in my case) will be full of such weird names! ;)

Anyways, another (probably best?) way is - just Google around! Then will tell you some ways to get around to this problem of mplayer not playing files with space characters. But, then you will miss all the scripting_fun! :D :P

Rhythmbox Music player - password window popping issue!

2010-05-29T06:08:00.000-07:00

I have been using Rhythmbox music player since almost 3 years now. Suddenly one day, it started popping a Give_ur_password type of window at every track. I used to close the box, then it would pop up again for thrice or so. It looked like the following.

I tried Google-ing of course, but could not find anything. There were some posts regarding why / how some plug-ins for Rhythmbox keep popping windows. I did not find anything related to password window specifically. Still, I opened Edit->Plug-ins and observed that few plug-ins were enabled. For no apparent reason, I disabled the plug-in for Cover art (as shown below) and never got those disturbing pop up windows again!

What I don't understand though is, that plug in was always enabled afaik, because I never explicitly enabled / disabled it in past, then why suddenly after almost 2.5 years, it started popping up windows?! Another thing is, as shown in first image, there is something called "xml.amazon-proxy caching server" - Have no clue how it could be related to cover art of an album!

Anyways, will definitely post the explanation if I can find one, for time being let me enjoy the pop-up_window_free Rhythmbox!

Approximate if you can do it in polytime!

2010-04-15T01:33:00.000-07:00

The main idea behind approximation algorithms is as follows. When one has to solve a problem which is proved to be NP-Complete, one has two choices.

Use usual algorithm and get answer in non-polynomial time (most of the times exponential) and be ready to wait until eternity OR
Compromise on the quality of answer but get things done faster

As suggested by 2 above, approximation algorithms have the same aim. Instead of finding the correct, optimal, best possible solution to the given problem, they try to find an approximated one; but they guarantee to find it in polynomial time so that they run faster. Now the question is - even if they are faster, how much does the quality of solution degrades? This is measured by Approximation factor or Approximation ratio. This factor or ratio basically captures how far we are from doing the best i.e. from the optimal. Consider the optimization in terms of minimization. Suppose a minimization problem is NP-Complete and suppose the optimal solution has the value 5; then if any proposed approximation algorithm, that is the algorithm that solves the problem correctly in polynomial time gives the solution whose value is 7.5 then we will say that we have got 1.5 times more than the optimal one; i.e. we are doing 1.5 times worse. So, here the approximation factor = approximation ratio = 7.5/5 = 1.5. Note that in case of minimization problem, if we call OPT as value of optimal solution and V as value of approximate solution then V >= OPT always; i.e. V can be equal or worse (here more, being minimization problem) than OPT and thus approximation ratio for minimization problem is always greater than or equal to 1. Similarly for maximization problem, V <= OPT and thus approximation ratio is always less than or equal to 1.

Thus, one comes up with an (approximation) algorithm A such that

It solves the problem in polynomial time
One can give proof of correctness for A
It does not degrade the quality of solution too much; i.e. one can come up with the reasonable approximation ratio for A and can prove that for the general case

There exists many ways to come up with approximation algorithms and to prove that the have a reasonable approximation ratio. One such recipe is as follows.

Suppose it is the maximization problem so that the value V <= OPT where V is the value for the solution returned by any approximation algorithm. So we have to come up with a ratio k s.t. V is at most k times worse than OPT.
In order to prove that V is at most k times worse than OPT, we have to know what is OPT which is not possible because that itself will take non-polynomial time in general case.
Suppose we know some number M such that the optimal solution has maximum cost of M then OPT <= M considering all possible cases.
Thus, if we can prove that V is at most k times worse than M, it is guaranteed that V is at most k times worse than OPT as well. Because, given that V >= k*M and M >= OPT, we get V >= k*OPT
For example, if the OPT is 100, and we know that M is 120. Then suppose V is 80 (the best we can achieve with approximation algorithm) then k = V / M = 80 / 120 = 2/3 meaning we can achieve 2/3 rd of the maximum possible solution value. We note that V / OPT is 80 / 100 = 4/5 which is always more than 2/3. Thus considering the upper bound M is enough.
For minimization problem as we shall see below, we have to consider lower bound.

I came across a nice lecture by Prof. Abhiram Ranade (IIT Bombay). In this lecture, in addition to introducing the basics of approximation algorithms and defining above mentioned notions, Sir also illustrates the approximation algorithms designed for two of the NP-Complete problems. One of them (my favorite and the easier one :D) is briefly explained below.

(Note: I have tried to explain the problem and its approximation algorithm the way I understood it, so it is recommended to go through the original lecture by Ranade Sir)

Metric TSP (Traveling Salesperson Problem)

Input: A graph G with N cities, a matrix D giving distance between a pair of cities s.t. D is a metric.

A distance D measure is a metric if

D[i,i] = 0 for all i
D[i,j] = D[j,i] for all i,j (D is Symmetric)
D[i,j] <= D[i,k] + D[k,j] for all i, j and k (D follows Triangle Inequality)

Output: A cycle in G that goes through every city exactly once and has shortest length (in terms of distance values in D)

Recipe:

This will follow the recipe mentioned above. This being the minimization problem, we will come up with a lower bound. For the given graph G, suppose we know the optimal tour and we remove an edge from it. As the required output is the cycle containing all vertexes in G, the edge_removed_tour is the spanning path in G (spanning = containing all vertexes). Thus this is the spanning tree of G because it contains all vertexes and no cycle. Consider the minimum spanning tree T in G which has cost M. Now as any other spanning tree in G is as expensive as T so the spanning tree that we came up with, by removing an edge from the optimal tour can have length no less than that of MST, which we called as M. So, length of edge_removed_tour >= M.

As, length of (optimal) tour > length of edge_removed_tour (obvious), so we get

length of (optimal) tour >= M
Here, LHS is nothing but OPT and thus M is the lower bound on OPT.

So, we come up with an algorithm that has solution with value V no more than twice of M i.e. V <= 2 * M.

The algorithm and its correctness is described in the lecture. The main idea used is triangle inequality property of D being a metric.

Precision-recall trade off ~ misclassification cost

2010-01-11T21:27:00.000-08:00

While I was reading Pattern recognition and ML by Bishop, I realized a beautiful relationship between misclassification cost and precision recall. This might be well-known of course, but it's important for me as I myself realized it. :)

We know that precision-recall concept comes from Information retrieval kind-of a setting while misclassification cost comes from decision theory problems. In IR, given a set of true results i.e. T, and a set of retrieved results say R, precision is simply the fraction of true results in the set R. i.e. precision = #intersection(R,T) / #R. For example, suppose we know *all* the web pages that exist in the Web; then the set of pages which should be retrieved by a search engine are say T of size |T|. The search engine may retrieve a set R with size |R|. Then, if only 10 of these |R| pages are in T, then R's precision is 10 / |R|. Intuitively, R is more precise / exact if it has less not_required results. One may think of |R| >> |T| and that R contains all pages from T and many more extra results. Here, one says that R could "recall" everything in T but at the same time, it also made lots of mistakes. It is like, it took many trials on a puzzle and solved a puzzle 100 times correct while num_trials were 1000+. But in general if it could retrieve 10 pages which are in T, we say that it's recall is 10 / |T| i.e. out of |T| correct pages, it could retrieve 10. This also shows that there is a trade off between precision and recall because as num_trials increase, recall improves at the cost of precision.

Little off-topic but relevant: It's said that "practice makes man perfect". If we consider practice being making many trials, then as num_trials increase, due to practice one gets better results.

While I was reading the book, on page 41, I came across the misclassification cost involved in the decision theory. It says- We account for the misclassification cost because we don't want to miss on important predictions. Like in the example given for the two-class classification about whether a person is healthy or has cancer, it's important that we make no mistakes when the true class is "cancer-patient" as the consequences of these mistakes are dangerous. It also says that misclassification costs are such that we don't mind if a healthy person is predicted to be having cancer in order to not miss any patient which has cancer in reality. This is basically a trade-off between decisions. By this, we are looking for more recall even though we lose the precision.

Beautiful theory of NP-completeness

2009-12-29T21:44:00.000-08:00

I am fascinated by this. I learned about it in one of the courses here at IIT Bombay and could not stop myself from reading this wonderful book (which one of my friends suggested to me). I hope you will enjoy the noted I wrote for myself, which are at this link.

Please give any suggestions / comments / point out errors / typos / serious mistakes or anything.

Happy reading!

Ph.D. info

2009-12-10T22:31:00.000-08:00

Well, this is a collection of articles / write ups etc. which I found / got from friends while gathering information about Ph.D. Hope this will be useful for others.

You and your research - Must read

The Ph.D. experience - Again, a must read

Another nice talk

Video cutter on ubuntu

2009-10-02T09:37:00.000-07:00

Since long I was searching for the video of a marathi movie song, which I just love to listen to. I could not find it on youtube. The only way (which my friend Aashay once suggested) was to cut the song from the movie itself and make my own video for that song. He being the expert at this, it was quite easy for him to just "suggect" that! Moreover, I use nothing else than ubuntu - thus I had to find some way on my own.

Little bit of Google-ing and I had the software installed on my laptop for video cutter / editor. It is "avidemux". It seemed to be an easy tool after I had gone through the ubuntu forum post messages. But..!! I opened that movie with avidemux and it popped an error message saying - "trouble initializing audio device". I checked preferences. (Note:- After you open the movie, it won't play automatically. You have to click "play". Also, while it is being played, it won't open "preferences" window.. weird though!! But, happens!!) In preferences, I set audio device to "default" as shown in the screen shot. I don't know why and how , but it did play audio too!! This ofcourse I got after some more Google-ing.

Well.. I loved it!! My first step towards becoming the "video-editor-by-hobby" ;) :D

Btw, this is the video that I cut with it. It's a nice classical based marathi song, sung by Sawanee Shende.

Inserting into mysql table from Java code

2009-08-29T22:21:00.000-07:00

This post gives sample implementation of an interface which will insert records in a given table.

Assumptions:
1. mysql_connector driver is in build path
2. List of records is built off line.
3. Database connection is already made.

Procedure:
As shown in the snippet of code, the method takes 5 parameters, as mentioned in the comment.

/**
* @param tableName name of the table
* @param numValues number of values to insert < = num columns
* @param dataTypes their datatypes (string, int etc)
* @param columnNames column names to add values for
* @param values actual values in same sequence
*/
public void insertIntoDb(String tableName,
int numValues,
ArrayList_of_String dataTypes,
ArrayList_of_String columnNames,
ArrayList_of_ArrayList_of_String values) {

One you have a list of column names, you can easuly concatenate them to form a string say,

String col = "(col1, col2, col3)";

Similarly, for values, using numValues, you can easily form a string say,

String val = "(?, ?, ?)";

Once this is done, make a query string like:

String query = "insert into " + tableName + " " + col + " values " + val + ";";

This will form a quert string similar to following:

"insert into myTable (name, id) values (?, ?);"

Then, write a loop which will iterate through the given ArrayList of values (actual records) and using PreparedStatement interface, fill in the values for '?' in the query string. Here you will use dataTypes provided; for example, if the corresponding data type is "string", write:

cs.setString(index_of_corresponding_?, current);

else for "integer", write:

cs.setInt(index_of_corresponding_?, Integer.parseInt(current));

where "current" is the current piece of text that you can get from ArrayList of records i.e. values.

You can verify whether your final prepared statement is correctly built, using toString() method for PreparedStatement interface. It will show something like:

"objectID1: insert into myTable (name, id) values ('abc', 1);"
"objectID2: insert into myTable (name, id) values ('xyz', 2);"

Then invoke executeQuery() method of the interface PreparedStatement which will actually insert these records in the table.

Easy!! No? :)

Select k records randomly from n records in Java

2009-08-28T09:57:00.000-07:00

Assume a situation wherein you have an array of records, let them be anything varying from marks (integers), names (string), student information (user defined class object), or ANYTHING ARBITRARY which derives itself from Java's object. Suppose, you have such n records, each with an identifier say, 1 to n, such that you have then stored in array of any form in Java.

Now, suppose you want to chose k random records from this array, this is how you proceed.

Procedure:
Given n, generate a random permutation of {1, 2, ... n} as {1', 2', ... , n'} and pick first k numbers from the permuted array.
For example, say you want to pick 3 numbers randomly from 8, s.t. n = 8, k = 3.
1. {1, ... 8} ===> {2,3,1,7,6,4,8,5} ===> {2,3,1}
2. {1, ... 8} ===> {8,4,3,1,6,5,7,2} ===> {8,4,3}
Thus, once you have k such indexes, you just need to probe your original array with these indexes and get corresponding k random records.

The Java code snippet is as shown:

The output as expected is :

randomly selecting 3 out of them
1 a
7 g
5 e
randomly selecting 5 out of them
6 f
5 e
7 g
8 h
2 b

I needed this for:
I had to partition dataset containing some @1,000 records, for cross-validation. It is a method of training a classifier when you have small number of training records. In this, you will first decide how many folds of dataset you want to make, say 5-fold, then 1,000 records will be partitioned in the ration 4 : 1 such that every time 4/5 th of the data records will be used for training and remaining 1/5 th for testing (validating in this case). Thus, I needed to randomly pick 4/5 * 1000 of my records every time, for which I wrote above piece of code.

Note 1: Not listing the code, but adding image so that, whoever wants to try it, will actually type in the things, and thus they will never forget the trick! :)
Note 2: Reference - Weka software

fundoo maths! in Data mining (ofcourse! ;))

2008-10-29T21:18:00.000-07:00

See the figure above. Consider X-Y plane where some points are scattered and you are asked to find a separator between them. The black line going through the points separates them well. There is a significant property that such a line has, which is - linear separation. In the sense, the line is expressed as a "linear combination of X-Y co-ordinates".

To understand this better, consider the figure below:

Can you separate these points (filled and not-filled red circles) by something which is a "linear separator"? If you consider to draw a line so as to separate these two types of point, you won't succeed. The reason being, the points are scattered such that no linear separator can separate them. Remember, by linear, I mean something (a line in 2D i.e. X-Y plane) which is expressible as a linear combination of its co-ordinates.

In fact, the points above can be separated by a circular / elliptical shape as you might have noticed. A circle / ellipse could be drawn covering the not-filled i.e. hollow red circles in the middle and separating them from filled ones. But as one can argue, ellipse / circle is NOT A LINEAR SEPARATOR. For X-Y i.e. 2D plane, equation of circle is:-

x^2 + Y ^2 = r ^2 ................. r being the radius

and the same for ellipse is:-

x^2 / a^2 + y^2 / b^2 = 1 .............. a and b being constants.

these square terms make these shapes "non-linear" in nature.

So, the Question is - Is there a way to separate these points by a linear separator?

And the answer is as follows:

Consider an added 3rd dimension to the above figure, say Z-axis. Imagine that we shift the hollow red circles behind along the Z-axis such that all of them are now in negative Z-axis region in 3D space. Now consider and plane parallel to X-Y plane such that it separates these negative Z-axis points from those with positive Z-coefficients. Such a plane now separates hollow and filled red circles well. See figure below for better understanding.

The important thing about such a plane is - It is now a linear separator. Because any 2D object, a plane here is a linear combination of its dimensions in 3D space.

---------------------
References:

It is a major issue in classification, a topic in Data mining. Google for it. Keywords like "feature transformation, nonlinear separator / classifier" should help.

Snoring

2008-01-25T02:32:00.000-08:00

During lunch, my friend was telling about her roommate and was laughing on the fact that she snores. I also snore. Obviously I felt bad. She was indirectly laughing at me. Since childhood I have been observing my father snoring and always thought of finding out the reasons behind the same. I thought now is the time because I myself is facing the same problem. Google-d for some information and this is just the compilation of what I found.
Last few months, I have realized that I snore. I happens sometimes during deep sleep, sometimes just after I go to bed. Most of the time its winter or my throat has some problem, whenever i snore. Normally it happens that I myself awake after some time I have been snoring and then I realize that I was snoring. I also noticed that whenever I am awake after snoring, I am sleeping on my back. I have never noticed myself getting awake due to snoring, when I am sleeping on my side. All this made sense after I read Google-d results.
Main reasons of snoring are:

Throat weakness causing the throat to close during sleep
Mispositioned jaw, often caused by tension in muscles
Fat gathering in and around the throat (tonsil problems, may be)
Obstruction in the nasal passageway
Obesity

A very good article about snoring, its reasons and treatments can be found at http://www.hinduonnet.com/thehindu/mag/2004/11/07/stories/2004110700350600.htm

The reasons numbered 1 and 2 ask for sleeping at the side to prevent the tongue from blocking the throat. Thats the reason I notice myself awake after snoring when I am sleeping on my back more than when I am sleeping on my side. Reason 4 might focus on the case that during winter I notice myself snoring. Reason 5 is directly applicable to me!
Reason 1, 2 and 5 are applicable to my father as well, who snores.

As known effects of snoring can be told as restlessness, hypertension (which may lead to heart-attack in very few cases), increased day time sleep, depression. (I go through almost all of these!)

A huge and useful information can also be found at my favorite wikipedia.. http://en.wikipedia.org/wiki/Snoring

As said in Hindu article, Snoring is no longer a laughing matter. It may lead to serious disease.

carbon trading

2008-01-16T05:26:00.000-08:00

I had heard this term.. of course without knowing the meaning.. During "Applied Economics" (!!) lecture, Sir explained it.. As follows:

Lets assume for a certain area, air pollution restrictions are laid.. in a sense.. no factory should emit (say) 100 units of carbon (carbon- in a sense, pollution causing element). Lets say in that area, there are 2 companies A and B.. A's factory is controlling its emission to say 80 units of carbon.. Then they are given a sort of "carbon saver" certificate, indication of "they are carbon savers" That certificate is considered of having a value. Because, if suppose company B's factory is not able to control their emission to 100, say they emit 120 amount of carbon.. they can acquire (may be, actually BUY) this "carbon saver" certificate from company A!! sort of license.. to emit higher amount of carbon.. So that total level of carbon in that area is kept constant (120 + 80) as well as because company A has saved carbon, they are given reward while company B which could not save carbon, is - in a sense - penalized.

Cool information.. isn't it?