Today is Thursday
October 19, 2017

Category: Uncategorized

May 3, 2017

Pepsodent to Patanjali transition of the society

by viggy — Categories: social, Uncategorized — Tags: , , , Leave a comment

Can you remember the era of toothpastes ad with actors dressed as doctors certifying that they would recommend this or that toothpaste to their patients. Do you remember the claim made by almost every toothpaste that 99% of members of some Dentist’s association has suggested them as the best toothpaste, etc. While the Indian society was changing to be a consumerist society, we accepted any garble from the idiot box and bought the products. From toothpaste to toothbrush, soap to fairness cream, everything was sold based on how much it was marketed and how much it made the buyer have a feel good factor. None of the claims were questioned, it was never verified, just broadcasted on all mediums.
They said the toothpaste had salt, we thought it must be good. They said it had neem, we accepted it. The put logos of all the certifications that product had obtained. What those certifications meant, why it mattered was never asked, never answered. As long as the product had the feel good factor, it was bought. Ask anyone the difference between one or other toothpaste, what mattered was only the offers it had or whether it had some taste or smell which they preferred, nothing to do with how it helped the teeth.
Then comes Patanjali, which also said that their toothpaste also had neem but now with certification of being nationalist, being swadeshi. People accepted it also. They again never questioned anything. While the marketting team of big FMCG companies are trying to understand why people are accepting Patanajali, where as they had themselves trained the society to accept anything with a feel good factor, not with any value in it.
Question still remains as to why do we need a toothpaste, why do we need a soap? What properties should the soap have? While we have never asked this question, we can see how a similar ignorance plagues our politics also.

In 1990s, while we were told to accept globalization as the answer to all our problems, we hailed the ones who were called as the architect of liberalized India, PV Narsimha Rao to Manmohan Singh. Everyone accepted that this was the way forward for the nation to leave the clutches of poverty. A grand vision of development, with roads, flyovers and skyscrapers was shown to everyone. Everyone accepted it, they had known of one or other who had benefitted from it. The cousin abroad, far relative who had a white collared job, industrialists in the town, everyone seemed to be benefiting and there was ‘development’ everywhere. No one was interested in trying to understand the fundamentals of the politics they were propagating. This idea of development made them feel better, brought a hope of a change which was more easy to accept.
Than came the nationalist forces which had been brewing silently in the country side making their base stronger. When they bought forward the same idea of development with nationalist way of implementing it, people have accepted it in the same way they shifted from Pepsodent to Patanjali. After all if the feel good factor is the criteria, all it takes is to show a vision which is more tempting than the current one.

If we accept this analogy, the natural question to ask is what would be the next direction for the society. In case of toothpastes, my guess is that very soon we will have local gurus and babas coming up with their own version of toothpastes. While the nationalist layer peels off from the society, society may well be painted with a layer of regionalism. As the efforts of nationalism fails, it may just be too easy to make the society to blame one region to other as the cause for all its problems. Some of the states already have a very good established regional party which would gladly accept this opportunity. Also the glaring economic divide between the south and the north will also make it possible for people to pass the blame on each other. Close to a decade ago, a friend of mine had suggested that India should get disintegrate into all the different states to solve its problems, I had laughed at him as an absurd idea. I wont be surprised if our political parties use such absurd idea to try their luck to political power.

April 22, 2017

Explanation of vectorized form of the for loop calculation of gradient descent in Exercise 1 in ML class in couresa

by viggy — Categories: Uncategorized — Tags: , , , Leave a comment

Source: Storing this for future reference

If you are wondering how the seemingly complex looking for loop can be vectorized and cramped into a single one line expression, then please read on. The vectorized form is:

theta = theta – (alpha/m) * (X’ * (X * theta – y))

Given below is a detailed explanation for how we arrive at this vectorized expression using gradient descent algorithm:

This is the gradient descent algorithm to fine tune the value of θ: enter image description here

Assume that the following values of X, y and θ are given:

m = number of training examples
n = number of features + 1


m = 5 (training examples)
n = 4 (features+1)
X = m x n matrix
y = m x 1 vector matrix
θ = n x 1 vector matrix
xi is the ith training example
xj is the jth feature in a given training example


h(x) = ([X] * [θ]) (m x 1 matrix of predicted values for our training set)
h(x)-y = ([X] * [θ] – [y]) (m x 1 matrix of Errors in our predictions)

whole objective of machine learning is to minimize Errors in predictions. Based on the above corollary, our Errors matrix is m x 1 vector matrix as follows:

To calculate new value of θj, we have to get a summation of all errors (m rows) multiplied by jth feature value of the training set X. That is, take all the values in E, individually multiply them with jth feature of the corresponding training example, and add them all together. This will help us in getting the new (and hopefully better) value of θj. Repeat this process for all j or the number of features. In matrix form, this can be written as:

This can be simplified as:

[E]’ x [X] will give us a row vector matrix, since E’ is 1 x m matrix and X is m x n matrix. But we are interested in getting a column matrix, hence we transpose the resultant matrix.

More succinctly, it can be written as:

Since (A * B)’ = (B’ * A’), and A” = A, we can also write the above as

This is the original expression we started out with:

theta = theta – (alpha/m) * (X’ * (X * theta – y))

April 15, 2017

The book of 0 and 1

by viggy — Categories: Uncategorized — Tags: , , Leave a comment

Imagine I had a nice fat and fancy notebook with great binding completely empty. I ask a young child who knows how to write 0 and 1 beautifully to fill that notebook completely, line by line, page by page with 0s and 1s. I give full liberty to the child to write it any sequence. At the end of the week, after 7 long days of writing the child returns and gives back the book very happy with the accomplishment. Now in a way of appreciation, I give the child a golden pen, which writes with golden ink and ask it to write a small sequence of 0 and 1 on the binding of the book.
Finally my book of 0 and 1 is ready. If at this moment, I would have come to you and tell you to buy the book from me for a small amount of money. Unless you are really feeling generous or pity on me, you would of course tell me that the book is worth nothing.

Now at this stage, I ask another random person, to choose any page of the book and any sequence of 0 and 1 and try to feed it into any of the machine he/she poses. What might happen. The machine understands the sequence and either return some gibberish or return something which we can understand. If the machine returns gibberish, I ask the person to change the machine and try again till it returns something more meaningful.

I do the same with many different people. Each choose their own random sequence from the book, their own machine to interpret it and then try to get something meaningful out of it what they can understand.

Now I have a book of some value as people are able to make some meaning out of it. Wonderful. I start a game, asking people to find the longest sequence in the book which makes sense. The game goes viral, there are now many machines which are able to make meaningful statements out of the sequence of the book. Very soon, we design machines specifically so that it returns meaningful statements just for the sequence of the book.
This book is the book of religion.

July 4, 2016

After the Dhaka cafe attack

by viggy — Categories: UncategorizedLeave a comment

ना खुरान कि आयथे आती हॆं, ना गीता का श्लोक आतॆं हॆं
हमें तो बस गालिब के प्यार के दो शायरी आतॆ हॆं
लेकिन डर नहीं हॆं मुझे तेरी बदुंक की गोली से
क्योंकी मुझे ये भी पता हॆ की ना खुरान कि आयथें ना गीता का श्लोक तुझे समझ आतें हॆं
मॆं नहीं कोसता तेरी झझ्बातों को
क्योंकी हम मॆं से ही कोई हॆं वो खाफ़िर जीसने बीस साल की उम्र मॆं ही
किताबों के बदले बंदुक थमा दी तेरी हाथों में
और तेरी मसुमियत को नफ़रत में बदल दिया!

तुने किसी को मार दिया
तुझे किसी ने मार दिया
ना उनको अपनी जीन्दगी मीली
ना तुझे अपनी जन्नत मीली
बस नफ़रत फ़ॆलाने वालों के ईरादें बुल्लंद हो गए
और प्यार करने वालॊं को जीने का एक और मकसद मील गया!

Wrote this after the Dhaka Cafe attack on July 2nd 2016 which allegedly was carried out by boys of 20-21 years of age and who selectively killed foreigners who were not able to recite Quran.

May 13, 2016

Nature is no mother

by viggy — Categories: UncategorizedLeave a comment

‘Oh the things that mother nature provide us’, ‘Look at how cruel humans are who destroy mother nature’, ‘Humans need to be in harmony with nature’. Often these are very common sentences we hear. People love to fantasize nature as if it is an active force taking care of the living beings. It isnt. Existence of nature as we understand is just a coincidence and the series of natural events happen with no purpose attached to it. As we understand, the existence of some cancerous cell in one’s body is just result of series of events prior to it as an avalanche on top of an mountain.

Some tectonic plate movements can cause a Tsunami killing thousands of humans and crores of organisms. Similarly some wandering meteoroid can hit the planet earth causing complete extinction of dinosaurs. There is no meaning to such events, they happen as a chain of cause and effects.

As humans, our quest throughout the evolution has been to understand this cause and effect and tame the nature to suit our survival. This could either be the discovery of fire which was used then to burn forests to create plainlands for agriculture or to domesticate animals as source of food and using them as cattle for grazing or making them do our heavy lifting jobs.

This taming of nature is what has differentiated humans as against other species and this taming of nature is what will decide how far our survival can go on. Can a stray asteroid destroy the human race or can we also tame the nature so that we can even use the asteroid for our own purposes.

As humans, what we need from nature for survival, we need to extract mercilessly. Animals, plants, mountains, oceans, planets, stars are all at disposal, it is upto us how we can utilize it. It will be foolish to not do so as we never know which other stray effect comes to destroy us and when it comes it is going to be merciless.

Where does it leave fellow human beings?

February 6, 2016

Cloaks of activism

by viggy — Categories: social, Uncategorized — Tags: , , , , , Leave a comment

Communist, Anarchist, Feminist, Anti-castist, Free Knowledge Hacktivist, Free Speech Advocate, Democrat, Atheist, Anti-racist, LGBT Activist, environmentalist, anti-imperialist and so many other cloaks that a liberal can wear. Ofcourse there are the opposite cloaks for a conservative person also. Amongst so many cloaks, which is the best fit for a person. A person can either have choice of renting one of the cloak for sometime or be forced to choose one of these based on his/her circumstances. Can a boy really wear the cloak of feminism, can he really internalize what a girl has to go through just due to an accident of her being born as a girl? Can an upper-caste really wear the anti-castist cloak without having ever faced with any discrimination or having to be aware of his/her caste identity in a public place? Ofcourse they can rent these cloaks for sometime, lending their voices in support but they can never represent the truly oppressed or shape their movement.

Ambedkar’s commandments to the oppressed, ‘Educate, Agitate, Organize’ represents very well as the direction for the people from the oppressed community. However, for the people from the non-oppressed who would like to support the oppressed and stand with them, in my opinion, the commandment should be to support the leaders from the oppressed rather than taking the role of leader themselves. And this ofcourse is the most challenging aspect for such a person who has always been habituated to lead and direct the decision process.

These lines from the ‘Dalit Marxist Manifesto‘ beautifully present the behaviour of an upper-caste in the dalit movement. “Such attitude is part of growing up upper-caste in India, they just can’t imagine how to look at the world without them being at the center of it, they can’t look at a lower-caste person except from above. Being progressive, radical, revolutionary are not just products of only honest, idealist and painstaking study and analysis of the world but also a resurfacing of the old theme of Higher-hood now denied to them, or they live in denial of, adjusted on a new surface.”

Gandhi and Ambedkar’s tussle in various aspects represents this very well. Gandhi did wear the cloak of being an anti-castist but he could never internalize it and so much that he always gave much more priority to freedom struggle against the Britishers than fighting the caste system within his own society. For Ambedkar, the freedom struggle against Britishers had very little significance as compared to the freedom struggle against the castist society.

This tussle between various cloaks that a liberal wears will always exist. Hence you see the feminist not talking about caste as an issue or even feminist not talking about LGBT rights as much. The environmentalists may talk about Tribal rights but will not talk about farmer suicides. Communists talk about exploitations in work place with their friends in living rooms while asking their wives to prepare food for the comrades.

Ofcourse this does not undermine their contributions to their own activism. Gandhi’s contribution to freedom struggle cannot be challenged and we can only respect him for his sacrifices and appreciate his bravery. This discussion of cloaks is still limited to people who choose to act, where as there are millions who continue to choose to be naked and whose silence allows oppression to be continued. Hence this tussle in whatever way still leads to something better for the society as against doing nothing.

October 20, 2015

Trek to beautiful grasslands of Kudremukh

Beautiful Grasslands of Kudremukh

Beautiful Grasslands of Kudremukh

After looking at the beautiful images of hills with stretches of grasslands, I always wanted to trek to Kudremukh. Me, Farrah and her colleague, Janaki booked our seats through TripHippie which had a fixed departure available for Oct 17th-18th weekend for Rs. 3500/- per person. The organizer of the trek was Nature Admire, which is run by Dev Balaji. I had met him couple of days before to talk about collaboration between TripHippie and Nature Admire. He has a vast experience and network in this domain and there is lot to learn for us from him. After booking through TripHippie, we received the detailed itinerary from Dev. We left for Kudremukh from Bangalore on Friday night around 10:00 pm in a tempo traveller. We reached Balegal by morning 7:00 am on Saturday. From here, we went to Mullodi in Jeep ride of 8 kms that I will never forget. It was only in videos that I had seen driving in such roads but this route will ensure that all the bones in your body is shook. The drivers here are skilful and had full control over the vehicle during the drive. After reaching the base camp at around 8:00 am, we quickly freshened ourselves and had breakfast. We then packed food items that we had brought from Bangalore along with lunch that was provided to us to carry along with other necessary items like torch, sprays and water bottles. Though as per the itinerary, we were supposed to be back to base camp by 6:00 pm and hence torch would not have been necessary, it later proved to be very helpful. With this, by 9:00 am, we were all set to start the trek.

One of the many streams during the trek with crystal clear water

One of the many streams during the trek with crystal clear water

We were quickly briefed by Balaji about the trek and do’s and don’ts during the trek. We had a local guide with us, who would lead the way for our group. There were more than 100 other trekkers who had come along with various other groups, so the trek route was crowded. The trek is 8 kms to the peak, with close to 4 kms of trek where you have steep climbing which is very tiring. However there are lot of places where you cross the streams with water so pure that you can just drink it directly. Again this was a first experience for me to just drink water directly from the stream.

View of the Kudremukh Peak

View of the Kudremukh Peak

It was very sunny and we were soon tired even just after 1 km of trekking. Balaji kept inspiring us that we will soon get comfortable once we get acclimatized to the new environment. Till now, we hadnt seen the actual peak that we had to reach. After around 1.5 kms of trek, Balaji showed us the Kudremukh Peak.

Image from Wikipedia explaining the name of the peak, kudremukh

Image from Wikipedia explaining the name of the peak, kudremukh

Kudremukh in Kannada means horse-face. Though a little difficult for me to imagine, others seem to get it easily imagine the peak as a horse face. The sight of the peak kept inspiring us apart from the sight of lush green grasslands. We were expecting rain and hence wanted to complete the trek before evening after which it would be really difficult to trek back if it rains. The local guide kept trekking easily and waiting for us to catch up with him. Our trekking team had people with varied experience with us being the complete newbies and a couple who had even visited Annapurna Base camp in Nepal. So they kept guiding us and helping us. Balaji kept reminding us to take some short breaks and have bananas, nuts and chocolates that we had brought along with us.

Our Trekking group, fully charged while climbing a steep path

Our Trekking group, fully charged while climbing a steep path

By around 1:00 pm, we were completely exhausted with close to 2 kms still to go. We had maintained a constant pace of 25-30 mins per km. Balaji stayed at the end encouraging us to keep moving forward. Looking back at the distance we had covered also kept pushing to achieve the final 2 kms to reach the peak. Also, the cloud had started becoming more dense and we expecting rain, we kept moving forward. By this time, a part of our group had already gone quite ahead.

Almost at the peak, we were able to see the clouds below us

Almost at the peak, we were able to see the clouds below us

The last 500m is almost plain and we were thrilled for having reached the peak. We were amidst the clouds and it was a great feeling of achievement and satisfaction.

Clouds at the peak

Clouds at the peak

A beautiful feeling to be amidst the clouds

A beautiful feeling to be amidst the clouds

We finally reached the peak at 2:00 pm after 5 and half hours of trek where others from our group were already waiting for us and taking rest.

TripHippie and NatureAdmire team at the peak of Kudremukh

TripHippie and NatureAdmire team at the peak of Kudremukh

From the peak where clouds were below us

From the peak where clouds were below us

We had the puliogre that we had got from base camp as lunch along with some oranges that we had carried from Bangalore. We also took a small nap of 10 mins to rest our legs. We then started our downhill trek at around 3:00 pm. Though we had expected it to be difficult due to some steep paths, we soon understood it was much easier to have climbed than to go down. Our joints and toes soon started to feel the pressure. We had thought that though difficult, we would still be faster in climbing downhill and hence would reach the base camp by around 7:00 pm with 4 hours to trek downhill. However one of our group mates sprained her leg very soon, just about 2 kms from the peak and it slowed us down. By around 6, were still at least 5 kms far from the base camp and it was already getting dark. This was where the torches that we had carried along helped us and we continued trekking downhill with the help of torch light. We had Balaji who had waited for us before the start of the dense forest. We would have definitely got lost had he not waited for us there. He kept assuring us that we keep moving slowly and carefully. We did not want anyone else to get hurt by twisting or falling. The steam which had been very welcoming while climbing up were now numbing cold. We still had to cross them and since we did not want to risk slipping on stones, we decided to walk through the numbing cold stream. It was our first experience trekking in evening through a dark forest and we were all scared except for Balaji who knew the route very well.

By around 9:00 pm, completely exhausted, we reached the basecamp and joined other members from our group. The dinner was ready along with the tents and sleeping bags for our night stay. Completely refreshed with the experience of the trek, we enjoyed the dinner and went to sleep in the cosy.

The next day, Sunday, we got up at around 9(we were supposed to get up by 6 :)) and visited the Somavathi falls. Since we were already late, we just enjoyed in the falls for an hour or so and rushed back to basecamp to pack our bags. Since most of us were exhausted from the trek, we had decided to go back to Balegal in a Jeep rather than trek back to Balegal which was the plan as per the itinerary. We also decided to skip visiting the Horanadu Temple and just stop at the Tea Estate for a while.

Beautiful Tea Estate

Beautiful Tea Estate

Valley between the tea estate

Valley between the tea estate

After this, we reached Kottigehara where we stopped for lunch. We enjoyed the Neer dosa being served there in a local hotel. Though we were expecting our body to ache due to the heavy trekking that we had done the previous day, surprisingly we did not have any such issues. Relieved we enjoyed the movies that was being played in the cab. We reached back to Bangalore at around 8:30 and as always were welcome with a huge traffic at Malleshwaram Maramma Circle. From the crystal clear water from streams and grasslands surrounding us, we were now back amidst the concrete structures of Bangalore.

Overall the trek to Kudremukh was a great experience and my first through TripHippie. I notice close to 150 more packages that are currently hosted in TripHippie which I need to experience.

October 12, 2015

Spectrum of oppression

by viggy — Categories: social, Uncategorized — Tags: , 1 Comment

I recently came across this article where Arundathi Roy mentions how she sees Gender as a spectrum. It made me wonder about oppression in general and how oppression also happens over a spectrum. All the “isms” look at oppression in a very closed boundary and all the “ists” conveniently choose to ignore the oppression when it does not suit them. Communists for long have been criticized for ignoring the caste oppression and choosing to talk only about class oppressions. They themselves again blame feminists for being too narrow minded and not fighting to uplift the poor. Atheists again blame others to not look at how religion is being used to create unrest and divide amongst people and why feminists/communists are not fighting it directly.

Amongst all these is an act of oppression and it comes in various shades. Oppressor does not choose his style of oppression, he chooses whom to oppress and for what benefit/profit. Then why is it that all the “ists” choose their oppression to fight against. How is it that we choose to ignore one form of oppression on one human to other while we choose to fight another form of oppression.

Ofcourse choosing to fight atleast one form of oppression is better than becoming an oppressor ourselves. Because on the other side, you have people who justify one form of oppression by giving example of another form of oppression. Nature is definitely very harsh, cruel and merciless. It treats living beings as equal as dust but that no way justifies what we do to each other. Human society has evolved fighting the harshness of nature, right from warm clothes we wear to food we cultivate and store.

August 23, 2015

Running GeoWave from command line for HBase options

by viggy — Categories: Uncategorized — Tags: , , , , , , Leave a comment

GeoWave provides GeoWaveMain class which can be used to run GeoWave from command line. For regular users GeoWave provides a wrapper script for the same which would be installed in case you are using the rpm or geowave through other packages. In my case, since I was running in a development environment, I am directly using the GeoWaveMain class with options from the command line. Here I document the results that I got while running the hbase related commands from the GeoWaveMain class from command line. The GeoWave documentation talks about running the commands from command line here. I used the same and the existing tests in GeoWave source code to deduce the commands.

In the following examples, I am running a local hbase and zookeeper instance.

First, to run the commands, we need to generate the tools jar using the command in the deploy source directory,

mvn package -P geowave-tools-singlejar  -Dmaven.test.skip=true

I am skipping tests as some of the tests fail in my local setup due to various reason and I do not want that to be reason for the maven build to fail.

Next I also generate the hbase datastore jar using the following mvn build command

mvn package -P hbase-container-singlejar -Dmaven.test.skip=true -Dfindbugs.onlyAnalyze=true

Here I am also asking FindBugs just to analyze as currently FindBugs fails the build due to the autogenerated code in FilterProtos, which is used for Filters in HBase. I have not yet been able to find a solution to fix the same and general solution was to skip findbugs for the build process using this switch. This would still generate the FindBugs.xml which can be analysed later using FindBugs GUI separately.

The last two command should generate geowave-deploy-0.8.8-SNAPSHOT-tools.jar and geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar in the target directory. Since we also use geotools-vector and gpx as formats in following commands, we need to copy their jar also here or refer to them in the classpath. In our case, we just copy the jar to the target directory.

Assuming that user’s current working directory is target, following command is used to run localhbaseingest:

java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-vector-0.8.8-SNAPSHOT.jar mil.nga.giat.geowave.core.cli.GeoWaveMain -localhbaseingest -z localhost:2181 -n geowave -f geotools-vector -b ../../test/data/hail_test_case/hail-box-temporal-filter.shp

To get hdfshbaseingest to work, you need to start Yarn from your hadoop installation. In this, we also use “gpx” format rather than “geotools-vector”(I need to understand why?). Hence we use the jar of gpx format and copy the same in our current working directory. Also this command expects that the directory gpx is already exists in HDFS file system. To create this, you can run the following command from your hadoop installation:

<Hadoop-Installation-Dir>/bin/hadoop fs -mkdir /gpx

Then we run the following command:

java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-gpx-0.8.8-SNAPSHOT.jar: mil.nga.giat.geowave.core.cli.GeoWaveMain -hdfshbaseingest -z localhost:2181 -n geowave -f gpx -b ../../test/data/hail_test_case/hail-box-temporal-filter.shp -hdfs localhost:9000 -hdfsbase “/” -jobtracker “localhost:8032”

-hdfs is the hostname:port of the hadoop installation

-hdfsbase is the parent directory in which we want to ingest

-jobtracker is the hostname:port of the yarn installation.

Currently for hdfshbasestage, I am getting the following error which need to be fixed:

➜  target git:(GEOWAVE-406) ✗ java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-gpx-0.8.8-SNAPSHOT-tools.jar:. mil.nga.giat.geowave.core.cli.GeoWaveMain -hdfshbasestage -b ~/workspace/geowave/extensions/formats/gpx/src/test/resources/ -hdfs localhost:9000 -hdfsbase “/gpx/” -f gpx
2015-08-23 04:41:27,085 FATAL [main] ingest.AbstractIngestHBaseCommandLineDriver ( – Error parsing plugins
java.lang.IllegalArgumentException: Unable to find SPI plugin provider for ingest format ‘gpx’
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.getPluginProviders(
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.applyArguments(
at mil.nga.giat.geowave.core.cli.GeoWaveMain.main(

Similarly for posthbasestage I am getting the same error:

➜  target git:(GEOWAVE-406) ✗ java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-gpx-0.8.8-SNAPSHOT-tools.jar:. mil.nga.giat.geowave.core.cli.GeoWaveMain -posthbasestage -hdfs localhost:9000 -hdfsbase “/gpx/” -f gpx  -z localhost:2181 -n geowave -jobtracker “localhost:8032”
2015-08-23 04:44:16,261 FATAL [main] ingest.AbstractIngestHBaseCommandLineDriver ( – Error parsing plugins
java.lang.IllegalArgumentException: Unable to find SPI plugin provider for ingest format ‘gpx’
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.getPluginProviders(
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.applyArguments(
at mil.nga.giat.geowave.core.cli.GeoWaveMain.main(

August 23, 2015

Using Google’s Protocol Buffer library to write GeoWave Filters for HBase datastore

by viggy — Categories: Uncategorized — Tags: , , , , Leave a comment

Accumulo provides Iterators which can be run on Tablet Servers as Filters during Scan. GeoWave uses this in form of local Client Filters and Distributable Filters which run on Tablet Servers when any scan is performed. As part of adding support for HBase, I needed to implement these filters in HBase. I have currently implemented two Filters, SingleEntryFilter and CqlHBaseQueryFilter which are counterparts for SingleEntryFilterIterator and CqlQueryFilterIterator in Accumulo.

Hbase makes use of Google’s Protocol Buffer to serialize data on client side and send it across to tablet servers. In this blog, I explain how I used the protobuf-java library to write the SingleEntryFilter for HBase in GeoWave.

Protobuf auto generates part of the code by using .proto file and its own code generator. The .proto file needs to contain the information about the arguments your class accepts in its constructor, which package the class needs to be generated in, etc. The arguments that class supports needs to be serializable. Since in our case, we are migrating from iterators, all the needed data are expected to be serializable as they need to be serialized even in case of iterators for Accumulo. I created a ‘protobuf’ directory inside local source directory ‘extensions/datastores/hbase/src/main/’ in GeoWave source code and in that created the following SingleEntryFilters.proto file.


option java_package = “mil.nga.giat.geowave.datastore.hbase.query.generated”;
option java_outer_classname = “FilterProtos”;
option java_generic_services = true;
option java_generate_equals_and_hash = true;
option optimize_for = SPEED;

message SingleEntryFilter {
required bytes adapterId = 1;
required bytes dataId = 2;


Now to generate the classes using protobuf, we need to install protobuf compiler on the machine. You can download the compiler from here.The README.txt given along with the compiler is quite explanatory for installing it.

After successful installation, by default the protoc compiler executable would be in src directory. Go to the source directory in which you want the generated package to be added. In my case, it was in <geowave-src-directory>/extensions/datastores/hbase/src/main/ .

Now, you can the following command.

<path-to-protoc-installation-dir>/src/protoc -I=. –java_out=java/ protobuf/SingleEntryFilters.proto

protobuf/SingleEntryFilter.proto is the path to the .proto file from your current directory.


This generated the necessary FilterProtos class. Now we need to create the SingleEntryFilter class. We use the FilterBase class provided by HBase to create new custom classes. Lars George’s book, HBase: The Definitive Guide  explains developing custom Filters for hbase and I used the example shared in the github repo for the book to develop Custom Filter for developing SingleEntryFilter.

It was only through that example that I came to know that I need to also implement toByteArray and parseFrom methods in the Custom Filter. Later I also found in HBase log that parseFrom method generates a DeserializationException which informs user about extending it in derived class.