My pensieve - Harder you try, luckier you get

September 20, 2016

Blackfire in 15 minutes helped me reduce my site load by almost 50%

by viggy — Categories: Tech, Triphippie — Tags: blackfire, pageload, php, prestashop, profiler, tech, triphippie — Leave a comment

Having been working on Triphippie, an adventure travel marketplace for almost one year now, its pageload time was always one of our biggest issue. We have used Prestashop for Triphippie as it helped us to go live with a complete e-commerce site in less than a week. We did not concentrate on pageload time early on as we were more focussed on improving the product pages and overall features. Due to this negligence, the page load time was around 12-14s which is pathetic, especially for an e-commerce site. However in last 2 weeks, I was forced to work on improving its pageload time. First thing that we worked on was to migrate our images to Cloudfront. We did this successfully and reduced our page load time to around 8s-9s. This was still not good.

It was clear that our server was just taking too much of time to send the response, our first byte response was still around 6.5s. It was time to use a profiler and understand what was causing this. My initial guess was that this was an inherent issue of Prestashop as it depended heavily on data stored in database. To select a profiler, coming from a Java background, my clear intuition was that I needed something which will not force me to modify code unless I wanted some very specific things. I looked around and then came across Blackfire. The main thing I noticed in its documentation is that it can used without modifying the code at all. This suited me and I immediately signed in for the free premium plan for trail.

With the registration, it took just less than 10 mins to get the Agent and PHP Probe installed on the Amazon Linux EC2 instance. I used the Google Chrome Companion of Blackfire to run the profiler. In the first run itself, I noticed that one of the module of Prestashop which we had enabled on the homepage was executing a SQL statement which took 1.1s to execute. And there was another SQL statement which took 600ms to execute. Blackfire has very intuitive UI to analyze profile runs and this makes it very easy to pinpoint issues immediately. Apart from this, most of the other time consuming part were the Smarty Template compilation which was expected.

Pageload time from Pingdom

I just disabled the module which was causing the first SQL statement and to my surprise, my page load time decreases to 4.6s, it just reduced by almost 50% of the previous time. This was probably the most efficient 15 mins that I had ever spent on improving the site. Its really great that I can now just work on improving the site while being able to very easily get the profiler run output on my test machines and make improvement based on the results.

July 4, 2016

After the Dhaka cafe attack

by viggy — Categories: Uncategorized — Leave a comment

ना खुरान कि आयथे आती हॆं, ना गीता का श्लोक आतॆं हॆं
हमें तो बस गालिब के प्यार के दो शायरी आतॆ हॆं
लेकिन डर नहीं हॆं मुझे तेरी बदुंक की गोली से
क्योंकी मुझे ये भी पता हॆ की ना खुरान कि आयथें ना गीता का श्लोक तुझे समझ आतें हॆं
मॆं नहीं कोसता तेरी झझ्बातों को
क्योंकी हम मॆं से ही कोई हॆं वो खाफ़िर जीसने बीस साल की उम्र मॆं ही
किताबों के बदले बंदुक थमा दी तेरी हाथों में
और तेरी मसुमियत को नफ़रत में बदल दिया!

तुने किसी को मार दिया
तुझे किसी ने मार दिया
ना उनको अपनी जीन्दगी मीली
ना तुझे अपनी जन्नत मीली
बस नफ़रत फ़ॆलाने वालों के ईरादें बुल्लंद हो गए
और प्यार करने वालॊं को जीने का एक और मकसद मील गया!

Wrote this after the Dhaka Cafe attack on July 2nd 2016 which allegedly was carried out by boys of 20-21 years of age and who selectively killed foreigners who were not able to recite Quran.

May 13, 2016

Nature is no mother

by viggy — Categories: Uncategorized — Leave a comment

‘Oh the things that mother nature provide us’, ‘Look at how cruel humans are who destroy mother nature’, ‘Humans need to be in harmony with nature’. Often these are very common sentences we hear. People love to fantasize nature as if it is an active force taking care of the living beings. It isnt. Existence of nature as we understand is just a coincidence and the series of natural events happen with no purpose attached to it. As we understand, the existence of some cancerous cell in one’s body is just result of series of events prior to it as an avalanche on top of an mountain.

Some tectonic plate movements can cause a Tsunami killing thousands of humans and crores of organisms. Similarly some wandering meteoroid can hit the planet earth causing complete extinction of dinosaurs. There is no meaning to such events, they happen as a chain of cause and effects.

As humans, our quest throughout the evolution has been to understand this cause and effect and tame the nature to suit our survival. This could either be the discovery of fire which was used then to burn forests to create plainlands for agriculture or to domesticate animals as source of food and using them as cattle for grazing or making them do our heavy lifting jobs.

This taming of nature is what has differentiated humans as against other species and this taming of nature is what will decide how far our survival can go on. Can a stray asteroid destroy the human race or can we also tame the nature so that we can even use the asteroid for our own purposes.

As humans, what we need from nature for survival, we need to extract mercilessly. Animals, plants, mountains, oceans, planets, stars are all at disposal, it is upto us how we can utilize it. It will be foolish to not do so as we never know which other stray effect comes to destroy us and when it comes it is going to be merciless.

Where does it leave fellow human beings?

February 6, 2016

Cloaks of activism

by viggy — Categories: social, Uncategorized — Tags: activism, ambedkar, cloaks, gandhi, liberal, Uncategorized — Leave a comment

Communist, Anarchist, Feminist, Anti-castist, Free Knowledge Hacktivist, Free Speech Advocate, Democrat, Atheist, Anti-racist, LGBT Activist, environmentalist, anti-imperialist and so many other cloaks that a liberal can wear. Ofcourse there are the opposite cloaks for a conservative person also. Amongst so many cloaks, which is the best fit for a person. A person can either have choice of renting one of the cloak for sometime or be forced to choose one of these based on his/her circumstances. Can a boy really wear the cloak of feminism, can he really internalize what a girl has to go through just due to an accident of her being born as a girl? Can an upper-caste really wear the anti-castist cloak without having ever faced with any discrimination or having to be aware of his/her caste identity in a public place? Ofcourse they can rent these cloaks for sometime, lending their voices in support but they can never represent the truly oppressed or shape their movement.

Ambedkar’s commandments to the oppressed, ‘Educate, Agitate, Organize’ represents very well as the direction for the people from the oppressed community. However, for the people from the non-oppressed who would like to support the oppressed and stand with them, in my opinion, the commandment should be to support the leaders from the oppressed rather than taking the role of leader themselves. And this ofcourse is the most challenging aspect for such a person who has always been habituated to lead and direct the decision process.

These lines from the ‘Dalit Marxist Manifesto‘ beautifully present the behaviour of an upper-caste in the dalit movement. “Such attitude is part of growing up upper-caste in India, they just can’t imagine how to look at the world without them being at the center of it, they can’t look at a lower-caste person except from above. Being progressive, radical, revolutionary are not just products of only honest, idealist and painstaking study and analysis of the world but also a resurfacing of the old theme of Higher-hood now denied to them, or they live in denial of, adjusted on a new surface.”

Gandhi and Ambedkar’s tussle in various aspects represents this very well. Gandhi did wear the cloak of being an anti-castist but he could never internalize it and so much that he always gave much more priority to freedom struggle against the Britishers than fighting the caste system within his own society. For Ambedkar, the freedom struggle against Britishers had very little significance as compared to the freedom struggle against the castist society.

This tussle between various cloaks that a liberal wears will always exist. Hence you see the feminist not talking about caste as an issue or even feminist not talking about LGBT rights as much. The environmentalists may talk about Tribal rights but will not talk about farmer suicides. Communists talk about exploitations in work place with their friends in living rooms while asking their wives to prepare food for the comrades.

Ofcourse this does not undermine their contributions to their own activism. Gandhi’s contribution to freedom struggle cannot be challenged and we can only respect him for his sacrifices and appreciate his bravery. This discussion of cloaks is still limited to people who choose to act, where as there are millions who continue to choose to be naked and whose silence allows oppression to be continued. Hence this tussle in whatever way still leads to something better for the society as against doing nothing.

December 30, 2015

Let us not ignore GoogleWebLight, another walled garden in the name of faster web content

by viggy — Categories: internet, social — Tags: basics, free, google, weblight — Leave a comment

Free Basics is evil but let us not ignore GoogleWebLight. It is a new firewall to the internet, one which centralizes the entry to the internet, tries to capture all the data with user privacy completely sacrificed. All this with a sugar coating of fast access to web content.
On the evil side of the digital world, now there are two gaints competing against each other on who can raise to the supreme position of ‘read-everything, know-all’ gaint. The end goal is very simple, understanding the user very personally for successful targeted advertisement. The battle that netizens/hacktivists are fighting against Free Basics from Facebook to protect NetNeutrality has received lot of attention and clearly we will find its results in due course. The battle is intense with Facebook having put gone full force to the extent that it even started attacking the NetNeutrality supporters directly in its full page print media ads about Free Basics.

However, the other gaint has been playing the more silent battle and winning quiet remarkably. Seriously, in terms of capturing the market and looking at the long term goals, Facebook can easily learn some major lessons from Google. Let us look at how an average internet user browses internet in India. There are lot of studies which show that smartphones are the basic device with which majority of internet users in India browse internet. This market is completely captured by the Android Operating System controlled by Google. In this, Chrome and the default Android Browser seem to the major browser that the user uses to browse, again both of which are controlled by Google.

Now since this was not enough, Google has added the feature in Chrome which redirects user to its own site, GoogleWebLight if it detects that the user has a slow internet and then loads the webpage within GoogleWebLight. This it suggests is to improve the speed with which the site is loaded. It naturally tries to remove the Javascript and CSS, may be some heavy images. Whether the site loads faster, maybe debatable, the fact that now Google is now redirecting all the users to its own site, instead of sending the user directly to the site which the user clicked seems completely against privacy of the user, breaks the internet as all the hyperlinks are now modified to redirect user to GoogleWebLight and also kind of acts like a firewall to the internet.

There is no doubt that large section of internet users in India are using internet through a slow connection. Hence there is need for a solution which helps to load the content faster and also such that their data usage is limited. But is redirecting the user to some other site such that the site which user intended to visit is embedded in the new site is definitely not the best design, infact it seems to be a very bad design. One simpler suggestion could just have been using a Chrome browser addons which disables loading of certain type of web elements when loading a site on the browser. Another way could have been developing a standard which encourages website developers to also have a site for slow internet users.

From the perspective of a website developer, GoogleWebLight is evil because now I have no idea how my website will be shown to the user. It seems you can add a meta tag to disable loading of the site through GoogleWebLight but then Chrome shows a warning to the user that ‘This page is slow to load’ or something to that effect. Definitely not something which I want my user to see while waiting for my page to load.

Overall, it is important for us to be careful on how GoogleWebLight evolves and as hacktivists, we need to come up with better ways to ensure that people with slow internet connection get better experience browsing internet.

December 21, 2015

Gandhi, law and emotions

by viggy — Categories: social — Tags: gandhi, godse, juvenile, law — Leave a comment

Two strand of thoughts have been in my mind for last one week. One is the popular news about the juvenile who raped Jyoti Singh (Nirbhaya) who is being released after completing three years in jail as what is prescribed by the law. The second thought was about more than half a century ago, the hanging of Nathuram Godse.
In the first case, as always, I had opinion opposite to most people who were thirsty for blood, who wanted him to be jailed for more time, probably forever. Their simple case was that it was dangerous to release him. Why? What exactly did he do on that day of incident? How exactly had he been for last 3 years? All this did not matter. They wanted blood and this was an easy case. For me personally it was never about the boy. I dont know his name or anything much about him. What mattered to me more was the kind of society that we were developing where 17 year old boys were becoming rapists. What led him to that path? In some news channel, I heard that the boy was on roads for 7 years away from his home, that means right at age of 10, he had been working, doing odd jobs.

I am not sure what exactly made me think of the Nathuram Godse’s hanging. But I kept thinking of how we had let down Gandhi by hanging his assassin. Wouldnt he himself suggested not to harm him had Gandhi been alive? This seemed to be first incident where we could make use of Gandhi’s teaching without his presence and we as a nation, just chose to ignore it. Now, more than 65 years of his death, no doubt that we have been making a mockery of all his teachings. This line of thought stayed with me for sometime.

But when I was thinking of the case of juvenile’s release along with Godse’s hanging, there seemed to two contradictory line of thoughts. In case of the juvenile’s release, I was looking at it from rule of law perspective and how since he had spent the stipulated time as per law, he had all rights to be released. However, in case of Godse’s hanging, I was looking at it from Gandhi’s perspective. However, when we look at rule of law, then in case of Godse, he did have to present his case in front of a court which then gave the judgement that he should be hanged till death.

So even if Gandhi would have been alive, there would have been no reason to not hang Godse, as that was the law at that time. Ofcourse, would Gandhi have intervened to make amendments to law such that death sentence would be abolished? I dont know. One of the popular reason, lot of left leaning intellectuals do not like Gandhi as much is because of the role he played in case of Bhagat Singh, hanging. Many believe that had Gandhi taken the issue up with the British government as a priority and asserted pressure, they could have been forced to release him or atleast not hang him and maybe give him life imprisonment.

October 20, 2015

Trek to beautiful grasslands of Kudremukh

by viggy — Categories: experience, social, Uncategorized — Tags: Balegal, kudremukh, mullodi, NatureAdmire, somavathi, Trek, triphippie — Leave a comment

Beautiful Grasslands of Kudremukh

After looking at the beautiful images of hills with stretches of grasslands, I always wanted to trek to Kudremukh. Me, Farrah and her colleague, Janaki booked our seats through TripHippie which had a fixed departure available for Oct 17th-18th weekend for Rs. 3500/- per person. The organizer of the trek was Nature Admire, which is run by Dev Balaji. I had met him couple of days before to talk about collaboration between TripHippie and Nature Admire. He has a vast experience and network in this domain and there is lot to learn for us from him. After booking through TripHippie, we received the detailed itinerary from Dev. We left for Kudremukh from Bangalore on Friday night around 10:00 pm in a tempo traveller. We reached Balegal by morning 7:00 am on Saturday. From here, we went to Mullodi in Jeep ride of 8 kms that I will never forget. It was only in videos that I had seen driving in such roads but this route will ensure that all the bones in your body is shook. The drivers here are skilful and had full control over the vehicle during the drive. After reaching the base camp at around 8:00 am, we quickly freshened ourselves and had breakfast. We then packed food items that we had brought from Bangalore along with lunch that was provided to us to carry along with other necessary items like torch, sprays and water bottles. Though as per the itinerary, we were supposed to be back to base camp by 6:00 pm and hence torch would not have been necessary, it later proved to be very helpful. With this, by 9:00 am, we were all set to start the trek.

One of the many streams during the trek with crystal clear water

We were quickly briefed by Balaji about the trek and do’s and don’ts during the trek. We had a local guide with us, who would lead the way for our group. There were more than 100 other trekkers who had come along with various other groups, so the trek route was crowded. The trek is 8 kms to the peak, with close to 4 kms of trek where you have steep climbing which is very tiring. However there are lot of places where you cross the streams with water so pure that you can just drink it directly. Again this was a first experience for me to just drink water directly from the stream.

View of the Kudremukh Peak

It was very sunny and we were soon tired even just after 1 km of trekking. Balaji kept inspiring us that we will soon get comfortable once we get acclimatized to the new environment. Till now, we hadnt seen the actual peak that we had to reach. After around 1.5 kms of trek, Balaji showed us the Kudremukh Peak.

Image from Wikipedia explaining the name of the peak, kudremukh

Kudremukh in Kannada means horse-face. Though a little difficult for me to imagine, others seem to get it easily imagine the peak as a horse face. The sight of the peak kept inspiring us apart from the sight of lush green grasslands. We were expecting rain and hence wanted to complete the trek before evening after which it would be really difficult to trek back if it rains. The local guide kept trekking easily and waiting for us to catch up with him. Our trekking team had people with varied experience with us being the complete newbies and a couple who had even visited Annapurna Base camp in Nepal. So they kept guiding us and helping us. Balaji kept reminding us to take some short breaks and have bananas, nuts and chocolates that we had brought along with us.

Our Trekking group, fully charged while climbing a steep path

By around 1:00 pm, we were completely exhausted with close to 2 kms still to go. We had maintained a constant pace of 25-30 mins per km. Balaji stayed at the end encouraging us to keep moving forward. Looking back at the distance we had covered also kept pushing to achieve the final 2 kms to reach the peak. Also, the cloud had started becoming more dense and we expecting rain, we kept moving forward. By this time, a part of our group had already gone quite ahead.

Almost at the peak, we were able to see the clouds below us

The last 500m is almost plain and we were thrilled for having reached the peak. We were amidst the clouds and it was a great feeling of achievement and satisfaction.

Clouds at the peak

A beautiful feeling to be amidst the clouds

We finally reached the peak at 2:00 pm after 5 and half hours of trek where others from our group were already waiting for us and taking rest.

TripHippie and NatureAdmire team at the peak of Kudremukh

From the peak where clouds were below us

We had the puliogre that we had got from base camp as lunch along with some oranges that we had carried from Bangalore. We also took a small nap of 10 mins to rest our legs. We then started our downhill trek at around 3:00 pm. Though we had expected it to be difficult due to some steep paths, we soon understood it was much easier to have climbed than to go down. Our joints and toes soon started to feel the pressure. We had thought that though difficult, we would still be faster in climbing downhill and hence would reach the base camp by around 7:00 pm with 4 hours to trek downhill. However one of our group mates sprained her leg very soon, just about 2 kms from the peak and it slowed us down. By around 6, were still at least 5 kms far from the base camp and it was already getting dark. This was where the torches that we had carried along helped us and we continued trekking downhill with the help of torch light. We had Balaji who had waited for us before the start of the dense forest. We would have definitely got lost had he not waited for us there. He kept assuring us that we keep moving slowly and carefully. We did not want anyone else to get hurt by twisting or falling. The steam which had been very welcoming while climbing up were now numbing cold. We still had to cross them and since we did not want to risk slipping on stones, we decided to walk through the numbing cold stream. It was our first experience trekking in evening through a dark forest and we were all scared except for Balaji who knew the route very well.

By around 9:00 pm, completely exhausted, we reached the basecamp and joined other members from our group. The dinner was ready along with the tents and sleeping bags for our night stay. Completely refreshed with the experience of the trek, we enjoyed the dinner and went to sleep in the cosy.

The next day, Sunday, we got up at around 9(we were supposed to get up by 6 :)) and visited the Somavathi falls. Since we were already late, we just enjoyed in the falls for an hour or so and rushed back to basecamp to pack our bags. Since most of us were exhausted from the trek, we had decided to go back to Balegal in a Jeep rather than trek back to Balegal which was the plan as per the itinerary. We also decided to skip visiting the Horanadu Temple and just stop at the Tea Estate for a while.

Beautiful Tea Estate

Valley between the tea estate

After this, we reached Kottigehara where we stopped for lunch. We enjoyed the Neer dosa being served there in a local hotel. Though we were expecting our body to ache due to the heavy trekking that we had done the previous day, surprisingly we did not have any such issues. Relieved we enjoyed the movies that was being played in the cab. We reached back to Bangalore at around 8:30 and as always were welcome with a huge traffic at Malleshwaram Maramma Circle. From the crystal clear water from streams and grasslands surrounding us, we were now back amidst the concrete structures of Bangalore.

Overall the trek to Kudremukh was a great experience and my first through TripHippie. I notice close to 150 more packages that are currently hosted in TripHippie which I need to experience.

October 12, 2015

August 23, 2015

Running GeoWave from command line for HBase options

by viggy — Categories: Uncategorized — Tags: commandline, Geowave, gpx, gsoc, hbase, jar, tools — Leave a comment

GeoWave provides GeoWaveMain class which can be used to run GeoWave from command line. For regular users GeoWave provides a wrapper script for the same which would be installed in case you are using the rpm or geowave through other packages. In my case, since I was running in a development environment, I am directly using the GeoWaveMain class with options from the command line. Here I document the results that I got while running the hbase related commands from the GeoWaveMain class from command line. The GeoWave documentation talks about running the commands from command line here. I used the same and the existing tests in GeoWave source code to deduce the commands.

In the following examples, I am running a local hbase and zookeeper instance.

First, to run the commands, we need to generate the tools jar using the command in the deploy source directory,

mvn package -P geowave-tools-singlejar -Dmaven.test.skip=true

I am skipping tests as some of the tests fail in my local setup due to various reason and I do not want that to be reason for the maven build to fail.

Next I also generate the hbase datastore jar using the following mvn build command

mvn package -P hbase-container-singlejar -Dmaven.test.skip=true -Dfindbugs.onlyAnalyze=true

Here I am also asking FindBugs just to analyze as currently FindBugs fails the build due to the autogenerated code in FilterProtos, which is used for Filters in HBase. I have not yet been able to find a solution to fix the same and general solution was to skip findbugs for the build process using this switch. This would still generate the FindBugs.xml which can be analysed later using FindBugs GUI separately.

The last two command should generate geowave-deploy-0.8.8-SNAPSHOT-tools.jar and geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar in the target directory. Since we also use geotools-vector and gpx as formats in following commands, we need to copy their jar also here or refer to them in the classpath. In our case, we just copy the jar to the target directory.

Assuming that user’s current working directory is target, following command is used to run localhbaseingest:

java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-vector-0.8.8-SNAPSHOT.jar mil.nga.giat.geowave.core.cli.GeoWaveMain -localhbaseingest -z localhost:2181 -n geowave -f geotools-vector -b ../../test/data/hail_test_case/hail-box-temporal-filter.shp

To get hdfshbaseingest to work, you need to start Yarn from your hadoop installation. In this, we also use “gpx” format rather than “geotools-vector”(I need to understand why?). Hence we use the jar of gpx format and copy the same in our current working directory. Also this command expects that the directory gpx is already exists in HDFS file system. To create this, you can run the following command from your hadoop installation:

<Hadoop-Installation-Dir>/bin/hadoop fs -mkdir /gpx

Then we run the following command:

java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-gpx-0.8.8-SNAPSHOT.jar: mil.nga.giat.geowave.core.cli.GeoWaveMain -hdfshbaseingest -z localhost:2181 -n geowave -f gpx -b ../../test/data/hail_test_case/hail-box-temporal-filter.shp -hdfs localhost:9000 -hdfsbase “/” -jobtracker “localhost:8032”

-hdfs is the hostname:port of the hadoop installation

-hdfsbase is the parent directory in which we want to ingest

-jobtracker is the hostname:port of the yarn installation.

Currently for hdfshbasestage, I am getting the following error which need to be fixed:

➜ target git:(GEOWAVE-406) ✗ java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-gpx-0.8.8-SNAPSHOT-tools.jar:. mil.nga.giat.geowave.core.cli.GeoWaveMain -hdfshbasestage -b ~/workspace/geowave/extensions/formats/gpx/src/test/resources/ -hdfs localhost:9000 -hdfsbase “/gpx/” -f gpx
2015-08-23 04:41:27,085 FATAL [main] ingest.AbstractIngestHBaseCommandLineDriver (AbstractIngestHBaseCommandLineDriver.java:applyArguments(146)) – Error parsing plugins
java.lang.IllegalArgumentException: Unable to find SPI plugin provider for ingest format ‘gpx’
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.getPluginProviders(AbstractIngestHBaseCommandLineDriver.java:196)
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.applyArguments(AbstractIngestHBaseCommandLineDriver.java:141)
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.run(AbstractIngestHBaseCommandLineDriver.java:75)
at mil.nga.giat.geowave.core.cli.GeoWaveMain.main(GeoWaveMain.java:48)

Similarly for posthbasestage I am getting the same error:

➜ target git:(GEOWAVE-406) ✗ java -cp geowave-deploy-0.8.8-SNAPSHOT-tools.jar:geowave-deploy-0.8.8-SNAPSHOT-hbase-singlejar.jar:geowave-format-gpx-0.8.8-SNAPSHOT-tools.jar:. mil.nga.giat.geowave.core.cli.GeoWaveMain -posthbasestage -hdfs localhost:9000 -hdfsbase “/gpx/” -f gpx -z localhost:2181 -n geowave -jobtracker “localhost:8032”
2015-08-23 04:44:16,261 FATAL [main] ingest.AbstractIngestHBaseCommandLineDriver (AbstractIngestHBaseCommandLineDriver.java:applyArguments(146)) – Error parsing plugins
java.lang.IllegalArgumentException: Unable to find SPI plugin provider for ingest format ‘gpx’
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.getPluginProviders(AbstractIngestHBaseCommandLineDriver.java:196)
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.applyArguments(AbstractIngestHBaseCommandLineDriver.java:141)
at mil.nga.giat.geowave.core.ingest.AbstractIngestHBaseCommandLineDriver.run(AbstractIngestHBaseCommandLineDriver.java:75)
at mil.nga.giat.geowave.core.cli.GeoWaveMain.main(GeoWaveMain.java:48)

August 23, 2015

Using Google’s Protocol Buffer library to write GeoWave Filters for HBase datastore

by viggy — Categories: Uncategorized — Tags: filter, Geowave, gsoc, hbase, iterator — Leave a comment

Accumulo provides Iterators which can be run on Tablet Servers as Filters during Scan. GeoWave uses this in form of local Client Filters and Distributable Filters which run on Tablet Servers when any scan is performed. As part of adding support for HBase, I needed to implement these filters in HBase. I have currently implemented two Filters, SingleEntryFilter and CqlHBaseQueryFilter which are counterparts for SingleEntryFilterIterator and CqlQueryFilterIterator in Accumulo.

Hbase makes use of Google’s Protocol Buffer to serialize data on client side and send it across to tablet servers. In this blog, I explain how I used the protobuf-java library to write the SingleEntryFilter for HBase in GeoWave.

Protobuf auto generates part of the code by using .proto file and its own code generator. The .proto file needs to contain the information about the arguments your class accepts in its constructor, which package the class needs to be generated in, etc. The arguments that class supports needs to be serializable. Since in our case, we are migrating from iterators, all the needed data are expected to be serializable as they need to be serialized even in case of iterators for Accumulo. I created a ‘protobuf’ directory inside local source directory ‘extensions/datastores/hbase/src/main/’ in GeoWave source code and in that created the following SingleEntryFilters.proto file.

option java_package = “mil.nga.giat.geowave.datastore.hbase.query.generated”;
option java_outer_classname = “FilterProtos”;
option java_generic_services = true;
option java_generate_equals_and_hash = true;
option optimize_for = SPEED;

message SingleEntryFilter {
required bytes adapterId = 1;
required bytes dataId = 2;
}

Now to generate the classes using protobuf, we need to install protobuf compiler on the machine. You can download the compiler from here.The README.txt given along with the compiler is quite explanatory for installing it.

After successful installation, by default the protoc compiler executable would be in src directory. Go to the source directory in which you want the generated package to be added. In my case, it was in <geowave-src-directory>/extensions/datastores/hbase/src/main/ .

Now, you can the following command.

<path-to-protoc-installation-dir>/src/protoc -I=. –java_out=java/ protobuf/SingleEntryFilters.proto

protobuf/SingleEntryFilter.proto is the path to the .proto file from your current directory.

This generated the necessary FilterProtos class. Now we need to create the SingleEntryFilter class. We use the FilterBase class provided by HBase to create new custom classes. Lars George’s book, HBase: The Definitive Guide explains developing custom Filters for hbase and I used the example shared in the github repo for the book to develop Custom Filter for developing SingleEntryFilter.

It was only through that example that I came to know that I need to also implement toByteArray and parseFrom methods in the Custom Filter. Later I also found in HBase log that parseFrom method generates a DeserializationException which informs user about extending it in derived class.

« Newer Posts — Older Posts »

April 2024
M	T	W	T	F	S	S
« Apr
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30