python

3 articles in category python / Subscribe

I remember it took me sometime to get this configured when I first started trying Jupyter and Spark out. Hopefully this is helpful for others. This works for Hadoop 2.6.0-CDH5.9.1, Spark 1.6.0 using python2.7 and Python3. For other versions, you need to adjust the path accordingly. Basically, you just need to tell spark 4 things: The location of your (Ana)conda installation The location of your Jupyter installation and its configuration The location of your Python installation Resources your Spark executors need Type the following from your bash terminal (If you are using Cloudera, this would be your Edge node. If …

Read more →

In case you are wondering how to use the awesome LIBSVM package with the awesome PyCharm IDE, here is a brief guide:

LIBSVM documentation  mention you need to

  • Download  libsvm(version 3.2 at the time of this writing)
  • Extract the zip to a folder. By default, this libsvm-3.20
  • Go to your command prompt, within the extracted directory, libsvm-3.20/, type “make”. This will create a file called libsvm.so.2. If this doesn’t work, google how to enable make command on Mac OS.
  • go to libsvm-3.20/python directory, type make. This will create 2 files: svm.py and svmutil.py

Now, within PyCharm, say that you have a folder called Project/code/ where your code is stored at this level. Copy svm.py and svmutil.py mentioned above to your Project/code/ directory and Copy libsvm.so.2 file to your Project/ directory.

From Project/code/somefile.py python file you can import the svm library with

from svmutil import *

That’s it. You should be all set.

 

My code is pulling JSON data from a web service I wrote. That JSON data is then loaded into a python dictionary. Things works fabulously when I debug in my local console running an internal web server provided by the App Engine SDK. But when I upload the application to Goole CLoud Engine, things break. This is a snippet of my code

states=dict()
population_data=""
url="http://something.com/json"
try:
   response=urllib2.urlopen(url)
   states=json.load(response,"utf-8")
except Exception as ex:
   self.response.write("Google says: %s"%(ex.message))
   self.response.write("Try refreshing the page (again)")
   SystemExit(0)

I have all the states abbreviation as keys. when I try to access the value of the dictionary directly using square brackets, like

states["IN"]

Google App Engine complains about KeyError exception. However if I try iterating through the key using a for loop, like

for each_state in states:
   self.response.write(states[each_state])

it works flawlessly in the App Engine. I also found out that the following also works perfectly

self.response.write(states.get("IN"))

The only thing I cannot do is accessing the dictionary using square brackets. That baffles me. If you know what’s up with that, drop me a line.