Monday, March 18, 2013

sshpass VS expect, what to expect on ssh without password?

SSHPASS VS EXPECT, what to expect when you are expecting?

Some time ago I was in a middle of a script and I found myself trying to connect to a SFTP server to download some files. First of all I tried the powerful 'wget' but this program does not support ssh connections. So let's see what options we have:

  1. sftp (secure file transfer program)
  2. scp (secure copy (remote file copy program))

And to make it harder I don't have access to the remote SFTP server to exchange SSL certificates to connect without password. So here are the options to make a ssh connection with password:

  1. sshpass
  2. expect 

For the first one (sshpass) it did not worked too well. The reason may be the FreeBSD I was using at that moment. But here is an amazing link where you can use this program very easily.

So if you have no luck with sshpass (as I did), there is one more solution. Here is expect. Just let me show you the quick solution :

expect -c 'spawn scp user@remote_host:/ /my/local/home; sleep 10; expect assword; send "password\n"; interact'

Let's see this in detail:

  1. The beginning of this command is to invoke the program " expect -c ".
  2. Then between single quotes (') the expect commands are specified.
  3.  "spawn" invokes our external program (sftp, ssh, scp, etc...) in my case 'spawn scp' and following the short-cut for ssh connections 'spawn myuser@remote_server:/path/file/i/want.ext /to/my/folder;'
  4. In my case the sftp server takes some time to respond to my request (it is a network issue so that's why I added this part). Let's wait at least 10 seconds for the server response 'sleep 10;'.
  5. At this point the scp command shows "BLAH BLAH server password:" or "BLAH BLAH password" or "BLAH Password:", this output may change in different unix distros. We tell expect to wait for the 'assword' text that will match with most of the responses showed before 'expect assword;'.
  6. Sends the password NOW!!!, 'password\n;'. And don't forget to set '\n;' because this is the 'return' key press.
  7. And continue every thing as usual ' interact '
Thanks for your time, and read more on this link.

Friday, March 23, 2012

What about file in GAE?

In addition to my post Creating Python Google App with Aptana and Pydev, the content for the file will be show here. So, what about

When I was developing my GAE app, I don't remember how I got this file but is nice to have in any GAE project.

#!/usr/bin/env python
# Copyright 2007 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# See the License for the specific language governing permissions and
# limitations under the License.

"""Sample Appstats Configuration.

There are four sections:

0) WSGI middleware declaration.
1) Django version declaration.
2) Configuration constants.
3) Configuration functions.

Also a section at the end for the remote_api handler.


import logging
import random
import re

# 0) WSGI middleware declaration.

# Only use this if you're not Django; with Django, it's easier to add
#   'google.appengine.ext.appstats.recording.AppstatsDjangoMiddleware',
# to your Django file.

# # def webapp_add_wsgi_middleware(app):
# #   from google.appengine.ext.appstats import recording
# #   app = recording.appstats_wsgi_middleware(app)
# #   return app

# 1) Django version declaration.

# If your application uses Django and requires a specific version of
# Django, uncomment the following block of three lines.  Currently
# supported values for the Django version are '0.96' (the default),
# '1.0', and '1.1'.

from google.appengine.dist import use_library
use_library('django', '1.2')
import django

# 2) Configuration constants.

# DEBUG: True of False.  When True, verbose messages are logged at the
# DEBUG level.  Also, this flag is causes tracebacks to be shown in
# the web UI when an exception occurs.  (Tracebacks are always logged
# at the ERROR level as well.)

appstats_DEBUG = True

# DUMP_LEVEL: -1, 0, 1 or 2.  Controls how much debug output is
# written to the logs by the internal dump() function during event
# recording.  -1 dumps nothing; 0 dumps one line of information; 1
# dumps more informat and 2 dumps the maximum amount of information.
# You would only need to change this if you were debugging the
# recording implementation.

appstats_DUMP_LEVEL = -1

# The following constants control the resolution and range of the
# memcache keys used to record information about individual requests.
# Two requests that are closer than KEY_DISTANCE milliseconds will be
# mapped to the same key (thus losing all information about the
# earlier of the two requests).  Up to KEY_MODULUS distinct keys are
# generated; after KEY_DISTANCE * KEY_MODULUS milliseconds the key
# values roll over.  Increasing KEY_MODULUS causes a proportional
# increase of the amount of data saved in memcache.  Increasing
# KEY_DISTANCE causes a requests during a larger timespan to be
# recorded, at the cost of increasing risk of assigning the same key
# to two adjacent requests.

appstats_KEY_DISTANCE = 100
appstats_KEY_MODULUS = 1000

# The following constants control the namespace and key values used to
# store information in memcache.  You can safely leave this alone.

appstats_KEY_NAMESPACE = '__appstats__'
appstats_KEY_PREFIX = '__appstats__'
appstats_KEY_TEMPLATE = ':%06d'
appstats_PART_SUFFIX = ':part'
appstats_FULL_SUFFIX = ':full'
appstats_LOCK_SUFFIX = ''

# Numerical limits on how much information is saved for each event.
# MAX_STACK limits the number of stack frames saved; MAX_LOCALS limits
# the number of local variables saved per stack frame.  MAX_REPR
# limits the length of the string representation of each variable
# saved; MAX_DEPTH limits the nesting depth used when computing the
# string representation of structured variables (e.g. lists of lists).

appstats_MAX_STACK = 10
appstats_MAX_LOCALS = 10
appstats_MAX_REPR = 100
appstats_MAX_DEPTH = 10

# Regular expressions.  These are matched against the 'code key' of a
# stack frame, which is a string of the form
# '::'.  If the code key of a stack frame
# matches RE_STACK_BOTTOM, it and all remaining stack frames are
# skipped.  If the code key matches RE_STACK_SKIP, that frame is not
# saved but subsequent frames may be saved.

appstats_RE_STACK_BOTTOM = r'dev_appserver\.py'
appstats_RE_STACK_SKIP = r'recording\.py|apiproxy_stub_map\.py'

# Timeout for memcache lock management, in seconds.

appstats_LOCK_TIMEOUT = 1

# Timezone offset.  This is used to convert recorded times (which are
# all in UTC) to local time.  The default is US/Pacific winter time.

appstats_TZOFFSET = 8*3600

# URL path (sans host) leading to the stats UI.  Should match app.yaml.
# If "builtins: - appstats: on" is used, the path should be /_ah/stats.

appstats_stats_url = '/_ah/stats'

# Fraction of requests to record.  Set this to a float between 0.0
# and 1.0 to record that fraction of all requests.

appstats_RECORD_FRACTION = 1.0

# List of dicts mapping env vars to regular expressions.  Each dict
# specifies a set of filters to be 'and'ed together.  The keys are
# environment variables, the values are *match* regular expressions.
# A request is recorded if it matches all filters of at least one
# dict.  If the FILTER_LIST variable is empty, all requests are
# recorded.  Missing environment variables are considered to have
# the empty string as value.  If a regular expression starts with
# '!', the sense of the match is negated (the value should *not*
# match the expression).

appstats_FILTER_LIST = []

# 3) Configuration functions.

# should_record() can be used to record a random percentage of calls.
# The argument is the CGI or WSGI environment dict.  The default
# implementation returns True iff the request matches FILTER_LIST (see
# above) *and* random.random() < RECORD_FRACTION.

def appstats_should_record(env):
    if appstats_FILTER_LIST:
        logging.debug('FILTER_LIST: %r', appstats_FILTER_LIST)
        for filter_dict in appstats_FILTER_LIST:
            for key, regex in filter_dict.iteritems():
                negated = isinstance(regex, str) and regex.startswith('!')
                if negated:
                    regex = regex[1:]
                value = env.get(key, '')
                if bool(re.match(regex, value)) == negated:
                    logging.debug('No match on %r for %s=%r', regex, key, value)
                logging.debug('Match on %r', filter_dict)
            logging.debug('Non-empty FILTER_LIST, but no filter matches')
            return False
        if appstats_RECORD_FRACTION >= 1.0:
            return True
        return random.random() < appstats_RECORD_FRACTION

# The following functions are called by the UI code only; they don't
# affect the recorded information.

# normalize_path() takes a path and returns an 'path key'.  The path
# key is used by the UI to compute statistics for similar URLs.  If
# your application has a large or infinite URL space (e.g. each issue
# in an issue tracker might have its own numeric URL), this function
# can be used to produce more meaningful statistics.

def appstats_normalize_path(path):
    return path

# extract_key() is a lower-level function with the same purpose as
# normalize_key().  It can be used to lump different request methods
# (e.g. GET and POST) together, or conversely to use other information
# on the request object (mostly the query string) to produce a more
# fine-grained path key.  The argument is a StatsProto object; this is
# a class defined in  Useful methods are:

#   - http_method()
#   - http_path()
#   - http_query()
#   - http_status()

# Note that the StatsProto argument is loaded only with summary
# information; this means you cannot access the request headers.

def appstats_extract_key(request):
    key = appstats_normalize_path(request.http_path())
    if request.http_method() != 'GET':
        key = '%s %s' % (request.http_method(), key)
    return key

# ########################################
# Remote_API Authentication configuration.

# See google/appengine/ext/remote_api/ for more information.
# In most cases, you will not want to configure this.


Sunday, January 15, 2012

Creating Python Google App with Aptana and Pydev

This document is focus on setup a development environment and start a web application based on Google App Engine (GAE) on Python SDK using Aptana IDE.

Google App Engine currently support create applications on three different languages: Java, GO and Python.  Many developers like to work their projects with a good text editor (Gedit, Notepad++, JEdit, TextMate, MCedit, Vim, etc.) but others go for a sophisticated IDE (Integrated Development Environment). Currently Google  gives a plugin for develop Java applications on Eclipse IDE, but not for Python and GO. Lets focus on start a whole project based on Python.

First than all let's start by setting the all environment by downloading the Python SDK from For Windows just install the ".msi" executable and for Linux just extract the ".zip" in your "home" folder. Don't forget to install Python on Windows.

Now let's go to get the Aptana IDE at For Windows just execute the installer and for Linux unpack the ".zip" file and make sure of install Sun Java JDK or JRE (Please not Open JDK). Aptana comes with PHP, HTML, CSS, Javascript, YAML, SQL and more editors. The most I like is the Django Template editor, but we'll get there. Aptana comes by default with a Git client but if you work with SVN take a look at plugin Subclipse.

Aptana is based on the Eclipse framework and therefore is possible to install many Eclipse plugins on it. The plugin that allow coding Python on Aptana (and Eclipse) is Pydev. Once you open Aptana for the first time select the Pydev perspective.

Let's specify the version of Python to Pydev in Aptana. Select "Window -> Preferences -> Pydev -> Intrepeter - Python". Click on the "Auto Config" button or select the path of your Python interpreter on the "New ..." button.

Lets start by creating a new Pydev Google App Engine Project. Select "File -> New -> Project -> Pydev -> Pydev Google App Engine Project".

Make sure to select the correct version of Python for the project and the path for the Google SDK. Look at the images below.

Now we have the development environment complete. I will assume that you have created an application on For the next example I will use my own app called djangoabdel (I called like this because I like Django a lot!)

When I started reading the documentation of GAE it was evident that there is a lot of freedom for create your own framework. So I will show my way but you can innovate or use any pure Python framework (Django?). Lets take a look at this image:

Here I have:
  • - This is the start point of the whole application.
  • - At the top level of your project is automatically imported bygoogle.appengine.ext.webapp.util.run_wsgi_app() to add middlewear to webapp applications. This is how I specify Django version (1.0, 1.1 or 1.2). Take a look at.
  •  app.yaml - The heart of the app configuration.
  • templates - folder that will contain static files like images, css files, javascript files, etc. This is the UI of your application.
  • controllers - python package that contains all the controllers or business scripts of our application.
  • models - python package that contains all the models or  Entities of our application.
  • libs - python package that contains all the third party libraries.
I will create a post for explain more detailed the different parts.

One thing that happened to me while creating this way of development was a problem trying to import the controllers, libs, models to the file. The solution is pretty simple, just add this to the file in the package folder:

import os, sys

path = os.path.abspath(".")

sys.path.append(os.path.join(path, __name__))

Even if sometimes you have trouble trying to import a package just restart the test server. Just keep this in mind (Oh, you will!).

Now lets create a "Run" configuration so we can run our development server. Go for "Run -> Run Configurations...". Here double click on "Pydev Google App Run" and set a name to this configuration and the at "Main Module" point to "" file. This file is in the SDK installation files.

The last of the configuration is click on the "Arguments" tab and on "Program Arguments" add the path to your application folder.

Now click on "Apply" button and then on "Run" button for run the app. Now all the messages of the application will appear on the 'Console" tab. Here you can stop the development server too.

Now that the "Run" configuration is created, the "Debug" configuration is created automatically. Take a look at the short-cut buttons. 

Just remember that for debugging set your break points.

While you will be coding you'll enjoy the auto-complete behaviour of Pydev (Ctrl + Space Bar). But the most I like is the Django Template Editor for the GAE template files. When opening a html file from the "Pydev Package Explorer" do a right click "Open with ->" and select "HTML.Templates Django Editor (Aptana)". GAE takes advantage of the Django template tags and filters, so lets  do the same on this editor.

The last part is upload our app to Google so that's easy as do a right click on our project "Pydev: Google App Engine -> Upload".

Thank you for your time.