Raymond Hettinger - Super considered super! - PyCon 2015

Video Link : https://www.youtube.com/watch?v=EiOglTERPEo

Main Takes:

1. super() function in Python is different from other OOP languages. Because it calls the next one in line NOT the ancestors (in context of Inheritance).

2. The next one in line is identified from MRO as below,







3. Catch No.1 -> Every (in between) Ancestor must have implemented "super()". If not use subclass while inheriting.
    Eg. https://stackoverflow.com/questions/13380819/multiple-inheritance-along-with-threading-in-python

4. Catch No.2 -> Who is upstream (or next in line) is determined by the derived class. So to stop the call chain a root class without super() is required.

5. Catch No.3 -> All class constructors must have implemented named parameters (**kwargs).

6. Class with most reusable code must be parent. (Other consideration must take back step)

Encounter with Python Generators

Recently faced a problem in analyzing log.
"Single line log file of 10GB size needs to be read and all IP address must be printed"

Issue: Cannot read line by line to avoid memory corruption. Have to go for character by character.

Solution:

#!/usr/bin/python
import re
def getIP():
        ip = re.compile('\d+|\\.')
        out = []
        with open("./ipaddr","r") as f:
                while True:
                        c = f.read(1)
                        if not c:
                                break
                        if ip.match(c):
                                out.append(c)
                                for i in range(14):
                                        c = f.read(1)
                                        if ip.match(c):
                                                out.append(c)
                                        else:
                                                if out:
                                                        yield "".join(out)
                                                out = []

print str([ipad for ipad in getIP()])

Any ideas to simplify ??

Expect scripting best practices that I learned so far

I use expect scripting for logging into remote machine and do some stuff and get back result as success or failure.

Here are my learning's..

1. Always setup keys at local and remote machines for logging in.

2. Setup your own prompt text at remote shell (> or # or $)

3. Use wait method to get exit value

1. REGEX matching:

If the remote machine's prompt is color coded with "$" in the prompt string, then for effective regex pattern match use follwing,

set prompt ".*(>|%|#|\\$).*"

expect -re $prompt

2. To know exit status of last command/script, add the following at the end of expect script,

lassign [wait -nowait] pid spawnid os_error_flag value

if {$os_error_flag == 0} {
    puts "exit status: $value"
} else {
    puts "errno: $value"
}

exit $value

3. Syntax issues:

While using "if" statements curly braces has to be in same line,

if {condition} {

   statement

}

While using expect_out, "no space" in between arguments(comma and string),

$expect_out(0,string)

4. Here document

If you ever has to use nested spawn (login to a shell again login to another shell), think of using here document 

set command "/opt/bin/another_shell << EOF\n"

append command "execute_command\n"

append command "exit\n"

append command "EOF"

send "$command\n" 

5. Solving issue of expect skipping conditional "send" statement

So, I had a part of script like,

set prompt ".*(>|%|#|\\$).*"

expect -re $prompt

if {[info exists args(-a)]} {
send "$args(-c) $args(-a)\n"
} else {
send "$args(-c)\n"
}

expect -re $prompt

send "echo \"returncode :$?\"\n"

exit 0

In the above snippet, "send" statements inside "if .. else" did not execute always. it gets skipped so many times.

Fix1: Adding delay (sleep 1) above last expect statement.

This worked well. Anyway I wanted a more elegant solution.

Fix2: using "$?"

if {[info exists args(-a)]} {
send "$args(-c) $args(-a);echo $?\n"
} else {
send "$args(-c);echo $?\n"
}

expect -re "0\r\n$prompt"

exit 0

Here the issue is, expect need to look for two lines 

0

PROMPT#

So, I added "\r\n" after "0" then the prompt statement.

Power of python dictionary

Scenario: Copy logs that were created after certain timestamp to a destination directory.
Catch: there are different types of logs - Alarms, Notifications, Errors etc.

Here is my initial ugly code,

#!/usr/bin/python
import glob
import os
import datetime, time
import math
import shutil

class logCollector:
    def __init__(self):
        self.toLogpath = "/home/appuser/automation/logs" 
        self.applogPath = "/cluster/storage/no-backup/coremw/var/log/saflog"
    def getFileTime(self,path):
        _mtime = math.floor(os.path.getmtime(path))
        return _mtime
    def getLogsAfter(self, filePath, start):
        logFiles = glob.glob(filePath)
        fileList = list()
        for file in logFiles:
                fileTime = self.getFileTime(file)
                if float(fileTime) >= math.floor(start):
                    fileList.append(file)
        return fileList
    def getNotificationLogAfter(self, start):
        filePath = self.applogPath+"/*otif*log"
        return self.getLogsAfter(filePath, start)
    def getAlarmLogAfter(self, start):
        filePath = self.applogPath+"/*larm*log"
        return self.getLogsAfter(filePath, start)
    def getErrorLogAfter(self, start):
        filePath = self.applogPath+"/*Err*log"
        return self.getLogsBWsecs(filePath, start)
    def copyTraces(self, time):
        start_time=time
        fileList = list()
            fileList.extend(self.getNotificationLogAfter(start_time))
            fileList.extend(self.getAlarmLogAfter(start_time))
            fileList.extend(self.getErrorLogAfter(start_time))
        for logfile in fileList:
            if os.path.isfile(logfile):
                shutil.copy(logfile, self.toLogpath)
            else:
                print(logfile+" is not found!!!")


After Learning dictionary in python,

#!/usr/bin/python
import glob
import os
import shutil
import math

class myTraceCollector:
        destination = "/home/appuser/automation/logs"
        appTraces ={
                        "Alarm Logs":"/cluster/storage/no-backup/coremw/var/log/saflog/*larm*log",
                        "Notification Logs":"/cluster/storage/no-backup/coremw/var/log/saflog/*otif*log",
                        "Error Logs":"/cluster/storage/no-backup/coremw/var/log/saflog/*Err*log"
                }

        def getTracesAfter(self, time):
                for fileType, filePath in self.appTraces.iteritems():
                        print fileType
                        tracesList = glob.glob(filePath)
                        for file in tracesList:
                                fileTime = os.path.getmtime(file)
                                if float(fileTime) >= math.floor(time):
                                        shutil.copy(file, self.destination)
                                        print file+" copied"

It is wow! experience for me...

Python, Automation, Problems, Solutions

Let me share, few of the problems that I faced during python automation here,

Handling Passwords across Python and Expect:

Scenario - Login to Hypervisor manager (where VNF is deployed) and reset (power off and on) a particular VM. Meanwhile monitor VNF node for status change. After that VM comes back to service perform assertions for setting test verdict.

Issue: password to VM manager must be captured using python and be sent to expect script as argument. password should not be printed / echoed anywhere. (catch: password may contain special characters)

Solution:
import getpass

pwd = getpass.getpass() # input: my#pwd

=> So password will not be echoed - partly solved

special characters must be escaped before send that as argument to expect script

pwd=re.escape(pwd)

cmd = "expect my_script.exp "+pwd  #expect my_script.exp my\#pwd
os.system(cmd)

=> Fully solved.

The Importance of "exit 0" in shell script:

 Scenario:
Python calls a shell script to kill a process. And based on exit status the Python takes further steps.

Problem:
When the processes that are being killed are interdependent, some time "no such process" error is thrown. So a non-zero exit status is returned to Python and It prints killing NOT successful, eventhough intended killing has been done.

Shell script:

#!/bin/bash

ps -ef | grep xxx | xargs -i kill {}

Python ():

import subprocess

def killme():
  p = subprocess.Popen(['./script.sh'], stdout=subprocess.PIPE)
  out, err = p.communicate()
  if p.returncode !=0:
    print "Killing NOT successful"
  print "Killed"

killme()


Solution:

Added exit 0 in shell script

#!/bin/bash

ps -ef | grep xxx | xargs -i kill {}
exit 0

Using Object of a class as an attribute of another class:

class1.py

class class1:
  def __init__(self):
    self.variable = "One"

class2.py

class class2:
  def __init__(self, obj):
    self.obj = obj

test.py

from class1 import class1
from class2 import class2

c1 = class1()
c2 = class2(c1)
c1.variable = "two"
print c2.obj.variable

#two

Grep LookAhead and LookBehind expressions

LookAhed ?=
 

LookBehind ?<=

?!
 

?<!

Biulding my own Framework for System Testing

    It happened in my career as well. I also developed yet another framework for automating system testing testcases.

First of all, I went through few blogs/presentations/posts to see If I can use any existing frameworks. There were so many content for Selenium and other web-based test frameworks. But my situation called for developing a new one. So, I built this on a basic idea. This idea has been repeated for numerous times in numerous websites. It goes as follows,

As you can see, It is a very simple yet powerful and widely used design. Here, the Automation Server holds all my scrips (Perl, Expect, Shell, Python ..etc) and few of these scripts runs on SUT and other few runs on AS itself. Actual test scenarios runs on SUT and logs are collected, then copied to Reports and Logs Server. URLs of these logs and other information are updated in database. When the test is completed, you get a URL to see the report. The presentation server takes care of tabulating and presenting the reports. In fact this Reports and Logs Server is a NFS mounted on Presentation Server.

But wait, who will initiate the test? There comes Jenkins!

Yes, you are right. Above framework is a part of our Continuous Integration architecture. Anyway, it was interesting to setup my own framework, though the same has been done by millions of other people.