Python3 generator with recursion

Requirement:

Generate sequential MAC addresses without any duplicates

Input:
integer
Output:
MAC Address

Problem:
Python2 does not support yield from keyword. So .. Python3

Solution:

mac = [0, 0, 0, 0, 0, 0]

def generateMAC(index):

    if index < 0:

        return

    else:

        generateMAC(index - 1)

    while mac[index] < 255:

        #print(':'.join(map(lambda x: "%02x" % x, mac)))

        mac[index] = mac[index] + 1

        yield ':'.join(map(lambda x: "%02x" % x, mac))

        yield from generateMAC(index - 1)

    mac[index] = 0

Rambling:

Maximum possible number of MAC Addresses are 255*255*255*255*255*255 = 274941996890625

So, 255*255=65025 combinations from first two octets are enough for my requirement. Hence this mess. But It works!

mac = [0, 0, 0, 0, 0, 0]

macs = []

def o_f_generateMAC(index):

    while mac[5] <= 255:

        macs.append(':'.join(map(lambda x: "%02x" % x, mac)))

        while mac[4] <= 255:

            macs.append(':'.join(map(lambda x: "%02x" % x, mac)))

            mac[4] = mac[4] + 1

        mac[4] = 0

        mac[5] = mac[5] + 1

    mac[5] = 0

    return macs

However when just multiplied the combinations by another 255 (ie., 255*255*255). Python interpretor took its own time. Anyway I just said enough.

mac = [0, 0, 0, 0, 0, 0]

macs = []

def o_f_generateMAC():

    while mac[5] <= 255:

        macs.append(':'.join(map(lambda x: "%02x" % x, mac)))

        while mac[4] <= 255:

            macs.append(':'.join(map(lambda x: "%02x" % x, mac)))

            while mac[3] <= 255:

                macs.append(':'.join(map(lambda x: "%02x" % x, mac)))

                mac[3] = mac[3] + 1

            mac[3] = 0

            mac[4] = mac[4] + 1

        mac[3] = 0

        mac[4] = 0

        mac[5] = mac[5] + 1

    mac[5] = 0

    return macs

When converting the same with better logic (of course! recursion), this arrived.

def f_generateMAC(index):

    if index < 0:

        return

    else:

        generateMAC(index - 1)

    while mac[index] < 255:

        macs.append(':'.join(map(lambda x: "%02x" % x, mac)))

        mac[index] = mac[index] + 1

        generateMAC(index - 1)

    mac[index] = 0

    return macs

The above function looks definitely better. Still memory inefficient. Why should the list "macs" be created at all?!. Thus, the need for the generator.

mac = [0, 0, 0, 0, 0, 0]

def generateMAC(index):

    if index < 0:

        return

    else:

        generateMAC(index - 1)

    while mac[index] < 255:

        mac[index] = mac[index] + 1

        yield ':'.join(map(lambda x: "%02x" % x, mac))

        generateMAC(index - 1)

    mac[index] = 0

Still, the above generator did not work as expected on Python2. Then, Python3 helped to achieve "generator delegating part of computing to another generator" with "yield from" keyword. Finally, arrived at the solution shown on the top.

Thanks!

Raymond Hettinger - Super considered super! - PyCon 2015

Video Link : https://www.youtube.com/watch?v=EiOglTERPEo

Main Takes:

1. super() function in Python is different from other OOP languages. Because it calls the next one in line NOT the ancestors (in context of Inheritance).

2. The next one in line is identified from MRO as below,







3. Catch No.1 -> Every (in between) Ancestor must have implemented "super()". If not use subclass while inheriting.
    Eg. https://stackoverflow.com/questions/13380819/multiple-inheritance-along-with-threading-in-python

4. Catch No.2 -> Who is upstream (or next in line) is determined by the derived class. So to stop the call chain a root class without super() is required.

5. Catch No.3 -> All class constructors must have implemented named parameters (**kwargs).

6. Class with most reusable code must be parent. (Other consideration must take back step)

Encounter with Python Generators

Recently faced a problem in analyzing log.
"Single line log file of 10GB size needs to be read and all IP address must be printed"

Issue: Cannot read line by line to avoid memory corruption. Have to go for character by character.

Solution:

#!/usr/bin/python
import re
def getIP():
        ip = re.compile('\d+|\\.')
        out = []
        with open("./ipaddr","r") as f:
                while True:
                        c = f.read(1)
                        if not c:
                                break
                        if ip.match(c):
                                out.append(c)
                                for i in range(14):
                                        c = f.read(1)
                                        if ip.match(c):
                                                out.append(c)
                                        else:
                                                if out:
                                                        yield "".join(out)
                                                out = []

print str([ipad for ipad in getIP()])

Any ideas to simplify ??

Expect scripting best practices that I learned so far

I use expect scripting for logging into remote machine and do some stuff and get back result as success or failure.

Here are my learning's..

1. Always setup keys at local and remote machines for logging in.

2. Setup your own prompt text at remote shell (> or # or $)

3. Use wait method to get exit value

1. REGEX matching:

If the remote machine's prompt is color coded with "$" in the prompt string, then for effective regex pattern match use follwing,

set prompt ".*(>|%|#|\\$).*"

expect -re $prompt

2. To know exit status of last command/script, add the following at the end of expect script,

lassign [wait -nowait] pid spawnid os_error_flag value

if {$os_error_flag == 0} {
    puts "exit status: $value"
} else {
    puts "errno: $value"
}

exit $value

3. Syntax issues:

While using "if" statements curly braces has to be in same line,

if {condition} {

   statement

}

While using expect_out, "no space" in between arguments(comma and string),

$expect_out(0,string)

4. Here document

If you ever has to use nested spawn (login to a shell again login to another shell), think of using here document 

set command "/opt/bin/another_shell << EOF\n"

append command "execute_command\n"

append command "exit\n"

append command "EOF"

send "$command\n" 

5. Solving issue of expect skipping conditional "send" statement

So, I had a part of script like,

set prompt ".*(>|%|#|\\$).*"

expect -re $prompt

if {[info exists args(-a)]} {
send "$args(-c) $args(-a)\n"
} else {
send "$args(-c)\n"
}

expect -re $prompt

send "echo \"returncode :$?\"\n"

exit 0

In the above snippet, "send" statements inside "if .. else" did not execute always. it gets skipped so many times.

Fix1: Adding delay (sleep 1) above last expect statement.

This worked well. Anyway I wanted a more elegant solution.

Fix2: using "$?"

if {[info exists args(-a)]} {
send "$args(-c) $args(-a);echo $?\n"
} else {
send "$args(-c);echo $?\n"
}

expect -re "0\r\n$prompt"

exit 0

Here the issue is, expect need to look for two lines 

0

PROMPT#

So, I added "\r\n" after "0" then the prompt statement.

Power of python dictionary

Scenario: Copy logs that were created after certain timestamp to a destination directory.
Catch: there are different types of logs - Alarms, Notifications, Errors etc.

Here is my initial ugly code,

#!/usr/bin/python
import glob
import os
import datetime, time
import math
import shutil

class logCollector:
    def __init__(self):
        self.toLogpath = "/home/appuser/automation/logs" 
        self.applogPath = "/cluster/storage/no-backup/coremw/var/log/saflog"
    def getFileTime(self,path):
        _mtime = math.floor(os.path.getmtime(path))
        return _mtime
    def getLogsAfter(self, filePath, start):
        logFiles = glob.glob(filePath)
        fileList = list()
        for file in logFiles:
                fileTime = self.getFileTime(file)
                if float(fileTime) >= math.floor(start):
                    fileList.append(file)
        return fileList
    def getNotificationLogAfter(self, start):
        filePath = self.applogPath+"/*otif*log"
        return self.getLogsAfter(filePath, start)
    def getAlarmLogAfter(self, start):
        filePath = self.applogPath+"/*larm*log"
        return self.getLogsAfter(filePath, start)
    def getErrorLogAfter(self, start):
        filePath = self.applogPath+"/*Err*log"
        return self.getLogsBWsecs(filePath, start)
    def copyTraces(self, time):
        start_time=time
        fileList = list()
            fileList.extend(self.getNotificationLogAfter(start_time))
            fileList.extend(self.getAlarmLogAfter(start_time))
            fileList.extend(self.getErrorLogAfter(start_time))
        for logfile in fileList:
            if os.path.isfile(logfile):
                shutil.copy(logfile, self.toLogpath)
            else:
                print(logfile+" is not found!!!")


After Learning dictionary in python,

#!/usr/bin/python
import glob
import os
import shutil
import math

class myTraceCollector:
        destination = "/home/appuser/automation/logs"
        appTraces ={
                        "Alarm Logs":"/cluster/storage/no-backup/coremw/var/log/saflog/*larm*log",
                        "Notification Logs":"/cluster/storage/no-backup/coremw/var/log/saflog/*otif*log",
                        "Error Logs":"/cluster/storage/no-backup/coremw/var/log/saflog/*Err*log"
                }

        def getTracesAfter(self, time):
                for fileType, filePath in self.appTraces.iteritems():
                        print fileType
                        tracesList = glob.glob(filePath)
                        for file in tracesList:
                                fileTime = os.path.getmtime(file)
                                if float(fileTime) >= math.floor(time):
                                        shutil.copy(file, self.destination)
                                        print file+" copied"

It is wow! experience for me...

Python, Automation, Problems, Solutions

Let me share, few of the problems that I faced during python automation here,

Handling Passwords across Python and Expect:

Scenario - Login to Hypervisor manager (where VNF is deployed) and reset (power off and on) a particular VM. Meanwhile monitor VNF node for status change. After that VM comes back to service perform assertions for setting test verdict.

Issue: password to VM manager must be captured using python and be sent to expect script as argument. password should not be printed / echoed anywhere. (catch: password may contain special characters)

Solution:
import getpass

pwd = getpass.getpass() # input: my#pwd

=> So password will not be echoed - partly solved

special characters must be escaped before send that as argument to expect script

pwd=re.escape(pwd)

cmd = "expect my_script.exp "+pwd  #expect my_script.exp my\#pwd
os.system(cmd)

=> Fully solved.

The Importance of "exit 0" in shell script:

 Scenario:
Python calls a shell script to kill a process. And based on exit status the Python takes further steps.

Problem:
When the processes that are being killed are interdependent, some time "no such process" error is thrown. So a non-zero exit status is returned to Python and It prints killing NOT successful, eventhough intended killing has been done.

Shell script:

#!/bin/bash

ps -ef | grep xxx | xargs -i kill {}

Python ():

import subprocess

def killme():
  p = subprocess.Popen(['./script.sh'], stdout=subprocess.PIPE)
  out, err = p.communicate()
  if p.returncode !=0:
    print "Killing NOT successful"
  print "Killed"

killme()


Solution:

Added exit 0 in shell script

#!/bin/bash

ps -ef | grep xxx | xargs -i kill {}
exit 0

Using Object of a class as an attribute of another class:

class1.py

class class1:
  def __init__(self):
    self.variable = "One"

class2.py

class class2:
  def __init__(self, obj):
    self.obj = obj

test.py

from class1 import class1
from class2 import class2

c1 = class1()
c2 = class2(c1)
c1.variable = "two"
print c2.obj.variable

#two

Grep LookAhead and LookBehind expressions

LookAhed ?=
 

LookBehind ?<=

?!
 

?<!