Author: TCTJr

DISCLAIMER: This blog was written some time ago. Software breaks once in a while and there was a ghost in my LazyWeb^TM machine. We are back to our regularly scheduled program. Read on Dear Reader, and your humble narrator apologizes.

The other day i was talking to someone about file manipulations and whatnot and mentioned how awesome Pandas and the magic of the df.DoWhatEverYaWant( my_data_object) via a dataframe was and they weren’t really familiar with Pandas. So being that no one knows everything i figured i would write a Snake_Byte[] about Pandas. i believe i met the author of pandas – Wes Mckinney at a PyData conference years ago at Facebook. Really nice human and has created one of the most used libraries for data wrangling.

One of the most nagging issues with machine learning, in general, is the access of high integrity canonical training sets or even just high integrity data sets writ large.

By my estimate over the years having performed various types of learning systems and algorithm development, machine learning is 80% data preparation, 10% data piping, 5% training, and 5% banging your head against the keyboard. Caveat Emptor – variable rates apply, depending on the industry vertical.

It is well known that there are basically three main attributes to the integrity of the data: complete, atomic, and well-annotated.

Complete data sets mean analytical results for all required influent and effluent constituents as specified in the effluent standard for a specific site on a specific date.

Atomic data sets are data elements that represent the lowest level of detail. For example, in a daily sales report, the individual items that are sold are atomic data, whereas roll-ups such as invoices and summary totals from invoices are aggregate data.

Well-annotated data sets are the categorization and labeling of data for ML applications. Training data must be properly categorized and annotated for a specific use case. With high-quality, human-powered data annotation, companies can build and improve ML implementations. This is where we get into issues such as Gold Standard Sets and Provenance of Data.

Installing Pandas:

Note: Before you install Pandas, you must bear in mind that it supports only Python versions 3.7, 3.8, and 3.9.

I am also assuming you are using some type of virtual environment.

As per the usual installation packages for most Python libraries:

pip install pandas

You can also choose to use a package manager in which case it’s probably already included.

#import pandas pd is the industry shorthand
import pandas as pd
#check the version
pd.__version__

[2]: '1.4.3'

Ok we have it set up correctly.

So what is pandas?

Glad you asked, i have always thought of pandas as enhancing numpy as pandas is built on numpy. numpy It is the fundamental library of python, used to perform scientific computing. It provides high-performance multidimensional arrays and tools to deal with them. A numPy array is a grid of values (of the same type) indexed by a tuple of positive integers, numpy arrays are fast, easy to understand, and give users the right to perform calculations across arrays. pandas on the other hand provides high-performance, fast, easy-to-use data structures, and data analysis tools for manipulating numeric data and most importantly time series manipulation.

So lets start with the pandas series object which is a one dimensional array of indexed data which can be created from a list or an array:

data = pd.Series([0.1,0.2,0.3,0.4, 0.5])
data

[5]: 0    0.1
     1    0.2
     2    0.3
     3    0.4
     4    0.5
     dtype: float64

The cool thing about this output is that Series creates and wraps both a sequence and the related indices; ergo we can access both the values and index attributes. To double check this we can access values:

[6]: data.values
[6]: array([0.1, 0.2, 0.3, 0.4, 0.5])

and the index:

[7]: data.index
[7]: RangeIndex(start=0, stop=5, step=1)

You can access the associated values via the [ ] square brackets just like numpy however pandas.Series is much more flexible than the numpy counterpart that it emulates. They say imitation is the highest form of flattery.

Lets go grab some data from the LazyWeb_^TM:

If one really thinks about the aspects of pandas.Series it is really a specialized version of a python dictionary. For those unfamiliar a dictionary (dict) is python structure that maps arbirtrary keys to a set of arbitrary values. Super powerful for data manipulation and data wrangling. Taking this is a step further pandas.Series is a structure that maps typed keys to a set of typed values. The typing is very important whereas the type-specific compiled code within numpy arrays makes it much more efficient than a python list. In the same vein pandas.Series is much more efficient python dictionaries. pandas.Series has an insane amount of commands:

Find Series Reference Here.

Next, we move to what i consider the most powerful aspect of pandas the DataFrame. A DataFrame is a data structure that organizes data into a 2-dimensional table of rows and columns, much like a spreadsheet. DataFrames are one of the most common data structures used in modern data analytics because they are a flexible and intuitive way of storing and working with data.

# Python code demonstrate creating 
# DataFrame from dict narray / lists 
# By default addresses.
 
import pandas as pd
 
# intialise data of lists.
data = {'Name':['Bob', 'Carol', 'Alice', ''],
        'Age':[18, 20, 22, 24]}
 
# Create DataFrame
df = pd.DataFrame(data)
 
# Print the output.
print(df)

 [8]:
    Name  Age
0    Bob   18
1  Carol   20
2  Alice   22
3          24       

Lets grab some data. nba.csv is a flat file of NBA statistics of players:

Get the NBA data file here.

i don’t watch or follow sports so i don’t know what is in this file. Just did a google search for csv statistics and this file came up.

# importing pandas package
import pandas as pd
 
# making data frame from csv file
data = pd.read_csv("nba.csv", index_col ="Name")
 
# retrieving row by loc method
first = data.loc["Avery Bradley"]
second = data.loc["R.J. Hunter"]
 
 
print(first, "\n\n\n", second)

[9]:
Team        Boston Celtics
Number                 0.0
Position                PG
Age                   25.0
Height                 6-2
Weight               180.0
College              Texas
Salary           7730337.0
Name: Avery Bradley, dtype: object 


Team        Boston Celtics
Number                28.0
Position                SG
Age                   22.0
Height                 6-5
Weight               185.0
College      Georgia State
Salary           1148640.0
Name: R.J. Hunter, dtype: object

How nice is this? Easy Peasy. It seems almost too easy.

For reference here is the pandas.Dataframe reference documentation.

Just to show how far reaching pandas is now in the data science world for all of you who think you may need to use Spark there is a package called PySpark. In PySpark A DataFrame is equivalent to a relational table in Spark SQL, and can be created using various functions. Once created, it can be manipulated using the various domain-specific-language (DSL) functions  much like your beloved SQL.

Which might be another Snake_Byte in the future.

i also found pandas being used in ye ole #HealthIT #FHIR for as we started this off csv manipulation. Think of this Snake_Byte as an Ouroboros.

This github repo converts csv2fhir ( can haz interoperability? ):

with pd.read_csv(file_path, **csv_reader_params) as buffer:
        for chunk in buffer:

            chunk: DataFrame = execute(chunk_tasks, chunk)

            # increment the source row number for the next chunk/buffer processed
            # add_row_num is the first task in the list
            starting_row_num = chunk["rowNum"].max() + 1
            chunk_tasks[0] = Task(name="add_row_num", params={"starting_index": starting_row_num})

            chunk: Series = chunk.apply(_convert_row_to_fhir, axis=1)

            for processing_exception, group_by_key, fhir_resources in chunk:
                yield processing_exception, group_by_key, fhir_resources

So this brings us to the end of this Snake_Byte. Hope this gave you a little taste of a great python library that is used throughout the industry.

Muzak To Blog By:

Mike Patton & The Metropole Orchestra – Mondo Cane – June 12th 2008 (Full Show) <- A true genius at work!

One other mention on the Muzak To Blog By must go to the fantastic Greek Composer, Evángelos Odysséas Papathanassíou (aka Vangelis) who recently passed away. We must not let the music be lost like tears in the rain, Vangelis’ music will live forever. Rest In Power, Maestro Vangelis. i have spent many countless hours listening to your muzak and now the sheep are truly dreaming. Listen here -> Memories Of Green.

Now… We are going in a loop.

~ Ramakrishna, Springs of Indian Wisdom

First, i trust everyone is safe.

Second, i’ll will be moving the frequency of Snake_Bytes [] to every other Wednesday. This is to provide higher quality information and also to allow me space and time to write other blogs. i trust dear reader y’all do not mind.

Third, i noticed i was remiss in explaining a function i used in a previous Snake_Byte [ ] that of the Python built-in function called range.

Range is a very useful function for, well, creating iterations on variables and loops.

# lets see how this works:
range(4)
[0,1,2,3]

How easy can that be?

Four items were returned. Now we can create a range or a for loop over that list – very meta huh?

Please note in the above example the list starts off with 0. So what if you want your range function to start with 1 base index instead of 0? You can specify that in the range function:

# Start with 1 for intial index
range (1,4)
[1,2,3]

Note the last number in the index in order to be inclusive for the entire index.

Lets try something a little more advanced with some eye candy:

%matplotlib inline
x_cords = range(-50,50)
y_cords = [x*x for x in x_cords]

plt.plot(x_cords, y_cords)
plt.show()

We passed a computation into the loop to compute over the indices of range x in this case.

In one of the previous Snake_Bytes[] i utilized a for loop and range which is extremely powerful to iterate over sequences:

for i in range (3):
    print(i,"Pythons")

0 Pythons
1 Pythons 
2 Pythons

For those that really need power when it comes to indexing, sequencing and iteration you can change the list for instance, as we move across it. For example:

L = [1,2,3,4,5,6]
#no add one to each row 
# or L[1] = L[i] +1 used all 
# the time in matrix operations
for i in range(len(L)): 
    L[i] += 1
print (L)

[2,3,4,5,6,7]

Note there is a more “slick” way to do this with list comprehension without changing the original list in place. However, that’s outside the scope if you will of this Snake_Byte[] . Maybe i should do that for the next one?

Well, i hope you have a slight idea of the power of range.

Also, i think this was more “byte-able” and not tl;dr. Let me know!

Until Then,

#iwshyouwater <- another good one here click!

@tctjr

Muzak To Blog By: Roger Eno & Brian Eno – Mixing Colors (this album is spectacular)

There is geometry in the humming of the strings, there is music in the spacing of the spheres.

Pythagoras

First, i  trust everyone is safe.

Second, this is the SB[3].  We are going to be covering some basics in Python of what constitutes a string, modifying a string, and explaining several string manipulation methods.

I also realized in the last Snake_Byte that i didn’t reference the book that i randomly open and choose the subject for the Snake_Byte.  I will be adding that as a reference at the end of the blog.

Strings can be used to represent just about anything.

They can be binary values of bytes, internet addresses, names, and Unicode for international localization.

They are part of a larger class of objects called sequences.  In fact, python strings are immutable sequences.  Immutability means you cannot change the sequence or the sequence does not change over time.

The most simplistic string is an empty string:

a = “ “ # with either singe or double quotes

There are numerous expression operations, modules, and methods for string manipulations.  

Python also supports much more advanced operations for those familiar with regular expressions (regex) it supports them via re.  Even more advanced operations are available such as XML parsing and the like.

Python is really into strings.

So let us get literal, shall we?

For String Literals there are countless ways to create and manipulate strings in your code:

Single Quotes:

a = `i w”ish you water’

Double Quotes;

A = “i w’ish you water”

Even triple quotes (made me think of the “tres commas” episode from Silicon Valley)

A = ```... i wish you water ```

Single and double quotes are by far the most used.  I prefer double quotes probably due to the other languages i learned before Python.

Python also supports the liberal use of backslashes aka escape sequences.  I’m sure everyone is familiar with said character `\`.  

Escape sequences let us embed bytecodes into strings that are otherwise difficult to type.

So let’s see here:

s = 't\nc\nt\njr'
print (s)
t
c
t
jr

So here i used ‘\n’ to represent the byte containing the binary value for newline character which is ASCII code 10.  There are several accessible representations:

‘\a\’ # bell

‘\b\’ #backspace

‘\f’ # formfeed for all the dot matrix printers we use 

‘\r’ #carriage return

You can even do different Unicode hex values:

‘\Uhhhhhhhh’ #32 bit hex count the number of h’s

With respect to binary file representations of note in Python 3.0 binary file content is represented by an actual byte string with operations similar to normal strings. 

One big difference between Python and another language like C is that that the zero (null) byte doesn’t terminate and in fact, there are no character string terminations in Python.  Also, the strings length and text reside in memory. 

s = 'a\0b\0c'
print (s)
len (s)
abc
5

So what can we do with strings in Python?

Well, we can concatenate:

a = "i wish"
print(len (a))
b = " you water"
print (len(b))
c = a + b
print (len(c))
print (c)
6
10
16
i wish you water

So adding two strings creates a new string object and a new address in memory.  It is also a form of operator overloading in place.  The ‘ + ‘ sign does the job for strings and can add numerics.  You also don’t have to “pre-declare” and allocate memory which is one of the advantages of Python.  In Python, computational processes are described directly by the programmer. A declarative language abstracts away procedural details however Python isn’t purely declarative which is outside the scope of the blog.  

So what else?  Well, there is indexing and slicing:

Strings are ordered collections of characters ergo we can access the characters by the positions within the ordering.

You access the component by providing a numerical offset via square brackets this is indexing.  

S = "i wish you water"
print (S[0], S[4], S[-1])
i s r

Since we can index we can slice:

S = "i wish you water"
print (S[1:3], S[2:10], S[9:10])
w wish you u

Slicing is a particular form of indexing more akin to parsing where you analyze the structure.

Python once again creates a new object containing the contiguous section identified by the offset pair.  It is important to note the left offset is taken to be the inclusive lower bound and the right is the non-inclusive upper bound. The inclusive definition is important here:  Including the endpoints of an interval. For example, “the interval from 1 to 2, inclusive” means the closed interval written [1, 2].  This means Python fetches all items from the lower bound up to but not including the upper bound. 

What about changing a string?

Let’s try it:

S = "i wish you water"
S[0] = "x"
---------------------------------------------------------------------------
TypeError Traceback (most recent call last) <ipython-input-67-a6fd56571822> 
in <module> 1 S = "i wish you water" ----> 2 S[0] = "x"
TypeError: 'str' object does not support item assignment

Ok, what just happened?  Well, remember the word immutable? You cannot change it in place.

To change a string you need to create a new one through various methods.  In the current case we will use a combination of concatenation, indexing, and slicing to bring it all together:

S = "i wish you water"
S = 'x ' + S[2]  +  S[3:17]
print (S)
x wish you water

This brings us to methods.

Stings in Python provide a set of methods that implements much more complex text processing.  Just like in other languages a method or function takes parameters and returns a value. A “method” is a specific type of function: it must be part of a “class”, so has access to the class’ member variables. A function is usually discrete and all variables must be passed into the function.

Given the previous example there is a replace method:

S = "i wish you water"
S = S.replace ('i wish you water', 'x wish you water')
print (S)
x wish you water

Let’s try some other methods;

# captialize the first letter in a string:
S = "i wish you water"
S.capitalize()
'I wish you water'

# capitalize all the letters in a string:
S = "i wish you water"
S.upper()
'I WISH YOU WATER'

# check if the string is a digit:
S = "i wish you water"
S.isdigit()
False

# check it again:
S = "999"
S.isdigit()
TRUE

# strip trailing spaces in a string:
S = "i wish you water     "
x = S.rstrip()
print("of all fruits", x, "is my favorite") 
of all fruits i wish you water is my favorite

The list is seemingly endless.  

One more caveat emptor you should use stings methods, not the original string module that was deprecated in Python 3.0

We could in fact write multiple chapters on strings by themselves.  However, this is supposed to be a little nibble of what the Snake language can offer.  We have added the reference that we used to make this blog at the end.  I believe it is one of the best books out there for learning Python.

Until Then,

#iwishyouwater

Tctjr

References:

Learning Python by Mark Lutz

Muzak To Blog By:  Mr. Robot Vol1 Original Television Soundtrack