# Python for DevOps  ## Metadata - Author: [[Noah Gift, Kennedy Behrman, Alfredo Deza, Grig Gheorghiu]] - Full Title: Python for DevOps - Category: #python #devops ## Highlights - IPython offers introspection (the ability to dynamically get information about objects), syntax highlighting, special magic commands (which we touch on later in this chapter), and many more features, making it a pleasure to use for exploring Python. ([Location 200](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=200)) - A spin-off from the iPython project, the Jupyter project allows documents containing text, code, and visualizations. These documents are powerful tools for combining running code, output, and formatted text. Jupyter enables the delivery of documentation along with the code. It has achieved widespread popularity, especially in the data science world. ([Location 208](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=208)) - Procedural programming is the issuing of instructions to a computer in an ordered sequence: ([Location 217](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=217)) - Python variables use dynamic typing. In practice, this means that they can be reassigned to values of different types or classes: ([Location 232](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=232)) - Functions are statements grouped as a unit. ([Location 259](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=259)) - for loops allow you to repeat a block of statements (a code block) once for each member of a sequence (ordered group of items). ([Location 337](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=337)) - while loops repeat a block as long as a condition evaluates to True: ([Location 358](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=358)) - Exceptions are a type of error causing your program to crash if not handled (caught). Catching them with a try-except block allows the program to continue. These blocks are created by indenting the block in which the exception might be raised, putting a try statement before it and an except statement after it, followed by a code block that should run when the error occurs: ([Location 379](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=379)) - The essential concepts to understand when working with objects are class instantiation (creating objects from classes) and dot syntax (the syntax for accessing an object’s attributes and methods). ([Location 401](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=401)) - Objects store data in attributes. These attributes are variables attached to the object or object class. Objects define functionality in object methods (methods defined for all objects in a class) and class methods (methods attached to a class and shared by all objects in the class), which are functions attached to the object. ([Location 413](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=413)) - In Python documentation, functions attached to objects and classes are referred to as methods. ([Location 416](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=416)) - Sequences are a family of built-in types, including the list, tuple, range, string, and binary types. Sequences represent ordered and finite collections of items. ([Location 432](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=432)) - Lists, one of the most commonly used Python data structures, represent an ordered collection of items of any type. The use of square brackets indicates a list syntax. ([Location 494](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=494)) - The contents of one list can be added to another using the extend method: ([Location 526](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=526)) - One of the most potent and idiomatic Python features, list comprehensions, allows you to use the functionality of a for loop in a single line. ([Location 545](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=545)) - Python f-strings use the same formatting language as the format method, but offer a more straightforward and intuitive mechanism for using them. f-strings are prepended with either f or F before the first quotation mark. ([Location 678](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=678)) - We highly recommend using f-strings for the majority of your string formatting. They combine the power of the specification mini-language with a simple and intuitive syntax. ([Location 691](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=691)) - Aside from strings and lists, dicts may be the most used of the Python built-in classes. A dict is a mapping of keys to values. The lookup of any particular value using a key is highly efficient and fast. The keys can be strings, numbers, custom objects, or any other nonmutable type. ([Location 708](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=708)) - A mutable object is one whose contents can change in place. Lists are a primary example; the contents of the list can change without the list’s identity changing. Strings are not mutable. You create a new string each time you change the contents of an existing one. ([Location 710](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=710)) - A more intuitive solution is to use the get() method. If you have not defined a key in a dict, it returns a supplied default value. ([Location 749](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=749)) - Similar to list comprehensions, dict comprehensions are one-line statements returning a dict by iterating through a sequence: ([Location 766](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=766)) - a function is a mechanism for encapsulating a block of code. ([Location 775](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=775)) - Functions are objects. They can be passed around, or stored in data structures. You can define two functions, put them in a list, and then iterate through the list to invoke them: ([Location 819](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=819)) - When you need to create a very limited function, you can create an unnamed (anonymous) one using the lambda keyword. Generally, you should limit their use to situations where a function expects a small function as a argument. ([Location 833](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=833)) - In addition to character sets, Python’s re offers character classes. These are premade character sets. ([Location 929](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=929)) - You can produce an iterator object using the finditer method. This object processes text until it finds a match and then stops. ([Location 983](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=983)) - Lazy evaluation is the idea that, especially when dealing with large amounts of data, you do not want process all of the data before using the results. ([Location 1036](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1036)) - You can use generators in a similar way as range objects. They perform some operation on data in chunks as requested. They pause their state in between calls. This means that you can store variables that are needed to calculate output, and they are accessed every time the generator is called. ([Location 1040](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1040)) - To write a generator function, use the yield keyword rather than a return statement. ([Location 1042](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1042)) - We can use generator comprehensions to create one-line generators. They are created using a syntax similar to list comprehensions, but parentheses are used rather than square brackets: ([Location 1072](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1072)) - You can use IPython to run shell commands. This is one of the most compelling reasons to perform DevOps actions in the IPython shell. ([Location 1091](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1091)) - If you get in the habit of using IPython, you should also get in the habit of using built-in magic commands. They are essentially shortcuts that pack a big punch. Magic commands are indicated by prepending them with %%. ([Location 1114](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1114)) - Another very useful command, %who, will show you what is loaded into memory. It comes in quite handy when you have been working in a terminal that has been running for a long time: ([Location 1134](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1134)) - One of Python’s most powerful features is its ability to manipulate text and files. In the DevOps world, you are continually parsing, searching, and changing the text in files, whether you’re searching application logs or propagating configuration files. ([Location 1156](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1156)) - You can use the open function to create a file object that can read and write files. ([Location 1165](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1165)) - It is a good practice to close a file when you finish with it. Python closes a file when it is out of scope, but until then the file consumes resources and may prevent other processes from opening it. ([Location 1181](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1181)) - A handy way of opening files is to use with statements. You do not need to close a file explicitly in this case. Python closes it and releases the file resource at the end of the indented block: ([Location 1192](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1192)) - If you are opening a binary file, such as a .jpeg image, you are likely to corrupt the data by this conversion if you open it as text. You can, however, read binary files by appending a b to mode: ([Location 1203](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1203)) - The open function creates a file if it does not already exist and overwrites if it does. If you want to keep existing contents and only append the file, use the append flag a. ([Location 1228](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1228)) - The Javascript Object Notation (JSON) format is widely used to store simple structured data in modern web services. It uses two data structures: a mapping of key-value pairs similar to a Python dict and a list of items somewhat similar to a Python list. It defines data types for numbers, strings, booleans (which hold true/false values), and nulls (empty values). ([Location 1249](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1249)) - The pprint module automatically formats Python objects for printing. Its output is often more easily read and is a handy way of looking at nested data structures. ([Location 1300](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1300)) - Another language commonly used in configuration files is YAML (“YAML Ain’t Markup Language”). It is a superset of JSON, but has a more compact format, using whitespace similar to how Python uses it. Ansible is a tool used to automate software configuration, management, and deployment. Ansible uses files, referred to as playbooks, to define actions you want to automate. ([Location 1321](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1321)) - Another language widely used for representing structured data is Extensible Markup Language (XML). It consists of hierarchical documents of tagged elements. Historically, many web systems used XML to transport data. One such use is for Real Simple Syndication (RSS) feeds. RSS feeds are used to track and notify users of updates to websites and have been used to track the publication of articles from various sources. ([Location 1362](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1362)) - XML allows for namespacing (using tags to group data). ([Location 1397](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1397)) - The csv reader object iterates through the .csv file one line at a time, allowing you to process the data one row at a time. Processing a file this way is especially useful for large .csv files that you do not want to read into memory all at once. ([Location 1427](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1427)) - The Pandas package is a mainstay in the data science world. It includes a data structure, the pandas.DataFrame, which acts like a data table, similar to a very powerful spreadsheet. ([Location 1430](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1430)) - formats. One widely used format is the Common Log Format (CLF). A variety of log analysis tools can understand this format. Below is the layout of this format: <IP Address> <Client Id> <User Id> <Time> <Request> <Status> <Size> What follows is an example line from a log in this format: 127.0.0.1 - swills [13/Nov/2019:14:43:30 -0800] "GET /assets/234 HTTP/1.0" 200 2326 ([Location 1479](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1479)) - Rather than loading the whole file into memory as you have done up until now, you can read one line at a time, process the line, and then move to the next. The lines are removed from memory automatically by Python’s garbage collector, freeing up memory. ([Location 1537](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1537)) - Notice that you can nest the with statements to open two files at once and loop through the source file object one line at a time. ([Location 1552](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1552)) - To be secure, user passwords must be stored encrypted. A common way to handle this is to use a one-way function to encrypt the password into a bit string, which is very hard to reverse engineer. Functions that do this are called hash functions. ([Location 1584](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1584)) - The cryptography library is a popular choice for handling encryption problems in Python. It is a third-party package, so you must install it with pip. Symmetric key encryption is a group of encryption algorithms based on shared keys. ([Location 1602](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1602)) - Asymmetric key encryption uses a pair of keys, one public and one private. The public key is designed to be widely shared, while a single user holds the private one. ([Location 1626](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1626)) - One very popular asymmetric key algorithm is Rivest-Shamir-Adleman (RSA), which is widely used for communication across networks. ([Location 1629](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1629)) - The os module is one of the most used modules in Python. This module handles many low-level operating system calls and attempts to offer a consistent interface across multiple operating systems, which is important if you think your application might run on both Windows and Unix-based systems. ([Location 1671](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1671)) - The os.path module offers a plethora of path-related methods for creating and manipulating paths as strings. ([Location 1740](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1740)) - Using files to configure an application at runtime is a common practice; files in Unix-like systems are named by convention as dotfiles ending with rc. Vim’s .vimrc file and the Bash shell’s .bashrc are two common examples. ([Location 1784](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1784)) - The os module offers a convenience function for walking directory trees called os.walk. This function returns a generator that in turn returns a tuple for each iteration. ([Location 1910](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1910)) - The pathlib library represents paths as objects rather than strings. ([Location 1932](https://readwise.io/to_kindle?action=open&asin=B082P97LDW&location=1932))