McConkey: Chapter 9

This chapter was fairly easy to read and was great for starting to understand how geometry objects (something everyone who has worked with ArcGIS should be familiar with) works with Python.

Chapter 9 goes over how to work with geometries, otherwise known as points, polylines, and polygons. This includes being able to read the properties of geometry objects from preexisting features as well as creating geometry objects with coordinate information.

The chapter explains how to use geometry cursors which allow you to work with or edit the vertices of a geometry feature object. To set a cursor you must use a geometry token rather than a field name.  For example, the SHAPE@ token gives access to the full geometry object and its properties.  Using specific geometry tokens may be easier such as SHAPE@XY, or SHAPE@LENGTH.  (SHAPE@XY results in a tuple of x,y coordinates for the feature’s centroid, and the SHAPE@LENGTH returns the feature’s length.)

When working with a polygon with holes, you may have some trouble. A geometry object that consists of a polygon with holes will return an exterior ring with one or more interior rings. Rings are made up of paths, or a series of vertices that begin with a starting vertex and finish with an end vertex. Exterior rings are clockwise and interior rings are counterclockwise. (For more info on polygons with holes look on page 299.)

There are several classes that can be used to make geometry objects. By using the Point object you can create a single vertice. You can also make multiple  Point objects to create polylines or polygon features.

 

 

 

McConkey – ArcPy: An Introduction (Video Tutorial Notes)

This video tutorial is part of the Python Essentials Learning Plan. The video is about 40 minutes long and gives a basic overview of using Python in ArcGIS.  The video has two presenters and switches back and forth between them during the presentation. One of them focuses are providing information about Python and its applications while the other actually demonstrates how the code is written for some of these applications. A lot of the details are things already covered in the book, but I will include several of the points made to provide an overview of the video.

Python makes coding a lot simpler by automating repetitive tasks such as data analysis and processing.  Scripts can be saved in a text file so they can be shared or used at a later time.  Unique tools that are not found in ArcGIS programs can be made from scratch by using Python. In fact, several tools in ArcGIS, that you may be familiar with are made from Python coding.

Python is dynamically typed. It can support strings, numbers, and functions as well as work with and produce files, lists, and dictionaries. (Dictionaries – objects with key-value pairs).

ArcPy is an API, or Application Programming Interface, and works for programs such as ArcGIS Pro, ArcGIS Server, and ArcGIS Desktop. It must be imported before running certain tools for those applications. If you have these applications you already have ArcPy, you just need to import it in a scripting window. ArcPy is composed of a series of unique modules, functions, classes, and tools.

When you open a tool in ArcGIS, you can select the blue question mark at the top right of the tool window to open syntax help, which will explain the function of the tool and its parameters. You will also be able to see a code sample written in Python at the bottom of the syntax help page which may be very useful when writing scripts for that tool.

You can open a Python window directly into ArcGIS Pro by going to the view tab and clicking the Python Window button.  If you type “arcpy.” then a list of available modules and functions will appear. You can narrow down your search by continuing to type out the tool you want and it should appear above. If there are multiple tools available you can scroll to the desired one with the arrow keys and then press the tab button to make a selection.

The parameters of the tool will be seen above and the Python window may automatically show optional inputs that are within your tab o contents. For example, if you have a series of shapefiles in your table of contents and the first parameter is an input feature class, then those shapefiles may appear above. (You can then use the arrow keys to scroll and then press tab to auto complete/select the layer.)  Once you one the tool, the results should be available in the history pane.

Another way to access Python in ArcGIS Pro is by opening a notebook. You can access or create a notebook by going to the insert tab and  pressing the New Notebook or Add Notebook buttons below.

 

 

Wade: Chapter 8: Manipulating Spatial and Tabular Data

 

Mack Wade

Chapter 8: manipulating Spatial and Tabular Data

 

Terms:

Comma separated Value (CSV): a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format. 

Data Lock: There are two states of a lock i.e locked and unlocked. A lock is a class in the threading module whose object is generated in the unlocked state and has two primary methods i.e acquire() and release() .

Mode:  refers to the most frequently occurring number found in a set of numbers.

Parsing: the processing of a piece of python program and converting these codes into machine language.

Postfix clause: An SQL postfix clause is positioned in the second position and will be appended to the SELECT statement, following the where clause. The SQL postfix clause is most commonly used for clauses such as ORDER BY.

Prefix clause: The PREFIX clause declares any abbreviations for URIs that you want to reference in a query. You can declare prefixes to simplify query text if your data includes long URI names. If you do not declare prefixes, you must include the full URI names in the query.

SQL clause: Clauses are in-built functions available to us in SQL. With the help of clauses, we can deal with data easily stored in the table. Clauses help us filter and analyze data quickly. When we have large amounts of data stored in the database, we use Clauses to query and get data required by the user.

SQL expression: An expression is a combination of one or more values, operators, and SQL functions that evaluate to a value. An expression generally assumes the datatype of its components.

SQL keyword: the reserved words that are used to perform various operations in the database. There are many keywords in SQL and as SQL is case insensitive, it does not matter if we use for example SELECT or select.

Structured query language (SQL): a programming language designed to get information out of and put it into a relational database. Queries are constructed from a command language that lets you select, insert, update and locate data.

Triple Quotes: allowing strings to span multiple lines, including verbatim NEWLINEs, TABs, and any other special characters. The syntax for triple quotes consists of three consecutive single or double quotes.

Review Questions

  1. What are the three different cursors, and what purpose does each one serve?

cursor.fetchall() fetches all the rows of a query result. It returns all the rows as a list of tuples. An empty list is returned if there is no record to fetch.

cursor.fetchmany(size) returns the number of rows specified by size argument. When called repeatedly, this method fetches the next set of rows of a query result and returns a list of tuples. If no more rows are available, it returns an empty list.

cursor.fetchone() method returns a single record or None if no more rows are available

  1. Explain how data locks occur and how they can be removed.
    1. A lock is a class in the threading module whose object generated in the unlocked state and has two primary methods i.e acquire() and release() .
  2. When writing SQL expressions, why are quotation marks sometimes an issue? 
    1. Single quotes are used to indicate the beginning and end of a string in SQL. Double quotes generally aren’t used in SQL, but that can vary from database to database. Stick to using single quotes.

Wade: Chapter 7: Debugging and Error Handling

Chapter 7: Debugging and Error Handling

Terms:

Breakpoint:  used to interrupt a running program immediately before the execution of a programmer-specified instruction. This is often referred to as an instruction breakpoint. … Breakpoints can also be used to interrupt execution at a particular time, upon a keystroke etc.

Commenting out code: to use comment syntax to remove something from the parsed code.

Custom class:  a developer defined class, based on one of the stock classes 

Debugging: the complete control over the program execution.

Error handling: the process of responding to the occurrence of exceptions – anomalous or exceptional conditions requiring special processing – during the execution of a program.

Exception:

Exception object: 

An event that occurs during the execution of a program that disrupts the normal flow of instructions is called an exception. 

Logic error: a mistake in a program’s source code that results in incorrect or unexpected behavior

Traceback: a report containing the function calls made in your code at a specific point.

Wade: Chapter 6: Exploring Spatial Data

Chapter 6: Exploring Spatial Data

Terms:

Dynamic (property): terminologies for attributes that are defined at runtime, after creating the objects or instances.

List comprehension: an easy and compact syntax for creating a list from a string or another list. It is a very concise way to create a new list by performing an operation on each item in the existing list. List comprehension is considerably faster than processing a list using the for loop.

System path

Wildcard: a symbol used to replace or represent one or more characters. Wildcards are used in computer programs, languages, search engines, and operating systems to simplify search criteria.

 

Review Questions:

  1. What are the key differences and similarities between the describe () and da.describe () functions?
    1. The Describe function returns a dictionary with multiple properties, such as data type, fields, indexes, and many others. The dictionary’s keys are dynamic, meaning that depending on what data type is described, different properties will be available for use.
  2. Explain the difference between system and catalog paths, and how they affect exploring data in a folder.
    1. The main difference between an absolute and a relative path is that an absolute path specifies the location from the root directory whereas relative path is related to the current directory.
  3. What is list comprehension, and when would you consider usng it.
    1. List Comprehension is an easy and compact syntax for creating a list from a string or another list. It is a very concise way to create a new list by performing an operation on each item in the existing list. List comprehension is considerably faster than processing a list using the for loop.

McConkey: Chapter 8

Chapter 8 covers manipulating spatial and tabular data. You may want to briefly refamiliarize yourself with “for” and “while” loops before reading this chapter. It would also be wise to familiarize yourself with Structured Query Language (SQL).

Important details to remember:

– Once a script creates an exclusive lock, the lock will continue to exist until the script releases the lock.

Terms

cursor – is a database technology term for accessing a set of records in a table. It works the way list functions work.

search cursor – retrieves rows

insert cursor – inserts rows

update cursor – updates and deletes rows

rows – records in a table

with statements – Good to use when you have two operations that you want to run as a pair

data locks – prevent multiple processes from changing the same table at the same time in two different applications. (includes shared lock and exclusive locks)

shared lock – is applied when a table or dataset is accessed

exclusive lock – are applied when changes are being made to a table or dataset

parsing – splitting the fully qualified name for a dataset into its components

Table 8.1 (pg 248) describes the cursor methods of each class

McConkey: Chapter 7

Chapter 7 reviews the types of errors you can come across while scripting and the most common procedures used to solve them.

Python IDEs have built-in error checking processes that will alert you of any known errors in the script. If you read the error carefully, you may be able to fix the error yourself, without seeking further help. You should keep indentation and spacing consistent in your scripts or you may produce an error. Syntax errors may be caught as they appear in Spyder and PyCharm. Spyder will display a small red x which opens a pop-up window for syntax errors that appear as you type. Spyder can catch other types of errors such as inconsistent indentation. In PyCharm you can produce a general report to see what types of errors are in your script by clicking Code (at the top of the menu) > Inspect code and then selecting the current script.

There are multiple ways to debug your code. The main ways include carefully reading error messages, adding print messages to your script, selectively commenting out code, or using a python debugger. Debugging will not tell you why a script failed, but it will tell you the line in which it failed. (See pg 224-234 for reference of these techniques.)

You can help handle exceptions by using a “try-except” statement (see pg 237). You can use these to produce text that can help you understand what went wrong. An example for a tuple is seen below:

except (ZeroDivisionError, ValueError):
print (“Your entries were not valid.”)

Using a Python debugger (pg 229):
1. Check for syntax errors then save the script.
2. Set breakpoints, allowing you to pause running the script at specific lines.
3. Run the script with a debugger.
4. Walk through the script while looking out for values of important variables.

Syntax errors – errors involved with spelling, punctuation, indentation, or other conventions of the Python language. If you click Run > Check Module and a syntax error appears, a red bar will appear on the line of which the error resides.

Excepts – Events that have no real error but are a result of the data and the script run. An example of this would be to write a script to produce a count of the feature classes in a feature class list. If there are zero feature classes in the list then the code may run perfectly fine but produce no output. This is not truly an error as the code has run correctly.

commenting out code – Placing a double hashtag (##) before lines of code helps cancel out that line in order to isolate parts of the script. This helps find errors in the script.

breakpoints – The code will run until it reaches a breakpoint. They are good for debugging scripts. To set a breakpoint, right-click a line of code and select Set Breakpoint. This will highlight the line yellow.

McConkey: Chapter 4

This chapter took a lot longer than the others but was probably the most engaging as you will finally be able to understand the concepts behind Python scripting. This chapter focuses on the fundamentals of Python scripting so that you will be able to make your own scripts. It would be wise to really pay attention to the chapter and to use it as a reference when creating your own code. Below is only some of the information provided by the chapter.

Python uses several different data types in its language. Data types determine what type of values can be held by an object. Some are the most common are strings, numbers, Booleans, and lists. Strings are usually text but can include numbers and other types of characters as well. Numbers can appear as integers (whole numbers) or floats (numbers with a decimal point). Boolean values are either noted as True or False and are useful for establishing or evaluating conditions. Lists are just as they sound and can contain a series of numbers, text, or other values.

You want to be careful when naming objects in Python as there are a number of ways of producing an error with a name (see pg 91-92). The chapter encourages consistency with how you name objects. One of the prefered methods is utilizing snake_case where each word is uncapitalized and instead of using spaces, underscores are used between each word to increase legibility. Python has several built-in functions and an error will result if you try to name an object with the same name as a function.

Furthermore, when writing scripts it may be wise to includes comments to provide context for the script. You can add comments by writing a line preceded by a number sign (#). Any text that comes after the number sign will not be considered code. An example of this could be (# Create output feature class) with the lines of code to create an output feature class found below.

Table References:

Table 4.1. Common operators for integers and floats in Python (pg. 89)

(Examples: * stands for multiplication and ** stands for exponent)

Table 4.2 Built-in functions in Python (pg. 102)

(Example: asbs(x) with “x” being some number will return the absolute value of that number)

Table 4.3 Comparison operators in Python

(Examples: == means equal to with != means not equal to)

McConkey: Chapter 3

Chapter 3 gives the rundown on how geoprocessing works in ArcGIS Pro with details on tools, models, and scripting. To complete some form of geoprocessing you must input some form of data (a feature class, raster, or table) into a geoprocessing tool (often with additional parameters) to produce an output.

Just like in ArcGIS Mapper, ArcGIS Pro contains toolboxes with tools inside them. You can locate geoprocessing tools in a variety of ways but you must first open the geoprocessing pane to do this. To do this navigate to Analysis Tab and click Tools. You can now search for tools by name and add frequently used tools to a list of favorites. Recently used tools will also show up in the favorites. When searching for tools the description of what the tool does will also appear. By right-clicking a tool and selecting “Batch” you can run the tool to produce multiple outputs with similar parameters.

Information about other types of tools is mentioned in the chapter. Most tools are built into the program but you can also utilize script tools made from Python code and model tools created using ModelBuilder. You can make your own model processes by opening ModelBuilder and creating your own models. The easiest way to open ModelBuilder is to navigate to the Analysis tab and click on the ModelBuilder button. This opens a window into which you can input data and tools and create connections between them. ModelBuilder is similar to Python coding but is more limited. It is still good to become familiar with ModelBuilder as it essentially allows you to create scripts visually rather than with text.