Author: Jay Tyler McConkey

Jay McConkey OWU '23 Favorite Color: Blue Special Interests: Running, Fungi, Reading

McConkey: Course Wrap Up

Posted onAuthorJay Tyler McConkey

Python Scripting for ArcGIS Pro

The book is helpful to start to get an understanding of Python, but it is difficult to interpret sometimes as it is prone to using jargon. It does define the jargon, or scripting terms, and explain how the terms relate to each other, but it does get confusing sometimes if you are unfamiliar with a lot of these concepts. The weakest part of the book is that it reads like a walkthrough, but there is no actual data package that comes with it so you are not able to perform all of the scripts it explains. Although the book works as a learning tool, I believe it works better as a reference guide.

Individual Project: ERSI Learning Courses

Even though I was not able to make as much progress as I wanted with the ESRI learning course, I do think the courses make it easier to learn Python much more than the book as you are able to actively learn through working with datasets and writing scripts. Additionally, you are immediately able to see the results of your scripts which makes you feel much more accomplished. There are many learning courses that you should be able to access and many of them are introductory level courses. The directions for the learning courses that I worked through were pretty thorough, so the biggest learning curve comes from navigating PyCharm than writing the code itself.

The work that was completed was certainly worthwhile; however, I hit an error that caused more errors and delays which prohibited me from completing the second learning course I was on.  The main error occurred after I added a python interpreter to PyCharm as instructed in the “Python Scripting for Geoprocessing” course. After the interpreter was added and I ran the completed code and an error code appeared in the console. It reads as follows: C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\numpy\__init__.py:143: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the-box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service from . import _distributor_init

I tried following the link but that lead to many dead ends. I also discovered that PyCharm stopped recognizing arcpy as a module around this time. This may have been from the version of Python I was using but after I tried changing it, the error still appeared.

I then tried following the file path listed in the error code: “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\numby\__init__.py:143.” I could not find “__init__.py:143” in the numpy folder.

I went to the settings and clicked on the Python interpreter and saw that the packages are actually listed. I found the mkl-service package in the list so I wasn’t sure why it wasn’t working. I clicked the plus sign to add a package, searched for the mkl-service package, and clicked install. However, an error occurred claiming that I did not “have the write permissions to the target environment.

I worked with someone from OWU tech support and we discovered that I did have administration access and we tried a couple of alternative methods but determined that the error may stem from a problem with the actual PyCharm software itself. Hopefully, you don’t encounter any of these errors as they are a pain to deal with.

Overall, I am proud of the work I was able to complete. I definitely think the learning courses are worth exploring and should maybe be the primary focus of the course, rather than reading the book. I think I retained knowledge better from the courses than from the book itself, but this could be a reflection of my learning style. The book is still a worthy investment. Taking down notes will slow your progress, but they definitely will help you retain more information.

McConkey: Python Scripting for Geoprocessing (ERRORS)

Posted onAuthorJay Tyler McConkey

Right off the bat, this course seems to cover directions in more detail than in the last course, Python for Everyone. I would actually recommend taking this course first so you can better interpret the directions of the other course. That is of course if you don’t encounter the same errors I came across. Sadly, I was not able to complete this course during the semester as I could not fix these errors.

Errors:

In the first exercise, an error may occur at step 5 when setting the environment for the script. When you set the Python Interpreter an error message may pop up in the Python console. It appears as written below (you may have to scroll to read the whole thing):

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\numpy\__init__.py:143: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the-box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init

Basically, it wants you to install something called an “mkl service package.” I discovered if you go into the settings and look at the interpreter there is actually an add (plus sign) button that you can add packages. I clicked the add button and made a search for the mkl service package. When I found it I pressed install but some type of administration error occurred. The error may be from the version of Python that you are using so you might have to do some research to find what version of Python works with ArcPro, which I believe is Python 3.7. I also tried to download this version but I am not sure if it was actually successful.

The script for exercise 1 should look like this:

import arcpy
# Set environments
arcpy.env.workspace = r"C:EsriTraining\PythonGP\Data\SanJuan.gdb"
arcpy.env.overwriteOutput = True
# Set parameters used to join the BufferDistance table to the Roads feature class
inFeatures = "Roads"
inField = "ROUTE_TYPE"
joinTable = "BufferDistance"
joinField = "ROUTE_TYPE"
# Script Joins table to feature class
arcpy.JoinField_management(inFeatures, inField, joinTable, joinField)
# Set parameters used to buffer Roads feature class
outBuffers = "RoadBuffers"
buffField = "DISTANCE"
# Buffer the roads based on DISTANCE attribute
arcpy.Buffer_analysis(inFeatures, outBuffers, buffField)

 

McConkey: Python for Everyone (OnlineCourse)

Posted onAuthorJay Tyler McConkey

Overview of the Course:

This course services as both an introduction and walkthrough basic Python scripting. The course is composed of reading material with easy to understand figures and exercises with step-by-step instructions for a hands on experience. The course also has simple multiple choice questions with immediate feedback for improved understanding. There is some supplemental reading provided by some links in the beginning that took about 25 minutes to complete.  At the end of the course there is a brief review and then 10 question exam. The exam generates different questions if you go off the page and you will need an 8 out 10 to receive certification that you passed the course. (The exam is not that difficult, but I do not know if you are able to take it multiple times as I passed it on my first attempt. I recommend double checking your answers before submitting.)

The walkthrough sections are a bit more challenging and are definitely going to make the course longer than it estimates. This is especially true if you are unfamiliar with PyCharm (which you will be asked to download) as you will be spending a good deal of time getting your bearings with software.  If there are any courses that go over the basic layout of PyCharm I would recommend taking it before tackling this course. You can get through the course if you’re not familiar with PyCharm, but it will take you some time getting used to it.

You will have to download PyCharm and some data. The data did not take long to download and there is a link provided so you can download PyCharm. The PyCharm download is more complicated and will take some time (~30 minutes). If you’re getting the free trail version it should last about 30 days and you will have to make a JetBrains account. Also, when you have downloaded the data you should put it in a folder named ErsiTraining in C so it resembles the walkthrough. You will actually want to extract the data before this step. The data path for the first exercise should look something like this: C:\EsriTraining\PythEveryone\RunningScripts/Polk_County. (I extracted the data after putting it into the EsriTraining folder so there were two PythEveryone folders back-to-back. Oops.)

Notes from the Course:

When choosing a script environment you may want to consider what you are trying to accomplish. If you are trying to test geoprocessing workflows and easily verify the results, you may want to stick with ArcGIS’s built in Python window. However, if you want to work with tools to enter, edit, and check syntax within your code, you may want to use an IDE, such as PyCharm. PyCharm also includes an integrated debugging environment.

Scripts can be built from data types, statements or functions. Statements and F=function both perform an action, but only functions return a value.

Tuples are a sequence of items, just like a list; however, tuples cannot be changed. Tuples are useful for when the sequence is significant in a script. Tuples are closed in parenthesis while lists are surrounded by brackets.

Reflection for the Exercises:

Make sure you follow the directions as best you can. If something isn’t working, you probably misread or skipped over something. If your positive you did not misread anything, try not to get to frustrated. You can try troubleshooting and if that does not work, it may be best to keep moving on. Every if you biffed one step (which can mess up the script in the end) it is better to see the process that the course is trying to teach you. Getting the end results are optimal though.

Not all of the directions are described as thoroughly as they should, especially when it comes to working with PyCharm. The exercise walkthroughs provide pictures that go along with the types of coding you will be asked to write out. (This is great but they do not provide images of the PyCharm screen  to improve your navigation of the software.) One thing you will have to look out for that some walkthroughs do not mention that you have to use print statements to make sure your scripts are working. A lot of times the walkthrough will tell you to write a script and then press enter and see the immediate output. From what I experienced this is misleading as this does not happen. Instead, I had to make a print statement and then run the script to see the output. (The output will be visible in the Python Console. This will appear on the bottom half of the screen when you run your scripts.) The good thing about this type of work is that even though it can be frustrating, the more you do what you intended, the more confident you get.

Review of Course:

Some of the directions for the exercises are a bit vague, especially when giving directions in PyCharm. I think PyCharm my have gone through an update so some of the actions you need to take may not directly correspond with what is being directed. It definitely took me longer than the estimated time, but that is probably due to me never using PyCharm before and only using ArcGIS Pro a few times before attempting this course. The course is in no way perfect but it does combine different learning styles and is well structured. The walkthroughs could be more thorough, but they also help you apply what you’ve learned.

McConkey: Chapter 9

Posted onAuthorJay Tyler McConkey

This chapter was fairly easy to read and was great for starting to understand how geometry objects (something everyone who has worked with ArcGIS should be familiar with) works with Python.

Chapter 9 goes over how to work with geometries, otherwise known as points, polylines, and polygons. This includes being able to read the properties of geometry objects from preexisting features as well as creating geometry objects with coordinate information.

The chapter explains how to use geometry cursors which allow you to work with or edit the vertices of a geometry feature object. To set a cursor you must use a geometry token rather than a field name.  For example, the SHAPE@ token gives access to the full geometry object and its properties.  Using specific geometry tokens may be easier such as SHAPE@XY, or SHAPE@LENGTH.  (SHAPE@XY results in a tuple of x,y coordinates for the feature’s centroid, and the SHAPE@LENGTH returns the feature’s length.)

When working with a polygon with holes, you may have some trouble. A geometry object that consists of a polygon with holes will return an exterior ring with one or more interior rings. Rings are made up of paths, or a series of vertices that begin with a starting vertex and finish with an end vertex. Exterior rings are clockwise and interior rings are counterclockwise. (For more info on polygons with holes look on page 299.)

There are several classes that can be used to make geometry objects. By using the Point object you can create a single vertice. You can also make multiple  Point objects to create polylines or polygon features.

 

 

 

McConkey – ArcPy: An Introduction (Video Tutorial Notes)

Posted onAuthorJay Tyler McConkey

This video tutorial is part of the Python Essentials Learning Plan. The video is about 40 minutes long and gives a basic overview of using Python in ArcGIS.  The video has two presenters and switches back and forth between them during the presentation. One of them focuses are providing information about Python and its applications while the other actually demonstrates how the code is written for some of these applications. A lot of the details are things already covered in the book, but I will include several of the points made to provide an overview of the video.

Python makes coding a lot simpler by automating repetitive tasks such as data analysis and processing.  Scripts can be saved in a text file so they can be shared or used at a later time.  Unique tools that are not found in ArcGIS programs can be made from scratch by using Python. In fact, several tools in ArcGIS, that you may be familiar with are made from Python coding.

Python is dynamically typed. It can support strings, numbers, and functions as well as work with and produce files, lists, and dictionaries. (Dictionaries – objects with key-value pairs).

ArcPy is an API, or Application Programming Interface, and works for programs such as ArcGIS Pro, ArcGIS Server, and ArcGIS Desktop. It must be imported before running certain tools for those applications. If you have these applications you already have ArcPy, you just need to import it in a scripting window. ArcPy is composed of a series of unique modules, functions, classes, and tools.

When you open a tool in ArcGIS, you can select the blue question mark at the top right of the tool window to open syntax help, which will explain the function of the tool and its parameters. You will also be able to see a code sample written in Python at the bottom of the syntax help page which may be very useful when writing scripts for that tool.

You can open a Python window directly into ArcGIS Pro by going to the view tab and clicking the Python Window button.  If you type “arcpy.” then a list of available modules and functions will appear. You can narrow down your search by continuing to type out the tool you want and it should appear above. If there are multiple tools available you can scroll to the desired one with the arrow keys and then press the tab button to make a selection.

The parameters of the tool will be seen above and the Python window may automatically show optional inputs that are within your tab o contents. For example, if you have a series of shapefiles in your table of contents and the first parameter is an input feature class, then those shapefiles may appear above. (You can then use the arrow keys to scroll and then press tab to auto complete/select the layer.)  Once you one the tool, the results should be available in the history pane.

Another way to access Python in ArcGIS Pro is by opening a notebook. You can access or create a notebook by going to the insert tab and  pressing the New Notebook or Add Notebook buttons below.

 

 

McConkey: Chapter 8

Posted onAuthorJay Tyler McConkey

Chapter 8 covers manipulating spatial and tabular data. You may want to briefly refamiliarize yourself with “for” and “while” loops before reading this chapter. It would also be wise to familiarize yourself with Structured Query Language (SQL).

Important details to remember:

– Once a script creates an exclusive lock, the lock will continue to exist until the script releases the lock.

Terms

cursor – is a database technology term for accessing a set of records in a table. It works the way list functions work.

search cursor – retrieves rows

insert cursor – inserts rows

update cursor – updates and deletes rows

rows – records in a table

with statements – Good to use when you have two operations that you want to run as a pair

data locks – prevent multiple processes from changing the same table at the same time in two different applications. (includes shared lock and exclusive locks)

shared lock – is applied when a table or dataset is accessed

exclusive lock – are applied when changes are being made to a table or dataset

parsing – splitting the fully qualified name for a dataset into its components

Table 8.1 (pg 248) describes the cursor methods of each class

McConkey: Chapter 7

Posted onAuthorJay Tyler McConkey

Chapter 7 reviews the types of errors you can come across while scripting and the most common procedures used to solve them.

Python IDEs have built-in error checking processes that will alert you of any known errors in the script. If you read the error carefully, you may be able to fix the error yourself, without seeking further help. You should keep indentation and spacing consistent in your scripts or you may produce an error. Syntax errors may be caught as they appear in Spyder and PyCharm. Spyder will display a small red x which opens a pop-up window for syntax errors that appear as you type. Spyder can catch other types of errors such as inconsistent indentation. In PyCharm you can produce a general report to see what types of errors are in your script by clicking Code (at the top of the menu) > Inspect code and then selecting the current script.

There are multiple ways to debug your code. The main ways include carefully reading error messages, adding print messages to your script, selectively commenting out code, or using a python debugger. Debugging will not tell you why a script failed, but it will tell you the line in which it failed. (See pg 224-234 for reference of these techniques.)

You can help handle exceptions by using a “try-except” statement (see pg 237). You can use these to produce text that can help you understand what went wrong. An example for a tuple is seen below:

except (ZeroDivisionError, ValueError):
print (“Your entries were not valid.”)

Using a Python debugger (pg 229):
1. Check for syntax errors then save the script.
2. Set breakpoints, allowing you to pause running the script at specific lines.
3. Run the script with a debugger.
4. Walk through the script while looking out for values of important variables.

Syntax errors – errors involved with spelling, punctuation, indentation, or other conventions of the Python language. If you click Run > Check Module and a syntax error appears, a red bar will appear on the line of which the error resides.

Excepts – Events that have no real error but are a result of the data and the script run. An example of this would be to write a script to produce a count of the feature classes in a feature class list. If there are zero feature classes in the list then the code may run perfectly fine but produce no output. This is not truly an error as the code has run correctly.

commenting out code – Placing a double hashtag (##) before lines of code helps cancel out that line in order to isolate parts of the script. This helps find errors in the script.

breakpoints – The code will run until it reaches a breakpoint. They are good for debugging scripts. To set a breakpoint, right-click a line of code and select Set Breakpoint. This will highlight the line yellow.

McConkey: Chapter 4

Posted onAuthorJay Tyler McConkey

This chapter took a lot longer than the others but was probably the most engaging as you will finally be able to understand the concepts behind Python scripting. This chapter focuses on the fundamentals of Python scripting so that you will be able to make your own scripts. It would be wise to really pay attention to the chapter and to use it as a reference when creating your own code. Below is only some of the information provided by the chapter.

Python uses several different data types in its language. Data types determine what type of values can be held by an object. Some are the most common are strings, numbers, Booleans, and lists. Strings are usually text but can include numbers and other types of characters as well. Numbers can appear as integers (whole numbers) or floats (numbers with a decimal point). Boolean values are either noted as True or False and are useful for establishing or evaluating conditions. Lists are just as they sound and can contain a series of numbers, text, or other values.

You want to be careful when naming objects in Python as there are a number of ways of producing an error with a name (see pg 91-92). The chapter encourages consistency with how you name objects. One of the prefered methods is utilizing snake_case where each word is uncapitalized and instead of using spaces, underscores are used between each word to increase legibility. Python has several built-in functions and an error will result if you try to name an object with the same name as a function.

Furthermore, when writing scripts it may be wise to includes comments to provide context for the script. You can add comments by writing a line preceded by a number sign (#). Any text that comes after the number sign will not be considered code. An example of this could be (# Create output feature class) with the lines of code to create an output feature class found below.

Table References:

Table 4.1. Common operators for integers and floats in Python (pg. 89)

(Examples: * stands for multiplication and ** stands for exponent)

Table 4.2 Built-in functions in Python (pg. 102)

(Example: asbs(x) with “x” being some number will return the absolute value of that number)

Table 4.3 Comparison operators in Python

(Examples: == means equal to with != means not equal to)

McConkey: Chapter 3

Posted onAuthorJay Tyler McConkey

Chapter 3 gives the rundown on how geoprocessing works in ArcGIS Pro with details on tools, models, and scripting. To complete some form of geoprocessing you must input some form of data (a feature class, raster, or table) into a geoprocessing tool (often with additional parameters) to produce an output.

Just like in ArcGIS Mapper, ArcGIS Pro contains toolboxes with tools inside them. You can locate geoprocessing tools in a variety of ways but you must first open the geoprocessing pane to do this. To do this navigate to Analysis Tab and click Tools. You can now search for tools by name and add frequently used tools to a list of favorites. Recently used tools will also show up in the favorites. When searching for tools the description of what the tool does will also appear. By right-clicking a tool and selecting “Batch” you can run the tool to produce multiple outputs with similar parameters.

Information about other types of tools is mentioned in the chapter. Most tools are built into the program but you can also utilize script tools made from Python code and model tools created using ModelBuilder. You can make your own model processes by opening ModelBuilder and creating your own models. The easiest way to open ModelBuilder is to navigate to the Analysis tab and click on the ModelBuilder button. This opens a window into which you can input data and tools and create connections between them. ModelBuilder is similar to Python coding but is more limited. It is still good to become familiar with ModelBuilder as it essentially allows you to create scripts visually rather than with text.

McConkey: Chapter 6

Posted onAuthorJay Tyler McConkey

        Chapter 6 took around an hour to read. It mostly handles how to confirm that data exists, creating lists to see where data is, and creating lists for specific data (ex. Isolating shapefiles). You can also use python scripting to describe the characteristics of a feature class, shapefile, raster, or table. An example of this would be to determine if a shapefile is a polyline through a Describe ( ) function. Using a script to produce a list can help later on when using batch processing. 

Key terms:

System paths – the paths recognized by the Windows operating system

Catalog paths – the paths that only ArcGIS Pro recognizes

List comprehension – used to create lists from existing files in a variety of ways

Wildcard – All functions have a wildcard parameter (*), which defines a name filter. A wildcard is a symbol that represents one or more characters. 

 

What key similarities and differences are there between Describe ( ) and da.Describe ( ) functions?

For the Describe ( ) function, the properties of the object are dynamic. For the da.Describe ( ) function, the keys of the dictionary are dynamic.