McConkey: Course Wrap Up

Posted onAuthorJay Tyler McConkey

Python Scripting for ArcGIS Pro

The book is helpful to start to get an understanding of Python, but it is difficult to interpret sometimes as it is prone to using jargon. It does define the jargon, or scripting terms, and explain how the terms relate to each other, but it does get confusing sometimes if you are unfamiliar with a lot of these concepts. The weakest part of the book is that it reads like a walkthrough, but there is no actual data package that comes with it so you are not able to perform all of the scripts it explains. Although the book works as a learning tool, I believe it works better as a reference guide.

Individual Project: ERSI Learning Courses

Even though I was not able to make as much progress as I wanted with the ESRI learning course, I do think the courses make it easier to learn Python much more than the book as you are able to actively learn through working with datasets and writing scripts. Additionally, you are immediately able to see the results of your scripts which makes you feel much more accomplished. There are many learning courses that you should be able to access and many of them are introductory level courses. The directions for the learning courses that I worked through were pretty thorough, so the biggest learning curve comes from navigating PyCharm than writing the code itself.

The work that was completed was certainly worthwhile; however, I hit an error that caused more errors and delays which prohibited me from completing the second learning course I was on.Ā  The main error occurred after I added a python interpreter to PyCharm as instructed in the “Python Scripting for Geoprocessing” course. After the interpreter was added and I ran the completed code and an error code appeared in the console. It reads as follows: C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\numpy\__init__.py:143: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the-box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service from . import _distributor_init

I tried following the link but that lead to many dead ends. I also discovered that PyCharm stopped recognizing arcpy as a module around this time. This may have been from the version of Python I was using but after I tried changing it, the error still appeared.

I then tried following the file path listed in the error code: “C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\numby\__init__.py:143.” I could not find “__init__.py:143” in the numpy folder.

I went to the settings and clicked on the Python interpreter and saw that the packages are actually listed. I found the mkl-service package in the list so I wasn’t sure why it wasn’t working. I clicked the plus sign to add a package, searched for the mkl-service package, and clicked install. However, an error occurred claiming that I did not “have the write permissions to the target environment.

I worked with someone from OWU tech support and we discovered that I did have administration access and we tried a couple of alternative methods but determined that the error may stem from a problem with the actual PyCharm software itself. Hopefully, you don’t encounter any of these errors as they are a pain to deal with.

Overall, I am proud of the work I was able to complete. I definitely think the learning courses are worth exploring and should maybe be the primary focus of the course, rather than reading the book. I think I retained knowledge better from the courses than from the book itself, but this could be a reflection of my learning style. The book is still a worthy investment. Taking down notes will slow your progress, but they definitely will help you retain more information.

McConkey: Python Scripting for Geoprocessing (ERRORS)

Posted onAuthorJay Tyler McConkey

Right off the bat, this course seems to cover directions in more detail than in the last course, Python for Everyone. I would actually recommend taking this course first so you can better interpret the directions of the other course. That is of course if you don’t encounter the same errors I came across. Sadly, I was not able to complete this course during the semester as I could not fix these errors.

Errors:

In the first exercise, an error may occur at step 5 when setting the environment for the script. When you set the Python Interpreter an error message may pop up in the Python console. It appears as written below (you may have to scroll to read the whole thing):

C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\numpy\__init__.py:143: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the-box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
from . import _distributor_init

Basically, it wants you to install something called an “mkl service package.” I discovered if you go into the settings and look at the interpreter there is actually an add (plus sign) button that you can add packages. I clicked the add button and made a search for the mkl service package. When I found it I pressed install but some type of administration error occurred. The error may be from the version of Python that you are using so you might have to do some research to find what version of Python works with ArcPro, which I believe is Python 3.7. I also tried to download this version but I am not sure if it was actually successful.

The script for exercise 1 should look like this:

import arcpy
# Set environments
arcpy.env.workspace = r"C:EsriTraining\PythonGP\Data\SanJuan.gdb"
arcpy.env.overwriteOutput = True
# Set parameters used to join the BufferDistance table to the Roads feature class
inFeatures = "Roads"
inField = "ROUTE_TYPE"
joinTable = "BufferDistance"
joinField = "ROUTE_TYPE"
# Script Joins table to feature class
arcpy.JoinField_management(inFeatures, inField, joinTable, joinField)
# Set parameters used to buffer Roads feature class
outBuffers = "RoadBuffers"
buffField = "DISTANCE"
# Buffer the roads based on DISTANCE attribute
arcpy.Buffer_analysis(inFeatures, outBuffers, buffField)

 

ArcPy Final Assignment

Posted onAuthormmwade2

Mack Wade

 

Get Started with Python in ArcGIS Pro

In this course I learned how to write code to determine the number of features for all the feature classes in a workspace. This course also introduced some of the basics of Python syntax as I wrote code into the Python window in ArcGIS Pro.Ā Ā 

In order to begin this project I had to download data from ESRIā€™s website. I really enjoyed this course and chose to do it because of the downloadable data that I could use as I followed along, unlike the book which did not provide downloadable datasets for itā€™s walkthroughs.Ā 

The course provided a few shapefiles to work out of, but for this exercise I used the Ambulances.shp.Ā 

First, I ran a tool using Python. In Arc there are many ways and routes to complete one task. This task was to find the count of the items in the shapefile. You can do this in the attribute table or by using the Get Count tool in the Toolbox. You can also find the count by running a script in the Python window.Ā 

To do this you take go to the toolbox history > right click the Get Count tool > Send to Python Window. After sending the window, I manually entered ā€œAmbulancesā€ since thatā€™s the shapefile I want to be working out of. The script looked like this:

arcpy.management.GetCount(“ambulances”)

After completing this task, I learned that when completing a script in the Python window, the history is updated to include the task.Ā 

The next task that I practiced was inputing an assignment statement.Ā 

y = 73Ā 

Then,Ā 

x = 37

Then,

x * yĀ 

Then, after pressing enter, the Python window answered with 2701

Next, I did a similar thing but with Temperature.Ā 

temp_c = 17

temp_f = temp_c * 9 / 5 + 32

print(temp_f)

The Python window responded with 62.6

The next assignment was learning how to use the Python window as a tool to help you when writing code. You can do this by hovering over the prompt and reading the information that the window provides.Ā 

There is also a system of code completion. For example, if you want the ā€œGet Countā€ tool, you can typer ā€œGetā€ in the Python Window and suggestions will pop up to help you finish the prompt easier.Ā 

I really enjoyed this tutorial. I thought that it was very easy to follow and was helpful. I see myself using more of ESRIā€™s resources in the future if or when needed when writing scripts. I think that if any student was interested in this course in the future these tutorials would be a better resource than the book we used.

I had some screen shots from this assignment but they did not get added into the blog format. If you would like me to send those over to you. let me know.

Frawley: R Geospatial Wrap Up

Posted onAuthormtfrawle

Background ā€”Ā 

This independent study was more so learning R rather than spending more time in project-based learning as I did with python. R is a very powerful data-driven language and is great for using large data sets. R is vastly used for geospatial analysis as well. I created a county population map of the United States using census data from built-in libraries whose primary use is for geospatial analysis. The code looks very simple, but it was not. I had significantly more problems with R than I did with Python. The syntax used in R is an absolute cluster and I despise it.Ā 

 

StepsĀ 

  1. Import libraries usmap and ggplot2.Ā 
  2. Add some data to plot the map, data = countypop draws the county lines and values creates data with the built-in census data via a CSV.Ā 
  3. Create a scale and create breaksĀ 
  4. Change the scale fill to one thatā€™s not a constant color for easier viewing
  5. Cry because you canā€™t create an interactive map with tmapĀ 

 

The Code ā€”Ā 

library(usmap)

library(ggplot2)

plot_usmap(data = countypop, values = “pop_2015”, color = “grey”) +Ā 

Ā Ā scale_fill_viridis_c(name = “County Population 2015”, label = scales::comma, option = ‘D’, trans = “log2”,Ā  breaks = trans_breaks(“log2”, function(x) 2^x))+

theme(legend.position = “right”)

 

 

Problems ā€”Ā 

Like I stated before, I had tons of issues with this project. I programmed the map at least 15 different ways and it never looked like how I wanted it to. I think R is a little over my head when it comes to the math-based stuff like the scale. The breaks were never how I wanted them but I chose one that seemed the most reasonable.

Ā Another issue that I had was this map is a static map, it cannot move and you cannot click on counties to see their population. I tried for at least three weeks to create an interactive map through the library T map. Itā€™s literally very simple, you just join the counties and the county population CSV to create the interactive map. The issue that I had was joining the two tables by their geoID. It never seemed to work and I literally even created my own spreadsheet with the countiesā€™ populations, names, states, and geoID. Didnā€™t work. This was the most frustrating part of the project that I dealt with.Ā 

Again, I took more of a book approach to this course so it definitely felt like I couldā€™ve figured it out by making other things within R. I can follow the code and see what someone is doing, but trying to do it on your own starting out is very difficult. Also, R is just way worse and not as user-friendly as Python. I feel that I could definitely become better at R if I just spent more time with it, and also brushed up on my math skills a little more.Ā 

 

Frawley: Python Wrap Up

Posted onAuthormtfrawle

Background – Ā  Ā For this independent study, I wanted to solve a problem that isnā€™t necessarily tedious but just annoying. For me, that was creating true color images through ArcPro. Working with Dr. Rowley has made me quite familiar with creating true colors, mostly creating several of them over a melt season for the Sermeq Kujalleq Ablation Region. By creating this script, it just makes everything a little easier so you donā€™t have to manually import which .TIFF files you want. Iā€™ve had some experience with python before starting this course so it definitely was a little easier than I expected, and the syntax of python is just leaps and bounds ahead of that stinky language R.Ā 

 

Introduction – (what each step does)

  1. Create a geodatabase or .gdb ā€” doing this creates a server like storage house your tiff files to make calculations and pull the data back down.Ā 
  2. Create Mosaic Dataset ā€” This creates a dataset to house your images, another important step before using the composite bandā€™s tool.Ā 
  3. Add rasters to Mosaic ā€” Add your images to the datasetĀ 
  4. Composite Bands Tool ā€” From the mosaic dataset, import them to the composite bandā€™s tool, which creates the Truecolor image raster for you.Ā 

 

 

 

What Did I learn? ā€”

I learned a lot from this project, firstly that ERSI software is insanely tedious when it comes to using their programs. I needed to jump through several hoops just to create a geodatabase. The geodatabase is created through ArcServer – everything must be run through the server and you need to have the software installed to even create one.Ā 

I learned to problem solve way better on my own, knowing exactly what to search to find solutions to my problem was a huge success. I genuinely didnā€™t have any problems with this project, nothing that was too halting that I felt like I needed to stop the project or require outside help. I dedicated about 8-10 hours a week to this project and it was mostly spent debugging and finding solutions to my code.Ā 

One issue that came across was at the very beginning of the project. To use the Arcpy library, you cannot have the newest version of python installed. The newest version is 3.10, but arcpy is currently only compatible with 3.7 and below. I felt like this project was a great introduction to the GIS developing world, and I am very glad that I took it. Overall, I am very confident in my work for this course. I believe that this script has significant use for GIS creators and researchers.Ā 

McConkey: Python for Everyone (OnlineCourse)

Posted onAuthorJay Tyler McConkey

Overview of the Course:

This course services as both an introduction and walkthrough basic Python scripting. The course is composed of reading material with easy to understand figures and exercises with step-by-step instructions for a hands on experience. The course also has simple multiple choice questions with immediate feedback for improved understanding. There is some supplemental reading provided by some links in the beginning that took about 25 minutes to complete.Ā  At the end of the course there is a brief review and then 10 question exam. The exam generates different questions if you go off the page and you will need an 8 out 10 to receive certification that you passed the course. (The exam is not that difficult, but I do not know if you are able to take it multiple times as I passed it on my first attempt. I recommend double checking your answers before submitting.)

The walkthrough sections are a bit more challenging and are definitely going to make the course longer than it estimates. This is especially true if you are unfamiliar with PyCharm (which you will be asked to download) as you will be spending a good deal of time getting your bearings with software.Ā  If there are any courses that go over the basic layout of PyCharm I would recommend taking it before tackling this course. You can get through the course if you’re not familiar with PyCharm, but it will take you some time getting used to it.

You will have to download PyCharm and some data. The data did not take long to download and there is a link provided so you can download PyCharm. The PyCharm download is more complicated and will take some time (~30 minutes). If you’re getting the free trail version it should last about 30 days and you will have to make a JetBrains account. Also, when you have downloaded the data you should put it in a folder named ErsiTraining in C so it resembles the walkthrough. You will actually want to extract the data before this step. The data path for the first exercise should look something like this:Ā C:\EsriTraining\PythEveryone\RunningScripts/Polk_County. (I extracted the data after putting it into the EsriTraining folder so there were two PythEveryone folders back-to-back. Oops.)

Notes from the Course:

When choosing a script environment you may want to consider what you are trying to accomplish. If you are trying to test geoprocessing workflows and easily verify the results, you may want to stick with ArcGIS’s built in Python window. However, if you want to work with tools to enter, edit, and check syntax within your code, you may want to use an IDE, such as PyCharm. PyCharm also includes an integrated debugging environment.

Scripts can be built from data types, statements or functions. Statements and F=function both perform an action, but only functions return a value.

Tuples are a sequence of items, just like a list; however, tuples cannot be changed. Tuples are useful for when the sequence is significant in a script. Tuples are closed in parenthesis while lists are surrounded by brackets.

Reflection for the Exercises:

Make sure you follow the directions as best you can. If something isn’t working, you probably misread or skipped over something. If your positive you did not misread anything, try not to get to frustrated. You can try troubleshooting and if that does not work, it may be best to keep moving on. Every if you biffed one step (which can mess up the script in the end) it is better to see the process that the course is trying to teach you. Getting the end results are optimal though.

Not all of the directions are described as thoroughly as they should, especially when it comes to working with PyCharm. The exercise walkthroughs provide pictures that go along with the types of coding you will be asked to write out. (This is great but they do not provide images of the PyCharm screenĀ  to improve your navigation of the software.) One thing you will have to look out for that some walkthroughs do not mention that you have to use print statements to make sure your scripts are working. A lot of times the walkthrough will tell you to write a script and then press enter and see the immediate output. From what I experienced this is misleading as this does not happen. Instead, I had to make a print statement and then run the script to see the output. (The output will be visible in the Python Console. This will appear on the bottom half of the screen when you run your scripts.) The good thing about this type of work is that even though it can be frustrating, the more you do what you intended, the more confident you get.

Review of Course:

Some of the directions for the exercises are a bit vague, especially when giving directions in PyCharm. I think PyCharm my have gone through an update so some of the actions you need to take may not directly correspond with what is being directed. It definitely took me longer than the estimated time, but that is probably due to me never using PyCharm before and only using ArcGIS Pro a few times before attempting this course. The course is in no way perfect but it does combine different learning styles and is well structured. The walkthroughs could be more thorough, but they also help you apply what you’ve learned.

McConkey: Chapter 9

Posted onAuthorJay Tyler McConkey

This chapter was fairly easy to read and was great for starting to understand how geometry objects (something everyone who has worked with ArcGIS should be familiar with) works with Python.

Chapter 9 goes over how to work with geometries, otherwise known as points, polylines, and polygons. This includes being able to read the properties of geometry objects from preexisting features as well as creating geometry objects with coordinate information.

The chapter explains how to use geometry cursors which allow you to work with or edit the vertices of a geometry feature object. To set a cursor you must use a geometry token rather than a field name.Ā  For example, the SHAPE@ token gives access to the full geometry object and its properties.Ā  Using specific geometry tokens may be easier such as SHAPE@XY, or SHAPE@LENGTH.Ā  (SHAPE@XY results in a tuple of x,y coordinates for the feature’s centroid, and the SHAPE@LENGTH returns the feature’s length.)

When working with a polygon with holes, you may have some trouble. A geometry object that consists of a polygon with holes will return an exterior ring with one or more interior rings. Rings are made up of paths, or a series of vertices that begin with a starting vertex and finish with an end vertex. Exterior rings are clockwise and interior rings are counterclockwise. (For more info on polygons with holes look on page 299.)

There are several classes that can be used to make geometry objects. By using the Point object you can create a single vertice. You can also make multipleĀ  Point objects to create polylines or polygon features.

 

 

 

McConkey – ArcPy: An Introduction (Video Tutorial Notes)

Posted onAuthorJay Tyler McConkey

This video tutorial is part of the Python Essentials Learning Plan. The video is about 40 minutes long and gives a basic overview of using Python in ArcGIS.Ā  The video has two presenters and switches back and forth between them during the presentation. One of them focuses are providing information about Python and its applications while the other actually demonstrates how the code is written for some of these applications. A lot of the details are things already covered in the book, but I will include several of the points made to provide an overview of the video.

Python makes coding a lot simpler by automating repetitive tasks such as data analysis and processing.Ā  Scripts can be saved in a text file so they can be shared or used at a later time.Ā  Unique tools that are not found in ArcGIS programs can be made from scratch by using Python. In fact, several tools in ArcGIS, that you may be familiar with are made from Python coding.

Python is dynamically typed. It can support strings, numbers, and functions as well as work with and produce files, lists, and dictionaries. (Dictionaries – objects with key-value pairs).

ArcPy is an API, or Application Programming Interface, and works for programs such as ArcGIS Pro, ArcGIS Server, and ArcGIS Desktop. It must be imported before running certain tools for those applications. If you have these applications you already have ArcPy, you just need to import it in a scripting window. ArcPy is composed of a series of unique modules, functions, classes, and tools.

When you open a tool in ArcGIS, you can select the blue question mark at the top right of the tool window to open syntax help, which will explain the function of the tool and its parameters. You will also be able to see a code sample written in Python at the bottom of the syntax help page which may be very useful when writing scripts for that tool.

You can open a Python window directly into ArcGIS Pro by going to the view tab and clicking the Python Window button.Ā  If you type “arcpy.” then a list of available modules and functions will appear. You can narrow down your search by continuing to type out the tool you want and it should appear above. If there are multiple tools available you can scroll to the desired one with the arrow keys and then press the tab button to make a selection.

The parameters of the tool will be seen above and the Python window may automatically show optional inputs that are within your tab o contents. For example, if you have a series of shapefiles in your table of contents and the first parameter is an input feature class, then those shapefiles may appear above. (You can then use the arrow keys to scroll and then press tab to auto complete/select the layer.)Ā  Once you one the tool, the results should be available in the history pane.

Another way to access Python in ArcGIS Pro is by opening a notebook. You can access or create a notebook by going to the insert tab andĀ  pressing the New Notebook or Add Notebook buttons below.

 

 

Wade: Chapter 8: Manipulating Spatial and Tabular Data

Posted onAuthormmwade2

 

Mack Wade

Chapter 8: manipulating Spatial and Tabular Data

Ā 

Terms:

Comma separated Value (CSV): a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format.Ā 

Data Lock: There are two states of a lock i.e locked and unlocked. A lock is a class in the threading module whose object is generated in the unlocked state and has two primary methods i.e acquire() and release() .

Mode: Ā refers to the most frequently occurring number found in a set of numbers.

Parsing: the processing of a piece of python program and converting these codes into machine language.

Postfix clause: An SQL postfix clause is positioned in the second position and will be appended to the SELECT statement, following the where clause. The SQL postfix clause is most commonly used for clauses such as ORDER BY.

Prefix clause: The PREFIX clause declares any abbreviations for URIs that you want to reference in a query. You can declare prefixes to simplify query text if your data includes long URI names. If you do not declare prefixes, you must include the full URI names in the query.

SQL clause: Clauses are in-built functions available to us in SQL. With the help of clauses, we can deal with data easily stored in the table. Clauses help us filter and analyze data quickly. When we have large amounts of data stored in the database, we use Clauses to query and get data required by the user.

SQL expression: An expression is a combination of one or more values, operators, and SQL functions that evaluate to a value. An expression generally assumes the datatype of its components.

SQL keyword: the reserved words that are used to perform various operations in the database. There are many keywords in SQL and as SQL is case insensitive, it does not matter if we use for example SELECT or select.

Structured query language (SQL): a programming language designed to get information out of and put it into a relational database. Queries are constructed from a command language that lets you select, insert, update and locate data.

Triple Quotes: allowing strings to span multiple lines, including verbatim NEWLINEs, TABs, and any other special characters. The syntax for triple quotes consists of three consecutive single or double quotes.

Review Questions

  1. What are the three different cursors, and what purpose does each one serve?

cursor.fetchall() fetches all the rows of a query result. It returns all the rows as a list of tuples. An empty list is returned if there is no record to fetch.

cursor.fetchmany(size) returns the number of rows specified by size argument. When called repeatedly, this method fetches the next set of rows of a query result and returns a list of tuples. If no more rows are available, it returns an empty list.

cursor.fetchone() method returns a single record or None if no more rows are available

  1. Explain how data locks occur and how they can be removed.
    1. A lock is a class in the threading module whose object generated in the unlocked state and has two primary methods i.e acquire() and release() .
  2. When writing SQL expressions, why are quotation marks sometimes an issue?Ā 
    1. Single quotes are used to indicate the beginning and end of a string in SQL. Double quotes generally aren’t used in SQL, but that can vary from database to database. Stick to using single quotes.

Wade: Chapter 7: Debugging and Error Handling

Posted onAuthormmwade2

Chapter 7: Debugging and Error Handling

Terms:

Breakpoint: Ā used to interrupt a running program immediately before the execution of a programmer-specified instruction. This is often referred to as an instruction breakpoint. … Breakpoints can also be used to interrupt execution at a particular time, upon a keystroke etc.

Commenting out code: to use comment syntax to remove something from the parsed code.

Custom class: Ā a developer defined class, based on one of the stock classesĀ 

Debugging: the complete control over the program execution.

Error handling: the process of responding to the occurrence of exceptions ā€“ anomalous or exceptional conditions requiring special processing ā€“ during the execution of a program.

Exception:

Exception object:Ā 

An event that occurs during the execution of a program that disrupts the normal flow of instructions is called an exception.Ā 

Logic error: a mistake in a program’s source code that results in incorrect or unexpected behavior

Traceback: a report containing the function calls made in your code at a specific point.