Maryland State GIS Conference (TuGIS)

The workshop evaluations are in

(if you want to cut to the chase, the workshop results are here).

I had a great time teaching our two workshops at the TuGIS conference.  In the morning, my students and I presented Spatial SQL: A Language for Geographers, and in the afternoon we taught Python for Geospatial.

We knew expectations would be high: both courses sold-out in 2 days, and we even expanded the class size to 38 people for each workshop!!  I knew that teaching 38 people would be a challenge, but it would also be a great lesson to see if we could corral so many cats into a single, technical workshop.  The workshop evaluations would be crucial to determine if we met our objectives.

The workshop evaluations were overwhelmingly positive.  For example:

  1. over 90% said they enjoyed the workshop.
  2. over 83% said it was much better than other GIS training they have been to.
  3. on a scale of 1-10, 95% of the attendees rated the course a 7 or above.
  4. 93% said they learned something new in the workshop.
  5. 89% said the workshops would help them in their careers.
  6. 91% said they would apply these skills to their job.

I decided to throw one curve-ball on the evaluation sheet and asked:

This was a half-day workshop. Most one-day GIS training classes cost around $600/day. If we developed other in-depth full-day workshops on topics like this for under $250, how likely would you be to participate in it?

it turned out that 89% of the respondents rated a 7 or higher, indicating that almost 90% of the people valued the training enough to pay $250 for a full day course (opposed to $600 for most GIS courses).  This means it is possible to offer really good, low cost training to GIS professionals.  Keep an eye out on this, as I am very likely to take these training classes on the road.

The comments the participants provided were great – it confirmed our belief that this was an excellent training course, and that the course needed to be expanded to 8 hours, rather than 4 hours – most everyone felt like their was simply too much information to absorb.

If you would like to see the results of the workshop evaluation, click the link below:

TuGIS Workshops – Google Forms

Finally, if you can’t make it to a live workshop, all of my video training courses are $30 or less, when you visit www.gisadvisor.com.  These courses can’t get into the level of depth that a live course gives, but you’ll see that after thousands of students taking the courses, close to 90% of them give the course 4 starts out of 5!

 

Workshops at the Maryland Geospatial Conference

tugisThe Maryland’s Geospatial Conference  () is on March 20/21, 2017.  I first attended TUgis in 1990, and it is always a great conference.  It is not too large, so it is  great way to have extended time with people.  So, if you had a technical question for someone from say ESRI, you could simply stop by their booth and have a chat.

This year I was asked to support the pre-conference workshops.  I will be presenting two workshops with the help of my students.  If you recall, my students are quite good at instructing others about GIS technology.  I’m really looking forward to the conference and interacting with people during the workshop.  Keep in mind, this is not something we are just throwing together – we’ve been spending a lot of time thinking about how to effectively move people through the material so that beginners do not get lost, and more technically savvy people are sufficiently challenged.  We are fanatical about making sure people’s learning experience is excellent.

A description of the courses are found here:

Spatial SQL: A Language for Geographers:  Are you stuck in a rut of only knowing how to use a GIS GUI? Do you want to learn how to automate tasks, but are afraid of computer programming. If so, SQL is the most powerful tool you can learn to help you perform complex GIS tasks. This hands-on course is designed to teach you how SQL can replicate many database and GIS tasks. We will start at a very basic overview and then proceed to more advanced topics related to GIS.

Topics to include:

  • Spatial is NOT Special
  • SQL Data Types
  • Traditional SQL
  • Spatial SQL for Vector and Raster Analysis
  • Spatial SQL for Classic Geographic Analysis

For this class, we’ll be using spatiaLite which is the spatial extension used with SQLite.  This is a great way to get started, as it is very similar to the functionality of Postgres/PostGIS.  If you want to move to enterprise GIS with Postgres or even Oracle or SQLServer, you’ll be in really good shape.

Python for Geospatial: If you are in the field of GIS, you’ve probably heard everyone talking about Python, whether it’s Arcpy in ArcGIS or special Python packages for doing things in open source.  In this hands-on workshop you will learn how Python is used to perform GIS analysis. The workshop will be an introduction to Python, with emphasis on integrating multiple Python plug-ins with ArcGIS and open source GIS.

Topics to include:

  • An overview of Python (variables, statements, I/O, writing code)
  • Python plug-ins for Geospatial (numpy, geocoder, pygal, Postgres)
  • A Taste of Arcpy
  • A Data Analytics Project with Python (for this, we will geocode addresses using Python, perform analysis with open source GIS, take the results into Arcpy to do more GIS analysis, compute statistical results with Python calling Excel, and then create charts and graphs of the results for use on the Internet—without ever opening up a single GIS product.)

If you want to learn more about how to use GIS technology, check out the 9 courses at gisadvisor.com.  

Undergraduate Geospatial Python Projects

earthballThis week my GIS Programming students presented their programming projects to ESRI. First, I cannot say enough to thank ESRI for taking time out of their schedule to meet with our students – the staff was helpful, encouraging, and provided great feedback to the students – what an honor it was to get their feedback.  I am so thankful to be a part of a GIS community that is so supportive of one another.

Now, this was a really special class of undergraduates – and some of them were part of that special group of students that presented their research at an undergraduate conference.  It was small, so we could do some really cool things.  In fact, in the middle of the semester, the students wrote a paper comparing the geocoding accuracies of Google Maps and the United States Census Bureau.

Things were going so well that I decided in lieu of a final exam, we expanded their final projects a little more, and arranged for the staff at ESRI Charlotte and ESRI Redlands to join us on a WebEx that included demonstrations and a code walk-through.  Below are each students’ presentation, and some of the Q&A from ESRI:

noahNoah Krach.  Noah is an amazing undergraduate.  Recently, we lost one of our graduate research associates, and Noah stepped in to provide technical support on a National Science Foundation project in Lake Victoria.  Without missing a beat, Noah was all over the project, and he used his time in my class to create an Arcpy tool to extract, translate, and load (ETL) gigabytes of Landsat imagery.  This tool does a lot, and I can’t even begin to describe all he did, you’ll simply have to watch and learn.

Check out his video, and you’ll see why we are so excited that Noah will be around for another semester.

cc

Caitlin Curry.  If you follow my blog, you’ve already met Caitlin.  She finished her summer internship I told you about, and during the middle of it, her boss wrote us to say what an excellent worker she was (he prefaced his email by saying he never does that, but was so impressed with Caitlin, he had to let us know).  We are impressed with Caitlin, too.  And, as I have now grown to expect, Caitlin did an amazing job with another ETL type tool using Arcpy, where she downloaded, unzipped, and processed earthquake data and critical infrastructure.

I did a lot of emergency response work with earthquakes in a previous life, and what Caitlin did here would have been so useful.  I think you will enjoy seeing how she integrated many different Python packages with Arcpy to provide an early warning application for emergency responders.  And just as a heads-up, Caitlin uses Python to download everything while the script is running – so you just give the script to a user and it works without any operator knowledge of the underlying data = really cool, and efficient.

mb

Matthew Bucklew.  After my first lecture this semester, Matt told me he built his own computer this summer – just for fun.  So, I knew he wasn’t your ordinary  geographer – he likes to try new things, and if something is done in a conventional way, Matt is going to try and be more innovative.  Matt created a great Arcpy application to locate renewable engery stations needed by automobiles.  His Python scripts use ArcGIS for analysis, but at the same time, seamlessly brings in the Google APIs to provide directions to the nearest locations.  For good measure, he also brings in other packages like heapq.

At the moment, Matt’s program works on a desktop, but his hope it to turn this application into a cloud based solution for use with mobile phones.  Keep an eye out for what Matt comes up with, and if you watch this, you’ll see it is an excellent tutorial on how to mash up bunches of Python packages with Arcpy.

jmJessica Molnar.  Like Caitlin, Jessica is another student you’ve seen before.  She’s got such a big heart, and is always looking for ways to apply GIS to humanitarian and ecological solutions.  In this project, Jessica created an Arcpy application to identify locations for community gardens in Baltimore City with special consideration for locations within food deserts, near churches and schools, and on suitable soils for growing food.  Jessica’s program also found those locations that were already owned by the City, but were vacant.  Let’s hope the City makes use of this to build a more beautiful Baltimore (BTW, Jessica wrote her program to work in any location in the State of Maryland, so any community can use this tool!).  I think Jessica may eventually roll this into a cloud based solution – hey Jessica, I think we found a project for graduate school!

 

jtJohn Tilghman.  John’s family owns an orthodontist practice, and John decided to use PostGRES/PostGIS along with a number of other different Python packages to perform market area analysis.  John integrated PostGRES, Google, and the Pygal libraries to create the first stages of a geodashboard to assess the effectiveness of marketing strategies, and other metrics.  In the video, you’ll also see how he created a distance decay algorithm in SQL to determine at what point customers drop off from visiting the practice.  With just a little bit of information (addresses and marketing strategies), John was able to extract a ton of business information – in fact, our guests from ESRI were surprised the John wasn’t already a business major!

This is an excellent presentation to watch for those of you who are interested in using Python with Open Source GIS – you’ll learn how to integrate FOSS4g and Python for a business analytics tool.

 

jyJosh Young.  Josh created an Arcpy script to assemble tons of location based data that might be useful for someone thinking about moving to a particular location.  Now, in Josh’s case, he chose location based data he deemed important for the neighborhood (download speeds, elementary school, crime statistics, distance to the downtown, etc.).  But ultimately, what Josh has shown us is how to create a template that integrates multiple Python packages and online data to provide very useful information.

It would be so easy to take Josh’s work and roll it into a site specific location-based analysis engine.  In fact, one of the people watching Josh’s presentation mentioned that he was moving, and saw how useful this could be for a community.  The best part of it is that Josh did it with all freely available online data for the State of Maryland, so any community can spin this up into a cloud-based solution.

 

image2

Robbie Stancil.  Robbie is our only non-geography major.  You’ve met him before when he worked with me on a National Science Foundation project to use Spatial Hadoop.  Like John, Robbie’s project used Postgres/PostGIS and the Google API to do something quite interesting: he created a mesh of points over community to determine how far the Google API will search in order to find a property address, and compared the concave hull of each series of points for an address to the actual property parcel.  This project got us thinking about some very creative uses – you’ll have to watch it until the end to see the interesting things we came up with.

 

Again, I have to give a huge shout out to the ESRI staff – they were wonderful guests, and really excellent mentors during the Q&A. As these students get ready to graduate in May, I know they will make excellent employees or graduate students – the future is really bright for them. If you are in academia, I hope that you are inspired to expect the very best of your students as I do, and you’ll be so pleased to see what they are capable of doing.

want to learn how to program geospatial solutions like these students? Check out the geospatial courses at gisadvisor.com.   

A Typical Class Project at Salisbury University: Evaluating Geocoding Accuracies

I’ve always been proud of our Salisbury University GIS students, and love to push them as far as their little minds can handle it.  You may recall that last Spring I had my Advanced GIS students perform independent GIS projects and present those projects as posters at an Undergraduate Research Conference.  Well, this Fall I am teaching GIS Programming, and have 7 awesome students (pictures and bios to follow).  We started the year off learning spatial SQL with Postgres and PostGIS.  We have now moved into Python, which includes Arcpy as well as other Python packages.

The semester was going so well, and the students were so responsive to anything I asked, I said what the heck, let’s try something crazy.  So, I showed the students how to use two Python geocoding packages (geocoder and censusgeocode) and then said:

why don’t we conduct a research project over the next week to test the match rates and positional accuracies of the Google API and the United Census Bureau API.  

So yeah, I gave them a week to put this together: design, code, analyze, and write.  And, like most of my students at this level, they didn’t disappoint me.  This meant they had to integrate a lot of what they have learned over the years (programming, GIS, statistics, etc.).

I just uploaded their work onto researchgate:

 Click for ResearchGate Article  

I was surprised by how little there is out there in terms of quantitative assessment of geocoding accuracies.  I hope you have a chance to click on the link and check out the working paper (we will submit it to a journal sometime soon).  Also, I included a short abstract below so that you can see the results of our work (note: our paper includes the original data and the source code for performing the geocoding):

Undergraduate Research in Action: Evaluating the positional differences between the Google Maps and the United States Census Bureau geocoding APIs

Abstract:  As part of a class assignment in GIS Programming at Salisbury University, students evaluated 106 geographically known addresses to determine the match rate and positional accuracy obtained using the Google and the United States Census Bureau geocoding application programming interface (API)s. The results showed that 96.2% of the addresses supplied by the Google API were successfully geocoded, while 84% of the addresses supplied by the Census Bureau API were successfully geocoded.  Further, the Google API matched 90% of the addresses with a ROOFTOP designation.  The average positional accuracy of the Google derived addresses were 80m overall, and 65m for those geocoded with the ROOFTOP designation while the Census Bureau positional accuracy was 271.09m.  

So yeah, this is what you can do with 7 GIS undergraduates at Salisbury University: they work hard, fast, and are a very creative bunch.

paper

 

 

Parallel Processing with QGIS

Once again, I am continuing my role as a mentor in a National Science Foundation (NSF) Research Experience for Undergraduate program.  This year we’ve decided to build a QGIS plug-in for terrain analysis, as it is embarrassingly parallel (slope, aspect, etc.).   We are doing four things as we generate slope for different size digital elevations models:

  1. A pure python implementation (for an easy plug-in)
  2. A serial-based C++ implementation (as a baseline for a non-parallel solution)
  3. A pyCUDA implementation (using the GPU for parallel processing)
  4. A C++ based parallel solution using the GPU

We plan to put our results on a shared GitHub site (we are in the process of cleaning up the code) so that people can start experimenting with it, and use our example to begin generating more parallel solutions for QGIS (or GDAL for that matter).

Here are some early results: Continue reading

Advanced Editing Capabilities with PostGIS

Today I want to introduce you to another one of my really great undergraduate students: Jeff Scarmazzi.  Jeff participated in the SUSRC conference that I posted about previously, but unfortunately, his photo and poster got deleted from my camera, so I never posted information about his project – I may eventually get around to posting Jeff’s poster, but for now it will have to wait.

 

Jeff is an exceptional student, and at this point can run circles around me with PostGIS, Leaflet, and Postgres – at the moment, I am still reserving the right to say that my spatial SQL is better than his, but that may not be true.  And, now that Jeff secured a prestigious internship with NASA over the summer, once the Fall rolls around it definitely won’t be true!

What I wanted to show you was a short presentation that Jeff gave on his work with Postgres and PostGIS in my Advanced GIS class.  For this project I had students create a geodatabase with ArcGIS that includes domains, subtypes, topology rules, etc.  Jeff decided to extend his work on this assignment and recreate the same kind of features using Postgres and PostGIS.  This presentation is just a small piece of what Jeff and his fellow students (Austin Barefoot, Carl Flint, and Jordan Roose) created, and focuses specifically on writing triggers and creating roles to automate and quality check digitizing procedures using Postgres.

The presentation is only about 15 minutes long, and you’ll be amazed to see what you can do with PostGIS to creating robust editing rules.

Warning Shameless plug: In case you wanted to learn how to implement PostGIS in an enterprise setting, check out my online course here.  Also, I have another course on how to write spatial SQL with PostGIS here.  

Undergraduate Research with GIS

our merry band of 12 undergraduate GIS students presenting their research posters at SUSRC.

Today was the Salisbury University Student Research Conference (SUSRC).  I love the SUSRC because it represents what is so great about Salisbury University: the undergraduate experience.  Our university prides itself in giving undergraduates a graduate school experience.  As a friend once told me, when you come to Salisbury University, you’ll find yourself in the lab, with a Professor, with a PhD.

Salisbury University has really shown me the joy of working with undergraduates.  I get to watch them enter as immature 18 year old freshmen, get excited about quantitative geography when they take Statistical Problem Solving in Geography their sophomore year, and continue to mature as they take GIS Programming and Advanced GIS, along with so many of the other great course offerings in Geography and Geosciences. Continue reading

From days to seconds: experiences with parallel processing and GIS (Part I, the team)

Many of you know that I have been working with parallel processing for GIS in the form of video card general purpose, graphical processing units (GPGPU).  However, this year I decided to change things up a bit, and focus more on CPU based parallel processing.  To that end we began working with Hadoop along with spatial Hadoop.

I plan to have around 7 or 8 blog posts on this over the next 4 or 5 weeks.  My initial outline is:

Part I, the team (this post)
Part II, the point-in-polygon problem
Part III, solving the problem in hours, not days
Part IV, solving the problem in minutes, not hours
Part V, solving the problem in seconds, not minutes
Part VI, lessons learned and challenge to the GIS community
Part VII, advice on building your own server

In the posts, I will tell you what we did, how we did it, and we will also assemble our code.

So, spoiler alert:  Yes, we actually took a classic GIS process that required days to complete, made some adjustments to complete the process in hours, created our own cluster of 4 computers with 16 CPUs to complete the problem in minutes using parallel processing, and finally, went all-out, and rented time on the Amazon EC2 server to complete the job in the realm of seconds (BTW, the rental time on EC2 cost around $5.00 to complete the job).

But first, I want to introduce you to the undergraduates that I worked with on this project with me.

My Research Interns

This summer, as part of a National Science Foundation (NSF) Research Experience for Undergraduates (REU), I spent my summer with two very smart guys: Alan Young, and Robbie Stancil.

Robbie Stancil

image2A Junior Math and Computer Science major, Robbie is an extraordinary student.  If there was something you could do in college, Robbie did it.  Just some of the things Robbie has done in 3 years at Salisbury University include being part of the Bellevance Honors Program, a Presidential Citizens Scholar, a Bellevance Honors Student Ambassador, Resident Assistant (RA) for the Honors Living Learning Community, Salisbury University student member of the Alumni Association, and the 2013-2014 Manokin Hall Residents Council Treasurer.  Outside of academics, Robbie has also held internships at NASA Wallops Flight Facility, and ADNET Systems.

As you might imagine, Robbie’s name has become a fixture on the Dean’s list at Salisbury University.   He sort of reminds me of when I was a student, with the only difference being my writeup would have said, Art Lembo was a mediocre college student, drank beer his freshman year, and was not smart enough to get a scholarship in college.  Other than that, we’re practically identical.

Alan Young

image1Alan Young comes to us from Berry College in Georgia, where he is a Sophomore Math and Computer Science major.  Alan holds multiple scholarships at Berry, including the Berry Academic scholarship, and the Georgia Zell Miller scholarship.  He has a job in the IT Department at Berry College, and is also involved in a number of organizations on campus including the math club, computer science club, and the Baptist collegiate ministries.  Alan has participated in a number of computer science programming competitions.  It is hard to believe he has only completed his Sophomore year in college!

Both of these guys are exceptionally smart and hard working students.  I loved working with them.  While I am thrilled that Robbie will be here next semester as a student, I am so sad to say goodbye to Alan as he returns to Berry College.  I can’t wait to start writing recommendation letters for these two in the next couple of years.

My next post will go over the problem we faced, the data we used, and the roadmap we set for ourselves. Stay tuned.

A quick look at ArcGIS Pro

One of my undergraduates was interested in incorporating a little more sophistication into an analysis he was conducting in Southeast Asia.  The long and the short of it is that he has polygons and lines, and he wants to quantify the number of lines within certain distances of each polygon (i.e. 5km, 10km, 20km, etc.).

An example dataset looks like this:

Slide1

This was rather small – 11,000 lines, and 120 polygons.  There were a bunch of steps to perform, and I won’t take time to talk about it here.  The most important step was to determine the distance between each line and each polygon that were within 5km of one another.

So, the first thing I did was to run the Generate Near Table tool to find the distance of every polygon to every line.  In ArcGIS 10.1, this took 420 seconds – I was disappointed because we have much bigger datasets to worry about.

So, I decided to give it a shot in ArcGIS Pro (here is a screen shot):

Slide2

The Generate Near Table in ArcGIS Pro ran in 17 seconds!  That is 24x faster than ArcGIS 10.1.  I’m going to be running some more tests on this over the summer and will report on what I find.  Next Fall, in my GIS Programming class, our undergraduates are going to write Arcpy scripts in both 10.1 and ArcGIS Pro, and we will show you the results.

Salisbury University students win design contest for app development

Rob and Hillary wowing the crowd

Rob and Hillary wowing the crowd

I am happy to announce that five of our students just took first place in an app development competition against 50 other teams.  Students are developing android and iPhone applications for the Sea Gull Century ride. The application is a collaboration in geography, computer science, and marketing.

I am very proud of our students!  The computer science students love what we are doing in GIS, and they have the skills to do some really innovating things.

More to come.