Salisbury Gives Back – free GIS tools

I want to invite you to join the Salisbury University Geography students as they present their final GIS programming projects, Monday December 18, at 4:30PM in Henson 153.  Snacks will be provided*.

Last year we had a really successful presentation of student programming assignments as part of our GIS Programming class.  For this year, we decided to do something different.  Rather than students selecting their own projects, I identified four topics that I thought were under served as traditional GIS tools.  That is, the tools simply don’t exist as far as I know.  And, because the tools aren’t readily available, they aren’t applied within our discipline – hence the name, Salisbury Gives Back.

So, our students created four separate ArcGIS Script tools that will run right in ArcGIS:

Join Count Analysis Tool – This tool focuses on the spatial autocorrelation method of join count analysis that evaluates area features with binary variables to determine if the data is random, dispersed or clustered.  An example would be to determine the spatial autocorrelation of voting patterns for the US Presidential Elections (i.e. red vs. blue states).  The tool calculates the expected and observed dissimilar joins, the Z score, and associated p-value.

Sadly, this useful method is rarely used because a tool does not currently exist to perform join count analysis.  However, with this tool, geographers can now evaluate spatial autocorrelation with binary variables.  In fact, with this tool’s ease-of-use, I expect to see Political Geographers make use of the analysis capabilities to evaluate elections all throughout the US and beyond.

Quadrat Analysis Tool – This tool focuses on the spatial autocorrelation of point patterns across a landscape. Using points and a grid of quadrats, this tool measures whether a point pattern is random, dispersed, or clustered and is frequently used in hazard analysis for things like wildfire distribution, tornado touchdowns, or dispersion of crime.   The tool calculates the variance to mean ratio, Chi-square value, and associated p-value.  In addition, the tool thematically shades the quadrats based on their counts.

Similar to the Join Count Analysis, there does not appear to be a tool that exists to accomplish the task, and therefore the approach is rarely used in our discipline.

Stratified Sampling Tool – This is not a spatial tool, but rather a tool that takes stratified samples and generates point and interval estimates from the stratums.  Generating estimates from stratified sampling is a very powerful statistical technique, and provides significant improvements over simple random sampling.  A good example is to estimate yearly household utility usage in a community by sampling homes from three different stratums: large, medium, and small.  The tool allows the user to enter the data for each stratum, along with the level of confidence (i.e. 90%, 95%), and provides the point estimate and confidence interval.

Stratified sampling is rarely used because there isn’t a tool that can perform the task.  However, with this tool, geographers can now easily determine confidence intervals for estimation of averages, totals, and proportions when considering different stratums.

ArcGIS and Google Route Optimization Tool – Currently, network analysis in ArcGIS requires Network Analyst. Network Analyst requires a significant effort to not only create a network, but also maintain it.  Further, most organizations don’t have up-to-date speed limits or real time traffic observations.  This tool integrates ArcGIS with the Google Maps API so that users can route the ArcGIS features over a Google network, using the Google Routing Engine.  Users can input their own geodatabase feature classes, and the tool returns the Google routes as a geodatabase feature class, along with driving directions.  In addition, the user can select different route types (i.e. driving, walking, mass transit).  If you ever wanted to route your ArcGIS data over a Google network, you want to stick around for this presentation!

We hope that you can join us for the presentations and code walk-through.  If you know of other people who might be interested in these tools, please do not hesitate to pass this email along.

* we plan to record all four sessions, and I will post them here next week, along with the associated toolbox (assuming I don’t mess up the recording!).  

 

Big Data GeoAnalytics – adding data

Continuing my series on big data geoanalytics, I wanted to show how to bring in large data sets so that we can start working with them. The data set we’ll use is the NYC taxi data that includes information on pickup and dropoffs. There are about 13 million records in a 2.2GB .csv file. That is not insanely large, but it is large enough for us to start messing around with it (don’t worry, I have a few 20GB+ data sets that I am working with and will eventually show that to you as well).

This video below will walk you through the steps I took to load and prepare the NYC taxi data inside of Manifold Future. My next posts will begin to look at how we can begin interrogating the data source to find meaningful information.

I hope you enjoy the video. Please comment below – I’d love to hear what people think.

 

Big Data GeoAnalytics – Turning Points to Lines

In my last video, I gave a short of mile-high view of how SQL can be used for big data geoanalytics.  I want to dive a little deeper, and explore the idea of create linear features from a time-series of points.

Once again, using some basic SQL and spatial SQL, we can perform basic time-series analysis.

I’m enjoying making these videos, as they are helping me put my course on big data and GIS together.  I hope you like them too.  Please comment down below so that I know this is something the user community enjoys and is learning from.

Also, if you are interested in learning more about how to perform spatial SQL in Microsoft SQL Server, Postgres, or Manifold, visit my other site, www.gisadvisor.com to sign up for my online video courses.

Big data geo-analytics with SQL

I’m getting ready to create a course in big data analytics with GIS.  I have lots of ideas as to what to do, but one thing I know is that I will be using spatial databases and SQL.  I’ll also be using Manifold Future.

ESRI has recently introduced their ArcGIS GeoAnalytics Server, which will introduce many GIS professionals to big data analytics with GIS.  They have some interesting scenarios and example data using NYC taxi cabs.  I think these will be really good case studies.

This video (just shy of 20 minutes) will use SQL and Manifold to try and address these big data problems.

Keep an eye on my blog as I will be rolling out new ideas as I prepare my course for the Spring.

if you like the video, and want to learn more about how to improve your spatial database skills, check out my videos at www.gisadvisor.com.

Great work by my undergraduates, Again!

You’ve heard about how good my undergraduate GIS students are here, here, and here.  Oh yeah, and here and here.    Well, over the last few years my undergraduate students have been working with the campus Department of Horticulture by surveying the Salisbury University campus under the direction of Dr. Dan Harris.  Did you know our beautiful campus is a registered arboretum?  Starting with my student Waverly Thompson two years ago, they surveyed all the trees, sidewalks, sprinkler heads, light poles, pretty much anything you can think of with survey grade instruments.

This past summer Zack Radziewicz, Josh Young, and Lindsey Pinder turned that survey work into a beautiful cartographic product, and yesterday put the map online here.  Zach has an art background (he’s also an awesome GIS student) so that really helped, and Josh has been helping me lead professional workshops in Postgres and Python.  Lindsey is a rising GIS star in our Department, so keep an eye out for more posts about her.

It never ceases to amaze me how good our undergraduate students are here at SU.  I’m proud of the work they do, and so thankful to get to work with them each day.

Great job Waverly, Thompson Zack, Josh, and Lindsey.

P.S. While I was in Korea and Dr. Harris was in Brazil, on their own, these students turned the entire map into a 3D visualization – I’ll post that soon.

New poll: Help me pick my next workshop.

I have had a great time giving live workshops – each one has been sold out, the the reviews have been fantastic (see here and here)

As people have asked me to do more advanced GIS workshops, I thought I’d ask some of you who read my blog what you would be interested in  As a side note, I am starting to put together a two-day Big Data Analytics for GIS workshop.  I will definitely offer that as a class at my University,  but I think I can also boil it down to a 16 hour, two day workshop – does that sound interesting to people?

Anyway, check out the poll, add some comments, and let me know if your community would like me out to give a workshop.

Don’t forget, if you want to learn advanced GIS training on your own time, you can grab one of my video courses here

BMC Spatial SQL Class Survey Results

I think this class has set a standard for quality professional development that will be hard to beat in the future. – Mara Kaminowitz, GISP, GIS Coordinator.

Easily one of the best workshops I have ever attended.  Sam Stanton, GIS Coordinator, Queen Anne’s County Information Technology.

sqlite

I had a great time at the Baltimore Metropolitan Council, leading a workshop on Spatial SQL using SQLite and SpatiaLite – thank you to Mara Kaminowitz for organizing it!  There were 25 people in attendance, and I just received the evaluations.  To cut to the chase, you can see the evaluation results here.

But, for the high level view, here is what people thought:

  1. Over 95% said the workshop was better than other GIS training they received.
  2. On a scale of 1-10, 95% of the attendees rated the course an 8 or above.
  3. 95% said they learned something new in the workshop.
  4. 100% said the workshops would help them in their careers.
  5. 87% said they would apply these skills to their job.

I decided to throw one curve-ball on the evaluation sheet and asked:

FullSizeRender

This was a half-day workshop. Most one-day GIS training classes cost around $600/day. If we developed other in-depth full-day workshops on topics like this for under $250, how likely would you be to participate in it?

It turned out that 91% of the respondents rated a 7 or higher, indicatin

g that the attendees valued the training enough to pay $250 for a full day course (opposed to $600 for most GIS courses).  This means it is possible to offer really good, low cost training to GIS professionals.  Keep an eye out on this, as I am very likely to take these training classes on the road.

FullSizeRender (1)The comments the participants provided were great – in  fact, the response was slightly higher than the response at the TuGIS conference.

The course evaluation will have lots of free-form comments you can read, but a couple of the ones I liked were:

One of the most valuable pieces of this workshop was how it drove home the fact that Spatial SQL is not as intimidating as it may seem. Going over the basic SQL functions and how closely it reads to standard english was very helpful in driving this home.

Gave me a much stronger foundation for SQL, the instructor was extremely knowledgeable and presented the information well. I am so happy to have a stronger understanding and feel this will help me immediately in my job.

Dr. Lembo was very effective in showing the analytical framework behind the use of SQL in the development of databases, geometric layers, and even modeling. He was really good in putting it all together; I had little exposure to SQL before the class, and now I feel confident on how to approach it to solve problems I encounter in my job.

What spatial database do you use?

How do you manage your spatial data?  Just looking to see what kind of consensus there is out there – please pass this along

More geocoding with Radian

radianintegrationI want to continue adding more nuances to my previous geocoding post here.  In this post, I wanted to show you how I am using Radian to call SQL Server to perform data selection, A Radian function to modify the data coming back from SQL Server, Google’s geocoding API to get X,Y coordinates from an address string, and then another Radian function to convert the geocoded coordinates into a geometry object.

 

 

if you want to learn more about writing SQL queries in SQL Server, Radian, or Postgres, check out the training classes at www.gisadvisor.com.