6  Field Projects

Field projects are defined here as projects that require doing fieldwork to collect novel physical samples or morphological data. These projects require travel of some sort to collect raw data. Raw data for these projects typically will consist of physical samples.

6.1 Pre-Data Infrastructure

6.1.1 Survey123 for tabular data, spatial data, and image collection

Our standardized tool for data collection for field projects is ArcGIS Survey123. Survey123 is a tool that allows you to create customized data entry and submission forms that you can then access through the Survey123 app on your phone. These forms can be created through the ArcGIS Online Survey123 portal. Although it is a proprietary tool, all faculty, students and staff at UMN have full, free access to the ArcGIS Online suite of tools which includes Survey123. Despite the fact that Survey123 is proprietary and not open source, it is highly functional and therefore the ease of use makes it our tool of choice. This chapter assumes you are familiar and have accessed Survey123. If you have not worked with Survey123 before, it is well supported and documented by ESRI and you can check out our Tools: Survey123 chapter #todo in this book.

6.1.2 Lab Standards for Survey123 Form

As you set up your Survey123 form, you should think through each of the pieces of information that will be important to collect in the field and provide a way to capture that information in your form. When in doubt - err on the side of including it in your form - you can always ignore it in the field while using the app if there is some redundancy.

If you are doing a field project, you should strongly consider using Survey123 to collect your plot or site-level data. Think of what unit of your sampling will have a unqiue latitude and longitude and that is the data collection unit that your survey form should be designed for. For example, if you have a project that requires taking soil cores, with multiple samples from different depths taken from each core, then the survey form should be designed around each of the cores, which will have a unqiue latitude and longitude, and not each of the samples (many of which from the same core will have the same latitude and longitude).

Survey123 excels at consolidating spatial data (latitudes and longitudes), tabular data (text, numbers, or X entered in response to questions on the form), and images (taken in response to questions on the form) in one location. When you submit each of your records, this raw data will live in the Survey123 database through ArcGIS Online that is associated with your form.

Our laboratory standards for Survey123 forms is that they should (at a minimum) collect the following information:

  • Name of initials of person recording data or submitting record
  • GPS location of record (autocollected in response to location question)
  • Date and time of record collection (autocollected in response to time question)
  • Unique plot or observation number in the context of the project

Your Survey123 form should also include fields to enter all relevant raw data as required by your project.

6.1.3 Additional Survey123 Functionality of Interest

Images. Survey123 is an excellent way to collect and organize images. Single image submission questions can be configured to collect multiple images. Once submitted, all images are stored with the associated record in the online database.

Deep Customization - XLSForms Deeper functionality for setting up forms in Survey123 requires a PC and the use of XLSForms and Survey123 Conect (a desktop app that only runs on Windows). While this is very annoying for us Mac users, it allows extensive customization of your Survey123 form, including but certainly not limited to:

  • Repeats: “Repeats” allow you to generate multiple records (rows) from a single survey entry/submission - so for example, you can generate multiple rows for the master sample log (each row is a sample) from one submission form (plot or pedon): https://community.esri.com/t5/arcgis-survey123-blog/survey123-tricks-of-the-trade-repeats/ba-p/898041
  • Customizing File Names for Submitted Images: Customaization through XLS Forms allows you to set the file names of submitted photos - this is really important - automatically annotate photo file names with project, plot, and question or anything else from designated fields in the form (as well as date/time). This functionality can save significant time when you go to curate your photo archive. https://community.esri.com/t5/arcgis-survey123-blog/survey123-tricks-of-the-trade-set-the-file-name-of/ba-p/1217125

6.1.4 Hard Copy Field Data Sheets

Survey123 is an excellent tool for collecting plot or site level data in the field with your phone. However, some types of field data collection are extremely difficult to digitize under field conditions. Examples of these are:

  • Soil morphology data (lots of specialized acronyms, notes, and chicken scratches add depth to the data, often difficult to use phone while textureing soil, many fields and extremely time consuming to use an app or tablet in the field).
  • Data consisting of many numbers or that exists at the sample level. Remember I said that Survey123 really shines when used at the level of unique geospatial coordinates which is typically the plot or site level. However, often times we may also be collecting data at the sample level. For example recording the dimensions of a permafrost soil core section that will go into a sample bag, or the field pH and EC of a soil sample. These data are best collected using hard copy forms. There are examples of hard copy forms for field data collection that Megan Andersen put together in the Jelinski Lab Field Handbook repository [here]#todo. Always print your hard copy forms on Rite-In-The-Rain!!

6.2 Raw Data Collection

After your pre-data infrastructure is set up, do some test runs and some mock data collection around campus. This will allow you to use your infrastructure, datasheets, and think through any missing pieces or functionality issues. Revise as needed and get ready for your travel - check out our [Field Manual]#todo for protocols and processes related to field work.

6.2.1 Tips for using Survey 123 in the field

Survey 123 is generally an excellent app in the field, and most new phones are sufficiently durable and waterproof to be used in the field. As recently as 2019 I carried a separate digital camera and GPS. Now I do everything (capture GPS, take pictures, use Survey123 for tabular data) on my phone. *NOTE - this also means you should have a backup field ready rugged battery pack to charge your phone during the day if necessary in the field. There are also some other tips when using Survey123 in the field.

  • Images. When choosing to use Survey123 as an image repository, ensure that you 1) take pictures with your phone and upload them to a survey response from your photo library (instead of taking them live through the app) - this ensures that if something goes wrong with the response you still have the image on your camera. The images will also be full quality this way, while taking them through the app may result in reduced quality and image compression. Additionally, make sure your survey is set up to receive and store original quality images - the default is compression so you need to change the default (google search how to do this and provide link here)#todo.

  • Offline/Online. Unless you are in a populated area, you may not be able to upload survey responses live in the field. That is fine. Survey 123 allows you to save draft responses to your outbox which are stored locally on your phone. When you get back to a good internet connection, you can submit your survey responses.

  • Phone GPS Accuracy. Even if you don’t have service or wifi signal, your phone GPS will be more accurate if you keep your cellular and wifi on. This will drain your battery faster in remote areas but the gain in accuracy is worth it. As mentioned above, for this reason, bring a field ready battery pack.

  • GPS Backup. Always ensure that your phone is not your only source of GPS data to insure against data loss. There are several combinations you can use to achieve this:

    • Phone + Hard Copy. Write down the Lat/Lon on your data sheet or notebook
    • Two Phone apps. Take GPS points with Survey123 & another app such as OnX.
    • Phone + GPS unit. Take an indendent GPS unit in the field with you and take a GPS point at every plot or site. Store on the GPS unit.

  • Data Backup. Your data should be backed up each evening when you return from the field. Do this by: exporting images from your phone to a computer or external drive, taking pictures of all hard copy data sheets collected that day and backing them up to a computer or external drive, if you have a wifi connection, uploading tabular data and survey responses to Survey123 and downloading the updated version of the database.

6.3 Raw Data Curation

Once your raw data has been collected, you are ready to curate it. Raw data curation involves assuring that your dataset is complete, formatted correctly, is properly stored and backed up, and includes additional notes not contained in the standard data formats.

6.4 Standards for Creating the Raw Data Archive

You should strive to meet the following standards when curating your raw data archive:

  • Your raw data archive should be generated within 1 week of returning from the field (if a single trip), or updated within 1 weeks of collection (for recurring work over longer periods).

  • Your raw data archive should consist of:

    • a shapefile in OGC Geopackage format containing lat lon and point or plot IDs
    • tabular data exported from Survey123
    • pictures or scans of your raw data sheets
    • a local image/video repository where images are placed in a uniqely named folder for each plot or site. NOTE: ideally the title of each folder, in addition to containing the plot or site code should also have a descriptive name alongside the plot code to jog your memory when you are looking back at your plots and images. For example “FRN-31 where we saw moose in lake”.
      • NOTE: your images should contain metadata in their image names. Nowadays most image files will be exported from your phone with a unique number ID such as “IMG_6249”. You should always retain that number in your file name; BUT you should add additional metadata to the file name by appending: 1) the project id, 2) the plot id, 3) a one or two word phrase regarding the question the image was taken in repsonse to. So, for example, an excellent image title for an image with the name and the folder above would be: “FRN-31_pedon_IMG_6249”. Note that this can be done in an automated way using XLSForms in Survey123. If you want to really take this to the next level, also include some descriptive text in the image file name between the Survey123 generated material and the image file name i.e. “FRN-31_pedon_Bwhorizon30cm_IMG_6249”. *Alternatively, you can also use an open source desktop image metadata editing app such as DigiKam.
    • a file (preferrably in [Markdown]#todo) format which contains any additional qualitative information, notes, observations, or a brain dump from the field data collection - if applicable.
    • our standardized [sample log file]#todo that details characteristics and names of physical samples taken.
    • a readme.txt file that contains metadata about your raw data archive. You can add to your existing readme file or Quarto book as well to include the files in your raw data subdirectory in the readme/metadata.

These files live in the raw data folder of your repo. However note that the image repository will be very large in some cases and gitHub is not a good place to store images. Therefore - in your [.gitignore file]#todo, you should include the image type or the image repository directory so that these do not upload to GitHub.

  • Your raw data repository should be stored locally and somewhere in the cloud. For everything except your image/video repository should be stored on GitHub. The image/video repository should be stored locally and either in Flickr or a Shared Google Drive (a truly shared drive, not through your personal account) or Google Photos (same re: shared). NOTE #todo: should image repositories also be required to upload to CFANS Drive?

  • Your raw data should also contain pre-data - including spreadsheet forms and an archive of the Survey123 form - in what format?#todo.

NOTE: folder repo structure should contain a pre-data folder!!!!!#todo

6.5 Physical Sample Curation

The short- and long-term curation of physical samples collected during fieldwork is not trivial. There are several issues to be solved - sample names, sample log, and connecting your sample log to a laboratory sample log.

6.5.1 Sample Names

See our field manual for detailed protocols on sample naming. Briefly, however, your physical samples should be collected in pre-labelled bags. These bags are pre-labelled with a two digit year and sample number. This allows you to collect samples in the field without labelling bags in difficult conditions. The sample log is then a critical piece of your raw data as it connects your sample numbers to metadata about the sample (depths, plot or site, and horizon, for example). Sample numbers then be something like “22-001”. NOTE; this assumes that you won’t collect more than 999 samples in a year (usually safe?). If projects are occuring at the same time, then we need a way of differentiating between these samples. We should therefore append the sample number with the last initials of the project lead who is collecting them in the field (i.e. “22-001-NAJ”).

*NOTE: if you are working on an externally sponsored project or a project that has a larger collaborative team or more parts, your sample names may be prescribed for you. That is fine - in that case, use the sample naming conventions that are part of the larger project. - how will this work with master sample log integration?

6.5.2 Sample Log

Use the sample log template provided in our Field Manual. Take pictures and backup pictures or scans of the sample log each night after data collection.

6.5.3 Connecting your sample log to a master laboratory sample log #todo

Big idea - use Make (Integromat) to trigger a watch on a Survey123 survey that automatically appends row (note need to have survey set up with repeat to produce multiple records of samples with one survey record submission) to a master laboratory physical sample sheet. The Google sheet should also somehow provide a sample name for each? Or you could have in your question at the site or plot level with multipl entries - what are the names/IDs of the samples collected at this plot/site? YES, this second is the best way to go.

6.5.4 Subsequent Laboratory Analysis of Physical Samples #todo

This should maybe be a separate chapter after lab analysis. then can be two stages - integrate with Google Sheets, pull from physical sample log, build on that.