The Shepherd Project is an open source software framework to support mark-recapture, molecular ecology, and social ecology studies.

View a one-hour presentation of the capabilities of the Shepherd Project

The biological and statistical communities already support a number of excellent tools, such as Program MARK,GenAlEx, and SOCPROG for use in analyzing wildlife data. The Shepherd Project is a complementary software application that:

• provides a scalable and collaborative platform for intelligent wildlife data storage and management, including advanced, consolidated searching
• provides an easy-to-use software suite of functionality that can be extended to meet the needs of wildlife projects, especially where individual identification is used
• provides an API to support the easy export of data to cross-disciplinary analysis applications (e.g., GenePop ) and other software (e.g., Google Earth)
• provides a platform that supports the exposure of data in biodiversity databases (e.g., GBIF and OBIS)
• provides a platform for animal biometrics that supports easy data access and facilitates matching application deployment for multiple species

The Shepherd Project started as a collaborative software platform for globally-coordinated whale shark (Rhincodon typus ) research as deployed in the ECOCEAN Whale Shark Photo-identification Library. After many requests to use our software outside of whale shark research, we are now working to make it an open source, community-maintained standard for mark-recapture studies.

• SPLASH Catalog for humpback whales (Megaptera novaeangliae)
• ECOCEAN Whale Shark Photo-identification Library for whale sharks (Rhincodon typus)
• University of Manitoba Polar Bear Library for polar bears (Ursus maritimus)
• ECOCEAN MantaMatcher for manta rays (Manta birostris and Manta alfredi)
• Oregon State University Marine Mammal Institute for
  • humpback whales
  • right whales
  • sperm whales
• PRETOMA for two species of turtles

The Shepherd Project is designed to produce successful, reproducible collaborations between biologists, biostatisticians, computer scientists, and citizen scientists by providing a Web-based software platform for collaboration.

If you answer yes to any of these questions, the Shepherd Project may be a very good choice for your research.

• Are you trying to track individual animals in a wildlife population using natural or genetic markings?
• Are you collecting biological samples from a wildlife population and performing genetic or chemical analyses (e.g., stable isotope measurements, haplotype determination, etc.)?
• Are you looking to increase wildlife data collection through citizen science?
• Are you looking to build a collaborative, distributed research network for a species?
• Are you looking to develop a new animal biometrics solution (e.g., pattern matching from photos) for one or more species?
• Are you collecting behavioral and/or social data for a wildlife study population?

Check out some screenshots of the Shepherd Project in action.

Using the web-based interface of the Shepherd Project Framework, a research team can:

Encounters (a.k.a. “captures” or “sightings”)

• Record an encounter (individual animal at a point in time and location) through a standardized data collection form
• Search a database of standardized encounter data using predefined search criteria
• Receive automated email updates when encounter reports are assigned to new or previously identified marked individuals
• Receive RSS feeds of encounter identifications as they are made
• View all reported encounters
• View individual encounter reports
• View thumbnail catalogs of submitted images
• View adopters of individual encounters (a fundraising tool)
• Edit encounter data with all changes tracked and permanently logged
• Assign encounters to new or existing marked individuals
• Set a list of additional email addresses to be notified with status and sighting updates for submitted encounters
• View an on-line calendar of whale shark encounter reports
• Restore accidentally deleted encounters
• Easily map a TapirLink provider or Integrated Publishing Toolkit (IPT) to the underlying relational database for exposure of mark-recapture data to biodiversity frameworks
• For species with marked individuals identified by unique spot patterning:
  1. Extract spot patterning from photographs and scan for matches across all patterns in library using two approaches to pattern recognition (I3S and Modified Groth).
  2. Share computing power among globally distributed machines.
  3. View pattern match reports and evaluate matches side-by-side, including direct spot remapping and statistical analysis of results.
• Record physical, acoustic, and satellite tag data deployed during an encounter
• Record tissue samples collected with an encounter
• Record the results of genetic sex, haplotype, microsatellite marker, stable isotope, and contaminant analyses performed on a tissue sample

Marked Individuals

• View all marked individuals
• View consolidated capture and sampling history of marked individuals
• Receive automated email updates when a marked individual is re-sighted
• Search a global database of marked individuals using predefined search criteria
• View consolidated mapping of sighting locations an individual
• View co-occurrence for the individual with other individual (e.g., social grouping)

Photos

• Define coimmon photo keywords for your species (e.g. scarring types)
• Add/remove keywords to encounter photos
• Search across photographs using assigned keywords
• Search all images\video using encounter data to create customized albums of images
• View photo metadata (e.g. EXIF data)

General

• Search all encounters and marked individual data using Google Search (when deployed as a web server)
• Assign alternate identifiers to encounters and marked individuals
• View sighting locations with satellite imagery incorporated from Google Maps (encounters, amrked individuals, and Encounter Search results)
• Export sighting data as a KML file or an Excel spreadsheet for use in Google Earth and other mapping applications (Encounter Search).
• Create and edit animal adoptions for project fundraising.
• Review access security logs and track the source of individual logins.
• Use a variety of relational and non-relational database types to store data. The full list of supported databases in available from the DataNucleus web site (our middleware).

API

The framework is open source and meant for you to extend it for your specific project! If it doesn't have the feature you need, use some simple Java programming and create it. Some things we have used it to do on whaleshark.org are:

• Quickly generate open and closed capture history files for population modeling in statistical packages, such as U-CARE, CloseTest, and Program MARK
• Generate statistical reports
• Rely on spam filters to block spurious data submission

The development of additional functionality is currently underway.

2013-5-12

We are pleased to announce the release of Shepherd Project 3.3.

Summary: Version 3.3 brings user account and role management to the Shepherd Project, removing the need to externally manage these through tomcat-users.xml. We use Apache Shiro to ensure that we are using a quality security system. This new feature reduces the technical complexity of the Shepherd Project and better supports collaborative research.

Features and fixes:

• NEW: Add Occurrence_ID to SRGD export/import process (GeneGIS)
• FIX: Fix Time_of_Day hours value in Encounter Search OBIS Excel export format
• NEW: Bundle MySQL JDBC driver by default (GeneGIS)
• FIX: encounters/reject.jsp has a non-functional Cancel button
• FIX: Encounter Search, when selecting the “alive” checkbox, the summary statement still defines the query parameter as “dead”
• FIX: minutes pulldown menu in submit.jsp has two :15 minutes options. One should be :20.
• FIX: Do not link “Unassigned” from the table in occurrence.jsp
• NEW: Integration of Apache Shiro for self-contained AAA security per project (GeneGIS)
• FIX: Duplicate EncounterSetVerbatimEventDate mapping in web.xml
• NEW: Support > = and < = operators for measurements and biomeasurements in Encounter Search and Individual Search (Chris Perkins request for white shark research)
• NEW: Support AND and OR keyword filtering in Encounter Search and Individual Search (Chris Perkins request for white shark research)

2013-4-27

Code development for Shepherd Project 3.3 is done, and we're prepping for testing on whaleshark.org. This release allows for the internal management of user accounts, rather than configuring these externally in tomcat-users.xml. This is a big usability improvement and a decrease in the complexity of the software overall. See the Roadmap for more details about all of the features and fixes in version 3.3.

2013-4-4

We are pleased to announce the release of Shepherd Project version 3.2.0, which can be downloaded from our downloads page. The following new features, fixes, and modifications were made in 3.2 or in the bundled interim release of 3.1.0.

• NEW: Simple data input format and importer (GeneGIS) - SRGD format
• NEW: SRGD CSV file export via URL
• MOD: Gracefully handle null for Encounter.verbatimLocality
• NEW: Add alive/dead status search filter to Individual Search
• NEW: DataNucleus 3.2.0 persistence layer upgrade
• FIX: Change sex and haplotype Encounter Search mapping to workaround a Google Maps issue causing all points to be mapped with one color
• MOD: Allow for multi-select of photo keyword attributes (MantaMatcher request)
• NEW: add “has photo/video” option to Individual Search
• NEW: “Remove All” function to clear out all data to prep for fresh import (GeneGIS)
• FIX: occurence.jsp mapping showing linked Markedindividual sightings, even if only one encounter is assigned to occurrence
• NEW: Simple mark-recapture export format from Individual Search
• MOD: Modify front page “jump to” behavior to more clearly select encounter objects
• NEW: Add sex and haplotype mapping to Individual Search mapped tab
• FIX: Ensure encounter dir exists when using EncounterAddImage
• FIX: Support photoless adoption submission
• MOD: Move BioMeasurement search field
• FIX: Remove legacy size measurement field from the calendar display
• MOD: Further null value handling to support cleaner data imports from legacy projects
• MOD: Improve null handling of Encounter information to support the import of legacy projects and their data
• FIX: Search results handle no icon elegantly
• FIX: Null exception handling improvements for CAPTURE export
• NEW: Encounter Search: support search for unidentified encounters
• FIX: Limit behavior field size in Encounter Search
• NEW: Add behavior filter option to Individual Search - it is date/location dependent
• MOD: Standardize maximum media upload file size across upload servlets and define its value in commonConfiguration.properties
• NEW: Support Encounter.country, including submission, editing, and searching capabilities. On/off should also be set through commonConfiguration.properties
• NEW: Add Encounter Search filter on biological sample ID to provide direct addressability of tissue samples
• MOD: Limit Fst to four sig. figures
• NEW: Add documentation link to the base menu
• NEW: Add connectivity displays to the maps of the Mapped Results of the Individual Search

Past news...

Development work for the Shepherd Project is currently being supported by:

• Oregon State University Marine Mammal Institute with the Office of Naval Research (ONR)
• Royal Caribbean Ocean Fund
• ECOCEAN USA
• Cascadia Research Collective
• PRETOMA

You can help move the Shepherd Project forward by making a donation! Your donation is tax deductible in the United States.

Please send feedback to jason at whaleshark dot org. Your ideas to improve the Shepherd Project are most welcome!

The Shepherd Project started as the ECOCEAN Whale Shark Photo-identification Library, which is a multiuser, web-based, research application for studying whale sharks (Rhincodon typus ). The aim of the ECOCEAN Library is to prevent individual “silos” of whale shark data and to promote a global, cooperative approach to whale shark research using the Web as a communications and research platform. The Library went on-line and began collecting whale shark encounter data from the web in January 2003. In early 2004, the pattern-recognition system that allows the Library to distinguish between individual whale sharks using natural spot patterning was integrated. Since its first line of code, the ECOCEAN Library has seen continuous feature additions, bug fixes, and performance enhancements. Our work to maintain and enhance the Library is ongoing and requires knowledge of Java, J2EE, JDO, PHP , Flash/Flex, HTML , XML , RSS , a wee bit of Python, and CSS .

The following publications have influenced our design and development of the Shepherd Project:

• Arzoumanian Z, Holmberg J & Norman B (2005) An astronomical pattern-matching algorithm for computer-aided identification of whale sharks Rhincodon typus . Journal of Applied Ecology 42, 999-1011.
• Hochachka WM, Fink D, Hutchinson RA, Sheldon D, Wong W-K, Kelling S. Data-intensive science applied to broad-scale citizen science. Trends Ecol Evol. 2012 Feb. 1;27(2):130–137.
• Madon, B., Gimenez, O., McArdle, B., Scott Baker, C., & Garrigue, C. (2011). A new method for estimating animal abundance with two sources of data in capture-recapture studies. Methods in Ecology and Evolution, 2(4), 390-400.
• Michener WK, Jones MB. Ecoinformatics: supporting ecology as a data-intensive science. Trends Ecol Evol. 2012 Jan.;27(2).
• Schwarz, C. J. (2009). Migration and movement – the next stage. Pages 325-350 in Modeling Demographic Processes in Marked Populations Series: Environmental and Ecological Statistics , Vol. 3. Thomson, David L.;Cooch, Evan G.; Conroy, Michael J. (Eds.). Springer, New York.
• Holmberg J, Norman B & Arzoumanian Z (2009) Estimating population size, structure, and residency time for whale sharks Rhincodon typus through collaborative photo-identification. Endangered Species Research, (7) 39-53.
• Holmberg J, Norman B & Arzoumanian Z (2008) Robust, comparable population metrics through collaborative photo-monitoring of whale sharks Rhincodon typus. Ecological Applications 18(1): 222-223.
• Williams BK, Nichols JD, Conroy MJ (2002) Analysis and management of animal populations. Academic Press, San Diego, CA.
• White GC, Burnham KP (1999) Program MARK: Survival estimation from populations of marked animals. Bird Study 46:120–138.
• Whitehead, H. Analyzing Animal Societies: Quantitative Methods for Vertebrate Social Analysis (University of Chicago Press, 2008).
• Wieczorek J, Bloom D, Guralnick R, Blum S, Döring M, et al. (2012) Darwin Core: An Evolving Community-Developed Biodiversity Data Standard. PLoS ONE 7(1): e29715. doi:10.1371/journal.pone.0029715

Jason Holmberg, Mark McBride, Charles Overbeck, and Ed Stastny are the software developers and architects for the Shepherd Project. Robert Jordan and Jason Holmberg are developing the technical documentation.

Do you want to join us? Please join our Google Group at:

http://groups.google.com/group/shepherd-project-developers

Esri and the Esri Logo are licensed trademarks of Environmental Systems Research Institute, Inc.