---
title: "Quality Control of SHARK Data"
output:
  rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Quality Control of SHARK Data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include=FALSE}
shark_available <- tryCatch({
  httr::status_code(httr::GET("https://shark.smhi.se/",
                               httr::timeout(5))) < 400
}, error = function(e) FALSE)

obis_available <- tryCatch({
  httr::status_code(httr::GET("https://api.obis.org/",
                               httr::timeout(5))) < 400
}, error = function(e) FALSE)
```

## Overview

The SHARK4R package provides a set of functions to perform quality control (QC) on SHARK data. These functions help identify missing or invalid values, spatial errors, and statistical outliers and are mainly intended for internal data validation. The tutorial covers:

- **Field validation** (required and datatype-specific fields)
- **Code validation** (project and platform codes)
- **Geospatial checks** (points on land, basins, proximity to shore)
- **Depth checks** (missing, negative, or implausible values)
- **Outlier detection** (parameter-specific thresholds)
- **Logical Parameter Checks** (parameter-specific rules)
- **Interactive QC** with a Shiny app

This workflow ensures SHARK data are consistent, valid, and ready for analysis. Several quality control components, originally developed by Provoost and Bosch (2024), have been adapted for compatibility with the SHARK format.

---

## Installation

You can install the latest version of `SHARK4R` from CRAN using:
```{r, eval=FALSE}
install.packages("SHARK4R")
```

Load the package along with `dplyr`:

```{r}
library(SHARK4R)
```

---

## Retrieve SHARK Data

You can fetch SHARK data using the same filtering options as the SHARK web interface. Explore available options with:

```{r, include=FALSE, eval=shark_available}
shark_options <- try(get_shark_options(), silent = TRUE)
if (inherits(shark_options, "try-error")) shark_available <- FALSE
```

```{r, eval=FALSE}
shark_options <- get_shark_options()
```

Filter datasets containing "Chlorophyll":

```{r, include=FALSE, eval=shark_available}
chlorophyll_datasets <- shark_options$datasets[grepl("Chlorophyll",
                                                     shark_options$datasets)]
selected_dataset <- chlorophyll_datasets[1]
```

```{r, eval=FALSE}
# Filter names using grepl
chlorophyll_datasets <- shark_options$datasets[grepl("Chlorophyll",
                                                     shark_options$datasets)]

# Select the first dataset for demonstration
selected_dataset <- chlorophyll_datasets[1]

# Print the name of the selected dataset
print(selected_dataset)
```

```{r, echo=FALSE, eval=shark_available}
print(selected_dataset)
```

Download the selected dataset as a data frame:

```{r, include=FALSE, eval=shark_available}
chlorophyll_data <- try(
  get_shark_datasets(selected_dataset, save_dir = tempdir(),
                     return_df = TRUE, verbose = FALSE),
  silent = TRUE
)
if (inherits(chlorophyll_data, "try-error")) shark_available <- FALSE
```

```{r, eval=FALSE}
chlorophyll_data <- get_shark_datasets(selected_dataset,
                                       save_dir = tempdir(),
                                       return_df = TRUE,
                                       verbose = FALSE)

print(chlorophyll_data)
```

```{r, echo=FALSE, eval=shark_available}
print(chlorophyll_data)
```

SHARK data can be downloaded and saved locally using the `save_dir` argument, then imported into R using the function `read_shark()` for both ZIP archives or text files.

---

## Step 1: Check Required Fields

Validate mandatory fields:

- **Global fields**: `check_datatype()`
- **Datatype-specific fields**: `check_fields()` with optional `field_definitions`

```{r, include=FALSE, eval=shark_available}
check_fields_result <- try(
  check_fields(data = chlorophyll_data, datatype = "Chlorophyll"),
  silent = TRUE
)
```

```{r, eval=FALSE}
check_fields(data = chlorophyll_data, datatype = "Chlorophyll")
```

```{r, echo=FALSE, eval=shark_available && !inherits(check_fields_result, "try-error")}
check_fields_result
```

---

## Step 2: Validate Project and Platform Codes

Ensure metadata codes follow SHARK conventions:

```{r, include=FALSE, eval=shark_available}
check_codes_result1 <- try(check_codes(chlorophyll_data), silent = TRUE)
check_codes_result2 <- try(
  check_codes(data = chlorophyll_data, field = "platform_code",
              code_type = "SHIPC", match_column = "Code"),
  silent = TRUE
)
```

```{r, eval=FALSE}
# Validate project codes
check_codes(chlorophyll_data)

# Validate ship/platform codes
check_codes(data = chlorophyll_data,
            field = "platform_code",
            code_type = "SHIPC",
            match_column = "Code")
```

```{r, echo=FALSE, eval=shark_available && !inherits(check_codes_result1, "try-error")}
check_codes_result1
```

```{r, echo=FALSE, eval=shark_available && !inherits(check_codes_result2, "try-error")}
check_codes_result2
```

---

## Step 3: Geospatial Checks

### Visualize Data Points

```{r, eval=shark_available && Sys.getenv("NOT_CRAN", unset = "FALSE") == "TRUE"}
plot_map_leaflet(chlorophyll_data)
```

```{r, echo=FALSE, eval=!Sys.getenv("NOT_CRAN", unset = "FALSE") == "TRUE"}
message("The interactive map is omitted here but appears in the online tutorial.")
```

### Identify Points on Land

```{r, include=FALSE, eval=shark_available && obis_available}
n_rows_on_land <- try(check_onland(chlorophyll_data), silent = TRUE)
onland_ok <- !inherits(n_rows_on_land, "try-error")
```

```{r, eval=FALSE}
n_rows_on_land <- check_onland(chlorophyll_data)
nrow(n_rows_on_land)
```

```{r, echo=FALSE, eval=shark_available && obis_available && onland_ok}
nrow(n_rows_on_land)
```

Optional geospatial QC functions:

- `positions_are_near_land()`
- `which_basin()` from the [iRfcb package](https://CRAN.R-project.org/package=iRfcb)

---

## Step 4: Depth Checks

Verify plausibility and consistency of depth values:

```{r, include=FALSE, eval=shark_available && obis_available}
depth_result1 <- try(check_depth(data = chlorophyll_data), silent = TRUE)
depth_result2 <- try(
  check_depth(data = chlorophyll_data, "water_depth_m"),
  silent = TRUE
)
depth_ok <- !inherits(depth_result1, "try-error")
```

```{r, eval=FALSE}
check_depth(data = chlorophyll_data) # default columns: min/max depth
check_depth(data = chlorophyll_data, "water_depth_m")
```

```{r, echo=FALSE, eval=shark_available && obis_available && depth_ok}
depth_result1
depth_result2
```

Checks performed:

- Missing depth column or empty values (warnings)
- Non-numeric or negative values (errors)
- Depth exceeding bathymetry or minimum > maximum (errors)

---

## Step 5: Outlier Detection

Retrieve reference statistics for your datatype:

```{r, include=FALSE, eval=shark_available}
shark_statistics <- try(
  get_shark_statistics(datatype = "Chlorophyll", fromYear = 2020,
                       toYear = 2024, verbose = FALSE),
  silent = TRUE
)
stats_ok <- !inherits(shark_statistics, "try-error")
```

```{r, eval=FALSE}
shark_statistics <- get_shark_statistics(datatype = "Chlorophyll",
                                         fromYear = 2020,
                                         toYear = 2024,
                                         verbose = FALSE)

print(shark_statistics)
```

```{r, echo=FALSE, eval=shark_available && stats_ok}
print(shark_statistics)
```

Detect extreme values using thresholds (e.g., 99th percentile):

```{r, eval=FALSE}
check_outliers(data = chlorophyll_data,
               parameter = "Chlorophyll-a",
               datatype = "Chlorophyll",
               threshold_col = "P99",
               thresholds = shark_statistics)
```

```{r, echo=FALSE, eval=shark_available && stats_ok}
check_outliers(data = chlorophyll_data,
               parameter = "Chlorophyll-a",
               datatype = "Chlorophyll",
               threshold_col = "P99",
               thresholds = shark_statistics)
```

Visualize anomalies:

```{r, eval=shark_available && stats_ok && Sys.getenv("NOT_CRAN", unset = "FALSE") == "TRUE"}
# Scatterplot with horizontal line at 99th percentile
scatterplot(chlorophyll_data,
            hline = shark_statistics$P99)
```

```{r, echo=FALSE, eval=!Sys.getenv("NOT_CRAN", unset = "FALSE") == "TRUE"}
message("The interactive plot is omitted here but appears in the online tutorial.")
```

---

## Step 6: Logical Parameter Checks

Use `check_parameter_rules()` to flag measurements that violate parameter-specific or row-wise logical rules.

```{r, eval=FALSE}
check_parameter_rules(data = chlorophyll_data)
```

```{r, echo=FALSE, eval=shark_available}
check_parameter_rules(data = chlorophyll_data)
```

* `return_df = TRUE` gives a data frame of violations.
* `return_logical = TRUE` gives logical vectors for each parameter.
* Only parameters present in the dataset are checked; available parameters are listed if none match.
* You can define custom rules by providing your own `param_conditions` or `rowwise_conditions` lists.

---

## Step 7: Station Matching

Verify station names against the official SHARK registry:

```{r, eval=FALSE}
station_match <- match_station(chlorophyll_data$station_name)
head(station_match)
```

```{r, echo=FALSE, eval=shark_available}
station_match <- match_station(chlorophyll_data$station_name)
head(station_match)
```

To plot stations and their distances from the station register in an interactive map:

```{r, eval=shark_available && Sys.getenv("NOT_CRAN", unset = "FALSE") == "TRUE"}
check_station_distance(data = chlorophyll_data,
                       plot_leaflet = TRUE)
```

```{r, echo=FALSE, eval=!Sys.getenv("NOT_CRAN", unset = "FALSE") == "TRUE"}
message("The interactive map is omitted here but appears in the online tutorial.")
```

To check if stations are nominal (comparing unique coordinates per station):

```{r, eval=FALSE}
check_nominal_station(data = chlorophyll_data)
```

```{r, echo=FALSE, eval=shark_available}
check_nominal_station(data = chlorophyll_data)
```

---

## Interactive QC with Shiny

For a more user-friendly interface, use the Shiny QC app:

```{r, eval=FALSE}
# Run the app
run_qc_app()

# Alternative, download support files and knit documents locally
check_setup(path = tempdir()) # using a temp folder in this example
```

The app provides point-and-click access to the same QC checks described above.

---

## Recommended Workflow Summary

1. **Check Required Fields** (`check_datatype()`, `check_fields()`)
2. **Validate Codes** (`check_codes()`)
3. **Geospatial Checks** (`plot_map_leaflet()`, `check_onland()`, optional `positions_are_near_land()`, `which_basin()`)
4. **Depth Checks** (`check_depth()`)
5. **Outlier Detection** (`check_outliers()`, `scatterplot()`)
6. **Logical Parameter Checks** (`check_parameter_rules()`)
7. **Station Matching** (`match_station()`)
8. **Final Review & Visualization** (interactive maps and scatterplots)

Following this order ensures comprehensive QC and prepares your SHARK data for analysis.

---

## Citation

```{r, echo=FALSE}
citation("SHARK4R")
```

```{r, echo=FALSE}
clean_shark4r_cache(0, clear_perm_cache = TRUE, verbose = FALSE)
```

## References
- Provoost P, Bosch S (2024). `obistools`: Tools for data enhancement and quality control. Ocean Biodiversity Information System. Intergovernmental Oceanographic Commission of UNESCO. R package version 0.1.0, <https://iobis.github.io/obistools/>.
