Misadentures Near the Mule Mountains

I am always trying to get to places where the species have not been recorded on iNaturalist. As the number of users in Southern California have increased that has become progressively more difficult. There are still a few patches though. One such patch is the Mule Mountains, about thirty miles southwest of Blythe. While there are a couple observations near the edges of the mountains the site has yet to have extensive documentation. Even related sites such as Calflora do not show much about the site.

With this in mind, I headed to Coon Hollow Campground dragging along my 4 and 6 year old boys. Being ten or more miles down a dirt road and in an area with very few observations, this campground seemed a good starting point for some adventures. So I set up my tent, ate a bit then headed for the Mule Mountains.

Before heading on this trip I had scouted the site as well as I could on Google. There was a little road called Bradshaw Trail which looked like it was in good shape up until the edge of the Mule Mountains. It was pretty clear my Prius wasn't up for a crossing of the mountains, but the road looked good on Google up until the edge of the mountains.

When I reached this road, it was clear there was a problem with my plan. About fifty feet down the road there was a warning that it was only 4 wheel drive accessible. This should have stopped me cold, but the road was in such good condition! I reasoned that with how good the road looked at the four wheel drive sign I should be able to go slowly until I was obviously out of my league then turn back.

So I drove about a half mile. Then I hit the sand. I continued through the sand and was on solid road again. That knocked my senses into me. I decided to turn around. The road was pretty small though and I wasn't sure I could do a u-turn without getting stuck. Also, I was only one patch of sand away from good road. I elected to back through the sand and turn around when I got to the cross street. This probably would have worked. However I was of the opinion that I needed to go quickly or I would get stuck in the sand. So I went faster than I should have. I ended up getting too close to the edge of the road and got solidly stuck in the much deeper sand there.

After a half hour of trying to dig myself out with some friendly motorcyclists which stopped to help (people are so much nicer when you are with kids), I realized I remarkably had cell phone service. So I just paid the exorbitant fees tow truck companies charge to pull someone out of the situation like that.

So I spent the next two hours waiting for the tow truck photographing plants and insects as I was really quite in the middle of nowhere. The kids played in the sand.

Luckily once towed out, the car still worked. So I returned to camp vowing not to start the car up again until it was time to go home. The area around the campground was pretty much undocumented, and it seemed like I could easily walk to the Mule Mountains a mile or so east of the campground.

The next morning I woke up to a small mantis near my head in the tent! Great! I had seen one of those the night before at an improvised bug attracting light but my kids stepped on it before I could get a picture. So I took a few photos.

Then I smelled smoke! Smoke? I realized it was coming from either my flash or my camera. I am pretty sure one of the capacitors had previously failed on the flash so I figured it was finally dying for real.

After breakfast we started east toward the Mule Mountains. The kids seemed excited enough and in the morning cool it was the perfect time to walk a couple miles.

Perhaps half way to the edge of the mountains the older one starts complaining about his feet. He says his shoes are hurting his feet. I look at his feet. It seemed like something got through the holes in his crocks and made his feet itch. Ugh, why did I let him bring crocs? I gave up on the Mule Mountains and started to carry him back to camp. Luckily about half way back he stopped complaining and started walking, but at that point I wasn't going to be able to motivate him to turn back towards the mountains.

So we spent the rest of the day playing games and searching the dry stream bed. Luckily my flash was still working apparently normally. The camera had been starting to act a little funny though. At about 6 PM while I was taking a photo of a beetle the camera said "Camera Error Turn power off then on." Turning it off did nothing at all. So I pulled the battery. Same problem. Pulled off the lens to find that the shutter was stuck closed. The Sony a6300 camera had survived three years and 267,331 shutter actuations but it was finally good and dead. This is the last photo that camera is likely to take:

Luckily the camera had survived three quarters of the trip. However the last night and the drive back I was stuck with my cell phone. The camera seems like it could be repaired but to do so would not be cheap. I ended up purchasing a new A7III as a replacement.

Could have done without the towing bill and dead camera, but other than that was a pretty good trip. Managed to get 359 observations in a really obscure and lightly documented part of the desert. Kids seemed to hold up well to camping in primitive conditions so I may have to try a similar place next spring.

Updated iNaturalist Upload Scripts

After a few rainy days I managed to come up with a python script using pyinaturalist which is at least as efficient as the existing upload method. For people uploading a large number of photos of the same species it is much more efficient.

The basic workflow is to put all the photos in a folder with the common name or scientific name and/or taxon number as the name of the folder.

You can put dozens of photos in any of these folders, so uploading fifty observations of the same species only requires running the script once. All the photos in every folder in the master folder will be uploaded as an individual observation.

What about uploading multiple photos to the same observation? This isn't much harder, add the photos to a sub-folder in the species folder. The script doesn't care what name, so I usually just leave these as "new folder." All the photos in this subfolder will be uploaded to an individual observation.

When you run the script, it gives you a few inputs to fill out:

The more annoying of these are "APP ID" and "APP Secret." You have to create an app for iNaturalist to upload through scripts. Fortunately this just takes a few seconds. The folder it wants is the main folder which contains all the species folders to be uploaded. Even if you are uploading just one species the folder with the photos will need to be in a master folder containing nothing but species folders to be uploaded.

Chances are you will want to go in the code and fill out default values for most of the entries so you will not have to fill them out every time you run the script. It is commented where to add them to the import_gui.py file.

Once the script is done, it moves all the files out of this folder, and puts them next to the main folder in a folder called "Uploaded." This should keep you from re-uploading everything if you lose connection mid-upload. Just run the script again, all the uploaded photos will now be gone. 

Since it got rather long, I uploaded it to Github as iNaturalist-Uploads. There are three files which all must be in the same folder. upload_folders.py is the file which is run as a python script. The other two (import_gui.py and import_functions.py) have functions which I preferred to put in a different folder to keep it less messy. All three files need to be saved to the same folder to run.

This probably doesn't make sense for most users as it is way less intuitive than the site submission tool, but if you are experienced with python or have a ton of photos of a limited number of species to upload this starts to make sense.

Uploading everything in a folder to iNaturalist with python

Since pyinaturalist recently came out I thought it would be a good time to try and write a script to automatically upload files to iNaturalist. Few if any example scripts are out there and this should make it easier for other people to write one which matches their workflow.

This script assumes a large number of photos of the same species. This might happen for example if you were trying to map every tree on a property. The workflow would consist of taking a single geotagged photo of each individual then separating out the photos so each one is in a folder which starts with its taxon ID. For example aphids would go in a folder named '52381' or '52381 Aphids' or '52381-Aphididae'

If you don't have python, I suggest installing Anaconda then pyinaturalist. You will then need to get an app ID.

Open this script, add your user name, password, app id, secret, and the time zone of the photos. Then run the script. It should upload everything jpg file in the folder as the file you select.

# Input your user name here:
user = ''

# Input your password here:
passw = ''

# Input your app ID and secret here:
app = ''
secret = ''


# Input the time zone for the photos here, options can be found at the
# website below
# https://gist.github.com/mjrulesamrat/0c1f7de951d3c508fb3a20b4b0b33a98
time_zone = 'America/Los_Angeles'



# tkinter used to choose a file
from tkinter import filedialog
from tkinter import Tk

# os used to get a folder name
import os

# pillow used to get exif data from the photos
import PIL
from PIL import ExifTags

# This is used to upload the photos.
import pyinaturalist

from pyinaturalist.rest_api import create_observations
from pyinaturalist.rest_api import get_access_token

print("Running")

# This code lets you choose a photo, can delete and replace with folder_name=''
root = Tk()
filename =  filedialog.askopenfilename(initialdir = "/",
                                    title = "Select one of the .jpg files in "
                                    "the folder to be uploaded. All files in "
                                    "the folder will be uploaded. The folder "
                                    "name should start with the taxon number",
                                    filetypes = (("jpeg files","*.jpg"),
                                    ("all files","*.*")))
root.withdraw()

#
folder_name = os.path.dirname(filename) +'/'
print('Uploading all photos in ' + folder_name + ' as a unique observation')

# Makes a list of all files in the folder inside element 2 of a tuple
for file in os.walk(folder_name):
    if file[0] == folder_name:
        files = file

# Creates list of all the file paths for every file in the folder.
file_paths = []
for file in files[2]:   # All files are in files[2]
    file_path = files[0] + file  # files[0] has the path to the folder
    file_paths.append(file_path) # Makes a big list of paths


# This function returns the latitude and longitude of a .jpg image
def get_lat_long(image):
    # Gets all the exif data from the photo
    exif = {
        PIL.ExifTags.TAGS[k]: v
        for k, v in image._getexif().items()
        if k in PIL.ExifTags.TAGS
    }

    # From all the exif data, pulls the GPS data
    gps_info = exif.get('GPSInfo')
    # The GPS data is in a odd format, so have to dig for it a bit. This was
    # only tested on files lightroom tagged.
    latitude_direction = str(gps_info.get(1)[0])
    latitude_degrees = float(gps_info.get(2)[0][0])
    minutes = float(gps_info.get(2)[1][0])
    multiplier = float(gps_info.get(2)[1][1])
    latitude_minutes = minutes/multiplier
    seconds = float(gps_info.get(2)[2][0])
    multiplier = float(gps_info.get(2)[2][1])
    latitude_seconds = seconds/multiplier
   
   
    # The sign is changed depending on if this is N or S
    if latitude_direction == 'N' or latitude_direction == 'n':
        latitude = latitude_degrees+latitude_minutes/60 + latitude_seconds/3600
    elif latitude_direction == 'S' or latitude_direction == 's':
        latitude = -(latitude_degrees+latitude_minutes/60 + latitude_seconds/3600)
       
    longitude_direction = gps_info.get(3)[0]
    longitude_degrees = gps_info.get(4)[0][0]
    minutes = float(gps_info.get(4)[1][0])
    multiplier = float(gps_info.get(4)[1][1])
    longitude_minutes = minutes/multiplier
    seconds = float(gps_info.get(4)[2][0])
    multiplier = float(gps_info.get(4)[2][1])
    longitude_seconds = seconds/multiplier
    # The sign is changed depending on if this is E or W
    if longitude_direction == 'E' or longitude_direction == 'e':
        longitude = longitude_degrees+longitude_minutes/60 +longitude_seconds/3600
    elif longitude_direction == 'W' or longitude_direction == 'w':
        longitude = -(longitude_degrees+longitude_minutes/60 +longitude_seconds/3600)
   
    latitude_longitude = [latitude, longitude]
   
    # Returns a list with both latitude and longiude in decimal format.
    return latitude_longitude
   
# Pulls the date information from
def get_date(image):
    # Gets all the exif data from the photo
    exif = {
        PIL.ExifTags.TAGS[k]: v
        for k, v in img._getexif().items()
        if k in PIL.ExifTags.TAGS
    }
    # Pulls the date and time from the exif format
    date = exif.get('DateTime').split()[0]
    time = exif.get('DateTime').split()[1]
    # Reformats the date to use - instead of :
    for character in date:
        if character == ':':
            date = date.replace(character, '-')
    # Combines the date and time to match the format pyinaturalist wants,
    date_time = str(date) + 'T' + str(time)
    # returns a date and time formatted to submit to iNaturalist with
    # pyinaturalist
    return date_time


# This presumes the name of the folder starts with the taxon number.It finds
# the taxon number by looking at the folder name and taking all the digits it
# sees. This allows you to name the folder "##### species name" to quickly
# tell where photos go. For example anything in '52381-Aphididae' is uploaded
# as an aphid.
def get_taxon(folder):
    taxon = ''
    folder =os.path.split(os.path.dirname(folder_name))[-1]
    for character in folder:
        if character.isdigit():
            taxon = taxon + character
    return taxon


# This is getting a token to allow photos to be uploaded.
token = get_access_token(username=user, password=passw,
                         app_id=app,
                         app_secret=secret)

# This goes to every file, checks if it is a jpg, gets the gps coordinates,
# get the time, and uploads it to iNaturalist.
for file in file_paths:
   if file[-3:] == 'jpg' or file[-3:] == 'JPG' or file[-3:] == 'Jpg':
       print('Uploading ' + file)
       try:
           img = PIL.Image.open(file)
           coordinates = get_lat_long(img)
       except:
           coordinates = 'No Coordinates'
       try:
           img = PIL.Image.open(file)
           date_time = get_date(img)
       except:
           date_time = 'No Date or Time' 
          
       # This requires the folder name to start with the taxon number.
       taxon = get_taxon(folder_name)   

       params = {'observation':
                    {'taxon_id': taxon,  # Vespa Crabro
                     'observed_on_string': date_time,
                     'time_zone': time_zone,
                     'description': '',
                     'tag_list': '',
                     'latitude': coordinates[0],
                     'longitude': coordinates[1],
                     'positional_accuracy': 50, # meters,
       

                     'observation_field_values_attributes':
                        [{'observation_field_id': '','value': ''}],
                     },}
       r = create_observations(params=params, access_token=token)
       
       new_observation_id = r[0]['id']

      
       from pyinaturalist.rest_api import add_photo_to_observation

       r = add_photo_to_observation(observation_id=new_observation_id,
                        file_object=open(file, 'rb'),
                        access_token=token)

print("Program complete")

Apple Health with Python

For about four years I have been storing my weight in the Apple Health app. At the same time, I am almost always carrying my phone so it has been recording my number of steps for at least as long. As Python seems to be a hot topic at the moment, I thought I would try using it to look for correlations between my weight and number of steps.

The whole project would have been way easier in Excel, but it seemed a good way to learn plotting and mathematics in Python.

First I plotted my weight and number of steps:

The weight plot definitely supports the hypothesis that it is easy to lose weight in the short term, and almost impossible over a five year period. I managed to drop fast when I first started recording but then slowly gained it all back and then some. The steps are more opaque. At first glance they don't say much.

So I tried plotting steps versus weight for the days where I have both sets of data. The X axis is how many steps I took that day, and the Y axis is what I weighed that morning:

Better, this shows that when I am above 190lbs I rarely walk more than 10,000 steps, but it is still an amorphous blob with a whole bunch of noise.

Some of the noise can be removed by instead plotting monthly averages. For every 30 day period in the dataset I averaged my steps and averaged by weight:

Again an amorphous blob, but  it does show that when I walk a lot I am more likely to be low weight and when I am heavy I am more likely to not walk much.

That seemed like it missed an important detail though, am I 190 pounds and dropping fast because I walk so much? or am I 180 pounds and gaining weight fast? That problem can be removed by instead looking at weight gain. So I averaged my number of steps for a 30 day period, then compared my weight in that 30 day period to my weight in the previous 30 day period:

 There is still a lot of noise, but when I lose more than two pounds in a month I always am walking more than 6,500 steps, and when I gain more than a pound in a week I am always walking less than 7,000 steps.

The code to do this simple analysis started with trying to get the data out of the export.xml file which apple provides. This was easy enough in excel, but I made it a point to use python. Eventually I came up with the following script which pulls all the weight and steps from the file:

import xml.etree.ElementTree
# This file location will need to be edited to match where the file is.
xDoc = xml.etree.ElementTree.parse(
        'E:/My Documents/Python files/export/apple_health_data/export.xml')

items = list(xDoc.getroot()) # Convert the XML items to a list

step_data = []

# Searches for steps in the XML file.
item_type_identifier='HKQuantityTypeIdentifierStepCount' # Desired data type
for i,item in enumerate(items):
    if 'type' in item.attrib and item.attrib['type'] == item_type_identifier:
        # Attributes to extract from the current item
        step_data.append((item.attrib['type'],
                         item.attrib['endDate'],
                         item.attrib['value']))

weight_data = []
# Searches for weights in the XML file
item_type_identifier='HKQuantityTypeIdentifierBodyMass'
for i,item in enumerate(items):
    if 'type' in item.attrib and item.attrib['type'] == item_type_identifier:
        # Attributes to extract from the current item
        weight_data.append((item.attrib['type'],
                         item.attrib['endDate'],
                         item.attrib['value']))


file_location = \
    'E:/My Documents/Python files/export/apple_health_data/Extract.csv'
file = open(file_location, 'w')

# Writes the list to a csv file by putting a , after every line and a \n at the
# end of a row.
i= 0
for i in range(len(step_data)):
    file.write(str(step_data[i][0])+',')
    file.write(str(step_data[i][1])+',')
    file.write(str(step_data[i][2])+',')
    file.write(str('\n'))
    i = i + 1

i= 0
for i in range(len(weight_data)):
    file.write(str(weight_data[i][0])+',')
    file.write(str(weight_data[i][1])+',')
    file.write(str(weight_data[i][2])+',')
    file.write(str('\n'))
    i = i + 1
      
file.close()

I could have just used the data directly, but I chose to divide this up into three scripts. Now that I have all the data in a csv file it is necessary to do some basic analysis on it to make the plots. The averaging and summing necessary was done in a second script which imports the data from the previous script and again saves to csv:

import csv
import pandas as pd
import datetime

file_location = \
    'E:/My Documents/Python files/export/apple_health_data/Extract.csv'
   
weights = []
weight = []

# This pulls the weights data out of the csv file and puts it into the object
# called weights
with open(file_location) as csvfile:
    readCSV = csv.reader(csvfile, delimiter=',')

    for row in readCSV:
        if row[0] == 'HKQuantityTypeIdentifierBodyMass':

            pounds = float(row[2])
           
            date_string = row[1].split()[0]
            year_int = int(date_string.split('-')[0])
            month_int = int(date_string.split('-')[1])
            day_int = int(date_string.split('-')[2])

            date = datetime.date(year_int, month_int, day_int)
           
            weight = [date, pounds]
            weights.append(weight)


step_counts = []
step_count = []
# This pulls the steps data out of the csv file and puts it into the object
# called step_counts
with open(file_location) as csvfile:
    readCSV = csv.reader(csvfile, delimiter=',')

    for row in readCSV:
        if row[0] == 'HKQuantityTypeIdentifierStepCount':

            steps = int(row[2])           
            date_string = row[1].split()[0]
            year_int = int(date_string.split('-')[0])
            month_int = int(date_string.split('-')[1])
            day_int = int(date_string.split('-')[2])
            date = datetime.date(year_int, month_int, day_int)
           
            step_count = [date, steps]
            step_counts.append(step_count)
           
# The steps from Apple come in smaller batches than make sense for this analysis
# Therefore I added up all the steps on any particular day and put it in the
# object daily_steps
summed_steps = []
daily_steps = 0
for entry in step_counts:
    if entry[0] == date:
        daily_steps += int(entry[1])
    else:
        days_steps = [date, daily_steps]
        summed_steps.append(days_steps)
        date = entry[0]
        daily_steps = int(entry[1])

# The weights and steps were moved from a list to a dataframe for ease of some
# of the later anaylsis
weights_data_frame=pd.DataFrame(weights,columns=['date', 'weight'])
steps_data_frame = pd.DataFrame(summed_steps,columns=['date','steps'])

# The weights and steps were combined into one dataframe.
combined_data = steps_data_frame.merge(weights_data_frame, how='outer')

# In case any data is out of order it is sorted
combined_data = combined_data.sort_values(by=['date'])

# Calculates the 7 day average weight.
weekly_average_weight = combined_data["weight"].rolling(min_periods=1,
                                                     center=True,
                                                     window=7).mean()
weekly_average_weight = weekly_average_weight.to_frame('weekly average weight')
combined_data = combined_data.join(weekly_average_weight)


# Calculates the 7 day average steps.
weekly_average_steps = combined_data["steps"].rolling(min_periods=1,
                                                    center=True,
                                                    window=7).mean()   
weekly_average_steps = weekly_average_steps.to_frame('weekly average steps')
combined_data = combined_data.join(weekly_average_steps)

# Calculates the 30 day average weight.
monthly_average_weight = combined_data["weight"].rolling(min_periods=1,
                                                     center=True,
                                                     window=30).mean()
monthly_average_weight = monthly_average_weight.to_frame(
                                                'monthly average weight')
combined_data = combined_data.join(monthly_average_weight)


# Calculates the 30 day average steps.
monthly_average_steps = combined_data["steps"].rolling(min_periods=1,
                                                    center=True,
                                                    window=30).mean()   
monthly_average_steps = monthly_average_steps.to_frame('monthly average steps')
combined_data = combined_data.join(monthly_average_steps)


# Compares the average weight to the average weight the week before to look
#for a gain.
i = 0
weekly_gain = []
for i in range(len(combined_data)-4):
    if combined_data.iloc[i+4][3] and combined_data.iloc[i-3][3] \
        and i +4< len(combined_data) and i-4>=0:
            gain = combined_data.iloc[i+4][3] - combined_data.iloc[i-3][3]

            dated_gain = [combined_data.iloc[i][0], gain]
            weekly_gain.append(dated_gain)
            i += 1

# Puts the gain into a dataframe
weekly_gain = pd.DataFrame(weekly_gain,columns=['date','weekly weight gain'])

# Merges the dataframe
combined_data = combined_data.merge(weekly_gain, how='outer')

# Compares the average weight to the average weight the month before to look
# for a gain.
i = 0
monthly_gain = []
for i in range(len(combined_data)-30):
    if combined_data.iloc[i+30][5] and combined_data.iloc[i-30][5] \
        and i +15< len(combined_data) and i-30>=0:
            gain = combined_data.iloc[i][5] - combined_data.iloc[i-30][5]
            print(combined_data.iloc[i])
            date_1 = combined_data.iloc[i][0]
           
            dated_gain = [date_1, gain]
            print(dated_gain)
            monthly_gain.append(dated_gain)
            i += 1
# Puts the gain into a dataframe
monthly_gain = pd.DataFrame(monthly_gain,columns=['date','monthly weight gain'])
print(monthly_gain)
# Merges the dataframe
combined_data = combined_data.merge(monthly_gain, how='outer')


print(combined_data)

combined_data.to_csv("output.csv")

Finally I had to make a bunch of plots out of the data which was stored to the csv. This was done in a third script which imported the formatted data from the second script.

import matplotlib
import matplotlib.pyplot as plt
import pandas as pd

# Imports the csv, will need to change file path to match the location.
file_path = 'E:/My Documents/Python files/export/apple_health_data/output.csv'

# Puts the imported data into a dataframe
combined_data = pd.read_csv(file_path, index_col=0)
combined_data['date'] = pd.to_datetime(combined_data['date'])

# Start of a plot of date versus monthly average steps, first line defines
# which values are plotted against each other.

plt.plot_date(combined_data['date'], combined_data['monthly average steps'])

# Rotates the labels since dates get long otherwise.
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)

# Labels the y axis, x axis not named since dates seem obvious.
plt.ylabel('Average Steps in 30 Day Period')

# The plot is made larger then saved.
fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('monthly_average_steps.png', dpi=100)

# Marks the end of one plot. 
plt.figure()

# Start of second plot.
plt.plot_date(combined_data['date'], combined_data['monthly average weight'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)

plt.ylabel('Average Weight in 30 Day Period')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('monthly_average_weight.png', dpi=100)

plt.figure()

# Start of third plot, this one is a bit different since it is a scatterplot
# rather than a date plot.
plt.scatter(combined_data['monthly average steps'], combined_data['weight'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)
plt.xlabel('Average Steps in 30 Day Period')
plt.ylabel('weight')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('monthly_average_steps_versus_weight.png', dpi=100)

plt.figure()

# Start of fourth plot
plt.scatter(combined_data['monthly average steps'], combined_data['monthly weight gain'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)
plt.xlabel('Average Steps in 30 Day Period')
plt.ylabel('Weight Gain Compared to Previous 30 Day Period')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('monthly_average_steps_versus_weight_gain.png', dpi=100)

plt.figure()

# Start of fifth plot
plt.scatter(combined_data['monthly average steps'], combined_data['monthly average weight'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)
plt.xlabel('Average Steps in 30 Day Period')
plt.ylabel('Monthly Average Weight')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('monthly_average_steps_versus_monthly_average_weight.png', dpi=100)

plt.figure()

# Start of sixth plot
plt.plot_date(combined_data['date'], combined_data['steps'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)

plt.ylabel('daily steps')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('steps.png', dpi=100)

plt.figure()

# Start of seventh plot
plt.plot_date(combined_data['date'], combined_data['weight'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)

plt.ylabel('daily weight')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('weight.png', dpi=100)

plt.figure()

# Start of eight plot
plt.scatter(combined_data['steps'], combined_data['weight'])
plt.setp(plt.gca().xaxis.get_majorticklabels(),rotation=90,)
plt.xlabel('Steps')
plt.ylabel('Weight')

fig = matplotlib.pyplot.gcf()
fig.set_size_inches(12, 8)
fig.savefig('steps_versus_weight.png', dpi=100)

plt.figure()

For years I have been considering getting an underwater case for my camera and starting to make more underwater iNaturalist observations. I made a few freediving with a GoPro but that camera wasn't really designed for macro shots. Getting a waterproof housing for my real camera terrifies me as I have heard too many stories of wrecked.

So hearing that someone I know bought an underwater drone definitely caught my attention.

The Power Vision PowerRay they bought can go a hundred feet down and stay down two hours. This certainly does seem a step up from freediving for a few seconds to take a few rapid photos. Compared to real scuba diving with a dive housing it isn't quite there, but it is certainly safer and less trouble and should be cheaper. Launching a ROV from a kayak seems much more practical than SCUBA diving from the same small vessels.

The Open ROV Trident however can go to 100 meters. Now we are starting to talk, that depth can certainly be done with SCUBA, but past 50 meters SCUBA starts getting pretty specialized. That means there must be some opportunity to find interesting creatures which haven't been all that well documented.

Then comes the Titan ROV which should start shipping any day. This drone reports a 150 meter maximum depth and 4 hour battery life. While it isn't exactly cheap, at around $3,000 this is certainly the lowest cost way to depths from 100-150 meters. It must be possible to track down creatures at that depth not only not posted to iNaturalist, but almost unstudied. Also, at the rate these drones seem to be improving within five years the cost of entry should be even more reasonable.

Too bad the last thing I need is another expensive hobby.

Globe Flash V3

I have been using the globe flash macro diffuser I made for the past six months. For close in macro images it really proved amazing. However it had a couple annoyances which I finally decided to address:

•  It had a paper towel on top of it and wouldn't work without it. That I stuck through six months of a camera that ugly is a good sign of how good of images it produced but it did feel silly. 
• It was useless beyond about two feet. At these distances the images got really hazy. This is an issue it actually shared with the Sony Twin Flash, and Sigma Ring Flash. Light from the flash would interfere with the image since they don't work with a lens hood.
• It was incompatible with Raynox macro filters. With these on a macro lens you can get about the best magnification which is usable in the field. 
• Reflections look a bit funny on some reflective subjects because you can clearly see circular shape of the lens. 
• It does not work with a lens cap.

Two tweaks fixed both issues. First, the size of the globe was increased to 12 inches. This means it is large enough to completely cover the flash from the perspective of the subject. Second was connecting the globe to a lens hood which blocked the light which was making images hazy.

The image quality for 1X macro is about the same as the previous version but without the previous issues.

 

 

To build it, I used the following materials:

• Twelve inch polyethylene globe lamp shade 
• 3.5 inch white acrylic disc. 
• Epoxy
• Screw in lens hood

The steps were pretty simple, although with the tools I had rather time consuming. Simply cut a hole in the acrylic disc for the lens to look through (I used a Dremmel), epoxy the disc to the lens hood, then epoxy the disc to the globe. Then epoxy the globe to the acrylic disc and cut the globe to shape(I used a hack saw). Since polyethylene is notoriously difficult to bond, I briefly. put it in a flame before making the connection.

Thus far this setup has been quite effective. I am a bit worried about the epoxy bond, it seems less sturdy than it was to the previous acrylic globe, but the last one survived six months of abuse so I expect this one will do alright. 

Globe Flash V2

Before getting into technical details, let me show three photos. All three are with the same camera and lens but were taken using different flashes:

The first uses a Sigma Ring Flash:

The second uses a Sony Twin Flash:

 

The third uses my latest DIY globe flash:

I chose ants because they move so fast they are really difficult to photograph without a flash you typically end up with a blurry mess. All three flashes do a reasonable job of stopping motion and at least making clear it is an ant. There is a real quality improvement from better diffusion though.

The Sigma flash always had a real problem with too little diffusion. Odd hot spots and dark spots really retract from the quality. It calls itself a ring flash, but due to lack of diffusion it is more of a twin flash with two fixed light sources.

The Sony flash was a solid flash, for $750 it better be! Still, even it ended up with some hot spots. I always wanted more diffusion when I worked with it.

The globe flash doesn't quite eliminate all specular reflections, some images still have hot spots when using it, but they are a much reduced level than the other two flashes. Typically they show up as a bright area rather than a totally over-exposed spot. Compared to any previous flash I have used this is great performance.

It isn't entirely a fair comparison since the photos are ordered chronologically and were taken a few months apart so I learned as I went. Also, the dedicated macro flashes give more control over shadows. Still, the DIY globe flash I am using is almost half the price of the Sigma flash, and and a quarter the price of the Sony flash.

The previous globe flash I used almost had this level of performance, but I found two somewhat annoying issues. Too little light got in meaning the flash took a long time to recharge and it had oddly shaped reflections on shiny surfaces. You can see the diffuser shaped reflections on the ants in the photo below:

So I upgraded from a 6" globe to an 8" globe. I chose 8" because that puts a subject at minimum focus distance right at the center of the light source. I obsessively keep subjects at minimum focus distance so I can add scale bars.

Going to an 8" globe created problems attaching the globe to the camera. The 8" globes do not come with smaller than 4" holes. That is larger than any cheap adapter I saw. So I had to purchase an acrylic disc to use to connect the step up ring to the globe.

The list of materials and tools for the project were:

•  8" acrylic globe shade. 
• 4" white acrylic disc 1/8" thick
• 62mm to 77mm step up ring
• Epoxy
• Dremel
• Paper Towel
• Packing Tape

First I used the dremel cut a hole in the acrylic disc matching the opening on the lens, then I cut off the flange attached to the globe and epoxied the step up ring, acrylic disc, and globe together. Once the epoxy dried I used the dremel to cut the globe into the shape I thought would work best:

As you can see the new flash diffuser is significantly larger than the old one. Just like last time, too little light makes it from the flash to the diffuser. This was easily solved by putting a paper towel above the two. In an attempt to ruggedize it, I covered the paper with packing tape.

If I used a 12" globe I could probably avoid the paper towel, and I will probably ultimately find a better way to trap the light than the paper towel. It does seem effective though.

This has been very successful. Unlike the old version I can fire off photos rapidly using this version. Also, while reflections aren't entirely gone, the smaller black area on the central hole and larger globe make the reflections less obviously the shape of my diffuser.

While the diffuser produces very good images, I have found some downsides:

1. The shape makes it difficult to image insects in a tight space. 
2. You look like a crazy person when you walk down the street carrying this camera. Honestly this is a problem with all macro flashes though.
3. Diffusion could still be improved! This seems the best trade off of usefulness and image quality but I still often find myself wanting more diffusion. The only viable way I see to get more diffusion without making the system hopelessly bulky would be to use two small flashes, one on each side of the globe. 
4. The paper towel is a pain to take on and off. Maybe a little aluminum foil hat for the flash? Maybe a white cloth with velcro?

This could be adapted to just about any macro lens. The only big change would be to use a different size step up ring to connect to whatever lens you want. I suspect it might even work on a superzoom camera with the on camera flash, but I have not tried.