Assignment2 Weather-Dates on X Axis - Submitted

 

 

Assignment 2

Before working on this assignment please read these instructions fully. In the submission area, you will notice that you can click the link to Preview the Grading for each step of the assignment. This is the criteria that will be used for peer grading. Please familiarize yourself with the criteria before beginning the assignment.

An NOAA dataset has been stored in the file data/C2A2_data/BinnedCsvs_d400/fb441e62df2d58994928907a91895ec62c2c42e6cd075c2700843b89.csv. This is the dataset to use for this assignment. Note: The data for this assignment comes from a subset of The National Centers for Environmental Information (NCEI) Daily Global Historical Climatology Network (GHCN-Daily). The GHCN-Daily is comprised of daily climate records from thousands of land surface stations across the globe.

Each row in the assignment datafile corresponds to a single observation.

The following variables are provided to you:

  • id : station identification code
  • date : date in YYYY-MM-DD format (e.g. 2012-01-24 = January 24, 2012)
  • element : indicator of element type
    • TMAX : Maximum temperature (tenths of degrees C)
    • TMIN : Minimum temperature (tenths of degrees C)
  • value : data value for element (tenths of degrees C)

For this assignment, you must:

  1. Read the documentation and familiarize yourself with the dataset, then write some python code which returns a line graph of the record high and record low temperatures by day of the year over the period 2005-2014. The area between the record high and record low temperatures for each day should be shaded.
  2. Overlay a scatter of the 2015 data for any points (highs and lows) for which the ten year record (2005-2014) record high or record low was broken in 2015.
  3. Watch out for leap days (i.e. February 29th), it is reasonable to remove these points from the dataset for the purpose of this visualization.
  4. Make the visual nice! Leverage principles from the first module in this course when developing your solution. Consider issues such as legends, labels, and chart junk.

The data you have been given is near Ann Arbor, Michigan, United States, and the stations the data comes from are shown on the map below.

In [1]:
#%matplotlib notebook

import matplotlib.pyplot as plt
import matplotlib as mpl
#mpl.get_backend()
import matplotlib.dates as mdates

import pandas as pd
import numpy as np
import datetime as dt

data_file = "fb441e62df2d58994928907a91895ec62c2c42e6cd075c2700843b89.csv"
raw_data = pd.read_csv(data_file)
raw_data.to_csv("raw_data.csv")

#print(raw_data.describe())
#print(raw_data.head())


# take the inverse of the mask to delete all leap days
raw_data = raw_data[~raw_data['Date'].str.contains("02-29")]

# separate 2005-2014 data and 2015 data
data_5_14 = raw_data[raw_data['Date'] < '2015-01-01' ]
data_2015 = raw_data[raw_data['Date'] > '2014-12-31']
#print(data_2015.head())

# separate TMAX data from TMIN data
max = data_5_14[data_5_14['Element']=='TMAX']
#max.to_csv('max.csv')
min = data_5_14[data_5_14['Element']=='TMIN']

# find max and min values for each date from among the
# measuring stations
max = max.groupby(['Date']).max()
min = min.groupby(['Date']).min()
data_2015_max = data_2015.groupby(['Date']).max()
data_2015_min = data_2015.groupby(['Date']).min()
data_2015 = data_2015_max.append(data_2015_min)
#print('new data_2015: \n', data_2015.describe())
#max.reset_index(inplace=True)
#max.to_csv('max_after_groupby.csv')

# drop unnecessary columns
max = max.drop(columns=['ID','Element'])
min = min.drop(columns=['ID','Element'])
data_2015.drop(columns=['ID','Element'])

# get the absolute highs and lows
max_high_temp = max.max()[0]
min_low_temp = min.min()[0]
#print('max_high_temp: ', max_high_temp)
#print('min_low_temp: ', min_low_temp)

# remove 20115 data between max_high_temp and min_low_temp
data_2015 = data_2015[(data_2015['Data_Value'] > max_high_temp) 
                      | (data_2015['Data_Value'] < min_low_temp)]

#print("for scatter plot: \n",data_2015)


# prepare the dataframes for plotting
max = max.reset_index()
min = min.reset_index()
data_2015 = data_2015.reset_index()

#print('max: \n', max.describe())
#print('min: \n', min.describe())
#print('data_2015: \n', data_2015.describe())

max_values = max['Data_Value'].to_numpy()
max_values = max_values / 10
max_dates = max['Date'].to_numpy()

# convert np array of string dates to list of datetime.dates
max_dates = \
    [dt.datetime.strptime(d,'%Y-%m-%d').date() for d in max_dates]

#print(max_dates)
#print(type(max_dates))
min_values = min['Data_Value'].to_numpy()
min_values = min_values / 10
min_dates = min['Date'].to_numpy()

# convert np array of string dates to list of datetime.dates
min_dates = \
    [dt.datetime.strptime(d,'%Y-%m-%d').date() for d in min_dates]

print("888888888", max_dates[0], max_dates[-1])
data_2015_values = data_2015['Data_Value'].to_numpy()
data_2015_values = data_2015_values / 10
data_2015_dates = data_2015['Date'].to_numpy()

print("******", type(data_2015_dates[0]))


#
# First Axes.....
#

first_axes = plt.gca()
x = first_axes.xaxis

# rotate the tick labels for the x axis
for item in x.get_ticklabels():
    item.set_rotation(-45)

first_axes.set_xlabel('\n Daily Highs and Lows 2004 through 2014')
first_axes.set_ylabel('Temperature (C)')
first_axes.set_title('2015 Temperatures that Exceed Maximum Highs and Lows from \
2004 through 2014 \n')

#first_axes.set_xlim(left=min_dates.min(), right=min_dates.max())

first_axes.xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y'))
first_axes.xaxis.set_major_locator(mdates.AutoDateLocator())
first_axes.set_ylim(top=70, bottom=-40)

plt.subplots_adjust(bottom=0.25, top=0.7)
f_axes_max_line, = first_axes.plot(max_dates, 
                                  max_values, 
                                  'r', 
                                  label='Maximum', 
                                  alpha=0.5, 
                                  linewidth=0.5)


f_axes_min_line, = first_axes.plot(min_dates, 
                                  min_values, 
                                  'blue', 
                                  label='Minimum', 
                                  alpha=0.5, 
                                  linewidth=0.5)

first_axes.fill_between(max_dates, max_values, min_values, facecolor='blue', alpha=0.25)

plt.subplots_adjust(bottom=0.25, top=0.55)
#
# build second axes
#


second_axes = first_axes.twiny() 
second_axes.set_xlabel("2015 Temps Exceeding 2004 to 2014 Highs and Lows\n")

second_axes.xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y'))
second_axes.xaxis.set_major_locator(mdates.DayLocator())

left = mpl.dates.datestr2num('2015-02-01')
right = mpl.dates.datestr2num('2015-02-28')
#left = mpl.dates.datestr2num('2015-01-01')
#right = mpl.dates.datestr2num('2015-12-31')
second_axes.set_xlim(left=left, right=right)
# rotate the tick labels for the upper x axis

# rotate the axis labels
top_x = second_axes.xaxis
for item in top_x.get_ticklabels():
    item.set_rotation(315)

# reduce the number of xticks to every other xtick in February
xticks = second_axes.get_xticks()
xticks = xticks[0::2]
second_axes.set_xticks(xticks)

#print(data_2015_dates)
#print(type(data_2015_dates[0]))
data_2015_dates_num = mpl.dates.datestr2num(data_2015_dates)
print(data_2015_dates[0])
print(type(data_2015_dates[0]))

'''
second_axes.annotate(s='some string', 
                     xy=(data_2015_dates[0],data_2015_values[0]), 
                     xytext=(-10.0,0.0),
                     textcoords='offset points',
                     ha='right')
'''
s_axis_scatter = second_axes.scatter(
    data_2015_dates_num, 
    data_2015_values) 

plt.legend([f_axes_max_line,f_axes_min_line, s_axis_scatter],
           ("2004/14 Max Temps",'2004/14 Min Temps','2015 Extremes'),
           loc='upper left',
           fontsize='small')

fig = plt.gcf()
fig.set_size_inches(10, 10)
#plt.gcf().autofmt_xdate()

#host.yaxis.get_label().set_color(p1.get_color())
#leg.texts[0].set_color(p1.get_color())

#par.yaxis.get_label().set_color(p2.get_color())
#leg.texts[1].set_color(p2.get_color())
 
888888888 2005-01-01 2014-12-31
****** <class 'str'>
2015-02-20
<class 'str'>
 
In [ ]:
 
In [ ]: