Introduction to the Open Data Portal and Tyler Tech's Data & Insights Tool
1) Getting Started in Data & Insights (Socrata) and the Maryland Open Data Site/How to Filter Data
This video will introduce you to opendata.maryland.gov and show you how to navigate the portal (once called Socrata by Tyler Technologies, Inc., this data management tool is now called Data & Insights).
Step-by-Step Video Guide
Finding a dataset (2:25)
- From the main page, click the words Data Catalog in the upper left-hand portion of the screen to access the full list of available datasets.
- If you are looking for a particular keyword, type it into the search bar at the top of the page. If you are interested in a particular category of data, you can filter out the unnecessary datasets by clicking one of the categories of interest on the left side of the page.
Navigating the dataset (3:25)
- After selecting a dataset, Data & Insights will first direct you to the metadata - a thorough description of what is in the dataset. Read over this section to be sure that this dataset is both up-to-date and relevant to the topic you are interested in.
- Selecting View Data in the upper right-hand corner will bring you to the entire dataset. The dataset's rows indicate each observation/recorded event, and its columns describe something about the event. Each intersection of rows and columns has a cell with a value, or unit of information.
- Some datasets contain tens of thousands of rows of information when you may only be interested in observations that fit a certain criteria. To pare down the data so that it only contains the observations of interest, select Filter on the right-hand side. Adding a filter will allow you to exclude rows that do not have a particular value in a column. Select a column you would like to filter by, then select how you would like the filter to function. You can filter data by the following:
- is - the dataset will return only rows with values in each column matching the exact value you typed in
- is not - the dataset will return only rows with values in each column excluding the exact value you typed in
- is less than - the dataset will return only rows with numeric values less than the value you typed in
- is at most - the dataset will return only rows with numeric values that do not exceed the value you typed in
- is greater than - the dataset will return only rows with numeric values greater than the value you typed in
- is at least - the dataset will return only rows with numeric values that are not lower than the value you typed in
- is between - the dataset will return only rows with numeric values between the two you typed in
- is blank - the dataset will return only rows where there is no data entered in that column
Saving your project (5:30)
- As soon as you begin editing and applying filters to your dataset, Data & Insights gives you the option of saving your project if you are logged into your account. You will be able to return to this data at a later time without having to reapply all of your filters. Click the Save As... button that has appeared in the yellow box at the top of the page.
- Give your project a name.
- You are now free to close your browser. When you are ready to access this project again, log into your account.
- This will land you on the My Profile page, which you can always access by clicking the user profile icon in the upper right corner. Find your project's name in the list labelled My Assets to pick up where you left off.
2) Visualizing the Data
This tutorial will teach you how to use Data & Insights' visualization feature to create your own graphs and charts from the datasets uploaded to the site.
You can find the Step-by-Step Video Guide below to follow along with this video, or you can scroll further down to the In-Depth Visualization Guide for explanations of the tools in the Visualization area.
Step-by-Step Video Guide
Creating A Graph (0:00)
- Find a dataset you are interested in making a graph for. Select the dataset, then select View Data from the metadata page.
- From the View Data page, click on Visualize on the upper right-hand side, followed by Launch New Visualization, opening up a new page for the Visualization tool.
- Select which type of graph or chart you would like to choose for your dataset. Your options are:
- Bar charts
- Column charts
- Pie charts
- Timeline charts
- Histograms
- Combo charts (which combine both column charts and timelines)
- Scatter plots
- Maps (if the data has been georeferenced)
- Calendars
Choosing Your Axes (1:31)
- Select what data will be represented along the X-axis by choosing a column from the dataset under the Dimension section on the left.
- Select what data will be represented in the Y-axis by choosing a column from the dataset under the Measure section on the left.
Changing The Way Your Visualization Looks (2:46)
- The Presentation and Axis tabs on the far left will allow us to create a more aesthetically pleasing visual. Beginning with the Axis tab, you can adjust the scale of the data to avoid any wasted space in your graph. For example, if your measures (the data in the Y-axis) tend to be very large values and all of your data points tend to be toward the top of the graph, you can select Scale to min and max values or Custom under the Scale tab to remove any wasted space.
- The Presentation tab allows you to edit the graph's title, axis names, and colors.
Publishing Your Visualization (3:40)
- Click Save Draft in the bottom right hand corner.
- Enter a name for your Visualization and click Save. You will be able to edit this project again before publishing it by accessing it under your My Profile screen.
- If you are ready to put this visualization onto the Open Data portal, click Publish... in the upper right.
3) In-Depth Visualization Guide
Once you have launched the Visualization tool, you'll first be prompted to select what type of chart you would like to make using one of the 9 options at the top of the screen:
- Bar chart
- Column chart
- Pie chart
- Timeline
- Histogram
- Combo chart (both a bar chart and a timeline chart)
- Scatter plot
- Map (which you can use if the dataset has a column with correctly-input addresses or coordinates
- Calendar
Once you have decided on what type of chart you would like to use, notice the tabs on the far left:
- Data
- Axis
- Presentation
- Legends and Flyouts
Data
Under this tab, you will decide what information will appear in your chart. Depending on which type of graph you select, the Data tab will display a varying number of sub-tabs for you to select from. In the example picture on the left, our available sub-tabs are:
- Data Selection, where you will choose what dimension, or field that orders/categorizes your data (x-axis), and measure (y-axis) your graph will have
- Group Dimension Values, where you can decide how you would like your data in the dimension area to be broken up (notice in the previous screenshot, all the data was recorded on a single blue line, while in the screenshot to the left, the data are recorded on separate green and blue lines)
- Timeline Display Options, where, if you have chosen a graph that uses a unit of time, you can decide what intervals are measured along the x-axis
Axis
Under this tab, you will decide how markers appear on your axes. In the example picture on the left, you will see the following sub-tabs:
- Scale, where you can decide if what data points your y-axis contains (you can have your y-axis begin at a number other than 0, and end at a number higher or lower than the largest observed value, if you so choose)
- Reference Lines, where you can add a reference line for your viewer in case there is a particular value you would like to draw attention to (in the example picture on the left, notice the dotted gray line at the value 60)
Presentation
Under this tab, you will decide on details regarding your graph's labels and other aesthetic traits:
- General, where you will title your graph, add a description, and whether or not the data table the graph is based on will appear
- Color, where you can choose the color scheme of your graph
- Labels, where you can decide if you would like individual data points to appear on the graph and how you would like your axes' labels to be oriented
- Axis Titles, where you will label each of your axes
Legends and Flyouts
Under this tab, you can decide what happens when a viewer moves their mouse over your graph:
- Flyout Unit Label allows you to edit what unit of measurement appears in the flyout
- Annotations allows you to mark a particular point of interest on your graph
- Legends allows you to edit qualities about the graph's legend
Filters
Like the dataset, the visualization tool also allows you to filter out data you are not interested in viewing. Select one of the following to keep data that fits the proper criteria, or to hide data that does not.
Click Add Filter, then select a column you would like to filter by, followed by an operator:
- IS - the graph will only show data points with the value you choose
- IS NOT - the graph will exclude data points with the value you choose
- STARTS WITH - the graph will only include data
- CONTAINS
- DOES NOT CONTAIN
The Final Product
Roll your cursor over the graph on the left to view some of the functionality in real time.
4) Uploading Your Own Data
This video will teach you how to import and publish your own dataset to the opendata.maryland.gov site.
Step-by-Step Video Guide
Creating a New Project (0:00)
- In the upper left hand corner, click the Create drop down menu, and select Dataset.
- Start your new project by giving it a name. This will be the title your project is given when you publish it online, and how you can identify your project from your profile screen while you are working on it.
Uploading a Dataset (0:33)
Click Add Data on the left. On the next screen, click Browse to search for a file on your computer to upload into Data & Insights. The following are some of the file types Data & Insights supports:
- .CSV (comma-separated values)
- .XLS (Microsoft Excel)
- .XLSX (Microsoft Excel)
- .TSV (tab-separated values)
Review and Maintenance of the Dataset (1:04)
After you've uploaded the dataset, Data & Insights will automatically take you to the Review & Configure Data screen. Look over your data to be sure it was properly translated from your file to Data & Insights. Data & Insights can tell what types of variables are in your dataset's columns; make sure they are correctly labelled by checking the drop down menu under the variable's name.
In Review and Configure Data, you can make a number of edits to clean up your dataset before publishing it:
- Changing the name of the column
- Providing a description of that column
- Moving the column to the left or right on the dataset in order to prioritize certain information
- Hiding or deleting columns you would not like to publish
- Editing individual cells in a column
Metadata (3:16)
Metadata--literally, data about the data--provides descriptions of a dataset and gives context for why you've included certain information, how frequently you expect to update your data, and any other descriptors that may be relevant to the upkeep of this dataset.
Click on Edit Dataset Metadata and Data & Insights will provide a list of all the metadata fields that need to be filled in before publishing the dataset.
- The Title of your dataset
- A description of your dataset
- Row labels for describing what each row in your dataset represents. For example, a dataset might be compromised of rows listing particular events, or, like in the example above, a list of legislation
- Tags. These will help users find your dataset using the search function
- Licensing (if applicable). If you found your dataset from another site, you'll want to list it here
- Attachments. You can attach other documents to your dataset, such as PDFs, that may supplement the data
- Time Period. Select how frequently your dataset is updated and what time period your dataset covers
Edit Column Metadata (4:55)
Similar to the Review & Configure Data section, the Edit Column Metadata section lets you describe what each of your columns mean in greater detail. You can also configure what order your columns will appear in when published.
Publishing (5:30)
Click Publish Dataset... in the upper right hand corner to submit your dataset for review.
5) Georeferencing Your Data
Once you know how to upload your data, this video will teach you how to use Data & Insights' built-in georeferencing tool to create a map of points relevant to your dataset.
Step-by-Step Video Guide
Format Your Dataset Properly
Before uploading your dataset, you'll have to have the data arranged in a way Data & Insights will recognize as physical locations.
For example, if you have a Microsoft Excel file with a list of all the locations of parking violations in Baltimore City between 2000 and 2005, you will have to be sure your data has either one of the following:
- The coordinates of the event (latitude and longitude)
- The full street address of the event (street number and name, city, state, and Zip code)
The easiest way to have your data ready to upload is to have each aspect in its own column. That is, latitude and longitude should have their own two separate columns, or the street number and name, the city, the state, and the Zip code should have their own four columns.
Upload Your Dataset (0:36)
- Like any regular dataset, click the Create drop down menu in the upper left, and select Dataset from the drop-down list.
- Give your new project a name.
- Click Add Data on the left.
- On the next screen, click Browse to search for the file on your computer to upload into Data & Insights.
Georeferencing with Latitude and Longitude (0:52)
- From the Review & Configure Data page, click Add Georeference on the left.
- Select Lat/Long as your source column type.
- The next two menus will ask which column from your dataset you would like to use for the latitude, and which you would like to use for your longitude. Choose the appropriate column for each. Data & Insights will recognize the first cell in each column as the title of that column, and those titles will show up accordingly in the menus.
- Create a name for the new georeferenced column. The Add Georeference tool is effectively allowing Data & Insights to create its own column with values it recognizes as points on a map.
- Click Add Column on the right, followed by Done at the bottom.
Georeferencing with the Street Address (2:09)
- From the Review & Configure Data page, click Add Georeference on the left.
- Select Address (separated) as your source column type.
- The next four menus will ask which column from your dataset you would like to use for the street number and name, which to use for the city, which to use for the state, and which to use for the Zip code. Choose the appropriate column for each. Data & Insights will recognize the first cell in each column as the title of that column, and those titles will show up accordingly in the menus.
- Create a name for the new georeferenced column. The Add Georeference tool is effectively allowing Data & Insights to create its own column with values it recognizes as points on a map.
- Click Add Column on the right, followed by Done at the bottom.
6) More Learning Resources
If you are interested in learning more about the various uses and features of Data & Insights and the Open Data Portal, the tools at support.socrata.com can offer you an even more in-depth look at Data & Insights' functionality. Below, you will find that site's articles on each of the topics covered in this asset: