randyzwitch - Articles

Building pyarrow with CUDA support

Fri, 03 Apr 2020 00:00:00 +0000

The other day I was looking to read an Arrow buffer on GPU using Python, but as far as I could tell, none of the provided pyarrow packages on conda or pip are built with CUDA support. Like many of the packages in the compiled-C-wrapped-by-Python ecosystem, Apache Arrow is thoroughly documented, but the number of permutations of how you could choose to build pyarrow with CUDA support quickly becomes overwhelming.

In this post, I’ll show how to build pyarrow with CUDA support on Ubuntu using Docker and virtualenv. These directions are approximately the same as the official Apache Arrow docs, just that I explain them step-by-step and show only the single build toolchain I used.

Step 1: Docker with GPU support

Even though I use Ubuntu 18.04 LTS on a workstation with an NVIDIA GPU, whenever I undertake a project like this, I like to use a Docker container to keep everything isolated. The last thing you want to do is to debug environment errors, changing dependencies for one project and breaking something else. Thankfully, NVIDIA Docker developer images are available via DockerHub:

docker run -it --gpus=all --rm nvidia/cuda:10.1-devel-ubuntu18.04 bash

Here, the -it flag puts us inside the container at a bash prompt, --gpus=all allows the Docker container to access my workstation’s GPUs and --rm deletes the container after we’re done to save space.

Step 2: Setting up the Ubuntu Docker container

When you pull Docker containers from DockerHub, frequently they are bare-bones in terms of libraries included, and usually can also be updated. For building pyarrow, it’s useful to install the following:

apt update && apt upgrade -y

apt install git \
wget \
libssl-dev \
autoconf \
flex \
bison \
llvm-7 \
clang \
cmake \
python3-pip \
libjemalloc-dev \
libboost-dev \
libboost-filesystem-dev \
libboost-system-dev \
libboost-regex-dev  \
python3-dev -y

In a later step, we’ll use the Arrow third-party dependency script to ensure all needed dependencies are present, but these are a good start.

Step 3: Cloning Apache Arrow from GitHub

Cloning Arrow from GitHub is pretty straightforward. The git checkout apache-arrow-0.15.0 line is optional; I needed version 0.15.0 for the project I was exploring, but if you want to build from the master branch of Arrow, you can omit that line.

git clone https://github.com/apache/arrow.git /repos/arrow
cd /repos/arrow
git submodule init && git submodule update
git checkout apache-arrow-0.15.0
export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"
export ARROW_TEST_DATA="${PWD}/testing/data"

Step 4: Installing remaining Apache Arrow dependencies

As mentioned in Step 2, some of the dependencies for building Arrow are system-level and can be installed via apt. To ensure that we have all the remaining third-party dependencies, we can use the provided script in the Arrow repository:

pip3 install virtualenv
virtualenv pyarrow
source ./pyarrow/bin/activate
pip install six numpy pandas cython pytest hypothesis
mkdir dist
export ARROW_HOME=$(pwd)/dist
export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH

cd cpp
./thirdparty/download_dependencies.sh $HOME/arrow-thirdparty

The script downloads all of the necessary libraries as well as sets environment variables that are picked up later, which is amazingly helpful.

Step 5: Building Apache Arrow C++ library

pyarrow links to the Arrow C++ bindings, so it needs to be present before we can build the pyarrow wheel:

mkdir build && cd build

cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DCMAKE_INSTALL_LIBDIR=lib \
-DARROW_FLIGHT=ON \
-DARROW_GANDIVA=ON \
-DARROW_ORC=ON \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_WITH_BROTLI=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=ON \
-DARROW_CUDA=ON \
..

make -j
make install

This is a pretty standard workflow for building a C or C++ library. We create a build directory, call cmake from inside of that directory to set up the options we want to use, then use make and then make install to compile and install the library, respectively. I chose all of the -DARROW_* options above just as a copy/paste from the Arrow documentation; Arrow doesn’t take long to build using these options, but it’s possibly the case that only -DARROW_PYTHON=ON and -DARROW_CUDA=ON are truly necessary to build pyarrow.

Step 6: Building pyarrow wheel

With the Apache Arrow C++ bindings built, we can now build the Python wheel:

cd /repos/arrow/python
export PYARROW_WITH_PARQUET=1
export PYARROW_WITH_CUDA=1
python setup.py build_ext --build-type=release --bundle-arrow-cpp bdist_wheel

As cmake and make run, you’ll eventually see the following in the build logs, which shows that we’re getting the behavior we want:

cmake --build . --config release --
[  5%] Compiling Cython CXX source for _cuda...
[  5%] Built target _cuda_pyx
Scanning dependencies of target _cuda
[ 11%] Building CXX object CMakeFiles/_cuda.dir/_cuda.cpp.o
[ 16%] Linking CXX shared module release/_cuda.cpython-36m-x86_64-linux-gnu.so
[ 16%] Built target _cuda

When the process finishes, the final wheel will be in the /repos/arrow/python/dist directory.

Step 7 (optional): Validate build

If you want to validate that your pyarrow wheel has CUDA installed, you can run the following:

(pyarrow) root@9260485caca3:/repos/arrow/python/dist# pip install pyarrow-0.15.1.dev0+g40d468e16.d20200402-cp36-cp36m-linux_x86_64.whl
Processing ./pyarrow-0.15.1.dev0+g40d468e16.d20200402-cp36-cp36m-linux_x86_64.whl
Requirement already satisfied: six>=1.0.0 in /repos/arrow/pyarrow/lib/python3.6/site-packages (from pyarrow==0.15.1.dev0+g40d468e16.d20200402) (1.14.0)
Requirement already satisfied: numpy>=1.14 in /repos/arrow/pyarrow/lib/python3.6/site-packages (from pyarrow==0.15.1.dev0+g40d468e16.d20200402) (1.18.2)
Installing collected packages: pyarrow
Successfully installed pyarrow-0.15.1.dev0+g40d468e16.d20200402
(pyarrow) root@9260485caca3:/repos/arrow/python/dist# python
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow import cuda
>>>

When the line from pyarrow import cuda runs without error, then we know that our pyarrow build with CUDA was successful.

A Beginner's Look at BenchmarkTools.jl

Mon, 16 Dec 2019 00:00:00 +0000

For the number of years I’ve been programming using Julia, I’ve never really been concerned with performance. Which is to say, I’ve appreciated that other people are interested in performance and have proven that Julia can be as fast as any other performance language out there. But I’ve never been one to pour over the Performance Tips section of the Julia manual trying to squeeze every last bit of performance.

But now that I’ve released OmniSci.jl, and as a company one of our major selling points is accelerated analytics, I figured it was time to stop assuming I wrote decent-ish code and really pay attention to performance. This post highlights my experience as a beginner, and hopefully will show how others can get started in learning to optimize their Julia code.

Read The Manuals!

As I mentioned above, I’ve written Julia for many years now, and in that time I’ve grown up with many of the tips in the performance tips section of the documentation. Things like “write type stable functions” and “avoid global variables” are things that I’ve internalized as good programming practices, as opposed to doing them just because they are performant. But with this long familiarity with the language comes laziness, and by not reading the BenchmarkTools.jl documentation, I started off benchmarking incorrectly. Consider this example:

julia> using Random, OmniSci, BenchmarkTools, Base.Threads

julia> #change defaults, since examples long-running
       BenchmarkTools.DEFAULT_PARAMETERS.seconds = 1000
1000

julia> BenchmarkTools.DEFAULT_PARAMETERS.samples = 5
5

julia> #generate test data
       gendata(x, T) = [rand(typemin(T):typemax(T)) for y in 1:x]
gendata (generic function with 1 method)

julia> int64_10x6 = gendata(10^6, Int64);

julia> #Test whether broadcasting more/less efficient than pre-allocating results array
       function preallocate(x)

           v = Vector{OmniSci.TStringValue}(undef, length(x))

           for idx in 1:length(x)
               v[idx] = OmniSci.TStringValue(x[idx])
           end

           return v
       end
preallocate (generic function with 1 method)

julia> @benchmark v61 = OmniSci.TStringValue.(int64_10x6)
BenchmarkTools.Trial:
  memory estimate:  297.55 MiB
  allocs estimate:  6000005
  --------------
  minimum time:     750.146 ms (0.00% GC)
  median time:      1.014 s (29.38% GC)
  mean time:        1.151 s (28.38% GC)
  maximum time:     1.794 s (43.06% GC)
  --------------
  samples:          5
  evals/sample:     1

julia> @benchmark v62 = preallocate(int64_10x6)
BenchmarkTools.Trial:
  memory estimate:  297.55 MiB
  allocs estimate:  6000002
  --------------
  minimum time:     753.877 ms (0.00% GC)
  median time:      1.021 s (28.30% GC)
  mean time:        1.158 s (28.10% GC)
  maximum time:     1.806 s (43.17% GC)
  --------------
  samples:          5
  evals/sample:     1

The benchmark above tests whether it’s worth pre-allocating the results array vs. using the more convenient dot broadcasting syntax. The idea here is that growing an array over and over can be inefficient when you know the result size at the outset. Yet, comparing the times above, for all statistics pre-allocating the array is slightly worse, even though we’re passing the compiler more knowledge up front. This didn’t sit well with me, so I consulted the BenchmarkTools.jl manual and found the following about variable interpolation:

A good rule of thumb is that external variables should be explicitly interpolated into the benchmark expression

Interpolating the int64_10x6 input array into the function takes it from being a global variable to a local, and sure enough, we see roughly a 6% improvement in the minimum time when we pre-allocate the array:

julia> @benchmark v61i = OmniSci.TStringValue.($int64_10x6)
BenchmarkTools.Trial:
  memory estimate:  297.55 MiB
  allocs estimate:  6000002
  --------------
  minimum time:     763.817 ms (0.00% GC)
  median time:      960.446 ms (24.02% GC)
  mean time:        1.178 s (28.68% GC)
  maximum time:     1.886 s (45.11% GC)
  --------------
  samples:          5
  evals/sample:     1

julia> @benchmark v62i = preallocate($int64_10x6)
BenchmarkTools.Trial:
  memory estimate:  297.55 MiB
  allocs estimate:  6000002
  --------------
  minimum time:     721.597 ms (0.00% GC)
  median time:      1.072 s (30.45% GC)
  mean time:        1.234 s (32.92% GC)
  maximum time:     1.769 s (44.51% GC)
  --------------
  samples:          5
  evals/sample:     1

Whether that 6% improvement will hold up over time or not, at least conceptually we’re no longer worse off for pre-allocating, which fits my mental model of how things should work.

Evaluate Your Benchmark Over the Range of Inputs You Care About

In the comparison above, I evaluate the benchmark over 10^6 observations. How did I choose 1 million as the “right” number of events to test, instead of just testing 1 or 10 events? My general goal for benchmarking this code is to speed up the methods of loading data into an OmniSciDB database. TStringValue is one of the internal methods as part of doing a row-wise table load, converting whatever data is present in an array or DataFrame from ::Type{T} into String (think iterating over a text file by line). Since users trying to accelerate their database operations are probably going to be using millions to billions of data points, I’m interested in understanding how the functions are performing at these volumes of data.

The other conscious decision I made was the environment to test on. I could test this on massive CPU- and GPU-enabled servers, but I’m testing this on my Dell XPS 15 laptop. Why? Because I’m actually interested in how things are performing under more real-world conditions for a realistic user. Testing the performance characteristics of a high-end server with tons of memory and cores would be fun, but I want to make sure any performance improvements are broadly applicable, instead of just because I am throwing more hardware at the problem.

Less important to me to control for was garbage collection, using a fresh session before each measurement or other “best case scenario” optimizations. I would expect my users to be more analytics and data science focused, so re-using the same session is going to be common. If the performance improvements aren’t completely obvious, I’m not going to incorporate them into the codebase.

Case Study: Speeding Up `TStringValue`

For my test, I evaluate the following as the methods to benchmark:

broadcasting: current library default
pre-allocating result array
pre-allocated result array with @inbounds macro
pre-allocated result array with threads
pre-allocated result array with threads and @inbounds

10x6 observations

For the first three on the left, these are comparisons of the single-threaded methods. You can see that pre-allocating the output array is marginally faster than broadcasting, and using the @inbounds macro is incrementally faster still, but neither method provides enough speedup to be worth implementing. The difference between the red and the blue bars represents a garbage collection occurring, but again, the three methods aren’t different enough to notice anything interesting.

For the multi-threaded tests, I’m using 6 threads (one per physical core), and we’re seeing roughly a 3x speedup. Like the single-threaded tests above, using @inbounds is only marginally faster, but not enough to widely implement for the cost of increased code complexity. Interestingly, doing these multi-threaded benchmarks didn’t trigger garbage collect at all across my five iterations; not sure if this is specific due to threading or not, but something to explore outside of this blog post.

10x7 observations

To see how these calculation methods might change at a larger scale, I bumped up the observations by an order of 10 and saw the following results:

Like at the 1 million data range, there isn’t much difference between the three single-threaded methods. All three of them are within a few percentage in either direction (all three methods triggered garbage collection in each of their five runs).

For the multi-threaded tests, an interesting performance scenario emerged. Like the 1 million point tests, it’s possible to get a run where garbage collection isn’t triggered, which leads to a large min/median difference in the multi-threaded tests. If you can avoid garbage collection, using six threads here gives nearly a 10x speedup, and at the median where both single-threaded and multi-threaded trigger garbage collection you still get a 2x speedup.

Parallelism > Compiler Hinting

In the case study above, I’ve demonstrated that for this problem, threading is the first way to pursue speeding up the OmniSci.jl load table methods. While pre-allocating the size of the output array and using the @inbounds did show some slight speedups, using threads to perform the calculations are where the largest improvements occurred. Incorporating the pre-allocation step naturally comes out from the way I wrote the threading methods, so I’ll incorporate that too. Disabling bounds-checking on arrays using @inbounds seems more dangerous than it is worth, even though none of these methods should ever get outside of their bounds.

Overall, I hope this post has demonstrated that you don’t have to fancy yourself a high-frequency trader or a bit-twiddler to find ways to improve your Julia code. The first step is reading the manuals for benchmarking, and then like any other pursuit, the only way to get a feeling for what works is to try things.

All of the code for this blog post can be found in this GitHub gist

RSiteCatalyst Version 1.4.16 Release Notes

Tue, 05 Nov 2019 00:00:00 +0000

It’s been a while since the last update, but RSiteCatalyst is still going strong! Thanks to Wen for submitting a fix/enhancement to enable the ability to use multiple columns from a Classification within the QueueDataWarehouse function. No other bug fixes were made, nor was any additional functionality added.

Version 1.4.16 of RSiteCatalyst was submitted to CRAN yesterday and should be available for download in the coming days.

Community Contributions

As I’ve mentioned in many a blog post before this one, I encourage all users of the software to continue reporting bugs via GitHub issues, and especially if you can provide a working code example. Even better, a fix via pull request will ensure that your bug will be addressed in a timely manner and for the benefit to others in the community.

Note: Please don’t email directly via the email in the RSiteCatalyst package, it will not be returned. Having a valid email contact in the package is a requirement to have a package listed on CRAN so they can contact the package author, it is not meant to imply I can/will provide endless, personalized support for free.

DataOps Summit: Streaming Real-time Telemetry With OmniSci and StreamSets

Thu, 31 Oct 2019 00:00:00 +0000

In this talk from the StreamSets DataOps 2019 conference, I provide an overview of the data pipeline for the OmniSci F1 Demo. Using StreamSets Data Collector in concert with Apache Kafka and OmniSciDB, you can create a full real-time data pipeline for telemetry data using only open-source components.

The talk outlines using the UDP listener for StreamSets to collect packets from the F1 2018 game, writing the packets to Kafka, reading from Kafka and using Groovy to parse the packets, and using the OmniSci JDBC driver to insert the data into one of nine OmniSciDB tables. With this workflow, you have a robust platform for accelerated analytics, using the power of GPUs for fast computation.

GitHub: https://github.com/omnisci/vehicle-telematics-analytics-demo

Speakerdeck: https://speakerdeck.com/omnisci/the-f1-demo-streaming-real-time-telemetry-using-apache-kafka-and-streamsets

ODSC webinar: End-to-End Data Science Without Leaving the GPU

Thu, 11 Jul 2019 00:00:00 +0000

In this webinar sponsored by the Open Data Science Conference (ODSC), I outline a brief history of GPU analytics and the problems that using GPU analytics solves relative to using other parallel computation methods such as Hadoop. I also demonstrate how OmniSci fits into the broader GPU-accelerated data science workflow, with examples provided using Python.

Check out the video, grab the Jupyter Notebook from the odscwebinar repo and get started with OmniSci and GPU-accelerated data science!

PyData NYC 2018: End-to-End Data Science Without Leaving the GPU

Fri, 01 Feb 2019 00:00:00 +0000

This talk is from October 2018, and so much has changed in the GOAI/RAPIDS ecosystem that it’s comical to see how much has changed! Regardless, the high-level concepts of how OmniSci works and the concepts behind GPU dataframes (then: pygdf, now: cudf) remain the same, so watching this talk still has value if you are interested in an end-to-end GPU workflow.

With the release of pymapd 0.7 a few days ago, getting started with GPU data science is just a matter of having an NVIDIA GPU and OmniSci Core (OSS) and a quick conda command to set up your environment:

conda install -c conda-forge -c nvidia -c rapidsai -c numba -c defaults pymapd cudf python=3.6

So check out the video, grab the Jupyter Notebook from the pydatanyc2018 GitHub repo and get started with GPU-accelerated data science!

Using pandas and pymapd for ETL into OmniSci

Tue, 16 Oct 2018 00:00:00 +0000

I’ve got PyData NYC 2018 in two days and rather finishing up my talk, I just realized that my source data has a silent corruption due to non-standard timestamps. Here’s how I fixed this using pandas and then uploaded the data to OmniSci.

Computers Are Dumb, MAKE THINGS EASIER FOR THEM!

Literally every data tool in the world can read the ISO-8601 timestamp format. Conversely, not every tool in the world can read Excel or whatever horrible other tool people use to generate the CSV files seen in the wild. While I should’ve been more diligent checking my data ingestion, I didn’t until I created a wonky report…

Let’s take a look at the format that tripped me up:

Month/Day/Year Hour:Minute:Second AM/PM feels very much like an Excel date format that you get when Excel is used as a display medium. Unfortunately, when you write CSV files like this, the next tool to read them has to understand 1) that these columns are timestamps and 2) if the user doesn’t specify the format, has to guess the format.

In my case, I didn’t do descriptive statistics on my timestamp columns and had a silent truncation(!) of the AM/PM portion of the data. So instead of having 24 hours in the day, the parser read the data as follows (the #AM and #PM are my comments for clarity):

datetime_beginning_utc
2001-01-01 01:00:00 #AM
2001-01-01 01:00:00 #PM
2001-01-01 02:00:00 #AM
2001-01-01 02:00:00 #PM
2001-01-01 03:00:00 #AM
2001-01-01 03:00:00 #PM
2001-01-01 04:00:00 #AM
2001-01-01 04:00:00 #PM
2001-01-01 05:00:00 #AM
2001-01-01 05:00:00 #PM
2001-01-01 06:00:00 #AM
2001-01-01 06:00:00 #PM
2001-01-01 07:00:00 #AM
2001-01-01 07:00:00 #PM
2001-01-01 08:00:00 #AM
2001-01-01 08:00:00 #PM
2001-01-01 09:00:00 #AM
2001-01-01 09:00:00 #PM
2001-01-01 10:00:00 #AM
2001-01-01 10:00:00 #PM
2001-01-01 11:00:00 #AM
2001-01-01 11:00:00 #PM
2001-01-01 12:00:00 #AM
2001-01-01 12:00:00 #PM

So while the data looks like it was imported correctly (because, it is a timestamp), it wasn’t until I realized that hours 13-23 were missing from my data that I realized I had an error.

Pandas To The Rescue!

Fixing this issue is as straight-forward as reading the CSV into python using pandas and specifying the date format:

import pandas as pd
import datetime
df = pd.read_csv("/mnt/storage1TB/hrl_load_metered/hrl_load_metered.csv",
                 parse_dates=[0,1],
                 date_parser= lambda x: datetime.datetime.strptime(x, "%m/%d/%Y %I:%M:%S %p"))

We can see from the code above that pandas has taken our directive about the format and it appears the data have been parsed correctly. A good secondary check here is that the difference in timestamps is -5, which is the offset of the East Coast of the United States relative to UTC.

Uploading to OmniSci Directly From Pandas

Since my PyData talk is going to be using OmniSci, I need to upload this corrected data or rebuild all my work (I’ll opt for fixing my source). Luckily, the pymapd package provides tight integration to an OmniSci database, providing a means of uploading the data directly from a pandas dataframe:

import pymapd

#connect to database
conn = pymapd.connect(host="localhost", port=9091, user="mapd", password="HyperInteractive", dbname="mapd")

#truncate table so that table definition can be reused
conn.execute("truncate table hrl_load_metered")

#re-load data into table
#with none of the optional arguments, pymapd infers that this is an insert operation, since table name exists
conn.load_table_columnar("hrl_load_metered", df)

I have a pre-existing table hrl_load_metered on the database, so I can truncate the table to remove its (incorrect) data but keep the table structure. Then I can use load_table_columnar to insert the cleaned up data into my table and now my data is correct.

Computers May Be Dumb, But Humans Are Lazy

At the beginning, I joked that computers are dumb. Computers are just tools that do exactly what a human programs them to do, and really, it was my laziness that caused this data error. Luckily, I did catch this before my talk and the fix is pretty easy.

I’d like to say I’m going to remember to check my data going forward, but in reality, I’m just documenting this here for the next time I make the same, lazy mistake.

Mirroring an FTP Using lftp and cron

Thu, 06 Sep 2018 00:00:00 +0000

As my Developer Advocate role leads me to doing more and more Sysadmin/Data Engineer type of work, I continuously find myself looking for more efficient ways of copying data folders to where I need them. While there are a lot of great GUI ETL tools out there, for me the simplest and fastest way tends to be using linux utilities. Here’s how to mirror an FTP using lftp, with a cron repeater every five minutes.

Data are on an FTP, Need Further Processing

The problem I have is data that exists on a remote FTP, but are in a binary format that is incompatible with loading directly into OmniSci. My current plan is to use Python to convert the binary format into CSV, but with the data on a server that I don’t control, I need to make a copy somewhere else.

It’s also the case that the data are roughly 300GB per day, streaming in at various intervals across the day, so I need to make sure that any copying I do is thoughtful. Downloading 300GB of data per day is bad enough, doing it multiple times even worse!

lftp `mirror` to the Rescue!

The best choice in my case seems to be copying the files onto a VM I own. lftp has an option mirror to do just that. Here is the one-liner I’m using:

lftp -e "mirror -c -e --parallel=20 --verbose /pub/data/nccf/com/nwm/prod /nwmftp/prod;quit;" ftp.government.gov

lftp -e : Execute command in quotes. In this case, the FTP allows anonymous access, so no user/pw arguments needed
mirror: Mirror command for lftp
-c: If download fails for whatever reason, keep trying (c = “continue”)
-e: Delete files on remote that are no longer on source (i.e. keep folders in perfect sync)
--parallel: Allow multiple connections for parallel downloading of multiple files
--verbose: Print lots of messages, helpful for debugging

With all of the flags in place, the last two arguments are the source (remote FTP) and destination (my VM) directories. Finally, I add a quit statement to exit lftp once the mirror process is over. This is mostly hygiene since I plan to run this on a cron scheduler and don’t want to leave the sessions open.

Run This Every Five Minutes, Forever

cron really is one of the greatest timesavers ever, especially in that it allows super-repetitive work to be automated away, usually with a single line. Here is the line I added after calling crontab -e on the command-line:

*/5 * * * * /home/username/pull_from_ftp.sh

Quite simply, “every 5 minutes, run pull_from_ftp.sh”. Creating pull_from_ftp.sh is as straightforward as creating a text file:

#!/bin/bash
lftp -e "mirror -c -e --parallel=20 --verbose /pub/data/nccf/com/nwm/prod /nwmftp/prod;quit;" ftp.government.gov

That’s It? YES!

With just a few characters short of a full tweet, you can mirror an entire folder from an FTP, automatically. lftp in combination with cron helped me factor out hundreds of lines of pre-existing Python code, which not only removed untested, copy-pasted code from the workflow but also added parallel downloading, increasing data throughput.

Not bad for a couple of free Linux utilities :)

Creating an OmniSci ODBC Connection in RStudio Server

Tue, 21 Aug 2018 00:00:00 +0000

Edit 10/1/2018: When I wrote this blog post, the company and product were named MapD. I’ve changed the title to reflect the new company name, but left the MapD references below to hopefully avoid confusion

In my post Installing MapD on Microsoft Azure, I showed how to install MapD Community Edition on Microsoft Azure, using Ubuntu 16.04 LTS as the base image. One thing I glossed over during the firewall/security section was that I opened ports for Jupyter Notebook and other data science tools, but I didn’t actually show how to install any of those tools.

For this post, I’ll cover how to install MapD ODBC drivers and create a connection within RStudio server.

1. Installing RStudio Server on Microsoft Azure

With an Ubuntu VM running MapD, installing RStudio Server takes but a handful of commands. The RStudio Server download/install page has fantastic instructions, but if you are looking for Azure-specific RStudio Server install instructions, this blog post from Jumping Rivers does a great job.

2. Installing an ODBC Driver Manager

There are two major ODBC driver managers for Linux and macOS: unixODBC and iODBC. I have had more overall ODBC driver installation success with unixODBC than iODBC; here are the instructions for building unixODBC from source:

#download source and extract
wget ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.7.tar.gz
gunzip unixODBC*.tar.gz
tar xvf unixODBC*.tar

#compile and install
cd unixODBC-2.3.7
./configure
make
sudo make install

If you want to check everything is installed correctly, you can run the following command:

odbc_config --cflags

#result
-DHAVE_UNISTD_H -DHAVE_PWD_H -DHAVE_SYS_TYPES_H -DHAVE_LONG_LONG -DSIZEOF_LONG_INT=8 -I/usr/local/include

3. Installing MapD ODBC Driver System-wide

With unixODBC installed, the next step is to install the MapD ODBC drivers. ODBC drivers for MapD are provided as part of MapD Enterprise Edition, so you’ll need to contact your sales representative to get the appropriate version for your MapD installation.

For Linux, the MapD ODBC drivers are provided as a tarball, which when extracted provides all of the necessary ODBC driver files:

#make a directory to extract files into
mkdir mapd_odbc && cd mapd_odbc
tar -xvf ../mapd_odbc_installer_linux_3.80.1.36.tar.gz

#move to /opt/mapd/mapd_odbc (or wherever the other MapD files are)
cd .. && mv mapd_odbc /opt/mapd/mapd_odbc

By convention, MapD suggests placing the ODBC drivers in the same directory as your installation (frequently, /opt/mapd). Wherever you choose to place the directory, you need add that location into the /etc/odbcinst.ini file:

[MapD Driver]
Driver          = /opt/mapd/mapd_odbc/libs/libODBC.so

At this point, we have everything we need to define a connection string within R using odbc:

library(odbc)

conn <- dbConnect(odbc::odbc(),
                  Driver    = "MapD Driver",
                  Server    = "localhost",
                  Database  = "mapd",
                  UID       = "mapd",
                  PWD       = "helloRusers!",
                  Port      = 9091)

Depending on your use case/security preferences, there are two downsides to this method: 1) the credentials are in plain-text in the middle of the script and 2) the RStudio Connection window also shows the credentials in connection window in plain-text until you delete the connection. This can be remedied by defining a DSN (data source name).

4. Defining A DSN

A DSN is what people usually think of when installing ODBC drivers, as it holds some/all of the actual details for connecting to the database. DSN files can be placed in two locations: system-wide in /etc/obdc.ini or in an individual user’s home directory (needs to be ~/.odbc.ini, a hidden file).

In order to have the credentials completely masked in the RStudio session, place the following in the /etc/obdc.ini file:

[MapD Production]
Driver=MapD Driver
PWD=helloRusers!
UID=mapd
HOST=localhost
DATABASE=mapd
PORT=9091

Within the RStudio Connection pane, we can now test our DSN:

With the DSN defined, the R connection code becomes much shorter, with no credentials exposed within the R session:

library(DBI)
con <- dbConnect(odbc::odbc(), "MapD Production")

ODBC: A Big Bag Of Hurt, But Super Useful

While the instructions above aren’t the easiest to work through, once you have ODBC set up and working one time, it’s usually just a matter of appending various credentials to the existing files to add databases.

From a MapD perspective, ODBC is supported through our Enterprise Edition, but it is the slowest way to work with the database. Up to this point, we’ve focused mostly on supporting Python through the pymapd package and the MapD Ibis backend, but there’s no reason technical reason why R can’t also be a first-class citizen.

So if you’re interested in helping develop an R package for MapD, whether using reticulate to wrap pymapd or to help develop Apache Thrift bindings and Apache Arrow native code, send me a Twitter message or connect via LinkedIn (or any other way to contact me) and we’ll figure out how to collaborate!

Data Science Without Leaving the GPU

Mon, 23 Jul 2018 00:00:00 +0000

Data has been growing rapidly for some time now, but CPU-based analytics solutions haven’t been able to sustain the same rate of growth in order to keep up. CPUs in desktop and laptop machines have started adding more cores, but even a 4- or 8-core CPU can only do so much work. Eventually the bottleneck will become not having enough bandwidth to keep all the CPU cores ‘fed’ with data to manipulate. Hadoop provides a framework for working with larger datasets, but its distributed nature can often feel like setting it up is more hassle than its worth.

GPU-based analytics solutions provide a great middle-ground; high-parallelism via thousands of GPU cores, while not having to automatically use a networked, multi-node architecture such as Hadoop. A single data science workstation with 2-4 GPUs can reasonably handle hundreds of millions of records, especially when using the Ibis backend for MapD.

In this webinar, I demonstrate how to do each step of a machine learning workflow, from exploring a dataset to adding features to estimating an xgboost model for predicting the amount of tip a user will give after a taxi ride. Because MapD incorporates Apache Arrow under the hood for its data transfer, this can all be done seamlessly by passing pointers, rather than needing expensive I/O operations, between each tool used. Not having to transfer the data off of the GPU has interesting implications for analytics, which I also discuss towards the end of the talk.

Enjoy!

Parallel, Disk-Efficient .zip to .gz Conversion

Mon, 18 Jun 2018 00:00:00 +0000

Similar to my last post about needing to merge shapefiles using Postgis, I recently downloaded a bunch of energy data from the federal government. 13,370 files to be exact. While the data size itself isn’t that large (~8GB, compressed), an open-source tool I was looking to evaluate only supports gzip compression instead of the zip compressed files I actually had.

While I could’ve used this opportunity to merge the files together into one and do all the data cleaning, I became obsessed with figuring out how to just switch the compression scheme. Here’s the one-liner that emerged:

$ find . -type f -name '*.zip' | parallel "unzip -p -q {} | gzip > {.}.gz && rm {}"

find . -type f -name '*.zip'
  - find all zip files in the current directory, including subdirectories

| parallel
  - take input list passed by find command, run some command against each argument in parallel

"unzip -p -q {} | gzip > {.}.gz && rm {}"
  - unzip a file, with flags -p to pass data to STDOUT and -q for quiet mode
  - {} represents the input file, which comes from list passed by find
  - gzip takes STDOUT as its input, writes to a file whose name is determined by {.}
    (the input file name going into parallel, where the . removes the file extension)
  - && rm {} is run after the gzip process finishes, removing the original .zip file

As a one-liner, it’s not the hardest to comprehend what’s going on, but it’s also not the most intuitive. The key idea here is that once we find all of the zip files, we can unzip/gzip the files in parallel. Note that this works because each process is independent from the other; a single file itself is being unzipped and then gzipped, we’re not unzipping and gzipping a single file in parallel. Just that multiple single-threaded processes are being kicked off at once instead of leaving the other cores in the CPU idle.

Once the unzip-to-gzip process has occurred, then I delete the original zip file. So for the most part, this process can be considered to take constant disk space (if you ignore that 4-8 files are being processed at one time).

Like many one-liners, this took longer to figure out than was actually worth the time savings. But such is life, and now it’s available in the wild for the next person who wonders how to do something like this!

Bulk Loading Shapefiles Into Postgres/Postgis

Fri, 01 Jun 2018 00:00:00 +0000

Recently I’ve been doing a fair bit of work with geospatial data, mostly on the data preparation side. While there are common data formats, I have found that because so much of this data are sourced from government agencies, the data are often in many files that can be concatenated.

In this example, I will show how to take a few dozen county-level shapefiles of parcel data from Utah and load it into a single table in Postgres/Postgis.

Step 1: Downloading Shapefiles

The following shell commands come from an in-progress collaboration with a friend, where we are going to analyze daily air quality in Utah over the past several years. Utah is open-data-friendly, providing shapefiles for every parcel of land in Utah.

While it may have been possible to use wget or curl to download every shapefile, they are stored within Google Drive with a bunch of hashed URLs, so I just clicked on each file instead of trying to be clever. So if you want to follow along with this blog post exactly, you’ll need to download the 25 zip files of Utah shapefiles:

$ ls -lrt utah_lir_shapefiles/
total 408688
-rw-rw-r-- 1 rzwitch rzwitch    954984 Jun  1 13:10 Parcels_Beaver_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   7183466 Jun  1 13:10 Parcels_BoxElder_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   9152777 Jun  1 13:10 Parcels_Cache_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   3279384 Jun  1 13:10 Parcels_Carbon_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch    356058 Jun  1 13:10 Parcels_Daggett_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch  18908413 Jun  1 13:10 Parcels_Davis_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   3900415 Jun  1 13:10 Parcels_Duchesne_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   2689950 Jun  1 13:10 Parcels_Garfield_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   2156109 Jun  1 13:10 Parcels_Grand_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   8107608 Jun  1 13:10 Parcels_Iron_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   1975537 Jun  1 13:10 Parcels_Juab_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   3273485 Jun  1 13:10 Parcels_Kane_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   2741403 Jun  1 13:10 Parcels_Millard_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   1110627 Jun  1 13:10 Parcels_Morgan_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   2970626 Jun  1 13:10 Parcels_Rich_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch 200183664 Jun  1 13:11 Parcels_SaltLake_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   1397522 Jun  1 13:11 Parcels_SanJuan_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   1576757 Jun  1 13:11 Parcels_Sanpete_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   7911261 Jun  1 13:11 Parcels_Summit_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   4480456 Jun  1 13:11 Parcels_Tooele_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch  69690149 Jun  1 13:11 Parcels_Utah_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch   5025674 Jun  1 13:11 Parcels_Wasatch_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch  35896908 Jun  1 13:11 Parcels_Washington_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch    298313 Jun  1 13:11 Parcels_Wayne_LIR.zip
-rw-rw-r-- 1 rzwitch rzwitch  23225130 Jun  1 13:11 Parcels_Weber_LIR.zip

Step 2: Bulk Unzip

With all of these files in the same directory at the same level (i.e. no subfolders), it’s pretty easy to bulk unzip the files, with one caveat: to move the contents of the unzipped files into a new directory, you need to use the -d flag:

mkdir utah_lir_shapefiles_unzipped && unzip utah_lir_shapefiles/\*.zip -d utah_lir_shapefiles_unzipped

The reason I created a new directory (mkdir) and then unzipped the files into a new directory is that when doing analysis, I always like to keep the source data separate, so that I always have the option of starting completely over. It also can make regular expression globs easier :)

Step 3: Creating Postgis Table Definition

After all of the county zip files are unzipped, you get 25 sub-directories structured like the following:

ls -ltr
total 10916
-rw-rw-rw- 1 rzwitch rzwitch   67868 Sep  3  2017 Parcels_Beaver_LIR.shx
-rw-rw-rw- 1 rzwitch rzwitch   28280 Sep  3  2017 Parcels_Beaver_LIR.shp.xml
-rw-rw-rw- 1 rzwitch rzwitch 1503304 Sep  3  2017 Parcels_Beaver_LIR.shp
-rw-rw-rw- 1 rzwitch rzwitch    3036 Sep  3  2017 Parcels_Beaver_LIR.sbx
-rw-rw-rw- 1 rzwitch rzwitch   83052 Sep  3  2017 Parcels_Beaver_LIR.sbn
-rw-rw-rw- 1 rzwitch rzwitch     425 Sep  3  2017 Parcels_Beaver_LIR.prj
-rw-rw-rw- 1 rzwitch rzwitch 9471508 Sep  3  2017 Parcels_Beaver_LIR.dbf
-rw-rw-rw- 1 rzwitch rzwitch       5 Sep  3  2017 Parcels_Beaver_LIR.cpg

The .shp files from the 25 counties all have the same format, which is very convenient. In this step, we can use the shp2pgsql utility that comes with Postgis to read a shapefile, determine the proper schema, then create the table in the database:

shp2pgsql -I -s 26912 -p utah_lir_shapefiles_unzipped/Parcels_Beaver_LIR/Parcels_Beaver_LIR.shp \
utahlirparcels  | psql -h localhost -U <username> <database>;

The key flag here is -p, which means ‘prepare mode’; the shapefile will get read, a table created, but no data loaded. By not loading the data in this step, it makes looping over the files easier later, as no special logic is required to keep the Parcels_Beaver_LIR.shp from being duplicated in Postgis (because it was never loaded in the first place).

Step 4: Bulk Loading Shapefiles into Postgis

The last steps of the loading process are to 1) get all of the shapefile locations and 2) feed them to shp2pgsql:

for i in $(find utah_lir_shapefiles_unzipped/ -type f -name '*.shp'); do
  shp2pgsql -I -s 26912 -a $i utahlirparcels  | psql -h localhost -U <username> <database>;
done;

To get all of the shapefile locations, I use find with flags -type f (files type) and name to search for the pattern within the directory. This command goes through the entire set of subdirectories and gets all the .shp files. From there, I iterate over the list of files using for i in..., then pass the value of $i into a similar shp2pgsql as above. However, rather than using flag -p for ‘prepare’, we are now going to use flag -a for ‘append’. This will perform an INSERT INTO utahlirparcels() statement for Postgres, loading in the actual data from the 25 shapefiles.

Spend Time Now To Save Time Later

Like so much of shell scripting, figuring out these commands took longer than I would’ve expected. Certainly, they took longer to figure out than it would’ve taken to copy-paste a shp2pgsql 25 times! But by taking the time upfront to figure out a generic method of looping over shapefiles, the next time (and every time after that) I find myself needing to do this, this code will be available to load multiple shapefiles into Postgis.

Using RSiteCatalyst With Microsoft PowerBI Desktop

Tue, 13 Mar 2018 00:00:00 +0000

With pretty regular frequency I get emails asking if RSiteCatalyst can be used with Microsoft Power BI. While admittedly I’m not a frequent user of the Windows operating system (nor dashboarding tools like Tableau or Power BI), I am pleased to report that it is fact possible to call the Adobe Analytics API with Power BI via RSiteCatalyst!

Step 1: Call Adobe Analytics API Using `Get Data` Menu

The majority of getting RSiteCatalyst to work within Power BI desktop is getting the R script correct. From the Get Data menu, choose the More... menu option to bring up all of the data import tools that Power BI defines:

Once you choose ‘R Script’, an input box will open where you can place your RSiteCatalyst function calls:

After hitting ‘OK’, Power BI will evaluate your R code, determining which statements return a data.frame (which is the only allowable data structure imported into Power BI). You can choose which data.frame(s) you want to import from the ‘Navigator’ window:

Once you hit ‘OK’, Power BI imports the data and you can use your Adobe Analytics data just as you would in R with RSiteCatalyst (or, like any other data source like CSV or database…)

Limitations

While it’s possible to call RSiteCatalyst through Power BI, there are some limitations to keep in mind.

First, RSiteCatalyst will only work with Microsoft Power BI Desktop, which is installed locally on your machine. The Power BI Service, which is more of a shared dashboard/data store environment, does not allow external API calls as part of its security model. So while you can analyze your data locally, you cannot share dashboards to the Power BI Service.

The second limitation I’ve noticed is that Power BI doesn’t read from from a .Renviron file (at least, not from the default Windows location that R GUI reads). So you will need to place your credentials directly in the R script, which is never really ideal (though, may not be a big deal all things considered).

Finally, the R script runs synchronously, so when placing multiple calls in the same R script you will need to wait for all of the data.frame results before you can use them within Power BI. This is the same default behavior within R, sans using Promises or parallelism of some sort, but it’s still important to keep in mind.

Dashboards, Dashboards, Dashboards!

With a few minutes work, I was able to create this rudimentary dashboard (R code):

Someone with more interesting/higher volume data could surely do better. But the most important thing in my opinion is that Microsoft has built an awesome integration with R and that creating dashboards in Power BI is waaaaaay easier than the last time I tried to create a dashboard using Excel and the Adobe Report Builder plugin.

Happy dashboarding!

Getting Started With OmniSci, Part 2: Electricity Dataset

Fri, 23 Feb 2018 00:00:00 +0000

In my previous MapD post, I loaded electricity data into MapD Community Edition, intentionally ignoring the what of the data to keep that post from being too overwhelming. Now let’s take a step back and explain the dataset, show how to format the data using Python that was loaded into MapD, then use the MapD Immerse UI to build a simple dashboard.

PJM Metered Load Data

I started off my career at PJM doing long-term electricity demand forecasting, to help power engineers do transmission line studies for reliability and to support expansion of the electrical grid in the U.S. Because PJM is a quasi-government agency, they provide over 25 years of hourly electricity usage for the Eastern and Central U.S., both in aggregate and by local power region (roughly, the local power company territories).

However, just because the data is available doesn’t mean it’s convenient, and unfortunately, the data are stored as Excel spreadsheets. This is easily remedied using pandas (v0.22.0, python3.6):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
import os
import pandas as pd

#change to directory with files for convenience
os.chdir("~/electricity_data")

#first sheet in workbook contains all info for years 1993-1999
df1993_1999 = [pd.read_excel(str(x) + "-hourly-loads.xls", usecols = "A:Z") for x in range(1993,1999)]

#melt, append df1993-df1999 together
df_melted = pd.DataFrame()
for x in df1993_1999:
    x.columns = df1993_1999[1].columns.tolist()
    x_melt = pd.melt(x, id_vars=['ACTUAL_DATE', 'ZONE_NAME'], var_name = "HOUR_ENDING", value_name = "MW")
    df_melted = df_melted.append(x_melt)

#multiple sheets to concatenate
#too much variation for a one-liner
d2000 = pd.read_excel("2000-hourly-loads.xls", sheet_name = [x for x in range(2,17)], usecols = "A:Z")
d2001 = pd.read_excel("2001-hourly-loads.xls", sheet_name = None, usecols = "A:Z")
d2002 = pd.read_excel("2002-hourly-loads.xls", sheet_name = [x for x in range(1,18)], usecols = "A:Z")
d2003 = pd.read_excel("2003-hourly-loads.xls", sheet_name = [x for x in range(1,19)], usecols = "A:Z")
d2004 = pd.read_excel("2004-hourly-loads.xls", sheet_name = [x for x in range(2,24)], usecols = "A:Z")
d2005 = pd.read_excel("2005-hourly-loads.xls", sheet_name = [x for x in range(2,27)], usecols = "A:Z")
d2006 = pd.read_excel("2006-hourly-loads.xls", sheet_name = [x for x in range(3,29)], usecols = "A:Z")
d2007 = pd.read_excel("2007-hourly-loads.xls", sheet_name = [x for x in range(3,29)], usecols = "A:Z")
d2008 = pd.read_excel("2008-hourly-loads.xls", sheet_name = [x for x in range(3,29)], usecols = "A:Z")
d2009 = pd.read_excel("2009-hourly-loads.xls", sheet_name = [x for x in range(3,29)], usecols = "A:Z")
d2010 = pd.read_excel("2010-hourly-loads.xls", sheet_name = [x for x in range(3,29)], usecols = "A:Z")
d2011 = pd.read_excel("2011-hourly-loads.xls", sheet_name = [x for x in range(3,32)], usecols = "A:Z")
d2012 = pd.read_excel("2012-hourly-loads.xls", sheet_name = [x for x in range(3,33)], usecols = "A:Z")
d2013 = pd.read_excel("2013-hourly-loads.xls", sheet_name = [x for x in range(3,34)], usecols = "A:Z")
d2014 = pd.read_excel("2014-hourly-loads.xls", sheet_name = [x for x in range(3,34)], usecols = "A:Z")
d2015 = pd.read_excel("2015-hourly-loads.xls", sheet_name = [x for x in range(3,40)], usecols = "B:AA")
d2016 = pd.read_excel("2016-hourly-loads.xls", sheet_name = [x for x in range(3,40)], usecols = "B:AA")
d2017 = pd.read_excel("2017-hourly-loads.xls", sheet_name = [x for x in range(3,42)], usecols = "B:AA")
d2018 = pd.read_excel("2018-hourly-loads.xls", sheet_name = [x for x in range(3,40)], usecols = "B:AA")

#loop over dataframes, read in matrix-formatted data, melt to normalized form
for ord in [d2000, d2001, d2002, d2003, d2004, d2005, d2006, d2007, d2008, d2009, d2010,
            d2011, d2012, d2013, d2014, d2015, d2016, d2017, d2018]:

    for key in ord:
        temp = ord[key]
        temp.columns = df1993_1999[1].columns.tolist() #standardize column names
        temp["ACTUAL_DATE"] = pd.to_datetime(temp["ACTUAL_DATE"]) #force datetime, excel reader wonky
        df_melted = df_melted.append(pd.melt(temp, id_vars=['ACTUAL_DATE', 'ZONE_NAME'], var_name = "HOUR_ENDING", value_name = "MW"))

#(4941384, 4)
#130MB as CSV
#remove any dates that are null, artifacts from excel reader
df_melted[pd.notnull(df_melted["ACTUAL_DATE"])].to_csv("hourly_loads.csv", index=False)

The code is a bit verbose, if only because I didn’t want to spend time to figure out how to programmatically determine how many tabs each workbook has. But the concept is the same each time: read an Excel file, get the data into a dataframe, then convert the data to long form. So instead of having 26 columns (Date, Zone, Hr1-Hr24), we have 4 columns, which is quite frequently a more convenient way to access the data (especially when using SQL).

The final statement writes out a CSV of approximately 4MM rows, the same dataset that was loaded using mapdql in the first post.

Top 10 Usage Days By Season

One of the metrics I used to monitor as part of my job was the top 5/top 10 peak electricity use days per Summer (high A/C usage) and Winter (electric space heating) seasons. Back in those days, I used to use SAS against an enterprise database and the results would come back eventually…

Obviously, it’s not a fair comparison to compare today’s GPUs vs. late ’90s enterprise databases in terms of performance, but back then it did take a non-trivial amount of effort to run this query to keep the report updated. With MapD, I can do the same report in ~100ms:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
--MapD doesn't currently support window functions, so need to precalculate maximum by day
with qry as (select
actual_date,
zone_name,
max(MW) as daily_max_usage
from hourly_loads
where zone_name = 'MIDATL' and actual_date between '2017-06-01' and '2017-09-30'
group by 1,2)
select
hl.actual_date,
hl.zone_name,
hl.hour_ending,
hl.MW
from hourly_loads as hl
inner join qry on qry.actual_date = hl.actual_date and qry.daily_max_usage = hl.mw
order by daily_max_usage desc
limit 10;

The thing about returning an answer in 100ms or so is that its fast enough where calling these results from a webpage/dashboard would be very responsive; that’s where MapD Immerse comes in.

Building A Dashboard Using MapD Immerse

Rather than copy/pasting the query in and running it, it’s pretty easy to build an automated report using the Immerse dashboard builder. I’m limited to a single data source because I’m using MapD Community Edition, but in just a few minutes I was able to create the following dashboard:

I took the query from above and built a view to encapsulate the query, so I didn’t have to worry about the with statement or joins, I could just use the view as if the results were pre-calculated. From there, adding in a results table and two bar charts was fairly quick (in the same drag-and-drop style of Tableau or other BI/reporting tools).

While this dashboard is pretty rudimentary in its design, were this data source set up as real-time using Apache Kafka or similar, this chart would always be up-to-date for use on a TV screen or as a browser bookmark without any additional data or web engineering.

Obviously, many dashboarding tools exist, but its important to note that no pre-aggregation or column indexing or other standard database performance tricks are being employed (outside of specialized hardware and fast GPU RAM caching). Even with 10 dashboard tiles updating serially 100ms at a time, you are still in the 1-2s page load time, on par with the fastest-loading dynamic webpages on the internet.

Programmatic analytics using pymapd

While dashboarding can be very effective for keeping senior management up-to-date, the real value of data is unlocked with more in-depth analytics and segmentation. In my next blog post, I’ll cover how to access MapD using pymapd in Python, doing more advanced visualizations and maybe even some machine learning…

RSiteCatalyst Version 1.4.14 Release Notes

Fri, 16 Feb 2018 00:00:00 +0000

Like the last several updates, this blog post will be fairly short, given only a single bug fix was added.

Thanks again to GitHub user leocwlau who reported that the GetReportSuiteGroups function added an additional field AND provided the solution. No other bug fixes were made, nor was any additional functionality added.

Version 1.4.14 of RSiteCatalyst was submitted to CRAN today and should be available for download in the coming days.

Community Contributions

Getting Started With OmniSci, Part 1: Docker Install and Loading Data

Thu, 01 Feb 2018 00:00:00 +0000

It’s been nearly five years since I wrote about Getting Started with Hadoop for big data. In those years, there have been incremental improvements in columnar file formats and dramatic computation speed improvements with Apache Spark, but I still wouldn’t call the Hadoop ecosystem convenient for actual data analysis. During this same time period, thanks to NVIDIA and their CUDA library for general-purpose calculations on GPUs, graphics cards went from enabling visuals on a computer to enabling massively-parallel calculations as well.

Building upon CUDA is MapD, an analytics platform that allows for super-fast SQL queries and interactive visualizations. In this blog post, I’ll show how to use Docker to install MapD Community Edition and load hourly electricity demand data to analyze.

Installing MapD CE using Docker/nvidia-docker

While CUDA makes it possible to do calculations on GPUs, I wouldn’t go as far as to say it is easy, including just getting everything installed! Luckily, there is Docker and nvidia-docker, which provide all-in-one containers with all necessary drivers and libraries installed to build upon. MapD provides instructions for installing MapD CE using nvidia-docker, with the main installation command as follows:

1
2
3
4
5
##nvidia-docker version 2
docker run --runtime=nvidia \
-v $HOME/mapd-docker-storage:/mapd-storage \
-p 9090-9092:9090-9092 \
mapd/mapd-ce-cuda

When you kickoff this command (I’m using a ssh terminal into a remote Ubuntu desktop), Docker will download all the required images from the mapd/mapd-ce-cudarepository and start a background process for the MapD database and the Immerse visualization interface/web server:

Once all of the images are downloaded, you can find the container that was created using docker container ls, then run docker exec -it <container id> bash to start the container and drop you into a Bash shell (on the container). From this point, MapD Community Edition will be running!

Loading Data Using the Immerse Interface

Once the Bash shell opens in the terminal, you can now interact with MapD via the Docker container. However, for beginning exploration, it’s much simpler to use the Immerse Web Interface at localhost:9092:

Uploading data via the Data Manager interface is reasonably performant for smaller files; a test file with four columns and million or so rows loaded in a few seconds (dependent on your upload speed, obviously):

Edit the column names and types if you want (the CSV reader gets it right for me most of the time). Then, once the ‘Save Table’ button is clicked, MapD will import the CSV data into a columnar binary format, so that the GPU can operate directly on the data rather than reading from the CSV each query.

Loading Data Using the Command Line

While browser GUIs are great for some things, I’m still very much a command-line guy, at least for things like loading data. MapD provides the mapdql interface to load data and query, very much like psql for Postgres and other databases. To load my 4.9 million * 4 column dataset, I used the following commands:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
51be4b888448        mapd/mapd-ce-cuda   "/bin/sh -c '/mapd/s…"   44 hours ago        Up 44 hours         0.0.0.0:9090-9092->9090-9092/tcp   nifty_heisenberg

$ docker exec -it 51be4b888448 bash
root@1f64b2dcc316:/mapd# bin/mapdql
Password: <default is "HyperInteractive">
User mapd connected to database mapd

mapdql> create table hourly_loads(
..> ACTUAL_DATE DATE,
..> ZONE_NAME TEXT,
..> HOUR_ENDING SMALLINT,
..> MW FLOAT);

mapdql> copy hourly_loads from 'hourly_loads.csv';
Result
Loaded: 4898472 recs, Rejected: 0 recs in 0.923000 secs

The DDL for MapD seems pretty much the same as every other database language. First you define a table’s columns and their types, then you can use the copy command to load data from a CSV. The statement that prints upon success begins to give an indication of the speed MapD provides, loading nearly 5 million records in less than a second.

Simplistic Query Performance

Up this point, I’ve intentionally not described the data I uploaded into MapD; in my next post, I’ll cover the dataset I’m using and how I converted the data from Excel spreadsheets into a CSV. But before ending this post, I wanted to show a brief summary of the performance of MapD:

The first query shows a simple record count by the hour_ending dimension in my table, something you might run if you weren’t too familiar with the table. You’ll notice that running this group by across the 5 million row dataset took 5143ms, which isn’t so fast. What’s going on?

Because this is the first query from a cold start, MapD needs to load data into GPU RAM. So while the first query takes a few seconds, the second query displays a warmed-up level of performance: 212ms to scan 5 million rows, filter by a few values of the zone_name column, then grouping by hour_ending. For reference, a human blink takes 100-400 ms, so this second query quite literally finished in the blink of an eye…

Dashboards, Streaming Data and more…

This first blog post just scratched the surface on what is possible using just the Community Edition of MapD. In future blog posts, I will provide the code to create the dataset, do some basic descriptive statistics, and even do some analysis and dashboarding of historical electricity demand.

Update, 2/1/2018 4:49 p.m.

Per Todd Mostak from MapD, the second query would likely even run faster than 212ms, had I run it again:

Also fyi that 212ms query will likely run faster if you run it again (even changing the literals). MapD compiles queries the first time it sees a new query plan but then can reuse the same compiled code if you change literals (like for zone_name).
— Todd Mostak (@ToddMostak) February 1, 2018

Parallelizing Distance Calculations Using A GPU With CUDAnative.jl

Mon, 14 Aug 2017 00:00:00 +0000

Hacker News discussion: link

Code as Julia Jupyter Notebook

Julia has the reputation as a “fast” language in that it’s possible to write high-performing code. However, what I appreciate most about Julia is not just that the code is fast, but rather that Julia makes high-performance concepts accessible without having to have a deep computer science or compiled language background (neither of which I possess!)

For version 0.6 of Julia, another milestone has been reached in the “accessible” high-performance category: the ability to run Julia code natively on NVIDIA GPUs through the CUDAnative.jl package. While CUDAnative.jl is still very much in its development stages, the package is already far-enough along that within a few hours, as a complete beginner to GPU programming, I was able to see in excess of 20x speedups for my toy example to calculate haversine distance.

Getting Started

The CUDAnative.jl introduction blog post and documentation cover the installation process in-depth, so I won’t repeat the details here. I’m already a regular compile-from-source Julia user and I found the installation process pretty easy on my CUDA-enabled Ubuntu workstation. If you can already do TensorFlow, Keras or other GPU tutorials on your computer, getting CUDAnative.jl to work shouldn’t take more than 10-15 minutes.

Julia CPU Implementation

To get a feel for what sort of speedup I could expect from using a GPU, I wrote a naive implementation of a distance matrix calculation in Julia:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#https://github.com/quinnj/Rosetta-Julia/blob/master/src/Haversine.jl
haversine(lat1::Float32,lon1::Float32,lat2::Float32,lon2::Float32) = 2 * 6372.8 * asin(sqrt(sind((lat2-lat1)/2)^2 + cosd(lat1) * cosd(lat2) * sind((lon2 - lon1)/2)^2))

function pairwise_dist(lat::Vector{Float32}, lon::Vector{Float32})

    #Pre-allocate, since size is known
    n = length(lat)
    result = Array{Float32}(n, n)

    #Brute force fill in each cell, ignore that distance [i,j] = distance [j,i]
    for i in 1:n
        for j in 1:n
            @inbounds result[i, j] = haversine(lat[i], lon[i], lat[j], lon[j])
        end
    end

    return result

end

#Example benchmark call
lat10000 = rand(Float32, 10000) .* 45
lon10000 = rand(Float32, 10000) .* -120
@time native_julia_cellwise = pairwise_dist(lat10000, lon10000)

The above code takes a pair of lat/lon values, then calculates the haversine distance between the two points. This algorithm is naive in that a distance matrix is symmetric (i.e. the distance between A to B is the same from B to A), so I could’ve done half the work by setting result[i,j] and result[j,i] to the same value, but as a measure of work for a benchmark this toy example is fine. Also note that this implementation runs on a single core, no CPU-core-level parallelization has been implemented.

Or to put all that another way: if someone wanted to tackle this problem without thinking very hard, the implementation might look like this.

CUDAnative.jl Implementation

There are two parts to the CUDAnative.jl implementation: the kernel (i.e. the actual calculation) and the boilerplate code for coordinating the writing to/from the CPU and GPU.

Kernel Code

The kernel code has similarities to the CPU implementation, with a few key differences:

Method signature is one lat/lon point vs. the lat/lon vectors, rather than a pairwise distance calculation
Boilerplate code for thread index on the GPU (0-indexed vs. normal Julia 1-indexing)
The trigonometric functions need to be prepended with CUDAnative., to differentiate that the GPU functions aren’t the same as the functions from Base Julia
Rather than return an array as part of the function return, we use the out keyword argument to write directly to the GPU memory

1
2
3
4
5
6
7
8
9
10
11
12
13
14
using CUDAnative, CUDAdrv

#Calculate one point vs. all other points simultaneously
function kernel_haversine(latpoint::Float32, lonpoint::Float32, lat::AbstractVector{Float32}, lon::AbstractVector{Float32}, out::AbstractVector{Float32})

    #Thread index
    #Need to do the n-1 dance, since CUDA expects 0 and Julia does 1-indexing
    i = (blockIdx().x-1) * blockDim().x + threadIdx().x

    out[i] =  2 * 6372.8 * CUDAnative.asin(CUDAnative.sqrt(CUDAnative.sind((latpoint-lat[i])/2)^2 + CUDAnative.cosd(lat[i]) * CUDAnative.cosd(latpoint) * CUDAnative.sind((lonpoint - lon[i])/2)^2))

    #Return nothing, since we're writing directly to the out array allocated on GPU
    return nothing
end

Coordination Code

The coordination code is similar to what you might see in a main() function in C or Java, where the kernel is applied to the input data. I am using the dev keyword with the default value of CuDevice(0) to indicate that the code should be run on the first (in my case, only) GPU device.

The remainder of the code has comments on its purpose, primarily:

Transfer Julia CPU arrays to GPU arrays (CuArray)
Set number of threads/blocks
Calculate distance between a point and all other points in the array, write back to CPU

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
#validated kernel_haversine/distmat returns same answer as CPU haversine method (not shown)
function distmat(lat::Vector{Float32}, lon::Vector{Float32}; dev::CuDevice=CuDevice(0))

    #Create a context
    ctx = CuContext(dev)

    #Change to objects with CUDA context
    n = length(lat)
    d_lat = CuArray(lat)
    d_lon = CuArray(lon)
    d_out = CuArray(Vector{Float32}(n))

    #Calculate number of calculations, threads, blocks
    len = n
    threads = min(len, 1024)
    blocks = Int(ceil(len/threads))

    #Julia side accumulation of results to relieve GPU memory pressure
    accum = Array{Float32}(n, n)

    # run and time the test
    secs = CUDAdrv.@elapsed begin
        for i in 1:n
            @cuda (blocks, threads) kernel_haversine(lat[i], lon[i], d_lat, d_lon, d_out)
            accum[:, i] = Vector{Float32}(d_out)
        end
    end

    #Clean up context
    destroy!(ctx)

    #Return timing and bring results back to Julia
    return (secs, accum)

end

#Example benchmark call
timing, result = distmat(lat10000, lon10000)
result ≈ native_julia_cellwise #validate results equivalent CPU and GPU

The code is written to process one row of the distance matrix at a time to minimize GPU memory usage. By writing out the results to the CPU after each loop iteration, I have n-1 extra CPU transfers, which is less performant than calculating all the distances first then transferring, but my consumer-grade GPU with 6GB of RAM would run out of GPU memory before completing the calculation otherwise.

Performance

The performance characteristics of the CPU and GPU calculations are below for various sizes of distance matrices. Having not done any GPU calculations before, I was surprised to see how much of a penalty there is writing back and forth to the GPU. As you can see from the navy-blue line, the execution time is fixed for matrices of size 1 to 1000, representing the fixed cost of moving the data from the CPU to the GPU.

Of course, once we get above 1000x1000 matrices, the GPU really starts to shine. Due to the log scale, it’s a bit hard to see the magnitude differences, but at 100000x100000 there is a 23x reduction in execution time (565.008s CPU vs. 24.32s GPU).

What I Learned

There are myriad things I learned from this project, but most important is that GPGPU processing can be accessible for people like myself without a CS background. Julia isn’t the first high-level language to provide CUDA functionality, but the fact that the code is so similar to native Julia makes GPU computing something I can include in my toolbox today.

Over time, I’m sure I’ll get better results as I learn more about CUDA, as CUDAnative.jl continues to smooth out the rough edges, etc. But the fact that as a beginner that I could achieve such large speedups in just an hour or two of coding and sparse CUDAnative.jl documentation bodes well for the future of GPU computing in Julia.

Code as Julia Jupyter Notebook

RSiteCatalyst Version 1.4.13 Release Notes

Sun, 23 Jul 2017 00:00:00 +0000

This blog post will be fairly short, given the minor nature of the update.

Several users complained about OAUTH2 authentication not working, which I didn’t know because I usually use the legacy authentication method! Luckily, GitHub user leocwlau reported the issue AND provided the solution. No other bug fixes were made, nor was any additional functionality added.

So if you ran into an issue where your login no longer worked, version 1.4.13 of RSiteCatalyst should remedy the issue. Even if you hadn’t run into this authentication issue, users should still upgrade, as all updates are cumulative in nature.

Community Contributions

RSiteCatalyst Version 1.4.12 (and 1.4.11) Release Notes

Mon, 10 Apr 2017 00:00:00 +0000

I released version 1.4.12 of RSiteCatalyst before I wrote the release notes for version 1.4.11, so this blog post will treat both releases as one. Users should upgrade directly to version 1.4.12 as the releases are cumulative in nature.

Get* method additions

Two Get* methods were added in this release: GetClickMapReporting and GetPreviousServerCalls, mostly for completeness. Analytics users will likely not need to use these methods, but they are useful for generating documentation.

Bug fixes

Fixed GetLogin function, adding selected_ims_group_list parameter to response (caused test suite failure)
Fixed issue with ViewProcessingRules where nested rules threw errors (#214)
Fixed issue with GetMarketingChannelRules where nested rules threw errors, matches column duplicated values across rows (#180)
Added ability to use a segment in QueueDataWarehouse function, which was previously implemented incorrectly (#216)
Fixed issue with QueueDataWarehouse not returning the proper number of results when enqueueOnly = FALSE
Fixed encoding for QueueDataWarehouse to correctly use UTF-8-BOM (#198)
Fixed parser for GetFeeds, to unnest ‘activity’ data frame into discrete columns
Fixed issue where message Error in if (!is.null(elements[i, ]$classification) && nchar(elements[i, : missing value where TRUE/FALSE needed displayed when using multiple elements in a Queue* function (#207)

Community Contributions (An Adobe Summit bounce?!)

In the past month, the number of GitHub issues submitted has increased dramatically, a good problem to have!

I encourage all users of the software to continue reporting bugs via GitHub issues, and especially if you can provide a working code example. Even better, a fix via pull request will ensure that your bug will be addressed in a timely manner and for the benefit to others in the community.

Self-Service Adobe Analytics Data Feeds!

Fri, 03 Mar 2017 00:00:00 +0000

I’ve written several posts about the Adobe Analytics Analytics (née Clickstream) Data Feed (links: 1,2,3) over the past several years. The Analytics Data Feed is an invaluable tool for moving beyond aggregate-level reporting information about your customers to really in-depth, customer-level analytics.

While the Analytics Data Feed is nowhere as easy to use as the Adobe UI, Report Builder, Analytics Workspace or even RSiteCatalyst, the hardest part for me when I was a digital analytics consultant was just obtaining the files in the first place (to say nothing of restarting a failed feed)! Luckily, Adobe has now built an interface accessible by any Adobe Analytics admin, removing the need for ClientCare to set up and maintain feeds.

In this post, I will briefly highlight how to set up an Analytics Data Feed from inside Adobe Analytics and give my impressions of the tool (as it exists at the time of writing). Note this post is not meant to be a substitute to the official documentation; Adobe provides detailed information about the entire process in their Help section.

Landing the Files via FTP/SFTP/S3

After clicking ‘Admin -> Data Feeds’ in the Adobe Analytics Admin menu/panel, you should see an interface similar to the following:

After clicking the “(+) Add” button, you will be brought to a page with three parts: Feed Information, Delivery Location and Data Column Definitions. The top half of the page looks as follows:

I find these two panels fairly self-explanatory. On the left, you give your data feed a name, choose the report suite(s) you want the data feed for, pick the delivery granularity, then choose the start/end date or choose ‘continuous’ for indefinite future delivery. On the right side, you provide your server information, FTP/SFTP or Amazon S3.

There are several things I love about this setup:

Delivery via SFTP and S3: For the longest time, FTP was the only choice, which disappointed security-minded folks who wanted SFTP. With S3 delivery, you can house your data files and read them into Hadoop directly (assuming you use AWS); this means you don’t necessarily have to do anything with the data until you need it, then you can fire up an EMR job to get your data. Dump the cluster when you’re done. Nice.
Multiple Report Suites, same window: I’m so happy to see that this interface supports choosing multiple report suites within the same “feed”. This makes it so much easier than having to create a separate feed instance, when in many cases the settings will stay the same (other than report suite of course).

What I don’t love:

Not saving FTP information: When you do choose to create separate feeds, there doesn’t appear to be a way to use the same credentials without typing them in yourself. Not a huge deal if your password is “Mom”, a little more annoying if you auto-generate a password that looks like “aHR0cDovL3JhbmR5endpdGNoLmNvbQ==”

Data Column Definitions

After choosing the report suites and providing the delivery details, the remaining step is to decide which data fields you want as part of the data feed. Luckily, there are templates that can be chosen, so that the user doesn’t have to remember every single field and the order that they need.

My favored practice when working with the data feeds is to have every feed have the same structure, even if certain report suites don’t have nearly the same amount of eVars and props implemented. I’m of the mind that taking all the fields, then letting the ETL process handle what to do with the data is much less error-prone. You can always re-run an ETL process if you forgot something; if you forget a column in your data feed, you need to re-process all of the feeds.

Empty columns are also much less wasteful in the age of distributed computing (such as Hadoop), as columnar data formats/databases remove much of the performance penalty of empty columns. Using dictionary encoding and columnar data structures, you can represent an entire column of missing values as {NA: } and spend mere bytes for data storage and still have lossless compression.

Save and Repeat

Once you have your data feed defined, you save your work and that’s it! I’m not sure what SLA might be in place, but it seems like there’s roughly a 48-hour delay between submitting a new feed and having the data start to process. Once the processing starts, your notification inbox will quickly look like this:

Automated Monitoring with RSiteCatalyst

If you want to have more control around monitoring than just staring at your inbox, you can use RSiteCatalyst to get the processing status of your feed:

1
2
3
4
5
6
7
8
9
10
library(RSiteCatalyst)

#Authentication (assumes credentials saved in .Renviron)
SCAuth(Sys.getenv("USER", ""), Sys.getenv("SECRET", ""))

#Get full list of report suites
rs <- GetReportSuites()

#Pass in all non-virtual report suites
feedstatus <- GetFeeds(rs[is.na(rs$virtual), ]$rsid)

GetFeeds will return a dataframe that returns the feed information, as well as the processing status:

It’s not hard to imagine what a cronjob would look like, polling the Adobe API every few minutes to see which feeds are complete, then kicking off an ETL process.

Data, Liberated!

This blog post has been relatively light on code and analysis, but I hope I’ve highlighted that the barriers to obtaining the most granular data Adobe can provide have been completely removed. With the new Analytics Data Feed interface, the possibility of bespoke customer analytics is only a few button clicks and an FTP away.

Building a Data Science Workstation (2017)

Wed, 18 Jan 2017 00:00:00 +0000

Update, 3/14/2018: While I’ve still maintained the same basic workstation, I have done some upgrades. Before GPU prices skyrocketed, I added a 1080ti to workstation along with the 1060. However, this required an upgrade of my case to fit both. I ended up getting a Corsair 760T, which is an enormous increase in space over the Corsair mid-tower I had before.

Since I was already in the case, I also added a AIO water-cooler to my CPU, a Corsair H115i Pro. While this wasn’t strictly necessary (I wasn’t having overheating issues with a single GPU and overclocked CPU), with two GPUs I figured that it was a good precaution. The water-cooler is also much quieter than my original air-cooled heatsink, which is nice.

So now I have a huge amount of open space, which is great for airflow, a dual-GPU rig and depending on how you calculate things, I’m at around $3000 for a workstation that should serve me well into the next several years (hopefully more!)

(original post below)

i need some kind of super computer to parse a multi-gigabyte XML file, @randyzwitch
— jason thompson (@usujason) January 17, 2017

For all the downsides of social media these days, the people I’ve met and the inside jokes bring me immense joy. From just this one tweet, several people messaged me aghast at 1) the idea of multi-gigabyte XML and that 2) I apparently have a computer for just this purpose!

Recently, I did build a workstation as a test bed for learning more about data science, specifically in the areas of Docker and GPU computing. Here’s what I built.

Machine Specs and Assembling from Parts

While there are several specialist workstation companies like Titan Computing that sell configurable workstations, assembling a computer from parts is pretty easy, and more importantly, can be significantly cheaper if you choose ‘consumer-grade’ parts instead of ‘server-grade’. Because I’m not planning on curing cancer or building Skynet with this computer, I opted not to get a dual-chip motherboard and used a single Intel i7 chip rather than going with Xeon workstation class chip(s). I also chose to use standard DDR4 RAM rather than ECC RAM. Here’s the full spec list:

PCPartPicker part list

Component	Details
CPU	Intel Core i7-5820K 3.3GHz 6-Core Processor
CPU Cooler	Cooler Master Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler
Motherboard	Asus X99-A II ATX LGA2011-3 Motherboard
Memory	Crucial Ballistix Sport LT 32GB (2 x 16GB) DDR4-2400 Memory
Memory	Crucial Ballistix Sport LT 32GB (2 x 16GB) DDR4-2400 Memory
Memory	Crucial Ballistix Sport LT 32GB (2 x 16GB) DDR4-2400 Memory
Memory	Crucial Ballistix Sport LT 32GB (2 x 16GB) DDR4-2400 Memory
Storage	Samsung 850 EVO-Series 250GB 2.5” Solid State Drive
Storage	Hitachi Deskstar 1TB 3.5” 7200RPM Internal Hard Drive
Video Card	EVGA GeForce GTX 1060 6GB 6GB SC GAMING Video Card
Case	Corsair SPEC-02 ATX Mid Tower Case
Power Supply	Corsair CXM 750W 80+ Bronze Certified Semi-Modular ATX Power Supply
Optical Drive	Asus DRW-24B1ST/BLK/B/AS DVD/CD Writer

Total: ~$2000

So for about $2000 and a few hours of time assembling, I have the rough equivalent of an r3.4xlarge instance on AWS ($1.33/hr on-demand at the time of writing). It would take about 1500 hours of usage to breakeven vs. using AWS, but cost isn’t really the point; the convenience of having the computer in my house and not having to do the startup/shutdown/EC2 images/S3/firewall/etc. dance is more than worth it to me so that I can focus on learning instead of operations.

Ubuntu 16.04LTS, CUDA, and Other Installs

While I did use Windows 10 to validate I put together my hardware correctly (and to mess with overlocking settings using the ASUS motherboard tools), I decided to use Ubuntu 16.04LTS as my base operating system. This allows for the most ‘server-like’ operations that I’m used to from a Linux environment. I enabled an internal static IP through my router, and for the most part, I either SSH into the machine from my MacBook Pro or use web UIs such as Jupyter Notebook (and again, remotely from my laptop).

I tried to install the NVIDIA drivers from source for the GTX1060 GPU, but eventually gave up and went the apt package manager route and everything works fine. Though I rarely sit at my desktop, I do have a 4K monitor hooked up to this computer which looks gorgeous with a video card of this caliber, and I have CUDA installed and working as well.

From there, I installed any number of tools from Python, R, Julia, MonetDB, Docker, Neo4j, Postgres, Spark, BlazingDB…any/all of which I hope to write about more in the near future.

RSiteCatalyst Version 1.4.10 Release Notes

Tue, 13 Dec 2016 00:00:00 +0000

Version 1.4.10 of RSiteCatalyst brings a handful of new Get* methods, QueueDatawarehouse and a couple of bugs fixes/low-level code improvements.

QueueDatawarehouse

The most useful user-facing change IMO is the addition of the QueueDatawarehouse method, which allows for submitting (and sometimes retrieving) Data Warehouse requests via R. This should be a huge timesaver for those of you using Data Warehouse as a substitute for the Adobe Analytics raw data feed (my employer alone pulls hundreds of Data Warehouse feeds per day).

In the coming days, I’ll write a blog post in more detail about how to use this method effectively to query Data Warehouse, but in the meantime, here’s a sample function call:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#API Credentials
SCAuth(Sys.getenv("USER", ""), Sys.getenv("SECRET", ""))
#FTP Credentials
FTP <- Sys.getenv("FTP", "")
FTPUSER <- Sys.getenv("FTPUSER", "")
FTPPW <- Sys.getenv("FTPPW", "")

#Write QueueWarehouse to FTP
report.id <- QueueDataWarehouse("report-suite",
                                "2016-11-01",
                                "2016-11-07",
                                c("visits", "pageviews"),
                                c("page"),
                                enqueueOnly=TRUE,
                                ftp = list(host = FTP,
                                           port = "21",
                                           directory = "/DWtest/",
                                           username = FTPUSER,
                                           password = FTPPW,
                                           filename = "myreport.csv")
)

ViewProcessingRules

ViewProcessingRules has also been added on an experimental basis, as this API method is not documented (as of the writing of this post), so Adobe may choose to modify or remove this in a future release. As the method name indicates, this new feature allows for the viewing of the processing rules for a list of report suites, including the behaviors that define the rules. There is currently no (public) method for setting processing rules via the API.

As processing rules are a super-user functionality, I would love it if someone in the community could verify that this method works for a large number of report suites/rules.

Get* method additions

Three Get* methods were added in this release: GetVirtualReportSuiteSettings, GetReportSuiteGroups, GetTimeStampEnabled, each corresponding to settings able to be viewed within the Adobe Analytics admin panel.

Bug fixes

GetRealTimeSettings: fixed bug where a passing a list of report suites lead to a parsing error
QueueSummary: Redefined method arguments to make date an optional parameter, allowing for more elegant use of date.to and date.from parameters.

Package-level improvements

At long last, I’ve added integration testing via the AppVeyor testing service. For the longest time, I’ve had a very nonchalant attitude towards Windows (since I don’t use it), but given that RSiteCatalyst is enterprise software and so many businesses use Windows, I figured it was time.

Luckily, there were none of the tests in the test suite throws any errors specifically due to Windows, so effectively this change is just defensive programming against that class of error in the future.

Community Contributions

As in the past several releases, there have been contributions from the community keeping RSiteCatalyst moving forward! Special thanks to Diego Villuendas Pellicero for writing the QueueDataWarehouse functionality and Johann de Boer for highlighting that QueueSummary could be improved.

WordPress to Jekyll: A 30x Speedup

Mon, 10 Oct 2016 00:00:00 +0000

About a month ago, I switched this blog from WordPress hosted on Bluehost to Jekyll on GitHub Pages. I suspected moving to a static website would be faster than generated HTML via PHP, and it is certainly cheaper (GitHub Pages is “free”). But it wasn’t until I needed a dataset for doing some dataset visualization development that I realize how much of an improvement it has been!

Packages, Packages, Packages

With the release of v0.5 of Julia, I’ve been working (less) on updating my packages and making new packages (more), because making new stuff is more fun than maintaining old stuff! One of the packages I’ve been building is for the ECharts visualization library (v3) from Baidu. While Julia doesn’t necessarily need another visualization library, visualization is something I’m interested in and learning is easier when you’re solving problems you like. And since the world doesn’t need another Iris example, I decided to share some real world website performance data :)

Line Chart

One of the first features I developed for ECharts.jl was X-Y charts, which I posit is the most common chart type in business. One thing that is great about the underlying ECharts JavaScript library is that interactivity is really easy to achieve:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
using ECharts, DataFrames

#Read in data
df = readtable("/assets/data/website_time_data.csv")

#Make data two different series that overlap, so endpoint touches
df[:pre] = [(x[1] <= "2016-09-06" ? x[2] : nothing) for x in zip(df[:date], df[:loadtime_ms])]
df[:post] = [(x[1] >= "2016-09-06" ? x[2] : nothing) for x in zip(df[:date], df[:loadtime_ms])]

#Graph code
l = line(df[:date], hcat(df[:pre], df[:post]))
l.ec_width = 800
seriesnames!(l, ["loadtime_ms", "post"])
colorscheme!(l, palette = ("acw", "FlatUI"))
yAxis!(l, name = "Load time in ms")
title!(l, text = "randyzwitch.com",
          subtext = "Switching from WordPress on Bluehost to Jekyll on GitHub (2016/09/06)")
toolbox!(l, chartTypes = ["bar", "line"])
slider!(l)

Even though I switched to Jekyll on WordPress on 9/6/2016, it appears that the page cache for Google Webmaster Tools didn’t really expire until 9/12/2016 or so. At the average case, the load time went from 1128ms to 38ms! Of course, this isn’t really a fair comparison, as presumably GitHub Pages runs on much better hardware than the cheap Bluehost hosting I have, and I didn’t reimplement most of the garbage I had on the WordPress version of the blog. But from a user-experience standpoint, good lord what an improvement!

Box Plots

Want to test out further functionality, here are some box plots of the load time variation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
using ECharts, DataFrames

#Read in data
df = readtable("/Users/randyzwitch/Desktop/website_load_time.csv")
df[:pre] = [(x[1] <= "2016-09-06" ? x[2] : nothing) for x in zip(df[:date], df[:loadtime_ms])]
df[:post] = [(x[1] >= "2016-09-12" ? x[2] : nothing) for x in zip(df[:date], df[:loadtime_ms])]

#Remove nulls
pre = [x for x in df[:pre] if x != nothing]
post = [x for x in df[:post] if x != nothing]

#Graph code
b = box([pre, post], names = ["WordPress", "Jekyll"])
b.ec_width = 800
colorscheme!(b, palette = ("acw", "VitaminC"))
yAxis!(b, name = "Load time in ms", nameGap = 50, min = 0)
title!(b, text = "randyzwitch.com",
           subtext = "Switching from WordPress on Bluehost to Jekyll on GitHub (2016/09/06)")
toolbox!(b)

Usually, a box plot comparison that is as smushed as the Jekyll plot vs the WordPress one would be a poor visualization, but in this case I think it actually works. The load time for the Jekyll version of this blog is so quick and so consistent that it barely registers as an outlier if it were WordPress! It’s crazy to think that the -1.5 * IQR time for WordPress is the mean/median/min load time of Jekyll.

Where To Go Next?

This blog post is really just an interesting finding from my experience moving to Jekyll on GitHub. As it stands now, ECharts.jl is stil in pre-METADATA mode. Right now, I assume that this would be a useful enough package to submit to METADATA some day, but I guess that depends on how much further I get smoothing the rough edges. If there are people who are interested in cleaning up this package further, I’d absolutely love to collaborate.

Bulk Downloading Adobe Analytics Data

Thu, 21 Jul 2016 08:25:02 +0000

This blog post also serves as release notes for RSiteCatalyst v1.4.9, as only one feature was added (batch report request and download). But it’s a feature big enough for its own post!

Recently, I was asked how I would approach replicating the market basket analysis blog post I wrote for 33 Sticks, but using a lot more data. Like, months and months of order-level data. While you might be able to submit multiple months worth of data in a single RSiteCatalyst call, it’s a lot more elegant to request data from the Adobe Analytics API in several calls. With the new batch-submit and batch-receive functionality in RSiteCatalyst, this process can be a LOT faster.

Non-Batched Method

Prior to version 1.4.9 of RSiteCatalyst, API calls could only be made in a serial fashion:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
library(RSiteCatalyst)
library(dplyr)

SCAuth(Sys.getenv("USER", ""), Sys.getenv("SECRET", ""))

combined_orders <- data.frame()
for(d in seq(as.Date("2016-06-01"), as.Date("2016-06-30"), by = "day")){

  d_ <- as.character(as.Date(d, origin = "1970-01-01"))
  print(d_)

  order_details <- QueueRanked(
    reportsuite.id = "reportsuite",
    date.from = d_,
    date.to = d_,
    elements = c('evar13', 'product'),
    metrics = c('revenue', 'units', 'orders'),
    top = c(50000, 10000),
    interval.seconds = 60
  )

  order_details$order_date <- d_
  combined_orders <- rbind.fill(combined_orders, order_details)
  rm(order_details)

}

The underlying assumption from a package development standpoint was that the user would be working in an interactive fashion; submit a report request, wait to get the answer back. There’s nothing inherently wrong with this code from an R standpoint that made this a slow process, you just had to wait until one report was calculated by the Adobe Analytics API until the next one was submitted.

Batch Method

Of course, most APIs can process multiple calls simultaneously, and the Adobe Analytics API is no exception. Thanks to user shashispace, it’s now possible to submit all of your report calls at once, then retrieve the results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
queued <- list()
for(d in seq(as.Date("2016-06-01"), as.Date("2016-06-30"), by = "day")){
  d_ <- as.character(as.Date(d, origin = "1970-01-01"))

  print(d_)

  reportid <- QueueRanked(
    reportsuite.id = "reportsuite",
    date.from = d_,
    date.to = d_,
    elements = c('evar13', 'product'),
    metrics = c('revenue', 'units', 'orders'),
    top = c(50000, 10000),
    interval.seconds = 1,
    enqueueOnly = TRUE
  )

  queued <- cbind(queued, reportid)

}

queued_df <- data.frame()
for (i in queued){
  queued_df <- bind_rows(queued_df, GetReport(i))
}

This code is nearly identical to the serial snippet above, except for 1) the addition of the enqueueOnly = TRUE keyword argument and 2) lowering the interval.seconds keyword argument to 1 second instead of 60. When you use the enqueueOnly keyword, instead of returning the report results back, a Queue* function will return the report.id; by accumulating these report.id values in a list, we can next retrieve the reports and bind them together using dplyr.

Performance gain: 4x speed-up

Although the code snippets are nearly identical, it is way faster to submit the reports all at once then retrieve the results. By submitting the requests all at once, the API will process numerous calls at once, and while you are retrieving the results of one call the others will continue to process in the background.

I wouldn’t have thought this would make such a difference, but retrieving one month of daily order-level data went from taking 2420 seconds to 560 seconds! If you were to retrieve the same amount of daily data, but for an entire year, that would mean saving 6 hours in processing time.

Keep The Pull Requests Coming!

The last several RSiteCatalyst releases have been driven by contributions from the community and I couldn’t be happier! Given that I don’t spend much time in my professional life now using Adobe Analytics, having improvements driven by a community of users using the library daily is just so rewarding.

So please, if you have a comment for improvement (and especially if you find a bug), please submit an issue on GitHub. Submitting questions and issues to GitHub is the easiest way for me to provide support, while also giving other users the possibility to answer your question before I might. It will also provide a means for others to determine if they are experiencing a new or previously-known problem.

Adobe Analytics Clickstream Data Feed: Calculations and Outlier Analysis

Tue, 24 May 2016 11:11:20 +0000

In a previous post, I outlined how to load daily Adobe Analytics Clickstream data feeds into a PostgreSQL database. While this isn’t a long-term scalable solution for large e-commerce companies doing millions of page views per day, for exploratory analysis a relational database structure can work well until a more robust solution is put into place (such as Hadoop/Spark).

Data Validation

Before digging too deeply into the data, we should validate that data from the data feed in our database (custom database view code) matches what we observe from other sources (mainly, the Adobe Analytics interface and/or RSiteCatalyst). Given the Adobe Analytics data feed represents an export of the underlying data, and Adobe provides the formulas in the data feed documentation, in theory you should be able to replicate the numbers exactly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# "Source 1": Pull data from the API using RSiteCatalyst
library("RSiteCatalyst")
SCAuth(Sys.getenv("USER", ""), Sys.getenv("SECRET", ""))
overtime <- QueueOvertime("zwitchdev",
                           date.from = "2016-04-01",
                           date.to = "2016-05-17",
                           metrics = c("pageviews", "visits", "visitors"),
                           date.granularity = "day")

# "Source 2": Pull data from Postgres database
library(RPostgreSQL)

# Connect to database
conn <- dbConnect(dbDriver("PostgreSQL"),
                 user="postgres",
                 password="",
                 host="localhost",
                 port=5432,
                 dbname="adobe")

dbdata <- dbGetQuery(conn,
                     "select
                     date(date_time) as date_localtime,
                     sum(CASE WHEN post_page_event = '0' THEN 1 END) as pageviews,
                     count(distinct ARRAY_TO_STRING(ARRAY[post_visid_high::text, post_visid_low::text, visit_num::text], '')) as visits,
                     count(distinct ARRAY_TO_STRING(ARRAY[post_visid_high::text, post_visid_low::text], '')) as visitors
                     from usefuldata
                     where date_time between '2016-04-01' and '2016-05-18' and exclude_hit = '0'
                     group by 1
                     order by 1;")

# Compare data sources
> diff_pv = table(overtime$pageviews - dbdata$pageviews)
> diff_pv

0
47

> diff_visits = table(overtime$visits - dbdata$visits)
> diff_visits

0
47

> diff_visitors = table(overtime$visitors - dbdata$visitors)
> diff_visitors

0
47

The code snippet above shows the validation, and sure enough, the “two different sources” show the same exact values (i.e. differences are 0), so everything has been loaded properly into the PostgreSQL database.

Finding Anomalies For Creating Bot Rules

With the data validated, we can now start digging deeper into the data. As an example, although I have bot filtering enabled, this only handles bots on the IAB bot list but not necessarily people trying to scrape my site (or worse).

To create a custom bot rule in Adobe Analytics, you can use IP address(es) and/or User-Agent string. However, as part of data exploration we are not limited to just these features (assuming, of course, that you can map your feature set back to an IP/User-Agent combo). To identify outlier behavior, I’m going to use a technique called ‘local outlier factors’ using the Rlof package in R with the following data features:

Distinct Days Visited
Total Pageviews
Total Visits
Distinct Pages Viewed
Pageviews Per Visit
Average Views Per Page

These aren’t the only features I could’ve used, but it should be pretty easy to view bot/scraper traffic using these metrics. Here’s the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Local outlier factor calculation
library(RPostgreSQL)
library(Rlof)

conn <- dbConnect(dbDriver("PostgreSQL"),
                  user="postgres",
                  password="",
                  host="localhost",
                  port=5432,
                  dbname="adobe")

metrics_lof <- dbGetQuery(conn,
                          "select
                          ip,
                          distinct_days_visited,
                          pageviews,
                          visits,
                          distinct_pages_viewed,
                          pageviews/visits::double precision as pv_per_visit,
                          pageviews/distinct_pages_viewed::double precision as avg_views_per_page
                          from
                          (
                          select
                          ip,
                          sum(CASE WHEN post_page_event = '0' THEN 1 END) as pageviews,
                          count(distinct ARRAY_TO_STRING(ARRAY[post_visid_high::text, post_visid_low::text, visit_num::text, visit_start_time_gmt::text], '')) as visits,
                          count(distinct post_pagename) as distinct_pages_viewed,
                          count(distinct date(date_time)) as distinct_days_visited
                          from usefuldata
                          where exclude_hit = '0'
                          group by 1
                          ) a
                          where visits > 1 and pageviews > 1;")


# The higher the value of k, the more likely lof will be calculated...
# ...but more generic the clusters
# NaN/Inf occurs with points on top of one another/div by zero, which is likely...
# ...with web data when most visitors have 1-2 sessions
df_lof<-lof(metrics_lof[, 2:7],k = 20)

# Append results, get top 500 worst scoring IP addresses
results <- cbind(metrics_lof, df_lof)[order(-df_lof),]
worst500 <- head(subset(results, !is.infinite(df_lof)), 500)

A local outlier factor greater than 1 is classified as a potential outlier. Here’s a visual of the lof scores for the top 500 worst scoring IP addresses (vegalite R graph code):

We can see from the graph that there are at least 500 IP addresses that are potential outliers (since the line doesn’t go below a lof value of 1). These points are now a good starting place to go back to our overall table and inspect the entire datafeed records by IP address.

But what about business value?

The example above just scratches the surface on what’s possible when you have access to the raw data from Adobe Analytics. It’s possible to do these calculations on my laptop using R because I only have a few hundred-thousand records and IP addresses. But this kind of ops work is pretty low-value, since unless you are trying to detect system hacking, trying to find hidden scrapers/spiders in your data to filter out just modifies the denominator of your KPIs it doesn’t lead to real money per se.

In the last post of this series, I’ll cover how to work with the datafeed using Spark, and provide an example of using Spark MLLib to increase site engagement.

Adobe: Give Credit. You DID NOT Write RSiteCatalyst.

Mon, 09 May 2016 12:31:51 +0000

EDIT 5/10/2016 1:30pm: Several folks from Adobe Analytics/Adobe Marketing Cloud have contacted me, and everything is resolved. I can’t untweet other people’s retweets/shares or delete comments on LinkedIn, but if everyone could stop sharing any more that would be great. 🙂

As an author of several open-source software projects, I’ve taken for granted that people using the software share the same community values as I do. Open-source authors provide their code “free” to the community so that others may benefit without having to re-invent the wheel. The only expectation (but not an actual requirement per se), is attribution to the package author(s) as a thank you for the time and effort they put into writing and maintaining a quality piece of software.

However, when others take direct credit for writing a package they did not, it crosses into a different realm. Adobe, you DID NOT write RSiteCatalyst, nor have you made any meaningful contributions. To take credit for RSiteCatalyst, either implicitly or explicitly, is a slight to the work of those who have contributed.

Adobe Summit 2014: Attribution!

In the beginning, there seemed to be no problem providing proper attribution. I count Ben Gaines as one of my stronger professional acquaintances (dare I say, even a friend), so I was honored that he not only mentioned me on stage at his Adobe Summit 2014 presentation, but also followed up with an official Adobe blog post re-capping his main points:

Perfect. My package got wide exposure to the intended audience, which in turn makes it easier to devote time for development and maintenance. The recognition also helped me professionally in that time period, so if I never thanked you publicly Ben, thank you!

Adobe Summit 2015: An Inconspicuous Absence

In 2015, RSiteCatalyst moved from a “Tip” to a full-fledged presentation. I was honored when I first heard that an entire hour would be dedicated to reviewing the package, but no attribution was given:

I mean, it was obviously okay to link to non-Adobe websites like statmethods.net (a great reference btw) and to Shiny…but okay, attribution is not a requirement.

Adobe Summit 2016: ‘We at Adobe…’

The non-mention at Adobe Summit 2015 could be attributed to an oversight; the following during the 2016 RSiteCatalyst Adobe Summit presentation cannot:

Just so we’re clear, this isn’t me noticing the slide notes in a PDF or PPT I shouldn’t have access to. The screenshot above is directly from the Adobe Summit video and the statement was said nearly verbatim during the presentation. And it’s not like this was a one-off comment…it’s the same damn presentation as 2015, and I KNOW this script went through several rounds of review and practice by the presenters.

It Costs $0 To Do What Is Right

It may be hard for RSiteCatalyst users to believe, but this was the first open-source project I ever wrote AND the means by which I learned how to write R code AND the first time I ever accessed an API. Since then, Willem Paling did an amazing job refactoring/re-writing a majority of the package when the Adobe Analytics API was updated from version 1.3 to 1.4, and there have been numerous other contributions from the user community. Maybe even one day, the repo will reach even 100 stars on GitHub…

But save for a single commit to a README file from an employee, Adobe you have contributed _zero_to the development and maintenance of this package. To claim otherwise is beyond distasteful to the ethos of open-source software. I’ve never asked for compensation of any kind; and again, I recognize that you don’t even need to attribute the work at all.

Just don’t take credit yourselves for providing this functionality to your customers. You did not write RSiteCatalyst Adobe, a community of (unpaid) volunteers did.

Travis CI: "You Have Too Many Tests LOLZ!"

Tue, 05 Apr 2016 11:41:53 +0000

As part of getting RSiteCatalyst 1.4.8 ready for CRAN, I’ve managed to accumulate hundreds of testthat tests across 63 test files. Each of these tests runs on Travis CI against an authenticated API, and the API frequently queues long-running reports. Long-story-short, my builds started failing, creating the error log message quoted below:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

Stalled Build?

The most frustrating about this error is that all my tests run (albeit, a looooong time) successfully through RStudio, so I wasn’t quite sure what the problem was with the Travis CI build. Travis CI does provide a comment about this in their documentation, but even then it didn’t solve my problem:

When a long running command or compile step regularly takes longer than 10 minutes without producing any output, you can adjust your build configuration to take that into consideration.

The shell environment in our build system provides a function that helps to work around that, at least for longer than 10 minutes.

If you have a command that doesn’t produce output for more than 10 minutes, you can prefix it with travis_wait, a function that’s exported by our build environment.

The travis_wait command would work if I were installing packages, but my errors were during tests, so this parameter isn’t the answer. Luckily, testthat provides a test filtering mechanism, providing a solution by allowing the tests to be broken up into smaller chunks.

Regex To The Rescue…

For many applications, the default testthat configuration example will work just well:

R CMD check
Create tests/testthat.R that contains:
library(testthat)
library(yourpackage)
test_check(“yourpackage”)

However, hidden within the test_check() arguments is filter, which will take a regular expression to filter which files in the test folder will get run when the command is triggered by R CMD check. Why is this important? Because each time a new test_check() function gets called, output gets written to stdout, and thus avoids 10 minutes passing without producing any output. Here’s an example of what my successful build logs now look like (GitHub code for the testthat code structure):

checking tests…
Running ‘testthat-build.R’
Running ‘testthat-get.R’ [5s/267s]
Running ‘testthat-queuefallout.R’ [1s/59s]
Running ‘testthat-queueovertime.R’ [3s/210s]
Running ‘testthat-queuepathing.R’ [2s/55s]
Running ‘testthat-queueranked.R’ [2s/183s]
Running ‘testthat-queuesummary.R’ [2s/136s]
Running ‘testthat-queuetrended.R’ [17s/346s]
Running ‘testthat-save.R’ [1s/46s]
OK

You can now see that instead of getting a single output message of Running testthat.R, I have nine separate test files running, none of which take 10 minutes to complete. For my package, each of my test files is labeled based on the function name, and I can end up using really simple regex literals such as the following:

1
2
library(testthat)
test_check("RSiteCatalyst", filter = "get")

So each file with the word “get” in the filename will be run by this function; I’m not worried about writing complex regexes here, since at worst I my matching is too broad and I run the same test multiple times.

…But Be Careful Of Case-Sensitivity!

The one caveat to simple regex filtering above is that if you’re not careful, you’ll get no match from your test_check() function, which will fail the build on Travis CI. I spent hours trying to figure out why my tests ran fine on OSX, but failed on Travis. Eventually, I even filed an issue against hadley’s repo, feeling silly as soon as I found out that my error was due to case-sensitivity in Linux by not OSX (or Windows for that matter).

So, pay attention, and if all else fails, go with filter = "summary|Summary" or similar to match the case of your filenames!

You Can Never Really Have Too Many Tests

Obviously, the title of this blog post is in jest; Travis CI doesn’t care what you’re running or comments on how many tests you run. But hopefully this blog post provides the answer to the next person down the line running into this issue. Don’t delete your tests, run multiple test_check() functions and the printing every few minutes of the file name to the console should resolve the problem.

RSiteCatalyst Version 1.4.8 Release Notes

Mon, 04 Apr 2016 10:05:15 +0000

For being in RSiteCatalyst retirement, I’m ending up working on more functionality lately ¯_(ツ)_/¯. Here are the changes for RSiteCatalyst 1.4.8, which should be available on CRAN shortly:

Segment Stacking

RSiteCatalyst now has the ability to take multiple values in the segment.id keyword for the Queue* functions. This functionality was graciously provided by Adam Gitzes, closing an issue that was nearly a year old. At times it felt like I was hazing him with change requests, but for Adam’s first open-source contribution, this is a huge addition in functionality.

So now you are able to pass multiple segments into a function call and get an ‘AND’ behavior like so:

1
2
3
4
5
6
7
stacked_seg <- QueueRanked("zwitchdev",
                          "2016-03-08",
                          "2016-03-09",
                          "pageviews",
                          "page",
                          segment.id = c("5433e4e6e4b02df70be4ac63", "54adfe3de4b02df70be5ea08")
                          )

The result (Visits from Social AND Visits from Apple Browsers):

QueueSummary: Now with `date.to` and `date.from` keywords

In response to GitHub issue #158, date.to and date.from parameters were added; this was a minor, but long-term oversight (it’s always been possible to do this in the Adobe Analytics API). So now rather than just specifying the date keyword and getting a full-year summary or a full-month, you can specify any arbitrary start/end dates.

Trivial Fixes: Silenced `httr` message, clarified documentation

Starting with the newest version of httr, you get a message for any API call where the encoding wasn’t set. So for long running Queue* requests, you may have received dozens of warnings to stdout about "No encoding supplied: defaulting to UTF-8." This has been remedied, and the warning should no longer occur.

Also, the documentation for the Queue* functions was clarified to show an example of using SAINT classifications as the report breakdown; hopefully this didn’t cause too much confusion to anyone else.

Volunteers Wanted!

As I referenced in the first paragraph, while I’m fully committed to maintaining RSiteCatalyst, I don’t currently have the time/desire to continue to develop the package to improve functionality. Given that I don’t use this package for my daily work, it’s hard for me to dedicate time to the project.

Thanks again to Adam Gitzes who stepped up and provided significant effort to close an outstanding feature request. I would love if others in the digital analytics community would follow Adam’s lead; don’t worry about whether you are ‘good enough’, get a working solution together and we’ll figure out how to harden the code and get it merged. Be the code change you want to see the world 🙂

Adobe Analytics Clickstream Data Feed: Loading To Relational Database

Fri, 18 Mar 2016 14:42:23 +0000

In my previous post about the Adobe Analytics Clickstream Data Feed, I showed how it was possible to take a single day worth of data and build a dataframe in R. However, most likely your analysis will require using multiple days/weeks/months of data, and given the size and complexity of the feed, loading the files into a relational database makes a lot of sense.

Although there may be database-specific “fast-load” tools more appropriate for this application, this blog post will show how to handle this process using only R and PostgresSQL.

File Organization

Before getting into the loading of the data into PostgreSQL, I like to sort my files by type into separate directories (remember from the previous post, you’ll receive three files per day). R makes OS-level operations simple enough:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#### 1. Setting directory to FTP folder where files incoming from Adobe
## Has ~2000 files in it from 2 years of data
setwd("~/Downloads/datafeed/")

#### 2. Sort files into three separate folders
## Manifests - plain text files
if(!dir.exists("manifest")){
  dir.create("manifest")
  lapply(list.files(pattern = "*.txt"), function(x) file.rename(x, paste("manifest", x, sep = "/")))
}

## Server calls tsv.gz
if(!dir.exists("servercalls")){
  dir.create("servercalls")
  lapply(list.files(pattern = "*.tsv.gz"), function(x) file.rename(x, paste("servercalls", x, sep = "/")))
}

## Lookup files .tar.gz
if(!dir.exists("lookup")){
  dir.create("lookup")
  lapply(list.files(pattern = "*.tar.gz"), function(x) file.rename(x, paste("lookup", x, sep = "/")))
}

Were there more file types, I could’ve abstracted this into a function instead of copying the code three times, but the idea is the same: Check to see if the directory exists, if it doesn’t then create it and move the files into the directory.

Connecting and Loading Data to PostgreSQL from R

Once we have our files organized, we can begin the process of loading the files into PostgreSQL using the RPostgreSQL R package. RPostgreSQL is DBI-compliant, so the connection string is the same for any other type of database engine; the biggest caveat of loading your servercall data into a database is the first load is almost guaranteed to require loading as text (using colClasses = "character" argument in R). The reason that you’ll need to load the data as text is that Adobe Analytics implementations necessarily change over time; text is the only column format that allows for no loss of data (we can fix the schema later within Postgres either by using ALTER TABLE or by writing a view).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
library(RPostgreSQL)

# Connect to database
conn = dbConnect(dbDriver("PostgreSQL"),
                 user="postgres",
                 password="",
                 host="localhost",
                 port=5432,
                 dbname="adobe")

#Set directory to avoid having to use paste to build urls
setwd("~/Downloads/datafeed/servercalls")

#Set column headers for server calls
column_headers <- read.delim("~/Downloads/datafeed/lookup/column_headers.tsv", stringsAsFactors=FALSE)

#Loop over entire list of files
#Setting colClasses to character only way to guarantee all data loads
#File formats or implementations can change over time; fix schema in database after data loaded
for(file in list.files()){
  print(file)
  df <- read.csv2(file, sep = "\t", header = FALSE, stringsAsFactors = FALSE, colClasses = "character")
  dbWriteTable(conn, name = 'servercalls', value = df, row.names=FALSE, append = TRUE)
  rm(df)
}

#Run analyze in PostgreSQL so that query planner has accurate information
dbGetQuery(conn, "analyze servercalls")

With this small amount of code, we’ve generated the table definition structure (see here for the underlying Postgres code), loaded the data, and told Postgres to analyze the table to gather statistics for efficient queries. Sweet, two years of data loaded with minimal effort!

Loading Lookup Tables Into PostgreSQL

With the server call data loaded into our database, we now need to load our lookup tables. Lucky for us, these do maintain a constant format, so we don’t need to worry about setting all the fields to text, RPostgreSQL should get the column types correct.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
library(RPostgreSQL)

# Connect to database
conn = dbConnect(dbDriver("PostgreSQL"),
                 user="postgres",
                 password="",
                 host="localhost",
                 port=5432,
                 dbname="adobe")

setwd("~/Downloads/datafeed/lookup/")

#Create function due to repetitiveness
#Since we're loading lookup tables with mostly same values each time, put source file in table
loadlookup <- function(tblname){
  df <- read.csv2(paste(tblname,".tsv", sep=""), sep = "\t", header = FALSE, stringsAsFactors = FALSE)
  df$file <- file
  dbWriteTable(conn, name = tblname, value = df, row.names=FALSE, append = TRUE)
}

#untar files, place in directory by day
for(file in list.files(pattern = "*.tar.gz")){
  print(file)
  untar(file)

  for(tbl in c("browser_type", "browser", "color_depth", "column_headers",
               "connection_type", "country", "event", "javascript_version",
               "languages", "operating_systems", "plugins", "referrer_type",
               "resolution", "search_engines")){

    loadlookup(tbl)

  }
}

SHORTCUT: The dimension tables that are common to all report suites don’t really change over time, although that isn’t guaranteed. In the 758 days of files I loaded (code), the only files having more than one value for a given key were: browser, browser_type, operating_system, search_engines, event (report suite specific for every company) and column_headers (report suite specific for every company). So if you’re doing a bulk load of data, it’s generally sufficient to use the newest lookup table and save yourself some time. If you are processing the data every day, you can use an upsert process and generally there will be few if any updates.

Let’s Do Analytics!!!!???!!!

Why is there always so much ETL work, I want to data science the hell out of some data

At this point, if you were uploading the same amount of data for the traffic my blog does (not much), you’d be about 1-2 hours into loading data, still having done no analysis. In fact, in order to do analysis, you’d still need to modify the column names and types in your servercalls table, update the lookup tables to have the proper column names, and maybe you’d even want to pre-summarize the tables into views/materialized views for Page View/Visit/Visitor level. Whew, that’s a lot of work just to calculate daily page views.

Yes it is. But taking on a project like this isn’t for page views; just use the Adobe Analytics UI!

In a future blog post or two, I’ll demonstrate how to use this relational database layout to perform analyses not possible within the Adobe Analytics interface, and also show how we can skip this ETL process altogether using a schema-on-read process with Spark.

Calling RSiteCatalyst From Python

Mon, 22 Feb 2016 10:34:44 +0000

@randyzwitch Do you know if anyone has gotten RSiteCat running in a Jupyter Notebook that ran RPY2? Tired of using 2 different environments

— Adam Gitzes (@FootballActuary) February 18, 2016

This will be a very short post, because the only “new” information I’m going to provide is the minimal example to answer the question. Yes, it is in fact possible to call RSiteCatalyst from Python and seems to work well. The most important things are 1) making sure you install rpy2 and 2) loading Pandas (since so much of RSiteCatalyst is API calls returning data frames). It doesn’t hurt to already have experience using RSiteCatalyst in R, since all we’re doing here is using Python to pass code to R.

Setup Code: rpy2 and Pandas

To call an R package from Python, the rpy2 package works very well, both from the REPL and Jupyter Notebook. For RSiteCatalyst, here is the set up code:

1
2
3
4
5
6
7
8
9
10
import pandas as pd

import rpy2.robjects.packages as rpackages
from rpy2.robjects import pandas2ri

# Activate ability to translate R objects to pandas data frames
pandas2ri.activate()

# Load RSiteCatalyst into Python
rsc = rpackages.importr('RSiteCatalyst')

With this code run, now you can make calls to the RSiteCatalyst R package, just as if you were in R itself.

Sample Call: GetReportSuites

Just to prove it works, here’s a code snippet using GetReportSuites():

1
2
3
# Call GetReportSuites to confim it works
grs = rsc.GetReportSuites()
pandas2ri.ri2py_dataframe(grs)

And in Jupyter Notebook, you would see something similar to:

But, Why?

So that’s about it…if you wanted to, you could call RSiteCatalyst from Python without much effort. There aren’t a whole lot of reasons to do so, unless like Adam above, you’d rather just use Python. I suppose if you wanted to use some other Python packages, such as Flask to create a dashboard or Seaborn for visualization you might want to do this. Until I got this tweet, it never occurred to me to do this, so YMMV.

Edit, 2/26/16: Adam Gitzes, who originally asked the question, also provides a different solution using Jupyter Notebook magics at his blog post here

RSiteCatalyst Version 1.4.7 (and 1.4.6.) Release Notes

Mon, 01 Feb 2016 09:24:04 +0000

It seems as though I missed release notes for version RSiteCatalyst 1.4.6, so we’ll do those and RSiteCatalyst 1.4.7 (now on CRAN) and the same time…

RSiteCatalyst 1.4.6

This release was mostly tweaking some settings, specifically:

Adding a second top argument within the Queue* functions for more control on results returned. It used to be the case that a breakdown report with the top argument would return, say, the top 10 values of the first variable and up to 50,000 values for the breakdown. Now you can control the second level breakdown as well, such as the top 10 pages and top 5 browsers for those pages.
Disable checking of the API call before submitting. I never ran into this, but a user was seeing that the API would return errors in validation under high volume. So if you have any weird issues, disable validation using the validate = FALSE keyword argument.
The package now handles situation where API returns an unexpected type for the reportID and automatically converts it to the proper type (low-level issue, not a user-facing issue)

Those changes carry forward into version RSiteCatalyst 1.4.7, so there is no reason for a user to stick with this release.

RSiteCatalyst 1.4.7 - No more Unicode Errors!

I was surprised it took so long for someone to report this error, but #151 finally reported a case from a user in Germany where search keywords were being mangled due to the presence of an umlaut. UTF-8 encoding is now the default for both calling the API and processing the results, so this issue will hopefully not arise again.

Additionally, a locale argument has been added, to set the proper locale for your report suite. This is specified through the SCAuth() function, with the list of possible locales provided by the Adobe documentation. So if the even after using 1.4.7 with UTF-8 encoding by default, you are still seeing errors, try setting the locale to the country you are in/country setting of the report suite.

Feature Requests/Bugs

As always, if you come across bugs or have feature requests, please continue to use RSiteCatalyst GitHub Issues page to submit issues. Don’t worry about cluttering up the page with tickets, please fill out a new issue for anything you encounter (with code you’ve already tried and is failing), unless you are SURE that it is the same problem someone else is facing.

However, outside of patching really serious bugs, I will likely not spend any more time improving this package in the future; my interests have changed, and RSiteCatalyst is pretty much complete as far as I’m concerned. That said, contributors are also very welcomed. If there is a feature you’d like added, and especially if you can fix an outstanding issue reported at GitHub, we’d love to have your contributions. Willem and I are both parents of young children and have real jobs outside of open-source software creation, so we welcome any meaningful contributions to RSiteCatalyst that anyone would like to contribute.

A Million Text Files And A Single Laptop

Thu, 28 Jan 2016 09:53:42 +0000

More often that I would like, I receive datasets where the data has only been partially cleaned, such as the picture on the right: hundreds, thousands…even millions of tiny files. Usually when this happens, the data all have the same format (such as having being generated by sensors or other memory-constrained devices).

The problem with data like this is that 1) it’s inconvenient to think about a dataset as a million individual pieces 2) the data in aggregate are too large to hold in RAM but 3) the data are small enough where using Hadoop or even a relational database seems like overkill.

Surprisingly, with judicious use of GNU Parallel, stream processing and a relatively modern computer, you can efficiently process annoying, “medium-sized” data as described above.

Data Generation

For this blog post, I used a combination of R and Python to generate the data: the “Groceries” dataset from the arules package for sampling transactions (with replacement), and the Python Faker (fake-factory) package to generate fake customer profiles and for creating the 1MM+ text files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
#R Code
library(arules)
data("Groceries")
write(Groceries, "groceries.txt", sep = ",")

#Python Code
import random, csv
from faker import Faker
fake = Faker()
from pandas import DataFrame
import pandas as pd

# Create customer file of 1,234,567 customers with fake data
# Use dataframe index as a way to generate unique customer id
customers = [fake.simple_profile() for x in range(0,1234567)]
customer_df = pd.DataFrame(customers)
customer_df["cust_id"] = customer_df.index

#Read in transactions file from arules package
with open("grocerydata.txt") as f:
    transactions = f.readlines()

#Remove new line character
transactions = [x[0:-1] for x in transactions]

#Generate transactions by cust_id

#file format:
#cust_id::int
#store_id::int
#transaction_datetime::string/datetime
#items::string

#for each customer...
for i in range(0,1234567):
    #...create a file...
    with open('/transactions/custfile_%s' % i, 'w') as csvfile:
        trans = csv.writer(csvfile, delimiter=' ', quotechar='"', quoting=csv.QUOTE_MINIMAL)
        #...that contains all of the transactions they've ever made
        for j in range(1, random.randint(1,365)):
            trans.writerow([i, fake.zipcode(), fake.date_time_this_decade(before_now=True, after_now=False), transactions[random.randint(0,len(transactions) - 1)]])

Problem 1: Concatenating (`cat * >> out.txt` ?!)

The cat utility in Unix-y systems is familiar to most anyone who has ever opened up a Terminal window. Take some or all of the files in a folder, concatenate them together….one big file. But something funny happens once you get enough files…

$ cat * >> out.txt

-bash: /bin/cat: Argument list too long

That’s a fun thought…too many files for the computer to keep track of. As it turns out, many Unix tools will only accept about 10,000 arguments; the use of the asterisk in the cat command gets expanded before running, so the above statement passes 1,234,567 arguments to cat and you get an error message.

One (naive) solution would be to loop over every file (a completely serial operation):

for f in *; do cat "$f" >> ../transactions_cat/transactions.csv; done

Roughly 10,093 seconds later, you’ll have your concatenated file. Three hours is quite a coffee break…

Solution 1: GNU Parallel & Concatenation

Above, I mentioned that looping over each file gets you past the error condition of too many arguments, but it is a serial operation. If you look at your computer usage during that operation, you’ll likely see that only a fraction of a core of your computer’s CPU is being utilized. We can greatly improve that through the use of GNU Parallel:

ls | parallel -m -j $f "cat {} >> ../transactions_cat/transactions.csv"

The $f argument in the code is to highlight that you can choose the level of parallelism; however, you will not get infinitely linear scaling, as shown below (graph code, Julia):

Given that the graph represents a single run at each level of parallelism, it’s a bit difficult to say exactly where the parallelism gets maxed out, but at roughly 10 concurrent jobs, there’s no additional benefit. It’s also interesting to point out what the -m argument represents; by specifying m, you allow multiple arguments (i.e. multiple text files) to be passed as inputs into parallel. This alone leads to an 8x speedup over the naive loop solution.

Problem 2: Data > RAM

Now that we have a single file, we’ve removed the “one million files” cognitive dissonance, but now we have a second problem: at 19.93GB, the amount of data exceeds the RAM in my laptop (2014 MBP, 16GB of RAM). So in order to do analysis, either a bigger machine is needed or processing has to be done in a streaming or “chunked” manner (such as using the “chunksize” keyword in pandas).

But continuing on with our use of GNU Parallel, suppose we wanted to answer the following types of questions about our transactions data:

How many unique products were sold?
How many transactions were there per day?
How many total items were sold per store, per month?

If it’s not clear from the list above, in all three questions there is an “embarrassingly parallel” portion of the computation. Let’s take a look at how to answer all three of these questions in a time- and RAM-efficient manner:

Q1: Unique Products

Given the format of the data file (transactions in a single column array), this question is the hardest to parallelize, but using a neat trick with the [tr](http://www.linfo.org/tr.html) (transliterate) utility, we can map our data to one product per row as we stream over the file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Serial method (i.e. no parallelism)
# This is a simple implementation of map & reduce; tr statements represent one map, sort -u statements one reducer

# cut -d ' ' -f 5- transactions.csv | \     - Using cut, take everything from the 5th column and over from the transactions.csv file
# tr -d \" | \                              - Using tr, trim off double-quotes. This leaves us with a comma-delimited string of products representing a transaction
# sort -u | \                               - Using sort, put similar items together, but only output the unique values
# wc -l                                     - Count number of unique lines, which after de-duping, represents number of unique products

$ time cut -d ' ' -f 5- transactions.csv | tr -d \" | tr ',' '\n' | sort -u | wc -l
331

real	292m7.116s

# Parallelized version, default chunk size of 1MB. This will use 100% of all CPUs (real and virtual)
# Also map & reduce; tr statements a single map, sort -u statements multiple reducers (8 by default)

$ time cut -d ' ' -f 5- transactions.csv | tr -d \" | tr ',' '\n' | parallel --pipe --block 1M sort -u | sort -u | wc -l
331

# block size performance - Making block size smaller might improve performance
# Number of jobs can also be manipulated (not evaluated)
# --500K:               73m57.232s
# --Default 1M:         75m55.268s (3.84x faster than serial)
# --2M:                 79m30.950s
# --3M:                 80m43.311s

The trick here is that we swap the comma-delimited transactions with the newline character; the effect of this is taking a single transaction row and returning multiple rows, one for each product. Then we pass that down the line, eventually using sort -u to de-dup the list and wc -l to count the number of unique lines (i.e. products).

In a serial fashion, it takes quite some time to calculate the number of unique products. Incorporating GNU Parallel, just using the defaults, gives nearly a 4x speedup!

Q2. Transactions By Day

If the file format could be considered undesirable in question 1, for question 2 the format is perfect. Since each row represents a transaction, all we need to do is perform the equivalent of a SQL Group By on the date and sum the rows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Data is at transaction level, so just need to do equivalent of 'group by' operation
# Using cut again, we choose field 3, which is the date part of the timestamp
# sort | uniq -c is a common pattern for doing a 'group by' count operation
# Final tr step is to trim the leading quotation mark from date string

time cut -d ' ' -f 3 transactions.csv | sort | uniq -c | tr -d \"

real	76m51.223s

# Parallelized version
# Quoting can be annoying when using parallel, so writing a Bash function is often much easier than dealing with escaping quotes
# To do 'group by' operation using awk, need to use an associative array
# Because we are doing parallel operations, need to pass awk output to awk again to return final counts

awksub () { awk '{a[$3]+=1;}END{for(i in a)print i" "a[i];}';}
export -f awksub
time parallel --pipe awksub < transactions.csv | awk '{a[$1]+=$2;}END{for(i in a)print i" "a[i];}' | tr -d \" | sort

real	8m22.674s (9.05x faster than serial)

Using GNU Parallel starts to become complicated here, but you do get a 9x speed-up by calculating rows by date in chunks, then “reducing” again by calculating total rows by date (a trick I picked up at this blog post.

Q3. Total items Per store, Per month

For this example, it could be that my command-line fu is weak, but the serial method actually turns out to be the fastest. Of course, at a 14 minute run time, the real-time benefits to parallelization aren’t that great.

It may be possible that one of you out there knows how to do this correctly, but an interesting thing to note is that the serial version already uses 40-50% of the available CPU available. So parallelization might yield a 2x speedup, but seven minutes extra per run isn’t worth spending hours trying to the optimal settings.

But, I’ve got MULTIPLE files…

The three examples above showed that it’s possible to process datasets larger than RAM in a realistic amount of time using GNU Parallel. However, the examples also showed that working with Unix utilities can become complicated rather quickly. Shell scripts can help move beyond the “one-liner” syndrome, when the pipeline gets so long you lose track of the logic, but eventually problems are more easily solved using other tools.

The data that I generated at the beginning of this post represented two concepts: transactions and customers. Once you get to the point where you want to do joins, summarize by multiple columns, estimate models, etc., loading data into a database or an analytics environment like R or Python makes sense. But hopefully this post has shown that a laptop is capable of analyzing WAY more data than most people believe, using many tools written decades ago.

21st Century C: Error 64 on OSX When Using Make

Thu, 31 Dec 2015 13:17:30 +0000

To end 2015, I decided to finally learn C, instead of making it a 2016 resolution! I had previously done the ‘Learn C The Hard Way’ tutorials, taken about a year off, and thus forgotten everything.

Rather than re-do the same material, I decided to get ’21st Century C’ from O’Reilly and work through that. Unfortunately, there is an error/misprint in the very beginning chapters that makes doing the exercises near impossible on OSX. This error manifests itself as c99: invalid argument 'all' to -W Error 64. If you encounter this error on OSX (I’m using OSX 10.11.2 El Capitan as of writing this post), here are three methods for fixing the issue.

Error 64!

When the discussion of using Makefiles begins on page 15, there is a discussion of the “smallest practicable makefile”, which is just six lines long:

1
2
3
4
5
6
7
P=program_name
OBJECTS=
CFLAGS= -g -Wall -O3
LDLIBS=
CC=c99

$(P): $(OBJECTS)

Unfortunately, this doesn’t quite work on OSX. Page 11 in the book sort-of references that a fix is needed, but the directions aren’t so clear…

Error 64, solution 1: Book Fix, updated

To use the book fix, you are supposed to:

Create a file named c99
Put the lines gcc -std=c99 $\* OR clang $\* in the c99 file
Add the file to your PATH in Terminal (such as export PATH="/Users/computeruser:$PATH" if the c99 file were located in /Users/computeruser directory)
Run chmod +x c99 on the file to make it executable

Once you add this work-around to your PATH, then open a fresh Terminal window (or run source .bash_profile to refresh the Bash settings), you should be able to use Make to compile your C code.

But to be honest, this seems like a really weird “fix” to me, as it overrides the C compiler settings for any program run via Terminal. I prefer one of two alternate solutions.

Error 64, solution 2: Makefile Change

As I was researching this, a helpful Twitter user noted:

@randyzwitch Remove space between CFLAGS and =, and replace c99 with cc. See man c99, -W is not -Wwarnings.

— Eugene Teo (@datajottings) December 30, 2015

When you switch the ‘c99’ reference to just ‘cc’ in the Makefile, everything works fine. Here’s the subtlety different, corrected Makefile:

1
2
3
4
5
6
7
P=program_name
OBJECTS=
CFLAGS= -g -Wall -O3
LDLIBS=
CC=cc

$(P): $(OBJECTS)

Error 64, solution 3: Switch to Clang

The final solution I came across is rather than using the GCC compiler, you can use an alternate compiler called Clang, which is also generally available on OSX (especially with XCode installed). Like solution 2 above, the Makefile is just subtlety different:

1
2
3
4
5
6
7
P=program_name
OBJECTS=
CFLAGS= -g -Wall -O3
LDLIBS=
CC=clang

$(P): $(OBJECTS)

Whether to use GCC or Clang as your compiler is really beyond the scope of this blog post; as 21st Century C discusses, it really shouldn’t matter (especially when you are just learning the mechanics of the language).

Error 64, Be Gone!

There’s not really much more to say at this point; this blog post is mainly documentation for anyone who comes across this error in the future. I’ve continued on through the book using Clang, but suffice to say, it’s not the compiler that writes poor-quality, non-compiling code, it’s the user. Ah, the fun of learning 🙂

Four Tactics For Well Thought Out Business Requirements

Fri, 21 Aug 2015 10:42:13 +0000

@randyzwitch how to get (reasonably) well thought-through requirements from business people?

— Art Webb (@arthurlwebb) August 21, 2015

One of the most common issues in business (especially large corporations) is trying to nail down the requirements for a given analysis request. The “business people” on the front-lines are talking to their higher-ups about what they think are important questions for the business to solve, but by the time the question gets to the analyst or developer, it sounds something like:

It would be interesting to model using SAS how our customers shop for our merchandise by channel and what overlaps there are between demographics, geography, product type and tenure. But we also have to timebox this, we can’t be boiling-the-ocean just looking for needles-in-a-haystack.

Say WHAT? Mr. Business Person, I cannot help you if you do not run that mess through Unsuck-It first.

In all seriousness, I’ve found there are a few great ways for an analyst to refine a “question” like the one above into an actionable plan of attack. So the next time you get a jargon-filled, completely generic analysis request such as the one above, try these four tactics.

1. All Requests Should Be Phrased In The Form Of A Question

The first thing to notice about the mock interaction above is that there are no question marks; it’s not a question! For an analyst or developer to work effectively, questions need to be presented, not bland statements. For example, a refinement series of questions from the analyst might include:

You need a model? What type of model? Do you mean a predictive model, a decision tree for understanding, a PivotTable for you to poke at, a one-page PowerPoint slide to give your boss?
You specified four attributes (demographics, geography, product type and tenure). Do you have a hypothesis around these attributes (or are you just brain-blabbing)?
What is meant by “shop”? Do you mean how do customers browse our goods online and in stores, the purchase cycle, what goods are frequently purchased together or something else?

Note that in all three of the refinement questions above, you are taking a generic idea and really drilling into what is needed. It is the analyst that is the expert in the techniques for analyzing data, so the analyst should be helping the business person to take a raw analysis request and make it into answerable questions.

2. Separate The Tools From The Question

The second thing to notice in the mock interaction above is the statement “using SAS”. I didn’t write that to pick on SAS, but rather, this exact statement was said to me early in my career. I had a boss who would try and guess which tool was appropriate for the question he was asking. I presume that he was trying to gauge how hard he thought the problem was, or try to signal to me how hard he thought the problem was. In the end, a plain SQL query with the results copied into an Excel table was all that was necessary.

As the analyst, confirm whether the tool is actually part of the deliverable. Meaning, if you need to deliver a Tableau workbook, ok, specifying “use Tableau” is an important part of the business question. But if the requirement is “production-quality visualizations”, Tableau may or may not be the right tool or might just be one part of a larger workflow.

3. Every Question Is Interesting To Someone. Solve The Valuable Ones.

Paraphrasing the aphorism “The path to hell is paved with good intentions”, the path to doing low-value work your entire career is answering questions that start ”Wouldn’t it be interesting if…”.

The basis for these statements are often tangents in other meetings, where high-level executives think there is information that should just be available at everyone’s fingertips. But if you were to ask “What business action would you take if you knew this piece of information?” or “Is it worth me stopping a project worth $1 million in Pre-Tax Profit per month to answer this for you?”, you’ll suddenly the question becomes a lot less interesting.

So always have estimates of the business impact of what you are currently working on and ask for the same estimate of those who ask for your time. Projects that are valuable to the business are “interesting”, everything else is just making work for other people.

4. Don’t Just Solved The Stated Question. Solve The Unstated Question Too.

Finally, when I read the mock interaction above, there are actually two questions:

Stated: Do we understand our customer’s purchasing behaviors?
Unstated: How do we optimize our business to take into account our customer’s purchasing behaviors?

For sure, a deep understanding of the customer base is important no matter the product. But the unstated question of “What are we doing to do about it?” is so much more valuable to answer (i.e. tactic #3).

So even if the refined question becomes ’Build a customer segmentation based on past purchases’, go one step further and figure out how to implement your findings. Create a test plan for increasing email click-through-rates based on the segments or optimize your display bidding, maybe build a recommender system for your website…implementation of new ideas is always going to be more valuable than just analyzing the past.

Always Be Assertive.

If the key to sales is “Always Be Closing”, the key to quality analysis is “Always Be Assertive”. Ask questions. Make people think about what they are doing, what they ask of others and what can be done to improve the business. It’s a rare, ego-centric co-worker who doesn’t appreciate collaborating to get to a better quality question (and answer!) than they originally started with.

Being able to read into what other people are asking for, estimating its value, then delivering more than they even knew they were asking for has helped me tremendously throughout my career. Hopefully by doing some or all of the tactics above, you’ll see a marked improvement in your analysis and career as well!

RSiteCatalyst Version 1.4.5 Release Notes

Mon, 17 Aug 2015 09:43:36 +0000

It’s only been a month since the last RSiteCatalyst update, and this update is also a pretty minor update in terms of functionality.

Set Your Own Endpoint

For the overseas users (or companies with weird setups), you can now use the endpoint argument in the SCAuth() function to specify your API endpoint. For the most part, this is not recommended, as RSiteCatalyst pings the Adobe Analytics API to evaluate the proper API endpoint to use, but if for some reason you are having issues, you can override what the Adobe API says.

New Functions

For this release, I briefly looked through the API explorer to see if there were any useful methods that had been missed. GetFunctions (Get definitions of all formula/functions in Adobe Analytics), QueueSummary (Get summary metrics for numerous report suites at once), GetPrivacySettings (Privacy Settings at a report suite level), and GetTemplate (Get template that a current report suite was built from). With the exception of QueueSummary(), none of these functions will likely get you much in the way of additional analytics capabilities, but they are there should you want to use them.

Feature Requests/Bugs

As always, if you come across bugs or have feature requests, please continue to use the RSiteCatalyst GitHub Issues page to submit issues. Don’t worry about cluttering up the page with tickets, please fill out a new issue for anything you encounter (with code you’ve already tried and is failing), unless you are SURE that it is the same problem someone else is facing.

Outside of patching really serious bugs, I will likely not spend any more time improving this package in the future; my interests have changed, and RSiteCatalyst is pretty much complete as far as I’m concerned. That said, contributors are also very welcomed. If there is a feature you’d like added, and especially if you can fix an outstanding issue reported at GitHub, we’d love to have your contributions. Willem and I are both parents of young children and have real jobs outside of open-source software creation, so we welcome any meaningful contributions to RSiteCatalyst that anyone would like to contribute.

JuliaCon 2015: Everyday Analytics and Visualization (video)

Fri, 14 Aug 2015 10:29:41 +0000

At long last, here’s the video of my presentation from JuliaCon 2015, discussion common analytics tasks and visualization. This is really two talks, the first being an example of using the citibike NYC API to analyze ridership of their public bike program, and the second a discussion of the Vega.jl package.

Speaking at JuliaCon 2015 at MIT CSAIL is the professional highlight of my year; hopefully even more of you will attend next year.

Enjoy!

Edit: For those of you who would like to follow-along using the actual presentation code, it is available on GitHub.

CitiBank Bike Data

Vega.jl Presentation

Apple MacBook Pro Model A1286 Declared Vintage - The End Of An Era

Fri, 07 Aug 2015 20:07:12 +0000

It’s hard to believe it’s been over 2.5 years since I wrote about my experience with Apple trying to get my Broken MacBook Pro Hinge fixed. Since that time, my Late 2008 MacBook Pro continued to work flawlessly, most of the time keeping up with the scientific programming I would do in R, Python or Julia.

Unfortunately, it seems near impossible (if not completely impossible) to get an OEM A1281 battery as a drop-in replacement. When I went to the Apple Store at Suburban Square, PA, the “Genius” that looked at my computer took 15-20 minutes to look on the Apple website (which I obviously did before arriving, so no value-add there), only to show me a battery in stock that didn’t fit my model of computer. Only after shaming him into looking up the actual part number, was he able to utter the phrase:

Oh, no, we don’t have those any more. Your model MacBook Pro was declared “Vintage”. No more original parts are available from Apple.

Of course it is. After getting home, I was able to find this service bulletin from Apple, which outlines which models are obsolete. Apparently, it’s a hard and fast rule that once five years from the end of manufacturing arrives, a model is declared vintage (unless local laws require longer service). So even though the only “problem” with my MacBook Pro is that I was only getting one hour of battery life per charge (or less if I’m compiling code), the computer is destined for a new life somewhere else.

“Vintage” For Me, Powerful For Thee

While I realize I could go the 3rd-party route and get a replacement battery, at some point, you can only spend so much money keeping older technology alive. Since I use computers pretty intensively, I ended up getting a “new” (used) 2011 MacBook Pro from a neighborhood listing that has decent life on the OEM battery. Surprisingly, I was able to get $360 for my Late-2008 MacBook Pro, being fully honest about the condition, issues and battery life. The older woman who I sold it to fully understood, but worked at a desk and didn’t care about the battery! She also said:

This is easily the most powerful computer I’ve ever owned.

Apple, like I said in my original post, you’ve got a customer for life. And while I’ve moved on to a newer machine, it’s beyond amazing to me that a 7-year old computer will continue to live on and work at a high level of performance. And with my 2011 MacBook Pro, I still have the option to upgrade the parts (though I don’t need to…SSD, 16GB of RAM and a quad-core i7 processor already!)

The Retina MacBook’s are nice, but very incremental. Here’s hoping the 2011 MacBook Pro lasts as long as my Late 2008 MacBook Pro did!

Authenticated API Testing Using Travis CI

Thu, 06 Aug 2015 20:21:50 +0000

As I’ve become more serious about contributing in the open-source community, having quality tests for my packages has been something I’ve spent much more time on than when I was just writing quick-and-dirty code for my own purposes. My most used open-sourced package is RSiteCatalyst, which accesses the Adobe Analytics (authenticated) API, which poses a problem: how do you maintain a project on GitHub with a full test suite, while at the same time not hard-coding your credentials in plain sight for everyone to see?

The answer ends up being using encrypted environment variables within Travis CI.

Testthat!

In terms of a testing framework, Hadley Wickham provides a great testing framework in testthat; while I wouldn’t go as far as he does to say that the package makes testing fun, it certainly makes testing easy. Let’s take a look at some of the tests in RSiteCatalyst from the QueueOvertime function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
test_that("Validate QueueOvertime using legacy credentials", {

  skip_on_cran()

  #Correct [masked] credentials
  SCAuth(Sys.getenv("USER", ""), Sys.getenv("SECRET", ""))

  #Single Metric, No granularity (summary report)
  aa <- QueueOvertime("zwitchdev",
                      "2014-01-01",
                      "2014-12-31",
                      "visits",
                      "")

  #Validate returned value is a data.frame
  expect_is(aa, "data.frame")

  #Single Metric, Daily Granularity
  bb <- QueueOvertime("zwitchdev",
                      "2014-01-01",
                      "2014-12-31",
                      "visits",
                      "day")

  #Validate returned value is a data.frame
  expect_is(bb, "data.frame")

  #Single Metric, Week Granularity
  cc <- QueueOvertime("zwitchdev",
                      "2014-01-01",
                      "2014-12-31",
                      "visits",
                      "week")

  #Validate returned value is a data.frame
  expect_is(cc, "data.frame")

  #Two Metrics, Week Granularity
  dd <- QueueOvertime("zwitchdev",
                      "2014-01-01",
                      "2014-12-31",
                      c("visits", "pageviews"),
                      "week")

  #Validate returned value is a data.frame
  expect_is(dd, "data.frame")

  #Two Metrics, Month Granularity, Social Visitors
  ee <- QueueOvertime("zwitchdev",
                      "2014-01-01",
                      "2014-12-31",
                      c("visits", "pageviews"),
                      "month",
                      "5433e4e6e4b02df70be4ac63")

  #Validate returned value is a data.frame
  expect_is(ee, "data.frame")

  #Two Metrics, Day Granularity, Social Visitors, Anomaly Detection
  ff <- QueueOvertime("zwitchdev",
                      "2014-01-01",
                      "2014-12-31",
                      c("visits", "pageviews"),
                      "day",
                      "5433e4e6e4b02df70be4ac63",
                      anomaly.detection = "1")

  #Validate returned value is a data.frame
  expect_is(ff, "data.frame")



})

From the code above, you can see the tests are fairly simplistic; for a given number of permutations of arguments of the function, I test to see if a data frame was returned. This is because, for the most part, RSiteCatalyst is just a means of generating JSON calls, submitting them to the Adobe Analytics API, then parsing the results into an R data frame.

Since there is very little additional logic in the package, I don’t spend a bunch of time testing what data is actually returned (i.e. what is returned depends on the Adobe Analytics API, not R). What is interesting is line 6; I reference Sys.getenv() twice in order to pass in my username and key for the Adobe Analytics API, which feels very “interactive R”, but the goal is automated testing. Filling in those two environment variables is where Travis CI comes in.

Travis CI Configuration

In order to have any automation using Travis CI, you need to create a .travis.yml configuration file. While you can read the Travis docs to create the .travis.yml file for R, you’re probably better off just using the use_travis function from devtools (also from Hadley, little surprise!) to create the file for you. In terms of creating encrypted keys to use with Travis, you’ll need to use the Travis CLI tool, which is distributed as a Ruby gem (i.e. package). If you view the RSiteCatalyst .travis.yml file, you can see that I define two global “secure” variables, the value of which are the output from running a command similar to the following in the Travis CLI tool:

$ travis encrypt RANDY=ZWITCH
Please add the following to your .travis.yml file:

  secure: "b6S4dBc7arvox8UpuFqkz+VP2UmAW/S/B/vgaAdZiZQqUp78YDR6VYdAYN3WisCK1VLGjOVVPQvGxLik0pQokF8FU3sjX0ekH6vSJeqg4utrEZmVtNvdDLEVAmagFy8Fyduow3U4CPW7rzXqvAE4cIVqGR5Lv2KLf8ANUGn+y3E="

Pro Tip: You can add it automatically by running with --add.

Note that if this seems insecure, every time you run the encrypt command with the same arguments, you get a different value; Travis CI is creating new public and private RSA keys each time.

Setting Up Authenticated Testing Locally

If you get as far as setting up encrypted Travis CI keys and tests using testthat, the final step is really for convenience. With the .travis.yml file, Travis CI sets the R environment variables on THEIR system; on your local machine, the environment variables aren’t set. Even if the environment variables were set, they would be set to the Travis CI hashed values, which is not what I want to pass to my authentication function in my R package.

To set the authentication variables locally, so that each time you hit ‘check’ to build and check against CRAN errors, you just need to modify the .Renviron file for R:

USER="myusername"
SECRET="mysecret"

With that minor change, in addition to the .travis.yml file, you’ll have a seamless environment for developing and testing R packages.

Testing Is Like Flossing…

As easy as the testthat and devtools packages make testing, and as inexpensively as Travis CI is as a service (free for open source projects!), there’s really no excuse to provide packaged-up code and not include tests. Hopefully this blog post has demonstrated that it’s possible to include tests even when authentication is necessary without compromising your credentials.

So let’s all be sure to include tests, not just pay lip service to the idea that testing is useful. Code testing only works if you actually do it 🙂

Getting Started: Adobe Analytics Clickstream Data Feed

Tue, 04 Aug 2015 09:00:35 +0000

“Well, first you need a TMS and a three-tiered data layer, then some jQuery with a node backend to inject customer data into the page asynchronously if you want to avoid cookie-based limitations with cross-domain tracking and be Internet Explorer 4 compatible…”

Blah Blah Blah. There’s a whole cottage industry around jargon-ing each other to death about digital data collection. But why? Why do we focus on tools, instead of the data? Because the tools are necessarily inflexible, so we work backwards from the pre-defined reports we have to the data needed to populate them correctly. Let’s go the other way for once: clickstream data to analysis & reporting.

In this blog post, I will show the structure of the Adobe Analytics Clickstream Data Feed and how to work with a day worth of data within R. Clickstream data isn’t as raw as pure server logs, but the only limit to what we can calculate from clickstream data is what we can accomplish with a bit of programming and imagination. In later posts, I’ll show how to store a year worth of data in a relational database, storing the same data in Hadoop and doing analysis using modern tools such as Apache Spark.

This blog post will not cover the mechanics of getting the feed delivered via FTP. The Adobe Clickstream Feed documentation is sufficiently clear in how to get started.

FTP/File Structure

Once your Adobe Clickstream Feed starts being delivered via FTP, you’ll have a file listing that looks similar to the following:

What you’ll notice is that with daily delivery, three files are provided, each having a consistent file naming format:

\d+-\S+_\d+-\d+-\d+.tsv.gz

This is the main file containing the server call level data

\S+_\d+-\d+-\d+-lookup_data.tar.gz

These are the lookup tables, header files, etc.

\S+_\d+-\d+-\d+.txt

Manifest file, delivered last so that any automated processes know that Adobe is finished transferring

The regular expressions will be unnecessary for working with our single day of data, but it’s good to realize that there is a consistent naming structure.

Checking md5 hashes

As part of the manifest file, Adobe provides md5 hashes of the files. There are at least two purposes to this, including 1) making sure that the files truly were delivered in full and 2) that the files haven’t been manipulated/tampered with. In order to check that your md5 hashes match the values provided by Adobe, we can do the following in R:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
setwd("~/Downloads/datafeed/")

#Read in Adobe manifest file
manifest <- read.table("zwitchdev_2015-07-13.txt", stringsAsFactors=FALSE)
names(manifest) <- c("key", "value")

#Use digest library to calculate md5 hashes
library(digest)
servercalls_md5 <- digest("01-zwitchdev_2015-07-13.tsv.gz", algo="md5", file=TRUE)
lookup_md5 <- digest("zwitchdev_2015-07-13-lookup_data.tar.gz", algo="md5", file=TRUE)

#Check to see if hashes contained in manifest file
servercalls_md5 %in% manifest$value #[1] TRUE
lookup_md5 %in% manifest$value #[1] TRUE

As we can see, both calculated hashes are contained within the manifest, so we can be confident that the files we downloaded haven’t been modified.

Unzipping and Loading Raw Files to Data Frames

Now that our file hashes are validated, it’s time to load the files into R. For the example files, I would be able to fit the entire day into RAM because my blog does very little traffic. However, I’m going to still limit the rows brought in, as if we were working with a large e-commerce website with millions of visits per day:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#Get list of lookup files from tarball
files_tar <- untar("zwitchdev_2015-07-13-lookup_data.tar.gz", list = TRUE)

#Extract files to _temp directory. Directory will be created if it doesn't exist
untar("zwitchdev_2015-07-13-lookup_data.tar.gz", exdir = "_temp")

#Read each file into a data frame
#If coding like this in R offends you, keep it to yourself...
for (file in files_tar) {
  df_name <- unlist(strsplit(file, split = ".tsv", fixed = TRUE))
  temp_df <- read.delim(paste("_temp", file, sep = "/"), header=FALSE, stringsAsFactors=FALSE)
  #column_headers not used as lookup table
  if (df_name != "column_headers"){
    names(temp_df) <- c("id", df_name)
  }
  assign(df_name, temp_df)
  rm(temp_df)
}

#gz files can be read directly into dataframes from base R
#Could also use `readr` library for performance
servercall_data <- read.delim("~/Downloads/datafeed/01-zwitchdev_2015-07-13.tsv.gz",
                       header=FALSE, stringsAsFactors=FALSE, nrows = 500)

#Use column_headers to label servercall_data data frame using first row of data
names(servercall_data) <- column_headers[1,]

If we were to be loading this data into a database, we’d be done with our processing; we have all of our data read into R and it would be a trivial exercise to load the data into a database (we’ll do this in a separate blog post). But since we’re going to be analyze this single day of clickstream data, we need to join these 14 data frames together.

SQL: The Most Important Language for Analytics

As a slight tangent, if you don’t know SQL, then you’re going to have a really hard time doing any sort of advanced analytics. There are literally millions of tutorials on the Internet (including this one from me), and understanding how to join and retrieve data from databases is the key to being more than just a report monkey.

The reason why the prior code creates 14 data frames is because the data is delivered in a normalized structure from Adobe. Now we are going to de-normalize the data, which is just a fancy way of saying “join the files together in order to make a gigantic table.”

There are probably a dozen different ways to join data frames using just R code, but I’m going to do it using the sqldf package so that I can use SQL. This will allow for a single, declarative statement that shows the relationship between the lookup and fact tables:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
library(sqldf)

query <-
"select
sc.*,
browser.browser as browser_name,
browser_type,
connection_type.connection_type as connection_name,
country.country as country_name,
javascript_version,
languages.languages as languages,
operating_systems,
referrer_type,
resolution.resolution as screen_resolution,
search_engines
from servercall_data as sc
left join browser on sc.browser = browser.id
left join browser_type on sc.browser = browser_type.id
left join connection_type on sc.connection_type = connection_type.id
left join country on sc.country = country.id
left join javascript_version on sc.javascript = javascript_version.id
left join languages on sc.language = languages.id
left join operating_systems on sc.os = operating_systems.id
left join referrer_type on sc.ref_type = referrer_type.id
left join resolution on sc.resolution = resolution.id
left join search_engines on sc.post_search_engine = search_engines.id
;
"

denormalized_df <- sqldf(query)

There are three lookup tables that weren’t used: color_depth, plugins and event. The first two don’t have a lookup column in my data feed (click link for a full listing of Adobe Clickstream data feed columns available). These columns aren’t really useful for my purposes anyway, so not a huge loss. The third table, the event list, requires a separate processing step.

Processing Event Data

As normalized as the Adobe Clickstream Data Feed is, there is one oddity: the events per server call come in a comma-delimited string in a single column with a lookup table. This implies that a separate level of processing is necessary, outside of SQL, since the column “key” is actually multiple keys and the lookup table specifies one event type per row. So if you were to try and join the data together, you wouldn’t get any matches.

To deal with this in R, we are going to do an EXTREMELY wasteful operation: we are going to create a data frame with a column for each possible event, then evaluate each row to see if that event occurred. This will use a massive amount of RAM, but of course, this is a feature/limitation of R which wouldn’t be an issue if the data were stored in a database.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#Create friendly names in events table replacing spaces with underscores
event$names_filled <- tolower(gsub(" ", "_", event$event))

#Initialize a data frame with all 0 values
#Dimensions are number of observations as rows, with a column for every possible event
event_df <- data.frame(matrix(data = 0, ncol = nrow(event), nrow = nrow(servercall_data)))
names(event_df) <- event$id

#Parse comma-delimited string into vector
#Each vector value represents column name in event_df, assign value of 1
for(row in servercall_data$post_event_list){
    if(!is.na(row)){
        for(event_ in strsplit(row, ",")){
          event_df[as.character(event_)] <- 1
        }
    }
}

#Rename columns with "friendly" names
names(event_df) <- event$names_filled

#Horizontally join datasets to create final dataset
oneday_df <- cbind(denormalized_df, event_df)

With the final cbind command, we’ve created a 500 row x 1562 column dataset representing a sample of rows from one day of the Adobe Clickstream Data Feed. Having the data denormalized in this fashion takes 6.13 MB of RAM…extrapolating to 1 million rows, you would need 12.26GB of RAM (per day of data you want to analyze, if stored solely in memory).

Next Step: Analytics?!

A thousand words in and 91 lines of R code and we still haven’t done any actual analytics. But we’ve completed the first step in any analytics project: data prep!

In future blog posts in this series, I’ll demonstrate how to actually use this data in analytics, from re-creating reports available in the Adobe Analytics UI (to prove the data is the same) to more advanced analysis such as using association rules, which can be one method for creating a “You may also like…” functionality such as the one at the bottom of this blog.

Example Files:

RSiteCatalyst Version 1.4.4 Release Notes

Mon, 13 Jul 2015 09:40:20 +0000

It’s been about six months since the last RSiteCatalyst update, and this update is really just a single bug fix, but a big bug fix at that!

Sparse Data = Opaque Error Messages

Numerous people have reported receiving an error message from RSiteCatalyst similar to the following:

‘names’ attribute [1] must be the same length as the vector [0]

This is about the least helpful message that could’ve been returned, but it was an R message indicating an internal function trying to overwrite the column names vector (which had non-zero length) with a vector of length zero (which is an error in the context of a data frame). Thankfully, Willem Paling was able to squash this bug (hopefully) once-and-for-all; the error occurs when a user tries to do a Queue*` report with multiple breakdowns, where NULL data is returned by the Adobe API for one of the breakdowns.

So hopefully, if you’ve run into this error before (which I have to imagine was quite frustrating), you shouldn’t see this again with v1.4.4 of RSiteCatalyst. Additionally, tests will be added to the test suite to attempt to trigger this warning, so that this horrible monster of a bug doesn’t appear again.

Authentication Messaging

The only other change of substance was to modify the message returned after calling SCAuth(); some users were having issues with API calls not working, after RSiteCatalyst having returned 'Authentication Succeeded' to the console. RSiteCatalyst never actually validates that your credentials are correct, just that they are stored within the session. The console message has been updated to reflect this.

Proper Punctuation Prevents Poor Documentation!

The Eagle-Eyed among you might have noticed that my DESCRIPTION file was out of CRAN spec for many months. This has now been fixed, so that the meaning is as clear as possible.

Feature Requests/Bugs

Contributors are also very welcomed. If there is a feature you’d like added, and especially if you can fix an outstanding issue reported at GitHub, we’d love to have your contributions. Willem and I are both parents of young children and have real jobs outside of open-source software creation, so we welcome any meaningful contributions to RSiteCatalyst that anyone would like to contribute.

Vega.jl, Rebooted

Thu, 21 May 2015 12:56:07 +0000

Mmmmm, baked goods!

Rebooting Vega.jl

Recently, I’ve found myself without a project to hack on, and I’ve always been interested in learning more about browser-based visualization. So I decided to revive the work that John Myles White had done in building Vega.jl nearly two years ago. And since I’ll be giving an analytics & visualization workshop at JuliaCon 2015, I figure I better study the topic in a bit more depth.

Back In Working Order!

The first thing I tackled here was to upgrade the syntax to target v0.4 of Julia. This is just my developer preference, to avoid using Compat.jl when there are so many more visualizations I’d like to support. So if you’re using v0.4, you shouldn’t see any deprecation errors; if you’re using v0.3, well, eventually you’ll use v0.4!

Additionally, I modified the package to recognize the traction that Jupyter Notebook has gained in the community. Whereas the original version of Vega.jl only displayed output in a tab in a browser, I’ve overloaded the writemime method to display :VegaVisualization inline for any environment that can display HTML. If you use Vega.jl from the REPL, you’ll still get the same default browser-opening behavior as existed before.

The First Visualization You Added Was A Pie Chart…

…And Followed With a Donut Chart?

Yup. I’m a troll like that. Besides, being loudly against pie charts is blowhardy (even if studies have shown that people are too stupid to evaluate them).

Adding these two charts (besides trolling) was a proof-of-concept that I understood the codebase sufficiently in order to extend the package. Now that the syntax is working for Julia v0.4, I understand how the package works (important!), and have improved the workflow by supporting Jupyter Notebook, I plan to create all of the visualizations featured in the Trifacta Vega Editor and other standard visualizations such as boxplots. If the community has requests for the order of implementation, I’ll try and accommodate them. Just add a feature request on Vega.jl GitHub issues.

Why Not Gadfly? You’re Not Starting A Language War, Are You?

No, I’m not that big of a troll. Besides, I don’t think we’ve squeezed all the juice (blood?!) out of the R vs. Python infographic yet, we don’t need another pointless debate.

My sole reason for not improving Gadfly is just that I plain don’t understand how the codebase works! There are many amazing computer scientists & developers in the Julia community, and I’m not really one of them. I do, however, understand how to generate JSON strings and in that sense, Vega is the perfect platform for me to contribute.

Collaborators Wanted!

If you’re interested in visualization, as well as learning Julia and/or contributing to a package, Vega.jl might be a good place to start. I’m always up for collaborating with people, and creating new visualizations isn’t that difficult (especially with the Trifacta examples). So hopefully some of you will be interested in enough to join me to adding one more great visualization library to the Julia community.

Sessionizing Log Data Using data.table [Follow-up #2]

Tue, 20 Jan 2015 09:01:19 +0000

Thanks to user dnlbrky, we now have a third way to accomplish sessionizing log data for any arbitrary time out period (see methods 1 and 2), this time using data.table from R along with magrittr for piping:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
library(magrittr)
library(data.table)

## Download, unzip, and load data (first 10,000 lines):
single_col_timestamp <- url("http://randyzwitch.com/wp-content/uploads/2015/01/single_col_timestamp.csv.gz") %>%
  gzcon %>%
  readLines(n=10000L) %>%
  textConnection %>%
  read.csv %>%
  setDT

## Convert to timestamp:
single_col_timestamp[, event_timestamp:=as.POSIXct(event_timestamp)]

## Order by uid and event_timestamp:
setkey(single_col_timestamp, uid, event_timestamp)

## Sessionize the data (more than 30 minutes between events is a new session):
single_col_timestamp[, session_id:=paste(uid, cumsum((c(0, diff(event_timestamp))/60 > 30)*1), sep="_"), by=uid]

## Examine the results:
#single_col_timestamp[uid %like% "a55bb9"]
single_col_timestamp[session_id %like% "fc895c3babd"]

I agree with dnlbrky in that this feels a little better than the dplyr method for heavy SQL users like me, but ultimately, I still think the SQL method is the most elegant and obvious to understand. But that’s the great thing with open-source software; pick any tool you want, accomplish whatever you choose using any method you choose.

Sessionizing Log Data Using dplyr [Follow-up]

Tue, 13 Jan 2015 16:24:52 +0000

Last week, I wrote a blog post showing how to sessionize log data using standard SQL. The main idea of that post is that if your analytics platform supports window functions (like Postgres and Hive do), you can make quick work out of sessionizing logs. Here’s the winning query:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
select
uid,
sum(new_event_boundary) OVER (PARTITION BY uid ORDER BY event_timestamp) as session_id,
event_timestamp,
minutes_since_last_interval,
new_event_boundary
from
			--Query 1: Define boundary events
			(select
			uid,
			event_timestamp,
			(extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp))/60 as minutes_since_last_interval,
			case when extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp) > 30 * 60 then 1 ELSE 0 END as new_event_boundary
			from single_col_timestamp
			) a;

One nested sub-query and two window functions are all it takes to calculate the event boundaries and create a unique identifier for sessions for any arbitrary timeout chosen.

It’s Hadley’s House, We’re Just Leasing

Up until today, I hadn’t really done anything using dplyr. But having a bunch of free time this week and hearing people talk so much about how great dplyr is, I decided to see what it would take to replicate this same exercise using R. dplyr has support for Postgres as a back-end, and has verbs that translate R code into window functions, so I figured it had to be possible. Here’s what I came up with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
###Sessionization using dplyr

library(dplyr)

#Open a localhost connection to Postgres
#Use table 'single_col_timestamp'
#group by uid and sort by timestamp for window function
#Do minutes calculation, working around missing support for extract(epoch from timestamp)
#Calculate event boundary and unique id via cumulative sum window function
sessions <-  
        src_postgres("logfiles") %>%
        tbl("single_col_timestamp") %>%
        group_by(uid) %>%
        arrange(event_timestamp) %>%
        mutate(minutes_since_last_event = (
                                           DATE_PART('day', event_timestamp - lag(event_timestamp)) * 24 +
                                           DATE_PART('hour', event_timestamp - lag(event_timestamp)) * 60 +
                                           DATE_PART('minute', event_timestamp - lag(event_timestamp)) * 60 +
                                           DATE_PART('second', event_timestamp - lag(event_timestamp))
                                           ) / 60
              ) %>%
        mutate(event_boundary = if(minutes_since_last_event > 30) 1 else 0,
               session_id = order_by(event_timestamp, cumsum(if(minutes_since_last_event > 30) 1 else 0)))

#Show query syntax
show_query(sessions)

#Actually run the query
answer <- collect(sessions)

Generally, I’m not a fan of the pipe operator, but I figured I’d give it a shot since everyone else seems to like it. This is one nasty bit of R code, but ultimately, it is possible to get the same result as writing SQL directly. I did need to take a few roundabout ways, specifically in calculating the minutes between timestamps and substituting the CASE expression into the window function rather than call it by name, but it’s basically the same logic.

Why Does This Work?

If you compare the SQL code above to the R code, you might be wondering why the dplyr code works. Certainly, working the dplyr way gives me cognitive dissonance, as you generally specify the verbs you are using in reverse order as you do in SQL. But calling show_query(sessions), you actually see that dplyr is generating SQL under-the-hood (I formatted the code for easier viewing):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
SELECT
	"uid",
	"event_timestamp",
	"minutes_since_last_event",
	"event_boundary",
	"session_id"
FROM (
		SELECT
			"uid",
			"event_timestamp",
			"minutes_since_last_event",
			CASE WHEN "minutes_since_last_event" > 30.0 THEN 1.0 ELSE 0.0 END AS "event_boundary",
			sum(CASE WHEN "minutes_since_last_event" > 30.0 THEN 1.0 ELSE 0.0 END) OVER (PARTITION BY "uid" ORDER BY "event_timestamp" ROWS UNBOUNDED PRECEDING) AS "session_id"
		FROM
			(
				SELECT
					"uid",
					"event_timestamp",
					(DATE_PART('day', "event_timestamp" - LAG("event_timestamp", 1, NULL) OVER (PARTITION BY "uid" ORDER BY "event_timestamp")) * 24.0
						+ DATE_PART('hour', "event_timestamp" - LAG("event_timestamp", 1, NULL) OVER (PARTITION BY "uid" ORDER BY "event_timestamp")) * 60.0
						+ DATE_PART('minute', "event_timestamp" - LAG("event_timestamp", 1, NULL) OVER (PARTITION BY "uid" ORDER BY "event_timestamp")) * 60.0
						+ DATE_PART('second', "event_timestamp" - LAG("event_timestamp", 1, NULL) OVER (PARTITION BY "uid" ORDER BY "event_timestamp"))) / 60.0 AS "minutes_since_last_event"
				FROM "single_col_timestamp"
				ORDER BY "uid", "event_timestamp"
			) AS "_W1"
	) AS "_W2"

Like all SQL-generating tools, the code is a bit inelegant; however, I have to say that I’m truly impressed the dplyr code was able to handle this scenario at all, given that this example has to be at least an edge-, if not a corner-case of what dplyr is meant for in terms of data manipulation.

So, dplyr Is Going To Become Part Of Your Toolbox?

While it was possible to re-create the same functionality, ultimately, I don’t see myself using dplyr a whole lot. In the case of using databases, it seems more efficient and portable just to write the SQL directly; at the very least, it’s what I’m already comfortable doing as part of my analytics workflow. For manipulating data frames, maybe I’d use it (I do use plyr extensively in my RSiteCatalyst package), but I’d probably be more inclined to use sqldf instead.

But that’s just me, not a reflection on the package quality. Happy manipulating, however you choose to do it! 🙂

Sessionizing Log Data Using SQL

Thu, 08 Jan 2015 11:57:56 +0000

Over my career as a predictive modeler/data scientist, the most important step(s) in any data project without question have been data cleaning and feature engineering. By taking the data you have, correcting flaws and reformulating raw data into additional business-specific concepts, you ensure that you move beyond pure mathematical optimization and actually solve a business problem. While “big data” is often held up as the future of knowing everything, when it comes down to it, a Hadoop cluster is more often a “Ha-dump” cluster: the place data gets dumped without any proper ETL.

For this blog post, I’m going to highlight a common request for time-series data: combining discrete events into sessions. Whether you are dealing with sensor data, television viewing data, digital analytics data or any other stream of events, the problem of interest is usually how a human interacts with a machine over a given period of time, not each individual event.

While I usually use Hive (Hadoop) for daily work, I’m going to use Postgres (via OSX Postgres.app) to make this as widely accessible as possible. In general, this process will work with any infrastructure/SQL-dialect that supports window functions.

Connecting to Database/Load Data

For lightweight tasks, I find using psql (command-line tool) is easy enough. Here are the commands to create a database to hold our data and to load our two .csv files (download here and here):

These files contain timestamps generated for 1000 uid values.

Query 1 (“Inner”): Determining Session Boundary Using A Window Function

In order to determine the boundary of each session, we can use a window function along with lag(), which will allow the current row being processed to compare vs. the prior row. Of course, for all of this to work correctly, we need to have our data sorted in time order by each of our users:

1
2
3
4
5
6
7
--Create boundaries at 30 minute timeout
select
uid,
event_timestamp,
(extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp))/60 as minutes_since_last_interval,
case when extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp) > 30 * 60 then 1 ELSE 0 END as new_event_boundary
from single_col_timestamp;

For this query, we use the lag() function on the event_timestamp column, and we use over partition by uid order by event_timestamp to define the window over which we want to do our calculation. To provide additional clarification about how this syntax works, I’ve added a column showing how many minutes have passed between intervals to validate that the 30-minute window is calculated correctly. The result is as follows:

For each row where the value of minutes_since_last_interval > 30, there is a value of 1 for new_event_boundary.

Query 2 (“Outer”): Creating A Session ID

The query above defines the event boundaries (which is helpful), but if we want to calculate session-level metrics, we need to create a unique id for each set of rows that are part of one session. To do this, we’re again going to use a window function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
select
uid,
sum(new_event_boundary) OVER (PARTITION BY uid ORDER BY event_timestamp) as session_id,
event_timestamp,
minutes_since_last_interval,
new_event_boundary
from
			--Query 1: Define boundary events
			(select
			uid,
			event_timestamp,
			(extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp))/60 as minutes_since_last_interval,
			case when extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp) > 30 * 60 then 1 ELSE 0 END as new_event_boundary
			from single_col_timestamp
			) a;

This query defines the same over partition by uid order by event_timestamp window, but rather than using lag() this time, we’re going to use sum() for the outer query. The effect of using sum() in our window function is to do a cumulative sum; every time 1 shows up, the session_id field gets incremented by 1. If there is a value of 0, the sum is still the same as the row above and thus has the same session_id. This is easier to understand visually:

At this point, we have a session_id for a group of rows where there have been no 30 minute gaps in behavior.

Final Query: Cleaned Up

Although the previous section is technically done, I usually concatenate the uid and session_id together. I do this concatenation just to highlight that the value is usually a ‘key’ value, not a metric in itself (though it can be). Concatenating the keys together and removing the teaching columns results in the following query:

1
2
3
4
5
6
7
8
9
10
11
12
13
--Query 3:  Outer query uses window function with sum to do cumulative sum as the id, concatentate to uid
select
uid,
uid || '-' || cast(sum(new_event_boundary) OVER (PARTITION BY uid ORDER BY event_timestamp) as varchar) as session_id,
event_timestamp
from
			--Query 1: Define boundary events
			(select
			uid,
			event_timestamp,
			case when extract(epoch from event_timestamp) - lag(extract(epoch from event_timestamp)) OVER (PARTITION BY uid ORDER BY event_timestamp) > 30 * 60 then 1 ELSE 0 END as new_event_boundary
			from single_col_timestamp
			) a;

Window Functions, Will You Marry Me?

The first time I was asked to try and solve sessionization of time-series data using Hive, I was sure the answer would be that I’d have to get a peer to write some nasty custom Java code to be able generate unique ids; in retrospect, the solution is so obvious and simple that I wish I would’ve tried to do this years ago. This is a pretty easy problem to solve using imperative programming, but if you’ve got a gigantic amount of hardware in a RDBMS or Hadoop, SQL takes care of all of the calculation without needing think through looping (or more complicated logic/data structures).

Window functions fall into a weird space in the SQL language, given that they allow you to do sequential calculations when SQL should generally be thought of as “set-level” calculations (i.e. no implied order and table-wide calculations vs. row/state-specific). But now that I’ve got a hang of them, I can’t imagine my analytical life without them.

RSiteCatalyst Version 1.4.3 Release Notes

Tue, 06 Jan 2015 13:00:40 +0000

It’s a new year, so…new version of RSiteCatalyst on CRAN! For the most part, this release fixes a handful of bugs that weren’t noticed with the prior release 1.4.2 (oops!), but there are pieces of additional functionality.

New functionality: Data Feed monitoring

For those of you having hourly or daily data feeds delivered via FTP, you can now find out the details of a data feed and all of a company’s feeds & the processing status of each using GetFeed() and GetFeeds() respectively.

For example, calling GetFeed() with a specific feed number will return the following information as a data frame:

Similarly, if you call GetFeeds("report-suite"), you’ll get the following information as a data frame:

I only have one feed set up for testing, but if there were more feeds delivered each day, they would show up as additional rows in the data frame. The interpretation here is that the daily feed for 1/5/15 was delivered (the 05:00:00 is GMT).

Bug Fixes

RSiteCatalyst v1.4.2 attempted to fix an issue where QueueRanked would error if two SAINT classifications were used. Unfortunately, by fixing that issue, QueueRanked ONLY worked with SAINT Classifications. This was only out in the wild for a month, so hopefully it didn’t really affect anyone.

Additionally, the segment.id and segment.name weren’t printing out to the data frame in the Queue* functions. This has also been fixed.

Test Suite Using Travis CI

To avoid future errors like the ones mentioned above, a full test suite using testthat has been added to RSiteCatalyst and monitored via Travis CI. While there is coverage for every public function within the package, there are likely additional tests that can be added for functionality I didn’t cover. If anyone out there has particularly weird cases they use and aren’t incorporated in the test suite, please feel free to file an issue or submit a pull request and I’ll figure out how to incorporate it into the test suite.

DataWarehouse API

Finally, the last bit of changes to RSiteCatalyst in v1.4.3 are internal preparations for a new package I plan to release in the coming months: AdobeDW. Several folks have asked for the ability to control Data Warehouse reports via R; for various reasons, I thought it made sense to break this out from RSiteCatalyst into its own package. If there are any R-and-Adobe-Analytics enthusiasts out there that would like to help development, please let me know!

Feature Requests/Bugs

And finally, like I end every blog post about RSiteCatalyst, please note that I’m not an Adobe employee. This hasn’t been an issue for a few months, so maybe next time I won’t end the post with this boilerplate :)

Review: Data Science at the Command Line

Mon, 15 Dec 2014 10:22:46 +0000

Admission: I didn’t really know how computers worked until around 2012.

For the majority of my career, I’ve worked for large companies with centralized IT functions. Like many statisticians, I fell into a comfortable position of learning SAS in a Windows environment, had Ops people to fix any Unix problems I’d run into and DBAs to load data into a relational database environment.

Then I became a consultant at a boutique digital analytics firm. To say I was punching above my weight was an understatement. All of the sudden it was time to go into various companies, have a one-hour kickoff meeting, then start untangling the spaghetti mess that represented their various technology systems. I also needed to figure out the boutique firm’s hacked together AWS and Rackspace infrastructure.

I’m starting off this review with this admission, because my story of learning to work from the command line parallels Data Science at the Command Line author Jeroen Janssens:

Around five years ago, during my PhD program, I gradually switched from using Microsoft Windows to GNU/Linux…Out of necessity I quickly became comfortable using the command line. Eventually, as spare time got more precious, I settled down with a GNU/Linux distribution known as Ubuntu…

Preface, pg. xi

Because a solid majority of people have never learned anything beyond point-and-click interface (Windows or Mac), the title of the book Data Science at the Command Line is somewhat unfortunate; this is a book for ANYONE looking to start manipulating files efficiently from the command line.

Getting Started, Safely

One of the best parts of Data Science at the Command Line is that it comes with a pre-built virtual machine with 80-100 or more command line tools installed. This is a very fast and safe way to get started with the command line, as the tools are pre-installed and no matter what command you run while you’re learning, you won’t destroy a computer you actually care about!

Chapters 2 and 3 move through the steps of installing the virtual machine, explaining the essential concepts of the command line, some basic commands showing simple (but powerful!) ways to chain command line tools together and how to obtain data. What I find so refreshing about these two chapters by Janssens is that the author assumes zero knowledge of the command line by the reader; these two chapters are the most accessible summary of how and why to use the command line I’ve ever read (Zed Shaw’s CLI tutorial is a close second, but is quite terse).

The OSEMN model

The middle portion of book covers the OSEMN model (Obtain-Scrub-Explore-Model-iNterpret) of data science; another way this book is refreshing is that rather than jump right into machine learning/predictive modeling, the author spends a considerable amount of time covering the gory details of real analysis projects: manipulating data from the format you receive (XML, JSON, sloppy CSV files, etc.) and taking the (numerous) steps required to get the format you want.

By introducing tools such as csvkit (csv manipulation), jq (JSON processor), and classic tools such as sed (stream editor) and (g)awk, the reader gets a full treatment of how to deal with malformed data files (which in my experience are the only type available in the wild!) . Chapter 6 (“Managing Your Data Workflow”) is also a great introduction into reproducible research using Drake (Make for Data Analysis). This is an area that I will personally be focusing my time on, as I tend to run a lot of one-off commands in HDFS and as of now, just copy them into a plain-text file. Reproducing = copy-paste in my case, which defeats the purpose of computers and scripting!

An Idea Can Be Stretched Too Far

Chapters 8 and 9 cover Parallel Processing using GNU Parallel and Modeling Data respectively. While GNU Parallel is a tool I could see using sometime in the future, I do feel like building models and creating visualizations straight from the command line is getting pretty close to just being a parlor trick. Yes, it’s obviously possible to do such things (and the author even wrote his own command line tool Rio for using R from the command line), but with the amount of iteration, feature building and fine-tuning that goes on, I’d rather use IPython Notebook or RStudio to give me the flexibility I need to really iterate effectively.

A Book For Everyone

As I mentioned above, I really feel that Data Science at the Command Line is a book well suited for anyone who does data analysis. Jeroen Janssens has done a fantastic job of taking his original “7 command-line tools for data science” blog post and extending the idea to a full-fledged book. This book has a prominent place in my work library next to Python for Data Analysis and in the past two months I’ve referred to each book at roughly the same rate. For under $30 for paperback at Amazon, there’s more than enough content to make you a better data scientist.

Introducing Twitter.jl

Mon, 08 Dec 2014 17:12:58 +0000

This is possibly the latest “announcement” of a package ever, given that Twitter.jl has existed on METADATA for nearly a year now, but that’s how things go sometimes. Here’s how to get started with Twitter.jl.

Hello, World!

If ‘Hello, World!’ is the canonical example of getting started with a programming language, the Twitter API is becoming the first place to start for people wanting to learn about APIs. Authenticating with the Twitter API using Julia is similar to using the R or Python packages, except that rather than doing the OAuth “dance”, Twitter.jl takes all four authentication values in one function:

1
2
3
4
5
6
7
8
9
10
11
using Twitter

apikey = "q8Qw7WJTVP..."
apisecret = "FIichPpGJxiOssN..."
accesstoken = "98689850-v0zZNr..."
accesstokensecret = "w7bDg9K0c493T..."

twitterauth(apikey,
            apisecret,
            accesstoken,
            accesstokensecret)

All four of these values can be found after registering at the Twitter Developer page and creating an application. Having all four values in your script is less secure than just providing the api key and api secret, but in the future, I’ll likely implement the full OAuth “handshake”. One thing to keep in mind with this function as it currently works is that no validation of your credentials is performed; the only thing this function does is define a global variable twittercred for later use by the various functions that create the OAuth headers. To shout “Hello, World!” to all of your Twitter followers, you can use the following code:

1
post_status_update("Hello, World!")

General Package/Function Structure

From the example above, you can see that the function naming follows the Twitter REST API naming convention, with the HTTP verb first and the endpoint as the remainder of the function name. As such, it’s a good idea at this early package state to have the Twitter documentation open while using this package, so that you can quickly find the methods you are looking for.

For each function/API endpoint, I’ve gone through and determined which parameters are required; these are required arguments in the Julia functions. For all other options, each function takes a second optional Dict{String, String} for any option shown in the Twitter documentation. While this Dict structure allows for ultimate flexibility (and quick definition of functions!), I do realize that it’s less than optimal that you don’t know what optional arguments each Twitter endpoint allows.

As an example, suppose you wanted to search for tweets containing the hashtag #julialang. The minimum function call is as follows:

1
julia_tweets = get_search_tweets("#julialang")

By default, the API will return the 15 most recent tweets containing the #julialang hashtag. To return the most recent 100 tweets (the maximum per API ‘page’), you can pass the “count” parameter via the Options Dict:

julia_tweets_100 = get_search_tweets("#julialang"; options = {"count" => "100"})

Composite Types and DataFrames definitions

The Twitter API is structured into 4 return data types (Places, Users, Tweets, and Entities), and I’ve mimicked these types using Julia Composite Types. As such, most functions in Twitter.jl return an array of specific type, such as Array{TWEETS,1} from the prior #julialang search example. The benefit to defining custom types for the returned Twitter data is that rudimentary DataFrame methods have also been defined:

1
df = DataFrame(julia_tweets_100)

I describe these DataFrames as ‘rudimentary’ as they parse the top level of JSON into columns, which results in some DataFrame columns having complex data types such as Dict() (and within the Dict(), nested Dicts!). As a running theme in this post, this is something I hope to get around to improving in the future.

Want to Get Started Developing Julia? Start Here!

One of the common questions I get asked is how to get started with Julia, both from a learning perspective and from a package development perspective. Hacking away on the core Julia codebase is great if you have the ability, but the code can certainly be intimidating (the people are quite friendly though). Creating a package isn’t necessarily hard, but you have to think about an idea you want to implement. The third alternative is…

…improve the Twitter package! If you go to the GitHub page for Twitter.jl, you’ll see a long list of TODO items that need to be worked on. The hardest part (building the OAuth headers) has already been taken care of. What’s left is re-factoring the code for simplification, factoring out the OAuth code in general into a new Julia library (also partially started), then building the Streaming API functions, cleaning up the DataFrame methods to remove the Dict column types, paging through API results…and so-on.

So if any of you are on the sidelines wanting to get some practice on developing packages, without needing to worry about learning Astrophysics first, I’d love to collaborate. And if any Julia programming masters want to collaborate, well that’s great too. All help and pull requests are welcomed.

In the meantime, hopefully some of you will find this package useful for natural language processing, social networking analysis or even creating bots 😉

RSiteCatalyst Version 1.4.2 Release Notes

Wed, 03 Dec 2014 23:01:05 +0000

RSiteCatalyst version 1.4.2 is now available on CRAN. This update was primarily bug fixes with one additional feature added.

Fixed QueueRanked function to allow multiple SAINT classifications to be specified. This allows for breaking down a SAINT classification with another SAINT classification, such as breaking down tracking codes by marketing channel and by campaign
Fixed bug in internal function, to allow for using the same element multiple times in a QueueRanked function call. This was a necessary fix for allowing multiple SAINT classifications in #1
Exported previous internal function SubmitJsonQueueReport to allow for submitting JSON requests directly to the Adobe Analytics API without all of the R function scaffolding. This approximates the same functionality as the Adobe API Explorer.

For the most part, this isn’t a release that most people will notice any differences from version 1.4.1. That said, special thanks go out to Jason Morgan (@framingeinstein) for identifying the two bugs that were fixed AND submitting fixes.

Feature Requests/Bugs

And finally, like I end every blog post about RSiteCatalyst, please note that I’m not an Adobe employee. Please don’t send me your API credentials, expect immediate replies (especially for you e-commerce folks sweating the holiday season!) or ask to set up phone calls to troubleshoot your problems. This is open-source software…Willem Paling and I did the hard part writing it, you’re expected to support yourself as best as possible unless you believe you’re encountering a bug. Then use GitHub.

Destroy Your Data Using Excel With This One Weird Trick!

Thu, 20 Nov 2014 10:07:55 +0000

All you pie-chart haters are wishing I used one here.

I often use Twitter as a place to vent about the horribleness of Excel, from the product itself to analyses its UI and workflow influences. Admittedly, some of this is snobbish preference: if everyone used my preferred tools, then the world would be a better place! But let me back off my snobbishness a bit and just say this: please feel free to use any tool you want, up to and including pencil-and-paper…JUST.STOP.USING.EXCEL.

Excel arbitrarily destroys data for fun, as evidenced by the example below.

Who Gives A ‘F’ About Seconds? I’m 10 minutes Late Everywhere!

CSV files have many flaws, but at least they are just plain text. It doesn’t take any special software to read them and you can open and close them without loss of fidelity…except if you open them with Excel.

Suppose you have a CSV file with timestamps in ISO8601 format. Depending on which text editor you use, it might look something like this:

Now, let’s open our file in Excel:

The first thing you might notice is that not only does Excel change the date formatting in the file to be more “‘Murica!”, they don’t even have the courtesy to use one of their existing date or time formats! And rather than keep the date the way it was, or standardize the dates to the way the rest of the world writes them, or even keep fixed-width columns, Excel feels like it should also hide the seconds! Makes sense…seconds are for other people to see, if/when they highlight an individual cell.

So, you’ve opened this file, but can’t remember if you made any changes outside of applying auto-width to the columns. The data still looks right, so you hit ‘Save’ when prompted by Excel. But you remember that your favorite programmer asked for a CSV file, and it’s already a CSV file, so you hit save, ignore the ‘features’ Excel brags about and email it back to your co-worker. Here’s what they receive:

Reading this back in our plain-text editor, we can now see we have a loss of fidelity of between 37 and 47 seconds on each cell of data. Whereas Excel keeps track of your timestamps while you’re in a SPREADSHEET, if you save as plain text, Excel assumes you want to keep the format it automatically applied to your data (automatically! silently!), and thus, destroys your file. In what world would you not care about seconds in your timestamps?

Remember, this mis-feature occurs even if the only thing you do is open a plain-text file in Excel and hit save. No other Excel actions are needed to destroy your data.

Excel: Only The Proper Tool If You Don’t Care

If you don’t care about using the proper tool for analytics, don’t want to learn something new, don’t want numerical accuracy, hate visually interesting graphics, don’t need reproducibility…use Excel. For everything else, there’s everything else. Don’t be a VLOOKUP guru, use SQL. Don’t store your data in Excel just because it allows for a million rows, use a database. If you need point-and-click graphics, at least spring for Tableau so the defaults look nicer.

Or, learn to code using open-source languages for a total licensing cost of $0. Every analyst would get value from knowing one open-source analytics language, even topically, so that you can write simple calculation scripts and document your thought process. A side benefit is that by coding, you can also use version control like Git or SVN. Then, you can have different versions of thought, and the next analyst down the line can see how your analysis has evolved.

And while I’m ranting, a special message for all you ‘top-tier’ analytics consultants out there: you should know SEVERAL of the common analytics languages. If you do your “analysis” in Excel, you are a hack or you are just providing reporting for $300/hr. Use better tools, your clients deserve better. I have infinitely more respect for someone who delivers a sloppy set of slides and a documented R script than someone who knows who to put drop-shadows on MS Office documents and makes fancy decks. You are being judged not just by the C-Suite, but also by snobs like me. And when contract renewal time comes around, they do ask my opinion and I do make comments on how sophisticated your toolset was that you used (or lack thereof if you’re using Excel).

It’s nearly 2015, do better. Stop Using Excel.

Code Refactoring Using Metaprogramming

Tue, 18 Nov 2014 09:11:06 +0000

It’s been nearly a year since I wrote Twitter.jl, back when I seemingly had MUCH more free time. In these past 10 months, I’ve used Julia quite a bit to develop other packages, and I try to use it at work when I know I’m not going to be collaborating with others (since my colleagues don’t know Julia, not because it’s bad for collaboration!).

One of the things that’s obvious from my earlier Julia code is that I didn’t understand how powerful metaprogramming can be, so here’s a simple example where I can replace 50 lines of Julia code with 10.

CTRL-A, CTRL-C, CTRL-P. Repeat.

Admittedly, when I started on the Twitter package, I fully meant to go back and clean up the codebase, but moved onto something more fun instead. The Twitter package started out as a means of learning how to use the Requests.jl library to make API calls, figure out the OAuth syntax I needed (which itself should be factored out of Twitter.jl), then copied-and-pasted the same basic function structure over and over. While fast, what I was left with was this (currently, the help.jl file in the Twitter package):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#############################################################
#
# Help section Functions for Twitter API
#
#############################################################

function get_help_configuration(; options=Dict{String, String}())

    r = get_oauth("https://api.twitter.com/1.1/help/configuration.json", options)

    return r.status == 200 ? JSON.parse(r.data) : r

end

function get_help_languages(; options=Dict{String, String}())

    r = get_oauth("https://api.twitter.com/1.1/help/languages.json", options)

    return r.status == 200 ? JSON.parse(r.data) : r

end

function get_help_privacy(; options=Dict{String, String}())

    r = get_oauth("https://api.twitter.com/1.1/help/privacy.json", options)

    return r.status == 200 ? JSON.parse(r.data) : r

end

function get_help_tos(; options=Dict{String, String}())

    r = get_oauth("https://api.twitter.com/1.1/help/tos.json", options)

    return r.status == 200 ? JSON.parse(r.data) : r

end

function get_application_rate_limit_status(; options=Dict{String, String}())

    r = get_oauth("https://api.twitter.com/1.1/application/rate_limit_status.json", options)

    return r.status == 200 ? JSON.parse(r.data) : r

end

It’s pretty clear that this is the same exact code pattern, right down to the spacing! The way to interpret this code is that for these five Twitter API methods, there are no required inputs. Optionally, there is the ‘options’ keyword that allows for specifying a Dict() of options. For these five functions, there are no options you can pass to the Twitter API, so even this keyword is redundant. These are simple functions so I don’t gain a lot by way of maintainability by using metaprogramming, but at the same time, one of the core tenets of programming is ‘Don’t Repeat Yourself’, so let’s clean this up.

For :symbol in symbolslist…

In order to clean this up, we need to take out the unique parts of the function, then pass them as arguments to the @eval macro as follows:

1
2
3
4
5
6
7
8
9
10
11
12
funcname = (:get_help_configuration, :get_help_languages, :get_help_privacy, :get_help_tos, :get_application_rate_limit_status)
endpoint = ("help/configuration.json", "help/languages.json", "help/privacy.json",  "help/tos.json", "application/rate_limit_status.json")

for (func, endp) in zip(funcname, endpoint)
	@eval function ($func)(; options=Dict{String, String}())

	        r = get_oauth($"https://api.twitter.com/1.1/$endp", options)

	        return r.status == 200 ? JSON.parse(r.data) : r

    	end
end

What’s happening in this code is that I define two tuples: one of function names (as symbols, denoted by :) and one of the API endpoints. We can then iterate over the two tuples, substituting the function names and endpoints into the code. When the package is loaded, this code evaluates, defining the five functions for use in the Twitter package.

Wha?

Yeah, so metaprogramming can be simple, but it can also be mind-bending. It’s one thing to not repeat yourself, it’s another to write something so complex that even YOU can’t remember how the code works. But somewhere in between lies a sweet spot where you can re-factor whole swaths of code and streamline your codebase. Metaprogramming is used throughout the Julia codebase, so if you’re interested in seeing more examples of metaprogramming, check out the Julia source code, the Requests.jl package (where I first saw this) or really anyone who actually knows what they are doing. I’m just a metaprogramming pretender at this point 🙂

To read additional discussion around this specific example, see the Julia-Users discussion at: https://groups.google.com/forum/#!topic/julia-users/zvJmqB2N0GQ

Edit, 11/22/2014: DarthToaster on Reddit provided another fantastic way to approach refactoring, using macros:

1
2
3
4
5
6
7
8
9
10
11
macro endpoint(name, path)
    quote
        function $(esc(name))(; options=Dict{String, String}())
            r = get_oauth($"https://api.twitter.com/1.1/$path", options)
            return r.status == 200 ? JSON.parse(r.data) : r
        end
    end
end

@endpoint get_help_configuration "help/configuration.json"
@endpoint get_help_languages "help/languages.json"

RSiteCatalyst Version 1.4.1 Release Notes

Mon, 10 Nov 2014 10:01:36 +0000

Changes

Version 1.4.1 of RSiteCatalyst is now available on CRAN. There were a handful of bug fixes and new features added, including:

Fixed bug in QueueRanked function where only 10 results were returned when requesting multiple element reports. Function now returns up to 50,000 per breakdown (API limit)
Created better error message to inform user to login with credentials instead of making function call without proper API credentials
Added support for using SAINT classifications in QueueRanked/QueueTrended functions
Added more error checking to make functions fail more elegantly
Added remaining GET methods from Reporting/Administration API

Additional GET methods

This version of RSiteCatalyst has roughly 20 new GET methods, mostly providing additional report suite information for those who might desire to generate their documentation programmatically rather than manually. New API methods include (but are not limited to):

GetMarketingChannelRules: Get a list of all criteria used to build the Marketing Channels report
GetReportDescription: For a given bookmark_id, get the report definition
GetListVariables: Get a list of the List Variables defined for a report suite
GetLogins: Get all logins for a given Company

If you were the type of person who enjoyed this blog post showing how to auto-generate Adobe Analytics documentation, I encourage you to take a look at these newly incorporated functions and use them to improve your documentation even further.

Feature Requests/Bugs

If you come across any bugs, or have any feature requests, please continue to use the RSiteCatalyst GitHub Issues page to make tickets. While I’ve responded to many of you via the maintainer email provided in the R package itself, it’s much more efficient (and you’re much more likely to get a response) if you use the GitHub Issues page. Don’t worry about cluttering up the page with tickets, please fill out a new issue for anything you encounter, unless you are SURE that it is the same problem someone else is facing.

And finally, like I end every blog post about RSiteCatalyst, please note that I’m not an Adobe employee. Please don’t send me your API credentials, expect immediate replies or ask to set up phone calls to troubleshoot your problems. This is open-source software…Willem Paling and I did the hard part writing it, you’re expected to support yourself as best as possible unless you believe you’re encountering a bug. Then use GitHub 🙂

Evaluating BreakoutDetection

Thu, 06 Nov 2014 21:24:00 +0000

A couple of weeks ago, Twitter open-sourced their BreakoutDetection package for R, a package designed to determine shifts in time-series data. The Twitter announcement does a great job of explaining the main technique for detection (E-Divisive with Medians), so I won’t rehash that material here. Rather, I wanted to see how this package works relative to the anomaly detection feature in the Adobe Analytics API, which I’ve written about previously.

Getting Time-Series Data Using RSiteCatalyst

To use a real-world dataset to evaluate this package, I’m going to use roughly ten months of daily pageviews generated from my blog. The hypothesis here is that if the BreakoutDetection package works well, it should be able to detect the boundaries around when I publish a blog post (of which the dates I know with certainty) and when articles of mine get shared on sites such as Reddit. From past experience, I get about a 3-day lift in pageviews post-publishing, as the article gets tweeted out, published on R-Bloggers or JuliaBloggers and shared accordingly.

Here’s the code to get daily pageviews using RSiteCatalyst (Adobe Analytics):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#Installing BreakoutDetection package
install.packages("devtools")
devtools::install_github("twitter/BreakoutDetection")
library(BreakoutDetection)

library("RSiteCatalyst")
SCAuth("company", "secret")

#Get pageviews for each day in 2014
pageviews_2014 <- QueueOvertime('report-suite',
                               date.from = '2014-02-24',
                               date.to = '2014-11-05',
                               metric = 'pageviews',
                               date.granularity = 'day')

#v1.0.1 of package requires specific column names and dataframe format
formatted_df <- pageviews_2014[,c("datetime","pageviews")]
names(formatted_df) <- c("timestamp", "count")

One thing to notice here is that BreakoutDetection requires either a single R vector or a specifically formatted data frame. In this case, because I have a timestamp, I use lines 17-18 to get the data into the required format.

BreakoutDetection - Default Example

In the Twitter announcement, they provide an example, so let’s evaluate those defaults first:

In order to validate my hypothesis, the package would need to detect 12 ‘breakouts’ or so, as I’ve published 12 blog posts during the sample time period. Mentally drawing lines between the red boundaries, we can see three definitive upward mean shifts, but far fewer than the 12 I expected.

BreakoutDetection - Modifying The Parameters

Given that the chart above doesn’t fit how I think my data are generated, we can modify two main parameters: beta and min.size. From the documentation:

beta: A real numbered constant used to further control the amount of penalization. This is the default form of penalization, if neither (or both) beta or (and) percent are supplied this argument will be used. The default value is beta=0.008.

min.size: The minimum number of observations between change points

The first parameter I’m going to experiment with is min.size, because it requires no in-depth knowledge of the EDM technique! The value used in the first example was 24 (days) between intervals, which seems extreme in my case. It’s reasonable that I might publish a blog post per week, so let’s back that number down to 5 and see how the result changes:

With 17 predicted intervals, we’ve somewhat overshot the number of blog posts mark. Not that the package is wrong per se; the boundaries are surrounding many of the spikes in the data, but perhaps having this many breakpoints isn’t useful from a monitoring standpoint. So setting the min.size parameter somewhere between 5 and 24 points would give us more than 3 breakouts, but less than 17. There is also the beta parameter that can be played with, but I’ll leave that as an exercise for another day.

Anomaly Detection - Adobe Analytics

From my prior post about Anomaly Detection with the Adobe Analytics API, Adobe has chosen to use Holt-Winters/Exponential Smoothing as their technique. Here’s what that looks like for the same time-period (code as GitHub Gist):

Even though the idea of both techniques are similar, it’s clear that the two methods don’t quite represent the same thing. In the case of the Adobe Analytics Anomaly Detection, it’s looking datapoint-by-datapoint, with a smoothing model built from the prior 35 points. If a point exceeds the upper- or lower-control limits, then it’s an anomaly, but not necessarily indicative of a true level shift like the BreakoutDetection package is measuring.

Conclusion

The BreakoutDetection package is definitely cool, but it is a bit raw, especially the default graphics. But the package definitely does work, as evidenced by how well it put boundaries around the traffic spikes when I set the min.size parameter equal to five.

Additionally, I tried to read more about the underlying methodology, but the only references that come up in Google seem to be references to the R package itself! I wish I had a better feeling for how the beta parameter influences the graph, but I guess that will come over time as I use the package more. But I’m definitely glad that Twitter open-sourced this package, as I’ve often wondered about how to detect level shifts in a more operational setting, and now I have a method to do so.

Visualizing Website Pathing With Sankey Charts

Wed, 10 Sep 2014 21:27:10 +0000

In my prior post on visualizing website structure using network graphs, I referenced that network graphs showed the pairwise relationships between two pages (in a bi-directional manner). However, if you want to analyze how your visitors are pathing through your site, you can visualize your data using a Sankey chart.

Visualizing Single Page-to-Next Page Pathing

Most digital analytics tools allow you to visualize the path between pages. In the case of Adobe Analytics, the Next Page Flow diagram is limited to 10 second-level branches in the visualization. However, the Adobe Analytics API has no such limitation, and as such we can use RSiteCatalyst to create the following visualization (GitHub Gist containing R code):

The data processing for this visualization is near identical to the network diagrams. We can use QueuePathing() from RSiteCatalyst to download our pathing data, except in this case, I specified an exact page name as the first level of the pathing pattern instead of using the ::anything:: operator. In all Sankey charts created by d3Network, you can hover over the right-hand side nodes to see the values (you can also drag around the nodes on either side if you desire!). It’s pretty clear from this diagram that I need to do a better job retaining my visitors, as the most common path from this page is to leave. 🙁

Many-to-Many Page Pathing

The example above picks a single page related to Hadoop, then shows how my visitors continue through my site; sometimes, they go to other Hadoop pages, some view Data Science related content or any number of other paths. If we want, however, we can visualize how all visitors path through all pages. Like the force-directed graph, we can get this information by using the ("::anything::", "::anything::") path pattern with QueuePathing():

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#Multi-page pathing
library("d3Network")
library("RSiteCatalyst")

#### Authentication
SCAuth("name", "secret")

#### Get All Possible Paths with ("::anything::", "::anything::")
pathpattern <- c("::anything::", "::anything::")
next_page <- QueuePathing("zwitchdev",
                          "2014-01-01",
                          "2014-08-31",
                          metric="pageviews",
                          element="page",
                          pathpattern,
                          top = 50000)

#Optional step: Cleaning my pagename URLs to remove to domain for clarity
next_page$step.1 <- sub("http://randyzwitch.com/","",
                        next_page$step.1, ignore.case = TRUE)
next_page$step.2 <- sub("http://randyzwitch.com/","",
                        next_page$step.2, ignore.case = TRUE)

#Filter out Entered Site and duplicate rows, >120 for chart legibility
links <- subset(next_page, count >= 120 & step.1 != "Entered Site")

#Get unique values of page name to create nodes df
#Create an index value, starting at 0
nodes <- as.data.frame(unique(c(links$step.1, links$step.2)))
names(nodes) <- "name"
nodes$nodevalue <- as.numeric(row.names(nodes)) - 1

#Convert string to numeric nodeid
links <- merge(links, nodes, by.x="step.1", by.y="name")
names(links) <- c("step.1", "step.2", "value", "segment.id", "segment.name", "source")

links <- merge(links, nodes, by.x="step.2", by.y="name")
names(links) <- c("step.2", "step.1", "value", "segment.id", "segment.name","source", "target")

#Create next page Sankey chart
d3output = "~/Desktop/sankey_all.html"
d3Sankey(Links = links, Nodes = nodes, Source = "source",
         Target = "target", Value = "value", NodeID = "name",
         fontsize = 12, nodeWidth = 50, file = d3output, width = 750, height = 700)

Running the code above provides the following visualization:

For legibility purposes, I’m only plotting paths that occur more than 120 times. But given a large enough display, it would be possible to visualize all valid combinations of paths.

One thing to keep in mind is that with the d3.js library, there is a weird hiccup where if your dataset contains “duplicate” paths such that both Source -> Target & Target -> Source exists, d3.js will go into an infinite loop/not show any visualization. My R code doesn’t provide a solution to this issue, but it should be trivial to remove these “duplicates” should they arise in your dataset.

Interpretation

Unlike the network graphs, Sankey Charts are fairly easy to understand. The “worst” path on my site in terms of keeping visitors on site is where I praised Apple for fixing my MacBook Pro screen out-of-warranty. The easy explanation for this poor performance is that this article attracts people who aren’t really my target audience in data science, but looking for information about getting THEIR screens fixed. If I wanted to engage these readers more, I guess I would need to write more Apple-related content.

To the extent there are multi-stage paths, these tend to be Hadoop and Julia-related content. This makes sense as both technologies are fairly new, I have a lot more content in these areas, and especially in the case of Julia, I’m one of the few people writing practical content. So I’m glad to see I’m achieving some level of success in these areas.

Hopefully this blog post and my previous post on visualizing your website visitors using network graphs have given a feel for the new functionality available in RSiteCatalyst v1.4, as well providing a new way of thinking about data visualization beyond just the default graphs provided by the Adobe Analytics interface.

Creating A Stacked Bar Chart in Seaborn

Tue, 09 Sep 2014 08:01:39 +0000

Download chart data

The other day I was having a heck of a time trying to figure out how to make a stacked bar chart in Seaborn. But in true open-source/community fashion, I ended up getting a response from the creator of Seaborn via Twitter:

@randyzwitch I don't really like stacked bar charts, I'd suggest maybe using pointplot / factorplot with kind=point

— Michael Waskom (@michaelwaskom) September 4, 2014

So there you go. I don’t want to put words in Michael’s mouth, but if he’s not a fan, then it sounded like it was up to me to find my own solution if I wanted a stacked bar chart. I hacked around on the pandas plotting functionality a while, went to the matplotlib documentation/example for a stacked bar chart, tried Seaborn some more and then it hit me…I’ve gotten so used to these amazing open-source packages that my brain has atrophied! Creating a stacked bar chart is SIMPLE, even in Seaborn (and even if Michael doesn’t like them 🙂 )

Stacked Bar Chart = Sum of Two Series

In trying so hard to create a stacked bar chart, I neglected the most obvious part. Given two series of data, Series 1 (“bottom”) and Series 2 (“top”), to create a stacked bar chart you just need to create:

1
Series 3 = Series 1 + Series 2

Once you have Series 3 (“total”), then you can use the overlay feature of matplotlib and Seaborn in order to create your stacked bar chart. Plot “total” first, which will become the base layer of the chart. Because the total by definition will be greater-than-or-equal-to the “bottom” series, once you overlay the “bottom” series on top of the “total” series, the “top” series will now be stacked on top:

Background: “Total” Series

Overlay: “Bottom” Series

End Result: Stacked Bar Chart

Running the code in the same IPython Notebook cell results in the following chart (download chart data):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import pandas as pd
from matplotlib import pyplot as plt
import matplotlib as mpl
import seaborn as sns
%matplotlib inline

#Read in data & create total column
stacked_bar_data = pd.read_csv("C:\stacked_bar.csv")
stacked_bar_data["total"] = stacked_bar_data.Series1 + stacked_bar_data.Series2

#Set general plot properties
sns.set_style("white")
sns.set_context({"figure.figsize": (24, 10)})

#Plot 1 - background - "total" (top) series
sns.barplot(x = stacked_bar_data.Group, y = stacked_bar_data.total, color = "red")

#Plot 2 - overlay - "bottom" series
bottom_plot = sns.barplot(x = stacked_bar_data.Group, y = stacked_bar_data.Series1, color = "#0000A3")


topbar = plt.Rectangle((0,0),1,1,fc="red", edgecolor = 'none')
bottombar = plt.Rectangle((0,0),1,1,fc='#0000A3',  edgecolor = 'none')
l = plt.legend([bottombar, topbar], ['Bottom Bar', 'Top Bar'], loc=1, ncol = 2, prop={'size':16})
l.draw_frame(False)

#Optional code - Make plot look nicer
sns.despine(left=True)
bottom_plot.set_ylabel("Y-axis label")
bottom_plot.set_xlabel("X-axis label")

#Set fonts to consistent 16pt size
for item in ([bottom_plot.xaxis.label, bottom_plot.yaxis.label] +
             bottom_plot.get_xticklabels() + bottom_plot.get_yticklabels()):
    item.set_fontsize(16)

Don’t Overthink Things!

In the end, creating a stacked bar chart in Seaborn took me 4 hours to mess around trying everything under the sun, then 15 minutes once I remembered what a stacked bar chart actually represents. Hopefully this will save someone else from my same misery.

Download chart data

Visualizing Website Structure With Network Graphs

Mon, 08 Sep 2014 06:40:38 +0000

Last week, version 1.4 of RSiteCatalyst was released, and now it’s possible to get site pathing information directly within R. Now, it’s easy to create impressive looking network graphs from your Adobe Analytics data using RSiteCatalyst and d3Network. In this blog post, I will cover simple and force-directed network graphs, which show the pairwise representation between pages. In a follow-up blog post, I will show how to visualize longer paths using Sankey diagrams, also from the d3Network package.

Obtaining Pathing Data With QueuePathing

Although the QueuePathing() function is new to RSiteCatalyst, its syntax should feel familiar (even with all of the breaking changes we made!). In the case of creating our network graphs, we want to download all pairwise combinations of pages, which is easy to do using the ::anything:: operator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
library("RSiteCatalyst")
library("d3Network")

#### Authentication
SCAuth("username", "secret")

#### Get Pathing data using ::anything:: wildcards
# Results are limited by the API to 50000
pathpattern <- c("::anything::", "::anything::")

queue_pathing_pages <- QueuePathing("zwitchdev",
                                    "2014-01-01",
                                    "2014-08-31",
                                    metric="pageviews",
                                    element="page",
                                    pathpattern,
                                    top = 50000)

Because we are using a pathing pattern of ("::anything::", "::anything::"), the data frame that is returned from this function will have three columns: step.1, step.2 and count, which is the number of occurrences of the path.

Plotting Graph Using d3SimpleNetwork

Before jumping into the plotting, we need to do some quick data cleaning. Lines 1-5 below are optional; I don’t set the Adobe Analytics s.pageName on each of my blog pages (a worst practice if there ever was one!), so I use the sub() function in Base R to strip the domain name from the beginning of the page. The other data frame modification is to remove the 'Entered Site' and 'Exited Site' from the pagename pairs. Although this is important information generally, these behaviors aren’t needed to show the pairwise relationship between pages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#Optional step: Cleaning my pagename URLs to remove to domain for graph clarity
queue_pathing_pages$step.1 <- sub("http://randyzwitch.com/","",
                                  queue_pathing_pages$step.1, ignore.case = TRUE)
queue_pathing_pages$step.2 <- sub("http://randyzwitch.com/","",
                                  queue_pathing_pages$step.2, ignore.case = TRUE)

#### Remove Enter and Exit site values
#This information is important for analysis, but not related to website structure
graph_links <- subset(queue_pathing_pages, step.1 != "Entered Site" & step.2 != "Exited Site")

#### First pass - Simple Network
# Setting standAlone = TRUE creates a full HTML file to view graph
# Set equal to FALSE to just get the d3 JavaScript
simpleoutput1 = "C:/Users/rzwitc200/Desktop/simpleoutput1.html"
d3SimpleNetwork(graph_links, Source = "step.1", Target = "step.2", height = 600,
                width = 750, fontsize = 12, linkDistance = 50, charge = -50,
                linkColour = "#666", nodeColour = "#3182bd",
                nodeClickColour = "#E34A33", textColour = "#3182bd", opacity = 0.6,
                standAlone = TRUE, file = simpleoutput1)

Running the above code results in the following graph:

Hmmm…looks like a blob of spaghetti, a common occurrence when creating graphs. We can do better.

Pruning Edges From The Graph

There are many complex algorithms for determining how to prune edges/nodes from a network. For the sake of simplicity, I’m going to use a very simple algorithm: each path has to occur more than 5 times for it to be included in the network. This will prune roughly 80% of the pairwise page combinations while keeping ~75% of the occurrences. This is simple to do using the subset() function in R:

1
2
3
4
5
6
7
8
9
#### Second pass: thin the spaghetti blob!
#Require path to happen more than some number of times (count > x)
#What constitutes "low volume" will depend on your level of traffic
simpleoutput2 = "C:/Users/rzwitc200/Desktop/simpleoutput2.html"
d3SimpleNetwork(subset(graph_links, count > 5), Source = "step.1", Target = "step.2", height = 600,
                width = 750, fontsize = 12, linkDistance = 50, charge = -100,
                linkColour = "#666", nodeColour = "#3182bd",
                nodeClickColour = "#E34A33", textColour = "#3182bd", opacity = 0.6,
                standAlone = TRUE, file = simpleoutput2)

The result of pruning the number of edges is a much less cluttered graph:

Even with fewer edges in the graph, we still lose some of the information about the pages, since we don’t know what topics/groups the pages represent. We can fix that using a slightly more complex version of the d3Network graph code.

Force-directed graphs

The graphs above outline the structure of randyzwitch.com, but they can be improved by adding color-coding to the nodes to represent the topic of the post, as well as making the edges thicker/thinner based on how frequently the path occurs. This can be done using the d3ForceNetwork() function like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
#### Force directed network

#Limit to more than 5 occurence like in simple network
fd_graph_links <- subset(graph_links, count > 5)

#Get unique values of page name to create nodes df
#Create an index value, starting at 0
fd_nodes <- as.data.frame(unique(c(fd_graph_links$step.1, fd_graph_links$step.2)))
names(fd_nodes) <- "name"
fd_nodes$nodevalue <- as.numeric(row.names(fd_nodes)) - 1

#Create groupings for node colors
#This is user-specific in terms of how to create these groupings
#Due to few number of pages/topics, I am manually coding this

grouping <- function(string){

  if(grepl("(hadoop|hive|pig)",string, perl=TRUE)){
    return(1)
  }else if(grepl("(julia|uaparser-jl)",string, , perl=TRUE)){
    return(2)
  }else if(grepl(("[r]?sitecatalyst|adobe-analytics|omniture"),string, perl=TRUE)){
    return(3)
  }else if(grepl("(wordpress|twenty-eleven|scrappy)",string, perl=TRUE)){
    return(4)
  }else if(grepl("data-science|ec2",string, perl=TRUE)){
    return(5)
  }else if(grepl("python",string, perl=TRUE)){
    return(6)  
  }else if(grepl("(digital-analytics|google-analytics|web-analyst)",string, perl=TRUE)){
    return(8)
  }else if(grepl("(macbook|iphone)",string, perl=TRUE)){
    return(9)
  }else if(grepl("(randyzwitch|about|page)",string, perl=TRUE)){
    return(10)
  }else if(grepl("(rstudio|rcmdr|r-language|jsonlite|r-language-oddities|tag/r|automated-re-install-of-packages-for-r-3-0|learning-r-sas|creating-dummy-variables-data-frame-r)",string, perl=TRUE)){
    return(7)
  }else{
    return(11)
  }

}

#Create group column
fd_nodes$group <- sapply(fd_nodes$name, grouping)

#Append numeric nodeid to pagename
fd_graph_links <- merge(fd_graph_links, fd_nodes[,1:2], by.x="step.1", by.y="name")
names(fd_graph_links) <- c("step.1", "step.2", "value", "source")

fd_graph_links <- merge(fd_graph_links, fd_nodes[,1:2], by.x="step.2", by.y="name")
names(fd_graph_links) <- c("step.1", "step.2", "value", "source", "target")

d3output = "C:/Users/rzwitc200/Desktop/fd_graph.html"
# Create force-directed graph
d3ForceNetwork(Links = fd_graph_links, Nodes = fd_nodes, Source = "source",
               Target = "target", NodeID = "name",
               Group = "group", opacity = 0.8, Value = "value",
               file = d3output,
               charge = -90,
               fontsize=12)

Running the code results in the following force-directed graph:

Interpretation

I’m not going to lie, all three of these diagrams are hard to interpret. Like wordclouds, network graphs can often be visually interesting, yet difficult to ascertain any concrete information. Network graphs also have the tendency to reinforce what you already know (you or someone you know designed your website, you should already have a feel for its structure!).

However, in the case of the force-directed graph above, I do see some interesting patterns. Specifically, there are a considerable number of nodes that aren’t attached to the main network structure. This may be occurring due to my method of pruning the network edges. More likely is that these disconnected nodes represent “dead-ends” in my blog, either because few pages link to them, there are technical errors, these are high bounce-rate pages or represent one-off topics that satiate the reader.

In terms of action I can take, I can certainly look up the bounce rate for these disconnected pages/nodes and re-write the content to make it more ‘sticky’. There’s also the case of the way my “Related Posts” plugin determines related pages. As far as I know, it’s quite naive, using the existing words on the page to determine relationships between posts. So one follow-up could be to create an actual recommender system to better suggest content to my readers. Perhaps that’s a topic for a different blog post.

Regardless of the actions I’ll end up taking from this information, hopefully this blog post has piqued some ideas of how to use RSiteCatalyst in a non-standard way, to extend the standard digital analytics information you are capturing with Adobe Analytics into creating interesting visualizations and potential new insights.

Example Data

For those of you who aren’t Adobe Analytics customers (or are, but don’t have API access), here are the data from the queue_pathing_pages data frame above. Just read this data into R, then you should be able to follow along with the d3Network code.

RSiteCatalyst Version 1.4 Release Notes

Mon, 01 Sep 2014 20:30:14 +0000

It felt like it would never happen, but RSiteCatalyst v1.4 is now available on CRAN! There are numerous changes in this version of the package, so unlike previous posts, there won’t be any code examples.

THIS VERSION IS ONE BIG BREAKING CHANGE

While not the most important improvement, it can’t be stressed enough that migrating to v1.4 of RSiteCatalyst is likely going to require re-writing some of your prior code. There are numerous reasons for the breaking changes, including:

Adobe made breaking changes to the API between v1.3 and v1.4, so we had to as well
I partnered with Willem Paling, who merged his RAA codebase into RSiteCatalyst to contribute most of the code in this version
Better consistency in R functions around keywords and options

Of the changes listed above, I think #2 and #3 are the biggest benefit to end-users of RSiteCatalyst. The codebase is now much cleaner and more consistent in terms of the keyword arguments, has better error handling, and having a second person helping maintain the project has led to a better overall package.

Where you’ll see the most difference is that all keyword arguments are now all lowercase and multi-word keyword arguments are now separated by a period instead of underscores or weird caMelCAse. We tried to maintain the same keyword order where possible to minimize code re-writes.

Pathing and Fallout Reports

Probably the most useful improvement to RSiteCatalyst comes from those breaking changes by Adobe, which is the inclusion of Pathing and Fallout reports! I can’t say with absolute certainty, but I think with these two additional reports, the API is pretty much at parity to the Adobe Analytics interface itself. So now you can create your funnels using ggplot2, make force-directed graphs or Sankey charts using d3Network or just simple reporting of top ‘Next Pages’ and the like.

Support for OAuth Authentication

As part of Adobe’s commitment to consolidating systems under the single Adobe Marketing Cloud, authentication with the API using OAuth is now possible. How to set up OAuth authentication is beyond the scope of this blog post, but you can get more information at this link: Adobe Marketing Cloud OAuth.

For those of you who don’t have OAuth credentials setup yet, the “legacy” version of authentication is still available in RSiteCatalyst.

GetClassifications, Inline Segmentation and More

Finally, there is now additional functionality on the descriptive side, as you can now download which Classifications are defined for a report suite, segments can be defined inline (i.e. from R) for the ‘Queue’ reports using the BuildClassificationValueSegment() function and functions that existed in previous versions of RSiteCatalyst tend to have more options defined than in previous versions.

Summary/We Want To Hear From You

While this new version of RSiteCatalyst has some annoying breaking changes, overall the package is much more robust than prior versions. I think the increase in functionality is well worth the minor annoyance of re-writing some code. Additionally, eventually Adobe will deprecate v1.3 of their API, so it’s better to move over sooner rather than later.

But for all of the improvements that have been made, there’s always room for improvement, whether it’s fixing unforeseen bugs, adding new features, improving the documentation or anything else. For all suggestions, bug fixes and the like, please submit them to the GitHub repository so that myself and Willem can evaluate and incorporate them. We’re also VERY open to any of you in the R community who are able to patch the code or add new features. As a friend in the data science community says, a Pull Request is always better than a Feature Request 🙂

Happy API’ing everyone!

Visualizing Analytics Languages With VennEuler.jl

Fri, 29 Aug 2014 15:16:24 +0000

It often doesn’t take much to get me off track, and on a holiday weekend…well, I was just begging for a fun way to shirk. Enter Harlan Harris:

someone redo this area-prop'l Venn w/ my Julia pkg! http://t.co/Mh8rXZbRgY http://t.co/RDWNQHTw3S http://t.co/ljujd9DG0T via @revodavid

— Harlan Harris (@HarlanH) August 29, 2014

Hey, I’m someone looking for something to do! And I like writing Julia code! So let’s have a look at recreating this diagram in Julia using VennEuler.jl (IJulia Notebook link):

Source: Revolution R/KDNuggets

http://blog.revolutionanalytics.com/2014/08/r-tops-kdnuggets-data-analysis-software-poll-for-4th-consecutive-year.html

Installing VennEuler.jl

Because VennEuler.jl is not in METADATA as of the time of writing, instead of using Pkg.add() you’ll need to run:

1
Pkg.clone("https://github.com/HarlanH/VennEuler.jl.git")

Note that VennEuler uses some of the more exotic packages (at least to me) like NLopt and Cairo, so you might need to have a few additional dependencies installed with the package.

Data

The data was a bit confusing to me at first, since the percentages add up to more than 100% (people could vote multiple times). In order to create a dataset to use, I took the percentages, multiplied by 1000, then re-created the voting pattern. The data for the graph can be downloaded from this link.

Code - Circles

With a few modifications, I basically re-purposed Harlan’s code from the package test files. The circle result is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
using VennEuler

data, labels = readcsv("/home/rzwitch/Desktop/kdnuggets_language_survey_2014.csv", header=true)
data = bool(data)
labels = vec(labels)

#Circles
eo = make_euler_object(labels, data, EulerSpec()) # circles, for now

(minf,minx,ret) = optimize(eo, random_state(eo), ftol=-1, xtol=0.0025, maxtime=120, pop=1000)
println("got $minf at $minx (returned $ret)")

render("/home/rzwitch/Desktop/kd.svg", eo, minx)

Since the percentage of R, SAS, and Python users isn’t too dramatically different (49.81%, 33.42%, 40.97% respectively) and the visualizations are circles, it’s a bit hard to tell that R is about 16% points higher than SAS and 9% points higher than Python.

Code - Rectangles

Alternatively, we can use rectangles to represent the areas:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
using VennEuler

data, labels = readcsv("/home/rzwitch/Desktop/kdnuggets_language_survey_2014.csv", header=true)
data = bool(data)
labels = vec(labels)

# Rectangles
eo = make_euler_object(labels, data, [EulerSpec(:rectangle), EulerSpec(:rectangle, [.5, .5, .4], [0, 0, 0]),
    EulerSpec(:rectangle)],
    sizesum=.3)


(minf,minx,ret) = optimize_iteratively(eo, random_state(eo), ftol=-1, xtol=0.0025, maxtime=5, pop=100)
println("phase 1: got $minf at $minx (returned $ret)")
(minf,minx,ret) = optimize(eo, minx, ftol=-1, xtol=0.001, maxtime=30, pop=100)
println("phase 2: got $minf at $minx (returned $ret)")

render("/home/rzwitch/Desktop/kd-rects.svg", eo, minx)

Here, it’s a slight bit easier to see that SAS and Python are about the same area-wise and that R is larger, although the different dimensions do obscure this fact a bit.

Summary

If I spent more time with this package, I’m sure I could make something even more aesthetically pleasing. And for that matter, it’s still a pre-production package that will no doubt get better in the future. But at the very least, there is a way to create an area-proportional representation of relationships using VennEuler.jl in Julia.

String Interpolation for Fun and Profit

Mon, 14 Jul 2014 12:01:10 +0000

In a previous post, I showed how I frequently use Julia as a ‘glue’ language to connect multiple systems in a complicated data pipeline. For this blog post, I will show two more examples where I use Julia for general programming, rather than for computationally-intense programs.

String Building: Introduction

The Strings section of the Julia Manual provides a very in-depth treatment of the considerations when using strings within Julia. For the purposes of my examples, there are only three things to know:

Strings are immutable within Julia and 1-indexed
Strings are easily created through the a syntax familiar to most languages:

julia> authorname = "randy zwitch"
"randy zwitch"

julia> typeof(authorname)
String

String interpolation is easiest done using dollar-sign notation. Additionally, parenthesis can be used to avoid symbol ambiguity:

julia> interpolated = "the author of this blog post is $(authorname)"
"the author of this blog post is randy zwitch"

If you are using large volumes of textual data, you’ll want to pay attention to the difference between the various string types that Julia provides (UTF8/16/32, ASCII, Unicode, etc), but for the purposes of this blog post we’ll just be using the ASCIIString type by not explicitly declaring the string type and only using ASCII characters.

EDIT, 9/8/2016: Starting with version 0.5, Julia defaults to the String type, which is an UTF-8 character encoding.

Example 1: Repetitive Queries

As part of my data engineering responsibilities at work, I often get requests to pull a sample of every table in a new database in our Hadoop cluster. This type of request is usually from the business owner, who wants to evaluate the data set has been imported correctly, but doesn’t actually want to write any sort of queries. So using the ODBC.jl package, I repeatedly do the same select * from <tablename> query and save to individual .tab files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
_
_       _ _(_)_     |  A fresh approach to technical computing
(_)     | (_) (_)    |  Documentation: http://docs.julialang.org
_ _   _| |_  __ _   |  Type "help()" to list help topics
| | | | | | |/ _` |  |
| | |_| | | | (_| |  |  Version 0.3.0-prerelease+4028 (2014-07-02 23:42 UTC)
_/ |\__'_|_|_|\__'_|  |  Commit 2185bd1 (11 days old master)
|__/                   |  x86_64-w64-mingw32
julia> using ODBC
julia> ODBC.connect("Production hiveserver2", usr="", pwd="")
ODBC Connection Object
----------------------
Connection Data Source: Production hiveserver2
Production hiveserver2 Connection Number: 1
Contains resultset? No
julia> tables = query("show tables in db;");
elapsed time: 0.167028049 seconds
julia> for tbl in tables[:tab_name]
query("select * from db.$(tbl) limit 1000;"; output="C:\\data_dump\\$(tbl).tab", delim='\t')
end

julia>

While the query is simple, writing/running this hundreds of times would be a waste of effort. So with a simple loop over the array of tables, I can provide a sample of hundreds of tables in .tab files with five lines of code.

Example 2: Generating Query Code

In another task, I was asked to join a handful of Hive tables, then transpose the table from “long” to “wide”, so that each id value only had one row instead of multiple. This is fairly trivial to do using CASE statements in SQL; the problem arises when you have thousands of potential row values to transpose into columns! Instead of getting carpal tunnel syndrome typing out thousands of CASE statements, I decided to use Julia to generate the SQL code itself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
#Starting portion of query, the groupby columns
julia> groupbycols ="select
interact.interactionid,
interact.agentname,
interact.agentid,
interact.agentgroup,
interact.agentsupervisor,
interact.sitename,
interact.dnis,
interact.agentextension,
interact.interactiondirection,
interact.interactiontype,
interact.customerid,
interact.customercity,
interact.customerstate,
interact.interactiondatetime,
interact.durationinms,"

#Generate CASE statements based on the number of possible values of queryid
julia> function casestatements(repetitions::Int64)
	for queryid in 1:repetitions
		println("MAX(CASE WHEN q.queryid = $queryid then q.score END) as q$(queryid)score,")
	end
	for queryid in 1:repetitions
		println("MIN(CASE WHEN q.queryid = $queryid then q.startoffsetinms END) as q$(queryid)startoffset,")
	end
	for queryid in 1:repetitions
		println("MAX(CASE WHEN q.queryid = $queryid then q.endoffsetinms END) as q$(queryid)endoffset,")
	end
	#Last clause, so repeat it up to number of repetitions minus 1, then do simple print to get line without comma at end
	for queryid in 1:repetitions-1
		println("SUM(CASE WHEN q.queryid = $queryid and q.score > q.mediumthreshold THEN 1 END) as q$(queryid)hits,")
	end
	println("SUM(CASE WHEN q.queryid = $repetitions and q.score > q.mediumthreshold THEN 1 END) as q$(repetitions)hits")
end

#Ending table statement
julia> tablestatements = "from db.table1 as interact
left join db.table2 as q on (interact.interactionid = q.interactionid)
left join db.table3 as t on (interact.interactionid = t.interactionid)
group by 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15;"

#Submitting all of the statements on one line is usually frowned upon, but this will generate my SQL code
julia> println(groupbycols);casestatements(5);println(tablestatements)
select
interact.interactionid,
interact.agentname,
interact.agentid,
interact.agentgroup,
interact.agentsupervisor,
interact.sitename,
interact.dnis,
interact.agentextension,
interact.interactiondirection,
interact.interactiontype,
interact.customerid,
interact.customercity,
interact.customerstate,
interact.interactiondatetime,
interact.durationinms,
MAX(CASE WHEN q.queryid = 1 then q.score END) as q1score,
MAX(CASE WHEN q.queryid = 2 then q.score END) as q2score,
MAX(CASE WHEN q.queryid = 3 then q.score END) as q3score,
MAX(CASE WHEN q.queryid = 4 then q.score END) as q4score,
MAX(CASE WHEN q.queryid = 5 then q.score END) as q5score,
MIN(CASE WHEN q.queryid = 1 then q.startoffsetinms END) as q1startoffset,
MIN(CASE WHEN q.queryid = 2 then q.startoffsetinms END) as q2startoffset,
MIN(CASE WHEN q.queryid = 3 then q.startoffsetinms END) as q3startoffset,
MIN(CASE WHEN q.queryid = 4 then q.startoffsetinms END) as q4startoffset,
MIN(CASE WHEN q.queryid = 5 then q.startoffsetinms END) as q5startoffset,
MAX(CASE WHEN q.queryid = 1 then q.endoffsetinms END) as q1endoffset,
MAX(CASE WHEN q.queryid = 2 then q.endoffsetinms END) as q2endoffset,
MAX(CASE WHEN q.queryid = 3 then q.endoffsetinms END) as q3endoffset,
MAX(CASE WHEN q.queryid = 4 then q.endoffsetinms END) as q4endoffset,
MAX(CASE WHEN q.queryid = 5 then q.endoffsetinms END) as q5endoffset,
SUM(CASE WHEN q.queryid = 1 and q.score > q.mediumthreshold THEN 1 END) as q1hits,
SUM(CASE WHEN q.queryid = 2 and q.score > q.mediumthreshold THEN 1 END) as q2hits,
SUM(CASE WHEN q.queryid = 3 and q.score > q.mediumthreshold THEN 1 END) as q3hits,
SUM(CASE WHEN q.queryid = 4 and q.score > q.mediumthreshold THEN 1 END) as q4hits,
SUM(CASE WHEN q.queryid = 5 and q.score > q.mediumthreshold THEN 1 END) as q5hits
from db.table1 as interact
left join db.table2 as q on (interact.interactionid = q.interactionid)
left join db.table3 as t on (interact.interactionid = t.interactionid)
group by 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15;

julia>

The example here only repeats the CASE statements five times, which wouldn’t really be that much typing. However, for my actual application, the number of possible values was 2153, leading to a query result which was 8157 columns! Suffice to say, I’d still be writing that code if I decided to do it by hand.

Summary

Like my ‘glue language’ post, I hope this post has shown that Julia can be used for more than grunting about microbenchmark performance. Whereas I used to use Python for doing weird string operations like this, I’m finding that the dollar-sign syntax in Julia feels more comfortable for me than the Python string formatting mini-language (although that’s not particularly difficult either). So if you’ve been hesitant to jump into learning Julia because you think it’s only useful for doing Mandelbrot calculations or complex linear algebra, Julia is just as at-home doing quick general programming tasks as well.

Maybe I Don't Really Know R After All

Thu, 26 Jun 2014 11:18:36 +0000

Lately, I’ve been feeling that I’m spreading myself too thin in terms of programming languages. At work, I spend most of my time in Hive/SQL, with the occasional Python for my smaller data. I really prefer Julia, but I’m alone at work on that one. And since I maintain a package on CRAN (RSiteCatalyst), I frequently spend my evenings bug fix programming in R. Then, there’s the desire to learn a Java-based language like Scala (or, Java)…maybe Spark for my Hadoop work…

So last night, when I ran into this series of follies with R, it really makes me wonder if I really understand how R works.

jsonlite:fromJSON

As part of the overall concept of my RSiteCatalyst package, I’m trying to make it as easy as possible for digital analysts to get their data via the Adobe Analytics API. As such, I abstract away the need to build JSON to request reports and parse the API answer from JSON to a data frame. Sometimes it’s easy, but sometimes you get something like this:

In case it’s not clear what’s going on here, fromJSON() from jsonlite returns a data frame as best as it can, but we have a list (of data frames!) nested inside of a column named “breakdown”. There are 12 rows here, but the proper data structure would be to take the data frame inside of ‘breakdown’ and append all of the fields from the original 12 rows, repeating the values down the rows. So something like 72 rows (12 original rows, 6 row data frames inside of the ‘breakdown’ column).

Loop and Accumulate

Because this is such a small data frame, and because *apply functions are too frustrating in most cases, to parse this I went with the tried-and-true loop and accumulate. But instead of immediately getting what I wanted, I got this fantastic R error message:

1
2
3
4
5
6
7
8
9
10
11
12
#Loop over df and accumulate results
parsed_df <- data.frame()
for(i in 1:nrow(df)){
  temp <- cbind(df[i,],breakdown_list[[i]])
  parsed_df <- rbind(parsed_df, temp)
}

There were 12 warnings (use warnings() to see them)
>warnings()
Warning messages:
  1: In data.frame(..., check.names = FALSE) :
  row names were found from a short variable and have been discarded

Row names from a short variable? Off to StackOverflow, the savior of all language hackers, which lets me know I just need to add an argument to my cbind() function. Trying again:

1
2
3
4
5
6
7
8
9
10
11
12
13
#Loop over df and accumulate results
#Adding row.names = NULL fixes error message
parsed_df <- data.frame()
for(i in 1:nrow(df)){
  temp <- cbind(df[i,],breakdown_list[[i]], row.names = NULL)
  parsed_df <- rbind(parsed_df, temp)
}

names(parsed_df)

> names(parsed_df)
 [1] "name"           "year"           "month"          "day"            "hour"           "minute"         "breakdownTotal"
 [8] "name"           "trend"          "counts"

So I successfully created an (84,10)-sized data frame, but cbind() allowed me to name two columns in the data frame “name”! Running ‘parsed_df$name’ at the REPL returns the first instance. So now, I have to use the unstable method of referring to the second ‘name’ column by position number if I want to access it (or, rename it using names() of course). The way I realized this behavior was occurring was that I tried to use plyr::rename and kept changing the name of two columns!

Final Solution

In order to get past my duplicate name issue, I eventually renamed the ‘name’ columns individually by each object, prior to cbind():

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#Separate breakdown list and original data frame into different objects
df <- ex_df$report$data
breakdown_list <- df$breakdown
df$breakdown <- NULL

#Loop over df and accumulate results
parsed_df <- data.frame()
for(i in 1:nrow(df)){
  right_df <-  breakdown_list[[i]]
  right_df <- rename(right_df, replace=c("name" = report_raw$report$elements$id[2]))
  temp <- cbind(df[i,],right_df, row.names = NULL)
  parsed_df <- rbind(parsed_df, temp)
}
parsed_df <- rename(parsed_df, replace=c("counts" = report_raw$report$metrics$id))

In the end, I found an answer to my solution, but it seems like every time I use R the more oddities I’m able to encounter/generate. At this point, I’m starting to question whether I really understand the underpinnings of how R works. It might be time to stop trying to be a language polyglot so much and focus on really learning a few of these tools in-depth.

Using Julia As A "Glue" Language

Tue, 24 Jun 2014 08:57:31 +0000

While much of the focus in the Julia community has been on the performance aspects of Julia relative to other scientific computing languages, Julia is also perfectly suited to ‘glue’ together multiple data sources/languages. In this blog post, I will cover how to create an interactive plot using Gadfly.jl, by first preparing the data using Hadoop and Teradata Aster via ODBC.jl.

The example problem I am going to solve is calculating and visualizing the number of airplanes by hour in the air at any given time in the U.S. for the year 1987. Because of the structure and storage of the underlying data, I will need to write some custom Hive code, upload the data to Teradata Aster via a command-line utility, re-calculate the number of flights per hour using a built-in Aster function, then using Julia to visualize the data.

Step 1: Getting Data From Hadoop

In a prior set of blog posts, I talked about loading the airline dataset into Hadoop, then analyzing the dataset using Hive or Pig. Using ODBC.jl, we can use Hive via Julia to submit our queries. The hardest part of setting up this process is making sure that you have the appropriate Hive drivers for your Hadoop cluster and credentials (which isn’t covered here). Once you have your DSN set up, running Hive queries is as easy as the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
using ODBC

#Connect to Hadoop cluster via Hive (pre-defined Windows DSN in ODBC Manager)
hiveconn = ODBC.connect("Production hiveserver2"; usr="your-user-name", pwd="your-password-here")

#Clean data, return results directly to file
#Data returned with have origin of flight, flight takeoff, flight landing and elapsed time
hive_query_string =
"select
origin,
from_unixtime(flight_takeoff_datetime_origin) as flight_takeoff_datetime_origin,
from_unixtime(flight_takeoff_datetime_origin + (actualelapsedtime * 60)) as flight_landing_datetime_origin,
actualelapsedtime
from
(select
origin,
unix_timestamp(CONCAT(year,\"-\", month, \"-\", dayofmonth, \" \", SUBSTR(LPAD(deptime, 4, 0), 1, 2), \":\", SUBSTR(LPAD(deptime, 4, 0), 3, 4), \":\", \"00\"))  as flight_takeoff_datetime_origin,
actualelapsedtime
from vw_airline
where year = 1987 and actualelapsedtime > 0) inner_query;"

#Run query, save results directly to file
query(hive_query_string, hiveconn;output="C:\\airline_times.csv",delim=',')

In this code, I’ve written my query as a Julia string, to keep my code easily modifiable. Then, I pass the Julia string object to the query() function, along with my ODBC connection object. This query runs on Hadoop through Hive, then streams the result directly to my local hard drive, making this a very RAM efficient (though I/O inefficient!) operation.

Step 2: Shelling Out To Load Data To Aster

Once I created the file with my Hadoop results in it, I now have a decision point: I can either A) do the rest of the analysis in Julia or B) use a different tool for my calculations. Because this is a toy example, I’m going to use Teradata Aster to do my calculations, which provides a convenient function called burst() to regularize timestamps into fixed intervals. But before I can use Aster to ‘burst’ my data, I first need to upload it to the database.

While I could loop over the data within Julia and insert each record one at a time, Teradata provides a command-line utility to upload data in parallel. Running command-line scripts from within Julia is as easy as using the run() command, with each command surrounded in backticks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#Connect to Aster (pre-defined Windows DSN in ODBC Manager)
asterconn = ODBC.connect("aster01"; usr="your-user-name", pwd="your-password")

#Create table to hold airline results
create_airline_table_statement =
"create table ebi_temp.airline
(origin varchar,
flight_takeoff_datetime_origin timestamp,
flight_landing_datetime_origin timestamp,
actualelapsedtime int,
partition key (origin))"

#Execute query
query(create_airline_table_statement, asterconn)

#Create airport table
#Data downloaded from http://openflights.org/data.html
create_airport_table_statement =
"create table ebi_temp.airport
(airport_id int,
name varchar,
city varchar,
country varchar,
IATAFAA varchar,
ICAO varchar,
latitude float,
longitude float,
altitude int,
timezone float,
dst varchar,
partition key (country))"

#Execute query
query(create_airport_table_statement, asterconn)

#Upload data via run() command
#ncluster_loader utility already on Windows PATH
run(`ncluster_loader -h 192.168.1.1 -U your-user-name -w your-password -d aster01 -c --skip-rows=1 --el-enabled --el-table e_dist_error_2 --el-schema temp temp.airline C:\\airline_times.csv`)

run(`ncluster_loader -h 192.168.1.1 -U your-user-name -w your-password -d aster01 -c --el-enabled --el-table e_dist_error_2 --el-schema temp temp.airport C:\\airports.dat`)

While I could’ve run this at the command-line, having all of this within an IJulia Notebook keeps all my work together, should I need to re-run this in the future.

Step 3: Using Aster For Calculations

With my data now loaded in Aster, I can normalize the timestamps to UTC, then ‘burst’ the data into regular time intervals. Again, all of this can be done via ODBC from within Julia:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
#Normalize timestamps from local time to UTC time
aster_view_string = "
create view temp.vw_airline_times_utc as
select
row_number() over(order by flight_takeoff_datetime_origin) as unique_flight_number,
origin,
flight_takeoff_datetime_origin,
flight_landing_datetime_origin,
flight_takeoff_datetime_origin - (INTERVAL '1 hour' * timezone) as flight_takeoff_datetime_utc,
flight_landing_datetime_origin - (INTERVAL '1 hour' * timezone) as flight_landing_datetime_utc,
timezone
from temp.airline
left join temp.airport on (airline.origin = airport.iatafaa);"

#Execute query
query(aster_view_string, asterconn)

#Teradata Aster SQL-H functionality, accessed via ODBC query
burst_query_string =
"create table temp.airline_burst_hour distribute by hash (origin) as
SELECT
*,
\"INTERVAL_START\"::date as calendar_date,
extract(HOUR from \"INTERVAL_START\") as hour_utc
FROM BURST(
     ON (select
        unique_flight_number,
        origin,
        flight_takeoff_datetime_utc,
        flight_landing_datetime_utc
        FROM temp.vw_airline_times_utc
)
     START_COLUMN('flight_takeoff_datetime_utc')
     END_COLUMN('flight_landing_datetime_utc')
     BURST_INTERVAL('3600')
);"

#Execute query
query(burst_query_string, asterconn)

Since it might not be clear what I’m doing here, the burst() function in Aster takes a row of data with a start and end timestamp, and (potentially) returns multiple rows which normalize the time between the timestamps. If you’re familiar with pandas in Python, it’s a similar functionality to resample on a series of timestamps.

Step 4: Download Smaller Data Into Julia, Visualize

Now that the data has been processed from Hadoop to Aster through a series of queries, we now have a much smaller dataset that can be loaded into RAM and processed by Julia:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#Calculate the number of flights per hour per day
flights_query = "
select
calendar_date,
hour_utc,
sum(1) as num_flights
from temp.airline_burst_hour
group by 1,2
order by 1,2;"

#Bring results into Julia DataFrame
flights_per_day = query(flights_query, asterconn)

using Gadfly

#Create boxplot, with one box plot per hour
set_default_plot_size(20cm, 12cm)
p = plot(flights_per_day , x="hour_utc", y="num_flights",
            Guide.xlabel("Hour UTC"),
            Guide.ylabel("Flights In Air"),
            Guide.title("Number of Flights In Air To/From U.S. By Hour - 1987"),
            Scale.y_continuous(minvalue=0, maxvalue=4000),
            Geom.boxplot)

The Gadfly code above produces the following plot:

Since this chart is in UTC, it might not be obvious what the interpretation is of the trend. Because the airline dataset represents flights either leaving or returning to the United States, there are many fewer planes in the air overnight and the early morning hours (UTC 7-10, 2-5am Eastern). During the hours when the airports are open, there appears to be a limit of roughly 2500 planes per hour in the sky.

Why Not Do All Of This In Julia?

At this point, you might be tempted to wonder why go through all of this effort? Couldn’t this all be done in Julia?

Yes, you probably could do all of this work in Julia with a sufficiently large amount of RAM. As a proof-of-concept, I hope I’ve shown that there is much more to Julia than micro-benchmarking Julia’s speed relative to other scientific programming languages. You’ll notice that in none of my code have I used any type annotations, as none would really make sense (nor would they improve performance). And although this is a toy example purposely using multiple systems, I much more frequently use Julia in this manner at work than doing linear algebra or machine learning.

So next time you’re tempted to use Python or R or shell scripting or whatever, consider Julia as well. Julia is just as at-home as a scripting language as a scientific computing language.

Five Hard-Won Lessons Using Hive

Thu, 12 Jun 2014 13:01:18 +0000

EDIT, 9/8/2016: Hive has come a long way in the two years since I’ve written this. While some of the code snippets might still work, it’s likely the case that this information is so out-of-date to be nothing more than a reflection of working with Hadoop in 2014.

I’ve been spending a ton of time lately on the data engineering side of ‘data science’, so I’ve been writing a lot of Hive queries. Hive is a great tool for querying large amounts of data, without having to know very much about the underpinnings of Hadoop. Unfortunately, there are a lot of things about Hive (version 0.12 and before) that aren’t quite the same as SQL and have caused me a bunch of frustration; here they are, in no particular order.

1. Set Hive Temp directory To Same As Final Output Directory

When doing a “Create Table As” (CTAS) statement in Hive, Hive allocates temp space for the Map and Reduce portions of the job. If you’re not lucky, the temp space for the job will be somewhere different than where your table actually ends up being saved, resulting in TWO I/O operations instead of just one. This can lead to a painful delay in when your Hive job says it is finished vs. when the table becomes available (one time, I saw a 30 hour delay writing 5TB of data).

If your Hive jobs seem to hang after the Job Tracker says they are complete, try this setting at the beginning of your session:

set hive.optimize.insert.dest.volume=true;

2. Column Aliasing In Group By/Order By

Not sure why this isn’t a default, but if you want to be able to reference your column names by position (i.e. group by 1,2) instead of by name (i.e. group by name, age), then run this at the beginning of your session:

set hive.groupby.orderby.position.alias=true;

3. Be Aware Of Predicate Push-Down Rules

In Hive, you can get great performance gains if you A) partition your table by commonly used columns/business concepts (i.e. Day, State, Market, etc.) and B) you use the partitions in a WHERE clause. These are known as partition-based queries. Otherwise, if you don’t use a partition in your WHERE clause, you will get a full table scan.

Unfortunately, when doing an OUTER JOIN, Hive will sometimes ignore the fact that your WHERE clause is on a partition and do a full table scan anyway. In order to get Hive to push your predicate down and avoid a full table scan, put your predicate on the JOIN instead of the WHERE clause:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
--#### Assume sales Hive table partitioned by day_id ####--

--Full Table Scan
select
employees.id,
b.sales
from employees
left join sales on (employees.id = sales.employee_id)
where day_id between '2014-03-01' and '2014-05-31';

--Partitioned-based query
select
employees.id,
b.sales
from employees
left join sales on (employees.id = sales.employee_id and sales.day_id between '2014-03-01' and '2014-05-31');

If you don’t want to think about the different rules, you can generally put your limiting clauses inside your JOIN clause instead of on your WHERE clause. It should just be a matter of preference (until your query performance indicates it isn’t!)

4. Calculate And Append Percentiles Using CROSS JOIN

Suppose you want to calculate the top 10% of your customers by sales. If you try to do the following, Hive will complain about needing a GROUP BY, because percentile_approx() is a summary function:

1
2
3
4
5
6
--Hive expects that you want to calculate your percentiles by account_number and sales
--This code will generate an error about a missing GROUP BY statement
select
account_number,
sales,
CASE WHEN sales > percentile_approx(sales, .9) THEN 1 ELSE 0 END as top10pct_sales

To get around the the need for a GROUP BY, we can use a CROSS JOIN. A CROSS JOIN is another name for a Cartesian Join, meaning all of the rows from the first table will be joined to ALL of the rows of the second table. Because the subquery only returns one row, the CROSS JOIN provides the desired affect of joining the percentile values back to the original table while keeping the same number of rows from the original table. Generally, you don’t want to do a CROSS JOIN (because relational data generally is joined on a key), but this is a good use case.

5. Calculating a Histogram

Creating a histogram using Hive should be as simple as calling the histogram_numeric() function. However, the syntax and results of this function are just plain weird. To create a histogram, you can run the following:

1
2
3
4
5
6
7
select
histogram_numeric(salary, 20) as salary_hist
from
sample_08;

--Results
[{"x":23507.68627450983,"y":255.0},{"x":31881.7647058824,"y":340.0},{"x":39824.11498257844,"y":287.0},{"x":47615.58011049725,"y":181.0},{"x":55667.01219512195,"y":164.0},{"x":59952.499999999985,"y":8.0},{"x":66034.67153284674,"y":137.0},{"x":75642.31707317074,"y":82.0},{"x":82496.13636363638,"y":44.0},{"x":91431.66666666667,"y":60.0},{"x":100665.71428571428,"y":21.0},{"x":107326.66666666667,"y":15.0},{"x":121248.74999999999,"y":16.0},{"x":142070.0,"y":2.0},{"x":153896.6666666667,"y":6.0},{"x":162310.0,"y":6.0},{"x":169810.0,"y":2.0},{"x":176740.0,"y":2.0},{"x":193925.0,"y":8.0},{"x":206770.0,"y":2.0}]

The results of this query comes back as a list, which is very un-SQL like! To get the data as a table, we can use LATERAL VIEW and EXPLODE:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
SELECT
   CAST(hist.x as int) as bin_center,
   CAST(hist.y as bigint) as bin_height
FROM (select
      histogram_numeric(salary, 20) as salary_hist
      from
      sample_08) a
LATERAL VIEW explode(salary_hist) exploded_table as hist;

--Results
	bin_center	bin_height
0	23507	255
1	31881	340
2	39824	287
3	47615	181
4	55667	164
5	59952	8
6	66034	137
7	75642	82
8	82496	44
9	91431	60
10	100665	21
11	107326	15
12	121248	16
13	142070	2
14	153896	6
15	162310	6
16	169810	2
17	176740	2
18	193925	8
19	206770	2

However, now that we have a table of data, it’s still not clear how to create a histogram, as the center of variable-width bins is what is returned by Hive. The Hive documentation for histogram_numeric() references Gnuplot, Excel, Mathematica and MATLAB, which I can only assume can deal with plotting the centers? Eventually I’ll figure out how to deal with this using R or Python, but for now, I just use the table as a quick gauge of what the data looks like.

Building JSON in R: Three Methods

Tue, 13 May 2014 13:27:39 +0000

When I set out to build RSiteCatalyst, I had a few major goals: learn R, build CRAN-worthy package and learn the Adobe Analytics API. As I reflect back on how the package has evolved over the past two years and what I’ve learned, I think my greatest learning was around how to deal with JSON (and strings in general).

JSON is ubiquitous as a data-transfer mechanism over the web, and R does a decent job providing the functionality to not only read JSON but also to create JSON. There are at least three methods I know of to build JSON strings, and this post will cover the pros and cons of each method.

Method 1: Building JSON using paste

As a beginning R user, I didn’t have the awareness of how many great user-contributed packages are out there. So throughout the RSiteCatalyst source code you can see gems like:

1
2
3
4
5
6
7
8
#"metrics" would be a user input into a function arguments
metrics <- c("a", "b", "c")

#Loop over the metrics list, appending proper curly braces
metrics_conv <- lapply(metrics, function(x) paste('{"id":', '"', x, '"', '}', sep=""))

#Collapse the list into a proper comma separated string
metrics_final <- paste(metrics_conv, collapse=", ")

The code above loops over a character vector (using lapply instead of a for loop like a good R user!), appending curly braces, then flattening the list down to a string. While this code works, it’s a quite brittle way to build JSON. You end up needing to worry about matching quotation marks, remembering if you need curly braces, brackets or singletons…overall, it’s a maintenance nightmare to build strings this way.

Of course, if you have a really simple JSON string you need to build, paste() doesn’t have to be off-limits, but for a majority of the cases I’ve seen, it’s probably not a good idea.

Method 2: Building JSON using sprintf

Somewhere in the middle of building version 1 of RSiteCatalyst, I started learning Python. For those of you who aren’t familiar, Python has a string interpolation operator %, which allows you to do things like the following:

1
2
3
In [1]: print "Here's a string subtitution for my name: %s" %("Randy")

Out[1]: "Here's a string subtitution for my name: Randy"

Thinking that this was the most useful thing I’d ever seen in programming, I naturally searched to see if R had the same functionality. Of course, I quickly learned that all C-based languages have printf/sprintf, and R is no exception. So I started building JSON using sprintf in the following manner:

1
2
3
4
5
elements_list = sprintf('{"id":"%s",
                          "top": "%s",
                          "startingWith":"%s",
                          "search":{"type":"%s", "keywords":[%s]}
                          }', element, top, startingWith, searchType, searchKW2)

In this example, we’re now passing R objects into the sprintf() function, with %s tokens everywhere we need to substitute text. This is certainly an improvement over paste(), especially given that Adobe provides example JSON via their API explorer. So I copied the example strings, replaced their examples with my tokens and voilà! Better JSON string building.

Method 3: Building JSON using a package (jsonlite, rjson or RJSONIO)

While sprintf() allowed for much easier JSON, there is still a frequent code smell in RSiteCatalyst, as evidenced by the following:

1
2
3
4
5
6
7
8
9
#Converts report_suites to JSON
if(length(report_suites)>1){
  report_suites <- toJSON(report_suites)
} else {
  report_suites <- toJSON(list(report_suites))
}

#API request
json <- postRequest("ReportSuite.GetTrafficVars",paste('{"rsid_list":', report_suites , '}'))

At some point, I realized that using the toJSON() function from rjson would take care of the formatting R objects to strings, yet I didn’t make the leap to understanding that I could build the whole string using R objects translated by toJSON()! So I have more hard-to-maintain code where I’m checking the class/length of objects and formatting them. The efficient way to do this using rjson would be:

1
2
3
4
5
6
7
#Efficient method
library(rjson)
report_suites <- list(rsid_list=c("A", "B", "C"))
request.body <- toJSON(report_suites)

#API request
json <- postRequest("ReportSuite.GetTrafficVars", request.body)

With the code above, we’re building JSON in a very R-looking manner; just R objects and functions, and in return getting the output we want. While it’s slightly less obvious what is being created by request.body, there’s literally zero bracket-matching, quoting issues or anything else to worry about in building our JSON. That’s not to say that there isn’t a learning curve to using a JSON package, but I’d rather figure out whether I need a character vector or list than burn my eyes out looking for mismatched quotes and brackets!

Collaborating Makes You A Better Programmer

Like any pursuit, you can get pretty far on your own through hard work and self-study. However, I wouldn’t be nearly where I am without collaborating with others (especially learning about how to build JSON properly in R!). A majority of the RSiteCatalyst code for the upcoming version 1.4 was re-written by Willem Paling, where he added consistency to keyword arguments, switched to jsonlite for better JSON parsing to Data Frames, and most importantly for the topic of this post, cleaned up the method of building all the required JSON strings!

Edit 5/13: For a more thorough example of building complex JSON using jsonlite, check out this example from the v1.4 branch of RSiteCatalyst. The linked example R code populates the required arguments from this JSON outline provide by Adobe.

Using SQL Workbench with Apache Hive

Fri, 25 Apr 2014 14:05:57 +0000

If you’ve spent any non-trivial amount of time working with Hadoop and Hive at the command line, you’ve likely wished that you could interact with Hadoop like you would any other database. If you’re lucky, your Hadoop administrator has already installed the Apache Hue front-end to your cluster, which allows for interacting with Hadoop via an easy-to-use browser interface. However, if you don’t have Hue, Hive also supports access via JDBC; the downside is, setup is not as easy as including a single JDBC driver.

While there are paid database administration tools such as Aqua Data Studio that support Hive, I’m an open source kind of guy, so this tutorial will show you how to use SQL Workbench to access Hive via JDBC. This tutorial assumes that you are proficient enough to get SQL Workbench installed on whatever computing platform you are using (Windows, OSX, or Linux).

Download Hadoop jars

The hardest part of using Hive via JDBC is getting all of the required jars. At work I am using a MapR distribution of Hadoop, and each Hadoop vendor platform provides drivers for their version of Hadoop. For MapR, all of the required Java .jar files are located at /opt/mapr/hive/hive-0.1X/lib (where X represents the Hive version number you are using).

Download all the .jar files in one shot, just in case you need them in the future

Since it’s not always clear which .jar files are required (especially for other projects/setups you might be doing), I just downloaded the entire set of files and placed them in a directory called hadoop_jars. If you’re not using MapR, you’ll need to find and download your vendor-specific version of the following .jar files:

hive-exec.jar
hive-jdbc.jar
hive-metastore.jar
hive-service.jar

Additionally, you will need the following general Hadoop jars (Note: for clarity/long-term applicability of this blog post, I have removed the version number from all of the jars):

hive-cli.jar
libfb303.jar
slf4j-api.jar
commons-logging.jar
hadoop-common.jar
httpcore.jar
httpclient.jar

Whew. Once you have the Hive JDBC driver and the 10 other .jar files, we can begin the installation process.

Setting up Hive JDBC driver

Setting up the JDBC driver is simply a matter of providing SQL Workbench with the location of all 11 of the required .jar files. After clicking File -> Manage Drivers, you’ll want to click on the white page icon to create a New Driver. Use the Folder icon to add the .jars:

For the Classname box, if you are using a relatively new version of Hive, you’ll be using Hive2 server. In that case, the Classname for the Hive driver is org.apache.hive.jdbc.HiveDriver (this should pop up on-screen, you just need to select the value). You are not required to put any value for the Sample URL. Hit OK and the driver window will close.

Connection Window

With the Hive driver defined, all that’s left is to define the connection string. Assuming your Hadoop administrator didn’t change the default port from 10000, your connection string should look as follows:

As stated above, I’m assuming you are using Hive2 Server; if so, your connection string will be jdbc:hive2://your-hadoop-cluster-location:10000. After that, type in your Username and Password and you should be all set.

Using Hive with SQL Workbench

Assuming you have achieved success with the instructions above, you’re now ready to use Hive like any other database. You will be able to submit your Hive code via the Query Window, view your schemas/tables (via the ‘Database Explorer’ functionality which opens in a separate tab) and generally use Hive like any other relational database.

Of course, it’s good to remember that Hive isn’t actually a relational database! From my experience, using Hive via SQL Workbench works pretty well, but the underlying processing is still in Hadoop. So you’re not going to get the clean cancelling of queries like you would with an RDBMS , there can be a significant lag to getting answers back (due to the Hive overhead), you can blow up your computer streaming back results larger than available RAM…but it beats working at the command line.

Real-time Reporting with the Adobe Analytics API

Mon, 10 Mar 2014 11:28:21 +0000

Starting with version 1.3.1 of RSiteCatalyst, you can now access the real-time reporting capabilities of the Adobe Analytics API through a familiar R interface. Here’s how to get started…

GetRealTimeConfiguration

Before using the real-time reporting capabilities of Adobe Analytics, you first need to indicate which metrics and elements you are interested in seeing in real-time. To see which reports are already set up for real-time access on a given report suite, you can use the GetRealTimeConfiguration() function:

1
2
#Get Real-Time reports that already set up
realtime_reports <- GetRealTimeConfiguration("<reportsuite>")

It’s likely the case that the first time you set this up, you’ll already see a real-time report for ‘Instances-Page-Site Section-Referring Domain’. You can leave this report in place, or switch the parameters using SaveRealTimeConfiguration().

SaveRealTimeConfiguration

If you want to add/modify which real-time reports are available in a report suite, you can use the SaveRealTimeConfiguration() function:

1
2
3
4
5
6
7
8
9
10
SaveRealTimeConfiguration("<report suite>",
  metric1 = "instances",
  elements1 = c("page", "referringdomain", "sitesection"),

  metric2 = "revenue",
  elements2 = c("referringdomain", "sitesection")

  metric3 = "orders",
  elements3 = c("products")
)

Up to three real-time reports are available to be stored at any given time. Note that you can mix-and-match what reports you want to modify, you don’t have to submit all three reports at a given time. Finally, keep in mind that it can take up to 15 minutes for the API to incorporate your real-time report changes, so if you don’t get your data right away don’t keep re-submitting the function call!

GetRealTimeReport

Once you have your real-time reports set up in the API, you can use the GetRealTimeReport() function in order to access your reports. There are numerous parameters for customization; selected examples are below.

Minimum Example - Overtime Report

The simplest function call for a real-time report is to create an Overtime report (monitoring a metric over a specific time period):

1
rt <- GetRealTimeReport("<report suite>", "instances")

The result of this call will be a DataFrame having 15 rows of one minute granularity for your metric. This is a great way to monitor real-time orders & revenue during a flash sale, see how users are accessing a landing page for an email marketing campaign or any other metric where you want up-to-the-minute status updates.

Granularity, Offset, Periods

If you want to have a time period other than the last 15 minutes, or one minute granularity is too volatile for the metric you are monitoring, you can add additional arguments to modify the returned DataFrame:

1
2
3
4
5
rt2 <- GetRealTimeReport("<reportsuite>",
                  "instances",
                  periodMinutes = "5",
                  periodCount = "12",
                  periodOffset = "10"

For this function call, we will receive instances for the last hour (12 periods) of five minute granularity, with a 10 minute offset (meaning, now - 10 minutes ago is the first time period reported).

Single Elements

Beyond just monitoring a metric over time, you can specify an element such as page to receive your metrics by:

1
2
3
4
5
GetRealTimeReport("<reportsuite>",
                  "instances",
                  "page",
                  periodMinutes = "9",
                  periodCount = "3")

This function call will return Instances by Page, for the last 27 minutes (3 rows/periods per page, 9 minute granularity…just because!). Additionally, there are other arguments such as algorithm, algorithmArgument, firstRankPeriod and floorSensitivity that allow for creating reports similar to what is provided in the Real-Time tab in the Adobe Analytics interface.

Currently, even through the Adobe Analytics API supports real-time reports with three breakdowns, only one element breakdown is supported by RSiteCatalyst; it is planned to extend these functions in RSiteCatalyst to full support the real-time capabilities in the near future.

From DataFrame to Something ‘Shiny’

If we’re talking real-time reports, we’re probably talking about dashboarding. If we’re talking about R and dashboarding, then naturally, ggvis/Shiny comes to mind. While providing a full ggvis/Shiny example is beyond the scope of this blog post, it’s my hope to provide a working example in a future blog post. Stay tuned!

RSiteCatalyst Version 1.3 Release Notes

Tue, 04 Feb 2014 09:44:19 +0000

Version 1.3 of the RSiteCatalyst package to access the Adobe Analytics API is now available on CRAN! Changes include:

Search via regex functionality in QueueRanked/QueueTrended functions
Support for Realtime API reports: Overtime and one-element Ranked report
Allow for variable API request timing in Queue*` functions
Fixed validate flag in JSON request to work correctly
Deprecated GetAdminConsoleLog (appears to be removed from the API)

Searching via Regex functionality

RSiteCatalyst now supports the search functionality of the API, similar in nature to using the Advanced Filter/Search feature within Reports & Analytics. Here are some examples for the QueueRanked function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#Top 100 Pages where the pagename starts with "Categories"
#Uses searchKW argument
queue_ranked_pages_search <- QueueRanked("<reportsuite>",
                                         "2013-01-01",
                                         "2014-01-28",
                                         c("pageviews", "visits"),
                                         "page",
                                         top = "100",
                                         searchKW = "^Categories"  
                                          )

#Top 100 Pages where the pagename starts with "Categories" OR contains "Home Page"
#Uses searchKW and searchType arguments
queue_ranked_pages_search_or <- QueueRanked("<reportsuite>",
                                            "2013-01-01",
                                            "2014-01-28",
                                            c("pageviews", "visits"),
                                            "page",
                                            top = "100",
                                            searchKW = c("^Categories", "Home Page"),
                                            searchType = "OR"
                                            )

QueueTrended function calls work in a similar manner, returning elements broken down by time rather than a single record per element name.

Realtime Reporting API

Accessing the Adobe Analytics Realtime API now has limited support in RSiteCatalyst. Note that this is different than just using the currentData parameter within the Queue* functions, as the realtime API methods provide data within a minute of that data being generated on-site. Currently, RSiteCatalyst only supports the most common types of reports: Overtime (no eVar or prop breakdown) and one-element breakdown.

Because of the extensive new functionality for the GetRealTimeConfiguration(), SaveRealTimeConfiguration() and GetRealTimeReport() functions, code examples will be provided as a separate blog post.

Variable request timing for Queue function calls

This feature is to fix the issue of having an API request run so long that RSiteCatalyst gave up on retrieving an answer. Usually, API requests come back in a few seconds, but in selected cases a call could run so long as to exhaust the number of attempts (previously, 10 minutes). You can use the maxTries and waitTime arguments to specify how many times you’d like RSiteCatalyst to retrieve the report and the wait time between calls:

1
2
3
4
5
6
7
8
9
10
11
12
#Change timing of function call
#Wait 30 seconds between attempts to retrieve the report, try 5 times
queue_overtime_visits_pv_day_social_anomaly2 <- QueueOvertime("<reportsuite>",
                                                              "2013-01-01",
                                                              "2014-01-28",
                                                              c("visits", "pageviews"),
                                                              "day",
                                                              "Visit_Social",
                                                              anomalyDetection = "1",
                                                              currentData = "1",
                                                              maxTries = 5,
                                                              waitTime = 30)

If you don’t specify either of these arguments, RSiteCatalyst will default to trying every five seconds to retrieve the report, up to 120 tries.

New Contributor: Willem Paling

I’m pleased to announce that I’ve got a new contributor for RSiteCatalyst, Willem Paling! Willem did a near-complete re-write of the underlying code to access the API, and rather than have multiple packages out in the wild, we’ve decided to merge our works. So look forward to better-written R code and more complete access to the Adobe Analytics API’s in future releases…

Support

If you run into any problems with RSiteCatalyst, please file an issue on GitHub so it can be tracked properly. Note that I’m not an Adobe employee, so I can only provide so much support, as in most cases I can’t validate your settings to ensure you are set up correctly (nor do I have any inside information about how the system works :) )

Edit 2/20/2014: I mistakenly forgot to add the new real-time functions to the R NAMESPACE file, and as such, you won’t be able to use them if you are using version 1.3. Upgrade to 1.3.1 to access the real-time functionality.

Getting Started With Hadoop, Final: Analysis Using Hive & Pig

Sun, 12 Jan 2014 20:25:32 +0000

We’ve finally made it to the final post in this tutorial! In my prior posts about getting started with Hadoop, we’ve covered the entire lifecycle from how to set up a small cluster using Amazon EC2 and Cloudera through how to load data using Hue. With our data loaded in HDFS, we can finally move on to the actual analysis portion of the airline dataset using Hive and Pig.

Basic Descriptive Statistics Using Hive

In part 4 of this tutorial, we used a Hive script to create a view named “vw_airline” to hold all of our airline data. Running a simple query is as easy as running the following in the Hive window in Hue. Note that this is ANSI-standard SQL code, even though we are submitting it using Hive:

A simple query like this is a great way to get a feel for the table, including determining whether or not the files were loaded correctly. Once the results are displayed, you can create simple visualizations like bar charts, line plots, scatterplots and pie charts. The results of the following query are shown below. Knowing this dataset, I can tell that the files were loaded incorrectly; the dips at Years 1994 and 2004 are too few records and will need to be reloaded.

1994 and 2004 have too few rows, which was validated using wc -l 1994.csv at the command line (outside of Hadoop)

Besides just simple counts, Hive supports nearly all standard SQL syntax relative to functions such as SUM, COUNT, MIN, MAX, etc., table joins, user-defined functions (UDF)`, window functions…pretty much everything that you are used to from other SQL tools. AFAIK, the only thing that Hive doesn’t support is nested sub-queries, but that’s on the Stinger initiative for improving Hive. However, depending on the nested subquery being performed, you might be able to accomplish the same thing using a LEFT SEMI JOIN.

Using Pig for Analytics

It’s important to realize that Hadoop isn’t just another RDBMS where you run SQL. Using Pig, you can write scripts for calculation in a similar manner to using other high-level languages such as Python or R.

For example, suppose we wanted to calculate the average distance for each route. A Pig script to calculate this might look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
--Load data from view to use
air = LOAD 'default.vw_airline' USING org.apache.hcatalog.pig.HCatLoader();

--Use FOREACH to limit data to origin, dest, distance
--Concatentate origin and destination together, separated by a pipe
--CONCAT appears to only allow two arguments, which is why the function is called twice (to allow 3 arguments)
origindest = FOREACH air generate CONCAT(origin, CONCAT('|' , dest)) as route, distance;

--Group origindest dataset by route
groupedroutes = GROUP origindest BY (route);

--Calculate average distance by route
avg_distance = FOREACH groupedroutes GENERATE group, AVG(origindest.distance);

--Show results in Pig shell
dump avg_distance;

--Write out results to text file, separated by tab (default)
store avg_distance into '/user/hue/avg_distance';

While it is possible to calculate average distance using Hive and a GROUP BY statement, one of the benefits to using Pig is having control over every step of the data flow. So while Hive queries tend to answer a single question at a time, Pig allows an analyst to chain together any number of steps in a data flow. In the example above, we could pass the average distance for each route to another transformation, join it back to the original dataset or do anything else our analyst minds can imagine!

Summary

Over these five blog posts, I’ve outlined how to get started with Hadoop and ‘Big Data’ using Amazon and Cloudera/Hortonworks. Hopefully I’ve been able to demystify the concepts and terminology behind Hadoop, shown that setting up a Hadoop using Cloudera on Amazon EC2 isn’t unsurmountable, and loading data and analyzing it using Hive and Pig isn’t dramatically different than using SQL on other database systems you may have encountered in the past.

While there’s a lot of hype around ‘Big Data’, data sizes aren’t going to be getting any smaller in the future. So spend the $20 in AWS charges and build a Hadoop cluster! There’s no better way to learn than by doing…

Quickly Create Dummy Variables in a Data Frame

Thu, 02 Jan 2014 13:58:51 +0000

On Quora, a question was asked about how to fix the error of the randomForest package in R not being able to handle more than 32 levels in a categorical variable. Seeing as how I’ve seen this question asked on Kaggle forums, StackOverflow and elsewhere, here’s the answer: code your own dummy variables instead of relying on Factors!

Code snippet

1
2
3
4
5
6
7
8
9
10
#Generate example dataframe with character column
example <- as.data.frame(c("A", "A", "B", "F", "C", "G", "C", "D", "E", "F"))
names(example) <- "strcol"

#For every unique value in the string column, create a new 1/0 column
#This is what Factors do "under-the-hood" automatically when passed to function requiring numeric data
for(level in unique(example$strcol)){
  example[paste("dummy", level, sep = "_")] <- ifelse(example$strcol == level, 1, 0)
}
view raw

As the code above shows, it’s trivial to generate your own 1/0 columns of data instead of relying on Factors. There are two things to keep in mind when creating your own dummy variables:

The problem you are trying to solve
How much RAM you have available

While it may make sense to generate dummy variables for Customer State (~50 for the United States), if you were to use the code above on City Name, you’d likely either run out of RAM or find out that there are too many levels to be useful. Of course, with any qualitative statement such as “too many levels to be useful”, oftentimes the only way to definitively know is to try it! Just make sure you save your work before running this code, just in case you run out of RAM. Or, use someone else’s computer for testing 😉

Edit 1/2/14: John Myles White brought up a good point via Twitter about RAM usage:

@randyzwitch If you're running out of RAM with dummy variables, you probably want to use a sparse matrix instead of a data.frame.

— John Myles White (@johnmyleswhite) January 2, 2014

Adobe Analytics Implementation Documentation in 60 Seconds

Mon, 09 Dec 2013 09:46:30 +0000

When I was working as a digital analytics consultant, no question quite had the ability to cause belly laughs AND angst as, “Can you send me an updated copy of your implementation documentation?” I saw companies that were spending six-or-seven-figures annually on their analytics infrastructure, multi-millions in salary for employees and yet the only way to understand what data they were collecting was to inspect their JavaScript code.

Luckily for Adobe Analytics customers, the API provides a means of generating the framework for a properly-documented implementation. Here’s how to do it using RSiteCatalyst.

Generating Adobe Analytics documentation file

The code below outlines the commands needed to generate an Excel file (see example) with six tabs containing the basic structure of an Adobe Analytics. This report contains all of the report suites you have access to, the elements that reports can be broken down by, traffic variables (props), conversion variables (eVars) and segments available for reporting.

Additionally, within each tab metadata is provided that contains the various settings for variables, so you’ll be able to document the expiration settings for eVars, participation, list variables, segment types and so on.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
library("RSiteCatalyst")
library("WriteXLS")

#Validate that underlying Perl modules for WriteXLS are installed correctly
#Will return "Perl found. All required Perl modules were found" if installed correctly
testPerl()

#### 1. Pull data for all report suites to create one comprehensive report ####

#Authenticate with Adobe Analytics API
SCAuth("user:company", "sharedsecret")

#Get Report Suites
report_suites <- GetReportSuites()

#Get Available Elements
elements <- GetElements(report_suites$rsid)

#Get eVars
evars <- GetEvars(report_suites$rsid)

#Get Segments
segments <- GetSegments(report_suites$rsid)

#Get Success Events
events <- GetSuccessEvents(report_suites$rsid)

#Get Traffic Vars
props <- GetProps(report_suites$rsid)

#### 2. Generate a single Excel file

#Create list of report suite objects, written as strings
objlist <- c("report_suites", "elements", "evars", "segments", "events", "props")

#Write out Excel file with auto-width columns, a bolded header row and filters turned on
WriteXLS(objlist, "/Users/randyzwitch/Desktop/adobe_analytics_implementation_doc.xlsx",
         AdjWidth = TRUE, BoldHeaderRow = TRUE, AutoFilter = TRUE)

The only “gotchas” to keep in mind when using the script above is that the user running this will only receive data for report suites they have access to (which is determined by Admin panel setting within Adobe Analytics) and that you need to have the WriteXLS package installed to write to Excel. The WriteXLS package uses Perl as the underlying code, so you’ll need to validate that the package is installed correctly, which is done using the testPerl() function in the package.

This is pretty bare-bones, no?

After you run this code, you’ll have an Excel file that has all of the underlying characteristics of your Adobe Analytics implementation. It’s important to realize that this is only the starting point; a great set of documentation will contain other pieces of information such as where/when the value is set (on entry, every page, when certain events occur, etc.), a layman’s explanation about what the data element means and other business information so your stakeholders can be confident they are using the data correctly. Additionally, you might consider creating a single Excel file for every report suite in your implementation. It’s trivial to modify the code above to subset each data frame used above for a single value of rsid, then write to separate Excel files. Regardless of how your structure your documentation, DOCUMENT YOUR IMPLEMENTATION! The employees that come after you (and your future self!) will thank you.

EDIT 2/4/2016: Thanks to reader @CSitty for pointing out the R code became a little stale. The documentation generating code should now work again for RSiteCatalyst versions >= 1.4 and WriteXLS >= 4.0 (basically, any current version as of the time of this update).

Using Amazon EC2 with IPython Notebook

Thu, 21 Nov 2013 20:13:11 +0000

Last week, I wrote a guest blog post at Bad Hessian about how to use IPython Notebook along with Amazon EC2 as your data science & analytics platform. I won’t reproduce the whole article here, but if you are interested in step-by-step instruction on how to setup an Amazon EC2 instance to use IPython Notebook, see the SlideShare presentation below which outlines the steps needed to setup a remote IPython Notebook environment (or, PDF download).

If you already have experience setting up EC2 images and just need the IPython Notebook settings, here are the commands that are needed to set up your IPython public notebook server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
#### Start IPython, generate SHA1 password to use for IPython Notebook server

$ ipython
Python 2.7.5 |Anaconda 1.8.0 (x86_64)| (default, Oct 24 2013, 07:02:20)
Type "copyright", "credits" or "license" for more information.

IPython 1.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from IPython.lib import passwd

In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'sha1:207eb1f4671f:92af695...'

#### Create nbserver profile

$ ipython profile create nbserver
[ProfileCreate] Generating default config file: u'/.ipython/profile_nbserver/ipython_config.py'
[ProfileCreate] Generating default config file: u'/.ipython/profile_nbserver/ipython_qtconsole_config.py'
[ProfileCreate] Generating default config file: u'/.ipython/profile_nbserver/ipython_notebook_config.py'
[ProfileCreate] Generating default config file: u'/.ipython/profile_nbserver/ipython_nbconvert_config.py'

#### Create self-signed SSL certificate

$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem

#### Modify ipython_notebook_config.py configuration file
#### Add these lines to the top of the file; no other changes necessary
#### Obviously, you'll want to add your path to the .pem key and your password

# Configuration file for ipython-notebook.

c = get_config()

# Kernel config
c.IPKernelApp.pylab = 'inline'  # if you want plotting support always

# Notebook config
c.NotebookApp.certfile = u'/home/ubuntu/certificates/mycert.pem'
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:207eb1f4671f:92af695...'
# It is a good idea to put it on a known, fixed port
c.NotebookApp.port = 8888

#### Start IPython Notebook on the remote server

$ ipython notebook --profile=nbserver

Happy IPython Notebooking!

Adding Line Numbers in IPython/Jupyter Notebooks

Tue, 19 Nov 2013 08:48:18 +0000

Lately, I’ve been using Jupyter Notebooks for all of my Python and Julia coding. The ability to develop and submit small snippets of code and create plots inline is just so useful that it has broken the stranglehold of using an IDE while I’m coding. However, the one thing that was missing for a smooth transition was line numbers in the cells; luckily, this can be achieved in two ways.

Keyboard Shortcut

The easiest way to add line numbers to a Jupyter Notebook is to use the keyboard shortcut, which is Ctrl-m to enter Command Mode, then type L. Just highlight the cell you are interested in adding line numbers to, then hit the keyboard shortcut to toggle the line numbers.

Add Line Numbers to All Cells at Startup

~~While the keyboard shortcut is great for toggling line numbers on/off, I prefer having line numbers always on. Luckily, the IPython Dev folks on Twitter were kind enough to explain how to do this:~~

~~@randyzwitch add `IPython.Cell.options_default.cm_config.lineNumbers = true;` to your custom.js~~

~~— IPython Dev (@IPythonDev) October 28, 2013~~

~~I use OSX with the default ‘profile_default’ profile, so the path for my custom.js file for IPython is:~~

/Users/randyzwitch/.ipython/profile_default/static/custom/

~~Similarly, you can do the same for IJulia:~~

/Users/randyzwitch/.ipython/profile_julia/static/custom

If you are using a different operating system than OSX, or you are using OSX and you don’t see a custom.js file in these locations, a quick search for custom.js will get you to the right file location. Once you open up the custom.js file, you can place the line of JavaScript anywhere in the file, as long as it’s not inside any of any pre-existing functions in the file.

Once you place the line of JavaScript in your file, you’ll need to restart IPython/IJulia completely for the change to take effect. After that, you’ll have line numbers in each cell, each Notebook!

Edit 11/4/2015: Thanks to reader Nat Dunn, I’ve been made aware that the above method no longer works, which isn’t a surprise given the amount of changes between IPython Notebook to the entire Jupyter project in the past 2 years.

For the (currently) correct method of adding line numbers to Jupyter Notebook by default, please see Nat’s post with the correct instructions on modifying the custom.js file.

RSiteCatalyst Version 1.2 Release Notes

Tue, 05 Nov 2013 08:29:16 +0000

Version 1.2 of the RSiteCatalyst package to access the Adobe Analytics API is now available on CRAN! Changes include:

Removed RCurl package dependency
Changed argument order for GetAdminConsoleLog to avoid error when date not passed
Return proper numeric type for metric columns
Fixed bug in GetEVars function
Added validate:true flag to API to improve error reporting
Removed remaining references to Omniture

For the most part, the only noticeable change for most users will be that you no longer need to call as.numeric() on a DataFrame after getting the results of an API call, as all functions now return the proper numeric type.

Changes from Development Version

For any of you out there that may have installed the 1.2 development version directly from GitHub, the only difference between the 1.2 development version and the stable, CRAN version of the package is that support for the Adobe Analytics Real Time API has been removed. This functionality will continue to be developed on the 1.3 development branch on GitHub.

Testing

For this release, I’ve made a more concerted effort to test RSiteCatalyst on various platforms outside of OSX (where I do my development). RSiteCatalyst works in the following environments:

OSX Lion and prior
Ubuntu 12.04 LTS
Windows 7 64-bit SP1
Windows 8.1 64-bit
R 2.15.2 and newer
R and RStudio

If your environment is not listed above, it is still likely the case that RSiteCatalyst will work in your environment, as there is no operating-system-specific code in the package. If you are finding issues, validate that you have all package dependencies installed, your Adobe account has Web Service Access privileges (set in Admin panel), you have permission access to the report suites you are trying to access (also an Admin panel setting) and that your company doesn’t have any firewall settings that would prevent API access.

Support

Clustering Search Keywords Using K-Means Clustering

Tue, 17 Sep 2013 14:41:01 +0000

One of the key tenets to doing impactful digital analysis is understanding what your visitors are trying to accomplish. One of the easiest methods to do this is by analyzing the words your visitors use to arrive on site (search keywords) and what words they are using while on the site (on-site search).

Although Google has made it much more difficult to analyze search keywords over the past several years (due to their passing of “(not provided)” instead of the actual keywords), we can create customer intent segments based on the keywords that are still being passed using unsupervised clustering methods such as k-means clustering.

Concept: K-Means Clustering/Unsupervised Learning

K-means clustering is one of many techniques within unsupervised learning that can be used for text analysis. Unsupervised refers to the fact that we’re trying to understand the structure of our underlying data, rather than trying to optimize for a specific, pre-labeled criterion (such as creating a predictive model for conversion). Unsupervised learning is a great technique for exploratory analysis in that the analyst enforces few assumptions on the data, so previously unexamined relationships can be determined then analyzed; contrast that with pre-defined relationships specified by the analyst (such as visitors from mobile or visitors from social), then evaluating how various metrics differ across these pre-defined groups.

Without getting too technical, k-means clustering is a method of partitioning data into ‘k’ subsets, where each data element is assigned to the closest cluster based on the distance of the data element from the center of the cluster. In order to use k-means clustering with text data, we need to do some text-to-numeric transformation of our text data. Luckily, R provides several packages to simplify the process.

Converting Text to Numeric Data: Document-Term Matrix

Since I use Adobe Analytics on this blog, I’m going to use the RSiteCatalyst package to get my natural search keywords into a dataframe. Once the keywords are in a dataframe, we can use the RTextTools package to create a document-term matrix, where each row is our search term and each column is a 1/0 representation of whether a single word is contained within natural search term.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#### 0. Setup
library("RSiteCatalyst")
library("RTextTools") #Loads many packages useful for text mining

#### 1. RSiteCatalyst code - Get Natural Search Keywords & Metrics

#Set credentials
SCAuth(<username:company>, <shared secret>)

#Get list of search engine terms
searchkeywords <- QueueRanked(<report_suite>, "2013-02-01","2013-09-16",
                  c("entries", "visits", "pageviews", "instances", "bounces"),
                  "searchenginenaturalkeyword", top="100000", startingWith = "1")

#### 2. Process keywords into format suitable for text mining

#Create document-term matrix, passing data cleaning options
#Stem the words to avoid multiples of similar words
#Need to set wordLength to minimum of 1 because "r" a likely term
dtm <- create_matrix(searchkeywords$'Natural Search Keyword',
                     stemWords=TRUE,
                     removeStopwords=FALSE,
                     minWordLength=1,
                     removePunctuation= TRUE)

Within the create_matrix function, I’m using four keyword arguments to process the data:

stemWords reduces a word down to its root, which is a standardization method to avoid having multiple versions of words referring to the same concept (e.g. argue, arguing, argued reduces to ‘argu’)
removeStopwords eliminates common English words such as “they”, “he” , “always”
minWordLength sets the minimum number of characters that constitutes a ‘word’, which I set to 1 because of the high likelihood of ‘r’ being a keyword
removePunctuation removes periods, commas, etc.

Popular Words

If you are unfamiliar with the terms that might be contained in your dataset, you can use the findFreqTerms to see which terms occur with a minimum frequency. Here are the terms that occur at least 20 times on this blog:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
> #Inspect most popular words, minimum frequency of 20
> findFreqTerms(dtm, lowfreq=20)
  [1] "15"           "2008"         "2009"         "2011"         "a"            "ad"           "add"          "adsens"      
  [9] "air"          "analyt"       "and"          "appl"         "at"           "back"         "bezel"        "black"       
 [17] "book"         "bookmark"     "break"        "broke"        "broken"       "bubbl"        "by"           "can"         
 [25] "case"         "chang"        "child"        "code"         "comment"      "comput"       "cost"         "cover"       
 [33] "crack"        "css"          "custom"       "data"         "delet"        "disabl"       "display"      "do"          
 [41] "doe"          "drop"         "edit"         "eleven"       "em209"        "entri"        "fix"          "footer"      
 [49] "footerphp"    "for"          "free"         "from"         "get"          "glue"         "googl"        "hadoop"      
 [57] "header"       "hing"         "how"          "i"            "if"           "imag"         "in"           "is"          
 [65] "it"           "laptop"       "late"         "lcd"          "lid"          "link"         "logo"         "loos"        
 [73] "mac"          "macbook"      "make"         "mobil"        "modifi"       "much"         "my"           "navig"       
 [81] "of"           "off"          "omnitur"      "on"           "page"         "permalink"    "php"          "post"        
 [89] "power"        "pro"          "problem"      "program"      "proud"        "r"            "remov"        "repair"      
 [97] "replac"       "report"       "sas"          "screen"       "separ"        "site"         "sitecatalyst" "store"       
[105] "tag"          "text"         "the"          "theme"        "this"         "tighten"      "to"           "top"         
[113] "turn"         "twenti"       "twentyeleven" "uncategor"    "unibodi"      "use"          "variabl"      "version"     
[121] "view"         "vs"           "warranti"     "was"          "what"         "will"         "with"         "wordpress"   
[129] "wp"           "you"

Guessing at ‘k’: A First Run at Clustering

Once we have our data set up, we can very quickly run the k-means algorithm within R. The one downside to using k-means clustering as a technique is that the user must choose ‘k’, the number of clusters expected from the dataset. In absence of any heuristics about what ‘k’ to use, I can guess that there are five topics on this blog: 1. Data Science

Digital Analytics
R
Julia
WordPress

Running the following code, we can see if the algorithm agrees:

1
2
3
4
5
6
7
8
9
10
11
12
13
#I think there are 5 main topics: Data Science, Web Analytics, R, Julia, Wordpress
kmeans5<- kmeans(dtm, 5)

#Merge cluster assignment back to keywords
kw_with_cluster <- as.data.frame(cbind(searchkeywords$'Natural Search Keyword', kmeans5$cluster))
names(kw_with_cluster) <- c("keyword", "kmeans5")

#Make df for each cluster result, quickly "eyeball" results
cluster1 <- subset(kw_with_cluster, subset=kmeans5 == 1)
cluster2 <- subset(kw_with_cluster, subset=kmeans5 == 2)
cluster3 <- subset(kw_with_cluster, subset=kmeans5 == 3)
cluster4 <- subset(kw_with_cluster, subset=kmeans5 == 4)
cluster5 <- subset(kw_with_cluster, subset=kmeans5 == 5)

Opening the dataframes to observe the results, it seems that the algorithm disagrees:

Cluster 1: “Free-for-All” cluster: not well separated (41.1% of terms)
Cluster 2: “wordpress” and “remove” (4.9% of terms)
Cluster 3: “powered by wordpress” (4.3% of terms)
Cluster 4: “twenty eleven” (13.5% of terms)
Cluster 5: “macbook” (36.2% of terms)

Of the clusters, the strongest cluster in terms of performance is cluster 5, which is pretty homogenous in terms of being about ‘macbook’ terms. Clusters 2-4 are all about WordPress, albeit different topics surrounding blogging. And cluster 1 is a large hodge-podge of terms that seem unrelated. Clearly, five clusters isn’t the proper value for ‘k’.

Selecting ‘k’ Using ‘Elbow Method’

Instead of randomly choosing values of ‘k’, then looking at each cluster result until we find one we like, we can take a more automated approach to picking ‘k’. For every kmeans object returned by R, there is a metric tot.withinss that provides the total of the squared distance metric for each cluster.

1
2
3
4
5
6
7
8
9
10
11
12
13
#accumulator for cost results
cost_df <- data.frame()

#run kmeans for all clusters up to 100
for(i in 1:100){
  #Run kmeans for each level of i, allowing up to 100 iterations for convergence
  kmeans<- kmeans(x=dtm, centers=i, iter.max=100)

  #Combine cluster number and cost together, write to df
  cost_df<- rbind(cost_df, cbind(i, kmeans$tot.withinss))

}
names(cost_df) <- c("cluster", "cost")

The cost_df dataframe accumulates the results for each run, which can then be plotted using ggplot2 (ggplot2 Gist here):

The plot above is a technique known informally as the ‘elbow method’, where we are looking for breakpoints in our cost plot to understand where we should stop adding clusters. We can see that the slope of the cost function gets flatter at 10 clusters, then flatter again around 20 clusters. This means that as we add clusters above 10 (or 20), each additional cluster becomes less effective at reducing the distance from the each data center (i.e. reduces the variance less). So while we haven’t determined an absolute, single ‘best’ value of ‘k’, we have narrowed down a range of values for ‘k’ to evaluate.

Ultimately, the best value of ‘k’ will be determined as a combination of a heuristic method like the ‘Elbow Method’, along with analyst judgement after looking at the results. Once you’ve determined your optimal cluster definitions, it’s trivial to calculate metrics such as Bounce Rate, Pageviews per Visit, Conversion Rate or Average Order Value to see how well the clusters actually describe different behaviors on-site.

Summary

K-means clustering is one of many unsupervised learning techniques that can be used to understand the underlying structure of a dataset. When used with text data, k-means clustering can provide a great way to organize the thousands-to-millions of words being used by your customers to describe their visits. Once you understand what your customers are trying to do, you can tailor your on-site experiences to match these needs, as well as adjusting your reporting/dashboards to monitor the various customer groups.

EDIT: For those who want to play around with the code but don’t use Adobe Analytics, here is the file of search keywords I used. Once you read in the .csv file into a dataframe and name it searchkeywords, you should be able to replicate everything in this blog post.

Fun With Just-In-Time Compiling: Julia, Python, R and pqR

Mon, 02 Sep 2013 19:57:45 +0000

Recently I’ve been spending a lot of time trying to learn Julia by doing the problems at Project Euler. What’s great about these problems is that it gets me out of my normal design patterns, since I don’t generally think about prime numbers, factorials and other number theory problems during my normal workday. These problems have also given me the opportunity to really think about how computers work, since Julia allows the programmer to pass type declarations to the just-in-time compiler (JIT).

As I’ve been working on optimizing my Julia code, I decided to figure out how fast this problem can be solved using any of the languages/techniques I know. So I decided to benchmark one of the Project Euler problems using Julia, Python, Python with Numba, PyPy, R, R using the compiler package, pqR and pqR using the compiler package. Here’s what I found…

Problem

The problem I’m using for the benchmark is calculating the smallest number that is divisible by all of the numbers in a factorial. For example, for the numbers in 5!, 60 is the smallest number that is divisible by 2, 3, 4 and 5. Here’s the Julia code:

1
2
3
4
5
6
7
8
9
10
11
function smallestdivisall(n::Int64)
    for i = 1:factorial(n)
        for j = 1:n
            if i % j !=0
                break
            elseif j == n
                return i
            end
        end
    end
end

All code versions follow this same pattern: the outside loop will run from 1 up to n!, since by definition the last value in the loop will be divisible by all of the numbers in the factorial. The inner loops go through and do a modulo calculation, checking to see if there is a remainder after division. If there is a remainder, break out of the loop and move to the next number. Once the state occurs where there is no remainder on the modulo calculation and the inner loop value of j equals the last number in the factorial (i.e. it is divisible by all of the factorial numbers), we have found the minimum number.

Benchmarking - Overall

Here are the results of the eight permutations of languages/techniques (see this GitHub Gist for the actual code used, this link for results file, and this GitHub Gist for the ggplot2 code):

Across the range of tests from 5! to 20!, Julia is the fastest to find the minimum number. Python with Numba is second and PyPy is third. pqR fares better than R in general, but using the compiler package can narrow the gap.

To make more useful comparisons, in the next section I’ll compare each language to its “compiled” function state.

Benchmarking - Individual

Python

Amongst the native Python code options, I saw a 16x speedup by using PyPy instead of Python 2.7.6 (10.62s vs. 172.06s at 20!). Using Numba with Python instead of PyPy nets an incremental ~40% speedup using the @autojit decorator (7.63s vs. 10.63 at 20!).

So in the case of Python, using two lines of code with the Numba JIT compiler you can get substantial improvements in performance without needing to do any code re-writes. This is a great benefit given that you can stay in native Python, since PyPy doesn’t support all existing packages within the Python ecosystem.

R/pqR

It’s understood in the R community that loops are not a strong point of the language. In the case of this problem, I decided to use loops because 1) it keeps the code pattern similar across languages and 2) I hoped I’d see the max benefit from the compiler package by not trying any funky R optimizations up front.

As expected, pqR is generally faster than R and using the compiler package is faster than not using the compiler. I saw ~30% improvement using pqR relative to R and ~20% incremental improvement using the compiler package with pqR. Using the compiler package within R showed ~35% improvement.

So unlike the case with Python, where you could just use Python with Numba and stay within the same language/environment, if you can use pqR and the compiler package, you can get a performance benefit from using both.

Summary

For a comparison like I’ve done above, it’s easy to get carried away and extrapolate the results from one simple test to all programming problems ever. “Julia is the best language for all cases ever!!!11111eleventy!” would be easy to proclaim, but all problems aren’t looping problems using simple division. Once you get into writing longer programs, other tasks such string manipulation and accessing APIs, using a technique from a package only available in one ecosystem but not another, etc., which tool is “best” for solving a problem becomes a much more difficult decision. The only way to know how much improvement you can see from different techniques & tools is to profile your program(s) and experiment.

The main thing that I took away from this exercise is that no matter which tool you are comfortable with to do analysis, there are potentially large performance improvements that can be made just by using a JIT without needing to dramatically re-write your code. For those of us who don’t know C (and/or are too lazy to re-write our code several times to wring out a little extra performance), that’s a great thing.

RSiteCatalyst Version 1.1 Release Notes

Sun, 25 Aug 2013 13:54:32 +0000

RSiteCatalyst version 1.1 is now available on CRAN. Changes from version 1 include:

Support for Correlations/Subrelations in the QueueRanked function
Support for Current Data in all Queue* functions
Support Anomaly Detection for QueueOvertime and QueueTrended functions (example usage with ggplot2 graph)
Decrease in wait time for API calls (from 5 seconds to 2 seconds) and extending total number of API tries before report failure (from 100 seconds to 10 minutes)

For those of you Adobe Analytics (Omniture) users who haven’t yet tried to use the Adobe Analytics API, I’ve created an introduction video to get started. There will also continue to be examples of using this package on this blog on the RSiteCatalyst tag. Enjoy!

Getting Started Using Hadoop, Part 4: Creating Tables With Hive

Thu, 22 Aug 2013 18:03:19 +0000

In the previous three tutorials (1, 2, 3), we’ve covered the background of Hadoop, how to build a proof-of-concept Hadoop cluster using Amazon EC2 and how to upload a .zip file to the cluster using Hue. In Part 4, we’ll use the data uploaded from the .zip file to create a master table of all files, as well as creating a view.

Creating Tables Using Hive

Like SQL for ‘regular’ relational databases, Hive is the tool we can use within Hadoop to create tables from data loaded into HDFS. Because Hadoop was built with large, messy data in mind, there are some amazingly convenient features for creating and loading data, such as being able to load all files in a directory (assuming they have the same format). Here’s the Hive statement we can use to load the airline dataset:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
-- Create table from yearly airline csv files
CREATE EXTERNAL TABLE airline (
  `Year` int,
  `Month` int,
  `DayofMonth` int,
  `DayOfWeek` int,
  `DepTime`  int,
  `CRSDepTime` int,
  `ArrTime` int,
  `CRSArrTime` int,
  `UniqueCarrier` string,
  `FlightNum` int,
  `TailNum` string,
  `ActualElapsedTime` int,
  `CRSElapsedTime` int,
  `AirTime` int,
  `ArrDelay` int,
  `DepDelay` int,
  `Origin` string,
  `Dest` string,
  `Distance` int,
  `TaxiIn` int,
  `TaxiOut` int,
  `Cancelled` int,
  `CancellationCode` string,
  `Diverted` string,
  `CarrierDelay` int,
  `WeatherDelay` int,
  `NASDelay` int,
  `SecurityDelay` int,
  `LateAircraftDelay` int)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
ESCAPED BY '\\'
STORED AS TEXTFILE
LOCATION '/user/hue/airline/airline';

The above statement starts by outlining the structure of the table, which is mostly integers with a few string columns. The next four lines of code specifies what type of data we have, which are delimited files where the fields are terminated (separated) by commas and where the delimiter is escaped using a backslash. Finally, we type the location of our files, which is the location of the directory where we uploaded the .zip file in part 3 of this tutorial. Note that we specify an “external table”, which means that if we drop the ‘airline’ table, we will still retain our raw data. Had we not specified the external keyword, Hive would’ve moved our raw data files into Hive, and had we decided to drop the ‘airline’ table, all our data would be deleted. Specifying external also lets us build multiple tables on the same underlying dataset if we so choose.

Creating a View Using Hive

One thing that’s slightly awkward above Hive is that you can’t specify that there is a header row in your files. As such, once the above code loads, we have 22 rows in our ‘airline’ table where the data is invalid. Another thing that’s awkward about Hive is that there is no row-level operations, so you can’t delete data! However, we can very easily fix our problem using a view:

1
2
3
4
-- Create view to "remove" 22 bad records from our table
create view vw_airline as
select * from airline
where uniquecarrier <> "UniqueCarrier";

Now that we have our view defined, we no longer have to explicitly exclude the rows in every future query we run. Just like in SQL, views are “free” from a performance standpoint, as they don’t require any additional data storage space (they just represent stored code references).

Time for Analysis?!

If you’ve made it this far, you’ve waited a long time to do some actual analysis! The next and final part of this tutorial will do some interesting things using Hive and/or Pig to analyze the data. The origin of this dataset was a data mining contest to predict why a flight would arrive late to its destination and we’ll do examples towards that end.

Anomaly Detection Using The Adobe Analytics API

Thu, 15 Aug 2013 10:57:56 +0000

As digital marketers & analysts, we’re often asked to quantify when a metric goes beyond just random variation and becomes an actual “unexpected” result. In cases such as A/B..N testing, it’s easy to calculate a t-test to quantify the difference between two testing populations, but for time-series metrics, using a t-test is likely not appropriate.

To determine whether a time-series has become “out-of-control”, we can use Exponential Smoothing to forecast the Expected Value, as well as calculate Upper Control Limits (UCL) and Lower Control Limits (LCL). To the extent a data point exceeds the UCL or falls below the LCL, we can say that statistically a time-series is no longer within the expected range. There are numerous ways to create time-series models using R, but for the purposes of this blog post I’m going to focus on Exponential Smoothing, which is how the anomaly detection feature is implemented within the Adobe Analytics API.

Holt-Winters & Exponential Smoothing

There are three techniques that the Adobe Analytics API uses to build time-series models:

Holt-Winters Additive (Triple Exponential Smoothing)
Holt-Winters Multiplicative (Triple Exponential Smoothing)
Holt Trend Corrected (Double Exponential Smoothing)

The formulas behind each of the techniques are easily found elsewhere, but the main point behind the three techniques is that time-series data can have a long-term trend (Double Exponential Smoothing) and/or a seasonal trend (Triple Exponential Smoothing). To the extent that a time-series has a seasonal component, the seasonal component can be additive (a fixed amount of increase across the series, such as the number of degrees increase in temperature in Summer) or multiplicative (a multiplier relative to the level of the series, such as a 10% increase in sales during holiday periods).

The Adobe Analytics API simplifies the choice of which technique to use by calculating a forecast using all three methods, then choosing the method that has the best fit as calculated by the model having the minimum (squared) error. It’s important to note that while this is probably an okay model selection method for detecting anomalies, this method does not guarantee that the model chosen is the actual “best” forecast model to fit the data.

RSiteCatalyst API call

Using the RSiteCatalyst R package version 1.1, it’s trivial to access the anomaly detection feature:

1
2
3
4
5
6
7
8
9
10
11
12
#Run until version > 1.0 on CRAN
library(devtools)
install_github("RSiteCatalyst", "randyzwitch", ref = "master")

#Run if version >= 1.1 on CRAN
library("RSiteCatalyst")

#API Authentication
SCAuth(<username:company>, <shared_secret>)

#API function call
pageviews_w_forecast <- QueueOvertime(reportSuiteID=<report suite>, dateFrom = "2013-06-01", dateTo="2013-08-13", metrics = "pageviews", dateGranularity="day", anomalyDetection="1")

Once the function call is run, you will receive a DataFrame of ‘Day’ granularity with the actual metric and three additional columns for the forecasted value, UCL and LCL. Graphing these data using ggplot2 (Graph Code Here - GitHub Gist), we can now see on which days an anomalous result occurred:

The red dots in the graph above indicate days where page views either exceeded the UCL or fell below the LCL. On July 23 - 24 timeframe, traffic to this blog spiked dramatically due to a blog post about the Julia programming language, and continued to stay above the norm for about a week afterwards.

Anomaly Detection Limitations

There two limitations to keep in mind when using the Anomaly Detection feature of the Adobe Analytics API:

Anomaly Detection is currently only available for ‘Day’ granularity
Forecasts are built on 35 days of past history

In neither case do I view these limitations as dealbreakers. The first limitation is just an engineering decision, which I’m sure could be expanded if enough people used this functionality.

For the time period of 35 days to build the forecasts, this is an area where there is a balance between calculation time vs. capturing a long-term and/or seasonal trend in the data. Using 35 days as your time period, you get five weeks of day-of-week seasonality, as well as 35 points to calculate a ‘long-term’ trend. If the time period is of concern in terms of what constitutes a ‘good forecast’, then there are plenty of other techniques that can be explored using R (or any other statistical software for that matter).

Elevating the discussion

I have to give a hearty ‘Well Done!’ to the Adobe Analytics folks for elevating the discussion in terms of digital analytics. By using statistical techniques like Exponential Smoothing, analysts can move away from qualitative statements like “Does it look like something is wrong with our data?” to actually quantifying when KPIs are “too far” away from the norm and should be explored further.

Tabular Data I/O in Julia

Tue, 06 Aug 2013 10:05:38 +0000

Importing tabular data into Julia can be done in (at least) three ways: reading a delimited file into an array, reading a delimited file into a DataFrame and accessing databases using ODBC.

Reading a file into an array using readdlm

The most basic way to read data into Julia is through the use of the readdlm function, which will create an array:

1
readdlm(source, delim::Char, T::Type; options...)

If you are reading in a fairly normal delimited file, you can get away with just using the first two arguments, source and delim:

1
2
3
4
5
6
7
julia> airline_array = readdlm("/Users/randyzwitch/airline/1987.csv", ',');

julia> size(airline_array)
(1311827,29)

julia> typeof(airline_array)
Array{Any,2}

It’s important to note that by only specifying the first two arguments, you leave it up to Julia to determine the type of array to return. In the code example above, an array of type Any is returned, as the .csv file I read in was not of homogenous type such as Int64 or ~~ASCII~~String. If you know for certain which type of array you want, you specify the data type using the type argument:

1
2
3
4
5
6
7
julia> airline_array = readdlm("/Users/randyzwitch/airline/1987.csv", ',' , String);

julia> size(airline_array)
(1311827,29)

julia> typeof(airline_array)
Array{String,2}

It’s probably the case that unless you are looking to do linear algebra or other specific mathy type work, you’ll likely find that reading your data into a DataFrame will be more comfortable to work with (especially if you are coming from an R, Python/pandas or even spreadsheet tradition).

To write an array out to a file, you can use the writedlm function (defaults to comma-separated):

1
writedlm(filename, array, delim::Char)

Reading a file into a DataFrame using readtable

As I covered in my prior blog post about Julia, you can also read in delimited files into Julia using the DataFrames package, which returns a DataFrame instead of an array. Besides just being able to read in delimited files, the DataFrames package also supports reading in gzippped files on the fly:

1
2
3
4
5
6
7
8
9
julia> using DataFrames

julia> airline_df = readtable("/Users/randyzwitch/airline/1987.csv.gz");

julia> size(airline_df)
(1311826,29)

julia> typeof(airline_df)
DataFrame  (use methods(DataFrame) to see constructors)

From what I understand, in the future you will be able to read files directly from Amazon S3 into a DataFrame (this is already supported in the AWS package), but for now, the DataFrames package works only on local files. Writing a DataFrame to file can be done with the writetable function:

1
writetable(filename::String, df::DataFrame)

By default, the writetable function will use the delimiter specified by the filename extension and default to printing the column names as a header.

Accessing Databases using ODBC

The third major way of importing tabular data into Julia is through the use of ODBC access to various databases such as MySQL and PostgreSQL.

Using a DSN

The Julia ODBC package provides functionality to connect to a database using a Data Source Name (DSN). Assuming you store all the credentials in your DSN (server name, username, password, etc.), connecting to a database is as easy as:

1
2
3
4
julia> using ODBC

julia> ODBC.connect("MySQL")
Connection 1 to MySQL successful.

Of course, if you don’t want to store your password in your DSN (especially in the case where there are multiple users for a computer), you can pass the usr and pwd arguments to the ODBC.connect function:

1
ODBC.connect(dsn; usr="", pwd="")

Using a connection string

Alternatively, you can build your own connection strings within a Julia session using the advancedconnect function:

1
2
3
4
5
6
7
#Amazon Redshift/Postgres connection string
Julia> red = advancedconnect("Driver={psqlODBC};ServerName=reporting.XXXXX.us-east-1.redshift.amazonaws.com;Username=XXXX;Password=XXXX;Database=XXXX;Port=XXXX");
Connection 1 to Driver={psqlODBC};ServerName=reporting.XXXXX.us-east-1.redshift.amazonaws.com;Username=XXXX;Password=XXXX;Database=XXXX;Port=XXXX successful.

#MySQL connection string
julia> my = advancedconnect("Driver={MySQL};user=root;server=localhost;database=airline;")
Connection 1 to Driver={MySQL};user=root;server=localhost;database=airline; successful.

Regardless of which way you connect, you can query data using the query function. If you want your output as a DataFrame, you can assign the result of the function to an object. If you want to save the results to a file, you specify the file argument:

1
2
3
4
5
6
7
8
9
10
11
12
13
julia> using ODBC

julia> ODBC.connect("MySQL")
Connection 1 to MySQL successful.

#Save query results into a DataFrame called 'results'
julia> results = query("Select * from a1987;");

julia> typeof(results)
DataFrame  (use methods(DataFrame) to see constructors)

#Save query results to a file, tab-delimited (default)
julia> query("Select * from a1987;", file="output.tab", delim = '\t');

Summary

Overall, importing data into Julia is no easier/more difficult than any other language. The biggest thing I’ve noticed thus far is that Julia is a bit less efficient than Python/pandas or R in terms of the amount of RAM needed to store data. In my experience, this is really only an issue once you are working with 1GB+ files (of course, depending on the resources available to you on your machine).

Edit 3/25/2016: A much more up-to-date method of reading CSV data into Julia can be found at this blog post by Clinton Brownley.

Hadoop Streaming with Amazon Elastic MapReduce, Python and mrjob

Wed, 31 Jul 2013 12:34:58 +0000

In a previous rant about data science & innovation, I made reference to a problem I’m having at work where I wanted to classify roughly a quarter-billion URLs by predicted website content (without having to actually visit the website). A few colleagues have asked how you go about even starting to solve a problem like that, and the answer is massively parallel processing.

Attacking the problem using a local machine

In order to classify the URLs, the first thing that’s needed is a customized dictionary of words relative to our company’s subject matter. When you have a corpus of words that are already defined (such as a digitized book), finding the population of words is relatively simple: split the text based on spaces & punctuation and you’re more or less done. However, with a URL, you have one continuous string with no word boundaries. One way to try and find the boundaries would be the following in Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import collections
import nltk

#Dictionary from Unix
internal_dict = open("/usr/share/dict/words")
#Stopwords corpus from NLTK
stopwords = nltk.corpus.stopwords.words('english')

#Build english_dictionary of prospect words
english_dictionary = []
for line in internal_dict:
    if line not in stopwords and len(line) > 4:  #make sure only "big", useful words included
        english_dictionary.append(line.rstrip('\n'))

#How many words are in the complete dictionary?        
len(english_dictionary)

#Import urls
urls = [line for line in open("/path/to/urls/file.csv")]

#Build counter dictionary
wordcount = collections.Counter()
for word in english_dictionary:    #Loop over all possible English words
  for url in urls:     #Loop over all urls in list
    if word in url:
      wordcount[word] += 1 #Once word found, add to dictionary counter

The problem with approaching the word searching problem in this manner is you are limited to the power of your local machine. In my case with a relatively new MacBook Pro, I can process 1,000 lines in 19 seconds as a single-threaded process. At 250,000,000 URLs, that’s 4.75 million seconds…197,916 minutes…3,298 hours…137 days…4.58 months! Of course, 4.58 months is for the data I have now, which is accumulating every second of every day. Clearly, to find just the custom dictionary of words, I’ll need to employ MANY more computers/tasks.

Amazon ElasticMapreduce = Lots of Horsepower

One thing you might notice about the Python code above is that the two loops have no real reason to be run serially; each comparison of URL and dictionary word can be run independently of each other (often referred to as “embarrassingly parallel”). This type of programming pattern is one that is well suited to running on a Hadoop cluster. With Amazon ElasticMapReduce (EMR), we can provision tens, hundreds, even thousands of computer instances to process this URL-dictionary word comparison, and thus getting our answer much faster. The one downside of using Amazon EMR to access Hadoop is that EMR expects to get a Java ``.jar` file containing your MapReduce code. Luckily, there is a Python package called MRJob that does the Python-to-Java translation automatically, so that users don’t have to switch languages to get massively parallel processing.

Writing MapReduce code

The Python code above, keeping a tally of words & number of occurrences IS a version of the MapReduce coding paradigm. Going through the looping process to do the comparison is the “Map” portion of the code and the sum of the word values is the “Reduce” step. However, in order to use EMR, we need to modify the above code to remove the outer URL loop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from mrjob.job import MRJob

class MRWordCounter(MRJob):    

  def mapper(self, english_dict, line):
  english_dict = ['aal', 'aalii', 'aam', 'aani'...'zythum', 'zyzomys', 'zyzzogeton']

  for word in english_dict:
            if word in line:
                yield word, 1

  def reducer(self, word, occurrences):
        yield word, sum(occurrences)

if __name__ == '__main__':
    MRWordCounter.run()

The reason why we remove the outer loop that loops over the lines of the URL file is because that is implicit to the EMR/Hadoop style of processing. We will specify a file that we want to process in our Python script, then EMR will distribute the URLs file across all the Hadoop nodes. Essentially, our 250,000,000 million lines of URLs will become 1,000 tasks of length 250,000 urls (assuming 125 nodes of 8 tasks each).

Calling EMR from the Python command line

Once we have our Python MRJob code written, we can submit our code to EMR from the command line. Here’s what an example code looks like:

1
python ~/Desktop/mapreduce.py -r emr s3://<s3bucket>/url_unload/0000_part_01 --output-dir=s3://<s3bucket>/url_output --num-ec2-instances=81

There are many more options that are possible for the MRJob package, so I highly suggest that you read the documentation for EMR options. One thing to also note is that MRJob uses a configuration file to host various options for EMR called “runners”. Yelp (the maker of the MRJob package) has posted an example of the mrjob.conf file with the most common options to use. In this file, you can specify your Amazon API keys, the type of instances you want to use (I use c1.xlarge spot instances for the most part), where your SSH keys are located and so on.

Results

In terms of performance, I have 8 files of 5GB’s each of URLs (~17.5 million lines per file) that I’m running through the MRJob code above. The first file was run with 19 c1.xlarge instances, creating on average 133 mappers and 65 reducers and taking 917 minutes (3.14 seconds/1000 lines). The second file was run with 80 c1.xlarge instances, creating 560 mappers and 160 reducers and taking 218 minutes (0.75 seconds/1000 lines). So using four times as many instances leads to one-fourth of the run-time.

For the most part, you can expect linear performance in terms of adding nodes to your EMR cluster. I know at some point, Hadoop will decide that it no longer needs to add any more mappers/reducers, but I haven’t had the desire to find out exactly how many I’d need to add to get to that point! 🙂

A Beginner's Look at Julia

Tue, 23 Jul 2013 12:16:34 +0000

Over the past month or so, I’ve been playing with a new scientific programming language called ‘Julia’, which aims to be a high-level language with performance approaching that of C. With that goal in mind, Julia could be a replacement for the ‘multi-language’ problem of needing to move between R, Python, MATLAB, C, Fortran, Scala, etc. within a single scientific programming project. Here are some observations that might be helpful for others looking to get started with Julia.

Get used to ‘Git’ and ‘make’

While there are pre-built binaries for Julia, due to the rapid pace of development, it’s best to build Julia from source. To be able to keep up with the literally dozen code changes per day, you can clone the Julia GitHub repository to your local machine. If you use one of the GitHub GUI’s, this is as easy as hitting the ‘Sync Branch’ button to receive all of the newest code updates.

To install Julia, you need to compile the code. The instructions for each supported operating system are listed on the Julia GitHub page. For Mac users, use Terminal to navigate to the directory where you cloned Julia, then run the following command, where ‘n’ refers to the number of concurrent processes you want the compiler to use:

1
make -j n

I use 8 concurrent processes on a 2013 MacBook Pro and it works pretty well. Certainly much faster than a single process. Note that the first time you run the make command, the build process will take much longer than successive builds, as Julia downloads all the required libraries needed. After the first build, you can just run the make command with a single process, as the code updates don’t take very long to build.

Package management is also done via GitHub. To add Julia packages to your install, you use the Pkg.add() function, with the package name in double-quotes.

Julia code feels very familiar

Text file import

Although the Julia documentation makes numerous references to MATLAB in terms of code similarity, Julia feels very familiar to me as an R and Python user. Take reading a .csv file into a dataframe and finding the dimensions of the resulting object

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#R: Read in 1987.csv from airline dataset into a dataframe
#No import statement needed to create a dataframe in R
airline1987 <- read.csv("~/airline/1987.csv")
dim(airline1987)
[1] 1311826      29

#Python: use pandas to create a dataframe
import pandas as pd
airline1987 = pd.read_csv("/Users/randyzwitch/airline/1987.csv")
airline1987.shape
Out[7]: (1311826, 29)

#Julia: use DataFrames to create a dataframe
using DataFrames
airline1987 = readtable("/Users/randyzwitch/airline/1987.csv")
size(airline1987)
(1311826,29)

In each language, the basic syntax is to call a ‘read’ function, specify the .csv filename, then the defaults of the function read in a basic file. I also could’ve specified other keyword arguments, but for purposes of this example I kept it simple.

Looping

Looping in Julia is similar to other languages. Python requires proper spacing for each level of a loop, with a colon for each evaluated expression. And although you generally don’t use many loops in R, to do so requires using parenthesis and brackets.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#Python looping to create a term-frequency dictionary

from collections import Counter

term_freq = Counter()
for word in english_dictionary:
  for url in url_list:
    if word in url_list:
      term_freq[word] += 1

#Julia looping to create a term-frequency dictionary

term_freq=Dict{String, Int64}()
for word in english_dictionary
    for url in url_list
        if search(line, word) != (0:-1)
            term_freq[word]=get(term_freq,word,0)+1
        end
    end
end

If you’re coming from a Python background, you can see that there’s not a ton of difference between Python looping into a dictionary vs. Julia. The biggest differences are the use of the end control-flow word and that Julia doesn’t currently have the convenience “Counter” object type. R doesn’t natively have a dictionary type, but you can add a similar concept using the hash package.

Vectorization

While not required to achieve high performance, Julia also provides the functional programming construct of vectorization and list comprehensions. In R, you use the *apply family of functions instead of loops in order to apply a function to multiple elements in a list. In Python, there are the map and reduce functions, but there is also the concept of list comprehensions. In Julia, both of the aforementioned functionalities are possible.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#Cube every number from 1 to 100

#Python map function
cubes = map(lambda(x): x*x*x, range(1,100))

#Python list comprehension
cubes= [x*x*x for x in range(1,100)]

#R sapply function
cubes <- sapply(seq(1,100), function(x) x*x*x)

#Julia map function
cubes = map((x)-> x*x*x, [1:100])

#Julia list comprehension
cubes = [x*x*x for x in [1:100]]

In each case, the syntax is just about the same to apply a function across a list/array of numbers.

A small, but intense community

One thing that’s important to note about Julia at this stage is that it’s very early. If you’re going to be messing around with Julia, there’s going to be a lot of alone-time experimenting and reading the Julia documentation. There are also several other resources including a Julia-Users Google group, Julia for R programmers, individual discussions on GitHub in the ‘Issues’ section of each Julia package, and a few tutorials floating around (here and here).

Beyond just the written examples though, I’ve found that the budding Julia community is very helpful and willing in terms of answering questions. I’ve been bugging the hell out of John Myles White and he hasn’t complained (yet!), and even when code issues are raised through the users group or on GitHub, ultimately everyone has been very respectful and eager to help. So don’t be intimidated by the fact that Julia has a very MIT and Ph.D-ness to it…jump right in and migrate some of your favorite code over from other languages.

While I haven’t moved to using Julia for my everyday workload, I am getting facility to the point where I’m starting to consider using Julia for selected projects. Once the language matures a bit more, ~~JuliaStudio starts to approach RStudio in terms of functionality~~, and I get more familiar with the language in general, I can see Julia taking over for at least one if not all of my scientific programming languages.

Getting Started Using Hadoop, Part 3: Loading Data

Wed, 22 May 2013 11:39:19 +0000

In part 2 of the “Getting Started Using Hadoop” series, I discussed how to build a Hadoop cluster on Amazon EC2 using Cloudera CDH. This post will cover how to get your data into the Hadoop Distributed File System (HDFS) using the publicly available “Airline Dataset”. While there are multiple ways to upload data into HDFS, this post will only cover the easiest method, which is to use the Hue ‘File Browser’ interface.

Loading data into HDFS using Hue

'File Browser' in Hue (Cloudera)

Loading data into Hadoop using Hue is by far the easiest way to get started. Hue provides a GUI that provides a “File Browser” like you normally see in Windows or OSX. The workflow here would be to download each year of Airline data to your local machine, then upload each file using the Upload -> Files menu drop-down.

While downloading files from one site on the Internet, then uploading files to somewhere else on the Internet is somewhat wasteful of time and bandwidth, as a tutorial to get started with Hadoop this isn’t the worst thing in the world. For those of you who are OSX users and comfortable using Bash from the command line, here’s some code so you don’t have to babysit the download process:

1
2
3
4
5
$ for i in {1987..2008}
> do
> curl http://stat-computing.org/dataexpo/2009/$i.csv.bz2 > $i.csv.bz2
> bunzip2 $i.csv.bz2
> done

Because you are going to be uploading a bunch of text files to your Hadoop cluster, I’d recommend zipping the files prior to upload. It doesn’t matter if you use .zip or .gz files with one key distinction: if you use .zip files, you will upload using the “Zip Files” button in the File Browser; if you choose .gz, then you must use the “Files” line in the File Browser. Not only will zipping the files make the upload faster, but it will also make sure you only need to do the process once (as opposed to hitting the upload button on each file). Using the .zip file upload process, you should something like the following…a new folder with all of the files extracted automatically:

.zip file automatically extracted into folder with files (Hortonworks)

Next Steps

With the airline .csv files loaded for each year, we can use Pig or Hive to load the tables into a master dataset & schema. That will be the topic of the next tutorial.

Innovation Will Never Be At The Push Of A Button

Fri, 17 May 2013 10:28:47 +0000

@randyzwitch @benjamingaines @usujason I am envisioning the data science equivalent of an autonomous vehicle pileup.

— Todd Belcher (@toddmetrics) May 16, 2013

Recently, I’ve been getting my blood pressure up reading (marketing) articles about “big data” and “data science”. What saddens me about the whole discussion is that there is the underlying premise that what is stopping companies from “harnessing the power of big data” is just the lack of an easy-to-use, push-button tool. Respectfully, if you believe this, you should bow out of the conversation altogether.

Math is hard and stuff.

The first article that really bothered me is titled “Do Predictive Modelers Need to Know Math?” This is a provocative title from a veteran in the data mining/data science industry, and his conclusion is basically ‘Yes, but not everyone on the team needs to be able to hand-solve equations.’ I think that’s a fair point within the context of needing to understand the mathematical concepts behind algorithms, but not needing to be bogged down by notation.

Extending that idea a little further, how far away from the math should a business be comfortable with an employee pushing the button on a machine learning algorithm? Should the CEO be building predictive models? The Intern? A Call Center Rep? For me, I think the answer falls back on the allegory of the highly-specialized tradesperson:

Driver: “How can you charge $100 for five minutes work? All you did was put a bolt on and turned the wrench a few times!

Mechanic: “I didn’t charge you for the parts, I charged you for knowing where to put the wrench…”

The value a data scientist brings to a business is not that he can push the buttons in a GUI like rattle for R, Weka, KnowledgeSeeker, or SAS Enterprise Miner. What your data scientist brings to the table is knowing the underlying assumptions that go into a model, how the algorithm works, which algorithm is appropriate for the business problem being solved and when to know the model/algorithm has failed.

Of all of the things listed, the experience of knowing when the model can/has failed is what you’re paying the money for. That knowledge doesn’t come from just pushing the GUI buttons a bunch of times. And if you’re making million-dollar decisions based on an algorithm, it’s worth paying the salary for a person who really understands the model.

Hire a mathematician, get a programmer free

The next article that bothered me is a “Business Analytics: Do we need data scientists?” debate over “Do we need data scientists at all?” The No argument boils down to an idea that only things that can be made easy and are sufficiently developed are useful/valuable. Thus, because a general analyst can’t use Excel, but rather might need to write a SQL query or write a program to put together a dataset, the problem-domain is too difficult. The No debater also refers to data scientists as being “adversarial”, “pretentious”, project “snobbery”, etc.

But here’s the thing…the problem-domain isn’t particularly difficult if you hire someone with above-average math proficiency. Any decent graduate program in mathematics, statistics, computer science, economics, finance, psychology and others will be using data through programming. Now, the languages may vary between Java, R, Python, Matlab, C++, SAS, Octave, Eviews or others, but the language doesn’t matter, they’ll learn whatever language your company is using once you hire them. They also will learn the systems you are using to store your data, whether it’s a standard relational database, a NoSQL database, or a parallel processing platform like Hadoop.

How can I be certain that the math person you hire will be able to learn all that’s necessary for data science? Because the type of person who likes math & programming is probably a ‘system builder’ type of person. The type of person who played with Legos growing up. The type of person built their own desktop computer back in the day. The type of person who thinks How It’s Made is much more interesting TV than mindless reality shows. The type of person who WANTS to know how a database is storing data, what new open-source technology is out there, wants to find out how many nodes they can connect together before their program won’t finish any faster.

As far as the adversarial/pretentious/snobby comment, all I can say is I’ve never witnessed that. Everyone I know in the data science community are the nicest people, willing to share code, collaborate on ideas and talk until they lose their voice about how to solve an interesting problem.

Data Science is about innovative research, not reporting

I’ve read four academic papers this week. I’m not in graduate school.

As some of you might know, I started a new position at a startup which provides real-time intelligence for the lead generation industry. As such, I’ve got access to billions of records of unstructured data and equally as much structured data. And as a startup, there are several warts that need to be fixed with respect to data storage. So for any given day, I might go from accessing a MySQL database, Amazon Redshift (columnar RDBMS), Amazon DynamoDB (NoSQL) and plain ol’ .csv files via Excel or massive .csv files on Amazon S3. To access this data, I’ve used a combination of R, Python, SQL Workbench, and MySQL Workbench using OSX, Ubuntu desktop and a ‘headless’ Ubuntu image on Amazon EC2.

Why am I giving you about all this jibber-jabber about research papers and tools? Because the idea of building a one-size-fits-all tool to solve the problem I’m working on just doesn’t make sense. And for that matter, I’m not even sure the problem I’m working on is worth solving. But that’s the thing…I don’t KNOW it’s not worth solving, so I need to find out. I’ve got a quarter-billion URLs that I think I can extract information from, just to give our clients ONE more data element to use to optimize their marketing strategies. There may be an already existing algorithm I can use, or maybe I’ll try this research paper on “word breaking” I found from Microsoft Research. Once I find out the answer, if it’s valuable, then I need to be able to implement my algorithm into our real-time API, because it’s likely whatever language I end up using isn’t going to be what our API is written in.

So if these aren’t the type of problems you’re working on, then maybe there is an all-in-one tool out there for you to use (and that’s okay). But these are the types of edge-case problems that I think about when I think about “data science”, and as such, it will always be custom and ad-hoc. There are many awesome open-source tools I will use to help me along the way, but it will never make sense to build an easy-to-use tool for a problem a few dozen companies may ever need to know the answer to.

Use the data you have to do something extraordinary

If you don't 'get' something, own it. Don't dump your dumb garbage into the world.

— marc maron (@marcmaron) May 16, 2013

I’m already 1100 words into this rant, so I’ll finish up with a few admissions. Yes, “data science” is somewhat a ridiculous name for the combination of advanced analytics and data engineering that it represents. And yes, there are plenty of vendors out there pedaling hype about the grandeur of ‘Big Data’ and why every business MUST jump on board or be left behind.

But rather than focusing on why something is “useless” or “stupid” or “hype”, just ask yourself “Can I solve the business problems I have today using the tools I currently have access to?” If the answer is yes, then great, get to work. If not, maybe you can find someone to help you get where you’re going (and that person may or may not call themselves a “Data Scientist”). Either way, let’s all move forward and do something extraordinary. It’s the least we can do for our customers.

Getting Started Using Hadoop, Part 2: Building a Cluster

Thu, 25 Apr 2013 17:33:48 +0000

In Part 1 of this series, I discussed some of the basic concepts around Hadoop, specifically when it’s appropriate to use Hadoop to solve your data engineering problems and the terminology of the Hadoop eco-system. This post will cover how to install your own Hadoop cluster on Amazon EC2 using Cloudera Manager.

Like prior posts talking about Amazon EC2, this post assumes you have some basic facility with Linux, submitting instructions via the command line, etc. Because really, if you’re interested in Hadoop, using the command line probably isn’t a limiting factor!

Building a 18-node Hadoop Cluster

The SlideShare presentation below shows the steps to building a 18-node Hadoop cluster, using a single m1.large EC2 instance as the ‘Name Node’ and 18 m1.medium EC2 instances as the ‘Data Nodes’. I chose 18 nodes because according to Cloudera, 20 is the maximum that can be activated at one time through the Amazon API, so let’s stay under the max to avoid any errors. It’s possible to add more instances later through the Cloudera Manager (up to 50 total), if so desired.

Note that going through this tutorial will cost $2.40/hr at current prices ($0.24/hr per m1.large instance and $0.12/hr per m1.medium instance).

Since the SlideShare presentation is potentially not so friendly on the eyes, I’ve also created a PDF download that’s full resolution.

Next Steps

Once you make it through all these steps to set up a Hadoop cluster, you are ready to do some analysis. Part 3 of this tutorial will cover how to upload data into HDFS using Hue.

Update, 7/13/13: As is the case with any open-source project, there have been several changes to the Cloudera Manager that makes setup easier. When getting started, on the screen where it asks “Which Cloudera do you want to deploy?”, choose ‘Cloudera Standard’. Also, once you get to slides 13-14 where you click on the link to get started with Hue, the link now works correctly (you don’t need to search for the Amazon DNS any more!)

Getting Started Using Hadoop, Part 1: Intro

Thu, 18 Apr 2013 12:47:15 +0000

For the last couple of days I’ve been at the eMetrics conference in San Francisco. There were several panels that discussed big data, both from an engineering standpoint as well as how to adopt newer technologies from a business perspective.

Unfortunately, there wasn’t a whole lot of practical information on how to actually get started using ‘big data’ technologies, of which Hadoop is one. Luckily, it’s fairly easy to create a proof-of-concept Hadoop cluster using Amazon EC2 and Cloudera.

This series will be at least 5 parts, as follows:

Intro to Hadoop ecosystem and concepts
Setting up Hadoop cluster on Amazon EC2 using Cloudera
Populating HDFS with airline dataset files using Hue
Use Hive and/or Pig to stack datasets into one master dataset
Doing analytics on the combined Airline dataset using Pig and/or Hive

My aim with this series is to simply explain why you might want to consider using Hadoop for your data storage and processing. There’s a lot of marketing & vendor ~~bullshit~~ excitement surrounding the term ‘big data’, so for this blog series, I’m just going to focus on the most important points for an analyst/marketer to understand. And other than this sentence, there will be no mentions of MS Excel in terms of ‘big data’, which is barely an appropriate tool for analysis in general, let alone analysis at scale.

What Is Hadoop & Why Are People Talking About It?

At it’s simplest, Hadoop provides a parallel-processing computing framework for data storage and processing. The reason why a parallel-processing framework is important for enterprise-level analysis is due to physical limitations on how quickly a single machine can process information.

As an example, suppose you want to create a report that looks at 1 trillion daily credit card transactions. It’s possible to do your calculations on your local desktop using a tool like SAS. However, the amount of time to process that much data on a desktop with 8GB-16GB of RAM might be 8 hours, 10 hours….24 hours?! So an analyst trying to get an answer can start a SINGLE business question at 8am and hope they get their answer before it’s time to leave at the end of the day. Suffice to say, not a particularly efficient way to run a business.

The solution might seem to add more processors and RAM to a desktop, but what happens when you add more users asking questions? Now you need an enterprise-class server such as Oracle or Teradata (and a few million dollars!). And for every terabyte of data you want to store, you’ll need a few thousand dollars. And that’s just for your nicely structured data…what happens when you want to start storing data such as free-form text that’s not so cleanly structured? Eventually, these types of engineering questions lead you towards a solution like Hadoop.

The reason why there is so much discussion around Hadoop as a data platform is that it solves the problems stated above: excessive time to process vast amounts of data and excessive cost of data storage. By using “commodity hardware” along with some fancy engineering, Hadoop provides an extremely cost-effective and flexible way to handle your enterprise data.

If Hadoop is so Great, Why Doesn’t Everyone Use It?

“Fast, Cheap And Good. Everyone should use Hadoop!” - Every vendor in marketplace

Just like you (probably) don’t use a screwdriver to stir a pot of chicken soup, not every data storage and analysis problem requires the extreme flexibility that Hadoop can provide. From the example above with credit card transactions, a standard relational database might continue to be an acceptable solution if you’re just running a basic SQL query to sum across the rows. But once your data starts moving beyond “rows and columns” and into things such as free-form text, images, clickstream data…the more Hadoop makes sense.

While it’s a tautology, how you know you need a solution like Hadoop is when you suspect you need a solution like Hadoop! If you already have a highly functioning data mart that answers your business questions, you probably don’t need to re-engineer everything just because. If you’re an Internet startup trying to create the next Facebook, then a standard relational database probably won’t cut it.

The best example I heard at eMetrics about the need for Hadoop was from Bob Page (now at Hortonworks, a Hadoop vendor): when Bob was at Ebay, for the longest time they were throwing away data, specifically images from the listings. So prior high storage costs leading to undesirable business outcome (deletion), unstructured data in the form of images…a Hadoop framework made sense to implement. Once implemented, Ebay could look across years of auctions to answer their business questions.

I’m An Analyst, Not An Engineer…What’s The Minimum I Need To Know To Get Started?

“MapReduce, Pigs, HCatalogs, Elephants, Bees, Zoos…Ooozie (Uzi’s)? WTF is everyone talking about?”

If you’ve made it this far and you’re not an engineer or DBA, you’re probably someone who’s interested in data science. You may be someone who already uses R, Python, Ruby or Java. Or, you’re a masochist. In any case, here are the minimum concepts I think you need to know to get started for later blog posts:

MapReduce: Not explicitly a Hadoop idea, but the idea that data can be split into chunks by a key (“Map”) and then processed into information by one or more functions/transformations (“Reduce”). In the Hadoop sense, MapReduce is generally a reference to a “job” written in Java that performs a data transformation
HDFS: Hadoop Distributed File System. Raw data gets imported into HDFS (either structured or unstructured), the distributed around to all of the various nodes to allow for parallel processing
Hive: SQL-like interface so that analysts don’t have to write MapReduce code directly
Pig: A scripting language used for analysis. Generally, an analyst will use Hive and/or Pig to do their work
HCatalog: A ‘Data Warehouse’ layer on top of HDFS, similar to how you define a database table (a series of columns in a table with formats)

Next Steps

With the above five Hadoop concepts in place, the next few posts will be to set up a proof-of-concept Hadoop cluster on Amazon EC2, processing ~12GB of publicly available data from the ‘Airline dataset’. That’s not ‘big’ as ‘big data’ goes, but it’s big enough to be fun to work with.

Instructions for Installing & Using R on Amazon EC2

Mon, 08 Apr 2013 16:36:07 +0000

If you’re an R user, you’ve surely heard all the hype around ‘big data’ and how R is commonly used to analyze these volumes of data. One thing that’s often missing from the discussion is HOW to work around issues using big data and R, specifically how to deal with the fact that R stores all its objects in-memory.

While you can use packages such as ff and bigmemory to overcome the in-memory limits of your local machine, these additional packages do require some re-engineering of your code. Instead, consider using Amazon EC2 to provision the resources you need. Here are two ways to get started…

Use a Pre-Made AMI

In the great open-source tradition, there are already R Amazon EC2 AMI images available out there to use. The way I got started was using the pre-built images that Louis Aslett provides on his website. Louis also provides great instructions on learning about EC2, so if you’ve never worked with R in the cloud or a just looking to get up and running fast, his website is a great means to do so.

Build Your Own Image

Alternatively, suppose you want to build your own customized image. For example, say you wanted to build a proof-of-concept ‘big data’ environment, so you want R, Python, MySQL and MongoDB. The commands to accomplish this are listed below. Note that I’m assuming you have a basic understanding of working through the Amazon Web Service Console (AWS), including being able to get to the ‘Classic Wizard’ for launching an EC2 instance. You also should have a basic understanding of working from the command line.

Setting Up Amazon EC2 Instance

Launch an Ubuntu 12.04.1 LTS 64-bit image. You can use a free “t1.micro” image while building, then provision more resources later once you’re ready for analysis.
Accept defaults until you get to Key-Pair tab. The Key-Pair is what allows you to login securely to your Amazon EC2 image without a password. Create and download a Key-Pair if you don’t already have one or choose an existing Key-Pair if you do.
When you get to the ‘Security Groups’ tab, create a security group that has the following ports open: 22 (SSH), 80 (HTTP), 443 (HTTPS), 3389 (RDP, optional), and 8787 (RStudio Server).
Work through the rest of the Wizard until your instance is launched.

Connecting to Amazon EC2 Instance

There are two ways to connect to your EC2 image, both of which can be found by going to the “Actions” tab in the AWS console, then selecting “Connect” from the drop-down. The rest of this tutorial assumes you connect via a stand-alone SSH client (such as Terminal for Mac OSX)
Connect to your instance by typing the code provided to you, such as: ssh -i me-aws.pem ubuntu@ec2-50-19-18-120.compute-1.amazonaws.com
Be sure that before you submit this code, you either modify the line to put the directory in front of your Key-Pair, or “cd” to the directory where the Key-Pair is located
After submitting the connect code, you will get a warning saying that the ‘authenticity can’t be established, do you want to continue?’ Type yes and hit enter to log in.

Installing Base R

Once you are logged in, there are about a dozen commands that need to be submitted. Some commands run quickly, others can take 10-15 minutes to run through the entire installation process. Depending on how quickly each command completes, you may or may not need to type “sudo” in front of each command to have proper access rights for installation. Submit each line one at a time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#Create a user, home directory and set password
sudo useradd rstudio
sudo mkdir /home/rstudio
sudo passwd rstudio
sudo chmod -R 0777 /home/rstudio

#Update all files from the default state
sudo apt-get update
sudo apt-get upgrade

#Add CRAN mirror to custom sources.list file using vi
sudo vi /etc/apt/sources.list.d/sources.list

#Add following line (or your favorite CRAN mirror)
deb http://lib.stat.cmu.edu/R/CRAN/bin/linux/ubuntu precise/

#Update files to use CRAN mirror
#Don't worry about error message
sudo apt-get update

#Install latest version of R
#Install without verification
sudo apt-get install r-base

While not strictly required to run R, I also like to run the following commands to install the Curl and XML packages as well, which are useful if you want to use R to connect to any web data/APIs.

1
2
3
#Install in order to use RCurl & XML
sudo aptitude install libcurl4-openssl-dev
sudo apt-get install libxml2-dev

With these commands run, you will now be able to run R from the command line just by typing “R” at the prompt. However, it would be a crime to do all this work and not install RStudio Server, which makes working in R so much easier.

Installing RStudio Server

Once you’ve installed the above commands, you can now access RStudio through your local browser. Navigate to the Public DNS of your image on port 8787, similar to:

http://ec2-50-19-18-120.compute-1.amazonaws.com:8787

The login and password will be the values you used in the image creation process (I used rstudio as my username above).

Installing MySQL, Python, and MongoDB

If you’ve made it this far, I’m sure you realize that installing additional packages will only take a line or two of code. Even better, Python is installed by default on Linux, so we really only need to install MySQL and MongoDB.

1
2
3
4
5
6
#Install MySQL

sudo apt-get install mysql-common
sudo apt-get install mysql-server

#Install MongoDB

sudo apt-get install mongodb

Summary

While the steps above can be intimidating if you’ve never used Linux or worked on the command line, but once you get the hang of it, your ability to use R on ‘big data’ (however you define it) will be much improved. For only a few pennies to up to a few dollars per hour, you can use hardware having 16-64GB of RAM or more.

EDIT, 4/9: The code is wrapping weird on some monitors. Click here for the commands in a .txt. file.

Automated Re-Install of Packages for R 3.0

Wed, 03 Apr 2013 10:10:09 +0000

With the big release of R 3.0 today comes an unfortunate side effect of needing to re-install all of your packages. Luckily, R provides a pretty easy method of getting all of your packages into a list for automated re-install. Here’s how to do it for OSX users with a default install to the Library:

For Windows users, the same general process should work, assuming you change the file reference in the installed.packages function to the proper Windows location. The one downside to this method is that only packages that are listed on CRAN will be reinstalled, so if you installed anything using devtools, you’ll need to re-install those packages again. But at the very least, the code snippet above is a quick way to re-install most of your packages. EDIT, 4/4/13: Per Noam below, you can also use a more direct method: update.packages(ask=FALSE, checkBuilt = TRUE)

The Fun of Error Trapping: R Package Edition

Mon, 25 Feb 2013 17:15:46 +0000

For the last month or so I’ve been working on an R package to make accessing the Adobe (Omniture) Digital Marketing Suite Reporting API easier. As part of this development effort, I’m at the point where I’m intentionally introducing errors into my function inputs, trying to guess some of the ways users might incorrectly input arguments into each function. Imagine my surprise when I saw this:

> result <- content(json)
Loading required package: XML
Error in parser(content, …) : could not find function “htmlTreeParse”
In addition: Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘XML’

The main idea behind the functions I’ve written is making REST calls to the Omniture API, which done correctly return valid JSON. From there, each JSON string is converted from binary or whatever formatting they come back as using the content function from the httr package. Without specifying any arguments to the content function, the function tries to guess at the proper translation method.

The guessing is all fine and good until you don’t pass a valid JSON string! In this case, the error message is guessing that it might be XML (the returned error is actually HTML), tries to load the XML package…then says it can’t load the XML package. A two-for-one error!

Maybe it’s just me, but I’m finding this hilarious after a long day of programming. Maybe it’s because I’m not longer intimidated by an error like this, and as such, I’ve gotten over the steep learning curve of R.

Note: Hadley, if you read this, I’m not saying your httr package has any sort of bug or anything. Just that I found this particular error amusing.

(not provided): Using R and the Google Analytics API

Fri, 11 Jan 2013 16:27:33 +0000

(not provided) terms from Google average 35%-60% of all Google organic search terms

For power users of Google Analytics, there is a heavy dose of spreadsheet work that accompanies any decent analysis. But even with Excel in tow, it’s often difficult to get the data just right without resorting to formula hacks and manual table formatting. This is where the Google Analytics API and R can come very much in handy.

Connecting to the Google Analytics API using R

I’m not going to say that connecting to the Google Analytics API is easy per se, but with the rga package written by “skardhamar” on GitHub, it’s easier than if you had to develop the connection code yourself! However, before you can get started making calls to the Google Analytics API, you need to register within the Google Analytics API console. There you can define a new project and then you’ll be able to make your API calls via R.

After you have your API access straightened out, the GitHub page for the rga package has all the details in how to authenticate using the rga.open function. I chose to use the where argument so that I can continuously hit the API across many sessions without having to do browser authentication each time.

1
rga.open(instance = "ga", where = "~/Documents/R/ga-api")

Analyzing (not provided) as a Google Analytics organic search term

Once connected to the Google Analytics API, now it’s time to submit our API calls. I used two API calls to create the graph at the top of the post, which shows the percentage of all Google organic search terms that are listed as “(not provided)” for the entire history of this blog. The two API calls were to download the number of total organic search term visits by date from Google and the number of “(not provided)” visits by date, also from Google. Here’s the API call for the “(not provided)” data (replace XXXXXXXX with your profile ID):

1
2
3
4
5
6
7
8
visits_notprovided.df <- ga$getData(XXXXXXXX,
start.date = "2011-01-01",
end.date = "2013-01-10",
metrics = "ga:visits",
filters = "ga:keyword==(not provided);ga:source==google;ga:medium==organic",
dimensions = "ga:date",
max = 1500,
sort = "ga:date")

The result of this API call provides an R data frame containing two columns: date and number of visits where the search term was “(not provided)”.

Munging the data using R

After pulling the data into R, all that’s left is to merge the data frames, do a few calculations, then make the boxplot. Because the default object returned by the rga package is a data frame, it’s trivial to use the merge function in R to join the data frames, then use a few calculated columns to create the percentage of visits that are “(not provided)”

What was that Google, only 10% of searches are supposed to be (not provided)?

By now, it’s beating a dead horse that the percentage of “(not provided)” search results from Google FAR exceeds what they said it would. This blog gets about 5,000 visits a month, and due to the technical nature of the blog many of the users are using Chrome (which does secure search automatically) or from iOS (which also does secure search). But at minimum, this graph illustrates the power of using the Google Analytics API via R; I can update this graph at my leisure by running my script, and I can create a graphic that’s not possible within Excel.

Full code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
#### Connecting to Google Analytics API via R
#### Uses OAuth 2.0
#### https://developers.google.com/analytics/devguides/reporting/core/v3/ for documentation

# Install devtools package & rga - This is only done one time
install.packages("devtools")
library(devtools)
install_github("rga", "skardhamar")


# Load rga package - requires bitops, RCurl, rjson
# Load lubridate to handle dates
library(rga)
library(lubridate)

# Authenticating to GA API. Go to https://code.google.com/apis/console/ and create
# an API application.  Don't need to worry about the client id and shared secret for
# this R code, it is not needed

# If file listed in "where" location doesn't exist, browser window will open.
# Allow access, copy code into R console where prompted
# Once file located in "where" directory created, you will have continous access to
# API without needing to do browser authentication
rga.open(instance = "ga", where = "~/Documents/R/ga-api")


# Get (not provided) Search results.  Replace XXXXXXXX with your profile ID from GA
visits_notprovided.df <- ga$getData(XXXXXXXX,
                                  start.date = "2011-01-01",
                                  end.date = "2013-01-10",
                                  metrics = "ga:visits",
                                  filters = "ga:keyword==(not provided);ga:source==google;ga:medium==organic",
                                  dimensions = "ga:date",
                                  max = 1500,
                                  sort = "ga:date")

names(visits_notprovided.df)<- c("hit_date", "np_visits")

# Get sum of all Google Organic Search results.  Replace XXXXXXXX with your profile ID from GA
visits_orgsearch.df <- ga$getData(XXXXXXXX,
                                    start.date = "2011-01-01",
                                    end.date = "2013-01-10",
                                    metrics = "ga:visits",
                                    filters = "ga:source==google;ga:medium==organic",
                                    dimensions = "ga:date",
                                    max = 1500,
                                    sort = "ga:date")

names(visits_orgsearch.df)<- c("hit_date", "total_visits")

# Merge files, create metrics, limit dataset to just days when tags firing
merged.df <- merge(visits_notprovided.df, visits_orgsearch.df, all=TRUE)
merged.df$search_term_provided <- merged.df$total_visits - merged.df$np_visits
merged.df$pct_np <- merged.df$np_visits / merged.df$total_visits
merged.df$yearmo <- year(merged.df$hit_date)*100 + month(merged.df$hit_date)

final_dataset = subset(merged.df, total_visits > 0)


# Visualization - boxplot by month
# Main plot, minus y axis tick labels
boxplot(pct_np~yearmo,data=final_dataset, main="Google (not provided)\nPercentage of Total Organic Searches",
        xlab="Year-Month", ylab="Percent (not provided)", col= "orange", ylim=c(0,.8), yaxt="n")

#Create tick sequence and format axis labels
ticks <- seq(0, .8, .2)
label_ticks <- sprintf("%1.f%%", 100*ticks)
axis(2, at=ticks, labels=label_ticks)

Video: SQL Queries in R using sqldf

Mon, 17 Dec 2012 13:22:51 +0000

This video covers how to run SQL queries using the ‘sqldf’ package within R. This sqldf tutorial was part of a Keystone Solutions podcast discussion about data science and what skills beginning analysts should be learning to improve their skill set.

The example files from this tutorial can be downloaded from this link:

Example Data files

Video: Overlay Histogram in R (Normal, Density, Another Series)

Fri, 09 Nov 2012 13:40:01 +0000

This video explains how to overlay histogram plots in R for 3 common cases: overlaying a histogram with a normal curve, overlaying a histogram with a density curve, and overlaying a histogram with a second data series plotted on a secondary axis.

Note: Towards the end of the video (maybe minute 14 or so), I make a language error when talking about the padj parameter in the mtext function…the setting doesn’t “left truncated” the label, I meant “right align”, “left align”, etc.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
#Step 0:  load/prepare data

#Read in data
sample_data <- read.csv("~/Desktop/test_data.csv")

# "Explode" counts by age back to unsummarized "raw" data
age.exploded <- rep.int(sample_data$age, sample_data$count)


#1. Histogram with normal distributon overlaid or density curve


#1A.  Create histogram
hist(age.exploded, xlim= c(0,20), ylim= c(0,.2), breaks=seq(min(age.exploded),
  max(age.exploded), length=22), xlab = "Age", ylab= "Percentage of Accounts",
  main = "Age Distribution of Accounts\n (where 0 <= age <= 20)",
  prob= TRUE, col= "lightgray")

#1B.  Do one of the following, either put the normal distribution on the histogram
#     or put the smoothed density function

#Calculate normal distribution having mean/sd equal to data plotted in the
#histogram above
points(seq(min(age.exploded), max(age.exploded), length.out=500),
       dnorm(seq(min(age.exploded), max(age.exploded), length.out=500),
             mean(age.exploded), sd(age.exploded)), type="l", col="red")

#Add smoothed density function to histogram, smoothness toggled using
#"adjust" parameter
lines(density(age.exploded, adjust = 2), col = "blue")

#2 Histogram with line plot overlaid

#2A.  Create histogram with extra border space on right-hand side

#Extra border space "2" on right  (bottom, left, top, right)
par(oma=c(0,0,0,2))

hist(age.exploded, xlim= c(0,20), ylim= c(0,.2),
     breaks=seq(min(age.exploded), max(age.exploded), length=22), xlab = "Age",
     ylab= "Percentage of Accounts", main = "Age Distribution of Accounts vs. Subscription Rate \n (where reported age <= 20)",
     prob= TRUE, col= "lightgray")

#2B.  Add overlaid line plot, create a right-side numeric axis
par(new=T)
plot(sample_data$subscribe_pct, xlab= "", ylab="", type = "b", col = "red", axes=FALSE)  
axis(4)

#2C.  Add right-side axis label

mtext(text="Subscription Rate",side=4, outer=TRUE, padj=1)

File Download:

Histogram overlay in R code and sample data file

Video: R, RStudio, Rcmdr & rattle

Fri, 07 Sep 2012 12:07:45 +0000

I did a screencast for my co-workers to show how to get started with R, specifically what a base installation of R looks like, then showing how to improve your workflow using RStudio, Rcmdr or rattle. The examples are somewhat pedestrian, but it gives a feel for what using R actually looks like.

If you have any questions, comments, or jeers about how bad I am at R, feel free to leave a comment in the comments section!

R Demo - Randy Zwitch from Keystone Solutions on Vimeo.

Getting Started Using R, Part 2: Rcmdr

Mon, 06 Aug 2012 10:54:18 +0000

In my first post in this series, I discussed RStudio, an IDE that adds significant functionality and consistency to a basic installation of R. In this post, I will discuss Rcmdr, a GUI that provides the ability to do basic business statistics without having to code in R.

Rcmdr (“R Commander”)

Example Rcmdr window with the "Statistics" menu expanded

Rcmdr is a package for R that was created by John Fox at McMaster University in Canada as a means of providing the basic statistics functionality for classroom use. In this way, Rcmdr is somewhat similar to SAS Enterprise Guide, a GUI that allows quick access to the power of SAS without the requirement of writing code.

While using Rcmdr won’t allow you to tap into every single advanced feature that R provides, it does provide a lot of great “general” functionality that can be used in everyday business such as summary statistics, t-tests, ANOVA, linear regression modeling, graphing and data re-coding.

Using Rcmdr

For the most part, the Rcmdr dialog boxes all look very similar. Only the most useful options are provided, such as the variable(s) you are looking to interrogate, variable(s) you’d like to break down your analysis by, what statistics you want the output to display (mean, median, mode, etc.) and so on. The dialog boxes vary depending on whether you are estimating a model or plotting a graph, but in my preliminary usage I haven’t found any dialog boxes that were so confusing that I needed to check the “Help” files.

For example, suppose I wanted to make a boxplot of my data, income by job type. To do so, I would go to the “Graphs” menu and select “Boxplot”, which provides me with the following dialog box:

Rcmdr options for creating a Boxplot

Boxplot output created by Rcmdr

Within this dialog box, there are only 3 choices: variable to plot (income), variable to break down the graph by (type), and “Identify outliers with mouse”, which allows for the user to point at the resulting graph to designate outliers to be labeled on the graph. When I click “OK” in the dialog box, the result is the boxplot shown above. We can see that the “bc” (blue-collar) group has a lower mid-point to the income range than “prof” (professors) and “wc” (white-collar).

One of the best features of Rcmdr is that not only do we get the output we requested, but the code window also shows the code that was necessary to create the boxplot. In this example, the underlying R code is relatively simple:

1
boxplot(income~type, ylab="income", xlab="type", data=Duncan)

By providing the underlying code, Rcmdr serves as a teaching tool to move the beginning user towards coding in R directly, or at least, modifying the tool-generated code to include titles or whatever options the user wants to add to the original analysis/output.

Installation of Rcmdr

Sadly, Rcmdr is one of those add-ins that seems to work better on Windows than Mac OSX, at least for the installation portion. I’ve been able to successfully install Rcmdr on my relatively old MacBook Pro, but it did take a bit of time to figure out. Luckily, the instructions to install Rcmdr on a Mac are fairly well laid out in this article.

However, once you get over the hurdle of downloading tcltk and XQuartz (X11 emulator), the program seems to work the same on both platforms.

Getting Started Using R, Part 1: RStudio

Sat, 04 Aug 2012 11:58:14 +0000

~~Despite my preference for SAS over R,~~ there are some add-ons to “basic” R that I’ve found that have made my learning process way easier. While I’m still in my infancy in learning R, I feel like once I found these additional tools, my ability to use R to get work done improved significantly.

In this first post of three, I’ll discuss RStudio, a more friendly access point to the default installation of R. My second post will discuss Rcmdr, a GUI developed for students taking a basic college-level course in Statistics. The third post will cover rattle, a GUI specifically designed for data mining (as opposed to more general statistics like Rcmdr).

RStudio

R Studio is an IDE that dramatically improves the R experience

RStudio is an open-source Integrated Development Environment (IDE) that provides a more consistent user experience to R. There are many great features of RStudio over “basic” R, including:

Consistent windowing between sessions (customizable by the user)
Point-and-click exploration of data frames and other data objects
Importing data files through dialog box functionality
Customizable code syntax highlighting, auto-complete, and Help menu access from the code editor
Ability to see all installed packages, turn on packages using a checkbox, and download libraries (and their dependencies) without having to write any code
Version Control using GitHub

While RStudio doesn’t provide a GUI that will help you run a regression model or build a graph, it provides a more “friendly” environment to work in as compared to the command-line interface of a default installation of R. I find that by having elements like the currently active data objects and available/active packages with links to the Help files “exposed” at all times, RStudio reminds me of where my analysis has been and gives me a quick way to think about “What Else?” to pursue if I hit a roadblock.

Installation of RStudio

RStudio installs like any other program for Windows or Mac OSX. As far as I can tell, there are no advantages to using RStudio in either environment, both the Windows and OSX versions seem to work equally well. The most important consideration is that RStudio is just an “add-on” so-to-speak, it does not include R itself. So be sure to go to one of the Comprehensive R Archive Network (CRAN) sites to download R first.

Learning R Has Really Made Me Appreciate SAS

Wed, 25 Jul 2012 11:34:03 +0000

EDIT, 9/9/2016: Four years later, this blog post is a comical look back in time. It’s hard to believe that I could think this way! Having used R (and Python, Julia), I will never return back to the constraints of using SAS. The inflexible nature of everything having to be a Dataset in SAS vs. the infinite flexibility of data structures in programming-oriented languages makes it no contest.

But I’ll leave this here to remind myself how today’s frustration leads to tomorrow’s breakthroughs.

For the past 18 months, it seems like all I’ve heard about in the digital marketing industry is “big data”, and with that, mentions of using Hadoop and R to solve these sorts of problems. Why are these tools the most often mentioned? Because they are open source, i.e. free of charge!

But as I’ve tried to learn R, I keep asking myself…are all of my colleagues out of their minds? Or, am I just beyond learning something new? As of right now, R is just one big hack on top of a hack to me, and the software is only “free” if you don’t consider lost productivity.

Need new functionality, just download another R package!

One of the biggest “pros” I see thrown around for R relative to a tool like SAS is that when new statistical techniques are invented, someone will code it in R immediately. A company like SAS make take 5 years to implement the feature, or it may not get implemented at all. That’s all fine and good, but the problem I’ve found is that there are 10 ways to do something in R, and I spend more time downloading packages (along with other packages that are dependencies) than I do learning A SINGLE WAY to do something correctly.

For example, take trying to get summary statistics by group. In SAS, you use a Proc Summary statement, with either a BY group statement or a CLASS statement. It’s fairly simple and it works.

proc summary data= hs0; var _numeric_; class prgtype; output out=results mean= /autolabel autoname inherit; run;

In R, I ran the following code, which should be roughly equivalent:

by(hs0, hs0$prgtype, mean)

Very simple, fewer lines…and technically wrong, throwing a 6 unhelpful errors for a single line of code. Because it was decided that “mean” as a function would be deprecated in R. WHY??? It’s so simple, why modify the language like that?

According to the error message, I’m supposed to use colMeans instead…but once you get to how, you’re on your own, the Help documentation is garbage. Some combination of “by” and “colMeans” might work, but I don’t have an example to follow.

Google sent me to the Quick-R website, and I found a “descriptive statistics” article with by group processing…with the recommendation of using the “psych” package or the “doBy” package. But CRAN won’t let me download all of the dependencies, so again, stuck trying to do the simplest thing in statistics.

Let’s be fast and run everything in RAM!

My next favorite hassle in R is that you are expected to continuously monitor how many data elements you have active in a workspace. R runs completely in RAM (as opposed to SAS which runs a combination of RAM for processing and hard disks for storage), so if you want to do something really “big”, you will quickly choke your computer. I tried to work with a single day of Omniture data from the raw data feed, and my MacBook Pro with 6GB of memory was shot. I believe the file was 700,000 rows by 300 columns, but I could be mis-remembering. That’s not even enough data to think about performance-tuning a program in SAS, any slop code will run quickly.

How does one solve these memory errors in R? Port to Amazon cloud seems to be the most commonly given suggestion. But that’s more setup time, getting an R instance over to Amazon, your data over to Amazon..and now you are renting hardware.

R is great for data visualization!

From what I’ve seen from the demo(graphics) tutorial, R does have some pretty impressive visualization capabilities. Contour maps, histograms, boxplots…there seems to be a lot of capability here beyond the realm of a tool like Excel (which, besides not being free, isn’t really for visualization). SAS has some graphics capabilities, but they are a bit hard to master.

But for all of the hassle to get your data formatted properly, downloading endless packages, avoiding memory errors, you could just pay for Tableau and get working. Then, once you have your visualizations done in Tableau, if you are using Tableau server you can share interactive dashboards with others. As far as I know, R graphics are static image exports, so you’re stuck with “flat” presentations.

Maybe, it’s just me

For R diehards, the above verbiage probably just sounds like whining from someone who is too new to appreciate the greatness of R or too stuck in the “old SAS way”. That’s certainly possible. But from my first several weeks of trying to use R, the level of frustration is way beyond anything I experienced when I was learning SAS.

Luckily, I don’t currently have any consulting projects that require R or SAS at the moment, so I can continue to try and learn why everyone thinks R is so great. But from where I sit right now, the licensing fee from SAS doesn’t seem so bad when it allows me to get to doing productive work instead of building my own statistics software piece-by-piece.

My Top 20 Least Useful Omniture Reports

Wed, 18 Jul 2012 00:14:04 +0000

Just because data CAN be captured doesn't mean it SHOULD be!

In a prior post about customizing the SiteCatalyst menu interface, I discussed how simple changes such as hiding empty Omniture variables/reports and re-organizing the menu structure will help improve understanding within your organization. In the spirit of even further interface optimization, here are 20 reports within Omniture that I feel that can be hidden due to their lack of business-actionable information.

Here are my Top 20, in no particular order:

Mobile: Color Depth
Mobile: Information Services
Mobile: Decoration Mail Support
Mobile: PTT
Mobile: Device Number Transmit
Mobile: Browser URL Length
Mobile: DRM
Mobile: Mail URL Length
Mobile: Java version
Mobile: Manufacturer
Technology: Connection Types
Technology: Monitor Color Depth
Technology: JavaScript Version
Technology: Monitor Resolutions
Visitor Profile: Top-Level Domains
Visitor Profile: Domains
Visitor Profile: Geosegmentation
Traffic Sources: All Search Page Ranking
Traffic Sources: Original Referring Domains
Custom Variable: s.server report

Mobile reports

For the most part, the information in the separate reports can determined just by knowing the device (which is also a default Omniture report). So, a single report can take the place of 10.

There’s also the pesky issue that the reports more often than not show “Unknown” for 90%+ of the mobile traffic (at least, in the U.S.). So not only can the data be determined from knowing the mobile device being used, the additional reports aren’t even well populated.

Technology reports

The “Connection Type” report, along with “Monitor Color Depth”, measure things that haven’t been an issue in too many years to continue reporting on. LAN, 16-bit or higher.

“Monitor resolution” is irrelevant in the face of also having “Browser Width” & “Browser Height” reports (the true size of the web page “real estate” on screen).

Finally, JavaScript version? The JavaScript report with “Enabled/Disabled” is likely more than enough information. Or, you can just include jQuery in your website and know with 100% certainty what version is being used.

Visitor Profile reports

My dislike of the identified Visitor Profile reports are due to halfway implementation. The “GeoSegmentation report shows a nice map representation, but only of traffic metrics like Page Views and Visits. Why not open this up to conversion variables and really make the visualization useful, instead of needing to rely on the “flat”, non-map Visitor Zip (s.zip) report?

For the “Domains” and “Top-Level Domains” report, you have granularity issues; the “Top-Level Domains” report is sort-of a country-level report, but the U.S. has several line items. The “Domains” report shows what ISP people are using to access the Internet (which I think is generally useless in itself), but again…it spans geography, so the ISP network someone is on may not even have the same technology. So what are we really measuring in these reports?

Traffic Sources reports

The “All Search Page Ranking” report seems like it could be useful, until you realize that 1) it aggregates all search engines (whose different algorithms provide different rankings and 2) with personalized search, rankings are no longer static. Literally every single person could see a different link position for the same search term. So while this report may have made sense for SEO measurement in the past, it’s really past it’s prime…use the right SEO tool for the job (Conductor, SEOmoz, and the like).

The “Original Referring Domains” report is weird in its own way…the absolute first URL that referred you to the site. Really? As Avinash has said, giving 100% credit to the first touchpoint is like giving your first girlfriend credit for you marrying your wife (paraphrased). This report is very limited in its usefulness IMO, especially given the advances in attribution modeling in the past several years.

Custom Variable: s.server report

The only custom variable report I have on this list is the s.server report; hopefully, all of your other custom variables are capturing only business-useful information!

The reason I dislike the s.server variable/report is the same reason I dislike the “All Search Page Ranking” report; use the right tool for the job. This is a lazy way of monitoring server volume for load balancing. But if you’re doing the job well on the back-end, shouldn’t every server have the same level of volume?

Even if the answer to the previous question is no (I’m not a network engineer, clearly), having an operational report like this doesn’t make much sense to me in a marketing reporting tool.

By hiding reports in the Omniture menu interface, this doesn’t mean the info stops being collected or becomes unavailable to all users. Rather, the option to use the reports isn’t immediately obvious (since they don’t show up in the menu). Power Users can still find these reports using the search box if necessary to answer an oddball question.

But in my experience, the information in these reports are generally not business useful, or are lacking in some critical way. If you can’t make regular, high impact decisions with the info, then you’re better off never looking at it at all.

Apple Has Earned a Customer for Life

Mon, 25 Jun 2012 20:38:57 +0000

Broken MacBook Pro hinge (due to glue failure)

I used to think that when people talked about the “legendary Apple customer service” that there was plenty of hyperbole thrown in for good measure. Until it happened to me with my broken MacBook Pro hinge.

“Broken MacBook Pro Hinge” - Plenty of search results

When the screen on my late 2008 15” MacBook Pro started separating from the hinge, the first thing I did was search Google. There I found more than enough search results to make me believe this was a widespread issue with this vintage of laptop. And since the laptop was out of warranty, most of the results talked about re-gluing the aluminum screen cover to the hinge.

After trying to re-attach the hinge to the screen using epoxy, I headed over to the Apple store in King of Prussia, PA. To say this first encounter at the Genius Bar was frustrating is an understatement.

You should’ve bought AppleCare

Apple ~~cashiers~~ “Geniuses” and fanboys alike are very big on pushing the AppleCare warranty, selling you with tales that Apple will fix anything in that extended time period. While that may be true, extended warranties generally don’t pay off for the consumer, and as such, I don’t buy them.

Not that it would have mattered for me anyway. My MacBook Pro is well beyond 3 years old, one of the first unibody models that came out. You think the Apple “Genius” would’ve known that after checking the serial number, but instead just kept repeating robotically:

“You should’ve bought AppleCare. You should’ve bought AppleCare.”

Even when I asked, “A glue failure doesn’t seem like a manufacturers defect?” or “I should’ve paid $349 for an extended warranty to protect against $0.05 of faulty glue?”

“You should’ve bought AppleCare.”

At that point, after being asked if I dropped the laptop, given a series of robotic answers, suggested that I should’ve spend $349 that wouldn’t have fixed my problem, and generally treated like a monkey, I felt like smashing the laptop right on the Genius Bar just to make a scene. Instead, I walked out feeling worse than when I arrived, with crippled MacBook Pro in hand.

Maybe an Apple Certified Repair facility can help

Since I wasn’t going back for a second round of stupidity at King of Prussia Apple Store, I decided to look up an independent shop to see what the cost of repair would be. The repair guy immediately said “Oh, I’ve seen this a few times recently…it’s probably around $500-$600 to fix.”

$%^$&%*(#! For $600, I’d be about 30-35% of a new 15” MacBook Pro. Again I left a store without doing anything, and feeling worse than when I arrived. I either need to pay $600 or pay $2000+ to get the newer equivalent of my laptop.

One more trip to the Apple Store

Several weeks had passed and my laptop became pretty much unusable. I decided to bite the bullet and pay to get the screen fixed. I also decided to go back to an Apple Store (this time, in Ardmore, PA) to have them fix it. I figured if I’d have to pay, might as well guarantee it would get fixed properly.

When I walked up to the Genius Bar, the Apple “Genius” still asked me if I dropped my laptop (sidebar: Is this part of the mind tricks they give everyone? There isn’t a scratch on the thing, let alone any dents). After the Apple employee looked over the laptop, I told him in my most dejected voice that I wanted to find out how much is was to replace the screen.

Apple Genius: “How about ‘free’?”

I damn near fell off the stool I was sitting on. How could the Apple Store in King of Prussia been so unhelpful, and then 5 minutes into the same explanation I get an offer to get the screen fixed FREE at the Suburban Square Apple Store in Ardmore?

Apple Genius: “And we can probably get this back to you by tomorrow.”

Needless to say, I didn’t want to do anything except hit ‘Accept’ on the electronic repair form. I’ve come too far to mess this gift up!

Apple, you’ve earned yourself a lifetime customer

Maybe I got lucky. Maybe it was perseverance. Maybe this screen/hinge defect has shown up too many times in the last six weeks and Apple could no longer ignore it.

Maybe it’s because I asked twice at two different Genius appointments. Or maybe Apple has realized I’ve spent several thousand dollars with them in the past several years, with this MacBook Pro, iMac, several iPhones and an iPad. That level of spend probably doesn’t even get me in the top 50% of non-business customers, but it’s not negligible either.

Whatever the reason, by comping me the $492.41, Apple has “bought” themselves a customer for life.

The cost of a broken MacBook Pro hinge? Apparently, $492.41!

Edit: To read the follow-up of what eventually ended up of this MacBook Pro, click here for an article about me replacement battery interaction with Apple.

For Maximum User Understanding, Customize the SiteCatalyst Menu

Wed, 06 Jun 2012 17:31:39 +0000

@nancyskoons @randyzwitch @shawncreed Best Practice #1: Customizing anything is better than customizing nothing. #measure #omniture
— Jason Egan (@jasonegan) June 6, 2012

Default Omniture report menu

Visits vs. Visitors vs. Unique Visitors…click-throughs, view-throughs, bounces…these concepts in digital analytics are fairly abstract, and many in business and marketing never really grasp the concepts fully. Knowing the enormous amount of learning that needs to take place for digital success, why do we make our internal stakeholders hunt for data that’s organized by TOOL definitions, instead of by business function?

In this case, the “tool” that I’m referring to here is Omniture SiteCatalyst. To be clear, there’s nothing excessively wrong about the default menu structure in Omniture, just that in my experience, understanding by end-users can be greatly enhanced by customizing the Omniture menu.

Simple modifications such as 1) Hiding Omniture variables and products not in use, 2) organizing reports by logical business function, and 3) placing custom reports and calculated metrics next to the standard SiteCatalyst reports will get users to making decisions with their data that much faster.

1) Hide Omniture variables and products not being used

Do your users a favor and hide the Omniture products such as Test & Target, Survey, and Genesis if you aren’t using them. Same thing with any custom traffic (props) and custom conversion variables (eVars) that aren’t being used. Nothing will distract your users faster than clicking on folders with advertisements (T&T, Survey) or worse, frustrate the user by making them wonder “What data is supposed to be in this report?”

Just by hiding or disabling these empty reports and tools advertisements, you should see an increased confidence in data quality. Or at the very least, keep the conversation from taking a detour.

2) Organize SiteCatalyst reports by logical business function

Your internal users aren’t thinking about Omniture variable structures when they are trying to find the answer to their business questions. So why do we keep our data artificially separated by “Custom Events”, “Custom Conversions” and “Custom Traffic”?

Worse yet, who remembers that the number of Facebook Likes can be found at “Site Metrics -> Custom Events -> Custom Events 21-30?” And why are Facebook Likes next to “Logins”? Does that mean Facebook Logins? Probably not.

Wouldn’t it be better for our users to organize reports by business function, such as:

Financial/Purchase Metrics (Revenue, Discounts, Shipping, AOV, Units, Revenue Per Visit)
Usability (Browser, Percent of Page Viewed, Operating System)
SEO (Non-campaign visits, Referring Domains)
Mobile (Device, browser, resolution)
Site Engagement (Page Views, Internal Campaigns, Logins)
Site Merchandising (Products Viewed, Cart Add Ratio, Cross-Sell)
Social (Facebook Likes, Pinterest Pins, Visits from Social domains)
Paid Campaigns (Email, Paid Search, Display)
Traffic (Total Visits, Geosegmentation)

The list above isn’t meant to be exhaustive, or necessarily how you should organize your SiteCatalyst menus. But for me, organizing the reports by the business function keeps my business thinking flowing, rather than trying to remember how Omniture was implemented by variable type.

3) Place custom reports and calculated metrics next to the standard SiteCatalyst reports

This is probably more like “2b” to the above, but there’s no reason to keep custom reports and calculated metric reports segregated either. Custom reports happen because of a specific business need, and the same thing with calculated metrics. By placing these reports along with the out-of-the-box reports from SiteCatalyst, you take away the artificial distinction between data natively in SiteCatalyst and business-specific data populated by a web developer.

Why you wouldn’t want to customize?

Shawn makes two great points in his post about (not) customizing the SiteCatalyst menu: users require special training and menu customization isn’t scalable.

Users need special training

Users need to be trained anyway. I don’t think either of us is suggesting moving all of the menus around after an implementation has been in place for years…but if you’re a company just starting out, why not start off customized?

Fellow Keystoner Tim Patten also commented to me via Twitter DM about power users being used to “default”, and it’s annoying have to learn a new menu when switching companies; I’m not really worried about power users, I’m thinking about the hundreds of users in thousands of organizations who can’t get beyond page views and visits. Power users can pick up a new menu quickly, switch back to default, or use the search box.

This is very much true. The larger the company, and the more complex and varied the tracking, inevitably menu customization isn’t particularly scalable. This is probably an area where specific dashboards are a much better strategy than customizing the menus.

Summary

For me, one of the first things I look for when working with a company looking to get their digital analytics program off the ground is whether they’ve customized their Omniture menu structure. As a free customization, it’s something that companies should at least consider. Organizing reports by business function requires a business to think about the questions they want to regularly answer, will keep novice users from focusing on implementation concepts, and overall is just better because it’s how I think 🙂

This blog post is a continuation of a Twitter conversation with Shawn C. Reed (@shawncreed), Jason Egan (@jasonegan), Tim Patten (@timpatten) and others. Shawn’s counter-argument can be found here. Jason wrote about Omniture menu customization a few years back. And finally, if you want to read more pros-and-cons about SiteCatalyst menu customization, see the Adobe blog posts here and here.

Effect Of Modified Bounce Rate In Google Analytics

Thu, 10 May 2012 08:05:02 +0000

A few months back, Justin Cutroni posted on his blog some jQuery code that modifies how Google Analytics tracks content. Specifically, the code snippet changes how bounce rate and time on site are calculated, creates a custom variable to classify whether visitors are “Readers” vs. “Scanners” and adds some Google Analytics events to track how far down the page visitors are reading.

Given that this blog is fairly technical and specific in nature, I was interested in seeing how the standard Google Analytics metrics would change if I implemented this code and how my changes compared to Justin’s. I’ve always suspected my bounce rate in the 80-90% range didn’t really represent whether people were finding value in my content. The results were quite surprising to say the least!

Bounce Rate - Dropped through the floor!

Starting April 24th, Bounce Rate drops considerably!

As expected, implementing the content tracking code caused a significant drop in bounce rate, due to counting scrolling as a page “interaction” using Google Analytics events. Thus, the definition of bounce rate changed from single page view visits to visitors that don’t interact with the page by scrolling at least 150 pixels.

In the case of my blog, the bounce rate dropped from 80-90% to 5-15%! This result tells me that people who arrive on-site aren’t arriving by accident, that they are specifically interested in the content. Sure, I could’ve validated this using incoming search term research, but this provides a second data point. The content I provide not only ranks well in Google, but once on-site also causes readers to want to see what the article contains.

Readers vs. Scanners

Even with the bounce rate drop above, I really don’t get a good feeling about whether people are actually reading the content. Sure, people are scrolling 150px or more, but due to the ADHD nature of the web, plenty of people scroll without reading just to see what else is on the page! That’s where the “Readers vs. Scanners” report comes in:

62% of visits only scan instead of read - Need to do better here!

The report above shows that only 38% of visits to the site actually READ an article, rather than just quickly scroll. This is disappointing, but now that I’ve got the information being tracked, I can set up a goal in Google Analytics with the aim of improving the ratio of actual readers vs. quick scrollers.

Average Visit Duration - Still useless

Like the bounce rate definition change above, average visit duration and average time on page also change definitions when using the jQuery content tracking code. Given that Google Analytics calculates time metrics by measuring the time between page views or events, by adding more events on the page, all time on site metrics have to increase (by definition).

Hard to see because of the Y-axis, but Avg. Visit Duration increases significantly as well.

That said, average visit duration is still a pretty useless metric, given that an increase/decrease in this metric doesn’t immediately tell you “good” or “bad”…

Content Consumption “Funnel”

Finally, the last change that occurs when you implement the content tracking code is a series of Google Analytics events that measure how far down the page visitors are actually seeing. This report, in combination with the Readers vs. Scanners report, helps understand reader engagement better than any generic “Time on Site” metric can do.

From this report, I can see that of the 2,102 articles loaded:

89.4% of the articles have a “StartReading” event fired
89.8% of those who start to read an article reach the bottom of the article.
19.7% of those who reach the end of the article scroll past the comments to reach the true end of page

The first metric above is analogous to subtracting the bounce rate from 1, the percentage of articles viewed that don’t bounce. The second metric (complete articles seen), with a success rate of 89.8% is ripe for segmentation. I stated above that only 38% actually READ an article, so segmenting the above report by “Readers” vs. “Scanners” will surely lower the success rate in the “Readers” population.

Finally, that <20% actually touch the true bottom of page is surprising to me, since this blog really doesn’t get many comments! If there were thousands of comments and the pages were really long, ok, no one sees the bottom…but here? I’ll have to think about this a bit.

Great update to Google Analytics default settings!

Overall, my impression of the jQuery code snippet developed by Justin and others is that it is extremely useful in understand interaction of visitors to content sites. The only downside I see here is that it changes the definition of bounce rate within Google Analytics, which could be confusing to others who 1) aren’t aware of the code snippet running on-site or 2) don’t quite understand the subtleties of Google Analytics implementation with respect to Events and the non-interaction setting.

But since this is my personal blog, I don’t need to worry about others mis-interpreting my Google Analytics data, so I’m going to keep this functionality installed!

Update 7/25/12: Google Analytics published a similar method to the one described above, using “setTimeout” to modify bounce rate based solely on time-on-page.

Adobe Discover 3: First Impressions

Fri, 27 Apr 2012 08:24:21 +0000

With yesterday’s code release, ~~Omniture~~ Adobe released version 3 of their “Discover” tool, THE way to perform web analysis within the Adobe Digital Marketing Suite. While SiteCatalyst has its place for basic reporting, to really dig deep into your data for actionable insights there’s no substitute to using Discover.

But as with every product overhaul, there is the potential to change things that users liked and while not make enough improvement to excite the user base…but luckily, that’s not the case with Discover 3. Here’s how I see the new features and design changes.

New “Darth Vader” interface

"Ooh, tough looking. Just like hardcore web analysts!"

Of all the cool things about Discover 3, I’m not sure the new color palette is one of them. Several reasons were given by Adobe for choosing the carbon colored interface, from trying to match analyst’s personalities (yuck!), reducing eye strain (ok), and consistent branding (eh). Of the three, I’ll say that reducing eye strain is a worthy goal, although Discover 3 never struck me as “eye-burning” in the past.

Maybe I’ll grow to like it, but right now, it seems really dark. The light gray text on dark gray background needs a bit more contrast, and in general, the interface feels kinda depressing.

Calendars - No more #^%&$ sliders!

Now we’re getting somewhere. The slider interface in Discover 2 never made sense to me. You pick your time period up front, open a report, and then to modify the time period within an individual report you needed to move a bunch of jerky sliders around.

In Discover 3, we now have the same style calendar interface as SiteCatalyst. Makes sense from a consistency standpoint within the Adobe Digital Marketing Suite and a general UX standpoint. Pointing at two dates on the calendar is way easier and faster than moving endpoints of a slider!

Heterogeneous Pathing

This is so completely badass and the best new feature of Discover 3. No longer are you confined to a fallout report that only includes just one Omniture variable type. So if I want to do a funnel that measures visits containing a few different pages, then triggering a Facebook ‘Like’ event, a Cart Open, then an Exit Link, I can now do so!

You can also switch from “Visit-level” to “Visitor-level” on the fly, which can also be useful depending on how your view your business. Some people like to think about every visit being an opportunity to convert on-site, whereas Avinash advocates in his Web Analytics 2.0 book that using Visitors as the denominator for conversion rate is the proper thought model. I won’t weigh in on the difference in this post, but it’s cool that we can now change back-and-forth to see what the differences in the data are.

Table Builder

Nice drag-and-drop options, very PivotTable like

Finally, the last really obvious difference between Discover 2 and Discover 3 is the table builder while using ranked reports. Like the eye-strain issue talked about above, the amount of time that it took for reports to build never really seemed like an issue to me. Perhaps that’s the ~~SAS~~ programmer side of me that often waits hours to return a result of a complex set of commands.

But now that I’ve used the table builder, it’s definitely an improvement on how data tables get built. You get to specify each element you want in the table first, THEN the data gets retrieved. It may sound like a small change, but when you already know what you want, not having to wait for the table to build while you keep dragging in metrics does feel like it’s way faster to get the table you are looking for.

Adobe Discover 3 - Definitely an improvement

There are probably 20 other things I haven’t noticed yet in the new Discover 3 interface, but from what I have used so far, this is a great upgrade in functionality! It feels faster to get things completed with the table builder and the new pathing functionality across all variable types is a long time coming. Now, if only there was a different color palette I could choose, it’d be perfect…maybe something like this?

You should prefer green, not carbon.

Using Omniture SiteCatalyst Target Report To Calculate YOY growth

Wed, 22 Feb 2012 08:15:18 +0000

Of the hundreds of stock reports and capabilities present within Adobe (Omniture) SiteCatalyst, calculating year-over-year growth isn’t the easiest thing to do. And while conversion reports (eVars) have the “Compare Dates” functionality within the calendar menu, we can’t quickly plot the difference between two time periods within a dashboard. This is where the Omniture SiteCatalyst Target report comes in handy.

Setting up your “Goal”

Within the Omniture Knowledge Base KB2153, I think Omniture does a disservice by stating:

Targets are quantifiable goals that you can place within the SiteCatalyst interface and compare against reports.

While this is a true statement, I think one of the reasons that the Omniture SiteCatalyst Target report isn’t more widely used is that it doesn’t have to be a future “goal” per se, any set numbers can be used. When last year’s numbers are used, the report becomes a year-over-year comparison!

For this example, I’m going to be comparing page views year-over-year. Here’s what the page views summary by month looks like for 2011:

Omniture Page Views Report - 2011

Using this report to set a year-over-year target, we can see the early months are in the few thousands of page views, increasing to 12,000 -14,000 later in the year.

Omniture SiteCatalyst Target interface - Inputting our numbers

Monthly Target Setup within Omniture Interface

Assuming you are using Omniture SiteCatalyst v15, you set up a Target report under Favorites -> Targets -> Manage Targets, then choose Add New once you’re in the Targets menu. I’ll be setting up a monthly target for Page Views, so I’ll just type it in instead of using the file upload capability. For this example, we want to apply this target to “Entire Site” for the “Page Views” Metric. The date range will be all of 2012, with “Monthly” granularity. This will give you 12 boxes to type in the 2011 Page View results, and once we hit “Ok” to save, we’ll have our year-over-year report set up.

Getting the Year-over-Year graph within Omniture

Showing the results of our newly created Target report is as easy as going to Favorites -> Targets, then choosing the appropriate Target. By default, the report will show like a normal metric report, with a green overlay for your targets:

Default view of Omniture SiteCatalyst Target report

The above report shows that Page Views for January 2012 are well above our January 2011 “Target” and that February has already exceeded the goal as well…which is great since we’ve got 7 more days left in the month!

If we want to show the year-over-year delta, however, we can choose the “Variance” report option at the top of the graph. Doing so will show the following report:

Omniture SiteCatalyst Target Report - "Variance" option

By placing this report in a dashboard, we can quickly evaluate whether Page Views have grown by month year-over-year. It’s disappointing that the only graph option Adobe provides is the raw metric higher/lower than the target instead of a percentage difference view, but the percentage difference is calculated as part of the data table view that goes along with this report.

Conclusion

When talking about “growth” many businesses aren’t content with just year-over-year growth, usually aiming for 10%, 20%…10,000% growth. These are goals that work well to track within the Omniture SiteCatalyst Target report. But year-over-year growth can be worth monitoring too, and the Omniture SiteCatalyst Target report is a great way to do so.

randyzwitch - Articles

Building pyarrow with CUDA support

Step 1: Docker with GPU support

Step 2: Setting up the Ubuntu Docker container

Step 3: Cloning Apache Arrow from GitHub

Step 4: Installing remaining Apache Arrow dependencies

Step 5: Building Apache Arrow C++ library

Step 6: Building pyarrow wheel

Step 7 (optional): Validate build

A Beginner's Look at BenchmarkTools.jl

Read The Manuals!

Evaluate Your Benchmark Over the Range of Inputs You Care About

Case Study: Speeding Up TStringValue

10x6 observations

10x7 observations

Parallelism > Compiler Hinting

RSiteCatalyst Version 1.4.16 Release Notes

Community Contributions

DataOps Summit: Streaming Real-time Telemetry With OmniSci and StreamSets

ODSC webinar: End-to-End Data Science Without Leaving the GPU

PyData NYC 2018: End-to-End Data Science Without Leaving the GPU

Using pandas and pymapd for ETL into OmniSci

Computers Are Dumb, MAKE THINGS EASIER FOR THEM!

Pandas To The Rescue!

Uploading to OmniSci Directly From Pandas

Computers May Be Dumb, But Humans Are Lazy

Mirroring an FTP Using lftp and cron

Data are on an FTP, Need Further Processing

lftp mirror to the Rescue!

Run This Every Five Minutes, Forever

That’s It? YES!

Creating an OmniSci ODBC Connection in RStudio Server

1. Installing RStudio Server on Microsoft Azure

2. Installing an ODBC Driver Manager

3. Installing MapD ODBC Driver System-wide

4. Defining A DSN

ODBC: A Big Bag Of Hurt, But Super Useful

Data Science Without Leaving the GPU

Parallel, Disk-Efficient .zip to .gz Conversion

Bulk Loading Shapefiles Into Postgres/Postgis

Step 1: Downloading Shapefiles

Step 2: Bulk Unzip

Step 3: Creating Postgis Table Definition

Step 4: Bulk Loading Shapefiles into Postgis

Spend Time Now To Save Time Later

Using RSiteCatalyst With Microsoft PowerBI Desktop

Step 1: Call Adobe Analytics API Using Get Data Menu

Limitations

Dashboards, Dashboards, Dashboards!

Getting Started With OmniSci, Part 2: Electricity Dataset

PJM Metered Load Data

Top 10 Usage Days By Season

Building A Dashboard Using MapD Immerse

Programmatic analytics using pymapd

RSiteCatalyst Version 1.4.14 Release Notes

Community Contributions

Getting Started With OmniSci, Part 1: Docker Install and Loading Data

Installing MapD CE using Docker/nvidia-docker

Loading Data Using the Immerse Interface

Loading Data Using the Command Line

Simplistic Query Performance

Dashboards, Streaming Data and more…

Parallelizing Distance Calculations Using A GPU With CUDAnative.jl

Getting Started

Julia CPU Implementation

CUDAnative.jl Implementation

Kernel Code

Coordination Code

Performance

What I Learned

RSiteCatalyst Version 1.4.13 Release Notes

Community Contributions

RSiteCatalyst Version 1.4.12 (and 1.4.11) Release Notes

Get* method additions

Bug fixes

Community Contributions (An Adobe Summit bounce?!)

Self-Service Adobe Analytics Data Feeds!

Landing the Files via FTP/SFTP/S3

Data Column Definitions

Save and Repeat

Case Study: Speeding Up `TStringValue`

lftp `mirror` to the Rescue!

Step 1: Call Adobe Analytics API Using `Get Data` Menu

QueueSummary: Now with `date.to` and `date.from` keywords

Trivial Fixes: Silenced `httr` message, clarified documentation

Problem 1: Concatenating (`cat * >> out.txt` ?!)