Insightful News

We are an addiction for those who love reading.

Insightful News

Visualizing huge scale Uber Movement Data

New York’s cab information visualization from Uber’s Engineering weblog

Last month considered one of my acquaintances in LinkedIn pointed me to an excessively attention-grabbing dataset. Uber’s Movement Dataset. It used to be attention-grabbing to discover their superior GUI and to play with the knowledge. However, their UI for exploring the dataset leaves a lot more to be desired, particularly the truth that we all the time need to specify supply and vacation spot to get related information and can not play with the entire dataset. Another limitation additionally used to be, the dataset does not come with any time part. Which in an instant threw out numerous issues I sought after to discover.

When I began having a look out if there may be every other publicly to be had dataset, I discovered one at Kaggle. And then slightly a couple of extra at Kaggle. But none of them gave the impression respectable, after which I discovered one launched through NYC – TLC which regarded lovely respectable and I used to be hooked.
To discover the knowledge I sought after to take a look at out OmniSci. I lately noticed a video of a communicate at jupytercon through Randy Zwitch the place he is going thru a demo of exploring an NYC Cab dataset the usage of OmniSci. And since my dataset used to be similar to that, I considered giving it a check out.

You can in finding the Jupyter Notebook right here:


Just as a toy experiment I attempted to respond to and visualize the next.

Can we visualize the choice of Uber Trips in a duration

Distribution in keeping with in keeping with hour, week and month

Estimated Monthly base income 

Distribution of visitors between months

Distribution in keeping with weekdays and weekends on quick and lengthy journeys

Distribution of Trip Duration

Trip Duration vs Trip Distance

And additionally a host of attention-grabbing information we will be able to glean from this dataset.

However, whilst attempting to do that, I spotted it is lovely arduous to paintings on an enormous dataset in jupyter at once in the event you load the entire dataset right into a dataframe anyway.

I used OmniSci’s Cloud interface to load up my information after which hook up with that dataset the usage of pymapd to learn the square information.

What I didn’t do used to be to be good and make the most of OmniSci’s tremendous tough mapd core and slice and cube the dataset within the cloud itself. Which cloud have stored me numerous time. For instance, the question I used to be working on one-sixth of the entire dataset used to be taking 25 mins.

You can check out a few of my tough concepts, tries and extra graphs on this Jupyter Notebook.

However, it kind of feels OmniSci additionally has a really perfect useful visualization internet interface as a part of OmniSci Cloud known as Immerse. And I used to be in a position to prepare dinner up those dashboards in lower than five mins.

And Immerse used to be in a position to crunch thru the entire dataset (now not one-sixth) nearly straight away and bring those charts for me. I’m lovely inspired with it up to now. And it kind of feels with help of pymapd and crafting some square queries, I will have to have the ability to harness this velocity as neatly. 

That can be my subsequent check out most probably.

What’s Next:

Since I spotted how tough OmniSci Immerse will also be and beginning to play with pymapd. My subsequent puppy undertaking is merging Uber Movement’s annually information with ward primarily based time sequence information. So that we will be able to recreate the entire dataset and analyze probably the most attention-grabbing facets of it as we did above. I’m most commonly to look (ideally in Bengaluru information)

  • Uber’s expansion thru time (and explicit job expansion in several wards)
  • Figuring from historic time sequence which wards and routes have maximum visitors during which hour (this additionally will have to allow us to expect which spaces would possibly face surge pricing)
  • See if the expansion has saturated in any explicit position (will have to give us higher threshold for that house)
  • If an increase in Uber Demand at once co-relate to trip time (perhaps the higher call for is inflicting visitors?)
  • Can we load it up in (extra in particular the usage of this demo as a template) and feature a pleasant timeseries visualization?
Should be a a laugh undertaking!


Since I wrote this put up I used to be additionally enjoying with what we will be able to do if we visualize this information in VR. And I’ve a initial asnwer 😀

Source hyperlink

Leave a Reply

Your email address will not be published. Required fields are marked *