(Liveblogging) Big Query and Prediction APIs (#io2010)to


Notes from the session:

Google infrastructure takes care of data storing, query scaling — and exposes Google’s deep analytics capability to users for leveraging in custom applications.

  1. Benefits – scalability, security, sharing,, easy integration with GAE, Google Spreadsheets
  2. 3 Steps: Upload (data to Google Storage), Process (Import to tables to train a model), Act (Run queries and make predictions)
  3. Big Query and Prediction APIs bridge user data and user apps directly in the cloud
  4. Security: SSL for securing interactions, user owns data (respects user ACLs)
  1. Many use cases: interactive tools, spam, trends detection, web dashboards, network optimization
  2. Deep dive: consider use case of monitoring large network of machines to detect network issues or threats.
  3. The M-Lab “open platform for advanced network research” (http://www.measurementlab.net) — we’ve imported their data into BigQuery so they can analyze their data with our tools. Doing a demo with 60 billion rows of data.
  4. BigQuery interface using simple SQL syntax — demo showing query that filtered that data, normalized results and returned them in a table — all within seconds (very real time responsive
  1. Key Capabilities of BQ: scalable (billions of rows), Simple (queries in SQL exposed via Web api), Fast.
  2. No need to worry about indices, sharding data or defining keys — BQ import takes care of it all
  3. No need to provision machines or resources – queries executed via simple API
  4. Writing Queries: Compact subset of SQL supported. Common functions (Math, String, time) supported for group or ordering. Also added statistical approximations (allow tradeoff of accuracy for speed) — e.g., TOP, COUNT DISTINCT
  5. API: Standard RESTful interface.
    GET /bigquery/v1/tables/{table name}
    GET /bigquery/v1/query?q={query}
    Returns JSON response

  6. Security and Privacy: support common Google Auth (Client Login, OAuth, AuthSub). HTTPs support (protect data, credentials), uses Google Storage for Developers to manage access
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: