Posted:
Ville
Navneet

By Navneet Joneja, Product Manager,
and Ville Aikas, Technical Lead


When evaluating options for cloud storage, customers often wonder, "How can we optimize our storage to get the highest performance possible?". We believe you shouldn't have to, so we do all the optimization for you – enabling you to focus on your application instead of the minutiae of storage optimization.

The performance of cloud storage services (and indeed most web services) depends on two main factors: the network that moves the data between us and the end user, and the performance of the storage service itself.

1. Network

When you make a request to Google Cloud Storage, one of the key determinants of performance is the network path between you and our servers. This path is critical because if the network is slow or unreliable, it doesn’t really matter how fast the backend is.

There are two main ways to make the network faster:
  • Serve the request from as close to the user as possible
  • Optimize the network routing between the end-user and the service, including avoiding pockets of network congestion and minimizing the number of network hops between the user and the service.
2. Storage

The other component of system performance is how quickly our servers process your request. The data needs to be managed optimally and once an end-user’s request reaches our servers, we need to serve the request as fast as possible. In a sense, Google Cloud Storage is a gigantic filesystem: authorization checks need to happen, the object in question needs to be looked up, and the data requested needs to be read from the physical storage medium and transferred to the end user, all as efficiently as possible.

So, how do we make sure your requests are served as fast as possible?
  • Google Cloud Storage is built on Google’s proprietary network and datacenter technology. We’ve spent more than a decade building out proprietary infrastructure and technology to power Google’s sites (after all, we believe that fast is better than slow). When you use Google Cloud Storage, the same network goes to work for your data.
  • We replicate data to multiple data centers and serve an end-user’s request from the nearest data center that holds a copy of the data. We also offer a choice of regions (currently U.S. and Europe) to allow you to keep your data close to where it’s most needed. We then take this one step further. When you upload an object and mark it as cacheable (by setting the standard HTTP Cache-Control header), we automatically figure out how best to serve it using Google’s broad network footprint, including caching it closer to the end-user if possible.
  • Finally, you don’t need to worry about optimizing your storage layout (like you would on a physical disk), or the lookups (i.e. directory and naming structure) like you would on most file systems and some other storage services. We take care of all the "file system" optimizations behind the scenes.
In other words, when you store your data on Google Cloud Storage, we do all the background work to make it fast so that you can focus on your application.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious almost-two-year-old.

Ville Aikas likes to work on tools and services that make developers lives easier and "just work". When not busy cranking out code, he loves to play soccer with his kids, build robots and watch F1.

Posted by Scott Knaster, Editor

Posted:
Author Photo
By Navneet Joneja, Product Manager

Google Cloud Storage enables you to use our storage and network infrastructure to store and access your data with high reliability, scale and performance. Today, we’re launching several frequently requested new features:

Signed URLs and updated browser uploads

Many of you have asked us for "virtual valet keys" that give limited access to specific data to the bearer for a short time, enabling them to implement application-managed access control for mobile applications, premium content distribution, and so on. You can now implement these applications and more using short-lived signed URLs to address any object stored in Google Cloud Storage. This feature gives your application another powerful tool to control access to any piece of data. You can also use this feature to enable browser-based uploads from your end users to Google Cloud Storage without requiring them to have Google accounts (browser-based uploads were previously limited to the interoperable API). URL signing is implemented using PKCS-12 keys and the industry-standard RSA algorithm and is currently experimental.

Cross-Origin Resource Sharing

We now support configuring storage buckets to return appropriate Cross-Origin Resource Sharing (CORS) headers, which streamlines the development of advanced JavaScript applications (such as browser-based games) using Google Cloud Storage. Other uses include serving web fonts from Google Cloud Storage and enabling trusted JavaScript access from your App Engine applications.

gsutil 3.0

We've been hard at work making gsutil easier to use at all levels. This latest release includes significant enhancements:
  • A refactored, cleaner code-base
  • Better in-tool documentation
  • Easy in-place update to new releases
  • Multithreaded operations
  • A hierarchical file tree abstraction layer that maps more closely to the way traditional file systems are organized.
To try all these features and more, download the latest version of gsutil (zip, tarball). Please note that wildcard and list bucket semantics have changed in gsutil to make the tool easier to use in a broad variety of use cases. You can read all about the latest update in the release notes.

We also recently reduced storage prices across all usage tiers by up to 15%.

As always, we welcome your feedback in our discussion group. If you haven’t tried Google Cloud Storage yet, you can sign up and get started here.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious almost-two-year-old.

Posted by Scott Knaster, Editor

Posted:
Author Photo
By Navneet Joneja, Product Manager

Google Cloud Storage is a robust, high-performance service that enables developers and businesses to use Google’s infrastructure to store and serve their data. Today, we’re announcing a new feature that gives you greater control over concurrent writes to the same object, and the availability of an App Engine Files API that makes it easier to read and write data from Java App Engine applications.

Write concurrency control

A number of our customers have asked us for greater control over concurrent writes, in order to implement features like strongly consistent write operations and distributed locking semantics in the cloud. In response to your feedback, we’re announcing the release of version-based concurrency control. Every time you update an object, it gets assigned a 32-bit, monotonically increasing sequence number. This version number is returned as a header with every GET or HEAD request. You can then use a conditional write operation to manage concurrent updates to the object (for example, when you want read-modify-write semantics). This feature is currently experimental.

AppEngine Files API for Java applications

Last fall, we announced the ability to read and write your Google Cloud Storage data using the App Engine Files API for Python applications. Today, we’re making the Files API available to Java App Engine applications too. This feature is currently experimental, and we’ll continue to enhance it in the months to come.

As always, we welcome your feedback in our discussion group. If you haven’t tried Google Cloud Storage yet, you can sign up and get started here.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious almost-two-year-old.

Posted by Scott Knaster, Editor

Posted:
Author Photo
By Dan Holevoet, Developer Relations Team

A platform is more than the sum of its component parts. You can read about it or hear about it, but to really learn what makes up a platform you have to try it out for yourself, play with the parts, and discover what you can build.

With that in mind, we started a project called Au-to-do: a full sample application implementing a ticket tracker, built using Google APIs, that developers can download and dissect.

Au-to-do screen shot

Au-to-do currently uses the following APIs and technologies:
Additional integrations with Google APIs are on their way. We are also planning a series of follow-up blog posts discussing each of the integrations in depth, with details on our design decisions and best practices you can use in your own projects.

By the way, if you’re wondering how to pronounce Au-to-do, you can say "auto-do" or "ought-to-do" — either is correct.

Ready to take a look at the code? Check out the getting started guide. Found a bug? Have a great idea for a feature or API integration? Let us know by filing a request.

Happy hacking!


Dan Holevoet joined the Google Developer Relations team in 2007. When not playing Starcraft, he works on Google Apps, with a focus on the Calendar and Contacts APIs. He's previously worked on iGoogle, OpenSocial, Gmail contextual gadgets, and the Google Apps Marketplace.

Posted by Scott Knaster, Editor



Posted:
Author Photo
By Navneet Joneja, Product Manager

Google Cloud Storage is a robust, high-performance service that enables developers and businesses to use Google’s infrastructure to power their data. Today, we’re announcing a new feature that makes it even easier to control and share your data.

Per-Bucket Default Object ACLs

Customers building a wide variety of applications have asked us for an easier mechanism to control the permissions granted on newly created objects. Now you can define your access control policy for a bucket once by specifying a Default Object ACL for any bucket, and we’ll automatically apply that ACL to any object without an explicitly defined ACL. You can always override the default by providing a canned ACL when you upload the object or by updating the object’s ACL afterwards. This mechanism simplifies wide variety of use cases, including data sharing, controlled-access data sets and corporate drop-boxes.

New buckets without Default ACLs

After analyzing how customers use our service, we’ve also decided to make a few small changes to the behavior of buckets that have no explicit default object ACL. Effective today, new buckets are created with an implied project-private default object ACL. In other words, project editors and owners will have FULL_CONTROL access to new objects, and project viewers will have READ access to them. This change better aligns the default behavior with how our customers use storage. You can change a bucket’s default object ACL at any time after creating the bucket.

Existing buckets have an effective default object ACL of "private", and they will continue to work as they always have until and unless you specify a new default object ACL for them.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious one-year-old.

Posted by Scott Knaster, Editor

Posted:

By Ju-kay Kwek, Product Manager

(Cross-posted on the Google App Engine Blog and the Google Enterprise Blog.)

Rapidly crunching terabytes of big data can lead to better business decisions, but this has traditionally required tremendous IT investments. Imagine a large online retailer that wants to provide better product recommendations by analyzing website usage and purchase patterns from millions of website visits. Or consider a car manufacturer that wants to maximize its advertising impact by learning how its last global campaign performed across billions of multimedia impressions. Fortune 500 companies struggle to unlock the potential of data, so it’s no surprise that it’s been even harder for smaller businesses.

We developed Google BigQuery Service for large-scale internal data analytics. At Google I/O last year, we opened a preview of the service to a limited number of enterprises and developers. Today we're releasing some big improvements, and putting one of Google's most powerful data analysis systems into the hands of more companies of all sizes.
  • We’ve added a graphical user interface for analysts and developers to rapidly explore massive data through a web application.
  • We’ve made big improvements for customers accessing the service programmatically through the API. The new REST API lets you run multiple jobs in the background and manage tables and permissions with more granularity.
  • Whether you use the BigQuery web application or API, you can now write even more powerful queries with JOIN statements. This lets you run queries across multiple data tables, linked by data that tables have in common.
  • It’s also now easy to manage, secure, and share access to your data tables in BigQuery, and export query results to the desktop or to Google Cloud Storage.

Michael J. Franklin, Professor of Computer Science at UC Berkeley, remarked that BigQuery (internally known as Dremel) leverages “thousands of machines to process data at a scale that is simply jaw-dropping given the current state of the art.” We’re looking forward to helping businesses innovate faster by harnessing their own large data sets. BigQuery is available free of charge for now, and we’ll let customers know at least 30 days before the free period ends. We’re bringing on a new batch of pilot customers, so let us know if your business wants to test drive BigQuery Service.


Ju-kay Kwek is a Product Manager for Google BigQuery Service.

Posted by Scott Knaster, Editor

Posted:
Author Photo
By Navneet Joneja, Product Manager for Google Cloud Storage

Google Storage for Developers is now out of Code Labs, and has a new name: Google Cloud Storage. In addition, we're also happy to announce some new features, and a significant price reduction.

App Engine File API Support

When we opened the service to all this summer, many of our customers asked for an easier way to use Google Cloud Storage with their App Engine applications. In response to your feedback, you can now read and write your data via the App Engine Files API, enabling you to quickly build your content management tools, data sharing applications, web games and more using the powerful combination of App Engine and Cloud Storage. This feature is experimental and currently Python-only, but we’re working on adding Java support and additional features.

Usage Information

We’re introducing a new API that gives you access to detailed usage information (including network access and storage use data). You can use this feature to analyze your usage, integrate with your analysis systems and build your own value-added applications using Google Cloud Storage. This feature is currently experimental.

Lower Prices

We're no longer charging for upload bandwidth into the Google cloud. In addition, we’re lowering our prices across the board and introducing volume discounts for our larger users. We are committed to offering an extremely high quality of service to all our customers. As the product has evolved, we’ve found ways to offer the same great service at a lower cost, so now our prices are lower too. For example, under our new prices, a customer storing a hundred terabytes of data, reading twenty terabytes and writing ten terabytes a month would pay approximately 40% less a month. The difference is even greater for customers with higher usage. Our new prices are retroactive to the beginning of October. Please see our updated pricing here.

As always, we welcome your feedback in our discussion group. If you haven’t yet tried Google Cloud Storage, you can sign up and get started here.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious one-year-old.

Posted by Scott Knaster, Editor

Posted:

By Navneet Joneja, Product Manager

For those of you who have been waiting to use Google Storage, we’re happy to announce that effective immediately, you can get a Google Storage for Developers account without needing to request an invitation.

We’ve also launched several significant enhancements to the service, including more flexible and powerful security features, simplified sharing, the ability to store data in Europe, support for larger objects (up to 5 TB), team-oriented accounts and a completely free promotional tier.

OAuth 2.0 Support
OAuth 2.0 is the new recommended Google Storage authentication and authorization scheme. OAuth 2.0 is an industry standard that we’ve adopted across Google, offering many benefits:
  • Simpler - never sign a request again! OAuth 2.0 uses access tokens for authentication, which obviate the need for complicated signature schemes. We recommend that all OAuth 2.0-authenticated requests be made over SSL.
  • More flexible and powerful - OAuth 2.0 allows for three-legged authentication, where a user can grant an application permission to access Google Storage on their behalf (and revoke the grant at any time if necessary).
  • Secure
    • Your user name and password are used once to create a long-lived (refresh) token which is scoped to allow access to Google Storage o behalf of that account. Your user name and password never needs to live on disk.
    • You don’t need to distribute your long-lived credentials to all the servers that need to access Google Storage. Instead, you can manage your long-lived credential centrally, and use it to create short-lived access tokens that you pass to your host cluster. This means that your long-lived tokens are safer.
    • You can revoke credentials at any time.
Learn more about using OAuth 2.0 with Google Storage for Developers here.

Simplified Sharing
We’re happy to announce that your customers and partners no longer need to sign up for Google Storage in order to be able to access data that you’ve shared with them; you can now share data with anyone who has a Google account.

New Storage Region: Europe
In response to popular demand, effective immediately, we’re adding a new option for data location. You can now choose to store your data in Europe - and get the same high-performance, massively scalable, reliable service as you do when you store your data in the United States.

Regardless of which region you choose, your data will be replicated to multiple geographically diverse Google data centers within the region in order to provide high levels of availability and reliability.

Learn more here.

Team-oriented accounts
Google Storage now uses a team-oriented account model, which allows you to create and manage your account in a more natural way.

We’ve moved to the Google API console for signup and administration, and adopted the same project-based account model that is used by other APIs. After creating a project, you configure billing for it once, and can then add any number of developers to it without requiring them to sign up for separate Google Storage accounts. This means individual developers on a team don’t need to configure billing in order to use Google Storage. This change also makes Google Storage account management consistent with other Google APIs.

Since buckets are now associated with projects rather than individuals, you will need to specify the new x-goog-project-id header with your list-buckets and create-bucket API calls. However, your existing code will continue to work using your new “default” project that already owns all the buckets you created before we implemented project-based accounts. Learn more about projects and the backwards-compatible default project mechanism here.

Support for chunked transfer encoding
Google Storage now allows you to upload your data to Google Storage without knowing the object size in advance, using the standard HTTP chunked transfer encoding mechanism. Using this feature, you can stream data into Google Storage instead of buffering it on your own server before sending it to Google.

New API Version
In order to continue to innovate and deliver high-impact features, we’ve found the need to make some backwards-incompatible changes. In order to deliver these features while retaining backwards compatibility for existing code that doesn’t need the new features, we’re introducing a new API version header. The new API version that implements this change is version 2. From version 2 onwards, all API calls will be versioned using the new x-goog-api-version header.

Version 2 of the Google Storage API does not support HMAC signature-based authentication. Although we strongly recommend that you move your code to use OAuth 2.0, explicitly specify the project ID when creating and listing buckets and use the new API version header, your existing code will continue to work; if you don’t specify the API version in your request, your requests will still be handled by our old API.

Free Trial Use
Starting right now, all new accounts will receive a monthly free quota of:
  • 5 GB of free storage,
  • 25 GB of free data transfer into Google Storage,
  • 25 GB of free data transfer out of Google Storage,
  • 2,500 free PUT, POST and LIST requests, and
  • 25,000 free GET, HEAD and other requests.
Existing accounts will continue to receive 100 GB of free storage until the July 1, 2011 and, in addition, will also receive the same free bandwidth and requests as new accounts. On July 1, 2011, they will automatically move to the new plan.

The new promotional plan will be effective until Dec 31, 2011. Please note that promotional usage will only apply to a user’s first project that uses Google Storage (for existing users, this is the project that was automatically created for them during the account migration).

If you’re an existing Google Storage for Developers user, thank you for using our product and for your valuable feedback that continues to help us evolve the service to meet your needs. As always, we continue to welcome your feedback in our discussion group. If you haven’t yet tried Google Storage, get your Google Storage account and get started for free today.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious one-year-old.

Posted by Scott Knaster, Editor

Posted:
Last month we introduced Google Storage for Developers with support for sharing with Google account holders. Today, we are enabling support for sharing with groups, giving you greater control and flexibility over how you share your data.

You can now easily share your Google Storage data with groups of users as well as individuals. Here are some examples of the things you can do with group sharing:
  • Share data with an entire department or mailing list
  • Make data publicly readable, but editable only by a group within your company
  • Keep access controls consistent with the current membership of a team
  • Add or remove a user's access to shared data via a single group membership change without having to change access control lists (ACLs) on all the buckets and objects involved.
You can share with groups by adding one or more Google Groups to the ACL of a bucket or object. A group member can be anyone with a Google Account, and members can authenticate using either their Google Storage credentials or in an authenticated browser session.

You can find more details on the new group sharing feature here. We have also updated our GSUtil command-line tool to support group sharing.

Thanks to everyone who has tested Google Storage so far. As always, we appreciate your feedback. Please let us know what you think on the Google Storage forum.

Posted:
To deliver our services, Google has had to develop sophisticated internal tools to process data more efficiently. We know that some of these tools could be useful to any developer, so we’ve been working to create external versions that we can share with the world.

We’re excited to introduce two new developer tools to get more from your data: BigQuery and Prediction API. These two tools can be used with your data stored on Google Storage for Developers.

BigQuery enables fast, interactive analysis over datasets containing trillions of records. Using SQL commands via a RESTful API, you can quickly explore and understand your massive historical data. BigQuery can help you analyze your network logs, identify seasonal sales trends, or find a needle in a haystack of big data.

Prediction API exposes Google’s advanced machine learning algorithms as a RESTful web service to make your apps more intelligent. The service helps you use historical data to make real-time decisions such as recommending products, assessing user sentiment from blogs and tweets, routing messages or assessing suspicious activities.

We are introducing BigQuery and Prediction API as a preview to a limited number of developers. There is no charge for using these services during the preview. To learn more and sign up for an invitation, please visit the BigQuery and Prediction API sites.

If you are in San Francisco for Google I/O, we look forward to meeting you. Please come to our session tomorrow to learn more.

Posted by Amit Agarwal and Jordan Breckenridge, BigQuery and Prediction API Teams

Posted:
As developers and businesses move to the cloud, there’s a growing demand for core services such as storage that power cloud applications. Today we are introducing Google Storage for Developers, a RESTful cloud service built on Google’s storage and networking infrastructure.

Using this RESTful API, developers can easily connect their applications to fast, reliable storage replicated across several US data centers. The service offers multiple authentication methods, SSL support and convenient access controls for sharing with individuals and groups. It is highly scalable - supporting read-after-write data consistency, objects of hundreds of gigabytes in size per request, and a domain-scoped namespace. In addition, developers can manage their storage from a web-based interface and use GSUtil, an open-source command-line tool and library.

We are introducing Google Storage for Developers to a limited number of developers at this time. During the preview, each developer will receive up to 100GB of data storage and 300GB monthly bandwidth at no charge. To learn more and sign up for the waiting list, please visit our website.

We’ll be demoing the service at Google I/O in our session and in the Developer Sandbox. We’re looking forward to meeting those of you who are attending.

Posted by Jessie Jiang, Google Storage for Developers Team