.. -*- mode: rst -*-

.. _storage_backends:

==================
 Storage Backends
==================

S3QL supports different *backends* to store data at different service
providers and using different protocols. A *storage url* specifies a
backend together with some backend-specific information and uniquely
identifies an S3QL file system. The form of the storage url depends on
the backend and is described for every backend below.

Furthermore, every S3QL commands that accepts a storage url also
accepts a :cmdopt:`--backend-options` parameter than can be used to
pass backend-specific options to the backend module. The available
options are documented with the respective backends below.

All storage backends respect the :envvar:`!http_proxy` (for plain HTTP
connections) and :envvar:`!https_proxy` (for SSL connections)
environment variables. However, S3QL currently supports only
CONNECT-style proxying. Therefore, there may be compatibility issues
with some proxy servers when using plain HTTP (i.e., when using the
:cmdopt:`no-ssl` backend option). A workaround for this case is to
either use SSL connections (for which proxying is fully supported) or
to use a direct connection without proxy.


Google Storage
==============

.. program:: gs_backend

`Google Storage <http://code.google.com/apis/storage/>`_ is an online
storage service offered by Google. To use the Google Storage backend,
you need to have (or sign up for) a Google account, and then `activate
Google Storage <http://code.google.com/apis/storage/docs/signup.html>`_
for your account. The account is free, you will pay only for the
amount of storage and traffic that you actually use. There are two
ways to access Google storage:

#. Use S3-like authentication. To do this, first `set a  default
   project
   <https://developers.google.com/storage/docs/migrating#defaultproj>`_.
   Then use the `key management tool
   <https://code.google.com/apis/console/#:storage:legacy>`_ to
   retrieve your *Google Storage developer access key* and *Google
   Storage developer secret* and use that as backend login and backend
   password.

#. Use OAuth2 authentication. In this case you need to use ``oauth2``
   as the backend login, and a valid OAuth2 refresh token as the
   backend password. To obtain a refresh token, you can use the
   :ref:`s3ql_oauth_client <oauth_client>` program. It will instruct
   you to open a specific URL in your browser, enter a code and
   authenticate with your Google account. Once this procedure is
   complete, :ref:`s3ql_oauth_client <oauth_client>` will print out
   the refresh token. Note that you need to do this procedure only
   once, the refresh token will remain valid until you explicitly
   revoke it.

To create a Google Storage bucket, you can use e.g. the `Google
Storage Manager`_. The storage URL for accessing the bucket in S3QL is
then ::

   gs://<bucketname>/<prefix>

Here *bucketname* is the name of the bucket, and *prefix* can be an
arbitrary prefix that will be prepended to all object names used by
S3QL. This allows you to store several S3QL file systems in the same
Google Storage bucket.

The Google Storage backend accepts the following backend options:

.. option:: no-ssl

   Disable encrypted (https) connections and use plain HTTP instead.

.. option:: ssl-ca-path=<path>

   Instead of using the system's default certificate store, validate
   the server certificate against the specified CA
   certificates. :var:`<path>` may be either a file containing
   multiple certificates, or a directory containing one certificate
   per file.

.. option:: tcp-timeout

   Specifies the timeout used for TCP connections. If no data can be
   exchanged with the remote server for longer than this period, the
   TCP connection is closed and re-established.

.. _`Google Storage Manager`: https://sandbox.google.com/storage/

Amazon S3
=========

.. program:: s3_backend

`Amazon S3 <http://aws.amazon.com/s3>`_ is the online storage service
offered by `Amazon Web Services (AWS) <http://aws.amazon.com/>`_. To
use the S3 backend, you first need to sign up for an AWS account. The
account is free, you will pay only for the amount of storage and
traffic that you actually use. After that, you need to create a bucket
that will hold the S3QL file system, e.g. using the `AWS Management
Console <https://console.aws.amazon.com/s3/home>`_. For best
performance, it is recommend to create the bucket in the
geographically closest storage region, but not the US Standard region
(see :ref:`durability` for the reason).

The storage URL for accessing S3 buckets in S3QL has the form ::

    s3://<bucketname>/<prefix>

Here *bucketname* is the name of the bucket, and *prefix* can be an
arbitrary prefix that will be prepended to all object names used by
S3QL. This allows you to store several S3QL file systems in the same
S3 bucket.

Note that the backend login and password for accessing S3 are not the
user id and password that you use to log into the Amazon Webpage, but
the *AWS access key id* and *AWS secret access key* shown under `My
Account/Access Identifiers
<https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key>`_.

The Amazon S3 backend accepts the following backend options:

.. option:: no-ssl

   Disable encrypted (https) connections and use plain HTTP instead.

.. option:: ssl-ca-path=<path>

   Instead of using the system's default certificate store, validate
   the server certificate against the specified CA
   certificates. :var:`<path>` may be either a file containing
   multiple certificates, or a directory containing one certificat                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       