Make it durable

Make it durable

This section introduces object storage.

OpenStack Object Storage (code-named swift) is open-source software that enables you to create redundant, scalable data storage by using clusters of standardized servers to store petabytes of accessible data. It is a long-term storage system for large amounts of static data that you can retrieve, leverage, and update. Unlike more traditional storage systems that you access through a file system, you access Object Storage through an API.

The Object Storage API is organized around objects and containers.

Similar to the UNIX programming model, an object, such as a document or an image, is a “bag of bytes” that contains data. You use containers to group objects. You can place many objects inside a container, and your account can have many containers.

If you think about how you traditionally make what you store durable, you quickly conclude that keeping multiple copies of your objects on separate systems is a good way strategy. However, keeping track of those multiple copies is difficult, and building that into an app requires complicated logic.

OpenStack Object Storage automatically replicates each object at least twice before returning ‘write success’ to your API call. A good strategy is to keep three copies of objects, by default, at all times, replicating them across the system in case of hardware failure, maintenance, network outage, or another kind of breakage. This strategy is very convenient for app creation. You can just dump objects into object storage and not worry about the additional work that it takes to keep them safe.

Use Object Storage to store fractals

The Fractals app currently uses the local file system on the instance to store the images that it generates. For a number of reasons, this approach is not scalable or durable.

Because the local file system is ephemeral storage, the fractal images are lost along with the instance when the instance is terminated. Block-based storage, which the Block Storage section discusses, avoids that problem, but like local file systems, it requires administration to ensure that it does not fill up, and immediate attention if disks fail.

The Object Storage service manages many of the tasks normally managed by the application owner. The Object Storage service provides a scalable and durable API that you can use for the fractals app, eliminating the need to be aware of the low level details of how objects are stored and replicated, and how to grow the storage pool. Object Storage handles replication for you. It stores multiple copies of each object. You can use the Object Storage API to return an object, on demand.

First, learn how to connect to the Object Storage endpoint:

from __future__ import print_function
import hashlib
from shade import *

conn = openstack_cloud(cloud='myfavoriteopenstack')

To begin to store objects, we must first make a container. Call yours fractals:

container_name = 'fractals'
container = conn.create_container(container_name)

You should see output such as:

Munch({u'content-length': u'0', u'x-container-object-count': u'0',
u'accept-ranges': u'bytes', u'x-container-bytes-used': u'0',
u'x-timestamp': u'1463950178.11674', u'x-trans-id':
u'txc6262b9c2bc1445b9dfe3-00574277ff', u'date': u'Mon, 23 May 2016
03:24:47 GMT', u'content-type': u'text/plain; charset=utf-8'})

You should now be able to see this container appear in a listing of all containers in your account:


[Munch({u'count': 0, u'bytes': 0, u'name': u'fractals'}),
Munch({u'count': 0, u'bytes': 0, u'name': u'fractals_segments'})]

The next logical step is to upload an object. Find a photo of a goat online, name it goat.jpg, and upload it to your fractals container:

file_path = 'goat.jpg'
object_name = 'an amazing goat'
container = conn.get_container(container_name)
object = conn.create_object(container=container_name, name=object_name, filename=file_path)

List objects in your fractals container to see if the upload was successful. Then, download the file to verify that the md5sum is the same:


[Munch({u'hash': u'd1408b5bf6510426db6e2bafc2f90854', u'last_modified':
u'2016-05-23T03:34:59.353480', u'bytes': 63654, u'name': u'an amazing
goat', u'content_type': u'application/octet-stream'})]
object = conn.get_object(container_name, object_name)

print(hashlib.md5(open('goat.jpg', 'rb').read()).hexdigest())


Finally, clean up by deleting the test object:

conn.delete_object(container_name, object_name)

Munch({u'content-length': u'0', u'x-container-object-count': u'0',
u'accept-ranges': u'bytes', u'x-container-bytes-used': u'0',
u'x-timestamp': u'1463950178.11674', u'x-trans-id':
u'tx46c83fa41030422493110-0057427af3', u'date': u'Mon, 23 May 2016
03:37:23 GMT', u'content-type': u'text/plain; charset=utf-8'})

Now, no more objects are available in the fractals container.



Back up the Fractals from the database on the Object Storage

Back up the Fractals app images, which are currently stored inside the database, on Object Storage.

Place the images in the fractals container:

container_name = 'fractals'

Next, back up all existing fractals from the database to the swift container. A simple loop takes care of that:


Replace IP_API_1 with the IP address of the API instance.

import base64
import cStringIO
import json
import requests

endpoint = 'http://IP_API_1'
params = { 'results_per_page': '-1' }
response = requests.get('%s/v1/fractal' % endpoint, params=params)
data = json.loads(response.text)
for fractal in data['objects']:
    r = requests.get('%s/fractal/%s' % (endpoint, fractal['uuid']), stream=True)
    with open(fractal['uuid'], 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
    conn.create_object(container=container_name, name=fractal['uuid'])

for object in conn.list_objects(container_name):


The example code uses the awesome Requests library. Before you try to run the previous script, make sure that it is installed on your system.

Configure the Fractals app to use Object Storage


Currently, you cannot directly store generated images in OpenStack Object Storage. Please revisit this section again in the future.

Extra features

Delete containers

To delete a container, you must first remove all objects from the container. Otherwise, the delete operation fails:

for object in conn.list_objects(container_name):
    conn.delete_object(container_name, object['name'])


It is not possible to restore deleted objects. Be careful.

Add metadata to objects

You can complete advanced tasks such as uploading an object with metadata, as shown in following example. For more information, see the documentation for your SDK.

This adds a “foo” key to the metadata that has a value of “bar”.


Swift metadata keys are prepended with “x-object-meta-” so when you get the object with get_object(), in order to get the value of the metadata your key will be “x-object-meta-foo”.

metadata = {'foo': 'bar'}
conn.create_object(container=container_name, name=fractal['uuid'],

Large objects

For efficiency, most Object Storage installations treat large objects, > 5GB, differently than smaller objects.

Shade’s create_object function has a “use_slo” parameter (that defaults to true) which will break your object into smaller objects for upload and rejoin them if needed.

Next steps

You should now be fairly confident working with Object Storage. You can find more information about the Object Storage SDK calls at:

Or, try one of these tutorial steps:

Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.