Make it durable

Make it durable

This section introduces object storage.

OpenStack Object Storage (code-named swift) is open-source software that enables you to create redundant, scalable data storage by using clusters of standardized servers to store petabytes of accessible data. It is a long-term storage system for large amounts of static data that you can retrieve, leverage, and update. Unlike more traditional storage systems that you access through a file system, you access Object Storage through an API.

The Object Storage API is organized around objects and containers.

Similar to the UNIX programming model, an object, such as a document or an image, is a “bag of bytes” that contains data. You use containers to group objects. You can place many objects inside a container, and your account can have many containers.

If you think about how you traditionally make what you store durable, you quickly conclude that keeping multiple copies of your objects on separate systems is a good way strategy. However, keeping track of those multiple copies is difficult, and building that into an app requires complicated logic.

OpenStack Object Storage automatically replicates each object at least twice before returning ‘write success’ to your API call. A good strategy is to keep three copies of objects, by default, at all times, replicating them across the system in case of hardware failure, maintenance, network outage, or another kind of breakage. This strategy is very convenient for app creation. You can just dump objects into object storage and not worry about the additional work that it takes to keep them safe.

Use Object Storage to store fractals

The Fractals app currently uses the local file system on the instance to store the images that it generates. For a number of reasons, this approach is not scalable or durable.

Because the local file system is ephemeral storage, the fractal images are lost along with the instance when the instance is terminated. Block-based storage, which the Block Storage section discusses, avoids that problem, but like local file systems, it requires administration to ensure that it does not fill up, and immediate attention if disks fail.

The Object Storage service manages many of the tasks normally managed by the application owner. The Object Storage service provides a scalable and durable API that you can use for the fractals app, eliminating the need to be aware of the low level details of how objects are stored and replicated, and how to grow the storage pool. Object Storage handles replication for you. It stores multiple copies of each object. You can use the Object Storage API to return an object, on demand.

First, learn how to connect to the Object Storage endpoint:

from __future__ import print_function
from import Provider
from import get_driver

auth_username = 'your_auth_username'
auth_password = 'your_auth_password'
auth_url = 'http://controller:5000'
project_name = 'your_project_name_or_id'
region_name = 'your_region_name'

provider = get_driver(Provider.OPENSTACK_SWIFT)
swift = provider(auth_username,


Libcloud 0.16 and 0.17 are afflicted with a bug that means authentication to a swift endpoint can fail with a Python exception. If you encounter this, you can upgrade your libcloud version, or apply a simple 2-line patch.


Libcloud uses a different connector for Object Storage to all other OpenStack services, so a conn object from previous sections will not work here and we have to create a new one named swift.

To begin to store objects, we must first make a container. Call yours fractals:

container_name = 'fractals'
container = swift.create_container(container_name=container_name)

You should see output such as:

<Container: name=fractals, provider=OpenStack Swift>

You should now be able to see this container appear in a listing of all containers in your account:


You should see output such as:

[<Container: name=fractals, provider=OpenStack Swift>]

The next logical step is to upload an object. Find a photo of a goat online, name it goat.jpg, and upload it to your fractals container:

file_path = 'goat.jpg'
object_name = 'an amazing goat'
container = swift.get_container(container_name=container_name)
object = container.upload_object(file_path=file_path, object_name=object_name)

List objects in your fractals container to see if the upload was successful. Then, download the file to verify that the md5sum is the same:

objects = container.list_objects()

[<Object: name=an amazing goat, size=191874, hash=439884df9c1c15c59d2cf43008180048, provider=OpenStack Swift ...>]
object = swift.get_object(container_name, object_name)

<Object: name=an amazing goat, size=954465, hash=7513986d3aeb22659079d1bf3dc2468b, provider=OpenStack Swift ...>
import hashlib
print(hashlib.md5(open('goat.jpg', 'rb').read()).hexdigest())


Finally, clean up by deleting the test object:



You must pass in objects and not object names to the delete commands.

Now, no more objects are available in the fractals container.

objects = container.list_objects()


Back up the Fractals from the database on the Object Storage

Back up the Fractals app images, which are currently stored inside the database, on Object Storage.

Place the images in the fractals container:

container_name = 'fractals'
container = swift.get_container(container_name)

Next, back up all existing fractals from the database to the swift container. A simple loop takes care of that:


Replace IP_API_1 with the IP address of the API instance.

import json

import requests

endpoint = 'http://IP_API_1'
params = { 'results_per_page': '-1' }
response = requests.get('%s/v1/fractal' % endpoint, params=params)
data = json.loads(response.text)
for fractal in data['objects']:
    response = requests.get('%s/fractal/%s' % (endpoint, fractal['uuid']), stream=True)
    container.upload_object_via_stream(response.iter_content(), object_name=fractal['uuid'])

for object in container.list_objects():

<Object: name=025fd8a0-6abe-4ffa-9686-bcbf853b71dc, size=61597, hash=b7a8a26e3c0ce9f80a1bf4f64792cd0c, provider=OpenStack Swift ...>
<Object: name=26ca9b38-25c8-4f1e-9e6a-a0132a7a2643, size=136298, hash=9f9b4cac16893854dd9e79dc682da0ff, provider=OpenStack Swift ...>
<Object: name=3f68c538-783e-42bc-8384-8396c8b0545d, size=27202, hash=e6ee0cd541578981c294cebc56bc4c35, provider=OpenStack Swift ...>


The example code uses the awesome Requests library. Before you try to run the previous script, make sure that it is installed on your system.

Configure the Fractals app to use Object Storage


Currently, you cannot directly store generated images in OpenStack Object Storage. Please revisit this section again in the future.

Extra features

Delete containers

To delete a container, you must first remove all objects from the container. Otherwise, the delete operation fails:

for object in container.list_objects():


It is not possible to restore deleted objects. Be careful.

Add metadata to objects

You can complete advanced tasks such as uploading an object with metadata, as shown in following example. For more information, see the documentation for your SDK.

This option also uses a bit stream to upload the file, iterating bit by bit over the file and passing those bits to Object Storage as they come. Compared to loading the entire file in memory and then sending it, this method is more efficient, especially for larger files.

file_path = 'goat.jpg'
object_name = 'backup_goat.jpg'
extra = {'meta_data': {'description': 'a funny goat', 'created': '2015-06-02'}}
with open('goat.jpg', 'rb') as iterator:
    object = swift.upload_object_via_stream(iterator=iterator,

Large objects

For efficiency, most Object Storage installations treat large objects, > 5GB, differently than smaller objects.

If you work with large objects, use the ex_multipart_upload_object call instead of the simpler upload_object call. The call splits the large object into chunks and creates a manifest so that the chunks can be recombined on download. Change the chunk_size parameter, in bytes, to a value that your cloud can accept.

swift.ex_multipart_upload_object(file_path, container, object_name,

Next steps

You should now be fairly confident working with Object Storage. You can find more information about the Object Storage SDK calls at:

Or, try one of these tutorial steps:

Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.