The OneOps system architecture

The OneOps system architecture

OneOps uses a web frontend written in Ruby on Rails to provide a self-service portal to users, as well as access to REST APIs to automate several tasks. It also has a middleware automation engine and a backend that stores data in several databases based on different needs. These broad components are further broken down into smaller components that connect and communicate with each other, either via databases or a message bus.

OneOps uses several technologies behind the scenes to create an ecosystem of services and applications.

OneOps detailed system architecture

Display

OneOps provides a web-based self-service portal written in Ruby on Rails named Display. Because Display is a OneOps's window to the world and is written in Rails, pretty much all aspects of it are customizable.

Note

If you are trying to install and run Display by itself on a Mac OS X machine, you will have to run the following command, as the dynamic linked postgres library has to be explicitly provided to the pg gem:

pg gem install issue on osx: gem install pg -v 0.17.0 -- --with-pg-config=/Library/PostgreSQL/9.1/bin/pg_config 
export DYLD_LIBRARY_PATH=/library/PostgreSQL/9.1/lib:$DYLD_LIBRARY_PATH

Once Display is installed, you can find settings in the /opt/oneops/config directory. Most of the settings can be set in two ways. You can either set the corresponding environment variable or change the setting file in the config directory. The environment variables always override the configuration set in the file. You can customize everything, starting with the logo, system name (default is OneOps), support e-mail, news URL, help URL, feedback URL, and URLs for terms of service and privacy policy. You can also configure OneOps to authenticate, either against a database or LDAP. OneOps also has support for OmniAuth, which means if you sign up for developer accounts on GitHub, Twitter, LinkedIn, and Google and plug in the keys and secrets to the configuration file, then users can use their user ID and password from these sites instead of creating a brand new one. All of these settings can be found in settings.yml, as shown in the following screenshot:

You can consider changing the following settings in your installation:

You can also change settings such that anyone is allowed to register for OneOps or users can register only if they have an invitation. By default, anyone can register. You can also make users click on a confirmation link which they will receive in their e-mail before they can log in to OneOps. By default, that requirement is not enforced.

You can also choose to authenticate against an active directory instead of a database. You can configure an active directory in ldap.yml. We have the following code:

A typical ldap.yml

The active directory configuration can be provided by the Windows Enterprise team specific to your organization's domain. This way, all or some of your users, depending on your configuration, can log in to your installation without having to create a new account, simply by using their Windows username and password.

Backend databases

OneOps has three main backend data stores, all in Postgres database. During installation, OneOps creates three databases named activitidb, kloopzapp, and kloopzdb, along with the corresponding roles. The activitidb database stores all the information related to current activities happening in OneOps. The kloopszpp database stores all the metadata about the app, including user login names, organizations, groups, user preferences, and so on. Kloopzdb is the main change management database and acts as the CMDB for OneOps.

In this database, you will find all the information about the change items, change attributes, the relationships between them, change management logs, current change item states, and other change-related information. If you are familiar with change management, CMDB, and ITIL, you can browse this database directly to find more information on your change items. OneOps also fires up an instance of Logstash to collect all the logs generated by OneOps itself, as well as all the CI that it deploys, in a centralized place. It uses collectors to channel all the logs to Logstash.

Besides these databases, OneOps has a few other data stores that it uses to store miscellaneous but important data. OneOps uses Elastic Search to store all the events generated by CMS, controller events such as work/action orders, and notifications. This helps OneOps to implement policies, generate costs, and maintain deployment/release history. OneOps also persists the health data about all its CI in a Cassandra database. It can then use these events to support functionality such as autorepair and autoscale.

Backend services

OneOps runs several backend services, each with a unique function. Running in conjunction, these services provide the actual functionality of OneOps. All the services are tied together by a messaging queue in the form of ActiveMQ. All services use either a dedicated or a shared queue to communicate with each other using Apache ActiveMQ.

Adapter

Adapter is a web application that runs under Tomcat and is available under port 8080. This application runs on whichever server Tomcat is installed on  which is usually the same machine on which all of OneOps is installed. Adapter provides a set of RESTful APIs for CRUD (short for create, read, update, and delete) operations on model, change items, deployments, and namespaces. The API provided by adapter is primarily consumed by Display.

Transistor

Transistor also runs as a web application under Tomcat and is available under port 8080. The transistor service is responsible for creating design and deployment plans and comparing what is currently deployed versus what has been designed by the user. The service also ensures that what is defined for deployment conforms to the pack design.

Data acquisition

Data acquisition (DAQ) is responsible for capturing data via collectors, which is then used to graph monitor details in Display. DAQ integrates with Logstash to collect all the logging data. DAQ collects all KPIs via Logstash and inserts them into Cassandra. It also publishes them into ActiveMQ queues for consumption by Display as needed.

Antenna

Antenna is also a web application/API available under Tomcat. It is used internally by the opamp and controller to send notifications about deployments and unhealthy events to configured notification sinks.

Note

A notification sink is nothing but a receiver that will capture a message and may choose to do something with the message. Often, the receiver will just display or log the message. Currently, OneOps comes with hooks for Jabber, HTTP, and Log4j.

Transmitter

Transmitter tracks changes to the change management system and publishes them as events to ActiveMQ. These events can then be picked up by the other systems to be actioned upon.

Note

You can quickly check the status of your transmitter by connecting to http://youroneopsinstance:8080/transmitter. This will show you whether your transmitter is running, your queue backlog, your CI events queue backlog, and the date and time of your transmitter's last run.

Controller

Controller is the main workhorse of OneOps doing the majority of the work once the button for deployment is pushed. It is an Activiti-based workflow engine responsible for processing work orders and action orders. Activiti is an open source business process management software available under Apache License. The Controller also runs as a web application under Tomcat and is available under port 8080.

Collector

Collectors are part of DAQ and collect metrics from managed instances, which they then channel to Logstash. As part of the collector framework, OneOps installs a small binary on all the VMs deployed and managed by OneOps, named Logstash-Forwarder. Logstash-Forwarder, written in Go, has a very small footprint and securely transmits all the logs from the VM using the lumberjack protocol to Collector, via a HAProxy setup. Collectors, in turn, aggregate all the logs and forward them to Logstash, where they get filtered and, ultimately, get stored in Cassandra.

Sensor

Sensor consumes metrics that are collected by collection and analyzes them against any thresholds that have been set against all the change items. If any threshold is reached, then sensor will generate an Ops event that can be actioned upon. Sensor is based on Esper open source technology by EsperTech, which is a complex event processing and event series analysis technology. Sensor uses Espers' API to detect events in the logs and data gathered by collectors.

Opamp

The opamp component process Ops events generated by the sensor, specifically, events generated by unhealthy CIs. The opamp process can consume such events and generate actions to compensate for them to make the CIs healthy again. Such actions can be autorepair, autoreplace, scale-up, scale-down, and so on. Opamp reacts to such events according to the rules set in each assembly.

Inductor

The inductor runs as a service in the background and consumes all the work orders and action orders generated by OneOps, actions them according to their priority, and posts the results back to the controller. The inductor is responsible for executing all the steps mentioned in an assembly. An inductor is configured to listen to a cloud-specific queue through the command-line tool available via the inductor gem. The same tool also posts the results from various clouds back to the controller queue. Inductor currently supports Chef cookbooks to run various commands, but is easily extensible to support other Infrastructure as Code tools, such as Ansible, Puppet, and so on.

WorkOrder

WorkOrder is RFC or Request For Change that will result in a physical change in a CI. This typically results in an addition, updating or deletion of component in OneOps. WorkOrders are sent by controllers to message queues. Once sent, an inductor then consumes a WorkOrder and executes the associated recipe using chef-solo using the data available in the WorkOrder. Once executed, the controller then posts the response from the WorkOrder back to the queue and updates the CMS.

ActionOrder

ActionOrder is similar to WorkOrder in all aspects, except it does not result in any physical alterations of CI. As such, ActionOrder consists of actions, such as start, stop, status, repair, reboot, snapshot, restore, and so on. You may think that repair might physically alter CI, but all it does is restore CI to its previous state, so it does not count as a physical alteration.

Note

ITILv3 defines a CI or configuration item as any component that needs to be managed to provide an IT service. In the context of OneOps, this includes compute instances, networking infrastructure, software components, and so on.

Useful backend tools

OneOps comes with some very useful tools that allow you to navigate your way through the backend, irrespective of whether you are troubleshooting a problem or learning OneOps.

Cms-admin

Cms-admin is installed as part of OneOps and is very handy, especially if you plan to extend OneOps. Cms-admin is installed as a web app under Tomcat and is available on port 8080 on the machine where Apache Tomcat is installed. If your OneOps is installed on a server named OneOps and the Tomcat installation is on the same machine, then you can access cms-admin at http://oneops:8080/cms-admin/.

Note

If you are using OneOps from an Amazon EC2 install, you will have to edit Security Group and enable the port 8080 to connect to cms-admin. Again, be careful and avoid opening up access to the whole world, as there is no authentication asked for to connect to this port.

Cms-admin, at the time of writing, provides four useful tools. It provides an easy way to browse all the CMS classes provided by OneOps. This goes above and beyond the documentation available on OneOps site and will be of great interest to Java developers who are interested in extending the core OneOps functionality via Java programming. Although it does not follow the Javadoc format, it does come very close to it and gives valuable information on classes, such as name, access level, description, attributes, data types, inherited from, default values, and from and to relationship with other classes.

The second very useful tool provided by cms-admin is the CI list. It's the second link from the top and allows you to browse a lot of things configured in OneOps without actually logging on to the server. We have the following screenshot:

Basic search interface to CMS

This simple tool allows you to browse a lot of backend data from OneOps. For example, if you enter the namespace path as / and press Enter , you will see all the organizations that you configured.

Clicking on a particular organization will show you the details of that particular organization, as shown in the following figure:

Entering a namespace path of /public/oneops/packs will show you all the built-in packs that OneOps supplies, as shown in the following screenshot:

Note

Think of packs as ready-made, prepackaged software that OneOps provides which is ready to install on any server.

Again, clicking on a particular pack name will give you details of that pack, such as the owner, the pack type, description, and category, as shown in the upcoming figure:

Similarly, if you provide /public/oneops/clouds as the namespace path, the cms-admin tool will show you all the currently available clouds under OneOps, as shown in the following screenshot:

And, similar to the previous examples, clicking on any of the clouds will provide descriptive details on that particular cloud, as shown in the following screenshot:

Besides the CI browser, cms-admin provides two more tools. The first is the Activiti monitor. The Activiti monitor shows all the tasks that are either in progress or have been finished by the Activiti Business Processes Manager that runs in the backend of OneOps, as described earlier. It provides details on the task that are in progress such as activity ID and start time. For finished tasks, it shows start time, end time, and duration. The final tool it offers is a report of stuck deployments. The stuck deployment report groups deployments under three categories:

This is a handy way for a system administrator to get a system-wide view of deployments that are not progressing.

OneOps CLI

The OneOps command-line interface allows you to control OneOps from the command line and do everything that you can do from the GUI from the command line. This comes in very handy if you want to script a lot of tasks from OneOps but don't want to make use of the REST interface provided by OneOps. The OneOps CLI does not come installed by default. You will have to download and install it manually. You can clone it from the OneOps git repository and install it, as follows:

$ git clone https://github.com/oneops/cli.git
$ gem build oneops.gemspec
$ gem install oneops

Note

If you want to install the gem universally, run the last two commands as root or with sudo privileges.

Once the gem is installed, it is available as the command oneops. You can then configure it to run as follows:

$ oneops config set site=http://localhost:9090 -g

This is assuming that you installed the oneops gem on the same box as OneOps. Also, if you are running OneOps on AWS, then you should change the port to 3000. You can then log in to the site from the command line. To log in, you will need to obtain the API key. You can obtain the key from the OneOps GUI. Log in to the OneOps GUI and click on the username on the lower left-hand side. Then, click on the authentication tab. You should see the API token on the right-hand side of the screen, as follows:

Once you obtain the API token, you can specify that as your login name with the OneOps CLI. Your password is your regular password. We have the following screenshot:

Use the oneops auth login command to log in. Once logged in, you can set the organization to your default organization. Remember that the CLI will not allow you to do anything that the GUI does not allow you to do. Running OneOps help will provide you with help on the commands that you can run from the OneOps CLI:

Once you are logged in, you can run all kinds of commands to get details on existing components, such as your organization, assemblies, catalog, clouds, and so on. You can even query your assembly to see the constituent pack components and can even transition the assembly from the command-line tool:

Control OneOps from command line

You can even get operations information on an assembly and perform an ops operation on an assembly via the command oneops operations.