The slides and a transcript for this session are available online. We will display a QR encoded, shortened URL for the tarball at the end, so don't worry about writing down the URLs now.
Before we begin, a quick rundown on what we'll cover:
We'll start out with a quick primer on Monasca, what it is, and what it does.
Next we'll take you on a guided tour of Monasca's architecture diagram. We'll intermingle this with information on which repository holds each component and some development hints.
After that we will show you various ways to build a development environment and how to test your code.
Finally we will show you how we organize the development and where you can find the topics to help us on.
We do assume you already are familiar with the OpenStack development process and tools. If you are not, please read the Code & Documentation Contributor Guide.
So, what is Monasca?
In a nutshell, it's Monitoring and Logging as a Service.
It's highly scalable: Monasca is built using micro-services architecture where each component can be clustered. In case of a bottleneck more machines can be added to resolve the problems.
The services communicate using Apache Kafka which is ridiculously fast and adds fault tolerance to the system. Even if some of the components temporarily cannot consume the messages they won't get lost, but will be held in the message queue until they can be consumed.
Last but not least, Monasca is multi-tenant. It stores metrics per project. The users authenticate with Keystone and get the measurements only for their project. It allows e.g. exposing application metrics running on customer instances.
Let's take a look at what Monasca can do for you:
We can gather all sorts of metrics and attach arbitrary dimensions to them. The most common one is of course the host name, but you could attach all sorts of other dimensions to identify a particular metric. For instance a VM's resource ID if you gather a particular metric for all VMs running on a host.
We have a real-time alerting engine which evaluates the measurements consumed from the message queue before they get persisted. This ensures very short latency for notifications.
The notification engine is pluggable and supports several notification channels, like email, webhook, Slack, Jira, Hipchat, etc.
Last but not least we also have an aggregation engine for turning a flood of individual metrics into a synthetic compound metric. Anybody who has ever worked with Grafana's aggregation functions knows why that is useful...
We have various places where we keep the documentation for Monasca:
The most up-to-date is usually the stuff on docs.openstack.org since that is generated directly from the monasca-api and monasca-log-api repositories.
The main contributors to Monasca are Fujitsu, StackHPC, SUSE and Universidade Federal de Campina Grande at the moment.
We will now give you a guided tour of Monasca's architecture mixed in with a quick run-down on each component.
We will include extra information relevant to developers where applicable.
The Monasca metrics API is the central piece of Monasca.
It receives metrics from agents, makes these metrics available to clients and is used for defining alarms and thresholds.
Alarm definitions, thresholds and various other things are stored in a configuration database.
This is a plain old SQL database, usually MariaDB since OpenStack has pretty much standardized on MariaDB.
Various Monasca components use it for things such as alarm state or user defined runtime state such as alarm definitions.
You will find the Metrics API in the monasca-api repository.
The Metrics API receives metrics data from metrics agents and exposes the metrics stored in the time series database (more on that later).
It is also the interface for modifying the configuration database.
Which is to say, if you want to define alarms or set alarm thresholds, the Metrics API is what you talk to. This happens either directly through a Monasca client or indirectly through the Monasca UI which uses the Monasca client in its backend.
For developers there are a couple of interesting things in the monasca-api repository:
1) It is the central documentation repository for Monasca. The things you find on https://docs.openstack.org/monasca-api/latest/ are generated from the monasca-api repository.
2) The repository contains full API reference documentation with all important concepts explained.
3) It contains the data model for the configuration database that is used by various Monasca services. Whenever you add or remove tables or columns you will need to edit the modules in the monasca_api/common/repositories directory. You should also create corresponding Alembic migration scripts.
There are two implementations of monasca-api in that repository: one in Python and one in Java. Nowadays everybody uses the Python implementation and we deprecated the Java one. So you won't need to target the Java implementation with your contributions.
monasca-agent)Another crucial component is the Monasca agent.
You will find the Monasca agent in the monasca-agent repository.
The agent is the boots-on-the-ground component of Monasca: it runs on the systems being monitored by Monasca, where it collects metrics and forwards them to monasca-api.
Custom plugins for metrics specific to your deployment can also be easily integrated: there are magic directories where an operator can simply drop them. Since you are prospective Monasca developers we strongly suggest you contribute your plugins upstream, though.
Let's take a look at how plugins work (this is the same for both official and "magic directory" plugins). There are two types of plugins:
1) Check plugins: You will find these in the monasca_agent/collector/checks_d directory. You create one of these if you want to add a new type of check to monasca-agent.
2) Detection plugins: You will find these in the monasca_setup/detection/plugins directory. These plugins are used by monasca-setup which automatically detects things to be monitored and configures monasca-agent accordingly.
If you add a new check plugin to the official monasca-agent source tree, please also try to add a detection plugin. Otherwise people will always have to configure your check manually.
Please refer to the documentation in the monasca-agent repository for information on plugin development. We have detailed instructions for both official and "magic directory" plugins in there.
You will find the Monasca client in the python-monascaclient repository.
Just like with other OpenStack components, python-monascaclient contains the client library for talking to the Monasca API, along with a command line client.
In Monasca's case the command client is mainly useful for listing/examining metrics and handling alarms. The client library is used by all Moasca components that talk to the Monasca API.
It should go without saying that you will need to modify python-monascaclient as well if you make changes to the Metrics API.
monasca-ui)You will find the Horizon plugin for Monasca in the monasca-ui repository.
The Monasca Horizon plugin is fairly bare bones. Its chief purpose is providing a friendly interface for dealing with alarms and notifications:
it allows the user to define these through a dialog based UI with sanity checks and tooltips. It also visualizes alarms and notifications.
Its secondary purpose is providing links to Grafana and Kibana dashboards which are used to visualize metrics and logs, respectively.
For the next component we do not have a repository of our own, but since it's part of the official architecture diagram we will mention it for completeness' sake: much like other OpenStack serivces Monasca uses a message queue to allow its components to communicate amongst each other.
While it is theoretically possible to use RabbitMQ for this (like other OpenStack services), most Monasca installations out there use Kafka for performance reasons.
The message queue is primarily used to get metrics and log entries to the respective persister services. Beyond that, monasca-threshold listens in on metrics (and triggers monasca-notification via the message queue if any alarm thresholds are exceeded).
monasca-thresh)The other side of notification is taken care of by the Monasca Threshold engine. You will find this component in the monasca-thresh repository.
This component listens in on metrics as they rush by on the message queue and checks whether they exceed any alarm thresholds. If they do, monasca-thresh publishes an alarm to the message queue. That message is then consumed by monasca-notification. At the same time the alarm will be recorded in the Monasca database so it can be visualized in the Monasca UI.
From a developer's point of view, monasca-thresh is a bit of an odd duck: it's one of the services that is entirely implemented in Java with no Python implementation existing at the moment.
monasca-thresh uses some shared code from the Java part of the monasca-common library, so you may have to modify monasca-common as well if you contribute code to monasca-thresh.
Last but not least, monasca-thresh does not do the heavy lifting all by itself. Instead, it uses Apache Storm to process metrics.
The Notification Engine is part of Monasca once more. You will find it in the monasca-notification repository.
As mentioned, it listens to messages from monasca-threshold on the the message queue.
Once these arrive it will send notifications through various channels. Currently we support E-Mail, IRC, Slack, and Jira among others.
Notifications are plugin based. If you have a communications channel you want to create a plugin for, just add a plugin to the monasca_notification/plugins/ directory.
To use your new plugin you need to register it in the Notification Engine's configuration file, notification.yaml.
monasca-transform)Another component in the metrics processing pipeline is monasca-transform. You will find it in the monasca-transform repository.
This component takes more of an active role: it consumes metrics the Metrics API publishes on the message queue and publishes the synthetic metrics that result from performing various transformations on them.
The most common transformation is aggregating individual metrics into bigger-picture metrics (e.g. summing up the m1.xlarge VMs on each compute node and publishing that sum as a synthetic metric for each compute node).
monasca-persister)monasca-persister is the key element in the metrics processing pipeline. You will find it in the monasca-persister repository.
When all is said and done, its job is very simple:
It consumes the metrics monasca-api and monasca-transformer publish to the message queue and stores them in the time series database.
There are two monasca-persister implementations: one in Python and one in Java. We are planning on deprecating the Java implementation, but for now contributions should still target both.
As with other components, monasca-persister uses shared code from monasca-common, so you may have to submit changes to monasca-common as well if you contribute to monasca-persister.
Last but not least, we have the heart of Monasca metrics: the time series database.
Again, that's a third-party application such as Cassandra, InfluxDB or Vertica.
This database is where we store our metrics data.
Since sometimes new time series databases pop up or become popular, we may have to add support for them to Monasca. To do that we will usually have to modify
1) monasca-common, which contains some database code shared by all components.
2) monasca-persister, which writes to the time series database.
3) monasaca-api, which reads from the time series database.
Metrics are but one side of the equation. Monasca also takes care of logs, and we've got another architecture diagram for that. This one is a bit shorter though.
Just like with metrics we have an API service as the center piece.
You will find its sources in the monasca-log-api repository.
The Log API receives log messages from agents and forwards them for further processing.
The monasca-log-api repository holds the sources for all logging specific documentation.
If you modify monasca-log-api, you may have to use monasca-common as well.
On the input side we have agents again, which send the log messages to the API.
Unlike the metric agent, these are not part of Monasca, though. You can use either logstash, beaver or fluentd with Monasca output plugin.
We currently recommend logstash because the Monasca plugin for Beaver has not been merged upstream and Beaver appears to be unmaintained.
Again, we use the message queue to pass log messages to the processing pipeline in the background. That processing pipeline consists of Logstash configurations for various purposes.
The first of these is the log metrics service. It processes the log messages in the message queue and counts the occurrences of log levels, such as INFO, WARN or DEBUG. These statistics are then published to Kafka as metrics for consumption on the metrics side of Monasca.
The next one is monasca-log-transformer. This service parses log messages and turns various metadata such as the time stamp or log level into proper JSON attributes. The parsed log message is then republished on the Kafka message queue.
The final step in the log processing pipeline is the log persister which writes the transformed log messages to Elasticsearch.
Not much to be said about elasticsearch. This is where we store our logs. We index them by tenant to support multi tenancy. Since Elasticsearch does not support Keystone authentication we also need some sort of Keystone enabled frontend.
That's what we've got Kibana with monasca-kibana-plugin for. That one does support Keystone authentication (it uses the Keystone auth token header).
For the future we are planning on allowing the retrieval of logs through monasca-log-api as well. That way any third party application that supports Keystone authentication will then be able to retrieve log data.
To give people a feel for how Monasca works from a user's perspective we have prepared an interactive Jupyter notebook.
While this is not aimed at developers, you might still want to give it a try if you haven't played with an active Monasca installation, yet.
It demonstrates the most important Monasca functionalities using hands-on examples and exercises. The material was prepared for the hands-on sessions on previous Summits.
Now that we've seen both the metrics and logging architecture, we'll take a look at the various ways to get a development environment going.
If you already have your own way to deploy Devstack, these are the local.conf settings you will need to deploy it with Monasca. It's the same as for other OpenStack projects: just enable the monasca-api plugin.
One setting of interest may be MONASCA_PERSISTER_IMPLEMENTATION_LANG and its analogues for other components. If you work on any components with dual implementations in Java and Python you may need these kinds of settings.
If you do not have a Devstack setup, yet, you can build one with a simple vagrant up in the monasca-api repository's devstack/ directory.
Last but not least you can deploy containerized Monasca with Docker Compose:
# git clone https://github.com/monasca/monasca-docker
# cd monasca-docker
# docker-compose upWorking with Docker containers may significantly speed up your development workflow.
Now for running tests...
Unit tests are pretty straightforward, same as for other OpenStack projects. Just cd to the repository you modified and run tox. If you don't want to wait too long, specify only the tests you are interested in.
Our Tempest tests are also standard OpenStack fare:
Before you deploy Devstack, enable the Monasca tempest plugin in your local.conf:
enable_plugin monasca-tempest-plugin https://git.openstack.org/openstack/monasca-tempest-pluginOnce you've got your Devstack instance up and running you can run them in the usual manner:
# cd /opt/stack/tempest
# tempest run -r monasca_tempest_tests.tests.api
# tempest run -r monasca_tempest_tests.tests.log_apimonasca-dockerIf you prefer Docker over DevStack, you can run tempest tests with monasca-docker as well.
To that end you will need to add this little snippet to docker-compose.yaml
tempest-tests:
image: monasca/tempest-tests:latest
environment:
KEYSTONE_SERVER: "keystone"
STAY_ALIVE_ON_FAILURE: "true"
MONASCA_WAIT_FOR_API: "true"...and launch the tempest tests with the following command:
# docker-compose up -d tempest-testsSo much for the technical part. We'd like to conclude with a few notes on the Monasca community and our contribution process.
We don't know how many of you already have your own reasons to contribute to Monasca, but here's a few from our side:
First of all you might be interested in contributing plugins to monasca-agent and/or monasca-notification if you are working on something that needs to be monitored or a communication tool that we do not have a notification plugin for.
For especially this part of Monasca is very modular and customizable. Hurdles for entry are very low and the benefit will be mutual: you can claim Monasca support and we get a fresh plugin.
And no matter the contribution you make, we are a small and friendly community. You can get things done fairly quickly in Monasca.
First of all, we do have a contributor guide with far more information than we could fit into this presentation, so be sure to check it out.
One notable thing about the Monasca community is that we use Storyboard for bugs and feature requests. So please don't look for Monasca bugs on Launchpad.
You will find our specifications in the openstack/monasca-specs repository.
If you think something is missing from Monasca, take a look at this repository
One thing you will also find in the specifications repository is our list of priorities for any given release, such as this one for Rocky.
For an overview of what we are working on, check out our storyboard page. You will often find discussion and background on the stories we maintain there.
As for things you can help with...the usual:
Like most Openstack projects we've got a packed review pipeline, so people who take some of the pressure off us are always welcome.
Then there's always bugs to fix, of course. Bug reports are welcome, too. There are a few monasca-agent plugins that don't see a lot of use but may nonetheless be broken. If you happen to use one of these and discover bugs please do report them.
Every OpenStack release brings one or two community goals, such as the implementation of Oslo's policy-in-code for Queens. Implementing them is usually not a glamorous job but it needs to be done, and often it involves a lot of refactoring. So we greatly appreciate help with that.
One sore point with Monasca is installation. You have seen how many moving parts Monasca has - setting it up is a major undertaking. So if you have Ansible, Chef, Docker, Helm or Kubernetes experience, by all means lets talk. More ways to set up Monasca are always a good thing.
Documentation is another a sore point. We've got a few very good pieces of documentation, such as the plugin development guide for monasca-agent, but we've also got some parts of Monasca where we have little to no documentation. If you happen to discover how one of these undocumented subsystems works the hard way, please consider writing down what you learned. We'd be happy to integrate that into our official documentation!