The purpose of these tests is to manually check functionality that is not tested using tempest tests, and to double-check the correctness and validity of a devstack in general.
Run
sudo systemctl status devstack@vitrage-graph.service
sudo systemctl status devstack@vitrage-persistor.service
sudo systemctl status devstack@vitrage-notifier.service
sudo systemctl status devstack@vitrage-api.service
Expected result
Make sure that the status is active
.
Check the logs for WARNING/ERROR/Exception/traceback
Run
sudo service devstack@vitrage-graph restart
sudo service devstack@vitrage-notifier restart
sudo service devstack@vitrage-persistor restart
Expected result
Make sure the restart is quick and does not reach a timeout
Run
vitrage service list
Expected result
+----------------------------------+------------+-------------+---------------------------+
| Name | Process Id | Hostname | Created At |
+----------------------------------+------------+-------------+---------------------------+
| ApiWorker worker(0) | 1084 | my-devstack | 2019-03-13T14:31:46+00:00 |
| EvaluatorWorker worker(0) | 1082 | my-devstack | 2019-03-13T14:31:46+00:00 |
| MachineLearningService worker(0) | 5956 | my-devstack | 2019-03-13T10:30:54+00:00 |
| PersistorService worker(0) | 22536 | my-devstack | 2019-03-13T14:14:15+00:00 |
| SnmpParsingService worker(0) | 6170 | my-devstack | 2019-03-13T10:30:56+00:00 |
| VitrageNotifierService worker(0) | 22746 | my-devstack | 2019-03-13T14:14:27+00:00 |
| vitrageuWSGI worker 1 | 2847 | my-devstack | 2019-03-13T10:30:47+00:00 |
| vitrageuWSGI worker 2 | 2848 | my-devstack | 2019-03-13T10:30:47+00:00 |
+----------------------------------+------------+-------------+---------------------------+
Run
ps -aux | grep vitrage-graph
Expected result
vitrage-graph: master process
vitrage-graph: EvaluatorWorker
vitrage-graph: ApiWorker
There may be more than one EvaluatorWorker processes, according to the definition in vitrage.conf (the default is one worker).
Run
vitrage healthcheck
Expected result
+----------+---------+
| Field | Value |
+----------+---------+
| detailed | False |
| reasons | [u'OK'] |
+----------+---------+
Most of the functionality is covered in tempest tests, but we have no automatic tests for the CLI itself.
Run
vitrage topology show
Expected result
Should display a graph with Vitrage entities and relationships.
What to execute | Expected results |
---|---|
vitrage template validate | Error, –path is required |
vitrage template validate -path ./v1_template.yaml | Validation failed - Unknown template type |
vitrage template validate –path ./v1_template.yaml –type standard | Template validation is OK |
vitrage template validate –path ./v1_template.yaml –type definition | Validation failed, definition template can not contain
scenarios block |
vitrage template validate –path ./v2_high_cpu_load.yaml | Template validation is OK |
vitrage template validate –path . | Validates all templates in the path. Some succeed and some fail. |
vitrage template validate –path ./v2_definition_template.yaml | Template validation is OK |
vitrage template validate –path v2_equivalence.yaml | No validation |
vitrage template validate –path v3_high_mem_consumption.yaml | Template validation is OK |
What to execute | Expected results |
---|---|
vitrage template list | An empty list |
vitrage template add | Error, –path is required |
vitrage template add –path ./v1_template.yaml | Template added with status ERROR: Unknown template type |
vitrage template add –path ./v1_template.yaml –type kuku | –type: invalid choice: kuku |
vitrage template add –path ./v1_template.yaml –type standard | Template added with status LOADING |
vitrage template add –path ./v1_template.yaml –type standard | ERROR: Duplicate template name |
vitrage template list | One template with status ACTIVE |
vitrage template delete | No output |
vitrage template list | An empty list |
vitrage template add –path ./v2_high_cpu_load.yaml | Template added with status LOADING |
vitrage template add –path ./v2_definition_template.yaml | Template added with status LOADING |
vitrage template add –path ./v2_with_include.yaml | Template added with status LOADING |
vitrage template add –path ./v2_with_invalid_include.yaml | ERROR: Trying to include a template that does not exist |
vitrage template add –path v2_equivalence.yaml | Template added with status LOADING |
vitrage template add –path v3_high_mem_consumption.yaml | Template validation is OK |
vitrage template list | Five templates with status ACTIVE and one in ERROR |
vitrage template show <uuid> | Shows the template json representation |
What to execute | Expected results |
---|---|
vitrage template validate –path v3_with_params.yaml | Failed to resolve parameter - template_name |
vitrage template validate –path v3_with_params.yaml –params template_name=template1 | Template validation is OK |
vitrage template validate –path v3_with_params.yaml –params alarm_name=alarm1 | Failed to resolve parameter - template_name |
vitrage template validate –path v3_with_params.yaml –params template_name=template1 alarm_name=alarm1 | Template validation is OK |
vitrage template add –path v3_with_params.yaml –params template_name=template1 alarm_name=alarm1 | Template added with status LOADING |
vitrage template add –path v2_with_params.yaml –params template_name=t1 alarm_name=a1 alarm_type=zabbix | Template added with status LOADING |
vitrage template show <uuid> | Shows the template json representation. All parameters
are resolved and the parameters section is removed. |
What to execute | Expected results |
---|---|
create an instance in Nova | An instance is created |
vitrage alarm list | An empty list |
vitrage event post –type=”High CPU load” –details=’ {“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ | Make sure to use hostname that contains an instance
No output |
vitrage alarm list | Shows ‘High CPU load’ on the host and also an alarm on the instance. |
vitrage alarm show <uuid> | Shows the alarm details. Verify for both alarms. |
vitrage rca show <uuid> | Shows the alarm RCA graph. Verify for both alarms. |
vitrage alarm count | Shows one WARNING and one CRITICAL alarm |
What to execute | Expected results |
---|---|
vitrage resource list | Shows a list of resources from different datasources. Does not show alarms |
vitrage resource list –filter ….. TBD | Shows a list of resources that match the given filter. |
vitrage resource show <uuid> | Shows a the defails of the selected resource. |
vitrage resource count | Shows how many resources there are of every type. |
TBD
Note: The resources and alarms are visible only to the tenant that created them.
admin
tenant.What to execute | Expected results |
---|---|
Create an instance/volume/network/stack | A new entity immediately appears in Vitrage entity graph and is connected to the right neighbors. |
Delete these resources | The entities are immediately removed from the graph |
What to execute | Expected results |
---|---|
Copy switch_and_nic.yaml under /etc/vitrage/static_datasources | The switche and nic should appear in the graph within 30 seconds |
What to execute | Expected results |
---|---|
aodh alarm create –name ‘test_1’ –state alarm –event-type ‘my.event’ –type event –query ‘traits.resource_id=string::my-devstack’ | An alarm is created in Aodh with state alarm .
Make sure to use as traits.resource_id the name of
your devstack. |
vitrage alarm list | The Aodh alarm appears and is connected to the devstack |
aodh alarm create –name ‘Gnocchi alarm 1’ –type gnocchi_resources_threshold –resource-type instance –resource-id f9335fc1-f3bf-4915-bed2-2c7350628ac9 –metric cpu_util –threshold 0.001 –granularity 300 –state alarm –aggregation-method mean –comparison-operator gt | An alarm is created in Aodh with state alarm .
Make sure to assign –resource-id with a valid instance
uuid. |
vitrage alarm list | Two Aodh alarms, connected to different resources |
aodh alarm delete <alarm UUID> | |
vitrage alarm list | One Aodh alarm remains |
What to execute | Expected results |
---|---|
vitrage template add –path ./host_down.yaml | Template added with status LOADING |
nova service-list | The state of nova-compute is up |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ | Make sure to use hostname of your devstack.
No output. |
vitrage alarm list | A compute.host.down alarm, connected to the right host |
nova service-list | The state of nova-compute is down |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”up”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ | Make sure to use hostname of your devstack.
No output. |
nova service-list | The state of nova-compute is up |
Copy test_web_server.py under /opt/stack Run: sudo pip install web.py
What to execute | Expected results |
---|---|
vitrage webhook list | Empty list |
python /opt/stack/test_web_server.py 8001 | server started (in a different console window) |
python /opt/stack/test_web_server.py 8002 | server started (in a different console window) |
vitrage webhook add –url http://localhost:8001/alarm | Outputs the webhook details |
vitrage webhook add –url http://localhost:8002/alarm –regex ‘{“vitrage_type”: “doctor”}’ | Outputs the webhook details |
vitrage webhook list | A list with the details of both webhooks |
vitrage webhook show <webhook uuid> | Shows the details of the webhook |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “compute-0-0”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ | Both webhooks print the details of compute.host.down alarm to the console. The webhook on port 8001 prints also the details of the deduced alarms to the console. |
vitrage webhook delete <webhook uuid> | Deletes a webhook |
vitrage webhook list | Shows only one webhook |
vitrage event post –type=”compute.host.down” –details= ‘{“hostname”: “compute-0-0”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”up”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ | The deleted webhook does not print anything. The other webhook prints the cleared alarm(s). |
What to execute | Expected results |
---|---|
mistral workflow-create ./workflow1.yaml | The workflow is created |
vitrage template add –path ./v3_execute_mistral.yaml | Template added with status LOADING |
vitrage event post –type=”alarm_for_mistral” –details= ‘{“hostname”: “my-devstack”,”source”:”sample_monitor”, “cause”:”link-down”,”severity”:”critical”,”status”:”down”, “monitor_id”:”monitor-1”,”monitor_event_id”:”123”}’ | Make sure to use hostname of your devstack.
No output. |
mistral execution-list | workflow1 is in the list. |
mistral execution-get <uuid of the latest execution> | Shows details about workflow1 execution. |
mistral execution-get-input <uuid of the latest execution> | “farewell”: “my-devstack” |
TBD
TBD
The UI tests are included in this document, so we’ll have a complete set of manual sanity tests in one place.
What to execute | Expected results |
---|---|
Go to compute->instances menu | The alarm banner should be on the top right corner with the correct number of alarms |
Click on the alarm banner | You should be redirected to Vitrage alarms view |
What to execute | Expected results |
---|---|
Raise an alarm of type doctor (examples above) |
The alarm appears in the Active Alarms list |
Filter By alarm type doctor |
Only doctor alarms remain |
Clear the doctor filter |
All alarms appear |
Sort by name | Alarms are sorted |
Click the RCA button for the High CPU load alarm |
An RCA graph displays the alarms on the host and on the instance(s). Make sure all properties are ok. |
Switch to Alarm History tab |
Several alarms should appear with different statuses.
The alarms that are currently active should not have
an Ended value. |
Click the RCA button for one of the alarms | An RCA graph displays the alarm(s) as inactive. |
Change the Ended filter and click Fetch Alarms |
The list of alarms is different |
Filter By resource type nova.host |
Only alarms on the host are displayed |
Go back to Active Alarms |
The list of active alarms is displayed |
What to execute | Expected results |
---|---|
Go to Topology View |
The sunburst shows the compute hierarchy in different colors. |
Switch tenants | The number of instances changes accordingly |
Drill down to the host and instances | The sunburst changes. On the left there may be alarms |
Click the RCA button on one of the alarms | The RCA view is opened |
What to execute | Expected results |
---|---|
Open the Entity Graph , zoom in and out |
The graph changes accordingly |
Click the Pin button, drag 1-2 entities, and refresh |
The graph is not changed |
Click the Unpin button |
The graph is changed |
Double-click two entites to pin them and drag them to different places. Then refresh the graph. | The pinned entities stay in the same location. The rest of the graph is changed. |
Click several entities, one at the time | The properties of the selected entity appear on the left panel |
Write a text in the search box | All entities with the selected text are highlighted |
Switch to a different tenant and see how the graph changes | There are less entities (all instances are gone) |
Filter by a specific Heat stack, modify the details level | The graph changes accordingly |
What to execute | Expected results |
---|---|
Open the Template view under Project->Vitrage ,
check the list of templates |
Few templates appear with Template validation is OK
Details.
There are no Add and Delete buttons. |
Write a name in the filter | Only templates with this name appear |
Clear the filter | All templates appear |
Click the Show icon |
The template content is displayed |
Expand some of the entities, relationships and scenarios | The details are displayed |
Switch to Yaml View |
A json representation is displayed |
Switch back to Simple View |
The simple view is displayed |
What to execute | Expected results |
---|---|
Open the Template view under Admin->Vitrage and do
the same checks as for tenant |
|
Delete all existing templates | Templates are deleted, the list is empty |
Click Add template and add v1_template.yaml. In the
Type drop-down, select definition |
ERROR: A template definition can not contain includes or scenarios blocks |
Click Add template and add v1_template.yaml. In the
Type drop-down, select standard |
The template is added with status ACTIVE |
Click Add template and add v2_definition_template.yaml |
The template is added with status ACTIVE |
Click Add template and add v2_equivalence.yaml |
The template is added with status ACTIVE and
and details No Validation |
Click Show icon for templates of types standard (v1,
v2 and v3), equivalence and definition |
All templates are displayed correctly |
Click Add template and add v2_wrong.yaml |
ERROR: Action definition must contain action_target field. The template is not added. |
What to execute | Expected results |
---|---|
Click Add template and add v2_with_params.yaml |
Error: Failed to resolve parameter - alarm_type The template is not added. |
Click Add template and add v2_with_params.yaml. Add
parameter alarm_type. |
Error: Failed to resolve parameter - alarm_name The template is not added. |
Click Add template and add v2_with_params.yaml. Add
parameters alarm_type, alarm_name, template_name and
new_state. Add and remove another parameter before
clicking Done. |
The template is added with status ACTIVE |
Click the Show icon |
There is no parameters section. All parameters are
resolved with the given values. |
Click Add template and add v2_with_params.yaml. Add
parameters alarm_type and alarm_name only |
The template is added with status ACTIVE |
Click the Show icon |
There is no parameters section. alarm_type and
alarm_name parameters are resolved with the given
values. new_state has default value of ERROR. |
Perform similar tests for v3_with_params.yaml | |
Click Add template and add v3_with_default_params.yaml |
The template is added with status ACTIVE |
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.