Seite wählen

NETWAYS Blog

ServiceNow Data Asset Import mit dem Director

Hallo Liebe Leser dieses Blogs,

nach einer weile der Abwesenheit hab ich heute die Freude ihnen etwas näher bringen zu dürfen.
Da viele unserer Kunden inzwischen auch ServiceNow verwenden und diese auch als Assetmanagement / CMDB verwenden.

Kommt doch die Frage auf wie man gegebenenfalls hier die Daten abfragen kann um Sie in den Director zu importieren um daraus dann Hosts/Services zur Überwachung zu formen.Ich versuche in diesem kleinen Blogpost zumindest einen Ansatzpunkt hierzu aufzuzeigen. Viele verwenden in diesem Fall ja ServiceNow als SaaS, aber es gibt auch Kunden welche es als On Prem einsetzen.
Aber fangen wir an 🙂 in diesem Fall ist der Anlaufpunkt hier die ServiceNow API, wer hätte es gedacht.
In diesem Fall ziehe ich mal die folgende API (Link) zur rate:

Hier wird die ‚cmdb-instance‘ in welcher die CMDB Daten landen dokumentiert. Es ist aber zu bedenken, dass die Instanz natürlich abweichen kann daher ist dies bitte nicht direkt 1:1 zu übernehmen.
Wir feuern per curl gegen die ServiceNow API, die folgende Abfrage… in Hoffnung, dass wir valides JSON zurückerhalten mit all unseren Objekten.


curl -k -s -S -i -u 'servicenow_apiuser:servicenow_apiuserpassword' -H 'Accept: application/json' -H 'X-HTTP-Method-Override: GET' -X POST 'https://instance.servicenow.com/api/now/cmdb/instance/cmdb_ci_linux_server'

und wir erhalten als Antwort das folgende JSON:


{
"result": {
...
"attributes": {
"firewall_status": "Intranet",
"os_address_width": "",
"sys_updated_on": "2020-07-08 11:16:51",
"sys_created_by": "glide.maint",
"warranty_expiration": "",
"ram": "2048",
"cpu_name": "",
"cpu_speed": "2800",
"classification": "Production",
"disk_space": "40",
"dns_domain": "",
"assigned": "2020-01-04 07:00:00",
"floppy": "",
...
"sys_class_name": "cmdb_ci_linux_server",
...
"cpu_count": "1",
...
"os_version": "2.6.9-22.0.1.ELsmp",
"serial_number": "",
"attributes": "",
...
"form_factor": "",
"cpu_core_count": "",
"sys_updated_by": "system",
"sys_created_on": "2008-10-26 17:17:28",
...
"name": "PS LinuxApp01",
"default_gateway": "",
"chassis_type": "",
"sys_id": "3a290cc60a0a0bb400000bdb386af1cf",
"po_number": "",
"checked_in": "",
...
"comments": "",
"os": "Linux Red Hat",
"sys_mod_count": "24",
"monitor": "false",
"model_id": {
"display_value": "Iris 5875",
"link": "https://instance.servicenow.com/api/now/table/cmdb_model/5f5fbcc3c0a8010e00f3b27814f3b96b",
"value": "5f5fbcc3c0a8010e00f3b27814f3b96b"
},
"ip_address": "192.168.178.1",
"duplicate_of": "",
"location": {
"display_value": "Somewhere Street, Someplace, State",
},
"category": "Do not migrate to asset",
"fault_count": "0",
"host_name": "",
"lease_id": ""
},
}

Den Output schreiben wir als Datei raus, so dass wir hier als Beipiel eine Datei haben, mit der wir weiterarbeiten können. Als Beispiel nennen wir diese servicenow.json
(Exclaimer) Ich habe das JSON hier als Beispiel mal extrem zusammengekürzt, damit wir thementechnisch nicht zu weit Abdriften.
Anyway, aber damit lässt sich arbeiten.
Damit wir mit diesem JSON arbeiten können, verwenden wir das Tool ‚jq‘, um relevante key/value Paare herauszufiltern.
Ich habe hier mal den folgenden ‚jq‘ Aufruf verwendet:


jq -j '.result.attributes.os,",",.result.attributes.ip_address,",",.result.attributes.classification,",",.result.attributes.sys_class_name,",",.result.attributes.name,",",.result.attributes.location.display_value,","' servicenow.json

Der somit gewonnene Output ist der folgende, bitte beachtet die ‚,‘ Kommata. Die meisten werden schon ahnen worauf ich gleich hinaus will.
Zurück zu dem nun mit ‚jq‘ aufbereiteteb Output, dieser ist der folgende:


Linux Red Hat,192.168.178.1,Production,cmdb_ci_linux_server,PS LinuxApp01,Somewhere Street, Someplace, State,

Mit etwas Bash Scripting kann man das schön automatisieren und hat am Ende dieses Vorgangs eine CSV Datei, die man hervorragend im Director per Fileshipper weiterverarbeiten kann.
Das mit dem CSV ist eine Geschmacksfrage, denn man kann auch direkt JSON per Fileshipper importieren … ich würde aber dazu tendieren, die Datenmenge durch etwas vorfiltern zu verkleinern. So eine Filterung, wie gesehen, kann auch per ‚jq‘ erfolgen, so dass man ein kleineres JSON File erhält.
Wer wissen möchte, wie man mit dem Director einen Fileshipper-Import durchführt, dem sei der Blogpost meines Kollegen Johannes empfohlen.

Blogpost Director Fileshipper Import

Das war schon mein kleines Intermezzo wie ServiceNow-Data-Assets per Fileshipper in den Director zu importiert sind.
Bis zum nächsten Mal

Mit freundlichem Gruß
David

David Okon
David Okon
Senior Systems Engineer

Weltenbummler David hat aus Berlin fast den direkten Weg zu uns nach Nürnberg genommen. Bevor er hier anheuerte, gab es einen kleinen Schlenker nach Irland, England, Frankreich und in die Niederlande. Alles nur, damit er sein Know How als IHK Geprüfter DOSenöffner so sehr vertiefen konnte, dass er vom Apple Consultant den Sprung in unser Professional Services-Team wagen konnte. Er ist stolzer Papa eines Sohnemanns und bei uns mit der Mission unterwegs, unsere Kunden zu glücklichen Menschen zu machen.

Icinga Web 2 – More Goodies for Developers

Tuesday version 2.7 of Icinga Web 2 has been released and introduced some interesting new functionality for module developers. Now I’d like to tell you, my fellow colleague, some more details about this.

 

jQuery v3 – Migration Required

First a friendly warning. We’ve upgraded the jQuery version we ship to v3.4.1. This has previously been v2.1.0 so now with this major upgrade some deprecated methods and interfaces are gone.

Though, don’t worry, you don’t need to hurry to avoid everyone complaining your module is incompatible with v2.7. We also ship jQuery migrate now which ensures that the usage of removed methods/interfaces still works. It also emits console warnings if it detects such a usage. The warnings are not active by default. They only appear when using the non-minimized javascript code. Put _dev in your address bar to instruct Icinga Web 2 to serve the non-minimized version. (e.g. /icingaweb2/dashboard?_dev)

Then start using the front-end as usual. Interact with all widgets you’ve written your own Javascript for and look for console entries starting with JQMIGRATE. Any of these messages will only appear once, repeated usages are not reported. If you’ve got a warning then, consult jQuery migrate’s warnings.md in order to get hints how to solve it.

jQuery migrate will be removed with Icinga Web 2 version 2.8.0. While this is still some time ahead, this (and the note in the upgrade documentation) is probably the only warning.

 

Persistence and Collapsible Containers

While we’re at it, let’s stay with the topic of Javascript. If you don’t already know about the localStorage and sessionStorage, it’s now time to inform yourself. (That’s an entire blogpost if described thoroughly)

There’s now an abstraction layer for this shipped with Icinga Web 2. It hides all the handling of complex datatypes and conflicts with other apps using the storage from you. That’s the object Icinga.Storage which utilizes the localStorage by default but also supports the sessionStorage. Take a look here to see how this is used for Icinga Web 2’s sidebar.

Though, this is only the basic stuff. If you need to store more complex data and want to benefit from a storage’s event processing, take a look at the object Icinga.Storage.StorageAwareMap. This is a proxy for Map and allows to subscribe to change events of particular keys in the map. It also keeps track of a key’s age and removes it automatically if it hasn’t been accessed for 90 days.

Another new addition are collapsible HTML containers. This is provided by a behavior which makes use of the StorageAwareMap, a perfect example use-case.

Making a container collapsible is as easy as possible. Just apply the CSS class collapsible and you’re done. If you’re not satisfied with the default height, apply the data attribute data-visible-height and give it the desired height in pixels. (For table’s and ul’s or ol’s there is also data-visible-rows.) Then, if you fancy a custom control by which users expand or collapse the container you can pass a CSS selector to data-toggle-element which (if a direct descendant of the container) then acts as the toggle.

 

Custom XHR Without Dirty Hacks

Have you ever wanted/tried to process link clicks or form submissions by yourself? Well, I have and it was a nightmare every single time. Most of Icinga Web 2’s processing is fine. But of course there ever is this single behavior or side-effect which keeps getting in the way. This has now come to an end.

Meet data attribute data-no-icinga-ajax which does exactly what it’s name suggests. Applied to an element it causes Icinga Web 2 to ignore click and submit events triggered by the element itself and all descendants.

Couldn’t be more simple, can it?

 

Hooks For Everyone

Previously hooks were only processed for logged-in users with the permission to access the module providing the hook. This for example prevented the audit module to register logins from users without the permission module/audit and also didn’t allow to log failed logins.

When providing a hook it is now possible to have it run always. ($this->provideHook($name, $implementation = null, $alwaysRun = false);)

Another case of hooks not being processed was the issue that, unlike in the web front-end, enabled modules were not loaded automatically on the CLI. Thus also their hooks were not registered. Now this has changed and also on the CLI all enabled modules are automatically loaded. If you’ve previously loaded the modules explicitly this is not required anymore. If you don’t need any other modules and want to avoid the overhead of loading them, you can disable this of course.

If you still don’t have enough of this, there’s also an entirely new hook available: ConfigFormEventsHook This hook enables you to influence every configuration form of Icinga Web 2. Extending a form’s validation or doing additional work once submission succeeded is now on the table.

 

That’s it. I hope these things are as useful to you as they were to us. And remember, we don’t mind any suggestions to further improve the integration of modules. You are the developer, you’ll know best what’s… best.

Johannes Meyer
Johannes Meyer
Lead Developer

Johannes ist seit 2011 bei uns und inzwischen, seit er 2014 die Ausbildung abgeschlossen hat, als Lead Developer für Icinga Web 2, Icinga DB Web sowie alle möglichen anderen Module und Bibliotheken im Web Bereich zuständig. Arbeitet er gerade mal nicht, macht er es sich bei schlechtem Wetter am liebsten zum zocken oder Filme/Serien schauen auf dem Sofa gemütlich. Passt das Wetter, geht's auch mal auf eines seiner Zweiräder. Motorisiert oder nicht.

OSMC Hackathon: Feature complete?

Our fourth OSMC hackathon started fully packed after an overwhelming conference – free form, and our introduction round provided many interesting topics to keep up with.
MQTT, OpenNMS, NSClient++, Elastic/Graylog, Migration from Nagios to Icinga 2, Zabbix, Prometheus, MySQL, ARM packages, Notification issues, Golang frameworks, Ansible, Puppet, Icinga Web 2 modules, Vsphere, Director, Maps, … holy moly, that’s super effective for just one day. Missed it? Join us next year!
Jens Schanz from Müller (Hint: The environment with 17k hosts and 200k services) mentioned a problem with missing notifications in his OSMC talk on Wednesday. So we did a deep dive in the production environment into one-time delayed notification without recovery for their ticket system integration. Turns out, that we’ve found a bug in Icinga, and also a quick workaround for production. Jens also knows how to use the debug console for future troubleshooting. Real #monitoringlove ❤️
Gian Arb from InfluxDB joined for the first hours, so „everyone who has questions, be quick“. Maybe he’ll stay longer next year, since he’s said that OSMC feels like home already. Rihards Olups shared Zabbix insights and focussed on improving MySQL monitoring. Our friends Tarus Balog and Ronny Trommer from OpenNMS looked into pushing sensor values into MQTT and OpenHAB, Tarus had his first success 🙂 Julien open-sourced an AMPQ adapter which forwards alerts from Prometheus.
Nicolai Buchwitz is building Icinga 2 for ARM hardware, and asked me to look into failing tests during the Debian package build. This turned out to be a very deep analysis with memory access violations and specific gcc and libstdc++ versions on this hardware, x86 works just fine. 50% are solved, more investigation needed.
Michael Medin implemented a huge new feature: Native Powershell support in NSClient++.  Cannot wait to see this being released! Dave Kempe started working on translation for the famous Icinga 2 book, together with Thomas Mr. Icinga Book. Or as they say: „Translating Austrian to Australian (English) …“ 😉 Thilo jumped right into Jinja templates for the Icinga Ansible modules, and also offered help on enhancing the graph integration in mail notifications.
Noah and Jean played with the Gin library in Golang, providing an HTTP framework. They also discussed the state logic in Icinga 2 together with Eric and our friends from Syseleven, Max and Maurice. Max also did a deep-dive into new enhancements for Icinga Web 2 with more advanced scheduled downtime options. Together with Eric, he found a bug in the forms too. First time hackathon attendees discovered how to write Icinga Web 2 with the help from Eric.
The Map module for Icinga Web 2 received a docs patch from Jens, while Thomas Mr. Director looked into improvements for the business process module, Director, Cube (PR from Nicolai) and certainly the newly released Vsphere module. Last but not least, Flo was for the rescue and helped Jens to modify his Batman theme for Icinga Web 2. Dave also shared the color blind theme for Icinga Web 2 with Jennifer, and to mention too, Jean helped Jens with a DNS/network problem on Icinga startup.
It is INCREDIBLE what we achieved together as OSMC hacking family. We are never feature complete, always exploring and developing new things. See you next year, mark November 4th–7th, 2019 in your calendar!

MSSQL databases from Icinga Web 2

Disclaimer: This guide is only for RedHat and CentOS 7, using the PHP SCL packages with the official icingaweb2 package.
Often when we help customers implement Icinga 2, Icinga Web 2 and Director, they use custom imports to pull in data to their monitoring config. Whenever data is in need to be pulled from a MSSQL database, this can be challenging. Their are multiple guides around the Internet, even from Microsoft.
This post should clear things how to achieve this with the official Icinga packages and their requirements.

Requirements

Basic Installation Icinga Web 2

First of all icingaweb2 and PHP needs to be installed – usually you should have this already…

# on RedHat
subscription-manager repos --enable rhel-7-server-optional-rpms
subscription-manager repos --enable rhel-server-rhscl-7-rpms
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# on CentOS
yum install centos-release-scl epel-release
yum install httpd icingaweb2
systemctl start httpd.service
systemctl enable httpd.service
echo "date.timezone = Europe/Berlin" > /etc/opt/rh/rh-php71/php.d/timezone.ini
systemctl start rh-php71-php-fpm.service
systemctl enable rh-php71-php-fpm.service

Please also see the official documentation.

MSSQL PDO driver for PHP

Their are multiple ways to install a MSSQL compatible driver, but with icingaweb2 and the ZendFramework below in mind, we need to use pdo_dblib with FreeTDS as driver implementation. I prepared a RPM spec file for you to build and install, it is based on the php-extras packages shipped with Fedora’s EPEL, only updated for PHP 7.1 SCL.
You can setup a small RPM build environment on a RedHat or CentOS machine with the required repositories.

sudo yum install rpm-build rpmdevtools yum-utils gcc gcc-c++ scl-utils scl-utils-build
rpmdev-setuptree
cd ~/rpmbuild/SPEC
wget https://github.com/lazyfrosch/rpm-php-extras/raw/epel7/php-extras.spec
cd ../SOURCES
spectool -gf ../SPECS/php-extras.spec
cd ~/rpmbuild
rpmbuild -bs SPECS/php-extras.spec
sudo yum-builddep SRPMS/rh-php71-php-extras*.src.rpm
rpmbuild --rebuild SRPMS/rh-php71-php-extras*.src.rpm

With the built RPM files under RPMS/ you are good to go to your Icinga machine. Basically you only need to copy rh-php71-php-mssql*.rpm over and install it there.

yum install rh-php71-php-mssql*.rpm
scl enable rh-php71 -- php -m

After restarting php-fpm the driver has been loaded.

systemctl restart rh-php71-php-fpm.service

Patching Icinga Web 2

In Icinga Web 2.5.1 and before there is an error in detecting MSSQL correctly, detection only works in PHP 5.x. A fix has been suggested in PR#3400.
To manually patch this, you only need to fix a single line in /usr/share/php/Icinga/Application/Platform.php

--- /usr/share/php/Icinga/Application/Platform.php.orig 2018-03-27 06:02:59.454240788 -0400
+++ /usr/share/php/Icinga/Application/Platform.php 2018-03-27 00:38:15.967639651 -0400
@@ -351,7 +351,7 @@
 */
 public static function hasMssqlSupport()
 {
- return static::extensionLoaded('mssql') && static::classExists('Zend_Db_Adapter_Pdo_Mssql');
+ return static::extensionLoaded('pdo_dblib') && static::classExists('Zend_Db_Adapter_Pdo_Mssql');
 }
 /**

I hope this helps you getting started, feel free to ask questions in the comments.

NETWAYS offers professional support for Icinga and other Open Source tools, check our Website about Support.
The image used in this article is from johnmartel.blogspot.de, we assume fair use by mentioning the author.

Icinga2 API und BitBar auf MacOs

preview1Wir wollen APIs, warum? Weil sie schnell, einfach zu integrieren und zu bedienen sind. Nun hat Icinga2 eine API und es entstehen ganz viele Möglichkeiten diese zu nutzen. Wir bauen uns Dashboards mit Dashing oder zeigen den Status von Hosts in Foreman an.
Ich bin letztens über ein Tool BitBar gestolpert, dieses Tool kann mit einfachen Skripten die eigene „Mac OS X menu bar“ erweitern. Hierfür braucht es nur die richtige Formatierung der Ausgabe und BitBar generiert ein weiteres Dropdown Menu.
Ich hab mir die Icinga2 API zu nutze gemacht und eine kleine Erweiterung gebaut um mir den Status von Icinga2 in meiner Menubar anzuzeigen.
Im Menu wird dann der globale Status entweder in grün oder rot, abhängig davon ob Hosts „down“ und „unhandled“ sind, angezeigt.
Der Aufruf dafür kann der Adresszeile im Browser entnommen werden.
/icingaweb2/monitoring/list/hosts?host_state=1&sort=host_severity&host_unhandled=1
Wenn wir am Ende dann ein „&format=json“ an die URL hängen, haben wir ein gängiges Format um das Ergebnis in jeglichen Applikationen zu verwenden.
[{"host_icon_image":"","host_icon_image_alt":"","host_name":"web01","host_display_name":"web01","host_state":"1","host_acknowledged":"0","host_output":"DOWN","host_attempt":"1\/3","host_in_downtime":"0","host_is_flapping":"0","host_state_type":"1","host_handled":"0","host_last_state_change":"1474556541","host_notifications_enabled":"1","host_active_checks_enabled":"0","host_passive_checks_enabled":"1"},
Mehr dazu gibts auf Github unter icinga2_api_examples oder natürlich in der Icinga2 Dokumentation.

Thilo Wening
Thilo Wening
Manager Consulting

Thilo hat bei NETWAYS mit der Ausbildung zum Fachinformatiker, Schwerpunkt Systemadministration begonnen und unterstützt nun nach erfolgreich bestandener Prüfung tatkräftig die Kollegen im Consulting. In seiner Freizeit ist er athletisch in der Senkrechten unterwegs und stählt seine Muskeln beim Bouldern. Als richtiger Profi macht er das natürlich am liebsten in der Natur und geht nur noch in Ausnahmefällen in die Kletterhalle.