Seite wählen

Guest Post by Alan Robertson: How The Assimilation Project Got Started

von | Jun 29, 2015 | Partner, Events, OSMC

Back in about 2010, I was working on a highly custom supercomputer research project with about 2.2 million cores called Cyclops64. It was a joint project between the University of Delaware, IBM and the NSA. It was interesting in a lot of respects, but what its most unusual feature was its networking. Each system was directly connected to 6 other systems by high-speed interconnects forming a cube, and only a few systems on one of the cube faces were connected to a “normal” ethernet network. The cube was 24x24x24 and each system had 160 cores (total: 2211840 cores).
As I thought about it, I realized that the way conventional monitoring systems like Nagios or Icinga work would be absolutely horrible in this environment. Imagine that the monitoring system was in one corner of the cube. Then each “Are You OK?” message would have to through 24 systems up, then 24 systems to the right, then 24 systems forward. The return message would have to follow the reverse return path – meaning that each round trip would touch 144 systems. Of course, the network in the vicinity of the central system would be totally destroyed by this traffic if it happened at any reasonable rate.
After thinking about it, I realized that it would be much better if each system monitored its six neighbors instead – delegating these “Are You OK?” queries to the machines being monitored. This totally changed the complexity of the task, and the network load – no network hot spots, and the central system had nothing to do most of the time – an amazing difference! I also realized that more conventional systems have these same problems of network congestion and ever growing centralized workload.
So, I adapted this idea to “normal” systems, and the Assimilation Project was born – providing greater scalability than any predecessors, while being incredibly simple. To keep the same topology-awareness that the original idea had, I realized that I needed to discover network connectivity. Once I’d done that, I realized that I could easily discover what services the system had which would eliminate manual configuration. Further, I could discover dependencies, which are essential to tracing problems to their root causes. Then I realized that this was all really a huge graph, so I adopted the Neo4j graph database. Lots of other valuable capabilities quickly became obvious results of having comprehensive and scalable discovery.
magic+unicorns-1200About this time, I realized that discovery was the real value and doing things like monitoring, and security compliance, network management and many other incredibly valuable applications naturally fell out from being able to easily know basically everything about everything and putting it into a graph-based configuration management database (CMDB).
So, it goes without saying that I’m excited to return to the OSMC in Nürnberg and talk about all the exciting new things we’re doing in the Assimilation Project. After all, the last time I was there, I had a great time and my talk got some pretty cool tweets.
*The Cyclops64 has special monitoring hardware making this unnecessary.

The Author Alan Robertson

Alan Robertson is an open source project leader and speaker on security, availability, discovery and monitoring. He founded Linux-HA (Pacemaker) and the Assimilation Project which maintains a scalable configuration management database driving monitoring and security.

0 Kommentare


  1. Why Create The Assimilation Suite? - Assimilation Systems Limited - […] This blog post is a reposting of material I wrote for Netway’s blog to promote my 2015 OSMC […]

Einen Kommentar abschicken

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert

Mehr Beiträge zum Thema Partner | Events | OSMC

OSMC 2023 | Save your Workshop Ticket!

Imagine a day filled with expert guidance, hands-on learning, and the opportunity to connect with like-minded individuals who share your passion. Our upcoming monitoring workshops are designed to do just that and more. Join us for deep insights, hands-on learning, and...

stackconf 2023 | Recap Day 2

After a fun evening event filled with drinks, socializing, and even some roulette, hopefully everyone got some rest. Once that first cup of coffee kicked in, the attendees were ready to dive into the second day's talks at stackconf 2023.   Let the Talks get...

stackconf 2023 | Excitement is in the Air!

Are you ready to dive headfirst into the world of cutting-edge cloud native infrastructure solutions? The wait is finally over as the highly anticipated stackconf 2023 is set to begin tomorrow! Hello Berlin! Yesterday, Katja and Lukas took their packed car and drove...