Module chargés dans le Broker
[2019-09-17 16:49:59] INFO : [ broker-master ] I correctly loaded the modules: [sla,Simple-log,WebUI,Livestatus,Graphite-Perfdata] |
Initialisation du module SLA - CHAPITRE [ INITIALISATION ]
Sur le Broker
Création du module
[2021-04-13 15:03:08] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] ============= Starting module initialisation ============== [2021-04-13 15:07:59] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating new migration process named : broker [ - Module: sla - migration] [2021-04-13 15:07:59] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating new archive process named : broker [ - Module: sla - archive] [2021-04-13 15:08:02] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating a queue manager process for the module. [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Reading module configuration [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating 1 workers |
Paramètre de connexion à la base mongo
[2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Parameter load for database connection [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - database ------------------------- :[shinken] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - replica_set ---------------------- :[] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - use_ssh_tunnel ------------------- :[False] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - ssh_user ------------------------- :[root] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - ssh_tunnel_timeout --------------- :[2] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - use_ssh_retry_failure ------------ :[1] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - auto_reconnect_max_try ----------- :[3] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - auto_reconnect_sleep_between_try - :[3] [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 8.69ms [2021-04-13 15:08:04] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 5.01ms |
Fin de l'initialisation du module
[2021-04-13 15:11:49] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Load from collection 28 elements info in cache done in 0.84ms [2021-04-13 15:11:49] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Found first element monitoring at 17-06-2020 10:42:52 [2021-04-13 15:11:49] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] ============= Module initialized in 234.97ms ============== |
Démarrage du processus de migration - chapitre "[ MIGRATION "
[2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Sub process ready to work. [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Migration process broker [ - Module: sla-writer - migration] initialization [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Parameter load for migrating [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - daily_clean_batch_size ----------------- :[10000] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - daily_clean_pause_time ----------------- :[2.0] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - broker_module_sla_migration_batch_size - :[1000] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - broker_module_sla_migration_pause_time - :[0.1] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - nb_stored_days ------------------------- :[INFINITE] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - time_when_delete_old_SLA --------------- :[03:02] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 6.62ms [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 4.18ms [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Load from collection 28 elements info in cache done in 0.92ms [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Migration process broker [ - Module: sla-writer - migration] initialized |
Erreurs - La connexion au serveur Mongo n'est pas établie
Avec Tunnel SSH
[2021-02-10 10:39:45] ERROR : [ WebUI ] [ sla ] [ INITIALISATION ] Initialisation Module: Error : [ WebUI -sla ] [ SSH TUNNEL ] [ MONGODB ] - mongo connection failure : localhost:43577 ==(ssh tunnel)==> 192.168.1.87:22 ==(mongodb)==> 192.168.1.87:27017. |
Sans Tunnel SSH
[YYYY-MM-DD HH:MM:SS] ERROR : [ WebUI ] [ sla ] [ INITIALISATION ] Initialisation Module: Error : [ broker-master -sla ] [ MONGODB ] - mongo connection failure to 192.168.1.87:27017 |
Démarrage du processus d'archive - CHAPITRE "[ ARCHIVE "
[2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Sub process ready to work. [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Starting initialization of archive process [broker [ - Module: sla-writer - archive]] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Reading configuration for sla archive writing [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - time_before_shinken_inactive ----------------------------------:[30] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - minimal_time_before_an_element_become_missing_data ------------:[0] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - minimal_time_before_an_element_become_missing_data_at_startup -:[0] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Parameter load for sla writing [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - size_chunk_to_archive --- :[10000] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - time_between_two_chunks - :[0.1] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - keep_raw_sla_day -------- :[7] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 16.50ms [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 6.95ms [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Load from collection 28 elements info in cache done in 0.79ms [2021-04-13 15:40:56] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Archive process broker [ - Module: sla-writer - archive] initialized in 51.06ms |
Démarrage des workers - CHAPITRE "[ WORKER"
[2021-04-13 15:11:50] INFO : [ broker-master ] [ sla ] [ WORKER MANAGER ] Starting worker 0 [2021-04-13 15:11:50] INFO : [ broker-master ] [ sla ] [ WORKER MANAGER ] The worker 0 is now started as pid:29585 |
[2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Sub process ready to work. [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Staring new worker. [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 5.85ms [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 15.31ms [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Load previous state of 28 elements done in 466.86ms. [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Parameter load for build raw sla [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - time_before_shinken_inactive - :[30] [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - store_output ----------------- :[True] [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - store_long_output ------------ :[True] [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - list_of_stored_output_status - :[set([])] [2021-04-13 15:40:57] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] New worker start in 468.01ms. |
Fonctionnement du module - chapitre [ ARCHIVING ]
Cas de 2 modules qui archive le même jour en base
[2019-09-18 00:08:45] ERROR : [ broker-master ] [ sla ] [ ARCHIVE (pid:1434) ] [ ARCHIVING ] Fail to archive collection 150_2020 (for date 29-05-2020) because it was lock by [broker-int-13 sla2 archive (pid:23460)] and i am [broker-int-13 sla2 archive (pid:23058)] (an other process try to archive this day) |
Dans le cas où 2 modules SLA écrivent dans la même base, les 2 modules vont tenter de faire l'archive du jour en même temps. C'est pourquoi nous avons mis un lock afin d'éviter les doublons dans les entrées SLA.
Si vous avez ce log, vérifiez votre configuration si vous n’avez pas 2 modules SLA ou 2 brokers qui écrivent dans la même base Mongo.
Fonctionnement du module - chapitre [ RUNNING ]
Ecriture des états futurs ( les états qu'on reporte sur les jours d'après )
Lors de l'écriture des états futurs, on a le log suivant :
[2021-04-19 10:36:14] INFO : [ broker-bmar-dev3 ] [ sla ] [ WORKER: 0 (pid:10277) ] [ RUNNING ] Save future states in [2.88ms]. Found [4] entries in base in [1.42ms]. Saving [1] new entries in [1.34ms]. Update [0] entries in [0.04ms] |
Une fois par jour, le module nettoie aussi les états futurs à supprimer
[2021-04-19 10:36:14] INFO : [ broker-bmar-dev3 ] [ sla ] [ WORKER: 0 (pid:10277) ] [ RUNNING ] The clean of future states found [1] entries to delete. There was deleted in [2.28ms (1.28ms for found entries, 1.03ms for delete entries)]. |
Log de migration des données SLA
A chaque démarrage du module, le module fait une recherche des données à migrer :
[2020-07-22 15:11:39] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Need to migrate 931144 archives |
Puis si il y a des données à migrer on log la progression :
[2020-07-22 15:11:39] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 1000/931144 archives [2020-07-22 15:11:39] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 2000/931144 archives [2020-07-22 15:11:39] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 3000/931144 archives [2020-07-22 15:11:40] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 4000/931144 archives |
Puis enfin :
Si il y a eu une migration :
[2020-07-22 15:18:37] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrate 931144 archive done in:7m 0.14s. |
Sinon:
[2020-07-22 09:58:47] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:10878) ] No archive need migration. Check done in:1.09ms. |