View Source

Module chargés dans le broker

[2019-09-17 16:49:59] INFO   : [broker-dev4    ] I correctly loaded the modules: [sla,Simple-log,WebUI,Livestatus,Graphite-Perfdata]

Création du module

[2019-09-17 15:33:19] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] Creating new SLA module name : sla for broker
[2019-09-17 15:33:20] INFO : [WebUI ] [sla ] [INITIALISATION ] Creating new SLA module name : sla for webui

Parametre de connection à la base mongo

[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] Parameter load for database connection
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - database ------------------------- :[shinken]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - replica_set ---------------------- :[]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - use_ssh_tunnel ------------------- :[False]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - ssh_user ------------------------- :[root]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - use_ssh_retry_failure ------------ :[1]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - auto_reconnect_max_try ----------- :[3]
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - auto_reconnect_sleep_between_try - :[3]

Démarrage du process d'archive

[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] Sub process ready to work.
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO] Parameter load for database connection
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - database ------------------------- :[shinken]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - replica_set ---------------------- :[]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - use_ssh_tunnel ------------------- :[False]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - ssh_user ------------------------- :[root]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - use_ssh_retry_failure ------------ :[1]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - auto_reconnect_max_try ----------- :[3]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO]    - auto_reconnect_sleep_between_try - :[3]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] Parameter load for build sla archive
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ]    - time_before_shinken_inactive:[30]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] Parameter load for archiving sla
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ]    - size_chunk_to_archive:[10000]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ]    - time_between_two_chunks:[0.1]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ]    - keep_raw_sla_day:[7]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ]    - Initialisation of the archive process done
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO] Open mongo connection done in 0.002s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO] Open mongo connection done in 0.002s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] [MONGO] Ensure mongo index done in 0.004s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [archive (pid:13227)   ] [INITIALISATION        ] Load 55 elements info in cache done in 0.001s

Archivage en cours

[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ] Archiving the day 17-09-2019
[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ] 55 items to archive in 1 blocks ( max size block : 10000 )
[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ]      -  block 0: archiving 55 elements
[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ] Found first monitoring start time at 10-09-2019 23:51:30
[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ]      -  block 0: insert 55 archive in mongo done in 0.003s
[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ]      -  block 0: archiving block elements done in 0.043s
[2019-09-18 00:08:45] INFO   : [broker-dev4    ] [sla             ] [archive (pid:1434)    ]  55 entries archive for the day 17-09-2019 in 0.157ss nb block 1

Démarrage du process de migration

[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] Sub process ready to work.
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO] Parameter load for database connection
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - database ------------------------- :[shinken]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - replica_set ---------------------- :[]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - use_ssh_tunnel ------------------- :[False]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - ssh_user ------------------------- :[root]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - use_ssh_retry_failure ------------ :[1]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - auto_reconnect_max_try ----------- :[3]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO]    - auto_reconnect_sleep_between_try - :[3]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] Parameter load for migrating
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ]    - daily_clean_batch_size:[10000]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ]    - daily_clean_pause_time:[2]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ]    - nb_stored_days:[INFINITE]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO] Open mongo connection done in 0.001s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO] Open mongo connection done in 0.001s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] [MONGO] Ensure mongo index done in 0.003s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] [INITIALISATION        ] Load 55 elements info in cache done in 0.001s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [migration (pid:13226) ] check raw sla collection format collections:[Date(yday=260, year=2019):True] all is migrate:[True]

Migration check :

Pas de migration :

[2019-09-18 00:08:55] INFO : [broker-dev4 ] [sla ] [migration (pid:1433) ] No archive need migration. Check done in:0.001s.

Démarrage d'un worker

[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] Sub process ready to work.
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] Staring new worker.
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO] Parameter load for database connection
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - database ------------------------- :[shinken]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - replica_set ---------------------- :[]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - use_ssh_tunnel ------------------- :[False]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - ssh_user ------------------------- :[root]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - use_ssh_retry_failure ------------ :[1]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - auto_reconnect_max_try ----------- :[3]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO]    - auto_reconnect_sleep_between_try - :[3]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO] Open mongo connection done in 0.002s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO] Open mongo connection done in 0.002s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] [MONGO] Ensure mongo index done in 0.002s
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] Load previous state of 1 elements done in 0.000s.
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] Parameter load for build raw sla
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ]    - time_before_shinken_inactive:[30]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ]    - store_output:[True]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ]    - store_long_output:[True]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ]    - list_of_stored_output_status:[set([])]
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [INITIALISATION        ] New worker start in 0.007s.
[2019-09-17 16:50:02] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] [sla[0]]: Now running..

Nombre de mise à jour en base par worker par secondes

[2019-09-17 16:50:11] INFO   : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] 0 raw data update in database for the day 17-09-2019

Cas de 2 modules qui archive le même jour en base

[2019-09-18 00:08:45] ERROR  : [broker-dev4    ] [sla             ] [archive (pid:1434)    ] Fail to archive collection 150_2020 (for date 29-05-2020) because it was lock by [broker-int-13 sla2 archive (pid:23460)] and i am [broker-int-13 sla2 archive (pid:23058)] (an other process try to archive this day)

Dans le cas où 2 modules SLA écrivent dans la même base, les 2 modules vont tenter de faire l'archive du jour en même temps. C'est pourquoi nous avons mis un lock afin d'éviter les doublons dans les entrées SLA.

Si vous avez ce log vérifiez votre configuration si vous n’avez pas 2 modules SLA ou 2 brokers qui écrivent dans la même base Mongo.

Ecriture des états futurs (les états qu'on reporte sur les jours d'après)

Lors de l'écriture des états futurs l'on a le log suivant :

[2019-09-18 00:08:45] ERROR  : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] Save future states in [31.00ms]. Found [10] entries in base in [0.00ms]. Saving [4] new entries in [31.00ms]. Update [0] entries in [0.00ms]

Une fois par jour on nettoie aussi les états future à supprimer

[2019-09-18 00:08:45] ERROR  : [broker-dev4    ] [sla             ] [Worker: 0 (pid:13266) ] The clean of future states found [4] entries to delete. There was deleted in [35.00ms (2.00ms for found entries, 33.00ms for delete entries)].

Log de migration des données SLA

A chaque démarrage du module on fait une recherche des données à migrer :

[2020-07-22 15:11:39] INFO   : [broker-master  ] [sla                   ] [migration (pid:32233) ] Need to migrate 931144 archives

Puis si il y a des données à migrer on log la progression :

[2020-07-22 15:11:39] INFO   : [broker-master  ] [sla                   ] [migration (pid:32233) ] Migrating in progress 1000/931144 archives
[2020-07-22 15:11:39] INFO   : [broker-master  ] [sla                   ] [migration (pid:32233) ] Migrating in progress 2000/931144 archives
[2020-07-22 15:11:39] INFO   : [broker-master  ] [sla                   ] [migration (pid:32233) ] Migrating in progress 3000/931144 archives
[2020-07-22 15:11:40] INFO   : [broker-master  ] [sla                   ] [migration (pid:32233) ] Migrating in progress 4000/931144 archives

Puis enfin :

Si il y a eu une migration :

[2020-07-22 15:18:37] INFO   : [broker-master  ] [sla                   ] [migration (pid:32233) ] Migrate 931144 archive done in:7m 0.14s.

Sinon:

[2020-07-22 09:58:47] INFO   : [broker-dev4    ] [sla                   ] [migration (pid:10878) ] No archive need migration. Check done in:1.09ms.

Erreurs

La connexion au serveur Mongo n'est pas établie à l'initialisation du module SLA

Avec Tunnel SSH

[2021-02-10 10:39:45] ERROR : [ WebUI ] [SLA][INITIALISATION] Initialisation Module: Error : [ WebUI -sla ] [ SSH TUNNEL ] [ MONGODB ] - mongo connection failure : localhost:43577 ==(ssh tunnel)==> 192.168.1.87:22 ==(mongodb)==> 192.168.1.87:27017.

Sans Tunnel SSH

[YYYY-MM-DD HH:MM:SS] ERROR : [ WebUI ] [SLA][INITIALISATION] Initialisation Module: Error : [ broker-master  -sla ] [ MONGODB          ]   - mongo connection failure to 192.168.1.87:27017