| Scroll Ignore | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
|
Module chargés dans le Broker
| Code Block |
|---|
[2019YYYY-09MM-17DD 16HH:49MM:59SS] INFO : [ broker-master ] I correctly loaded the modules: [sla,Simple-log,WebUI,Livestatus,Graphite-Perfdata] |
Initialisation du module SLA - CHAPITRE [ INITIALISATION ]
Sur le Broker
Création du module
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:03MM:08SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] ============= Starting module initialisation ============== [2021YYYY-04MM-13DD 15HH:07MM:59SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating new migration process named : broker [ - Module: sla - migration] [2021YYYY-04MM-13DD 15HH:07MM:59SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating new archive process named : broker [ - Module: sla - archive] [2021YYYY-04MM-13DD 15HH:08MM:02SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating a queue manager process for the module. [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Reading module configuration [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Creating 1 workers |
Paramètre de connexion à la base mongo
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Parameter load for database connection [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - database ------------------------- :[shinken] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - replica_set ---------------------- :[] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - use_ssh_tunnel ------------------- :[False] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - ssh_user ------------------------- :[root] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - ssh_tunnel_timeout --------------- :[2] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - use_ssh_retry_failure ------------ :[1] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - auto_reconnect_max_try ----------- :[3] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] - auto_reconnect_sleep_between_try - :[3] [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 8.69ms [2021YYYY-04MM-13DD 15HH:08MM:04SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 5.01ms |
Fin de l'initialisation du module
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:11MM:49SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Load from collection 28 elements info in cache done in 0.84ms [2021YYYY-04MM-13DD 15HH:11MM:49SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] Found first element monitoring at 17-06-2020 10:42:52 [2021YYYY-04MM-13DD 15HH:11MM:49SS] INFO : [ broker-master ] [ sla ] [ INITIALISATION ] ============= Module initialized in 234.97ms ============== |
Démarrage du processus de migration - chapitre "[ MIGRATION "
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Sub process ready to work. [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Migration process broker [ - Module: sla-writer - migration] initialization [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Parameter load for migrating [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - daily_clean_batch_size ----------------- :[10000] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - daily_clean_pause_time ----------------- :[2.0] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - broker_module_sla_migration_batch_size - :[1000] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - broker_module_sla_migration_pause_time - :[0.1] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - nb_stored_days ------------------------- :[INFINITE] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] - time_when_delete_old_SLA --------------- :[03:02] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 6.62ms [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 4.18ms [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Load from collection 28 elements info in cache done in 0.92ms [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:31197) ] [ INITIALISATION ] Migration process broker [ - Module: sla-writer - migration] initialized |
Erreurs - La connexion au serveur Mongo n'est pas établie
Avec Tunnel SSH
| Code Block |
|---|
[2021YYYY-02MM-10DD 10HH:39MM:45SS] ERROR : [ WebUI ] [ sla ] [ INITIALISATION ] Initialisation Module: Error : [ WebUI -sla ] [ SSH TUNNEL ] [ MONGODB ] - mongo connection failure : localhost:43577 ==(ssh tunnel)==> 192.168.1.87:22 ==(mongodb)==> 192.168.1.87:27017. |
Sans Tunnel SSH
| Code Block |
|---|
[YYYY-MM-DD HH:MM:SS] ERROR : [ WebUI ] [ sla ] [ INITIALISATION ] Initialisation Module: Error : [ broker-master -sla ] [ MONGODB ] - mongo connection failure to 192.168.1.87:27017 |
Démarrage du processus d'archive - CHAPITRE "[ ARCHIVE "
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Sub process ready to work. [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Starting initialization of archive process [broker [ - Module: sla-writer - archive]] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Reading configuration for sla archive writing [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - time_before_shinken_inactive ----------------------------------:[30] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - minimal_time_before_an_element_become_missing_data ------------:[0] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - minimal_time_before_an_element_become_missing_data_at_startup -:[0] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Parameter load for sla writing [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - size_chunk_to_archive --- :[10000] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - time_between_two_chunks - :[0.1] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] - keep_raw_sla_day -------- :[7] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 16.50ms [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 6.95ms [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Load from collection 28 elements info in cache done in 0.79ms [2021YYYY-04MM-13DD 15HH:40MM:56SS] INFO : [ broker-master ] [ sla ] [ ARCHIVE (pid:31201) ] [ INITIALISATION ] Archive process broker [ - Module: sla-writer - archive] initialized in 51.06ms |
Démarrage des workers - CHAPITRE "[ WORKER"
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:11MM:50SS] INFO : [ broker-master ] [ sla ] [ WORKER MANAGER ] Starting worker 0 [2021YYYY-04MM-13DD 15HH:11MM:50SS] INFO : [ broker-master ] [ sla ] [ WORKER MANAGER ] The worker 0 is now started as pid:29585 |
| Code Block |
|---|
[2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Sub process ready to work. [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Staring new worker. [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Creating connection to sla database [shinken] [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Mongo connection established in 5.85ms [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] [ MONGO ] Ensure mongo index done in 15.31ms [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Load previous state of 28 elements done in 466.86ms. [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] Parameter load for build raw sla [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - time_before_shinken_inactive - :[30] [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - store_output ----------------- :[True] [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - store_long_output ------------ :[True] [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] - list_of_stored_output_status - :[set([])] [2021YYYY-04MM-13DD 15HH:40MM:57SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:31252) ] [ INITIALISATION ] New worker start in 468.01ms. |
Fonctionnement du module - chapitre [ ARCHIVING ]
Cas de 2 modules qui archive le même jour en base
| Code Block |
|---|
[2019YYYY-09MM-18DD 00HH:08MM:45SS] ERROR : [ broker-master ] [ sla ] [ ARCHIVE (pid:1434) ] [ ARCHIVING ] Fail to archive collection 150_2020 (for date 29-05-2020) because it was lock by [broker-int-13 sla2 archive (pid:23460)] and i am [broker-int-13 sla2 archive (pid:23058)] (an other process try to archive this day) |
Dans le cas où 2 modules SLA écrivent dans la même base, les 2 modules vont tenter de faire l'archive du jour en même temps. C'est pourquoi nous avons mis un lock afin d'éviter les doublons dans les entrées SLA.
Si vous avez ce log, vérifiez votre configuration si vous n’avez pas 2 modules SLA ou 2 brokers qui écrivent dans la même base Mongo.
Fonctionnement du module - chapitre [ RUNNING ]
Ecriture des états futurs ( les états qu'on reporte sur les jours d'après )
Lors de l'écriture des états futurs, on a le log suivant :
| Code Block |
|---|
[2021YYYY-04MM-19DD 10HH:36MM:14SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:10277) ] [ RUNNING ] Save future states in [2.88ms]. Found [4] entries in base in [1.42ms]. Saving [1] new entries in [1.34ms]. Update [0] entries in [0.04ms] |
Une fois par jour, le module nettoie aussi les états futurs à supprimer
| Code Block |
|---|
[2021YYYY-04MM-19DD 10HH:36MM:14SS] INFO : [ broker-master ] [ sla ] [ WORKER: 0 (pid:10277) ] [ RUNNING ] The clean of future states found [1] entries to delete. There was deleted in [2.28ms (1.28ms for found entries, 1.03ms for delete entries)]. |
Log de migration des données SLA
A chaque démarrage du module, le module fait une recherche des données à migrer :
| Code Block |
|---|
[2020YYYY-07MM-22DD 15HH:11MM:39SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Need to migrate 931144 archives |
Puis si il y a des données à migrer on log la progression :
| Code Block |
|---|
[2020YYYY-07MM-22DD 15HH:11MM:39SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 1000/931144 archives [2020YYYY-07MM-22DD 15HH:11MM:39SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 2000/931144 archives [2020YYYY-07MM-22DD 15HH:11MM:39SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 3000/931144 archives [2020YYYY-07MM-22DD 15HH:11MM:40SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrating in progress 4000/931144 archives |
Puis enfin :
Si il y a eu une migration :
| Code Block |
|---|
[2020YYYY-07MM-22DD 15HH:18MM:37SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:32233) ] Migrate 931144 archive done in:7m 0.14s. |
Sinon:
| Code Block |
|---|
[2020YYYY-07MM-22DD 09HH:58MM:47SS] INFO : [ broker-master ] [ sla ] [ MIGRATION (pid:10878) ] No archive need migration. Check done in:1.09ms. |