Discussion:
KVM Problem by deploying VPC
(too old to reply)
Andrija Panic
2018-05-21 11:17:18 UTC
Permalink
NetworkManager ? I though it was advised to not run it...
Ben,
Can you put the KVM agent in debug mode and post the logs?
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
- Si
________________________________
Sent: Friday, May 18, 2018 2:20 AM
To: Cloudstack Mailinglist
Subject: KVM Problem by deploying VPC
Hi Together,
im currently testing the Configuration of KVM Hosts in our Test
environment.
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:12:08 kvm-test01-sb systemd: Started Network Manager Script
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up' [vnet0]: new
request (4 scripts)
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up' [vnet0]: start
running ordered scripts...
6251: warning : qemuDomainObjTaint:5378 : Domain id=2 name='r-31-VM'
uuid=ff22b439-e0d0-44d1-a3cc-8dd23afb82eb is tainted: high-privileges
service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.
machine1.service'
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine and
Container Registration Service...
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully activated
service 'org.freedesktop.machine1'
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine and
Container Registration Service.
May 18 09:12:08 kvm-test01-sb systemd-machined: New machine qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Unit iscsi.service cannot be
reloaded because it is inactive.
May 18 09:12:08 kvm-test01-sb kvm: 1 guest now active
May 18 09:12:12 kvm-test01-sb kernel: kvm [48292]: vcpu0 disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44 kvm-test01-sb kernel: kvm [48292]: vcpu0 disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46 kvm-test01-sb sh: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
May 18 09:12:46 kvm-test01-sb sh: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Asking libvirt to refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6454] manager: (enp8s0f0.233): new VLAN device
(/org/freedesktop/NetworkManager/Devices/15)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6470] device (enp8s0f0.233): carrier: link connected
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 1(enp8s0f0.233)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 1(enp8s0f0.233)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device enp8s0f0.233 entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 2(vnet1)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 2(vnet1)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device vnet1 entered promiscuous mode
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6648] manager: (vnet1): new Tun device (/org/freedesktop/
NetworkManager/Devices/16)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6662] device (vnet1): state change: unmanaged -> unavailable
(reason 'connection-assumed', sys-iface-state: 'external')
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6674] device (vnet1): state change: unavailable -> disconnected
(reason 'none', sys-iface-state: 'external')
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0) entered
disabled state
May 18 09:13:10 kvm-test01-sb kernel: device vnet0 left promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0) entered
disabled state
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5339] device (vnet0): state change: activated -> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (cloud0): bridge port vnet0 was detached
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (vnet0): released from master device cloud0
service name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
May 18 09:13:10 kvm-test01-sb systemd: Starting Network Manager Script
Dispatcher Service...
May 18 09:13:10 kvm-test01-sb kernel: device vnet1 left promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: brenp8s0f0-233: port 2(vnet1)
entered disabled state
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Successfully activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:13:10 kvm-test01-sb systemd: Started Network Manager Script
Dispatcher Service.
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down' [vnet0]: new
request (4 scripts)
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down' [vnet0]: start
running ordered scripts...
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): state change: disconnected -> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): released from master device brenp8s0f0-233
1519: error : qemuMonitorIO:704 : internal error: End of file from qemu
monitor
May 18 09:13:10 kvm-test01-sb kvm: 0 guests now active
May 18 09:13:10 kvm-test01-sb systemd-machined: Machine qemu-2-r-31-VM
terminated.
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching uuid 'ff22b439-e0d0-44d1-a3cc-8dd23afb82eb'
(r-31-VM)
May 18 09:13:11 kvm-test01-sb kernel: device enp8s0f0.233 left promiscuous
mode
May 18 09:13:11 kvm-test01-sb kernel: brenp8s0f0-233: port 1(enp8s0f0.233)
entered disabled state
May 18 09:13:11 kvm-test01-sb NetworkManager[925]: <info>
[1526627591.2137] device (enp8s0f0.233): released from master device
brenp8s0f0-233
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) No existing libvirtd connection
found. Opening a new one
May 18 09:13:11 kvm-test01-sb sh: libvirt: LXC Driver error : Domain not
found: No domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) Can not find a connection for
Instance r-31-VM. Assuming the default connection.
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
2018-05-18 09:13:11,284 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Failed to start instance VM[DomainRouter|r-31-VM]
com.cloud.utils.exception.ExecutionException: Unable to start
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb due to error in finalizeStart,
not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:4691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.VmWorkJobDispatcher.runJob(
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:558)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-18 09:13:11,289 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Cleaning up resources for the vm VM[DomainRouter|r-31-VM]
in Starting state
2018-05-18 09:13:11,291 DEBUG [c.c.a.t.Request] (Work-Job-Executor-30:ctx-67604a44
Sending { Cmd , MgmtId: 109952567336, via: 10(kvm-test01-sb), Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-
31-VM","executeInSequence":false,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
Processing: { Ans: , MgmtId: 109952567336, via: 10, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request] (Work-Job-Executor-30:ctx-67604a44
Received: { Ans: , MgmtId: 109952567336, via: 10(kvm-test01-sb), Ver: v1,
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG [c.c.n.g.ControlNetworkGuru]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Released nic: NicProfile[65-31-null-null-null
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Successfully released network resources for the vm
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Successfully cleanued up resources for the vm
VM[DomainRouter|r-31-VM] in Starting state
Deployment of Isolatet Networks and redundant VPCs, works wothout any
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal (libvirt-3.9.0-14.el7_5.2.x86_64,
qemu-kvm-1.5.3-156.el7.x86_64, kernel-3.10.0-693.el7.x86_64)
Someone got an i idea ?
Kind regards
Ben
--
Andrija Panić
Benjamin Naber
2018-05-23 11:08:16 UTC
Permalink
Hi Andrija,

first of all thanks for your reply. I have now testet the setup on a Ubuntu Xenial. Same issu with Default VPC Offering. Redundant VPC offering works also without any problems. Same as on CentOS.

See delow the debug log (censored public Ips):

2018-05-23 12:58:05,161 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:0806b407) Proccess agent ready command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:0806b407) Ready command is processed: agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:0806b407) Seq 15-5506494969390563335: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0806b407) Request:Seq 15-5506494969390563336: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.ReadyCommand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0806b407) Processing command: com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0806b407) Proccess agent ready command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0806b407) Ready command is processed: agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0806b407) Seq 15-5506494969390563336: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110, [{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq 15-5506494969390563337: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2","uuid":"6129c76d-67e5-4454-9bc9-4e9a5a152610","id":3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ece6c19d4b89db41de3","hvm":false,"displayText":"SystemVM Template (KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"ROOT-37","size":349945344,"volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}] }
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:ef8b353e) Processing command: org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool 2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool 2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool 2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool 2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Creating volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template 6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool 2258aa76-7813-354d-b274-961fb337e716 (RBD) with size 349945344
2018-05-23 12:58:43,225 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Trying to perform a RBD clone (layering) since we are operating in the same storage pool
2018-05-23 12:58:43,270 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully connected to Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) The source image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD format 2. We will perform a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Checking if RBD snapshot rbd/6129c76d-67e5-4454-9bc9-***@cloudstack-base-snap exists prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Found 1 snapshots on RBD image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) RBD snapshot rbd/6129c76d-67e5-4454-9bc9-***@cloudstack-base-snap already exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully cloned 6129c76d-67e5-4454-9bc9-***@cloudstack-base-snap to fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:ef8b353e) Seq 15-5506494969390563337: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}] }
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Request:Seq 15-5506494969390563338: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" vpccidr=10.59.0.0/21 domain=test.internal dns1=8.8.8.8 privategateway=None template=domP name=r-37-VM eth0ip=169.254.2.247 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56 port=8080","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBcCscv7A","params":{},"uuid":"87a45b1c-bf35-431e-82a0-c0c961feab05","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"RAW","provisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHost":"ceph-mon","managed":"false","storagePort":"6789","volumeSize":"349945344"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}],"guestOsDetails":{}},"hostIp":"10.253.250.230","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.2.247","port":3922,"interval":6,"retries":100,"name":"r-37-VM","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.ip":"169.254.2.247","router.name":"r-37-VM"},"wait":0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId":1,"networkRateMbps":200,"defaultNic":true,"pxeDisable":true,"nicUuid":"29bd4c33-47dd-4357-aa3d-8c5ae56169b5","uuid":"787cf778-230f-4e17-a648-f63171bdc81c","ip":"82.119.177.188","netmask":"255.255.255.240","gateway":"82.119.177.177","mac":"1e:00:87:00:00:0d","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://233","isolationUri":"vlan://233","isSecurityGroupEnabled":false},"instanceName":"r-37-VM","vmType":"DomainRouter","details":{},"wait":0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://233","vlanGateway":"82.119.X.X","vlanNetmask":"255.255.255.240","vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"trafficType":"Public","newNic":false}],"accessDetails":{"router.guest.ip":"82.X.X.X","router.ip":"169.254.2.247","zone.network.type":"Advanced","router.name":"r-37-VM"},"wait":0}},{"com.cloud.agent.api.routing.SetSourceNatCommand":{"ipAddress":{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"broadcastUri":"vlan://233","vlanGateway":"82.X.X.X","vlanNetmask":"255.255.255.240","vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"trafficType":"Public","newNic":false},"add":true,"accessDetails":{"router.ip":"169.254.2.247","zone.network.type":"Advanced","router.name":"r-37-VM"},"wait":0}},{}] }
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:null) (logid:ef8b353e) remaining mem36873936896
2018-05-23 12:58:43,649 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool 2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool 2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not find volume fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage volume not found: no storage vol with matching name 'fa3dba5d-364b-43ce-9ebf-d15a3324f765'
2018-05-23 12:58:43,664 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Refreshing storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Found volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage pool 2258aa76-7813-354d-b274-961fb337e716 after refreshing the pool
2018-05-23 12:58:44,204 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool 2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool 2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-5:null) (logid:ef8b353e) nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /bin/bash -c ip route | grep 169.254.0.0/16
2018-05-23 12:58:44,248 DEBUG [utils.script.Script] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:null) (logid:ef8b353e) starting r-37-VM: <domain type='kvm'>
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>87a45b1c-bf35-431e-82a0-c0c961feab05</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd' name='rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765'>
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-961fb337e716'/>
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.253.250.230' passwd='-Wf7MGedC0usYmBcCscv7A'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-37-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-37-VM.org.qemu.guest_agent.0'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>

2018-05-23 12:58:46,460 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%privategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:58:46,498 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:58:49,567 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to 169.254.2.247
2018-05-23 12:58:54,568 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:58:57,630 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to 169.254.2.247
2018-05-23 12:59:02,631 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to 169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%privategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:02,669 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-2:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:80cd5d21) Processing command: com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-2:null) (logid:) Execution is successful.
2018-05-23 12:59:04,772 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-2:null) (logid:) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent] (UgentTask-2:null) (logid:) Sending ping: Seq 15-2: { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"r-37-VM":{"state":"PowerOn","host":"kvm-test01-sb"}},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}] }
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) (logid:) Received response: Seq 15-2: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}] }
2018-05-23 12:59:05,726 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to 169.254.2.247
2018-05-23 12:59:10,727 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:59:12,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to 169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:275957d4) Processing command: com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:275957d4) Trying to fetch storage pool 2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-2:null) (logid:275957d4) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:275957d4) Asking libvirt to refresh storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:275957d4) Succesfully refreshed pool 2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to 169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%privategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:17,803 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:59:17,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to 169.254.2.247
2018-05-23 12:59:22,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:59:38,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to 169.254.2.247
2018-05-23 12:59:43,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to 169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py -n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%privategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,285 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null) (logid:ef8b353e) Ping command port, 169.254.2.247:3922
2018-05-23 12:59:43,286 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to 169.254.2.247
2018-05-23 12:59:43,287 DEBUG [resource.wrapper.LibvirtOvsVpcRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null) (logid:ef8b353e) Ping command port succeeded for vm r-37-VM
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-5:null) (logid:ef8b353e) nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 233 -p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) matchPifFileInDirectory: file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Transforming com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem, copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
2018-05-23 12:59:47,767 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) [INFO] update_config.py :: Processing incoming file => ip_associations.json[INFO] Processing JSON file ip_associations.json
2018-05-23 12:59:47,768 DEBUG [resource.virtualnetwork.VirtualRoutingResource] (agentRequest-Handler-5:null) (logid:ef8b353e) Processing ScriptConfigItem, executing update_config.py ip_associations.json took 186ms
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:ef8b353e) Seq 15-5506494969390563338: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" vpccidr=10.59.0.0/21 domain=test.internal dns1=8.8.8.8 privategateway=None template=domP name=r-37-VM eth0ip=169.254.2.247 eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56 port=8080","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBcCscv7A","vncAddr":"10.253.250.230","params":{},"uuid":"87a45b1c-bf35-431e-82a0-c0c961feab05","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"RAW","provisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHost":"ceph-mon","managed":"false","storagePort":"6789","volumeSize":"349945344"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}],"guestOsDetails":{}},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.10.0 Mon Oct 9 06:45:25 UTC 2017","scriptsVersion":"748f68e99144d82158a4c4d3dcbfef04\n","result":true,"details":"Cloudstack Release 4.10.0 Mon Oct 9 06:45:25 UTC 2017&748f68e99144d82158a4c4d3dcbfef04\n","wait":0}},{"com.cloud.agent.api.PlugNicAnswer":{"result":true,"details":"success","wait":0}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: null","null - failed: [INFO] update_config.py :: Processing incoming file => ip_associations.json[INFO] Processing JSON file ip_associations.json"],"result":false,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }
2018-05-23 12:59:47,835 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:ef8b353e) Request:Seq 15-5506494969390563341: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:47,836 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.StopCommand
2018-05-23 12:59:47,836 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:47,854 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py destroy_network_rules_for_vm --vmname r-37-VM --vif vnet0
2018-05-23 12:59:48,041 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:48,044 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:ef8b353e) Cleaning the metadata of vm snapshots of vm r-37-VM
2018-05-23 12:59:48,045 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:54,585 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) (logid:ef8b353e) successfully shut down vm r-37-VM
2018-05-23 12:59:54,586 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls /sys/class/net/cloud0
2018-05-23 12:59:54,591 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,593 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls /sys/class/net/cloud0/brif | tr '
' ' '
2018-05-23 12:59:54,598 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,600 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) (logid:ef8b353e) unable to get a vNet ID from name cloud0
2018-05-23 12:59:54,600 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls /sys/class/net/brenp8s0f0-233
2018-05-23 12:59:54,604 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,605 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls /sys/class/net/brenp8s0f0-233/brif | tr '
' ' '
2018-05-23 12:59:54,610 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,612 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -o delete -v 233 -p enp8s0f0 -b brenp8s0f0-233
2018-05-23 12:59:54,763 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,766 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:ef8b353e) Seq 15-5506494969390563341: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq 15-5506494969390563342: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:ef8b353e) Processing command: com.cloud.agent.api.StopCommand
2018-05-23 12:59:54,853 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Can not find KVM connection for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection at: lxc:///
2018-05-23 12:59:54,854 INFO [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) No existing libvirtd connection found. Opening a new one
2018-05-23 12:59:54,856 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Successfully connected to libvirt at: lxc:///
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Can not find LXC connection for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,857 WARN [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Can not find a connection for Instance r-37-VM. Assuming the default connection.
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:54,858 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:54,859 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py destroy_network_rules_for_vm --vmname r-37-VM
2018-05-23 12:59:55,043 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get vm :Domain not found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:55,047 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:null) (logid:ef8b353e) VM r-37-VM doesn't exist, no need to stop it
2018-05-23 12:59:55,048 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:ef8b353e) Seq 15-5506494969390563342: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:ef8b353e) Request:Seq 15-5506494969390563343: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged":false}},"name":"ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"RAW","provisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"wait":0}}] }
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:ef8b353e) Processing command: org.apache.cloudstack.storage.command.DeleteCommand
2018-05-23 12:59:56,941 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Trying to fetch storage pool 2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:56,952 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-1:null) (logid:ef8b353e) Looking for libvirtd connection at: qemu:///system
2018-05-23 12:59:56,960 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully refreshed pool 2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232 Used: 22173237211 Available: 1426660573184
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Attempting to remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Unprotecting and Removing RBD snapshots of image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765 prior to removing the image
2018-05-23 12:59:57,228 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully connected to Ceph cluster at ceph-mon:6789
2018-05-23 12:59:57,290 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Fetching list of snapshots of RBD image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:59:57,302 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully unprotected and removed any remaining snapshots (0) of rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765 Continuing to remove the RBD image
2018-05-23 12:59:57,302 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Closing image and destroying context
2018-05-23 12:59:57,307 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:ef8b353e) Instructing libvirt to remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,681 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:ef8b353e) Seq 15-5506494969390563343: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2018-05-23 13:00:04,666 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-3:null) (logid:) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 13:00:04,770 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-3:null) (logid:) Execution is successful.
2018-05-23 13:00:04,772 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-3:null) (logid:) Looking for libvirtd connection at: qemu:///system
2018-05-23 13:00:04,774 DEBUG [cloud.agent.Agent] (UgentTask-3:null) (logid:) Sending ping: Seq 15-3: { Cmd , MgmtId: -1, via: 15, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessible":true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}] }
2018-05-23 13:00:04,819 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) (logid:) Received response: Seq 15-3: { Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}] }

any ideas ?


king regards

Ben
Post by Andrija Panic
NetworkManager ? I though it was advised to not run it...
Ben,
Can you put the KVM agent in debug mode and post the logs?
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
* Si
________________________________
Sent: Friday, May 18, 2018 2:20 AM
To: Cloudstack Mailinglist
Subject: KVM Problem by deploying VPC
Hi Together,
im currently testing the Configuration of KVM Hosts in our Test
environment.
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:12:08 kvm-test01-sb systemd: Started Network Manager Script
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up' [vnet0]: new
request (4 scripts)
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up' [vnet0]: start
running ordered scripts...
6251: warning : qemuDomainObjTaint:5378 : Domain id=2 name='r-31-VM'
uuid=ff22b439-e0d0-44d1-a3cc-8dd23afb82eb is tainted: high-privileges
service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.
machine1.service'
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine and
Container Registration Service...
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully activated
service 'org.freedesktop.machine1'
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine and
Container Registration Service.
May 18 09:12:08 kvm-test01-sb systemd-machined: New machine qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Unit iscsi.service cannot be
reloaded because it is inactive.
May 18 09:12:08 kvm-test01-sb kvm: 1 guest now active
May 18 09:12:12 kvm-test01-sb kernel: kvm [48292]: vcpu0 disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44 kvm-test01-sb kernel: kvm [48292]: vcpu0 disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46 kvm-test01-sb sh: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
May 18 09:12:46 kvm-test01-sb sh: INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Asking libvirt to refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6454] manager: (enp8s0f0.233): new VLAN device
(/org/freedesktop/NetworkManager/Devices/15)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6470] device (enp8s0f0.233): carrier: link connected
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 1(enp8s0f0.233)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 1(enp8s0f0.233)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device enp8s0f0.233 entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 2(vnet1)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port 2(vnet1)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device vnet1 entered promiscuous mode
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6648] manager: (vnet1): new Tun device (/org/freedesktop/
NetworkManager/Devices/16)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6662] device (vnet1): state change: unmanaged -> unavailable
(reason 'connection-assumed', sys-iface-state: 'external')
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6674] device (vnet1): state change: unavailable -> disconnected
(reason 'none', sys-iface-state: 'external')
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0) entered
disabled state
May 18 09:13:10 kvm-test01-sb kernel: device vnet0 left promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0) entered
disabled state
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5339] device (vnet0): state change: activated -> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (cloud0): bridge port vnet0 was detached
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (vnet0): released from master device cloud0
service name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
May 18 09:13:10 kvm-test01-sb systemd: Starting Network Manager Script
Dispatcher Service...
May 18 09:13:10 kvm-test01-sb kernel: device vnet1 left promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: brenp8s0f0-233: port 2(vnet1)
entered disabled state
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Successfully activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:13:10 kvm-test01-sb systemd: Started Network Manager Script
Dispatcher Service.
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down' [vnet0]: new
request (4 scripts)
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down' [vnet0]: start
running ordered scripts...
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): state change: disconnected -> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): released from master device brenp8s0f0-233
1519: error : qemuMonitorIO:704 : internal error: End of file from qemu
monitor
May 18 09:13:10 kvm-test01-sb kvm: 0 guests now active
May 18 09:13:10 kvm-test01-sb systemd-machined: Machine qemu-2-r-31-VM
terminated.
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching uuid 'ff22b439-e0d0-44d1-a3cc-8dd23afb82eb'
(r-31-VM)
May 18 09:13:11 kvm-test01-sb kernel: device enp8s0f0.233 left promiscuous
mode
May 18 09:13:11 kvm-test01-sb kernel: brenp8s0f0-233: port 1(enp8s0f0.233)
entered disabled state
May 18 09:13:11 kvm-test01-sb NetworkManager[925]: <info>
[1526627591.2137] device (enp8s0f0.233): released from master device
brenp8s0f0-233
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) No existing libvirtd connection
found. Opening a new one
May 18 09:13:11 kvm-test01-sb sh: libvirt: LXC Driver error : Domain not
found: No domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) Can not find a connection for
Instance r-31-VM. Assuming the default connection.
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: libvirt: QEMU Driver error : Domain not
found: no domain with matching name 'r-31-VM'
2018-05-18 09:13:11,284 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Failed to start instance VM[DomainRouter|r-31-VM]
com.cloud.utils.exception.ExecutionException: Unable to start
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb due to error in finalizeStart,
not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:4691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.VmWorkJobDispatcher.runJob(
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:558)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-18 09:13:11,289 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Cleaning up resources for the vm VM[DomainRouter|r-31-VM]
in Starting state
2018-05-18 09:13:11,291 DEBUG [c.c.a.t.Request] (Work-Job-Executor-30:ctx-67604a44
Sending { Cmd , MgmtId: 109952567336, via: 10(kvm-test01-sb), Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-
31-VM","executeInSequence":false,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
Processing: { Ans: , MgmtId: 109952567336, via: 10, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request] (Work-Job-Executor-30:ctx-67604a44
Received: { Ans: , MgmtId: 109952567336, via: 10(kvm-test01-sb), Ver: v1,
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG [c.c.n.g.ControlNetworkGuru]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Released nic: NicProfile[65-31-null-null-null
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Successfully released network resources for the vm
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Successfully cleanued up resources for the vm
VM[DomainRouter|r-31-VM] in Starting state
Deployment of Isolatet Networks and redundant VPCs, works wothout any
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal (libvirt-3.9.0-14.el7_5.2.x86_64,
qemu-kvm-1.5.3-156.el7.x86_64, kernel-3.10.0-693.el7.x86_64)
Someone got an i idea ?
Kind regards
Ben
--
Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
E-Mail: ***@coders-area.de
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben, dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
Andrija Panic
2018-05-23 22:37:13 UTC
Permalink
Hi Ben,

interesting parts seems to be:
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Executing:
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1

What I believe we see here, is that IP association fails for some reason
(exit value is 1) and after that ACS will just stop the VM and do the
cleanup (all log lines after this one)...

Can you check your broadcast URI in the DB ? ACtually... I see it vlan 223
- I did have some issues in past releases, when we wanted to use untagged
vlans for Public network, but it seems not to be case here...

Not sure, if it's possible that you also SSH to the VR during this creation
process, in order to collect logs from inside the VR - before Qemu destroys
the VR?

ssh -p 3922 -i .ssh/id_rsa.cloud root@ 169.254.2.247 and then try to fetch
(SCP or something) whole /var/log/ folder to the localhost - from there,
there is cloud.log and auth.log where most of the commands outputs are
located (success or failure)

or something like rsync -av -e "ssh -p 3992 -i .ssh.id_rsa.cloud"
***@169.......
/local/dir/
and keep repeating until you fetch whatever data to be able to analyze it

Perhaps someone else will have better suggestion...

Best
Andrija
Post by Benjamin Naber
Hi Andrija,
first of all thanks for your reply. I have now testet the setup on a
Ubuntu Xenial. Same issu with Default VPC Offering. Redundant VPC offering
works also without any problems. Same as on CentOS.
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Request:Seq
15-5506494969390563336: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.ReadyCo
mmand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563337: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"
{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/sys
temvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2
3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ec
e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM Template
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.P
rimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337
e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"
rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&
STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged"
:false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO
":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"
QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hy
pervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}]
}
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Creating volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template
6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool
2258aa76-7813-354d-b274-961fb337e716 (RBD) with size 349945344
2018-05-23 12:58:43,225 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to perform a RBD
clone (layering) since we are operating in the same storage pool
2018-05-23 12:58:43,270 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) The source image
rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD format 2. We will perform
a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Checking if RBD snapshot
prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Found 1 snapshots on RBD
image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) RBD snapshot
exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully cloned
fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"
{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Request:Seq
15-5506494969390563338: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.StartCo
mmand":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter"
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0
(64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" vpccidr=
10.59.0.0/21 domain=test.internal dns1=8.8.8.8 privategateway=None
template=domP name=r-37-VM eth0ip=169.254.2.247 eth0mask=255.255.0.0
type=vpcrouter disable_rp_filter=true baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","params":{},"uuid":"87a45b1c-bf35-431e-82a0-
c0c961feab05","disks":[{"data":{"org.apache.cloudstack.storage.to
.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hy
pervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-
43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHo
st":"ceph-mon","managed":"false","storagePort":"6789","
volumeSize":"349945344"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-
b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-
af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.
0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broa
dcastType":"LinkLocal","type":"Control","isSecurityGroupEnab
led":false}],"guestOsDetails":{}},"hostIp":"10.253.250.230",
"executeInSequence":false,"wait":0}},{"com.cloud.agent.api.
check.CheckSshCommand":{"ip":"169.254.2.247","port":3922,"
interval":6,"retries":100,"name":"r-37-VM","wait":0}},{"
com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId"
true,"nicUuid":"29bd4c33-47dd-4357-aa3d-8c5ae56169b5","uuid"
:"787cf778-230f-4e17-a648-f63171bdc81c","ip":"82.119.177.
188","netmask":"255.255.255.240","gateway":"82.119.177.
177","mac":"1e:00:87:00:00:0d","broadcastType":"Vlan","type"
:"Public","broadcastUri":"vlan://233","isolationUri":"vl
an://233","isSecurityGroupEnabled":false},"instanceName":"r-
37-VM","vmType":"DomainRouter","details":{},"wait":0}},{"
com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses
":[{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"
"vlan://233","vlanGateway":"82.119.X.X","vlanNetmask":"255.255.255.240","
vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"
trafficType":"Public","newNic":false}],"accessDetails":{"
router.guest.ip":"82.X.X.X","router.ip":"169.254.2.247","
zone.network.type":"Advanced","router.name":"r-37-VM"},"
wait":0}},{"com.cloud.agent.api.routing.SetSourceNatComman
d":{"ipAddress":{"accountId":2,"publicIp":"82.X.X.X","sourc
eNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"
broadcastUri":"vlan://233","vlanGateway":"82.X.X.X","
00:0d","networkRate":200,"trafficType":"Public","newNic"
:false},"add":true,"accessDetails":{"router.ip":"169.254.2.
247","zone.network.type":"Advanced","router.name":"r-37-VM"},"wait":0}},{}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) remaining mem36873936896
2018-05-23 12:58:43,649 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not find volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage volume not found: no
storage vol with matching name 'fa3dba5d-364b-43ce-9ebf-d15a3324f765'
2018-05-23 12:58:43,664 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Refreshing storage pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Found volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage pool
2258aa76-7813-354d-b274-961fb337e716 after refreshing the pool
2018-05-23 12:58:44,204 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /bin/bash -c ip
route | grep 169.254.0.0/16
2018-05-23 <http://169.254.0.0/162018-05-23> 12:58:44,248 DEBUG
[utils.script.Script] (agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) starting r-37-VM: <domain
type='kvm'>
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>87a45b1c-bf35-431e-82a0-c0c961feab05</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd' name='rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765'>
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-961fb337e716'/>
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.253.250.230'
passwd='-Wf7MGedC0usYmBcCscv7A'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-37-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-
37-VM.org.qemu.guest_agent.0'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>
2018-05-23 12:58:46,460 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:58:46,498 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:49,567 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:58:54,568 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:57,630 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:02,631 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:02,669 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Execution is successful.
2018-05-23 12:59:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent] (UgentTask-2:null)
(logid:) Sending ping: Seq 15-2: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}] }
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
2018-05-23 12:59:05,726 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:10,727 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:12,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:275957d4) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Asking libvirt to refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:17,803 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:17,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:22,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:38,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:43,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,285 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port, 169.254.2.247:3922
2018-05-23 12:59:43,286 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,287 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port succeeded for vm r-37-VM
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 233
-p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.LibvirtComputingResource]
file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
2018-05-23 12:59:47,767 DEBUG [kvm.resource.LibvirtComputingResource]
Processing incoming file => ip_associations.json[INFO] Processing JSON file
ip_associations.json
2018-05-23 12:59:47,768 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing ScriptConfigItem,
executing update_config.py ip_associations.json took 186ms
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":37,"name":"
r-37-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"
maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux
5","bootArgs":" vpccidr=10.59.0.0/21 domain=test.internal dns1=8.8.8.8
privategateway=None template=domP name=r-37-VM eth0ip=169.254.2.247
eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true
baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0Rh
IGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","vncAddr":"10.253.250.230","params":{},"uuid":"
87a45b1c-bf35-431e-82a0-c0c961feab05","disks":[{"data":{"org.apache.
cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO"
:{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"po
olType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","isManaged":false}},"name":"
ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","
accountId":2,"format":"RAW","provisioningType":"THIN","id":3
7,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsRea
dRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq
":0,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","type"
:"ROOT","_details":{"storageHost":"ceph-mon","managed":"
false","storagePort":"6789","volumeSize":"349945344"}}],"
nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable"
:true,"nicUuid":"226f4319-b15c-4e48-8a74-6a28b574a992","
uuid":"76d57f99-f4af-42b8-af0a-41f0a2cd0ba5","ip":"169.
254.2.247","netmask":"255.255.0.0","gateway":"169.254.0.1","
{}},"result":true,"wait":0}},{"com.cloud.agent.api.check.Che
ckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.
api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.10.0
Mon Oct 9 06:45:25 UTC 2017","scriptsVersion":"748f68
e99144d82158a4c4d3dcbfef04\n","result":true,"details":"Cloudstack Release
4.10.0 Mon Oct 9 06:45:25 UTC 2017&748f68e99144d82158a4c4d3d
cbfef04\n","wait":0}},{"com.cloud.agent.api.PlugNicAnswer"
:{"result":true,"details":"success","wait":0}},{"com.cloud.
agent.api.routing.GroupAnswer":{"results":["null - success: null","null -
failed: [INFO] update_config.py :: Processing incoming file =>
ip_associations.json[INFO] Processing JSON file
ip_associations.json"],"result":false,"wait":0}},{"com.
cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous failure","wait":0}}] }
2018-05-23 12:59:47,835 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e) Request:Seq
15-5506494969390563341: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:47,836 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:47,836 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,854 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM --vif vnet0
2018-05-23 12:59:48,041 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:48,044 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Cleaning the metadata of vm
snapshots of vm r-37-VM
2018-05-23 12:59:48,045 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:54,585 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) successfully shut down vm
r-37-VM
2018-05-23 12:59:54,586 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0
2018-05-23 12:59:54,591 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,593 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0/brif | tr '
' ' '
2018-05-23 12:59:54,598 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,600 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) unable to get a vNet ID from
name cloud0
2018-05-23 12:59:54,600 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233
2018-05-23 12:59:54,604 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,605 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233/brif | tr '
' ' '
2018-05-23 12:59:54,610 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,612 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -o
delete -v 233 -p enp8s0f0 -b brenp8s0f0-233
2018-05-23 12:59:54,763 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,766 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563342: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:54,853 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find KVM connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: lxc:///
2018-05-23 12:59:54,854 INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) No existing libvirtd
connection found. Opening a new one
2018-05-23 12:59:54,856 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Successfully connected to
libvirt at: lxc:///
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find LXC connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,857 WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find a connection
for Instance r-37-VM. Assuming the default connection.
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,858 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,859 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM
2018-05-23 12:59:55,043 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get vm :Domain not
found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:55,047 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) VM r-37-VM doesn't exist, no
need to stop it
2018-05-23 12:59:55,048 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e) Request:Seq
15-5506494969390563343: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"
data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"wait":0}}]
}
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.DeleteCommand
2018-05-23 12:59:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:56,952 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-1:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:56,960 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Attempting to remove volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Unprotecting and Removing
RBD snapshots of image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765 prior to
removing the image
2018-05-23 12:59:57,228 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:59:57,290 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Fetching list of snapshots
of RBD image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:59:57,302 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully unprotected and
removed any remaining snapshots (0) of rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
Continuing to remove the RBD image
2018-05-23 12:59:57,302 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Closing image and destroying
context
2018-05-23 12:59:57,307 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Instructing libvirt to
remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,681 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2018-05-23 13:00:04,666 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 13:00:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Execution is successful.
2018-05-23 13:00:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 13:00:04,774 DEBUG [cloud.agent.Agent] (UgentTask-3:null)
(logid:) Sending ping: Seq 15-3: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessib
le":true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}]
}
2018-05-23 13:00:04,819 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
any ideas ?
king regards
Ben
Post by Andrija Panic
NetworkManager ? I though it was advised to not run it...
Ben,
Can you put the KVM agent in debug mode and post the logs?
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
* Si
________________________________
Sent: Friday, May 18, 2018 2:20 AM
To: Cloudstack Mailinglist
Subject: KVM Problem by deploying VPC
Hi Together,
im currently testing the Configuration of KVM Hosts in our Test
environment.
When i try to deploy a VPC the Hypervisor shows me the
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
Post by Andrija Panic
service 'org.freedesktop.nm_dispatcher'
May 18 09:12:08 kvm-test01-sb systemd: Started Network Manager
Script
Post by Andrija Panic
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:12:08 kvm-test01-sb libvirtd: 2018-05-18
6251: warning : qemuDomainObjTaint:5378 : Domain id=2
name='r-31-VM'
high-privileges
Post by Andrija Panic
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.machine1'
unit='dbus-org.freedesktop.
Post by Andrija Panic
machine1.service'
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual
Machine and
Post by Andrija Panic
Container Registration Service...
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
Post by Andrija Panic
service 'org.freedesktop.machine1'
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
and
Post by Andrija Panic
Container Registration Service.
May 18 09:12:08 kvm-test01-sb systemd-machined: New machine
qemu-2-r-31-VM.
Post by Andrija Panic
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Unit iscsi.service
cannot be
Post by Andrija Panic
reloaded because it is inactive.
May 18 09:12:08 kvm-test01-sb kvm: 1 guest now active
May 18 09:12:12 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
Post by Andrija Panic
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
Post by Andrija Panic
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
Post by Andrija Panic
(agentRequest-Handler-5:) (logid:b911ffae) Trying to fetch
storage pool
Post by Andrija Panic
2258aa76-7813-354d-b274-961fb337e716 from libvirt
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
Post by Andrija Panic
(agentRequest-Handler-5:) (logid:b911ffae) Asking libvirt to
refresh
Post by Andrija Panic
storage pool 2258aa76-7813-354d-b274-961fb337e716
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6454] manager: (enp8s0f0.233): new VLAN device
(/org/freedesktop/NetworkManager/Devices/15)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6470] device (enp8s0f0.233): carrier: link
connected
Post by Andrija Panic
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
Post by Andrija Panic
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
Post by Andrija Panic
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device enp8s0f0.233
entered
Post by Andrija Panic
promiscuous mode
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
Post by Andrija Panic
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
Post by Andrija Panic
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device vnet1 entered
promiscuous mode
Post by Andrija Panic
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6648] manager: (vnet1): new Tun device
(/org/freedesktop/
Post by Andrija Panic
NetworkManager/Devices/16)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6662] device (vnet1): state change: unmanaged ->
unavailable
Post by Andrija Panic
(reason 'connection-assumed', sys-iface-state: 'external')
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6674] device (vnet1): state change: unavailable ->
disconnected
Post by Andrija Panic
(reason 'none', sys-iface-state: 'external')
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
Post by Andrija Panic
disabled state
May 18 09:13:10 kvm-test01-sb kernel: device vnet0 left
promiscuous mode
Post by Andrija Panic
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
Post by Andrija Panic
disabled state
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5339] device (vnet0): state change: activated ->
unmanaged
Post by Andrija Panic
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (cloud0): bridge port vnet0 was
detached
Post by Andrija Panic
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (vnet0): released from master device
cloud0
Post by Andrija Panic
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
May 18 09:13:10 kvm-test01-sb systemd: Starting Network
Manager Script
Post by Andrija Panic
Dispatcher Service...
May 18 09:13:10 kvm-test01-sb kernel: device vnet1 left
promiscuous mode
Post by Andrija Panic
May 18 09:13:10 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
Post by Andrija Panic
entered disabled state
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Successfully
activated
Post by Andrija Panic
service 'org.freedesktop.nm_dispatcher'
May 18 09:13:10 kvm-test01-sb systemd: Started Network Manager
Script
Post by Andrija Panic
Dispatcher Service.
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): state change: disconnected
-> unmanaged
Post by Andrija Panic
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): released from master device
brenp8s0f0-233
Post by Andrija Panic
May 18 09:13:10 kvm-test01-sb libvirtd: 2018-05-18
1519: error : qemuMonitorIO:704 : internal error: End of file
from qemu
Post by Andrija Panic
monitor
May 18 09:13:10 kvm-test01-sb kvm: 0 guests now active
May 18 09:13:10 kvm-test01-sb systemd-machined: Machine
qemu-2-r-31-VM
Post by Andrija Panic
terminated.
Domain not
Post by Andrija Panic
found: no domain with matching uuid
'ff22b439-e0d0-44d1-a3cc-8dd23afb82eb'
Post by Andrija Panic
(r-31-VM)
May 18 09:13:11 kvm-test01-sb kernel: device enp8s0f0.233 left
promiscuous
Post by Andrija Panic
mode
May 18 09:13:11 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
Post by Andrija Panic
entered disabled state
May 18 09:13:11 kvm-test01-sb NetworkManager[925]: <info>
[1526627591.2137] device (enp8s0f0.233): released from master
device
Post by Andrija Panic
brenp8s0f0-233
Domain not
Post by Andrija Panic
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: INFO
[kvm.resource.LibvirtConnection]
Post by Andrija Panic
(agentRequest-Handler-3:) (logid:94c38c31) No existing
libvirtd connection
Post by Andrija Panic
found. Opening a new one
Domain not
Post by Andrija Panic
found: No domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: WARN
[kvm.resource.LibvirtConnection]
Post by Andrija Panic
(agentRequest-Handler-3:) (logid:94c38c31) Can not find a
connection for
Post by Andrija Panic
Instance r-31-VM. Assuming the default connection.
Domain not
Post by Andrija Panic
found: no domain with matching name 'r-31-VM'
Domain not
Post by Andrija Panic
found: no domain with matching name 'r-31-VM'
Domain not
Post by Andrija Panic
found: no domain with matching name 'r-31-VM'
Domain not
Post by Andrija Panic
found: no domain with matching name 'r-31-VM'
Domain not
Post by Andrija Panic
found: no domain with matching name 'r-31-VM'
2018-05-18 09:13:11,284 ERROR [c.c.v.VirtualMachineManagerIm
pl]
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
Post by Andrija Panic
(logid:94c38c31) Failed to start instance
VM[DomainRouter|r-31-VM]
Post by Andrija Panic
com.cloud.utils.exception.ExecutionException: Unable to start
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb due to error in
finalizeStart,
Post by Andrija Panic
not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:4691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.VmWorkJobDispatcher.runJob(
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.
java:558)
Post by Andrija Panic
at org.apache.cloudstack.managed.context.
ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.callWithContext(DefaultManagedContext
.java:103)
Post by Andrija Panic
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.runWithContext(DefaultManagedContext.
java:53)
Post by Andrija Panic
at org.apache.cloudstack.managed.context.
ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-18 09:13:11,289 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
Post by Andrija Panic
(logid:94c38c31) Cleaning up resources for the vm
VM[DomainRouter|r-31-VM]
Post by Andrija Panic
in Starting state
2018-05-18 09:13:11,291 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
Post by Andrija Panic
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
Sending { Cmd , MgmtId: 109952567336, via: 10(kvm-test01-sb),
Ver: v1,
Post by Andrija Panic
Flags: 100011, [{"com.cloud.agent.api.StopCom
mand":{"isProxy":false,
Post by Andrija Panic
"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-
31-VM","executeInSequence":false,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-14:null) (logid:) Seq
Processing: { Ans: , MgmtId: 109952567336, via: 10, Ver: v1,
Flags: 10,
Post by Andrija Panic
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}]
}
Post by Andrija Panic
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
Post by Andrija Panic
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
10(kvm-test01-sb), Ver: v1,
Post by Andrija Panic
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG [c.c.n.g.ControlNetworkGuru]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
Post by Andrija Panic
(logid:94c38c31) Released nic: NicProfile[65-31-null-null-null
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
Post by Andrija Panic
(logid:94c38c31) Successfully released network resources for
the vm
Post by Andrija Panic
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
Post by Andrija Panic
(logid:94c38c31) Successfully cleanued up resources for the vm
VM[DomainRouter|r-31-VM] in Starting state
Deployment of Isolatet Networks and redundant VPCs, works
wothout any
Post by Andrija Panic
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal (libvirt-3.9.0-14.el7_5.2.x86_64,
qemu-kvm-1.5.3-156.el7.x86_64, kernel-3.10.0-693.el7.x86_64)
Someone got an i idea ?
Kind regards
Ben
--
Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt
vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht
der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben,
dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die
eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den
Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und
löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
--
Andrija Panić
Benjamin Naber
2018-05-24 13:36:59 UTC
Permalink
Hi Andrija,

ty for reply. i solved the error. Now i have an other error. Problem was that it failed witout a bonding device. i created a bonding device (bond0) wiht a single nic now it works without problems On Ubuntu.

Example (didnt work):

auto enp8s0f0
iface enp8s0f0 inet manual

auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports enp8s0f0
bridge_fd 5
bridge_stp off
bridge_maxwait 1

Example (works fine):

auto enp8s0f0
iface enp8s0f0 inet manual
bond-master bond0

auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none


auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1

Now i moved back to centos7 and have another problem. all vlan bridges that will be created by cloudstack-agent not manual get in ifup state. if i ifup the vlan bridges manual all works fine. Have someone a idea how i can force centos to automaticly bring up new if devices ?


Kind regards

Ben
Post by Andrija Panic
Hi Ben,
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
What I believe we see here, is that IP association fails for some reason
(exit value is 1) and after that ACS will just stop the VM and do the
cleanup (all log lines after this one)...
Can you check your broadcast URI in the DB ? ACtually... I see it vlan 223
* I did have some issues in past releases, when we wanted to use untagged
vlans for Public network, but it seems not to be case here...
Not sure, if it's possible that you also SSH to the VR during this creation
process, in order to collect logs from inside the VR - before Qemu destroys
the VR?
(SCP or something) whole /var/log/ folder to the localhost - from there,
there is cloud.log and auth.log where most of the commands outputs are
located (success or failure)
or something like rsync -av -e "ssh -p 3992 -i .ssh.id_rsa.cloud"
/local/dir/
and keep repeating until you fetch whatever data to be able to analyze it
Perhaps someone else will have better suggestion...
Best
Andrija
Post by Benjamin Naber
Hi Andrija,
first of all thanks for your reply. I have now testet the setup on a
Ubuntu Xenial. Same issu with Default VPC Offering. Redundant VPC offering
works also without any problems. Same as on CentOS.
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Request:Seq
15-5506494969390563336: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.ReadyCo
mmand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563337: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"
{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/sys
temvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2
3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ec
e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM Template
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.P
rimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337
e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"
rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&
STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged"
:false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO
":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"
QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hy
pervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}]
}
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Creating volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template
6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool
2258aa76-7813-354d-b274-961fb337e716 (RBD) with size 349945344
2018-05-23 12:58:43,225 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to perform a RBD
clone (layering) since we are operating in the same storage pool
2018-05-23 12:58:43,270 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) The source image
rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD format 2. We will perform
a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Checking if RBD snapshot
prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Found 1 snapshots on RBD
image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) RBD snapshot
exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully cloned
fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"
{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Request:Seq
15-5506494969390563338: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.StartCo
mmand":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter"
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0
(64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" vpccidr=
10.59.0.0/21 domain=test.internal dns1=8.8.8.8 privategateway=None
template=domP name=r-37-VM eth0ip=169.254.2.247 eth0mask=255.255.0.0
type=vpcrouter disable_rp_filter=true baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","params":{},"uuid":"87a45b1c-bf35-431e-82a0-
c0c961feab05","disks":[{"data":{"org.apache.cloudstack.storage.to
.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hy
pervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-
43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHo
st":"ceph-mon","managed":"false","storagePort":"6789","
volumeSize":"349945344"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-
b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-
af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.
0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broa
dcastType":"LinkLocal","type":"Control","isSecurityGroupEnab
led":false}],"guestOsDetails":{}},"hostIp":"10.253.250.230",
"executeInSequence":false,"wait":0}},{"com.cloud.agent.api.
check.CheckSshCommand":{"ip":"169.254.2.247","port":3922,"
interval":6,"retries":100,"name":"r-37-VM","wait":0}},{"
com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId"
true,"nicUuid":"29bd4c33-47dd-4357-aa3d-8c5ae56169b5","uuid"
:"787cf778-230f-4e17-a648-f63171bdc81c","ip":"82.119.177.
188","netmask":"255.255.255.240","gateway":"82.119.177.
177","mac":"1e:00:87:00:00:0d","broadcastType":"Vlan","type"
:"Public","broadcastUri":"vlan://233","isolationUri":"vl
an://233","isSecurityGroupEnabled":false},"instanceName":"r-
37-VM","vmType":"DomainRouter","details":{},"wait":0}},{"
com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses
":[{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"
"vlan://233","vlanGateway":"82.119.X.X","vlanNetmask":"255.255.255.240","
vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"
trafficType":"Public","newNic":false}],"accessDetails":{"
router.guest.ip":"82.X.X.X","router.ip":"169.254.2.247","
zone.network.type":"Advanced","router.name":"r-37-VM"},"
wait":0}},{"com.cloud.agent.api.routing.SetSourceNatComman
d":{"ipAddress":{"accountId":2,"publicIp":"82.X.X.X","sourc
eNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"
broadcastUri":"vlan://233","vlanGateway":"82.X.X.X","
00:0d","networkRate":200,"trafficType":"Public","newNic"
:false},"add":true,"accessDetails":{"router.ip":"169.254.2.
247","zone.network.type":"Advanced","router.name":"r-37-VM"},"wait":0}},{}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) remaining mem36873936896
2018-05-23 12:58:43,649 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not find volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage volume not found: no
storage vol with matching name 'fa3dba5d-364b-43ce-9ebf-d15a3324f765'
2018-05-23 12:58:43,664 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Refreshing storage pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Found volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage pool
2258aa76-7813-354d-b274-961fb337e716 after refreshing the pool
2018-05-23 12:58:44,204 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /bin/bash -c ip
route | grep 169.254.0.0/16
2018-05-23 <http://169.254.0.0/162018-05-23> 12:58:44,248 DEBUG
[utils.script.Script] (agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) starting r-37-VM: <domain type='kvm'>
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>87a45b1c-bf35-431e-82a0-c0c961feab05</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd' name='rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765'>
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-961fb337e716'/>
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.253.250.230' passwd='-Wf7MGedC0usYmBcCscv7A'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-37-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r- 37-VM.org.qemu.guest_agent.0'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>
2018-05-23 12:58:46,460 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:58:46,498 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:49,567 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:58:54,568 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:57,630 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:02,631 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:02,669 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Execution is successful.
2018-05-23 12:59:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent] (UgentTask-2:null)
(logid:) Sending ping: Seq 15-2: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}] }
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
2018-05-23 12:59:05,726 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:10,727 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:12,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:275957d4) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Asking libvirt to refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:17,803 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:17,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:22,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:38,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:43,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,285 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port, 169.254.2.247:3922
2018-05-23 12:59:43,286 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,287 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port succeeded for vm r-37-VM
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 233
-p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.LibvirtComputingResource]
file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
2018-05-23 12:59:47,767 DEBUG [kvm.resource.LibvirtComputingResource]
Processing incoming file => ip_associations.json[INFO] Processing JSON file
ip_associations.json
2018-05-23 12:59:47,768 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing ScriptConfigItem,
executing update_config.py ip_associations.json took 186ms
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":37,"name":"
r-37-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"
maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux
5","bootArgs":" vpccidr=10.59.0.0/21 domain=test.internal dns1=8.8.8.8
privategateway=None template=domP name=r-37-VM eth0ip=169.254.2.247
eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true
baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0Rh
IGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","vncAddr":"10.253.250.230","params":{},"uuid":"
87a45b1c-bf35-431e-82a0-c0c961feab05","disks":[{"data":{"org.apache.
cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO"
:{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"po
olType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","isManaged":false}},"name":"
ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","
accountId":2,"format":"RAW","provisioningType":"THIN","id":3
7,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsRea
dRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq
":0,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","type"
:"ROOT","_details":{"storageHost":"ceph-mon","managed":"
false","storagePort":"6789","volumeSize":"349945344"}}],"
nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable"
:true,"nicUuid":"226f4319-b15c-4e48-8a74-6a28b574a992","
uuid":"76d57f99-f4af-42b8-af0a-41f0a2cd0ba5","ip":"169.
254.2.247","netmask":"255.255.0.0","gateway":"169.254.0.1","
{}},"result":true,"wait":0}},{"com.cloud.agent.api.check.Che
ckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.
api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.10.0
Mon Oct 9 06:45:25 UTC 2017","scriptsVersion":"748f68
e99144d82158a4c4d3dcbfef04\n","result":true,"details":"Cloudstack Release
4.10.0 Mon Oct 9 06:45:25 UTC 2017&748f68e99144d82158a4c4d3d
cbfef04\n","wait":0}},{"com.cloud.agent.api.PlugNicAnswer"
:{"result":true,"details":"success","wait":0}},{"com.cloud.
agent.api.routing.GroupAnswer":{"results":["null - success: null","null -
failed: [INFO] update_config.py :: Processing incoming file =>
ip_associations.json[INFO] Processing JSON file
ip_associations.json"],"result":false,"wait":0}},{"com.
cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous failure","wait":0}}] }
2018-05-23 12:59:47,835 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e) Request:Seq
15-5506494969390563341: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:47,836 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:47,836 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,854 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM --vif vnet0
2018-05-23 12:59:48,041 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:48,044 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Cleaning the metadata of vm
snapshots of vm r-37-VM
2018-05-23 12:59:48,045 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:54,585 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) successfully shut down vm
r-37-VM
2018-05-23 12:59:54,586 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0
2018-05-23 12:59:54,591 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,593 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0/brif | tr '
' ' '
2018-05-23 12:59:54,598 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,600 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) unable to get a vNet ID from
name cloud0
2018-05-23 12:59:54,600 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233
2018-05-23 12:59:54,604 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,605 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233/brif | tr '
' ' '
2018-05-23 12:59:54,610 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,612 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -o
delete -v 233 -p enp8s0f0 -b brenp8s0f0-233
2018-05-23 12:59:54,763 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,766 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563342: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:54,853 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find KVM connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: lxc:///
2018-05-23 12:59:54,854 INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) No existing libvirtd
connection found. Opening a new one
2018-05-23 12:59:54,856 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Successfully connected to
libvirt at: lxc:///
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find LXC connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,857 WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find a connection
for Instance r-37-VM. Assuming the default connection.
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,858 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,859 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM
2018-05-23 12:59:55,043 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get vm :Domain not
found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:55,047 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) VM r-37-VM doesn't exist, no
need to stop it
2018-05-23 12:59:55,048 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e) Request:Seq
15-5506494969390563343: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"
data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"wait":0}}]
}
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.DeleteCommand
2018-05-23 12:59:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:56,952 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-1:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:56,960 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Attempting to remove volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Unprotecting and Removing
RBD snapshots of image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765 prior to
removing the image
2018-05-23 12:59:57,228 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:59:57,290 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Fetching list of snapshots
of RBD image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:59:57,302 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully unprotected and
removed any remaining snapshots (0) of rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
Continuing to remove the RBD image
2018-05-23 12:59:57,302 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Closing image and destroying
context
2018-05-23 12:59:57,307 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Instructing libvirt to
remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,681 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2018-05-23 13:00:04,666 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 13:00:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Execution is successful.
2018-05-23 13:00:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 13:00:04,774 DEBUG [cloud.agent.Agent] (UgentTask-3:null)
(logid:) Sending ping: Seq 15-3: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessib
le":true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}]
}
2018-05-23 13:00:04,819 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
any ideas ?
king regards
Ben
Post by Andrija Panic
NetworkManager ? I though it was advised to not run it...
Ben,
Can you put the KVM agent in debug mode and post the logs?
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
* Si
________________________________
Sent: Friday, May 18, 2018 2:20 AM
To: Cloudstack Mailinglist
Subject: KVM Problem by deploying VPC
Hi Together,
im currently testing the Configuration of KVM Hosts in our Test
environment.
When i try to deploy a VPC the Hypervisor shows me the
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:12:08 kvm-test01-sb systemd: Started Network Manager
Script
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:12:08 kvm-test01-sb libvirtd: 2018-05-18
6251: warning : qemuDomainObjTaint:5378 : Domain id=2
name='r-31-VM'
high-privileges
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.machine1'
unit='dbus-org.freedesktop.
machine1.service'
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual
Machine and
Container Registration Service...
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.machine1'
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
and
Container Registration Service.
May 18 09:12:08 kvm-test01-sb systemd-machined: New machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Unit iscsi.service
cannot be
reloaded because it is inactive.
May 18 09:12:08 kvm-test01-sb kvm: 1 guest now active
May 18 09:12:12 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Trying to fetch
storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Asking libvirt to
refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6454] manager: (enp8s0f0.233): new VLAN device
(/org/freedesktop/NetworkManager/Devices/15)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6470] device (enp8s0f0.233): carrier: link
connected
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device enp8s0f0.233
entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device vnet1 entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6648] manager: (vnet1): new Tun device
(/org/freedesktop/
NetworkManager/Devices/16)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6662] device (vnet1): state change: unmanaged ->
unavailable
(reason 'connection-assumed', sys-iface-state: 'external')
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6674] device (vnet1): state change: unavailable ->
disconnected
(reason 'none', sys-iface-state: 'external')
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
disabled state
May 18 09:13:10 kvm-test01-sb kernel: device vnet0 left
promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
disabled state
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5339] device (vnet0): state change: activated ->
unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (cloud0): bridge port vnet0 was
detached
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (vnet0): released from master device
cloud0
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
May 18 09:13:10 kvm-test01-sb systemd: Starting Network
Manager Script
Dispatcher Service...
May 18 09:13:10 kvm-test01-sb kernel: device vnet1 left
promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered disabled state
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:13:10 kvm-test01-sb systemd: Started Network Manager
Script
Dispatcher Service.
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): state change: disconnected
-> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): released from master device
brenp8s0f0-233
May 18 09:13:10 kvm-test01-sb libvirtd: 2018-05-18
1519: error : qemuMonitorIO:704 : internal error: End of file
from qemu
monitor
May 18 09:13:10 kvm-test01-sb kvm: 0 guests now active
May 18 09:13:10 kvm-test01-sb systemd-machined: Machine
qemu-2-r-31-VM
terminated.
Domain not
found: no domain with matching uuid
'ff22b439-e0d0-44d1-a3cc-8dd23afb82eb'
(r-31-VM)
May 18 09:13:11 kvm-test01-sb kernel: device enp8s0f0.233 left
promiscuous
mode
May 18 09:13:11 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered disabled state
May 18 09:13:11 kvm-test01-sb NetworkManager[925]: <info>
[1526627591.2137] device (enp8s0f0.233): released from master
device
brenp8s0f0-233
Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: INFO
[kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) No existing
libvirtd connection
found. Opening a new one
Domain not
found: No domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: WARN
[kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) Can not find a
connection for
Instance r-31-VM. Assuming the default connection.
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
2018-05-18 09:13:11,284 ERROR [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Failed to start instance
VM[DomainRouter|r-31-VM]
com.cloud.utils.exception.ExecutionException: Unable to start
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb due to error in
finalizeStart,
not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:4691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.VmWorkJobDispatcher.runJob(
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.
java:558)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.callWithContext(DefaultManagedContext
.java:103)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.runWithContext(DefaultManagedContext.
java:53)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-18 09:13:11,289 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Cleaning up resources for the vm
VM[DomainRouter|r-31-VM]
in Starting state
2018-05-18 09:13:11,291 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
Sending { Cmd , MgmtId: 109952567336, via: 10(kvm-test01-sb),
Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCom
mand":{"isProxy":false,
"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-
31-VM","executeInSequence":false,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-14:null) (logid:) Seq
Processing: { Ans: , MgmtId: 109952567336, via: 10, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}]
}
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
10(kvm-test01-sb), Ver: v1,
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG [c.c.n.g.ControlNetworkGuru]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Released nic: NicProfile[65-31-null-null-null
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Successfully released network resources for
the vm
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Successfully cleanued up resources for the vm
VM[DomainRouter|r-31-VM] in Starting state
Deployment of Isolatet Networks and redundant VPCs, works
wothout any
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal (libvirt-3.9.0-14.el7_5.2.x86_64,
qemu-kvm-1.5.3-156.el7.x86_64, kernel-3.10.0-693.el7.x86_64)
Someone got an i idea ?
Kind regards
Ben
--
Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt
vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht
der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben,
dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die
eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den
Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und
löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
--
Andrija Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
E-Mail: ***@coders-area.de
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben, dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
Adam Witwicki
2018-05-24 14:03:08 UTC
Permalink
Hi Ben

" Now i moved back to centos7 and have another problem. all vlan bridges that will be created by cloudstack-agent not manual get in ifup state. if i ifup the vlan bridges manual all works fine. Have someone a idea how i can force centos to automaticly bring up new if devices ?"

I had this issue with 4.9 and ending up writing a dirty script to check the down bridges every minute via cron and if up them.

iface=`/usr/sbin/ip add | grep -E "brbond.*DOWN" | awk -F ':' '{print $2}'`
echo "$iface" | while read -r a; do /usr/sbin/ip link set dev $a up; done


Thanks

Adam

-----Original Message-----
From: Benjamin Naber <***@coders-area.de>
Sent: 24 May 2018 14:37
To: ***@cloudstack.apache.org
Subject: Re: KVM Problem by deploying VPC

** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **

Hi Andrija,

ty for reply. i solved the error. Now i have an other error. Problem was that it failed witout a bonding device. i created a bonding device (bond0) wiht a single nic now it works without problems On Ubuntu.

Example (didnt work):

auto enp8s0f0
iface enp8s0f0 inet manual

auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports enp8s0f0
bridge_fd 5
bridge_stp off
bridge_maxwait 1

Example (works fine):

auto enp8s0f0
iface enp8s0f0 inet manual
bond-master bond0

auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none


auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1

Now i moved back to centos7 and have another problem. all vlan bridges that will be created by cloudstack-agent not manual get in ifup state. if i ifup the vlan bridges manual all works fine. Have someone a idea how i can force centos to automaticly bring up new if devices ?


Kind regards

Ben
Post by Andrija Panic
Hi Ben,
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
What I believe we see here, is that IP association fails for some reason
(exit value is 1) and after that ACS will just stop the VM and do the
cleanup (all log lines after this one)...
Can you check your broadcast URI in the DB ? ACtually... I see it vlan 223
* I did have some issues in past releases, when we wanted to use untagged
vlans for Public network, but it seems not to be case here...
Not sure, if it's possible that you also SSH to the VR during this creation
process, in order to collect logs from inside the VR - before Qemu destroys
the VR?
(SCP or something) whole /var/log/ folder to the localhost - from there,
there is cloud.log and auth.log where most of the commands outputs are
located (success or failure)
or something like rsync -av -e "ssh -p 3992 -i .ssh.id_rsa.cloud"
/local/dir/
and keep repeating until you fetch whatever data to be able to analyze it
Perhaps someone else will have better suggestion...
Best
Andrija
Post by Benjamin Naber
Hi Andrija,
first of all thanks for your reply. I have now testet the setup on a
Ubuntu Xenial. Same issu with Default VPC Offering. Redundant VPC offering
works also without any problems. Same as on CentOS.
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Request:Seq
15-5506494969390563336: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.ReadyCo
mmand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563337: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"
{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/sys
temvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2
3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ec
e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM Template
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.P
rimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337
e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"
rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&
STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged"
:false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO
":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"
QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hy
pervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}]
}
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Creating volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template
6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool
2258aa76-7813-354d-b274-961fb337e716 (RBD) with size 349945344
2018-05-23 12:58:43,225 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to perform a RBD
clone (layering) since we are operating in the same storage pool
2018-05-23 12:58:43,270 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) The source image
rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD format 2. We will perform
a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Checking if RBD snapshot
prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Found 1 snapshots on RBD
image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) RBD snapshot
exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully cloned
fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"
{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Request:Seq
15-5506494969390563338: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.StartCo
mmand":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter"
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0
(64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" vpccidr=
10.59.0.0/21 domain=test.internal dns1=8.8.8.8 privategateway=None
template=domP name=r-37-VM eth0ip=169.254.2.247 eth0mask=255.255.0.0
type=vpcrouter disable_rp_filter=true baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","params":{},"uuid":"87a45b1c-bf35-431e-82a0-
c0c961feab05","disks":[{"data":{"org.apache.cloudstack.storage.to
.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hy
pervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-
43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHo
st":"ceph-mon","managed":"false","storagePort":"6789","
volumeSize":"349945344"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-
b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-
af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.
0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broa
dcastType":"LinkLocal","type":"Control","isSecurityGroupEnab
led":false}],"guestOsDetails":{}},"hostIp":"10.253.250.230",
"executeInSequence":false,"wait":0}},{"com.cloud.agent.api.
check.CheckSshCommand":{"ip":"169.254.2.247","port":3922,"
interval":6,"retries":100,"name":"r-37-VM","wait":0}},{"
com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId"
true,"nicUuid":"29bd4c33-47dd-4357-aa3d-8c5ae56169b5","uuid"
:"787cf778-230f-4e17-a648-f63171bdc81c","ip":"82.119.177.
188","netmask":"255.255.255.240","gateway":"82.119.177.
177","mac":"1e:00:87:00:00:0d","broadcastType":"Vlan","type"
:"Public","broadcastUri":"vlan://233","isolationUri":"vl
an://233","isSecurityGroupEnabled":false},"instanceName":"r-
37-VM","vmType":"DomainRouter","details":{},"wait":0}},{"
com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses
":[{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"
"vlan://233","vlanGateway":"82.119.X.X","vlanNetmask":"255.255.255.240","
vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"
trafficType":"Public","newNic":false}],"accessDetails":{"
router.guest.ip":"82.X.X.X","router.ip":"169.254.2.247","
zone.network.type":"Advanced","router.name":"r-37-VM"},"
wait":0}},{"com.cloud.agent.api.routing.SetSourceNatComman
d":{"ipAddress":{"accountId":2,"publicIp":"82.X.X.X","sourc
eNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"
broadcastUri":"vlan://233","vlanGateway":"82.X.X.X","
00:0d","networkRate":200,"trafficType":"Public","newNic"
:false},"add":true,"accessDetails":{"router.ip":"169.254.2.
247","zone.network.type":"Advanced","router.name":"r-37-VM"},"wait":0}},{}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) remaining mem36873936896
2018-05-23 12:58:43,649 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not find volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage volume not found: no
storage vol with matching name 'fa3dba5d-364b-43ce-9ebf-d15a3324f765'
2018-05-23 12:58:43,664 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Refreshing storage pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Found volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage pool
2258aa76-7813-354d-b274-961fb337e716 after refreshing the pool
2018-05-23 12:58:44,204 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /bin/bash -c ip
route | grep 169.254.0.0/16
2018-05-23 <http://169.254.0.0/162018-05-23> 12:58:44,248 DEBUG
[utils.script.Script] (agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) starting r-37-VM: <domain type='kvm'>
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>87a45b1c-bf35-431e-82a0-c0c961feab05</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd' name='rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765'>
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-961fb337e716'/>
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.253.250.230' passwd='-Wf7MGedC0usYmBcCscv7A'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-37-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r- 37-VM.org.qemu.guest_agent.0'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>
2018-05-23 12:58:46,460 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:58:46,498 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:49,567 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:58:54,568 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:57,630 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:02,631 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:02,669 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Execution is successful.
2018-05-23 12:59:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent] (UgentTask-2:null)
(logid:) Sending ping: Seq 15-2: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}] }
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
2018-05-23 12:59:05,726 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:10,727 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:12,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:275957d4) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Asking libvirt to refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:17,803 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:17,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:22,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:38,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:43,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,285 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port, 169.254.2.247:3922
2018-05-23 12:59:43,286 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,287 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port succeeded for vm r-37-VM
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 233
-p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.LibvirtComputingResource]
file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
2018-05-23 12:59:47,767 DEBUG [kvm.resource.LibvirtComputingResource]
Processing incoming file => ip_associations.json[INFO] Processing JSON file
ip_associations.json
2018-05-23 12:59:47,768 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing ScriptConfigItem,
executing update_config.py ip_associations.json took 186ms
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":37,"name":"
r-37-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"
maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux
5","bootArgs":" vpccidr=10.59.0.0/21 domain=test.internal dns1=8.8.8.8
privategateway=None template=domP name=r-37-VM eth0ip=169.254.2.247
eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true
baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0Rh
IGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","vncAddr":"10.253.250.230","params":{},"uuid":"
87a45b1c-bf35-431e-82a0-c0c961feab05","disks":[{"data":{"org.apache.
cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO"
:{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"po
olType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","isManaged":false}},"name":"
ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","
accountId":2,"format":"RAW","provisioningType":"THIN","id":3
7,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsRea
dRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq
":0,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","type"
:"ROOT","_details":{"storageHost":"ceph-mon","managed":"
false","storagePort":"6789","volumeSize":"349945344"}}],"
nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable"
:true,"nicUuid":"226f4319-b15c-4e48-8a74-6a28b574a992","
uuid":"76d57f99-f4af-42b8-af0a-41f0a2cd0ba5","ip":"169.
254.2.247","netmask":"255.255.0.0","gateway":"169.254.0.1","
{}},"result":true,"wait":0}},{"com.cloud.agent.api.check.Che
ckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.
api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.10.0
Mon Oct 9 06:45:25 UTC 2017","scriptsVersion":"748f68
e99144d82158a4c4d3dcbfef04\n","result":true,"details":"Cloudstack Release
4.10.0 Mon Oct 9 06:45:25 UTC 2017&748f68e99144d82158a4c4d3d
cbfef04\n","wait":0}},{"com.cloud.agent.api.PlugNicAnswer"
:{"result":true,"details":"success","wait":0}},{"com.cloud.
agent.api.routing.GroupAnswer":{"results":["null - success: null","null -
failed: [INFO] update_config.py :: Processing incoming file =>
ip_associations.json[INFO] Processing JSON file
ip_associations.json"],"result":false,"wait":0}},{"com.
cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous failure","wait":0}}] }
2018-05-23 12:59:47,835 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e) Request:Seq
15-5506494969390563341: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:47,836 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:47,836 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,854 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM --vif vnet0
2018-05-23 12:59:48,041 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:48,044 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Cleaning the metadata of vm
snapshots of vm r-37-VM
2018-05-23 12:59:48,045 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:54,585 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) successfully shut down vm
r-37-VM
2018-05-23 12:59:54,586 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0
2018-05-23 12:59:54,591 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,593 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0/brif | tr '
' ' '
2018-05-23 12:59:54,598 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,600 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) unable to get a vNet ID from
name cloud0
2018-05-23 12:59:54,600 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233
2018-05-23 12:59:54,604 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,605 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233/brif | tr '
' ' '
2018-05-23 12:59:54,610 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,612 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -o
delete -v 233 -p enp8s0f0 -b brenp8s0f0-233
2018-05-23 12:59:54,763 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,766 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563342: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:54,853 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find KVM connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: lxc:///
2018-05-23 12:59:54,854 INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) No existing libvirtd
connection found. Opening a new one
2018-05-23 12:59:54,856 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Successfully connected to
libvirt at: lxc:///
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find LXC connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,857 WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find a connection
for Instance r-37-VM. Assuming the default connection.
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,858 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,859 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM
2018-05-23 12:59:55,043 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get vm :Domain not
found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:55,047 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) VM r-37-VM doesn't exist, no
need to stop it
2018-05-23 12:59:55,048 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e) Request:Seq
15-5506494969390563343: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"
data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"wait":0}}]
}
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.DeleteCommand
2018-05-23 12:59:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:56,952 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-1:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:56,960 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Attempting to remove volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Unprotecting and Removing
RBD snapshots of image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765 prior to
removing the image
2018-05-23 12:59:57,228 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:59:57,290 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Fetching list of snapshots
of RBD image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:59:57,302 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully unprotected and
removed any remaining snapshots (0) of rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
Continuing to remove the RBD image
2018-05-23 12:59:57,302 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Closing image and destroying
context
2018-05-23 12:59:57,307 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Instructing libvirt to
remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,681 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2018-05-23 13:00:04,666 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 13:00:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Execution is successful.
2018-05-23 13:00:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 13:00:04,774 DEBUG [cloud.agent.Agent] (UgentTask-3:null)
(logid:) Sending ping: Seq 15-3: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessib
le":true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}]
}
2018-05-23 13:00:04,819 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
any ideas ?
king regards
Ben
Post by Andrija Panic
NetworkManager ? I though it was advised to not run it...
Ben,
Can you put the KVM agent in debug mode and post the logs?
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
* Si
________________________________
Sent: Friday, May 18, 2018 2:20 AM
To: Cloudstack Mailinglist
Subject: KVM Problem by deploying VPC
Hi Together,
im currently testing the Configuration of KVM Hosts in our Test
environment.
When i try to deploy a VPC the Hypervisor shows me the
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:12:08 kvm-test01-sb systemd: Started Network Manager
Script
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:12:08 kvm-test01-sb libvirtd: 2018-05-18
6251: warning : qemuDomainObjTaint:5378 : Domain id=2
name='r-31-VM'
high-privileges
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.machine1'
unit='dbus-org.freedesktop.
machine1.service'
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual
Machine and
Container Registration Service...
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.machine1'
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
and
Container Registration Service.
May 18 09:12:08 kvm-test01-sb systemd-machined: New machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Unit iscsi.service
cannot be
reloaded because it is inactive.
May 18 09:12:08 kvm-test01-sb kvm: 1 guest now active
May 18 09:12:12 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Trying to fetch
storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Asking libvirt to
refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6454] manager: (enp8s0f0.233): new VLAN device
(/org/freedesktop/NetworkManager/Devices/15)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6470] device (enp8s0f0.233): carrier: link
connected
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device enp8s0f0.233
entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device vnet1 entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6648] manager: (vnet1): new Tun device
(/org/freedesktop/
NetworkManager/Devices/16)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6662] device (vnet1): state change: unmanaged ->
unavailable
(reason 'connection-assumed', sys-iface-state: 'external')
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6674] device (vnet1): state change: unavailable ->
disconnected
(reason 'none', sys-iface-state: 'external')
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
disabled state
May 18 09:13:10 kvm-test01-sb kernel: device vnet0 left
promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
disabled state
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5339] device (vnet0): state change: activated ->
unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (cloud0): bridge port vnet0 was
detached
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (vnet0): released from master device
cloud0
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
May 18 09:13:10 kvm-test01-sb systemd: Starting Network
Manager Script
Dispatcher Service...
May 18 09:13:10 kvm-test01-sb kernel: device vnet1 left
promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered disabled state
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:13:10 kvm-test01-sb systemd: Started Network Manager
Script
Dispatcher Service.
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): state change: disconnected
-> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): released from master device
brenp8s0f0-233
May 18 09:13:10 kvm-test01-sb libvirtd: 2018-05-18
1519: error : qemuMonitorIO:704 : internal error: End of file
from qemu
monitor
May 18 09:13:10 kvm-test01-sb kvm: 0 guests now active
May 18 09:13:10 kvm-test01-sb systemd-machined: Machine
qemu-2-r-31-VM
terminated.
Domain not
found: no domain with matching uuid
'ff22b439-e0d0-44d1-a3cc-8dd23afb82eb'
(r-31-VM)
May 18 09:13:11 kvm-test01-sb kernel: device enp8s0f0.233 left
promiscuous
mode
May 18 09:13:11 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered disabled state
May 18 09:13:11 kvm-test01-sb NetworkManager[925]: <info>
[1526627591.2137] device (enp8s0f0.233): released from master
device
brenp8s0f0-233
Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: INFO
[kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) No existing
libvirtd connection
found. Opening a new one
Domain not
found: No domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: WARN
[kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) Can not find a
connection for
Instance r-31-VM. Assuming the default connection.
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
2018-05-18 09:13:11,284 ERROR [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Failed to start instance
VM[DomainRouter|r-31-VM]
com.cloud.utils.exception.ExecutionException: Unable to start
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb due to error in
finalizeStart,
not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:4691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.VmWorkJobDispatcher.runJob(
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.
java:558)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.callWithContext(DefaultManagedContext
.java:103)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.runWithContext(DefaultManagedContext.
java:53)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-18 09:13:11,289 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Cleaning up resources for the vm
VM[DomainRouter|r-31-VM]
in Starting state
2018-05-18 09:13:11,291 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
Sending { Cmd , MgmtId: 109952567336, via: 10(kvm-test01-sb),
Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCom
mand":{"isProxy":false,
"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-
31-VM","executeInSequence":false,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-14:null) (logid:) Seq
Processing: { Ans: , MgmtId: 109952567336, via: 10, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}]
}
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
10(kvm-test01-sb), Ver: v1,
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG [c.c.n.g.ControlNetworkGuru]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Released nic: NicProfile[65-31-null-null-null
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Successfully released network resources for
the vm
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Successfully cleanued up resources for the vm
VM[DomainRouter|r-31-VM] in Starting state
Deployment of Isolatet Networks and redundant VPCs, works
wothout any
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal (libvirt-3.9.0-14.el7_5.2.x86_64,
qemu-kvm-1.5.3-156.el7.x86_64, kernel-3.10.0-693.el7.x86_64)
Someone got an i idea ?
Kind regards
Ben
--
Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehängter Dateien enthält
vertrauliche und / oder rechtlich geschützte Informationen. Wenn Sie nicht
der richtige Adressat sind und diese E-mail irrtümlich erhalten haben,
dürfen Sie weder den Inhalt dieser E-mail nutzen noch dürfen Sie die
eventuell angehängten Dateien öffnen und auch keine Kopie fertigen oder den
Inhalt weitergeben / verbreiten. Bitte verständigen Sie den Absender und
löschen Sie diese E-mail und eventuell angehängte Dateien umgehend.
--
Andrija Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
E-Mail: ***@coders-area.de
___________________________________________________
Diese E-mail einschließlich eventuell angehängter Dateien enthält vertrauliche und / oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtümlich erhalten haben, dürfen Sie weder den Inhalt dieser E-mail nutzen noch dürfen Sie die eventuell angehängten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verständigen Sie den Absender und löschen Sie diese E-mail und eventuell angehängte Dateien umgehend.
Disclaimer Notice:
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Regis
Benjamin Naber
2018-05-24 15:45:25 UTC
Permalink
Hi Adam,


got it ! the problem is this tiny script: /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh

at this lines you can see that ifconfig is used to bring bridges up.

ifconfig |grep -w $vlanBr > /dev/null
if [ $? -gt 0 ]
then
ifconfig $vlanBr up
fi
return 0

the simple solution for centos7: just install the package net-tools over package manager! Centos in basic installation have no ifconfig installed. Ubuntu and Debian derivates have it.

Kind regards and thanks for all impressions

Ben
Post by Andrija Panic
Hi Ben
" Now i moved back to centos7 and have another problem. all vlan bridges that will be created by cloudstack-agent not manual get in ifup state. if i ifup the vlan bridges manual all works fine. Have someone a idea how i can force centos to automaticly bring up new if devices ?"
I had this issue with 4.9 and ending up writing a dirty script to check the down bridges every minute via cron and if up them.
iface=/usr/sbin/ip add | grep -E &quot;brbond.*DOWN&quot; | awk -F &#39;:&#39; &#39;{print $2}&#39;
echo "$iface" | while read -r a; do /usr/sbin/ip link set dev $a up; done
Thanks
Adam
-----Original Message-----
Sent: 24 May 2018 14:37
Subject: Re: KVM Problem by deploying VPC
** This mail originated from OUTSIDE the Oakford corporate network. Treat hyperlinks and attachments in this email with caution. **
Hi Andrija,
ty for reply. i solved the error. Now i have an other error. Problem was that it failed witout a bonding device. i created a bonding device (bond0) wiht a single nic now it works without problems On Ubuntu.
auto enp8s0f0
iface enp8s0f0 inet manual
auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports enp8s0f0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
auto enp8s0f0
iface enp8s0f0 inet manual
bond-master bond0
auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none
auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
Now i moved back to centos7 and have another problem. all vlan bridges that will be created by cloudstack-agent not manual get in ifup state. if i ifup the vlan bridges manual all works fine. Have someone a idea how i can force centos to automaticly bring up new if devices ?
Kind regards
Ben
Post by Andrija Panic
Hi Ben,
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
What I believe we see here, is that IP association fails for some reason
(exit value is 1) and after that ACS will just stop the VM and do the
cleanup (all log lines after this one)...
Can you check your broadcast URI in the DB ? ACtually... I see it vlan 223
* I did have some issues in past releases, when we wanted to use untagged
vlans for Public network, but it seems not to be case here...
Not sure, if it's possible that you also SSH to the VR during this creation
process, in order to collect logs from inside the VR - before Qemu destroys
the VR?
(SCP or something) whole /var/log/ folder to the localhost - from there,
there is cloud.log and auth.log where most of the commands outputs are
located (success or failure)
or something like rsync -av -e "ssh -p 3992 -i .ssh.id_rsa.cloud"
/local/dir/
and keep repeating until you fetch whatever data to be able to analyze it
Perhaps someone else will have better suggestion...
Best
Andrija
Post by Benjamin Naber
Hi Andrija,
first of all thanks for your reply. I have now testet the setup on a
Ubuntu Xenial. Same issu with Default VPC Offering. Redundant VPC offering
works also without any problems. Same as on CentOS.
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Request:Seq
15-5506494969390563336: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.ReadyCo
mmand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Proccess agent ready
command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent]
agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563337: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"
{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/sys
temvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.qcow2.bz2
3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ec
e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM Template
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.P
rimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337
e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"
rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&
STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged"
:false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO
":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"
QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hy
pervisorType":"KVM"}},"executeInSequence":true,"options":{},"options2":{},"wait":0}}]
}
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Creating volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template
6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool
2258aa76-7813-354d-b274-961fb337e716 (RBD) with size 349945344
2018-05-23 12:58:43,225 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to perform a RBD
clone (layering) since we are operating in the same storage pool
2018-05-23 12:58:43,270 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) The source image
rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD format 2. We will perform
a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Checking if RBD snapshot
prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Found 1 snapshots on RBD
image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) RBD snapshot
exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully cloned
fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"
{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","accountId":0,"format":"RAW","id":0}},"result":true,"wait":0}}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Request:Seq
15-5506494969390563338: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.StartCo
mmand":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter"
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 5.0
(64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" vpccidr=
10.59.0.0/21 domain=test.internal dns1=8.8.8.8 privategateway=None
template=domP name=r-37-VM eth0ip=169.254.2.247 eth0mask=255.255.0.0
type=vpcrouter disable_rp_filter=true baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","params":{},"uuid":"87a45b1c-bf35-431e-82a0-
c0c961feab05","disks":[{"data":{"org.apache.cloudstack.storage.to
.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hy
pervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-
43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHo
st":"ceph-mon","managed":"false","storagePort":"6789","
volumeSize":"349945344"}}],"nics":[{"deviceId":0,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-
b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-
af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.
0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broa
dcastType":"LinkLocal","type":"Control","isSecurityGroupEnab
led":false}],"guestOsDetails":{}},"hostIp":"10.253.250.230",
"executeInSequence":false,"wait":0}},{"com.cloud.agent.api.
check.CheckSshCommand":{"ip":"169.254.2.247","port":3922,"
interval":6,"retries":100,"name":"r-37-VM","wait":0}},{"
com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId"
true,"nicUuid":"29bd4c33-47dd-4357-aa3d-8c5ae56169b5","uuid"
:"787cf778-230f-4e17-a648-f63171bdc81c","ip":"82.119.177.
188","netmask":"255.255.255.240","gateway":"82.119.177.
177","mac":"1e:00:87:00:00:0d","broadcastType":"Vlan","type"
:"Public","broadcastUri":"vlan://233","isolationUri":"vl
an://233","isSecurityGroupEnabled":false},"instanceName":"r-
37-VM","vmType":"DomainRouter","details":{},"wait":0}},{"
com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses
":[{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"
"vlan://233","vlanGateway":"82.119.X.X","vlanNetmask":"255.255.255.240","
vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"
trafficType":"Public","newNic":false}],"accessDetails":{"
router.guest.ip":"82.X.X.X","router.ip":"169.254.2.247","
zone.network.type":"Advanced","router.name":"r-37-VM"},"
wait":0}},{"com.cloud.agent.api.routing.SetSourceNatComman
d":{"ipAddress":{"accountId":2,"publicIp":"82.X.X.X","sourc
eNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"
broadcastUri":"vlan://233","vlanGateway":"82.X.X.X","
00:0d","networkRate":200,"trafficType":"Public","newNic"
:false},"add":true,"accessDetails":{"router.ip":"169.254.2.
247","zone.network.type":"Advanced","router.name":"r-37-VM"},"wait":0}},{}]
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) remaining mem36873936896
2018-05-23 12:58:43,649 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not find volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage volume not found: no
storage vol with matching name 'fa3dba5d-364b-43ce-9ebf-d15a3324f765'
2018-05-23 12:58:43,664 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Refreshing storage pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Found volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage pool
2258aa76-7813-354d-b274-961fb337e716 after refreshing the pool
2018-05-23 12:58:44,204 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully refreshed pool
22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) (logid:ef8b353e) Executing: /bin/bash -c ip
route | grep 169.254.0.0/16
2018-05-23 <http://169.254.0.0/162018-05-23> 12:58:44,248 DEBUG
[utils.script.Script] (agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.LibvirtStartCommandWrapper]
(agentRequest-Handler-5:null) (logid:ef8b353e) starting r-37-VM: <domain type='kvm'>
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>87a45b1c-bf35-431e-82a0-c0c961feab05</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd' name='rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765'>
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-961fb337e716'/>
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.253.250.230' passwd='-Wf7MGedC0usYmBcCscv7A'/>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-37-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r- 37-VM.org.qemu.guest_agent.0'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>
2018-05-23 12:58:46,460 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:58:46,498 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:49,567 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:58:54,568 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:58:57,630 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:02,631 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:02,669 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-2:null) (logid:) Execution is successful.
2018-05-23 12:59:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent] (UgentTask-2:null)
(logid:) Sending ping: Seq 15-2: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}] }
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
2018-05-23 12:59:05,726 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:10,727 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:12,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:275957d4) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Asking libvirt to refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) (logid:275957d4) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:17,803 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:17,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:22,804 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:38,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not connect to
169.254.2.247
2018-05-23 12:59:43,239 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon to
169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.py
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=test.internal%dns1=8.8.8.8%priv
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,285 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port, 169.254.2.247:3922
2018-05-23 12:59:43,286 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to connect to
169.254.2.247
2018-05-23 12:59:43,287 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper] (agentRequest-Handler-5:null)
(logid:ef8b353e) Ping command port succeeded for vm r-37-VM
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 233
-p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.LibvirtComputingResource]
file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri for
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing FileConfigItem,
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
2018-05-23 12:59:47,767 DEBUG [kvm.resource.LibvirtComputingResource]
Processing incoming file => ip_associations.json[INFO] Processing JSON file
ip_associations.json
2018-05-23 12:59:47,768 DEBUG [resource.virtualnetwork.VirtualRoutingResource]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing ScriptConfigItem,
executing update_config.py ip_associations.json took 186ms
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":37,"name":"
r-37-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"
maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux
5","bootArgs":" vpccidr=10.59.0.0/21 domain=test.internal dns1=8.8.8.8
privategateway=None template=domP name=r-37-VM eth0ip=169.254.2.247
eth0mask=255.255.0.0 type=vpcrouter disable_rp_filter=true
baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71Xu0Rh
IGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ host=172.16.11.56
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","vncAddr":"10.253.250.230","params":{},"uuid":"
87a45b1c-bf35-431e-82a0-c0c961feab05","disks":[{"data":{"org.apache.
cloudstack.storage.to.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO"
:{"uuid":"2258aa76-7813-354d-b274-961fb337e716","id":14,"po
olType":"RBD","host":"ceph-mon","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","isManaged":false}},"name":"
ROOT-37","size":349945344,"path":"fa3dba5d-364b-43ce-
9ebf-d15a3324f765","volumeId":37,"vmName":"r-37-VM","
accountId":2,"format":"RAW","provisioningType":"THIN","id":3
7,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsRea
dRate":0,"iopsWriteRate":0,"hypervisorType":"KVM"}},"diskSeq
":0,"path":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","type"
:"ROOT","_details":{"storageHost":"ceph-mon","managed":"
false","storagePort":"6789","volumeSize":"349945344"}}],"
nics":[{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable"
:true,"nicUuid":"226f4319-b15c-4e48-8a74-6a28b574a992","
uuid":"76d57f99-f4af-42b8-af0a-41f0a2cd0ba5","ip":"169.
254.2.247","netmask":"255.255.0.0","gateway":"169.254.0.1","
{}},"result":true,"wait":0}},{"com.cloud.agent.api.check.Che
ckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.
api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack Release 4.10.0
Mon Oct 9 06:45:25 UTC 2017","scriptsVersion":"748f68
e99144d82158a4c4d3dcbfef04\n","result":true,"details":"Cloudstack Release
4.10.0 Mon Oct 9 06:45:25 UTC 2017&748f68e99144d82158a4c4d3d
cbfef04\n","wait":0}},{"com.cloud.agent.api.PlugNicAnswer"
:{"result":true,"details":"success","wait":0}},{"com.cloud.
agent.api.routing.GroupAnswer":{"results":["null - success: null","null -
failed: [INFO] update_config.py :: Processing incoming file =>
ip_associations.json[INFO] Processing JSON file
ip_associations.json"],"result":false,"wait":0}},{"com.
cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous
failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped
by previous failure","wait":0}}] }
2018-05-23 12:59:47,835 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e) Request:Seq
15-5506494969390563341: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:47,836 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:47,836 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:47,854 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM --vif vnet0
2018-05-23 12:59:48,041 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:48,044 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Cleaning the metadata of vm
snapshots of vm r-37-VM
2018-05-23 12:59:48,045 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:54,585 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) (logid:ef8b353e) successfully shut down vm
r-37-VM
2018-05-23 12:59:54,586 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0
2018-05-23 12:59:54,591 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,593 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/cloud0/brif | tr '
' ' '
2018-05-23 12:59:54,598 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,600 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) unable to get a vNet ID from
name cloud0
2018-05-23 12:59:54,600 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233
2018-05-23 12:59:54,604 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,605 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Executing: /bin/bash -c ls
/sys/class/net/brenp8s0f0-233/brif | tr '
' ' '
2018-05-23 12:59:54,610 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,612 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -o
delete -v 233 -p enp8s0f0 -b brenp8s0f0-233
2018-05-23 12:59:54,763 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:54,766 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563342: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
checkBeforeCleanup":false,"forceStop":false,"vmName":"r-37-
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
com.cloud.agent.api.StopCommand
2018-05-23 12:59:54,853 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find KVM connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,854 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: lxc:///
2018-05-23 12:59:54,854 INFO [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) No existing libvirtd
connection found. Opening a new one
2018-05-23 12:59:54,856 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Successfully connected to
libvirt at: lxc:///
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find LXC connection
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,857 WARN [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Can not find a connection
for Instance r-37-VM. Assuming the default connection.
2018-05-23 12:59:54,857 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:54,858 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,859 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.LibvirtComputingResource]
/usr/share/cloudstack-common/scripts/vm/network/security_group.py
destroy_network_rules_for_vm --vmname r-37-VM
2018-05-23 12:59:55,043 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Execution is successful.
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Failed to get vm :Domain not
found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:55,046 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) Try to stop the vm at first
2018-05-23 12:59:55,047 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-4:null) (logid:ef8b353e) VM r-37-VM doesn't exist, no
need to stop it
2018-05-23 12:59:55,048 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e) Request:Seq
15-5506494969390563343: { Cmd , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"
data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"hypervisorType":"KVM"}},"wait":0}}]
}
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
org.apache.cloudstack.storage.command.DeleteCommand
2018-05-23 12:59:56,941 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Trying to fetch storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:56,952 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-1:null) (logid:ef8b353e) Looking for libvirtd
connection at: qemu:///system
2018-05-23 12:59:56,960 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully refreshed pool
22173237211 Available: 1426660573184
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Attempting to remove volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,215 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Unprotecting and Removing
RBD snapshots of image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765 prior to
removing the image
2018-05-23 12:59:57,228 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully connected to
Ceph cluster at ceph-mon:6789
2018-05-23 12:59:57,290 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Fetching list of snapshots
of RBD image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:59:57,302 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Succesfully unprotected and
removed any remaining snapshots (0) of rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
Continuing to remove the RBD image
2018-05-23 12:59:57,302 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Closing image and destroying
context
2018-05-23 12:59:57,307 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) (logid:ef8b353e) Instructing libvirt to
remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,681 DEBUG [cloud.agent.Agent]
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2018-05-23 13:00:04,666 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Executing: /usr/share/cloudstack-common/s
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 13:00:04,770 DEBUG [kvm.resource.LibvirtComputingResource]
(UgentTask-3:null) (logid:) Execution is successful.
2018-05-23 13:00:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 13:00:04,774 DEBUG [cloud.agent.Agent] (UgentTask-3:null)
(logid:) Sending ping: Seq 15-3: { Cmd , MgmtId: -1, via: 15, Ver: v1,
Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"
newGroupStates":{},"_hostVmStateReport":{},"_gatewayAccessib
le":true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}]
}
2018-05-23 13:00:04,819 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null)
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
}
any ideas ?
king regards
Ben
Post by Andrija Panic
NetworkManager ? I though it was advised to not run it...
Ben,
Can you put the KVM agent in debug mode and post the logs?
sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
Then restart the agent.
o Si
________________________________
Sent: Friday, May 18, 2018 2:20 AM
To: Cloudstack Mailinglist
Subject: KVM Problem by deploying VPC
Hi Together,
im currently testing the Configuration of KVM Hosts in our Test
environment.
When i try to deploy a VPC the Hypervisor shows me the
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:12:08 kvm-test01-sb systemd: Started Network Manager
Script
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:12:08 kvm-test01-sb nm-dispatcher: req:1 'up'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:12:08 kvm-test01-sb libvirtd: 2018-05-18
6251: warning : qemuDomainObjTaint:5378 : Domain id=2
name='r-31-VM'
high-privileges
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.machine1'
unit='dbus-org.freedesktop.
machine1.service'
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual
Machine and
Container Registration Service...
May 18 09:12:08 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.machine1'
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
and
Container Registration Service.
May 18 09:12:08 kvm-test01-sb systemd-machined: New machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Started Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Starting Virtual Machine
qemu-2-r-31-VM.
May 18 09:12:08 kvm-test01-sb systemd: Unit iscsi.service
cannot be
reloaded because it is inactive.
May 18 09:12:08 kvm-test01-sb kvm: 1 guest now active
May 18 09:12:12 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44 kvm-test01-sb kernel: kvm [48292]: vcpu0
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Trying to fetch
storage pool
2258aa76-7813-354d-b274-961fb337e716 from libvirt
May 18 09:12:46 kvm-test01-sb sh: INFO
[kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:) (logid:b911ffae) Asking libvirt to
refresh
storage pool 2258aa76-7813-354d-b274-961fb337e716
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6454] manager: (enp8s0f0.233): new VLAN device
(/org/freedesktop/NetworkManager/Devices/15)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6470] device (enp8s0f0.233): carrier: link
connected
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device enp8s0f0.233
entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered blocking state
May 18 09:13:05 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered disabled state
May 18 09:13:05 kvm-test01-sb kernel: device vnet1 entered
promiscuous mode
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6648] manager: (vnet1): new Tun device
(/org/freedesktop/
NetworkManager/Devices/16)
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6662] device (vnet1): state change: unmanaged ->
unavailable
(reason 'connection-assumed', sys-iface-state: 'external')
May 18 09:13:05 kvm-test01-sb NetworkManager[925]: <info>
[1526627585.6674] device (vnet1): state change: unavailable ->
disconnected
(reason 'none', sys-iface-state: 'external')
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
disabled state
May 18 09:13:10 kvm-test01-sb kernel: device vnet0 left
promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: cloud0: port 1(vnet0)
entered
disabled state
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5339] device (vnet0): state change: activated ->
unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (cloud0): bridge port vnet0 was
detached
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5342] device (vnet0): released from master device
cloud0
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Activating
service name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
May 18 09:13:10 kvm-test01-sb systemd: Starting Network
Manager Script
Dispatcher Service...
May 18 09:13:10 kvm-test01-sb kernel: device vnet1 left
promiscuous mode
May 18 09:13:10 kvm-test01-sb kernel: brenp8s0f0-233: port
2(vnet1)
entered disabled state
May 18 09:13:10 kvm-test01-sb dbus[883]: [system] Successfully
activated
service 'org.freedesktop.nm_dispatcher'
May 18 09:13:10 kvm-test01-sb systemd: Started Network Manager
Script
Dispatcher Service.
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:13:10 kvm-test01-sb nm-dispatcher: req:1 'down'
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): state change: disconnected
-> unmanaged
(reason 'unmanaged', sys-iface-state: 'removed')
May 18 09:13:10 kvm-test01-sb NetworkManager[925]: <info>
[1526627590.5772] device (vnet1): released from master device
brenp8s0f0-233
May 18 09:13:10 kvm-test01-sb libvirtd: 2018-05-18
1519: error : qemuMonitorIO:704 : internal error: End of file
from qemu
monitor
May 18 09:13:10 kvm-test01-sb kvm: 0 guests now active
May 18 09:13:10 kvm-test01-sb systemd-machined: Machine
qemu-2-r-31-VM
terminated.
Domain not
found: no domain with matching uuid
'ff22b439-e0d0-44d1-a3cc-8dd23afb82eb'
(r-31-VM)
May 18 09:13:11 kvm-test01-sb kernel: device enp8s0f0.233 left
promiscuous
mode
May 18 09:13:11 kvm-test01-sb kernel: brenp8s0f0-233: port
1(enp8s0f0.233)
entered disabled state
May 18 09:13:11 kvm-test01-sb NetworkManager[925]: <info>
[1526627591.2137] device (enp8s0f0.233): released from master
device
brenp8s0f0-233
Domain not
found: no domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: INFO
[kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) No existing
libvirtd connection
found. Opening a new one
Domain not
found: No domain with matching name 'r-31-VM'
May 18 09:13:11 kvm-test01-sb sh: WARN
[kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:) (logid:94c38c31) Can not find a
connection for
Instance r-31-VM. Assuming the default connection.
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
Domain not
found: no domain with matching name 'r-31-VM'
2018-05-18 09:13:11,284 ERROR [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Failed to start instance
VM[DomainRouter|r-31-VM]
com.cloud.utils.exception.ExecutionException: Unable to start
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb due to error in
finalizeStart,
not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
VirtualMachineManagerImpl.java:4691)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.VmWorkJobDispatcher.runJob(
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.
java:558)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.callWithContext(DefaultManagedContext
.java:103)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext.runWithContext(DefaultManagedContext.
java:53)
at org.apache.cloudstack.managed.context.
ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.
AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-05-18 09:13:11,289 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Cleaning up resources for the vm
VM[DomainRouter|r-31-VM]
in Starting state
2018-05-18 09:13:11,291 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
Sending { Cmd , MgmtId: 109952567336, via: 10(kvm-test01-sb),
Ver: v1,
Flags: 100011, [{"com.cloud.agent.api.StopCom
mand":{"isProxy":false,
"checkBeforeCleanup":false,"forceStop":false,"vmName":"r-
31-VM","executeInSequence":false,"wait":0}}] }
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-14:null) (logid:) Seq
Processing: { Ans: , MgmtId: 109952567336, via: 10, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}]
}
2018-05-18 09:13:11,627 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1) (logid:94c38c31) Seq
10(kvm-test01-sb), Ver: v1,
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG [c.c.n.g.ControlNetworkGuru]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Released nic: NicProfile[65-31-null-null-null
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Successfully released network resources for
the vm
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG [c.c.v.VirtualMachineManagerIm
pl]
(Work-Job-Executor-30:ctx-67604a44 job-169/job-170
ctx-ec9dfbf1)
(logid:94c38c31) Successfully cleanued up resources for the vm
VM[DomainRouter|r-31-VM] in Starting state
Deployment of Isolatet Networks and redundant VPCs, works
wothout any
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal (libvirt-3.9.0-14.el7_5.2.x86_64,
qemu-kvm-1.5.3-156.el7.x86_64, kernel-3.10.0-693.el7.x86_64)
Someone got an i idea ?
Kind regards
Ben
--
Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt
vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht
der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben,
dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die
eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den
Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und
löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
--
Andrija Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben, dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
This email has been sent by Oakford Technology Limited, while we have checked this e-mail and any attachments for viruses, we can not guarantee that they are virus-free. You must therefore take full responsibility for virus checking.
This message and any attachments are confidential and should only be read by those to whom they are addressed. If you are not the intended recipient, please contact us, delete the message from your computer and destroy any copies. Any distribution or copying without our prior permission is prohibited.
Internet communications are not always secure and therefore Oakford Technology Limited does not accept legal responsibility for this message. The recipient is responsible for verifying its authenticity before acting on the contents. Any views or opinions presented are solely those of the author and do not necessarily represent those of Oakford Technology Limited.
Registered address: Oakford Technology Limited, 10 Prince Maurice Court, Devizes, Wiltshire. SN10 2RT.
Registered in England and Wales No. 5971519
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
E-Mail: ***@coders-area.de
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben, dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
Andrija Panic
2018-05-25 14:41:21 UTC
Permalink
Hi Benjamin,

good that you solved it - now, I can propose you update the documentation
over here :) if you have time:
http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/latest/hypervisor/kvm.html
Just write special requirement for the Centos7 Basic install.

Cheers
Post by Benjamin Naber
Hi Adam,
got it ! the problem is this tiny script: /usr/share/cloudstack-common/
scripts/vm/network/vnet/modifyvlan.sh
at this lines you can see that ifconfig is used to bring bridges up.
ifconfig |grep -w $vlanBr > /dev/null
if [ $? -gt 0 ]
then
ifconfig $vlanBr up
fi
return 0
the simple solution for centos7: just install the package net-tools over
package manager! Centos in basic installation have no ifconfig installed.
Ubuntu and Debian derivates have it.
Kind regards and thanks for all impressions
Ben
Post by Andrija Panic
Hi Ben
" Now i moved back to centos7 and have another problem. all vlan
bridges that will be created by cloudstack-agent not manual get in ifup
state. if i ifup the vlan bridges manual all works fine. Have someone a
idea how i can force centos to automaticly bring up new if devices ?"
Post by Andrija Panic
I had this issue with 4.9 and ending up writing a dirty script to
check the down bridges every minute via cron and if up them.
Post by Andrija Panic
iface=/usr/sbin/ip add | grep -E &quot;brbond.*DOWN&quot; | awk -F
&#39;:&#39; &#39;{print $2}&#39;
Post by Andrija Panic
echo "$iface" | while read -r a; do /usr/sbin/ip link set dev $a up;
done
Post by Andrija Panic
Thanks
Adam
-----Original Message-----
Sent: 24 May 2018 14:37
Subject: Re: KVM Problem by deploying VPC
** This mail originated from OUTSIDE the Oakford corporate network.
Treat hyperlinks and attachments in this email with caution. **
Post by Andrija Panic
Hi Andrija,
ty for reply. i solved the error. Now i have an other error. Problem
was that it failed witout a bonding device. i created a bonding device
(bond0) wiht a single nic now it works without problems On Ubuntu.
Post by Andrija Panic
auto enp8s0f0
iface enp8s0f0 inet manual
auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports enp8s0f0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
auto enp8s0f0
iface enp8s0f0 inet manual
bond-master bond0
auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none
auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
Now i moved back to centos7 and have another problem. all vlan
bridges that will be created by cloudstack-agent not manual get in ifup
state. if i ifup the vlan bridges manual all works fine. Have someone a
idea how i can force centos to automaticly bring up new if devices ?
Post by Andrija Panic
Kind regards
Ben
Post by Andrija Panic
Hi Ben,
2018-05-23 12:59:47,213 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) getting
broadcast uri for
Post by Andrija Panic
Post by Andrija Panic
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
FileConfigItem,
Post by Andrija Panic
Post by Andrija Panic
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
/usr/share/cloudstack-common/scripts/network/domr/router_
proxy.sh
Post by Andrija Panic
Post by Andrija Panic
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
What I believe we see here, is that IP association fails for
some reason
Post by Andrija Panic
Post by Andrija Panic
(exit value is 1) and after that ACS will just stop the VM and
do the
Post by Andrija Panic
Post by Andrija Panic
cleanup (all log lines after this one)...
Can you check your broadcast URI in the DB ? ACtually... I see
it vlan 223
Post by Andrija Panic
Post by Andrija Panic
* I did have some issues in past releases, when we wanted
to use untagged
Post by Andrija Panic
Post by Andrija Panic
vlans for Public network, but it seems not to be case
here...
Post by Andrija Panic
Post by Andrija Panic
Not sure, if it's possible that you also SSH to the VR
during this creation
Post by Andrija Panic
Post by Andrija Panic
process, in order to collect logs from inside the VR -
before Qemu destroys
Post by Andrija Panic
Post by Andrija Panic
the VR?
and then try to fetch
Post by Andrija Panic
Post by Andrija Panic
(SCP or something) whole /var/log/ folder to the
localhost - from there,
Post by Andrija Panic
Post by Andrija Panic
there is cloud.log and auth.log where most of the
commands outputs are
Post by Andrija Panic
Post by Andrija Panic
located (success or failure)
or something like rsync -av -e "ssh -p 3992 -i
.ssh.id_rsa.cloud"
Post by Andrija Panic
Post by Andrija Panic
/local/dir/
and keep repeating until you fetch whatever data to be
able to analyze it
Post by Andrija Panic
Post by Andrija Panic
Perhaps someone else will have better suggestion...
Best
Andrija
On 23 May 2018 at 13:08, Benjamin Naber <
Post by Benjamin Naber
Hi Andrija,
first of all thanks for your reply. I have now
testet the setup on a
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Ubuntu Xenial. Same issu with Default VPC
Offering. Redundant VPC offering
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
works also without any problems. Same as on CentOS.
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407)
Proccess agent ready
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Set
agent id 15
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407)
agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 110,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"com.cloud.agent.api.
ReadyAnswer":{"result":true,"wait":0}}] }
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407)
Request:Seq
109952567336, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100111, [{"com.cloud.agent.api.ReadyCo
mmand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407)
com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407)
Proccess agent ready
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Set
agent id 15
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407)
agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 110,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"com.cloud.agent.api.
ReadyAnswer":{"result":true,"wait":0}}] }
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e)
Request:Seq
109952567336, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100111, [{"org.apache.cloudstack.
storage.command.CopyCommand":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
srcTO":{"org.apache.cloudstack.storage.to.
{"path":"6129c76d-67e5-4454-
9bc9-4e9a5a152610","origUrl":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
http://packages.shapeblue.com.
s3-eu-west-1.amazonaws.com/sys
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
temvmtemplate/4.10/dnsmasq/
systemvm64template-4.10-kvm.qcow2.bz2
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
","uuid":"6129c76d-67e5-4454-
3,"format":"QCOW2","accountId"
:1,"checksum":"bc2eac46f16a2ec
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM
Template
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(KVM)","imageDataStore":{"org.
apache.cloudstack.storage.to.P
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rimaryDataStoreTO":{"uuid":"
2258aa76-7813-354d-b274-961fb337
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
e716","id":14,"poolType":"RBD"
,"host":"ceph-mon","path":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rbd","port":6789,"url":"RBD://
ceph-mon/rbd/?ROLE=Primary&
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
STOREUUID=2258aa76-7813-354d-
b274-961fb337e716","isManaged"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:false}},"name":"routing-3","
hypervisorType":"KVM"}},"destTO
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
":{"org.apache.cloudstack.
storage.to.VolumeObjectTO":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"
org.apache.cloudstack.storage.to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"
poolType":"RBD","host":"ceph-mon
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
isManaged":false}},"name":"
ROOT-37","size":349945344,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
volumeId":37,"vmName":"r-37-
VM","accountId":2,"format":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
QCOW2","provisioningType":"
THIN","id":37,"deviceId":0,"hy
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
pervisorType":"KVM"}},"executeInSequence":true,"
options":{},"options2":{},"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e)
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Trying to fetch storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Succesfully refreshed pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Trying to fetch storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Succesfully refreshed pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Creating volume
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template
6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool
2258aa76-7813-354d-b274-961fb337e716 (RBD) with
size 349945344
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,225 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Trying to perform a RBD
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
clone (layering) since we are operating in the
same storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,270 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Succesfully connected to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) The
source image
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD
format 2. We will perform
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Checking if RBD snapshot
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rbd/6129c76d-67e5-4454-9bc9-
prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Found 1 snapshots on RBD
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) RBD
snapshot
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rbd/6129c76d-67e5-4454-9bc9-
exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Succesfully cloned
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
6129c76d-67e5-4454-9bc9-
fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 110,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"org.apache.cloudstack.
storage.command.CopyCmdAnswer":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
newData":{"org.apache.cloudstack.storage.to.
{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","accountId":0,"
format":"RAW","id":0}},"result":true,"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e)
Request:Seq
109952567336, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100111, [{"com.cloud.agent.api.StartCo
mmand":{"vm":{"id":37,"name":"
r-37-VM","type":"DomainRouter"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(64-bit)","platformEmulator":"Debian GNU/Linux
5","bootArgs":" vpccidr=
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
10.59.0.0/21 domain=test.internal dns1=8.8.8.8
privategateway=None
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
template=domP name=r-37-VM eth0ip=169.254.2.247
eth0mask=255.255.0.0
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
type=vpcrouter disable_rp_filter=true
baremetalnotificationsecurityk
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_
lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
baremetalnotificationapikey=
6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ
host=172.16.11.56
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
port=8080","enableHA":true,"limitCpuUse":false,"
enableDynami
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
callyScaleVm":false,"
vncPassword":"-Wf7MGedC0usYmBc
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Cscv7A","params":{},"uuid":"
87a45b1c-bf35-431e-82a0-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
c0c961feab05","disks":[{"data":{"
org.apache.cloudstack.storage.to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
volumeType":"ROOT","dataStore":{"
org.apache.cloudstack.storage.to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"
poolType":"RBD","host":"ceph-mon
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
isManaged":false}},"name":"
ROOT-37","size":349945344,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
path":"fa3dba5d-364b-43ce-
37,"vmName":"r-37-VM","
accountId":2,"format":"RAW","p
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rovisioningType":"THIN","id":37,"deviceId":0,"
bytesReadRate"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:0,"bytesWriteRate":0,"iopsReadRate":0,"
iopsWriteRate":0,"hy
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
pervisorType":"KVM"}},"
diskSeq":0,"path":"fa3dba5d-364b-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
43ce-9ebf-d15a3324f765","type"
:"ROOT","_details":{"storageHo
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
st":"ceph-mon","managed":"
false","storagePort":"6789","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
volumeSize":"349945344"}}],"nics":[{"deviceId":0,"
networkRateMbps":-1,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
defaultNic":false,"pxeDisable"
:true,"nicUuid":"226f4319-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
b15c-4e48-8a74-6a28b574a992","
uuid":"76d57f99-f4af-42b8-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
af0a-41f0a2cd0ba5","ip":"169.
254.2.247","netmask":"255.255.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
0.0","gateway":"169.254.0.1","
mac":"0e:00:a9:fe:02:f7","broa
"Control","isSecurityGroupEnab
{}},"hostIp":"10.253.250.230",
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
"executeInSequence":false,"
wait":0}},{"com.cloud.agent.api.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
check.CheckSshCommand":{"ip":"
169.254.2.247","port":3922,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
interval":6,"retries":100,"
name":"r-37-VM","wait":0}},{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
com.cloud.agent.api.GetDomRVersionCmd":{"
accessDetails":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
router.ip":"169.254.2.247","router.name
0}},{"com.cloud.agent.api.
PlugNicCommand":{"nic":{"deviceId"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:1,"networkRateMbps":200,"
true,"nicUuid":"29bd4c33-47dd-
4357-aa3d-8c5ae56169b5","uuid"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:"787cf778-230f-4e17-a648-
f63171bdc81c","ip":"82.119.177.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
188","netmask":"255.255.255.
240","gateway":"82.119.177.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
177","mac":"1e:00:87:00:00:0d"
,"broadcastType":"Vlan","type"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:"Public","broadcastUri":"
vlan://233","isolationUri":"vl
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
an://233","isSecurityGroupEnabled":false}
,"instanceName":"r-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
37-VM","vmType":"DomainRouter"
,"details":{},"wait":0}},{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
com.cloud.agent.api.routing.IpAssocVpcCommand":{"
ipAddresses
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
":[{"accountId":2,"publicIp":"
82.X.X.X","sourceNat":true,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
add":true,"oneToOneNat":false,
"vlan://233","vlanGateway":"
82.119.X.X","vlanNetmask":"255.255.255.240","
00:0d","networkRate":200,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
trafficType":"Public","newNic"
:false}],"accessDetails":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
router.guest.ip":"82.X.X.X","
router.ip":"169.254.2.247","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
zone.network.type":"Advanced","router.name
":"r-37-VM"},"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
wait":0}},{"com.cloud.agent.
api.routing.SetSourceNatComman
2,"publicIp":"82.X.X.X","sourc
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
eNat":true,"add":true,"
oneToOneNat":false,"firstIP":true,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
broadcastUri":"vlan://233","
vlanGateway":"82.X.X.X","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
vlanNetmask":"255.255.255.240"
00:0d","networkRate":200,"
trafficType":"Public","newNic"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:false},"add":true,"accessDetails":{"router.ip":"
169.254.2.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
247","zone.network.type":"Advanced","router.name
":"r-37-VM"},"wait":0}},{}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e)
com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.
LibvirtStartCommandWrapper]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
remaining mem36873936896
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,649 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to fetch storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Succesfully refreshed pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not find volume
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage
volume not found: no
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
storage vol with matching name
'fa3dba5d-364b-43ce-9ebf-d15a3324f765'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,664 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Refreshing storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Found volume
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage
pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 after
refreshing the pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:44,204 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to fetch storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Succesfully refreshed pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) (logid:ef8b353e)
Executing: /bin/bash -c ip
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
route | grep 169.254.0.0/16
2018-05-23 <http://169.254.0.0/162018-05-23>
12:58:44,248 DEBUG
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[utils.script.Script]
(agentRequest-Handler-5:null) (logid:ef8b353e)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.
LibvirtStartCommandWrapper]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
starting r-37-VM: <domain type='kvm'>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0
(64-bit)</description>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software
Foundation</entry>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<entry name='product'>CloudStack KVM
Hypervisor</entry>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<entry name='uuid'>87a45b1c-bf35-
431e-82a0-c0c961feab05</entry>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd'
name='rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765'>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-
961fb337e716'/>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-
common/vms/systemvm.iso'/>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes'
listen='10.253.250.230' passwd='-Wf7MGedC0usYmBcCscv7A'/>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-
37-VM.agent'/>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-
37-VM.org.qemu.guest_agent.0'/>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
<address type='virtio-serial'/>
<target type='virtio'
name='org.qemu.guest_agent.0'/>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>
2018-05-23 12:58:46,460 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/hypervisor/kvm/patchviasocket.py
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%
name=r-37-VM%eth0ip=169.254.2.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
247%eth0mask=255.255.0.0%type=
vpcrouter%disable_rp_filter=
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
true%baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpa
sA7dmX9UtCRILq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
RjLA%baremetalnotificationapikey=
6a0ehX9xAk7wTIm4knUtIY0Qd1e
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
95U03mXqoNnZD42UMvKAsZJxpOgnVn
wRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:46,498 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:58:49,567 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:58:54,568 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:58:57,630 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:02,631 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Unable to logon to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/hypervisor/kvm/patchviasocket.py
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%
name=r-37-VM%eth0ip=169.254.2.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
247%eth0mask=255.255.0.0%type=
vpcrouter%disable_rp_filter=
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
true%baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpa
sA7dmX9UtCRILq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
RjLA%baremetalnotificationapikey=
6a0ehX9xAk7wTIm4knUtIY0Qd1e
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
95U03mXqoNnZD42UMvKAsZJxpOgnVn
wRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:02,669 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.
LibvirtComputingResource]
/usr/share/cloudstack-common/s
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cripts/vm/network/security_group.py
get_rule_logs_for_vms
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:80cd5d21)
com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(UgentTask-2:null) (logid:) Execution is
successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:04,772 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(UgentTask-2:null) (logid:) Looking for libvirtd
qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent]
(UgentTask-2:null)
-1, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 11, [{"com.cloud.agent.api.
PingRoutingWithNwGroupsCommand":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
newGroupStates":{},"_
"PowerOn","host":"kvm-test01-
true,"_vnetAccessible":true,"
hostType":"Routing","hostId":15,"wait":0}}] }
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent]
(Agent-Handler-3:null)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(logid:) Received response: Seq 15-2: { Ans: ,
15, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.PingAns
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
wer":{"_command":{"hostType":"
Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:59:05,726 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:10,727 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:12,767 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:275957d4)
com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4)
Trying to fetch storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4)
Asking libvirt to refresh
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4)
Succesfully refreshed pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Unable to logon to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/hypervisor/kvm/patchviasocket.py
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%
name=r-37-VM%eth0ip=169.254.2.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
247%eth0mask=255.255.0.0%type=
vpcrouter%disable_rp_filter=
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
true%baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpa
sA7dmX9UtCRILq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
RjLA%baremetalnotificationapikey=
6a0ehX9xAk7wTIm4knUtIY0Qd1e
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
95U03mXqoNnZD42UMvKAsZJxpOgnVn
wRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:17,803 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:17,804 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:22,804 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:38,239 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Could not connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,239 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Unable to logon to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/hypervisor/kvm/patchviasocket.py
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%
name=r-37-VM%eth0ip=169.254.2.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
247%eth0mask=255.255.0.0%type=
vpcrouter%disable_rp_filter=
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
true%baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpa
sA7dmX9UtCRILq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
RjLA%baremetalnotificationapikey=
6a0ehX9xAk7wTIm4knUtIY0Qd1e
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
95U03mXqoNnZD42UMvKAsZJxpOgnVn
wRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:43,285 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e)
com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG
[resource.wrapper.LibvirtOvsVp
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-5:null)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(logid:ef8b353e) Ping command port,
169.254.2.247:3922
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:43,286 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Trying to connect to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,287 DEBUG
[resource.wrapper.LibvirtOvsVp
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-5:null)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(logid:ef8b353e) Ping command port succeeded for
vm r-37-VM
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e)
com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/network/domr/router_proxy.sh
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e)
com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/network/vnet/modifyvlan.sh -v 233
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
-p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e)
com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
getting broadcast uri for
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Transforming
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
com.cloud.agent.api.routing.IpAssocVpcCommand to
ConfigItems
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:47,581 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Processing FileConfigItem,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
copying 257 characters to ip_associations.json
took 340ms
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:47,582 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/network/domr/router_proxy.sh
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Exit value is 1
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:47,767 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Processing incoming file =>
ip_associations.json[INFO] Processing JSON file
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ip_associations.json
2018-05-23 12:59:47,768 DEBUG
[resource.virtualnetwork.VirtualRoutingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
Processing ScriptConfigItem,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
executing update_config.py ip_associations.json
took 186ms
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 110,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"com.cloud.agent.api.
StartAnswer":{"vm":{"id":37,"name":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
r-37-VM","type":"DomainRouter"
,"cpus":1,"minSpeed":500,"
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian
GNU/Linux
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
5","bootArgs":" vpccidr=10.59.0.0/21
domain=test.internal dns1=8.8.8.8
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
privategateway=None template=domP name=r-37-VM
eth0ip=169.254.2.247
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
eth0mask=255.255.0.0 type=vpcrouter
disable_rp_filter=true
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
baremetalnotificationsecurityk
ey=Gs_GstLMdaO8CWiW_-wN71Xu0Rh
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
IGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpa
sA7dmX9UtCRILqRjLA
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
baremetalnotificationapikey=
6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ
host=172.16.11.56
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
port=8080","enableHA":true,"limitCpuUse":false,"
enableDynami
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
callyScaleVm":false,"
vncPassword":"-Wf7MGedC0usYmBc
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Cscv7A","vncAddr":"10.253.250.
230","params":{},"uuid":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
87a45b1c-bf35-431e-82a0-
c0c961feab05","disks":[{"data":{"org.apache.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cloudstack.storage.to.VolumeObjectTO":{"uuid":"
fa3dba5d-364b-43ce-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
9ebf-d15a3324f765","volumeType":"ROOT","dataStore"
:{"org.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
apache.cloudstack.storage.to.PrimaryDataStoreTO"
:{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"po
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
olType":"RBD","host":"ceph-
mon","path":"rbd","port":6789,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
url":"RBD://ceph-mon/rbd/?ROLE=Primary&STOREUUID=
2258aa76-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
7813-354d-b274-961fb337e716","
isManaged":false}},"name":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ROOT-37","size":349945344,"
path":"fa3dba5d-364b-43ce-
37,"vmName":"r-37-VM","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
accountId":2,"format":"RAW","
provisioningType":"THIN","id":3
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
7,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsRea
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
dRate":0,"iopsWriteRate":0,"
hypervisorType":"KVM"}},"diskSeq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
":0,"path":"fa3dba5d-364b-
43ce-9ebf-d15a3324f765","type"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:"ROOT","_details":{"storageHost":"ceph-mon","
managed":"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
false","storagePort":"6789","
volumeSize":"349945344"}}],"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
nics":[{"deviceId":0,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:true,"nicUuid":"226f4319-
b15c-4e48-8a74-6a28b574a992","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
uuid":"76d57f99-f4af-42b8-
af0a-41f0a2cd0ba5","ip":"169.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
254.2.247","netmask":"255.255.
0.0","gateway":"169.254.0.1","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
mac":"0e:00:a9:fe:02:f7","
"Control","isSecurityGroupEnabled":false}
{}},"result":true,"wait":0}},{
"com.cloud.agent.api.check.Che
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ckSshAnswer":{"result":true,"
wait":0}},{"com.cloud.agent.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack
Release 4.10.0
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Mon Oct 9 06:45:25 UTC
2017","scriptsVersion":"748f68
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
e99144d82158a4c4d3dcbfef04\n",
"result":true,"details":"Cloudstack Release
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
4.10.0 Mon Oct 9 06:45:25 UTC
2017&748f68e99144d82158a4c4d3d
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cbfef04\n","wait":0}},{"com.
cloud.agent.api.PlugNicAnswer"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
:{"result":true,"details":"
success","wait":0}},{"com.cloud.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
agent.api.routing.GroupAnswer":{"results":["null
- success: null","null -
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
failed: [INFO] update_config.py :: Processing
incoming file =>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
ip_associations.json[INFO] Processing JSON file
ip_associations.json"],"
result":false,"wait":0}},{"com.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cloud.agent.api.Answer":{"
result":false,"details":"Stopped by previous
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
failure","wait":0}},{"com.
cloud.agent.api.Answer":{"result":false,"details":"Stopped
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
by previous failure","wait":0}}] }
2018-05-23 12:59:47,835 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Request:Seq
109952567336, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100011, [{"com.cloud.agent.api.
StopCommand":{"isProxy":false,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
checkBeforeCleanup":false,"
forceStop":false,"vmName":"r-37-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:47,836 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e)
com.cloud.agent.api.StopCommand
2018-05-23 12:59:47,836 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:47,854 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/network/security_group.py
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
destroy_network_rules_for_vm --vmname r-37-VM
--vif vnet0
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:48,041 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:48,044 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
Cleaning the metadata of vm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
snapshots of vm r-37-VM
2018-05-23 12:59:48,045 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e) Try
to stop the vm at first
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,585 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
successfully shut down vm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
r-37-VM
2018-05-23 12:59:54,586 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Executing: /bin/bash -c ls
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
/sys/class/net/cloud0
2018-05-23 12:59:54,591 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,593 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Executing: /bin/bash -c ls
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
/sys/class/net/cloud0/brif | tr '
' ' '
2018-05-23 12:59:54,598 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,600 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
unable to get a vNet ID from
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
name cloud0
2018-05-23 12:59:54,600 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Executing: /bin/bash -c ls
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
/sys/class/net/brenp8s0f0-233
2018-05-23 12:59:54,604 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,605 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Executing: /bin/bash -c ls
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
/sys/class/net/brenp8s0f0-233/brif | tr '
' ' '
2018-05-23 12:59:54,610 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,612 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/network/vnet/modifyvlan.sh -o
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
delete -v 233 -p enp8s0f0 -b brenp8s0f0-233
2018-05-23 12:59:54,763 DEBUG
[kvm.resource.BridgeVifDriver]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-3:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,766 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 10,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"com.cloud.agent.api.
StopAnswer":{"result":true,"wait":0}}] }
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e)
Request:Seq
109952567336, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100011, [{"com.cloud.agent.api.
StopCommand":{"isProxy":false,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
checkBeforeCleanup":false,"
forceStop":false,"vmName":"r-37-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
VM","executeInSequence":false,"wait":0}}] }
2018-05-23 12:59:54,852 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e)
com.cloud.agent.api.StopCommand
2018-05-23 12:59:54,853 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:54,854 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Can
not find KVM connection
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,854 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: lxc:///
2018-05-23 12:59:54,854 INFO [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) No
existing libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection found. Opening a new one
2018-05-23 12:59:54,856 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Successfully connected to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
libvirt at: lxc:///
2018-05-23 12:59:54,857 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Can
not find LXC connection
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
for Instance: r-37-VM, continuing.
2018-05-23 12:59:54,857 WARN [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Can
not find a connection
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
for Instance r-37-VM. Assuming the default
connection.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:54,857 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:54,858 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
org.libvirt.LibvirtException: Domain not found: no
domain with matching
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
name 'r-37-VM'
2018-05-23 12:59:54,859 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
org.libvirt.LibvirtException: Domain not found: no
domain with matching
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
org.libvirt.LibvirtException: Domain not found: no
domain with matching
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
name 'r-37-VM'
2018-05-23 12:59:54,860 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
/usr/share/cloudstack-common/
scripts/vm/network/security_group.py
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
destroy_network_rules_for_vm --vmname r-37-VM
2018-05-23 12:59:55,043 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Execution is successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:55,046 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e)
Failed to get vm :Domain not
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
found: no domain with matching name 'r-37-VM'
2018-05-23 12:59:55,046 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Try
to stop the vm at first
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:55,047 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) VM
r-37-VM doesn't exist, no
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
need to stop it
2018-05-23 12:59:55,048 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 10,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"com.cloud.agent.api.
StopAnswer":{"result":true,"wait":0}}] }
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e)
Request:Seq
109952567336, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100011, [{"org.apache.cloudstack.
storage.command.DeleteCommand":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
data":{"org.apache.cloudstack.
storage.to.VolumeObjectTO":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"
org.apache.cloudstack.storage.to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"
poolType":"RBD","host":"ceph-mon
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
","path":"rbd","port":6789,"
url":"RBD://ceph-mon/rbd/?ROLE
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
=Primary&STOREUUID=2258aa76-
7813-354d-b274-961fb337e716","
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
isManaged":false}},"name":"
ROOT-37","size":349945344,"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
path":"fa3dba5d-364b-43ce-
37,"vmName":"r-37-VM","
accountId":2,"format":"RAW","p
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
rovisioningType":"THIN","id":37,"deviceId":0,"
hypervisorType":"KVM"}},"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:59:56,940 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e)
org.apache.cloudstack.storage.
command.DeleteCommand
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:56,941 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Trying to fetch storage pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:56,952 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Looking for libvirtd
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:56,960 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Succesfully refreshed pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
22173237211 Available: 1426660573184
2018-05-23 12:59:57,215 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Attempting to remove volume
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from pool
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,215 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Unprotecting and Removing
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
RBD snapshots of image rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
prior to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
removing the image
2018-05-23 12:59:57,228 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Succesfully connected to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Ceph cluster at ceph-mon:6789
2018-05-23 12:59:57,290 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Fetching list of snapshots
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
of RBD image rbd/fa3dba5d-364b-43ce-9ebf-
d15a3324f765
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:57,302 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Succesfully unprotected and
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
removed any remaining snapshots (0) of
rbd/fa3dba5d-364b-43ce-9ebf-d15a3324f765
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Continuing to remove the RBD image
2018-05-23 12:59:57,302 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Closing image and destroying
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
context
2018-05-23 12:59:57,307 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-1:null) (logid:ef8b353e)
Instructing libvirt to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
remove volume fa3dba5d-364b-43ce-9ebf-d15a3324f765
from pool
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:57,681 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1,
Flags: 10,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}]
}
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 13:00:04,666 DEBUG [kvm.resource.
LibvirtComputingResource]
/usr/share/cloudstack-common/s
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
cripts/vm/network/security_group.py
get_rule_logs_for_vms
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 13:00:04,770 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(UgentTask-3:null) (logid:) Execution is
successful.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 13:00:04,772 DEBUG [kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(UgentTask-3:null) (logid:) Looking for libvirtd
qemu:///system
2018-05-23 13:00:04,774 DEBUG [cloud.agent.Agent]
(UgentTask-3:null)
-1, via: 15, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Flags: 11, [{"com.cloud.agent.api.
PingRoutingWithNwGroupsCommand":{"
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
newGroupStates":{},"_hostVmStateReport":{},"_
gatewayAccessib
true,"hostType":"Routing","hostId":15,"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 13:00:04,819 DEBUG [cloud.agent.Agent]
(Agent-Handler-3:null)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
(logid:) Received response: Seq 15-3: { Ans: ,
15, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.PingAns
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
wer":{"_command":{"hostType":"
Routing","hostId":15,"wait":0},"result":true,"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
}
any ideas ?
king regards
Ben
hat am 21. Mai 2018 um
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
NetworkManager ? I though it was advised to
not run it...
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
On 18 May 2018 at 16:11, Simon Weller
Ben,
Can you put the KVM agent in debug
mode and post the logs?
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
sed -i 's/INFO/DEBUG/g'
/etc/cloudstack/agent/log4j-cloud.xml
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Then restart the agent.
o Si
________________________________
From: Benjamin Naber <
Sent: Friday, May 18, 2018 2:20
AM
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
To: Cloudstack Mailinglist
Subject: KVM Problem by
deploying VPC
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Hi Together,
im currently testing the
Configuration of KVM Hosts in our Test
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
environment.
When i try to deploy a VPC the
Hypervisor shows me the
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:08 kvm-test01-sb
dbus[883]: [system] Successfully
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
activated
service 'org.freedesktop.nm_
dispatcher'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:08 kvm-test01-sb
systemd: Started Network Manager
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Script
Dispatcher Service.
May 18 09:12:08 kvm-test01-sb
nm-dispatcher: req:1 'up'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:12:08
kvm-test01-sb nm-dispatcher: req:1 'up'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:12:08
kvm-test01-sb libvirtd: 2018-05-18
qemuDomainObjTaint:5378 : Domain id=2
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
name='r-31-VM'
high-privileges
May 18 09:12:08
kvm-test01-sb dbus[883]: [system] Activating
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
service
name='org.freedesktop.machine1'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
unit='dbus-org.freedesktop.
machine1.service'
May 18 09:12:08
kvm-test01-sb systemd: Starting Virtual
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Machine and
Container Registration
Service...
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:08
kvm-test01-sb dbus[883]: [system] Successfully
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
activated
service
'org.freedesktop.machine1'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:08
kvm-test01-sb systemd: Started Virtual Machine
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
and
Container Registration
Service.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:08
kvm-test01-sb systemd-machined: New machine
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
qemu-2-r-31-VM.
May 18 09:12:08
kvm-test01-sb systemd: Started Virtual Machine
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
qemu-2-r-31-VM.
May 18 09:12:08
kvm-test01-sb systemd: Starting Virtual Machine
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
qemu-2-r-31-VM.
May 18 09:12:08
kvm-test01-sb systemd: Unit iscsi.service
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
cannot be
reloaded because it is
inactive.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:08
kvm-test01-sb kvm: 1 guest now active
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:12
kvm-test01-sb kernel: kvm [48292]: vcpu0
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:44
kvm-test01-sb kernel: kvm [48292]: vcpu0
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
disabled perfctr
wrmsr: 0xc1 data 0xabcd
May 18 09:12:46
kvm-test01-sb sh: INFO
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(agentRequest-Handler-5:)
(logid:b911ffae) Trying to fetch
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
storage pool
2258aa76-7813-354d-b274-961fb337e716
from libvirt
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:12:46
kvm-test01-sb sh: INFO
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(agentRequest-Handler-5:)
(logid:b911ffae) Asking libvirt to
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
refresh
storage pool
2258aa76-7813-354d-b274-961fb337e716
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:05
kvm-test01-sb NetworkManager[925]: <info>
(enp8s0f0.233): new VLAN device
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(/org/freedesktop/
NetworkManager/Devices/15)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:05
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627585.6470] device
(enp8s0f0.233): carrier: link
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
connected
May 18 09:13:05
kvm-test01-sb kernel: brenp8s0f0-233: port
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
1(enp8s0f0.233)
entered blocking state
May 18 09:13:05
kvm-test01-sb kernel: brenp8s0f0-233: port
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
1(enp8s0f0.233)
entered disabled state
May 18 09:13:05
kvm-test01-sb kernel: device enp8s0f0.233
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
entered
promiscuous mode
May 18 09:13:05
kvm-test01-sb kernel: brenp8s0f0-233: port
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2(vnet1)
entered blocking state
May 18 09:13:05
kvm-test01-sb kernel: brenp8s0f0-233: port
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2(vnet1)
entered disabled state
May 18 09:13:05
kvm-test01-sb kernel: device vnet1 entered
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
promiscuous mode
May 18 09:13:05
kvm-test01-sb NetworkManager[925]: <info>
(vnet1): new Tun device
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(/org/freedesktop/
NetworkManager/Devices/16)
May 18 09:13:05
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627585.6662] device
(vnet1): state change: unmanaged ->
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
unavailable
(reason
'connection-assumed', sys-iface-state: 'external')
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:05
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627585.6674] device
(vnet1): state change: unavailable ->
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
disconnected
(reason 'none',
sys-iface-state: 'external')
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:10
kvm-test01-sb kernel: cloud0: port 1(vnet0)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
entered
disabled state
May 18 09:13:10
kvm-test01-sb kernel: device vnet0 left
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
promiscuous mode
May 18 09:13:10
kvm-test01-sb kernel: cloud0: port 1(vnet0)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
entered
disabled state
May 18 09:13:10
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627590.5339] device
(vnet0): state change: activated ->
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
unmanaged
(reason 'unmanaged',
sys-iface-state: 'removed')
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:10
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627590.5342] device
(cloud0): bridge port vnet0 was
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
detached
May 18 09:13:10
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627590.5342] device
(vnet0): released from master device
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
cloud0
May 18 09:13:10
kvm-test01-sb dbus[883]: [system] Activating
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
service
name='org.freedesktop.nm_dispatcher'
unit='dbus-org.freedesktop.nm-dispatcher.service'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:10
kvm-test01-sb systemd: Starting Network
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Manager Script
Dispatcher Service...
May 18 09:13:10
kvm-test01-sb kernel: device vnet1 left
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
promiscuous mode
May 18 09:13:10
kvm-test01-sb kernel: brenp8s0f0-233: port
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2(vnet1)
entered disabled state
May 18 09:13:10
kvm-test01-sb dbus[883]: [system] Successfully
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
activated
service
'org.freedesktop.nm_dispatcher'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:10
kvm-test01-sb systemd: Started Network Manager
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Script
Dispatcher Service.
May 18 09:13:10
kvm-test01-sb nm-dispatcher: req:1 'down'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[vnet0]: new
Post by Andrija Panic
request (4 scripts)
May 18 09:13:10
kvm-test01-sb nm-dispatcher: req:1 'down'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
[vnet0]: start
Post by Andrija Panic
running ordered scripts...
May 18 09:13:10
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627590.5772] device
(vnet1): state change: disconnected
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
-> unmanaged
(reason 'unmanaged',
sys-iface-state: 'removed')
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:10
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627590.5772] device
(vnet1): released from master device
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
brenp8s0f0-233
May 18 09:13:10
kvm-test01-sb libvirtd: 2018-05-18
qemuMonitorIO:704 : internal error: End of file
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
from qemu
monitor
May 18 09:13:10
kvm-test01-sb kvm: 0 guests now active
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:10
kvm-test01-sb systemd-machined: Machine
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
qemu-2-r-31-VM
terminated.
May 18 09:13:11
Domain not
found: no domain with
matching uuid
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
'ff22b439-e0d0-44d1-a3cc-
8dd23afb82eb'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(r-31-VM)
May 18 09:13:11
kvm-test01-sb kernel: device enp8s0f0.233 left
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
promiscuous
mode
May 18 09:13:11
kvm-test01-sb kernel: brenp8s0f0-233: port
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
1(enp8s0f0.233)
entered disabled state
May 18 09:13:11
kvm-test01-sb NetworkManager[925]: <info>
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[1526627591.2137] device
(enp8s0f0.233): released from master
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
device
brenp8s0f0-233
May 18 09:13:11
Domain not
found: no domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
kvm-test01-sb sh: INFO
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(agentRequest-Handler-3:)
(logid:94c38c31) No existing
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
libvirtd connection
found. Opening a new one
May 18 09:13:11
Domain not
found: No domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
kvm-test01-sb sh: WARN
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
[kvm.resource.
LibvirtConnection]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(agentRequest-Handler-3:)
(logid:94c38c31) Can not find a
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
connection for
Instance r-31-VM. Assuming
the default connection.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
Domain not
found: no domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
Domain not
found: no domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
Domain not
found: no domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
Domain not
found: no domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
May 18 09:13:11
Domain not
found: no domain with
matching name 'r-31-VM'
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2018-05-18 09:13:11,284 ERROR
[c.c.v.VirtualMachineManagerIm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
pl]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ctx-ec9dfbf1)
(logid:94c38c31) Failed to start
instance
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VM[DomainRouter|r-31-VM]
Unable to start
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VM:ff22b439-e0d0-44d1-a3cc-8dd23afb82eb
due to error in
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
finalizeStart,
not retrying
at com.cloud.vm.
VirtualMachineManagerImpl.orchestrateStart(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VirtualMachineManagerImpl.java:1099)
at com.cloud.vm.
VirtualMachineManagerImpl.orchestrateStart(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VirtualMachineManagerImpl.java:4691)
at sun.reflect.
NativeMethodAccessorImpl.invoke0(Native Method)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at sun.reflect.
NativeMethodAccessorImpl.invoke(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
NativeMethodAccessorImpl.java:62)
at sun.reflect.
DelegatingMethodAccessorImpl.invoke(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.
invoke(Method.java:498)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at com.cloud.vm.VmWorkJobHandlerProxy.
handleVmWorkJob(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.
VirtualMachineManagerImpl.handleVmWorkJob(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VirtualMachineManagerImpl.java:4852)
at com.cloud.vm.
VmWorkJobDispatcher.runJob(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.
framework.jobs.impl.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
AsyncJobManagerImpl$5.runInContext(
AsyncJobManagerImpl.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
java:558)
at org.apache.cloudstack.managed.
context.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ManagedContextRunnable$1.run(
ManagedContextRunnable.java:49)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at org.apache.cloudstack.managed.
context.impl.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
DefaultManagedContext$1.call(
DefaultManagedContext.java:56)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at org.apache.cloudstack.managed.
context.impl.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
DefaultManagedContext.callWithContext(
DefaultManagedContext
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
.java:103)
at org.apache.cloudstack.managed.
context.impl.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
DefaultManagedContext.runWithContext(
DefaultManagedContext.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
java:53)
at org.apache.cloudstack.managed.
context.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ManagedContextRunnable.run(
ManagedContextRunnable.java:46)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at org.apache.cloudstack.
framework.jobs.impl.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
AsyncJobManagerImpl$5.run(
AsyncJobManagerImpl.java:506)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at java.util.concurrent.
Executors$RunnableAdapter.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
call(Executors.java:511)
at java.util.concurrent.
FutureTask.run(FutureTask.java:266)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
at java.util.concurrent.
ThreadPoolExecutor.runWorker(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ThreadPoolExecutor.java:1149)
at java.util.concurrent.
ThreadPoolExecutor$Worker.run(
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.
java:748)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2018-05-18 09:13:11,289 DEBUG
[c.c.v.VirtualMachineManagerIm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
pl]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ctx-ec9dfbf1)
(logid:94c38c31) Cleaning up resources
for the vm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VM[DomainRouter|r-31-VM]
in Starting state
2018-05-18 09:13:11,291 DEBUG
[c.c.a.t.Request]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Seq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Sending { Cmd , MgmtId: 109952567336,
via: 10(kvm-test01-sb),
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.StopCom
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
mand":{"isProxy":false,
"checkBeforeCleanup":false,"
forceStop":false,"vmName":"r-
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
31-VM","executeInSequence":false,"wait":0}}]
}
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2018-05-18 09:13:11,627 DEBUG
[c.c.a.t.Request]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(AgentManager-Handler-14:null)
(logid:) Seq
109952567336, via: 10, Ver: v1,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Flags: 10,
[{"com.cloud.agent.api.
StopAnswer":{"result":true,"wait":0}}]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
}
2018-05-18 09:13:11,627 DEBUG
[c.c.a.t.Request]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170 ctx-ec9dfbf1)
(logid:94c38c31) Seq
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
10(kvm-test01-sb), Ver: v1,
Flags: 10, { StopAnswer } }
2018-05-18 09:13:11,639 DEBUG
[c.c.n.g.ControlNetworkGuru]
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ctx-ec9dfbf1)
NicProfile[65-31-null-null-null
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
2018-05-18 09:13:11,646 DEBUG
[c.c.v.VirtualMachineManagerIm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
pl]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ctx-ec9dfbf1)
(logid:94c38c31) Successfully released
network resources for
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
the vm
VM[DomainRouter|r-31-VM]
2018-05-18 09:13:11,646 DEBUG
[c.c.v.VirtualMachineManagerIm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
pl]
(Work-Job-Executor-30:ctx-67604a44
job-169/job-170
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
ctx-ec9dfbf1)
(logid:94c38c31) Successfully cleanued
up resources for the vm
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
VM[DomainRouter|r-31-VM] in Starting
state
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Deployment of Isolatet Networks and
redundant VPCs, works
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
wothout any
problems.
CS Version: 4.10
KVM Host: CentOS 7.4 minimal
(libvirt-3.9.0-14.el7_5.2.x86_64,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
qemu-kvm-1.5.3-156.el7.x86_64,
kernel-3.10.0-693.el7.x86_64)
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Someone got an i idea ?
Kind regards
Ben
--
Panić
______________________________
_____________________
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Benjamin Naber • Holzstraße 7 •
D-73650 Winterbach
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Post by Andrija Panic
Mobil: +49 (0) 152.34087809
______________________________
_____________________
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Diese E-mail einschließlich eventuell angehÀngter
Dateien enthÀlt
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
vertrauliche und / oder rechtlich geschÃŒtzte
Informationen. Wenn Sie nicht
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
der richtige Adressat sind und diese E-mail
irrtÃŒmlich erhalten haben,
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen
noch dÃŒrfen Sie die
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
eventuell angehÀngten Dateien öffnen und auch
keine Kopie fertigen oder den
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
Inhalt weitergeben / verbreiten. Bitte
verstÀndigen Sie den Absender und
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
löschen Sie diese E-mail und eventuell angehÀngte
Dateien umgehend.
Post by Andrija Panic
Post by Andrija Panic
Post by Benjamin Naber
--
Andrija Panić
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt
vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht
der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben,
dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die
eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den
Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und
löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
Post by Andrija Panic
This email has been sent by Oakford Technology Limited, while we
have checked this e-mail and any attachments for viruses, we can not
guarantee that they are virus-free. You must therefore take full
responsibility for virus checking.
Post by Andrija Panic
This message and any attachments are confidential and should only be
read by those to whom they are addressed. If you are not the intended
recipient, please contact us, delete the message from your computer and
destroy any copies. Any distribution or copying without our prior
permission is prohibited.
Post by Andrija Panic
Internet communications are not always secure and therefore Oakford
Technology Limited does not accept legal responsibility for this message.
The recipient is responsible for verifying its authenticity before acting
on the contents. Any views or opinions presented are solely those of the
author and do not necessarily represent those of Oakford Technology Limited.
Post by Andrija Panic
Registered address: Oakford Technology Limited, 10 Prince Maurice
Court, Devizes, Wiltshire. SN10 2RT.
Post by Andrija Panic
Registered in England and Wales No. 5971519
___________________________________________________
Benjamin Naber • Holzstraße 7 • D-73650 Winterbach
Mobil: +49 (0) 152.34087809
___________________________________________________
Diese E-mail einschließlich eventuell angehÀngter Dateien enthÀlt
vertrauliche und / oder rechtlich geschÃŒtzte Informationen. Wenn Sie nicht
der richtige Adressat sind und diese E-mail irrtÃŒmlich erhalten haben,
dÃŒrfen Sie weder den Inhalt dieser E-mail nutzen noch dÃŒrfen Sie die
eventuell angehÀngten Dateien öffnen und auch keine Kopie fertigen oder den
Inhalt weitergeben / verbreiten. Bitte verstÀndigen Sie den Absender und
löschen Sie diese E-mail und eventuell angehÀngte Dateien umgehend.
--
Andrija Panić
Andrija Panic
2018-05-25 14:36:20 UTC
Permalink
Hi Benjamin,

no experience here with Centos7 vs KVM really...

What I do remember, but that is VXLAN related, with kernel 4.0 and on-wards
(we are on Ubuntu), if you try to provision more vxlan interfaces then
max_igmp_memebership ( /proc/sys/net/ipv4/igmp_max_memberships ) it will
start interfaces but they will not be UP and proper cloud-stack message is
shown in logs (message that OS sends when you try to UP the interface, we
use ACS 4.8) - so it is easy to see issue and failed ations (though the VM
would be started fine :) )

Again, you should be able to examine agent logs and see which script is
failing... I expect either UP action is failing or not even attempted (that
would be perhaps a bug in ACS, but I don't expect that...)

Cheers
Post by Andrija Panic
Hi Ben
" Now i moved back to centos7 and have another problem. all vlan bridges
that will be created by cloudstack-agent not manual get in ifup state. if i
ifup the vlan bridges manual all works fine. Have someone a idea how i can
force centos to automaticly bring up new if devices ?"
I had this issue with 4.9 and ending up writing a dirty script to check
the down bridges every minute via cron and if up them.
iface=`/usr/sbin/ip add | grep -E "brbond.*DOWN" | awk -F ':' '{print $2}'`
echo "$iface" | while read -r a; do /usr/sbin/ip link set dev $a up; done
Thanks
Adam
-----Original Message-----
Sent: 24 May 2018 14:37
Subject: Re: KVM Problem by deploying VPC
** This mail originated from OUTSIDE the Oakford corporate network. Treat
hyperlinks and attachments in this email with caution. **
Hi Andrija,
ty for reply. i solved the error. Now i have an other error. Problem was
that it failed witout a bonding device. i created a bonding device (bond0)
wiht a single nic now it works without problems On Ubuntu.
auto enp8s0f0
iface enp8s0f0 inet manual
auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports enp8s0f0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
auto enp8s0f0
iface enp8s0f0 inet manual
bond-master bond0
auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none
auto cloudbr0
iface cloudbr0 inet static
address 10.253.250.230
gateway 10.253.250.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1
Now i moved back to centos7 and have another problem. all vlan bridges
that will be created by cloudstack-agent not manual get in ifup state. if i
ifup the vlan bridges manual all works fine. Have someone a idea how i can
force centos to automaticly bring up new if devices ?
Kind regards
Ben
Post by Andrija Panic
Hi Ben,
2018-05-23 12:59:47,213 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) getting broadcast uri
for
Post by Andrija Panic
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
FileConfigItem,
Post by Andrija Panic
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
What I believe we see here, is that IP association fails for some
reason
Post by Andrija Panic
(exit value is 1) and after that ACS will just stop the VM and do the
cleanup (all log lines after this one)...
Can you check your broadcast URI in the DB ? ACtually... I see it
vlan 223
Post by Andrija Panic
* I did have some issues in past releases, when we wanted to use
untagged
Post by Andrija Panic
vlans for Public network, but it seems not to be case here...
Not sure, if it's possible that you also SSH to the VR during this
creation
Post by Andrija Panic
process, in order to collect logs from inside the VR - before Qemu
destroys
Post by Andrija Panic
the VR?
to fetch
Post by Andrija Panic
(SCP or something) whole /var/log/ folder to the localhost - from
there,
Post by Andrija Panic
there is cloud.log and auth.log where most of the commands outputs
are
Post by Andrija Panic
located (success or failure)
or something like rsync -av -e "ssh -p 3992 -i .ssh.id_rsa.cloud"
/local/dir/
and keep repeating until you fetch whatever data to be able to
analyze it
Post by Andrija Panic
Perhaps someone else will have better suggestion...
Best
Andrija
On 23 May 2018 at 13:08, Benjamin Naber <
Post by Benjamin Naber
Hi Andrija,
first of all thanks for your reply. I have now testet the
setup on a
Post by Andrija Panic
Post by Benjamin Naber
Ubuntu Xenial. Same issu with Default VPC Offering. Redundant
VPC offering
Post by Andrija Panic
Post by Benjamin Naber
works also without any problems. Same as on CentOS.
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Proccess agent
ready
Post by Andrija Panic
Post by Benjamin Naber
command, agent id = 15
2018-05-23 12:58:05,161 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,162 INFO [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Ready command
agent id = 15
2018-05-23 12:58:05,162 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:0806b407) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}]
}
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Request:Seq
15-5506494969390563336: { Cmd , MgmtId: 109952567336, via: 15,
Ver: v1,
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100111, [{"com.cloud.agent.api.ReadyCo
mmand":{"dcId":1,"hostId":15,"wait":0}}] }
2018-05-23 12:58:05,292 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Processing
com.cloud.agent.api.ReadyCommand
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Proccess agent
ready
Post by Andrija Panic
Post by Benjamin Naber
command, agent id = 15
2018-05-23 12:58:05,292 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Set agent id 15
2018-05-23 12:58:05,293 INFO [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Ready command
agent id = 15
2018-05-23 12:58:05,294 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) (logid:0806b407) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.ReadyAnswer":{"result":true,"wait":0}}]
}
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,019 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Request:Seq
15-5506494969390563337: { Cmd , MgmtId: 109952567336, via: 15,
Ver: v1,
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100111, [{"org.apache.cloudstack.
storage.command.CopyCommand":{"
Post by Andrija Panic
Post by Benjamin Naber
{"path":"6129c76d-67e5-4454-9bc9-4e9a5a152610","origUrl":"
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/sys
temvmtemplate/4.10/dnsmasq/systemvm64template-4.10-kvm.
qcow2.bz2
Post by Andrija Panic
Post by Benjamin Naber
3,"format":"QCOW2","accountId":1,"checksum":"bc2eac46f16a2ec
e6c19d4b89db41de3","hvm":false,"displayText":"SystemVM
Template
Post by Andrija Panic
Post by Benjamin Naber
(KVM)","imageDataStore":{"org.apache.cloudstack.storage.to.P
rimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-b274-961fb337
e716","id":14,"poolType":"RBD","host":"ceph-mon","path":"
rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE=Primary&
STOREUUID=2258aa76-7813-354d-b274-961fb337e716","isManaged"
:false}},"name":"routing-3","hypervisorType":"KVM"}},"destTO
":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
uuid":"fa3dba5d-364b-43ce-9ebf-d15a3324f765","
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
storage.to
Post by Andrija Panic
Post by Benjamin Naber
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
volumeId":37,"vmName":"r-37-VM","accountId":2,"format":"
QCOW2","provisioningType":"THIN","id":37,"deviceId":0,"hy
pervisorType":"KVM"}},"executeInSequence":true,"
options":{},"options2":{},"wait":0}}]
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:58:43,020 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Processing
org.apache.cloudstack.storage.command.CopyCommand
2018-05-23 12:58:43,021 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch
storage pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,021 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,032 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully
refreshed pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232
22039007177 Available: 1426988417024
2018-05-23 12:58:43,032 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to fetch
storage pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,032 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-4:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,041 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully
refreshed pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232
22039007177 Available: 1426988417024
2018-05-23 12:58:43,223 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Creating volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 from template
6129c76d-67e5-4454-9bc9-4e9a5a152610 in pool
2258aa76-7813-354d-b274-961fb337e716 (RBD) with size 349945344
2018-05-23 12:58:43,225 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Trying to
perform a RBD
Post by Andrija Panic
Post by Benjamin Naber
clone (layering) since we are operating in the same storage
pool
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,270 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully
connected to
Post by Andrija Panic
Post by Benjamin Naber
Ceph cluster at ceph-mon:6789
2018-05-23 12:58:43,321 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) The source image
rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610 is RBD format 2. We
will perform
Post by Andrija Panic
Post by Benjamin Naber
a RBD clone using snapshot cloudstack-base-snap
2018-05-23 12:58:43,321 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Checking if RBD
snapshot
exists
Post by Andrija Panic
Post by Benjamin Naber
prior to attempting a clone operation.
2018-05-23 12:58:43,343 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Found 1
snapshots on RBD
Post by Andrija Panic
Post by Benjamin Naber
image rbd/6129c76d-67e5-4454-9bc9-4e9a5a152610
2018-05-23 12:58:43,344 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) RBD snapshot
already
Post by Andrija Panic
Post by Benjamin Naber
exists.
2018-05-23 12:58:43,468 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-4:null) (logid:ef8b353e) Succesfully
cloned
Post by Andrija Panic
Post by Benjamin Naber
fa3dba5d-364b-43ce-9ebf-d15a3324f765
2018-05-23 12:58:43,474 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"
{"size":349945344,"path":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","accountId":0,"format":"RAW","id":0}},"
result":true,"wait":0}}]
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Request:Seq
15-5506494969390563338: { Cmd , MgmtId: 109952567336, via: 15,
Ver: v1,
Post by Andrija Panic
Post by Benjamin Naber
Flags: 100111, [{"com.cloud.agent.api.StartCo
mmand":{"vm":{"id":37,"name":"r-37-VM","type":"DomainRouter"
268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux 5.0
Post by Andrija Panic
Post by Benjamin Naber
(64-bit)","platformEmulator":"Debian GNU/Linux
5","bootArgs":" vpccidr=
Post by Andrija Panic
Post by Benjamin Naber
10.59.0.0/21 domain=test.internal dns1=8.8.8.8
privategateway=None
Post by Andrija Panic
Post by Benjamin Naber
template=domP name=r-37-VM eth0ip=169.254.2.247
eth0mask=255.255.0.0
Post by Andrija Panic
Post by Benjamin Naber
type=vpcrouter disable_rp_filter=true
baremetalnotificationsecurityk
Post by Andrija Panic
Post by Benjamin Naber
ey=Gs_GstLMdaO8CWiW_-wN71Xu0RhIGWdJ27Y_Q_
lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILqRjLA
Post by Andrija Panic
Post by Benjamin Naber
baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e95U03
mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ
host=172.16.11.56
Post by Andrija Panic
Post by Benjamin Naber
port=8080","enableHA":true,"limitCpuUse":false,"enableDynami
callyScaleVm":false,"vncPassword":"-Wf7MGedC0usYmBc
Cscv7A","params":{},"uuid":"87a45b1c-bf35-431e-82a0-
c0c961feab05","disks":[{"data":{"org.apache.cloudstack.
storage.to
Post by Andrija Panic
Post by Benjamin Naber
.VolumeObjectTO":{"uuid":"fa3dba5d-364b-43ce-9ebf-
d15a3324f765","
Post by Andrija Panic
Post by Benjamin Naber
volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
storage.to
Post by Andrija Panic
Post by Benjamin Naber
.PrimaryDataStoreTO":{"uuid":"2258aa76-7813-354d-
b274-961fb337e716","id":14,"poolType":"RBD","host":"ceph-mon
","path":"rbd","port":6789,"url":"RBD://ceph-mon/rbd/?ROLE
=Primary&STOREUUID=2258aa76-7813-354d-b274-961fb337e716","
isManaged":false}},"name":"ROOT-37","size":349945344,"
37,"vmName":"r-37-VM","accountId":2,"format":"RAW","p
rovisioningType":"THIN","id":37,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hy
pervisorType":"KVM"}},"diskSeq":0,"path":"fa3dba5d-364b-
43ce-9ebf-d15a3324f765","type":"ROOT","_details":{"storageHo
st":"ceph-mon","managed":"false","storagePort":"6789","
volumeSize":"349945344"}}],"nics":[{"deviceId":0,"
networkRateMbps":-1,"
Post by Andrija Panic
Post by Benjamin Naber
defaultNic":false,"pxeDisable":true,"nicUuid":"226f4319-
b15c-4e48-8a74-6a28b574a992","uuid":"76d57f99-f4af-42b8-
af0a-41f0a2cd0ba5","ip":"169.254.2.247","netmask":"255.255.
0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:f7","broa
dcastType":"LinkLocal","type":"Control","isSecurityGroupEnab
led":false}],"guestOsDetails":{}},"hostIp":"10.253.250.230",
"executeInSequence":false,"wait":0}},{"com.cloud.agent.api.
check.CheckSshCommand":{"ip":"169.254.2.247","port":3922,"
interval":6,"retries":100,"name":"r-37-VM","wait":0}},{"
com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
0}},{"com.cloud.agent.api.PlugNicCommand":{"nic":{"deviceId"
true,"nicUuid":"29bd4c33-47dd-4357-aa3d-8c5ae56169b5","uuid"
:"787cf778-230f-4e17-a648-f63171bdc81c","ip":"82.119.177.
188","netmask":"255.255.255.240","gateway":"82.119.177.
177","mac":"1e:00:87:00:00:0d","broadcastType":"Vlan","type"
:"Public","broadcastUri":"vlan://233","isolationUri":"vl
an://233","isSecurityGroupEnabled":false},"instanceName":"r-
37-VM","vmType":"DomainRouter","details":{},"wait":0}},{"
com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses
":[{"accountId":2,"publicIp":"82.X.X.X","sourceNat":true,"
"vlan://233","vlanGateway":"82.119.X.X","vlanNetmask":"
255.255.255.240","
Post by Andrija Panic
Post by Benjamin Naber
vifMacAddress":"1e:00:87:00:00:0d","networkRate":200,"
trafficType":"Public","newNic":false}],"accessDetails":{"
router.guest.ip":"82.X.X.X","router.ip":"169.254.2.247","
zone.network.type":"Advanced","router.name":"r-37-VM"},"
wait":0}},{"com.cloud.agent.api.routing.SetSourceNatComman
d":{"ipAddress":{"accountId":2,"publicIp":"82.X.X.X","sourc
eNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"
broadcastUri":"vlan://233","vlanGateway":"82.X.X.X","
00:0d","networkRate":200,"trafficType":"Public","newNic"
:false},"add":true,"accessDetails":{"router.ip":"169.254.2.
247","zone.network.type":"Advanced","router.name":"r-37-
VM"},"wait":0}},{}]
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:58:43,625 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
com.cloud.agent.api.StartCommand
2018-05-23 12:58:43,647 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,648 DEBUG [resource.wrapper.
LibvirtStartCommandWrapper]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) remaining
mem36873936896
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,649 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch
storage pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:43,649 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:43,657 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully
refreshed pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232
22039007177 Available: 1426988417024
2018-05-23 12:58:43,663 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not find
volume
Post by Andrija Panic
Post by Benjamin Naber
fa3dba5d-364b-43ce-9ebf-d15a3324f765: Storage volume not
found: no
Post by Andrija Panic
Post by Benjamin Naber
storage vol with matching name 'fa3dba5d-364b-43ce-9ebf-
d15a3324f765'
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:43,664 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Refreshing
storage pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:58:43,947 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Found volume
fa3dba5d-364b-43ce-9ebf-d15a3324f765 in storage pool
2258aa76-7813-354d-b274-961fb337e716 after refreshing the pool
2018-05-23 12:58:44,204 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to fetch
storage pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:58:44,204 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:58:44,211 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Succesfully
refreshed pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232
22039007177 Available: 1426979717120
2018-05-23 12:58:44,211 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Control-169.254.2.247-null]
2018-05-23 12:58:44,212 DEBUG [utils.script.Script]
/bin/bash -c ip
Post by Andrija Panic
Post by Benjamin Naber
route | grep 169.254.0.0/16
2018-05-23 <http://169.254.0.0/162018-05-23> 12:58:44,248
DEBUG
Post by Andrija Panic
Post by Benjamin Naber
[utils.script.Script] (agentRequest-Handler-5:null)
(logid:ef8b353e)
Post by Andrija Panic
Post by Benjamin Naber
Execution is successful.
2018-05-23 12:58:44,251 DEBUG [resource.wrapper.
LibvirtStartCommandWrapper]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) starting
r-37-VM: <domain type='kvm'>
Post by Andrija Panic
Post by Benjamin Naber
<name>r-37-VM</name>
<uuid>87a45b1c-bf35-431e-82a0-c0c961feab05</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>87a45b1c-bf35-
431e-82a0-c0c961feab05</entry>
Post by Andrija Panic
Post by Benjamin Naber
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>262144</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:02:f7'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='network'>
<driver name='qemu' type='raw' cache='none' />
<source protocol='rbd' name='rbd/fa3dba5d-364b-43ce-
9ebf-d15a3324f765'>
Post by Andrija Panic
Post by Benjamin Naber
<host name='ceph-mon' port='6789'/>
</source>
<auth username='admin'>
<secret type='ceph' uuid='2258aa76-7813-354d-b274-
961fb337e716'/>
Post by Andrija Panic
Post by Benjamin Naber
</auth>
<target dev='vda' bus='virtio'/>
<serial>fa3dba5d364b43ce9ebf</serial></disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='10.253.250.230'
passwd='-Wf7MGedC0usYmBcCscv7A'/>
Post by Andrija Panic
Post by Benjamin Naber
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-
37-VM.agent'/>
Post by Andrija Panic
Post by Benjamin Naber
<address type='virtio-serial'/>
<target type='virtio' name='r-37-VM.vport'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/r-
37-VM.org.qemu.guest_agent.0'/>
Post by Andrija Panic
Post by Benjamin Naber
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>
2018-05-23 12:58:46,460 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
patchviasocket.py
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:58:46,496 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is
successful.
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:58:46,498 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:58:49,567 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:58:54,568 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:58:57,630 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:02,631 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon
to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:02,632 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
patchviasocket.py
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:02,667 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is
successful.
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:02,669 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:04,665 DEBUG [kvm.resource.
LibvirtComputingResource]
/usr/share/cloudstack-common/s
Post by Andrija Panic
Post by Benjamin Naber
cripts/vm/network/security_group.py get_rule_logs_for_vms
2018-05-23 12:59:04,731 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) (logid:80cd5d21) Processing
com.cloud.agent.api.GetHostStatsCommand
2018-05-23 12:59:04,770 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(UgentTask-2:null) (logid:) Execution is successful.
2018-05-23 12:59:04,772 DEBUG [kvm.resource.LibvirtConnection]
qemu:///system
2018-05-23 12:59:04,777 DEBUG [cloud.agent.Agent]
(UgentTask-2:null)
Post by Andrija Panic
Post by Benjamin Naber
(logid:) Sending ping: Seq 15-2: { Cmd , MgmtId: -1, via: 15,
Ver: v1,
Post by Andrija Panic
Post by Benjamin Naber
Flags: 11, [{"com.cloud.agent.api.
PingRoutingWithNwGroupsCommand":{"
Post by Andrija Panic
Post by Benjamin Naber
true,"_vnetAccessible":true,"hostType":"Routing","hostId":15,"wait":0}}]
}
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:04,876 DEBUG [cloud.agent.Agent]
(Agent-Handler-3:null)
Post by Andrija Panic
Post by Benjamin Naber
15, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAns
wer":{"_command":{"hostType":"Routing","hostId":15,"wait":0}
,"result":true,"wait":0}}]
Post by Andrija Panic
Post by Benjamin Naber
}
2018-05-23 12:59:05,726 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:10,727 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:12,767 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:13,532 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) (logid:275957d4) Processing
com.cloud.agent.api.GetStorageStatsCommand
2018-05-23 12:59:13,533 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4) Trying to fetch
storage pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 from libvirt
2018-05-23 12:59:13,533 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-2:null) (logid:275957d4) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:13,539 INFO [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4) Asking libvirt
to refresh
Post by Andrija Panic
Post by Benjamin Naber
storage pool 2258aa76-7813-354d-b274-961fb337e716
2018-05-23 12:59:13,989 DEBUG [kvm.storage.
LibvirtStorageAdaptor]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-2:null) (logid:275957d4) Succesfully
refreshed pool
Post by Andrija Panic
Post by Benjamin Naber
2258aa76-7813-354d-b274-961fb337e716 Capacity: 1500336095232
22173237211 Available: 1426660573184
2018-05-23 12:59:17,767 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon
to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:17,768 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
patchviasocket.py
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:17,802 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is
successful.
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:17,803 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:17,804 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:22,804 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:38,239 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Could not
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,239 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Unable to logon
to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,248 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
patchviasocket.py
Post by Andrija Panic
Post by Benjamin Naber
-n r-37-VM -p %vpccidr=10.59.0.0/21%domain=
test.internal%dns1=8.8.8.8%priv
Post by Andrija Panic
Post by Benjamin Naber
ategateway=None%template=domP%name=r-37-VM%eth0ip=169.254.2.
247%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=
true%baremetalnotificationsecuritykey=Gs_GstLMdaO8CWiW_-wN71
Xu0RhIGWdJ27Y_Q_lJ7vnFR44NkHMRknSyuMp4eRm9WXpasA7dmX9UtCRILq
RjLA%baremetalnotificationapikey=6a0ehX9xAk7wTIm4knUtIY0Qd1e
95U03mXqoNnZD42UMvKAsZJxpOgnVnwRNQC3pC32eIwyYnyWHBX1e8vY5cQ%
host=172.16.11.56%port=8080
2018-05-23 12:59:43,283 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is
successful.
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:43,285 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,286 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
com.cloud.agent.api.check.CheckSshCommand
2018-05-23 12:59:43,286 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-5:null)
Post by Andrija Panic
Post by Benjamin Naber
(logid:ef8b353e) Ping command port, 169.254.2.247:3922
2018-05-23 12:59:43,286 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Trying to
connect to
Post by Andrija Panic
Post by Benjamin Naber
169.254.2.247
2018-05-23 12:59:43,287 DEBUG [resource.wrapper.LibvirtOvsVp
cRoutingPolicyConfigCommandWrapper]
(agentRequest-Handler-5:null)
Post by Andrija Panic
Post by Benjamin Naber
(logid:ef8b353e) Ping command port succeeded for vm r-37-VM
2018-05-23 12:59:43,287 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
com.cloud.agent.api.GetDomRVersionCmd
2018-05-23 12:59:43,289 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
/usr/share/cloudstack-common/scripts/network/domr/router_
proxy.sh
Post by Andrija Panic
Post by Benjamin Naber
get_template_version.sh 169.254.2.247
2018-05-23 12:59:43,456 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is
successful.
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:43,459 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
com.cloud.agent.api.PlugNicCommand
2018-05-23 12:59:43,459 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e)
nic=[Nic:Public-82.119.177.188-vlan://233]
2018-05-23 12:59:43,474 DEBUG [kvm.resource.BridgeVifDriver]
/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh
-v 233
Post by Andrija Panic
Post by Benjamin Naber
-p enp8s0f0 -b brenp8s0f0-233 -o add
2018-05-23 12:59:43,504 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-5:null) (logid:ef8b353e) Execution is
successful.
Post by Andrija Panic
Post by Benjamin Naber
2018-05-23 12:59:47,202 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
com.cloud.agent.api.routing.IpAssocVpcCommand
2018-05-23 12:59:47,203 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) (logid:ef8b353e) Looking for
libvirtd
Post by Andrija Panic
Post by Benjamin Naber
connection at: qemu:///system
2018-05-23 12:59:47,212 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e)
file name 'enp8s0f0.233'
2018-05-23 12:59:47,213 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) getting
broadcast uri for
Post by Andrija Panic
Post by Benjamin Naber
pif enp8s0f0.233 and bridge brenp8s0f0-233
2018-05-23 12:59:47,213 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Transforming
com.cloud.agent.api.routing.IpAssocVpcCommand to ConfigItems
2018-05-23 12:59:47,581 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
FileConfigItem,
Post by Andrija Panic
Post by Benjamin Naber
copying 257 characters to ip_associations.json took 340ms
2018-05-23 12:59:47,582 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
/usr/share/cloudstack-common/scripts/network/domr/router_
proxy.sh
Post by Andrija Panic
Post by Benjamin Naber
update_config.py 169.254.2.247 ip_associations.json
2018-05-23 12:59:47,766 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Exit value is 1
2018-05-23 12:59:47,767 DEBUG [kvm.resource.
LibvirtComputingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) [INFO]
Processing incoming file => ip_associations.json[INFO]
Processing JSON file
Post by Andrija Panic
Post by Benjamin Naber
ip_associations.json
2018-05-23 12:59:47,768 DEBUG [resource.virtualnetwork.
VirtualRoutingResource]
Post by Andrija Panic
Post by Benjamin Naber
(agentRequest-Handler-5:null) (logid:ef8b353e) Processing
ScriptConfigItem,
Post by Andrija Panic
Post by Benjamin Naber
executing update_config.py ip_associations.json took 186ms
2018-05-23 12:59:47,775 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) (logid:ef8b353e) Seq
{ Ans: , MgmtId: 109952567336, via: 15, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":37,"name":"
r