SpringCloud Finchley.SR2配置Sleuth+Rabbit+Zipkin+ELK链路追踪

前言

SpringCloud-Sleuth是SpringCloud分布式服务追踪解决方案的一种实现方式。它借鉴了Google的分布式服务追踪技术Dapper的学术术语。

Span是分布式追踪服务的基本工作单元,发送一个RPC请求并得到一个response响应是一个Span,一个Span由64位长度的字符串标识,其中包含描述信息、时间戳、由K-V构成的Tags、触发Span事件的相关Span-ID、IP地址信息组成。Span有start和stop两个动作,start一个Span,就必须在追踪处理完成后stop这个Span。在初始化一个Span时,创建一个root-span,它的Span-ID也是服务追踪的Trace-ID。

Trace是由一系列的Span组成的树状结构。

Annotation是用来记录事件的发生过的痕迹。每次发起请求,哪个是客户端,哪个是服务端,请求从哪里发起,从哪里结束都要通过指定具体的事件通知Zipkin(Zipkin负责对收集的服务追踪痕迹数据进行解析展示),从SpringCloud-Sleuth-2.0.x版本开始使用Brave框架收集事件痕迹数据,而不用像之前的版本那样指定事件就可以让Zipkin图形化的把事件展示出来。Annoatation包含以下几种内容:

cs(Client Sent):客户端发起一个请求,cs标识一个Span的开始。

sr(Server Received):服务端收到请求,并开始处理请求。当前的时间戳减去cs的时间戳,从而获取网络等待时间。

ss(Server Sent):标记服务端处理完请求,当前时间宠减去sr时间戳,得到服务端处理请求的时间。

cr(Client Received):标记Span结束完成,客户端已经成功接收到服务端的响应。减去cs时间戳,得到整个请求从发起到接收到服务端响应整个过程的耗时。

下图展示了在一个应用系统中,Anotation、Span、Trace信息在系统的微服务中的创建和传递过程。

Trace Info propagation

Zipkin展示服务追踪痕迹

Traces Info propagation

Sleuth、Zipkin、ELK架构图

SpringCloud-Sleuth、Zipkin、ELK通常三者一起配合使用,Zipkin负责对链路追踪的调用链依赖关系和服务之间请求等待时间信息进行监控,ELK可以对Sleuth日志进行存储、查询、分析。

Sleuth+Zipkin实现服务追踪

示例使用springcloud实现的微服务应用3个,创建eureka-server、eureka-service-1、eureka-service-2三个微服务分布式应用,eureka-service-1中实现一个restcontroller接口,接口中eureka-service-1使用feignclient调用eureka-service-2的服务接口。痕迹数据发送给Zipkin有两种方式:HTTP和MQ。

通过HTTP异步发送服务追踪痕迹

部署Zipkin

下载Zipkin(示例使用zipkin-server-2.10.1-exec.jar),在本地启动Zipkin。

1
java -jar zipkin-server-2.10.1-exec.jar

parent-pom

parent-pom.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.3.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>sleuth-rabbit-zipkin</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>sleuth-rabbit-zipkin</name>
<description>Demo project for Spring Boot</description>

<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Finchley.RELEASE</spring-cloud.version>
</properties>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>https://repo.spring.io/milestone</url>
</repository>
</repositories>

</project>

eureka-server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.example</groupId>
<artifactId>sleuth-rabbit-zipkin</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>eureka-server</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>eureka-server</name>
<description>Demo project for Spring Boot</description>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

</project>
1
2
3
4
5
6
7
8
9
@EnableEurekaServer
@SpringBootApplication
public class EurekaServerApplication {

public static void main(String[] args) {
SpringApplication.run(EurekaServerApplication.class, args);
}

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
spring:
application:
name: eureka-server
jackson:
time-zone: GMT+8
server:
port: 8000
eureka:
instance:
hostname: eureka-server
client:
register-with-eureka: false
fetch-registry: false
service-url:
defaultZone: http://localhost:8000/eureka/

eureka-service-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package com.example.eurekaservice1;

import brave.Tracer;
import com.example.eurekaservice2.feigns.Svc2FeignClient;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@Slf4j
@RestController
@EnableDiscoveryClient
@EnableFeignClients(basePackages = "com.example.eurekaservice2.feigns")
@SpringBootApplication
public class EurekaService1Application {

@Autowired
private Svc2FeignClient svc2FeignClient;

@Autowired
private Tracer tracer;

public static void main(String[] args) {
SpringApplication.run(EurekaService1Application.class, args);
}

@RequestMapping(path = "/svc/1",method = RequestMethod.GET)
public String test(){
log.info("========/svc/1=======");
tracer.currentSpan().tag("svc-1","/abc/efg");
return "ForwardLee " + svc2FeignClient.test();
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
server:
port: 8001
spring:
application:
name: eureka-service-1
zipkin:
base-url: http://localhost:9411
enabled: true
sender:
type: web
sleuth:
sampler:
probability: 1.0
eureka:
client:
service-url:
defaultZone: http://localhost:8000/eureka/

eureka-service-2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
package com.example.eurekaservice2.feigns;

import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

/**
* @program: sleuth-rabbit-zipkin
* @description: ${description}
* @author: Forwardlee
* @create: 2019-03-11
**/
@FeignClient("eureka-service-2")
public interface Svc2FeignClient {

@RequestMapping(path = "/svc/2",method = RequestMethod.GET)
public String test();
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
package com.example.eurekaservice2;

import brave.Tracer;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@Slf4j
@RestController
@EnableDiscoveryClient
@SpringBootApplication
public class EurekaService2Application {

@Autowired
private Tracer tracer;

public static void main(String[] args) {
SpringApplication.run(EurekaService2Application.class, args);
}


@RequestMapping(path = "/svc/2",method = RequestMethod.GET)
public String svc2(){
log.info("=========/svc/2==========");
tracer.currentSpan().tag("svc-2","/svc/2");
return "ok";
}

}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
server:
port: 8002
spring:
application:
name: eureka-service-2
zipkin:
base-url: http://localhost:9411
enabled: true
sender:
type: web
sleuth:
sampler:
probability: 1.0
eureka:
client:
service-url:
defaultZone: http://localhost:8000/eureka/

Zipkin展示SpringCloud-Sleuth痕迹

使用浏览器或者postman请求微服务http://localhost:8001/svc/1,返回正常response。浏览器打开http://localhost:9411,查看sleuth数据。

image-20190312190823933

点击展示出的eureka-service-1查看具体的痕迹信息。

停掉eureka-service-2服务,再请求一次http://localhost:8001/svc/1,返回错误response。再刷新Zipkin页面,显示如下。

image-20190312190912566

通过MQ发送服务追踪痕迹

可以使用RabbitMQ或Kafka接收Sleuth数据,Zipkin订阅MQ获取Sleuth数据。示例使用RabbitMQ接收Sleuth数据。

部署RabbitMQ

示例在mac环境下安装rabbitmq

1
brew install rabbitmq

启动rabbitmq

1
brew services start rabbitmq

部署完Zipkin后,在浏览器地址栏输入http://localhost:15672查看RabbitMQ,查找Queue,Zipkin会创建一个zipkin的默认队列,zipkin队列就是用来接收Sleuth消息数据的。

image-20190312185155437

部署Zipkin

使用MQ接收Sleuth数据时,Zipkin的方式略有不同。

1
RABBIT_ADDRESSES=localhost java -jar zipkin-server-2.10.1-exec.jar

配置RabbitMQ启动Zipkin时可配置的相关参数如下:

属性 环境变量 描述
zipkin.collector.rabbitmq.concurrency RABBIT_CONCURRENCY 并发消费者数量,默认为1
zipkin.collector.rabbitmq.connection-timeout RABBIT_CONNECTION_TIMEOUT 建立连接时的超时时间,默认为 60000毫秒,即 1 分钟
zipkin.collector.rabbitmq.queue RABBIT_QUEUE 从中获取 span 信息的队列,默认为 zipkin
zipkin.collector.rabbitmq.uri RABBIT_URI 符合 RabbitMQ URI 规范 的 URI,例如amqp://user:pass@host:10000/vhost

设置了URI,以下参数可忽略:

属性 环境变量 描述
zipkin.collector.rabbitmq.addresses RABBIT_ADDRESSES 用逗号分隔的 RabbitMQ 地址列表,例如localhost:5672,localhost:5673
zipkin.collector.rabbitmq.password RABBIT_PASSWORD 连接到 RabbitMQ 时使用的密码,默认为 guest
zipkin.collector.rabbitmq.username RABBIT_USER 连接到 RabbitMQ 时使用的用户名,默认为guest
zipkin.collector.rabbitmq.virtual-host RABBIT_VIRTUAL_HOST 使用的 RabbitMQ virtual host,默认为 /
zipkin.collector.rabbitmq.use-ssl RABBIT_USE_SSL 设置为true则用 SSL 的方式与 RabbitMQ 建立链接

parent-pom

同上

eureka-server

同上

eureka-service-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.example</groupId>
<artifactId>sleuth-rabbit-zipkin</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>eureka-service-1</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>eureka-service-1</name>
<description>Demo project for Spring Boot</description>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

<!-- 增加这块 start -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
</dependency>
<!-- 增加这块 end -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.example</groupId>
<artifactId>eureka-service-2</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>https://repo.spring.io/milestone</url>
</repository>
</repositories>

</project>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package com.example.eurekaservice1;

import brave.Tracer;
import com.example.eurekaservice2.feigns.Svc2FeignClient;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@Slf4j
@RestController
@EnableDiscoveryClient
@EnableFeignClients(basePackages = "com.example.eurekaservice2.feigns")
@SpringBootApplication
public class EurekaService1Application {

@Autowired
private Svc2FeignClient svc2FeignClient;

@Autowired
private Tracer tracer;

public static void main(String[] args) {
SpringApplication.run(EurekaService1Application.class, args);
}

@RequestMapping(path = "/svc/1",method = RequestMethod.GET)
public String test(){
log.info("========/svc/1=======");
tracer.currentSpan().tag("svc-1","/abc/efg");
return "Li Fenghua " + svc2FeignClient.test();
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
server:
port: 8001
spring:
application:
name: eureka-service-1
zipkin:
base-url: http://localhost:9411
enabled: true
sender:
#这里不同
type: rabbit
sleuth:
sampler:
probability: 1.0
#这里不同
rabbitmq:
addresses: localhost:5672
username: guest
password: guest
eureka:
client:
service-url:
defaultZone: http://localhost:8000/eureka/

eureka-service-2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.example</groupId>
<artifactId>sleuth-rabbit-zipkin</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.example</groupId>
<artifactId>eureka-service-2</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>eureka-service-2</name>
<description>Demo project for Spring Boot</description>

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>

<!-- 增加这块 start -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.amqp</groupId>
<artifactId>spring-rabbit</artifactId>
</dependency>
<!-- 增加这块 end -->

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>https://repo.spring.io/milestone</url>
</repository>
</repositories>

</project>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
package com.example.eurekaservice2.feigns;

import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

/**
* @program: sleuth-rabbit-zipkin
* @description: ${description}
* @author: Forwardlee
* @create: 2019-03-11
**/
@FeignClient("eureka-service-2")
public interface Svc2FeignClient {

@RequestMapping(path = "/svc/2",method = RequestMethod.GET)
public String test();
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
package com.example.eurekaservice2;

import brave.Tracer;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

@Slf4j
@RestController
@EnableDiscoveryClient
@SpringBootApplication
public class EurekaService2Application {

@Autowired
private Tracer tracer;

public static void main(String[] args) {
SpringApplication.run(EurekaService2Application.class, args);
}


@RequestMapping(path = "/svc/2",method = RequestMethod.GET)
public String svc2(){
log.info("=========/svc/2==========");
tracer.currentSpan().tag("svc-2","/svc/2");
return "ok";
}

}

Zipkin展示SpringCloud-Sleuth痕迹

使用浏览器或者postman请求多次微服务http://localhost:8001/svc/1,返回正常response。浏览器打开http://localhost:9411,查看sleuth数据。

image-20190312190232718

停掉eureka-service-2服务,再请求一次http://localhost:8001/svc/1,返回错误response。浏览器打开http://localhost:9411,查看sleuth数据,错误的请求痕迹被标记成红色。

image-20190312190447696

Sleuth+ELK实现服务追踪日志分析

SpringCloud默认集成了slf4j的logback实现,我们在微服务项目中依赖net.logstash.logback:logstash-logback-encoder:4.9包,就可以实现logstash日志文件输出或者日志数据直接发送给logstash服务。

ELK使用Elasticsearch、Logstash、Kibana三种技术实现的一套日志存储、检索、分析、展示解决方案。Elasticsearch负责存储、检索、分析日志,Logstash负责收集日志,Kibana负责动态图形化展示日志分析的结果。我们的SpringCloud-Sleuth只需要把生成的Trace数据按照Logstash的约定格式提供给Logstash服务即可,Logstash再把收集的日志输出给Elasticsearch进行存储、查询、分析。Logstash接收服务追踪日志数据有两种形式:读取指定位置日志文件、网络接口服务接收日志数据。示例使用Logstash的网络服务接口接收日志数据。

微服务集成logstash

在eureka-service-1和eureka-service-2两个pom文件中添加如下依赖:

1
2
3
4
5
6
7
8
9
10
<!-- setup logstash -->
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</dependency>
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.9</version>
</dependency>

PS:首先,官方文档中推荐依赖net.logstash.logback:logstash-logback-encoder:4.6,但是本机尝试只有版本升到4.9服务才能启动起来;其次,在两个项目配置文件目录必须创建bootstrap.yml,并把spring.application.name设置在bootstrap.yml中,否则下面的logback-spring.xml中的springApplicationName将优先加载却找不到配置而导致应用报错。

logback-spring.xml配置如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>

<springProperty scope="context" name="springAppName" source="spring.application.name"/>
<!-- Example for logging into the build folder of your project -->
<!--<property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${springAppName}"/>-->
<property name="LOG_FILE" value="/Users/Cattlee/logs/logstash/${springAppName}"/>

<!-- You can override this to have a custom pattern -->
<property name="CONSOLE_LOG_PATTERN"
value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/>

<!-- Appender to log to console -->
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<!-- Minimum logging level to be presented in the console logs-->
<level>DEBUG</level>
</filter>
<encoder>
<pattern>${CONSOLE_LOG_PATTERN}</pattern>
<charset>utf8</charset>
</encoder>
</appender>

<!--&lt;!&ndash; Appender to log to file &ndash;&gt;-->
<!--<appender name="flatfile" class="ch.qos.logback.core.rolling.RollingFileAppender">-->
<!--<file>${LOG_FILE}</file>-->
<!--<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">-->
<!--<fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.gz</fileNamePattern>-->
<!--<maxHistory>7</maxHistory>-->
<!--</rollingPolicy>-->
<!--<encoder>-->
<!--<pattern>${CONSOLE_LOG_PATTERN}</pattern>-->
<!--<charset>utf8</charset>-->
<!--</encoder>-->
<!--</appender>-->

<!-- 日志输出给logstash服务 -->
<appender name="logstash"
class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>localhost:9250</destination>
<!-- 日志输出编码 -->
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<timestamp>
<timeZone>GMT+8</timeZone>
</timestamp>
<pattern>
<pattern>
{
"severity": "%level",
"service": "${springAppName:-}",
"trace": "%X{X-B3-TraceId:-}",
"span": "%X{X-B3-SpanId:-}",
"parent": "%X{X-B3-ParentSpanId:-}",
"exportable": "%X{X-Span-Export:-}",
"pid": "${PID:-}",
"thread": "%thread",
"class": "%logger{40}",
"rest": "%message"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>

<root level="INFO">
<appender-ref ref="console"/>
<!-- uncomment this to have also JSON logs -->
<appender-ref ref="logstash"/>
<!--<appender-ref ref="flatfile"/>-->
</root>
</configuration>

配置和部署ELK

示例重点在于ELK收集、分析、查询、展示的实现方式,对于ELK环境的相关配置参数一笔带过。

配置和启动Logstash服务

logback中配置了logstash的网络接口服务接收日志数据,那么在ELK中的logstash如下配置,启动网络服务接口。(LOGSTASH_HOME/config/XXX.conf)input配置Logstash的启动方式(网络应用和本地后台应用),output配置elasticsearch的相关信息,index用来标识logstash发送给Elasticsearch的日志数据标识,在Kibana展示日志数据时会用到。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

# For detail structure of this file
# Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input {
# For detail config for log4j as input,
# See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
tcp {
mode => "server"
host => "localhost"
port => 9250
}
}
filter {
#Only matched data are send to output.
}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
# index => "eureka-service"
#user => "elastic"
#password => "changeme"
}
}

启动logstash:

1
./bin/logstash -f config/logstash-demo.conf

配置和启动Elasticsearch

Elasticsearch不用特殊配置,默认端口9200,直接启动。

1
./bin/elasticsearch

配置和启动Kibana

Kibana不用特殊配置,默认端口5601,直接启动。

1
./bin/kibana

使用Kibana

在浏览器中输入http://localhost:5601进入Kibana页面,进入IndexPattern管理页面,创建一个index pattern。这个index pattern可以用来正则匹配logstash配置文件中output部分的elasticsearch index的值。

image-20190314120659395

配置完index pattern后,再进入Discover,就可以查看SpringCloud-Sleuth日志分析详情。

image-20190314143351458

谢谢你请我吃糖果