prometheus和tidb集群exporter兼容性问题

BraveChen · 2025 年7 月 14 日 01:56

在v7.5.6中，安装包自带的prometheus 是2.27.1版本，自家prometheus统一监控是v3.0，拉取某些metric时似乎协议不兼容，大家有碰到这种问题吗，怎么解的tiflash exporter occured scrape error when using prometheus:v3.0.1 · Issue #10304 · pingcap/tiflash · GitHub

xfworld · 2025 年7 月 14 日 03:21

不用自带的 prometheus，用自己的就好了

Kongdom · 2025 年7 月 14 日 05:38

直接用自家的prometheus就可以了。
https://docs.pingcap.com/zh/tidb/stable/deploy-monitoring-services/

有猫万事足 · 2025 年7 月 14 日 16:40

github.com/prometheus/prometheus

Non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target

已打开 02:14PM - 29 Nov 24 UTC

已关闭 06:53PM - 17 Dec 24 UTC

insider89

### What did you do? Upgrade prometheus from 2.55.1 to 3.0.0 ### What did you …expect to see? Continue scrape metrics ### What did you see instead? Under which circumstances? Prometheus failed to scrape metrics from dshackle pods with the error: In the UI I see ``` Error scraping target: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target ``` In the logs I see ``` fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" ``` Works on previous version. ### System information Linux/k8s ### Prometheus version ```text time=2024-11-29T14:13:34.722Z level=INFO source=main.go:694 msg="operational information" build_context="(go=go1.23.3, platform=linux/amd64, user=root@aa286d0eb00a, date=20241114-16:40:43, tags=netgo,builtinassets,stringlabels)" host_details="(Linux 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 prometheus-prometheus-server-58b678cdd6-w55gq (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)" ``` ``` ### Prometheus configuration file _No response_ ### Alertmanager version _No response_ ### Alertmanager configuration file _No response_ ### Logs ```text time=2024-11-29T14:13:34.723Z level=INFO source=main.go:770 msg="Leaving GOMAXPROCS=32: CPU quota undefined" component=automaxprocs time=2024-11-29T14:13:34.726Z level=INFO source=web.go:650 msg="Start listening for connections" component=web address=0.0.0.0:9090 time=2024-11-29T14:13:34.726Z level=INFO source=main.go:1239 msg="Starting TSDB ..." time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1731974400018 maxt=1732039200000 ulid=01JD3AFKRCP18NS8P3F8RAEQ8R time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732039200000 maxt=1732104000000 ulid=01JD51D54YYNWNW169BMXFCFWF time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732104000001 maxt=1732168800000 ulid=01JD762MM0R1HFY71AYCQQDC0J time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732168800005 maxt=1732233600000 ulid=01JD93W848ZZBES8WJF0S3MRD6 time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732233600018 maxt=1732298400000 ulid=01JDB1NQB6DYMPAFZJE0K508YX time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732298400013 maxt=1732363200000 ulid=01JDCZFACJP5JY9CBZG5EAR0RW time=2024-11-29T14:13:34.726Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732363200011 maxt=1732428000000 ulid=01JDEX8S4HGWP9SF3TWJ7HDQQV time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732428000020 maxt=1732492800000 ulid=01JDGV2BWVYNESWAPZB2RR1Z13 time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732492800020 maxt=1732557600000 ulid=01JDJRVW3JSSNKWWEKTX8W2WD5 time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732557600020 maxt=1732622400000 ulid=01JDMPND9J6MAZ6E4C05DJJHQ9 time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732622400019 maxt=1732687200000 ulid=01JDPMEYQMZABW9PBJ1JD5YB0Y time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732687200010 maxt=1732752000000 ulid=01JDRJ8G2NY1TAQBNYKHVYVH8R time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732752000003 maxt=1732816800000 ulid=01JDTG1PRQCVWE52MW6WTNFDB3 time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732816800000 maxt=1732838400000 ulid=01JDTXS5K9E129BG097E40Q9GA time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732860000027 maxt=1732867200000 ulid=01JDVJC2PA57ZBBM69WZSSCVQ6 time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732867200009 maxt=1732874400000 ulid=01JDVS7SEQT963ZHGENGHK45YS time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732838400037 maxt=1732860000000 ulid=01JDVS81W1Y3E66A1WFNMD262E time=2024-11-29T14:13:34.727Z level=INFO source=repair.go:54 msg="Found healthy block" component=tsdb mint=1732874400004 maxt=1732881600000 ulid=01JDW03H6ACXGT2R3CFWGQTKWH time=2024-11-29T14:13:34.728Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=:9090 time=2024-11-29T14:13:34.728Z level=INFO source=tls_config.go:350 msg="TLS is disabled." component=web http2=false address=:9090 time=2024-11-29T14:13:34.856Z level=INFO source=head.go:628 msg="Replaying on-disk memory mappable chunks if any" component=tsdb time=2024-11-29T14:13:35.256Z level=INFO source=head.go:715 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=400.662001ms time=2024-11-29T14:13:35.256Z level=INFO source=head.go:723 msg="Replaying WAL, this may take a while" component=tsdb time=2024-11-29T14:13:35.852Z level=INFO source=head.go:760 msg="WAL checkpoint loaded" component=tsdb time=2024-11-29T14:13:37.019Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58546 maxSegment=58555 time=2024-11-29T14:13:39.026Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58547 maxSegment=58555 time=2024-11-29T14:13:40.635Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58548 maxSegment=58555 time=2024-11-29T14:13:40.777Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58549 maxSegment=58555 time=2024-11-29T14:13:41.886Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58550 maxSegment=58555 time=2024-11-29T14:13:43.025Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58551 maxSegment=58555 time=2024-11-29T14:13:43.593Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58552 maxSegment=58555 time=2024-11-29T14:13:45.292Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58553 maxSegment=58555 time=2024-11-29T14:13:45.755Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58554 maxSegment=58555 time=2024-11-29T14:13:45.756Z level=INFO source=head.go:795 msg="WAL segment loaded" component=tsdb segment=58555 maxSegment=58555 time=2024-11-29T14:13:45.756Z level=INFO source=head.go:832 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=595.854405ms wal_replay_duration=9.903472583s wbl_replay_duration=411ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=400.662001ms total_replay_duration=10.900037991s time=2024-11-29T14:13:45.972Z level=INFO source=main.go:1260 msg="filesystem information" fs_type=EXT4_SUPER_MAGIC time=2024-11-29T14:13:45.972Z level=INFO source=main.go:1263 msg="TSDB started" time=2024-11-29T14:13:45.972Z level=INFO source=main.go:1446 msg="Loading configuration file" filename=/etc/config/prometheus.yml time=2024-11-29T14:13:45.974Z level=INFO source=kubernetes.go:327 msg="Using pod service account via in-cluster config" component="discovery manager scrape" discovery=kubernetes config=prometheus-pushgateway time=2024-11-29T14:13:45.979Z level=INFO source=kubernetes.go:327 msg="Using pod service account via in-cluster config" component="discovery manager scrape" discovery=kubernetes config=kubernetes-pods time=2024-11-29T14:13:45.979Z level=INFO source=kubernetes.go:327 msg="Using pod service account via in-cluster config" component="discovery manager scrape" discovery=kubernetes config=kubernetes-nodes time=2024-11-29T14:13:45.979Z level=INFO source=kubernetes.go:327 msg="Using pod service account via in-cluster config" component="discovery manager scrape" discovery=kubernetes config=kubernetes-service-endpoints time=2024-11-29T14:13:45.980Z level=INFO source=main.go:1485 msg="updated GOGC" old=100 new=75 time=2024-11-29T14:13:45.980Z level=INFO source=main.go:1495 msg="Completed loading of configuration file" db_storage=1.142µs remote_storage=2.645µs web_handler=521ns query_engine=1.884µs scrape=316.788µs scrape_sd=5.376961ms notify=2.574µs notify_sd=1.142µs rules=537.505µs tracing=6.533µs filename=/etc/config/prometheus.yml totalDuration=8.387554ms time=2024-11-29T14:13:45.980Z level=INFO source=main.go:1224 msg="Server is ready to receive web requests." time=2024-11-29T14:13:45.980Z level=INFO source=manager.go:168 msg="Starting rule manager..." component="rule manager" time=2024-11-29T14:13:51.881Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.5.178:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:52.062Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.216:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:52.419Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.220:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:52.576Z level=WARN source=scrape.go:1880 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.112:9650/ext/metrics num_dropped=8 time=2024-11-29T14:13:53.946Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.231:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:54.335Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.3.218:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:55.214Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.217:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:55.693Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.135:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:56.021Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.202:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:56.714Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.233:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:57.118Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.219:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:57.380Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.199:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:58.340Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.218:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:58.406Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.214:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:59.115Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.230:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:59.382Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.3.228:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:13:59.638Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.3.220:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:00.347Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.113:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:00.768Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.232:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:01.315Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.225:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:02.013Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.200:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:02.676Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.223:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:02.730Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.224:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:02.901Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.205:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:02.938Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.203:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:03.434Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.142:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:03.477Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.5.173:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:03.781Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.216:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:03.951Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.92:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:04.348Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.134:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:04.605Z level=WARN source=scrape.go:1880 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.111:9650/ext/metrics num_dropped=8 time=2024-11-29T14:14:05.783Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.227:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:06.881Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.5.178:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:07.037Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.216:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:07.398Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.220:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:07.542Z level=WARN source=scrape.go:1880 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.112:9650/ext/metrics num_dropped=8 time=2024-11-29T14:14:08.922Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.231:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:09.310Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.3.218:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:10.190Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.217:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:10.694Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.135:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:10.996Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.202:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:11.690Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.233:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:12.092Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.219:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:12.356Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.199:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:13.316Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.218:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:13.381Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.214:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:14.090Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.230:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:14.358Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.3.228:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:14.614Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.3.220:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:15.355Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.4.113:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:15.742Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.6.232:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:16.288Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.9.225:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" time=2024-11-29T14:14:16.988Z level=ERROR source=scrape.go:1585 msg="Failed to determine correct type of scrape target." component="scrape manager" scrape_pool=kubernetes-pods target=http://11.12.7.200:8081/metrics content_type="" fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target" ``` ```

prometheus的bug。

5月26日才彻底修好。

这个issue里面的处理方法和版本应该更新到那个版本我研究了一下没搞懂，这块你可以自己研究一下。但是这个回复说，应该在http头里面加

Content-Type: text/plain; version=0.0.4

我看明白了。

实在不行你给2.27.1的exporter都加个反向代理，代理回传的时候加这个http头，应该就能解决兼容性问题。