Skip to content

Incorrect Disk usage is reported when using Ceph erasure code pool #13718

Open
Bug
Open
Incorrect Disk usage is reported when using Ceph erasure code pool#13718
Bug
Assignees
wido
Labels
component:cephcomponent:managed-storagecomponent:storagestatus:needs-functional-definition
Milestone
unplanned

Description

@kiranchavala
kiranchavala
opened on Jul 27, 2026

problem

Incorrect Disk usage is reported when a Ceph storage

versions

ACS 4.22.1
ceph version
ceph version 20.2.2 (0fcffee29411e3a38036764817b6e1afc59741cc) tentacle (stable - RelWithDebInfo)
Ubuntu 24.0.4 kvm host

The steps to reproduce the bug

CloudStack added support for ceph erasure code pool in the 4.21 release

github.com/apache/cloudstack/pull/9808

Steps to reproduce the issue

  1. Create a ceph-storage with Erasure code setting
ceph osd erasure-code-profile set cloudstack-ec k=2 m=1 crush-failure-domain=host
ceph osd pool create cloudstack-ec 64 erasure cloudstack-ec
ceph osd pool set cloudstack-ec allow_ec_overwrites true
ceph osd pool application enable cloudstack-ec rbd

ceph osd pool create cloudstack-ec-metadata 32 replicated
ceph osd pool set cloudstack-ec-metadata size 3
rbd pool init cloudstack-ec-metadata
  1. Add the storage pool in cloudstack
createStoragePool scope=zone zoneid=<ZONE_ID> hypervisor=KVM \
  name=MyCephECCluster provider=DefaultPrimary \
  url=rbd://cloudstack:<RADOS_SECRET>@192.168.55.11/cloudstack-ec-metadata \
  details[0].key=dataPool details[0].value=cloudstack-ec \
  tags=RBD-EC
  1. Check the disk usage from ceph side

root@ceph1:/home/ubuntu# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    150 GiB  150 GiB  200 MiB   200 MiB       0.13
TOTAL  150 GiB  150 GiB  200 MiB   200 MiB       0.13

--- POOLS ---
POOL                    ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL

cloudstack-ec            3   64    8 KiB        1   12 KiB      0     95 GiB
cloudstack-ec-metadata   4   32  7.9 KiB        3   36 KiB      0     47 GiB
  1. Check the Disk used size from Cloudstack
Image

  1. Deploy a data disk of size lets say 5gb on the ceph storage pool

  2. Attach it to a running vm

  3. Populate some data on the data disk

dd if=/dev/urandom of=/dev/vdb bs=1M count=2048 status=progress oflag=direct

  1. Check the ceph disk usage
root@ceph1:/home/ubuntu# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    150 GiB  147 GiB  3.2 GiB   3.2 GiB       2.13
TOTAL  150 GiB  147 GiB  3.2 GiB   3.2 GiB       2.13

--- POOLS ---
POOL                    ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr                     1    1  577 KiB        2  1.7 MiB      0     46 GiB
cloudstack               2   64  6.8 KiB        3   32 KiB      0     46 GiB
cloudstack-ec            3   64  2.0 GiB      513  3.0 GiB   2.11     93 GiB
cloudstack-ec-metadata   4   32  9.1 KiB        6   62 KiB      0     46 GiB
  1. Check the cloudstack storage used
Image

Cloudstack is reporting the used size from the replicated metadata pool and not the actual erasure code pool

What to do about it?

CloudStack should report the correct disk size used from the erasure -code pool and not the replicated metadata stats

Activity

  1. kiranchavala
    added
    component:managed-storage
    component:storage
    bug
    on Jul 27, 2026
  2. kiranchavala
    added the
    Bug
    issue type on Jul 27, 2026
  3. kiranchavala
    changed the title [-]Incorrect Disk usage is reported when a Ceph storage[/-] [+]Incorrect Disk usage is reported when using Ceph erasure code pool[/+] on Jul 27, 2026

  4. kiranchavala commented on Jul 27, 2026

    @kiranchavala
    kiranchavala
    on Jul 27, 2026
    MemberAuthor

    Based on the discussion in

    #5500

    CloudStack gets the used bytes information from libvirt.

    root@Cloudstack-Kvm-Host3:/home/ubuntu# virsh pool-info 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
Name:           15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
UUID:           15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
State:          running
Persistent:     no
Autostart:      no
Capacity:       149.99 GiB
Allocation:     62.89 KiB
Available:      146.79 GiB

  5. wido commented on Jul 27, 2026

    @wido
    wido
    on Jul 27, 2026
    Contributor

    Based on the discussion in

    #5500

    CloudStack gets the used bytes information from libvirt.

    root@Cloudstack-Kvm-Host3:/home/ubuntu# virsh pool-info 15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
Name:           15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
UUID:           15e9a03c-6eb1-3d3c-8cac-5918f0c75f0f
State:          running
Persistent:     no
Autostart:      no
Capacity:       149.99 GiB
Allocation:     62.89 KiB
Available:      146.79 GiB

    That is correct. CloudStack leverages Libvirt for this. If this is broken or wrong in Libvirt, it has to be fixed there. There isn't much CloudStack can do here.

    I've set the priority to low as this is outside CloudStack's scope and assigned it to myself.

    Will probably close this one soon(ish).

  6. wido
    self-assigned this
    on Jul 27, 2026

  7. kiranchavala commented on Jul 27, 2026

    @kiranchavala
    kiranchavala
    on Jul 27, 2026
    MemberAuthor

    @wido Wondering if we should consider it as an enhancement

    Bypassing libvirt entirely and querying Ceph directly via restful api to get the usage stats of a erasure code pool or a ceph storage pool

    curl -k -u admin: https://<>:8443/api/pool

    The usage reporting in cloudstack could also get affected because of querying the data from libvirt

  8. wido commented on Jul 27, 2026

    @wido
    wido
    on Jul 27, 2026
    Contributor

    @wido Wondering if we should consider it as an enhancement

    Bypassing libvirt entirely and querying Ceph directly via restful api to get the usage stats of a erasure code pool or a ceph storage pool

    curl -k -u admin: https://<>:8443/api/pool

    The usage reporting in cloudstack could also get affected because of querying the data from libvirt

    I doubt if that's a good idea. Technically, yes, it could be done, but I wonder if we can get this to work. Ceph is implemented in the DefaultPrimartStorageDriver and this relies on Libvirt. If we want to do this cleanly we should write a separate Ceph driver for CloudStack, a plugin.

    Maybe Libvirt can be improved, getting a patch into Libvirt might be better and more future proof.

  9. DaanHoogland commented on Jul 28, 2026

    @DaanHoogland
    DaanHoogland
    on Jul 28, 2026
    Contributor

    this sounds like we want storage extensions now ;)

  10. DaanHoogland
    added this to the unplanned milestone on Jul 28, 2026
  11. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Jul 28, 2026
  12. DaanHoogland
    moved this to Discuss in Apache CloudStack BugFest - Issueson Jul 28, 2026
  13. DaanHoogland
    added
    status:needs-functional-definition
    and removed
    bug
    on Jul 28, 2026

  14. wido commented on Jul 28, 2026

    @wido
    wido
    on Jul 28, 2026
    Contributor

    Maybe @DaanHoogland ! But at this moment I don't see this being changed. Somebody has to invest time into this to get it fixed.

  15. weizhouapache
    added
    component:ceph
    on Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:cephcomponent:managed-storagecomponent:storagestatus:needs-functional-definition

Type

Bug

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions