blktrace output error in docker container

my docker container base image is ubuntu, and I ran it with full privilege options that means in run command I use these switches:

--cap-add=SYS_ADMIN --security-opt apparmor:unconfined

I wanna use blktrce using below command:

  • Running docker as non-root user OR running jenkins on tomcat as root user
  • Simple PHP-CLI Cron Supervisor Docker not working
  • Kubernetes - Creating a specific namespace for “services”
  • Kubernetes can't start due to too many open files in system
  • How to setup a quick deployment process?
  • Using custom shell script as docker entrypoint
  • sudo blktrace -d / -a issue -o - | blkparse -f "%p %T.%9t %D %S ^C %d\n" -i - >stream.out
    

    but, first time when is use this command I get this error:

    Debugfs is not mounted at /sys/kernel/debug
    

    i searched and found this solution which led to use this command:

    mount -t debugfs none /sys/kernel/debug
    

    after that, when I use the blktrace command again, i get this error:

    BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device
    Thread 1 failed open /sys/kernel/debug/block/(null)/trace1: 2/No such file or directory
    Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory
    Thread 2 failed open /sys/kernel/debug/block/(null)/trace2: 2/No such file or directory
    Thread 4 failed open /sys/kernel/debug/block/(null)/trace4: 2/No such file or directory
    Thread 3 failed open /sys/kernel/debug/block/(null)/trace3: 2/No such file or directory
    Thread 5 failed open /sys/kernel/debug/block/(null)/trace5: 2/No such file or directory
    Thread 7 failed open /sys/kernel/debug/block/(null)/trace7: 2/No such file or directory
    Thread 8 failed open /sys/kernel/debug/block/(null)/trace8: 2/No such file or directory
    Thread 6 failed open /sys/kernel/debug/block/(null)/trace6: 2/No such file or directory
    Thread 12 failed open /sys/kernel/debug/block/(null)/trace12: 2/No such file or directory
    Thread 10 failed open /sys/kernel/debug/block/(null)/trace10: 2/No such file or directory
    Thread 13 failed open /sys/kernel/debug/block/(null)/trace13: 2/No such file or directory
    Thread 15 failed open /sys/kernel/debug/block/(null)/trace15: 2/No such file or directory
    Thread 14 failed open /sys/kernel/debug/block/(null)/trace14: 2/No such file or directory
    Thread 17 failed open /sys/kernel/debug/block/(null)/trace17: 2/No such file or directory
    Thread 16 failed open /sys/kernel/debug/block/(null)/trace16: 2/No such file or directory
    Thread 18 failed open /sys/kernel/debug/block/(null)/trace18: 2/No such file or directory
    Thread 11 failed open /sys/kernel/debug/block/(null)/trace11: 2/No such file or directory
    Thread 19 failed open /sys/kernel/debug/block/(null)/trace19: 2/No such file or directory
    Thread 20 failed open /sys/kernel/debug/block/(null)/trace20: 2/No such file or directory
    Thread 9 failed open /sys/kernel/debug/block/(null)/trace9: 2/No such file or directory
    Thread 21 failed open /sys/kernel/debug/block/(null)/trace21: 2/No such file or directory
    Thread 22 failed open /sys/kernel/debug/block/(null)/trace22: 2/No such file or directory
    Thread 23 failed open /sys/kernel/debug/block/(null)/trace23: 2/No such file or directory
    FAILED to start thread on CPU 0: 1/Operation not permitted
    FAILED to start thread on CPU 1: 1/Operation not permitted
    FAILED to start thread on CPU 2: 1/Operation not permitted
    FAILED to start thread on CPU 3: 1/Operation not permitted
    FAILED to start thread on CPU 4: 1/Operation not permitted
    FAILED to start thread on CPU 5: 1/Operation not permitted
    FAILED to start thread on CPU 6: 1/Operation not permitted
    FAILED to start thread on CPU 7: 1/Operation not permitted
    FAILED to start thread on CPU 8: 1/Operation not permitted
    FAILED to start thread on CPU 9: 1/Operation not permitted
    FAILED to start thread on CPU 10: 1/Operation not permitted
    FAILED to start thread on CPU 11: 1/Operation not permitted
    FAILED to start thread on CPU 12: 1/Operation not permitted
    FAILED to start thread on CPU 13: 1/Operation not permitted
    FAILED to start thread on CPU 14: 1/Operation not permitted
    FAILED to start thread on CPU 15: 1/Operation not permitted
    FAILED to start thread on CPU 16: 1/Operation not permitted
    FAILED to start thread on CPU 17: 1/Operation not permitted
    FAILED to start thread on CPU 18: 1/Operation not permitted
    FAILED to start thread on CPU 19: 1/Operation not permitted
    FAILED to start thread on CPU 20: 1/Operation not permitted
    FAILED to start thread on CPU 21: 1/Operation not permitted
    FAILED to start thread on CPU 22: 1/Operation not permitted
    FAILED to start thread on CPU 23: 1/Operation not permitted    
    

    how can i solve that?

    Update1:

    there is sda folder in /sys/kernel/debug/block/ and into this folder there are these files:

    trace0 trace1 trace2 etc.
    

    Update2:

    @abligh thank for your answer, but did not help. The strace output about ioctl is:

    ioctl(3, BLKTRACESETUP, {act_mask=64, buf_size=524288, buf_nr=4, start_lba=0, end_lba=0, pid=0}, 0x7ffe8a4ceac0) = -1 ENOTTY (Inappropriate ioctl for device)
    write(2, "BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device\n", 61) = 61
    ioctl(3, BLKTRACESTOP, 0x7f6fd19789d0) = -1 ENOTTY (Inappropriate ioctl for device)
    ioctl(3, BLKTRACESTOP, 0x7ffe8a4ce540) = -1 ENOTTY (Inappropriate ioctl for device)
    ioctl(3, BLKTRACESTOP, 0x7f6fd19789e0) = -1 ENOTTY (Inappropriate ioctl for device)`
    ioctl(3, BLKTRACETEARDOWN, 0x7f6fd19789e0) = -1 ENOTTY (Inappropriate ioctl for device)
    

    to answer your question about why i run blktrace in container, i would say I’m using containers as cluster, so i need the trace of each node.

  • The best practices for PostgreSQL docker container initialization with some data
  • Docker file for java project
  • Cannot launch interactive session in Windows IIS Docker container
  • Bluemix IBM Container with Mongodb connection failed
  • SBT Native Packager Plugin Docker Error
  • Have sshd forward logins of git user to a (GitLab) Docker container
  • One Solution collect form web for “blktrace output error in docker container”

    I don’t have enough information to debug this, but you asked for an answer to be posted.

    The root of the problem seems to be here:

    BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device
    

    This is indicating that the ioctl call to set up block tracing is failing. It is failing inside the container, but according to the comments works outside the container. This would indicate that the problem is with the container setup or a limitation in the kernel that prevents that ioctl from being used in a container at all.

    The error inappropriate ioctl for device is errno 25, i.e. ENOTTY. I can’t immediately see what would be returning that unless it can’t find the device node at all (given you’ve already demonstrated from the fact it works outside the container that the block trace code is compiled in). I can’t remember whether this is in a module, but it would be worth trying block tracing outside the container first (then check it inside the container), just to check this isn’t a module loading issue.

    The first step in debugging this would be to use strace tool (as suggested above) so you know exactly which system call is being made with what parameters. EG run:

    sudo strace -f -s2048 -o/tmp/trace blktrace -d / -a issue -o - | blkparse -f "%p %T.%9t %D %S ^C %d\n" -i - >stream.out
    

    and look at /tmp/trace afterwards. strace will list all the system calls made. See if you can determine from that which ioctl is failing.

    Secondly, I’d ensure that the block device that you are trying to trace actually appears within the correct place(s) in /proc/ and /sys/. Something wrong is happening here:

    Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory
    

    Note the (null) in the debug line, which clearly should not be there – that should be the name of the block device. This is possibly a consequence of the failed ioctl, or possibly indicative of a problem within the /sys/ hierarchy.

    BLKTRACESETUP is handled in the kernel here. This eventually calls doblktracesetup here. I cannot immediately see any reason why this would not work from an appropriately permissioned container, which makes me suspect your /sys hierarchy might not be set up right. But the output of strace would be helpful.

    Also, the inevitable comment: why are you doing this in a container? Why not run it outside the container?

    EDIT: Looks like it might be a kernel thing. See https://github.com/scaleway/kernel-tools/issues/107 – this suggests you (a) need to modprobe the specific modules first, and (b) may need a specific kernel.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.