Kubernetes Persistent Volume is not working on GCE

I am trying to make my elastic search pods persistent so that data is preserved when deployment or pods are recreated.Elastic search is a part of Graylog2 setup.

After I set everything up, I sent a few logs to Graylog and I could see them appear on the dashboard. However, I deleted elasticsearch pod and after it was recreated all the data was lost on Graylog dashboard.

  • specify max log json file size in docker compose
  • git clone project from inside docker container
  • backports/lzma/_lzmamodule.c:115:18: fatal error: lzma.h: No such file or directory
  • apachectl -DFOREGROUND exits immediately
  • How do I integrate my git PHP repo into an official Docker Hub container?
  • Does file touch create a copy in the docker container layer?
  • I am using GCE.

    Here is my persistent volume config:

    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: elastic-pv
      labels:
        type: gcePD
    spec:
      capacity:
        storage: 200Gi
      accessModes:
        - ReadWriteOnce
      gcePersistentDisk:
        fsType: ext4
        pdName: elastic-pv-disk
    

    Persistent volume claim config:

    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: elastic-pvc
      labels:
        type: gcePD
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 200Gi
    

    and here is my elasticsearch deployment:

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: elastic-deployment
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            type: elasticsearch
        spec:
          containers:
          - name: elastic-container
            image: gcr.io/project/myelasticsearch:v1
            imagePullPolicy: Always
            ports:
            - containerPort: 9300
              name: first-port
              protocol: TCP
            - containerPort: 9200
              name: second-port
              protocol: TCP
            volumeMounts:
                - name: elastic-pd
                  mountPath: /data/db
          volumes:
          - name: elastic-pd
            persistentVolumeClaim:
              claimName: elastic-pvc
    

    Output of kubectl describe pod:

    Name:       elastic-deployment-1423685295-jt6x5
    Namespace:  default
    Node:       gke-sd-logger-default-pool-2b3affc0-299k/10.128.0.6
    Start Time: Tue, 09 May 2017 22:59:59 +0500
    Labels:     pod-template-hash=1423685295
            type=elasticsearch
    Status:     Running
    IP:     10.12.0.11
    Controllers:    ReplicaSet/elastic-deployment-1423685295
    Containers:
      elastic-container:
        Container ID:   docker://8774c747e2a56363f657a583bf5c2234ed2cff64dc21b6319fc53fdc5c1a6b2b
        Image:      gcr.io/thematic-flash-786/myelasticsearch:v1
        Image ID:       docker://sha256:7c25be62dbad39c07c413888e275ae419a66070d37e0d98bf5008e15d7720eec
        Ports:      9300/TCP, 9200/TCP
        Requests:
          cpu:      100m
        State:      Running
          Started:      Tue, 09 May 2017 23:02:11 +0500
        Ready:      True
        Restart Count:  0
        Volume Mounts:
          /data/db from elastic-pd (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from default-token-qtdbb (ro)
        Environment Variables:  <none>
    Conditions:
      Type      Status
      Initialized   True
      Ready     True
      PodScheduled  True
    Volumes:
      elastic-pd:
        Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  elastic-pvc
        ReadOnly:   false
      default-token-qtdbb:
        Type:   Secret (a volume populated by a Secret)
        SecretName: default-token-qtdbb
    QoS Class:  Burstable
    Tolerations:    <none>
    No events.
    

    Output of kubectl describe pv:

    Name:       elastic-pv
    Labels:     type=gcePD
    StorageClass:
    Status:     Bound
    Claim:      default/elastic-pvc
    Reclaim Policy: Retain
    Access Modes:   RWO
    Capacity:   200Gi
    Message:
    Source:
        Type:   GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
        PDName: elastic-pv-disk
        FSType: ext4
        Partition:  0
        ReadOnly:   false
    No events.
    

    Output of kubectl describe pvc:

    Name:       elastic-pvc
    Namespace:  default
    StorageClass:
    Status:     Bound
    Volume:     elastic-pv
    Labels:     type=gcePD
    Capacity:   200Gi
    Access Modes:   RWO
    No events.
    

    Confirmation that real disk exists:

    enter image description here

    What could be the reason Persistent Volume is not persistent?

  • Spring Boot in Docker
  • How to get the docker containers information with Ruby and monitor it
  • Docker exporting container
  • Setup global environnement variables for all Docker containers
  • What are the advantages of docker containers over statically linked EXEs?
  • How to connect mysql in a rails appliation with docker?
  • One Solution collect form web for “Kubernetes Persistent Volume is not working on GCE”

    In the official images, the Elasticsearch data is stored at /usr/share/elasticsearch/data and not /data/db. It would appear that you needed to updated the mount to be /usr/share/elasticsearch/data instead to get the data storing on the persistent volume.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.